Shared jobs

Shared jobs allow you to bundle entire workflows into a single custom component and then use those custom components anywhere else in the project. Only orchestration jobs (and the transformation jobs they link to) can be shared in this manner. If a job calls another job via Run Orchestration or Run Transformation components, then all jobs will be included in the shared job. To this end, a shared job can include as many jobs as the user wishes.

Within a shared job, the Run Orchestration component can't run an orchestration job that hasn't originally been packaged inside the shared job. If you try this, the following error will occur:

{{Parameter Validation Failure:
Job Name - Could not find job with name [<JobName>]}}

Therefore, all orchestration jobs required by the workflow should be included in the shared job.

Creating a shared job

To create a shared job, right-click on any job in the project's job list, and click Generate Shared Job. The Generate Shared Job wizard will open, where you can configure your shared job. The next section gives information about how to complete the wizard.

Note

The job that you right-click on is the "root" job that the new shared job relates to.

Shared job configuration

The Generate Shared Job wizard contains three pages of configuration settings you are required to complete. On the first page, complete the following fields:

New Shared Job: Select either Create New to create an entirely new shared job, or Copy From Existing to create a shared job based on an already existing job.
Existing: If you selected Copy From Existing, you need to select an existing shared job on which to base the new shared job. Select the job from the drop-down list.
Package: The package name for the new shared job. Nested packages can be specified using a period (dot) to separate package names. For example, toplevel.nesteda.nestedb specifies that the package nestedb is inside package nesteda, which is inside package toplevel.
Name: A name for the new shared job. Names may contain letters, numbers, underscores, single spaces, parentheses, and hyphens. Jobs can have identical names if they're within different parts of the shared jobs resource tree.
Revision: The revision number of the shared job. When first creating the shared job, the default revision is 1.
Icon: Click Browse... to locate and select an icon for this shared job. The icon must be a .png file. Matillion ETL provides a default icon if you don't select one here.
Description: A meta description of the shared job. This description will be visible in the component's Help tab. Simple HTML can be used for formatting.

When you have completed these fields, click Next to move to the second page of the dialog and complete these fields:

Root Job: The root job is the job that the shared job originates from. By default, the job that was right-clicked to open the Generate Shared Job dialog is set as the root job. You can change this by selecting a different job from the drop-down menu. There can only be one root job.
Additional Jobs: Jobs listed here will be packaged within the shared job. This list should include any jobs that the root job is expected to call. Any jobs called using the Run Orchestration or Run Transformation components within the root job are added to this list by default. Jobs that may be called dynamically from the root job should also be included but must be added manually.

There are buttons at the bottom of the dialog associated with additional jobs:

Click + to open the Add Job dialog, and select from one of the existing jobs in the drop-down menu.
Click Clear to remove a job from the list.
Click Auto to populate the list with explicitly called jobs.

A warning may appear at the bottom of the wizard, called Contained Shared Jobs will not be bundled. This warning applies to the process of exporting shared jobs. When you export a shared job, the jobs listed in this warning won't be exported along with it. You should therefore export all referenced jobs separately, to individual .melt files. These exported files should all be shipped and imported along with the main job. For a full explanation of how shared jobs are exported to .melt files, read Manage Shared Jobs.

When you have completed these fields, click Next to move to the third page of the wizard, Parameter Configuration:

When you use a shared job, it includes a set of configurable parameters that are dictated by the root job's job variables. These job variables populate the Parameter Configuration list, and include the following properties:

Name: The name of this job variable as it exists in the root job.
Display Name: The name of the parameter created from this job variable. You can edit this to give the parameter a more meaningful or user-friendly name.
Required: This setting can be enabled or disabled for each parameter. All parameters are required by default. When a shared job is created and used, all required parameters must have a value set for the shared job to validate successfully, but parameters not marked as "required" can remain empty. For shared jobs that pre-date this feature, all parameters are assumed to be required.

When you have completed the Generate Shared Job configuration wizard, click OK to create the shared job. Click Back to return to the previous dialog pages. Click Cancel to terminate the shared job setup.

Using shared jobs

Created shared jobs are available from the Shared Jobs panel in the lower-left of your Matillion ETL UI. Right-click a shared job to open a context menu with the following options:

Open Shared Job: Opens the shared job so that you can navigate through the jobs and components contained within. This is a read-only view of the shared job.
Unpack: Unpacks the constituent jobs that were included when the shared job was generated. This shows you what orchestration and transformation jobs are included in the shared job. You can edit these as required, and then regenerate the shared job.
Export: Opens the Export Shared Jobs dialog, which is described in Manage Shared Jobs.
View jobs using this shared job: Opens a tab, and lists any existing jobs where the shared job is being used. If the shared job is currently not being used by an existing job, then "[SharedJobName] is not used in any jobs" will be displayed.

Exporting variables

The variables used inside shared jobs can be exported:

Create one or more job variables. To do this, right-click the component in the job canvas, and click Manage Job Variables. For more information, read Job Variables.
Create the shared job. You may want to include these variables as parameters when you configure the job.
Create a set of Job Variables in the intended job that will be included in the newly created shared job. You will be using these to store the exported variables, so ensure they're of the expected data types.
Click on the shared job in the job canvas, select the Export tab, and then click Edit.
Take the variables from the shared job in the Source column, and map them to variables from the current (calling) job in the Target Variable column.
Click OK.

Read Component Exports for more information on the export process.

Parallel transformation jobs

It's recommended to use Shared jobs as a means of running parallel transformation jobs on HA clusters. When multiple instances of the same transformation job are running at the same time on an HA Cluster, the values passed to those transformation jobs will not reflect those used by the job and will result in unexpected behaviour. For more information, read Designing a Job for a High Availability Cluster.

Shared jobs are unique entities on each execution and a shared job can be run many times in parallel on an HA Cluster without issue.

It's advised to package any transformation jobs as shared jobs when wanting to run them in parallel (or they are at risk of being run in parallel) in an HA cluster.

In this example we'll demonstrate how you can package a transformation job as a shared job in your Matillion ETL instance. To package a transformation job, we must include an orchestration job in our shared job.

Here we have a Matillion ETL orchestration job that uses the Jira Query component to load all the data from the Epics data source. We want to display this data in a transformation job, which we'll create now, so we can reference its name in our orchestration job, using the Run Transformation component.

Orchestration Job

In the transformation job we've just created, we'll use a Job Variable to store the name of the target table we created earlier in our orchestration job. We'll use the Table Input component to read columns from the target table, by using the table_name job variable. We'll apply a Filter to the "Jira results" table that sets conditions to only show Epics that have been completed. We'll now use the Table Output component to store the filtered data in the original table by using the table_name variable again, as the target table.

Transformation Job

We'll now package this job into a shared Job. To do this, refer to Creating a shared job.

Begin by right-clicking on the orchestration job in the jobs panel, and select Generate Shared Job. In the Generate Shared Job wizard, we'll enter details as below:

Create Shared Job

In the next step of the wizard select the job that this shared job is based on. This will be the name of our orchestration job. The page will also display the transformation job in the Additional Jobs section. In other words, any job in addition to the Root Job that is packaged within our shared job. In the final step of the wizard you'll see the Parameter Configuration page. This will be blank. We didn't add any job variables to the orchestration job (root job). Any variables that are used in this job type will be displayed here. Click OK to generate the shared job.

The "Epics" shared job will appear in the Shared Jobs panel in our Matillion ETL instance. Drag the shared job on to the job canvas.

Shared Job on Canvas

For this example we want to open our shared job in a read-only format. Right-click it, and click Open Shared Job to view our orchestration and transformation jobs:

Open Shared Job