Skip to content

UI and basic functions

This is a quick introduction to the user interface (UI) and basic functionality of Matillion ETL. While there are a wide array of functions and options available, the goal of this guide is to get new users started and running simple tasks in minutes. It will discuss the basics of Matillion ETL, explain common terminology, and show how to use general interface features.

Note

  • Before beginning, a working instance of Matillion ETL will need to be running. Read Launching Matillion ETL overview if help is required setting up an instance.
  • While some features may differ between cloud data warehouses, functionality of the Matillion ETL instance remains the same.

Interface features

Matillion ETL UI

The majority of Matillion ETL features can be accessed using the following system features:

  1. Project menu
  2. Navigation panel
  3. Job canvas
  4. Components, Shared Jobs, and Environments panels
  5. Properties panel
  6. Tasks panel
  7. Admin menu
  8. Help menu

Project menu

The Project drop-down menu provides a list of options regarding the current project. These options include:

Project

  • Switch Project: Browse to or create a project. Jobs inside other projects can also be opened from here, automatically switching to their parent project.
  • Manage Project: Set a name, description, and permissions for sharing the current project. Read Project user access for more details.
  • Rename Project Group: Change the name of the group associated with the current project.
  • Delete Project: Delete the current project.
  • Input Data Report: Generate an Input data report on all jobs run in the current session within the current project.

Version

  • Manage Versions: Create, delete, and switch versions, or change the details of the current version. Default and locked versions can't be renamed, and current versions can only be renamed if more than one version is available. Read Manage versions for more details.
  • Recycle Bin: Delete the current version. Default and locked versions can't be deleted.

Environment

  • Add Environment: Add an environment. Read Environments for more details.
  • Manage Environment Variables: Define variables for the current environment. Read Environment variables for more details.

Data sharing

  • Export: Export selected jobs, environments, and variables for use in other projects. Exported as a .json file. Read Import - Export for more details.
  • Import: Import .json files containing jobs, environments, and variables for use in the current project. Read Import - Export for more details.

Administration and authentication

  • Manage Credentials: Configure credentials to be used to integrate with the Matillion ETL instance environment. Read Manage credentials for more details.
  • Manage Schedules: Schedule Matillion ETL jobs. Read Manage schedules for more details.
  • Task History: Show a detailed list of successfully and unsuccessfully completed tasks. Read Task management for more details.
  • Manage OAuth: Configure OAuth for various APIs from which Matillion ETL will source data. Read Manage OAuth for more details.
  • Manage Project Group Passwords: Configure username-password pairs to be stored for use in different parts of Matillion ETL. Read Manage passwords for more details.
  • Manage Query Profiles: Configure query profiles to describe a JSON-based or XML-based API for importing data into tables. Read Matillion ETL API and Manage query profiles for more details.
  • Manage Extract Profiles: Configure extract profiles to describe JSON- or XML-based API for importing data into tables. Read Matillion ETL API and Manage extract profiles for more details.
  • Manage Webhook Payloads: Configure new and edit existing webhook payloads. Read Manage webhook payloads for more details.
  • Manage Shared Jobs: Bundle workflows into a single custom component and use those components elsewhere in the project. Read Managed shared jobs for more details.
  • Change My Password: Change your Matillion ETL password. Read Change my password for more details.
  • Manage Error Reporting: Set up Matillion ETL to report component or job errors to a webhook at a user-specified interval. Read Manage error reporting for more details.

Additional options

  • Manage SQS Configuration: Configure Matillion ETL with Amazon Simple Queue Service (SQS). Read Manage SQS configuration for more details.
  • Manage Change Data Capture: Synchronize cloud data warehouse data with the local database to ensure all data is continually and automatically kept up to date. Read Manage CDC for more details.
  • Manage Pub/Sub Configurations: Enable and configure Pub/Sub services when using a Matillion ETL as part of a larger process. Read Manage Pub-Sub configuration for more details.
  • Migrate: Send Matillion ETL resources from one Matillion ETL instance to another. Read Migrate for more details.
  • Git: Initialize a local Git repository or clone a non-Matillion repository to be integrated into the Matillion ETL instance. Read Git integration for more details.

Logout

Logout: Log the current user out of the current Matillion ETL instance.

Each Matillion ETL project is made up of jobs. All available jobs are displayed within this panel. There is no limit to the number of jobs that may be contained within a project. Double-clicking a job on the list will open it on the Job canvas. Alternatively, right-clicking anywhere within the panel will open a pop-up menu, which allows you to create, delete, import, and export jobs, as well as creating job folders.

There are two kinds of job:

  • Orchestration job: Generally loads and unloads data to tables and file storage, also known as staging the data.
  • Transformation job: Generally manipulates data in the form of tables.

Each job is made up of a series of tasks, each represented by a "component". Multiple components can be linked together to tackle more sophisticated tasks. Each type of job has its own set of components that can only be used within their respective job canvas. Orchestration jobs may also be linked to transformation jobs, allowing large pipelines to be planned and executed using a single job.

Job canvas

The job canvas shows the contents of a selected job. Each open job will be represented by a tab above the job canvas. Open jobs can be toggled between by clicking on these tabs, without affecting other open jobs.

When creating a new job, the job canvas will be empty and ready for the user to fill with components. These components can be found in the Components panel and dragged onto the job canvas.

These components will then need to be linked to one another in the order in which they will need to be executed. This can be done by clicking on a component icon on the job canvas to reveal the connection rings around the component. Clicking and dragging on a ring will extend a connection that can be affixed to another component. These rings represent four potential connection options:

  • Green/Success: Continue on to the next component if executed successfully.
  • Red/Failed: Continue on to the next component if failed to execute.
  • Grey/Unconditional: Continue on to the next component regardless of success or failure.
  • Blue/Iterate: Continue on to another component (regarding iterator components).

Note

  • These connection options allow users to direct the job according to whether the current run is successful or not.
  • All orchestration jobs must begin with a Start component and thus can't receive links from other components.

When using multiple components, a job canvas can quickly become quite cluttered or complex. To aid in understanding these jobs, you can add notes by right clicking anywhere on the job canvas, then clicking Create Note. Notes can be used to:

  • Add explanations of functions, components and workflows directly on the job canvas.
  • Create blocks to group multiple components together to make sense of large workflows.
  • Provide information for other users working on the same job.

Components panel

This is a collapsible list of all components available to the currently selected job type. Switching between an orchestration job and a transformation job will automatically update the available components, such that it's impossible to place a transformation component into an orchestration job or vice versa. A search field at the top of the panel allows you to search for specific components (matching components by name only).

Each component embodies a particular function—from moving data into file storage to performing custom calculations on table columns. Clicking and dragging a component onto the job canvas will create a component of that type in the current job. There is no limit to the number of components that can apply to a job.

Components are named according to their function, although it's common for a single component to have multiple functions that aren't strictly related to one another. Thus, it is recommended that users acquaint themselves with the available components such that they can make informed decisions on which components to use to best achieve a given goal.

Shared Jobs panel

Shared jobs allow users to bundle entire workflows into a single custom component that can then be used anywhere else in the project. These custom components and other pre-made shared jobs are listed in the Shared Jobs panel, and may be dragged onto the job canvas and used in the same way as a normal component. Read Shared Jobs for more details.

Environments panel

In Matillion ETL, an environment describes a single connection to a database, on which jobs can be run. Numerous environments can be set up at once, meaning different users can use different environments on a single Matillion ETL instance. All environments associated with the current instance are listed in this panel, along with the schemas, tables and columns contained within each environment. Read Environments for more details.

Properties panel

Clicking a component icon on the job canvas reveals detailed information about the component in the Properties panel below. Components generally share a similar set of information tabs, but with different contents in each. Common tabs include:

  • Properties: A unique set of user-defined properties that effect the function of the selected component—changes made to these properties will not affect the properties of any other component, even components of the same type. Read the specific component's documentation for more details.
  • Export: Export certain values from the selected component as a variable to be used elsewhere. Read Variables for more details.
  • Sample: Many components allow you to produce a sample output of data up to a specified number of rows—this can be used to confirm a component is set up correctly before using it in a live job.
  • Metadata: Display data for each column featured in the sample data, including column names, types, sizes, and scales. This tab is only available in transformation components.
  • Lineage: Track columns across transformations to understand how that column has been affected by components up to the point of inspection. This tab is only available in Enterprise Mode.
  • SQL: Display the SQL equivalent of the component and its current setup.
  • Plan: Get the plan from the SQL query generated by the job up to (and including) the selected component.
  • Help: Display a snippet of the help documentation for the selected component—including an explanation of its function and properties, as well as a link to the full documentation.

Tasks panel

The Tasks tabs show information about tasks that have been run during the current session. Right-clicking anywhere in the canvas, you have the option to run the current job as it currently stands. Right-clicking on a component will also allow you to run just that single component. In both cases, the run takes the form of a task that is documented in the Tasks tabs. For more information on tasks, read Task management. These tabs include:

  • Tasks: A list of up to the last 20 tasks attempted by the current user in the current project session—including queued, failed, or cancelled tasks. Read Task management for more details.
  • Search: Allows you to search the current project for matching components. Read Search tab for more details.
  • Console: The Matillion ETL instance console.
  • Command Log: An archive of interactions between the current browser session and the Matillion ETL server.
  • Notices: A list of notifications concerning the Matillion ETL project—including updates to Matillion ETL.

Admin menu

The Admin drop-down menu allows users with "Admin" privileges to manage the Matillion ETL instance. Read Admin menu for more details. The following options are available:

  • Download Server Log: Download the available Tomcat Catalina server logs of the Matillion ETL instance.
  • Matillion ETL Updates: Check for available updates for Matillion ETL. Read Updating Matillion ETL for more details.
  • User Configuration: Configure and manage user privileges and access rights. Read User configuration for more details.
  • SSL: Configure the Matillion ETL instance to listen for HTTP and HTTPS requests.
  • Restart Server: Restart the current Matillion ETL instance.
  • Manage Database Drivers: Upload JDBC drivers to use with the Database Query component. Read Manage database drivers for more details.
  • Manage Backups: Configure automatic backups and test backup schedules. Read Manage backups for more details.
  • Manage Groups: Create, edit, and remove user groups. Available in Enterprise Mode only.
  • Manage Permissions: Adjust the permission settings for user groups. Available in Enterprise Mode only.
  • Audit: View a log of all activity on the Matillion ETL instance. Read Audit log for more details.
  • Product Improvement Metrics: Choose to allow the collection of anonymous usage data to help Matillion build a better product. Read Product improvement metrics for more details and to see a list of what data is collected.

Help menu

The Help drop-down menu options provide useful information about Matillion ETL. This include:

  • Get Support: Open a new browser window with contact details and information on Matillion ETL support.
  • Support Information: Provides a template to send advanced enquiries to Matillion ETL support. Read Getting support for more details.
  • Manage Connections: Manage multiple connections to the same Matillion ETL instance to enable users to see what other users are doing within the instance. Read Manage connections for more details.
  • License: View and manage Matillion ETL licenses.
  • About: Display information about the current Matillion ETL version.

Note

If more than one user is currently logged into a project, these users will be represented by boxes alongside Admin and Help. Hovering over one of these boxes will display that user's current job status.

Running a job

Jobs are automatically saved with each change, so there is no need to manually save the progress—meaning they can be closed and reopened at any time to continue editing. At no stage will a job reach a "final" state where it can't be further iterated upon.

Jobs can be run at any stage by simply right-clicking on the job and clicking Run on the job canvas menu. Alternatively, jobs can be scheduled to run at a future time. Read Manage schedules for more details.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement