Pipeline Observability dashboard

The Pipeline Observability dashboard is a feature of the Hub that lists all Data Productivity Cloud pipelines, and any Matillion ETL instances currently running jobs. This list shows a real-time State of all instances, and is checked every 5 seconds, but you can click the refresh icon at the top of the page to ensure you're seeing the latest results.

Note

Your Matillion ETL instances must be of version 1.65 or later to work with Pipeline Observability.

Accessing the dashboard

There are two ways you can access the Pipeline Observability dashboard:

Log in to the Hub.
Click ☰ → Manage → Pipeline Runs.

Or:

Log in to the Hub.
The Home tab will be open. Click View pipeline runs.

Either of these methods will open the Data Productivity tab of the Your activity page.

Pipeline run history

The Data Productivity Cloud tab gives you a comprehensive view of your historic pipeline run details. You can view your pipeline run history information for the past 12 months, you can search for a pipeline name, and you can change time frames based on your preference.

Note

The record of a pipeline run is kept for one year; records within a pipeline, called Pipeline run details, are kept for 90 days. To access Pipeline run details, click on the name of your chosen pipeline. All pipeline names are displayed as a link.

For the Data Productivity Cloud, you can read information about the following:

Pipeline: The name of the pipeline that has been executed. Clicking this link will take you to a separate page, which will give you metadata about the pipeline, such as pipeline and individual component details, as well as any error logs.
Source: Where the pipeline was executed.
Project: The name of your project.
Status: An indicator if your pipeline is Running, Successful, or if it's Failed.
Environment: The environment your pipeline was executed in.
Started at/by: The timestamp of when the pipeline was run, and the username or mechanism (such as schedule), it was run by.
Ended at/duration: The timestamp of when the pipeline finished executing, and how long it took.

Click the link to the right of your pipeline to open your project.

Switch to the Matillion ETL tab to find Matillion ETL instances currently running jobs. This list shows a real-time State of all instances. Your instance's status denotes if it's currently Running, Stopped, or there's been an Error.

This list is checked every 5 seconds, but users can click the refresh icon to ensure they're seeing the latest results. Click the pencil icon next to the instance name to add a description for the intended instance. Selecting an instance will expand its details.

Click on either the Matillion ETL instance name or the downward facing arrow to the right of the status to reveal the following properties:

Platform: The cloud platform your instance resides in.
Data Warehouse: The cloud data warehouse your instance is using.
Region: The region your instance resides in. See your cloud platform provider's documentation for details on available regions.
Size: The instance machine size.
Matillion version: The version of Matillion ETL running on your instance.
Last called in: The latest timestamp from your Matillion ETL instance contacting the Hub. This is done every 5 seconds for running instances. Stopped or Errored instances will cease to "call in".

Note

If you run jobs in your Matillion ETL instance, the details of those jobs will appear here. If no jobs have been run, a message will appear stating "No jobs have been found on this instance. Please run some jobs in Matillion ETL to see them appear here." For more information, read Job activity.

Job activity

Selecting a Matillion ETL instance in the Matillion ETL tab will expand the listing to job activity within the last 24 hours. All job activity is updated in real-time. Entering a job name in the search bar will filter the list.

Jobs have the following properties:

Started at: The timestamp for when this job began running.
Job name: The name of the job as given within the Matillion ETL instance. Clicking the job name will open your Matillion ETL instance in a new browser window and navigate to that job run's entry in the Task History.
Duration: The duration of the job run.
Status: Either Success or Failed, depending on the job's state when ending.

Cancel pipeline runs

To cancel a pipeline that is running:

Click ☰ → Manage → Pipeline Runs.
Ensure you're in the Activity tab in the left sidebar and the Data Productivity Cloud tab.
Locate your running pipeline and click ... .
Click Cancel run or Force cancel.
Cancel run: This is a "soft" cancellation, and will allow the current task to complete. After that, no other pipeline tasks will run.
Force cancel: This is a "hard" cancellation, and will interrupt and terminate the current pipeline task, set the pipeline status to "Stopped", and prevent the pipeline from running any more tasks.

You can also cancel pipeline runs within Designer.