Skip to content

Pipeline Observability dashboard

The Pipeline Observability dashboard is a feature of the Hub that lists all Data Productivity Cloud pipelines, and any Matillion ETL instances available in your account.

Note

Your Matillion ETL instances must be of version 1.65 or later to work with Pipeline Observability.

Accessing the dashboard

There are two ways you can access the Pipeline Observability dashboard:

  1. Log in to the Hub.
  2. Click ☰ → Manage → Pipeline Runs.

Or:

  1. Log in to the Hub.
  2. On the Services page, click View pipeline runs.

Either of these methods will open the Pipeline run history tab of the Your activity page.

Pipeline run history

The Pipeline run history tab gives you a comprehensive view of your historic pipeline run details. You can view your pipeline run history information for the past 12 months, you can search for a pipeline name, and you can change time frames based on your preference.

For the Data Productivity Cloud, you can read information about the following:

  • Pipeline: The name of the pipeline that has been executed. Clicking this link will take you to a separate page, which will give you metadata about the pipeline, such as pipeline and individual component details, as well as any error logs.
  • Source: Where the pipeline was executed.
  • Project/branch: Project name and branch information.
  • Status: An indicator if your pipeline is Running, Successful, or if it's Failed.
  • Environment: The environment your pipeline was executed in.
  • Started at/by: The timestamp of when the pipeline was run, and the username or mechanism (such as schedule), it was run by.
  • Ended at/duration: The timestamp of when the pipeline finished executing, and how long it took.

Matillion ETL instance overview

Switch to the Instances tab to find Matillion ETL instances associated with your account automatically listed here. This list shows a real-time State of all instances. This list is checked every 5 seconds, but users can click the refresh icon to ensure they're seeing the latest results. Click the pencil icon next to the instance name to add a description for the intended instance. Selecting an instance will expand its details.

New instances can also be launched from this page by clicking Add instance. See Launching Matillion ETL for more information.

Listed instances have the following properties:

  • State: Whether the instance is currently Running, Stopped, or in Error.
  • Platform: The cloud platform your instance resides in.
  • Data Warehouse: The cloud data warehouse your instance is using.
  • Region: The region your instance resides in. See your cloud platform provider's documentation for details on available regions.
  • Size: The instance machine size.
  • METL version: The version of Matillion ETL running on your instance.
  • Last called in: The latest timestamp from your Matillion ETL instance contacting the Hub. This is done every 5 seconds for running instances. Stopped or Errored instances will cease to "call in".

Job activity

Selecting an instance in the list will expand the listing to job activity within the last 24 hours. All job activity is updated in real-time. Entering a job name in the search bar will filter the list.

Jobs have the following properties:

  • Started at: The timestamp for when this job began running.
  • Job name: The name of the job as given within the Matillion ETL instance. Clicking the job name will open your Matillion ETL instance in a new browser window and navigate to that job run's entry in the Task History.
  • Duration: The duration of the job run.
  • Status: Either Success or Failed, depending on the job's state when ending.