Manage Streaming pipelines

Streaming pipelines are managed through the Streaming tab of your project.

Read Streaming pipelines for a deeper look at the concepts and architecture of Streaming pipelines in the Data Productivity Cloud.

View Streaming pipelines

To view the list of Streaming pipelines:

Log in to the Matillion Data Productivity Cloud.
Click ☰ → Designer.
Select your project.
Choose the Streaming tab.

From here, you can create a new Streaming pipeline.

For each created pipeline, the following information is shown on this screen:

Name: The name assigned to the pipeline.
Status: The pipeline will have one of the following status codes:
- Running: The pipeline is active and streaming data.
- Not Running: The pipeline is configured but isn't currently streaming data.
- Unknown: The status of the pipeline can't be determined at this time. This may be because it's incorrectly set up, or because there is a problem communicating with the agent, or some other issue.
Source: The pipeline's configured source.
Destination: The pipeline's configured destination.

From this tab you can:

Edit a pipeline. Click ... next to the pipeline, and then click Edit.
Start a pipeline. Click ... next to the pipeline, and then click Start. The pipeline will begin streaming changes. This option is only available if the pipeline is currently in a Not Running state.
Stop a pipeline. Click ... next to the pipeline, and then click Stop. The pipeline will no longer stream changes. This option is only available if the pipeline is currently in a Running state.
Delete a pipeline. Click ... next to the pipeline, and then click Delete.
View detailed information on the Pipeline dashboard. Click the name of a pipeline to see its dashboard.

Pipeline dashboard

Click the name of any pipeline in the list of pipelines to see a dashboard showing more details about that pipeline. At the top is the pipeline name and current status, and below this is a Pipeline information panel, which shows the source and destination. Click More details to show full configuration details for the source and destination as well as information about the Streaming agent running the pipeline.

Below the pipeline information are two tabs, Overview and Snapshotting.

From the dashboard you can perform the following actions:

To edit the pipeline configuration, click Edit.
To stop a Running pipeline, click Stop
To start a Not Running pipeline, click Start
To return to the pipeline list, click Streaming pipelines in the top left.

Overview

With the Overview tab selected, you can see the following tabs.

Activity: Shows a graph of the number of rows changed over a period of the last hour, day, seven days, or 30 days.
Tables selected: Lists all tables selected for the pipeline.
Diagnostics: Lists a set of pipeline metrics for debugging and support purposes. In case of an issue with your pipelines, Matillion support may ask you to copy this information and send it to us. To send diagnostics to Matillion support, click the refresh icon in the top left of the panel and then click Copy all, and paste the information into an email, or paste it into a file which can be attached to a support ticket.

Note

Diagnostics are intended for Matillion support only, and it's not expected that you will use this information yourself to diagnose issues.

The Pipeline events panel displays a log of events generated by the running pipeline. Each event has an Event type and a Timestamp. Pipeline Error events can be expanded to show the error message or further information about the event. You can sort the list by ascending or descending Timestamp.

If Email users on pipeline error is toggled on, all project users will be emailed an alert when a pipeline produces a Pipeline Error event type (but not on other event types such as Pipeline Started).

Snapshotting

The Snapshotting tab displays details of snapshot requests currently queued. From here, you can monitor progress of the queue. Each snapshot request will be processed sequentially and removed from the queue on snapshot start. The pipeline must be running to show the snapshot queue.

To make a new snapshot request, click Request a snapshot. New requests will be added to the queue.

The Snapshot events tab displays a log of events generated by running snapshots. Each event has an Event type and a Timestamp. Events can be expanded to show further information about the event. You can sort the list by ascending or descending Timestamp.

The Metrics tab provides a breakdown by table of snapshots performed since the pipeline started. If multiple snapshots have been performed on the same table, only the latest is shown.