Manage Streaming pipelines

Streaming pipelines are managed through the Streaming tab of your project.

Read Streaming pipelines for a deeper look at the concepts and architecture of Streaming pipelines in the Data Productivity Cloud.

View Streaming pipelines

To view the list of Streaming pipelines:

Log in to the Data Productivity Cloud.
In the left navigation, click the Projects icon .
Select your project.
Choose the Streaming tab.

From here, you can create a new Streaming pipeline.

For each created pipeline, the following information is shown on this screen:

Name: The name assigned to the pipeline.
Status: The pipeline will have one of the following status codes:
- Running: The pipeline is active and streaming data.
- Not Running: The pipeline has been created but isn't currently streaming data, and may not be fully configured correctly. Check the pipeline configuration against the details given in Create a Streaming pipeline to ensure the configuration is correct.
- Unknown: The status of the pipeline can't be determined at this time. This may be because it's incorrectly set up, or because there is a problem communicating with the agent, or some other issue. Check the pipeline configuration against the details given in Create a Streaming pipeline to ensure the configuration is correct.
Source: The pipeline's configured source.
Destination: The pipeline's configured destination.

From this tab you can:

Edit a pipeline. Click ... next to the pipeline, and then click Edit.
Start a pipeline. Click ... next to the pipeline, and then click Start. The pipeline will begin streaming changes. This option is only available if the pipeline is currently in a Not Running state.
Stop a pipeline. Click ... next to the pipeline, and then click Stop. The pipeline will no longer stream changes. This option is only available if the pipeline is currently in a Running state.
Delete a pipeline. Click ... next to the pipeline, and then click Delete.
View detailed information on the Pipeline dashboard. Click the name of a pipeline to see its dashboard.

Pipeline dashboard

Click the name of any pipeline in the list of pipelines to see a dashboard showing more details about that pipeline. At the top is the pipeline name and current status, and below this is a Pipeline information panel, which shows the source and destination. Click More details to show full configuration details for the source and destination as well as information about the Streaming agent running the pipeline.

Below the pipeline information are two tabs, Overview and Snapshotting.

From the dashboard you can perform the following actions:

To edit the pipeline configuration, click Edit.
To stop a Running pipeline, click Stop
To start a Not Running pipeline, click Start
To return to the pipeline list, click Streaming pipelines in the top left.

Overview

With the Overview tab selected, you can see the following tabs.

Activity: Shows a graph of the number of rows changed over a period of the last hour, day, seven days, or 30 days.
Tables selected: Lists all tables selected for the pipeline.
Diagnostics: Lists a set of pipeline metrics for debugging and support purposes. In case of an issue with your pipelines, Matillion support may ask you to copy this information and send it to us. To send diagnostics to Matillion support, click the refresh icon in the top left of the panel and then click Copy all, and paste the information into an email, or paste it into a file which can be attached to a support ticket.

Note

Diagnostics are intended for Matillion support only, and it's not expected that you will use this information yourself to diagnose issues.

The Pipeline events panel displays a log of events generated by the running pipeline. Each event has an Event type and a Timestamp. Pipeline Error events can be expanded to show the error message or further information about the event. You can sort the list by ascending or descending Timestamp.

If Email users on pipeline error is toggled on, all project users will be emailed an alert when a pipeline produces a Pipeline Error event type (but not on other event types such as Pipeline Started).

Snapshotting

The Snapshotting tab displays details of snapshot requests currently queued. From here, you can monitor progress of the queue. Each snapshot request will be processed sequentially and removed from the queue on snapshot start. The pipeline must be running to show the snapshot queue.

To make a new snapshot request, click Request a snapshot. New requests will be added to the queue.

The Snapshot events tab displays a log of events generated by running snapshots. Each event has an Event type and a Timestamp. Events can be expanded to show further information about the event. You can sort the list by ascending or descending Timestamp.

The Metrics tab provides a breakdown by table of snapshots performed since the pipeline started. If multiple snapshots have been performed on the same table, only the latest is shown.

Source database maintenance

Source databases may undergo scheduled or ad hoc maintenance processes, including:

Vacuuming
Backups
Log switching
Log archiving

Make sure that tables and data are not locked or otherwise inaccessible when the initial snapshot phase of a pipeline is carried out. These processes can potentially disrupt streaming processes and give the impression that the pipeline is stalled. For example, if an agent log shows data was being processed but then shows no progress for a prolonged period, check with your source database administrator or admin that the data is accessible.

Note

PostgreSQL automated vacuuming can cause pipelines to stall. In these circumstances, terminate the vacuuming process to allow the streaming pipeline to continue.