Manage streaming pipelines

Streaming pipelines are managed through the Streaming tab of your project.

Read Streaming pipelines for a deeper look at the concepts and architecture of streaming pipelines in the Data Productivity Cloud.

View streaming pipelines

To view the list of streaming pipelines:

Log in to the Matillion Hub.
Click ☰ → Designer.
Select your project.
Choose the Streaming tab.

From here, you can create a new streaming pipeline.

For each created pipeline, the following information is shown on this screen:

Name: The name assigned to the pipeline.
Status: The pipeline will have one of the following status codes:
- Running: The pipeline is active and streaming data.
- Not Running: The pipeline is configured but isn't currently streaming data.
- Unknown: The status of the pipeline can't be determined at this time. This may be because it's incorrectly set up, or because there is a problem communicating with the agent, or some other issue.
Source: The pipeline's configured source.
Destination: The pipeline's configured destination.

From this tab you can:

Edit a pipeline. Click ... next to the pipeline, and then click Edit.
Start a pipeline. Click ... next to the pipeline, and then click Start. The pipeline will begin streaming changes. This option is only available if the pipeline is currently in a Not Running state.
Stop a pipeline. Click ... next to the pipeline, and then click Stop. The pipeline will no longer stream changes. This option is only available if the pipeline is currently in a Running state.
Delete a pipeline. Click ... next to the pipeline, and then click Delete.
View pipeline information on the Pipeline dashboard.

Pipeline dashboard

Click the name of any pipeline in the list of pipelines to see a dashboard showing a summary of details for that pipeline. This dashboard offers more details about your pipeline, including its full configuration and current status. Information on the dashboard is arranged into the following panels:

Pipeline information: A summary of the pipeline configuration. Click More details to expand this panel.
Metrics: This panel has three tabs:
- Overview: Shows a graph of the number of rows changed over a period of the last hour, day, seven days, or 30 days.
- Tables selected: Lists all tables selected for the pipeline, with a count of rows changed.
- Snapshotting: Displays details of snapshot operations carried out by the pipeline.
Events: This panel displays a log of events generated by the running pipeline. Each event has a Type, Timestamp, and brief Information message. The panel also contains a toggle button for Email users on error. If this is toggled to On, all project users will be emailed an alert when a pipeline produces an Error event type.

From the dashboard you can perform the following actions:

Click Edit to edit the pipeline configuration.
Click Stop to stop a Running pipeline, or Start to start a Not Running pipeline.
Click Streaming pipelines in the top left to return to the pipeline list.