GCP Streaming agent install

The Streaming agent is a component within the Data Productivity Cloud that serves as a bridge between the source database and the target cloud data lake or cloud storage, enabling the execution and scheduling of streaming pipelines. The agent will be hosted in your own infrastructure, using a Hybrid SaaS solution.

Once the Streaming agent is configured and started, it operates autonomously without requiring much intervention. The agent continuously monitors all changes occurring in the source database, consumes those changes from the low-level logs, and delivers them to the designated target data lake or storage. This ensures a continuous and reliable change data capture process.

This topic explains how to create an agent in your GCP infrastructure.

Note

Each Matillion Streaming agent can run only one Streaming pipeline. Each Streaming pipeline requires a new agent installation.

Prerequisites

A Data Productivity Cloud account. To register, read Registration. Once you have signed up, log in to the Data Productivity Cloud.
An account in GCP to host the agent.
Access to a cloud secrets service, which is a secure storage system for storing authentication and connection secrets. These secrets are used by the agent to authenticate itself with the source database and establish a secure connection for capturing the data changes.

Create an agent

Click Agents & Instances → Agents. The Agents page lists all agents currently created, showing their Status, Platform, and Type.
Click Add agent.
Click Streaming.
Complete the following properties:
- Agent name: A unique name for your new agent. Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
- Description: Optionally enter a brief description of the agent.
- Cloud provider: The cloud platform that the agent will be deployed to. Select GCP.
- Deployment: The agent deployment method. Choose from GCE (Google Compute Engine) or GKE (Google Kubernetes Engine).
Click Create agent.
This creates an agent definition in the Data Productivity Cloud, and displays the following parameters on the Agent details page. You will need these parameters to set up the agent in your GCP infrastructure.
- ACCOUNT_ID
- AGENT_ID
- MATILLION_REGION
- OAUTH_CLIENT_ID
- OAUTH_CLIENT_SECRET
The screen will also show any optional environment variables needed by the agent.

Click Reveal to make the OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET values visible before copying.
The agent's status is set to Pending, which means it is not yet ready to run pipelines. The next step is to deploy the agent application into your GCP infrastructure, as described below.

Set up the agent in your GCP infrastructure

Setting up agents requires access to the GCP platform and services, as well as a degree of familiarity with that platform. We recommend going through this process with your GCP platform administrator.

Note

GCP resources may come with their own pricing independent of any billing you receive from Matillion.

Recommendations:

Create new resources specifically for streaming use rather than attempt to use existing cloud resources.
Set up a resource group for your new resources for better organization and billing ease.
Consult your cloud/network administrator for advice on GCP permissions, roles, access and other considerations such as GCP regions.
Keep resources in the same Google Cloud region. Note that all resources and services may not be available in all regions; it is recommended you research your desired region in advance.

Check agent status

After deploying the agent in your GCP infrastructure, you should return to the Data Productivity Cloud to verify that it's correctly connected and running.

Click Agents & Instances → Agents. The Agents screen lists all agents currently created.
Locate the agent in the list and check the status:
- Pending: The agent has been created but has not yet connected to the Data Productivity Cloud.
- Running: The agent is connected and available for running Streaming pipelines, or is connected and already running a Streaming pipeline.
- Stopped: The agent has been stopped.
- Unknown: The agent is in an unknown state. This typically means the agent has lost connection to the Data Productivity Cloud without being stopped, for example due to networking issues.
When the agent status shows Running, it's ready to use. It can be selected in the Agent drop-down when you create a new Streaming pipeline, as long as a pipeline is not already assigned.