Skip to content

Create a Streaming agent🔗

Agent Type:
Streaming
Agent Platform:
AWS ✅ Azure ✅ GCP ✅

The Streaming agent is a component within Maia that serves as a bridge between the source database and the target cloud data lake, enabling the execution and scheduling of streaming pipelines. The Streaming agent will be hosted in your own infrastructure, using a Hybrid SaaS solution.

Once the Streaming agent is configured and started, it operates autonomously without requiring much intervention. The Streaming agent continuously monitors all changes occurring in the source database, consumes those changes from the low-level logs, and delivers them to the designated target data lake or storage. This ensures a continuous and reliable change data capture process.

Create a Streaming agent in your own infrastructure. You can create Streaming agents in the user interface or provision them programmatically when deploying streaming environments at scale. We currently support Streaming agents running in AWS, Azure, and GCP infrastructure.

Note

Each Streaming agent can run only one Streaming pipeline. Each Streaming pipeline requires a new Streaming agent installation.

Prerequisites🔗

  • A Maia account. To register, read Registration. Once you have signed up, log in to the Maia.
  • An account in Azure, AWS, or GCP to host the Streaming agent.
  • Access to a cloud secrets service, which is a secure storage system for storing authentication and connection secrets. These secrets are used by the Streaming agent to authenticate itself with the source database and establish a secure connection for capturing the data changes.
  • Allowlisted IP addresses configured for your environment. For more information, read Hybrid SaaS agents and Git repositories.

Note

Your source database will also require configuration to work with Streaming pipelines. This is independent from the Streaming agent installation process. More information can be found in the documentation for each supported data source.

Create a Streaming agent in Maia🔗

  1. In the left navigation, click the Agents & Instances icon . Then, select Agents from the menu.
  2. All currently created Streaming agents are listed showing their Status, Platform (AWS, Azure, or GCP), and Type (Maia or Streaming).
  3. Click Add agent.
  4. Click Streaming.

  5. Complete the following properties:

    • Agent name: A unique name for your new Streaming agent. Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
    • Description: Optionally enter a brief description of the Streaming agent.
    • Cloud provider: The cloud platform that the Streaming agent will be deployed to. Currently, AWS, Azure, and GCP are supported.
    • Deployment: The supported deployment method for the given cloud provider. Currently, Fargate and EKS for AWS, ACI and AKS for Azure, and GKE and GCE for GCP are supported.

  6. Click Create agent.

This creates a Streaming agent definition in Maia, and displays the Streaming agent's parameters on the Agent details screen. The Streaming agent's status is set to Pending, which means it is not yet ready to run pipelines. The next step is to deploy the Streaming agent application into your cloud infrastructure, as described below.

Set up the Streaming agent in your cloud infrastructure🔗

After creating the Streaming agent in Maia with the above process, the Streaming agent needs to be installed into your cloud infrastructure. There are several different ways of doing this, and you can use whichever method suits you:

To complete these processes, you will require certain details of the created Streaming agent. To obtain these details, locate the Streaming agent in the list of Streaming agents, and click the three dots ... next to it, then click Agent details. The parameters and values in the sections Agent image URI, Agent environment variables, and Credentials are required when configuring your Streaming agent in your cloud infrastructure.

You can also use Terraform to deploy Streaming agents and pipelines across AWS, Azure, and GCP. For more information, see Deploy Streaming agents and pipelines with Terraform.

Check Streaming agent status🔗

After deploying the Streaming agent in your cloud infrastructure, you should return to Maia to verify that it's correctly connected and running.

  1. Click the menu button in the top left of any Maia screen, then click Manage → Agents.

  2. Locate the Streaming agent in the list and check the status:

    • Pending: The Streaming agent has been created but has not yet connected to Maia.
    • Running: The Streaming agent is connected and available for running Streaming pipelines, or is connected and already running a Streaming pipeline.
    • Stopped: The Streaming agent has been stopped.
    • Unknown: The Streaming agent is in an unknown state. This typically means the Streaming agent has lost connection to Maia without being stopped, for example due to networking issues.

  3. When the Streaming agent status shows Running, it's ready to use. It can be selected in the Agent drop-down when you create a new Streaming pipeline, as long as a pipeline is not already assigned.

Deleting Streaming agents🔗

To delete a Streaming agent from Maia, locate the Streaming agent on the Agents screen, then click the three dots ... → Remove agent.

This action is irreversible, so be sure that you want to continue.

Deleting the Streaming agent from the Agents screen doesn't remove the underlying AWS, Azure, or GCP resources. You should go into the AWS Console, Azure Portal, or GCP Console and clean up any resources that you no longer require.

Warning

Deleting a Streaming agent that is currently running may interrupt pipelines that are currently running. Therefore, you should always stop the Streaming agent service before deleting it.

© Copyright 2026 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement