Create an agent in your infrastructure
This topic explains how to create an agent in your own infrastructure. For a discussion of why you might want to deploy in your own infrastructure rather than use a Matillion Full SaaS solution, read Agent overview. We currently support agents running in AWS or Azure infrastructure.
There are two different types of agent:
- Data Productivity Cloud agent: Required to run Designer pipelines.
- Streaming agent: Required to run Streaming pipelines.
The installation process is similar for both agents, but you must select the correct agent type for the type of pipeline you want to run. You can install both agents in the same infrastructure if you intend to run both types of pipeline.
Read Scaling best practices to understand how to size your agent for the level of performance you need.
Prerequisites
- A Hub account. To register, read Registration. Once you have signed up, log in to the Hub.
- An account in either Azure or AWS to host the agent.
Create an agent
- Click the menu button in the top left of any Data Productivity Cloud screen, then click Manage → Agents. The Agents screen lists all agents currently created, showing their Status, Platform (AWS or Azure), and Type (Data Productivity Cloud or Streaming).
- Click Add agent.
- Click either Data Productivity Cloud or Streaming, depending on which type of pipeline you intend to run.
-
Complete the following properties:
- Agent name: A unique name for your new agent. Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
- Description: Optionally enter a brief description of the agent.
- Cloud provider: The cloud platform that the agent will be deployed to. Currently, AWS and Azure are supported.
- Deployment: The supported deployment method for the given cloud provider. Currently, Container App/ACI for Azure and Fargate for AWS are supported.
-
Click Create agent.
Set up the agent in your cloud infrastructure
When created with the above process, the agent needs to be set up in your cloud infrastructure. There are several different ways of doing this, and you can use whichever method suits you:
-
For AWS agents:
-
For Azure agents:
To complete these processes, you will require certain details of the created agent. To obtain these details, locate the agent in the list of agents, and click ... next to it, then click Agent details. The parameters and values in the sections Agent image URI, Agent environment variables, and Credentials are required when configuring your agent in your cloud infrastructure.
Check agent status
After deploying the agent in your cloud infrastructure, you should return to the Data Productivity Cloud to verify that it's correctly connected and running.
- Click the menu button in the top left of any Data Productivity Cloud screen, then click Manage → Agents.
-
Locate the agent in the list and check the status:
- Running: The agent is connected and ready for development and pipeline tasks.
- Unknown: The agent is in an unknown state.
- Pending: The agent has been created but has not yet connected to the Hub.
- Stopped: The agent has been stopped.
-
When the agent status shows Running, it's ready to use.
Restarting agents
To restart an agent, follow the steps in Restart an agent.
There are a number of reasons why you may need to restart your agent, such as receiving agent updates.
Deleting agents
To delete an agent from the Data Productivity Cloud, locate the agent on the Agents screen, then click ... → Remove agent.
Deleting the agent from the Agents screen does not remove the underlying AWS or Azure resources. You should go into the AWS Console or Azure Portal and clean up any resources that you no longer require.
Warning
Deleting an agent that is currently running may interrupt scheduled pipelines or pipelines that are currently running. Therefore, you should always stop the agent service before deleting it.