Agent setup

Use this page to troubleshoot errors that you may encounter while configuring and working with streaming agents.

General database prerequisites

We recommend that you set up your source database infrastructure for streaming pipelines before creating an agent.
Ensure you have the required database privileges to manage datastores.

Note

If you're not a database administrator, follow your company guidance to secure access to create a stack in your infrastructure environment.

AWS prerequisites

An active AWS account is available.
The Amazon instance is running.
SELECT privileges are granted to the database user.
Secret Key needs to be uploaded to the AWS Secret Manager.
Subnets within a customer private cloud with outbound access to the relevant data sources and destinations, as well as to ECR and to the Matillion CDC websocket endpoint.
Security group controlling ingress/egress within the subnet(s).
CloudWatch log group for container log ingestion.
S3 bucket for data ingestion, with permissions granted to the IAM role.
IAM task execution role with permissions to pull from ECR Public repositories.
A valid log group needs to be created, where Matillion pushes the logs. You should also have valid AWS credentials (access and secret keys) of an IAM user who has CreateLogStream , DeleteLogStream, and PutLogEvents permissions on the log group created.

Template usage troubleshooting

The Matillion Streaming agent is a container image that's required to be deployed on any supporting containerized infrastructure and as part of this installation process, the agent requires some specific environment variables to be configured for platform authentication and functionality.

Some of these environment variables are automatically generated by the Matillion CDC platform and stated on the Installation Instructions page in CDC UI.

Ensure you take note of the environment variables listed on the Installation Instructions page as they will be required when you install and configure agent in your database environment.

There are templates available for every database to install agent setup configurations. You can refer to the required templates in Deploying Streaming agent.

AWS Template - Basic/Advanced
- Basic Templates: Basic templates should be used by users with more stringent requirements around networking and permissions. In some cases such users may need to set up manually in a way that meets their company policy.
- Advanced Templates: Advanced templates contains required resources for the users that will automatically update and fill required fields while configuring the stack in AWS. This option is suitable for a user looking to try out the software.

Accessing agent logs

Agent logs provide verbose descriptions of agent information, warnings, and errors to help you diagnose and fix any installation and platform communication issues. If you are not able to resolve your issue with this information, contact Matillion Support.

Agents are automatically configured to write logs to the platform's native logging facility, and therefore the process for accessing the logs will depend on the platform used.

AWS

Agents deployed to AWS use the AWS CloudWatch service. To view the logs, log in to your AWS account and browse to the CloudWatch service. The logs will be located under the Log group you created when you installed the agent.

Azure

Agents deployed to Azure use the Azure Log Analytics service. To view the logs, log in to your Microsoft Azure portal account, select the resource group that hosts your agent, and select the Log Analytics workspace you created when you installed the agent.

Google Cloud Platform

Agents deployed to GCP store logs on the virtual machine instance that hosts the agent. To view the logs, log in to your Google Cloud Platform account, select the VM that the agent is deployed to, and browse to the Observability information.

Azure Streaming agent continually restarting

When you set up an Azure agent and it successfully runs, you may encounter a situation where the agent keeps restarting and a message of "no active profile set" displays. To fix this, make sure you grant identity permission to access the Azure key vault. For more information, read Assign an access policy.

The issue with Azure agents restarting also means the agent log will clear upon each restart. You need to use Log Analytics to view the agent's log history.

Got feedback or spotted something we can improve?

We'd love to hear from you. Join the conversation in the Documentation forum!