Agent (AWS) template
It's possible to use a Kubernetes platform to manage Streaming agent containers. The provided Kubernetes pod templates will allow the agent image to be retrieved from Amazon ECR and deployed to any compatible x86 Linux Kubernetes platform.
See the official documentation on Kubernetes pods for more information.
It should be understood that using the Matillion Streaming agent in a Kubernetes platform is among the more manual methods of setting up a Data Loader CDC pipeline with a great burden of knowledge on the user. This should only be attempted by users who are adept with Kubernetes and AWS.
The template provides a blueprint for installation that you may use verbatim, but you may need to modify it to suit your own needs and rules governing your cloud infrastructure.
Prerequisites
Resources
This template is intended for users who are accustomed to setting up their own Kubernetes platforms and offers a degree of freedom that other templates (such as CloudFormation) do not. As such, we cannot dictate which resources you may choose to use to build your CDC stack.
However, this template does require AWS Secrets Manager resources and permissions to the agent image on the ECR Public Gallery.
-
AWS Secrets Manager secrets for the following:
- Platform Key
- Database passwords
-
IAM Role for Task Execution:
ecr-public:GetAuthorizationTokensts:GetServiceBearerToken
-
IAM Role for Tasks:
secretsmanager:GetSecretValue
See the AWS documentation for more information.
AWS Template
Parameters
The following environment variables are defined in the AWS Template. This template requires you to fill in environment variables that allow Data Loader to identity the agent.
| Environment Variable | Description |
|---|---|
| ID_ORGANIZATION | This is provided to you by the Data Loader client when setting up a new agent. |
| ID_AGENT | This is provided to you by the Data Loader client when setting up a new agent. |
| PLATFORM_KEY_NAME | The name of the key storing your platform secret that's generated the first time you attempt to create an agent. |
| PLATFORM_KEY_PROVIDER | The service that supplies your platform key. This must be aws-secrets-manager for the AWS Template. |
| PLATFORM_WEBSOCKET_ENDPOINT | This value must be set to wss://ws-<region>.matillion-cdc-prod.matillion.com:443/ws where <region> is either eu or us depending on the Data Loader region you are building the pipeline in. |
| SECRET_PROVIDERS | The service that holds your database passwords. This must be aws-secrets-manager:1 for the AWS Template. |