Deploying a CDC agent in Azure using the CLI

Use this guide to add a CDC agent in Data Loader and deploy the agent using the Azure command line interface (CLI). Creating and deploying an agent are required steps before you can set up a CDC pipeline in Data Loader.

Note

For best performance, your Azure region should be geographically similar to your Data Productivity Cloud account region.

Prerequisites

Complete the Creating a CDC agent in Data Loader and Azure Prerequisite Steps sections in Quick guide for deploying a CDC agent in Azure.

Deploy your CDC agent using the Azure CLI

Export the following variables from your Azure environment (see the variable, value, and description):

RESOURCE_GROUP your-resource-group-name Target resource group. Should be the same group containing your target Vnet and Subnet if using private networking (recommended).

CONTAINER_NAME desired-container-name Provide a descriptive name for the container.

CONTAINER_IMAGE public.ecr.aws/matillion/cdc-agent:2 Public agent image path.

CONTAINER_CPU 4 We recommend 4 vCPU.

CONTAINER_RAM 8 We recommended 8 GB RAM.

AZURE_LOCATION For example, southcentralus Target location name.

AZURE_VNET_NAME For example, vnet-southcentralus Target Vnet, if using private networking.

AZURE_SUBNET For example, subnet-cdc-containers Target subnet, if using private networking.

LOG_ANALYTICS_WORKSPACE For example, azure-us-mssql-blob-cli Target Log Analytics workspace name—for shipping logs.

LOG_ANALYTICS_WORKSPACE_KEY For example, lYfsWJqa6J0q9+PDDLMqyjk5iXXXXXXXXXXXgcIzz4sBrsZ7/n9C8eSoqnyRong8Uxag4yfw== Target Log Analytics workspace name—for shipping logs.

AZURE_CLIENT_ID For example, 72de73c2-XXXX-XXXX-9fcf-c14b20d34bb1 App Registration's Application/Client ID.

ID_ORGANIZATION For example, b4a525af-XXXX-XXXX-7c1f-ef898f4d6d49 Hub Organization ID.

ID_AGENT For example, 91470067-XXXX-XXXX-963d-588a1f729ee3 Hub Agent ID.

PLATFORM_WEBSOCKET_ENDPOINT For example, wss://ws-eu.matillion-cdc-prod.matillion.com:443/ws URL of the Hub Websocket endpoint.

AZURE_SECRET_KEY_VAULT_URL https://your-key-vault-name.vault.azure.net/ URL of Key Vault containing Hub key.

AZURE_TENANT_ID For example, 09d19996-XXXX-XXXX-8332-37120f9bba10 Hub Directory/Tenant ID.

PLATFORM_KEY_NAME For example, agent-rsa Name of the key vault secret containing the Hub key.

PLATFORM_KEY_PROVIDER azure-key-vault Static value.

SECRET_PROVIDERS azure-key-vault:1 Static value.

AZURE_CLIENT_SECRET For example, RbB8Q~EfgoqLG21E-XXXXXXXX~Z9B4em6I6bdt App Registration's client secret value (not secret ID).

AZURE_STORAGE_KEY For example, FOHf+HsmBATWsdMa5f97BAvIx50XXXXXXXXXXXDlaY3nBrSrpB+RnDYY7+E4jdNCQYkm3A== Key of target storage account.

Open the Azure CLI and run the following command:

az container create \
    --resource-group $RESOURCE_GROUP \
    --name $CONTAINER_NAME \
    --image $CONTAINER_IMAGE \
    --os-type Linux \
    --cpu $CONTAINER_CPU \
    --memory $CONTAINER_RAM \
    --location $AZURE_LOCATION \
    --restart-policy Always \
    --environment-variables AZURE_CLIENT_ID=$AZURE_CLIENT_ID ID_ORGANIZATION=$ID_ORGANIZATION ID_AGENT=$ID_AGENT PLATFORM_WEBSOCKET_ENDPOINT=$PLATFORM_WEBSOCKET_ENDPOINT AZURE_SECRET_KEY_VAULT_URL=$AZURE_SECRET_KEY_VAULT_URL AZURE_TENANT_ID=$AZURE_TENANT_ID PLATFORM_KEY_NAME=$PLATFORM_KEY_NAME PLATFORM_KEY_PROVIDER=$PLATFORM_KEY_PROVIDER SECRET_PROVIDERS=$SECRET_PROVIDERS \
    --secure-environment-variables AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY \
    --ip-address Private \
    --vnet $AZURE_VNET_NAME \
    --subnet $AZURE_SUBNET \
    --ports 8080 \
    --protocol TCP \
    --log-analytics-workspace $LOG_ANALYTICS_WORKSPACE \
    --log-analytics-workspace-key $LOG_ANALYTICS_WORKSPACE_KEY

In Data Loader, your created CDC agent's status should display as Connected and offer the Add Pipeline button.

Got feedback or spotted something we can improve?

We'd love to hear from you. Join the conversation in the Documentation forum!