Deploying a Streaming agent in Azure using the CLI

Use this guide to add a Streaming agent in Data Loader and deploy the agent using the Azure command line interface (CLI). Creating and deploying an agent are required steps before you can set up a CDC pipeline in Data Loader.

Note

For best performance, your Azure region should be geographically similar to your Hub account region.

Prerequisites

Complete the Creating a Streaming agent in Data Loader and Azure Prerequisite Steps sections in Quick guide for deploying a Streaming agent in Azure.

Deploy your Streaming agent using the Azure CLI

Export the following variables from your Azure environment (see the variable, value, and description):

RESOURCE_GROUP your-resource-group-name Target resource group. Should be the same group containing your target Vnet and Subnet if using private networking (recommended).

CONTAINER_NAME desired-container-name Provide a descriptive name for the container.

CONTAINER_IMAGE public.ecr.aws/matillion/cdc-agent:2 Public agent image path.

CONTAINER_CPU 4 We recommend 4 vCPU.

CONTAINER_RAM 8 We recommended 8 GB RAM.

AZURE_LOCATION For example, southcentralus Target location name.

AZURE_VNET_NAME For example, vnet-southcentralus Target Vnet, if using private networking.

AZURE_SUBNET For example, subnet-cdc-containers Target subnet, if using private networking.

LOG_ANALYTICS_WORKSPACE For example, azure-us-mssql-blob-cli Target Log Analytics workspace name—for shipping logs.

LOG_ANALYTICS_WORKSPACE_KEY For example, lYfsWJqa6J0q9+PDDLMqyjk5iXXXXXXXXXXXgcIzz4sBrsZ7/n9C8eSoqnyRong8Uxag4yfw== Target Log Analytics workspace name—for shipping logs.

AZURE_CLIENT_ID For example, 72de73c2-XXXX-XXXX-9fcf-c14b20d34bb1 App Registration's Application/Client ID.

ID_ORGANIZATION For example, b4a525af-XXXX-XXXX-7c1f-ef898f4d6d49 Hub Organization ID.

ID_AGENT For example, 91470067-XXXX-XXXX-963d-588a1f729ee3 Hub Agent ID.

PLATFORM_WEBSOCKET_ENDPOINT For example, wss://ws-eu.matillion-cdc-prod.matillion.com:443/ws URL of the Hub Websocket endpoint.

AZURE_SECRET_KEY_VAULT_URL https://your-key-vault-name.vault.azure.net/ URL of Key Vault containing Hub key.

AZURE_TENANT_ID For example, 09d19996-XXXX-XXXX-8332-37120f9bba10 Hub Directory/Tenant ID.

PLATFORM_KEY_NAME For example, agent-rsa Name of the key vault secret containing the Hub key.

PLATFORM_KEY_PROVIDER azure-key-vault Static value.

SECRET_PROVIDERS azure-key-vault:1 Static value.

AZURE_CLIENT_SECRET For example, RbB8Q~EfgoqLG21E-XXXXXXXX~Z9B4em6I6bdt App Registration's client secret value (not secret ID).

AZURE_STORAGE_KEY For example, FOHf+HsmBATWsdMa5f97BAvIx50XXXXXXXXXXXDlaY3nBrSrpB+RnDYY7+E4jdNCQYkm3A== Key of target storage account.

Open the Azure CLI and run the following command:

az container create \
    --resource-group $RESOURCE_GROUP \
    --name $CONTAINER_NAME \
    --image $CONTAINER_IMAGE \
    --os-type Linux \
    --cpu $CONTAINER_CPU \
    --memory $CONTAINER_RAM \
    --location $AZURE_LOCATION \
    --restart-policy Always \
    --environment-variables AZURE_CLIENT_ID=$AZURE_CLIENT_ID ID_ORGANIZATION=$ID_ORGANIZATION ID_AGENT=$ID_AGENT PLATFORM_WEBSOCKET_ENDPOINT=$PLATFORM_WEBSOCKET_ENDPOINT AZURE_SECRET_KEY_VAULT_URL=$AZURE_SECRET_KEY_VAULT_URL AZURE_TENANT_ID=$AZURE_TENANT_ID PLATFORM_KEY_NAME=$PLATFORM_KEY_NAME PLATFORM_KEY_PROVIDER=$PLATFORM_KEY_PROVIDER SECRET_PROVIDERS=$SECRET_PROVIDERS \
    --secure-environment-variables AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY \
    --ip-address Private \
    --vnet $AZURE_VNET_NAME \
    --subnet $AZURE_SUBNET \
    --ports 8080 \
    --protocol TCP \
    --log-analytics-workspace $LOG_ANALYTICS_WORKSPACE \
    --log-analytics-workspace-key $LOG_ANALYTICS_WORKSPACE_KEY

In Data Loader, your created Streaming agent's status should display as Connected and offer the Add Pipeline button.