Quick guide for deploying a Streaming agent in Azure

Use this guide to add a Streaming agent in Data Loader and then deploy that agent in Microsoft Azure, using the ARM template. Creating and deploying an agent are required steps to set up a CDC pipeline in Data Loader.

For the best performance, your Azure region should be geographically similar to your Hub account region.

Create a Streaming agent in Data Loader

Register and log in to the Hub.
The My Accounts lists any accounts you have already created or joined. At the bottom of this list, click Add new account. Read Create an Account to learn more about this topic.

Note

Each Hub account can generate one unique platform key that your Streaming agent will use to communicate with Data Loader. With this in mind, create the Streaming agent in the account that matches the platform key you want to use. For more information, read Platform Keys.
Click Load data on the What do you want to do today? page.
On the Data Loader dashboard, scroll to the lower-right of the UI and choose your region.
In the Pipelines dashboard, select Agents in the left sidebar and click Add agent.
Give your agent a sensible Agent name and Description. Click Continue.
Since this guide is for Azure, select Azure as your cloud provider.
Choose ARM as the service to provision and deploy your cloud resources from for the Streaming agent installation.
The Agent setup page will be displayed with useful information and prerequisites you need to know before you can deploy your agent. Please also note on this page the following environment variable values:
- ID_ORGANIZATION: This value is unique per agent.
- ID_AGENT: This value is unique per agent.
- PLATFORM_WEBSOCKET_ENDPOINT: The value is unique for the Data Loader region (US or EU).
Manage key pair: This is a generated value. If you haven't generated a platform secret for your account yet, Data Loader will prompt you to do so when creating a CDC pipeline. You need to store this value in Azure Key Vault where your Streaming agent can access it. For security reasons, this key pair/platform key can only be generated and shown once per account, so make sure to copy and save it for future use.
If a key pair has been configured, click Return to agent list where you can begin the process of adding an agent.
The Agent containers page will list all current agents in your Hub account, including the one you have just created. Click the Agent setup icon next to the intended agent, and refer to the following section to Deploy your Streaming agent in Azure.

Azure prerequisite steps

You need an Azure account with an active subscription.
Resource Group: Create a new resource group if you do not have any existing. You can follow the instructions in the Azure documentation.
Virtual Network and Subnet: Create a virtual network and a subnet associated with it. When deploying an agent in a private network, this is required. You can follow the instructions provided in the Azure documentation to quickly create a virtual network using the Azure portal.
Network Address Translation (NAT) Gateway: Create a NAT gateway and associate it with the subnet you created. You can refer to the Azure documentation for detailed instructions on creating a NAT gateway.

Deploy your Streaming agent in Azure

Follow these steps to deploy your Streaming agent using the ARM template:

Download the Azure ARM template.json.
Log in to the Azure portal.
Search for Deploy a custom template.
The Select a template tab will open. Select Build your own template in the editor.
Click Load file at the top of the page, and choose the template file downloaded in step 1.
The template file will now be loaded in Data Loader. Click Save to open the Custom deployment page in Azure.
Complete the template parameters:
- Subscription: Select the desired subscription from the drop-down menu. This field is likely to default to your preferred subscription. Please note, all resources in an Azure subscription will be billed together.
- Resource Group: Assign a resource group to the selected subscription, or create a new one for your instance. A resource group is a collection of resources that share the same permissions and policies.
- Region: This should be the same region (either US or EU) that you chose when you created the Streaming agent in Data Loader.
- Agent Id: This is the ID_AGENT value you copied from the Prerequisites for agent setup dialog—step 9 of Create a Streaming agent in Data Loader.
- Organization Id: This is the ID_ORGANIZATION value you copied from the Prerequisites for agent setup dialog—step 9 of Create a Streaming agent in Data Loader.
- Platform Websocket Endpoint: The PLATFORM_WEBSOCKET_ENDPOINT value, Prerequisites for agent setup dialog—step 9 of Create a Streaming agent in Data Loader.
- Private Key Data: The private key from the Data Loader UI. Just the content between the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- tags.
- Network Type: This is the network deployment type to be used by the agent. You can select for public or private deployment.
- V Net Resource Group: This is the name of the resource group that holds the target virtual network for agent deployment. Only required if private network is being used.
- V Net: This is the target virtual network for agent deployment. Only required if private network is being used.
- Subnet Name: Target subnet name for agent deployment. Only required if private network is being used.
- Container Image URL: The repository of the Data Loader Streaming agent. The template should autofill this value.
- Location: The Azure Region for your resources to be located.
- Cpu Core: Number of CPU cores the container can use. Can be with a maximum of two decimals. Maximum of 2.0 vCPUs (cores).
- Memory Size: Amount of memory (in gibibytes, GiB) allocated to the container up to 4GiB. Can be with a maximum of two decimals. Ratio with CPU cores must be equal to 2. Default value is 8. For more detail you can refer Sizing Streaming agent.
Click Review + Create, and check you're happy with the information you've provided.
Click Create to deploy your custom template in Azure.

In Data Loader, your created Streaming agent's status will display as Connected, and offer the Add Pipeline button. To add a pipeline to your agent, refer to CDC pipeline overview.