Agent installation using an ARM template

This topic details how to install a Matillion agent in your Azure infrastructure using an ARM template provided by Matillion. You can also, if you wish, download the supplied ARM template and edit as needed.

Note

If you are installing a Streaming pipeline, the process is slightly different. Read Streaming agent installation on Azure for that process.

If you require multiple agents, each one must be installed into a different resource group.

For troubleshooting and frequently asked questions encountered in this process, read Azure agent troubleshooting.

Prerequisites

To get started, you'll need:

An Azure subscription with appropriate permissions to provision cloud resources in the Azure environment and manage access control, specifically for managing resource groups, virtual networks, key vaults, and container apps. Read ARM template permissions for details.
A suitable resource group already defined in your Azure environment. Instructions for creating this—if it doesn't already exist—are included below.
A suitable virtual network already defined in your Azure environment, to secure access between the agent and other resources, reducing the scope of possible access to those resources. The virtual network must:
- Be fully configured as desired, including routing to on-premises resources.
- Allow egress to Matillion's Data Productivity Cloud IP ranges.
- Have room for one additional subnet with at least /28 IP range.
Note

For purposes of the agent, /28 provides 14 usable IPs, which are sufficient for use by agent container replicas for normal operation, and to allow extra IPs for new nodes to come online in a rolling restart/update. Read Environment selection in the Azure documentation for further details.
Optionally, a suitable key vault already defined in your environment. This isn't a mandatory requirement, as the installation process will create a new key vault by default.
Minimum permissions to include the following:
- Create:
  - Subnet
  - Managed identity
  - Key Vault
  - Log Analytics Workspace
  - Container App & Container App Environment
- Modify:
  - Subnet delegation
- Permissions:
  - Role assignments in the resource group
  - Key Vault
Read ARM template permissions for full details of the fine-grained permissions required.

We recommend that you read the documentation and prerequisites before beginning this process. You may require input from your organization's cloud administrator for access and permissions.

Note

The container app for the agent must be deployed with a Dedicated workload profile type. Otherwise, the agent will be unable to update without manual intervention when Matillion issues an updated version.

This document includes instructions for creating a resource group, virtual network, key vault, and NAT gataway, if you don't already have these resources. In all cases, consult Azure's own documentaiton for further details.

Completing the installation

Ensure you have created the appropriate prerequisites in your Azure environment.

Add a suitable subnet to the virtual network, using the following command in the Azure CLI:

az network vnet subnet create \
    --resource-group <YourResourceGroup> \
    --vnet-name <YourVNetName> \
    --name <YourPrivateSubnetName> \
    --address-prefix 10.0.2.0/28 \
    --service-endpoints Microsoft.Storage Microsoft.KeyVault \
    --delegations Microsoft.App/environments

Note the subnet's full ID.

If desired, add a NAT gateway. Apply any security groups and user-defined routing that your network configuration may require, for example to achieve connectivity to the Internet, Azure resources, and resources available via site-to-site VPN. For a full discussion of these topics, read Networking in Azure Container Apps environment in the Azure documentation.
In the Data Productivity Cloud, follow the steps to create a new agent, with Azure as the cloud provider and Container App as the deployment type.
On the Agent details screen, click Reveal credentials in the Credentials section and note the client_id and client_secret.
Click the Launch button in the Install using ARM section. This will open the Azure portal at the Custom deployment page in a new browser tab.

Complete the following details on the Custom deployment page.

Property	Description
Subscription	Select the subscription you will deploy the agent into.
Resource group	Select the resource group you created previously.
Region	Select the Azure region you will deploy the agent to, or leave the default value to deploy the agent in the same region as the selected resource group.
Create Key Vault	Select true if you want the ARM template to create a new key vault. This is the recommended method. If you prefer to use a key vault you have created yourself, select false.
Agent Name	Enter a unique and descriptive name for the agent.
Location	This is set automatically by the template and shouldn't be changed.
Matillion Cloud Region	This is set automatically by the template and shouldn't be changed. It should be the same as the MATILLION_REGION parameter shown on the Agent details screen.
Container App Name	This is set automatically by the template and shouldn't be changed.
Default Key Vault	If you are using an existing key vault, enter its name here. If you are allowing the ARM template to create the key vault, don't edit the parameter string displayed in this field.
Default Key Vault Resource Group	If you are using an existing key vault, enter the name of its resource group here. If you are allowing the ARM template to create the key vault, don't edit the parameter string displayed in this field.
Subnet ID	Enter the ID of the subnet you created perviously.
Account ID	This is set automatically by the template and shouldn't be changed. It should be the same as the ACCOUNT_ID parameter shown on the Agent details screen.
Agent ID	This is set automatically by the template and shouldn't be changed. It should be the same as the AGENT_ID parameter shown on the Agent details screen.
Client ID	Enter the client_id credential you obtained from the Agent details screen.
Client Secret	Enter the client_secret credential you obtained from the Agent details screen.
Container Registry Username	This can be left blank.
Container Registry Password Secret Ref	This can be left blank.
Tags	Optionally, add any tags that your organization's tagging policy requires. Tags should be added as a JSON string, for example: `{"Category": "Development","Business Unit": "Data Pipelines","Owner": "owner@organization.com"}`.

The following properties are optional and should only be set if needed. For more information on these, read Optional agent parameters.

Property	Description
Proxy HTTP	If using a proxy, enter the HTTP proxy details. For example: `http://myproxy.com:8080`.
Proxy HTTPS	If using a proxy, enter the HTTPS proxy details. For example: `https://myproxy.com:8443`.
Proxy Excludes	If using a proxy, specify any addresses to exclude. Separate a list of addresses with the pipe character `\|`. For example: `example.com\|example.net`.
Custom Certificate Location	If using custom certificates, specify the storage location. For example: `my_storage/my_certs`.
Extension Library Location	Specify the location of any additional Python libraries you want to use.
External Driver Location	Specify the location of any external driver files you want to use.

Click Review & Create.

It will take at least 10 minutes for the Container App to be created and the Data Productivity Cloud agent application to load and initialize.

Return to the Agents screen in the Data Productivity Cloud, and ensure that the agent's status is Running.

Create a resource group

You need an Azure resource group that will contain all of the Azure resources for the agent. The agent deployment template requests the name of the resource group to deploy into and will create all resources there.

If you don't have a suitable existing resource group, create a new one as follows:

Log in to the Azure Portal.
Use the search bar to search for "resource groups", and click the Resource groups result.
On the Resource groups screen, click + Create.
On the Create resource group screen, select the Subscription you want to use for the agent from the drop-down.
Enter a new name for the Resource group.
Select an Azure Region for your resource group.
If your organization has a policy that mandates resource group tags, click Tags and assign tags as required.
Click Review + create, and then click Create if you are satisfied with your resource group configuration.

For more information, read Manage Azure resource groups by using the Azure portal in the Microsoft developer documentation.

Create a virtual network

You need a virtual network to secure access between the agent and other resources, reducing the scope of possible access to those resources.

If you don't have a suitable existing virtual network, create a new one as follows:

Log in to the Azure Portal.
Use the search bar to search for "virtual networks", and click the Virtual networks result.
On the Virtual networks screen, click + Create.
On the Create virtual network screen, select the resource group you created previously.
Enter a new Virtual network name.
Select an Azure Region for your virtual network. By default, this should be the same region as your resource group.
If your organization has a policy that mandates virtual network tags, click Tags and assign tags as required.
Click Review + create, and then click Create if you are satisfied with your virtual network configuration.
On the Virtual networks screen, locate the virtual network just created and click it.
Click Service endpoints.
Click + Add and select the required service endpoints from the Service drop-down. These are required if the subnet doesn't have access to the Internet (via Internet gateway or NAT gateway). As a minimum, the following service endpoints will be required:
- Microsoft.KeyVault
- Microsoft.Storage

Select a subnet from the Subnets drop-down. The subnet must have been previously created via the Azure CLI using the following command:

az network vnet subnet create \
    --resource-group <YourResourceGroup> \
    --vnet-name <YourVNetName> \
    --name <YourPrivateSubnetName> \
    --address-prefix 10.0.2.0/28 \
    --service-endpoints Microsoft.Storage Microsoft.KeyVault \
    --delegations Microsoft.App/environments

Click Add.

Create a key vault

The agent requires access to an Azure key vault. However, you don't need to create a key vault yourself; the deployment template will automatically create a key vault in the same resource group as the agent.

If you prefer, the agent can reference a different key vault you have previously created. The existing key vault may be in the same or a different resource group. You might want to reference a key vault in a different resource group so that you can have multiple agents referencing the same secrets.

If you are using an existing key vault, you will need to complete the configuration steps described in Configuring a key vault for Azure agent.

Create a NAT gateway

Log in to the Azure Portal.
Use the search bar to search for "nat gateways", and click the NAT gateways result.
Click Create NAT gateway.
On the Basics tab, enter details of the Subscription, Resource group, and Region, ensuring that these match the details you have used for your virtual network. Enter a suitable Name for the gateway, and keep the suggested defaults for Availability zone (No Zone) and TCP idle timeout (4 minutes).
Click Next: Outbound IP.
On the Outbound IP tab drop-down, click Create a new public IP address. Provide a Name for the IP address and click OK.
Click Next: Subnet.
On the Subnet tab, select the Virtual network containing the target subnet. In the list of subnets that appears, select the required subnet.
If your organization has a policy that mandates resource group tags, click Next: Tags and assign tags as required.
Click Review + create, and then click Create if you are satisfied with the configuration.