Streaming agent installation on Azure

This topic details how to install a Matillion streaming agent in your Azure infrastructure using an ACI template provided by Matillion.

It is possible to install multiple agents into a single resource group. Multiple agent installations require you to provide a different name for each container app you launch a template into. Otherwise, the process is identical to that described here.

For convenience, you might want to download and amend the template so that the agents you install will have consistent properties.

For troubleshooting and frequently asked questions encountered in this process, read Azure agent troubleshooting.

Resources deployed

The agent installation process will deploy the following resources in your Azure environment:

Resource group: A new or existing resource group to contain all deployed resources.
Virtual network and subnet: A virtual network with a specified subnet size (/23 or larger).
Key vault (optional): A key vault for storing secrets. If you prefer, the agent can use an existing key vault.
Managed identity: A managed identity to access the key vault and other resources securely.
Container app: The container app that runs the Matillion agent.
Log analytics workspace: A workspace for collecting and analyzing logs from the deployed resources.
Role assignments: Various role assignments to ensure appropriate permissions for the deployed resources.
Other associated resources: Additional resources required for the deployment, such as storage accounts and container registries, depending on the configuration specified in the template.

Prerequisites

To get started, you'll need:

A Hub account. To register, read Registration.
An Azure subscription with appropriate permissions to provision cloud resources in the Azure environment and manage access control, specifically for managing resource groups, virtual networks, key vaults, and container apps. Read ARM template permissions for details.
A suitable resource group and virtual network already defined in your Azure environment. Instructions for creating these—if they don't already exist—are included below.
Optionally, a suitable key vault already defined in your environment. This isn't a mandatory requirement, as the installation process will create a new key vault by default.

We recommend that you read the documentation and prerequisites before beginning this process. You may require input from your organization's cloud administrator for access and permissions.

Note

The container app for the agent must be deployed with the Consumption Only Environment Type, or, if using the Workload Profiles Environment Type, the Workload Profile must be the Dedicated type. Otherwise, the agent will be unable to update without manual intervention when Matillion issues an updated version.

Create a resource group

You need an Azure resource group that will contain all of the Azure resources for the agent. The agent deployment template requests the name of the resource group to deploy into and will create all resources there.

If you don't have a suitable existing resource group, create a new one as follows:

Log in to the Azure Portal.
Use the search bar to search for "resource groups", and click the Resource groups result.
On the Resource groups screen, click + Create.
On the Create resource group screen, select the Subscription you want to use for the agent from the drop-down.
Enter a new name for the Resource group.
Select an Azure Region for your resource group.
If your organization has a policy that mandates resource group tags, click Tags and assign tags as required.
Click Review + create, and then click Create if you are satisfied with your resource group configuration.

For more information, read Manage Azure resource groups by using the Azure portal in the Microsoft developer documentation.

Create a virtual network

You need a virtual network to secure access between the agent and other resources, reducing the scope of possible access to those resources.

If you don't have a suitable existing virtual network, create a new one as follows:

Log in to the Azure Portal.
Use the search bar to search for "virtual networks", and click the Virtual networks result.
On the Virtual networks screen, click + Create.
On the Create virtual network screen, select the resource group you created previously.
Enter a new Virtual network name.
Select an Azure Region for your virtual network. By default, this should be the same region as your resource group.
Click IP addresses.
Edit the default subnet, or create a new subnet, to be a minimum size of /23 (512 addresses). This size is required for a "Consumption only" environment type, as described in the Environment selection Azure documentation.
If your organization has a policy that mandates virtual network tags, click Tags and assign tags as required.
Click Review + create, and then click Create if you are satisfied with your virtual network configuration.
On the Virtual networks screen, locate the virtual network just created and click it.
Click Service endpoints.
Click + Add and select the required service endpoints from the Service drop-down. These are required if the subnet doesn't have access to the Internet (via Internet gateway or NAT gateway). As a minimum, the following service endpoints will be required:
- Microsoft.KeyVault
- Microsoft.Storage
Select the subnet you previously edited from the Subnets drop-down.
Click Add.

Create a key vault

The agent requires access to an Azure key vault. However, you don't need to create a key vault yourself; the deployment template will automatically create a key vault in the same resource group as the agent.

If you prefer, the agent can reference a different key vault you have previously created. The existing key vault may be in the same or a different resource group. You might want to reference a key vault in a different resource group so that you can have multiple agents referencing the same secrets.

If you are using an existing key vault, you will need to complete the configuration steps described in Configuring a key vault for Azure agent.

Launch the ARM template in Azure

Ensure that the prerequisites of resource group, virtual network, and key vault (if required) have all been created, and that you are logged in to an Azure account with sufficient permissions to deploy the template in a container app.

In the Data Productivity Cloud, follow the steps to create a new agent, with Streaming as the agent type and Azure as the cloud provider.
On the Agent details screen, click Reveal credentials in the Credentials section and note the client_id and client_secret.
Click the Launch button in the Install using ARM section. This will open the Azure portal at the Custom deployment page in a new browser tab.

Complete the following details on the Custom deployment page.

Property	Description
Subscription	Select the subscription you will deploy the agent into.
Resource group	Select the resource group you created previously.
Region	Select the Azure region you will deploy the agent to, or leave the default value to deploy the agent in the same region as the selected resource group.
Resource Prefix	The prefix used for all resources deployed by this process.
Location	This is set automatically by the template and shouldn't be changed.
Account ID	This is set automatically by the template and shouldn't be changed. It should be the same as the ACCOUNT_ID parameter shown on the Agent details screen.
Agent ID	This is set automatically by the template and shouldn't be changed. It should be the same as the AGENT_ID parameter shown on the Agent details screen.
Matillion Cloud Region	This is set automatically by the template and shouldn't be changed. It should be the same as the MATILLION_REGION parameter shown on the Agent details screen.
CPU Core	This is set automatically by the template and shouldn't be changed.
Memory Size	This is set automatically by the template and shouldn't be changed.
Existing Key Vault URI	To connect to an already existing Key Vault, enter the URI of the vault. If you leave this blank, the template will create a new Key Vault.
Client ID	Enter the client_id credential you obtained from the Agent details screen.
Client Secret	Enter the client_secret credential you obtained from the Agent details screen.
Existing Managed Identity Name	The name of an existing Managed Identity. Leave this empty if you want the template to create a new Managed Identity.
Existing Managed Identity Resource Group	The resource group where the existing Managed Identity is located. Leave this empty if the template is creating a new Managed Identity.
Network Type	The network deployment type to be used by the agent. Select Public or Private.
Virtual Network Name	Enter the name of the virtual network you created previously. Leave this blank if the Nwtwork Type is Public.
Virtual Network Resource Group	The resource group where the Virtual Network is located. Leave this blank if the Network Type is Public.
Subnet Name	Enter the name of the subnet you created previously. Leave this blank if the Network Type is Public.
Tags	Optionally, add any tags that your organization's tagging policy requires. Tags should be added as a JSON string, for example: `{"Category": "Development","Business Unit": "Data Pipelines","Owner": "owner@organization.com"}`.

Click Review & Create.