Launching Matillion ETL via Azure CLI

Warning

New and existing customers must go through the Hub to choose their preferred cloud provider and cloud data warehouse before starting their Matillion ETL journey.

This guide explains how to find and launch existing Matillion ETL products via the Azure command line interface (CLI). You can use the Azure CLI for the following:

• To create a "bring your own licence" (BYOL) Matillion ETL instance. Information about setting up all required Azure resources will be covered in this guide.
• To launch a legacy marketplace listing for customers who aren't registered with the Hub.

Note

Make sure you choose a Matillion ETL instance size based on the supported Azure VM size in the Azure Instance Recommendations section of the Non-hub: Instance Sizes Guide.

---

Prerequisites

Prior to launching a Matillion ETL instance you need to be an existing non-hub customer, and are required to have:

• Adequate knowledge about the cloud service account (AWS, Azure, GCP), and Cloud Data Warehouse (Snowflake, Redshift or Google BigQuery), you want to launch.
• A user with admin permissions who can access the intended cloud service account.
• Access to a cloud storage bucket (S3, Azure, Blob Storage or Google Cloud Storage) to house the transient staging files Matillion used to load data to the cloud.
• A network path to access the intended data sources. This may involve working with your network team to enable access to on-premise databases.

You are also required to:

• Install and authenticate the Azure CLI at https://learn.microsoft.com/en-us/cli/azure/install-azure-cli.
• Download and install jq at https://stedolan.github.io/jq/download/.

Note

The commands provided in this guide are written for Bash. However, using similar commands you can generate the same results using PowerShell or other shells.

---

Locating the available images

You are required to locate URN values to launch the published Matillion ETL images using the relevant CLI. The published images are available in the following format:

publisher:offer:sku:version

Displayed in the table below, are the specific offers from Matillion ETL:

Publisher	Offer	SKU	Payment Method
Matillion	matillion-etl-snowflake	matillion-etl-for-snowflake	Marketplace billing
Matillion	matillion-azure-synapse	matillion-etl-for-synapse	Marketplace billing
Matillion	matillion-etl-snowflake	matillion-etl-byol	Bring your own licence (BYOL)
Matillion	matillion-azure-synapse	matillion-etl-synapse-byol	Bring your own licence (BYOL)
Matillion	matillion	matillion-etl-for-snowflake	Hub
Matillion	matillion	matillion-etl-for-synapse	Hub
Matillion	matillion	matillion-etl-for-deltalake	Hub

To find the specific versions for the table above, run the following command:

az vm image list --publisher matillion --all | jq -r '.[].urn'

An example of what this command will return is printed below:

matillion:matillion:matillion-etl-for-deltalake:1.52.0
matillion:matillion:matillion-etl-for-deltalake:1.53.8
matillion:matillion:matillion-etl-for-snowflake:1.53.8
matillion:matillion:matillion-etl-for-synapse:1.53.8
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.48.4
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.50.9
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.51.5
matillion:matillion-etl-azure-synapse:matillion-etl-for-synapse:1.53.8
matillion:matillion-etl-azure-synapse:matillion-etl-synapse-byol:1.53.8
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.48.4
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.50.9
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.51.5
matillion:matillion-etl-snowflake:matillion-etl-for-snowflake:1.53.8
matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8

Next, select the URN/product/version combination that you wish to launch, and continue to the next step. For the purpose of this example, the following URN will be used:

matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8

---

Launching an image: Required resources

To launch an image, the following resources must be created:

• An IP address to assign. See example.
• A network interface (NIC) to assign to the machine image. See example.

The following dependencies are required for you to launch a Matillion ETL virtual machine and access it successfully:

• A resource group. See example.
• A storage account. See example.
• A network and subnetwork. See example.
• A network security group. See example.
• A network security rule. See example.
• An identity to assign to the machine image. See example.

Examples for each of these resources are displayed in this document. In many cases, users will already have a pre-existing resource group, storage account, network and subnetwork, network security group, network security rule, and identity set up. Therefore, the next section of this document will provide instructions for setting up the IP, the NIC, and the virtual machine (VM).

---

Creating an IP

Explanation of the CLI command:

CLI call
# Public IP Creation
# - Give the new IP a name
# - Select the resource group to associate the IP with.
# - Select the location which you wish to create the IP.
az network public-ip create \
--name example-public-ip \
--resource-group example-resource-group \
--location UKWest \
--allocation-method Dynamic

Example CLI command:

az network public-ip create \
--name example-public-ip \
--resource-group example-resource-group \
--location UKWest \
--allocation-method Dynamic

---

Creating a network interface (NIC)

Explanation of the CLI command:

Network Interface Card Creation
# - Give the new NIC a name
# - Select the resource group to associate the NIC with.
# - Select the location which you wish to create the NIC.
# - Select the network to associate this NIC with
# - Select the subnetwork to associate this NIC with
# - Add the network security group which you created earlier, or one which already exists
# - Assign the public IP which we created earlier

Example CLI command:

az network nic create \
--name example-nic \
--resource-group example-resource-group \
--location UKWest \
--vnet-name example-network \
--subnet example-subnetwork \
--network-security-group example-nsg \
--public-ip-address example-public-ip

---

Creating a virtual machine (VM)

Explanation of the CLI command:

# Azure VM Creation
# - Give the new VM a name.
# - Select the resource group to associate the VM with.
# - Select the location in which you wish to create the VM.
# - Use the image URN that was found earlier.
# - Optionally, specify that an unmanaged disk will be used.
# - Define the Azure Storage Account where the unmanaged disk will be stored (not needed for managed disks).
# - Define the Container in the Azure Storage Account to store the unmanaged disk (not needed for managed disks).
# - Choose a name for the root disk.
# - Add an identity to the machine to allow it to authenticate to Azure
# - Assign the NIC which was created earlier.
# - There are other options if you wish you use an existing key (see:
  https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az_vm_create)
# - Specify the size of the OS disk you wish to deploy.
# - Specify the machine name size which you wish to launch.

Example CLI command:

az vm create \
--name example-vm \
--resource-group example-resource-group \
--location UKWest \
--image matillion:matillion-etl-snowflake:matillion-etl-snowflake-byol:1.53.8 \
--use-unmanaged-disk \
--storage-account matillionsaexample \
--storage-container-name system \
--os-disk-name example-machine-disk.vhd \
--assign-identity example-managed-identity \
--nics example-nic \
--admin-username centos \
--generate-ssh-keys \
--ssh-dest-key-path /home/centos/.ssh/authorized_keys \
--os-disk-size-gb 60 \
--size Standard_B2s

For the --assign-identity parameter you should specify the actual User Identity you want to associate with the VM. See Roles and Permissions (Azure) for details. This assumes you want to use a user-assigned identity. If you want to use a System Assigned Identity (which is unique to the VM) instead, use the --assign-identity parameter without specifying the name of a user identity:

--assign-identity \

The way you configure a managed identity will be dependent on how you want to control your Matillion ETL VM's access to a storage account. This is described in the article Roles and Permissions (Azure), which also describes how to review your existing Matillion ETL VM and how its identity is set up.

Additionally, we recommend that users read Azure's documentation on Configure managed identities for Azure resources on an Azure VM using Azure CLI for more details and examples.

The above example assumes you are creating a VM with an Unmanaged Disk, which requires the following three parameters:

-use-unmanaged-disk \
--storage-account matillionsaexample \
--storage-container-name system \

You can instead choose to create the VM with a Managed Disk, in which case these three parameters should be omitted. See here for an explanation of the differences between managed and unmanaged disks.

Note

When deploying with an unmanaged disk, the --os-disk-name can't have the same name as an existing file in the storage container you have specified.

The recommended typical default disk size (specified with --os-disk-size-gb) is 60GB. Matillion has found that this size suits most user's needs; however, you might need to look at the size of your existing Matillion ETL VM and see how much space is available to determine whether 60GB is sufficient for your specific use.

The biggest consumer of disk space on a Matillion ETL VM is typically the on-instance Postgres database. This is Matillion ETL's "persistence layer", where all things created by users will be stored. Typically, the biggest consumer of space in the Postgres database is task history information. Matillion ETL will by default retain 365 days of task history, after which it will be purged. The frequency and volume of job executions and the messages generated by those jobs will be the biggest factor in the overall storage footprint of task history.

---

Final launching steps

Once all of the above has been accomplished, Matillion ETL can be accessed (you may need to wait a few minutes for the image to be prepared).

If the End User Licence Agreement (EULA) hasn't been accepted from the product before launching, you may receive an error like the below:

Message: Offer with PublisherId: 'matillion', OfferId: 'matillion' cannot be purchased due to validation errors. For more information see details. Correlation Id: '33a6e641-d6cb-4f6c-999a-34aaff514739' You have not accepted the legal terms on this subscription: '47b72173-e519-41b6-afd1-10018741e1d3' for this plan. Before the subscription can be used, you need to accept the legal terms of the image. To read and accept legal terms, use the Azure CLI commands described at https://go.microsoft.com/fwlink/?linkid=2110637 or the PowerShell commands available at https://go.microsoft.com/fwlink/?linkid=862451.

To fix or prevent this error, accept the terms of the URN by running this command:

az vm image terms accept --subscription <your-subscription-id> --urn <product urn>

Once you have run this command, run the create virtual machine (VM) command again.

---

Command responses (optional reading)

This section includes the responses to the commands made above.

Creating an IP response

{
  "publicIp": {
    "ddosSettings": null,
    "dnsSettings": null,
    "etag": "W/\"a0a4aafb-ccf4-417b-979e-60313b016c1e\"",
    "extendedLocation": null,
    "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/publicIPAddresses/example-public-ip",
    "idleTimeoutInMinutes": 4,
    "ipAddress": null,
    "ipConfiguration": null,
    "ipTags": [],
    "location": "ukwest",
    "name": "example-public-ip",
    "provisioningState": "Succeeded",
    "publicIpAddressVersion": "IPv4",
    "publicIpAllocationMethod": "Dynamic",
    "publicIpPrefix": null,
    "resourceGroup": "example-resource-group",
    "resourceGuid": "82455538-8034-4803-84f1-1fd10d97653e",
    "sku": {
      "name": "Basic",
      "tier": "Regional"
    },
    "tags": null,
    "type": "Microsoft.Network/publicIPAddresses",
    "zones": null
  }
}

Creating a network interface (NIC) response

{
  "NewNIC": {
    "dnsSettings": {
      "appliedDnsServers": [],
      "dnsServers": [],
      "internalDnsNameLabel": null,
      "internalDomainNameSuffix": "mqc30qfsjbvu5ebczbbi3bz3hc.cwx.internal.cloudapp.net",
      "internalFqdn": null
    },
    "dscpConfiguration": null,
    "enableAcceleratedNetworking": false,
    "enableIpForwarding": false,
    "etag": "W/\"aa2272f0-9f56-4cb5-b135-fd3d9a0b7d11\"",
    "extendedLocation": null,
    "hostedWorkloads": [],
    "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkInterfaces/example-nic",
    "ipConfigurations": [
      {
        "applicationGatewayBackendAddressPools": null,
        "applicationSecurityGroups": null,
        "etag": "W/\"aa2272f0-9f56-4cb5-b135-fd3d9a0b7d11\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkInterfaces/example-nic/ipConfigurations/ipconfig1",
        "loadBalancerBackendAddressPools": null,
        "loadBalancerInboundNatRules": null,
        "name": "ipconfig1",
        "primary": true,
        "privateIpAddress": "10.0.0.4",
        "privateIpAddressVersion": "IPv4",
        "privateIpAllocationMethod": "Dynamic",
        "privateLinkConnectionProperties": null,
        "provisioningState": "Succeeded",
        "publicIpAddress": {
          "ddosSettings": null,
          "dnsSettings": null,
          "etag": null,
          "extendedLocation": null,
          "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/publicIPAddresses/example-public-ip",
          "idleTimeoutInMinutes": null,
          "ipAddress": null,
          "ipConfiguration": null,
          "ipTags": null,
          "location": null,
          "name": null,
          "provisioningState": null,
          "publicIpAddressVersion": null,
          "publicIpAllocationMethod": null,
          "publicIpPrefix": null,
          "resourceGroup": "example-resource-group",
          "resourceGuid": null,
          "sku": null,
          "tags": null,
          "type": null,
          "zones": null
        },
        "resourceGroup": "example-resource-group",
        "subnet": {
          "addressPrefix": null,
          "addressPrefixes": null,
          "delegations": null,
          "etag": null,
          "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network/subnets/example-subnetwork",
          "ipAllocations": null,
          "ipConfigurationProfiles": null,
          "ipConfigurations": null,
          "name": null,
          "natGateway": null,
          "networkSecurityGroup": null,
          "privateEndpointNetworkPolicies": null,
          "privateEndpoints": null,
          "privateLinkServiceNetworkPolicies": null,
          "provisioningState": null,
          "purpose": null,
          "resourceGroup": "example-resource-group",
          "resourceNavigationLinks": null,
          "routeTable": null,
          "serviceAssociationLinks": null,
          "serviceEndpointPolicies": null,
          "serviceEndpoints": null
        },
        "type": "Microsoft.Network/networkInterfaces/ipConfigurations",
        "virtualNetworkTaps": null
      }
    ],
    "location": "ukwest",
    "macAddress": null,
    "name": "example-nic",
    "networkSecurityGroup": {
      "defaultSecurityRules": null,
      "etag": null,
      "flowLogs": null,
      "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg",
      "location": null,
      "name": null,
      "networkInterfaces": null,
      "provisioningState": null,
      "resourceGroup": "example-resource-group",
      "resourceGuid": null,
      "securityRules": null,
      "subnets": null,
      "tags": null,
      "type": null
    },
    "primary": null,
    "privateEndpoint": null,
    "provisioningState": "Succeeded",
    "resourceGroup": "example-resource-group",
    "resourceGuid": "dce4915d-9be1-46ec-8c19-9e4d66c11ca5",
    "tags": null,
    "tapConfigurations": [],
    "type": "Microsoft.Network/networkInterfaces",
    "virtualMachine": null
  }
}

Creating a virtual machine (VM) response

{
  "fqdns": "",
  "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/example-vm",
  "identity": {
    "systemAssignedIdentity": "",
    "userAssignedIdentities": {
      "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example-managed-identity": {
        "clientId": "3f7da7e7-dda6-46de-82d2-8c82a8bd9d09",
        "principalId": "1b0be934-b012-416f-b1a8-79b8cc40ec30"
      }
    }
  },
  "location": "ukwest",
  "macAddress": "00-22-48-1A-2F-92",
  "powerState": "VM running",
  "privateIpAddress": "10.0.0.4",
  "publicIpAddress": "51.104.44.10",
  "resourceGroup": "example-resource-group",
  "zones": ""
}

---

Commands for additional resources

This section provides explanations, examples, and responses for the commands for the following Azure resources: - Resource groups - Storage accounts - Networks and subnetworks - Network security groups - Network security rules - Identities

If you already have any of these resources set up, you don't need to create another.

Creating a resource group

Explanation of the CLI command:

# Resource Group Creation
# - Name of Resource Group
# - Location for Resource Group

Example CLI command:

az group create \
--name example-resource-group \
--location UKWest

Example response:

{
  "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group",
  "location": "ukwest",
  "managedBy": null,
  "name": "example-resource-group",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

---

Creating a storage account

Explanation of the CLI command:

# Storage Account Creation
# - Name of Storage Account (If required - globally unique)
# - Select the resource group to associate the storage account with.

Example CLI command:

az storage account create \
--name matillionsaexample \
--resource-group example-resource-group

Example response:

{
  "accessTier": "Hot",
  "allowBlobPublicAccess": null,
  "azureFilesIdentityBasedAuthentication": null,
  "blobRestoreStatus": null,
  "creationTime": "2021-04-24T08:13:22.972959+00:00",
  "customDomain": null,
  "enableHttpsTrafficOnly": true,
  "encryption": {
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "requireInfrastructureEncryption": null,
    "services": {
      "blob": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2021-04-24T08:13:23.066708+00:00"
      },
      "file": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2021-04-24T08:13:23.066708+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "failoverInProgress": null,
  "geoReplicationStats": null,
  "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Storage/storageAccounts/matillionsaexample",
  "identity": null,
  "isHnsEnabled": null,
  "kind": "StorageV2",
  "largeFileSharesState": null,
  "lastGeoFailoverTime": null,
  "location": "ukwest",
  "minimumTlsVersion": null,
  "name": "matillionsaexample",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://matillionsaexample.blob.core.windows.net/",
    "dfs": "https://matillionsaexample.dfs.core.windows.net/",
    "file": "https://matillionsaexample.file.core.windows.net/",
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": "https://matillionsaexample.queue.core.windows.net/",
    "table": "https://matillionsaexample.table.core.windows.net/",
    "web": "https://matillionsaexample.z35.web.core.windows.net/"
  },
  "primaryLocation": "ukwest",
  "privateEndpointConnections": [],
  "provisioningState": "Succeeded",
  "resourceGroup": "example-resource-group",
  "routingPreference": null,
  "secondaryEndpoints": {
    "blob": "https://matillionsaexample-secondary.blob.core.windows.net/",
    "dfs": "https://matillionsaexample-secondary.dfs.core.windows.net/",
    "file": null,
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": "https://matillionsaexample-secondary.queue.core.windows.net/",
    "table": "https://matillionsaexample-secondary.table.core.windows.net/",
    "web": "https://matillionsaexample-secondary.z35.web.core.windows.net/"
  },
  "secondaryLocation": "uksouth",
  "sku": {
    "name": "Standard_RAGRS",
    "tier": "Standard"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": "available",
  "tags": {},
  "type": "Microsoft.Storage/storageAccounts"
}

---

Creating a network and subnetwork

Explanation of the CLI command:

# Network / Subnetwork Creation
# - Name of Network (If required)
# - Select the resource group to associate the network with.
# - Prefixes for network addresses (Tune to what is appropriate to your deployment)
# - Name of the subnetwork
# - Prefixes for subnet addresses (Tune to what is appropriate to your deployment)

Example CLI command:

CLI call
az network vnet create \
--name example-network \
--resource-group example-resource-group \
--address-prefixes 10.0.0.0/16 \
--subnet-name example-subnetwork \
--subnet-prefixes 10.0.0.0/24

Example response:

{
  "newVNet": {
    "addressSpace": {
      "addressPrefixes": [
        "10.0.0.0/16"
      ]
    },
    "bgpCommunities": null,
    "ddosProtectionPlan": null,
    "dhcpOptions": {
      "dnsServers": []
    },
    "enableDdosProtection": false,
    "enableVmProtection": null,
    "etag": "W/\"a1d5b34b-ebe8-42c5-a29f-4bd345876036\"",
    "extendedLocation": null,
    "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network",
    "ipAllocations": null,
    "location": "ukwest",
    "name": "example-network",
    "provisioningState": "Succeeded",
    "resourceGroup": "example-resource-group",
    "resourceGuid": "40dd0564-48b2-4f6b-9022-c8428e873d3a",
    "subnets": [
      {
        "addressPrefix": "10.0.0.0/24",
        "addressPrefixes": null,
        "delegations": [],
        "etag": "W/\"a1d5b34b-ebe8-42c5-a29f-4bd345876036\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/virtualNetworks/example-network/subnets/example-subnetwork",
        "ipAllocations": null,
        "ipConfigurationProfiles": null,
        "ipConfigurations": null,
        "name": "example-subnetwork",
        "natGateway": null,
        "networkSecurityGroup": null,
        "privateEndpointNetworkPolicies": "Enabled",
        "privateEndpoints": null,
        "privateLinkServiceNetworkPolicies": "Enabled",
        "provisioningState": "Succeeded",
        "purpose": null,
        "resourceGroup": "example-resource-group",
        "resourceNavigationLinks": null,
        "routeTable": null,
        "serviceAssociationLinks": null,
        "serviceEndpointPolicies": null,
        "serviceEndpoints": null,
        "type": "Microsoft.Network/virtualNetworks/subnets"
      }
    ],
    "tags": {},
    "type": "Microsoft.Network/virtualNetworks",
    "virtualNetworkPeerings": []
  }
}

---

Creating a network security group

Explanation of the CLI command:

# Network Security Group Creation
# - Name of NSG (If required)
# - Select the resource group to associate the nsg with.

Example CLI command:

az network nsg create \
--name example-nsg \
--resource-group example-resource-group

Example response:

{
  "NewNSG": {
    "defaultSecurityRules": [
      {
        "access": "Allow",
        "description": "Allow inbound traffic from all VMs in VNET",
        "destinationAddressPrefix": "VirtualNetwork",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Inbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowVnetInBound",
        "name": "AllowVnetInBound",
        "priority": 65000,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "VirtualNetwork",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      },
      {
        "access": "Allow",
        "description": "Allow inbound traffic from azure load balancer",
        "destinationAddressPrefix": "*",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Inbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowAzureLoadBalancerInBound",
        "name": "AllowAzureLoadBalancerInBound",
        "priority": 65001,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "AzureLoadBalancer",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      },
      {
        "access": "Deny",
        "description": "Deny all inbound traffic",
        "destinationAddressPrefix": "*",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Inbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/DenyAllInBound",
        "name": "DenyAllInBound",
        "priority": 65500,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "*",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      },
      {
        "access": "Allow",
        "description": "Allow outbound traffic from all VMs to all VMs in VNET",
        "destinationAddressPrefix": "VirtualNetwork",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Outbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowVnetOutBound",
        "name": "AllowVnetOutBound",
        "priority": 65000,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "VirtualNetwork",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      },
      {
        "access": "Allow",
        "description": "Allow outbound traffic from all VMs to Internet",
        "destinationAddressPrefix": "Internet",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Outbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/AllowInternetOutBound",
        "name": "AllowInternetOutBound",
        "priority": 65001,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "*",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      },
      {
        "access": "Deny",
        "description": "Deny all outbound traffic",
        "destinationAddressPrefix": "*",
        "destinationAddressPrefixes": [],
        "destinationApplicationSecurityGroups": null,
        "destinationPortRange": "*",
        "destinationPortRanges": [],
        "direction": "Outbound",
        "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
        "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/defaultSecurityRules/DenyAllOutBound",
        "name": "DenyAllOutBound",
        "priority": 65500,
        "protocol": "*",
        "provisioningState": "Succeeded",
        "resourceGroup": "example-resource-group",
        "sourceAddressPrefix": "*",
        "sourceAddressPrefixes": [],
        "sourceApplicationSecurityGroups": null,
        "sourcePortRange": "*",
        "sourcePortRanges": [],
        "type": "Microsoft.Network/networkSecurityGroups/defaultSecurityRules"
      }
    ],
    "etag": "W/\"02367ddb-b299-413e-bd71-9ac4ca20a535\"",
    "flowLogs": null,
    "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg",
    "location": "ukwest",
    "name": "example-nsg",
    "networkInterfaces": null,
    "provisioningState": "Succeeded",
    "resourceGroup": "example-resource-group",
    "resourceGuid": "def8de26-7836-4bb7-8737-9a55b27c862b",
    "securityRules": [],
    "subnets": null,
    "tags": null,
    "type": "Microsoft.Network/networkSecurityGroups"
  }
}

---

Creating a network security rule

Note

The networking policy can be locked down once the initial launch is established and working as expected based upon customer-specific networking policies. Matillion ETL needs 80 or 8080 (HTTP) and 443 or 8443 (HTTPS) traffic.

Explanation of the CLI command:

# Network Security Rule Creation
# Name of NSGR (If required)
# Select the resource group to associate the nsg with.
# Use the example nsg from earlier
# Assign a priority to the rule
# This can be changed to a CIDR range to keep things more secure
# Matillion uses either http or https

Example CLI command:

az network nsg rule create \
--name example-nsg-rule \
--resource-group example-resource-group \
--nsg-name example-nsg \
--description "Example description of rule for Matillion Traffic." \
--priority 1000 \
--source-address-prefixes '*' \
--source-port-ranges '*' \
--destination-address-prefixes '*' \
--destination-port-ranges '*' \
--access Allow \
--protocol Tcp

Example response:

{
  "access": "Allow",
  "description": "Example description of rule for Matillion Traffic.",
  "destinationAddressPrefix": "*",
  "destinationAddressPrefixes": [],
  "destinationApplicationSecurityGroups": null,
  "destinationPortRange": null,
  "destinationPortRanges": [
    "*"
  ],
  "direction": "Inbound",
  "etag": "W/\"b361c7f5-8a1c-497f-8bf5-fe4ad16e52db\"",
  "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourceGroups/example-resource-group/providers/Microsoft.Network/networkSecurityGroups/example-nsg/securityRules/example-nsg-rule",
  "name": "example-nsg-rule",
  "priority": 1000,
  "protocol": "Tcp",
  "provisioningState": "Succeeded",
  "resourceGroup": "example-resource-group",
  "sourceAddressPrefix": "*",
  "sourceAddressPrefixes": [],
  "sourceApplicationSecurityGroups": null,
  "sourcePortRange": null,
  "sourcePortRanges": [
    "*"
  ],
  "type": "Microsoft.Network/networkSecurityGroups/securityRules"
}

---

Creating a managed identity

Explanation of the CLI command:

# Managed Identity Creation
# Name of Managed Identity (If required)
# Select the resource group to associate the managed-identity with.

Example CLI command:

az identity create \
--name example-managed-identity \
--resource-group example-resource-group

Example response:

{
  "clientId": "3f7da7e7-dda6-46de-82d2-8c82a8bd9d09",
  "clientSecretUrl": "<redacted>"",
  "id": "/subscriptions/47b72173-e519-41b6-afd1-10018741e1d3/resourcegroups/example-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example-managed-identity",
  "location": "ukwest",
  "name": "example-managed-identity",
  "principalId": "1b0be934-b012-416f-b1a8-79b8cc40ec30",
  "resourceGroup": "example-resource-group",
  "tags": {},
  "tenantId": "09d19996-a185-4b6c-8332-37120f9bba10",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

---

Viewing and managing Matillion ETL resources

You can view your billable resources in the Azure Portal by clicking into Resource Groups, and then selecting your resource group that you created earlier.