Roles & permissions (Azure)

For Matillion ETL to detect Azure Blob Storage containers, additional credentials may be required. Matillion ETL can either use instance credentials or user defined credentials, the latter of which requires you to gather credentials from your Azure account, and enter them into Matillion ETL.

Matillion ETL also requires read access to your Azure subscription, which can be enabled by following the process below.

Note

Each credential connection is specific to one Azure subscription. If you wish to access resources in multiple subscriptions, you will need to create multiple apps with access to the required resources in a specific subscription. If you give access to credentials in multiple subscriptions, only resources from one subscription will be available.

Using identities (instance credentials)

To use instance credentials, your Matillion ETL virtual machine (VM) must already be set up. If you wish to use a user-assigned managed identity (as opposed to a system-assigned managed identity, which is unique to the VM) then you will need to search for the Managed Identities blade on the Azure Portal and set one up.

If you haven't already done so, please follow the steps below:

Log in to the Azure Portal.
Click Virtual machines, or click More services to locate it from the full list of Azure services.
Select the virtual machine containing your instance then select Identity from the center-left panel.
In the Identity panel, click the User assigned tab at the top, and click + Add.
You will see a list of user-assigned managed identities on the Add user assigned managed identity panel to the right. Select your Subscription from the drop-down menu and search for the user identity you want to add. Select one or more identities that you want to assign to the resource, then click the Add button.
Return to the Azure portal and select the Storage accounts tile, then select the Blob Storage account that you want Matillion ETL to access.
Once you've selected your Blob Storage account, click Access control (IAM) from the center-left panel.
Select + Add at the top of the page and then Add role assignment from the small menu below the + Add button.
The Add role assignment page will be displayed on the Role tab. Select Storage Account contributor from the list of Job function roles and click Next, which will take you to the Members tab.
For user-assigned identities:
1. Select User, group, or principle in the Assign access to field.
2. Select +Select members to open the Select members panel at the right. Search for and select the users you want to assign access to, then click Select.
For system-assigned identities:
1. Select Managed identity in the Assign access to field.
2. Select your Subscription and choose the relevant virtual machine from the Managed identity drop-down menu.
3. Search for and select the users you want to assign access to, then click Select.
Click Review + Assign or Next at the bottom of the window to go to the Review and assign tab to review your selections.
Click Review + Assign again to confirm and save your selections.

Using apps (user defined credentials)

To add storage accounts to Matillion ETL, we must first create an app. This requires a user with the Application administrator directory role.

Log in to the Azure Portal.
Click App registrations, or click More services to locate it from the full list of Azure services.
Select + New registration at the top of the page.
On the Register an application page, provide the following details:
- Name: A clear and descriptive name.
- Supported account types: Tick the checkbox next to Accounts in any organizational directory (Any Azure AD directory – Multi-tenant) and personal Microsoft accounts (e.g. Skype, Xbox).
- Redirect URI (optional): Select Web in the drop-down field and paste the Callback URL copied from the Manage OAuth window in Matillion ETL. Note that although the page states this field is optional, you must complete it.
Click Register.
The browser will redirect to the Overview page on the app's newly created dashboard. From here, copy the credentials to the right of Application (client) ID and Directory (tenant) ID as they will be required for use in Matillion ETL.

Note

If you haven't already, add a Blob Storage resource to this storage account.
Return to the storage account and select the Blob account from the list of storage accounts, and click Overview, then click Containers.
To add a new container, click the + Container button at the top. Give it a name and access level, and click Create.

Now, when you import details from your app into Matillion ETL, your client will be able to discover the buckets that the app has ownership of. To use this app in your Matillion ETL client, see the next section.

Gathering Azure credentials

For a Matillion ETL instance to take advantage of Azure resources, you are required to provide credentials in the form of a Tenant ID, which is unique to your Azure account, and a Client ID and Secret Key, which are taken from a registered app.

Tenant ID

From the Azure Portal, browse to Azure Active Directory, then click Properties from the sidebar on the left, and copy the Tenant ID.

Client ID

Browse from the Azure Portal to Azure Active Directory, then click App Registrations, and select an app that's associated with your desired storage accounts. Copy the Application (client) ID, for your Client ID.

Secret Key

Browse from the Azure Portal to Azure Active Directory, then click App Registrations. Select the App associated with your desired storage accounts.
Next, click Certificates & secrets on the sidebar on the left. Then, in the Certificates & secrets window, click + New client secret, situated underneath the Client Secret section.
The Add a client secret pop-up window will appear at the right of the page. Provide details for the following fields:
- Description: Provide a description of the client secret.
- Expires: Tick the checkbox next to when the client secret should expire, then click Add.
Returning to the Certificates & secrets window, the new client secret will appear on the list in the Client secrets section. Copy the Value of the relevant client secret, as you will need to refer to this in Matillion ETL.

Note

Make sure to copy the client secret value right away as it may appear only once.
Additionally, when copying the client secret value, some browsers may add a space to the end of the string. Watch out for this as it will cause the credentials to fail.

Test credentials

You can test credentials by clicking Project → Manage Credentials.

If instance credentials are available, you can test them by clicking the Test button at the top of the dialog. This will check access to any services that Matillion ETL uses. You may continue even if the tests fail; however, some features of the product may be impaired or non-functional without appropriate credentials.

Azure User Defined Credentials are listed by name under their respective tabs. New user defined credentials can be added by using the + button, edited using the pencil icon, or deleted using the X icon by each entry. When creating or editing credentials, a Test button is made available in the new dialog to check the details before finalizing your credentials.

To learn more, read Manage credentials.

If you need help with connecting to Azure Blob Storage from Matillion ETL, visit the page Troubleshooting connections to Azure Blob Storage for more information.

Accessing the subscription

Matillion ETL requires read access to your Azure subscription. To enable this access, follow this procedure.

Log in to the Azure Portal.
Click Virtual machines. You may need to click More services to locate the option in the full list of Azure services.
Select the virtual machine containing your Matillion ETL instance.
Click Access control (IAM).
Click Add → Add role assignment.
Click Reader to select it, then click Next.
Click + Select members.
In the Select members panel, click the name of the Matillion ETL virtual machine to select it, then click Select.
Click Review + assign.

Now restart the Tomcat service from inside the Matillion ETL user interface, by clicking Restart Server on the Admin menu.