Connect to Azure Blob
Matillion CDC can load data from your pipelines into Azure Blob storage. Follow the steps on this page to configure Azure Blob as a destination in Matillion CDC.
Azure Blob Storage prerequisites
To start using Azure Blob Storage as a destination, some requirements must be met. These prerequisites ensure that a working connection can be set up to successfully transfer data to your Azure Blob storage container.
- Your destination should be an Azure Blob storage account that supports containers, such as
BlobStorage
,Storage
, orStorageV2
. - At a minimum, the roles
Reader
&Data Access
are required for sufficient permissions. These roles should be applicable for the Azure Blob storage account in which your destination container is located. - The destination container needs to use an access key for authentication.
- The agent container needs to use a Shared Key injected as an environment variable for authentication to the storage container. For more information about environment variables, read Environment Variables.
- If your storage account only allows access from selected networks, you need to allow certain IPs.
- For Snowflake, configure secure access to data files stored in an Azure storage integration. To learn more, read here.
Select destination
- After you configure the source during a CDC pipeline creation, you'll be directed to choose a destination you would like to load your data into.
- On the Choose destination page, select Azure Blob.
Configure Azure Blob connection settings
Specify the following settings in the Connect to Azure Blob destination page:
Property | Description |
---|---|
Container | This refers to the name of the Azure Blob storage container you want to use as a destination. Find your container name in the Azure portal by navigating to Storage accounts and opening the account in which the container you want to use is located. |
Prefix | Prefix is the name of the 'folder' or a location within the Azure Blob storage bucket that all CDC data for this pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes. |
Account Name | A unique name of the Azure Blob storage account. The storage account provides a unique namespace for your Azure storage data. |
Azure Key Vault | When you create a storage account, Azure generates storage account access keys for that account. These keys can be used to authorize access to data in your storage account. Find your secret key in the Azure Portal by navigating to Azure Storage > select the Storage Account → Security + networking → Access keys. The access key to storage account will need to be saved in Azure key vault as a secret entry. |
Test connection
You can test your connection by clicking Test connection. If the test is successful, click Test and Continue.