Streaming to an Azure Blob Storage destination

Streaming pipelines can use Azure Blob Storage as a direct storage destination. This page describes the prerequisites, process, and other considerations of using Azure Blob Storage as a Streaming pipeline destination.

You should have arrived here by first reading Create a Streaming pipeline.

Prerequisites

You need an Azure Blob Storage account that supports containers, such as BlobStorage, Storage, or StorageV2.
At the minimum, the role Reader & Data Access is required for sufficient permissions. The role should be applicable for the Azure Blob Storage account in which your destination container is located.
The destination container needs to use an access key for authentication.
The agent container needs to use a Shared Key injected as an environment variable for authentication to the storage container.
If your storage account only allows access from selected networks, you need to allow certain IPs.

Destination configuration

Refer to this section to complete the Destination configuration section of the Create Streaming pipeline screen.

Container = string

The name of the Azure Blob Storage container you want to use as a destination. This must be the base directory of the Azure Blob Storage container, not a sub-directory.

Find your container name in the Azure portal by navigating to Storage accounts and opening the account in which the container you want to use is located.

Prefix = string

Prefix is the name of the "folder", a location within the Azure Blob Storage container that all data for this Streaming pipeline should be saved to. Assigning a unique prefix prevents naming conflicts and enables the reuse of the same container for multiple pipelines.

Account name = string

A unique name of the Azure Blob Storage account. The storage account provides a unique namespace for your Azure storage data.

Secrets Manager = drop-down

Choose the service you use to manage secrets.

Account key secret name = string

A named entry created in your chosen secrets manager which holds your Azure Blob Storage account credentials.

This completes the destination setup. The next groups of properties on this screen are for source setup and pipeline configuration.

Schema drift

Schema drift is supported for this destination. Read Schema drift to learn more.