Streaming to an Azure Blob Storage destination

Streaming pipelines can use Azure Blob Storage as a direct storage destination. This page describes the prerequisites, process, and other considerations of using Azure Blob Storage as a streaming pipeline destination.

Prerequisites

You need an Azure Blob Storage account that supports containers, such as BlobStorage, Storage, or StorageV2.
At the minimum, the role Reader & Data Access is required for sufficient permissions. The role should be applicable for the Azure Storage account in which your destination container is located.
The destination container needs to use an access key for authentication.
The agent container needs to use a Shared Key injected as an environment variable for authentication to the storage container.
If your storage account only allows access from selected networks, you need to allow certain IPs.

Destination connection

Refer to this section to complete the Destination connection section of the Create streaming pipeline wizard.

Container = string

This refers to the name of the Azure Blob Storage container you want to use as a destination. Find your container name in the Azure portal by navigating to Storage accounts and opening the account in which the container you want to use is located.

Prefix = string

Prefix is the name of the 'folder' or a location within the Azure Blob Storage bucket that all data for this streaming pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes.

Account Name = string

A unique name of the Azure Blob Storage account. The storage account provides a unique namespace for your Azure storage data.

Secrets Manager = drop-down

Choose the service you use to manage secrets.

Secret name = string

The name of the secret in your secret manager service that references your Azure Blob Storage credentials.

JDBC parameters and specific connection settings = column editor

This parameter is optional.

Specify any parameter:value pairs as part of an advanced connection. Click Save to finish adding parameters.

Click Connect.

Destination configuration

Refer to this section to complete the Destination configuration section of the Create streaming pipeline wizard.

Schema drift

Schema drift is supported for this destination. Read Schema drift to learn more.