Azure Blob Storage lifecycle policy for file management

By combining Azure Blob Storage lifecycle management with Azure Functions and Event Grid, you can automatically delete objects based on their tags from the new CDC shared jobs. The process may require some custom implementation, but it provides a flexible and automated solution for managing object deletion based on tags in Azure Blob Storage.

Here's how you can configure a solution to delete objects tagged with matillion_cdc_processed = true from the new shared jobs:

Enable lifecycle management for the storage account

Log in to the Azure Portal.
Navigate to Storage Accounts.
Choose a BlobStorage storage account.
In the left-hand sidebar, type "lifecycle" into the search bar and click Lifecycle management.
Click Enable.

Create an Azure Function

Create an Azure Function that will handle the deletion of objects with the specified tag matillion_cdc_processed = true.
In the Azure Function, implement the logic to query the blob container using the Azure Blob Storage SDK or Azure Blob Storage REST API.
Iterate through the objects in the blob container and check for the matillion_cdc_processed = true tag.
Delete the objects that meet the tag criteria.

Trigger the Azure Function with Azure Event Grid

Set up Azure Event Grid to monitor the blob container for new object creations or changes.
Configure an Event Grid subscription to trigger the Azure Function when a new object is created or changed in the blob container.

Add the tag during object upload

Ensure that the new shared jobs add the tag matillion_cdc_processed = true to the objects they upload to the blob container. Tags can be added using the Azure Blob Storage SDK or Azure Blob Storage REST API during the object upload process.

When the new shared jobs upload objects with the matillion_cdc_processed = true tag, Event Grid will trigger the Azure Function. The Azure Function will then delete the objects from the blob container based on the tag criteria.