May 2022 changelog
2022 - May 5th
New features and improvements
Batch pipelines
- New features and functionalities have been added to the Batch pipeline page:
- View alerts.
- View pipeline logs.
- View total rows moved.
- View pipeline settings.
- Batch pipeline control features have been introduced on the pipeline dashboard, including:
- Run Pipeline Now
- Enable/Disable Pipeline Schedule
- Set Sync preferences
- Support for the following Batch pipeline data sources:
Billing integration
Data Loader adds several new capabilities:
- Billing integration has been added for Batch and CDC data processing. Users will be charged through the Hub based on their usage. For more information, visit our pricing page.
Unified Batch and CDC pipelines
- Users can now create and manage Change Data Capture (CDC) pipelines in the same user interface as Batch pipelines.
CDC pipelines
- CDC is available for Oracle, PostgreSQL, and MS SQL sources. CDC data is stored in Amazon S3 or Azure Blob Storage.
- Matillion ETL Shared Jobs are available to ingest CDC data from Amazon S3 or Azure Blob Storage, transform the data, and load the data into Snowflake, Amazon Redshift, or Delta Lake on Databricks as final destinations. These configurable jobs can be scheduled to run in sync with your CDC agent process once you have set up your CDC pipelines.
- CDC pipelines can handle schema drift as change events in your source.