2024 changelogπ
Here you'll find the 2024 changelog for the Data Productivity Cloud. Just want to read about new features? Read our New Features blog.
Decemberπ
December 12π
DesignerNew features π
- Added the
Retrieve Runendpoint to the dbt Cloud Flex connector.
AgentImprovements π§
- The agent template for Azure now deploys with an environment type of "Workload profile - Dedicated" (was previously "Consumption Only"). This allows the agent to better facilitate complex networking, while keeping the ability for the agent to auto-update.
December 9π
CDC
- Updated the CDC agent version from
2.98.7to2.101.2. - For customers who may have experienced an out-of-memory issue with the agent, we have added a new advanced property
matillion.compact-historythat can be set totruein any affected pipeline definition. This will reduce the memory footprint of the agent by compacting the schema history of the pipeline. - For pipelines that stream data directly to Snowflake, the agent will create a new column
MTLN_CDC_LOAD_TIMESTAMPin each target table to store the load timestamp of each row. - Library updates and security fixes.
December 6π
APINew Endpoints π
The following endpoints have been added to the Data Productivity Cloud REST API:
Credit consumption
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/consumption | Get a breakdown of your credit consumption for flat-rated products |
| GET | /v1/consumption/etl-users | Get a breakdown of your credit consumption for ETL users |
December 5π
DesignerNew features π
- Added the new Skip on run action to the component action bar. This allows you to temporarily disable a component in a pipeline, without removing it. For more information, read Skip components.
New transformation component:
- Cortex Parse Document is a transformation component that uses Snowflake Cortex to extract content from a document on an internal or external stage in the form of an object that contains JSON-encoded objects as strings.
December 4π
DesignerNew features π
- Added the option to copy and paste pipelines and pipeline folders in Designer. For more information, read Managing pipelines and Copying pipeline folders.
December 3π
DesignerNew features π
- Added the option to rename pipeline folders in Designer. For more information, read Pipeline folders.
- Added a new Logging tab to the detailed view that appears when you click a task in the Task history tab. This shows a detailed log of each step in a task, as described in Logging.
StreamingImprovements π§
- Added a graph of number of rows changed over time to the Manage streaming pipelines dashboard.
Novemberπ
November 19π
DesignerNew features π
New connectors:
- Added a Kafka connector. The Kafka connector is an orchestration component that lets you load data from Kafka topics into either schemaless JSON or JSON schema objects. New to Kafka? Read Apache Kafka's Introduction.
- Added the following Flex connectors for developing data pipelines:
Git actions in Designer:
- Added a new Compare changes feature that allows you to compare your local changes to the previous commit.
- Added commit indicators that show how many commits your local branch is ahead of or behind the remote branch when committing, pushing, or pulling changes.
API:
- Added a GET pipeline executions endpoint, which returns a list of summarized pipeline executions.
November 15π
APINew Endpoints π
The following endpoint has been added to the Data Productivity Cloud REST API:
Pipeline execution
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/pipeline-executions | Returns a list of summarized pipeline executions |
November 13π
CDC
- Updated the CDC agent version from
2.98.4to2.98.7. - Enhanced the resilience of connecting the agent to the Data Productivity Cloud.
- Library updates and security fixes.
November 12π
DesignerNew features πImprovements π§
New connectors:
Component improvements:
- The Python Pushdown component now includes a
Warehouseparameter, allowing you to specify which Snowflake warehouse to run your Python scripts on. - The Detect Changes component is now available on Databricks projects.
November 7π
DesignerNew features πImprovements π§
- Maia (formerly Copilot) is now available for Databricks and Amazon Redshift projects (previously only available for Snowflake projects). In the Data Productivity Cloud, Maia introduces a new era of AI-driven data productivity tools, letting you prompt generative AI to create data pipelines, improving your efficiency in managing and processing data.
- Updated the S3 Unload component for Amazon Redshift projects. Previously, only CSV file types were supported. Now, the component also supports delimited, fixed with, and Parquet file types.
November 6π
CDC
- Updated the CDC agent version from
2.98.2to2.98.4. - Library updates and security fixes.
November 5π
Data Productivity CloudNew features π
- The Data Productivity Cloud is now available in Snowflake Partner Connect, allowing a quick and simple way to create an automatically configured Matillion account directly from a Snowflake instance.
Octoberπ
October 31π
DesignerNew features π
Connect your own Git repo:
- You can now connect your own Bitbucket repository to the Data Productivity Cloud. This feature lets you connect your Bitbucket repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
In case you missed our announcements in previous changelogs, we also support GitHub, GitLab, and Azure DevOps. Read Connect your own Git repo for an overview.
Agents:
- You can now refresh an agent's credentials, which generates a new client_secret to authenticate with the agent application in your infrastructure (Hybrid SaaS deployments only).
October 29π
DesignerNew features π
Components:
- JDBC Table Metadata to Grid is now available for Amazon Redshift projects.
- Truncate Table is now available for Databricks projects.
October 22π
CDC
- Updated the CDC agent version from
2.97.14to2.98.2. - Change behaviour of the Snowflake destination - if the Replication Type is
Copy TableorCopy Table with Soft Deletesand the source table does not have a primary key then the Replication Type will be set toChange Logfor that table. - Library updates and security fixes.
October 15π
DesignerImprovements π§
Component updates:
- Updated Python Pushdown. Added a
Snowflake Secretparameter, which is activated when an external access integration is selected. You can select Snowflake secrets and assign them to Snowflake variables, which can then be used in the Python script.
October 9π
CDC
- Updated the CDC agent version from
2.97.12to2.97.14. - Fixed Azure Blob Storage authentication flow.
October 8π
DesignerNew features π
New orchestration components:
- Azure Document Intelligence. This component uses the Azure AI Document Intelligence API to automate the extraction of text, handwriting, layout elements, and other key data from forms and documents. The output format can be either Markdown or text.
- Azure Speech Transcribe. This component uses the Azure Speech service to transcribe speech to text from an audio file in your specified Azure Blob location and Speech resource endpoint. Speaker diarization clarifies which speaker spoke and when, increasing the accuracy of your transcriptions.
New transformation components:
- Databricks Vector Search. This component lets you perform a search on an input table to find content that best answers specific questions, using vector embeddings to identify suitable answers within data located in your Databricks account. Only available for Databricks projects.
October 3π
DesignerNew features π
New orchestration components:
- Add Partition. This orchestration component lets users define the S3 directory structure for partitioned external table data. Amazon Redshift projects only.
- Create External Table. This orchestration component creates a table to reference data stored in an Amazon S3 bucket. The table references external data, meaning the table itself does not hold the data. Amazon Redshift projects only.
October 2π
CDCImprovements π§
- Updated the CDC agent version from
2.95.33to2.97.12. - Improved the process of agent shutdowns so that on clean shutdown the agent status shows as Stopped rather than Unknown.
- Library updates and security fixes.
Septemberπ
September 26π
DesignerImprovements π§
Projects:
- Snowflake projects now allow the choice of key-pair authentication when configuring environments, so a Snowflake private key can be used.
- Databricks projects now require a Databricks personal access token for authentication when configuring environments. Username/password has been deprecated by Databricks and can no longer be used.
Git version control:
- Additional options are available for Git reset, allowing a reset to your selected working state.
New orchestration components:
- Bash Pushdown. This component enables users running Full SaaS configurations (Matillion-hosted agents) to use SSH connections to run their Bash scripts in their own instances.
September 24π
DesignerNew features π
New orchestration components:
- Snowflake Vector Upsert. This component converts data into vector embeddings in a Snowflake table.
Component updates:
- You can now cross-query two databases with the Table Input and Multi Table Input transformation components for Amazon Redshift projects.
Designer UI:
- You can now reorder your pipeline tabs in Designer with click-and-drag.
September 12π
DesignerNew features π
New transformation components:
- Vector Search. This component uses Snowflake vectors to perform a retrieval augmented generation (RAG) search on an input table. This component is available in public preview for Snowflake projects.
Component updates:
- Added SFTP as a data input type in the File Iterator component.
- Table Iterator is now available on Amazon Redshift projects.
Designer:
- The Git Commit dialog now lists all uncommitted changes.
- Added a command palette to Designer. In its first phase, the following command is supported:
- Use
cmd + korctrl + kto activate a file explorer dialog. Alternatively, click the search icon in the upper-right corner of the UI.
- Use
September 10π
CDCImprovements π§
- Updated the CDC agent version from
2.95.27to2.95.33. - Library updates and security fixes.
September 5π
DesignerNew features π
New orchestration components:
- Amazon Textract. This component uses the Textract API to extract text, handwriting, layout elements, and data from scanned documents. You can optionally choose to include footers and page numbers in the extraction process.
- Amazon Transcribe. This component uses the Transcribe API to extract data from media and audio files and then converts that data into transcriptions.
Component updates:
- Chunk Text is now out of public preview and generally available for all Snowflake projects.
September 3π
DesignerNew features π
New components:
- Cortex Embed. This transformation component lets you convert English-language text into a Cortex vector embedding.
- Remove From Grid. This orchestration component takes in grid variable data and remove data based on the user's definition.
New connector:
- NetSuite SuiteAnalytics. This new connector is an orchestration component that uses NetSuite SuiteAnalytics Connect to retrieve data and load it into a table.
Augustπ
August 29π
DesignerNew features π
- You can now connect your own GitLab repository to the Data Productivity Cloud. This feature lets you connect your GitLab repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
- Updated Schedules so that you can now specify a time and date to enable pipeline schedules from.
- Updated the "Connect your own GitHub" workflow so that you can now view the list of authorized accounts or organizations after a successful authorization with GitHub.
August 27π
StreamingNew features π
- Added the following destinations to streaming pipelines:
August 20π
DesignerNew features π
- Shared pipelines are now available in the Data Productivity Cloud. Any pipeline can be saved as a shared pipeline, helping you to maintain consistency of core functionality over multiple different projects across the organization.
- Added the Run Shared Pipeline component, which allows you to call a shared pipeline from within another pipeline.
- Added improvements to environments. When you create an environment, you can now set a default access level for that environment. This access level can also be set on a per-user basis.
- As part of this change, all users in existing environments have been set to have Admin access by default, to avoid any disruption. Project admins can change a user's environment access.
August 16π
DesignerNew features π
- Added a Document AI Predict component, which invokes the Snowflake PREDICT function and allows users to call their Document AI models from the context of a Data Productivity Cloud pipeline. This component is available in public preview.
August 14π
CDCImprovements π§
- Updated the CDC agent version from
2.95.22to2.95.27. - CDC will now retain snapshot metrics once a pipeline snapshot completes and streaming begins. Note: these will be lost after an agent restart.
- Improved resilience of connection between Streaming agents and the platform.
- Library updates and security fixes.
August 8π
StreamingNew features π
- Streaming pipelines are now available in the Data Productivity Cloud. Streaming pipelines are a complete end-to-end solution for near-real-time data ingestion, allowing you to capture and synchronize data from source databases and write to a target data warehouse.
- Supported data sources are:
- Supported data targets are:
- Streaming pipelines are available on our Hybrid SaaS offering, meaning you'll need to create a streaming agent in your AWS or Azure infrastructure. Read Create an agent in your infrastructure to set up your agent.
DesignerNew features π
- Options to run a single component or run a pipeline starting from a specific component have been added to the pipeline canvas.
- Options to control the behavior of variables in a branching pipeline have been added.
- The Push local changes dialog now shows a list of commits to be pushed.
August 6π
DesignerNew features π
- Users can now terminate running pipelines from the Pipeline Observability Dashboard.
Julyπ
July 30π
Data Productivity Cloud APINew features π
- The Pipeline execution API now supports overriding scalar or grid variables when making a call. This enhancement allows for greater flexibility and customization during pipeline execution.
CDCImprovements π§
- Updated CDC agent version from
2.95.21to2.95.22.
July 25π
DesignerNew features π
- Added the Cloud Pub/Sub component, which allows messages to be published to a topic on Google Cloud Platform (GCP). Other applications can subscribe to these topics to receive such messages.
July 23π
DesignerNew features π
- Google Cloud Storage is now available for Snowflake projects in the following areas:
- Added the Google Cloud Storage Load component, which lets you load data stored on the Google Cloud Storage service into an existing Snowflake table.
- Added the Google Cloud Storage Unload component, which writes data from a Snowflake table or view into a specified Google Cloud Storage (GCS) bucket in your specified file format (CSV, JSON, or Parquet).
- Added Google Cloud Storage as a staging platform to all "Query" components (Facebook Query, Salesforce Query, Shopify Query, and so on).
- Added Google Cloud Storage as a source and destination to the Data Transfer component.
- Added Google Cloud Storage as a source and destination to the Excel Query component.
- Added Google Cloud Storage as a source and destination to the File Iterator component.
Note
To interact with Google Cloud Platform services, you'll need to set up GCP cloud credentials and associate your credentials with your environment.
CDCImprovements π§
- Updated CDC agent version from
2.95.1to2.95.21. - Fixed an issue where the thread cap for table writers could be reached when capturing changes on a large number of tables at once and cause the agent to stop streaming.
- The Oracle connector will now gracefully handle skipping transactions when the retention.ms property is configured.
- Library updates and security fixes.
July 18π
DesignerNew features πImprovements π§
- Updated the font across the Data Productivity Cloud to a more modern, readable font.
- Added HTML as a supported data format to the Chunk Text component.
- Added the ability to use multiple Azure Key Vault vaults to store secrets. When you add a new secret definition, you can choose which of your key vaults the secret is stored in.
July 16π
DesignerNew features π
- Added a JDBC orchestration component for the Hybrid SaaS offering. The JDBC component allows you to run queries on a JDBC-compatible data source that you have a driver for. You can upload your drivers to the Data Productivity Cloud, and so connect to any sources that Matillion doesn't provide drivers for as standard.
- Added a Chunk Text orchestration component, which performs pushdown text chunking using a Python user-defined function (UDF) in Snowflake via the computational power of your Snowflake warehouse. You can choose text or Markdown as your data format.
- Added the following Flex connectors for developing data pipelines:
- Added a Refresh Table orchestration component, which invalidates cached entries of a specified table or view, including both data and metadata. Databricks projects only.
July 4π
DesignerNew features π
- Added a new Postgres Vector Upsert component, which lets you convert text data stored in your cloud data warehouse into embeddings and then store these embeddings as vectors in your Postgres vector database.
- Added Postgres as a vector database option when retrieval augmented generation (RAG) is enabled on the following components:
- You can now use Databricks volumes as your staging platform when using our Query components (Facebook Query, Salesforce Query, and so on).
July 2π
DesignerNew features πImprovements π§
- Added support for AWS Guardrails to the Amazon Bedrock Prompt component.
- Removed
Pinecone EnvironmentandPinecone Project IDparameters from AI components where users can interact with Pinecone as these parameters are no longer required. - The following new components that support Databricks LLM operations are available for Databricks projects in public preview:
Juneπ
June 25π
DesignerNew features π
- Connecting your own Azure DevOps repo to Data Productivity Cloud projects is now generally available. This feature lets you connect your Azure DevOps repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
- Added the following Flex connectors for developing data pipelines:
- Added an SAP ODP orchestration component, for connecting directly through SAP ODP to access available data sources in SAPI and ABAP CDS views.
- Added the ability to upload external drivers to allow you to enable third-party connectors that don't have a driver supplied by default in a Hybrid SaaS configuration, such as SAP ODP.
- Added support for the OAuth 2.0 client credentials authentication type in Designer when using custom connectors.
- Updated data sampling for orchestration components to include a drop-down menu where you can set the number of rows sampled, for example 1, 10, 250, 1000. The default is 25 rows.
June 20π
CDCImprovements π§
- Updated the CDC agent version from
2.93.8to2.95.1. - Added an option to map date and time source columns to the corresponding Snowflake data types when using Snowflake as a target.
- Changed the
MTLN_CDC_LAST_COMMIT_TIMESTAMPcolumn when using Snowflake as a target with the Change Log transformation type. - Library updates and security fixes.
June 18π
DesignerNew features π
- Added a component action bar that will be displayed when you click on a component in the canvas. The action bar currently supports copying the component and deleting the component.
- Updated data sampling for transformation components to include a drop-down menu where you can set the number of rows sampled, for example 1, 10, 250, 1000. The default is 25 rows.
- Added an Append To Grid component, which lets you append or prepend data to a pre-existing grid variable in the Data Productivity Cloud. Running this component can add rows to existing columns, but doesn't add columns to the grid.
- Added the following Flex connectors for developing data pipelines:
June 10π
DesignerNew features π
- Project administrators can now delete projects.
June 6π
DesignerNew features π
- Added support for GPT-4o to the OpenAI Prompt component. When this model is selected, the Image input type is available. Images should be in either Base64 format or a direct, public URL of the image. If you're feeding images in Base64 format, you will need to add
data:image/{format_of_your_image};base64,before the encoded image. - Added a Volume to Delta Table orchestration component for Databricks, which lets users transfer data from a pre-existing volume in Databricks in to a Delta Lake table without replacing or deleting any existing data.
- Added a Convert String To Struct transformation component, which takes a string input and converts it to a struct data type.
- Renamed the Extract Nested Data component to Extract Structured Data for Databricks. This transformation component unpacks arrays of structured data into columns and rows of data in a table. Any existing pipelines will still display an Extract Nested Data component.
- The Query Result To Grid component is now available for Databricks projects.
- Added the following Flex connectors for developing data pipelines:
- A user's account ID is now visible in the user menu (top right of the Data Productivity Cloud UI), to help users easily identify their account.
Mayπ
May 31π
Custom ConnectorNew features π
- Added a Script paging option for connector pagination. Paging scripts give you complete control over response pagination in Custom Connector.
May 30π
DesignerNew features π
- Maia (formerly Copilot) for the Data Productivity Cloud is now available in public preview for transformation pipelines in Snowflake projects. Maia lets you prompt generative AI to create data pipelines, improving your efficiency in managing and processing data. Read our docs to get started:
- The Query Result to Scalar component is now available for Databricks projects. This orchestration component allows you to query a table and return a scalar value. This value can then be mapped to a project or pipeline variable for use in other pipeline components.
May 23π
DesignerNew features π
- Added Amazon Bedrock as an embedding provider and added Titan Embeddings G1 - Text as an embedding model for the following AI components:
- Added the ability to configure agents to work with a proxy server in AWS and Azure environments.
May 21π
DesignerNew features π
- The Snowpark Container Prompt is now available in public preview. This orchestration component lets you host your large language models within your Snowpark Container Services.
- The Pinecone Vector Upsert component is now available for Databricks and Amazon Redshift projects.
- Added the following Flex connectors for developing data pipelines:
- The Data Productivity Cloud agent for Azure is now available in public preview. This lets you run a Hybrid SaaS configuration with the Matillion agent hosted in your own Azure infrastructure.
May 16π
DesignerNew features π
- The AI prompt components are now available for Databricks projects:
- The following new components that support Snowflake Cortex LLM operations are available for Snowflake projects:
May 14π
DesignerNew features π
- The AI prompt components are now available for Amazon Redshift projects:
- Added two new connectors:
- Gmail, which lets you query the Gmail API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
- SurveyMonkey, which lets you query the SurveyMonkey API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
- Added the following components for Snowflake projects:
- Azure Queue Storage Message lets you post a message to Azure Queue Storage. Other applications can read those messages and perform further processing.
- JDBC Table Metadata to Grid takes the metadata from a JDBC table and uses this data to populate a grid variable.
- Table Metadata to Grid takes the metadata from a table and uses this data to populate a grid variable with data.
- The Git menu can now be accessed by clicking the active branch.
- The Data Transfer component now supports SFTP as a target.
May 10π
CDCImprovements π§
- Updated CDC agent version from
2.93.1to2.93.8. - Improved messaging and logging for some scenarios.
- Library updates and security fixes.
May 2π
DesignerGA release π
- The Python Pushdown component is now generally available for Snowflake users. This component lets you execute a Python script using the Snowpark service in your Snowflake account.
May 1π
DesignerNew features π
- Amazon Redshift is now generally available as a cloud data warehouse in the Data Productivity Cloud.
Aprilπ
April 30π
DesignerNew features πImprovements π§
- Added a Microsoft Exchange component, which lets you query the Microsoft Exchange API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
- Added a ServiceNow Query component, which lets you query the ServiceNow API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
- Added logging to Custom Connector and to Flex and custom connectors used in Designer. You can choose one of five logging options: Error, Warn, Info, Trace, or Debug.
- Improved component sampling to enable column resizing.
April 24π
CDCImprovements π§
- Updated Streaming agent version from
2.92.3to2.93.1. - Removed a number of unnecessary dependency overrides and updated some minor dependencies.
April 18π
DesignerNew features π
- Added the following Flex connectors for developing data pipelines:
April 17π
DesignerNew features π
- Maia (formerly Copilot) is now available as an individual assignable service in the Edit User menu.
- Super Admins can enable and disable Maia access for all users in an account.
- User Admins can enable and disable Maia access for all User Admins and Users in an account (not the Super Admin).
April 11π
DesignerImprovements π§
- Added an improvement to connecting a GitHub repository to a project. Now, when you connect an existing repository, any active branches will be cloned into Designer and selectable from the Branches tab. Previously only the
mainbranch was cloned. If any of the active branches in your repository only include files that are not Data Pipeline Language (DPL) files, those branches will appear as empty while in Designer until you create a pipeline.
April 4π
DesignerNew features π
- Added the following Flex connector for developing data pipelines:
April 2π
CDCImprovements π§
- Updated CDC agent version from
2.92.0to2.92.3 - Library updates and security fixes
- Improved agent logging around data handling edge cases to help with debugging
Marchπ
March 28π
DesignerNew features π
- Added the ability to export pipelines and import them into a new project.
- Added the following connectors for developing data pipelines:
- Databricks connector
- Mandrill flex connector
March 20π
CDCImprovements π§
- Updated CDC agent version from
2.91.5to2.92.0. - Added optional
STREAMING_AGENT_DEBUG_DISABLE_COMPRESSIONEnvironment Variable to disable compression when uploading to Snowflake Staging area. - Library updates and security fixes.
March 19π
DesignerNew features π
- The following AI components are now generally available to all Data Productivity Cloud users in Designer:
- Prompt components:
- Vector database operations:
- AI Note is also now generally available, which lets you invoke generative AI to annotate your data pipelines using notes.
- The project details page in the Data Productivity Cloud now displays project metadata including the agent type, the cloud data platform, and the type of Git repository (Matillion hosted repository or customer's GitHub repository).
March 14π
DesignerNew features π
- The ability to connect your own GitHub repo to the Data Productivity Cloud is now generally available. This feature lets you connect your GitHub repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
APINew Endpoints π
- The Data Productivity Cloud API is now generally available and includes the following endpoints:
Projects
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/projects | List all projects |
Environments
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/projects/{projectId}/environments | Lists all environments in a project |
Pipelines
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/projects/{projectId}/published-pipelines | Lists all published pipelines in a project |
Pipeline execution
| Method | Endpoint | Description |
|---|---|---|
| POST | /v1/projects/{projectId}/pipeline-executions | Execute a published pipeline |
| GET | /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId} | Returns the status of a pipeline |
| PATCH | /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId} | Terminate a pipeline |
| GET | /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}/steps | Represents a single execution of a pipeline triggered from a step of another pipeline |
March 13π
DesignerNew features π
- Added a Text mode option to components and grid variables, to facilitate rapid input and editing of multiple properties.
CDCImprovements π§
- Updated CDC agent version from
2.90.16to2.91.5. - Increased the Streaming agent JVM memory allocation to 60%.
- Fixed a bug with the Direct to Snowflake flow, in which a change to the scale of a numeric column of a table in the source database would not trigger a schema change in the target Snowflake table.
- Library updates and security fixes.
March 12π
DesignerNew features π
- Added support for lineage for transformation pipelines. With lineage, you get a visual representation of each transformation, which can make understanding the evolution of your data faster and easier. Lineage is currently in public preview and includes:
- Transformation lineage at runtime.
- Table-level lineage.
- Column-level lineage.
- Table lists.
- Table metadataβcolumn information and data types.
- Added the following Flex connectors for developing data pipelines:
March 5π
DesignerImprovements π§
- Updated the AI Note feature with the ability to refine notes. When you click Refine when creating a note, you can elaborate on the note (adding detail), shorten the note's length and detail, or regenerate the note.
- Added a
Load Strategyparameter to connectors that support storage-only outputs. Choose between appending files in folder and overwriting files in folder, based on exact structure matching.
Februaryπ
February 29π
DesignerNew features π
- Added the ability to access the Snowflake
Sessionobject from the Python Pushdown component, to allow reading from and writing to the Snowflake database from within a Python script. - Added the following Flex connectors for developing data pipelines:
- Added Azure Blob Storage as a data input type for the Excel Query component.
- Added Azure Blob Storage as a source and target type for the Data Transfer component.
CDCImprovements π§
- Updated CDC agent version from
2.90.16to2.90.35. - Logging improvements.
- Library updates and security fixes.
February 28π
DesignerNew features π
- Added the Query Result to Grid component, which queries a table and returns rows of data that are loaded into a predefined grid variable.
February 27π
DesignerNew features π
- Improved user interface for easier pipeline creation. Read Designer UI basics for details.
February 21π
CDCNew features π
- Added Snapshot Overview to display improved metrics when a pipeline is snapshotting.
February 20π
DesignerNew features π
- Added Azure Blob Storage as a data input type for the File Iterator component.
February 9π
DesignerNew features π
- Added support for grid variables in pipelines.
- Added a Grid Iterator component for working with grid variables.
February 8π
DesignerNew features π
- Added the Mailchimp component, which lets you query the Mailchimp API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
- Added the following Flex connector for developing data pipelines:
- Added a toggle to the Schedules tab where users can easily enable or disable their schedules.
February 6π
DesignerNew features π
- Added the Azure SQL component, which lets you run an SQL query on an Azure SQL database and copy the results to a table. You can then use transformation components to enrich and manage the data in permanent tables.
February 1π
DesignerNew features π
- Databricks is now a generally available cloud data platform for the Data Productivity Cloud.
- The following components have been added to the Data Productivity Cloud for Databricks:
CDCImprovements π§
- Updated CDC agent version from
2.89.4to2.90.16. - Library updates and security fixes.
Januaryπ
January 25π
DesignerNew features π
- Added a Commands for dbt Core component. This component empowers you to run dbt commands as part of your orchestration pipelines.
Data Productivity CloudImprovements π§
- Improved the β° navigation menu, adding links to manage your custom connectors and agents.
January 23π
DesignerNew features πImprovements π§
- Added a Python Pushdown component to Designer. This component lets you execute a Python script using the Snowpark service in your Snowflake account. This component is currently available as a public preview.
- Improved the schedule feature to include a Standard mode and an Advanced mode.
- Standard mode lets you define intervals with the units Days (default), Weeks, Hours, and Minutes.
- Advanced mode retains the Cron expression scheduling feature.
January 17π
CDCNew features π
- Added option to configure or disable the table name prefix when using Snowflake as a target.
January 15π
DesignerNew features π
- Added the following Flex connector for developing data pipelines:
January 11π
DesignerNew features π
- Added the following Flex connector for developing data pipelines:
Improvements π§
- Fixed an issue where using the Rename component would reorder columns.
- Fixed an issue where using the Convert Type component would reorder columns.
January 9π
DesignerNew features π
- Added the following Flex connector for developing data pipelines:
January 5π
DesignerNew features π