2023 changelogπ
Here you'll find the 2023 changelog for the Data Productivity Cloud. Just want to read about new features? Read our New Features blog.
Decemberπ
December 12π
DesignerNew features π
- Added Google Cloud Storage as a "direct to storage" option when using Flex connectors.
- Added REST API service support to SharePoint Query. This service requires OAuth authentication.
- Added the following Flex connector for developing data pipelines:
December 11π
CDCImprovements π§
- Updated CDC agent version from
2.88.3to2.89.4. - Added option to disable the table name prefix when using Snowflake as a target.
- Fix support for Snowflake when using the
QUOTED_IDENTIFIERS_IGNORE_CASEproperty. - Additional logging when unable to connect to the Matillion Platform.
December 8π
DesignerNew features π
- Added support for "connect your own GitHub repo" to the Data Productivity Cloud. This feature lets you connect your GitHub repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
December 7π
DesignerNew features πImprovements π§
- New Update Scalar component. This component allows you to change the value of any pipeline variable or project variable during pipeline execution, to a new value decided at the time the component is configured.
- New Print Variable component. This component prints the value that a pipeline or project variable has at that specific point in the pipeline execution.
- Added the following Flex connector for developing data pipelines:
- Improved the Git Merge workflow to let users merge changes that have been pushed to a remote branch into your local branch. This improvement empowers users to validate and test changes before pushing merged changes to the remote repository.
Novemberπ
November 28π
DesignerNew features π
- Added Azure as a staging location to all connectors.
- Added the following Flex connector for developing data pipelines:
Data Productivity CloudNew features π
- Added a Recent pipelines menu to the Data Productivity Cloud UI. The Data Productivity Cloud will dynamically display your most recent data pipelines, with identifying metadata such as the project platform, Git branch, and pipeline folder name.
November 21π
DesignerNew features π
- Added the following component for developing data pipelines:
November 14π
DesignerNew features π
- Added the following Flex connector for developing data pipelines:
November 8π
BatchNew features πImprovements π§
- Improved the Set frequency page of the batch pipeline wizard, including adding an advanced mode with support for Quartz cron expressions.
CDCImprovements π§
- Fixed issue where data would not load into tables for pipelines with Snowflake as a destination.
November 7π
DesignerNew features π
November 2π
DesignerNew features π
- Added the Workday Custom Reports connector to Designer. This component uses the Workday RaaS (Reporting as a Service) API to retrieve and store dataβsuch as employee, financial, and business-related dataβfrom Workday, to be loaded into a table.
November 1π
CDCNew features πImprovements π§
- Added Copy Table with Soft Deletes as a transformation type for pipelines with Snowflake as a destination.
- Updated CDC agent version from
2.87.8to2.88.2. - Improved upload speed when writing to a Snowflake staging table.
Octoberπ
October 31π
DesignerNew features π
- Added the following connectors for developing data pipelines:
- Added Azure Storage as a storage destination for the following connectors:
- ActiveCampaign
- Amplitude
- Anaplan
- Braze
- Brevo
- Chargebee
- CircleCI
- Concord
- Confluence
- Datadog
- Delighted
- Eventbrite
- Freshdesk
- Gong
- Klaviyo
- LaunchDarkly
- PagerDuty
- PayPal
- Pendo
- Productboard
- Recurly
- Slack
- Smartsheet
- Snyk
- TikTok
- Toggl
- Twilio
October 27π
DesignerNew features π
- Added the Anaplan connector.
October 26π
DesignerImprovements π§
- Updated the Add Components panel with clickable filter buttons under the search bar. Click a filter button to only display components tagged for that filter. You can activate multiple filter buttons.
- Updated the Add component call-to-action (CTA) dialog to include an output connection setting. When adding a component to the canvas, you can now confirm whether that component should be run:
- On success (green connection line).
- Unconditionally (grey connection line).
- On failure (red connection line).
- You can now sample Query components in orchestration pipelines. To do this:
- Click on a validated Query component in your orchestration pipeline, e.g. Salesforce Query.
- Click the Sample tab.
- Click Sample data.
- Added row count when sampling components in a transformation pipeline. To view the total rows:
- Click on a validated transformation component, e.g, Table Input.
- Click the Sample tab.
- Click the refresh button.
Custom ConnectorNew features π
- The ability to import pre-existing connectors using the Custom Connector import feature. This feature supports two types of connectors:
- Custom connectors that have been previously exported.
- Matillion ETL connectors.
October 18π
DesignerNew features π
-
Added support for Flex connectors to Designer. In the Data Productivity Cloud, Flex connectors let you connect to a curated set of endpoints to load data. 20 Flex connectors are initially available for Designer:
- Amplitude
- Brevo
- CircleCI
- Concord
- Confluence
- Datadog
- Delighted
- Freshdesk
- Klaviyo
- LaunchDarkly
- PagerDuty
- Pendo
- Productboard
- Recurly
- Smartsheet
- Snyk
- TikTok
- Toggl
- Twilio
October 12π
DesignerNew features π
- Added the Assert View component, which lets you verify that certain conditions are true of a view, or otherwise stop the query.
October 10π
DesignerNew features πImprovements π§
- Updated the Add Components panel to un-nest components. We believe this will make finding the component you're looking for much easier. Components are listed alphabetically and include a descriptor keyword, e.g. "Connectors" or "Flow". You can use these descriptors to filter out components that don't match that keyword. For example, typing "flow" in the search bar will return only the Flow components (And, End Failure, End Success, If, Or, Retry, Restart). The search also includes a number of synonym keywords to help filter components. For example, typing "logic" into the search bar will also return the Flow components.
- Updated the pipeline canvas to include a + call-to-action button. When you select a component on the canvas, the + button will appear to the right. Clicking this button will open the Add component dialog. This dialog works similarly to the Add Component panel.
- Components are listed alphabetically.
- Descriptor and synonym keywords help you refine your search.
- Clicking a component in this dialog will add that component to the canvas in place of the + button and close the dialog. The new component will be connected to the previously selected component.
- This newly added component will become the selected component, meaning the + button will shift to the right.
- Added the Table Update component, which lets you update a target table with a set of input rows. The rows to update are based on matching keys.
October 5π
DesignerImprovements π§
- The Git Commit changes and push option has been split into two separate operations: Commit and Push local changes.
Septemberπ
September 20π
DesignerImprovements π§
- The Data Transfer component now supports SFTP as a source type.
September 19π
DesignerNew features πImprovements π§
- Oracle is now provided as a driver and a supported database type in the Database Query and RDS Query components.
- Added support for multi-line secret values for Full SaaS, Matillion hosted projects when creating secrets and secret definitions.
- Added an Add schedule call-to-action button to the Designer UI to easily navigate from Designer to the Create a new schedule menu.
- Added the ability to associate newly created cloud credentials to an environment in a single workflow. You can associate credentials from multiple providers, but only one set of credentials for each cloud provider. For example, you can associate both AWS and Azure credentials, but not two different AWS credentials.
- Added a "hard reset" functionality to Git in Designer. Hard reset lets users reset their branch to the last local commit.
September 18π
DesignerNew features π
- Added the following component:
September 14π
Data Productivity CloudNew features πImprovements π§
- Registered Data Productivity Cloud customers with Account and User Administrator privileges can edit the Account name and Subdomain name for their account.
- Data Loader Batch pipeline runs are now observable from the Pipeline Observability dashboard.
- Pipeline error messages are now displayed at the top of the Pipeline run details page.
September 12π
CDCNew features πImprovements π§
- Updated CDC agent version from
2.83.5to2.87.8. - Added "Copy Table With Soft Deletes" as a transformation type for pipelines with Snowflake as a destination.
- Fixed an issue where Snowflake role names could not contain non-standard characters.
- Improved connection reuse for Snowflake as a destination.
September 7π
DesignerNew features π
- Added the following components:
Augustπ
August 31π
DesignerNew features π
- Added the ability to store cloud provider credentials for authenticating with AWS and Azure.
- The following components are now supported in a Matillion Full SaaS environment (previously were Hybrid SaaS only):
- Added the ability to use an S3 location as the stage platform on all query (load) components in a Matillion Full SaaS environment (previously was a Hybrid SaaS function only).
CDCNew features πImprovements π§
- Updated CDC agent version from
2.83.5to2.87.1. - Added on-demand snapshots for pipelines with Db2 for IBM i as a source.
- Added Change Log as a transformation type for pipelines with Snowflake as a destination.
August 30π
CDCImprovements π§
- Added the in-client Help bar with contextual documentation to the "Create Pipeline" pages.
August 24π
DesignerNew features πImprovements π§
- Added the following components:
- Replaced the + button with an Add button that opens a context menu for creating pipelines and folders.
August 23π
CDCImprovements π§
- Updated CDC Agent version from
2.81.12to2.83.5. - Fixed issue where on-demand snapshots could fail to trigger.
- Additional validation of the stage format for the Snowflake destination.
- Added
rs_idandssnmetadata fields to the Oracle change records.
August 22π
DesignerNew features π
- Added the following components:
- Added a Getting started wizard for users who load into an instance of Designer with no existing pipelines.
August 16π
DesignerNew features π
- Added the following component:
August 9π
DesignerNew features π
- Added new item Invite your Teammates to the help widget's Task checklist, to facilitate collaboration.
August 3π
DesignerNew features πImprovements π§
- Added the following components:
- Added the ability to organize pipelines into folders.
- Tooltips added to the Designer UI for type of pipeline and type of component.
- The Designer product tour has been replaced with a checklist to help users with creating a pipeline, adding components, and running a pipeline.
CDCNew features πImprovements π§
- Updated CDC Agent version from
2.81.6to2.81.12. - Added Direct to Snowflake as a target. Read Connect to Snowflake to learn more.
Julyπ
July 27π
DesignerImprovements π§
- Improved the code editor to provide features such as:
- Syntax highlighting.
- IntelliSense.
- Validation support for SQL, Python, and Bash.
- Command palette menu via
F1. - Visual Studio Code shortcut keys, e.g. CMD/CTRL for multi-cursor editing.
July 25π
DesignerNew features π
- Added the following components:
CDCImprovements π§
- Updated CDC Agent version from
2.77.1to2.81.6. - Added validation for Oracle Signal Table on PDBs.
- Changed "Unavailable Value" placeholderβread PostgreSQL connector to learn more.
July 19π
CDCNew features π
- Enabled on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases.
July 18π
DesignerNew features π
- Added the SharePoint Query component. This component's driver currently supports:
- Windows SharePoint Services 3.0
- Microsoft Office SharePoint Server 2007
- SharePoint Server 2010
- SharePoint Server 2013
- SharePoint Server 2016
- SharePoint Online
CDCNew features πImprovements π§
- Updated CDC Agent version from
2.68.1to2.76.1. - Updated CDC Agent version from
2.76.1to2.77.1. - Added on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases.
July 4π
DesignerNew features π
- Added the Data Transfer component. Users can transfer files from a chosen source to a chosen target using common network protocols.
- Currently supported data sources include: FTP, HTTP, HTTPS, Amazon S3, and Windows Fileshare.
- Currently supported targets: Amazon S3 and Windows Fileshare.
Juneπ
June 20π
CDCNew features π
- Added on-demand snapshot functionality for PostgreSQL. This feature requires your agent to be version 2.65.0 or higher.
June 14π
CDCImprovements π§
- CDC agent version updated to
2.67.4. - Updates to the DB2 for IBM i connector.
- Pipelines using this connector will now fail on startup if any of the journals for the tracked tables do not have IMAGES(*BOTH) configured.
- Pipelines using this connector will now reconnect to the DB2 instance between snapshotting and streaming. This should rectify issues where the connection times out after a long-running snapshot.
Aprilπ
April 12π
BatchNew features π
- DB2 for IBM i is now available as a Batch loading pipeline.
April 5π
BatchNew features π
- Added support for POST API methods to Matillion Custom Connector.
- Added BODY parameter support for Matillion Custom Connector.
CDCImprovements π§
- Improved performance related to journal lookup for the Db2 for i connector.
Marchπ
March 15π
CDCImprovements π§
- Improved error handling during the pipeline startup process.
- Db2 for IBM i connector: Fix for the SQL catalog and system name differences.
Note
This connector only works for a single system. Remote system journaling is not supported.
March 1π
CDCImprovements π§
- Users can now choose a schema before choosing a table when building a CDC pipeline. Read Create a CDC Pipeline to learn more
Februaryπ
February 27π
CDCImprovements π§
- Introduced a
startingstatus to pipelines that are starting up. - Made the request to stop a pipeline non-blocking.
- Db2 for i connector: Fix handling of tables where the table name and system name differ.
February 13π
CDCImprovements π§
- Users can now edit pipelines in-place, without the need to delete and recreate. Read Create and manage a CDC Pipeline to learn more.
February 1π
BatchNew features π
- Users can now stop a pipeline. Do this from the Dashboard page or the Pipeline Details page.
- The Stop pipeline button will display the current status of a running pipeline.
Januaryπ
January 25π
BatchNew features π
- Users can now observe the URL being queried, so they know the endpoint they're working with. This feature is specific to the Custom Connector pipeline.
CDCImprovements π§
- Added an improvement so that users will be alerted via email if their CDC pipeline fails.
- Streaming agent deployment approach has improved. Users can now quick-create an agent within AWS, using the link from the Data Loader UI (it will pre-fill values into the template for you). Read CloudFormation ECS Fargate Advanced Template for more information.
- Fixed an issue that prevented data from being properly erased while creating destinations. Now, when editing one destination after another or adding new destinations, users should be able to establish pipelines with cleared data.
January 20π
CDCImprovements π§
- Users can now manually reset their own key pair without needing to contact Matillion support. Read Platform Keys for more information.
January 18π
BatchImprovements π§
- Users can now download error logs for their pipelines as a
.txtfile. - Improved validation when creating and editing pipelines by adding per-field validation.
- Users can now set table names when creating a pipeline (as opposed to the table name being assigned on the source name.)
CDCImprovements π§
- Column names are now sanitized to ensure conformity with the Avro file format rules.
- Fixed an issue where special characters in table names would cause issues in the pipeline configuration.
- Improvement added to now log an error if your Organization ID or Agent ID values are not in the correct format.