What's new in the Data Productivity Cloud?

Want to see the raw changes? Check out the Changelog

Connect to Square and put your query results to good use

DesignerNew features 🎉

Square connector

This week we've added the much-requested Square connector. This is a Flex connector so you can edit it through the Custom Connector feature to add your own endpoints and more.

Query Result to Scalar

We're also happy to announce the new Query Result to Scalar component that lets you write a custom SQL query and map its scalar output to a variable. Variables can be used in other components or even passed to Transformation pipelines.

Query Result to Scalar means you can have jobs dynamically react to live data and act accordingly. The component also comes with a basic mode so you can build a query without requiring any SQL at all!

Scheduling improvements, connectors and more!

BatchNew features 🎉Improvements 🔧CDC

Scheduling improvements

Batch pipelines allow you to quickly grab as many sources as you like from your service and load them into a data warehouse... but the real power comes from scheduling these to run more frequently! We've added deeper support for Quartz cron expressions so you can have total control over your schedule frequencies.

Cron expression scheduling

New connectors including Workday support

And as usual we're committed to getting your favorite services integrated natively with the Data Productivity Cloud so you can find the following connector components in Designer, today:

Soft deletes for Snowflake CDC

Earlier this quarter, we released direct-to-Snowflake for CDC pipelines which allows you to replicate your data directly to your Snowflake data warehouse. We're now expanding that with a new "Copy Table with Soft Deletes" feature. This allows you to produce a replica in Snowflake of each captured source table, but rows deleted in the source are not deleted in the target table. As well as each column mapped from the corresponding source table, an additional boolean column called MTLN_CDC_DELETED is included that tells you whether that row is deleted in the source or not.

See the CDC Snowflake Destination documentation for more information.

Connectors keep coming!

DesignerNew features 🎉

We're committed to making your data loading journey as simple as possible and we're backing that up with yet more native integrations this week. Check out the following list of connector components available in the Designer today:

We're also happy to announce Azure Storage as a storage destination for many of our connectors, letting you land your data directly into Blob storage. We're starting with the following connectors, with many more on their way! Here we go...

ActiveCampaign, Amplitude, Anaplan, Braze, Brevo, Chargebee, CircleCI, Concord, Confluence, Datadog, Delighted, Eventbrite, Freshdesk, Gong, Klaviyo, LaunchDarkly, PagerDuty, PayPal, Pendo, Productboard, Recurly, Slack, Smartsheet, Snyk, TikTok, Toggl, and Twilio!

Azure storage

A better way to build pipelines

A new week, and new improvements in the Data Productivity Cloud's pipeline designer. With the huge number of integrations being added, we wanted to make it easier to find components and keep the pipeline building experience quick and intuitive. We've added the ability to choose which output connection the new component should be added to, meaning less reconfiguring for you to do after it's added.

Once your data loading components are on the canvas, you might notice the new Sample tab. This means you can sample loaded data directly from the load component without the need for a transformation job, letting you check your data loads immediately and troubleshoot as you build.

It's pretty straightforward:

  1. Click on your validated Query component (for example, Excel Query).
  2. Click the Sample tab.
  3. Click Sample data.

Sampling inside transformation pipelines is also better than ever now that it reports a total row count for the table data, making it easier than before to verify your load is working as expected. Just hit the refresh button on the Sample tab to get a row count.

Keep custom connecting

We've added an import feature to Custom Connector that allows you to import existing custom connectors from a Matillion ETL instance, making switching to the Data Productivity Cloud easier than ever.

Sample data

Flex Connectors Galore!

Designer Data Loader New features 🎉

We're thrilled to announce the integration of Flex connectors to the Designer and the addition of new connectors for batch pipelines, offering you even more flexibility and connectivity options for seamless data integration.

Flex Connectors Now in Designer

The Designer now supports Flex connectors! Flex connectors can added directly to the canvas when building pipelines and are like a preconfigured blueprint for that service. You can edit this blueprint whenever you like to update all of the components created from it, making it easier than ever to keep multiple pipelines up to date. You can now easily connect to the following 20 endpoints... deep breath!

ActiveCampaign, Amplitude, Brevo, CircleCI, Concord, Confluence, Datadog, Delighted, Freshdesk, Klaviyo, LaunchDarkly, PagerDuty, Pendo, Productboard, Recurly, Smartsheet, Snyk, TikTok, Toggl, Twilio

Check out Flex connectors for more information on Flex connectors in Designer.

Data Loader Flex connectors

In the batch pipelines department, we're also welcoming new Flex connectors!

  • CircleCI
  • Salesforce Pardot
  • Slack
  • TikTok
  • Dropbox

Don't see the connector you need? Let us know at the Ideas Portal.

Enhanced Designer Features and New Flex Connector!

Designer Data Loader New features 🎉

We're excited to bring you a suite of enhancements in the Designer and a new addition to the batch pipelines' Flex connectors. Here's a breakdown of the latest features designed to streamline your workflow.

Designer Updates

  • Assert View Component: We've introduced the Assert View component, enabling you to verify specific conditions of a view and halt the query if the conditions aren't met, ensuring data accuracy and integrity.
  • Revamped Add Components Panel: Finding the component you need is now a breeze with the updated panel that lists components alphabetically, un-nested, and paired with descriptor keywords like "Connectors" or "Flow" for easy filtering and search.
  • Enhanced Pipeline Canvas: The addition of a + call-to-action button streamlines the component addition process. It appears when a component is selected, and clicking it opens the Add component dialog for a smooth, intuitive workflow.
  • Table Update Component: Say hello to the new Table Update component that allows you to update a target table with input rows based on matching keys, enhancing data manipulation and management.
  • Refined Git Operations: The Git Commit changes and push option are now two separate operations, offering greater control and flexibility in managing your version control tasks.

New Flex Connector for Batch Pipelines

Batch pipelines are now enriched with the Concord Flex Connector, enhancing your options and efficiency in developing batch pipelines.

Expanding Batch Pipelines with New Flex Connectors!

Data Loader New features 🎉

We're extending your data integration options with the introduction of new Flex connectors for batch pipelines:

  • ActiveCampaign Connector: Streamline the integration and management of your marketing data with ease.
  • Confluence Connector: Optimize the extraction and handling of your Confluence data for enhanced collaboration and insights.
  • Eventbrite Connector: Simplify the management of event data, making it quicker and more efficient.
  • Recurly Connector: Enhance the processing and analysis of your subscription data seamlessly.

Flex connectors are designed to be both quick to use and customisable. They are built on the Custom Connector framework to be highly customisable, yet are preconfigured to quickly connect to your data. Check out Flex connectors for more information on Flex connectors.

Enhanced Designer and Batch Pipelines with New Functionalities and Connectors!

Designer Data Loader New features 🎉Improvements 🔧

This week we've rolled out of several updates enhancing both the Designer and batch pipelines. From added support for different source types and database components to new connectors, here's a detailed look at the recent enhancements.

Designer Boosts

  • Data Transfer Enhancement: The Data Transfer component in the Designer now supports SFTP as a source type, broadening your options for secure data transfer.
  • Oracle Integration: Oracle has been added as a supported database type in the Database Query and RDS Query components, opening up new avenues for data extraction and manipulation.
  • Multi-line Secret Values Support: Users of Full SaaS, Matillion hosted projects can now include multi-line secret values when creating secret definitions, adding an extra layer of security and flexibility.
  • Scheduling Made Easy: We've included an 'Add schedule' call-to-action button in the Designer UI, making the navigation to the 'Create a new schedule' menu straightforward.
  • Streamlined Cloud Credentials Association: Now, you can associate newly created cloud credentials to an environment in one workflow, simplifying the setup and management process.
  • Git ‘Hard Reset' Functionality: A new "hard reset" option for Git in Designer allows users to reset their branch to the last local commit, enhancing version control.
  • In addition to these, the Google Ads Query component has joined the Designer's component family, offering tailored functionalities for extracting and handling Google Ads data.

Batch Pipelines Connector

Amplitude now has a connector in Data Loader's batch pipelines. Easily connect to Amplitude and pull in one or more data sources straight to data warehouse tables.

Enhancements in Batch Pipelines, Hub, and CDC Agent!

Data Loader Agents Hub New features 🎉

We're excited to roll out our latest features and improvements to the Data Productivity Cloud. Here's what's new and improved in batch pipelines, Hub, and CDC agent.

Batch Pipelines Connectors

We've introduced new Flex connectors - Brevo, Delighted, and Toggl for use in Data Loader's batch pipelines. Flex connectors bring the best of both worlds with out-of-the-box connectivity as well as being customisable and expandable.

Hub Features

For our registered Hub customers with Account and User Administrator privileges, you now have the flexibility to edit the Account name and Subdomain name directly. We've also enhanced the Pipeline Observability dashboard to include visibility into Data Loader Batch pipeline runs. Plus, to make diagnosing issues simpler, pipeline error messages will now be prominently displayed at the top of the Pipeline run details page.

CDC Agent Improvements

We've updated the CDC agent to version 2.87.8. A new transformation type, "Copy Table With Soft Deletes," has been introduced for pipelines with Snowflake as a destination. We've also fixed an issue with Snowflake role names and improved connection reuse for enhanced performance and reliability when using Snowflake as a destination. For more information on agents see the documentation.

New Components and a Flex Connector to Enhance Your Data Workflows!

Designer Data Loader New features 🎉

We're here with another round of exciting updates! The Designer has welcomed a series of new components to broaden your data integration and transformation capabilities, and batch pipelines have a new Flex connector onboard.

Newly Added Designer Components

Expanding your data handling options, we've introduced several new components to the Designer:

  • Azure Blob Storage Load and Unload for efficient data loading and extraction with Azure Blob Storage.
  • Dynamics 365 Query to seamlessly pull and manipulate your Dynamics 365 data.
  • Facebook Ads Query and Facebook Query to integrate and handle data from Facebook and Facebook Ads efficiently.
  • Salesforce Marketing Cloud Query to unlock easy access and manipulation of your marketing data housed in Salesforce.

A New Addition to Batch Pipelines

Batch pipelines also got a boost with the addition of the Snapchat Flex Connector, offering tailored connectivity to develop batch pipelines, ensuring a more streamlined and effective data transformation process.

Cloud Credentials Storage, Expanded SaaS Support, and More!

Designer Data Loader New features 🎉Improvements 🔧

We're excited to share our latest updates designed to optimize your experience with enhanced features in cloud credential storage, component support, and an array of new Flex connectors. Here's the lowdown.

Designer Enhancements

Users can now store cloud provider credentials within the Designer, streamlining authentication with AWS and Azure. We've broadened the scope of components supported in a Matillion Full SaaS environment, including Data Transfer, Excel Query, File Iterator, RDS Bulk Output, RDS Query, S3 Load, S3 Unload, SNS Message, and SQS Message. Plus, utilizing an S3 location as the stage platform on all query components is now a breeze in a Matillion Full SaaS setting.

CDC Pipeline Upgrades

The CDC agent has advanced to version 2.87.1. Now, pipelines with Db2 for IBM i as a source can benefit from on-demand snapshots. We've introduced the Change Log as a transformation type for pipelines targeting Snowflake. Also, the “Create Pipeline” pages got a facelift with an in-client Help bar, offering contextual documentation to assist you seamlessly through the creation process.

Batch Pipelines Boost

Meet the new arrivals in Flex connectors – PagerDuty, Snyk, Datadog, Freshdesk, Klaviyo, LaunchDarkly, Productboard, Smartsheet, and Twilio – each designed to amplify your batch pipeline development and data transformation capabilities.

New Components, Enhanced UI, and CDC Agent Upgrades!

Designer CDC New features 🎉Improvements 🔧

We're excited to unveil a series of updates, featuring new components for the Designer, an enhanced user interface, and significant improvements to the CDC Agent.

Designer's Growing Suite of Components

The Designer has been enriched with the addition of the Marketo Query component, making integration with Marketo smoother than ever. Google BigQuery Query and NetSuite Query also joined the roster, offering more versatility in querying data.

We've revamped the UI for a streamlined user experience. The ‘+' button is now an ‘Add' button accompanied by a context menu, simplifying the process of creating pipelines and folders. For those stepping into an instance of Designer with no existing pipelines, a 'Getting Started' wizard is at your service to make the setup a breeze.

CDC Agent Gets a Revamp

The CDC Agent was upgraded to version 2.83.5, with several imrprovements to snapshotting reliability. Snowflake users will also benefit from the additional validation of the stage format.

We also introduced ‘rs_id' and ‘ssn' metadata fields to the Oracle change records, providing more detailed insights for effective data tracking and management.

Bing Ads Query Component, Enhanced Collaboration, and a New Flex Connector!

Designer Data Loader New features 🎉Improvements 🔧

The Designer welcomes the Bing Ads Query component, a new feature to boost team collaboration, and a fresh Flex connector for batch pipelines. Here's a quick overview!

Bing Ads Query

Harness the power of the Bing Ads Query component in the Designer for efficient data extraction and handling from Bing Ads.

Boosting Collaboration

We've added an ‘Invite your Teammates' item to the help widget's Task checklist in the Designer, making team collaboration straightforward and efficient.

New Flex Connector

For batch pipelines, meet the new Pendo Flex connector, expanding your options and making data transformation more versatile.

New Components, Organizational Features, and Direct to Snowflake for CDC!

Designer Data Loader New features 🎉Improvements 🔧

With the addition of new components in the Designer, intuitive organizational features, and updates to the CDC agent, navigating through your data transformation tasks just got easier.

Designer Enhancements

We've introduced the File Iterator and Stream Input components to expand your data processing capabilities. Organizing your pipelines is now a breeze with the newly added ability to sort them into folders, ensuring a cleaner and more efficient workspace. Tooltips have been added to offer instant insights on pipelines and components.

Designer also now has the File Iterator component, allowing your orchestration jobs to loop through files in storage such as S3, opening up many possibilities for adding automation logic to your workflows.

The new Stream Input transformation component allows you to read chosen columns from a Snowflake stream. Be sure to create a stream using the Create Stream component, first!

CDC Agent and Pipelines update

The CDC Agent isn't left behind in this wave of updates; we've moved up to version 2.81.12. For those working with Snowflake, you'll be pleased to know that Direct to Snowflake has now been added as a target.

New Code Editor, New Components, and CDC Agent Updates!

Designer CDC New features 🎉Improvements 🔧

We're rolling out updates to enhance your experience: a revamped code editor for a more intuitive coding journey, additional components to broaden your data handling capabilities, and updates to the CDC Agent for an enriched performance. Here's a detailed look!

Revamped Code Editor

The Designer's code editor has been refined. Now featuring syntax highlighting and IntelliSense for quicker code writing and reviewing, plus enhanced validation support for SQL, Python, and Bash. Access a suite of commands with a simple tap on F1 and smoothly navigate your code with Visual Studio Code shortcut keys.

New Components Alert!

We've introduced Email Query, HubSpot Query, LDAP Query, and Stripe Query to the Designer. These additions are geared to facilitate more streamlined and versatile data extraction and management.

CDC Agent Update

We've updated the CDC Agent to version 2.81.6. Oracle Signal Table validation on PDBs and an updated "Unavailable Value" placeholder in the PostgreSQL connector are part of this update, offering more tools at your disposal.

Sharepoint comes to Designer and CDC agent updates

Designer CDC New features 🎉

The new Sharepoint Query connector allows you connect to a number of Sharepoint services and load data as part of Designer's orchestration pipelines.

The component's driver currently supports:

  • Windows SharePoint Services 3.0
  • Microsoft Office SharePoint Server 2007
  • SharePoint Server 2010
  • SharePoint Server 2013
  • SharePoint Server 2016
  • SharePoint Online

CDC agent updates

We've added on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases. You'll need to update your CDC agent to get these benefits.

  • Updated CDC Agent version from 2.68.1 to 2.76.1.
  • Updated CDC Agent version from 2.76.1 to 2.77.1.