What's new in the Data Productivity Cloud?

Want to see the raw changes? Check out the Changelog

Full SaaS is better than ever with Python Pushdown

DesignerNew features ๐ŸŽ‰

Weโ€™re thrilled to announce the ability to incorporate Python scripts into your pipelines with the new Python Pushdown component to Designer. This component lets you execute Python scripts using the Snowpark service in your Snowflake account, bringing high-code capabilities to Full SaaS solutions in the Data Productivity Cloud.

New connectors in January

Data loading made easy

Weโ€™ve also made scheduling pipelines easier by introducing two modes: standard and advanced. The new standard mode lets you set the schedule in simple terms of Days, Weeks, Hours, and Minutes, rather than writing a Cron expressionโ€”which is required by the advanced mode.

And as always, weโ€™ve added a new connector! Check out the Intercom flex connector to easily load Intercom data to your cloud data warehouse.

AI notes are here!

DesignerNew features ๐ŸŽ‰Improvements ๐Ÿ”ง

Automatically document pipelines with AI Notes

Adding notes can be a great way to document your work and make your teamโ€™s life a lot easier. But why not sit back and let GenAI do the work for you?

Introducing AI notes, a one-click solution to start documenting your pipelines. Simply select the component(s) you want to document, right click and select Add note using AI. Notes can be regenerated or added to the canvas for later use or editing.

For more information, visit the documentation for this wonderful feature.

Connector improvements

Itโ€™s another great week for connectors. Google Cloud Storage has been added as a direct-to-storage option to all Flex connectors in the Designer. This means you can load data from your service of choice directly into Google Cloud Storage with a single component.

Make sure to add your GCP credentials to the Cloud provider credentials menu before setting your components Destination property to Cloud Storage. Weโ€™ve also added a new SendGrid Flex connector and added REST API service support to the SharePoint Query connector!

Do more with the Data Productivity Cloud API!

DesignerNew features ๐ŸŽ‰

Pipeline API

We're thrilled to announce the release of the Data Productivity Cloud's first public API. This collection allows you to check your projects, pipelines and their status. You can execute pipelines via the API for easier automation and terminate running pipelines, just in case.

You can check out our API reference for these endpoints or get started with the API in the documentation. But why not give our video guide a watch first to cover the basics?

Low code variables

Want to enjoy the flexibility of implementing variables into your jobs but don't want to mess around with the Python Script component? We've implemented a couple of new low-code components that will help you out.

This helps you build more flexible pipelines that report on their own progress for easier debugging and monitoring.

Shopify connector

And of course we've added a new connector! Shopify is available in the Designer right now and can also be extended from the Custom Connector menu. Enjoy!

Azure Blob Storage is here!

DesignerNew features ๐ŸŽ‰

Land your data straight into Azure Blob Storage

This week we're extremely pleased to announce that Azure Blob Storage has been added as a staging option to all connectors in the Designer.

To get plugged in, simply add your Azure credentials to the Cloud provider credentials menu, then select a connector that has been added to your pipeline and select the Stage Platform property. From there, select Existing Azure Blob Storage Location to select from your Blob Storage containers.

For Flex connectors, users can select Cloud Storage as a Destination and then select Azure Blob Storage from there.

Happy loading!

Recent Pipelines

We know what it's like to have a lot of projects and branches, so we've added a "Recent Pipelines" section to the front Hub page that fast-tracks you straight to the designer to easily access your most recently-visited pipelines.

Zendesk Talk

Finally, it wouldn't be a release without another connector dropping, and we've added the Zendesk Talk - just add it to your pipeline canvas to get connected. Remember to visit the Custom Connector menu to set up your Zendesk Talk configuration and even add new endpoints as desired.

Connect to Square and put your query results to good use

DesignerNew features ๐ŸŽ‰

Square connector

This week we've added the much-requested Square connector. This is a Flex connector so you can edit it through the Custom Connector feature to add your own endpoints and more.

Query Result to Scalar

We're also happy to announce the new Query Result to Scalar component that lets you write a custom SQL query and map its scalar output to a variable. Variables can be used in other components or even passed to Transformation pipelines.

Query Result to Scalar means you can have jobs dynamically react to live data and act accordingly. The component also comes with a basic mode so you can build a query without requiring any SQL at all!

Scheduling improvements, connectors and more!

BatchNew features ๐ŸŽ‰Improvements ๐Ÿ”งCDC

Scheduling improvements

Batch pipelines allow you to quickly grab as many sources as you like from your service and load them into a data warehouse... but the real power comes from scheduling these to run more frequently! We've added deeper support for Quartz cron expressions so you can have total control over your schedule frequencies.

Cron expression scheduling

New connectors including Workday support

And as usual we're committed to getting your favorite services integrated natively with the Data Productivity Cloud so you can find the following connector components in Designer, today:

Soft deletes for Snowflake CDC

Earlier this quarter, we released direct-to-Snowflake for CDC pipelines which allows you to replicate your data directly to your Snowflake data warehouse. We're now expanding that with a new "Copy Table with Soft Deletes" feature. This allows you to produce a replica in Snowflake of each captured source table, but rows deleted in the source are not deleted in the target table. As well as each column mapped from the corresponding source table, an additional boolean column called MTLN_CDC_DELETED is included that tells you whether that row is deleted in the source or not.

See the CDC Snowflake Destination documentation for more information.

Connectors keep coming!

DesignerNew features ๐ŸŽ‰

We're committed to making your data loading journey as simple as possible and we're backing that up with yet more native integrations this week. Check out the following list of connector components available in the Designer today:

We're also happy to announce Azure Storage as a storage destination for many of our connectors, letting you land your data directly into Blob storage. We're starting with the following connectors, with many more on their way! Here we go...

ActiveCampaign, Amplitude, Anaplan, Braze, Brevo, Chargebee, CircleCI, Concord, Confluence, Datadog, Delighted, Eventbrite, Freshdesk, Gong, Klaviyo, LaunchDarkly, PagerDuty, PayPal, Pendo, Productboard, Recurly, Slack, Smartsheet, Snyk, TikTok, Toggl, and Twilio!

Azure storage

A better way to build pipelines

A new week, and new improvements in the Data Productivity Cloud's pipeline designer. With the huge number of integrations being added, we wanted to make it easier to find components and keep the pipeline building experience quick and intuitive. We've added the ability to choose which output connection the new component should be added to, meaning less reconfiguring for you to do after it's added.

Once your data loading components are on the canvas, you might notice the new Sample tab. This means you can sample loaded data directly from the load component without the need for a transformation job, letting you check your data loads immediately and troubleshoot as you build.

It's pretty straightforward:

  1. Click on your validated Query component (for example, Excel Query).
  2. Click the Sample tab.
  3. Click Sample data.

Sampling inside transformation pipelines is also better than ever now that it reports a total row count for the table data, making it easier than before to verify your load is working as expected. Just hit the refresh button on the Sample tab to get a row count.

Keep custom connecting

We've added an import feature to Custom Connector that allows you to import existing custom connectors from a Matillion ETL instance, making switching to the Data Productivity Cloud easier than ever.

Sample data

Flex Connectors Galore!

Designer Data Loader New features ๐ŸŽ‰

We're thrilled to announce the integration of Flex connectors to the Designer and the addition of new connectors for batch pipelines, offering you even more flexibility and connectivity options for seamless data integration.

Flex Connectors Now in Designer

The Designer now supports Flex connectors! Flex connectors can added directly to the canvas when building pipelines and are like a preconfigured blueprint for that service. You can edit this blueprint whenever you like to update all of the components created from it, making it easier than ever to keep multiple pipelines up to date. You can now easily connect to the following 20 endpoints... deep breath!

ActiveCampaign, Amplitude, Brevo, CircleCI, Concord, Confluence, Datadog, Delighted, Freshdesk, Klaviyo, LaunchDarkly, PagerDuty, Pendo, Productboard, Recurly, Smartsheet, Snyk, TikTok, Toggl, Twilio

Check out Flex connectors for more information on Flex connectors in Designer.

Data Loader Flex connectors

In the batch pipelines department, we're also welcoming new Flex connectors!

  • CircleCI
  • Salesforce Pardot
  • Slack
  • TikTok
  • Dropbox

Don't see the connector you need? Let us know at the Ideas Portal.

Enhanced Designer Features and New Flex Connector!

Designer Data Loader New features ๐ŸŽ‰

We're excited to bring you a suite of enhancements in the Designer and a new addition to the batch pipelines' Flex connectors. Here's a breakdown of the latest features designed to streamline your workflow.

Designer Updates

  • Assert View Component: We've introduced the Assert View component, enabling you to verify specific conditions of a view and halt the query if the conditions aren't met, ensuring data accuracy and integrity.
  • Revamped Add Components Panel: Finding the component you need is now a breeze with the updated panel that lists components alphabetically, un-nested, and paired with descriptor keywords like "Connectors" or "Flow" for easy filtering and search.
  • Enhanced Pipeline Canvas: The addition of a + call-to-action button streamlines the component addition process. It appears when a component is selected, and clicking it opens the Add component dialog for a smooth, intuitive workflow.
  • Table Update Component: Say hello to the new Table Update component that allows you to update a target table with input rows based on matching keys, enhancing data manipulation and management.
  • Refined Git Operations: The Git Commit changes and push option are now two separate operations, offering greater control and flexibility in managing your version control tasks.

New Flex Connector for Batch Pipelines

Batch pipelines are now enriched with the Concord Flex Connector, enhancing your options and efficiency in developing batch pipelines.

Expanding Batch Pipelines with New Flex Connectors!

Data Loader New features ๐ŸŽ‰

We're extending your data integration options with the introduction of new Flex connectors for batch pipelines:

  • ActiveCampaign Connector: Streamline the integration and management of your marketing data with ease.
  • Confluence Connector: Optimize the extraction and handling of your Confluence data for enhanced collaboration and insights.
  • Eventbrite Connector: Simplify the management of event data, making it quicker and more efficient.
  • Recurly Connector: Enhance the processing and analysis of your subscription data seamlessly.

Flex connectors are designed to be both quick to use and customisable. They are built on the Custom Connector framework to be highly customisable, yet are preconfigured to quickly connect to your data. Check out Flex connectors for more information on Flex connectors.

Enhanced Designer and Batch Pipelines with New Functionalities and Connectors!

Designer Data Loader New features ๐ŸŽ‰Improvements ๐Ÿ”ง

This week we've rolled out of several updates enhancing both the Designer and batch pipelines. From added support for different source types and database components to new connectors, here's a detailed look at the recent enhancements.

Designer Boosts

  • Data Transfer Enhancement: The Data Transfer component in the Designer now supports SFTP as a source type, broadening your options for secure data transfer.
  • Oracle Integration: Oracle has been added as a supported database type in the Database Query and RDS Query components, opening up new avenues for data extraction and manipulation.
  • Multi-line Secret Values Support: Users of Full SaaS, Matillion hosted projects can now include multi-line secret values when creating secret definitions, adding an extra layer of security and flexibility.
  • Scheduling Made Easy: We've included an 'Add schedule' call-to-action button in the Designer UI, making the navigation to the 'Create a new schedule' menu straightforward.
  • Streamlined Cloud Credentials Association: Now, you can associate newly created cloud credentials to an environment in one workflow, simplifying the setup and management process.
  • Git โ€˜Hard Reset' Functionality: A new "hard reset" option for Git in Designer allows users to reset their branch to the last local commit, enhancing version control.
  • In addition to these, the Google Ads Query component has joined the Designer's component family, offering tailored functionalities for extracting and handling Google Ads data.

Batch Pipelines Connector

Amplitude now has a connector in Data Loader's batch pipelines. Easily connect to Amplitude and pull in one or more data sources straight to data warehouse tables.

Enhancements in Batch Pipelines, Hub, and CDC Agent!

Data Loader Agents Hub New features ๐ŸŽ‰

We're excited to roll out our latest features and improvements to the Data Productivity Cloud. Here's what's new and improved in batch pipelines, Hub, and CDC agent.

Batch Pipelines Connectors

We've introduced new Flex connectors - Brevo, Delighted, and Toggl for use in Data Loader's batch pipelines. Flex connectors bring the best of both worlds with out-of-the-box connectivity as well as being customisable and expandable.

Hub Features

For our registered Hub customers with Account and User Administrator privileges, you now have the flexibility to edit the Account name and Subdomain name directly. We've also enhanced the Pipeline Observability dashboard to include visibility into Data Loader Batch pipeline runs. Plus, to make diagnosing issues simpler, pipeline error messages will now be prominently displayed at the top of the Pipeline run details page.

CDC Agent Improvements

We've updated the CDC agent to version 2.87.8. A new transformation type, "Copy Table With Soft Deletes," has been introduced for pipelines with Snowflake as a destination. We've also fixed an issue with Snowflake role names and improved connection reuse for enhanced performance and reliability when using Snowflake as a destination. For more information on agents see the documentation.

New Components and a Flex Connector to Enhance Your Data Workflows!

Designer Data Loader New features ๐ŸŽ‰

We're here with another round of exciting updates! The Designer has welcomed a series of new components to broaden your data integration and transformation capabilities, and batch pipelines have a new Flex connector onboard.

Newly Added Designer Components

Expanding your data handling options, we've introduced several new components to the Designer:

  • Azure Blob Storage Load and Unload for efficient data loading and extraction with Azure Blob Storage.
  • Dynamics 365 Query to seamlessly pull and manipulate your Dynamics 365 data.
  • Facebook Ads Query and Facebook Query to integrate and handle data from Facebook and Facebook Ads efficiently.
  • Salesforce Marketing Cloud Query to unlock easy access and manipulation of your marketing data housed in Salesforce.

A New Addition to Batch Pipelines

Batch pipelines also got a boost with the addition of the Snapchat Flex Connector, offering tailored connectivity to develop batch pipelines, ensuring a more streamlined and effective data transformation process.

Cloud Credentials Storage, Expanded SaaS Support, and More!

Designer Data Loader New features ๐ŸŽ‰Improvements ๐Ÿ”ง

We're excited to share our latest updates designed to optimize your experience with enhanced features in cloud credential storage, component support, and an array of new Flex connectors. Here's the lowdown.

Designer Enhancements

Users can now store cloud provider credentials within the Designer, streamlining authentication with AWS and Azure. We've broadened the scope of components supported in a Matillion Full SaaS environment, including Data Transfer, Excel Query, File Iterator, RDS Bulk Output, RDS Query, S3 Load, S3 Unload, SNS Message, and SQS Message. Plus, utilizing an S3 location as the stage platform on all query components is now a breeze in a Matillion Full SaaS setting.

CDC Pipeline Upgrades

The CDC agent has advanced to version 2.87.1. Now, pipelines with Db2 for IBM i as a source can benefit from on-demand snapshots. We've introduced the Change Log as a transformation type for pipelines targeting Snowflake. Also, the โ€œCreate Pipelineโ€ pages got a facelift with an in-client Help bar, offering contextual documentation to assist you seamlessly through the creation process.

Batch Pipelines Boost

Meet the new arrivals in Flex connectors โ€“ PagerDuty, Snyk, Datadog, Freshdesk, Klaviyo, LaunchDarkly, Productboard, Smartsheet, and Twilio โ€“ each designed to amplify your batch pipeline development and data transformation capabilities.

New Components, Enhanced UI, and CDC Agent Upgrades!

Designer CDC New features ๐ŸŽ‰Improvements ๐Ÿ”ง

We're excited to unveil a series of updates, featuring new components for the Designer, an enhanced user interface, and significant improvements to the CDC Agent.

Designer's Growing Suite of Components

The Designer has been enriched with the addition of the Marketo Query component, making integration with Marketo smoother than ever. Google BigQuery Query and NetSuite Query also joined the roster, offering more versatility in querying data.

We've revamped the UI for a streamlined user experience. The โ€˜+' button is now an โ€˜Add' button accompanied by a context menu, simplifying the process of creating pipelines and folders. For those stepping into an instance of Designer with no existing pipelines, a 'Getting Started' wizard is at your service to make the setup a breeze.

CDC Agent Gets a Revamp

The CDC Agent was upgraded to version 2.83.5, with several imrprovements to snapshotting reliability. Snowflake users will also benefit from the additional validation of the stage format.

We also introduced โ€˜rs_id' and โ€˜ssn' metadata fields to the Oracle change records, providing more detailed insights for effective data tracking and management.

Bing Ads Query Component, Enhanced Collaboration, and a New Flex Connector!

Designer Data Loader New features ๐ŸŽ‰Improvements ๐Ÿ”ง

The Designer welcomes the Bing Ads Query component, a new feature to boost team collaboration, and a fresh Flex connector for batch pipelines. Here's a quick overview!

Bing Ads Query

Harness the power of the Bing Ads Query component in the Designer for efficient data extraction and handling from Bing Ads.

Boosting Collaboration

We've added an โ€˜Invite your Teammates' item to the help widget's Task checklist in the Designer, making team collaboration straightforward and efficient.

New Flex Connector

For batch pipelines, meet the new Pendo Flex connector, expanding your options and making data transformation more versatile.

New Components, Organizational Features, and Direct to Snowflake for CDC!

Designer Data Loader New features ๐ŸŽ‰Improvements ๐Ÿ”ง

With the addition of new components in the Designer, intuitive organizational features, and updates to the CDC agent, navigating through your data transformation tasks just got easier.

Designer Enhancements

We've introduced the File Iterator and Stream Input components to expand your data processing capabilities. Organizing your pipelines is now a breeze with the newly added ability to sort them into folders, ensuring a cleaner and more efficient workspace. Tooltips have been added to offer instant insights on pipelines and components.

Designer also now has the File Iterator component, allowing your orchestration jobs to loop through files in storage such as S3, opening up many possibilities for adding automation logic to your workflows.

The new Stream Input transformation component allows you to read chosen columns from a Snowflake stream. Be sure to create a stream using the Create Stream component, first!

CDC Agent and Pipelines update

The CDC Agent isn't left behind in this wave of updates; we've moved up to version 2.81.12. For those working with Snowflake, you'll be pleased to know that Direct to Snowflake has now been added as a target.

New Code Editor, New Components, and CDC Agent Updates!

Designer CDC New features ๐ŸŽ‰Improvements ๐Ÿ”ง

We're rolling out updates to enhance your experience: a revamped code editor for a more intuitive coding journey, additional components to broaden your data handling capabilities, and updates to the CDC Agent for an enriched performance. Here's a detailed look!

Revamped Code Editor

The Designer's code editor has been refined. Now featuring syntax highlighting and IntelliSense for quicker code writing and reviewing, plus enhanced validation support for SQL, Python, and Bash. Access a suite of commands with a simple tap on F1 and smoothly navigate your code with Visual Studio Code shortcut keys.

New Components Alert!

We've introduced Email Query, HubSpot Query, LDAP Query, and Stripe Query to the Designer. These additions are geared to facilitate more streamlined and versatile data extraction and management.

CDC Agent Update

We've updated the CDC Agent to version 2.81.6. Oracle Signal Table validation on PDBs and an updated "Unavailable Value" placeholder in the PostgreSQL connector are part of this update, offering more tools at your disposal.

Sharepoint comes to Designer and CDC agent updates

Designer CDC New features ๐ŸŽ‰

The new Sharepoint Query connector allows you connect to a number of Sharepoint services and load data as part of Designer's orchestration pipelines.

The component's driver currently supports:

  • Windows SharePoint Services 3.0
  • Microsoft Office SharePoint Server 2007
  • SharePoint Server 2010
  • SharePoint Server 2013
  • SharePoint Server 2016
  • SharePoint Online

CDC agent updates

We've added on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases. You'll need to update your CDC agent to get these benefits.

  • Updated CDC Agent version from 2.68.1 to 2.76.1.
  • Updated CDC Agent version from 2.76.1 to 2.77.1.