March 2024 changelog

Here you'll find the March 2024 changelog for the Data Productivity Cloud. Just want to read about new features? Read our New Features blog.

---

28th March

DesignerNew features 🎉

• Added the ability to export pipelines and import them into a new project.
• Added the following connectors for developing data pipelines:
  • Databricks connector
  • Mandrill flex connector

---

20th March

CDCImprovements 🔧

• Updated CDC agent version from 2.91.5 to 2.92.0.
• Added optional STREAMING_AGENT_DEBUG_DISABLE_COMPRESSION Environment Variable to disable compression when uploading to Snowflake Staging area.
• Library updates and security fixes.

---

19th March

DesignerNew features 🎉

• The following AI components are now generally available to all Data Productivity Cloud users in the Designer:
  • Prompt components:
    • OpenAI Prompt
    • AzureOpenAI Prompt
    • Amazon Bedrock Prompt
  • Vector database operations:
    • Pinecone Vector Upsert
    • Pinecone Vector Query
• AI Note is also now generally available, which lets you invoke generative AI to annotate your data pipelines using notes.
• The project details page in Data Productivity Cloud now displays project metadata including the agent type, the cloud data platform, and the type of Git repository (Matillion hosted repository or customer's GitHub repository).

---

14th March

DesignerNew features 🎉

• The ability to connect your own GitHub repo to the Data Productivity Cloud is now generally available. This feature lets you connect your GitHub repository when you create a project in the Data Productivity Cloud. Follow the documentation to get started:
  • Prerequisites for connecting a GitHub repository to the Data Productivity Cloud
  • Installing the Matillion Data Productivity Cloud app in GitHub
  • Troubleshooting guide for connecting a GitHub repository to the Data Productivity Cloud
• The Data Productivity Cloud API is now generally available and includes the following endpoints:
  • GET projects
  • GET environments
  • GET pipelines
  • POST execute published pipeline
  • GET pipeline status
  • PATCH pipeline termination
  • GET pipeline steps status

---

13th March

DesignerNew features 🎉

• Added a Text mode option to components and grid variables, to facilitate rapid input and editing of multiple properties.

CDCImprovements 🔧

• Updated CDC agent version from 2.90.16 to 2.91.5.
• Increased the Streaming agent JVM memory allocation to 60%.
• Fixed a bug with the Direct to Snowflake flow, in which a change to the scale of a numeric column of a table in the source database would not trigger a schema change in the target Snowflake table.
• Library updates and security fixes.

---

12th March

DesignerNew features 🎉

• Added support for data lineage for transformation pipelines. With data lineage, you get a visual representation of each transformation, which can make understanding the evolution of your data faster and easier. Data lineage is currently in public preview and includes:
  • Transformation lineage at runtime.
  • Table-level lineage.
  • Column-level lineage.
  • Table lists.
  • Table metadata—column information and data types.
• Added the following Flex connectors for developing data pipelines:
  • Bugcrowd
  • Salesloft
  • Zendesk Ticketing

---

5th March

DesignerImprovements 🔧

• Updated the AI Note feature with the ability to refine notes. When you click Refine when creating a note, you can elaborate on the note (adding detail), shorten the note's length and detail, or regenerate the note.
• Added a Load Strategy parameter to connectors that support storage-only outputs. Choose between appending files in folder and overwriting files in folder, based on exact structure matching.