Skip to content

Harness the power of LLMs right inside Designer!

DesignerNew features 🎉

Ready for AI-augmented data engineering? Have we got components for you!

  • Open AI Prompt
  • Azure OpenAI Prompt
  • Amazon Bedrock Prompt
  • Pinecone Vector Upsert
  • Pinecone Vector Query

Our new prompt components use large language models (LLMs) to provide responses to user-composed prompts. The components take inputs from your source table, combine those inputs with user prompts, and then send this data to the LLMs for processing. The components even let you augment your queries to the LLM using retrieval augmented generation (RAG).

Speaking of RAG, the Pinecone Vector Upsert component lets you convert data stored in your cloud data warehouse into embeddings, and then store the embeddings as vectors in your Pinecone vector database. The vectors are then available to use in retrieval augmented generation through one of our AI prompt components (or in other projects).

You can use the Pinecone Vector Query component to ingest text search strings from a table and return text data associated with similar vectors in your Pinecone vector database. This means your AI prompt components can benefit from multiple RAG sources at once!

"Our new AI Prompt components enable data engineers to harness the power of prompt engineering when designing data pipelines in the Data Productivity Cloud. This helps to transform data in an unprecedented fashion, using natural language as the main input. It's as easy as defining transformations using the language of the business through prompts, in low-code, graphical components. Anyone can start integrating the reasoning capabilities of LLMs to transform and enrich data, and take data processing to the next level. For instance, turning unstructured data into structured content is now a breeze: you can now make the data business-ready faster with AI thanks to these new components. What was previously the realm of tech-savvy data scientists can now be done at scale by data engineers."

— Cyril Sonnefraud, Principal Product Manager at Matillion

You'll find all the new AI components in the Components tab of the Designer; just click the AI filter.

AI Prompt components in Designer

For specifics on each component, see the documentation, and make sure to check out the videos for hands-on configuration guidance:

AI Note is now in general availability

It's also the perfect time to mention that our AI Note feature is now in GA. When it's this easy, there's no excuse not to annotate your data pipelines with explanations helpful to collaborators. On the canvas, just select the components you want to document, right-click and select Add note using AI. Happy with the note generated? If you think it could be better, click Refine. Then, add more detail by clicking Elaborate, or reduce the detail by clicking Shorten.

Read the documentation and video for more information.