Data Productivity Cloud
The Data Productivity Cloud is the Matillion platform that allows users to connect their data, put it to work, and get value faster. The Data Productivity Cloud provides many services that provide low-code solutions to your ETL and data loading needs with minimal time investment to get started.
To begin learning about the Data Productivity Cloud, see the below sections describing the core services and follow the links to learn more about each.
What makes up the Data Productivity Cloud?
Hub
The Hub is the access point for your Matillion services and an administrative centre for user administration, billing, subscription management, and observability. It also provides overviews for resources such as custom connectors and agents.
For more information, read Hub overview.
Designer
Designer is a SaaS tool for designing sophisticated ETL workflows by splitting logic between orchestration tasks and transformation tasks. It is designed to be low-code solution that emphasizes ease of use and collaborative development. For more information, read Designer overview.
Designer require an agent to execute tasks, although it should be noted that an agent will be provided that can be used without any setup required from the user via a Matillion Full SaaS solution. For more information, read Designer agent overview.
Data Loader
Data Loader allows users to easily load data from many different sources. It offers both batch loading and near real-time replication using Change Data Capture (CDC). These two types of pipeline are usually discussed separately as their setup differs significantly.
Batch
Data Loader's batch pipelines perform batch loads from a target database into your cloud data warehouse. These pipelines can be scheduled to run periodically, meaning you always have recent data to work from. For more information, read Batch loads introduction.
Batch pipelines support a long list of sources straight out of the box, but can also run on Custom Connectors that can ingest JSON responses from a REST API of your choosing.
CDC
Data Loader's CDC pipelines connect to a target database and provide near real-time replication to a replication database while also outputting changes to a storage area (such as an S3 bucket).
CDC pipelines require a CDC agent to be installed on the user's cloud platform. Read CDC agent installation for details. For more information on CDC pipelines, read Change Data Capture.
Matillion ETL
Matillion ETL is a deep ETL/ELT tool that allows customers to load, transform, sync, and orchestrate their data via a virtual machine within their cloud data platform.
Although Matillion ETL is deployed on a user's private cloud, it does make contact with the hub for the purposes of billing. To launch a Matillion ETL instance, see Matillion ETL instance creation
Additional services
Manage Agents
Agents allow Matillion services to execute pipelines on a user's private cloud. Both Designer and Data Loader (CDC) rely on agents to perform their functions and each has its own respective agent. It should be noted that using Designer with Matillion Full SaaS solution requires no setup from the user and the agent will be hosted by Matillion.
The Manage Agents page will allow users to see all of their agents in one place, monitor their statuses and remove them as required.
For Designer agents, read Designer agent overview. For CDC agents, read CDC agent overview
Manage Custom Connectors
Custom connectors are a way of connecting any REST API that returns a JSON format response to a Matillion Data Loader batch pipeline. These connectors can be stored and managed from one place via the Manage Custom Connectors page. For more information, read Custom Connector overview