Skip to content

Secrets overview

Several functions in the Data Productivity Cloud require access to secrets stored in your cloud provider's secret manager. These include:

  • Identification between installed agents and your Hub account.
  • Storing database credentials to be used when configuring a pipeline's source.
  • Storing source credentials for use in connectors.

Using the cloud provider's secret manager means that the Data Productivity Cloud never needs to store the values of your passwords or keys—it's all handled by your cloud provider. This provides an extra assurance of security for your credentials.

Secret manager flow in Data Productivity Cloud

Secret manager flow in the Data Productivity Cloud

  • Users authenticate themselves through the Hub to access the Data Productivity Cloud—where secrets, including customer references, are managed securely.
  • Users invoke Designer to create and configure data pipelines and to define secrets for an existing cloud provider's secret within the corresponding secret definition.
  • Task requests are sent to the Agent Gateway for direct communication with customer-hosted agents, and the scheduler coordinates pipeline executions based on schedules and triggers.
  • A Matillion-hosted agent securely stores and retrieves customer secrets from the Matillion-hosted vault as necessary for pipeline execution.
  • Customer secret vaults securely store and retrieve customer secrets. A customer-hosted agent provides the option to execute data pipelines on-premises or in the customer's cloud network.

Security Benefits

  • The Data Productivity Cloud never directly stores passwords or keys, relying instead on your cloud provider for secure storage.
  • Your cloud provider's secret manager offers robust security measures for protecting credentials, ensuring the confidentiality and integrity of sensitive information.

Agents

The agent is responsible for processing pipeline tasks, which are individual units of work within a data integration workflow. These tasks handle data integration and transformation operations by securely connecting to data sources and targets.

The agent can be configured in two ways:

  • Matillion-hosted agent (Full SaaS): Fully managed by Matillion and resides in Matillion’s VPC.
  • Customer-hosted agent (Hybrid SaaS): Runs inside a user’s VPC.

Note

Agents can access stored secrets, which serve as the repository for all your secrets. In your projects, information is limited to the names of agents and the secrets they can access. The Data Productivity Cloud doesn't provide direct access to the values of secrets. However, these secrets can be used within your projects to access your data services.

Using secrets

In the Data Productivity Cloud, secret definitions are stored at the project level. To use a secret:

  1. Create a named secret in your cloud provider's secret manager.
  2. Add the secret name to the secrets stored in the Data Productivity Cloud. Doing this stores only the name and location of the secret, not the secret's actual value.
  3. Call the secret by name when you need to use the credentials—for example in a data source connector. The secret name is resolved at runtime to obtain the credentials stored in the secret key.

Note

If you need to store multiple passwords and keys, each should be in a separate, named secret.

Error

If an agent cannot access a password, it will result in an error says "The agent can't access the customer's secret manager".

Secret managers

To learn more about your cloud provider's secret manager technology, read the corresponding documentation: