Skip to content

Secret definitions

A secret in the Data Productivity Cloud is a pointer to a secure location where a value is stored for sensitive information such as a password or secure API access token.

Once a secret definition is created, it can be used in many places throughout the Designer where sensitive information must be given, such as password fields. Secret definitions can be viewed and created in the Secret Definitions tab of your project. Not added a project yet? Read Add project.

The way a secret is stored (and, therefore, the way you create the secret definition) depends on whether you are using a Matillion Full SaaS or Hybrid SaaS operating model.

  • In a Matillion Full SaaS implementation, Matillion holds your secrets in an AWS Secrets Manager instance hosted in Matillion's own AWS infrastructure. You don't need to be concerned with the management of the secrets manager.
  • In a Hybrid SaaS implementation, you are responsible for managing your own AWS Secrets Manager or Azure Key Vault in your own cloud infrastructure.

Add a secret definition - Matillion Full SaaS

In a Matillion Full SaaS deployment model, follow this process to create a secret.

  1. From the Your projects menu, select your project.
  2. Navigate to the Secret definitions tab.
  3. Click Add secret definition.

  4. Configure your secret definition by completing the following proerties:

    Property Description
    Secret definition name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector). Any whitespace added to the start or end of the secret definition name is automatically trimmed.
    Description An optional description for the secret. Maximum 256 characters.
    Use multi-line text input? Toggle "Yes" to enable multi-line text input for secret values that require multi-line input, for example, certificates and SSH keys.
    Secret value Enter the value of the secret, for example the password, SSH key, etc.

    Warning

    It is recommended that secret names and secret definition names do not end with a hyphen followed by six characters. To quote the AWS documentation:

    Do not end your secret name with a hyphen followed by six characters. If you do so, you risk confusion and unexpected results when searching for a secret by partial ARN. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN.

  5. Click Create secret.

Once the secret has been created, the secret value is securely hidden from all users and can't be viewed or edited. If you later need to change the value (for example, if you have been issued a new API key to connect to a third party), you will have to delete and recreate the secret.

Add a secret definition - Hybrid SaaS

In a Matillion Hybrid SaaS deployment model, you will store secrets in either AWS Secrets Manager or Azure Key Vault in your own AWS or Azure infrastructure. When you created the Data Productivity Cloud agent, you should have identified which secrets manager or key vaults it has access to.

If you're using an Azure agent, you can store secrets in any Azure key vault that your agent has access to. When you add a new secret definition, you can choose which of your key vaults the secret is stored in.

The Data Productivity Cloud secret definition doesn't hold the secret directly. Instead, it's simply a pointer to the appropriate secret in your own secrets manager/key vault. Before you create this "pointer", you must first create the secret for it to point to. Read either Adding a secret to the AWS Secrets Manager or Adding a secret to Azure Key Vault, below, if you don't know how to do this.

To then create the secret in the Data Productivity Cloud, follow these steps:

  1. From the Your projects menu, select your project.
  2. Navigate to the Secret definitions tab.
  3. Click Add secret definition.

  4. Configure your secret definition.

    Property Description
    Secret definition name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector), regardless of the secret name used in the secrets manager or key vault. Any whitespace added to the start or end of the secret definition name is automatically trimmed.
    Description An optional description for the secret. Maximum 256 characters.
    Agent Select the agent that will be used by the pipelines that use this secret. The agent you select will determine which secrets manager or key vault the secret will be found in.
    Vault name For Azure deployments only. Select the Azure key vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
    Secret name Select a named entry created in AWS Secrets Manager or Azure Key Vault.
    Secret Key For AWS deployments only. Select a named secret key tied to your secret name.

    Warning

    If using AWS Secrets Manager, it is recommended that secret names and secret definition names do not end with a hyphen followed by six characters. To quote the AWS documentation:

    Do not end your secret name with a hyphen followed by six characters. If you do so, you risk confusion and unexpected results when searching for a secret by partial ARN. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN.

  5. Click Create secret.

Managing secret definitions

You can sort the Secret definitions list by name or description.

Once you've created a secret definition, click the more button ... next to your secret and click View secret details. This will display the details of the secret defintion, but won't show the value of the secret itself.

Delete a secret definition

To delete a secret definition:

  1. Enter the Secret definitions tab.
  2. Click the more button ... on the corresponding row of a secret definition you want to delete.
  3. Click Delete secret.
  4. Click Yes, delete to confirm deletion. Otherwise, click Cancel.

In a Matillion Full SaaS environment, this will also remove the secret from the AWS Secrets Manager. In a Hybrid SaaS solution, you would have to manually delete the secret from your own secrets manager/key vault.

Adding a secret to the AWS Secrets Manager

  1. Log in to the AWS account that houses your agent.
  2. Browse to the Secrets Manager service.
  3. Ensure you're in the same AWS region as your agent.
  4. Click Store a new secret.
  5. Click Other type of secret.
  6. Enter a memorable key name and then your secret's value as the value.
  7. Click Next until the secret creation is completed.

Adding a secret to Azure Key Vault

  1. Log in to the Azure portal.
  2. Click Key vaults and click the name of the key vault you want to reference. If there is only one, then this will be the [Default] key vault configured in the agent.
  3. Click Objects and then click Secrets.
  4. Click + Generate/Import.

  5. On the Create a secret screen enter the following:

    Property Description
    Upload options Select Manual.
    Name Type a name for the secret. The secret name must be unique within a key vault. The name must be a 1-127 character string, starting with a letter and containing only 0-9, a-z, A-Z, and -. For more information on naming, read Key Vault objects, identifiers, and versioning.
    Secret value Type a value for the secret.

  6. Click Create.

Adding a Snowflake private key

When creating an environment for a Snowflake data warehouse, you can choose to use key-pair authentication. With this authentication method, the private key will be stored as a secret.

Read Using Snowflake key-pair authentication to learn more.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement