Skip to content

Secrets and secret definitions

A secret in the Data Productivity Cloud is a pointer to a secure location where a value is stored. A secret definition can represent a password or secure API access token.

Once a secret definition is created, it can be used in many places throughout the Designer where sensitive information must be given, such as password fields. Secret definitions can be viewed and created in the Secret Definitions tab of your project. Not added a project yet? Read Add project.

The way a secret is stored (and, therefore, the way you create the secret definition) depends on whether you are using a Matillion Full SaaS or Hybrid SaaS operating model.

  • In a Matillion Full SaaS implementation, Matillion holds your secrets in an AWS Secrets Manager instance hosted in Matillion's own AWS infrastructure. Matillion maintains this AWS Secrets Manager instance for you.
  • In a Hybrid SaaS implementation, you are responsible for managing your own AWS Secrets Manager or Azure Key Vault in your own cloud infrastructure.

Create a secret definition - Matillion Full SaaS

In a Matillion Full SaaS deployment model, you can add new secret definitions from the Your projects menu or in Designer.

Once the secret has been created, the secret value is securely hidden from all users and can't be viewed or edited. If you later need to change the value (for example, if you have been issued a new API key to connect to a third party), update the secret definition.

Warning

When creating a secret, it is recommended that secret definition names do not end with a hyphen followed by six characters. To quote the AWS documentation:

Do not end your secret name with a hyphen followed by six characters. If you do so, you risk confusion and unexpected results when searching for a secret by partial ARN. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN.

Create a secret in Designer

To create a new secret in Designer:

  1. Add a component that requires a secret definition to the canvas.
  2. Click the property that requires a secret definition.
  3. In the Secret drop-down menu, click Add secret. The Add secret definition panel will open.

  4. Configure your secret definition by completing the following properties:

    Property Description
    Name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector). The name may only contain letters, numbers, underscores, single spaces, parentheses, and hyphens. Any whitespace added to the start or end of the secret definition name is automatically trimmed. Choose a memorable, descriptive name because it cannot be changed after the secret is created.
    Description An optional description for the secret. Maximum 256 characters.
    Secret value Enter the value of the secret, for example the password, SSH key, etc.
    Use multi-line text input? Toggle "Yes" to enable multi-line text input for secret values that require multi-line input, for example, certificates and SSH keys.

  5. Click Add secret definition.

The Add secret definition panel will close, and your new secret definition will be automatically selected in the drop-down menu.

Create a secret from Your projects

To create a new secret from the Your projects menu:

  1. From the Your projects menu, select your project.
  2. Navigate to the Secret definitions tab.
  3. Click Add secret definition.

  4. Configure your secret definition by completing the following proerties:

    Property Description
    Secret definition name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector). The name may only contain letters, numbers, underscores, single spaces, parentheses, and hyphens. Any whitespace added to the start or end of the secret definition name is automatically trimmed. Choose a memorable, descriptive name because it cannot be changed after the secret is created.
    Description An optional description for the secret. Maximum 256 characters.
    Use multi-line text input? Toggle "Yes" to enable multi-line text input for secret values that require multi-line input, for example, certificates and SSH keys.
    Secret value Enter the value of the secret, for example the password, SSH key, etc.

  5. Click Create secret.

Create a secret definition - Hybrid SaaS

In a Matillion Hybrid SaaS deployment model, you will store secrets in either AWS Secrets Manager or Azure Key Vault in your own AWS or Azure infrastructure. When you created the Data Productivity Cloud agent, you should have identified which secrets manager or key vaults it has access to.

If you're using an Azure agent, you can store secrets in any Azure key vault that your agent has access to. When you add a new secret definition, you can choose which of your key vaults the secret is stored in.

The Data Productivity Cloud secret definition doesn't hold the secret directly. Instead, it's simply a pointer to the appropriate secret in your own secrets manager/key vault. Before you create this "pointer", you must first create the secret for it to point to. Read either Adding a secret to the AWS Secrets Manager or Adding a secret to Azure Key Vault, below, if you don't know how to do this.

You can then create a secret from the Your projects menu or in Designer.

Create a secret in Designer

To create a new secret in Designer:

  1. Add a component that requires a secret definition to the canvas.
  2. Click the property that requires a secret definition.
  3. In the Secret drop-down menu, click Add secret. The Add secret definition panel will open.

  4. Configure your secret definition by completing the following properties:

    Property Description
    Secret definition name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector), regardless of the secret name used in the secrets manager or key vault. The name may only contain letters, numbers, underscores, single spaces, parentheses, and hyphens. Any whitespace added to the start or end of the secret definition name is automatically trimmed.
    Description An optional description for the secret definition. Maximum 256 characters.
    Vault name For Azure deployments only. Select the Azure key vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
    Secret name Select a named entry created in AWS Secrets Manager or Azure Key Vault. In addition to any character restrictions imposed by your cloud provider, you cannot use @, ~, or whitespace characters in your secret names.
    Secret Key For AWS deployments only. Select a named secret key tied to your secret name.

  5. Click Add secret definition.

The Add secret definition panel will close, and your new secret definition will be automatically selected in the drop-down menu.

Create a secret from Your projects

To create a new secret from the Your projects menu:

  1. From the Your projects menu, select your project.
  2. Navigate to the Secret definitions tab.
  3. Click Add secret definition.

  4. Configure your secret definition.

    Property Description
    Secret definition name A name for the secret definition. You will use this name to refer to the secret from elsewhere in the Data Productivity Cloud (for example, when specifying a password in a connector), regardless of the secret name used in the secrets manager or key vault. The name may only contain letters, numbers, underscores, single spaces, parentheses, and hyphens. Any whitespace added to the start or end of the secret definition name is automatically trimmed.
    Description An optional description for the secret. Maximum 256 characters.
    Agent Select the agent that will be used by the pipelines that use this secret. The agent you select will determine which secrets manager or key vault the secret will be found in.
    Vault name For Azure deployments only. Select the Azure key vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
    Secret name Select a named entry created in AWS Secrets Manager or Azure Key Vault. In addition to any character restrictions imposed by your cloud provider, you cannot use @, ~, or whitespace characters in your secret names.
    Secret Key For AWS deployments only. Select a named secret key tied to your secret name.

    Warning

    If using AWS Secrets Manager, it is recommended that secret definition names do not end with a hyphen followed by six characters. To quote the AWS documentation:

    Do not end your secret name with a hyphen followed by six characters. If you do so, you risk confusion and unexpected results when searching for a secret by partial ARN. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN.

  5. Click Create secret.

Managing secret definitions

You can sort the Secret definitions list by name or description.

Once you've created a secret definition, click the more button ... next to your secret and click View secret details. This will display the details of the secret definition, but won't show the value of the secret itself.

Update a secret definition

If you use the Full SaaS deployment model, you can update the value of a secret stored in Matillion's AWS Secrets Manager, for example if you change an account password that you use for authentication. You cannot update the name of a secret definition because this would cause pipeline errors.

To update the value of a secret:

  1. Navigate to the Secret definitions tab.
  2. Click the more button ... on the row of the secret definition you want to update.
  3. Click Edit secret value.
  4. Enter the new value for this secret.
  5. Click Update.

Delete a secret definition

To delete a secret definition:

  1. Navigate to the Secret definitions tab.
  2. Click the more button ... on the row of the secret definition you want to delete.
  3. Click Delete secret.
  4. Click Yes, delete to confirm deletion. Otherwise, click Cancel.

In a Matillion Full SaaS environment, this will also remove the secret from the AWS Secrets Manager. In a Hybrid SaaS solution, you would have to manually delete the secret from your own secrets manager/key vault.

Adding a secret to the AWS Secrets Manager

  1. Log in to the AWS account that houses your agent.
  2. Browse to the Secrets Manager service.
  3. Ensure you're in the same AWS region as your agent.
  4. Click Store a new secret.
  5. Click Other type of secret.
  6. Enter a memorable key name and then your secret's value as the value.
  7. Click Next until the secret creation is completed.

Adding a secret to Azure Key Vault

  1. Log in to the Azure portal.
  2. Click Key vaults and click the name of the key vault you want to reference. If there is only one, then this will be the [Default] key vault configured in the agent.
  3. Click Objects and then click Secrets.
  4. Click + Generate/Import.

  5. On the Create a secret screen enter the following:

    Property Description
    Upload options Select Manual.
    Name Type a name for the secret. The secret name must be unique within a key vault. The name must be a 1-127 character string, starting with a letter and containing only 0-9, a-z, A-Z, and -. For more information on naming, read Key Vault objects, identifiers, and versioning.
    Secret value Type a value for the secret.

  6. Click Create.

Adding a Snowflake private key

When creating an environment for a Snowflake data warehouse, you can choose to use key-pair authentication. With this authentication method, the private key will be stored as a secret.

Read Using Snowflake key-pair authentication to learn more.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement