Manage Passwords
Overview
Many components in Matillion ETL require passwords to provide access to various services on behalf of the user. Users can input a password directly into a component and it will be stored securely within that component. However, if you use multiple components, managing all the required passwords individually can become laborious, especially if those passwords change or expire regularly.
Warning
For projects using Git, passwords stored in the component will be saved in the repository as plain text. To avoid this, use the password manager.
The password manager provides an alternative to individually stored component passwords, and allows the user to store passwords as named entries. When a component requests a password, the identifying name can be entered and will draw the corresponding password from the manager. Thereafter, if a password should change, the password need only be edited once in Manage Passwords and not in individual components.
There are two ways in which passwords can be managed in Matillion ETL:
- Internal: The password is stored on the Matillion ETL server. Internal passwords can use various encryption methods, with availability depending on the cloud platform you are using. These are:
- AWS Key Management Services (AWS platforms)
- Key Vault Store (Azure platforms)
- GCP Key Management Services (GCP platforms)
- External: The password is stored in a third-party secret manager.
When using external passwords, you will first need to set up Matillion ETL to use an appropriate third-party secret manager, as described in Integrating Matillion ETL with Secret Managers.
Note
- Passwords can also be managed via the Matillion API. For more information, read API v1 - Passwords.
- Internal passwords are stored at the project group level and can be shared with, and accessed from, all other projects within the same Group.
Adding Encoded Passwords
Encoded passwords are encoded and stored in metadata. However, this data is not encrypted or hashed, merely obfuscated. We advise using the other encryption options available such as the key systems from your cloud provider.
- Click Project and then click Manage Project Group Passwords to open the Manage Passwords dialog.
- In the Passwords tab of the Manage Passwords dialog, click + in the bottom left to add a new password entry.
- The Create Password dialog will load. The fields you will be presented with in this dialog will depend on your cloud platform and the Password Type and Encryption Type you want to use. For an internal, encoded password, you will need to complete the following.
- Password Name: A descriptive name for the password to be stored.
- Password Type: Select Internal.
- Password: The password to be stored.
- Encryption Type: Select Encoded.
- Description: A detailed description of the password and its use (this is optional).
- Once complete, click OK. If created successfully, the new password will appear on list of passwords on the Manage Passwords dialog.
Note
It is possible to edit a password's description after creating a password. However, it is not possible to edit or recover a plaintext password through the password manager once it has been entered.
Using AWS Key Management Services
AWS Key Management Services (KMS) is an alternative to "Encoded" encryption that is only available on AWS platforms. Setting the Encryption Type field to KMS during password creation will reveal an additional field:
- Master Key: Select one of the pre-defined AWS KMS master keys to encrypt the password.
AWS KMS master keys must be set up through the associated AWS account. Please refer to AWS Key Management Service Documentation for more details.
Instance credentials dictate key availability. KMS keys must be enabled and based in the same region as the Matillion ETL instance. Additionally, Matillion ETL must have the following IAM Roles:
- kms:ListAliases
- kms:Encrypt
- kms:Decrypt
User-defined credentials cannot be referenced to access KMS.
Warning
If KMS is used for a password but is unavailable for any reason at a component's runtime, the component will fail as though an incorrect password has been entered.
Using Azure Key Vault Store
Key Vault Store is an alternative to "Encoded" encryption that is only available on Azure platforms. Setting the Encryption Type field to Key Vault Store during password creation will reveal these additional fields:
- Encryption Algorithm: Select the algorithm to be used to encrypt the password.
- Resource Group: Select the resource group to which the key vault belongs.
- Key Vault: Select the key vault in which the key is stored.
- Key: Select the name of the key to be used to encrypt the password.
Note
- Resource groups, Azure key vaults, and keys must be pre-defined through the Azure Portal. For more information, read Creating secrets in Azure Key Vault or read the Azure Key Vault documentation.
- The Matillion ETL instance must have at least Reader access to the resource group containing the selected key vault.
- Additionally, key vaults require separate access permissions, requiring the Matillion ETL instance to also have Encrypt and Decrypt access to the Key Vault Key. Access to a key vault must be configured separately, as permissions are not inherited—this can be done via Access policies.
Using GCP Key Management Services
GCP Key Management Services (KMS) is an alternative to "Encoded" encryption that is only available on GCP platforms. Setting the Encryption Type field to KMS during password creation will reveal these additional fields:
- Project: Select a project associated with the GCP account.
- Location: Select a location within the project.
- Key Ring: Select a key ring within the location.
- Key: Select a key associated with the key ring in which to store the password.
Note
- GCP KMS keys must be set up through the associated GCP account. For more information, read Creating symmetric encryption keys.
- Environment credentials dictate from which GCP account the project, key, and key ring will be sourced. Additionally, Matillion ETL must have the following predefined roles:
- cloudkms.admin or viewer
- cloudkms.cryptoKeyEncrypterDecrypter
Warning
If KMS is used for a password but is unavailable for any reason at a component's runtime, the component will fail as though an incorrect password has been entered.
Using external password stores
The External password type allow you to store the password in a third-party secret manager. You will first need to set up Matillion ETL to use an appropriate third-party secret manager, as described in Integrating Matillion ETL with Secret Managers. Setting the Password Type field to External during password creation will reveal an additional field:
- Secret Manager: Select a third-party secret manager that you have configured Matillion ETL to use.