Skip to content

Creating and listing secret references

Try in Console

In this guide, you will learn how to use Matillion's APIs to create and list secret references that point to the name and location of the secret stored in your cloud provider's secrets manager. You can then use these secret references to access databases or connectors when configuring a pipeline. For more information about secret references, read Secrets overview.

Prerequisites

Before you begin, make sure you have the following:

Create a secret reference

Use this API endpoint to create a new secret reference in a specified project. Currently, the only supported secretReferenceType is PASSWORD. If you use a Hybrid SaaS setup, you can create secret references that point to your cloud provider's secret manager. If you use a Full SaaS setup, you can create secret references hosted by Matillion.

To create a secret reference in any project, you first need to find and note your project ID and agent ID.

Find your project ID

Retrieve the list of projects using the Project API and copy the id value–this is your project ID.

Base URL: GET /v1/projects

Example response:

{
  "page": 0,
  "results": [
    {
      "description": "project-1",
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "name": "Test-project"
    }
  ],
  "size": 0,
  "total": 0
}

Find your agent ID

Click the menu button in the top left of any Data Productivity Cloud screen, then click Manage → Agents.

If you use a Hybrid SaaS setup (customer-hosted agents), in the row for the agent, click ... → Agent details. Scroll down to Agent environment variables and copy the AGENT_ID value.

Alternatively, if you use a Full SaaS setup (Matillion-hosted agents), hover over the agent and click Copy to clipboard next to the agent ID.

Create a secret reference for an AWS customer-hosted project

To create a secret reference for a project connected to the AWS Secrets Manager, you will need your project ID, agent ID, and AWS Secrets Manager secret key and secret name.

Base URL: POST https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references

Example request body:

{
  "agentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "agentType": "AWS",
  "description": "My secret reference",
  "secretReferenceType": "PASSWORD",
  "secretKey": "aws-secret-key",
  "secretName": "aws-secret-name"
}

Create a secret reference for an Azure customer-hosted project

To create a secret reference for a project connected to an Azure Key Vault, you will need your project ID, agent ID, and Azure Key Vault secret key and secret name.

Base URL: POST https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references

Example request body:

{
  "agentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "agentType": "Azure",
  "description": "My secret reference",
  "secretReferenceType": "PASSWORD",
  "secretName": "azure-secret-name",
  "vaultName": "azure-vault-name"
}

Create a secret reference for a Matillion-hosted project

To create a secret reference that is hosted and managed by Matillion, you will need your project ID and the agent ID of your Matillion-hosted agent.

Base URL: POST https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references

Example request body:

{
  "agentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "agentType": "Matillion hosted",
  "description": "My secret reference",
  "secretReferenceType": "PASSWORD",
  "secretValue": {"password":"password-plain-text-value"}
}

List secret references

Use this API endpoint to retrieve a list of secret references within a specific project. It supports pagination for large result sets and optional filtering by secret reference type and providers. If no value is provided for secretReferenceTypes, only PASSWORD types will be returned.

To retrieve a list of secret references, you will need your project ID—see Find your project ID for how to find this—and any secret reference types that you want to use to filter the results.

List secret references for an AWS or Azure customer-hosted project

To list secret references for an AWS or Azure customer-hosted project, you will need your project ID. The response body will vary based on where the agent used in the project is hosted, e.g. AWS or Azure.

Base URL: GET https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references

The example response for an AWS customer-hosted project is shown below. For an Azure customer-hosted project, the values for vaultSecretKey and vaultSecretName will be your Azure secret key and secret name. The vaultType value will be Azure.

{
"page": 1073741824,
  "results": [
    {
      "description": "My secret reference",
      "name": "Data-Productivity-Cloud-secret-name",
      "vaultSecretKey": "aws-secret-key",
      "vaultSecretName": "aws-secret-name",
      "type": "PASSWORD",
      "vaultType": "AWS"
    }
  ],
  "size": 1073741824,
  "total": 9007199254740991
}

List secret references for a Matillion-hosted project

To list secret references for a Matillion-hosted project, you will need your project ID

Base URL: GET https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references

Example response:

{
  "page": 1073741824,
  "results": [
    {
      "name": "my-secret",
      "secretReferenceType": "PASSWORD",
      "vaultType": "Matillion hosted"
    }
  ],
  "size": 1073741824,
  "total": 9007199254740991
}

Copy secret references from one project to another

Use this endpoint to list the secret references in one project (the source project). A post-response import script will then build the request body to create these secret references in another project (the destination project).

Before you begin, make sure you have the following:

  • Postman to enter environment variables and the post-response import script.
  • The project IDs for the source project and destination project—see Find your project ID for how to find these project IDs.
  • The agent ID for the destination project–see Find your agent ID for how to find this.

To copy secret references fro one project to another, follow these steps:

  1. Make a request using the following base URL, with the source project ID as a path variable: GET https://{region}.api.matillion.com/v1/projects/{projectId}/secret-references.
  2. Configure the following environment variables:
    • projectId: Enter the destination project ID.
    • agentId: Enter the agent ID for the destination project.
    • public_api_host: Enter https://{region}.api.matillion.com. Replace {region} with either eu1 or us1as, depending on the region where your project is located.
    • secretsData: Leave this empty—this will be populated by the post-response import script.
  3. Add the post-response import script in the Scripts → Post-response section.
  4. Click Send.

Post-response import script:

// Extract secrets from API response
let secrets = pm.response.json().results || [];

// Format secrets and store in an environment variable
pm.environment.set("secretsData", JSON.stringify(secrets.map(secret => ({
    secretReferenceName: secret.name,
    secretReferenceType: secret.secretReferenceType,
    secretKey: secret.vaultSecretKey,
    secretName: secret.vaultSecretName,
    agentId: pm.environment.get('agentId'),
    agentType: secret.vaultType
}))));

// Send create requests
JSON.parse(pm.environment.get("secretsData")).forEach(secret => {
    let { secretReferenceName, ...body } = secret;

    let requestUrl = `${pm.environment.get('public_api_host')}/v1/projects/${pm.environment.get('projectId')}/secret-references/${secretReferenceName}`;

    pm.sendRequest({
        url: requestUrl,
        method: "POST",
        header: {
            "Content-Type": "application/json",
            "Authorization": `Bearer ${pm.environment.get('jwt')}`
        },
        body: { mode: "raw", raw: JSON.stringify(body) }
    }, (err, res) => console.log(err ? "Request failed:" : "Request successful:", err || res));
});

Run In Postman

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement