Skip to content

Cloud provider credentials

Credentials are required for the Data Productivity Cloud to authenticate with your cloud provider. Credentials are configured at the project level and can then be used by any pipeline running in that project.

Create cloud provider credentials

To set up a new cloud provider credential, follow these steps:

  1. In your project, click the Cloud credentials tab.
  2. Click Add cloud credential.
  3. Complete the following fields:
    • Credential name: A unique, descriptive name for your cloud provider credential.
    • Provider: Click the appropriate tile to select the cloud provider.
    • Agent: This is only required if you host your own agent in a Hybrid SaaS environment. Use the drop-down to select the agent that will be used to store these credentials securely in Data Productivity Cloud.
  4. Complete the fields required to authorize with the cloud provider.
  5. Click Save to move to the Associate cloud credentials screen.
  6. Optionally, select an environment to associate this cloud credential with.
    • Credential name: This displays the name you gave this credential on the previous screen.
    • Environment: Select an environment from the drop-down.
  7. To associate the credential with the selected environment, click Associate. To save the credential without first associating it with an environment, click Associate later.

To use a credential, you must associate it with an environment; however, this can be done at a later stage, as described under Associate cloud provider credentials with an environment, below.

Delete a cloud provider credential

  1. In your project, click the Cloud provider credentials tab.
  2. Click the more button ... on the corresponding row of a credential you want to delete.
  3. Click Delete cloud provider credential.
  4. Click Yes, delete to confirm deletion. Otherwise, click Cancel.

Associate cloud provider credentials with an environment

Each environment in your project must have at least one set of cloud credentials associated with it. This will allow you to access account resources on different platforms other than that hosting your project. For example, if your project is on AWS and you want to access resources in Azure, you need to associate your Azure cloud credentials with the environment.

You can associate credentials from multiple providers, but only one set of credentials for each cloud provider. For example, you can associate both AWS and Azure credentials, but not two different AWS credentials.

You can associate credentials with an environment when you first Create cloud provider credentials, or you can associate them later as follows:

  1. In your project, click the Environments tab.
  2. Click the more button ... on the corresponding row of the environment you want to associate, and select Associate Credentials.
  3. Select the credentials from the drop-down lists. You can associate one set of credentials for each cloud provider.
  4. Click Associate.

Acquiring AWS credentials

Obtain the Access key ID and Secret access key for an existing Amazon IAM user.

  1. Log in to the AWS Console.

  2. On the navigation menu, click Users. If you don't see the option, search for "Users" in the search bar at the top of the page and select Users IAM Feature from the results.

    Note

    You must have an appropriate level of administrator access to AWS to view user details.

  3. Click the name of the IAM user you want to obtain an access key for.

  4. Click the Security credentials tab, and under Access keys click Create access key.
  5. On the Access key best practices & alternatives page, choose Other and click Next.
  6. Click Create access key.
  7. On the Retrieve access keys page, click Show to reveal the value of your user's secret access key. Copy the Access key and the revealed Secret access key value to use as your cloud provider credentials.
  8. Click Done to complete the creation of the key.

Acquiring Azure credentials

Obtain the Tenant ID, Client ID, and Secret key from the Azure portal.

  1. Log in to the Microsoft Azure Portal.
  2. Click Azure Active Directory from the Azure services menu.
  3. On the App registrations page, click + New registration.
  4. On the Register an application page, provide details for the following fields:
    • Name: A name for the app.
    • Supported account types: Select Accounts in any organizational directory (Any Azure AD directory - Multitenant).
  5. Click Register.
  6. Your browser will redirect to the Overview page on the app's newly created dashboard. From here, copy the credentials to the right of Application (client) ID and Directory (tenant) ID.
  7. Click Certificates & secrets on the menu on the left, and on the Certificates & secrets page click + New client secret.
  8. The Add a client secret page will appear to the right. Provide details for the following fields and then click Add:
    • Description: Provide a description of the client secret.
    • Expires: Use the drop-down to select when the client secret should expire

  9. You will automatically be returned to the Certificates and secrets page, where the new client secret will appear in the list in the Client secrets tab. Copy the client secret Value.

    Note

    Make sure to copy the client secret right away, as you won't be able to see it again.

Acquiring GCP credentials

Create a service account in the GCP portal to obtain its Service account key.

  1. Log in to the Google Cloud Console to be taken to the homepage.
  2. Choose an appropriate Organization using the drop-down menu at the top-left. The select a resource dialog will open, where you can choose your project.
  3. Click the IAM and admin tile from the quick-access area at the bottom of the homepage.
  4. Click Service accounts from the IAM and admin menu. A list of current service accounts will be displayed.
  5. Click CREATE SERVICE ACCOUNT at the top of the page.
  6. Give a display name for your Service account name.
  7. We recommend that you keep the Service account ID as it autofills (matching your Service account name). Regardless, take note of the Service account name.
  8. Describe what this service account will do. Filling in the Service account description is optional.
  9. Click CREATE AND CONTINUE.
  10. The Grant this service account access to the project section will expand. This section is optional. Add all roles required for this service account using the Select a role drop-down menu. To add another role, click the Add Another Role button. For more information, read Manage access to projects, folders, and organizations.
  11. Click Continue.
  12. The final section, Grant users access to this service account, will expand. This section is optional. In the first field, grant users the permission to deploy jobs and virtual machines (VMs) with this service account. In this second field, grant users the permission to administer this service account.
  13. Click Done. You will return to the list of current service accounts for your project, where your new service account will be displayed.

Adding a service account key

  1. Click ☰, to the right of your service account.
  2. Click Manage Keys.
  3. Click the ADD KEY drop-down menu, and select Create new key.
  4. A new dialog will appear. Ensure the Key Type JSON is selected.
  5. Click Create.
  6. Clicking create will download the JSON file containing the private key.

Note

Store the JSON file securely because the private key cannot be recovered if it's lost.

The private key should look like this:

{
 "type": "service_account",
 "project_id": "abcde",
 "private_key_id": "",
 "private_key": "",
 "client_email": "abcde@appspot.gserviceaccount.com",
 "client_id": "XXXXXXXXXXXXX",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "token_uri": "https://accounts.google.com/o/oauth2/token",
 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
 "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abcde%40appspot.gserviceaccount.com"
}

Roles and permissions for Cloud Storage

To load your data directly into your preferred cloud storage destination, such as S3, Azure Blob Storage, or Google Cloud Storage, you'll need to apply the following roles and permissions to your cloud accounts:

Cloud Storage Role/policy Permissions
S3 An appropriate policy name
  • s3:DeleteObject
  • s3:GetObject
  • s3:PutObject
  • s3:ListAllMyBuckets
Azure storage Storage Blob Data Contributor Actions:
  • Microsoft.Storage/storageAccounts/blobServices/containers/delete
  • Microsoft.Storage/storageAccounts/blobServices/containers/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/write
  • Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action
DataActions:
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
Storage Account Contributor Actions:
  • Microsoft.Authorization/*/read
  • Microsoft.Insights/alertRules/*
  • Microsoft.Insights/diagnosticSettings/*
  • Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action
  • Microsoft.ResourceHealth/availabilityStatuses/read
  • Microsoft.Resources/deployments/*
  • Microsoft.Resources/subscriptions/resourceGroups/read
  • Microsoft.Storage/storageAccounts/*
  • Microsoft.Support/*
Google Cloud Storage Storage Admin storage.buckets.*