Skip to content

IAM roles & permissions (GCP)

Overview

Google Cloud Platform (GCP) credentials are required for Matillion ETL instances to access various services such as discovering Cloud Storage buckets, PubSub, and KMS.

Appropriate permissions must be given via your GCP admin console and details of your GCP account must be entered into the Matillion ETL instance via Project → Manage Credentials where credentials for other platforms may also be entered.

Important Information

GCP & BigQuery Roles

When using Matillion ETL for GCP and BigQuery or even when using BigQuery components on other Matillion ETL platforms, it is required that the user has access to a GCP account with the BigQuery roles.

The required roles while creating a Service Account for GCP are:

Heading Role
Project Editor
Browser
BigQuery BigQuery Admin
BigQuery Data Editor
BigQuery Data Owner
BigQuery Data Viewer
BigQuery User
Storage Storage Admin
Storage Object Admin
Storage Object Creator
Storage Object Viewer
PubSub Pubsub Admin
Pubsub Editor
Pubsub Publisher
Pubsub Subscriber
KMS kms ListAliases
kms Encrypt
kms Decrypt

Matillion ETL uses admin BigQuery roles as shown below:

roles/bigquery.admin

The admin BigQuery role includes the following roles:

Role Description
roles/bigquery.user Provides permissions to run jobs, including queries, within the project.
roles/bigquery.dataViewer

When applied to a dataset, dataViewer provides permissions to:

  • Read the dataset's metadata and to list tables in the dataset.
  • Read data and metadata from the dataset's tables.

When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

roles/bigquery.dataEditor

When applied to a dataset, dataEditor provides permissions to:

  • Read the dataset's metadata and to list tables in the dataset.
  • Create, update, get, and delete the dataset's tables.

When applied at the project or organization level, this role can also create new datasets.

roles/bigquery.dataOwner

When applied to a dataset, dataOwner provides permissions to:

  • Read, update, and delete the dataset.
  • Create, update, get, and delete the dataset's tables.

When applied at the project or organization level, this role can also create new datasets.

Matillion ETL requires the Storage admin role:

 roles/storage.admin

The Storage admin role includes the following roles:

Role Description
roles/storage.objectCreator Allows users to create objects. Does not give permission to delete or overwrite objects.
roles/storage.objectViewer Grants access to view objects and their metadata, excluding ACLs.
roles/storage.objectAdmin Grants full control of objects.

The PubSub includes the following roles in Matillion:

roles/pubsub.admin
Roles Description
roles/pubsub.admin Full access to the topics, subscriptions, and snapshots.
roles/pubsub.editor Modify topics and subscriptions, publish and consume messages.
roles/pubsub.publisher Publish messages to a topic
roles/pubsub.subscriber Consume messages from a subscription, attach subscriptions to a topic, and seek to a snapshot.

The KMS includes the following roles in Matillion:

Roles Description
kms:ListAliases Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt Enables Matillion to store an encrypted password.
kms:Decrypt Enables Matillion to retrieve and use an encrypted password.

Managing and Testing GCP Credentials

When using Matillion ETL the credentials are attached to your Environment definition.

Manage Credentials

  1. In Matillion ETL, in the top left corner of the screen, click Project → Manage Credentials.

    Project dropdown menu

    Project dropdown menu

  2. Now, in the Manage Credentials window, if the instance credentials are available, you can Test them by using the Test button at the top of the screen.

  3. On the Manage Credentials window, new User Defined Credentials can be added by using the + button. Make sure to select the GCP tab in User Defined Credentials section.

    Manage Credentials Window

    Manage Credentials Window

  4. Next you enter the details required to create a new credential. Then, click Test.

    • Name – Enter the name for the user credential.
    • Service Account – Browse and select the appropriate service account, which you have created while setting up an account for GCP.
    Create GCP Credential

    Create GCP Credential

    5. If further information is needed for the Service Account, please read the GCP Account Setup for BigQuery and Storage guide.

  5. User defined credentials are then listed by name under the GCP tab. You select the User Credential you have created from the list, and click Test at the bottom of the manage credentials window.

    New created User Test

    New Created User Test

Please Note

  • You can use 🖉 for editing or X icon for any deletion in the each entry listed. When creating or editing credentials, a Test button is made available in the new dialog to check the details before finalising your credentials.
  • This Test will check access to any services that Matillion ETL uses. You may continue even if the tests fail, however some parts of the product may be impaired or non-functional without appropriate credentials.
  • Different environments can use different credentials if required.

Add Credentials to an Environment

  1. Expand the Environment panel and choose the environment you wish to modify. Right click on the environment and select Add Environment.

    Add Environment

    Add Environment

  2. Enter the details to create Environment and then, click Test.
    • Environment Name – Enter the environment you wish to create.
    • GCP Credentials – Select the GCP credential from the dropdown
    • Default project – Select the project from the dropdown.
    • Default Dataset – Select the dataset from the from the dropdown.

    Once all settings and testing done, click Finish.

    Create Environment

    Create Environment

Testing GCP Credentials

  1. Begin by launching your Matillion Instance and select Create Project if you do not already have existing project in your instance.

  2. Browser will direct you to Create Project window. Enter the Project Details and, then click Next.

    • Project group – Select the Project group from the dropdown.
    • Project Name – Enter the project name.
    • Project Description – Provide a description for the project.
    Create project

    Create Project

  3. On the next page of Environment, enter details and click Test. Then, click Finish.

    • Environment name – Enter the name for the environment to create.
    • GCP Credentials – Select the GCP credentials, from the dropdown or click Manage to select.
    • Default project – Select the Default project from the dropdown.
    • Default Database – Select the default database name.
    Environment details

    Environment details

  4. Now the browser will take you to the new Project in Matillion instance, go to the Manage Credentials window by selecting Project menu,, select the newly created user credential and click Test, you should acknowledge a success for BigQuery, GoogleCloudStorage, PubSub, and KMS in the new project.

    Successful GCP Credential Test

    Successful GCP Credential Test