IAM roles & permissions (GCP)

Overview

Google Cloud Platform (GCP) credentials are required for Matillion ETL instances to access various services such as discovering Cloud Storage buckets, PubSub, and KMS.

Appropriate permissions must be given via your GCP admin console and details of your GCP account must be entered into the Matillion ETL instance via Project → Manage Credentials where credentials for other platforms may also be entered.

Important Information

Each Matillion ETL instance takes a single set of GCP credentials.
If you would like to set up a new Matillion ETL instance to work with a new GCP Project, it is advised you to refer the following documents:

You can grant access to other GCP services using Access Control

GCP & BigQuery Roles

When using Matillion ETL for GCP and BigQuery or even when using BigQuery components on other Matillion ETL platforms, it is required that the user has access to a GCP account with the BigQuery roles.

The required roles while creating a Service Account for GCP are:

Heading	Role
Project	Editor Browser
BigQuery	BigQuery Admin BigQuery Data Editor BigQuery Data Owner BigQuery Data Viewer BigQuery User
Storage	Storage Admin Storage Object Admin Storage Object Creator Storage Object Viewer
PubSub	Pubsub Admin Pubsub Editor Pubsub Publisher Pubsub Subscriber
KMS	kms ListAliases kms Encrypt kms Decrypt

Matillion ETL uses admin BigQuery roles as shown below:

roles/bigquery.admin

The admin BigQuery role includes the following roles:

Role	Description
roles/bigquery.user	Provides permissions to run jobs, including queries, within the project.
roles/bigquery.dataViewer	When applied to a dataset, dataViewer provides permissions to: Read the dataset's metadata and to list tables in the dataset. Read data and metadata from the dataset's tables. When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.
roles/bigquery.dataEditor	When applied to a dataset, dataEditor provides permissions to: Read the dataset's metadata and to list tables in the dataset. Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.
roles/bigquery.dataOwner	When applied to a dataset, dataOwner provides permissions to: Read, update, and delete the dataset. Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.

Matillion ETL requires the Storage admin role:

 roles/storage.admin

The Storage admin role includes the following roles:

Role	Description
roles/storage.objectCreator	Allows users to create objects. Does not give permission to delete or overwrite objects.
roles/storage.objectViewer	Grants access to view objects and their metadata, excluding ACLs.
roles/storage.objectAdmin	Grants full control of objects.

The PubSub includes the following roles in Matillion:

roles/pubsub.admin

Roles	Description
roles/pubsub.admin	Full access to the topics, subscriptions, and snapshots.
roles/pubsub.editor	Modify topics and subscriptions, publish and consume messages.
roles/pubsub.publisher	Publish messages to a topic
roles/pubsub.subscriber	Consume messages from a subscription, attach subscriptions to a topic, and seek to a snapshot.

The KMS includes the following roles in Matillion:

Roles	Description
kms:ListAliases	Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt	Enables Matillion to store an encrypted password.
kms:Decrypt	Enables Matillion to retrieve and use an encrypted password.

Managing and Testing GCP Credentials

When using Matillion ETL the credentials are attached to your Environment definition.

Manage Credentials

In Matillion ETL, in the top left corner of the screen, click Project → Manage Credentials.

Project dropdown menu
Now, in the Manage Credentials window, if the instance credentials are available, you can Test them by using the Test button at the top of the screen.
On the Manage Credentials window, new User Defined Credentials can be added by using the + button. Make sure to select the GCP tab in User Defined Credentials section.

Manage Credentials Window
Next you enter the details required to create a new credential. Then, click Test.
- Name – Enter the name for the user credential.
- Service Account – Browse and select the appropriate service account, which you have created while setting up an account for GCP.
Create GCP Credential

If further information is needed for the Service Account, please read the GCP Account Setup for BigQuery and Storage guide.

User defined credentials are then listed by name under the GCP tab. You select the User Credential you have created from the list, and click Test at the bottom of the manage credentials window.

New Created User Test

Please Note

You can use 🖉 for editing or X icon for any deletion in the each entry listed. When creating or editing credentials, a Test button is made available in the new dialog to check the details before finalising your credentials.
This Test will check access to any services that Matillion ETL uses. You may continue even if the tests fail, however some parts of the product may be impaired or non-functional without appropriate credentials.
Different environments can use different credentials if required.

Add Credentials to an Environment

Expand the Environment panel and choose the environment you wish to modify. Right click on the environment and select Add Environment.

Add Environment
Enter the details to create Environment and then, click Test.
- Environment Name – Enter the environment you wish to create.
- GCP Credentials – Select the GCP credential from the dropdown
- Default project – Select the project from the dropdown.
- Default Dataset – Select the dataset from the from the dropdown.
Once all settings and testing done, click Finish.

Create Environment

Testing GCP Credentials

Begin by launching your Matillion Instance and select Create Project if you do not already have existing project in your instance.
Browser will direct you to Create Project window. Enter the Project Details and, then click Next.
- Project group – Select the Project group from the dropdown.
- Project Name – Enter the project name.
- Project Description – Provide a description for the project.
Create Project
On the next page of Environment, enter details and click Test. Then, click Finish.
- Environment name – Enter the name for the environment to create.
- GCP Credentials – Select the GCP credentials, from the dropdown or click Manage to select.
- Default project – Select the Default project from the dropdown.
- Default Database – Select the default database name.
Environment details
Now the browser will take you to the new Project in Matillion instance, go to the Manage Credentials window by selecting Project menu,, select the newly created user credential and click Test, you should acknowledge a success for BigQuery, GoogleCloudStorage, PubSub, and KMS in the new project.

Successful GCP Credential Test