GCP service accounts

Overview

Matillion ETL for BigQuery requires users to set up their existing GCP Project with BigQuery and GCP authentication for the Matillion ETL instance itself. This is done via Service accounts, a special type of account within your GCP Project that is used by virtual machines. Like a regular account, a Service account has assigned roles and permissions, access keys and so on.

Importantly, your service account should be assigned the correct roles and be linked to a Billing account.

The following instructions all begin from the Google Cloud Platform Console to which you must be logged in.

Creating Service Accounts

1. From the top menu bar, select (or create) your Project.
2. Through the upper-left dropdown menu, browse to the IAM & Admin area of the dashboard.



3. At the left-hand menu, browse to Service Accounts. A list of current service accounts will be present on this page and new service accounts can be added using the Create Service Account button at the top.
4. Name your service account as you wish. We recommend that you keep the Service account ID as it autofills (matching your Service account name). Regardless, take note of the Service account name.



5. Click Create
6. A new section, Grant this service account access to the project will expand. In this section, you are required to add all roles required for this service account using the Select a roleAdd Another Role button.



The required roles are as below. This is the minimum required for basic Matillion ETL usage and does not cover all potential services. For information consult the documentation on IAM Roles and Permissions in GCP.

Heading	Role
Project	Editor Browser
BigQuery	BigQuery Admin BigQuery Data Editor BigQuery Data Owner BigQuery Data Viewer BigQuery User
Cloud Storage	Storage Admin Storage Object Admin Storage Object Creator Storage Object Viewer

7. Ensure each of the above roles are selected before clicking Continue to move on.
8. A third section, Grant users access to this service account, will expand. Enter your users and admins as desired (if even required).



9. Click Done.

Adding Roles to a Service account

If you already have a Service account you wish to use, you may want to add roles to it for use with Matillion ETL.

1. Browse to IAM & Admin via the top-left menu.
2. From the PERMISSION tab list, find your Service account ID and click the pencil icon to its right.
3. Click Add Role and select a Role from the dropdown.

Ensure the below roles are present. This is the minimum required for basic Matillion ETL usage and does not cover all potential services. For information consult the documentation on IAM Roles and Permissions in GCP.

Heading	Role
Project	Editor Browser
BigQuery	BigQuery Admin BigQuery Data Editor BigQuery Data Owner BigQuery Data Viewer BigQuery User
Cloud Storage	Storage Admin Storage Object Admin Storage Object Creator Storage Object Viewer

4. Click Save

Linking Service Accounts to a Billing Account

It is important that your chosen GCP Project has billing enabled, otherwise BigQuery will be inaccessible. Billing accounts are defined outside of projects and applied to each project as desired. For more information, see the google documentation.

1. From the GCP Console left-hand menu, select Billing
2. Select a Billing account for your chosen Project in the list.
• For more information on changing billing for projects, see here.
• If you do not have a billing account or wish to make a new one, see here.

Keys

Service accounts use keys to access services and you will need to know these to properly configure your Matillion ETL instance.

1. From the top-left menu, Select IAM & Admin → Service Accounts
2. Find your Service account in the list and click the three-dot menu to the right, the Manage Keys
3. Click ADD KEY → Create new key
4. Ensure JSON is selected and click Create
5. This should initiate the download of a private key to your computer, keep this safe

The private key should look like this:

{
 "type": "service_account",
 "project_id": "abcde",
 "private_key_id": "",
 "private_key": "",
 "client_email": "abcde@appspot.gserviceaccount.com",
 "client_id": "XXXXXXXXXXXXX",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "token_uri": "https://accounts.google.com/o/oauth2/token",
 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
 "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abcde%40appspot.gserviceaccount.com"
}