Skip to content

Connect to Google Cloud Storage

Matillion CDC can load data from any of your pipelines into Google Cloud Storage. Follow the steps on this page to configure Google Cloud Storage as a destination.

Most of the setup for this should be done when creating your CDC agent and giving it access to a GCP Storage Bucket.

Google Cloud Storage prerequisites

  • You need a Google Cloud Platform account with administrative permissions to create resources. Signing up is free.
  • Permissions to create and manage IAM Roles on Google Cloud Storage buckets. Users must be able to create a bucket (if one doesn't already exist), add/modify bucket policies, and upload files to the bucket.
  • The Google Service account used by the Agent container should have the following permissions for the Google Cloud Storage bucket.
    • storage.buckets.get
    • storage.objects.get
    • storage.multipartUploads.abort
    • storage.multipartUploads.create
    • storage.multipartUploads.list
    • storage.multipartUploads.listParts
    • storage.objects.create
    • storage.objects.delete
    • storage.objects.get
    • storage.objects.list
    • storage.objects.update
  • Provide a unique prefix name for each pipeline.

:::info{title='Note'} Read Google's Cloud Storage documentation to learn more about the service. :::

Connect to Google Cloud Storage

Select destination

  • After you configure the source during a CDC pipeline creation, you will be directed to choose a destination you would like to load your data into.
  • In the Choose destination page, select Google Cloud Storage.

Configure Google Cloud Storage Connection Settings

Specify the following settings in the Connect to Google Cloud Storage destination page:

Field Name Description
Bucket This refers to the name of the Google Cloud Storage bucket you want to use as a destination. Find your bucket name in the Google Cloud Management Console.
Prefix Prefix is the name of the 'folder' or a location within the storage bucket that all CDC data for this pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes.

:::info{title='Note'} A pipeline must have one unique prefix. :::

Test Connection

You can test your connection by using the Test connection button at the bottom; if the test is successful, click Test and Continue.