Connect to Google Cloud Storage🔗
Matillion CDC can load data from any of your pipelines into Google Cloud Storage. Follow the steps on this page to configure Google Cloud Storage as a destination.
Most of the setup for this should be done when creating your Streaming agent and giving it access to a GCP Storage Bucket.
Google Cloud Storage prerequisites🔗
- You need a Google Cloud Platform account with administrative permissions to create resources. Signing up is free.
- Permissions to create and manage IAM Roles on Google Cloud Storage buckets. Users must be able to create a bucket (if one doesn't already exist), add/modify bucket policies, and upload files to the bucket.
- The Google Service account used by the Agent container should have the following permissions for the Google Cloud Storage bucket.
storage.buckets.getstorage.objects.getstorage.multipartUploads.abortstorage.multipartUploads.createstorage.multipartUploads.liststorage.multipartUploads.listPartsstorage.objects.createstorage.objects.deletestorage.objects.getstorage.objects.liststorage.objects.update
- Provide a unique prefix name for each pipeline.
Read Google's Cloud Storage documentation to learn more about the service.
Connect to Google Cloud Storage🔗
Select destination🔗
- After you configure the source during a CDC pipeline creation, you will be directed to choose a destination you would like to load your data into.
- In the Choose destination page, select Google Cloud Storage.
Configure Google Cloud Storage Connection Settings🔗
Specify the following settings in the Connect to Google Cloud Storage destination page:
| Field Name | Description |
|---|---|
| Bucket | This refers to the name of the Google Cloud Storage bucket you want to use as a destination. Find your bucket name in the Google Cloud Management Console. |
| Prefix | Prefix is the name of the 'folder' or a location within the storage bucket that all CDC data for this pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes. A pipeline must have one unique prefix. |
Test Connection🔗
You can test your connection by using the Test connection button at the bottom; if the test is successful, click Test and Continue.