Configuring the Streaming agent for MySQL

This page offers information about setting up the Streaming agent deployment for MySQL using the following cloud providers:

Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)
- Using GCP Terraform templates

Each cloud provider section includes information about the MYSQL_DRIVER_SHA512 environment variable, and gives you examples of how to implement it.

The method for generating the SHA512 value is unique for different operating systems. For more information, read Generating SHA512 value.

Prerequisites

To download the appropriate MySQL drivers. Data Loader officially supports version 8.0.29.
The environment variables you need to set are dependent on where you are choosing to deploy the agent. However, one of the MYSQL_DRIVER_SHA512 or DISABLE_MYSQL_DRIVER_SHA512 variables must always be set regardless of the chosen cloud provider.
We strongly recommend using the MYSQL_DRIVER_SHA512 variable, and providing the SHA512 digest of the driver .jar file you are using. For example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"

Note

It's not recommended to set the DISABLE_MYSQL_DRIVER_SHA512 variable to "true", because this means the agent will no longer validate the hash of the driver file for tampering.
If a hash is provided in MYSQL_DRIVER_SHA512, and it doesn't match the hash of the downloaded file, then the agent won't start.

Amazon Web Services

Upload the driver package into the S3 bucket.
The Streaming agent needs access to the S3 bucket, especially the permission s3:GetObject to download the package.
Define the SHA512, S3 bucket, and S3 key values using the following environment variables:
- MYSQL_DRIVER_SHA512
- MYSQL_DRIVER_S3_BUCKET
- MYSQL_DRIVER_S3_KEY

Example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
MYSQL_DRIVER_S3_BUCKET="bucket-name"
MYSQL_DRIVER_S3_KEY="path/to/driver.jar"

Note

The MYSQL_DRIVER_S3_BUCKET, and MYSQL_DRIVER_S3_KEY environment variables, and their values need to be provided to download the files. The agent will launch only if the files are successfully downloaded.
If you are using an AWS-managed MySQL-compatible database as a source for AWS, read the MySQL for AWS documentation.

Microsoft Azure

Upload the driver package into an Azure Blob Storage location that's accessible by the Streaming agent.
The Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read permission is required for the agent to download the driver. For more information, read Get Blob.
Define the SHA512, account name, container name, and Azure Blob name values using the following environment variables:
- MYSQL_DRIVER_SHA512
- MYSQL_DRIVER_ABS_ACCOUNT_NAME
- MYSQL_DRIVER_ABS_CONTAINER_NAME
- MYSQL_DRIVER_ABS_BLOB_NAME

Example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
MYSQL_DRIVER_ABS_ACCOUNT_NAME="storageaccount"
MYSQL_DRIVER_ABS_CONTAINER_NAME="storagecontainer"
MYSQL_DRIVER_ABS_BLOB_NAME="path/to/driver.jar"

Note

The MYSQL_DRIVER_ABS_ACCOUNT_NAME, MYSQL_DRIVER_ABS_CONTAINER_NAME, and MYSQL_DRIVER_ABS_BLOB_NAME environment variables, and their values need to be provided to download the files. The agent will launch only if the files are successfully downloaded.

Note

The "Quick Create" ARM template will create a managed identity that the Streaming agent uses to authenticate against required Azure resources. When adding a MySQL driver to a storage account, you must also grant the role storage blob data contributor to the created managed identity against the storage account that the driver has been placed into.

Google Cloud Platform

Upload the driver package to the Google Cloud Storage location.
The Streaming agent needs access to the Cloud Storage bucket, especially the permission storage.objects.get to download the package.
Define the SHA512, GCS bucket, and GCS blob name values using the following environment variables:
- MYSQL_DRIVER_SHA512
- MYSQL_DRIVER_GCS_BUCKET
- MYSQL_DRIVER_GCS_BLOB_NAME

Example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
MYSQL_DRIVER_GCS_BUCKET="bucket"
MYSQL_DRIVER_GCS_BLOB_NAME="path/to/driver.jar"

Note

The MYSQL_DRIVER_GCS_BUCKET, and MYSQL_DRIVER_GCS_BLOB_NAME environment variables, and their values need to be provided to download the files. The agent will launch only if files are successfully downloaded.

Using GCP Terraform templates

When deploying the Streaming agent for MySQL in the Google Cloud Platform using the Terraform templates that are available here, the necessary templates require additional environment variables.

Refer to the list of variables that you need to add, to deploy the Streaming agent for MySQL, in the relevant Terraform templates.

Add the following three variables to the end of variables.tf:

variable "mysql_bucket" {
description = "GCS Bucket where the MySQL JDBC driver resides."
type        = string
}
variable "mysql_blob_name" {
description = "Path to the MySQL JDBC Driver."
type        = string
}
variable "mysql_driver_sha512" {
description = "MySQL JDBC Driver SHA512."
type        = string
}

Add the following after Line 44 in Compute-engine-template.tf:

    {
        name  = "MYSQL_DRIVER_GCS_BUCKET"
        value = var.mysql_bucket
      },
      {
        name  = "MYSQL_DRIVER_GCS_BLOB_NAME"
        value = var.mysql_blob_name
      },
      {
        name = "MYSQL_DRIVER_SHA512"
        value = var.mysql_driver_sha512
      }

Add the following additional entries to the end of Matillion-cdc-agent.tfvars:

mysql_bucket = ""
mysql_blob_name = ""
mysql_driver_sha512 = ""

Generating the SHA512 value

Each of the following operating systems requires a unique method for generating the SHA512 value for the MYSQL_DRIVER_SHA512 environment variable. In all cases, begin by opening your chosen terminal, and entering the following into the command line:

Operating System	Driver File	Example
Windows	`certutil`	`certutil -hashfile "/path/to/file.jar" SHA512`
MacOS	`shasum`	`shasum -a 512 /path/to/file.jar`
Linux	`sha512sum`	`sha512sum path/to/file.jar`