Skip to content

MySQL connector

The Matillion CDC agent's MySQL connector can capture changes within tables that are selected for monitoring in a pipeline. These changes are then written to your desired cloud storage location.

Note

We are using the Debezium MySQL connector to connect to this source. For more information, read Debezium connector for MySQL.

Supported configurations

Data Loader supports the following MySQL configurations:

Supported Category Supported Values
Database version 8.0 and above
Database driver version 8.0.29

Prerequisites

  • MySQL version is 8.0 or higher. To check this version, enter the following command in your MySQL instance: mysql -V
  • MySQL database port (usually 3306).
  • MySQl user account with the necessary permissions (SELECT, RELOAD, CREATE, and ALTER).
  • Binary log (BinLog) replication must be enabled.

CDC agent using MySQL

These instructions involve setting up environment variables for the S3 bucket and key values where the MySQL driver package is stored, as well as an optional SHA512 digest check for the driver file.

  1. Make sure you download the appropriate MySQL drivers, and ensure they are version 8.0.29 or higher.
  2. Upload the downloaded MySQL driver package (.jar file) to your cloud storage location.

Set environment variables for the CDC agent to use MySQL on AWS:

  1. MYSQL_DRIVER_S3_BUCKET:
    • Set this variable to the name of your S3 bucket where the driver package is stored.
    • Example: MYSQL_DRIVER_S3_BUCKET="bucket-name"
  2. MYSQL_DRIVER_S3_KEY:
    • Set this variable to the path of the driver .jar file within the S3 bucket.
    • Example: MYSQL_DRIVER_S3_KEY="path/to/driver.jar"
  3. Provide one of the following:
    • MYSQL_DRIVER_SHA512:
      • Set this variable to the SHA512 digest of the driver .jar file.
      • Example: MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
    • OR DISABLE_MYSQL_DRIVER_SHA512:
      • Set this variable to true to disable checking the .jar file's SHA512 digest (not recommended).
      • DISABLE_MYSQL_DRIVER_SHA512="true"

Warning

  • Disabling the SHA512 variable will skip the SHA512 check. This is not recommended.
  • If the MYSQL_DRIVER_S3_BUCKET and MYSQL_DRIVER_S3_KEY environment variables are set, then the agent will attempt to download the file. If the download fails, the agent will fail to start.
  • If the SHA512 value doesn't match the driver provided, the agent will fail to start.

Set environment variables for CDC agent to use MySQL on Azure:

Set the following environment variables to provide necessary information for the CDC agent:

  1. MYSQL_DRIVER_AZURE_STORAGE_ACCOUNT_NAME:
    • Set this variable to the name of your Azure Storage Account where the driver package is stored.
    • Example: MYSQL_DRIVER_AZURE_STORAGE_ACCOUNT_NAME="yourstorageaccount"
  2. MYSQL_DRIVER_AZURE_STORAGE_CONTAINER_NAME:
    • Set this variable to the name of the container within your Azure Storage Account where the driver .jar file is located.
    • Example: MYSQL_DRIVER_AZURE_STORAGE_CONTAINER_NAME="yourcontainer"
  3. MYSQL_DRIVER_AZURE_STORAGE_BLOB_NAME:
    • Set this variable to the name of the blob (driver .jar file) within the specified container.
    • Example: MYSQL_DRIVER_AZURE_STORAGE_BLOB_NAME="path/to/driver.jar"
  4. Provide one of the following:
    • MYSQL_DRIVER_SHA512:
    • Set this variable to the SHA512 digest of the driver .jar file.
    • Example: MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
    • OR DISABLE_MYSQL_DRIVER_SHA512:
    • Set this variable to true to disable checking the .jar file's SHA512 digest (not recommended).

Quick create (ARM Template) deployment example

Manually edit the template to add these environment variables with correct values:

{
    "name": "DISABLE_MYSQL_DRIVER_SHA512",
    "value": "true"
  },
  {
  "name": "MYSQL_DRIVER_ABS_ACCOUNT_NAME",
  "value": "[variables('storageAccountName')]"
},
 {
  "name": "MYSQL_DRIVER_ABS_CONTAINER_NAME",
  "value": "[variables('storageContainerName')]"
  },
  {
  "name": "MYSQL_DRIVER_ABS_BLOB_NAME",
  "value": "mysql-connector-j-8.1.0.jar"
 }

Adjust the values as per the storage account configuration, considering the appropriate cloud platform, whether it's AWS or Azure.

  • For AWS, make sure the values correspond to your S3 bucket and key where the MySQL driver package is stored.
  • For Azure, make sure the values align with your Azure Storage Account details such as account name, container name, and blob name.

Connect to MySQL

When selecting MySQL as a source during pipeline creation, you will be required to provide the following information:

Property Description
Pipeline Name The name for your new pipeline. Must be unique.
Server address The server address of your MySQL database.
Port The port number used to access your MySQL database.
Username The username used to log in to the specified database.
Secret Provider Choose the provider for your secrets manager that contains your database password.
Secret Name Enter the name of the secret that corresponds to your database password. Read Manage secrets to learn how to add a named secret to your provider's secret manager.