MySQL streaming connector

The MySQL connector can monitor and capture row-level changes within MySQL schemas in a non-intrusive and performant manner. The connector produces a change event record for every insert, update, and delete event occurring in any tables being monitored by the streaming agent.

Note

We are using the Debezium MySQL connector to connect to this source. For more information, read Debezium connector for MySQL.

Prerequisites

The Data Productivity Cloud supports the following MySQL configurations:

Database versions 8.0 and above. Check your MySQL version with mysql -V.
Database driver version 8.0.29

Source setup

Refer to this section when you create a streaming pipeline.

Server address = string

The server address of your MySQL database.

Port = integer

The port number used to access your MySQL database. The default is 3306.

Database name = string

The name of your MySQL database.

Read Replica = boolean

Set to true ✅ if the source database is an always-on, read-only replica database.

Username = string

The username used to log in to the specified database.

Secrets Manager = drop-down

Choose the service you use to manage secrets.

Secret name = string

The name of the secret in your secret manager service that references your MySQL password.

JDBC parameters and specific connection settings = column editor

This parameter is optional.

Specify any parameter:value pairs as part of an advanced connection. Click Save to finish adding parameters.

Click Connect to establish the connection to your source database.

Configure new tables

When adding tables to a pipeline, they must be correctly configured to allow the on-demand snapshot to take place and to ensure change data capture will stream future changes on those tables. Read Configure your MySQL database for a more comprehensive outline of these requirements.

Grant permissions to the streaming user

To enable on-demand snapshots and streaming on new tables, the user configured for streaming must have the expected permissions. These permissions should be applied by the root user. Use the following SQL statement:

-- Grant required privileges on all tables. Check with your database admin for an appropriate mask.
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO '<streaming_user>'@'%';

Replace <streaming_user> with the name of the user configured for streaming.

By following these steps, you ensure that new tables added to the pipeline will be properly configured for on-demand snapshots and streaming, allowing historical data to be captured without losing synchronization with the target. If on-demand snapshots are disabled, new tables will only capture changes from the point of addition and the pipeline will have a STREAMING status immediately.