Drop CDC Tables shared job
Overview
This shared job drops all the tables created by the Sync Single Table and Sync All Tables shared jobs.
Warning
This will drop all CDC tables created by your existing process. This may be required if snapshotting your CDC tables is re-started from scratch, or there is an update to the shared jobs that is a breaking change and necessitates starting from a clean position. You should also clear down the relevant cloud storage area. Please liaise with Support if you are unsure of the steps to follow.
Properties
Cloud Storage URL = string
The URL to the location where the CDC change event files are stored. This should have one of the following formats:
s3://<bucket>/<prefix>/azure://<account>.blob.core.windows.net/<container>/<prefix>/gs://<bucket>/<prefix>/
Target Database = string
The Snowflake database containing the external table with change events and the target table.
Target Schema = string
The Snowflake schema containing the external table with change events and the target table.
External Table = string
The external table in Snowflake containing the CDC change events.
Target Table Prefix = string
Prefix to prepend to the source table name to generate the target table name. If no target prefix is specified, the target table will have the same name as the source table.
Fully Qualify Target Table = string
Optionally includes the source database and schema in the target table name. Enter Y or N:
- Y: The target table name will have the format
<source_database>_<source_schema>_<source_table>. - N: The target table name will have the format
<source_table>.
The default value is Y.
Note
We strongly recommend that you leave this set to Y, unless you are confident that your source table names will always be unique. If the target table name is not fully qualified, and there are source tables with the same name in different databases or schemas, this job will see them as the same table and attempt to merge their metadata and contents.
Azure Key Vault URL = string
Required for Azure Blob Storage. The URL of the Azure Key Vault used to store the Azure Blob Storage connection string. This is also called the vault's "DNS Name".
Azure Blob Storage Account Key Secret = string
Required for Azure Blob Storage. The name of the Azure Key Vault secret which stores the Azure Blob Storage account key.
Actually Drop the Tables = string
Optionally drop the CDC tables from the target database. Enter one of the following:
- Y: The CDC tables will be dropped.
- N: The CDC tables won't be dropped. A list of the tables can be exported from the
cdc_tablesgrid variable.
The default value is N.
CDC Tables (for exporting only) = column editor
Don't set this parameter, it will have no effect. A cdc_tables grid variable can be exported, containing the full list of tables created by the Sync All Tables shared job.
Cloud Storage URL = string
The URL to the location where the CDC change event files are stored. This should have one of the following formats:
s3://<bucket>/<prefix>/azure://<account>.blob.core.windows.net/<container>/<prefix>/
Target Database = string
The Delta Lake database containing the target table.
Stage Table Prefix = string
A prefix to prepend to the source table name to generate the stage table names.
Target Table Prefix = string
Prefix to prepend to the source table name to generate the target table name. If no target prefix is specified, the target table will have the same name as the source table.
Fully Qualify Target Table = string
Optionally includes the source database and schema in the target table name. Enter Y or N:
- Y: The target table name will have the format
<source_database>_<source_schema>_<source_table>. - N: The target table name will have the format
<source_table>.
The default value is Y.
Note
We strongly recommend that you leave this set to Y, unless you are confident that your source table names will always be unique. If the target table name is not fully qualified, and there are source tables with the same name in different databases or schemas, this job will see them as the same table and attempt to merge their metadata and contents.
Azure Key Vault URL = string
Required for Azure Blob Storage. The URL of the Azure Key Vault used to store the Azure Blob Storage connection string. This is also called the vault's "DNS Name".
Azure Blob Storage Account Key Secret = string
Required for Azure Blob Storage. The name of the Azure Key Vault secret which stores the Azure Blob Storage account key.
Actually Drop the Tables = string
Optionally drop the CDC tables from the target database. Enter one of the following:
- Y: The CDC tables will be dropped.
- N: The CDC tables won't be dropped. A list of the tables can be exported from the
cdc_tablesgrid variable.
The default value is N.
CDC Tables (for exporting only) = column editor
Don't set this parameter, it will have no effect. A cdc_tables grid variable can be exported, containing the full list of tables created by the Sync All Tables shared job.
Cloud Storage URL = string
The URL to the location where the CDC change event files are stored. This should have one of the format gs://<bucket>/<prefix>/.
Target Project = string
The GCP project containing the target table.
Target Dataset = string
The BigQuery dataset containing the target table.
Stage Table Prefix = string
A prefix to prepend to the source table name to generate the stage table names.
Target Table Prefix = string
Prefix to prepend to the source table name to generate the target table name. If no target prefix is specified, the target table will have the same name as the source table.
Fully Qualify Target Table = string
Optionally includes the source database and schema in the target table name. Enter Y or N:
- Y: The target table name will have the format
<source_database>_<source_schema>_<source_table>. - N: The target table name will have the format
<source_table>.
The default value is Y.
Note
We strongly recommend that you leave this set to Y, unless you are confident that your source table names will always be unique. If the target table name is not fully qualified, and there are source tables with the same name in different databases or schemas, this job will see them as the same table and attempt to merge their metadata and contents.
Actually Drop the Tables = string
Optionally drop the CDC tables from the target database. Enter one of the following:
- Y: The CDC tables will be dropped.
- N: The CDC tables won't be dropped. A list of the tables can be exported from the
cdc_tablesgrid variable.
The default value is N.
CDC Tables (for exporting only) = column editor
Don't set this parameter, it will have no effect. A cdc_tables grid variable can be exported, containing the full list of tables created by the Sync All Tables shared job.
| Snowflake | Delta Lake on Databricks | Amazon Redshift | Google BigQuery | Azure Synapse Analytics |
|---|---|---|---|---|
| ✅ | ✅ | ❌ | ✅ | ❌ |