JDBC

The JDBC orchestration component allows you to run queries on a JDBC-compatible data source that you have a driver for. You can upload your drivers to the Data Productivity Cloud, and so connect to any sources that Matillion doesn't provide drivers for as standard.

Note

This component is only available for use with Hybrid SaaS agents.
When adding new drivers, an agent restart is required to recognize the configuration. Once the agent has restarted, you can choose the driver from the Database Type property in the component.

To use this component, you must upload a suitable JDBC driver for your data source, following the process given in Uploading external drivers to the agent. You can upload drivers for multiple different data sources, and select the required source from a drop-down when configuring the component.

You must also create a manifest file, which will contain configuration information for your data sources.

Properties

Name = string

A human-readable name for the component.

Database Type = drop-down

Select the database type. The drop-down will contain all databases that you have included in the manifest file.

Connection URL = string

The URL for your chosen JDBC database. If the field displays a template URL for the database, replace any placeholder values with the actual values of your database's URL.

Although many parameters and options can be added to the end of the URL, it's generally easier to add them in the Connection Options property, documented below.

Authentication Type = drop-down

Currently only supports Username & Password or None.

Username = string

Your username for the JDBC data source.

Password = drop-down

Displays a drop-down list of secret definitions. Select the secret definition that references the password corresponding to the username.

Connection Options = column editor

Parameter: A JDBC parameter supported by the database driver. The parameters available to your database will usually be explained in the database's data model.
Value: A value for the given parameter.

Mode = drop-down

Basic: This mode will build a query for you using settings from the Data Source, Data Selection, and Data Source Filter parameters. In most cases, this mode will be sufficient.
Advanced: This mode will require you to write an SQL-like query to call data from your data source.

Note

While the query is exposed in an SQL-like language, the exact semantics can be surprising, for example, filtering on a column can return more data than not filtering on it. This is an impossible scenario with regular SQL.

SQL Query = code editor

This is an SQL-like SELECT query. Treat collections as table names, and fields as columns. Only available in Advanced mode.

Schema = drop-down

If your data source uses schemas, then use the drop-down to select the required schema. This property should be left blank if the data source does not use schemas.

Data Source = drop-down

Select a data source.

Data Selection = dual listbox

Choose one or more columns to return from the query. The columns available are dependent upon the data source selected. Move columns left-to-right to include in the query.

Data Source Filter = column editor

Input Column: Select an input column. The available input columns vary depending upon the data source.
Qualifier:
- Is: Compares the column to the value using the comparator.
- Not: Reverses the effect of the comparison, so "Equals" becomes "Not equals", "Less than" becomes "Greater than or equal to", etc.
Comparator: Choose a method of comparing the column to the value. Possible comparators include: "Equal to", "Greater than", "Less than", "Greater than or equal to", "Less than or equal to", "Like", "Null". "Equal to" can match exact strings and numeric values, while other comparators, such as "Greater than" and "Less than", will work only with numerics. The "Like" operator allows the wildcard character % to be used at the start and end of a string value to match a column. The Null operator matches only null values, ignoring whatever the value is set to. Not all data sources support all comparators, meaning that it is likely that only a subset of the above comparators will be available to choose from.
Value: The value to be compared.

Combine Filters = drop-down

Select whether to use the defined filters in combination with one another according to either And or Or.

Limit = integer

Set a numeric value to limit the number of rows that are loaded.

Select your cloud data warehouse.

SnowflakeDatabricksAmazon Redshift

Destination = drop-down

Snowflake: Load your data into Snowflake. You'll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Snowflake or Cloud Storage tab on this page for documentation applicable to that destination type.

SnowflakeCloud Storage

Warehouse = drop-down

The Snowflake warehouse used to run the queries. The special value, [Environment Default], will use the warehouse defined in the environment. Read Overview of Warehouses to learn more.

Database = drop-down

The Snowflake database. The special value, [Environment Default], will use the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.

Schema = drop-down

The Snowflake schema. The special value, [Environment Default], will use the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.

Table Name = string

The name of the table to be created.

Load Strategy = drop-down

Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.

Clean Staged files = boolean

Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.

Stage Platform = drop-down

Choose a data staging platform using the drop-down menu.

S3: Stage your data on an AWS S3 bucket.
Snowflake: Stage your data on a Snowflake internal stage.
Azure Storage: Stage your data in an Azure Blob Storage container.

Click one of the tabs below for documentation applicable to that staging platform.

Amazon S3SnowflakeAzure StorageGoogle Cloud Storage

Amazon S3 Bucket = drop-down

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Internal Stage Type = drop-down

A Snowflake internal stage type. Currently, only type User is supported.

Read Choosing an Internal Stage for Local Files to learn more about internal stage types and the usage of each.

Storage Account = drop-down

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container = drop-down

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Storage Integration= drop-down

Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.

GCS Bucket= drop-down

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite = boolean

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Load Strategy = drop-down

Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure `uniqueID/timestamp-partX` where X is the part number, starting from 1. For example, `1da27ea6-f0fa-4d15-abdb-d4e990681839/20240229100736969-part1`.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-timestamp-partX` where X is the part number, starting from 1. For example, `folder/prefix-20240229100736969-part1`.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-partX` where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path = string

The folder path of the written files.

File Prefix = string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage = drop-down

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3Azure StorageGoogle Cloud Storage

Amazon S3 Bucket = drop-down

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Storage Account = drop-down

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container = drop-down

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

GCS Bucket= drop-down

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite = boolean

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Destination = drop-down

Databricks: Load your data into Databricks. You'll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Databricks or Cloud Storage tab on this page for documentation applicable to that destination type.

DatabricksCloud Storage

Catalog = drop-down

Select a Databricks Unity Catalog. The special value, [Environment Default], will use the catalog specified in the Data Productivity Cloud environment setup. Selecting a catalog will determine which schema are available in the next parameter.

Schema = drop-down

Select the Databricks schema. The special value, [Environment Default], will use the schema specified in the Data Productivity Cloud environment setup.

Table Name = string

The name of the table to be created.

Load Strategy = drop-down

Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.

Clean Staged Files = boolean

Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.

Stage Platform = drop-down

Choose a data staging platform using the drop-down menu.

S3: Stage your data on an AWS S3 bucket.
Azure Storage: Stage your data in an Azure Blob Storage container.

Click one of the tabs below for documentation applicable to that staging platform.

Amazon S3Azure StorageGoogle Cloud Storage

Amazon S3 Bucket = drop-down

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Storage Account = drop-down

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container = drop-down

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

Storage Integration= drop-down

Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.

GCS Bucket= drop-down

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite = boolean

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Load Strategy = drop-down

Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure `uniqueID/timestamp-partX` where X is the part number, starting from 1. For example, `1da27ea6-f0fa-4d15-abdb-d4e990681839/20240229100736969-part1`.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-timestamp-partX` where X is the part number, starting from 1. For example, `folder/prefix-20240229100736969-part1`.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-partX` where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path = string

The folder path of the written files.

File Prefix = string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage = drop-down

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3Azure StorageGoogle Cloud Storage

Amazon S3 Bucket = drop-down

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Storage Account = drop-down

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container = drop-down

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

GCS Bucket= drop-down

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite = boolean

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Destination = drop-down

Redshift: Load your data into Amazon Redshift. You'll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.

Click either the Amazon Redshift or Cloud Storage tab on this page for documentation applicable to that destination type.

Amazon RedshiftCloud Storage

Schema = drop-down

Select the Redshift schema. The special value, [Environment Default], will use the schema defined in the environment. For information about using multiple schemas, read Schemas.

Table Name = string

The name of the table to be created.

Load Strategy = drop-down

Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.

Clean Staged Files = boolean

Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.

Amazon S3 Bucket = drop-down

An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Load Strategy = drop-down

Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.

See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:

Configuration	Description
Append files in folder with defined folder path and file prefix.	Files will be stored under the structure `uniqueID/timestamp-partX` where X is the part number, starting from 1. For example, `1da27ea6-f0fa-4d15-abdb-d4e990681839/20240229100736969-part1`.
Append files in folder without defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-timestamp-partX` where X is the part number, starting from 1. For example, `folder/prefix-20240229100736969-part1`.
Overwrite files in folder with defined folder path and file prefix.	Files will be stored under the structure `folder/prefix-partX` where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.	Validation will fail. Folder path and file prefix must be supplied for this load strategy.

Folder Path = string

The folder path of the written files.

File Prefix = string

A string of characters to include at the beginning of the written files. Often used for organizing database objects.

Storage = drop-down

A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.

Click the tab that corresponds to your chosen cloud storage service.

Amazon S3Azure StorageGoogle Cloud Storage

Amazon S3 Bucket = drop-down

An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.

Storage Account = drop-down

Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.

Container = drop-down

Select a Blob container to be used for staging the data. For more information, read Introduction to Azure Blob storage.

GCS Bucket= drop-down

The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.

Overwrite = boolean

Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.

Manifest file

The manifest file is a JSON format text file that includes configuration information needed to connect to the data source. Every data source that you upload an external driver for must have a corresponding entry in the manifest file.

The manifest file must be named jdbc-driver-manifest.json and saved to the storage location that also holds the uploaded external driver files.

The following example shows the format of a manifest file. In this example we've configured a single data source, MySQL.

[
    {
      "name": "MySQL",
      "driverClassName": "com.mysql.jdbc.Driver",
      "baseUrl": "jdbc:mysql://",
      "limitStrategy": "limit-inline",
      "fetchSize": 500,
      "defaultProperties": {
        "connectTimeout": "20000",
        "allowLoadLocalInfile": "false",
        "allowUrlInLocalInfile": "false",
        "allowLocalInfile": "false"
      },
      "blacklistedProperties": [
      ],
      "propertyOverrides": {
        "username": "user",
        "password": "password"
      }
    }
]

The parameters in this file should be configured as follows:

name: This can be anything. The name will appear in the component's Database Type drop-down.
driverClassName: The base class name for the JDBC driver.
baseUrl: A template URL used to prompt the user to enter the correct format URL in the component's Connection URL property.
limitStrategy: Controls how the component gets metadata about the query. Different databases have different ways of supporting this. Enter the method required by the data source you are configuring:
- top-n: SELECT TOP 1 * FROM (query).
- fetch-first-n: query FETCH FIRST 1 ROWS ONLY.
- limit-outer: SELECT * FROM (query) LIMIT 1.
- limit-inline: query LIMIT 1.
- rownum: SELECT * FROM (query) WHERE ROWNUM <=1.
- none: query.
fetchSize: The number of rows fetched at once, if supported by the database driver.
defaultProperties: Define any required defaults for driver properties. These can be overridden by the user when the component is configured.
blacklistedProperties: List any properties that you don't want to be displayed in the UI. For example, if you have set a default property here in the manifest file that you don't want to user to override.
propertyOverrides: If the driver is expecting property names that are different from those used by the component, you can map the expected names here. For example, MySQL expects "user" instead of "username" as used by the component, so you would enter "username": "user", here (this is shown in the example file, above).

You can download an example of this file here to use as a template for creating your own manifest file.

The following manifest can be used to configure SAP Hana as a data source:

[
    {
      "name" : "SAPHana",
      "driverClassName" : "com.sap.db.jdbc.Driver",
      "baseUrl" : "jdbc:sap://",
      "fetchSize" : 500,
      "limitStrategy" : "top-n",
      "defaultProperties": {
      },
      "blacklistedProperties": [
      ],
      "propertyOverrides": {
        "username": "user",
        "password": "password"
      }
    }
]

Deactivate soft delete for Azure blobs (Databricks)

If you intend to set your destination as Databricks and your stage platform as Azure Storage, you must turn off the "Enable soft delete for blobs" setting in your Azure account for your pipeline to run successfully. To do this:

Log in to the Azure portal.
In the top-left, click ☰ → Storage Accounts.
Select the intended storage account.
In the menu, under Data management, click Data protection.
Untick Enable soft delete for blobs. For more information, read Soft delete for blobs.

Snowflake	Databricks	Amazon Redshift
✅	✅	✅