Connect to Delta Lake on Databricks
Connect to Delta Lake on Databricks via Data Loader and use it as your destination for batch-loading a pipeline.
Add AWS credentials
Select an existing AWS cloud credential, or click +Add AWS cloud credential to make a new one.
| Property | Description |
|---|---|
| AWS credential label | A unique, descriptive name for your new AWS credential. |
| Access Key ID | An AWS access key. Read Understanding and getting your AWS credentials for more information. |
| Secret Access Key | An AWS secret access key. Read Understanding and getting your AWS credentials for more information. |
Click Test and save to continue.
Connect to Delta Lake on Databricks
| Property | Description |
|---|---|
| Destination label | A unique, descriptive label for your destination. |
| Workspace ID | Your Databricks workspace identifier. Read Get workspace, cluster, notebook, folder, model, and job identifiers to learn more. |
| Token | A Databricks personal access token. Read https://docs.databricks.com/dev-tools/api/latest/authentication.html to learn more. |
| Username | Your Databricks login username. |
| Password | A managed entry representing your Databricks login password. Choose an existing password from the dropdown menu or click Manage and then click Add new password to configure a new managed password entry. Give the password a label, which is what you can see in the password dropdown menu, and then input the value of the password. Read Manage Passwords to learn more. |
Click Test and continue to test your settings and move forward. You can't continue if the test fails for any reason.
Configure destination
To configure the destination, you need to provide the following properties:
| Property | Description |
|---|---|
| Endpoint/Cluster | Select a Databricks cluster from the drop-down list. The available options are populated based on your Databricks account. |
| AWS S3 | This section is related to configuring the Amazon S3 settings. If you select AWS S3, you can add AWS cloud credentials. |
| Schema | Select a schema for the target tables. |
| S3 bucket | Select an Amazon S3 bucket where you want to load the data into. |
| Table prefix | (Optional) Specify a string to prepend to the table names. This can be useful for organization or naming conventions. |
| Personal | This section is related to personal settings. |
| Default Catalog | Select the default catalog for the Delta Lake tables from the drop-down list. |
| Default Schema | Select the default schema for the Delta Lake tables from the drop-down list. |
Click Continue