Data Transfer
The Data Transfer component enables users to transfer files from a chosen source to a chosen target.
This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component's other properties will change to reflect those choices.
Currently supported data sources include:
- Azure Blob Storage
- FTP
- SFTP
- HTTP
- HTTPS
- Amazon S3
- Windows Fileshare
- Microsoft SharePoint
Currently supported targets include:
- Azure Blob Storage
- Amazon S3
- Windows Fileshare
Note
- To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
- When reading from or writing to Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
- When the Source Type property is set to FTP, SFTP, HTTP, HTTPS, or Windows Fileshare, users will need to set a Source URL property. This property provides a template URL, with placeholder values inside square brackets [ ]. Please replace the placeholder values with the values of your actual source URL.
Properties
Name
= string
A human-readable name for the component.
Source Type
= drop-down
Select the type of data source. The source type will determine which source properties are required.
Currently supported data sources include:
- Azure Blob Storage
- FTP
- SFTP
- HTTP
- HTTPS
- Amazon S3
- Windows Fileshare
- Microsoft SharePoint
Unpack ZIP File
= boolean
Select Yes if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type
= drop-down
Select the target type for the new file. The target type will determine which target properties are required.
Currently supported targets:
- Azure Blob Storage
- Amazon S3
- Windows Fileshare
Gzip Data
= boolean
Select Yes if you wish to gzip the transferred data when it arrives at the target.
Target Object Name
= string
The filename of the new file.
Source properties
Blob Location
= string
The URL that points to the source file that exists on Azure Blob Storage.
Clicking this property will open the Blob Location dialog. This displays a list of all existing storage accounts. Select a storage account, then a container, and then a subfolder if required. This constructs a URL with the following format:
AZURE://<account>/<container>/<path>
You can also type the URL directly into the Storage Accounts path field, instead of selecting listed elements. This is particularly useful when using variables in the URL, for example:
AZURE://${jv_blobStorageAccount}/${jv_containerName}
Special characters used in this field must be URL-safe.
Set Home Directory as Root
= boolean
- Yes: URLs are relative to the user's home directory.
- No: (default) URLs are relative to the server root.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
The source URL must start with sftp://
or else an authentication failure message will be returned.
Source Username
= string
This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
This is your URL connection password. It is optional and will only be used if the data source requests it.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
The source URL must start with sftp://
or else an authentication failure message will be returned.
Source Username
= string
This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
This is your URL connection password. It is optional and will only be used if the data source requests it.
Perform Certificate Validation
= drop-down
Check that the SSL certificate for the host is valid before taking data.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
The Source URL must start with sftp://
or else an authentication failure message will be returned.
Source Username
= string
Your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
The corresponding password. It is optional and will only be used if the data source requests it.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
When a user enters a forward slash character /
after a folder name, a validation of the file path is triggered. This works in the same manner as the Go button.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.
Source Domain
= string
The domain that the source file is located on.
Source Username
= string
Your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
The corresponding password. It is optional and will only be used if the data source requests it.
Service Type
= drop-down
The type of API that the component connects to. REST requires OAuth authentication, whereas SOAP requires basic authentication (username and password).
Authentication
= drop-down
If connecting to a REST Service Type, select a configured OAuth profile from the drop-down menu. Click Manage to navigate to the OAuth tab in a new browser tab and review your existing OAuth profiles and create new ones. To set up a new OAuth profile:
- Click Add OAuth connection.
- Provide a unique, descriptive OAuth name.
- Select your provider. In this case, SharePoint.
- Set the authentication type to OAuth 2.0 Authorization Code Grant.
- Provide your SharePoint URL.
- Click Authorize. A new browser tab will open to log in to
https://login.microsoftonline.com/
. - Provide your username and click Accept.
- The browser tab will close and you will be returned to the Data Productivity Cloud.
- Return to the Authentication parameter in your Data Transfer component and locate your new OAuth profile in the drop-down menu. If the profile has not propagated, click out of the dialog and then try again.
URL
= string
If connecting to a SOAP Service Type, enter the web address that you visit to log in to your SharePoint account. For example, https://companyname.sharepoint.com
.
User
= string
If connecting to a SOAP Service Type, enter a valid SharePoint username to use for authentication.
Password
= drop-down
If connecting to a SOAP Service Type, select the secret definition storing the password for your SharePoint account. Your password should be saved as a secret definition before using this component.
SharePoint Edition
= drop-down
Select your edition of Microsoft SharePoint. Options are SharePoint Online and SharePoint On-Premise.
Connection Options
= column editor
- Parameter: A JDBC parameter supported by the database driver. The available parameters are explained in the data model. Manual setup isn't usually required, since sensible defaults are assumed.
- Value: A value for the given parameter.
File Type
= drop-down
Select whether the file type is a Document or Attachment.
Library
= drop-down
The SharePoint library where the file is stored. The drop-down will be populated with libraries contained within your SharePoint instance.
For more information, read Introduction to libraries.
File URL
= string
The URL that points to the file.
Set Home Directory as Root
= boolean
- Yes: URLs are relative to the user's home directory.
- No: URLs are relative to the server root.
Source URL
= string
The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
The Source URL must start with sftp://
or else an authentication failure message will be returned.
Source Username
= string
Your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
The corresponding password. It is optional and will only be used if the data source requests it.
Source SFTP Key
= string
Your SFTP private key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
The following private key formats are currently supported:
- DSA
- RSA
- ECDSA
- Ed25519
In a Hybrid SaaS configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:
ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem
Target properties
Target Object Name
= string
The filename of the new file.
Blob Location
= string
The URL that points to the target file location on Azure Blob Storage.
Clicking this property will open the Blob Location dialog. This displays a list of all existing storage accounts. Select a storage account, then a container, and then a subfolder if required. This constructs a URL with the following format:
AZURE://<account>/<container>/<path>
You can also type the URL directly into the Storage Accounts path field, instead of selecting listed elements. This is particularly useful when using variables in the URL, for example:
AZURE://${jv_blobStorageAccount}/${jv_containerName}
Special characters used in this field must be URL-safe.
Target Object Name
= string
The filename of the new file.
Target URL
= string
The URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
Access Control List Options
= drop-down
Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
Encryption
= drop-down
Decide how the files are encrypted inside the S3 bucket. This property is available when using an existing Amazon S3 location for staging.
- None: No encryption.
- SSE KMS: Encrypt the data according to a key stored on KMS. Read AWS Key Management Service [AWS KMS] to learn more.
- SSE S3: Encrypt the data according to a key stored on an S3 bucket. Read Using server-side encryption with Amazon S3-managed encryption keys [SSE-S3] to learn more.
KMS Key ID
= drop-down
The ID of the KMS encryption key you have chosen to use in the Encryption property.
Target Object Name
= string
The filename of the new file.
Target URL
= string
The URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets []
or angled brackets <>
. Replace the placeholder values with the actual values of your source URL.
Target Domain
= string
The domain that the newly created file is to be located on.
Target Username
= string
Your URL connection username. It is optional and will only be used if the data source requests it.
Target Password
= string
The corresponding password. It is optional and will only be used if the data source requests it.
Snowflake | Databricks | Amazon Redshift |
---|---|---|
✅ | ✅ | ✅ |