Data Transfer
The Data Transfer component enables users to transfer files from a chosen source to a chosen target.
This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component's other properties will change to reflect those choices.
Currently supported data sources include: FTP, SFTP, HTTP, HTTPS, Amazon S3, and Windows Fileshare.
Currently supported targets: Amazon S3, and Windows Fileshare.
Note
- To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
- When reading from or writing to Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
- When the Source Type property is set to FTP, SFTP, HTTP, HTTPS, or Windows Fileshare, users will need to set a Source URL property. This property provides a template URL, with placeholder values inside square brackets [ ]. Please replace the placeholder values with the values of your actual source URL.
Warning
Currently, this component cannot move data between S3, Azure Blob, and GCP.
Properties
Name
= string
A human-readable name for the component.
Source Type
= drop-down
Select the type of data source. The source type will determine which source properties are required.
Unpack ZIP File
= boolean
Select Yes if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type
= drop-down
Select the target type for the new file. The target type will determine which target properties are required.
Gzip Data
= boolean
Select Yes if you wish to gzip the transferred data when it arrives at the target.
Target Object Name
= string
The filename of the new file.
Source properties
Set Home Directory as Root
= boolean
- Yes: URLs are relative to the user's home directory.
- No: (default) URLs are relative to the server root.
Source URL
= string
The URL, including full path and filename, that points to the source file. The source URL template includes placeholder values inside either square brackets [ ] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
For S3 sources, this property opens a Source URL dialog that allows you to select from a list of your existing S3 buckets.
Source Username
= string
This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password
= string
This is your URL connection password. It is optional and will only be used if the data source requests it.
Source SFTP Key
= string
Your SFTP private key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
The following private key formats are currently supported:
- DSA
- RSA
- ECDSA
- Ed25519
In a Hybrid Cloud configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:
ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem
Perform Certificate Validation
= drop-down
Check the SSL certificate for the host is valid before taking data. Required for HTTPS sources only.
Source Domain
= string
The domain that the source file is located on. Required for Windows Fileshare sources only.
Target properties
S3 Target URL
= string
The URL of the S3 bucket to get the files from, including sub-folders. This follows the format s3://bucket-name/path/
.
This property opens a Target URL dialog that allows you to select from a list of your existing S3 buckets and sub-folders.
Warning
When you select a bucket and sub-folder, the Target URL string will not have a terminating /
. You must add this to the Target URL string manually. Failure to do so will cause the URL to be incorrectly resolved.
S3 Access Control List Options
= drop-down
Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found in the Amazon documentation.
S3 Encryption
= drop-down
Decide on how the files are to be encrypted inside the target S3 Bucket:
- None: No encryption.
- SSE Encryption: Encrypt the data according to a key stored on KMS. Read AWS Key Management Service (AWS KMS) to learn more.
- S3 Encryption: Encrypt the data according to a key stored on an S3 bucket. Read Using server-side encryption with Amazon S3-managed encryption keys (SSE-S3) to learn more.
S3 KMS Key ID
= drop-down
The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
Windows Fileshare Target URL
= string
The URL (without filename) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [ ] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
Windows Fileshare Target Domain
= string
The domain that the newly created file is to be located on.
Windows Fileshare Target Username
= string
This is your URL connection username. It is optional and will only be used if the data source requests it.
Windows Fileshare Target Password
= string
This is your URL connection password. It is optional and will only be used if the data source requests it.