Skip to content

Data Transfer

The Data Transfer component enables users to transfer files from a chosen source to a chosen target.

This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component's other properties will change to reflect those choices.

Currently supported data sources include: FTP, SFTP, HTTP, HTTPS, Amazon S3, and Windows Fileshare.

Currently supported targets: Amazon S3, and Windows Fileshare.

Note

  • To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
  • When reading from or writing to Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
  • When the Source Type property is set to FTP, SFTP, HTTP, HTTPS, or Windows Fileshare, users will need to set a Source URL property. This property provides a template URL, with placeholder values inside square brackets [ ]. Please replace the placeholder values with the values of your actual source URL.

Warning

Currently, this component cannot move data between S3, Azure Blob, and GCP.

Properties

Name = string

A human-readable name for the component.

Source Type = drop-down

Select the type of data source. The source type will determine which source properties are required.

Unpack ZIP File = boolean

Select Yes if the source data is a ZIP file that you wish to unpack before being transferred.

Target Type = drop-down

Select the target type for the new file. The target type will determine which target properties are required.

Gzip Data = boolean

Select Yes if you wish to gzip the transferred data when it arrives at the target.

Target Object Name = string

The filename of the new file.

Source properties

Set Home Directory as Root = boolean

  • Yes: URLs are relative to the user's home directory.
  • No: (default) URLs are relative to the server root.

Source URL = string

The URL, including full path and filename, that points to the source file. The source URL template includes placeholder values inside either square brackets [ ] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.

For S3 sources, this property opens a Source URL dialog that allows you to select from a list of your existing S3 buckets.

Source Username = string

This is your URL connection username. It is optional and will only be used if the data source requests it.

Source Password = string

This is your URL connection password. It is optional and will only be used if the data source requests it.

Source SFTP Key = string

Your SFTP private key. It is optional and will only be used if the data source requests it.

This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.

The following private key formats are currently supported:

  • DSA
  • RSA
  • ECDSA
  • Ed25519

In a Hybrid Cloud configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:

ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem

Perform Certificate Validation = drop-down

Check the SSL certificate for the host is valid before taking data. Required for HTTPS sources only.

Source Domain = string

The domain that the source file is located on. Required for Windows Fileshare sources only.

Target properties

S3 Target URL = string

The URL of the S3 bucket to get the files from, including sub-folders. This follows the format s3://bucket-name/path/.

This property opens a Target URL dialog that allows you to select from a list of your existing S3 buckets and sub-folders.

Warning

When you select a bucket and sub-folder, the Target URL string will not have a terminating /. You must add this to the Target URL string manually. Failure to do so will cause the URL to be incorrectly resolved.

S3 Access Control List Options = drop-down

Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found in the Amazon documentation.

S3 Encryption = drop-down

Decide on how the files are to be encrypted inside the target S3 Bucket:

S3 KMS Key ID = drop-down

The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.

Windows Fileshare Target URL = string

The URL (without filename) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [ ] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.

Windows Fileshare Target Domain = string

The domain that the newly created file is to be located on.

Windows Fileshare Target Username = string

This is your URL connection username. It is optional and will only be used if the data source requests it.

Windows Fileshare Target Password = string

This is your URL connection password. It is optional and will only be used if the data source requests it.