Skip to content

Data Transfer

The Data Transfer component enables users to transfer files from a chosen source to a chosen target.

This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component's other properties will change to reflect those choices.

Currently supported data sources include: FTP, SFTP, HTTP, HTTPS, Amazon S3, and Windows Fileshare.

Currently supported targets: Amazon S3, and Windows Fileshare.

Note

  • To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
  • When reading from or writing to Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
  • When the Source Type property is set to FTP, SFTP, HTTP, HTTPS, or Windows Fileshare, users will need to set a Source URL property. This property provides a template URL, with placeholder values inside square brackets [ ]. Please replace the placeholder values with the values of your actual source URL.

Warning

Currently, this component cannot move data between S3, Azure Blob, and GCP.


Properties

Name = string

A human-readable name for the component.


Source Type = drop-down

Select the type of data source. The source type will determine which source properties are required.


Unpack ZIP File = boolean

Select Yes if the source data is a ZIP file that you wish to unpack before being transferred.


Target Type = drop-down

Select the target type for the new file. The target type will determine which target properties are required.


Gzip Data = boolean

Select Yes if you wish to gzip the transferred data when it arrives at the target.


Target Object Name = string

The filename of the new file.


Source properties

Set Home Directory as Root = boolean

  • Yes: URLs are relative to the user's home directory.
  • No: (default) URLs are relative to the server root.

Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.

The source URL must start with sftp:// or else an authentication failure message will be returned.


Source Username = string

This is your URL connection username. It is optional and will only be used if the data source requests it.


Source Password = string

This is your URL connection password. It is optional and will only be used if the data source requests it.

Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.

The source URL must start with sftp:// or else an authentication failure message will be returned.


Source Username = string

This is your URL connection username. It is optional and will only be used if the data source requests it.


Source Password = string

This is your URL connection password. It is optional and will only be used if the data source requests it.

Perform Certificate Validation = drop-down

Check that the SSL certificate for the host is valid before taking data.


Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.

The Source URL must start with sftp:// or else an authentication failure message will be returned.


Source Username = string

Your URL connection username. It is optional and will only be used if the data source requests it.


Source Password = string

The corresponding password. It is optional and will only be used if the data source requests it.

Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.

When a user enters a forward slash character / after a folder name, a validation of the file path is triggered. This works in the same manner as the Go button.

Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.


Source Domain = string

The domain that the source file is located on.


Source Username = string

Your URL connection username. It is optional and will only be used if the data source requests it.


Source Password = string

The corresponding password. It is optional and will only be used if the data source requests it.

Set Home Directory as Root = boolean

  • Yes: URLs are relative to the user's home directory.
  • No: URLs are relative to the server root.

Source URL = string

The URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.

The Source URL must start with sftp:// or else an authentication failure message will be returned.


Source Username = string

Your URL connection username. It is optional and will only be used if the data source requests it.


Source Password = string

The corresponding password. It is optional and will only be used if the data source requests it.


Source SFTP Key = string

Your SFTP private key. It is optional and will only be used if the data source requests it.

This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.

The following private key formats are currently supported:

  • DSA
  • RSA
  • ECDSA
  • Ed25519

In a Hybrid SaaS configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:

ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem

Target properties

Target Object Name = string

The filename of the new file.


Target URL = string

The URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.


Access Control List Options = drop-down

Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.


Encryption = drop-down

Decide how the files are encrypted inside the S3 bucket. This property is available when using an existing Amazon S3 location for staging.


KMS Key ID = drop-down

The ID of the KMS encryption key you have chosen to use in the Encryption property.

Target Object Name = string

The filename of the new file.


Target URL = string

The URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets <>. Replace the placeholder values with the actual values of your source URL.


Target Domain = string

The domain that the newly created file is to be located on.


Target Username = string

Your URL connection username. It is optional and will only be used if the data source requests it.


Target Password = string

The corresponding password. It is optional and will only be used if the data source requests it.


Snowflake Databricks (preview) Amazon Redshift (preview)