Bash Pushdown

Bash Pushdown is an orchestration component that enables users running Full SaaS configurations (Matillion-hosted agents) to use SSH connections to run their Bash scripts in their own instances. Any output from your Bash scripts is displayed in task messages. If you cancel a pipeline task while a Bash script is running, then the script terminates.

Any command line tools or credentials that you wish to use must be installed and set up on the instance you're using.

Refer to the Bash Reference Manual where appropriate.

Cloud credentials

To ensure that cloud credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.

Properties

Reference material is provided below for the Connection, Authentication, and Execution details properties.

Connection

Host = string

Your SSH host.

Username = string

Your SSH username.

Run echo "$USER" to prompt your shell for your username.

Connection Timeout (ms) = integer

Length of time in milliseconds before the connection will timeout. Default is 3000. The purpose of the timeout is to ensure scripts will never run forever even if they enter an infinite loop or are blocked by an external resource.

Port = integer

Enter your port number. The default is set to port 22.

Authentication

Authentication Type = drop-down

Choose whether to use a basic password or a key pair to authenticate. The default is key pair.

Password = drop-down

Available when Authentication Type is set to Basic Password.

Use the drop-down menu to select the corresponding secret definition that denotes the value of your password.

Read Secret definitions to learn how to create a new secret definition.

Private Key = drop-down

Available when Authentication Type is set to Key Pair.

Use the drop-down menu to select the corresponding secret definition that denotes the value of your private key.

Read Secret definitions to learn how to create a new secret definition.

Warning

For AWS users:

If you're storing a multi-line secret in AWS Secrets Manager:

Add your key and value to the Key/value tab of the Secret value section when storing your secret.
Click the Plaintext tab.
Replace any whitespace characters before and after ----- with \n. Do not remove whitespace characters in the BEGIN/END RSA PRIVATE KEY parts.

Example: {"dwh-bash-private-key":"-----BEGIN RSA PRIVATE KEY-----\nline1\nline2\nline3\n-----END RSA PRIVATE KEY-----"}

Alternatively, you can run the following code in your terminal, replacing values where appropriate:

    PEM_CONTENT=$(awk '{printf "%s\\n", $0}' /path/to/your/file.pem)

    aws secretsmanager create-secret \
      --name "MyKeyValueSecretWithPem" \
      --description "Secret with PEM file content" \
      --secret-string "{\"pem\":\"$PEM_CONTENT\"}"

Warning

For Azure users:

Do not store multi-line secrets via the Azure Key Vault GUI, as newlines may be stripped. Instead, use the Azure CLI depending on your use case. Read Store a multi-line secret in Azure Key Vault to work around this issue.

The following CLI command will maintain newlines:

    az keyvault secret set --vault-name <vault-name> --name <secet-name> --file <key-file-name>

Require Passphrase = boolean

Choose whether to require a passphrase as part of using key pair authentication. Default is false (no).

Passphrase = drop-down

Use the drop-down menu to select the corresponding secret definition that denotes the value of your passphrase.

Read Secret definitions to learn how to create a new secret definition.

Execution details

Script = text editor

The Bash script to run.

All the usual variables are made available in the bash environment and any changes made to such variables will never be visible outside of the current script execution.

Exit Script On Failure = boolean

Choose whether to terminate the script if any command fails. Default is true (yes).

Snowflake	Databricks	Amazon Redshift
✅	✅	✅