Optional agent parameters

When creating an agent, there are a number of optional parameters that you may wish to set. These are listed on the Agent details screen under the heading Optional parameters that may be required. Optional parameters only need to be specified if you want to use the functionality they relate to.

To set optional parameters during agent creation, follow the instructions given in the appropriate document:

In the following sections, we describe the Data Productivity Cloud functionality that requires optional parameters to be set.

Any changes to the optional parameters on an existing agent will require the agent to be restarted. Read Restart an agent to learn more.

Using a proxy server and custom certificates

While agents don't interact with proxies by default, if your configuration requires a proxy then you should specify the following parameters.

Parameter	Description
PROXY_HTTP	Your HTTP proxy server and port. For example: `myproxy.com:3000`.
PROXY_HTTPS	Your HTTPS proxy server and port. For example: `myproxy.com:3000`.
PROXY_EXCLUDES	Addresses the proxy should ignore. Separate multiple addresses using the pipe character, `\|`. For example: `website.com\|website.net`.
CUSTOM_CERT_LOCATION	The storage location for custom certificates. For example: `my_storage/my_certs`.

Only the .cer, .pem, and .crt file types will be recognized as certificates. Any other file types in the storage location will be ignored by the agent.

For instructions on how to store certificate files and make them accessible to the agent, read Storing external files, below.

Note

Currently there is no functionality for handling proxy support in a Matillion Full SaaS environment.

Proxying AWS service and container traffic

When both AWS service traffic and container traffic must pass through a proxy, the following steps will be needed.

Deploy the service on an EC2 instance to gain more control over network configurations.
Export the proxy configuration at node level. Configure the EC2 instance to route all outbound traffic through the proxy.
Configure the proxy parameters on the agent, described above.

Loading external drivers

The AWS and Azure hosted agents natively include drivers for a large number of third-party connectors. There are, however, certain external drivers that aren't supplied by default. If a driver you need isn't supplied by default, you can upload your own drivers to the agent at runtime by specifying a storage location in your infrastructure that will hold the driver files. You specify the location in the following agent parameter.

Parameter	Description
EXTERNAL_DRIVER_LOCATION	The storage location for any additional driver files. For example: `my_storage/my_drivers`.

Only the .jar and .so file types will be recognized as external drivers. Any other file types in the storage location will be ignored by the agent.

For instructions on how to store these files and make them accessible to the agent, read Storing external files, below.

Note

Adding additional unsupported drivers to the agent will result in increased startup time with no guarantee that they will work, so we don't recommend doing this. Supported drivers are listed here.
Currently there is no support for importing external drivers to Matillion-hosted agents in a Full SaaS environment.

Loading additional Python libraries

The Python Script component lets you import modules from any Python library. To do this, you need to put the Python library into an Amazon S3 or Azure Blob Storage location that is accessible to the agent, and set the following parameter to point to the location.

Parameter	Description
EXTENSION_LIBRARY_LOCATION	The storage location of any additional Python libraries. For example: `my_storage/my_pythonlibs`.

For instructions on how to store library files and make them accessible to the agent, read Storing external files, below.

Storing external files

For the agent to access external files (such as certificates or drivers), the files must be stored in an accessible location in your infrastructure. Follow the appropriate instructions, below, depending on whether your agent is hosted on AWS or Azure infrastructure.

All files of one type must be stored in the same location. Different types may be stored in different locations, however. For example, all certificates must be stored together in one location pointed to by CUSTOM_CERT_LOCATION, while all Python libraries can be in a different location pointed to by EXTENSION_LIBRARY_LOCATION.

AWS-hosted agents

To make external files accessible to an AWS-hosted agent, store the files in an S3 bucket that's in the same account space as the agent and accessible by the agent.

The IAM role for the account the agent is hosted in will need to have at least the following permissions to access S3 buckets:

s3:ListAllMyBuckets
s3:ListBucket
s3:GetObject
s3:GetBucketLocation

These will be applied automatically if you created the agent using the provided CloudFormation template.

Set the appropriate environment variables to point to the file location, as described above. For example, s3://my-bucket/my-files. You can optionally omit s3:// from this, as the agent will assume the connection is to an S3 bucket and automatically use the correct protocol.

If you are updating an existing agent, you will need to create a new revision of the task definition in use, and add the environment variable with the appropriate path. Then, restart the service using the new task definition.

Azure-hosted agents

To make external files accessible to an Azure-hosted agent, store the files in an Azure Blob container in a storage account that's in the same resource group as the agent and accessible by the agent.

The storage account's managed identity associated with the agent's container will need to have at least the following permissions:

Storage Account Contributor
Storage Blob Data Contributor
Storage Blob Data Reader

These will be applied automatically if you create the agent using the provided ARM template.

Set the appropriate environment variables to point to the file location, as described above. For example, https://mystorageaccount.blob.core.windows.net/my-files. You can optionally omit https:// from this, as the agent will assume the connection is to a Blob container and automatically use the correct protocol.

If you are updating an existing agent, you will need to edit and deploy the existing container, and add the environment variable with the appropriate path. Then, create the new revision.