Using a proxy server with the agent
Agents do not interact with proxy servers by default, but can be configured to do so if required. This is true of both AWS and Azure hosted agents.
There is no support for using proxy servers with Matillion hosted agents in a Full-SaaS solution.
Note
This article is not intended as a guide for how to set up a proxy server; we assume you are already using a proxy in your infrastructure, and need to know how to use your Data Productivity Cloud agent with it.
Environment variables for proxy support
There are a number of optional environment variables that must be configured to use the agent with a proxy server. See below for how to configure these variables in an AWS or Azure based agent. The variables are:
| Variable | Description |
|---|---|
| PROXY_HTTP | Holds your HTTP proxy server name and port used. For example: myproxy.com:3000. |
| PROXY_HTTPS | Holds your HTTPS proxy server name and port used. For example: myproxy.com:3000. |
| PROXY_EXCLUDES | Lists addresses for the proxy to ignore. Separate multiple addresses using a pipe character. For example: example.com|example.net. |
| CUSTOM_CERT_LOCATION | Points to the storage location for custom certificates. For example, my_storage/my_certs |
Valid certificate file types
Only the .cer and .pem file types will be downloaded for use as certificates. When using external storage to supply the agent with certificates, any other file types in the storage location will not be downloaded by the agent.
AWS hosted agents
Loading externally hosted certificates
To load certificates into an AWS-hosted agent for proxy-routed communications, store the certificates in an S3 bucket that is in the same account space as the agent and accessible by the agent.
The IAM role for the account the agent is hosted in will need to have at least the following permissions to access S3 buckets:
- s3:ListAllMyBuckets
- s3:ListBucket
- s3:GetObject
- s3:GetBucketLocation
These will be applied automatically if you created the agent using the provided CloudFormation template.
The agent must have the CUSTOM_CERT_LOCATION environment variable added and set to the location of the bucket, for example s3://my-additional-libraries. You can omit s3:// from this, as the agent will assume the connection is to an S3 bucket and automatically use the correct protocol.
When you launch a new agent with the CloudFormation template, the configuration page in the AWS console will have a field for each environment variable you have created. Enter the proxy values you need for each variable.
If you are updating an existing agent, you will need to create a new revision of the task definition in use, and add CUSTOM_CERT_LOCATION plus any other optional environment variables you require, along with their required values. Then, restart the service using the new task definition.
Proxying service and container traffic
When both AWS service traffic and container traffic must pass through a proxy, the following additional steps will be needed:
- Deploy the service on an EC2 instance to gain more control over network configurations.
- Export the proxy configuration at node level. Configure the EC2 instance to route all outbound traffic through the proxy.
- Apply proxy variable settings for the agent, as described above.
Azure hosted agents
Loading externally hosted certificates
To load certificates into an Azure-hosted agent, store the certificates in an Azure Blob container in a storage account that is in the same resource group as the agent and accessible by the agent.
The storage account's managed identity associated with the agent's container will need to have at least the following permissions:
- Storage Account Contributor
- Storage Blob Data Contributor
- Storage Blob Data reader
These will be applied automatically if you create the agent using the provided ARM template.
The agent must have the CUSTOM_CERT_LOCATION environment variable added and set to the location of the Azure Blob container, for example https://mystorageaccount.blob.core.windows.net/my-certificates. You can omit https:// from this, as the agent will assume the connection is to a Blob container and automatically use the correct protocol.
When you launch a new agent with the ARM template, the configuration page in the Azure Portal will have a field named Custom Certificate Location, along with fields for each other environment variable you have created. Enter the proxy values you need for each variable.
If you are updating an existing agent, you will need to edit and deploy the existing container and add CUSTOM_CERT_LOCATION plus any other optional environment variables you require, along with their required values.