Skip to content

Configuring a connection from Matillion ETL to Hub

Overview

Your Matillion ETL instance connects to the Hub to confirm your active subscription and which edition is licensed to be used. This action is commonly referred to as "phoning home", a particular form of data egress. After launching your Matillion ETL instance, you will need to ensure your instance can properly phone home before it is ready to use.

There are two main options available for egress to the Hub:

  1. Configuring a security group to allow outbound traffic on a port and for an IP address.
  2. Using a proxy server to allow outbound traffic to a domain.

The following sections will outline the above methods, and explain what to expect if your Matillion ETL instance cannot phone home to the Hub. In all cases, Matillion advises consulting your cloud provider administrator on the best course of action.


Connecting via security groups

For new VPCs (AWS) and VNets (Azure), a default security group is enabled that allows all outbound IPv4 traffic over all ports, and as such will always allow Matillion ETL to phone home. However, many organizations will use security groups with much stricter rules than this, and in this case you must ensure that those rules allow connection to the Hub. To configure your security group to allow the Hub connection, follow these steps.

1. Allow the following outbound IPs (on port 443) in your security group:

  • 13.248.217.21
  • 76.223.69.85

2. SSH into your instance that hosts the Matillion ETL client and open the following file for editing:

/usr/share/emerald/WEB-INF/classes/Emerald.properties

3. Set the BILLING_API_BASE_URL property to https://staticapi.billing.matillion.com. Example below:

BILLING_API_BASE_URL=https://staticapi.billing.matillion.com

4. Save and close the file.

5. Restart Tomcat using the following command:

sudo service tomcat restart

Read the following documentation for information about configuring security groups:

AWS-hosted
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html

Azure-hosted
https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview


Connecting via proxy servers

Proxy servers may be implemented to provide an additional layer of security, or to act as an intermediary between your servers and the internet. Depending on the scenario, proxy servers may help with URL and web content filtering, IDS/IPS, data loss prevention, monitoring, and advanced threat protection. With egress to the Hub in mind, a proxy server can provide a safe way of allowing outbound traffic specifically to the Hub and nowhere else.

Matillion ETL is a Java application and is hosted on an Apache Tomcat application server. To configure Tomcat to use a proxy server for HTTP and HTTPS communication (which Matillion ETL inherits), follow these instructions:

Please Note

  • Besides Matillion ETL, there are other applications that do not depend on Tomcat yet use the proxy server configuration of the underlying Linux operating system. Examples of these include: any system processes/services that run in the background, AWS CLI, Matillion ETL's Bash Script component, or Python scripts run using the Python 2/3 interpreters in Matillion ETL.
  • This guide assumes that your proxy server is already configured and is reachable from the Matillion ETL instance and that any ports used for communication are open on the respective security groups.

1. SSH to the Matillion ETL instance.

2. Edit the following file: /etc/sysconfig/tomcat.

3. In this file, edit the value of the JAVA_OPTS variable, so that the variable is set as follows:

JAVA_OPTS='-Djavax.net.ssl.trustStore=/usr/lib/jvm/jre/lib/security/cacerts -Djavax.net.ssl.trustStorePassword=changeit -Djava.security.egd=file:/dev/./urandom -XX:+UseG1GC -XX:OnOutOfMemoryError=/usr/share/emerald/WEB-INF/classes/scripts/oom.sh -Dhttps.proxyHost=[proxy host ip/name]; -Dhttps.proxyPort=[port number] -Dhttp.nonProxyHosts=localhost\\\\|127.*\\\\|169.254.169.254'

Replace [proxy host IP/name] and [port number] with the details of your own proxy server. This will tell outbound traffic to be directed to your proxy server but not for loopback addresses—on which some services rely.

Important

Although we use HTTPS as a best-practice example, you can choose to use HTTP or HTTPS—but importantly, you can only use one. Use one of the following property sets below with an appropriate proxy as detailed:

  • -Dhttp.proxyHost and -Dhttp.proxyPort properties should point to a proxy server that listens on HTTP/80
  • -Dhttps.proxyHost and -Dhttps.proxyPort properties should point to a proxy server that listens on HTTPS/443

In either case, -Dhttp.nonProxyHosts should remain as given in the snippet.

4. Save and close the file.

5. Next, you will need to edit the Emerald.properties file. Open the following file for editing:

/usr/share/emerald/WEB-INF/classes/Emerald.properties

6. On a new line in this file, add the desired environment variables, replacing the placeholder values [proxy host ip/name] and [port number] in the example below with your real proxy server values:

HTTP_PROXY=http://[proxy host ip/name]:[port number]/
HTTPS_PROXY=https://[proxy host ip/name]:[port number]/

Important

As in step 3, it is important to note that you can use HTTP or HTTPS, but again, only one can be used and this should correspond to what your proxy is configured to listen to:

  • HTTP_PROXY property should point to a proxy server that listens on HTTP/80
  • HTTPS_PROXY property should point to a proxy server that listens on HTTPS/443

7. Save the file and restart Tomcat using the following command:

sudo service tomcat restart

8. Finally, allow the address api.billing.matillion.com on your proxy server if necessary.


Successful connection

If you are now able to phone home, the connection has been established and will allow you to start using your instance. However, if this instance has not yet been associated with your Hub account, then the Hello there, we haven't seen this instance before dialog will be displayed, where you will be required to associate your Matillion ETL instance with your Hub account. To do this, click Associate instance. For more information, read Hub - Associating a Matillion ETL Instance.

Instance not associated with account


Unsuccessful connection

When you initially load your Matillion ETL instance, the Unable to Retrieve Account pop-up dialog will appear if you are experiencing connection issues to the Hub and your instance is unable to phone home. Click Retry. This will temporarily disable the button in an attempt to phone home again.

If after clicking Retry your instance still fails to phone home, the Unable to Retrieve Account dialog will again display, and this time provide information about why your instance cannot connect to the server. Click Retry again.

Unable to retrieve account Unable to retrieve account

Customers who have established a connection can use their Matillion ETL instance for up to a maximum of 14 days without having to phone home. Your instance will continue to work but will be set to a "Disconnected" state. In the Unable to Connect dialog, click Retry, and your instance will attempt to manually phone home.

If after 14 days, your instance is still showing as "Disconnected", it will move to a "Suspended" state. In the Unable to Connect dialog, click Retry, and your instance will attempt to manually phone home.

Unable to retrieve account Unable to retrieve account

Please Note

In the Unable to Connect pop-up dialog, click Contact Support for further assistance if you are unable to phone home. For more information, read Getting Support.