Skip to content

Has your Matillion ETL instance stopped working?

If you can't access the Matillion ETL login screen, you may have been impacted by the Hazelcast license expiry issue. You can verify if you are affected by this issue by checking the server logs on your Matillion ETL virtual machine for the following error:

SEVERE [localhost-startStop-1] com.matillion.bi.emerald.server.cluster.hazel.HazelClusterManager.<init> Hazel Failed to launch com.hazelcast.license.exception.InvalidLicenseException: Enterprise License has expired! Please contact your sales representative or contact support@hazelcast.com

The quickest way to fix this issue is to run the patching script in How to fix with internet connection via an SSH session or similar to the Matillion ETL instance:

How to fix with internet connection

Warning

You'll need your SSH key. If you've lost it, skip below to the section How to recover your SSH key (also available in the table of contents on the right).

This option lets you deploy the single fix for this licensing issue, without changing your Matillion ETL version.

If your Matillion ETL instance can connect to the internet—specifically to our Yum upgrade host, artifacts.matillion.com—we have provided an update script for the instance.

  1. SSH in to your Matillion ETL instance. Click here if you need help doing this.
  2. Run the following script:
sudo su
wget https://artifacts.matillion.com/jar/hazelcast/hazelcast_update.sh
bash hazelcast_update.sh

What steps does the script perform?

  1. Downloads a .jar file containing the fix.
  2. Stops the Matillion ETL service.
  3. Replaces the required file in the Matillion ETL library folder.
  4. Restarts the Matillion ETL service.

Note

The .jar file is verified before the server restart takes place. This operation typically requires only a few minutes of downtime.

How to fix without internet connection

Warning

You'll need your SSH key. If you've lost it, skip below to the section How to recover your SSH key (also available in the table of contents on the right).

This option lets you deploy the single fix for this licensing issue, without changing your Matillion ETL version

If your instance is isolated from the internet, you can still patch via SSH:

  1. Download the updated jar file https://artifacts.matillion.com/jar/hazelcast/emerald-<your version>.jar separately, where <your version> would be the Matillion ETL version. For example, 1.65.9.
  2. Replace the existing emerald-[your version].jar in the libraries folder, /usr/share/emerald/WEB-INF/lib/.
  3. Restart Matillion ETL, either by:
    1. Using systemctl restart tomcat8 in the command line in pre-1.69 versions.
    2. Using systemctl restart tomcat in post-1.69 versions.
    3. Choosing Restart Server from the Admin menu within Matillion ETL.

Confirm fix success

Confirm via license expiry date

  1. SSH in to your Matillion ETL instance.

  2. Switch to a super user:

    sudo bash

  3. Execute this command:

    Matillion ETL versions before v1.69:

    grep -i "hazelcast.*expiryDate" /var/log/tomcat8/catalina.out

    If the above doesn't work, your instance is on a version after v1.69. Try the filepath without 8 in tomcat:

    grep -i "hazelcast.*expiryDate" /var/log/tomcat/catalina.out

  4. Check that the last entry is expiryDate=Mar/31/2026 instead of expiryDate=Mar/31/2024:

Note

Some logs may display expiryDate=03/31/2026 23:59:59 and some may display expiryDate=Mar/31/2026 (03 versus Mar).

Invalid: ❌

26-Jan-2024 11:57:37.435 INFO [localhost-startStop-1] com.hazelcast.instance.NodeExtension.null [172.31.31.147]:5701 [Matillion] [3.12.13-SNAPSHOT] License{allowedNumberOfNodes=6, expiryDate=Mar/31/2024 23:59:59, featureList=[ Management Center, Clustered JMX, Clustered Rest, Security ], type=null, companyName=null, ownerEmail=null, keyHash=di4GH0ObvwFLK8cug4OIPHBScL3B8an5VNg7Fx1gyto=, No Version Restriction}

Valid ✅

26-Jan-2024 11:57:37.435 INFO [localhost-startStop-1] com.hazelcast.instance.NodeExtension.null [172.31.31.147]:5701 [Matillion] [3.12.13-SNAPSHOT] License{allowedNumberOfNodes=6, expiryDate=Mar/31/2026 23:59:59, featureList=[ Management Center, Clustered JMX, Clustered Rest, Security ], type=null, companyName=null, ownerEmail=null, keyHash=di4GH0ObvwFLK8cug4OIPHBScL3B8an5VNg7Fx1gyto=, No Version Restriction}

Confirm via number of nodes

  1. SSH in to your Matillion ETL instance.

  2. Switch to a super user:

    sudo bash

  3. Execute this command:

    Matillion ETL versions before v1.69:

    grep -i "hazelcast.*allowedNumberOfNodes" /var/log/tomcat8/catalina.out

    Matillion ETL versions after v1.69:

    grep -i "hazelcast.*allowedNumberOfNodes" /var/log/tomcat/catalina.out

  4. If the response is allowedNumberOfNodes=9999 then you have successfully patched your instance. ✅

  5. If the response is allowedNumberOfNodes=6, your instance still requires the patch in the steps above. ❌

How to recover your SSH key

Note

Once you recover your SSH key, read the appropriate section of this document:

If you have lost your SSH key to the Matillion ETL instance, there are several knowledge base (KB) articles that you should find helpful:

AWS

Azure

GCP

Need support?

If you need any help with this issue, contact the Matillion support portal.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement