AWS thread count increases leading to failing instances

10th October 2024

This tech note applies to Matillion ETL versions 1.75.10 and 1.76.0 and later for Snowflake on AWS.

We have received reports from customers using Matillion ETL for Snowflake on AWS infrastructure, observing AWS SDK thread counts increasing on their instances, to the extent that their instances may fail to respond and a restart is required.

---

Advised fix

A new configuration property, AWS_CLIENT_MAX_SCHEDULER_THREADS, has been introduced to control the maximum number of client AWS SDK threads allowed for AWS clients to connect to various AWS resources.

The default value is 500. Users can adjust this value by setting a valid value in the Emerald.properties file. This allows for greater flexibility, depending on the needs of the specific installation.

---

Instance configuration

1. SSH into your instance that hosts the Matillion ETL client and open the following file for editing:

   /usr/share/emerald/WEB-INF/classes/Emerald.properties
   

2. Add the following line:

   AWS_CLIENT_MAX_SCHEDULER_THREADS=<max_number_of_threads>
   

   Where <max_number_of_threads> is the user-defined maximum number of AWS SDK threads allowed on the instance.

   Ensure that the value is appropriate for the system capacity and expected AWS resource connections.

3. Save and close the file.

4. Restart Tomcat using the following command:

   sudo service tomcat restart