Tech note - AWS Redshift RingBuffer exceeding expected limits
An instability issue has been detected between the AWS Redshift JDBC driver version
126.96.36.199 and Matillion ETL for Redshift versions:
This tech note provides a workaround for Matillion ETL for Redshift users who are experiencing either or both of the following when processing large data volumes:
- Out-of-memory (OOM) exception errors
- High CPU usage
The cause of this issue is that the AWS Redshift JDBC driver is not releasing memory from the following object queue,
RedshiftRowsBlockingQueue, which is causing exponential growth and leading to an out-of-memory exception after loading 1+ million rows.
Two workaround articles are available from the Matillion support team. One for reading data and one for writing data:
A GitHub issue submitted to AWS can be found here.