Tech note - AWS Redshift RingBuffer exceeding expected limits
An instability issue has been detected between the AWS Redshift JDBC driver version 2.1.0.9
and Matillion ETL for Redshift versions:
- 1.68.x
- 1.69.x
- 1.70.x
- 1.71.x
This tech note provides a workaround for Matillion ETL for Redshift users who are experiencing either or both of the following when processing large data volumes:
- Out-of-memory (OOM) exception errors
- High CPU usage
Cause
The cause of this issue is that the AWS Redshift JDBC driver is not releasing memory from the following object queue, RedshiftRowsBlockingQueue
, which is causing exponential growth and leading to an out-of-memory exception after loading 1+ million rows.
Workarounds
Two workaround articles are available from the Matillion support team. One for reading data and one for writing data:
Additional information
A GitHub issue submitted to AWS can be found here.