Skip to content

Tech note - AWS Redshift RingBuffer exceeding expected limits

Note

Matillion ETL version 1.74.0 includes an updated Redshift driver (2.1.0.18). This updated Redshift driver does not have the issue described in this tech note.

An instability issue has been detected between the AWS Redshift JDBC driver version 2.1.0.9 and Matillion ETL for Redshift versions:

  • 1.68.x
  • 1.69.x
  • 1.70.x
  • 1.71.x

This tech note provides a workaround for Matillion ETL for Redshift users who are experiencing either or both of the following when processing large data volumes:

  • Out-of-memory (OOM) exception errors
  • High CPU usage

Cause

The cause of this issue is that the AWS Redshift JDBC driver is not releasing memory from the following object queue, RedshiftRowsBlockingQueue, which is causing exponential growth and leading to an out-of-memory exception after loading 1+ million rows.

Workarounds

Two workaround articles are available from the Matillion support team. One for reading data and one for writing data:

  1. Reading from Amazon Redshift
  2. Writing to Amazon Redshift

Additional information

A GitHub issue submitted to AWS can be found here.