Enable agent heap dumps

Agent Type:

Data Productivity Cloud Hybrid

Agent Platform:

AWS ✅ Azure ✅

A heap dump can be a useful tool for diagnosing out of memory (OOM) events in the Data Productivity Cloud agent. If you enable heap dumps for an agent, they can be sent to Matillion for analysis in the event of errors. This document provides instructions on how to enable heap dumps for the Data Productivity Cloud agent, as well as options for sanitizing heap dumps and creating heap dump reports to protect sensitive data.

---

Enable agent heap dumps

Heap dumps can be enabled for Java OOM events by configuring an environment variable for your agent. Set the following environment variable:

ENABLE_HEAP_DUMP = TRUE

Setting this will create heap dumps in the agent service, writing them to /home/agentuser/heap-dump by default. To change this location, set the environment variable HEAP_DUMP_PATH to the required directory.

Read Adding or updating AWS environment variables or Adding or updating Azure environment variables for instructions on how to set environment variables for your agent.

---

Storing heap dumps

With the agent deployed in a containerized environment, heap dumps won't persist when the container restarts. To persist the heap dump, it will have to be stored externally to the agent container. This will require HEAP_DUMP_PATH to be mapped externally.

The method used for storing heap dumps externally will depend on your environment and setup. You have a number of options for this, discussed in the following documents:

• Options for AWS (mounting Amazon EFS and transferring files to S3):

  • Use Amazon EFS volumes with Amazon ECS - Amazon Elastic Container Service.
  • What is AWS DataSync? - AWS DataSync.
  • Configuring AWS DataSync transfers with Amazon EFS - AWS DataSync.
  • Configuring AWS DataSync transfers with Amazon S3 - AWS DataSync.

• Options for Azure (ACI):

  • Mount Azure Files volume to container group - Azure Container Instances.

Your DevOps team will be able to advise on the best option for your environment.

---

Sanitize heap dumps

Before sending heap dumps externally we recommend sanitizing the heap dump files. This process removes variable data to protect sensitive information.

We recommend the Java application Heap Dump Tool is used to sanitize heap dump files. You will need Java installed to use the tool.

To download the tool:

1. Browse to https://mvnrepository.com/artifact/com.paypal/heap-dump-tool.
2. Choose the latest version.
3. Select the jar download link.

To run the tool, use the following command, replacing <heapdump-file-name> with the name of your heap dump file:

java -jar heap-dump-tool.jar sanitize <heapdump-file-name>.hprof heapdump-sanitised.hprof

This will create a new file called heapdump-sanitised.hprof with variable data removed. This file can be shared with Matillion for analysis without exposing sensitive data.

---

Create heap dump report

If preferred, you can further reduce the data sent to Matillion by creating a heap dump report. This process uses the Eclipse Memory Analyzer Tool (MAT) to create a report from the heap dump file, which can be shared with Matillion for analysis.

You can obtain the MAT from the Eclipse Foundation website at https://eclipse.dev/mat/.

To use the tool:

1. Move your sanitized heap dump file to a new directory, so it's the only file in the directory. The MAT will create additional files in this directory.
2. Open your sanitized heap dump with the MAT.
3. Once the heap dump has loaded, select Leak Suspects Report in the Getting Started Wizard. This will create report files in the same directory as the heap dump.
4. This file heapdump-sanitised_Leak_Suspects.zip contains an HTML Leak Suspects report. You can send just this zip file to Matillion for analysis.

For more information on using the tool, read the MAT documentation.