[FIXED] What goes into the "other" section of java's native memory tracking output?

Issue

I'm trying to debug a Java application with high memory usage. Using native memory tracking I found that there's a lot of memory being used by a section called "Other". I can't seem to find any information online about what might be stored here, what might cause it to be high, or how to debug it further. Any help along these lines would be greatly appreciated. For reference here's the output I get from jcmd:

Total: reserved=49GB, committed=48GB
-                 Java Heap (reserved=32GB, committed=32GB)
                            (mmap: reserved=32GB, committed=32GB)

-                    Thread (reserved=1GB, committed=0GB)
                            (thread #1291)
                            (stack: reserved=1GB, committed=0GB)

-                        GC (reserved=2GB, committed=2GB)
                            (mmap: reserved=1GB, committed=1GB)

-                     Other (reserved=14GB, committed=14GB)
                            (malloc=14GB #1520)

Solution

Other section of NMT report counts off-heap memory allocated by an application either through Direct ByteBuffers or immediately via Unsafe.allocateMemory. ByteBuffer.allocateDirect calls Unsafe.allocateMemory under the hood.

You can monitor the number of Direct ByteBuffers and the total memory occupied by them via JMX - look for BufferPool MBean:

Async-profiler can help to find where off-heap memory is allocated. Run

profiler.sh -d <duration> -e jdk.internal.misc.Unsafe.allocateMemory -f profile.html <pid>

This will create a flame graph, showing stack traces of all places where Unsafe.allocateMemory (including ByteBuffer.allocateDirect) is called from:

For more information about ByteBuffers and their relation to off-heap memory usage, see this presentation.

Answered By - apangin
Answer Checked By - Cary Denson (JavaFixing Admin)