[TechNote] Interpreting svmon output to determine if a native memory leak is the cause of a java.lang.OutOfMemoryError

Interpreting svmon output to determine if a native memory leak is the cause of a java.lang.OutOfMemoryError

Technote (troubleshooting)

Problem(Abstract)

A java.lang.OutOfMemoryError issued by the Java™ Virtual Machine can indicate a memory leak.

Monitoring heap memory usage in the verboseGC output shows decreasing amounts of free space on the Java heap if the java.lang.OutOfMemoryError exception is due to a leak of Java objects. However, if the verboseGC output does not show this behavior, it is probable that the java.lang.OutOfMemoryError is caused by a native memory leak.

This technote describes how to interpret AIX svmon data to determine if a native memory leak has caused the java.lang.OutOfMemoryError.

Cause

A native memory leak is caused by a native library that is allocating memory, but not freeing it.

Resolving the problem

This technote assumes you have already followed MustGather: Out of Memory errors on AIX, Linux, or Windows and collected the necessary documentation.

Use the svmon data to determine which segments of the process are growing. If the growth is in the native heap segments of the process, the problem is due to a native memory leak. Here is a sample output from SVMON:

-----------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
90190 java 198140 3252 1857 191470 N Y N

Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
33899 4 work working storage - 65536 0 0 65536
1760 3 work working storage - 65519 0 0 65519
2f136 6 work working storage - 44158 0 0 44158
149aa d work loader segment - 8622 0 19 8679
0 0 work kernel segment - 4779 3115 1838 5631
359ba - pers /dev/lv04:2053 - 1839 0 - -
13949 5 work working storage - 798 0 0 798
116c8 - pers /dev/lv01:2125 - 571 0 - -
3991c 7 work working storage - 560 0 0 560


This is the output from a screenshot of the svmon utility:

• ESID is the segment ID for the segment being used. The ESID corresponds to the segment numbers referenced above.
  
• INUSE is the memory (in pages) used by the segment. Each page is 4KB. The maximum number of pages that may be taken up by a segment is 65536.

Determining if there is a native memory leak
Check INUSE value by taking svmon snapshots over time. A memory leak exists if INUSE increases for any ESIDs that correspond to the native heap in each subsequent snapshot.

This sample shows svmon data for segment ESID from five consecutive svmon snapshots. Notice that the INUSE value of ESID 5 (also known as segment 5) appears to be increasing over time. This is evidence of a memory leak in that segment:

 Vsid  Esid Type Description     LPage Inuse   Pin   Pgsp  Virtual
 13949  5   work working storage   -   798     0     0     798 
 13949  5   work working storage   -   1798    0     0     798 
 13949  5   work working storage   -   3798    0     0     798 
 13949  5   work working storage   -   5798    0     0     798 
 13949  5   work working storage   -   10000   0     0     798 
 and so on...
 

INUSE growth can indicate a problem with:

• Java SDK Java Native Interface (JNI) methods
  
• Native calls elsewhere
  
  For example, JNI calls from a user application, JDBC™ drivers for DB2®, WebSphere® MQ libraries, or third party native code.
  
  Check for:
  
  • Nonstandard libraries in the application; determine if the native leak is related to these.
  • Which segment is leaking memory; this can be an indication of the application that is leaking memory. For example, DB2 CLI code uses segment 3, while MQ uses segment 8.
• The Just In-Time (JIT) compiler might be using the native memory. Disabling JIT might eliminate the memory leak. If this does prevent a memory leak, this is indicative of a problem in the JIT compiler of the JDK. Report this to IBM support.