[TechNote] OutOfMemoryError (OOM) and Leak Detected in IBM PoolManager When Using Oracle JDBC Driver

OutOfMemoryError (OOM) and Leak Detected in IBM PoolManager When Using Oracle JDBC Driver

Technote (troubleshooting)

Problem(Abstract)

OutOfMemoryError or unexpectedly high heap utilization when using Oracle JDBC Driver and stored procedures. Leak detection logic may point to com.ibm.ejs.j2c.PoolManager however the leak is due to how stored procedures are processed by the JDBC Driver.

Symptom

OutOfMemoryError or high heap consumption while using stored procedures with Oracle JDBC. Current heap analyzers will point to the com.ibm.ejs.j2c.PoolManager as the source of the leak.

Cause

The PoolManager pools database connections and provide a performance improvement to hosted applications. Customers who experience an OutOfMemoryError where the leak suspect is the PoolManager may be mislead into thinking the database connection pool is exceeding the configured size. The underlying problem is that the Oracle JDBC Driver accumulates oracle.jdbc.driver.T2CStatement objects until the database connection is closed. The T2CStatement object represents the cursor of a stored procedure’s result set and is not closed when the result set is closed. Oracle is working on a design change under bug # 9092263.

Diagnosing the problem

Identify a large accumulation of T2CStatement objects by loading the heapdump (or system core) into the Memory Analyzer Tool and running the following OQL:

select * from instanceof oracle.jdbc.driver.T2CStatement

This will show all of the T2CStatement objects however the T2CStatement’s are often linked together similar to a link list. The following OQL string will select the first T2CStatement object in each link. select * from instanceof oracle.jdbc.driver.T2CStatement s where (classof(dominatorof(s)).getName()!="oracle.jdbc.driver.T2CStatement")

The Memory Analyzer Tool can be downloaded as either an ISA Plug-in or as a standalone tool from
eclipse. The standalone version of MAT requires a DTFJ plug-in to open heap dumps from the IBM JDK.
IBM Support Assistant download page:
http://www-01.ibm.com/software/support/isa/
Memory Analyzer Tool homepage:
http://www.eclipse.org/mat/
Instructions on adding DTFJ plug-in to standalone Memory Analyzer Tool:http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html

Resolving the problem

Set a connection-aged timeout on the database pool until the design change is made available from Oracle. The recommended value is ¼ the time it takes for the server to reach OOM after restart. For example, if it takes 2 hours for the server to reach OOM then set the aged timeout to 10800 (30 minutes).

Aged Timeout in the WebSphere Application Server 6.1 Information Center:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1//index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpoolset.html