Problem(Abstract)
WebSphere Application Server was migrated from 6.1.0.43 to 8.0.0.3 and the application started getting Java OutOfMemoryError (OOM).
Symptom
The log shows:
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError".
The Heap analyzer tool shows a memory leak. The prime suspect is org.apache.axis2.description.AxisService
Cause
The problem was triggered by changes in default behaviors between WebSphere Application Server 6.1.x and 8.0.x.
Changes that could affect WebSphere Application Server V8 and memory usage when using JAX-WS dynamic ports, which are those added using the service method addPort, might have additional memory requirements starting in WebSphere® Application Server Version 8.0. In previous releases, a single instance of a dynamic port could be shared across multiple service instances. In version 8.x, dynamic ports are now scoped to the instance of the service that added them. If a JAX-WS client has multiple service instances which refer to a dynamic port of the same name, those instances are no longer shared. This can potentially increase the memory requirements for that client. The memory used by dynamic ports is released when the service instance goes out of scope. However, if you encounter issues related to increased memory usage, it is possible to revert the behavior so that dynamic ports are again shared across service instances.
Note, version 8 includes a new default Garbage Collection policy, generational (genconn). In version 6.1, the default is optthruput.
Changes that could affect WebSphere Application Server V8 and memory usage when using JAX-WS dynamic ports, which are those added using the service method addPort, might have additional memory requirements starting in WebSphere® Application Server Version 8.0. In previous releases, a single instance of a dynamic port could be shared across multiple service instances. In version 8.x, dynamic ports are now scoped to the instance of the service that added them. If a JAX-WS client has multiple service instances which refer to a dynamic port of the same name, those instances are no longer shared. This can potentially increase the memory requirements for that client. The memory used by dynamic ports is released when the service instance goes out of scope. However, if you encounter issues related to increased memory usage, it is possible to revert the behavior so that dynamic ports are again shared across service instances.
Note, version 8 includes a new default Garbage Collection policy, generational (genconn). In version 6.1, the default is optthruput.
Environment
JAX-WS application running on WebSphere Application Server migrated from V6.1.0.43 to V8.0.0.3
Diagnosing the problem
Logs, javacores and heapdumps were collected for the OutOfMemory condition. Analysis of these files showed the leak suspect to be java/util/HashMap$Entry, which are allocated in class org/apache/axis2/jaxws/description/impl/ServiceDescriptionImpl.
Using PMAT tool on the GC trace in the logs confirmed the growth.
Using PMAT tool on the GC trace in the logs confirmed the growth.
Resolving the problem
Consider making the following changes:
- Increased the Java heap size
- Set the GC policy to the v6.1 default setting of gcPolicy:
-Xgcpolicy:optthruput - To revert the behavior so that dynamic ports are again shared across service instances, set the JVM custom property jaxws.share.dynamic.ports.enable = true. However, note that doing so can cause some other issues, such as having policy set attachments incorrectly applied across shared dynamic ports. If you set this flag to true and encounter some of these issues, then you should remove the flag setting.
- For WebSphere for z/OS if more than one Servant region is used consider adding the following WebSphere custom property:
com.ibm.ws390.routing.EnableCustomSessionCookieName=true
댓글 없음:
댓글 쓰기