[TechNote] Setting the X-Forwarded-For header

Setting the X-Forwarded-For header using IBM HTTP Server

Question

How do I configure IBM HTTP Server to set a header called "X-Forwarded-For" equal to the IP address of the client?

Cause

The X-Forwarded-For header is not required by IBM HTTP Server (IHS) or by IBM WebSphere Application Server (WAS). But it may be required by software from other vendors.

Answer

For more information about the X-Forwarded-For header, see the definition on Wikipedia here: http://en.wikipedia.org/wiki/X-Forwarded-For

NOTE: No configuration is necessary when IBM HTTP Server is configured as a reverse proxy, as this header is added automatically by mod_proxy_http.

To configure IBM HTTP Server to set the X-Forwarded-For header to the client IP address, in a non-proxy request, you need to enable the mod_rewrite and mod_headers modules, and then add the following lines in the IBM HTTP Server config (httpd.conf):

RewriteEngine on RewriteRule ^(.*) - [E=CLIENT_IP:%{REMOTE_ADDR},L] RequestHeader set x-forwarded-for %{CLIENT_IP}e

In WebSphere Application Server, J2EE APIs that access the client IP address are not affected by the X-Forwarded-For header. The IBM web server Plug-in sends the client IP address to WebSphere Application Server in proprietary ($WS) headers. To learn more about HTTP headers, including the IBM $WS headers, please see the following webcast replay:
http://www.ibm.com/support/docview.wss?uid=swg27014842 

[TehcNote] WebSphere v8.x Plugin - UseInsecure (Plugin <==> WAS http offload)

WAS - transport chains - secure - disable
plugin - UseInSecure = true


Plug-in regeneration updates https transports in the plugin-cfg.xml file after the WCInboundDefaultSecure transport chain is disabled

Problem(Abstract)
The plug-in regeneration persists to update the plugin-cfg.xml file with the https transport ports for the WebSphere Application Server even after the WCInboundDefaultSecure transport chain is disabled.
Symptom
The plugin-cfg.xml would still display https transport port settings for the WebSphere Application Server.

Cause
If the user prefers to disable https communications between the plug-in and the application server, there is an additional https transport chain starting with WebSphere 7.0 and higher named HttpQueueInboundDefaultSecure that uses the same port as the WCInboundDefaultSecure chain which should be disabled.
Resolving the problem
Both HttpQueueInboundDefaultSecure and WCInboundDefaultSecure chains must be disabled before regenerating the plugin.


Procedure to disable https communication between the plug-in and WebSphere Application Server:

1) Login to the WebSphere Administrative console
2) Go to Servers -> Server Types > Application Servers -> server_name -> Web Container Settings -> Web container transport chains -> WCInboundDefaultSecure
3) Uncheck the box beside the "Enabled" option.
4) Click Apply and save settings.

Using the same procedure, disable the HttpQueueInboundDefaultSecure transport chain:

5) Go to Servers -> Server Types > Application Servers -> server_name -> Web Container Settings -> Web container transport chains -> click on HttpQueueInboundDefaultSecure

6) Uncheck the box beside the "Enabled " option under HttpQueueInboundDefaultSecure

7) Scroll to the bottom of the page and Click Apply, OK then click "Save directly to the Master configuration"

8) Remove the Web Container Secure Transport entry under Host Aliases:

a) From the left hand pane of the Admin console, Go to Environment -> Virtual Hosts -> Virtual Host_Name -> Host Aliases
b) Check the box beside the Web Container Secure Transport entry (default is 9443) and click the Delete button.
c) Click Apply, OK and "Save directly to the Master configuration."
d) Regenerate and propagate the plug-in.
e) Restart the application server and the web server to re-load the updated plug-in settings

PI10757: ALLOW PLUG-IN CUSTOM PROPERTY "USEINSECURE"IN A V7 DMGR, FOR GENERATING PLUGIN-CFG.XML INTENDED FOR A V8R5 RUNTIME.

...... skip ......

Problem conclusion
This custom property was previously only settable
when generating the plugin configuration in WebSphere
Application Server V8.5 or higher but with this apar, the
custom property can be set and picked up when generating the
configuration in WebSphere Application Server V7 or V8.
NOTE-This custom property is only recognized (and
only needed) in web server plugin v8.5 and higher.

Set the custom property UseInsecure=true on the Servers >
Web Servers > Web_server_name > Plug-in properties > Custom
properties page.

[TechNote] Enable SSLv3 설정 (v8.5.5.10 이후)

IBM은 문서가 잘 갖춰진 밴더 중 하나이나 너무 광범위하거나 여러 버전에 있어 잘 못된 정보를 가지고 일할때도 있는데, 동영상으로 명확히 가이드 해주는 내용을 찾으니 감사할 따름이다.
실력있는 엔지니어를 내쫓은 대신에 이런 거라도 있으니....

PI54960: PROVIDE PROPERTY TO SET JAVA SECURITY ALGORITHM RELATED PROPERTIES

Problem summary
****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server                                      *****************************************************************
* PROBLEM DESCRIPTION: The server needs a way to set java      *
*                      security properties                     *
*                      jdk.tls.disabledAlgorithms and         *
*                      jdk.certpath.disabledAlgorithms.       *****************************************************************
* RECOMMENDATION:                                              *
****************************************************************

WebSphere Application Server will setjdk.tls.disabledAlgorithms andjdk.certpath.disabledAlgorithms properties programmatically.Problem conclusionThe JRE has started disabling algorithms that are weak or areconsidered vulnerable.  The JRE disables these algorithms bysetting them on the jdk.tls.disabledAlgorithms andjdk.certpath.disabledAlgorithms Security properties in thejava.security file.  The jdk.tls.disabledAgorithms property isused to disable algorithms during TLS handshaking.  Thejdk.certpath.disabledAlgorithms is used to disable algorithmsduring certification path processing.   WebSphere does notmodify the java.security file in the service stream.   To  makesure the server is at the recommended level of securityWebSphere will be programmatically setting these properties.During server startup jdk.tls.disabledAlgorithms will be set toSSLv3, RC4, DH keySize < 768, MD5withRSA andjdk.certpath.disabledAlgorithms will be set to MD2, RSA keySize< 1024, MD5 programmatically.   And informational message willbe printed in the SystemOut.log file informing users whatWebSphere is setting them to.
There are 2 new WebSphere security custom properties that userscan use to either customize what is set by the Securityproperties or to tell WebSphere to not programmatically set theproperties at all.

1.  The com.ibm.websphere.tls.disabledAlgorithms security customproperty can either be used to tell WebSphere to set a customlist of algorithms to disable during TLS handshaking or if userdo not want WebSphere to programmatically set the java Securityproperty java.tls.disabledAlgorithms they can setcom.ibm.websphere.tls.disabledAlgorithms to none.2.  The com.ibm.websphere.certpath.disabledAlgorithms securitycustom property can either be used to tell WebSphere to set acustom list of algorithms to disable during certification pathprocessing or if user do not want WebSphere to programmaticallyset the java Security property java.certpath.disabledAlgorithmsthey can set com.ibm.websphere.certpath.disabledAlgorithms tonone.

To set a security custom property on the Admin Console go to:Security > Global security > Custom propertiesSelect New,  in the box labled Name addcom.ibm.websphere.tls.disabledAlgorithms orcom.ibm.websphere.certpath.disabledAlgorithms and in the boxlabeled Value enter either a comma separated list algorithms ornone if you don't want WebSphere to set the Security properties.Apply and  Save the changes.
The server will need to be restarted for the properties to takeeffect.

Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Security Bulletin

Summary

SSLv3 contains a vulnerability that has been referred to as the Padding Oracle On Downgraded Legacy Encryption (POODLE) attack. SSLv3 is enabled by default in IBM WebSphere Application Server. These fixes will disable SSLv3 completely.

Vulnerability Details

CVE ID: CVE-2014-3566
DESCRIPTION: IBM WebSphere Application could allow a remote attacker to obtain sensitive information, caused by a design error when using the SSLv3 protocol. A remote user with the ability to conduct a man-in-the-middle attack could exploit this vulnerability via a POODLE (Padding Oracle On Downgraded Legacy Encryption) attack to decrypt SSL sessions and access the plaintext of encrypted connections.

CVSS Base Score: 4.3
CVSS Temporal Score: See https://exchange.xforce.ibmcloud.com/vulnerabilities/97013 for the current score
CVSS Environmental Score*: Undefined
CVSS Vector: (AV:N/AC:M/Au:N/C:P/I:N/A:N)

Affected Products and Versions

This vulnerability affects all versions and releases of IBM WebSphere Application Server, IBM WebSphere Application Server Full Profile, IBM WebSphere Application Server Liberty Profile and IBM WebSphere Application Server Hypervisor Edition.

Remediation/Fixes

Please refer to the Security Bulletin for IBM HTTP Server to remediate your webserver.

Please note that these fixes will remove SSLv3 completely.

If you have SSL hard coded in your application code, such as SSLContext.getInstance("SSL") then you should install the interim fixes listed below since the current implementation defaults that context to SSLv3. The interim fix is an enhancement in the IBM JDK.

The following table lists some common standard protocol label names for Java 5 and above:

Protocol	Prior to this fix	After this fix
SSL	SSL v3.0	see chart below
SSLv3	SSL v3.0	Connection will fail
TLS	TLS v1.0 (defined in RFC 2246)	TLS v1.0 (defined in RFC 2246)
TLSv1	TLS v1.0 (defined in RFC 2246)	TLS v1.0 (defined in RFC 2246)
TLSv1.1	TLS v1.1 (defined in RFC 4346)	TLS v1.1 (defined in RFC 4346) (not available for version 6.1)
TLSv1.2	TLS v1.2 (defined in RFC 5246)	TLS v1.2 (defined in RFC 5246) (not available for version 6.1)
SSL_TLS	Enables all SSL V3.0 and TLS 1.0 protocols	TLS 1.0
SSL_TLSv2	Enables all SSL V3.0 and TLS 1.0, 1.1 and 1.2 protocols	TLS 1.0, 1.1 and 1.2 protocols (not available for version 6.1 and for version 7.0 only available with fix pack 7.0.0.23 or later.)

The IBM® SDK Java™ Technology Edition that is shipped with IBM WebSphere Application Server will be updated per the chart below, so that SSL Protocol alias label of "SSL" would mean the TLS levels marked.

Java Version	TLS 1.0	TLS 1.1	TLS 1.2
Java 7 Server	x	x	x
Java 7 Client	x
Java 6	x
Java 5	x

The interim fixes will disable SSLv3 by default. If you need to re-enable SSLv3, then there is a new java system property to enable SSLv3 with the protocols listed above. Set the system property either statically or dynamically as described in the product documentation for the IBM SDK Java Technology Edition you are using or in the Setting generic JVM arguments technote Warning: Reenabling this will make you vulnerable to the POODLE attack. 

-Dcom.ibm.jsse2.disableSSLv3=false   


Second, update the Oracle system property jdk.tls.disabledAlgorithms by removing SSLv3 from the list. This property is defined in the java.security file and any protocols in this list will be disabled. You can modify the setting to remove SSLv3 or add further algorithms with the following syntax: 

For the IBM SDK, Java Technology Edition that is used by IBM WebSphere Application Server Full Profile: 
For Java 7 or Java 8:

• Edit the java.security file and turn off SSLv3 by adding: jdk.tls.disabledAlgorithms=SSLv3

For the IBM SDK, Java Technology Edition or Oracle Java SDK, that is used by IBM WebSphere Application Server Liberty Profile:

• Edit the java.security file and turn off SSLv3 by adding: jdk.tls.disabledAlgorithms=SSLv3

This security property has precedence over the IBM system property com.ibm.jsse2.disableSSLv3 
Please refer to the knowledge center for more information: http://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.security.component.70.doc/security-component/jsse2Docs/disablesslv3.html?lang=en

[Tech-WebSphere] ExtendedDocumentRoot

WebSphere Application Server - ExtendedDocumentRoot config
원문:
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.iseries.doc/ae/rweb_jsp_staticfile.html

When the com.ibm.ws.webcontainer.ServeWelcomeFileFromExtendedDocumentRoot property is set to true, the web container searches in a static file extended document root for a static welcome file to serve for a request comprising a valid partial URL. For example, if a request specifies only the application context, and a welcome file is specified as index.html, the property must be set if index.html is to be served from a static file extended document.

The WebContainer has been modified to search an application defined ExtendedDocumentRoot when looking for a welcome file when the following WebContainer custom property is set to 
"true" (default is "false"): \\ \\ 

com.ibm.ws.webcontainer.ServeWelcomeFileFromExtendedDocumentRoot = true
com.ibm.ws.webcontainer.enablepartialurltoextendeddocumentroot = true


** ibm-web-ext.xmi

Static files:
<fileServingAttributes xmi:id="<user provided name>" name="extendedDocumentRoot"
 value="<user provided value>"/>

JSP files:
<jspAttributes xmi:id="<user provided name>" name="extendedDocumentRoot"
 value=<user provide value>"/>


** ibm-web-ext.xml

Static files:
<file-serving-attribute name="extendedDocumentRoot" value="/opt/extDocRootDir" />

JSP files:
<jsp-attribute name="extendedDocumentRoot" value="/opt/extDocRootDir","${MY_CUSTOM_VARIABLE}"  />

[Technote] Troubleshooting connection pooling (J2C) problems in WebSphere Application Server

Troubleshooting connection pooling (J2C) problems in WebSphere Application Server

Problem(Abstract)

Troubleshooting for connection pooling problems in IBM® WebSphere® Application Server. This should help address common issues with this component before calling IBM support and save you time.

Note: The troubleshooting steps in this document are applicable to JDBC connection problems using a standard data source in WebSphere Application Server. This document is not applicable to Version 4 data sources. You can also use this document for troubleshooting JMS connection problems and enterprise information system (EIS) connection problems.

Resolving the problem

Tab navigation

• Learn more
• Troubleshoot- selected tab,
• Collect data

Steps to help resolve connection pooling problems

1. Are you having a problem using a data source to establish a JDBC connection to a database? Most exceptions that occur when a data source is used to connect to a database will have the com.ibm.ws.rsadapter package in the stack trace of the exception. This includes java.sql.SQLExceptions and WebSphere Application Server messages that begin with DSRA.
   • Yes, go to the Troubleshooting JDBC connection problems section.
     
   • No, continue to question 2.
     
2. Are you having a problem using a JMS connection factory to establish a JMS connection to WebSphere MQ or another messaging system? Most exceptions that occur when a connection factory is used to connect to a messaging system will have the com.ibm.ejs.jms package in the stack trace of the exception. This includes javax.jms.JMSExceptions and WebSphere Application Server messages that begin with WMSG.
   • Yes, go to the Troubleshooting JMS connection problems section.
     
   • No, continue to question 3. 
     
3. Are you seeing J2CA0045E errors with ConnectionWaitTimeoutExceptions or slow performance in getting a connection?
   • Yes, go to the Troubleshooting connection wait issues, connection leaks, and performance problems section.
     
   • No, continue to question 4.
     
4. Are you seeing StaleConnectionExceptions or having problems recovering from invalid, or stale, connections in the pool?
   • Yes, go to the Troubleshooting stale connection problems section.
     
   • No, continue to question 5.
     
5. Go to the Miscellaneous connection pooling problems section. If your problem is not covered in that section or addressed elsewhere in this document, review the WebSphere Application Server V8.0, V8.5.5, or V9.0 product documentation, or the WebSphere Application Server Support site for additional information that might help you to resolve the problem, or continue to the MustGather for database connection and connection pooling problems.

Troubleshooting JDBC connection problems

1. This section will help you to troubleshoot problems that occur when using a data source defined in WebSphere Application Server to establish a JDBC connection to a database. JDBC connection problems usually occur at runtime when an application attempts to connect to the database and perform SQL queries. This section does not include ConnectionWaitTimeoutExceptions or StaleConnectionExceptions, which are covered in subsequent sections.
   
   To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. When diagnosing a database connection problem, the first step is to use the Test Connection button for the data source in the administrative console to test the connection to the database. You can find the Test Connection button in the data source configuration panel.
   
2. Was the attempt to connect to the database using the Test Connection button successful?
   • Yes, continue to question 3.
   • No, continue to question 4.
     
3. If you can use the Test Connection button to successfully connect to the database, but a failure occurs when your application tries to get a connection, the problem is likely caused by a failed JNDI lookup. Check the SystemOut.log to see if a NameNotFoundException occurs when the application tries to use the data source. Does a NameNotFoundException occur?
   • Yes, the root cause of the problem is that the JNDI lookup of the data source by the application fails. Check the data source JNDI name and ensure that it is bound to the JNDI namespace when the application server starts. If the JNDI name is jdbc/ds, you should see this entry in the SystemOut.log:
     WSVR0049I: Binding ds as jdbc/ds
     Also, check the application code and ensure that it is looking up the correct JNDI name. If you are doing an indirect JNDI lookup (i.e. java:comp/env/jdbc/ds), ensure that the binding is correct in the data source resource reference. The resource reference is configured in the web, EJB, or application client deployment descriptor. Checking for these things can help you to resolve many JNDI problems. However, if you can't determine the cause of the problem, continue to the Troubleshooting JNDI and naming problems.
     
   • No, continue to question 8.
     
4. The error message shown in the administrative console when the Test Connection button fails is important to understanding the cause of the problem. You can also check the SystemOut.log to get more information about the error and see the stack trace.
   
   Does the error message and the accompanying SQLException indicate a problem with the user ID and password that are used to connect to the database? The actual error and exception will differ depending on the JDBC driver and database that you're using.
   • Yes, the root cause of the problem is either that no userid and password are passed to the database, or the userid or password is not correct. Ensure that a J2C authentication alias, containing the correct userid and password, is specified on the data source. Also check with your DBA to make sure that the userid and password that you are using are correct for connecting to the database.
     
   • No, continue to question 5.
     
5. If you are using a Type 2 driver or Type 3 JDBC driver, does a java.lang.UnsatisfiedLinkError occur when you use the Test Connection button? A java.lang.UnsatisfiedLinkError occurs when the JVM is unable to load a native library that is needed by the JDBC driver.
   • Yes, the root cause of the problem is that the application server JVM is not properly configured to load the native libraries. Follow these instructions if you are trying to connect to a DB2® database. If you are trying to connect to an Oracle database, ensure that the ORACLE_HOME and the LIBPATH (on AIX®), LD_LIBRARY_PATH (on Sun Solaris or Linux®), or SHLIB_PATH (on HP-UX) JVM environment entries are set. The required JVM environment entries might be different for other databases.
     
     You should also make sure that the user that is being used to run the application server has the proper permissions to access the native libraries. Incorrect permissions are another common cause of the java.lang.UnsatisfiedLinkError.
     
     Finally, remember that 32-bit native libraries cannot be used with a 64-bit application server JVM and vice versa. Mismatches between the two can also cause the java.lang.UnsatisfiedLinkError.
     
   • No, continue to question 6.
     
6. If you are testing a connection to a Microsoft® SQL Server database, does the following error occur?
   java.sql.SQLException: [IBM][SQLServer JDBC Driver][SQLServer]Could not find stored procedure 'master..xp_jdbc_open'.
   • Yes, the root cause of the problem is that the stored procedures for JTA are not installed on the SQL Server database. These stored procedures are required for XA connections and for using the Test Connection button. Review this technote to resolve the problem.
     
   • No, continue to question 7.
     
7. If the error that you are getting is not one of the common errors described in the previous steps, review the error message and SQLException that occur when using the Test Connection button. Check the properties that you have specified on the data source to ensure that they are correct. Refer to your database documentation or your DBA for more information about why a particular error or exception might have occurred. An example of a typical error message that might occur is:
   
   
   DSRA8040I: Failed to connect to the DataSource.  Encountered : java.lang.Exception:          java.sql.SQLException: Io exception: The Network Adapter could not establish the connectionDSRA0010E: SQL State = null, Error Code = 17,002
   If you still cannot resolve the problem, continue to question 8.
   
8. Finally, if none of the previous troubleshooting steps helped to resolve the problem, try to connect to the database server using a stand-alone java program to determine if you are able to successfully connect to the database using the same properties that you have set in your data source outside of WebSphere Application Server. This will allow you to determine whether the problem is specific to WebSphere Application Server or if the problem occurs with the JDBC driver independently of WebSphere Application Server..
   • If a direct JDBC connection fails with the same error that occurs within WebSphere Application Server, the problem is not specific to your WebSphere Application Server environment. Work with your DBA to troubleshoot the problem.
     
   • If the direct JDBC connection is successful, continue to the MustGather for database connection and connection pooling problems.

Troubleshooting JMS connection problems

1. This section will help you to troubleshoot problems that occur when using a JMS connection factory to establish a connection to a messaging system such as WebSphere MQ. Only JMS connection problems are covered here.
   
   To troubleshoot this type of problem, you should have access to the SystemOut.log for your application server. When troubleshooting a JMS connection problem to MQ, it is useful to determine the MQ reason code associated with the JMSException. In the SystemOut.log, find the JMSException and review the stack trace. You should see one or more linked exceptions. Find the last linked exception. You should see a line like this:
   ---- Begin backtrace for Nested Throwables
   com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009
   
   The four-digit number is the reason code. You can review this document to find out more about the reason code that you see. The reason code helps you to understand the root cause of the problem.
   
2. Are you receiving a JMSException with MQ reason code 2009?
   • Yes, the root cause of the problem is that the connection to the MQ queue manager is broken or invalid. Review this technote to resolve the problem.
     
   • No, continue to question 3.
     
3. Are you receiving a JMSException with MQ reason code 2019?
   • Yes, the root cause of the problem could be that the connection pool Purge Policy is not set correctly. Review this technote to resolve the problem.
     
   • No, continue to question 4.
     
4. Are you receiving the following exception?
   
   
   J2CA0020E: The Connection Pool Manager could not allocate a Managed Connection: java.lang.IllegalStateException: Internal Error: cannot find the PoolManager Reference
   
   • Yes, the root cause of the problem is that the application caches or fails to close JMS session objects. Review this technote to resolve the problem.
   • No, continue to question 5.
     
5. Are either of the following authentication errors occurring when trying to establish a JMS connection?
   
   
   MSGS0508E: The JMS Server security service was unable to authenticate userid: myuser
   
   javax.jms.JMSSecurityException: MQJMS2013: invalid security authentication supplied for MQQueueManager
   
   • Yes, the root cause of the problem is either that no userid and password are passed to the queue manager, or the userid or password is not correct. Ensure that a J2C authentication alias is specified on your JMS connection factory. If you are using the WebSphere JMS Provider in V5 (Embedded Messaging), ensure that the userid specified in the J2C authentication alias is also in the user registry. 
   • No, continue to question 6.
     
6. Is the problem that you observe (using a tool such as netstat) more TCP/IP connections than you expect between your application server process and an MQ queue manager?
   • Yes, this is not necessarily a problem. This condition is explained in great detail in this technote. You should also review the technote if you want to gain an understanding of how JMS connection pooling in WebSphere Application Server works.
     
   • No, continue to question 7.
     
7. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather for connection pooling problems.

Troubleshooting connection wait issues, connection leaks, and performance problems

1. This section will help you to troubleshoot connection wait issues, including those that are caused by application connection leaks, and general performance issues that are related to connection pooling. These types of problems could occur for any type of backend system, including databases, messaging systems, and enterprise information systems.
   
   To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. It is also preferable for you to have access to the source code for your application(s). Additionally, obtaining javacores (also known as thread dumps) may help you to resolve the problem. 
   
   
2. The following technote can be used to diagnose what is causing the J2CA0045E errors, How to troubleshoot J2CA0045E connection pooling problems.
   
3. Are you experiencing a connection wait problem? Symptoms might include ConnectionWaitTimeoutExceptions in the SystemOut.log file, slower performance or delays when the application tries to get a connection, or threads hanging while waiting for a free connection (this can be seen in a javacore). An example stack trace for a thread that is waiting for a free connection is:
   
   "Servlet.Engine.Transports : 2833" (TID:0x31680590, sys_thread_t:0x78C96AA0, state:CW, native ID:0x86C7) prio=5
       at java.lang.Object.wait(Native Method)
       at com.ibm.ejs.j2c.poolmanager.FreePool.queueRequest(FreePool.java(Compiled Code))
       at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java(Compiled Code))
       at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
   ...
   • Yes, continue to question 4.
     
   • No, continue to question 9.
     
4. This situation would occur when the connection pool is at its maximum size (defined by the Maximum Connections property on the connection pool), all of the connections are in use, and the application requests more connections. The application must wait for a free connection to become available.
   
   Does the problem eventually clear up or does it persist until you restart the application server?
   • If the problem eventually clears up, the root cause of the problem is likely that Maximum Connections is not set high enough for the amount of load on the application. You should increase Maximum Connections for the connection pool. You should conduct thorough load testing to find the optimal value for Maximum Connections. You can enable PMI and monitor the connection pool counters in the Tivoli Performance Viewer to help you tune this.
     
   • If the problem does not clear up until you restart the application server, continue to question 5.
     
5. Is the application calling close() on every connection object that it obtains from a WebSphere Application Server connection pool? When an application gets a connection from the pool, it is considered "in use" until the application calls close() on the connection, which then returns the connection to the free pool. If the application does not call close(), the connection is leaked and never returns to the free pool. Eventually, the pool might become filled with leaked connections, causing connection wait problems.
   
   To determine if an application is leaking connections, you should review the application source code. Note that the application should close connections in a finally block to ensure that the connections are closed even in error conditions. In addition, you can use WebSphere Application Server tracing to identify connection leaks. The trace output can show you that a connection leak exists, and it can also show exactly what application code is leaking connections. Finally, the PMI connection pool countersmight also be useful in determining whether or not there is a connection leak.
   • If the application is failing to call close() on all connection objects, that is the root cause of the problem. You should fix the application code to resolve the problem.
     
   • If you determine that the application server is not leaking connections, continue to question 6.
     
6. Connection wait problems and slow application performance when using connection pooling can occur when the web container is not configured properly. If the web container thread pool size is set too high relative to the Maximum Connectionssetting for the connection pool, resource contention for the available connections could occur. It is also strongly recommended not to check the isGrowable checkbox for the web container thread pool.
   Is the web container thread pool size set too high or is the isGrowable checkbox checked in your configuration?
   • Yes, then this configuration is likely the root cause of the problem. Correct the issue and then test to see if the problem is resolved.
     
   • No, continue to question 7.
     
7. Other issues might cause connection wait problems. Applications should follow the "get/use/close" pattern and close all connections immediately after finishing using them. If an application takes too long to close connections, or caches connections, the connections will be in use longer, which could lead to connection wait issues. Even if the "get/use/close" pattern is followed, connections will not return to the free pool until the transaction in which the connection is obtained is committed. If the transaction remains active for a long time after a connection is closed, connection wait issues could occur. This frequently occurs in servlets when a shareable connection is obtained in a local transaction containment (LTC). More details about this scenario and solutions for it are documented in this article.
   
   Is your application following the "get/use/close" pattern and not caching connections, without long periods of time elapsing before the transaction is committed?
   • Yes, continue to question 8.
     
   • No, then the root cause of the problem is that connections are "in use" for too long before returning to the free pool. Correct the issue and then test to see if the problem is resolved.
     
8. Are there any network or database issues at the time that the problem occurs? Connection wait problems and performance problems might occur when network traffic is slow or when the database server is busy or performing slowly.
   • Yes, then the root cause of the problem is that connections are "in use" for long periods of time due to the network or database slowdown. Correct the problem with the network or the database server.
     
   • No, continue to question 9.
     
9. If you are seeing slow performance related to connection pooling without a connection wait issue, check the SystemOut.log. Are WSVR0605W warnings appearing in the SystemOut.log file?
   • Yes, this means that there is a thread that has been running as long as the threshold defined in the hang detection policy. Generate javacores to analyze why the thread is hung. This should reveal the root cause of the problem.
     
   • No, continue to question 10.
     
10. If none of the conditions described in this section are occurring, but the application is performing poorly or the application server appears to be unresponsive or hung, then use the appropriate MustGather document for an application server hang to analyze the threads of the application server to determine the cause of the problem.
    
11. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather for database connection and connection pooling problems.

Troubleshooting stale connection problems

1. This section will help you to troubleshoot problems with stale or invalid connections in the connection pool. WebSphere Application Server can determine that a connection is stale based on the exception that is returned by the backend when trying to establish a connection. When there is a stale database connection, WebSphere Application Server issues a StaleConnectionException. For other backends, a fatal connection error occurs. As a result, WebSphere Application Server purges the connection pool based on the Purge Policy setting and applications are able to recover from the problem. More information about this type of problem can be found here.
   
   To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. It is also preferable for you to have access to the source code for your application(s).
   
2. Is the backend system that you are connecting to a database, a messaging system, or an enterprise information system?
   • If it is a database, continue to question 3.
     
   • If it is a messaging system or an enterprise information system, continue to question 6.
     
3. If you are seeing StaleConnectionExceptions in the SystemOut.log, do they occur for every connection attempt or do they occur intermittently or only after a certain time period has elapsed?
   • If it occurs for every connection attempt, the root cause of the problem is likely that the data source properties are misconfigured. Follow the steps in the Troubleshooting JDBC connections section.
     
   • If it occurs intermittently or only after a certain time period has elapsed, continue to question 4.
     
4. Is the application handling the StaleConnectionException properly by catching the exception and then retrying the connection?
   • Yes, continue to question 5.
     
   • No, then the root cause of the problem is that the application is not catching the exception and retrying, so it cannot recover from the problem. Review the documentation on how to handle the StaleConnectionException and implement this in your application code.
     
5. A data source can be configured to pretest connections to ensure that they are valid before they are allocated to the application. This is documented here. Have you configured your data source to do this?
   • Yes, continue to question 6.
     
   • No, the problem can probably be resolved by configuring the data source to pretest connections before they are allocated to the application.
     
6. Is the Purge Policy set to EntirePool for the connection pool? This means that when a StaleConnectionException occurs, every free connection in the pool will be purged, which makes it easier for the application to recover. Documentation about the Purge Policy can be found here.
   • Yes, continue to question 7.
     
   • No, the problem symptoms can be alleviated by changing the Purge Policy to EntirePool.
     
7. Is the Minimum Connections property in the connection pool set to 0? Setting it to 0 enables the pool maintenance thread to clean up all of the connections in the pool after they are unused for more than the Unused Timeout number of seconds. If Minimum Connections is set to a value greater than 0, WebSphere Application Server must keep at least that number of connections in the pool indefinitely. The longer that a connection remains in the pool, the more susceptible it is to becoming stale.
   • Yes, continue to question 8.
     
   • No, the problem symptoms can be alleviated by changing the value of Minimum Connections to 0.
     
8. Does a firewall exist between the application server and the backend system that it is connecting to?
   • Yes, the root cause of the problem is likely that the firewall is timing out and dropping connections between the application server and the backend. To avoid this possibility, set the Unused Timeout to half the value of the timeout setting on the firewall. This way, WebSphere Application Server can clean up its unused connections before the firewall drops them.
     
   • No, continue to question 9.
     
9. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather for database connection and connection pooling problems.
   

Miscellaneous connection pooling problems

1. Are you seeing J2CA0075W warnings in the SystemOut.log?
   J2CA0075W: An active transaction should be present while processing method allocateMCWrapper.
   • Yes, the root cause of the problem is that the application is spawning its own threads and obtaining connections on those threads. Review this technote to resolve the problem.
     
   • No, continue to question 2.
     
2. Are you seeing J2CA0106E errors in the SystemOut.log?
   J2CA0106E: A 5.0 DataSource was attempted to be used in a WebModule that was not level 2.3
   • Yes, the root cause of the problem is that a servlet at the 2.2 specification level is trying to use a standard data source, which is not allowed. Review this technote to resolve the problem.
     
   • No, continue to question 3.
     
3. Are you running WebSphere Application Server V7, V8.5.5 or V9 and seeing J2CA0294W warnings in the SystemOut.log?
   J2CA0294W: Deprecated usage of direct JNDI lookup of resource jdbc/ds. The following default values are used: [Resource-ref settings]
     res-auth: 1 (APPLICATION)
     res-isolation-level: 0 (TRANSACTION_NONE)
     res-sharing-scope: true (SHAREABLE)
     loginConfigurationName: null
     loginConfigProperties: null
     [Other attributes]
   
     res-resolution-control: 999 (undefined)
     isCMP1_x: false (not CMP1.x)
     isJMS: false (not JMS)
   • Yes, it is not necessarily a problem and it can be prevented by creating a resource reference and doing an indirect JNDI lookup of your data source or connection factory. Review this technote to resolve the problem.
     
   • No, continue to question 5.
     
4. Are you seeing the following error in the SystemOut.log?
   J2CA0079E: Method getManagedConnection has detected an internal illegal state and is throwing an IllegalStateException. The exception is: java.lang.IllegalStateException: setManagedConnection: illegal state exception. State = STATE_INACTIVE
   • Yes, the root cause of the problem is a WebSphere Application Server defect that only occurs when tracing is enabled. Review this technote for further details and to resolve the problem.
     
   • No, continue to question 6.
     
5. Are you seeing J2CA0086W warnings in the SystemOut.log?
   J2CA0086W: Shareable connection MCWrapper id 686bbdf9 Managed connection WSRdbManagedConnectionImpl@6156bdf9 State:STATE_TRAN_WRAPPER_INUSE from resource jdbc/ds was used within a local transaction containment boundary.
   • Yes, this means that two or more connections were obtained within one local transaction containment (LTC). This doesn't necessarily indicate a problem, but you should review this technote for more details.
     
   • No, continue to question 7.
     
6. Are you seeing high memory usage or java.lang.OutOfMemoryErrors due to a high statement cache size?
   • Yes, the problem can probably be resolved by tuning the statement cache size. Review this technote for the details.
     
   • No, continue to question 8.
     
7. Are you seeing java.lang.ClassCastExceptions or other errors when using vendor specific, non-standard JDBC methods in your application?
   • Yes, the problem can probably be resolved by using the WSCallHelper class, provided by WebSphere Application Server. Refer to the WebSphere Application Server Information Center for information on the use of this class.
     
   • No, continue to question 9.
     
8. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather for database connection and connection pooling problems.

What to do next 
If the preceding troubleshooting steps did not solve your problem, see the MustGather for Connection Pooling problems to continue investigation.