Problem(Abstract)
When the IHS reverse proxy with IBM WebSphere Application Server 7 is used with Rational Team Concert (RTC) Enterprise Extensions' long running functions such as builds, scans, queries, or searches, 500 Internal Server Errors can occur if the serverIOTimeout value in the IHS plugin is set to the default of 60 seconds.
Symptom
Similar errors will show in the dialog windows and in the logs such as:
2013-11-14 12:40:41,199 [WebContainer : 12] INFO com.ibm.team.jfs.app.http.client.JazzHttpClient - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request: The target server failed to respond
or
CRJAZ0098I The "com.ibm.team.build.internal.common.ITeamBuildRequestService(/com/service/com.ibm.team.build.internal.common.ITeamBuildRequestService)" service fails. The server returned the HTTP error 500 with error text "Internal Server Error".
or
CRJAZ0098I The "com.ibm.team.build.internal.common.ITeamBuildRequestService(/com/service/com.ibm.team.build.internal.common.ITeamBuildRequestService)" service fails. The server returned the HTTP error 500 with error text "Internal Server Error".
Cause
Due to increased network traffic or long running functions, the WebSphere Application Server running the Rational Team Concert CCM application may not respond to the build agent, z/OS, or the client within the time required or vice versa. The IHS server being the intermediate handler will not process the request from the application server within this time and an ambiguous 500 Internal Server Error is returned.
Timeout values can be set higher for the build agent and the client to overcome individual problems, but if IHS is running as the reverse proxy, the 500 errors may still randomly occur.
Timeout values can be set higher for the build agent and the client to overcome individual problems, but if IHS is running as the reverse proxy, the 500 errors may still randomly occur.
There have been some defects resolved that do affect specific timeout issues. If after applying these, the 500 Internal Server Errors still occur, you should investigate the IHS plugin settings.
Environment
Enterprise Extension dependency builds, promotions, scans, context aware search, or any function that is long running through the IHS reverse proxy with WebSphere Application Server 7 with a default value of 60 as ServerIOTimeout can cause these errors.
Diagnosing the problem
The best method of diagnosing is to enable logging in the IHS plugin and monitor the access.log, error.log, and httpd_plugin.log for the response times and any 500 errors over a specific time frame.
To enable logging in the IHS http and plugin:
- Enable mod_status
- Enable mod_mpmstats
- Set LogFormat to include %T
- Enable the http plugin trace
In the httpd.conf, this example shows mod_mpmstats and mod_status enabled:
- # Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
#
<IfModule mod_status.c>
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from specific_company.com
</Location>
</IfModule>
# mod_mpmstats logs statistics about server activity to the
# main
# error log. No records are written while the server is idle.
LoadModule mpmstats_module modules/debug/mod_mpmstats.so
<IfModule mod_mpmstats.c>
ReportInterval 60
# Include details of active module in the statistics.
TrackModules On
</IfModule>
Also, in the httpd.conf, edit LogFormat to include %T to determine which requests consistently take the longest amount of time
LogFormat "%h %l %u %t TIME: %T \"%r\" %>s %b" common
This edit does require a restart of the IHS server.
Lastly, for the IHS plugin trace, change the log level to trace. Setting the plugin trace in plugin-cfg.xml. does not need a server reboot to apply.
As an example,
<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/Plugins/logs/http_plugin.log"/>
Monitor the access.log and error.log and httpd_plugin.log for errors and the times.
Once the longest response time is determined, then the value for serverIOTimeout should be set to 2-3 times higher than the longest response time.
Resolving the problem
If 500 Internal Server errors from the application server are seen during long builds or functions when using Rational Team Concert Enterprise Extensions and the IHS reverse proxy, check the serverIOTimeout setting for the IHS plugin, plugin-cfg.xml.
In the plugin-cfg.xml, the value is listed as ServerIOTimeout=60 as below.
<Server ConnectTimeout="5" ExtendedHandshake="false" MaxConnections="-1" Name="Server1" ServerIOTimeout="60" WaitForContinue="false">
<Transport Hostname="Emeric" Port="9081" Protocol="http"/>
<Transport Hostname="Arlin" Port="9444" Protocol="https">
<Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb"/>
<Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.sth"/>
</Transport>
In WebSphere Application Server 8, the ServerIOTimeout default is 900 seconds.
Note: Setting to 0 indicates no timeout and could cause the IHS server to crash as incoming threads are not released until they are responded to. A valid value should be determined and set.
댓글 없음:
댓글 쓰기