레이블이 IBM.WebSphere.Problem인 게시물을 표시합니다. 모든 게시물 표시
레이블이 IBM.WebSphere.Problem인 게시물을 표시합니다. 모든 게시물 표시

2021년 1월 7일 목요일

 SSL handshake fails due to TLSv1 used on IBM JDK


SSL handshake fails due to TLSv1 used on IBM JDK

SSL handshake failure caused by wrong TLS version used by Mule. This article explains how to force Mule runtime running on IBM JDK to use TLSv1.1/1.2.


SYMPTOM

Mule server is running on IBM JDK. When making an outbound call to Salesforce which only supports TLSv1.2, the following error is received:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure 
at com.ibm.jsse2.k.a(k.java:15) ~[?:8.0 build_20180305] 
at com.ibm.jsse2.k.a(k.java:23) ~[?:8.0 build_20180305] 
at com.ibm.jsse2.av.b(av.java:343) ~[?:8.0 build_20180305] 


CAUSE

TLSv1 is used by IBM JDK to make the external HTTP call. 

SOLUTION

Please specify the supported TLS version using the following system parameters:
jdk.tls.client.protocols=TLSv1.1,TLSv1.2
https.protocols=TLSv1.1,TLSv1.2

 Problem: ArcGIS resources are inaccessible through HTTPS when the Web Adaptor is deployed on WebSphere (to tls1.2 https)



Description

In ArcGIS 10.6.1, resources are inaccessible through HTTPS when the Web Adaptor is deployed on WebSphere. This can manifest itself in the following ways:

  • When configuring ArcGIS Server with the Java Web Adaptor, the user interface displays the following error message:
Error:
Unable to register the ArcGIS Server with the Web Adaptor. Please make sure that the server machine is running and that the account specified has administrative privileges to the site.
  • When configuring Portal for ArcGIS with the Java Web Adaptor, the user interface displays the following error message:
Error:
Unable to configure Portal with the Web Adaptor. Please make sure that the Portal machine is running and that the account specified has administrative privileges to the Portal.
  • The error in the WebSphere application server logs is
Error:
Unable to get token for user <username>, Error in admin request for get machines Received fatal alert: handshake_failure.

Cause

Starting at ArcGIS 10.6.1, the secure socket layer (SSL) protocol TLSv1 is disabled by default for both Portal for ArcGIS and ArcGIS Server due to Payment Card Industry (PCI) compliance. By default, WebSphere uses SSL-TLS as the SSL handshake protocol, which is SSLv3 and TLSv1, and hence terminates HTTPS connections from the Web Adaptor to the ArcGIS Server and Portal for ArcGIS applications.

Note:
Starting with ArcGIS Web Adaptor for Java 10.3, SSLv3 is no longer supported to prevent the POODLE vulnerability.

Solution or Workaround

To use HTTPS with the ArcGIS Web Adaptor on WebSphere, update the WebSphere configuration to use SSL_TLSv2 as the SSL handshake protocol, which is SSLv3 and TLSv1, TLSv1.1, and TLSv1.2.

Follow the instructions below.

  1. Log in to the WebSphere Application Server Integrated Solutions Console.
  2. Click Security > SSL certificate and key management, and under Related Items, click SSL configurations.
  3. Click NodeDefaultSSLSettings (default).
  4. Under Additional Properties, click Quality of protection (QoP) settings. QoP settings define the strength of the SSL encryption, the integrity of the signer, and the authenticity of the certificate.
  5. Select a protocol for the SSL handshake SSL_TLSv2.
  6. Click OK and save directly to the main configuration.

In addition, the WebSphere Java Virtual Machine must be updated to start with SSL protocol TLSv1.1 and TLSv1.2. For this, follow the steps below.
 
In the WebSphere Administrative console:

  1. Click Servers > Server Types > WebSphere application servers, and click server1 to open it.
  2. Under Server Infrastructure, click Java and Process Management > Process definition.
  3. Under Additional Properties, click Java Virtual Machine.
  4. In the Generic JVM arguments text box, enter -Dhttps.protocols=TLSv1.1,TLSv1.2
  5. Click Apply, OK, and save directly to the main configuration.
  6. Restart the application server.

Note: 
The above solution is for Java Web Adaptor working with WebSphere 9. Review the WebSphere documentation for further details on security configuration for the organization level requirements

2016년 1월 5일 화요일

[Trobleshooting] Out Of Memory Error - org.apache.axis2.description.AxisService

After migrating WebSphere Application Server from V6.1 to V8.0, JAX-WS application started getting Java OutOfMemoryError


Problem(Abstract)

WebSphere Application Server was migrated from 6.1.0.43 to 8.0.0.3 and the application started getting Java OutOfMemoryError (OOM).

Symptom

The log shows:
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError".
The Heap analyzer tool shows a memory leak. The prime suspect is org.apache.axis2.description.AxisService 

Cause

The problem was triggered by changes in default behaviors between WebSphere Application Server 6.1.x and 8.0.x.

Changes that could affect WebSphere Application Server V8 and memory usage when using JAX-WS dynamic ports, which are those added using the service method addPort, might have additional memory requirements starting in WebSphere® Application Server Version 8.0. In previous releases, a single instance of a dynamic port could be shared across multiple service instances. In version 8.x, dynamic ports are now scoped to the instance of the service that added them. If a JAX-WS client has multiple service instances which refer to a dynamic port of the same name, those instances are no longer shared. This can potentially increase the memory requirements for that client. The memory used by dynamic ports is released when the service instance goes out of scope. However, if you encounter issues related to increased memory usage, it is possible to revert the behavior so that dynamic ports are again shared across service instances. 

Note, version 8 includes a new default Garbage Collection policy, generational (genconn). In version 6.1, the default is optthruput.

Environment

JAX-WS application running on WebSphere Application Server migrated from V6.1.0.43 to V8.0.0.3

Diagnosing the problem

Logs, javacores and heapdumps were collected for the OutOfMemory condition. Analysis of these files showed the leak suspect to be java/util/HashMap$Entry, which are allocated in class org/apache/axis2/jaxws/description/impl/ServiceDescriptionImpl.
Using PMAT tool on the GC trace in the logs confirmed the growth.

Resolving the problem

Consider making the following changes:
  • Increased the Java heap size
  • Set the GC policy to the v6.1 default setting of gcPolicy:
    -Xgcpolicy:optthruput
  • To revert the behavior so that dynamic ports are again shared across service instances, set the JVM custom property jaxws.share.dynamic.ports.enable = true. However, note that doing so can cause some other issues, such as having policy set attachments incorrectly applied across shared dynamic ports. If you set this flag to true and encounter some of these issues, then you should remove the flag setting.
  • For WebSphere for z/OS if more than one Servant region is used consider adding the following WebSphere custom property:
    com.ibm.ws390.routing.EnableCustomSessionCookieName=true

Related information

2015년 2월 17일 화요일

[TechNote] WebSphere Customization Toolbox (WCT) will not start on Linux Operating System


Problem(Abstract)

Unable to start the WebSphere Customization Toolbox (WCT) GUI from commandline
/opt/IBM/WebSphere/Toolbox/WCT/wct.sh

Symptom

In some cases when the problem occurs there will be no noticeable error displayed on the commandline, while in other cases the following popup message may occur



The following error is seen in the log ..ibm/WebSphere/workspaces/WCT85/.metadata/.log

java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons: 
/root/.ibm/WebSphere/configurations/WCT85/org.eclipse.osgi/bundles/45/1/.cp/libswt-pi-gtk-3659.so (libXtst.so.6: cannot open shared object file: No such file or directory)

Resolving the problem

This problem can be resolved by installing the linux packages gtk2-engines.i686 and libXtst.i686
e.g.
# yum install gtk2-engines.i686
# yum install libXtst.i686

2014년 10월 24일 금요일

[TechNote] While running ISD services in AIX, the Node Agent often crashes and generates javacore files showing "Failed to create a thread"

http://www-01.ibm.com/support/docview.wss?uid=swg21454120

While running ISD services in AIX, the Node Agent often crashes and generates javacore files showing "Failed to create a thread"

Technote (troubleshooting)


Problem(Abstract)

We are using Information Services Director (ISD) in AIX and when running several ISD services the Node Agent is crashing often. It seems to happen often when multiple services run concurrently. The javacore files show:

Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError":"Failed to crate a thread: retVal -1073741830, errno 11" received

After restarting Information Server the problem doesn't happen again.

Cause

This error is suggesting that the java process was unable to create a new Thread. This usually indicates some sort of kernel level limit being reached.

Resolving the problem

To verify that there are no limitation to create processes, please verify the following:
  1. That the Node Agent is running as root or that it has been properly configured to run as non-root. See tech note 1415257.
  2. That the ulimit setting (if any) for processes that a user can create. In some releases of AIX, you can set the maximum number of processes per user. Run

    ulimit -a

    to check this value. If set to a low value try to increase or set it to unlimited using

    ulimit -u nnnn

    where nnnn is the new value.
  3. That the maxuproc is set to a high value. This is a kernel parameter defines the maximum number of processes that a user can have at a given time. This is a system level parameter and applies to all users. To check the current value of maxuproc run:

    lsattr -E -l sys0|grep maxuproc

    If you current value is lower than 4096, try to set it to this value and monitor after the change (if you are running a high number of processes you might need to use a higher value). To change maxuproc use the following command

    chdev -l sys0 -a maxuproc='nnnn'

    where nnnn is the new value for this parameter.

2014년 4월 15일 화요일

[TechNote] OutOfMemoryError (OOM) and Leak Detected in IBM PoolManager When Using Oracle JDBC Driver

OutOfMemoryError (OOM) and Leak Detected in IBM PoolManager When Using Oracle JDBC Driver


Technote (troubleshooting)


Problem(Abstract)

OutOfMemoryError or unexpectedly high heap utilization when using Oracle JDBC Driver and stored procedures. Leak detection logic may point to com.ibm.ejs.j2c.PoolManager however the leak is due to how stored procedures are processed by the JDBC Driver.

Symptom

OutOfMemoryError or high heap consumption while using stored procedures with Oracle JDBC. Current heap analyzers will point to the com.ibm.ejs.j2c.PoolManager as the source of the leak.

Cause

The PoolManager pools database connections and provide a performance improvement to hosted applications. Customers who experience an OutOfMemoryError where the leak suspect is the PoolManager may be mislead into thinking the database connection pool is exceeding the configured size. The underlying problem is that the Oracle JDBC Driver accumulates oracle.jdbc.driver.T2CStatement objects until the database connection is closed. The T2CStatement object represents the cursor of a stored procedure’s result set and is not closed when the result set is closed. Oracle is working on a design change under bug # 9092263.

Diagnosing the problem

Identify a large accumulation of T2CStatement objects by loading the heapdump (or system core) into the Memory Analyzer Tool and running the following OQL:
select * from instanceof oracle.jdbc.driver.T2CStatement

This will show all of the T2CStatement objects however the T2CStatement’s are often linked together similar to a link list. The following OQL string will select the first T2CStatement object in each link. select * from instanceof oracle.jdbc.driver.T2CStatement s where (classof(dominatorof(s)).getName()!="oracle.jdbc.driver.T2CStatement")

The Memory Analyzer Tool can be downloaded as either an ISA Plug-in or as a standalone tool from
eclipse. The standalone version of MAT requires a DTFJ plug-in to open heap dumps from the IBM JDK.
IBM Support Assistant download page:
http://www-01.ibm.com/software/support/isa/
Memory Analyzer Tool homepage:
http://www.eclipse.org/mat/
Instructions on adding DTFJ plug-in to standalone Memory Analyzer Tool:http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html

Resolving the problem

Set a connection-aged timeout on the database pool until the design change is made available from Oracle. The recommended value is ¼ the time it takes for the server to reach OOM after restart. For example, if it takes 2 hours for the server to reach OOM then set the aged timeout to 10800 (30 minutes).

Aged Timeout in the WebSphere Application Server 6.1 Information Center:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1//index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpoolset.html

2014년 2월 24일 월요일

[TechNote] Troubleshooting: Workload Management Problems

Troubleshooting: Workload Management Problems

Technote (troubleshooting)


Problem(Abstract)

Troubleshooting for problems with the Workload Management component of IBM WebSphere Application Server. This should help address common issues with this component before calling IBM support and save you time.

Resolving the problem

Information about the Workload Management Service
The Workload Management (WLM) component in IBM WebSphere Application Server provides routing services for incoming application requests so that these can be distributed to application server resources like Enterprise Java Beans (EJBs), Servlets and other server side application resources capable of processing such requests. WLM also provides failover capabilities when the servers hosting the applications are members of a cluster.
More details about Workload Management are available from the WebSphere Application Server V8.0 Information Center under the following topics:

What symptoms are you experiencing?
  1. Application requests fail to get serviced and runtime exceptions are seen in the SystemOut logs

    • org.omg.CORBA.TRANSIENT: SIGNAL_RETRY: This is a transient exception and suggests a routing attempt by the workload management routing service connecting to a target server.

      This is the exception thrown to the client in the case where a request is sent out to a target server and no reply ever comes back. The exception informs the application that another target might be available to satisfy the request, but the request could not be failed over transparently by WLM because the completion status was not determined to be “no”. In this case the client application needs to determine if they wish to resend the request.

    • org.omg.CORBA.NO_IMPLEMENT: This exception is thrown when none the servers participating in the workload management of the Enterprise JavaBeans (EJB) are available and the routing service cannot locate a suitable target for the request.

      The exception is created, for example, if the cluster is stopped or if the application does not have a path to any of the cluster members. There are many kinds of NO_IMPLEMENT which can be distinguished by the associated message or minor code with the NO_IMPLEMENT exception.

    • NO_IMPLEMENT: Retry Limit Reached & NO_IMPLEMENT: Forward Limit Reached: Each of these exceptions is thrown in the case where WLM is attempting to route a request to a server and receives an exception which is considered retryable, or the request is being forwarded.

      In order to avoid an infinite selection loop, these exceptions will be thrown if errors are received or forwarding is done on the same server ten consecutive times.

    • NO_IMPLEMENT: No Cluster Data: This exception, often seen in the Node Agent, is thrown when WLM is asked to make a selection over a cluster, but no data has been found or gathered for that particular cluster.

      This error is often seen for a short time when first requests for a cluster are made after startup of a cell, if this is seen they can usually be resolved by the setting of custom WLM properties to true. For information on the custom properties Exceptions of this type which remain persistent should be sent to IBM support.

    • NO_IMPLEMENT: No Available Target: This is a more general exception meaning that we may have some cluster data (perhaps not all), but with whatever data we currently have available to us, WLM cannot find a valid target for the request from that data.

      It is possible that members have been marked unusable or just that we do not have the current data necessary to route the request to the intended resource.

    • NoAvailableTargetException: This exception is internal to IBM only, you may see it printed out in traces with the WLM trace spec enabled, but this exception is caught and handled internal to the WLM code.


      This exception is often expected, especially in failover and startup scenarios and if a real problem exists, it would manifest itself as one of the NO_IMPLEMENT exceptions above.

  2. Enterprise JavaBeans requests are not distributed to all servers

    1. Make sure the target servers are started. Use the Administrative console to try starting them, or if a target server that is failing to service requests is already started, try restarting it.

    2. Try accessing the enterprise bean directly on the problem server. Perhaps the issue is not related to workload management. If this fails, review topic Enterprise bean cannot be accessed from a servlet, a JSP file, a stand-alone program, or other client in the WebSphere Application Server V8.0 Information Center.

    3. Check your configuration. Review the Troubleshooting the Workload Management component topics in the WebSphere Application Server V8.0 Information Center.

    4. Make sure the server is "in view" in reference to HAManager and the core group the server belongs to. If the server is not in view then it may be islanded from the rest of the cell and not seen as being available by the other servers, and subsequently the client.

    5. Make sure static routing is not enabled, this can cause the same islanding problem as above.

    6. Make sure HAManager is enabled, disabling it also causes WLM to function improperly.

  3. Enterprise JavaBeans requests are not distributed evenly

    • Possible reasons for this behavior are:
      • Improper configuration
      • Environment issues such as the availability of servers or applications.
      • A large numbers of requests that involve transactional affinity, or
      • A small number of clients

    • Things to consider:
      • WLM sprays a variety of different requests (referred to as WLMable requests).

        If you are only tracking the spraying of a particular application request and it is unbalanced, that does not mean that WLM is spraying improperly. The classic example is a 2 cluster member cluster with the same server weights and an client application which loops on 2 WLMable requests, operationA and operationB. If a tracking system only looks at how operationA requests are being distributed, they will all be sent to one server. This is not a bug or problem, as all operationB requests are sent to the other server as well. This “pattern problem” is often seen in small test environments with only a few servers and is rarely seen in production systems with more cluster members.
      • The Workload Management service uses a weighted proportional scheme to distribute Enterprise JavaBeans requests.

        The WLM selection logic had certain feedback mechanisms that can change our routing behavior on the fly. We react to various scenarios and even server load when making routing decisions, so it is entirely possible that WLM can function perfectly and the requests will not be balanced exactly to the configured server weights. For example, if a customer has a cluster with 2 machines, one is a powerful 8-way with lots of RAM, and the other is a single processor desktop machine, even if the server weights are set to 2 for both servers, you could potentially see 80% or more of the request go to the 8-way machine, simply because the desktop machine cannot keep up as fast as the 8-way.

  4. A failing server still receives Enterprise JavaBeans requests (failover fails)


    Some possible causes are:

    • The client might have been in a running transaction with an Enterprise JavaBeans on the server that went down.

      This might be working as designed by letting this particular exception flow back to the client. Since the transaction might have completed, failing over this request to another server could result in this request being serviced twice. This function is referred to as “Quiesce mode”. Quiesce mode is entered when a server is asked to shut down. While in Quiesce mode, the server will reject all incoming requests which it determines are new work, but still allow in-flight requests to complete. This is primarily designed to allow transaction work to finish as above to prevent unnecessary TRANSACTION ROLLBACK exceptions, but requests other than transaction can be allowed into a server. By default, Quiesce mode will last for a maximum of 3 minutes (this is configurable), although it is possible for a server to exit quiesce earlier if all registered components agree that it is okay to do so based on their own criteria. If a request gets rejected by a server in quiesce mode, WLM will be called with a org.omg.CORBA.COMM_FAILURE with a completion status of “no”, and this request will be automatically retried by WLM.

    • If the requests sent to the servers come back to the client with any other exceptions consistently, it might be that no servers are available.

      In this case, follow the resolution steps as outlined in Troubleshooting the Workload Management component in the WebSphere Application Server V8.0 Information Center.

  5. Stopped or hung servers do not share the workload after being restored


    This error occurs when the servers that were unavailable are not recognized by the Workload Management component after they are restored.

    There is an unusable interval determined by the property com.ibm.websphere.wlm.unusable.interval during which the workload manager waits to send to a server that has been marked unusable. By default this is 5 minutes. You can confirm that this is the problem by ensuring that the servers is up and running and then waiting for the unusable interval to elapse before checking to determine whether failover occurs. If the server still does not participate in workload, reference bullet #2 above for additional reasons why this could occur.


  6. What to do Next?

    If the above scenarios and steps taken did not help resolve your problem, you may access the information available from the WebSphere Application Server Support site and browse for current and known problems and their resolution.

    If no useful hints are found in the support site, then see the MustGather for WLM problems and collect the information requested before opening a PMR.

2014년 1월 23일 목요일

[TechNote] WebSphere Application Server: global security setting changes do not reflect on Intelligent Management web server connectors



Technote (troubleshooting)


Problem(Abstract)

If you enable Intelligent Management on your web server, and change the global security setting, the HTTP or HTTPS protocol does not change on your Intelligent Management web server connectors.

Resolving the problem

In order for the the security change to reflect on the Intelligent Management web server connectors , run the refreshCellForIntelligentManagement command on local and remote cells.
To propagate the security change to remote cells, use the refresh button on the remote cell. In the administrative console for the remote cell, go to Servers > Server Types > Web servers >web_server_name > Intelligent Management, and click the Refresh button.

Related information

2014년 1월 2일 목요일

[TechNote] Getting Heapdumps on the Solaris platform



Technote (troubleshooting)


Problem(Abstract)

By default, Sun's Java™ Virtual Machine (JVM) does not produce heapdumps for memory diagnostics. The required steps to enable this functionality (if supported) is dependant on what level of the JVM is installed.

Resolving the problem


1.6.0
To produce a heapdump in the event of an OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError

Heapdumps can be taken on demand from the new improved jmap tool which uses the built in heapdumper code (unlike the jmap tools in 1.4.2 and 5.0). The histogram option in Java™ 6 also uses a much improved built in mechanism. To manually create a heapdump, use the following command:


jmap -dump:live,format=b,file= <heapdump_filename> [JVM PID]


To print class histograms, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written to native_stdout.log:
-XX:+PrintClassHistogram





SDK 1.5.0_16 or later
To force a heapdump by sending the Java Virtual Machine (JVM) a SIGQUIT - kill -3 [JVM PID]:
-XX:+HeapDumpOnCtrlBreak

To produce a heapdump in the event of an OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written to native_stdout.log:
-XX:+PrintClassHistogram




SDK 1.5.0_07 to 1.5.0_15
Note: Manual generation of heapdumps is still not implemented. -XX:+HeapDumpOnCtrlBreak only works on 1.5.0_16 or later.

To produce a heapdump in the event of an OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written to native_stdout.log:
-XX:+PrintClassHistogram





SDK 1.4.2_12 or later
In 1.4.2_12, Sun implemented a built-in heapdump facility, with a new command line option.

To force a heapdump by sending the Java Virtual Machine (JVM) a SIGQUIT - kill -3 [JVM PID]:
-XX:+HeapDumpOnCtrlBreak

To produce a heapdump in the event of an OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written to native_stdout.log:
-XX:+PrintClassHistogram






SDK 1.5.0 GA to 1.5.0_06

SDK 1.4.2_09 to 1.4.2_11
Note: Due to limitations using jmap, the most reliable solution for getting a heapdump will be to upgrade the JDK. Click here for more information.

The jmap executable is contained in the same directory as the Java executable (for example: [path to JDK]/bin), and can be run in a number of ways:

The following produces a heapdump in the current working directory, named heap.bin:


jmap -heap:format=b [JVM PID]

The following produces a heapdump in the same way as the previous command, but from a core file rather than a running process:


jmap -heap:format=b [Java executable] [core file]


The following produces a summary of the state of the heap, including details of the heap configuration (although this frequently seems to be incorrect):


jmap -heap [JVM PID] > heapsummaryfilename


The following walks the heap and produces a list of all object types, sorted by total size:


jmap -histo [JVM PID] > histofilename

The following produces classloader statistics for the permanent generation. Among other things, the results detail the size and number of classes loaded by each classloader:


jmap -permstat [JVM PID] > histofilename





kill -3 [JVM PID] (for example: The same as the trigger for a threaddump):

A thread dump is written to native_stdout.log, after which you should see text similar to the following:
Dumping heap to java_pid21745.hprof.1146911952593 ...
Heap dump file created [288590804 bytes in 13.043 secs]

The heapdump file is written to the current working directory.

Notes:
The latest SDK and WSAS updates can be found here:
Recommended Fixes for WebSphere Application Server


1. The Heapdumps generated using any of the preceding methods can be processed using the latest version of HeapAnalyzer.

2. The heapdumps produced are written in the HPROF binary format and can also be examined using Sun'sHAT tool (or preferably jhat, included with JDK 6.0).
3. Finally, the built-in heapdumper works for all types of heap apart from the parallel GC - throughput collector (-XX:+UseParallelGC)
Note: Support for the throughput collector is being targeted by Sun for 1.4.2_15, under Sunbug 6452074


Additional notes on jmap:
Because jmap attaches to the JVM, the JVM has no control over when the heapdump occurs. Therefore, the heapdumps produced by jmap can be incomplete or inconsistent. They are also likely to contain "floating garbage" (for example: "dead" objects that will be collected in the next).

Jmap also has severe performance and stability issues. It can take an extremely long time (for example: an hour) to write a heapdump, especially if the object tree is deep and complex (as is often the case for our customers). The dump might fail completely if the heap is larger than approximately 500MB, due to flaws in the design of jmap (is uses a lot of native memory, which can lead to native OutOfMemoryErrors when the process size grows to approximately 4GB).

Finally, jmap only works for standard heaps. For example, it cannot attach to heaps employing CMS.

For more details:




Change history
Last Updated:18 June 2010
  • 18 June 2010: Added WebSphere Application Server 7.0 (Java 6) information.

2013년 4월 16일 화요일

[TechNote] java.io.IOException: Async IO operation failed (3), reason: RC: 55 The specified network resource or device is no longer available.

Recovering from service transaction timeout messages

Use the information in this topic to recover from service transaction timeout messages.
If you receive service transaction timeout message, you may time out while waiting for a thread in the pool to become available, or see a message similar to the following:
 
java.io.IOException: Async IO operation failed (3), reason: RC: 55 
The specified network resource or device is no longer available.
 
 To overcome this problem, increase the maximum size of the web container thread pool and increase the size of the HTTP outbound connector connection pool for all JVM web services.
  1. Navigate to the WebSphere® Application Server administrative console.
  2. Select the Servers > Server types > Websphere application servers.
  3. Click the client server links. For example: IICCltSvcs01.
  4. In the Server Infrastructure section, select Java and Process Management > Process definition.
  5. In the Additional Properties section, select Java virtual machine > Custom Properties
  6. Set the value for com.ibm.websphere.webservices.http.maxConnection to a higher value (for example, 500).
  7. Restart the server.

2013년 1월 28일 월요일

[TechNote] Configuring the WebSphere Application Server hung thread detector to automatically produce a javacore or thread dump

Configuring the WebSphere Application Server hung thread detector to automatically produce a javacore or thread dump


Problem(Abstract)

By default, WebSphere Application Server reports potentially hung threads using the hung thread detection policy. When a thread has been running for a certain interval (default 10 minutes), the thread might be reported as hung and a WSVR0605W message is printed in the SystemOut.log file. By setting a custom property in WebSphere Application Server, the hung thread detection policy will generate a javacore (or thread dump on Solaris and HP-UX) when a potentially hung thread is reported.

Symptom

A javacore, or thread dump, will be created each time a potentially hung thread is detected and a WSVR0605W message is written out to the SystemOut.log file:
WSVR0605W: Thread <threadname> has been active for <time> and may be hung. There are <totalthreads> in total in the server that may be hung.


The javacore can then be used to determine what code is running in the potentially hung thread and if there are other bottlenecks existing in the JVM, causing the reported hung thread. This can be helpful in troubleshooting server hangs and performance issues.

Note: It might be normal for certain threads to be running for long periods of time. These might be falsely reported as hung threads.

Environment

The com.ibm.websphere.threadmonitor.dump.java property was enabled in WebSphere Application Server V6.0.2.29, 6.1.0.19, and 7.0 and later.

Resolving the problem

Set the com.ibm.websphere.threadmonitor.dump.java property to true:

Application Servers:
  1. From the administrative console, click Servers > Application Servers > server_name.
  2. Under Server Infrastructure, click Administration > Custom Properties.
  3. Click New and add the following property:

    Name: com.ibm.websphere.threadmonitor.dump.java
    Value: true

  4. Click Apply.
  5. Click OK and save the configuration changes.
  6. Restart the Application Server for the changes to take effect.
Node Agent:
  1. From the administrative console, click System Administration > Node Agents > nodeagent.
  2. Under Additional Properties, click Administration Services
  3. Under Additional Properties, click Custom Properties
  4. Click New and add the following property:

    Name: com.ibm.websphere.threadmonitor.dump.java
    Value: true

  5. Click Apply.
  6. Click OK and save the configuration changes.
  7. Restart the Node Agent for the changes to take effect.

2013년 1월 25일 금요일

[TechNote] Configuring the WebSphere Application Server hung thread detector to automatically produce a javacore or thread dump

Configuring the WebSphere Application Server hung thread detector to automatically produce a javacore or thread dump

Problem(Abstract)

By default, WebSphere Application Server reports potentially hung threads using the hung thread detection policy. When a thread has been running for a certain interval (default 10 minutes), the thread might be reported as hung and a WSVR0605W message is printed in the SystemOut.log file. By setting a custom property in WebSphere Application Server, the hung thread detection policy will generate a javacore (or thread dump on Solaris and HP-UX) when a potentially hung thread is reported.

Symptom

A javacore, or thread dump, will be created each time a potentially hung thread is detected and a WSVR0605W message is written out to the SystemOut.log file:
WSVR0605W: Thread <threadname> has been active for <time> and may be hung. There are <totalthreads> in total in the server that may be hung.


The javacore can then be used to determine what code is running in the potentially hung thread and if there are other bottlenecks existing in the JVM, causing the reported hung thread. This can be helpful in troubleshooting server hangs and performance issues.

Note: It might be normal for certain threads to be running for long periods of time. These might be falsely reported as hung threads.

Environment

The com.ibm.websphere.threadmonitor.dump.java property was enabled in WebSphere Application Server V6.0.2.29, 6.1.0.19, and 7.0 and later.

Resolving the problem

Set the com.ibm.websphere.threadmonitor.dump.java property to true:

Application Servers:
  1. From the administrative console, click Servers > Application Servers > server_name.
  2. Under Server Infrastructure, click Administration > Custom Properties.
  3. Click New and add the following property:

    Name: com.ibm.websphere.threadmonitor.dump.java
    Value: true

  4. Click Apply.
  5. Click OK and save the configuration changes.
  6. Restart the Application Server for the changes to take effect.
Node Agent:
  1. From the administrative console, click System Administration > Node Agents > nodeagent.
  2. Under Additional Properties, click Administration Services
  3. Under Additional Properties, click Custom Properties
  4. Click New and add the following property:

    Name: com.ibm.websphere.threadmonitor.dump.java
    Value: true

  5. Click Apply.
  6. Click OK and save the configuration changes.
  7. Restart the Node Agent for the changes to take effect.

2012년 9월 13일 목요일

[TechNode] 세션 클러스터링 중 계속 JSESSIONID가 바뀌는 현상


세션 클러스터링 중 계속 JSESSIONID가 바뀌는 현상

- 작성: IBM 이정운 과장

이부분은 두가지 측면에서 접근을 하셔야 합니다.
먼저 서버간 JSESSIONID가 충돌하여 발생할수도 있고,
애플리케이션간 JSESSIONID가 충돌하여 발생할수도 있습니다.

물론 다른 여러가지 이유에서 발생할수도 있지만. WAS엔지니어가 제일먼저 시도해 보아야 할부분은 엔진 셋팅을
바꿔가면서 테스트를 해보고 범위를 줄여나가는게 아닌가 생각됩니다.

첫번째로 서버간 JSESSIONID가 충돌하여 발생하는 경우
대부분 ND환경에서는 잘 일어나지 않고 한 머신에 BASE를 두대 설치했을경우 일어나는 경우가 많습니다.

예를들어
한 BOX에 두개의 BASE 엔진을 설치후 각각 서로 다른 응용어플리케이션을 설치 했을경우
첫번째 BASE엔진에 설치되어있는 어플리케이션의 JSP1화면(로그인상태)에서 두번째 BASE엔진에 설치되어있는 어플리케이션의 JSP2화면을 호출(팝업창으로)하게 되면 JSP1화면에 세션이 사라지면서 로그아웃 되는 현상이 발생하는 경우입니다.

처음 WAS에서 세션를 생성하면 JSESSION ID가 생깁니다.(JSESSION:session id , domain or ip 정보등)
세션이 생성후에는 jsessionid를 통해 세션 object를 찾아서 세션을 유지하게 되는데,
한 BOX에서 두개의 WAS가 실행되면 jsessionid에 충돌이 발생하게 됩니다.

두개의 어플리케이션이 서로 다른 이름에 세션을 사용하더라도 마찬가지로 JSESSIONID가 중복되므로

Application Server > server1 > 세션 관리 화면에서 가운데쯤 “쿠키사용가능”이 링크가 걸려있는데 이부분을
클릭하고 들어가면 쿠키이름이 “JSESSIONID”라고 되어있다.

이부분에 이름을 두 WAS엔진이 틀리게 가져가게끔 바꿔줘야 합니다.
예)WAS1 엔진 = JSESSIONID
   WAS2 엔진 = JSESSIONID2

두번째 애플리케이션간 JSESSIONID가 충돌하여 발생하는 경우
- 어플리케이션 레벨의 세션관리로 이동하여 “세션 관리 대체” 를 체크해 줍니다.
이부분을 체크하면 서버단의 JSESSIONID 를 읽지 않고 어플리케이션 단의 JSESSIONID를 읽게 됩니다.
이후 각각의 어플리케이션 마다 JSESSIONID를 틀리게 주셔야 합니다.

참고로 한서버에 여러 어플리케이션을 사용한다면, 엔터프라이즈 응용프로그램 > 응용프로그램명 > 세션 관리 > 쿠키
에서 쿠키경로를 다른 이름으로 각각 셋팅해 주면 됩니다.