[TechNote] General guidelines for installation of WebSphere Application Server on Networked Filesystems (NFS), Mapped Network Drives, or Storage Area Network (SAN) resources

General guidelines for installation of WebSphere Application Server on Networked Filesystems (NFS), Mapped Network Drives, or Storage Area Network (SAN) resources

Question

What do WebSphere Application Server V6.1, V7.0, V8.0, V8.5, and V8.5.5 require in terms of filesystem support on networks? How do networked filesystems affect the way IBM Support provides support for WebSphere Application Server products?

Cause

WebSphere Application Server is tested on operating systems which use local filesystems, meaning that the filesystems are located on disks which are directly attached to the physical hardware hosting the system.

There are other methods of attaching filesystems to a host system, such as using Networked Filesystems (NFS) on UNIX or Linux-based systems, Mapped Network Drives on Windows-based systems, or Storage Area Network (SAN) resources. This technote describes some general guidelines associated with installing the product in environments which use these filesystem solutions.

In general, WebSphere Application Server tolerates the use of these methods to attach filesystems to a host system. However, the filesystems must be attached to the host system in a manner which makes it appear indistinguishable from a local filesystem. The system administrators must be completely responsible for the configuration and maintenance of these filesystem solutions.

Please note that this article primarily applies to traditional WebSphere Application Server, and does not necessarily apply to Liberty Profiles.

Answer

Network filesystem support

For the sake of brevity, the term Networked Filesystem (NFS) is used for representing a UNIX or Linux Networked Filesystem, a Windows Mapped Network Drive (or other Windows shared resource), a Storage Area Network (SAN), or any other configuration where files are hosted on a resource which is not considered "local" to the system it is running on.

WebSphere Application Server can be installed to or from a networked filesystem as long as the environment meets certain requirements. As stated below, these are fairly strict requirements, and it is entirely the responsibility of the system administrator to ensure that the filesystems behave properly.

If IBM Support suspects that a networked filesystem's behavior is causing an issue, IBM Support may ask for the problem to be reproduced using a local filesystem. If the problem cannot be reproduced on a local filesystem, IBM Support may ask the system administrator to address the root cause of the problem from the perspective of the network and filesystem.


Filesystem requirements for UNIX and Linux-based systems

• File locking and lock daemon processes:
  The WebSphere Application Server installer requires the capability to lock files in the user's home directory, as well as files in the temporary directory and files in the targeted installation directory. If the network filesystem does not properly handle file locking, the product installer might hang on startup. For this reason, the NFS client must have file-locking capabilities. For some types of NFS, this means that you may need to run the lock daemon processes, such as lockd or nfslock, and statd. For more information, refer to technote 1316694.
  
• Mount point permissions:
  When a remote filesystem is mounted, it is typically attached to an empty directory which serves as the mount point. The user installing the software does not need read and write access to the mount point, they need read and execute permissions on the mount point. This lets them correctly enter and list the attributes of the mounted filesystem.The permissions of that empty directory mount point must allow the product's users to have appropriate read/execute access. The permissions on that empty directory mount point can affect access to files in the mounted filesystem, even though the permissions on the mount point become "hidden" when the filesystem is mounted. If the permissions are incorrect, this can cause issues with the product installer, such as the "Illegal character at index 18" error. For more information, refer to technote 1316005.

Filesystem requirements for Windows-based systems

• Do not use UNC paths with the product installer:
  As stated in the Information Center article for installing WebSphere Application server (such as the article for V7.0 Network Deployment), the WebSphere Application Server installer has issues installing the product from UNC paths. When installing WebSphere Application Server from a networked filesystem that is accessed from a UNC path (for example, it is accessed using the convention \\ServerName\ShareName\Directory), some product files might not be installed. It is better to install WebSphere Application Server from a Mapped Network Drive if it is being installed from a network resource.
  
• Use UNC paths after the product is installed:
  Once the product is installed, it is best to avoid any further use of Mapped Network Drives. Instead, use UNC paths to reference networked locations after the product is installed. This recommendation stems from the fact that different Windows users do not have access to other users Mapped Network Drives, which can cause issues if the product is configured to run as a Windows Service. For more information, refer to technote 1316456.

Networked Filesystem (NFS) server requirements

• NFS availability must be consistent and uninterrupted:
  The NFS server must be capable of providing uninterrupted connectivity to the filesystem which contains WebSphere Application Server. If the NFS connection is interrupted or the operating system reports a timeout while attempting to access the files on the NFS, then the product installation process will be interrupted. If an application server is running on an NFS and the connection is interrupted, then application server is at risk of hanging or crashing. From the perspective of the application server process, there is no way to mitigate the impact of interruptions to accessing the NFS. The NFS must be consistently available.
  
• NFS server must be configured properly:
  The system administrator must configure the NFS server, and understand how to configure elements such as /etc/exports, /etc/hosts.allow, /etc/hosts.deny, Windows file permissions, Windows domains, UNIX file permissions, and so on. Refer to the appropriate NFS documentation for handling this. IBM WebSphere Application Server support cannot assist with questions regarding NFS configuration or best practices.
  
• NFS server permissions:
  The application server installer must have complete read-write access to the directory that WebSphere Application Server will be installed to.

Networked Filesystem client requirements

• Concurrent file access:The product files must only be used by running server processes on one system at any given time. WebSphere Application Server does not support multiple systems running application servers or product tools from a single install located on a shared networked filesystem. In short, installing WebSphere Application Server once on an NFS and using multiple systems to run from that one install is not supported.
  
  Note that it is possible to install WebSphere Application Server on a shared networked filesystem, then use high-availability software to allow one system in a high-availability cluster to run application servers. If a system in the cluster fails, then the high-availability software can shut down and disconnect one system from the shared networked filesystem, then connect a different system to the shared networked filesystem. The other system can then start and run application servers. This is acceptable because only one system is using the product files at any given time. For more information about support for high-availability solutions, see technote 1419214.
  
  As noted earlier, this article is focused on support of traditional WebSphere Application Server, and not Liberty Profiles. In particular, the binaries in Liberty Profiles can be configured in a manner which allows shared concurrent file access.
  
• Client software:
  Be sure to run NFS client software which is up-to-date. Check with the operating system and filesystem vendor to ensure that the NFS software is appropriately updated.
  
• Host access and firewalls:
  The NFS client must run appropriate processes to establish the connection with the host, such as the portmap process. The NFS client must have appropriate consistent access through a firewall.
  
• NFS client permissions:
  The NFS client must have permission to access files on the host system. If the client and host are different operating system types (such as an AIX client accessing files shared on a Windows host, or a Windows client accessing files stored on a Solaris host), then the system administrator must ensure that file permissions and symbolic links are maintained consistently when the files are accessed using that client.
  
• NFS client performance:
  WebSphere administrators should be aware that running WebSphere Application Server from a networked filesystem can adversely affect is performance. The networked filesystem should allow access to the product files as quickly and consistently as possible, performing similarly to local filesystems.

[TechNote] How to clear the WebSphere class caches

How to clear the WebSphere class caches

Problem(Abstract)

Instructions on clearing the java class caches in IBM WebSphere Application Server. Both the JVM's shared class cache and WebSphere Application Server's OSGi class cache.

Resolving the problem

IBM Support may ask you to clear the class cache. There are several reasons for this. After an upgrade, it is possible that the class cache's are still holding onto previous versions of classes. It is also possible that the caches became corrupted.

Please remember there are two caches that will need to be cleared, the JVM's cache and the OSGi cache. The server has to be stopped before clearing the cache.

To clear the OSGi class cache:

For UNIX platforms, run the following script in each profile:
<WebSphere_HOME>/profiles/profile_name/bin/osgiCfgInit.sh

For Windows platforms, run the following script in each profile:
<WebSphere_HOME>\profiles\profile_name\bin\osgiCfgInit.bat

To clear the JVM's class cache (Windows, Linux and AIX only) :

For UNIX platforms, run the following script:
<WebSphere_HOME>/bin/clearClassCache.sh

For Windows platforms, run the following script:
<WebSphere_HOME>\bin\clearClassCache.bat

And clear the following directory contents:
<WinUsers_home>\Local Settings\ApplicationData\javasharedresources\

where WinUsers_home is either C:\Documents and Settings\DefaultUser OR C:\Users depending on your current version of Windows.

[TechNote] Issues with setting Read/Write Timeout of "-1"

Issues with setting Read/Write Timeout of "-1"

Problem(Abstract)

When "Read/Write timeout" of "Web server plug-in properties" is set to
-1 via "WebSphere Integrated Solutions Console", The property becomes invalid. In the sense, the console view for it is greyed out.

Symptom

On a WebSphere AppServer 8.5 Console, to disable ServerIOTimeout variable when trying to set the Read/Write Timeout of "0" , if inadvertently an input of "-1" is provided instead, The following is observed

1. Setting Read/Write Timeout "-1" via Console.



2. Once saved, The Console view gets greyed out with a value of "0"


3> Now, The console view for Read/Write Timeout does not revert to "0" for any other negative value. Only for "-1". However, the plugin-cfg.xml shows the correct value of "-1" for ServerIOTimeout. This in effect renders the JVM to get marked down after a ServerIOTimeout of "1" second. A highly unrealistic value for ServerIOTimeout.

NOTE: 1. Read/Write Timeout from WebSphere Console is reflected as ServerIOTimeout in the plugin-cfg.xml file.

2. This is observation is consistent in WebSphere 6.1/7.0/8.0/8.5 versions.

Cause

In the past (pre WebSphere plugin 6.1), a ServerIOTimeout of "-1" was documented as "waiting for the OS to time out of the connection". i.e effectively disabling the ServerIOTimeout property.

Since WebSphere 6.1, we added a feature. A negative ServerIOTimeout effectively means that the JVM will be marked down after that finite ServerIOTimeout value elapses. This also means that a ServerIOTimeout of "-1" would render the JVM marked down in if the request is not served in "1" second.

Resolving the problem

If customer wishes to deactivate ServerIOTimeout, they will need to set it to zero. The use of a negative value has been incorporated to logically mark the JVM offline from the plugin request handling and load-balancing perspective.

Although, the observation noticed is a bug from the console perspective, from a plugin perspective it functions as intended.

APAR PM94593 corrects the behavior of -1 not look like the timer is disabled but actually look like the mark down after 1 second setting. This applies to WAS70 and higher.

[TechNote] Renewing personal WAS certificate fails if plugin-key.kdb is unavailable

Renewing personal WAS certificate fails if plugin-key.kdb is unavailable

Problem(Abstract)

When the personal certificate in your WAS NodeDefaultKeyStore or CellDefaultKeyStore expires, you need to renew it (either manually or automatically using the expiration monitor).
This might fail with an error message, that the CMSKeystore is not available.

Symptom

The expiring certificate is not renewed in the NodeDefaultKeyStore, although the ISC already shows the new personal certificate.

Steps to recreate:

1. Login to the AdminConsole (ISC)
2. Go to Security -> SSL Certificate and Key Management
3. Key Stores and certificates -> NodeDefaultKeyStore -> Personal Certificates
4. select the personal certificat that you want to renew (normally it's called "default")
5. Press the "Renew" Button
6. If the plugin-key.kdb cannot be accessed by the DMgr, you will an error similar to this:
   
   An error occurred renewing default: com.ibm.websphere.crypto.KeyException: KeyStore "C:/WebSphere/AppServer/profiles/Dmgr01/wstemp/1623776755/workspace/cells/winwas70dCell01/nodes/winwas70dNode01/servers/webserver1/plugin-key.kdb" does not exist.
   
   Along with a CWPKI0033E error in the DMgr's SystemOut.log, telling the same problem.
7. Despite of the error above, the list of certificates will already show you the new certificate with the new expiration date, but this is a false notification!

If you logout and login again, you will find that there is still the old personal certificate in the NodeDefaultKeyStore!

Cause

The WebServer Plugin needs the signer certificate from the Node's personal certificate to ensure a secure Plugin-WAS connection.

If the plugin-key.kdb ist not available at the defined location of the CMSKeyStore (or not accessible due to file permission problems), then the automatic signer exchange is not possible for the Deployment Manager - hence the renewal of the certificate cannot complete and is interrupted.

Resolving the problem

You either need to ensure, that all keystores and truststores which are defined in the cell are accessible, including the CMSKeyStore for the plugin.
If for some reason the plugin-key.kdb has been removed and is no longer required in this cell, the CMSKeyStore definition should also be deleted in the list of keystores in the ISC.

[TechNote] The self-signed certificate is automatically renewed, but is not automatically propagated to webserver directory.

The self-signed certificate is automatically renewed, but is not automatically propagated to webserver directory.

Problem(Abstract)

Customer can define IHS SSL virtualhost and create CMS keystore for that SSL virtualhost from WAS admin console. And WAS expiration management will automatically renew the self-signed certificate in that CMS keystore. However, the renewed self-signed certificate is not automatically propagated to the keystore in IHS webserver directory.

Resolving the problem

The typical IHS keystore and certificate management is done through Ikeyman, or GSKit command line certificate management, which is outside of WAS scope.

If key stores for IHS are managed in this way, two steps are required to have the changes take effect.

* Customer has to manually click "Copy to Web server key store directory" to make the renewed self-signed certificate take effect.



* Customer must restart IHS (changes to key stores are not picked up dynamically.

[TechNote] Insufficient ulimit -u (NPROC) Value Contributes to Native OutOfMemory

Insufficient ulimit -u (NPROC) Value Contributes to Native OutOfMemory

Technote (troubleshooting)

Problem(Abstract)

An out of memory may be observed on a system running WebSphere Application Server on Linux. Further investigation may reveal a "Failed to create a thread:" message within the generated javacore which would indicate a native out of memory issue has been encountered. The cause of the problem may be an insufficient ulimit setting. While this type of issue can occur on any level of Linux, more recent releases, such as Redhat Enterprise Linux 6, are more likely to encounter this issue due to a decreased default value for certain ulimits. The following will outline how to identify if a process ulimit is the culprit and what WebSphere Application Server Support recommends to fix the case.

Symptom

An out of memory Dump Event such as:

"systhrow" (00040000) Detail "java/lang/OutOfMemoryError"
"Failed to create a thread: retVal -1073741830, errno 11" received

Note: This message will appear only in javacores.

Diagnosing the problem

When using WebSphere Application Server, ulimits can be set to fix or tune around a number of problems. For more on how to set a ulimit refer to the "Guidelines for setting ulimits" Technote which goes into detail on setting different ulimits on various operating systems and the difference between the soft and hard limit. This article is concerned particularly with the "-u" ulimit or "nproc" limit which affects the number of processes allowed by a single user running WebSphere Application Server.

The nproc limit usually only counts processes on a server towards determining this number. Linux systems running WebSphere Application Server are a particular case. The nproc limit on Linux counts the number of threads within all processes that can exist for a given user. To determine the ulimit settings of a WebSphere Application Server process running on Linux refer to "How to determine the ulimit settings of a running WebSphere Application Server process on Linux".

User Limits (in bytes except for NOFILE and NPROC)

type	soft limit	hard limit
RLIMIT_AS	unlimited	unlimited
RLIMIT_CORE	unlimited	unlimited
RLIMIT_CPU	unlimited	unlimited
RLIMIT_DATA	unlimited	unlimited
RLIMIT_FSIZE	unlimited	unlimited
RLIMIT_LOCKS	unlimited	unlimited
RLIMIT_MEMLOCK	65536	65536
RLIMIT_NOFILE	65536	65536
RLIMIT_NPROC	131072	131072

For most cases of older versions of Linux this value will be defaulted to around 2048. For out of the box Red Hat Enterprise Linux (RHEL) 6 the default value for nproc will be set to 1024. This low default setting for larger systems will not allow for enough threads in all processes.

WebSphere Application Server Support recommends setting the ulimit -u or nproc to a value of 131072 when running on Linux to safely account for all the forked threads within processes that could be created.

By using this recommended value a sufficient number of threads in all processes will be allowed and will not be a limiting factor for the environment. Increasing the limit to the suggested value should have no negative impact. When the number of threads in all processes reaches the -u ulimit, an out of memory error message will be thrown. This issue can be avoided by increasing this limit. Be aware that if the number of threads/processes reaches the recommended number of 131072 or close, the issue may be deeper and continuing to increase the -u ulimit will only prove to be a temporary fix.

[Info] New and Noteworthy in the WebSphere Application Server Migration Toolkit V3.5.3

New and Noteworthy in the WebSphere Application Server Migration Toolkit V3.5.3

What’s New and Noteworthy?

WebSphere Application Server Migration Toolkit Version 3.5.3 includes updates to all five of the migration tools:

• Application Migration Tool – WebSphere Version to Version
• Application Migration Tool – Apache Tomcat to WebSphere
• Application Migration Tool – JBoss AS to WebSphere
• Application Migration Tool – OracleAS to WebSphere
• Application Migration Tool – WebLogic to WebSphere

WebSphere Version to Version migration

The version to version application migration tool includes new rules and updates to help migrate application JSP, Java, and XML files.

Competitive application server migration

The competitive application server migration tools include new rules and updates to help migrate application JSP, Java, XML, and property files.

The WebLogic tool adds rules for several WebLogic APIs where more information is available on how to migrate to WebSphere Application Server. The JBoss tool adds rules for several JBoss specific strings commonly used in applications.

All the tools include rules for JSP files including recognition of Java keywords in expression language elements.

Hibernate to WebSphere JPA

A new rule set is available with rules to help migrate applications using the session-based Hibernate programming model. This set of 25 rules is provided with the framework migration feature in the competitive migration tools, but they are not automatically selected with the any of the competitive application server migration rule sets. Analyze your code with this new rule set when you want to learn more about migrating from Hibernate to WebSphere JPA.

New support for:

Eclipse 4.3.1 for Java EE Developers (Kelper SR1)
The latest WebSphere Developer Tools v8.5.5 available from WASdev
Rational Application Developer v8.5.5
Rational Software Architect v8.5.5

For more details on the release content see the migration toolkit forum.

Get it now!

[Info] WebSphere Application Server V8.5.5 license terms have been updated to relax load balancing and failover restrictions

WebSphere Application Server V8.5.5 license terms have been updated to relax load balancing and failover restrictions

Abstract

WebSphere Application Server Version 8.5.5 includes a license update to modify the server load balancing and failover restrictions currently in the program licenses for some WebSphere Application Server editions.

Content

The server load balancing and failover restrictions currently in the program licenses for some WebSphere Application Server Version 8.5.5 programs have been updated. The terms have been changed to allow an unlimited number of server profiles for load balancing and failover, rather than the previously specified limitations.

Versions affected ("affected products")
This update applies to the following version 8.5.5 editions:

· IBM WebSphere Application Server Version 8.5.5 (5724-J08)

· IBM WebSphere Application Server – Express Version 8.5.5 (5724-I63)

· IBM WebSphere Application Server Liberty Core Version 8.5.5 (5725-L29)

Only the WebSphere Application Server Version 8.5.5 editions that specified a load balancing and failover limitation are included in this update. Other editions, such as Network Deployment and z/OS, do not include a load balancing and failover restriction.

The program license information is available from the IBM Software License Agreements website. To search for a program license, select "Search for a specific program license agreement" and then search using the program number or name (for example, IBM WebSphere Application Server Liberty Core V8.5.5). Select the most recent V8.5.5 license.

http://www.ibm.com/software/sla/sladb.nsf