[HTTP] Tuning IBM HTTP Server to maximize the number of client connections to WebSphere Application Server

Tuning IBM HTTP Server to maximize the number of client connections to WebSphere Application Server

Problem(Abstract)

Out of the box, IBM HTTP Server supports a maximum of 600 concurrent connections. Performance will suffer if load dictates more concurrent connections, as incoming requests will be queued up by the host operating system.

You can increase the number of maximum connections allowed by IBM HTTP Server by editing the httpd.conf file.

Resolving the problem

First and foremost, you must determine the maximum number of simultaneous connections required for this Web server. Using mod_status or mod_mpmstats (available with ihsdiag) to display the active number of threads throughout the day will provide some starting data.

There are 3 critical aspects to MPM (Multi-processing Module) tuning in IBM HTTP Server.

1. Configuring the maximum number of simultaneous connections (MaxClientsdirective)
   
2. Configuring the maximum number of IBM HTTP Server child processes (ThreadsPerChild directive)
   
3. Less importantly, configuring the ramp-up and ramp-down of IBM HTTP Server child processes (MinSpareThreads, MaxSpareThreads, StartServers)

The first setting (MaxClients) has the largest immediate impact, but the latter 2 settings help tune IBM HTTP Server to accommodate per-process features in Apache modules, such as the WebSphere Application Server Web server plug-in.

Configuring the maximum number of simultaneous connections

If the number of concurrent connections required is smaller than the current value of MaxClients (default 600), then MaxClients should remain unchanged and further tuning is likely not required.

If the number of concurrent connections is larger than the current value of MaxClients, this will be a very serious inhibitor of performance. MaxClients represents the total number concurrent connections IBM HTTP Server can process.

1. Determine a new value for MaxClients, which is the desired number of total simultaneous connections you require the server to support. This new value should remain a multiple of ThreadsPerChild.
   
2. If the new value of MaxClients is beyond 2000, consider a more horizontal topology where the IBM HTTP Server workload is distributed amongst different physical servers.
   
3. Update MaxClients in the httpd.conf file to reflect the desired capacity.
4. In order to increase MaxClients,it is typically necessary to make a corresponding increase in the value of the ServerLimit directive. ServerLimit does not directly change the available capacity, but acts as a limit on the total number of active IBM HTTP Server child processes. 
   
   Calculate a new ServerLimit by dividing MaxClients by ThreadsPerChild(this ratio is the required number of processes). If ServerLimitexceeds this prescribed ratio,MaxClientscan then be increased during a graceful restart of IBM HTTP Server.
5. If MaxSpareThreads was previously equal to MaxClients, increase MaxSpareThreads to MaxClients.

Configuring the maximum number of HTTP Server child processes

The number of IBM HTTP Server child processes is controlled by the ratio of MaxClients divided by ThreadsPerChild. Since MaxClients is dictated exclusively by the amount of load the system will need to handle, ThreadsPerChild is the directive used to change this ratio.

This does not affect how much load IBM HTTP Server can handle, just the way that load is distributed among separate processes.

Some features of various Apache components might perform better when fewer child processes, each with more threads, are used. For example, the WebSphere Application Server Web server plug-in MaxConnections parameter and ESI cache are each per-process and are more effective with fewer child processes (higher ThreadsPerChild).

The number of IBM HTTP Server child processes should be independent of any WebSphere Application Server or front-end load balancer settings.

There are reasons to avoid a high value for ThreadsPerChild. Some operating system facilities might require per-process locks that more threads suffer contention on, such as access to the heap. A child process crash results in all threads terminating, so in environments with frequent child process crashes it is reasonable to use a lowerThreadsPerChild.

Third-party modules might have per-process caches or connection pools that might favor a larger ThreadsPerChild for efficiency purposes. However, their architectures could just as likely favor a smaller ThreadsPerChild if a per-process resource is protected by a mutex.

Generally, heavily loaded systems doing SSL favor a small ThreadsPerChild due to contention on the native heap.

If the ESI invalidation servlet is configured in the WebSphere Plugin, each additional process causes a dedicated web container thread to be permanently consumed.

If a change to the number of IBM HTTP Server child processes is required:

1. Determine the number of child processes desired, and divide the current value of MaxClients by this number.
   
2. Replace ThreadsPerChild with the value obtained from step 1.
   
3. Replace ThreadLimit with the value obtained from step 1.
   
4. If ThreadsPerChild has increased by a large amount, evaluate the current value of StartServers (the initial number of child processes created). Note that StartServers is not a very important tunable, because IBM HTTP Server re-evaluates the number of available threads every second.

Configuring the ramp-up and ramp-down of HTTP Server child processes

With the default httpd.conf file, IBM HTTP Server tries to maintain a pool of at least 25 spare processing threads and no more then 75 at any given time. These are specified with the MinSpareThreads and MaxSpareThreads directives respectively.

MaxSpareThreads

It is suggested that you disable the periodic termination of IBM HTTP Server child processes by setting MaxSpareThreads equal to MaxClients. Under this configuration, IBM HTTP Server child processes are created based on demand but not killed when the demand recedes. This is the recommended configuration because of complexities of graceful process termination and long-running requests; child processes are replaced as soon as they are scheduled for termination but might take considerable time to fully exit.

If periodic graceful child process termination is desired, MaxSpareThreads should be set to any value substantially smaller than MaxClients. If a module is maintaining a per-process cache or connection pool, like the WebSphere Application Server Web server plug-in, these will be discarded and created anew for the replacement child process.

MinSpareThreads

IBM HTTP Server compares the number of available processing threads to MinSpareThreads every second to decide whether or not to spawn additional child processes. Each additional child process contains ThreadsPerChild number of threads.

MinSpareThreads is usually safely set to the same value (or double) ThreadsPerChild. Since this value is checked every second, it is not an especially sensitive setting.

Example 1:

This configuration uses a single IBM HTTP Server child process with 2000 threads.

With this configuration, there will be only ONE child process with 2000 threads. Consequently, there will also only be ONE plug-in instance. This is much better than having many processes and many plug-in instances.

This also maximizes the effectiveness of the ESI cache as well as the shared information about marked-down Application Servers, which are both maintained by the WebSphere Application Server Web server plug-in on a per-process basis.

However, a crash in this child process takes out all 2000 threads. Heavy SSL usage, or heavy load under old operating systems, might consume extra CPU.

<IfModule worker.c>

ServerLimit 1
ThreadLimit 2000
StartServers 1
MaxClients 2000
MinSpareThreads 2000
MaxSpareThreads 2000
ThreadsPerChild 2000
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512

# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>

Example 2:

This configuration uses multiple IBM HTTP Server child processes, with a fixed capacity of 2000 concurrent connections spread amongst 20 child processes.

If MaxConnections is used with the WebSphere Application Server Web server plug-in, MaxConnections will refer to the number of backend connections permitted in each IBM HTTP Server child process. While the number of processes is predictable in this case, the distribution of users over these processes is not. For this reason MaxConnections would be difficult to use in such an environment.

If the ESI cache is used, it is fragmented or duplicated between the child processes, which is ineffective.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry -- this information is not shared between child processes.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.

<IfModule worker.c>

ServerLimit 20

# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with all 20 processes
StartServers 20

MaxClients 2000
MinSpareThreads 2000

# Don't kill child processes in response to load
MaxSpareThreads 2000

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512

# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>

Example 3:

This configuration uses multiple IBM HTTP Server child processes, with a variable capacity of 2000 concurrent connections spread amongst 1-20 child processes.

If the ESI cache is used, it is fragmented or duplicated between the child processes, which is ineffective.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry -- this information is not shared between child processes.

To mitigate the impact to the WebSphere Application Server Web server plug-in, you can configure IBM HTTP Server to conservatively create new child processes and aggressively kill off old ones. This ensures that the smallest working set of child processes will know about ESI, markdowns, and so on.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.

Note: While MinSpareThreads and MaxSpareThreads do not have to be multiples ofThreadsPerChild, it is convenient to use multiples of ThreadsPerChild to ensure that there is no churn between creating a new child process and immediately killing it.

<IfModule worker.c>

ServerLimit 20

# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with only 2 processes
StartServers 2

MaxClients 2000

# Don't create a new process until we

# have one process worth free.
MinSpareThreads 100

# Kill a child when we've got an entire extra process
MaxSpareThreads 100

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512

# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>

Notes:

• Logging of static files might contribute to load on your IBM HTTP Server system, see the "conditional logging" topic of the IBM HTTP Server information center for further information:
  http://publib.boulder.ibm.com/httpserv/manual60/logs.html
• IHSDIAG has more detailed performance tuning information: