IBM WebSphere Application Server V8.5 enhances the High Performance Extensible Logging (HPEL) feature that was first introduced in V8.0. As the name suggests, HPEL was created to significantly outperform the existing application server logging and tracing mechanism, plus provide a powerful way to extend log and trace entries with extra fields. Beyond that, HPEL provides a number of ease of use improvements for working with log and trace content. Enabling HPEL is quick and easy, and does not require any code changes to your applications.
This article provides a primer for administrators considering trying HPEL.
Throughout this article, "HPEL" will refer to the new log and trace system in WebSphere Application Server V8.5, and "Basic" will refer to the pre-existing log and trace system.
HPEL is a new log and trace content storage and access system for WebSphere Application Server. HPEL stores log and trace content produced by the existing log and trace APIs, in particular the java.util.logging API.
HPEL addressses a number of common needs:
- Log and trace performanceIf you have worked with IBM support teams, you might have been asked to turn on tracing for certain components. This might have been problematic to do on your production systems, because enabling tracing could significantly affect the performance of your applications. HPEL log and trace is significantly faster than Basic log and trace, making tracing much less impactful to your application performance. Using the DayTrader benchmark and EJBContainer=fine trace specification, HPEL boosted overall benchmark performance by 33% compared to using the same trace specification with Basic log and trace (measured using WebSphere Application Server V8.0).
- Log and trace entry extensibilityBeing able to extend log and trace records can be very useful. For example, you might want to add a userID field to every log and trace record so that you can search your logs or traces later to find records with a particular userID value. There is no mechanism for doing this with the Basic log and trace system.
- Automating log/trace analysisAs a WebSphere Application Server administrator, you might have had a need to automate the scan of SystemOut.log and SystemErr.log for significant events. This is non-trivial with the Basic log and trace system because log and trace entries can span multiple lines in the log/trace files. The SystemOut.log and SystemErr.log files also rolled (meaning that when they reached a certain size, they were renamed and truncated back to 0 length), adding further complexity to automation.
- Application separationTo facilitate problem determination, you might want to separate the logs and traces for each application into a separate file. There is no mechanism with the Basic log and trace system to make that easy to do. If you had created your own log files per application (perhaps using Log4J), you might have found it cumbersome to correlate content in your log files with WebSphere Application Server log files, or to correlate log and trace content between applications when looking for patterns.
- Request trackingYou might have wanted to gather the log and trace records that made up a single request to your server, where a request might have been served by multiple application server threads, or different application server JVMs (such as for web services requests). The Basic log and trace system does not provide that capability. HPEL delivers this when used in combination with Cross Component Trace (XCT).
- Remote access to logs/traceThe Basic log and trace system provides the ability to view chunks of the SystemOut.log, SystemErr.log, or trace.log in the administrative console (WebSphere Integrated Solutions Console). HPEL provides a greatly improved user interface capable of filtering and browsing content more easily. HPEL also provides the ability to export log and trace content from the administrative console without need for operating system access. HPEL log and trace content can also be remotely accessed via a JMX MBean interface, or via the HPEL API.
- Access to z/OS® logs/traceIf you run your applications on the z/OS platform, you might have difficulty getting your application developers to look at the log and trace content when problems occur, as they are more comfortable in a Windows® or Linux® environment. HPEL lets z/OS and other operating system teams work with logs and trace in a consistent way.
The Basic log and trace system stores log content in multiple places, including the SystemOut.log, the service log (activity.log), and the trace.log (whenever trace is enabled). System.err content is stored separately – in the SystemErr.log – making it challenging to correlate with content in the SystemOut.log. Trace content is stored in the trace.log. This makes the trace.log the primary log for problem determination when trace is enabled, and the SystemOut.log the primary log for problem determination when trace is not enabled.
In contrast, HPEL stores log, System.out, and System.err content in a log data repository, and trace content in a trace data repository (Figure 1).
To view the content from the log data repository and trace data repository, you can use the
logViewer command line tool. The logViewer command provides access to all content in both repositories. It also enables live monitoring – similar to running tail -f SystemOut.log – and enables filtering so you can restrict what content it provides as output.HPEL also provides a text log (which is optional, but enabled by default). The text log mirrors the content of the log data repository, but does so in plain text, with the same format you're used to from SystemOut.log. The text log can be enabled to additionally mirror the content of the trace data repository, though for performance reasons, you should avoid doing that on a production system. The text log is handy if you need a plain text log for any reason.
Figure 1. Comparison of where content is stored in Basic and HPEL modes
Configuration
The switch from Basic mode to HPEL mode is done on a server-by-server basis. The main Troubleshooting > Logs and Trace > serverName panel provides a button with which to switch each server to HPEL. Switching back to Basic is just as easy, but is done via the Change log and trace mode link on the same panel. In either case, you will need to restart the server whose log and trace mode you changed in order for changes to take effect.
The administrative console also provides HPEL configuration options for the log data repository, trace data repository, and text log, and lets you independently specify data retention and buffering options for each.
Be aware that HPEL can be configured for use on your deployment manager and node agent processes as well. Enabling HPEL on your node agent process is a good idea, as it enables you to use the Log Viewer in the deployment manager GUI to access the logs of any server on that node, even when those servers are stopped.
HPEL was first introduced in WebSphere Application Server V8.0. The initial design included the ability for the log and trace data repositories to have extensible records, but that capability was not surfaced to application developers or administrators.
In WebSphere Application Server V8.5, the application server uses this extensibility to store the application name in log and trace records created during requests to Java™ EE applications. It also uses this capability to store the Cross Component Trace requestID, created when Cross Component Trace is enabled, in the log and trace records. Finally, an API is provided in V8.5 to enable you to add your own extensions to log and trace records.
To make use of these extension fields, the
logViewer command line tool was extended to include an -includeExtensions option, enabling it to filter log and trace records by the content of extension fields.Out of the box, WebSphere Application Server V8.5 tracks
appName and requestID extensions, but you can feel free to add your own extensions to your log and trace entries using the com.ibm.websphere.logging.hpel.LogRecordContext API.Let's take a practical look at how you might typically use HPEL.
When problems occur on your production application servers, it can be a time consuming task to discover and isolate the most important content in your log and trace files. Imagine that somewhere in your server's log files an error occurred on one of your systems.
Traditionally, you would have three options for finding problems in your log files:
- Look at each log manually, perhaps grep-ing for " E " or " W " or "Error" or "Exception," and so on.
- Write scripts to do the above searches.
- Use tools to mine your logs for warnings, errors, and exceptions.
In its simplest form, you can output all content from your logs and trace using this simple command from your profile bin directory (output shown in Figure 2):
./logViewer.shFigure 2. Output from logViewer.sh command
Let's try to do something more interesting though. To find any warning or error messages in your log file you can use this command (output shown in Figure 3):
./logViewer.sh -minLevel warningFigure 3. Output from logViewer.sh command
Notice in Figure 3 that the server was restarted, and by default the logViewer output includes the content from all ”instances” of the server. Importantly, logViewer never alters the repositories, it just provides a filtered, formatted view.
Now, let's further restrict your query to only see content since the last time you restarted your server. That's easy too, with the -latestInstance option (output shown in Figure 4):
./logViewer.sh -minLevel warning -latestInstance Figure 4. Output from logViewer.sh command
Maybe, you've even got a dozen applications running on the same WebSphere Application Server JVM -- and you'd like to check for warning or errors in your LemonadeStand application. Using WebSphere Application Server V8.5, you can filter your log and trace content by Java EE application name, leveraging the appName extension as follows (output shown in Figure 5):
./logViewer.sh -minLevel warning -latestInstance -includeExtensions appName="LemonadeStand"Figure 5. Output from logViewer.sh command
Now that you've found an interesting error in your logs, you might want to see what happened during the rest of the request on the same thread. This time, you can also eliminate any trace records by setting the minLevel option to info (output shown in Figure 6):
./logViewer.sh -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel infoFigure 6. Output from logViewer.sh command
If you are using HPEL in combination with the new Cross Component Trace (XCT) capability introduced in WebSphere Application Server V8.5, and the error you are interested in occurred during an HTTP or SIBus request, you can filter your logs by (the XCT supplied) requestID. To see the requestIDs, rerun the previous query with the -format advanced option (output shown in Figure 7):
./logViewer.sh -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel info -format advancedFigure 7. Output from logViewer.sh command
With the output in advanced format, you can see the requestID for that message. As another tip, viewing your logs in advanced format is also a great way to get the full logger name from the log entry (which appears as the 'source' field in log/trace records); this is helpful in case you want to enable more granular tracing for a particular logger. Similarly, you can see the name of the thread that logged/traced each entry in the advanced format.
Armed with the requestID of the request that had the error you are interested in, you can now query the repository to see other log and trace entries that are present in the same request. HPEL stores the requestID as an extension to log and trace records, so you use the -includeExtensions option, as follows (output shown in Figure 8):
./logViewer.sh -includeExtensions requestID=AAAKLwUVkuH-AAAAAAAAAACFigure 8. Output from logViewer.sh command
Having isolated a problem in your log files, you might want to send the logs for your LemonadeStand application to a teammate. You can store your output in a log in plain text using the -outLog option:
./logViewer.sh -includeExtensions appName="LemonadeStand" -outLog /tmp/myLogs.logAlternatively, you could store your filtered output as a new HPEL repository, using the
extractToNewRepository option, so your colleagues can use the logViewer command on their system to continue to manipulate the data:./logViewer.sh -includeExtensions appName="LemonadeStand" -extractToNewRepository /mySharedDrive/myLogsUpon receiving word of your great work, your colleague could then access what you stored in the log. To do so, they would specify the location of the repository using the
repositoryDir option:./logViewer.sh -repositoryDir /mySharedDrive/myLogsWhen developing or troubleshooting applications, it's often helpful to be able to monitor the log output in real time. Perhaps you are aware of a problem and are trying to get it to reoccur, or you want to watch for a specific event as part of testing your application. At one point or another, you've probably found a need to fire up a shell window and execute a
tail -f SystemOut.log to monitor your log output. HPEL can make this type of task easy by enabling you to monitor a log repository for new events without having to monitor multiple log files or worry about files rolling over, interfering with your ability to tail them. Try it out by simply invoking the HPEL logViewer tool with the
-monitor option:./logViewer.sh -monitorA key benefit with using HPEL's monitor option is that it works across platforms, providing you a consistent option for monitoring new log and trace events, regardless of your hardware or operating system. This includes the z/OS® platform, so it's possible to easily view or work with your application logs on z/OS without needing to be a z/OS system administrator or have access to log spools.
The monitor feature can be combined with filtering options to watch for specific events. Imagine you need to monitor a verbose logging application for critical error messages that pertain to a specific username that is part of a message. You could do this with something like the following to see all new severe or fatal log events that contain the string “user33” as part of their message, without the need for grep or other tooling:
./logViewer.sh -monitor -minLevel SEVERE -message “*user33*”As you might expect, it is possible to combine multiple filter options to further fine tune what log events you want displayed. For example, you could further limit the records displayed to those generated by the LemonadeStand application as follows:
./logViewer.sh -monitor -minLevel SEVERE -message “*user33*” -includeExtensions appName=”LemonadeStand”By combining options for the logViewer tool, administrators can easily view all or only the specific log data that is required directly from the HPEL repository. LogViewer provides a consistent and powerful tool for viewing WebSphere Application Server log and trace data regardless of platform. It does not require that any special tooling be installed or administrators to copy log files to their workstations.
Now that you've seen how easy it can be to use HPEL to view live log data and filter that output from a command line, lets look at how easy it is to view log data from within the WebSphere Application Server administrative console.
Another common scenario that many system administrators may have faced is the need to ensure limited operating system access to key production application servers, while still allowing application support teams or development groups access to the production application's logs. WebSphere Application Server now provides an improved web-based log viewing and export utility via the administrative console when HPEL is enabled.
While Basic mode log and trace has for some time provided simple log viewing capabilities, HPEL provides an improved console log viewing experience. The HPEL Log Viewer (not to be confused with the logViewer command line tool) can be used to view HPEL log and trace repository data by anyone authorized to access the administrative console. An application support specialist with Monitor role access to the administrative console can view log data without needing OS level access to the production servers or having to wait for their system administrator to collect the log files on their behalf.
Figure 9. Administrative Console view of log contents
The console's Log Viewer provides all the same powerful filtering options found in the command line logViewer tool. To view log and trace data via the administrative console, navigate to Troubleshooting > Logs and trace and select your HPEL enabled application server from the list, then choose the View HPEL logs and trace link on the Logging and tracing panel.
The full log repository (or even a filtered subset of the repository) can be exported to a .zip file for local analysis, if required, directly from the administrative console's Log Viewer. Simply select the export button at the top of the Log Viewer. You can choose to export the entire repository or just your current (filtered selection) to a new binary HPEL repository. Anyone you share the repository with can use the logViewer command to manipulate the file later. If preferred, you can export the entire repository or the current selection to a text file in basic or advanced format as well. The logViewer command-line tool can read HPEL repositories from inside an exported zip file -- no unzipping needed.
Figure 10. Administrative Console export log dialog
HPEL utilizes a new way to store log and trace data for improved performance and extensibility. At the same time, HPEL maintains compatibility with your existing log manipulation scripts by providing the same data formats you are already familiar with from WebSphere Application Server logs. New command line and administrative console tools expedite the tasks of log and trace monitoring and filtering. In addition to the tools provided, tool providers can utilize the new HPEL Java API, which provides access to HPEL repositories locally as well as remotely, and output results in basic or advanced text formats, or even store them in a new HPEL repository (see Resources).
Switching to and from HPEL in WebSphere Application Server can be done quickly in the administrative console and requires a server restart to take effect. This ability to easily switch between logging modes means you can try out HPEL and switch back if you find any need for the Basic mode. IBM support teams are able to work with HPEL repositories – whether exported from your administrative console, output from the logViewer command, or simply zipped up from your server logs directory.
Resources
Learn
- WebSphere Application Server V8.5 product information
- WebSphere Application Server V8.5 Information Center
- High Performance Extensible Logging (HPEL)
- IBM Education Assistant - High Performance Extensible Logging
- Javadoc – com.ibm.websphere.logging.hpel.reader
- Javadoc – com.ibm.websphere.logging.hpel.writer
- Cross Component Trace (XCT)
- IBM developerWorks WebSphere
댓글 없음:
댓글 쓰기