RSS Feed
Download our iPhone app
Browse DevX
Sign up for e-mail newsletters from DevX


Real-Time Tracking and Tuning for Busy Tomcat Servers : Page 4

Learn a technique for configuring your Tomcat servers to handle heavy traffic while determining how appropriate your current configuration is for changing load demands.


How It Works

As each request comes into Tomcat connector, the custom, performance-tracking Valve gets invoked. This Valve gathers information from the MBeans about the utilization of the pooled resources and then prints that information into a log file, along with the timestamp and other potentially useful information about the nature of the request. This log file is available for monitoring in real time, as well as for later analysis of the server's performance and capacity.

For my solution, I chose to modify the AccessLogValve supplied with the Tomcat source code. AccessLogValve produces a log file formatted according to Web logging standards, so its output looks identical to the standard Apache logs. Since these log statements contain very useful information by themselves, I preferred to keep them and just append extra resource pool state information to the log output. Of course, you can choose whatever information you want logged.

The following are the specific implementation steps for the solution:

  1. Implement a custom resource-tracking Valve. Tomcat supplies a predefined BaseValve that you can extend for this purpose. It is located in the catalina.jar file. Add this jar file to your classpath and have your custom Valve extend it as follows:
    public class ResourceTrackingAccessLogValve 
        extends ValveBase
        implements Lifecycle {

    (Note: I also implemented the Lifecycle interface. That is not important. It just adds a marks object with extra manageability features.)

    When extending the base Valve, override the method as follows:

       public void invoke(Request request, Response response)
            throws IOException, ServletException {

    This contains the code that will query MBeans for resource utilization and log that information to the log files.

  3. Inside the invoke method, access the resource pools MBeans to read the performance attributes as follows (See the accompanying source code and its comments for details):
               //Lazy instantiate MBeanServer using Tomcat's core  
                // MbeanUtility class 
               if ( this.mbeanServer == null ) { 
                     this.mbeanServer = MBeanUtils.createServer ();
                }//end if
                //Get the instance of the db pool MBean object
                // by fetching it through the ObjectName (connectionPoolMBeanID)  
               if ( this.dbPoolObjectName == null ) {
                    this.connectionPoolMBeanID.put( "name","\"" + this.getJdbcName() + "\"" );            
                   this.connectionPoolMBeanID.put( "class", "javax.sql.DataSource" );            
                    this.connectionPoolMBeanID.put ( "host", this.getHost() );        
                    this.connectionPoolMBeanID.put( "path", this.getApplicationPath() );
                    this.connectionPoolMBeanID.put( "type", "DataSource" );                 
                    this.dbPoolObjectName = ObjectName.getInstance( this.domain , 
    this.connectionPoolMBeanID ); } MBeanInfo mBeanInfo = mbeanServer.getMBeanInfo ( this.dbPoolObjectName ); MBeanAttributeInfo attributeInfo[] = mBeanInfo.getAttributes(); for (int i = 0; i < attributeInfo.length; i++) { if ( attributeInfo[ i ].getName().equals( this.DB_POOL_NUM_ACTIVE_ATTRIBUTE ) ) { this.dbConnectionCount = mbeanServer.getAttribute( dbPoolObjectName,
    attributeInfo[ i ].getName() ).toString(); } if ( attributeInfo[ i ].getName().equals( this.DB_POOL_MAX_ACTIVE_ATTRIBUTE ) ) { this.dbMaxSize = mbeanServer.getAttribute( dbPoolObjectName,
    attributeInfo[ i ].getName() ).toString(); } }//end for
  4. Log out the performance information each time the Valve is accessed, as follows:
           result.append( " | busy threads: " + this.threadsBusyCount ); 
           result.append( " active threads: " + this.threadsCurrentCount  ); 
           result.append( " active db connections: " + this.dbConnectionCount );
           result.append ( " max db connections : " + this.dbMaxSize );

    These log statements will produce output in the log file that looks similar to this:
 [29/Aug/2006:12:58:05:265 -0400] 12 200 896 /Test/  GET - - - - | busy  
    threads: 2 active connector threads: 51 active db connections: 8 max db connections : 30
  5. Register your Valve. Insert the following statement into Tomcat's server.xml file, right before the closing </Host> tag:
           <Valve className=" tomcat.tracking.ResourceTrackingAccessLogValve"
                   directory="logs"  prefix="localhost_resource_log." suffix=".txt"
                   pattern="%a %t %D %s %b %U %q %m" resolveHosts="false"/>   

    This directive instructs Tomcat to enable your custom Valve, and it configures the Valve to produce standard Web server log output into a localhost_resource_log.txt file.

  6. Install the Valve. I chose to deploy the custom Valve as a jar file and place it inside the Tomcat's server/lib directory. Putting this jar into the server's classpath provides it access to all of the Tomcat internal classes that the custom monitoring process utilizes.

  7. Parse and interpret the logs. As you run load and stress tests against your application, ResourceTrackingAccessLogValve will fill the localhost_resource_log.txt file with invaluable information related to the resource consumption and pooled resources utilization at run time.

This information will be of tremendous value for your production configuration efforts. Observe the maximum number of threads, and take note of the ratio between busy connector threads and active database connections. This ratio will tell you a lot about the health of your application, not only in terms of what your application consumes under various loads, but also how truly scalable your application is. Scalable applications tend to stay conservative on database connection usage as the number of Web connections grows.

Further Functions

The technique described in this article provides a practical understanding of Tomcat's resource utilization under heavy loads. In my experience, understanding key configuration parameters such as resource pools and understanding the real-time needs of production applications are essential to running scalable and responsive applications. However, you can use my proposed approach for more than just tracking resource utilization.

If you examine the Valve included with the accompanying source code and spend some time learning the access method for MBeans, you will probably find it useful for some other needs such as security tracking and pure performance monitoring.

I also strongly encourage you to use tools such as JConsole to explore the richness of Tomcat's MBeans API and learn more about the internals of the server as well as the runtime behavior of the application.

Edmon Begoli a software architect with 14 years of professional experience on large commercial and federal software projects. He is a member of the R&D staff at Oak Ridge National Laboratory.
Email AuthorEmail Author
Close Icon
Thanks for your registration, follow us on our social networks to keep up-to-date