Monitoring and Managing Tomcat Clusters Using JMX

he latest version of the Tomcat 5 servlet container provides built-in support for monitoring server components using the Java Management Extensions (JMX) API. This article concentrates on the clustering and load balancing components, and provides an overview of configuring and using JMX MBeans in managing the elements in a J2EE cluster. You’ll also see how to use the JMX API to automate various administration tasks in the Tomcat container.

A sample Web application (see the downloadable sample code) to demonstrate the steps involved in configuring Tomcat server instances with load balancing and clustering features enabled. This application provides the ability to monitor the run-time statistics of the cluster elements such as load balancer, cluster members, session manager, and replication valve.

What You Need
Tomcat 5, Ant, MX4J, Commons Modeler, Commons Digester, Commons Collections, Eclipse

Clustering, Load Balancing, and Fault Tolerance
Before getting into the design and implementation details of a Tomcat cluster, here’s a brief overview of clustering in J2EE servers. A cluster is a group of application servers that transparently run a J2EE application as if it were a single entity. In other words, the application runs on multiple cooperating servers. Clustering helps achieve scalability (a system’s ability to support increasing numbers of users) because you can add additional servers to the cluster, and also helps achieve high availability (providing redundancy in the system so if one server instance fails for some reason, another member in the cluster can transparently take over the Web requests).

Load balancing, also known as “high availability switch over,” is a mechanism that distributes server load to different nodes within the server cluster based on a load balancing policy. For example, rather than execute an application on a specific single server, the system executes that application code on a dynamically selected server. When a client requests a service, the load-balancing system selects one (or more) of the cooperating servers to execute the request.

Fault tolerance is a system’s ability to allow a computation to continue or “fail over”?as transparently to the end user as possible?on another server when a server in the cluster goes down. The cluster service should detect when a server instance is no longer available to take requests and stop sending requests to that instance. It should also periodically check to see if a failed cluster member is available again, and automatically add it to the pool of active cluster nodes.

Many Web applications need to store per-user values on the server. The values for all the individual active sessions are collectively called session data. The problem is that each server in the cluster must have equal access to the session data for load-balancing and fail-over to work. One way to handle the problem is to persist session state at multiple servers in the cluster, a process called session replication. With session replication, a client can seamlessly get session information from another server in the cluster if the original server, on which the client first established a session, fails. The main goal of session replication is to prevent the loss of any session information if a cluster member crashes.

Clustering in Tomcat 5
Clustering was available in Tomcat’s previous version (version 4.1) as a third-party library (JavaGroups is a popular choice for adding clustering capabilities in Tomcat 4.1); but in Tomcat 5 clustering is built-in. It’s available in the main installation package. Session replication in Tomcat 5 is an all-to-all replication of session state, meaning that session attributes are constantly propagated to all other cluster members.

However, load balancing capability was not available in the previous versions of Tomcat. Until now, people have used the Apache Web server as the popular choice for load balancing HTTP requests. Tomcat 5 provides load balancing in three different ways: using the JK native connector, using Apache 2 with mod_proxy and mod_rewrite, or using the balancer Web application.

For more details on Tomcat 5 support for clustering refer to the article “Clustering and Load Balancing in Tomcat 5, Part 1“.

Server Manageability Using JMX
The Java Management Extensions (JMX) specification defines the architecture, services, and API for the distributed management of resources using Java. As a Java technology standard for instrumentation, JMX introduces a JavaBeans model for representing the manageability of resources and a set of services to help manage these resources. Similar to JDBC (for accessing relational databases) and JNDI (for accessing application and directory resources), JMX is transparent and hides the details of managing enterprise resources. It acts as an isolation layer between the applications and the management systems. In real-world applications, JMX is used for JDBC connection pool management, HTTP session management, cache monitoring, and JMS queue management.

JMX Architecture
JMX has three main components (layers) that work together to provide manageability for J2EE application resources.

  • Instrumentation Layer: This layer contains managed beans (MBeans) and their manageable resources.
  • Agent Layer: This contains JMX agents used to expose MBeans, such as MBean Server and monitoring, timing, relation, and class-loading services. These standard agent services include M-Let Service, Timer Service, Monitoring Service, Relation Service.
  • Distributed Layer: This layer contains components that enable management applications to communicate with JMX agents. The distributed layer consists of one or more adapters that are used to translate between JMX components and their respective management systems. Some of these adapters are HTTP Adapter (Web pages for viewing management data), RMI Adapter, SOAP Adapter, and SNMP Adapter.

Implementing a Typical JMX Application
In the instrumentation layer you first define a managed bean (MBean). MBeans are Java objects that encapsulate a manageable resource and expose it for monitoring and management purposes. A manageable resource is any resource that can be accessed in the J2EE server.

After creating the MBean, you need to make it available for any client that needs to access it. You do this by registering the MBean in a JMX agent such as MBeanServer. MBeanServer is the registry for all MBeans in the server, and all management operations go through MBeanServer. You can use MBeanServer to find registered MBeans, get the list of operations available for a specific MBean, and determine the return data type of each operation).

Finally, from the distributed layer, you will use an adapter (usually an HTML adapter) to communicate with your agent to access all registered MBeans, view their readable attributes, update the writable attributes, and invoke their operations (methods).

JMX JSRs:
Multiple Java Specification Requests (JSRs) relate to JMX technology. Table 1 summarizes the various JSRs currently listed on the JCP Web site.

Table 1. JSR Requests related to JMX and Application Management

JSR# Name Description
3 Java Management Extensions (JMX) Specification The JMX specification will provide a management architecture, APIs and services for building Web-based, distributed, dynamic and modular solutions to manage Java enabled resources.
70 IIOP Protocol Adapter for JMX Specification This specification will establish an IIOP-based adapter for the JMX specification to allow CORBA clients access JMX agents.
71 JMX-TMN Specification This JSR specifies the interoperability between the Telecommunication Management Network (TMN) standards and JMX.
77 J2EE Management Provide server vendors and tool vendors with a standard model for managing the J2EE Platform. J2EE Management will be defined using standard management information formats and protocols, such as JMX, WBEM, and SNMP, in such a way as to provide information that can be managed by standard system management applications.
146 WBEM Services: JMX Provider Protocol Adapter Define the architecture and API for WBEM support for Java. This specification will define how JMX instrumentation can be mapped to CIM and the definition of a JMX Provider Protocol Adapter for WBEM Services.
160 Java Management Extensions (JMX) Remote API Extend the Java Management Extensions (JMX) 1.0 specification, by adding the Client APIs. These APIs provide to any Java Manager discovery and access to JMX Agents abstracting the underlying protocol.
174 Monitoring and Management for the JVM machine A specification for APIs for monitoring and management of the Java virtual machine. These APIs will provide Java applications, system management tools and RAS-related tools with the ability to monitor the health of the Java virtual machine as well as manage certain run-time controls such as Class load/unload, Memory allocation and Garbage collection statistics, Thread details, and Object info (object count in the Java heap).

The JavaSoft Web site provides links to products using JMX technology. Another link on the site lists open source and commercial implementations of the JMX specification.

Some popular implementations of JMX technology are JBoss, WebLogic Server, Tomcat 5, and MX4J. All WebLogic server resources such as JMS servers and JDBC connection pools are managed through the JMX-based services. You can also build your own management utilities that use these JMX services to manage WebLogic Server resources and applications.

MBean Types:
There are four types of MBeans:

  • Standard MBeans: These MBeans are the simplest to design and implement. The management interface consists of method names (getters and setters), used to store static management information where the structure of the managed component is well defined and is not likely to change.
  • Dynamic MBeans: These MBeans expose a more flexible management interface at runtime via generic methods such as getAttribute, setAttribute, and invoke. They must implement the DynamicMBean interface. Dynamic MBeans are useful when the management interface is changing.
  • Open MBeans: These are dynamic MBeans that use a subset of universal Java types. They have a specific set of metadata classes to manage the resource.
  • Model MBeans: These are special highly dynamic MBeans that are configurable at runtime. Model MBeans are self contained, self described, and are configurable. You provide the required metadata information about the classes that are being managed (as opposed to those managed classes implementing interfaces as in the case of Standard MBeans). These MBeans allow instrumentation of resources where you might not be able to modify or enhance the existing code to weave in the instrumentation logic.

JMX Support in Tomcat 5
JMX support was available in Tomcat 4.1 release, but has been enhanced in Tomcat 5 to provide a comprehensive server monitoring and administration using JMX. JMX is now a core part of Catalina, Tomcat’s servlet implementation. Tomcat 5 uses JMX MBeans for implementing the manageability of Tomcat. You can now manage all Tomcat internals?such as Service, Engine, Host etc.?using JMX technology. Also, the Tomcat server itself can be launched as an embedded server using JMX.

Tomcat uses the JMX open source framework MX4J to implement JMX support. You don’t need to install MX4J separately to start using JMX; the Tomcat 5 installation package includes MX4J as part of the bundle. The JAR file for the MX4J API is located in the TOMCAT_HOME/bin directory (in the file jmx.jar, which also includes the core JMX API).

Here’s a short list of the various types of managed resources available in Tomcat 5 server:

  • Architectural components: Server, Service, Connector, Engine, Host, Context, Default Context, and Cluster.
  • Nested components: Realm, Logger, Loader, Valve, and Manager.
  • Runtime Data Objects: User Database, User, Group, and Role.
  • Resource Objects: Naming Resources, Environment, Resource, Resource Link, WebModule, Request Processor, Cache, and Thread Pool.

The book Professional Apache Tomcat 5 contains a very good discussion on all these components. It provides a list of the attributes for all the manageable components listed above.

Tomcat provides two ways to access these manageable components: the JMX Proxy Servlet provided as part of the Manager application (this is based on the HTTP protocol adapter) and the RMI connector (this adapter is implemented using MX4J framework). Visit the MBeans home page on the Tomcat 5 Web site for more details on Tomcat’s MBean support.

The sample Web application described in this article uses the JMX API to manage a Tomcat cluster. The Model MBeans approach shown here gives you a great deal of flexibility in instantiating managed beans for cluster elements. Model MBeans are:

  • Generic, flexible, and extensible
  • Able to use a common template for different MBeans
  • Configurable at run-time
  • Rapidly instrumentable

The sample code uses the Jakarta Commons-Modeler framework to simplify the tasks of instantiating the model MBeans and accessing the components in the cluster.

The Jakarta Commons-Modeler Framework
The Jakarta group created the Commons-Modeler Framework to provide a flexible way of working with MBeans. This framework is based on Model MBeans which let you define MBeans for many different components without having to write a specific implementation for each MBean. You can use the Commons-Modeler API to simplify setting up metadata about a managed component (Java class) such as its constructor, attributes, and methods. You supply the required runtime metadata information in an XML MBean descriptor file (The XML file must conform to the DTD supplied with the Modeler installation package). Using an XML file to describe the Model MBean details is powerful and extensible, and a much better approach than hardcoding the information.

 
Figure 1. Tomcat Cluster Hierarchy: The figure shows the hierarchy of server components in a Tomcat cluster.

The Commons-Modeler framework also provides a registry and a base Model MBean to work with, which makes accessing the manageable components relatively simple. Tomcat 5 uses this framework extensively to simplify its JMX implementation. Every Tomcat MBean component has an attribute called modelerType that describes the component’s MBean type.

Figure 1 shows the hierarchy of the server components in a Tomcat cluster.

The theoretical Tomcat cluster setup described here consists of four instances of Tomcat server. One Tomcat instance serves as a load balancer and the other three instances are part of the cluster. The cluster was set up with a vertical scaling method (multiple Tomcat server instances running on a single machine) in which one server group and two clones were configured in the cluster. The clones were configured with the identical Web application directory structure and contents as the server group to optimize session replication.

The Sample JMX Application
The sample Web application, called jmxapp, is based on the JMXProxyServlet application provided with the Tomcat 5 installation. It uses an MBean descriptor file located at jmxapp/WEB-INF/classes directory/mbeans-descriptor.xml. A Model MBean is instantiated within the JMX MBean Server for each manageable component listed in Table 2.

Custom MBeans
I wrote custom Java classes to make the cluster elements JMX components. The cluster itself is not shown as part of the accessible JMX components accessible when you launch the Tomcat server. These custom classes provide useful statistics for a cluster monitoring and management framework. For example, session statistics such as default session timeout and number of currently active sessions are exposed as MBean attributes using the custom session manager class. You instantiate and register these MBeans when Tomcat server is started (Make sure you copy these class files to the %TOMCAT_HOME%/server/classes directory because Tomcat requires these classes at startup time).

MBean Naming Syntax:
To provide unique names, JMX specifies a two part naming convention for MBeans. The parts are:

  • JMX Domain: The domain name is a case-sensitive string that defines a top level within the JMX namespace. For example, the domain in Tomcat server is called “Catalina”. Therefore, all the manageability components have MBeans names with a “Catalina” prefix.
  • Key-Value Pairs: The second part of an MBean name is a set of one or more name-value pairs. These key properties create unique object names within a specific JMX domain and they provide information about the MBean. This set doesn’t have to be ordered; in other words, the order of the key-value pairs is not significant, but the name must begin with the domain. Using key-value pairs, you can supply information such as names, host name, port, locations, application paths, and so forth as shown below
   domain:Name=name,Type=type[,key=value]

In the preceding line of code, Name is the string that you provide when you created the managed resource that the MBean represents and Type is the interface class of which the MBean is an instance.

Here’s a more expansive example. The following three lines represent the MBeans names for the Service, Host, and Replication Valve components in Tomcat server.

   Catalina:serviceName=Catalina,type=Service   Catalina:host=localhost,type=Host"    Catalina:type=Valve,name=ReplicationValve,      host=localhost" 

Table 2 shows the list of MBeans monitored using the JMX Web application and their object names.

Table 2. Cluster MBean Names

Managed Component MBean Name
Load Balancer Catalina:host=localhost,name=SimpleLoadBalancer,type=LoadBalancer
Cluster Catalina:host=localhost,name=SimpleTcpCluster,type=Cluster
Session Manager Catalina:host=localhost,path=/jmxapp,type=Manager
Replication Valve Catalina:type=Valve,name=CustomReplicationValve,host=localhost

Table 3 shows the list of attributes for the MBeans monitored by the sample Web application.

Table 3. Cluster MBeans and their attributes.

Cluster Element Attributes
Cluster ManagerClassNameCluster MembersMembershipServiceContainerCluster SenderClusterReceiver
Load Balancer Load Balancing Rule Name (Round Robin or Random)Server InstanceTotal number of instancesTCP Listen AddressTCP Listen Port
Session Manager Number of sessionsManager Class NameNumber of ActiveSessionsInvalidated SessionsRejected SessionsMaximum Inactive IntervalSession Id, Creation Time, Last Accessed Time for each session
Replication Valve ControllerContainerRequest Filter details

JMX Web App Framework Elements
The main components in the JMX Web application framework are:

  • JMXUtility: The utility class for defining the cluster MBeans instantiation and registration in the MbeanServer. The custom servlet context listener class calls the methods in this class to register MBeans when jmxapp Web application context is initialized.
  • JMXMonitor: This class has methods to get MBean attribute details for the manageable server components. The jmx console script (jmxmonitor.jsp) calls these methods to get the details of JMX components.
  • CustomDeltaManager: This is the custom session manager class created to access the session information. It provides the getter methods for session attributes to get the details of HTTP Session.
  • CustomReplicationValve: This is the custom replication valve created to access the request filter details. It was created so the filter pattern (*.js, *.jpg, *.html, etc) can be viewed and modified dynamically using MBean methods.
  • SimpleLoadBalancer: This class represents MBean for the load balancer used in the cluster Web application. SimpleLoadBalancer encapsulates the load-balancing algorithm, number of cluster members, their IP addresses, and port numbers.
  • RuleBean: This is a java bean to store the load balancer rules used by Tomcat server to perform load balancing. It has the getters and setters to access and manipulate the load balancing algorithm rules.
  • XMLUtility: A utility class to parse the load balancer rules XML file and populate the RuleBean. It uses Commons Digester package to simplify the XML processing to parse the load balancing details stored in rules.xml file and populate RuleBean object.
  • ClusterJMXContextListener: This is the custom servlet context listener created to call the MBean instantiation methods (in JMXUtility) when Tomcat server is started.
  • WebAppLogger: Implementation of Log4J for logging debug messages in the Web application.
 
Figure 2. JMX Web Application Class Diagram: The diagram shows the relationship among the classes that administer Load Balancer and Clustering components and their relationship.

Figure 2 shows the class diagram with the custom java classes created to administer Load Balancer and Clustering components and their relationship.

The sample project contains only two JSP files. JmxMonitor.jsp is the main page for the Web application. It is used to check the status of the cluster components. JmxOutput.jsp displays the MBean details for the cluster elements. There are HTML links for each component to view the MBean attributes using the HTML adapter.

Configuration Files
You need the following three configuration files:

  • server.xml configures clustering in a Tomcat server instance. The version that ships with the Tomcat installation has the cluster configuration commented out. You enable the cluster by uncommenting the XML element. You also specify the custom cluster classes in this configuration file.
  • Web.xml specifies whether Web application session data needs to be replicated. You also use this file to configure the custom servlet context listener.
  • rules.xml defines the custom load-balancing rules. This is the file where you specify which load-balancing rule you want to use to distribute the load among the cluster members.

You’ll find one additional file called log4jconfig.properties. This is the properties file created for Log4J configuration.

The sample code provided with this article includes all the source files you need to run the Web application.

Web App Request Flow:
Here’s the sequence of events for each Web request handled by the sample Web application:

  1. Launch the startup page (http://localhost:8080/jmxapp/jmxmonitor.jsp).
  2. JSP calls the methods in the JMXMonitor class to retrieve the MBeans details and forwards the request to jmxoutput.jsp.
  3.  
    Figure 3. JMX Web Application Sequence Diagram: The figure shows a visual representation of the sequence of events in the sample Web application.
  4. jmxoutput.jsp displays the MBean details for the elements in the Tomcat cluster. This page also has links to show more MBean information using the jmx-html HTML adapter.
  5. Click on the jmx-html view link to see the MBean attributes using the HTML adapter user interface.
  6. Open a new Web browser and launch the cluster application’s main page (http://localhost:8080/balancer/testLB.jsp).
  7. Modify the session data by adding or changing a session attribute.
  8. Refresh the MBean details on the jmxoutput.jsp Web page and ensure that the session details have been updated to reflect the changes you made.

The sequence diagram in Figure 3 provides a visual representation of these steps.

JMX/MBean Monitoring Console

 
Figure 4. Cluster MBean Details Using the HTML Adapter: The figure shows the sample Web page containing jmx-html output for the Cluster component.

The jmx-html adapter provided by the servletsuite.com group has a simple and easy to navigate interface for displaying MBean attribute details (see Figure 4). You can use this console to view the cluster MBeans details.

Web Application Setup
You need to have the JAR files in the classpath to get the sample Web application to compile and run without any problems. Table 4 shows the list of JAR files required to run the Web application and their locations.

Table 4. JMX Web Application Setup?Required JAR Files.

JAR File API Directory
commons-logging-api.jar Common Logging TOMCAT_HOMEin
jmx.jar Core JMX, MX4J TOMCAT_HOMEin
servlet-api.jar Servlet API TOMCAT_HOMEcommonlib
catalina.jar Core Tomcat classes TOMCAT_HOMEserverlib
commons-digester.jar Commons Digester TOMCAT_HOMEserverlib
catalina-cluster.jar Tomcat Cluster API TOMCAT_HOME webappsjmxappWEB-INFlib
commons-collections-3.1.jar Commons Collections TOMCAT_HOME webappsjmxappWEB-INFlib
commons-modeler.jar Commons Modeler TOMCAT_HOME webappsjmxappWEB-INFlib
log4j-1.2.8.jar Log4J Logging TOMCAT_HOME webappsjmxappWEB-INFlib

The Eclipse project file (jmxapp) provided with the downloadable source code has all these JAR files in the project’s build path.

This article concentrated mainly on accessing the cluster elements from a monitoring standpoint. But you can also use JMX technology to monitor and manage other areas, such as logging, connection pooling, object caching, and application configuration.

Although the sample application currently doesn’t provide the ability to update the attributes of a particular MBean component, look for that feature when the Tomcat 5 container becomes more JMX-based as the later versions are released. For example, you might want to do such things as provide centralized statistics from all cluster members so the administrator can decide if more servers need to be added to the cluster, or if a particular server instance can be brought down for maintenance or system upgrades, etc., without impacting the server availability.

Finally, although I didn’t cover it here, another important piece of JMX is notifications. Notifications allow for the transmission of events?anything from changing an MBean’s attribute to the registration of a new MBean in the MBean server. You can use notifications to broadcast any changes in a cluster configuration and perform the required updates in all the cluster members.

Share the Post:
Share on facebook
Share on twitter
Share on linkedin

More From DevX