henever you have to fix a problem in a production environment, one of the most common issues is retrieving some useful debug information to tell you what happened. Of course, to avoid needless notifications and enormous log files, most production systems are configured to log only errors. To avoid having to modify your Log4J configuration and then redeploy the correct version of the code with the modified logging level, why not just implement a mechanism for dynamically changing the logging level?

Log4J registers a number of MBeans, one of which provides access to the root logger. You can dynamically modify the logging level of the root logger by implementing the following servlet to set the appropriate MBean property through JMX:

public class SetLog4jLevelServlet extends HttpServlet {    private static final Log log = LogFactory.getLog(SetLog4jLevelServlet.class);    protected void doGet(HttpServletRequest request,             HttpServletResponse response)             throws ServletException, IOException {        List list = MBeanServerFactory.findMBeanServer(null);        MBeanServer server  = (MBeanServer)list.iterator().next();        try {            String loggingLevel = request.getParameter("level");            Attribute attribute = new Attribute("priority", loggingLevel);            ObjectName objectName = new ObjectName("log4j:logger=root");            server.setAttribute(objectName, attribute);        } catch (Exception e) {            log.error("log4j:logger=root", e);        }    }}

The first two lines of the doGet method look up the MBeanServer. Log4J registers the root logger with the name log4j:logger=root. This MBean provides a property to set the logging level of the root logger. You set the priority of the root logger by passing the ObjectName of the MBean to operate on, along with an instance of javax.management.Attribute to the MBean Server. In this case, you want to set the “priority” attribute. The value will be the new logging level you have specified.

A Log4J MBean for Each Log4J Configuration

Although this technique seems simple, if your Tomcat cluster hosts a number of Web applications?each with its own Log4J configuration that allows it to log to a separate log file?then you will run into your first problem. Given multiple Log4J configurations, only the first one loaded by the JVM will have a root logger MBean associated with it. The reason for this is Log4J hard-codes the JMX Domain that it uses when registering its MBeans as log4j, and the JMX server doesn’t allow duplicate names.

The way round this is to leave a Tomcat server Log4J configuration (jars under server/lib and property files under server/classes) registered under the default Log4J name log4j:logger=root. Then register the root logger for each Web application in a startup servlet, using the Web application name to identify the domain:

public class RegisterLog4JMBeanServlet extends HttpServlet {    private static final Log log = LogFactory.getLog(RegisterLog4JMBeanServlet.class);    public void init(ServletConfig servletConfig)             throws ServletException {        ServletContext context = servletConfig.getServletContext();        String webappName = context.getServletContextName();        Hashtable nameParameters = new Hashtable();        nameParameters.put("logger", "root");        Logger rootLogger = LogManager.getRootLogger();        LoggerDynamicMBean mbean = new LoggerDynamicMBean(rootLogger);        List list = MBeanServerFactory.findMBeanServer(null);        MBeanServer server  = (MBeanServer)list.iterator().next();        ObjectName objectName = null;        try {            objectName = new ObjectName(webappName, nameParameters);            server.registerMBean(mbean, objectName);        } catch (Exception e) {            log.error("Problems registering Log4J MBean", e);        }    }}

You will end up with an MBean named <web-app-name>: logger=root for each Web application. The webappName is retrieved from the <display-name> element in your application’s web.xml file, and the application’s root logger is retrieved by calling LogManager.getRootLogger(). Then you can use the LoggerDynamicMBean class from Log4J to create your MBean from the root logger definition.

To create a distinct ObjectName for the new MBean, you should use the Web application name as the JMX domain (this is the text before the colon) and then register the Root Logger MBean with the new ObjectName. This will guard against conflicts with other Log4J MBeans.

Now that you have registered your Root Logger under a different domain, you must modify the servlet that sets the logging level. So in SetLog4JlevelServlet, change this:

ObjectName objectName = new ObjectName("log4j:logger=root");

To this:

ServletContext context = servletConfig.getServletContext();String webappName = context.getServletContextName();Hashtable nameParameters = new Hashtable();nameParameters.put("logger", "root");ObjectName objectName = new ObjectName(webappName, nameParameters);

When setting up your Log4J configuration, give your Log4J appenders unique names. Otherwise, they will not get registered to the JMX server and you will get lots of the following errors in your log files at startup:

ERROR main org.apache.log4j.jmx.LoggerDynamicMBean ? - Could not add appenderMBean for [<appender_name>].javax.management.InstanceAlreadyExistsException: log4j:appender=<appender_name>

Again, the reason for this is because the JMX domain name his hard-coded to log4j, so if you have repeated appender names then only the first of these will be registered.

At this point, configuring the MX4J HTTP adaptor in your Tomcat server might be useful. This will give you a visual representation of the MBeans you are creating and manipulating, and show you what else is exposed via MBeans within Tomcat. To do this, you must put the mx4j-tools.jar file (download it at mx4j.sourceforge.net) in your common/lib directory. Then configure your server.xml file to set up the connector as follows:

When you start your server, you will be able to access the MX4J HTTP connector through your browser at http://localhost:8013, assuming you have no other Tomcat connector running on port 8010.

Set the Logging Level for All the Nodes in Your Cluster

Once you have made the previously discussed changes, you will be able to set the Log4J logging level dynamically for any application running in Tomcat. The next challenge is to make sure you can set the logging level for every node in the cluster. You could either call this servlet manually on each node in the cluster or get the servlet to send a message to every other node in the cluster so the logging levels are always the same across the cluster.

To send a message to the other nodes in a Tomcat cluster, you need to call the send(ClusterMessage) method on the org.apache.catalina.cluster.SimpleTcpCluster class. When implementing a ClusterMessage, you must populate the details of the cluster node that was responsible for sending the request. (Listing 1 provides an example that retrieves the information through the Tomcat MBeans.) The first step is to provide an implementation of the ClusterMessage interface to pass the new logging level and to determine for which application you want to set the logging level.

It may seem strange, but you must use JMX over RMI to send the message to the cluster because the interface implemented in Listing 1 is defined within the Tomcat server class loader (i.e., it exists in a jar file in the server directory). You will instantiate that class in a Web application, which uses a totally separate class loader (it exists in the WEB-INF directory of your Web application). This means that the cluster message you instantiate in your Web application and send to the SimpleTcpCluster class will be different from the cluster message definition that is placed in the Tomcat server class loader. If you try to send your message directly through JMX, your cluster message instance will not match the cluster message definition accessible to the SimpleTcpCluster class and the call will fail. (See the Tomcat documentation on class loaders for a definition of how the class loaders are configured for the Tomcat server.)

The workaround is to use JMX over RMI so the cluster message is serialized in the Web application class loader, instantiated in the Tomcat server class loader (and that is the important point), and then de-serialized. The following method opens a connection to the local JVM’s JMX server over RMI, looks up the MBean that wraps the SimpleTcpCluster, and then invokes the send(ClusterMessage) operation on the MBean:

public static void sendLogMessageEvent(String applicationName, String logLevel) {    String port = System.getProperty("com.sun.management.jmxremote.port");    String urlForJMX = JMX_SERVICE_PREFIX + HOST + ":" + port + JMX_SERVICE_SUFFIX;    try {        JMXServiceURL url = new JMXServiceURL(urlForJMX);        JMXConnector connector = JMXConnectorFactory.connect(url, null);        MBeanServerConnection server = connector.getMBeanServerConnection();        ObjectName cluster = new ObjectName("Catalina:type=Cluster,host=localhost");        log.debug("Cluster cluster: " + cluster.toString());        LogLevelMessage message = new LogLevelMessage();        message.setLoggingLevel(logLevel);        message.setApplication(applicationName);        Object[] params = new Object[] {message};        String[] types = new String[] {"org.apache.catalina.cluster.ClusterMessage"};        server.invoke(cluster, "send", params, types);    } catch (Exception e) {        log.error(e.getMessage(), e);    }}

You should call this method in your SetLog4JLevelServlet to ensure that a message is sent to the other members of the cluster when you set the logging level.

You will need to configure your Tomcat server to allow a JMX connection to be opened up over RMI. Simply add the following line to your catalina.bat file in the bin directory:

set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote.port=8012 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

For a quick guide on applying security to this configuration, see the Tomcat documentation. For a more in-depth look at configuring JMX Remote, see the Sun documentation.

Receiving the Message in the Other Nodes

Once you are able to send these messages to the other nodes in the cluster, you will need to implement a Tomcat ClusterListener to receive the events. This implementation will set the logging level for the correct MBean determined by the application name that is passed as part of the message:

public class ClusterLoggingListener extends ClusterListener {    private static final Log log = LogFactory.getLog(ClusterLoggingListener.class);        public void messageReceived(ClusterMessage message) {        if(message instanceof LogLevelMessage) {            LogLevelMessage logMessage = (LogLevelMessage) message;            List list = MBeanServerFactory.findMBeanServer(null);            MBeanServer server  = (MBeanServer)list.iterator().next();            Hashtable nameParameters = new Hashtable();            nameParameters.put("logger", "root");            try {                ObjectName logBean = new ObjectName(logMessage.getApplication(), nameParameters);                Attribute attribute = new Attribute("priority", logMessage.getLoggingLevel());                server.setAttribute(logBean, attribute);            } catch (Exception e) {                log.error("Problem setting the logging level for application " +                         logMessage.getApplication() + " to level " +                         logMessage.getLoggingLevel(),e);            }        }    }    public boolean accept(ClusterMessage message) {        return message instanceof LogLevelMessage;    }}

You must deploy this class in the server directory and enter it in the Tomcat server.xml file as a ClusterListener within the element:

You should now be able to set the logging level for any application, over all nodes, in the cluster by calling SetLog4JLevelServlet on any node in the cluster.

Deploying the Example Code

To deploy the example code for setting the logging level in your cluster, take the following steps:

1. Load the Eclipse project and build it using the dist target. This will create set-log-levels.war and set-log-levels-server.jar in the dist directory.
2. Put the set-log-levels-server.jar in the Tomcat serverlib directory and deploy the set-log-levels.war file across the cluster.
3. Use the SetLog4jLevelServlet (e.g., /setLogLevel?level=debug) to set the logging level and use PerformLogTestServlet (/testLogLevel) to test the logging level changes.