ack in 2000 while taking a walk in downtown Houston, I was approached by a couple of Astros baseball fans who asked me to join them in testing the plumbing system in the newly constructed Houston Astros Stadium. Inside the stadium, about 1,000 other Astros fans participated. All we had to do was flush the urinals at once. The team treated us to pizza and soda for taking part in this test?a very practical way to load/stress test the stadium but also expensive.
While the Astros team had no problem getting so many people to participate in their load test effort, testing your software the same way is unfathomable. It is not cost effective to do an exact production simulation load test on a pre-production or test environment, which has created a market for numerous software tools that perform simulated load tests. All the testing tools basically run automated tests, during which you can collect other vital operating system information to calculate your software’s performance capacity.
Because development teams seldom address load test issues during development, initial load test runs are prone to crashing. Lots of load testing time can be saved if problems such as open cursors and overly used CPU-intensive queries are addressed during the development phase itself. This article walks through how to analyze load test data and do capacity planning for your enterprise Java applications while they are in development.
If you leave database-related processing cursors such as statements and results sets open (without explicit closing instructions), they will eventually max out and throw the Max Open Cursors Exceeded exception. So while doing load tests, use this simple query to watch for open cursors:
select s.SQL_TEXT , count(*) from v$sql s, v$open_cursor ocwhere oc.hash_value = s.hash_valuegroup by s.SQL_TEXT having count(*) > 1
The query will tell you which query cursors are left open. If the cursor for a particular query is not closed, the count corresponding to the query will constantly increase. Once you’ve identified the query, it is easy to fix the code.
CPU and Memory-Intensive Queries
The V$sqlarea view has many useful columns for identifying CPU-intensive and memory-intensive queries. The CPU_TIME column specifies the accumulated microseconds of CPU time that your SQL uses for parsing, executing, and fetching. The RUNTIME_MEM column specifies the fixed amount of memory required during the execution of a cursor. Queries using clauses such as
ORDER BY tend to demand more CPU time and runtime memory. During low loads these queries do not have much of an effect, but as loads increase they can have a cumulative effect. While there is no workaround for these queries, if the architecture group knows about them beforehand, they can minimize such queries in your application.
Comment Out All
System.out statements put enormous load directly on the JVM, as well as indirectly on the underlying operating system. Too many
System outs increase CPU utilization exponentially. So comment out all the System printout statements prior to running load tests (as you will see later, capacity planning involves CPU utilization values). For example, the following sample code takes 30 seconds to run using 80 percent of CPU capacity:
for(int i = 0; i
However, the same code without the print statements takes only 9 milliseconds with less than 10 percent of CPU utilization:
for(int i = 0; i
|Figure 1. Load Averages Using Uptime
Data to Collect During Load Test
If you use *inx environments for your application server deployments, collecting the load on the OS is very important. All *inx systems use the /proc file system to provide an easy way to view kernel information and information about currently running processes. The Proc pseudo file system is a real-time, memory-resident file system that tracks the processes running on the machine and the state of the system. Process-specific information is stored within /proc under the process ID directory.
For load tests, pick the most time-consuming or computation-intensive API of your enterprise application or one iteration of all user activities on the system. Start the load test with 100 concurrent users and run it for at least 20 minutes. While the load test is running, use the
uptime command to periodically record the load averages on the OS.
Uptime gives the load on the system during the previous 1-, 5-, and 15-minute intervals. Load average specifies the number of processes waiting to compete for the CPU. Figure 1 gives the sample output of an
You do not want too many processes waiting in line for the CPU, but at the same time you do not want too few processes in the system. The optimum load will have values hovering between 2 and 3. The 'top' command in the Figure 2 sample output shows CPU utilization and a memory snapshot of the system.
|Figure 2. CPU and Memory Snapshot Using Top
As you proceed with load tests, keep gradually increasing the number of concurrent users, paying attention to the CPU utilization, memory stats, and load averages until the load averages reach the optimum values (between 2 and 3).
In J2EE applications, the max-http-connections configuration controls the maximum allowable HTTP connections. Though the theoretical value is infinite, after a certain number the response time will gradually increase for each request to the server. At this juncture, it is better to have another instance of the server. Usually the documentation of the application server provides these values for optimum performance. Multiple J2EE instances can be installed on the same server with different port numbers. A load balancer is needed if multiple instances of the application are used.
Once the load test is successful, analyzing the load test data and computing your software capacity is straight forward. Suppose you run the previous example load test and determine that two application server instances on a single server are able to handle 10,000 users each with optimum load averages on the OS (between 2 and 3). Say your server has 1 GHz processor and 1 GB of RAM, and the capacity of each server is 20,000 users. The capacity of your enterprise software is 20,000 concurrent users on a server with a 1 GHz processor with 1 GB of memory. If your application has to handle 100,000 users during peak loads, you would request five such servers, assuming unlimited availability of network bandwidth and database resources.
Knowing Your Software's Limits
Many major software vendors specify minimum OS requirements for installing their products, but how the software fares under different user loads is left for the clients to figure out. It is about time that software vendors provide load ratings for their products. The academic community is not doing any research to compute a software load-rating factor, but I have formulated my own definition: software load rating is the result of evaluating software response time in applications where the number of concurrent users on a system is the only measure of poor performance for a given server.
It is always good to compute the operating limits of your software. As the cost of memory and CPUs keeps going down, the load rating of your software keeps going up.