A series of tests
were performed in order to identify scaling characteristics. The tests were
run for 10 min with a warm-up and cool down period of 30s. The test period
was long enough for the setup to stabilize and consistent reading were seen
during all tests.
There were also
a number of endurance tests performed where the application was run for
more than 12 hours to show that there weren’t any signs of performance
degradation over time. These tests were run at a consistently high load,
approximately half of the maximum limit.
user had a 10 second think time once a request had been fully received.
This means that each user performs one request every 10s + TTLB, where TTLB
is the time it takes from the request has been made to all the data has
been received. With 1000 concurrent users you would have slightly less than
The database was reset before each test-run to contain the same
products, with totally 50 000 items
• 100,000 users
• 10,000 orders
and Purchase Test
During the “Browse and Purchase” test, two scripts
where run simultaneously on separate ACT clients to simulate a near 50/50
mix of browsing and purchasing. A number of operations were performed to
simulate a visitor browsing the website.
1 Go to the site
2 Perform three (3) free text searches on 1000 products
3 Examine the products in three (3) different categories, clicking the next
button three (3) times
4 Examine the details of three (3) different product items
number of operations were performed to simulate a visitor who is making
1 Perform two
(2) free text searches on 1000 products
2 Sign in using a random UserID from one of 100 000 available customers
3 Add two (2) random items, from 50 000, to the shopping cart
4 Checkout, verify account information and place the order
5 The user signs out
6 Perform one last free text search
A single web service test was performed were the clients used HTTP
GET call with a supplied OrderID, whereby the web service returned that
order including customer and delivery information. The following information
• Order Items
• Credit Card Information
• Shipping Address
• Billing Address
• Customer Contact Information
Finally, a test was performed using distributed transactions. The
following actions were completed:
1 Sign in using
a random UserID from one of 100 000 available customers
2 Add a random item from 50 000 to the cart, order, verify account information
and check out. Do this totally ten (10) times
3 Sign out
the distributed transaction test, NLB was disabled because it was interfering
with the DTC. Each client was assigned a specific server to make calls to.
The maximum number
of pages served when running flat out with four application servers in the
web farm was 66 million pages per day (24 billion pages per year) with sub
second page response times. If only 1 in 1000 of those page visits lead
to an actual purchase of lets say 20 USD, this e-commerce company would
be selling virtual pets with an annual turnover of 482 MUSD. Without discussing
how reasonable this market forecast is, we can conclude that it would have
been a major operation and there is still a lot of room to scale up the
database tier and scale out the web farm.
A note on the
diagrams. Different diagrams can have different scales. Also note that req/s
are measured on the left vertical axis, whereas response times (TTLB) are
measured on the right vertical axis.
thing to note is that we have used processor utilisation as a measure of
how hard a server is working. This is of course not the only limiting factor,
but we found it to be a pretty practical metric that is easy to understand.
Rather than trying to weigh in all the components of the server to create
a more accurate (and more cryptic) custom index, we trust your common sense
to draw the right conclusions regarding this.
The first test involves a simple Browse and Purchase experience.
This is the most common activity on an e-commerce site. A snappy browsing
experience is important in order to give the user a positive relationship
with the web shop. A browser request shouldn’t take more than a second
to fulfil, including the network roundtrip.
Test with one
(1) application server: The diagram in figure 8 (immediately below) tells
us how the average page response time, measured as Time To Last Byte (TTLB)
received by the calling client, decreases as the number of concurrent users
increase. The average processor utilisation increases from 15% to 89%, beyond
this point, response times will drop due to exhausting the processors. The
response times are fantastic, even with 2000 concurrent users each page
is delivered in less than 100ms leaving 900ms for network roundtrip and
to load graphics until a full second has passed. This allows very fast browsing.
The test also shows that the amount of available memory on the application
server only decreases from 648 Mb to 610 Mb. We could have done with a lot
less memory, and there is plenty of room if the application grows in complexity.
Test with four
(4) application servers: The diagram in figure 9 (below) shows the same
test running on 4 Network Load Balanced (NLB) application servers. Even
running flat out with 8000 concurrent users making 771 req/s (the equivalent
of 66 million requests per day), the response time is as low as 540ms leaving
plenty of room for network roundtrip. At this point the application servers
are running with 91% processor utilisation and the database server at 72%
processor utilisation. The strain on the network is high at this point and
the workstations in the client simulation tier were getting a small number
of network socket errors (58 in all, equivalent to 0.01%).
The average network
bandwidth used by the client/application server communication during this
test amounted to 11.6 Mb/s, which is slightly less than half of the maximum
(25 Mb/s) possible with NLB on a single 1 Gigabit network.
note is that the shared state server only utilised 14.8% processing power
when running 8000 concurrent users with 4 application servers in the web
farm (figure 11). This indicates that using a shared session server won’t
be a bottleneck when scaling the application further.
Looking at the
diagram in figure 10 emphasises our findings. The number of pages served
scales linearly when adding new servers to the web farm until the database
server becomes the bottleneck. We used a four-way 2GHz Xeon server, when
increasing the load further you will want to scale up the database server.
Recent TPC tests running SQL Server on 32-way Itanium servers indicate that
you will have plenty of room to grow…
As you can see
in the diagram of figure 12, response times can be improved by adding more
servers to the web farm when running with a constant load (in this case
2000 concurrent users). This is good news, proving that adding servers to
the web farm can be used to improve response times, given that the servers
in the web farm are the bottleneck.
The web service
test mainly puts pressure on the database. With 1000 concurrent users, the
database was running at 99.9% processor utilisation with the single application
server only at 39.4%. This makes a lot of sense since we performed few and
uncomplicated business rules, basically only gathering and presenting data
(see the test description in section 8).
Note that the
response times increased significantly when we passed 800 concurrent users.
At this point it would be advisable to scale up the database server. This
is further emphasised by the tests with 4 application servers, which shows
more or less the same curve.
The last test that was performed was a distributed transaction test. During
the test, which ran for 12 hours on three dual processor PIII application
servers with Windows 2003, a total of 310k distributed transactions where
committed, none aborted. This corresponds to 433 transactions per minute.
The test wasn’t
designed to try the limits of the solution under stress; rather, it was
performed to show how the application responded to extended use under normal
pressure. There was no degradation of performance over time.
|Number of Transactions