Diagnosing the Problem
Because it uses a different thread pool for ASP and non-ASP requests, IIS can scale to huge numbers of requests for static content with little impact on its ability to process requests for ASP pages. However, that also means that where a majority of the content consists of ASP pages, you can exhaust the ASP request capacity even while you still have excess capacity to service other request types. In other words, requests for ASP pages can queue up (due to exhaustion of the ASP thread pool), causing longer response times for users, while static (non-ASP) content (provided via the ATQ) continues to be serviced as normal. Other common symptoms of this condition include a non-responsive or slowly responding Web server with low CPU utilization. The key is that the limitation isn't caused by CPU, network, or memory capacity, but rather by the limitation in the number of threads available to handle the Web requests.

The Windows performance monitor can help to visualize the problem. To know for sure if you are having trouble with running out of ASP threads, open PerfMon and add two performance counters: Active Server Pages/Requests Executing and Active Server Pages/Requests Queued. In a healthy environment, the value of Requests Executing will stay well below the maximum allowed on the machine. The maximum number of Requests Executing is equal to the number of processors times the value of the IIS metabase property ASPProcessorThreadMax. Therefore, if you have a dual processor Web server with the default value of twenty-five for ASPProcessorThreadMax, the maximum number of simultaneous requests will be 25 x 2, or fifty. Therefore, you should strive to keep the number of simultaneously executing requests well below fifty to maintain a healthy ASP server environment.

Likewise, Requests Queued should never rise above zero. When this value rises above zero, the pool of ASP threads allocated to processing ASP requests has been saturated—new requests that are received when the server is in this state will be placed in a first-in-first-out (FIFO) queue to be processed as ASP threads become available.

In Figure 1, notice that the Requests Executing counter is almost flat-lined out at a value of approximately fifty. Because requests continue to arrive while the server is in this state, the Requests Queued value rises at a rate equal to the difference between the rate of arrival of new requests and the message processing rate of the server. In such a situation, unless something occurs to dramatically lower the response times of the ASP requests, the server will have little chance of recovering.

Figure 1. Performance Monitor Counters: The figure shows the Requests Executing counter essentially "flat-lined" at 50, causing IIS to queue increasing numbers of new requests (Requests Queued).

The Solution
Most of the fixes for this problem cannot be implemented quickly, especially in a "live" context where you find your server suffering from these symptoms. The quickest, albeit most dangerous, solution is to simply modify the value of the ASPProcessorThreadMax metabase property. Using the adsutil.vbs script available in the inetpub\adminScripts folder, you can modify that value easily—note that you'll need to restart the IIS service for the new setting to take effect. If the server capacity is being only slightly outpaced by the rate of incoming transactions, it might be possible to raise the value of ASPProcessorThreadMax enough to offset the IIS throughput shortfall. This is a reasonable short-term solution, but others should be considered that will provide the ability to scale up by several orders of magnitude or more.

Two other solutions involve throwing hardware at the solution. By adding additional Web servers or additional processors, you can achieve near-linear growth in capacity. The benefits of this solution are that it will likely require no fundamental change to application architecture of the solution and that it can be implemented relatively rapidly—the only real cost is money. However, as growth of this nature is generally a good thing for business, the solution can probably be easily cost justified.

The solutions discussed so far may get you into the All-Star game batting lineup at the end of the season, but to get into the Hall of Fame requires an architectural paradigm shift. The key is to decouple the request from the response in a way that does not tie up valuable server resources while the long running resource is being accessed. Just as early operating system designers found that multitasking let processors execute other processes while one process was I/O-bound, the idea here is to use the ASP thread only long enough to queue a request for the long running resource and then disconnect—you can come back later to get the response from the resource.

To disconnect the request and response requires three phases of execution. First, the Web client posts data to an ASP script that queues a request for the resource. Second, the client periodically polls the server for the response. Lastly, when the client finds that the response has arrived, open an ASP page that will display the response from the resource.

This is quite a shift from conventional thinking for Web page authors. First, it requires an identifier to identify individual requests. This identifier must uniquely identify every request, even requests executing on other servers, and it must be generated before sending the request to the resource—it will be used as the voucher used by the client when polling for a specific response from the resource. A GUID (Globally Unique Identifier) is a good choice for this identifier as you can safely generate GUIDs in staggering number across any number of servers. Microsoft guarantees that you will never generate a duplicate GUID—if you meet several easily met conditions.

The system also requires a way to queue requests for the resource. The queuing mechanism holds requests when the ASP script that sends the request for the resource completes execution. A good (and free) choice is Microsoft's Message Queuing Server (MSMQ), which ships with all versions of Windows 2000 and higher (it was also available for Windows NT 4.0 via the Option Pack). MSMQ is both easy to configure and very simple to use.

The last requirement for the system is a means of holding the response from the resource until the Web client comes looking for it again. You could also use MSMQ for this purpose, but SQL Server (or any other database) is a much better choice. MSMQ is optimized for providing prioritized FIFO messaging, but is not optimized for providing random access to messages in a queue. The key design goal for storing responses is to optimize the retrieval of those responses from the system. An indexed database table provides this easily.

With all the requirements in place, it's time to walk through a sample data flow under the proposed model. The first step is to provide an ASP page that generates a GUID and, using MSMQ, submits the request for the long-running resource (attaching the GUID to the request). Finally, the ASP page redirects the client to an intermediate ASP page that periodically checks for the response each time it refreshes. Next, you need an application that picks up the requests from MSMQ, and satisfies the requests against the resource. When the application receives a response from the resource, it inserts the response into a SQL Server table, indexed by the previously generated GUID. The next time the ASP page executes after the response has been written to the SQL table, it redirects the browser to a Web page that displays the response to the user. Long-winded as the explanation may be, it is really is quite a simple process.