Running the Application
Our user tests were designed with two main goals in mind:
- Verify that the system performed well in real-life scenarios: The main goal is simply to verify that testers can manage the basic tasks that real customers would want to perform.
- Exercise the full feature-set of the application: In addition to testing standard goals, it was important to make sure that the complete feature set of the application was tested as well. Testers were guided to parts of the system that might not necessarily be on a most-likely-path scenario, in order to make sure that the entirety of the system worked as expected.
To accomplish these goals, we gave our testers scenarios
that included both common tasks and special-case scenarios designed to guide the user toward special situations. A sample script might look like this:
TASK ONE (Product-Number-Driven Ordering)
TASK TWO (Catalog-Based Ordering and Shopping Cart Review)
- You noticed a product in a magazine that is sold at the IBuySpy store. The product number was listed as 3-6-0. Purchase this product.
TASK THREE (Review Previous Orders)
- Needing more power to persuade, you want to buy the Persuasive Pencil, a product found in the Communications category. Please purchase one unit.
- Sustaining damage to your car on your last mission, you want to purchase the Universal Repair System, but only if it is safe for cars. Check the product description and if it is safe, purchase two.
- Deciding that it may also make sense to have a Persuasive Pencil for your vacation home, you decide it makes more sense to buy two pencils instead of one. Update your order.
- Finish the transaction and make a note of the order ID. End the call.
- With the order ID from the previous task, check to see if your order shipped.
- You've become confused about how many Persuasive Pencils you ordered. Check the order details to verify the quantity shipped.
Test subjects were given account numbers and PINs to log into their account, but otherwise were left alone to complete the tasks. Tests were repeated with a number of different test subjects and over a number of successive product revisions.