These are the three types of standard work we currently do for measuring website effectiveness. We are developing a fourth method: remote testing of websites, which we hope will be available early in 2005.
This is a study of existing and/or potential users of a website (or other service). As long as the sample is carefully stratified, and the scope of the study is carefully defined, quite a small sample is enough to provide useful and representative information. For this, we use a technique of cognitive interviewing, to produce mental maps of the website (or other service) being researched. The output is a detailed report, showing the different ways in which users see the service. Indicative cost: 2 000 USD (if no travel is required). Minimum time: about 1 week.
For this, we use the co-discovery method developed initially by Philips Electrical for usability studies of consumer products. We have found that it works very well with website and document usability studies, as well as products. For a website, this involves developing a search exercise. Pairs of people sit in front of a computer, and are asked to find 10 items of information that exist somewhere on the website being studied. For most websites, a sample of 12 (6 pairs) is sufficient. We measure the time they take to do this, the success rate, and their level of satisfaction (or frustration) with the site. We also record their comments, which are useful in troubleshooting difficult navigation problems. Indicative cost: 5 000 USD. Minimum time: about 2 weeks.
The consensus group method is similar to focus groups, but with more interactive input from users. The method was developed by our principal, Dennis List, for estimating the likely take-up of new services and products. This method involves having two or three groups of 10-12 people meet for several hours, interacting with and discussing stimulus materials (e.g. web pages, documents, prototypes), discussing them, and coming to some conclusions about them. Indicative cost: 5 000 USD. Minimum time: about 2 weeks.
We can do a wide range of larger customized studies, but our advice is that a succession of small studies usually produces better data than one large study.
For all of these types of research, we normally produce detailed reports, usually 20-30 pages, and a small-group presentation. On request, we can produce a wide range of other reporting formats, including Powerpoint shows, web-based reports, workshops, and so on.
And for all of them, the "value triangle" applies: in other words, the work can be low-cost, high-quality, and/or done very quickly - pick any two. If you try to achieve all three at once, you will find that it's quality that will suffer, since itıs the least measurable - as NASA found when it adopted its FBC ("faster, better, cheaper") policy a few years ago, and the space shuttle crashed. However, if you have no information at all about your users, a competently-done small study can be very informative indeed - as long as the sample is carefully selected.