Audience Dialogue

Know Your Audience: chapter 15, part C
Using the Internet for qualitative research

Which types of qualitative research can be done on the Internet? As qualitative research is based on words instead of numbers, it uses open-ended questions (in situations with an interviewer and respondent), or discussions (when multiple respondents are present).

There are two main ways of doing qualitative research on the internet: synchronous and asynchronous. Synchronous means "all at the same time" - this is the internet research equivalent of a group discussion, or live programming. Asynchronous software (email, for example) lets people respond whenever it is convenient for them. Both methods have their strengths.

The main use of synchronous software is with chat groups.

Asynchronous software comes in a wide variety of types, including listservs, threaded discussions, groupware, and forms with open-ended text areas.

Chat groups

An internet chat group is a group of people who "meet" online at a particular internet location, at an agreed time. Using special software, they "chat" by typing in statements at their keyboards, and sending these statements to the host computer. As soon as each statement is received, it is transmitted to everybody else taking part in the group. An online moderator may ask participants to discuss a particular issue, and later study a transcript of the entire discussion. In other words, it's very like a normal group discussion, except that everything is in writing.

If you'd like to try this, there are some web sites where you can set up a chat group. It will cost you nothing, but they will display
advertisements. Some examples are
Parachat ( www.parachat.com )
Anexa ( www.anexa.com )
Talk City ( www.talkcity.com )

I tried using Parachat for qualitative research, and found that it worked well when:

If you expect your sample to be people who can type well, and you are studying something that doesn't attract emotional reactions, possibly chat groups may be successful - but also consider the other methods listed below.

How well do these online focus groups work? There are conflicting reports. A study by the U.S. market research company NFO comparing online focus groups with normal focus groups found that both methods produced much the same results.

Other researchers have found that the "conversation" is stilted and doesn't flow well, and that most of the communication is between the online moderator and individual participants. I suspect this lack of "conversation" occurs because most people are slow typists. If they are worried about making mistakes in spelling or grammar, they may not concentrate on the subject being discussed. Perhaps in a few years' time, if most people are using microphones instead of keyboards, these chat groups will be more productive for research.

Listservs

A listserv is a program that sends email to a group of people. All you need is a list of email addresses, and some listserv software. Any message sent through the listserv email address will automatically be sent to everybody on the list. Anybody can reply, either to everybody else (with a single command) or to the organizer.

Some providers of free listservs are:
Yahoo groups ( groups.yahoo.com )
Topica ( www.topica.com )
Tight Circle ( www.tightcircle.com ) [not working, June 2006]

The reason these programs are free is that they include advertising - but it's usually unobtrusive. If you don't want advertising, you'll need to pay for the service. Some low-cost services (a few cents per message) are
Sparklist ( www.sparklist.com )
Lyris ( lyris.net ).

Many ISPs also have such software available for their clients to use, often at no extra charge. Majordomo is a widely used program of this type, but it's not very user-friendly.

The listserv method is normally used for newsletters: when a list is built up, members are sent messages over and over again. It can be used for a one-off survey, but I don't recommend this. In effect, it's an email survey - and (as explained above) there are many problems with email surveys. Another disadvantage of using a listserv to send a questionnaire is that a lot of people inadvertently send their reply to the whole list (i.e. everybody else) not just the organizer - and naturally they are upset when their privacy is lost.

The listserv is best used for participatory interactive research - where it's actually an advantage for people to see each other's responses. The participants can respond to each other, so in this way listserv research is similar to a group discussion. But a group discussion is limited to a period of a few hours, and participants must be physically present, With a listserv, participants can be anywhere, and can respond at any time. Usually a listserv discussion will run for up to a week. Though it takes longer to get the data, participants have a lot more time to think about their answers - and the answers are often very thoughtful and well-considered.

Like a mail survey, listserv research doesn't work with some populations, but if the participants are reasonably literate, and motivated to take part in the discussion, results can be very useful.

Groupware

This is a term for a whole class of software that a group of people can use to communicate between themselves, either simultaneously or at short intervals. It could include bulletin boards, guestbooks, forums (newsgroups), threaded discussions, ICQ, and conferencing software designed for either businesses or education. Even the humble listserv could be considered groupware. I'll discuss several of these types of software, which could be useful in research.

Threaded discussions

These were first developed in internet Newsgroups, a form of hypertext which existed long before the Worldwide Web did. The term newsgroups is misleading: these are not about news in the journalistic sense, and nor are they fixed groups. The recent term forums is a better description: they are discussions on a particular theme, or thread. At any time, a thread can branch into several new threads. Each thread is made up of a number of contributions from readers. One contribution is usually a single paragraph.

The contents page of a threaded discussion looks like an outline, usually with a single-line summary of each comment, showing the topic and the name of the commentator. When a line is clicked on, the whole comment will be displayed, with links to the previous and following comments. Any reader can respond either to a specific comment or a group of comments.

Here's an example of a summary tree, with varying indentation representing a hierarchy of messages. Each underlined item is the title of a message, with its size shown to the left (e.g. 1K = 1,000 characters - about one screenful). Clicking on the underlined item will display the full message.


09 Feb 01 14:50 2K Today's meeting - Bertha

09 Feb 01 15:05 1K    Re: Today's meeting - Anna

09 Feb 01 15:53 1K    Re: Today's meeting - Vovster

10 Feb 01 08:45 1K Yesterday's meeting - Bertha

10 Feb 01 09:02 1K    Re: Yesterday's meeting - Joe

10 Feb 01 10:30 1K       Re: Re: Yesterday's meeting - Iki

The technology of threaded discussions is now widely used in a variety of software types, including bulletin boards and business and educational groupware. A wide variety of software is available, but a lot of it is not easy to use. A common problem is that you can't read more than one note at a time: clicking on one item in a tree such as the above will display the comment - but to see the replies you need to go back to the tree and click each one individually. This makes it difficult to follow the discussion.

Some of the best threaded-discussion software packages are:
Discus ( www.discusware.com )
Webboard ( webboard.oreilly.com )
Wimba ( www.wimba.com ) - the only one (yet) based on spoken messages - so users need speakers and microphones on their computers.
Donald Woolley's website Forum software for the web is a useful and comprehensive review of all types of forum software.

Prices vary greatly: for example, the simpler version of Discus is free (and works very well), Wimba is initially free, while the more elaborate versions of Webboard cost thousands of dollars.

The threaded discussion system works well for answering specific questions: for example, in newsgroups, discussing problems with software. But the sheer number of threads makes it difficult to read a whole set of contributions. If you want a lot of people to participate in a discussion, the threading system makes this difficult, because participants only follow the threads they are most interested in. Threaded discussions are therefore best suited to a large number of participants - perhaps 1000 or more, mostly making very occasional contributions. Qualitative research has traditionally used much smaller samples, with a lot of words from each participant - but the threaded discussion model is quite different. A researcher would want to know how many people had read each comment. Though this can be found out from site logs (if each comment is a separate web page), it would be good if threaded-discussion software provided this information. I don't know of any that does.

Guestbooks

A guestbook is the computer equivalent of the guestbook that you find in some old-style hotels: visitors are invited to sign the guestbook and write in a comment. Most guestbooks found on the Web are trashy, or filled with thinly veiled advertising, but it's possible to use this format for qualitative research - for example, to get visitors' reactions to a web site. Instead of the questions you usually see ("Who is your favourite pop star?" and so on) you could ask questions such as "Of the pages on this site that you've seen, which do you like most, and why?"

One of the main attractions of a guestbook is that everybody can see everybody else's responses - and even comment on them. From a research point of view, this could be either a strength or a weakness: a guestbook more resembles a group discussion than an individual interview, in that the respondents will influence one another.

The weakness of a normal guestbook, from a research point of view, is the sampling. If only a small proportion of site visitors fill in the guestbook, the comments can't be taken as representative.

Many ISPs provide guestbook software that their clients can use on their own web sites. If yours doesn't, the Web has numerous sites offering free and low-cost guestbook software. Some of the most popular providers of this software are
Bravenet ( www.bravenet.com )
Guestbook Star from Webgenie ( www.webgenie.com )
Site Gadgets ( www.sitegadgets.com ).

Other groupware

Two major types of users of groupware are large organizations and universities. The large organizations use it to improve communication between their geographically-dispersed managers, and universities are beginning to use it for tutorials, in online distance learning.

The most widely used software among organizations is Lotus Notes, which is very powerful, very expensive, and doesn't seem particularly well suited to qualitative research - partly because all participants need a copy of it. If you are going to survey your audience on a casual basis, they can hardly be expected to install a copy of Lotus Notes (or similar software) on their computers to join in your research - even if you pay for it. Even if you set up a regular panel, and help them install the software, they still must learn to use it. It could be a lot of work for a small return.

Another class of groupware programs is those used for online learning. Many universities are now using these, to run the online equivalent of tutorials. Examples of such programs are Commonspace, Aspects, WebCT, Courseinfo, and Daedalus. These programs combine threaded discussions with other facilities and may be usable for qualitative research, though they may be too teaching-oriented. Commonspace has a particularly clear layout, with the main text in one column, and comments in other columns to the right. Unfortunately it's not yet available on the Web.

A German program called Zeno, described as group mediation software, includes facilities for rating and voting, and looks as if it might be excellent for qualitative research. Details can be found on the Zeno website, zeno.fhg.de

For groupware to be successful in audience research, it must

Advantages and disadvantages of the Internet compared with face-to-face discussions

It often happens in group discussions that a participant wants to say something, in response to a comment somebody has just made. If another participant gets in first, the first would-be commentator often gives up; the appropriate moment passes, or he or she forgets. A big advantage of online discussions, if they run over an extended time, is that it's always possible to comment on what somebody else has said - e.g. with a threaded discussion.

Normal group discussions can be fairly expensive. Because participants have to come to the same place at the same time, and many people are not very willing to do this, participants are usually paid to attend - at a higher rate than interviewers are paid. Also, only about one person in every three eligible will end up attending a group discussion. This is in effect a low response rate, and it makes you wonder what opinions the non-attending majority might have.

Both of these problems are solved with online groups - if you're dealing with a population who already have internet access. Participants don't need to be paid (or not much), and for many people it's much easier to "attend" a group at home than to find a research venue they may have never seen before.

If an online group continues over several days, participants have more time to consider their responses, and to think of points they might miss in a live group.

Online groups have one major disadvantage. Most of the emotional and sub-verbal content of the discussion is lost. You don't see the expressions on people's faces, or hear their tone of voice, and they can't look at each other. If there's a conflict between atavistic emotion and social desirability, a normal group discussion will balance the two, but an online discussion is likely to produce the socially desirable response. An example of this would be a discussion of whether a particular time slot on a TV channel should have a news program or entertainment. I suspect an online group would conclude that news was the most desirable, even though diary or meter data would show that entertainment had a much larger audience.

Another problem that you sometimes find with email is that somebody might write a comment whose meaning isn't quite clear. Perhaps it's a joke, perhaps not. But somebody else might be very offended by it, then a lot of others will join in. In all the attacks and counter-attacks, the research is forgotten.

You can partly overcome this by encouraging people to type emoticons or smileys: keyboard-created pictures such as a smile :-) or a frown :-( or a wink ;-| or a shout :-0

There are hundreds of these (including such gems as :-(=<q "person not happy at giving birth to a squirrel") but it's best to keep to a few that most participants can remember.

Another method of expressing emotions is to ask people to type them in, like stage directions in a play, such as [feeling bored] or [disagreeing] or [happy]. These can actually be more direct than actions in a normal group setting; an online moderator can encourage participants to write their actual feelings, not just the way they'd respond in real life - for example [agreeing] instead of [clapping].

As well as encouraging people to express their emotions clearly, the research organizers can ask participants to always pause for a moment before pressing the Send button, and consider the effect they might have on readers. If they are feeling a strong while writing a message, it's often best to leave it a few hours, then edit it again before sending it.

Forms with open-ended questions

One of the easiest ways of doing qualitative research on the Internet is to adapt a normal web survey. HTML forms include two ways of getting open-ended responses: the Text input (one line), and the Text Area (a multiple-line box, with scroll bars).

Unlike the other qualitative research methods described above, this is not a group-based method. Normally, only the survey organizers will read these comments, not other respondents.

You might expect that, because of problems with typing, people wouldn't give much detail in open-ended questions. However, this is not what I've found; I've been pleasantly surprised at the coherence and detail in open-ended answers. Even though I was surveying populations with above-average education, I've used the same populations in mail and telephone surveys, and found that the web surveys reflected no problems relating to typing.

My tentative conclusion is that if the population is right (i.e. above average education), and the open-ended questions are well worded (neither too specific, nor too vague), and the subject of the survey is of interest to the sample, HTML forms work very well.

However I should add that other people have tried this method, and found that it doesn't work well at all - judged by the number and length of responses, and the care that has been taken in writing them.