Audience Dialogue

Know Your Audience: chapter 15, part A
Internet audience research

Every two-way communication medium has been used for market research: face to face, mail, and telephone. So it's not surprising that when the Internet became popular, it was quickly put to use for audience research.

Internet audience research is still at an early stage, and many technical problems can occur. However the cost savings are so great that this method is well worth pursuing.

Internet jargon

This chapter includes some terms that will be familiar to experienced internet users, but not to most other people. The first time each term appears, it's briefly explained. Definitions of the 100-odd commonest terms used in internet research can be found on the Audience Dialogue Jargoncracker page.

1. Possibilities of internet research

Even in the richest countries, most people are not regular internet users. Though the percentage of people using the internet has been growing fast, the growth rate is sure to slow down. I'd be surprised if by 2005 as many as 90% of the population of any country are regular internet users - unless computers become far easier to use.

When less than 90% of a population use a medium, it's dangerous to do a survey and expect that the non-users would give the same answers, thus making the results true of everybody. This is why telephone surveys weren't widely used until the 1980s, when in the richest countries about 90% of households had a telephone. And it took 100 years for the telephone to reach that level of coverage. Penetration of the internet is already much faster than that, but for the next few years any surveys done on the internet will have to be based on specific populations already using the internet.

An aspect of surveys which the internet may change is sample sizes. With the internet, it costs little more to survey a million people than a hundred. So why bother with a random sample? Why not interview everybody - such as all visitors to a web site?

This is an attractive idea, because sampling error disappears, and personal computers are now fast enough to analyse huge samples. However, there are disadvantages. Completed questionnaires always need to be checked carefully, and computers can't do this as well as humans.

Also, large samples are no substitute for accurate samples. If a million people visit a web site, and 100,000 of them respond to a questionnaire, the results could still be completely wrong - if the 900,000 who didn't respond are different in some important way. And if the response rate is only 10% they probably will be different. So it's still important to ensure a good response rate, of 70% or more. To achieve a high response rate, the same criteria apply as for other surveys:

Advantages of internet research

1. Low cost

The main attraction of internet research is its low cost. Because the equipment has been paid for (mostly by respondents), the cost to researchers is very low. No printed questionnaires, no interviewer wages, no mail-out costs, no data entry costs. And instead of paying for every phone call made (as with a telephone survey), questionnaires can be emailed to an unlimited number of people for the cost of a single phone call. Even the costs of producing reports can be done away with: reports can be distributed by email or on the Web.

This will have a revolutionary effect on the market research industry. Costs will drop enormously: when the computer setting-up costs have been paid, the only remaining major costs for a survey will be the labour involved in questionnaire development and analysis. And when a survey is repeated (e.g. to monitor trends) the same questionnaire can be used over and over again, and the analysis can be computerized. With all these advantages, I predict that within a few years most professional surveys in Western countries will use the internet whenever possible..

Until then, only one population can be researched using the internet, with no worries about sampling problems. That population is (of course) people who already use the internet. So the internet itself can be used to do audience research about internet sites.

2. Fast response

With a typical email or web survey, most of the responses come back in less than one day. No other survey method can match this - but in fact, such a fast response isn't always desirable. For example, in Australia, where elections are normally held on a Saturday, some survey organizations do polls on the Friday night before the election. Often these are less accurate than the polls done earlier in the week. I suspect this is because a lot of people go out on Friday night, so are unable to be interviewed in a one-night poll. And if the votes of people who go out on Friday nights are different from the votes of those who stay home, the election results won't be predicted so accurately.

A similar situation exists for broadcasting: if a survey is done too quickly, without several callbacks, the people who are out a lot will be less likely to be interviewed. Because most TV viewing is done at home, a too-quick survey will overestimate TV audiences. Generally, about 3 days are needed for a fully accurate survey. With the Internet, this is easily possible - only a highly-staffed telephone survey can match it.

3. Ability to show a wide range of stimuli

As long as respondents have modern computers, they can see far more on an internet questionnaire than a paper one. Coloured pictures, even moving pictures, and sounds are all possible. If you are asking "Have you ever seen Eugene Shurple's TV program?" you can show a short clip from it, to make sure that respondents know exactly what is being asked. Though many people don't yet have a computer capable of showing all this, within a few years they will.

Problems with internet research

On the Internet, there are new problems, which seldom occur with more traditional surveys. The commonest problems (apart from poor samples and low response rates) are multiple submissions, and lies.

Problem 1. Multiple submissions

Many people inadvertently submit a completed questionnaire several times, by mistake. Often, they click a SUBMIT button to send in their questionnaire. Nothing seems to happen, so they click it again. Perhaps again - to make sure it is sent. So three identical questionnaires arrive at the host computer. How can the researchers know if this was really the same person sending in a questionnaire three times, or three different people who just happened to give the same answers?

Another reason for multiple submissions occurs when respondents are offered a very attractive incentive - such as a prize. So, when possible, use cookies to prevent people from making multiple submissions. Cookies are small files which can be automatically sent to a visitor's computer), but they don't solve the problem completely. If a computer has more than one user, a cookie will prevent a legitimate respondent from sending in a questionnaire. If a respondent sends in a questionnaire by mistake before it's completed, the cookie will stop them from making a correction.

Multiple submissions are also a problem with email surveys, but these are easier to check, because the sender's email address is always known. However, some skilled computer users can hide or fake their email - and many people have more than one email address.

Problem 2. Lies

On the internet, respondents lie! This happens sometimes with mail surveys and self-administered questionnaires, but it's usually rare. With real interviewers - whether face-to-face or by telephone - respondents don't have time to lie. But with internet surveys, I've often found blatant lying. Though it's often possible to detect a lie (if you study the responses carefully enough), you don't know what the correct answer is. Most people don't realize that some of their answers can be checked. For example, the country that somebody is in can usually be worked out from the IP number of their computer.

Part of the reason why internet respondents lie is poor questionnaire design. I've even lied myself, from frustration. With a live interviewer, or a paper questionnaire, at least you can write in an answer for a multiple-choice question which doesn't offer an answer to fit your situation. But with Web surveys, it's possible to force respondents to give one of several answers. If no space is provided for comments, their only alternatives are to lie or to give up on the questionnaire.

Recently, for example, I was filling in a Web questionnaire. I was asked: "Which state do you live in?" The 50 US states were then listed. Perhaps the survey was only intended for Americans (though it didn't say so), or perhaps the writer didn't realize that the questionnaire could be seen by anybody in the world. So, to find out what questions came next, I pretended to live in some US state. I clicked the mouse somewhere, without even looking at what state I chose.

Anybody who carefully analysed the results of that survey could have found that I had an Australian IP number, but supp-osedly lived in the USA. So it was possible for them to discover that I'd lied - but, judging from the obvious lack of care taken in the rest of the questionnaire, I suspect they never knew that I'd lied.

So if you don't want respondents to lie, let them give their own answers, without being forced to choose one of yours. I strongly suggest that, on a Web questionnaire, every question should have a space for comments or "other" answers - even a question as seemingly simple as "Which sex are you?" (Sometimes men and women jointly complete a questionnaire. In this situation, they'll answer "both sexes.")

Another advantage of having plenty of open-ended questions is that liars will give themselves away with their comments - or at least, raise strong suspicions. But of course, if you're getting thousands of responses a week and have only one researcher on the job, there's no time to read through the data file and assess its accuracy.

2. Uses of internet research

The ways in which internet research is used is limited by the representativeness of the samples that can be obtained. Even in countries where two thirds of the population now have internet access (in some form), these people are not average in any way, and to draw conclusions about the whole population would produce very inaccurate results.

One of the main uses of internet research is to evaluate web sites (and other internet outputs, such as email newsletters). There are no problems with sampling, because all users of a site can be reached using the same method that the site uses: e.g. web surveys to evaluate web sites, and email surveys to research email newsletters.

Even though there should be no problems with sampling, there are often problems with response rates. In 1996, I organized a survey for a web site by simply putting a link on its home page, saying "Our survey" - or some such words. Internet surveys at that time were a novelty, and this was the first I'd been involved with. However the response rate was fairly low. We didn't know how low, because we couldn't count the number of different visitors to the site (cookies weren't available back then), but after studying the server logs, the response rate seemed to be around 40%. Some other internet surveys I've read about had response rates less than 10%.

With such low response rates, it is those who take part in a survey who are (by definition) unusual. So you must ask yourself how respondents are likely to differ from non-respondents. If internet audiences act in the same way as broadcast audiences, the respondents (when these are a small minority) are likely to be the people who value the web site most, and feel some sort of inner pressure to respond. This means that the results are likely to be biased in favour of the web site, in its current (or recent) form.

Non-respondents are more likely to be people who visited the site for the first and only time, and can't be bothered becoming involved with improving it.

Therefore - as with every other type of research - it is very important to get a high response rate. It should be as easy as possible for people to complete a questionnaire, and there should be some sort of incentive to complete the questionnaire - either a financial incentive, or an emotional one. When the response rate is over 70% or so, the non-respondents would have to be very different from respondents, to substantially change the results.