Audience Dialogue

Know Your Audience: chapter 3, part C
Producing questionnaires

7. Format of questionnaires

Questionnaires that are intended to be read aloud should be laid out quite differently from questionnaires which respondents will fill in themselves. With spoken questionnaires, the interviewers are trained in using the questionnaire for each survey, and will repeat it many times, interviewing a variety of different people. When interviewers are used, a spoken questionnaire can omit a lot of detail. Interviewers don’t need to be shown how to indicate answers, how to follow arrows to skip destinations, and so on.

But a written questionnaire is read and filled in by the respondent. Each respondent will see only one questionnaire (for a particular survey), so everything has to be explained in detail - but not so much detail that the respondent gets bored with reading it all.

The method of answering each question needs to be spelled out. Does the question need one answer only, or all answers that apply, or some limited number of answers? Is the question answered by ticking a box, circling a code, or writing in a full answer? A written questionnaire needs to explain all this clearly. It must look easy, attractive, interesting, error-free, and professional - otherwise the response rate and completion quality will suffer.

A spoken questionnaire should make allowance for the difficulty that interviewers have in recording answers while they are standing up, perhaps outdoors in wind or rain or poor light, resting the questionnaire on a clipboard. If you don’t leave enough space between codes, some interviewers may circle the wrong code by mistake.

Therefore spoken questionnaires shouldn’t try to cram too much onto each page. You may spend a little more on paper with this type of layout, but repeating a single interview may cost more than hundreds of sheets of paper.

Using answer sheets instead of questionnaires

In a spoken survey, two different approaches can be taken to laying out questionnaires. The usual method is to have one questionnaire per respondent, with the respondent’s answers written on the questionnaire. This uses a lot of paper, which increases costs and gives the interviewers heavy weights to carry around.

The other way is to have one questionnaire per interviewer, with answers entered on separate answer sheets. Answer sheets are laid out like tables, with replies from each respondent entered on a single line. This layout requires more attention by the interviewers (to make sure that the answers correspond with the right questions), but the survey will use far less paper.

Two types of layout are possible for answer sheets: one respondent per sheet, and multiple respondents per sheet. When a questionnaire occupies more than one page, and open-ended questions are used extensively, use a full-page answer sheet for each respondent.

If a survey has few questions and most of these have multiple-choice answers, you can have a single answer sheet for many questionnaires. The simplest layout of all is to have one line per respondent and one column per question - or vice versa.

To determine which layout is best, consider the maximum space that one respondent’s answers can occupy, and how the results are to be processed. There is little point in using an answer sheet if it has to be the size of a desk to fit all the answers, or if it needs more than one page. Answer sheets are best when questionnaires are short and have few open-ended questions.

If results will be processed manually and the questionnaire occupies more than a single page, using answer sheets simplifies the analysis, because all answers from each respondent can be seen on the same sheet of paper.

If the results will be processed by computer, it’s more convenient not to have to handle lots of questionnaires when entering the data, so the answer sheet is the quickest layout for computer data entry.

The main problem with answer sheets (and the reason they’re not often used) is that it’s easier for the interviewer to make a mistake. The interviewer must read a question from one piece of paper and write the answer on another. This makes it very easy for the answer to a question to be entered in the answer space for a different question, so steps have to be taken to guard against this — for example by varying the answer codes for successive questions. It is less likely interviewers will make a mistake if they are writing answers on the same questionnaire from which they are reading out the questions, so if your interviewers are inexperienced it’s best to avoid using answer sheets.

The exception is when a questionnaire is very short, and will fit on a single page. In this case, you can print the questions on the left-hand side of the page, and a number of columns of answers on the right, one column for each respondent. When you use this format, and a question is skipped (not asked of some respondents) some answer code should always be written in - otherwise, interviewers tend to write the next respondent’s answers in the current respondent’s space.

Questionnaire layout

When all the questions on a questionnaire have been finalized, it’s time to consider how the questionnaire should be laid out. These suggestions may help

Multiple-choice questions

Multiple-choice answers can be given in several ways. Boxes can be ticked, alternatives can be circled, or codes can be written in. When respondents fill in questionnaires themselves, tick-boxes are normally used, as they are an accepted convention and require no explanation to the public - at least, in countries where most people have completed high school.

When interviewers are entering the answers, it’s usual to have a short code next to each possible answer, and for the interviewers to circle the codes that apply. An example of a coding system is to write in M for males and F for females. Many surveys use numeric codes for multiple-choice questions, allotting number 1 to the first possible answer, 2 to the second, and so on. For example:

Sex of respondent
Circle number for answer given

1 Male
2 Female

If the codes are being circled, it doesn’t matter whether they are numeric (easier for computers) or alphabetic (easier for people): the interviewer simply circles the correct code.

Another method of answering is to have interviewers write in the correct code. In this case, it’s a good idea to use a mnemonic coding system: for example, M or F to indicate the respondent’s sex. You can even use two letters: for example if a statement is made and respondents are asked to indicate whether they agree or disagree, strongly or mildly, these codes can be used:

Strongly agree = SA
Mildly agree = MA
Strongly disagree = SD
Mildly disagree = MD
Can’t say = CS

Using mnemonic codes reduces errors due to wrong coding. For example, using M or F to indicate sex means that the interviewer doesn’t have to remember whether 1 means male or female.

Even when the codes are written right there on the questionnaire, interviewers still make mistakes. If you work in a comfortable office, it’s easy for you to say "the interviewers must be more accurate." But an interviewer who is questioning one person, being distracted by another, and trying to stop the questionnaire from blowing away (all at the same time) needs to be helped as much as possible to enter the correct answers.

Although this slows computer data entry, it’s better to make sure that the correct answer is recorded in the first place.

Open-ended questions

Don’t leave blank spaces for the answers: use dotted lines. Without dotted lines, handwriting strays, and is much harder to read. Leave at least 8mm vertical spacing — which on a typewriter is double spacing, and in a word processor is 24 points. (If the respondent is to fill in the questionnaire - usually sitting down - less space is needed: about 6mm, or 18 points.)

For each free choice question, you need to judge how much space to allow for an answer. Interviewers have a way of controlling how much most respondents say (except for the very talkative). The volume of answers rises to fit the space available. I suggest about 3 or 4 full-width dotted lines in answer to most questions. If many people try to give a longer answer, your question is probably worded too generally, and you should consider either narrowing its scope, or splitting it into several tighter questions. Most respondents drift off the subject if allowed too much time or space to comment.

Indicate types of questionnaire content

The work of interviewers can be made easier by clearly distinguishing between the three types of words that appear on a questionnaire: questions, answers, and instructions. A common convention is to type all questions in lower-case letters, and possible answers in capitals, with interviewer instructions enclosed in brackets. However, I’ve found that interviewers make fewer mistakes (reading out parts they should not, or not reading out parts they should) if you put quotation marks around any words they must speak.

If you are using a modern word processor, it works well to print the interviewers’ words in bold type, possible answers in normal type, and instructions in italics. Whatever you do, be consistent, and explain the system to the interviewers.

Indicate filtered questions

The final aid to interviewers is to visually mark any questions which are not asked of all respondents. For example, there might be a filter question: "Do you ever listen to the FM99 breakfast program?" People who answer Yes are then asked three other questions about that program, and those who do not answer Yes are not asked these questions, and skip straight on to the next series of questions.

With large samples, such skipping can run for pages at a time. But when the sample is already small, further reductions should be minimized. If all questions in a group to be skipped are on the same page, a good way to show the skip is to indent the questions which not everybody is asked, draw a box around them, and perhaps draw an arrow or two to clarify the flow of questions.

Layout on pages

Don’t split a question across two different pages: each question, including any filter instructions at the beginning or end, and all the possible answers, should be visible on the same page. Otherwise you will have problems - unless the interviewers are very well trained indeed.

If a question doesn’t quite fit on a page, you can either change the order, or (if that isn’t possible) squeeze it up into a smaller space, or leave a large blank space at the end of the page, and put the question on the next page.

The easiest ways to compress questions are: to reduce the margins, to use smaller print, to reduce the blank spaces between lines.

Interview logs

Another necessary piece of paper is the interview log: the interviewer’s record of the attempts made to contact each selected household. With cluster surveys, when the interviewer visits a number of different households in a neighbourhood, a single log is used for the cluster, with one line per household. Usually there are between 4 and 20 households in a cluster.

With surveys that don’t use clustering - such as telephone surveys - there is one page per household, or per telephone number. In this case, the logs can be smaller: often A5 size. The advantage of having a separate log for each interview is that the logs can be sorted into heaps: for finished interviews, numbers that will not produce interviews (e.g. refusals), pending appointments, and numbers not yet contacted.

It’s possible to do a telephone survey without using logs: the same information is recorded at the beginning of each questionnaire. However, a lot of paper will be wasted, because many telephone numbers do not result in completed interviews.

In cluster surveys, logs are almost essential, because they give interviewers a record of their progress in the cluster, on a single sheet of paper.

Chapter 4 has an example of a general purpose log, which can be copied, varied as necessary, and used in many surveys.

8. Testing questionnaires

This section forms a bridge between the chapter on questionnaires and the chapter on interviewing. Both interviewers and questionnaire writers should read it.

When you have written a questionnaire, and checked it against the criteria in this chapter, the next stage is not to rush into the survey proper, but to test it with some typical respondents.

There are two stages in testing a questionnaire: rehearsal, and pilot testing.

Rehearsal

First find two people who have had little or no involvement in developing the questionnaire. One should be an interviewer, and the other will play the part of respondent. This "respondent" should be somebody you know well enough that they will frankly discuss any shortcomings of the questions.

The interviewer, before the interview, should read through the questionnaire. Doing an interview is like learning lines in a play. You must decide where to pause, and which words to emphasize. If the phrasing of a question isn’t clear, read it aloud in several different ways, and choose the way that most clearly conveys the intended meaning.

In the real survey, consistency between interviewers is important; all interviewers should phrase each question in the same way. At the early stages, it’s best to go through the questionnaire with a pencil, marking it to show pauses and emphasis. If a sentence in a spoken questionnaire could have an ambiguous meaning when read with the wrong emphasis, it’s a good idea to underline the words to be emphasized.

The "respondent" in the practice interview should not only answer questions, but also take notes on any problems associated with the questionnaire. The interviewer checks for tongue-twisters, omitted explanations, errors in skip instructions, and anything else that may create a problem.

When the respondent has answered all the questions (perhaps several times, playing different roles), the questionnaire writers, the interviewer and respondent all meet and discuss whether the written answers give a true impression of the respondent. This is more than a simple matter of accurate wording; it also involves whether the respondent felt that some important questions were not asked.

If possible, the whole interview should be tape-recorded. It is valuable for interviewers to listen to tapes of themselves interviewing somebody, and improve their style. When the interviewer and respondent get together to review the "rehearsal" interview, they should both listen to the tape of their dialogue. By asking such questions as "What did you think I meant by that?" and "How did you feel about that?" both the wording and the presentation of questions can be much improved.

Tape recordings don’t replace written notes - but they allow the written notes to be made afterwards, with both the interviewer and the respondent listening to the tape.

At least two of these pseudo-interviews should be done, with different respondents, and different interviewers. Any one interviewer-respondent combination may be quite atypical.

During the "rehearsal", when the interviewer comes to a question that seems longer or more complex than most, it’s interesting to ask the respondent to repeat the question, word for word, back to the interviewer, before answering. If the respondent can’t repeat the question accurately, the question needs to be simplified and/or shortened.

Also at this stage, the questionnaire’s duration should be checked — but the actual duration is not as important as the psychological duration. At the end of a pilot interview, the interviewer should ask the respondent how long the interview lasted. If it really lasted 10 minutes, but the respondent thought it took 20, this is not a good sign — the questionnaire need not necessarily be shortened, but perhaps the questions are dull or irrelevant to the respondent. Sometimes it’s a good idea to throw in a few opinion questions, not because you need the results, but to improve the motivation of respondents.

This rehearsal will nearly always bring up some questions that need to be reworded, and other small improvements that can be made to the questionnaire.

Pilot testing

When these improvements have been made, the next stage is the pilot test, using real interviewers and real respondents, in the same situation as the real survey, e.g. in the respondents’ homes.

For a pilot test, I suggest a sample of 10 to 20 interviews. If the questionnaire is short and simple (no more than 10 minutes, with little or no filtering), and the budget is small, 10 will be enough. For a high-cost survey or a long and complex questionnaire, 20 may not be enough. It’s not necessary to decide the exact number in advance: just keep going till you have no more problems.

Each interviewer used at this stage should do 2 interviews, exactly following the instructions. After the introduction, but before beginning the questionnaire proper, the interviewer should say to the respondent something like: "We’re still trying out this questionnaire, so if anything isn’t clear, please tell me straight away."

It’s a good idea to tape some of these interviews, so that the tapes can be played back and all interviewers can discuss them. Respondents’ permission should be obtained for the taping. If a respondent seems worried, explain that you only want to get their answers accurately, and that you’ll erase the tape as soon as you’ve done this.

By this stage, any changes required should be fairly minor. If extensive changes are made as a result of the pilot test, more interviews should be done. Any change of question order, in particular, can bring unforeseen repercussions.

Printing questionnaires

When the pilot testing is finished, the questionnaires can be printed. If you don’t do comprehensive two-stage testing, as outlined above, you are almost certain to find - on the first day of real interviewing - that the questionnaire contains mistakes. In a large survey, with thousands of pages printed, it is safer to photocopy a few questionnaires for the first day’s work, and make any corrections before printing the bulk of the questionnaires.

Print more questionnaires than the number of interviews you are planning. A lot of questionnaires cannot be used: some copies are spoiled by interviewers, often some are not properly printed, and spare copies often cannot be transferred to other interviewers.

I suggest printing 20% more than the planned number of interviews. Give each interviewer 10% more than they should need, and keep 10% back in the office - for later distribution to interviewers who run out of questionnaires, for office use, for giving to interested people, and a few to be filed away, to be used when the next survey is done.