Audience Dialogue

Manual analysis of surveys

From Know Your Audience: chapter 5

Not many surveys are analysed manually these days, even in developing countries, but manual analysis can be quicker than using a computer, when:

Manual counting works better if questionnaires are printed on fairly thick paper (i.e. not very floppy), and answer codes are shown on the questionnaires.

There are two possible ways to hand-count questionnaires:

1. to go through each questionnaire once, noting on a separate piece of paper the answers to every question.

2. to go through each question one at a time, counting questionnaires.

In theory, method 1 should be faster - but in practice, it’s not, because it’s too easy to make a mistake. If you are interrupted while doing this. you are almost certain to lose count, and you’ll have to start again. In practice, method 2 is best: one heap for each possible answer to the question you are currently tabulating. Though this means that you have to re-sort the questionnaires for every question, it also means that any errors are easily discovered - and, with hand-counting, there will always be a few errors.

For example, to hand-count the results of the question "Which sex are you?" you need to make space for 2 heaps (or possibly 3, if the sex of some respondents wasn’t noted). Sort the questionnaires into male and female heaps (and maybe a not-stated heap), then count the number in each heap. As you count each questionnaire, double-check that it’s in the right heap. The total of all the heaps should be the known total number of questionnaires. If it isn’t, count again.

After tabulating the responses to each question (e.g. 81 male, 119 female, 2 not stated), take the next question, and create a new set of heaps.

To make a manual count for a multiple-answer question (such as "Which languages do you understand?") is not so easy, because people who gave two answers should be in two heaps - but they only have one questionnaire. The solution is to make a separate count for each possible answer, with two heaps for each answer: those who gave that answer (e.g. understand English) and those who did not (e.g. do not understand English).

If you want to do manual counting of a questionnaire that takes up more than one piece of paper, this is possible, but each questionnaire will have to be folded so that you can always see the answer to the question you’re counting at the time.

Cross-tabulation by hand

For this, you need not just a row of heaps of questionnaires, but a matrix of heaps. If one of the two questions being cross-tabulated has 3 possible answers, and the other has 5, you’ll need to make space for 15 heaps: 5 across, and 3 down, or vice versa. If your table isn’t large enough, you can do this on the floor - as long as there’s no wind!

Piles of pages

If the two questions being cross-tabulated aren’t on the same page of the questionnaire, this needs to be done in two steps: first create heaps for a single variable (as described above), then turn over each questionnaire and re-sort each heap for the other variable. The arrangement you end up with is like a printed table - simply count the number of questionnaires in each heap to get the total for that cell. As with any table, the total number of questionnaires in each row and column should be the same as the original count for that question.