Audience Dialogue

Was your event a success?

Outline:
1. The standard approach
2. Problems with it
3. An improved approach
4. Triad discussions
5. Evaluating beyond the event

The standard approach

Often, at the end of a seminar, a talk, a course - or in fact any kind of event when an audience is in a room other than for entertainment - questionnaires are handed out to audience members at the end. They are requested to fill in their questionnaire, giving their opinion of the talk (or whatever), and to hand it up.

When writing this web page, I looked for some published research on this kind of evaluation. I couldn't find much at all. Perhaps this kind of research seems too trivial to take seriously. However, a lot of people put a lot of effort into these events; they are genuinely interested in audience reactions, and how the event could be improved next time.

Because not much seems to have been written on this, and because Audience Dialogue has helped evaluate several hundred such events in the last few years, I thought it was worthwhile to try to record some of the principles we've learned: how to do it, and how not to do it.

The questionnaires are sometimes called "happy sheets," suggesting that the participants give too favourable an opinion. The implication is that if they were asked the same questions after the event, by a third party, opinions would be less favourable. Actually, that's not what Audience Dialogue has found. If you ask audience members to rate the event they've just attended on a scale between 0 and 10 - where 0 is the worst possible rating, and 10 the best possible - the average answer seems to be around 7 out of 10, no matter when the questions are asked or who asked them.

Problems with the standard approach

The standard approach to event evaluation has four big problems:

  1. It measures only attitudes, not behaviour or knowledge.
  2. It happens too soon. Normally, the questionnaires are filled in at the end of the event. This allows no time for learning, or behavioural change.
  3. But in a long course or event, it can also happens too late. By the end of a day, participants may have forgotten the points they thought of hours ago.
  4. It evaluates only a small part of the process. Think about what an event is trying to achieve, and you soon realize that questionnaires given out to gather the audience's opinions are only a small part of the process - see Beyond the event (below) for more on this.

An improved approach to event evaluation: summary principles

To solve those problems, the scope of the evaluation needs to be extended, and more time needs to be allowed. Some ways to achieve these goals are...

1. Always think of the event in its context. The study is never solely about the event as experienced by participants: that's just one part of the evaluation. Every event is done for some purpose, and those attending it usually don't know the full purpose. It's useful to think of events using program logic, like this:

With that framework in mind, an event is evaluated by answering these questions:
  1. What inputs were used? (Money, time, resources)
  2. What activities were done? (An event was organized?)
  3. What outputs were produced? (E.g. X people attended the event.) For indicators of efficiency, calculate how much input it took to produce a given output.
  4. What were the impacts? (Partly from the questionnaire given out after the event, partly from other sources.)
  5. What are the outcomes? (E.g. reviewing the event a few months later, what effects did it have, among what people?) For indicators of effectiveness, compare these outcomes with the initial goals.
Considering the entire planning and effects of the event, you can see that a questionnaire filled in on the spot produces only a small proportion of the information needed to evaluate the event, and the program it forms part of. Consider all the people involved - participants, those affected, and those who did not attend but were still affected in some way. Even if an event is a flop at the box-office, it may still have important effects on artistic life. Perhaps the spending on sets for a play helped to keep some precious skill alive in the local area.

And even if this particular event wasn't a box-office success, and had no effects on artistic life, it can still fulfil a broader purpose. For example, if a local drama group produces an avant-garde play, this may help to attract the attention of distant funding sources. (But if that's one of the purposes of such a production, the achievement of that purpose shouldn't be left to chance: it should be sought as a planned outcome, in the framework mentioned above.)

Broadening the context further still, consider benchmarking your event against others. This is done by (i) gathering data in a standard format, then (ii) comparing the results for your event with other results in the same format. One such format is the Transfer of Training Evaluation Model (TOTEM) which can be used in a wide variety of educational evaluation contexts. This can be found at the US Department of Energy's Knowledge Transfer Website at www.t2ed.com, though benchmarking data doesn't seem to be available there.

In general, I suggest that you try to find and use a standard evaluation scale, rather than trying to develop your own. There are many pitfalls in developing a new scale, some of which are not obvious till it's too late.

Another aspect of context is peer review. Other people and organizations that produce events of this kind can be useful sources of evaluation - even if they are biased. Though they'll all have different viewpoints, if a wide range of experts agree on a criticism, you'd better take it seriously. For successful peer evaluations, you need to have 3 or more people present, with experience in the same type of event. Get them to fill in a special questionnaire (based on the same one that ordinary participants fill in, but with extra questions), and see if they all agree about the strong points or weak points of your event. If they do, you should take notice.

Even when you have all this information, you can still be left wondering. Perhaps you asked participants to rate the event on a scale of 0 to 10, and the average rating was 7 out of 10. Is that high or low? (In fact, it's a little below average - based on our results from hundreds of surveys). Unless you have a context to place the results in, such figures will be meaningless. This is an argument that lots of little evaluations are more useful than one big one.

For long courses and events - more than about half a day - participants often forget suggestions they thought of making. And if communication isn't working well in a course that runs for a week, it's no help to discover that at the very end. Quick evaluation sessions - using both written and spoken form - at the beginning or end of each day can be very helpful.

Improving event questionnaires

Mix multiple-response questions (easy to answer, but not very informative) with open-ended (more valuable responses, if people take the time to think about the answers, and if the questions are fully relevant).

Keep it to one page (A4 or letter size) if possible, but definitely to one sheet of paper. If both sides of the paper are printed, write PTO or OVER or MORE at the bottom right of both pages. Consider the nature and size of the surface that will people have to write on. For example, if they are sitting in theatre-type seats, without table tops, will they rest the questionnaire on their knees to fill it in? In that case, maybe it should be printed on card, not on thin paper.

Open-ended questions should span a range of generality. For example, if you ask the very general question "What other comments would you like to make about this seminar?" nobody's comments are excluded, but many people will not have time to think of comments. (Usually, at the end of an event, most people are in a rush to leave.) On the other hand, if you ask only specific questions, such as "Which slides, if any, had writing that was too small for you to read?" people who had problems you hadn't expected will have nowhere to give an answer. The solution: use both types of open-ended question, the specific as well as the general.

Ask behaviour questions as well as attitude questions. Questions such as "How would you rate the quality of tonight's performance, on a scale of 0 to 10?" are about attitudes or feelings. While these are perfectly valid, they don't necessarily relate to future behaviour - which may interest you more. Perhaps what you really want to know is "If we put on another play like this in a few months' time, how likely are you to attend?" A behavioural intention question like that, though far from a perfect prediction, normally produces more useful results than an attitude question.

Other useful behavioural intention questions are along the lines of "What changes will you make in your organization as a result to attending today's workshop?" A list of actions can then be presented, and respondents invited to tick those that apply. The interesting thing about this approach is that it can be (for some people) self-fulfilling: the act of making the choice on the questionnaire can actually cause them to carry out their intention.

Even more accurate than behavioural intention is behavioural reporting. For this to work, you could collect their name and address on the questionnaire given out at the event, and ask their permission to recontact them later. Perhaps a month or two later you can recontact those respondents and ask what they have actually done as a result of attending that event. If the results are favourable - that is, if the respondents have done what they said they'd do - this can be a very powerful argument for seeking more funding.

Improving the environment for evaluation

Though the questionnaire wording and layout is important, its environment is even more important. Ideally, you want everybody present to fill in their questionnaire, and you want honest answers from them.

Improving the response rate

Imagine you're a member of the audience at a seminar. What you heard and saw over the last hour or two was quite interesting, but it's getting late now, and you have to go home and cook dinner for the children. Everybody is asked to pick up a questionnaire on their way out, fill it in, and put it in a box. You don't really feel like doing this (it seems hardly worth the effort to simply record "It was OK") but the compere asked everybody to make the effort and fill in the form. So you pick up a form off the heap as you leave. It's long - about 20 questions - and they seem to be repetitive. Some look quite difficult to answer, but obviously worthwhile. They want your comments or suggestions for "next time" - not that you plan to come along "next time." Some of the wording is hard to understand.

For example, one question was "How adequately did the presentation meet your learning expectations?" This was to be answered on a 0-10 scale. So, when you have figured out exactly what this question is asking, what might a score of 10 out of 10 mean? "I expected it to be perfect, and it was." Or (equally valid) "I expected it to be useless - and it was." In practice, the question was so opaque that you didn't read it very carefully, and just gave a general rating out of 10 based on what you thought you had learned from the seminar.

So you decide to fill it in (giving that question 7 out of 10), but then you realize you don't have a pen with you. Maybe you can borrow one. Also, you have nothing to rest the paper on when you fill it in. People around you are putting their forms up to the wall, and trying to write with ballpoints - which don't work well unless pointing down. The lights are dim, and the questionnaire is printed on blue paper - very hard to read. Also, you can't see the place where the presenter said the completed forms should be left. So you put the questionnaire in your bag, and take it home. Maybe you'll fill it in later tonight, and mail it back to the organizers tomorrow.

But when you get home, there is a minor crisis (perhaps the cat was sick) and you forget to fill in the form. You put it away for later, then lose it. A week later, it surfaces in a heap of paper. By then you've forgotten what you were going to write, there's no address on the questionnaire for you to mail it back to, and by now it's probably too late anyway. Still, you're reluctant to throw it out, so you move it into a heap of papers that you might think about some day. Maybe six months later, you find it and finally throw it out.

That story (not so uncommon) shows why response rates for event evaluations are often so low. Organizers have been known to congratulate themselves for getting a 20% response rate, falsely believing the average is 3%. If 100 people attend a seminar, and only 20 forms are returned, what did the other 80 people think? Did they believe the seminar was so great that they had nothing to add? Did they think it was so terrible that they'd be embarrassed to hand in a form full of criticism? Or were they so underwhelmed that it made no impression on them at all? The organizers will never know.

That's why it's vital to get a high response rate. If you get at least two thirds of the questionnaires back, the other third of the audience would need to have very different opinions to make a large difference to the results. And the way to get that two-thirds response is to remove the barriers that prevent people from completing and returning the questionnaire. The steps needed can be grouped into five main headings:

1. Make the questionnaire easy to fill in.
- Keep the questionnaire short and relevant.
- Avoid questions that need a lot of thinking time (unless you distribute the questionnaires before the event begins)
- Also avoid questions that encourage an instant, thoughtless response.

2. Allow enough time.
- If you put the questionnaires on the seats before the audience arrives, people will be able to fill them in at dull moments during the event.

3. Encourage response.
- The more strongly the presenter encourages people to fill in the forms, the better the response rate. However the presenter should ask respondents to be critical, and should not collect the questionnaires in person.

4. Avoid barriers to completion.
- Provide pens for people who haven't brought one. Ballpoint pens are very cheap, when you buy 100 at a time. The cheaper they are, the less likely people are to take them away - specially if they can't be closed, because you've removed the end-caps. Then you can use them again next time. Another approach is to tell participants that you're giving them a free pen, but that in return you'd like them to complete the questionnaire.
- Provide a surface to write on. If that's not possible, print the questionnaires on thick paper, or in a small size, folded.
- Make the questionnaires easily readable, in the conditions that will exist at the venue. For example, if the lighting is low, don't use thin, small type on dark-coloured paper. (Garamond 10 point is about the worst; Comic Sans 12 point is among the best.)

5. Make the form easy to return.
- If audience members will have other papers, print the questionnaire distinctively, so that it will stand out and be harder to lose - e.g. on bright yellow paper.
- Provide a collection box at every exit from the venue - or better still, have people standing there to collect the forms. The collectors must not appear to look at the completed forms, which might inhibit frankness.
- You could provide a reply-paid envelope for people who want to take their forms home and fill in them in later. Though people think they might do this, in practice hardly anybody does. Giving everybody a reply-paid envelope only encourages them to take the form home, so it's generally not a good idea. However it is a good idea to print a mailing address on each questionnaire, for the few people who really will post them back. Better still, use a freepost address, so they don't have to find a stamp. On a one-page questionnaire on a business topic, mention your fax number, so that respondents can easily fax it back to you.

Triad discussions

Though multiple-choice questions on an evaluation questionnaire enable the comparison of different events (e.g. in a series of events), they don't provide useful information for improving an event. If all you know is that 73% of respondents disliked the event, how can you use this information? You can't, and that is why you need to include open-ended questions.

But open-ended questions have their problems too. Because they rely on respondents to think of their own answers, you tend to get a lot of unique responses. This makes it hard to summarize the results. If 3% of respondents commented favourably about some aspect of the event, does that mean the other 97% disliked it? Or didn't they even notice it?

Another common problem with open-ended questions is that people write cryptic comments. They know what they mean, but to the person processing the completed questionnaires the answer is unclear or ambiguous. This is usually because the answer is too short.

After thinking about these problems, I developed a solution: triad discussions. It works like this:

  1. Everybody fills in a questionnaire in the normal way: alone. The questionnaire includes quite a lot of open-ended questions, such as
    - "What did you like most about this event?"
    - "What did you like least about this event?"
    - "How do you think this event could have been improved?"
    - "Are there any other comments or suggestions you'd like to make?"
  2. The participants then divide into triads: groups of three. One is appointed as secretary (whoever admits having the most readable handwriting), and is given a new blank questionnaire, printed on paper of a different colour.
  3. Now the people in the triad discuss their answers. Each person in turn reads out his or her own answers to a question. If one of the other two doesn't understand it, they say so, and the person who wrote the response should add a few words to make it clear.
  4. To prevent people automatically agreeing, not wanting to upset the others in their group, each of the three can be assigned a particular role: supporter, clarifier, and critic.
    - The person who first made the statement obviously believes it to be true. For example, take the statement "Letters were tiny."
    - The secretary's role can be to clarify the wording, and make the statement clearer. The statement might become "Letters on slides were too small to read." (For statements made by the secretary, somebody else must be the clarifier.)
    - The third person's role is to challenge the statement, to limit its scope, or to admit its subjectivity. After the challenge, the statement might read: "The letters on some slides were too small to read from the back of the room."
  5. When the statement is clear, the others vote on it. The secretary writes the statement on the different-coloured group questionnaire, followed by the number who agree (1, 2, or 3) - e.g. "The letters on some slides were too small to read from the back of the room (3)".
  6. The coloured questionnaires from all triads are posted on the walls of the room, and all participants are given some sticky dots to vote on the statements they agree with most strongly.
  7. Because many statements will be similar, the researchers can later combine their numbers of votes. (It would be better to do this before the sticky-dot voting, but that's usually not feasible: while the statements were being compared, the event would be finished and the participants would be wanting to leave.)
The advantages of the triad method are that

If you have read about our consensus group technique, you'll recognize the method of triads as a miniature consensus group.

Groups of four or more people can also do this. However, the larger the group, the longer it takes - and there's usually not much time left for an evaluation at the end of an event. The triad process can often be finished in five minutes, for an event questionnaire with up to about 6 open ended questions.

Evaluation beyond the event

Whatever kind of event you are evaluating, consider why it was held. What was the purpose of having that event in the first place? What are you hoping the audience will do in response to the event? Even if the event is pure entertainment, you would probably like the audience to follow it up in some way. Should they urge their friends to attend later performances? Should they attend the next performance at your venue? Should they change their lives because of this event?

If the event is some form of training, the Kirkpatrick model will apply. Donald Kirkpatrick (in his book Evaluating Training Programs, published by Berrett-Koehler, San Francisco, in 1994) described a 4-level model for evaluating the success of training...

  1. Were the trainees pleased with the event? This is an aspect of customer satisfaction, as commonly assessed in the kind of survey mentioned above.
  2. How much did they learn? This can be assessed by educational tests, exams, etc.
  3. How much did they change their behaviour? In the case of industrial training, this can be assessed by supervisors, on-the-job performance measures, etc.
  4. How much did that changed behaviour contribute towards the organization's goals? (E.g. a training department would hope that its activities increased the organization's operating efficiency).
With the Kirkpatrick model, success at each level depends largely on success at the previous levels. If the trainees didn't like the course, they probably won't learn much. If they don't learn much, they probably won't change their behaviour. And if they don't change their behaviour, the organizational goals for the course probably won't be achieved. Notice the word "probably" - there might be the odd exception, but it's much harder to achieve a higher level of success if the lower levels haven't also been achieved.

The higher the level, the more difficult it is to be sure how much difference the course made. Participant satisfaction is easily measured, but it's often not clear to what extent a course might have increased a company's profit. For that reason, success at Kirkpatrick's Level 4 is often judged too difficult to assess.

When we tried the Kirkpatrick model, we found that it omitted some important questions, that Kirkpatrick perhaps took for granted...

When answers have been gathered for the above questions, interesting cost ratios can be worked out - such as how much per person attending it cost to achieve the goals of the event. If that figure seems unduly high, it's worth considering a different method of achieving the goals.

If you've read our page on program logic models you'll realize the direction this is heading: there's nothing as useful as a logic model in evaluating the success of anything - including a simple event.