Audience Dialogue

Content analysis of an open-ended question

Case study 27

From the four types of content analysis described in our main content analysis page, here's an example of focused content analysis from a personal source. (The other types included article-based sources, and unfocused analysis.)

Audience Dialogue has been helping with the evaluation program of the National Institute of Manufacturing Management (Australia) - now defunct.  This Institute organize seminars on manufacturing-related topics, and is very keen on evaluation. At every seminar, a short questionnaire is distributed to the audience members, who are asked to answer questions on the relevance and value of the seminar. One of these questions is "What are the main issues facing your business in the coming year?" Space is provided for three issues: one dotted line each, labelled 1, 2, and 3. Each dotted line is wide enough for audience members to write 6-10 words.

The focus in this example is to find out to what extent audience members (mostly small manufacturers) saw the major issues facing them as (broadly) external to their firms, or internal.

From 2000-odd questionnaires completed during 2001-02, 800-odd people gave at least one answer to the question on the main issues facing their business in the coming year. There were a total of 1364 comments - after weeding out...

The comments were originally entered in an SPSS file, along with the rest of the data for each questionnaire. There was one record for each questionnaire, and three fields allowed for answers to this question.

We do not recommend SPSS for this purpose. SPSS was designed to handle numerical and coded data, and has no way of handling open-ended text. Excel, despite its weaknesses in handling text, is better than SPSS for data entry of comments. Therefore we transferred the comments from an SPSS file to an Excel file. We put one comment on each row, with these column headings:

Column A. Event number:
The particular seminar being referred to. A separate file gave details for each event: date, time, duration, venue,presenter, topic, people present, etc.

Column B. ID number of questionnaire:
An arbitrary number, starting from 1 for each event. Questionnaires were stored in order of event number, and for each event they were in ID number order. Thus by knowing the event number and ID number for a questionnaire, it could be quickly found.

Column C. type of organization: <br) we="" pasted="" a="" column="" from="" the="" spss="" file="" to="" excel="" comments="" file,="" showing="" whether="" respondent="" worked="" in="" manufacturing="" and="" size="" of="" their="" organization="" (manufacturing="" or="" non-manufacturing,="" under="" over="" 100="" staff="" -="" 4="" possible="" answers).="" this="" was="" because="" wanted="" compare="" types="" made="" by="" small="" manufacturers,="" large="" non-manufacturers.="" <p=""> Column D. Text of the comment:
Each row in the file corresponded to a single comment. As the question being studied allowed for up to 3 answers per person, there were between 0 and 3 rows per person, depending on the number of answers given.

Columns E to J. Comment categories:
We defined six categories of comments that interested us for this purpose:

- E. about technology
- F. about costs
- G. about human and organizational issues
- H. about technology
- I. about the economy
- J. about competitors

Then (in effect) we asked each comment six questions:
"Are you about E?"- "Are you about F?" - "Are you about G?" - "Are you about H?" - "Are you about I?" - "Are you about J?"

When the comment answered Yes, we put a 1 in that row, for that column. When the answer was No, we put 0. If the answer wasn't clear, we entered 2. All this was done by the main coder.

When the preliminary coding had been finished, three of us met: Kym (the main coder), Chris (the technology diffusion expert), and myself (the evaluator). We met in Adelaide's Cafe 31, and went through the problem codes, numbered 2. Our goal was to convert all the 2s into 0s or 1s. Ruthlessly, we drank coffee, and sorted out the problems. The rule was simple: if we didn't all agree that a code should be 1 ("Yes, it applies") we made it 0 ("No, it doesn't apply"). Before our coffee was cold, we'd solved all the problems, and no 2 codes remained.

With this system, some comments fitted more than one category, while others (e.g. about marketing) fitted none of the 6 categories. In fact, the average comment answered Yes to about 1.3 categories of the 6.

When the coding was finished, we simply summed each column to get the number of 1 answers, using the SUM command in Excel. This enabled us to calculate the number of comments (usually a little greater than the number of people) who gave each of the six types of answer.

This is very different from the way that open-ended answers are normally coded. The usual process (that you find in books on content analysis) is to first to create a list of categories, then to code each comment into one of those categories. In that case, we could have used only column F, and entered a number from 1 to 6 to show which code best applied.

The reason we chose not to code the comments in the traditional way was that we could achieve greater consistency by effectively asking the comment 6 questions with Yes/No answers, instead of one question with 6 possible answers: "Which one of these 6 categories do you come into?"

Though you might expect it would take more time to ask each comment 6 questions than one, in fact it probably takes less. Instead of asking "Are you more about A, or more about B?" we only needed to ask "Are you about A? Are you about B? Are you about C?" and so on, up to F. If the comment was about more than one category, that was easily coded.

Coding long comments can be quite subjective, so to achieve a high level of credibility requires a coding process that's as explicit as possible. We find that this can best be achieved by treating each code as an answer to a separate question.

The practical problem in this case was that some comments (about 7% of them) were so short that we couldn't be certain of the respondent's meaning.

This is the easiest application of content analysis: when the beginnings and endings of comments are clearly marked, and when everybody is answering the same question. If two people are given the same task, of fitting such open-ended answers into a specified coding frame, and the instructions are clear, they usually agree about 90% of the time. For the example just given, the two reviewing coders agreed with the main coder in about 98% of cases. The high figure arose partly because we'd spent an hour discussing the issues before the coding began, and then we prepared a very explicit set of written instructions. If you don't make detailed preparations like this, you'll find it difficult to reach 80% agreement.

Another factor is the wording of the original question. The more specific the wording, the higher is the agreement between coders. A vague, loose question will usually be interpreted in different ways by different people, and will produce not just different answers, but answers to different questions.

Back to Content Analysis page

To learn more about audience research click on the link here to access our comprehensive book Know your Audience.