Audience Dialogue

Measuring controversiality

Recently we were conducting a workshop to help people envision the future of their region until 2025. For this, we covered the walls of the room with 257 numbered photos of the region. The workshop participants each filled in a questionnaire with the numbers of photos showing (a) aspects of the local environment they wanted to keep, and (b) aspects they wanted to get rid of.

A main purpose of the workshop was to discover how much agreement existed between different stakeholder groups: residents, workers in the area, business owners, administrators, and visitors. So when we collected the completed questionnaires, we needed a method to measure the differences in ratings between the different groups. You might call this a measure of controversy, or controversiality.

The questionnaire data was recorded on a spreadsheet, in a table that showed for each photo the number voting in favour of it and against it. Superficially, it seems simple to measure controversiality: just use a measure of dispersion, such as standard deviation or variance. To do this, score a positive rating as +1, a negative rating as -1, and calculate the dispersion for each photo. However, because there were so many photos, people only wrote the numbers of the scenes they liked and disliked: the average respondent mentioned about 50 photos. This suggested adding a third count for each photo: the number of people who didn't mention it; this might be scored as zero. So we did that.

The problem was that when we calculated standard deviations, they didn't produce useful results. For example, if 9 people voted in favour of a photo and 1 against it, the standard deviation was much higher than when 5 voted in favour and 5 against. In that situation, this didn't make sense. We realized that controversiality, though similar to dispersion, is not exactly the same thing.

So I came up with a new measure of controversiality. If the number of positive votes for a photo is A, the number of negative votes is B, and the total sample is N, the formula for controversiality (let's call it C) is

C = min(A,B) / max(A,B) x (A + B) / N

min(A,B) means the smallest value, whether that is A or B) and max(A,B) is the larger of the two figures. If A is the same as B, the minimum is also the maximum, and min(A,B)/max(A,B) = 1.

This measure ranges from a minimum of 0 (wheneverybody who voted on a photo either liked it or disliked it) to a maximum of 1 (when everybody in the sample voted, splitting exactly 50-50. If everybody votes on every issue, you don't need to include (A+B)/N because it's always 1. This is a much better behaved measure (in this context) than standard deviation. It's also very easy to program on a spreadsheet.

This measure is so simple that probably somebody has come up with it before, but a quick search of the web and some statistics books didn't locate it. (If it's anywhere, it might be in Numerical Taxonomy by Sokal and Sneath - a great book that I lent to somebody years ago and never got back). So if anybody wants to use this measure of controversy, here it is: a tiny contribution to the toolbox of statistics.