Surveys and The Slaughtered Ox Problem
The wisdom of crowds only works some of the time
I was recently pitched a report based on a survey of 1500 or so Americans who had lost their jobs in the pandemic, asking them about their expectations about getting back to work. I’ll leave aside the specific findings and the group behind the research, but I will use the experience to explain why we should not take research like this seriously.
The Wisdom of the Crowd
We have a tendency to believe that polling a group of people on some topic is likelier to yield helpful insights than randomly asking a single person. This is why we rely on trial by jury to enact justice, or poll registered voters on who they plan to vote for.
It turns out that this works only in certain circumstances.
The statistician Francis Galton attended a Plymouth England county fair in 1906, and observed 800 people joined a contest to guess the weight of a slaughtered ox:
Galton observed that the median guess, 1207 pounds, was accurate within 1% of the true weight of 1198 pounds. This has contributed to the insight in cognitive science that a crowd’s individual judgments can be modeled as a probability distribution of responses with the median centered near the true value of the quantity to be estimated. [via Wikipedia]
But note that the question ‘what is the weight of this ox?’ has only one correct answer, to which the crowd’s wisdom is directed. And also note that 800 people were involved, which made it a crowd. Had there only been eight people guessing, the median of their guesses would not have been so close (unless they were experts, like butchers, who may have been excluded from the county fair contest).
It turns out that the wisdom of the crowd only surfaces when the group is diverse, and when there is a correct answer to the question being asked.
Contrast that with the circumstances around the pandemic jobs survey. There is no ‘correct’ answer to the question ‘when will you be back at work?’, or ‘what work will you be doing when you are employed again?’. It is all hypothesis, based on many uncontrollable variables, and each individual’s future is independent of all the others in the survey. So, for example, if the median answer to ‘when will you be back at work?’ is December 6, 2020, that shouldn’t be interpreted as the day that all 1500 respondents would start new jobs. Divergence of possibilities cannot lead to convergence of that sort.
We have to discount the aggregation of the crowd’s response to the survey’s questions about the future because unlike the case of the county fair, there is no slaughtered ox on display, only a collection of hypotheses.
There is no shortcut to the hard work of researching when people will be going back to work, and what work they will be doing.
It is a wicked problem involving epidemiology, economics, politics, and human psychology, and so, we are better off talking to experts in those domains, who play the role of the butchers in this example. But you are still likely to get better answers by increasing the diversity of the group and minimizing their communication with each other during the survey activity. Taking those precautions decreases convergent thinking.
But the result is still the hard work of balancing the diverse opinions of a collection of experts who are coming at the issue from many different perspectives and domains of knowledge. The crowd won’t save us that work, because there is no slaughtered ox, only wishes.