Experts and Influence

David Levy and I have sketched out a paper in which we check the expertise of the past.  We can do that since the data that were used by Francis Galton -- people's guesses about the weight of an ox -- have been preserved.  The paper first talks about Galton's use of the data (he chose the median guess as the best estimate of the ox weight) and then about how Galton's results were re-told, and changed over time.  Here are a couple of paragraphs from the paper, which looks at this as a case study of the role of expertise:

The question of expertise enters into this because, if experts pass along false information (wittingly or unwittingly), they become part of a process by which errors are diffused. If experts are trusted (and if they trust the experts whose work they cite), mistakes are diffused and remain uncorrected. If, by contrast, expert results are not accepted automatically, then their results are more often subject to the scrutiny associated with replication. Knowing this, an expert whose work might be scrutinized will check the work carefully. An expert who expects deference, acceptance of results without checking, will check the work less carefully. Paradoxically, such trust increases the probability of mistakes being made or passed along. Thus, we expect that the errors made by experts will tend to be systematic because they will be repeated by those who trust them.

Our examination begins with Galton’s contribution on voting as estimation as a way to bound influence. But this is only the beginning of our story. Even more interesting, perhaps, than Galton’s contribution itself, is the “retelling” that has occurred since. Though Galton was defended the median as the estimator for the ox weight, the tale of Galton’s median was changed soon after. In addition to the median location of central tendency, Pearson suggested the mean. Along the way in the historical retelling, embellishments were added so that Galton’s original procedure was entirely misconstrued. Our story recounts the misconstruing. As we do so, a key question is whether the tale was changed deliberately (falsified) or whether, not knowing the truth, the retold (and different) tale was passed on unwittingly.

A pdf of the paper will be available when we fix it.

Here's a scan of some of Galton's data:

Comments

[James Surowiecki]

(I've already posted much of this at Overcoming Bias.)

Sandra, before accusing me of disseminating a "falsehood" in my book, you and David Levy might have wanted to read Galton's original work more carefully. Your article is predicated on the idea that Galton himself did not calculate the mean of the ox-weighing guessers, and that it was Karl Pearson's "retelling of the ox judging tale" that was the "starting point" for my use of the Galton story. You then point to my telling of the story (and others' uncritical repetition of it) as evidence that "mistakes, once made by experts, will be widely diffused and persistent."

But I didn't make the mistake. You did. Galton did, in fact, calculate the mean, long before Karl Pearson. Galton's calculation appeared in Nature, Vol. 35, No. 1952 (3/28/07), in a response to letters regarding his original article. One of the correspondents had gone ahead and calculated a mean from the data that Galton had provided in his original piece, and had come up with the number 1196. Galton writes, "he makes it [the mean] 1196 lb. . . . whereas it should have been 1197 lb."

The fact that you wrote an entire article on Galton (and on my use of him) while somehow managing never to come across this passage and, apparently, without reading the footnotes to my book (where Galton's response is cited), is absurd in its own right. But it's even more absurd when you consider that your paper is about the the diffusion of errors by experts who "pass along false information (wittingly or unwittingly)," and is itself based on false information.

(It's also bizarre that you caution that "The expectation of being careful
seems to substitute for actually being careful," and yet you somehow were unable to figure out how to spell my name correctly.)

I'm happy to enter into a discussion of whether the median or the mean should be used in aggregating the wisdom of crowds. And there's no doubt that Galton himself thought that the median was, on the whole, a better choice. But my use of the ox-weighing story wasn't intended to be -- and didn't pretend to be -- a discussion of which method Francis Galton thought was more correct. It was intended to be an illustration of the power of group judgments -- and it was an accurate illustration of that. Galton did calculate the mean. It was 1197 lbs., and it was 1 lb. away from the actual weight of the ox. The only "falsehoods" being perpetrated here are the ones you've put in your paper.

[James Surowiecki]

With all due respect, you should remove the link to your paper, now that you know that much of what's in it is not true.

[peart]

James, I took last night off to watch the ballgames -- you'll see the paper is now down, thanks for pointing out the error. sjp

[Ray]

Hi. In the future I'm going to keep here links to their sites. But I do not worry about the sites where my link is removed. So if you do not want to see a mountain of links, simply delete this message. After 2 weeks, I will come back and check.