David Levy and I have sketched out a paper in which we check the expertise of the past. We can do that since the data that were used by Francis Galton -- people's guesses about the weight of an ox -- have been preserved. The paper first talks about Galton's use of the data (he chose the median guess as the best estimate of the ox weight) and then about how Galton's results were re-told, and changed over time. Here are a couple of paragraphs from the paper, which looks at this as a case study of the role of expertise:
The question of expertise enters into this because, if experts pass along false information (wittingly or unwittingly), they become part of a process by which errors are diffused. If experts are trusted (and if they trust the experts whose work they cite), mistakes are diffused and remain uncorrected. If, by contrast, expert results are not accepted automatically, then their results are more often subject to the scrutiny associated with replication. Knowing this, an expert whose work might be scrutinized will check the work carefully. An expert who expects deference, acceptance of results without checking, will check the work less carefully. Paradoxically, such trust increases the probability of mistakes being made or passed along. Thus, we expect that the errors made by experts will tend to be systematic because they will be repeated by those who trust them.
Our examination begins with Galton’s contribution on voting as estimation as a way to bound influence. But this is only the beginning of our story. Even more interesting, perhaps, than Galton’s contribution itself, is the “retelling” that has occurred since. Though Galton was defended the median as the estimator for the ox weight, the tale of Galton’s median was changed soon after. In addition to the median location of central tendency, Pearson suggested the mean. Along the way in the historical retelling, embellishments were added so that Galton’s original procedure was entirely misconstrued. Our story recounts the misconstruing. As we do so, a key question is whether the tale was changed deliberately (falsified) or whether, not knowing the truth, the retold (and different) tale was passed on unwittingly.
A pdf of the paper will be available when we fix it.
Here's a scan of some of Galton's data: