Judging Scientific Information

Steven Wiley has an article in The Scientist explaining the basis for his advice at a recent conference:

Keep current with experimental technologies, and evaluate papers on a technical basis before trusting their conclusions.

Basically, if you don't trust someones methods, don't trust their observations, and if you don't trust their observations, don't trust their conclusions. Wiley properly recognizes that you have to keep up with changing techniques to properly evaluate the methods.

Wiley uses the example of a series of articles by two scientists, a senior author named Racker and a graduate student who apparently falsified experiments. He writes:

Thankfully, fraud this outlandish is rare in biology. What fascinated me the most about the case, however, was the lack of recognition by Racker (and apparently the paper's reviewers) of the technical implausibility of what the authors were describing. Exciting ideas and Racker's past accomplishments apparently blinded him and many other people in the field. The harm to science was minimal, but the damage to Racker's distinguished career was severe.

Of course, all of us ought to be regularly evaluating the results of new scientific research for which we do not have the scientific training to adequately judge methods. "Information literacy" then involves the set of skills involved in warranting information that we are not professionally competent to warrant.

Wiley was surprised, as he should have been, that a paper with a distinguished senior author in a peer reviewed journal should have proven fraudulent. These are good indicators of confiability. Indeed, scientific knowledge system imposes serious sanctions on a scientists demonstrated to have added his/her name to a fraudulent publication or other professional document.

Belief for a scientist is a relative thing. One believes ones own results only after checking them and replicating them. One believes them more after they have passed peer review, and still more after they have been replicated by others. One greets results from others, if they are important, with skepticism, increasing belief when one has replicated them in one's own lab, and still more when they have been replicated by others, and still more when they have been incorporated in a body of theory. Even results that were as well established as Newton's theory of gravity after a couple of hundred years can be reinterpreted and revised when an Einstein comes along.

Indeed, sometimes old, rejected scientific beliefs are revived. The evidence that the rate of increase in velocity of distant galaxies was not constant lead to a revival of an old and once discredited possibility advanced by Einstein, and the resulting theories of the existence of "dark energy" seem similar to the idea that space was filled with a substance known as ether. Thus scientists are (or at least should be) prepared to increase their belief in things that they once believed to be highly unlikely.

There is a good lesson in the scientific attitude toward belief for the rest of us. If information appears to come from a reliable source and is trusted by others we trust, assume that it is probably true, but recognize it may be untrue. Trust but verify. Be prepared to readjust the confidence you have in your beliefs as new information arrives.

Think of betting on a horse race. What is the probability you would assign to each horse winning? What odds would you take to make a bet on your favorite. How sure are you of your estimates? How big a bet would you be willing to make if you got your odds?