Scientific Publications Change Credibility of Hypotheses

There is an article in the science section of the New York Times on the reproducibility of results published in scientific journals. I think that it is based on a misperception of the role of scientific publications.

Let me start with an example. Times of sunrise, sunset, high tide and low tide are very accurately measured and predicted. If a scientific journal were to publish an article on the measurement of such events, those results could be reproduced widely and easily. The science is well established, its credibility assured. Thus no scientific journal would now publish such results (unless of course, the rotation of the earth suddenly changed in an unexpected way).

So suppose a scientist comes up with a hypothesis that the sun rises not in the east, but in the west in a specific place on earth. That is a very surprising hypothesis, one that seems very likely, a priori, to be wrong. He goes there and measures the location of the sunrise for a year, and indeed his data show it to have risen in a westerly direction. He submits an article to a good journal; it establishes he is a credible person, and peer reviewers can find no fault in his methods, so his paper is published. Of course, his hypothesis is still judged to be wildly improbable, but others would seek to replicate his observations. If they did, an important lead would then be available for new discoveries for earth scientists. If, as would be likely, it was found that there was an abnormality in the magnetic field or that his instruments were flawed in a way he had failed to detect, and that the sun indeed rose in the east, that would be the end of it. The results would not be reproduced and confidence would be restored in the prior theory.

Scientific journals seek to publish interesting results, those which show evidence for something that was thought a priori to be unlikely. The most interesting experimental results are just those which most challenge the probabilities as estimated before the experiments. It seems reasonable that these anomalous results will not stand up; that published scientific research will be difficult to reproduce. something is unlikely to be true, as any scientist will tell you, an experiment that indicates it is true is likely to be faulty.

This would seem to be especially true as scientists explore things very far from common experience, where the experiments themselves become very difficult. Think of particle experiments at very high energies, superconductivity experiments at very low temperatures, astronomical observations at very long distances, etc.

An experiment that appears to challenge a well established idea, doing so by achieving very unusual and difficult to achieve experimental conditions, may be very suggestive of new ways to understand the world. Such an experiment would also seem likely to be an artifact

•  a set of observations that would occur only very improbably, but which by the laws of random chance occurred in a specific experiment
• a methodological error, difficult to detect, that fell upon a reputable researcher. 

That is why replication is important, and indeed why scientific integrity is important. Often the supporting or challenging evidence will be obtained by alternative experiments, done with different equipment, and done in other laboratories.

So irreproducible results are not indicative of something being wrong with science or scientific publication, but are a natural result of the use of scientific journals to direct attention at potentially important findings as the frontiers of science advance into more and more difficult experimental domains.

This incidentally is why the public should not rush off to demand doctors prescribe a new drug based on reading a newspaper report that it had very unexpectedly cured a condition in an experiment on rats. A single unlikely result leaves the theory still unlikely. Consumers should wait until the FDA, based on the advice of expert, experienced scientists judge that the evidence is then sufficiently credible of safety, efficacy and effectiveness to allow doctors to prescribe the medication.