Experts make mistakes too

The Economist has an article about a new journal, Rejecta Mathematica. which prints only reports that have already been rejected. I recall early in my career when a couple of friends and I submitted an article and waited for months for a response from the editors. Eventually we were told that the manuscript had literally fallen between two desks, and when the editors found it they had already published another paper that had arrived more recently but was too similar to ours to allow ours to be published.

The Economist reports:

PAUL LAUTERBUR, the father of magnetic-resonance imaging, had his seminal paper rejected when he first submitted it to Nature. Peter Higgs, eponymous predictor of physics’s missing boson, faced similar trouble with Physics Letters. But Lauterbur went on to win a Nobel prize for his work, and Dr Higgs is an odds-on favourite to get one soon. A good, rejected paper, then, is by no means an oxymoron.

We know that a lot of articles that prove to be flawed are published. Scientists look for results that would not occur by chance more than once in 20 or once in 100 times, and there are hundreds of thousands submitted each year. And I am not even mentioning the articles that are deliberately falsified, or in which the authors convince themselves a few inconvenient results must have been due to errors in the experimental setup.

We know that journals love to publish surprising results and tend not to publish negative results, biasing the portfolio of published documents. And we know that people sometimes misinterpret the meaning and implications of their observations.

But sometimes experts are just wrong and offer bad advice to the editors.

I spent a lot of time dealing with peer review of research projects. I came to believe that it was necessary to deal with the individual reviews as pieces of information that modify the probability that the project under review was worth funding. I also came to the conclusion that a good panel discussion aided with the judgments, that people are willing to rate proposals on a scale of value rather than simply giving a pass or fail rating, and that the rating scale can improve the information. And I found it useful to have an outside observer reporting on whether the review had been fair.

Making good decisions ain't easy, even if you have expert advice.