Thoughts on scientific merit review.

A international meeting of senior officials from agencies funding science recently produced a statement of principles for scientific merit review.

I spent a couple of decades managing or advising on scientific merit review, and the basic fact is that the proponents of new projects, whose professional careers depend on getting funded and who seek to identify a great project that they would be happy to implement, spend a lot longer and more serious time preparing a proposal than do the peers in preparing their review.

A second basic fact is that key decisions are made at the margin. It is likely that the best proposal in a competition will be funded and the worst will not be funded, if the process works at all. But say an agency is going to fund 15 percent of proposals. How accurate is the distinction between the 15th and the 16th best proposals? My guess is that few review processes are good enough to be accurate at this level. Indeed, the proposal is at best a poor surrogate for the quality that the project would have if funded.

Consider further, an agency that is planning to fund projects in say four different areas will have to judge not only whether the 15th proposal in one group is really better than the 16th, but also whether it would not be better to shift funds among groups to fund 14 proposals in one or two groups in order to fund more an another group. There will not be true peers of different scientific areas.

A principle that might have been added was to entrust the management of the scientific merit review to well trained professionals -- those who have experience in finding peer experts, in balancing peer review panels, in dealing with borderline conflict of interest issues, in determining whether reviews are adequate for decision making or whether the decision should be postponed to obtain still more information, and in managing the process of obtaining written reviews and conducting in person peer review meetings.

Another important principle of peer review is that there must be an appeal process. Reviewers make mistakes. Review managers make mistakes. Project proponents must have an opportunity to respond to comments on their proposals and appeal unjust decisions.

The effort to develop internationally approved standards for peer review is important, and the report is useful. But there are lots of errors that can be made in implementing a good set of principles.