Scientific Peer Review versus Science Advice

Both scientific peer review panels and scientific advisory panels involve scientists providing advice. So what is the difference?

The peer review process ideally (in my opinion) involves a group of working scientists providing advice on scientific issues similar to those on which they themselves are working. Often a well structured peer review panel involves scientists with slightly differing specializations chosen so that their combined expertise encompasses all aspects of the issue under review. Of course there are a lot of details that have to be worked out to get high quality peer review, but the process must involve disinterested review by real peers in the sense of scientists with directly relevant scientific experience and expertise.

I am thinking of scientific advice such as that provided by the U.S. National Science Board (which provides its advice to the National Science Foundation) or the President's Council of Science Advisors. In such panels, scientists provide their advice on issues of national policy which transcend their specific scientific expertise. Ideally, such a panel is asked to advise and restricts its advice to those aspects of the policy for which their scientific training, expertise and experience provides a basis for more expert opinion than that of the the normal participants of the decision making process in which they function. Normally the members of such a panel have quite different scientific backgrounds one from another. Nor are they peers of a person submitting a proposal or a product for their review.

People being people, the participants in either a peer review panel or a scientific advisory panel will generally try to provide the advice requested from them, whether they are specially qualified to provide that advice or not. Indeed, they may go beyond their charter to advise on aspects of the issue on which they have no special expertise.

People being people, the participants in a panel may also be wrong, and the farther they stray from their areas of expertise the more likely their advice is to be problematical.

In both cases, panels might best be considered "nominal groups". That is, they will normally not have the time or use processes that allow them to debate to a true consensus. Thus I have found that it is often useful to record all the full range of opinions expressed by either panel.

An under explored area of both peer review and scientific advice is how to construct a process that effectively utilizes the advisory services of scientists incorporating that advice appropriately in decision making.

I note, for example, that the scientists serving on a scientific advisory panel are often quite senior people who spend at best a portion of their time doing research. They are generally people who have achieved a wide expertise in their field, and often are administrators of scientific or educational organizations, who have had responsibilities in their professional societies. Sometimes they have broad experience in public policy, and generally they are better informed on issues of public policy than is the general public. (The advice requested of the National Science Board is often closely related to the member's expertise in science and educational administration, while that asked of the President's science advisors deals with broader issues.)

Thus when science advisors stray beyond the specific scientific issues on which they are acknowledged experts, they may still have valuable insights and opinions.

As one deals with broader issues, scientific advice ought to be combined with the advice from other expert groups (diplomats, economists, military experts, political experts, etc.) in the decision making process. Given how hard it is simply to manage the scientific advice, the coordination of many sources of expertise in decision making on broader issues is truly daunting. It is easier to retrospectively criticize failures in the processes used in past decision that led to unfortunate decisions than to prospectively organize processes well for involving scientific expertise effectively for future decision making.