To steal from Churchill, peer review is the worst way to make decisions on professional issues except for all the other ways.
I spent a lot of time running peer review, and I came to the conclusion that it is hard to do well. You need people who really understand the issues on which they are asked for opinions, who are willing to spend time and effort to do their jobs well, and who are disinterested, or at least whose interests are declared and known.
The peer review process has to be well managed. People do better in face to face panel meetings rather than mail reviews, but conducting panel meetings involves a lot of work, and heavy demands on reviewer time. I found it useful to avoid pressing for consensus in such meetings. I also found it very useful to have observers in the peer review meeting to evaluate the process and help interpret the advice and decide how to use it.
It is important to recognize that peer reviewers are not giving revealed truth, but only their informed opinion. Think about a probability distribution of reviewer values of a paper or proposal, based on the distribution of such objects. That can be considered an a priori probability distribution for quality. If you get a quality rating from a reviewer, think of Bayes Rule as a means of finding an a posteriori distribution of ratings. Thus the next reviewer may disagree strongly with the first.
We also know that there are all sorts of biases that reviewers have. Some are reluctant to deviate from average judgments, some are unwilling to make other than extreme judgments. Less knowledgeable or less serious reviewers are likely to provide less information in their reviews as to what others might do. I actually found one reviewer who on average disagreed with his peers -- if he said to accept something, it was more likely that others would vote to reject it.
Still, how else are you going to review well complex objects that are highly technical?
Tuesday, May 06, 2008
About Peer Review
I just posted a comment on the info-fetishist blog, and thought I would share it with the readers of my blog since it deals with knowledge and decision making:
Subscribe to:
Post Comments (Atom)
2 comments:
Hi - thanks for the thoughtful comment! I think there is so much to value in peer review - the potential it offers for mentoring and learning that Historiann mentioned, and the ideas about expertise you articulate here both contribute to that value. My concerns come when it's presented to students (well, to anyone, but I see this most often with students) as a critical-thinking shortcut. That because they can identify an article that has been peer-reviewed as such, they can just accept that it is of a certain quality.
That really bothers me for a lot of reasons, but one of the most important is that every time those same students hear about a plagiarized article that got through peer review, or a computer-generated article that got through peer review -- they have nothing else they can draw upon to see the value of peer review.
If it's not the magic bullet they've been told it is - what is it? And with increasingly excellent, non-peer-reviewed, dynamic knowledge bases out there -- today's students don't *have* to take the time to figure out the answers to those questions. So I worry that we could lose something valuable because we're not willing to teach it critically, and talk about it warts and all.
Thanks again!
Thanks amd.
I see your point. Students and the rest of the world should recognize that peer reviewed information is but one kind of information, and it may be wrong as any kind of information may be wrong.
Perhaps more importantly, peer review works when it works because experts have learned ways to evaluate information -- the clarity of hypotheses, the appropriateness of the sample, the quality of the instruments used, etc. We can all use these same criteria to evaluate information. We don't usually because we only need information that is "good enough" and it is not worth while to spend more time and effort getting information of higher quality that one needs.
I have been impressed by the failure of my students in the past to see that they can transfer their information literacy from one field to another. Teaching scientific experts to use their skills to evaluate foreign policy information is an example of something that seems easy but is really quite difficult.
Post a Comment