[O]nly 8% members of the Scientific Research Society agreed that 'peer review works well as it is.' (Chubin and Hackett, 1990; p.192)I agree that peer review is a problem, but I think it's mostly a problem because we lie to ourselves and to the public about what purpose peer review serves. Peer review is about protecting what Kuhn called "normal science," by which he meant incrementalist, paradigmatic, non-revolutionary progress in understanding the world. Every once in a while, the paradigm shifts and normal science becomes impossible; the old understanding of the world is dead and a new paradigm is established. Kuhn says that scientists who worked in the old paradigm can't even work as scientists under the new paradigm, as their entire way of understanding the world has been undermined.
"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)
Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001) This has been statistically proven and reported by an increasing number of journal editors.
But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).
This is the purpose of The 2nd International Symposium on Peer Reviewing: ISPR 2010 (http://www.sysconfer.org/ispr) being organized in the context of The SUMMER 4th International Conference on Knowledge Generation, Communication and Management: KGCM 2010 (http://www.sysconfer.org/kgcm), which will be held on June 29th - July 2nd, in Orlando, Florida, USA.
Peer review is meant to paper over the cracks and preserve the old paradigm as long as possible. Reviewers are gatekeepers, who allow into our best journals only those papers that sustain the current paradigm. In this way, scientists are trained only to propose and test incremental contributions to our understanding. Eventually, the cracks become too large and the old paradigm crumbles.
Peer review also promotes good methods and good analysis, although not best methods and best analysis. Moreover, methods and analysis are only two values among many. If the theory is interesting or the data is unusual, reviewers will let defects in methods and/or analysis slide. The real problem, though, is that peer review is in no way an audit of the paper or data despite our letting people assume that it is. If data is bad, no reviewer will be able to ferret that out, nor is review of the raw data a routine part of peer review. We assume that the authors did what they say they did, and dealt honestly with the data they found.
7 comments:
Peer review is also used to aid friends and allies and keep out rivals and outsiders.
However, it does have a few good points. It is a way of coercing busy scientists to provide a critical perspective. Ordinarily, diplomacy and laziness would preclude such criticism. Critical commentary is often helpful in improving papers.
Kuhn was wrong.
Researchers regularly embrace paradigm-busting innovations overnight, and the examples are so numerous that it amazes me that anybody still pays any attention to him.
A short list:
Modification of genomes via natural selection.
Relativity.
Germ theory.
Plate tectonics.
There are many more.
Re: plate tectonics, didn't that arise from Wegener's continental drift theory which took decades to catch on?
I don't have the reference handy but Bill Bryson observed in his book on science that new theories generally get adopted only once the old guard dies out.
Ali;
I pondered that point as well, but I let it slide because once the theory was adopted, it was adopted very rapidly. It was very much not a case of the old guard non-believers dying off and being replaced by new believers. The majority of existing practioners switched within a few years.
Bryson is just parroting Kuhn.
It's true, Wegener advanced his theory around 1920, but he didn't offer any conceivable mechanism. As soon as the Lamont-Doherty guys came up with a mechanism (using technology Wegener didn't have), Wegener's theory was accepted overnight.
Similarly with Darwin. There were lots of theories out there about extinction/succession, but they lacked convincing force because they were, as David likes to say, just-so stories.
Descent with modification via natural selection was at least testable through observation, and somebody (forget who, it's been 30 years since I studied this stuff) went back and determined that within 3 years of the publication of 'Origin,' virtually all professional biologists were on board (except the French, who, for pete's sake, are still waffling about Newton).
Curiously, and tending to prove my point, by 30 years after 'Origin,' professionals were entertaining real doubts about natural selection, because some things had been discovered that seemed hard to explain with it.
Like Einstein delving down a level and providing not a refutation of but a more comprehensive understanding of the Newtonian view, the discovery of genes around 1900 restored natural selection.
Whatever we may say about resistance to change, we cannot say scientists, as a rule, maintain their early views until they die.
One living example is Milford Wolpoff of the U. of Michigan, who for over 30 years was the best known exponent of the theory of mulitiple evolutions of Homo sapiens. A few years ago, analysis of mitochrondrial DNA knocked the props out from his theory (based largely on morphology), and he chucked his theory within a week.
I am sure that peer review could work better.
About six months ago, Mike Beversluis posted an article by a (IIRC) a physicist writing about his peer review travails.
It was both funny and depressing.
Of course, as the history of Warmenism has shown, what really works is not "peer" review, but rather "open source" review.
Put everything on the intertubes, and let whoever wants to take a whack at it.
Post a Comment