[O]nly 8% members of the Scientific Research Society agreed that 'peer review works well as it is.' (Chubin and Hackett, 1990; p.192)I agree that peer review is a problem, but I think it's mostly a problem because we lie to ourselves and to the public about what purpose peer review serves. Peer review is about protecting what Kuhn called "normal science," by which he meant incrementalist, paradigmatic, non-revolutionary progress in understanding the world. Every once in a while, the paradigm shifts and normal science becomes impossible; the old understanding of the world is dead and a new paradigm is established. Kuhn says that scientists who worked in the old paradigm can't even work as scientists under the new paradigm, as their entire way of understanding the world has been undermined.
"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)
Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001) This has been statistically proven and reported by an increasing number of journal editors.
But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).
This is the purpose of The 2nd International Symposium on Peer Reviewing: ISPR 2010 (http://www.sysconfer.org/ispr) being organized in the context of The SUMMER 4th International Conference on Knowledge Generation, Communication and Management: KGCM 2010 (http://www.sysconfer.org/kgcm), which will be held on June 29th - July 2nd, in Orlando, Florida, USA.
Peer review is meant to paper over the cracks and preserve the old paradigm as long as possible. Reviewers are gatekeepers, who allow into our best journals only those papers that sustain the current paradigm. In this way, scientists are trained only to propose and test incremental contributions to our understanding. Eventually, the cracks become too large and the old paradigm crumbles.
Peer review also promotes good methods and good analysis, although not best methods and best analysis. Moreover, methods and analysis are only two values among many. If the theory is interesting or the data is unusual, reviewers will let defects in methods and/or analysis slide. The real problem, though, is that peer review is in no way an audit of the paper or data despite our letting people assume that it is. If data is bad, no reviewer will be able to ferret that out, nor is review of the raw data a routine part of peer review. We assume that the authors did what they say they did, and dealt honestly with the data they found.