Dear Dr. O'Gorman,
Concerning your essay "The (Frustrating) State of the Peer Review" from the IAPR Newsletter (January 2008), this is to point you to a similar reading by Donald Geman (in case you were not aware of it):
"Ten Reasons Why Conference Papers Should Be Abolished" http://cis.jhu.edu/people/faculty/geman/
While I agree with many of the ideas you expressed, in particular that good reviewers are overloaded these days, here are some points that I think should be touched if you plan to publish a "2nd edition" of your essay:
1. The peer review process is still the best alternative we have. I believe that the scientific method is the most honest effort humanity has produced to this day to explain "life, the universe and everything", and it largely achieved this through the peer review process, especially in the last three hundred years.
2. Computer Science is different from other sciences in that conference papers are peer reviewed publications. This may come from the high pressure the field is submitted to, both from government, society and industry: grants, money, research positions, they all come at a price: # of papers we publish and where (as an "objective" measure of quality).
3. Many more papers are published today compared to 20 years ago. There are thousands of universities, institutes and research facilities on this planet. They all want recognition from international forums, and this is normal.
4. Although the peer review process is far from perfect, there is a sort of "natural selection" process: the thing that counts is not to have published hundreds of papers, but how many times they are cited. I see this like a sort of impact factor of somebody's work: how much other researchers refer to it.
However, as a reviewer, I tend to agree with you: as the number of submissions increases, it becomes more and more difficult to peer review them. Moreover, the quality of the submissions have decreased a lot, and I think this is a real problem. So many times I accepted to review a paper after reading an abstract only to find later that it was just not worth it. Let me explain: as a reviewer I "donate" to the author of the paper one or two days of my working time, time I could have
spent on my own projects and I feel very frustrated when I see that the paper I committed to review is badly written, the authors did not pay attention to the structure and organization of their ideas, often use incorrect or improper English language (difficult to understand) or simply the ideas presented are trivial or just a slight rework of previous published work. In a word, since there is no penalty, many authors shoot in the dark at conferences, and sometimes get papers accepted. I would very much like that the research community become more aware that when submitting a paper we also ask several persons to spend time to review it. We thus, should pay them with a rewarding insightful paper.
In this regard, I think that the system used at ICCV07 last year, was a good idea: 10 reviewers voted
after only a light reading of the paper if the paper should be considered for peer review or not.
Institut TELECOM - TELECOM ParisTech