Skip to content

Peer Review

Peer review as practiced in journals today is somewhat antithetical to the spirit of the web: journals filter then publish, the web publishes then filters. I’ve shared (aka whined) my thoughts in the past on poor peer review and have offered a developmental model of scholarship. Which means I’m predisposed to finding articles like this very satisfying:

Scientists worship at the altar of peer review, and I use that metaphor deliberately because it is rarely if ever questioned. Somehow the process of peer review is supposed to sprinkle some sort of magical dust over a text which makes it “scientific” or “worthy”, yet while we quibble over details of managing the process, or complain that we don’t get paid for it, rarely is the fundamental basis on which we decide whether science is formally published examined in detail.

There is a good reason for this. THE EMPEROR HAS NO CLOTHES! [sorry, had to get that off my chest]. The evidence that peer review as traditionally practiced is of any value at all is equivocal at best…

The author then goes on to present the opposite of what I advocate (my position: publish it all, let people comment on it and filter it through their discussions and citations): “My solution to this is to radically cut the number of peer reviewed papers probably by 90-95% leaving the rest to be published as either pure data or pre-prints.”

8 Comments

  1. Hi, thanks for the comments. I’m not sure that my thinking is really the opposite of what you’re suggesting, maybe different in degree. I think we both are saying “publish everything”, with maybe a different slant on what we mean by “publish” and I’m certainly in favour of much more fine grained filtering.

    But I think there is a value in focussing a close review, both peer-based and editorial, on papers that have something that might be important to say. Still not sure that pre-publication peer review is worth the effort but a refining process of some sort is good. What I’d be really interested in is some proper scientific evidence on which approaches work.

    Thursday, February 11, 2010 at 2:31 pm | Permalink
  2. Mark Nichols wrote:

    Hi George, have to disagree with you here. I think particularly in e-learning circles the emperor is well dressed when it comes to the difference between peer reviewed and not. The sheer volume of material available means there is substantial value in the filtering process… and it leads to a different level of conversation. I edit a journal, and I see some *shocking* submissions… and as you know I am no fan of the ‘wisdom of the crowd’ when it comes to ideas becoming popular. We could end up with Prensky all over again…

    You could also argue that through blogs it is already possible to have open publications, and popular blog posts can spread quickly.

    Thursday, February 11, 2010 at 3:41 pm | Permalink
  3. Mark Bullen wrote:

    Hi George:

    To use another overused expression, I think you’re trying to throw out the baby with the bathwater. Peer review isn’t infallible but it serves a valuable purpose. It ensures that anybody publishing research must reveal in detail their methodology and allow it to be open for scrutiny. This supports the principle of openness by ensuring that authors disclose how they conducted their research. We have a good example of what happens when material isn’t subject to peer review and readers are left to do their own filtering when we see what has happened with the net generation discourse as I think Mark Nichols is referring to. Speculative articles by people like Prensky and proprietary or closed “research” by the likes of Tapscott is accepted uncritically. Peer review isn’t perfect but it does serve a quality assurance role. Let’s not get carried away with the “wisdom of the crowd”.

    Saturday, February 13, 2010 at 1:33 am | Permalink
  4. ted nellen wrote:

    peer review is the second of the three tenets of scholarship. publish it, engage in peer review, and pass it on.

    i agree with your comments of peer review and have been using them since the early 90′s in CyberEnglsih

    well done,

    ted

    Saturday, February 13, 2010 at 7:29 pm | Permalink
  5. Geoff Cain wrote:

    I recently got back a proposal that was said to have gone through a “rigorous peer-review process” and it contained no comments! They didn’t say “no” of course because in this economy they need the registration fees, but I think a peer review process should be part of the conversation, not a way to filter. Right now it is only a good way to impede innovation and ideas that are not yet very well understood. George, I am glad you did not wait for the traditional “rigorous peer review” route before getting your ideas out; the world of education would be far less interesting.

    Monday, February 15, 2010 at 5:13 pm | Permalink
  6. gsiemens wrote:

    @Mark Nichols – I’m quite happy to have you disagree with me :) .

    I agree with you about the shocking number of poor articles that are submitted to journals (based on special editions I’ve been involved with). I guess this comes down to the role that we see peer-review playing. Is it about filtering out “junk”? Is it about improving the quality of articles that are published? Is it about subjecting articles to rigorous critique?

    Do you think peer review would have prevented Prensky’s digital immigrants/natives from spreading/developing?

    Wednesday, February 17, 2010 at 1:08 pm | Permalink
  7. gsiemens wrote:

    @Mark Bullen – great point: “It [peer review] ensures that anybody publishing research must reveal in detail their methodology and allow it to be open for scrutiny.”

    I’m not a fan of “crowdsourcing” or “wisdom of the crowds” as it is commonly understood. I appreciate experts and expertise.

    For me, this centres on a discussion of what the internet enables and what it renders obsolete. NBER asked, a few years ago, whether peer review is in decline (http://www.nber.org/papers/w13272). Notable economists are able to by-pass peer review and disseminate their work via the web. What problems does this create? What will be the damage of this?

    As the Sokal hoax, Elsevier, climategate, and other incidents indicate, peer review does fall short of its ideals at times. My argument is that not only should the methodology/research of the paper author be transparent, but the process of review should be as well. And I’m not saying this needs to be done by blogs or only social media…

    Wednesday, February 17, 2010 at 1:17 pm | Permalink
  8. gsiemens wrote:

    Hi Geoff – the lack of comments is what I find particularly frustrating. Review should not only evaluate a paper, but also recommend ways to improve it. That was my main point here: http://www.elearnspace.org/Articles/journal.htm

    Mind you, Geoff, the other option is that your paper was excellent and no comments were needed!

    Wednesday, February 17, 2010 at 1:19 pm | Permalink

One Trackback/Pingback

  1. Things I learned this week – #7 | dougbelshaw.com/blog on Sunday, February 14, 2010 at 5:31 am

    [...] George Siemens reckons/hopes/is-indifferent-about the end will come to peer-review of journals. I’m not too sure whether that’s a good thing or a bad thing. [...]