Not a Good Day for Science

Bad news. Some MIT students have pointed out that the Emperor has no clothes, i.e. the "peer review" process of science has some gross defects. I ran across this item in Derek Lowe's In the Pipeline website:

A comment here the other day mentioned Scigen, which I hadn’t seen before. Some folks at MIT have whipped up a bit of code and a database of computer science topics, phrases, and graphs, and developed a quick paper generator. The paper will make no sense at all, of course, but it is quick. And what they’ve found is that making no sense isn’t as much of a handicap as you might think when it comes to some conferences and some journals.

Scigen papers have been accepted for presentation at some of the less prestigious meetings, and have been sent to various cheesy journals, which have cheerfully “reviewed” them once details of payment were cleared up. This is not a good sign for your field when total gibberish can be passed off like this, although one assumes that it says more about the sorts of conferences and journals that are accepting these things.

And yes, a comparison to the Sokal hoax comes to mind immediately. That one was even more damning, though, because the gibberish paper that Sokal came up with wasn’t sent to some sleazy fee-generating publication mill, but to what was considered one of the better journals in the field (Social Text). Who (famously) published it anyway. The editors later backtracked by saying that they thought the paper, you know, lacked originality, that it wasn’t well written, that they (ahem!) just accepted it as a favor to a physicist visiting their rigorous area of study, and so on – but the fact remained (and remains) that an editor should be able to distinguish a valid paper from a sticky pile of superglued nonsense.

The reason the Scigen papers aren’t picked up on, clearly, is that no one’s looking at them, at least no one with any knowledge of computer science. The editors and organizers who let them through are interested in collecting the registration and editorial fees first, and after that, well, that’s not really their department.

Hopefully this prank will cause science organizations to police themselves better. Science is a democracy of ideas, and like all democracies, it is the "best of a bad lot". That is, other systems have been tried but show even greater weaknesses. In the long run democracies tend to be self-correcting. So hopefully this incident will provoke the scientific community to police itself more effectively.

I remember reviewing papers for a prof while I was a grad student. Apparantly this is common practice. The prof gets the brownie points for "civic duty" but handing a paper to a grad student means there is no "quality control". I found my prof really wasn't concerned about laying out ground rules or checking up on me. That meant the whole system of "peer review" was pretty weak.