Feels like Censorship
I just got informed that my second paper on basic block versioning, an extension of my previous work, has been rejected. Most academics don’t really talk about these things. You probably shouldn’t publicly say that your paper has been rejected, because you want to project some kind of image of never-ending flawless success. The calculated, business-like, aseptic thing to do is to keep quiet, rework your paper, submit it somewhere else, rinse and repeat.
I’m talking about it. I need to let out some steam, express my frustrations a little bit. If that’s a bad career move, well, so be it. I don’t want to spend my life hiding behind a façade, pretending I’m perfect and always cheerful. Living life without ever expressing yourself is a fast path to depression, if you ask me. At the moment, I’m both frustrated and sad. I’ve spent months working on this paper. It was a good paper. Somehow though, it wasn’t good enough. It didn’t make the cut. Better luck next time. Call me cynical, but it is a little depressing considering this conference has an acceptance rate of about 45%. Damn.
I’ve worked with a conference’s program committee before. I’ve had to evaluate a paper about a programming language that consisted of a hand-written AST encoded in XML, with no tool support. I think the paper was 8 pages long. They were pitching this as a revolutionary new idea. This was back in the day of the XML-all-the-things craze. Are you telling me that my latest submission is in the same category as the XML one? I guess when it comes to computer science conferences, you’re either a zero or a one. There is no middle ground. Your idea is either deserving of publication, or piped into /dev/null.
The perverse thing is that this constant stream of rejection discourages exploration. As an academic, you really want your papers to get accepted. Your funding and ultimately your academic career depend on it. I’ve already started to adapt the way that I work. When I started my PhD, I had no idea how the paper game was played. Now, when I have a new idea for my research, I have to ask myself: it this publishable? It’s really interesting, it has a lot of potential, but is it publishable?
To publish your idea, you should craft the smallest possible publishable unit. It needs to be sexy and trendy. It needs to be about JavaScript. It needs to reference as many recent papers as possible, and ideally, point in the same direction as those papers. Contradicting established wisdom is not smart. Suggesting alternatives to the established wisdom is not very smart either. You’re contradicting iron-clad, proven, mathematical facts, which means you are wrong.
The reason why conferences have limited acceptance rates dates back to the days when conference papers were published in these books called “proceedings” which were purchased, printed and shipped in the mail. You couldn’t accept every paper, it wasn’t physically or financially possible. Nowadays, it’s estimated the Google server farms have a total storage capacity rated in multiple exabytes. Conceivably, we could make all submissions to all conferences available on conference websites.
Why do so many computer science papers come without any source code? Because the current practices in our field discourage replication and encourage “massaging” of results. In the spirit of transparency, we could make all submissions available, along with all of the reviewer comments. Maybe we don’t want all papers to be on the same footing. Maybe your paper would get ranked into class A, B, C or D, maybe you’d get some score on a 5 or 10 point scale. Certainly, not everyone could realistically be invited to come and give a talk. Still, is there really a need to silently discard 50 to 90% of all submissions to a conference?
It feels like censorship. When a paper is rejected, it strongly discourages further exploration of that research avenue. You’re telling me that my idea doesn’t deserve to be seen. Worse, you’re giving my academic competitors a chance to beat me to the punch. Science is about proving and disproving things, but it’s also about playing with ideas. In the world of computer science conferences, there’s very little room for disproving anything, and even less room for playing with ideas. We don’t have time for that. The next conference deadline is coming up real soon, and we have funding applications to write. Peer reviews can become peer pressure, a civilized form of hazing.
Fortunately, my paper is already online on arXiv. It’s timestamped. It’s out there. I don’t know if I’ll have time to publish this paper at an academic conference before the end of my PhD, I’m being pressed to finish as soon as possible, and submit more papers. If it gets rejected one or two or three more times, it might never get into any conference. I can at least take some comfort in the idea that some of my research was published, and my latest work is out there. It might inspire someone to explore a similar research direction.
My personal opinion is that academic research in compilers is dying. It’s going to go the way of operating systems research. Why? Because there’s too much infrastructure to build. It takes too long. It’s just not practical to publish about. These days, the game-changing, innovative work in compilers is largely happening in the industry, and it’s being done by people who left academia.
Pointers Gone Wild 
I also feel like I’ve done a lot of interesting work in a similarish area that is also not very “hot”. A major issue I face is comparing against existing work: some of my work is designed for a rigid environment that no other system works in, but there are unique and useful ideas within that work that can generalize. Generalizing my work is non-trivial and hard to justify though. It really brings me down :-(
If publishing in a journal rather than a conference is feasible for you then take a look at PeerJ Computer Science https://peerj.com/computer-science
The editorial criteria in PeerJ fits with what you’re feeling: if the research is valid, then it should be published, even if it isn’t always ground-breaking. As a bonus it is Open Access and indexed where you’d expect with a fairly heavy-weight editorial board.
Publishing an article in a impact-factor journal is the way to get a grant, not to do Science. “Printing” is a notion associated with paper, journals and copyright transfer, not with Science. To do Science, you should just upload what you think (“your pre-print”) to an appropriate server where your peers could genuinely discuss and debate about it, without notion of “accept” or “reject”, which again are concepts associated with industrial chains and not Science. Such a natural way to do is now possible with Internet and will give us back one major thing the current system has taken from us: the pleasure to discuss with peers and share ideas, the pleasure to feel that we do contribute positively to the collective building of scientific knowledge. Open community-wide discussions is obviously also the way to top quality.
arXiv is a good thing to share pre-prints but is not fit to discuss. If you upload it the wrong day, it will have no impact. You may want to have a look at this novel repository : http://www.sjscience.org and http://www.sjscience.org/article?id=46 . It’s up to scientists to start putting their value in something more than impact-factor journals, because change will never happen from the top of this mad and frustrating system.
I agree with the sense of frustration but I would say you are still quite lucky: your research involve somehow JavaScript and it makes it cool and easier to publish than other kinds of research. I think there is definitely a strong influence on focusing on certain topics or adopt certain publication strategies. In general this could be even positive but the problem is that the few people steering the computer science field are not necessarily doing the best choices.
Luckily we have things like arXiv or this blog, which will be probably read more often than most papers anyway, so I cannot agree on the risk of censorship. However there is a strong risk of being forced to leave Academia if someone do not want to “play the publication game” adopting the optimal strategies.
It is also because of this attitude that after getting my PhD I moved to the industry. It seems less artificial to me, there are not all these barriers, it seems a much open world to me.
Don’t blame academics. Here’s how the movie plays. Each time.
—
Academic Reviewer Lambda: “Fine idea, the evaluation could be stronger, but we should accept.”
Industrial Expert Beta: “This is nonsense.”
L: “What do you mean?”
B: “It was implemented in System X five years ago.”
L: “Where is that written up?”
B: “We don’t write, we ship.”
L: “Ok, where is it in the code?”
B: “Look two hundred revisions ago, you’ll find traces. Now we are doing something else.”
L: “How can you expect others to know about something you tried and discarded?”
B: “We are open source. Just look at the code.”
B: “Anyway, the system in the paper, it’s not as good as System X.”
L: “System X is the result of over 200 man/years of work, the paper is done by a team of one.”
—
The outcome depends on Lambda’s persistence and on Beta’s ability to accept that you can’t compare work done by a single person to a large team.
—
For some reason the JavaScript ecosystem is full of opinionated Betas.
—
My first paper had a review that said: “This has already been done in Common Lisp ten years before” … and that was 21 years ago.
> B: “We don’t write, we ship.”
“Real Life” :)
> B: “Look two hundred revisions ago, you’ll find traces. Now we are doing something else.”
We don’t walk, we run and we don’t look back.
There are pros and cons to that. More pros than cons,
Did you see this post by Bertrand Meyer about Nastiness in CS?: http://m.cacm.acm.org/blogs/blog-cacm/123611-the-nastiness-problem-in-computer-science/fulltext
I didn’t. Thanks for pointing it out.