My Paper was Rejected… Again
A month ago, I submitted an updated version of my basic block versioning paper to an academic conference (the previous version can be viewed here). We have an improved system, more working benchmarks and better results across all metrics. I was quite proud of this paper, and thought our odds of getting into the said conference were quite high. Unfortunately, the improved paper was again rejected. Comments from the reviewers were mixed. Some quite positive, others extremely negative and harsh. Notably, the paper was shot down by a famous industry researcher who has done extensive work on trace compilation, a competing approach.
Below are some excerpts of the comments from four different reviewers:
The paper is well written and the examples help to understand the tradeoffs of the approach. Overall the approach seems to be sound and deserves further discussion/publication.
I fail to see the difference to trace compilation (and the predecessor of trace compilation, dynamic binary translation) […] Constant propagation and conditional elimination in a trace compiler lead to the same type check elimination that you present.
I have had a peek at the code and it very elegant. The paper is well written too. The paper has some weaknesses but I don’t believe that they are show stoppers. […] Overall this is a nice idea well worth looking at in more detail.
The one big problem I have with the paper is that it does not motivate and put into context lazy block versioning properly. The paper needs to do a better job at explaining which precise problems of current Javascript JIT approaches that are used in production are solved by lazy basic block versioning.
In this improved paper, we’ve managed to show that basic block versioning reduced the number of type tests much more effectively than a static analysis could, and that our approach was able to yield significant speedups. We’ve also addressed one of the most important criticisms of such an approach by showing that it does not result in a code size explosion in practice, but only a moderate code size increase. This, however, wasn’t sufficient for some reviewers. The main criticism they raised is that basic block versioning has similarities with tracing, and that as such, we should have compared its performance against that of a tracing JIT.
I have no issue with writing a detailed comparison of both approaches on paper, but it seems the only way to compare the performance of basic block versioning against that of tracing in a fair manner would be for me to implement a tracing JIT compiler in Higgs. Not only that, I’d probably be expected to implement a tracing JIT with all the latest developments in tracing technology. This could well require an entire year of work. I feel like there’s an unfortunate disregard for the amount of infrastructure work required to do compiler research. It’s becoming increasingly difficult to do academic research on compiler architecture. There’s a high amount of pressure to publish often, but at the same time, conferences expect us to pour several man-years into each publication and to immediately provide an absolute proof that ideas brought forward are better than everything else out there. Curious exploration of new ideas is systematically discouraged.
I was understandably feeling quite frustrated and discouraged after this second rejection. This blog post by Daniel Lemire lifted my spirits a little. It’s clear in my mind that basic block versioning has potential. We have something clearly novel, and it works! No matter what, I can at least take some comfort in the knowledge that basic block versioning has already been presented at several industry conferences, such as DConf, mloc.js and Strange Loop.
I may not be doing very well at the publication game thus far, but thousands of people have watched my talks online. The idea has already been spread around, and I’ve gotten positive responses. I haven’t put this much work into Higgs and my thesis to just give up. I’ve already found two conferences where I can submit further updated versions of the paper, and the deadlines are luckily coming up soon. In addition to this, I’ve already started work towards implementing typed shapes in Higgs, which should lead to another publication next year.
Pointers Gone Wild 
I read the early version of the paper you linked and in my mind there is no need for a comparison to tracing as your approach is focused on profile-driven generation of type specialized variants within a traditional SSA-oriented compiler. I am not terribly familiar with the latest in the tracing world, but it doesn’t seem like the problem you’re solving is the same problem they’re solving. As the reviewer said, the type specialization comes “for free” with tracing, but then you’re stuck with the downsides of tracing you pointed out such as needing to reconstruct the control flow graph from the traces.
In some ways I think you’re both trying to achieve the same goal, but through opposite approaches – your approach starts with inlining a traditional SSA representation to create blocks that can be usefully specialized; the tracing approach starts from traces that must be outlined in order to reconstruct a flow graph (and avoid trace explosion). Your approach further intrigues me because of the possibility of using it in a profile-directed optimization mode to generate meaningful source-level annotations that could be inserted in the code to bypass generation of uncalled block versions in future runs, something I think would be more difficult in tracing.
The level of fighting between academics who favor different approaches or ideas is one of the main reasons I left academia. A colleague of mine who has been finishing his PhD has been suffering this as well, so it sounds like the problem hasn’t gotten any better. It seems like the main “solution” is to find a friendly venue for the publication to bypass the reviewers who want to diss anything that isn’t their religion, but this just leads to even more of a silo problem.
Many years after I left, I read Olin Shivers’ advice on writing dissertations and it occurred to me that many of the issues I had with conference reviewers could have been avoided had I followed his advice. Perhaps there’s a way to focus the abstract and introduction on a clear thesis that can’t be rejected or disputed by a comparison to tracing. But I haven’t read the newest version of your paper, so please don’t take this as a critique of something I haven’t read :-)
I haven’t put the most up to date version online because we’re resubmitting it very soon. I thought it was a very nice improvement over the first version, however. The results are much, much better, we show improvements in execution time, and we worked on improved examples/illustrations to better explain our approach.
I’d be curious to read Olin Shivers’ advice. Do you have a link to a blog post?
Olin’s advice is here: http://www.ccs.neu.edu/home/shivers/diss-advice.html
Perhaps you have already accomplished this in the new version of your paper – which I look forward to reading. It was merely my own reflection that the advice to focus the thesis and repeat it as necessary to demonstrate the point would have helped with some of my reviewer issues – and later was very helpful in a corporate research context.
As someone who got a PhD and left Academia immediately after I would say that this blog and the code you are producing are far more important than some additional publications. Counting publications is a very stupid way to count the points and once outside it because so perfectly clear that the real world does not really care about academic papers :D
So keep posting and coding and do not lose too much time doing useless thing to satisfy some old, unfair reviewer.
I’m currently finishing up a PhD. I’m dealing with into the same thing. What a stupid system. What a waste of time, effort, money, and sanity. I don’t know of any solution other than to leave (hopefully with the PhD though). Good luck, I hope it works out for you!
I am getting a little frustrated at how many barriers are in the way of research in the academic world. It’s pretty clear to me that I don’t want to continue on the academic path after my PhD. I like to try and innovate without having to constantly worry about what is or isn’t publishable and what’s the smallest publishable unit we can try and quickly get out the door.
Academics are often pretty conservative and are generally encouraged to have strong opinions and group into cliques. This makes innovative research extremely hard to publish, especially when some other people already worked on the problem you are trying to address with a different technique. Another problem is the quality of reviews, a recent paper of mine (in PL) got rejected from a prestigious conference with very short reviews filled with typos, misunderstandings, and plain wrong statements. Your area is harsh, because, as you phrase it, the infrastructure necessary for experiments is huge, I think your work is interesting (if only because it is alternative) and deserves publication. You should really be proud that industry people receive your work positively, they are often interested by actual results; as opposed to some bad researchers that spend most of their energy making their ideas the state of the art by simply denying to consider other’s.