How not to alienate your reviewers, aka writing a decent rebuttal?

[forwarded from https://nebelwelt.net/blog/20180704-rebuttal.html]

Assuming you have given everything to write the best and most beautiful paper you can ever create, it is obvious that the reviewers must see your points and therefore write you a favorable review with a recommendation of strong accept. Unfortunately, this is not always the case and reviewers may miss some points or misunderstand some of your contributions.

Many conferences have therefore introduced a rebuttal phase that allows authors to respond to the (initial) set of reviews. The rebuttal is an opportunity to clarify misunderstandings, answer questions the reviewers may have, or to expand on a given point the reviewers complained about. There are many different forms of rebuttals with slight twists. Generally, a rebuttal allows you to discuss and clarify certain aspects in a review but it is not intended to add new material, so keep it short and focused.

Reviewing generally is not an adversarial setting and most reviewers are not against you or against your research. Due to the increasing review burden, some reviews may end up being on the short end or not as deep as you would have wanted. The rebuttal is not the time to complain about such reviews. As mentioned above, the rebuttal serves the purpose to clarify and to respond to the reviews. If you must complain about the reviews themselves, consider taking it up with the PC chairs.

Over time, I've settled on the following three step process to write rebuttals, which helps me work through the reviews and to extract the points reviewers raised. I encourage my students to always write rebuttals even if a conference is not using a rebuttal process. Rebuttals allow you to digest reviews and to reflect on your paper from the reviewer's point of view, hopefully identifying the weaknesses and, if the paper is not accepted, improve the paper for the next submission.

Read the reviews

Reading reviews is an art. It is incredibly difficult to read between the lines. Try to identify what annoyed the reviewer: where did they stop paying attention? What is, according to their view, the main issue with the paper? What are the shortcomings? Additionally, try to figure out what they liked and what they think the strength of the paper is. Great reviews also contain a section that highlights the path to acceptance, i.e., what the reviewer thinks needs to change to get the paper accepted. If no such section is present, try to identify what would have helped swing the reviewer in your favor.

Reading reviews can be disturbing. You may ask yourself why reviewers did not get a certain point as it was clearly discussed in the paper. After going through the reviews, it is best to take some time off to digest the reviews, allowing you to regain your objectivity.

Extract the main criticisms, group, and rank

Start marking the main criticisms in the paper. Pay attention to the topics identified in the first phase and highlight them. Scribble over the reviews to highlight individual comments. In this second phase your goal is to identify the main topics that need to be addressed. Creating an outline of these main points can be helpful. As you are working through the reviews again and again, start grouping the comments of individual reviewers based on topics, and then rank the topics according to importance. If multiple reviewers brought up the same points it may be crucial to clarify that aspect.

An interesting question that often pops up is what aspects a rebuttal should focus on. Should the ranking be purely technical, according to reviewer expertise, or according to the review score? For example, is it better to convince a non-expert weak accept to bump up their score or to clarify some issues that an expert raised? I've heard many different approaches and each approach has pros/cons. Also, having seen the process from the other side as a reviewer, I cannot say if any given approach has advantages. In my rebuttals I generally try to address the technical points, not focusing on individual reviewers or experts too much. If an expert is strongly polarizing, it may be worthwhile to highlight some misunderstanding or to keep the discussion of that review short. But these issues quickly evolve into politics and may be for people with more social skills.

The key issue you want to likely avoid is alienating reviewers. Keep sarcasm, irony, and other subtle forms of communication out of your rebuttal and stick to technical facts. Try to clarify technical items and write in a way that gives reviewers a way out to adjust their scores for the better. I.e., instead of writing "reviewer A is a moron who ignored our section 2.1 where we clearly describe the design of our Flubb system" write something along the lines: "In section 2.1 we describe how Flubb satisfies the Blubb assumption. We will clarify these constraints based on reviewer A's feedback." If a reviewer takes the time to note a certain point as part of their review then they felt that this was an issue and it is the author's job to clarify that issue. The reviewer is not wrong but may have been misguided by the paper. Improving your writing will make it easier for the reviewer to digest your points.

Formulating a response

Now that you are clear about the major (perceived) weaknesses of your paper and after you have identified the main topics that need clarification, it is time to write the actual rebuttal. I like to write the rebuttal based on topics and then highlight which reviewers have raised that topic. Note that at a top tier PC, reviewers have 20-25 papers on their stack and reading 25 rebuttals can be taxing, make it easy for them to identify which parts address their points. Also important: stick to the word limit, many reviewers hate overlong rebuttals and I've seen great rebuttals ignored if they were over (I've also seen rebuttals that were 4x the allowed length). It is good tone to start the rebuttal by thanking the reviewers for their reviews and to highlight any general issues such as that you plan to open source your implementation or to give a quick one sentence introduction into the main topic.

After the initial lead in you can dive into the individual points starting with a quick introduction that summarizes the issue or question and an answer. Try to keep the discussion short. You're not writing a new paper but are clarifying some details. The rebuttal is not the place to introduce new topics but you may mention that you have some additional results or to highlight certain trade-offs.

Generally, keep the tone polite. An aggressive rebuttal will rarely be read to end and is not helpful in convincing the reviewers of your case. Snarky comments or insults are not a good idea either.

When you submit the rebuttal, note that HotCRP sends out an email with the rebuttal to all the reviewers. I've received several rebuttals that were heavily modified and received a couple of updates. It is generally in your interest to only submit the latest version, especially if earlier versions are not yet polished.

Edit

Thanks to Nathan Burow for feedback on the article. I updated the discussion of politics and rephrased the outline construction slightly.