Anonymity and Review

Anonymous review is a way of achieving a fairer process. The ongoing discussion among many in our field led to us examining how well this was really working, and rethinking how anonymity was implemented for COLING this year.

One step we took was to make sure that area chairs did not know who the authors were. This is important because area chairs are the ones putting forward recommendations based on reviews; area chairs are the people who mediate between borderline papers and acceptance, or who assess reviewer ratings to decide if they put a paper on the wrong side of the acceptance boundary. This is a critical and powerful role. So, we should be extra sure that if a venue has chosen to run an anonymized process, the area chairs don’t see paper authors’ names.

This policy caused a little initial surprise but everyone has adapted quickly. In order for this to work, authors must continue to hide their identity, especially through author response to chairs—the current process.

We also increased anonymity in reviewer discussion: reviewers did not and still do not know each others’ identity. To keep review tone professional, we will reveal reviewer identities to each other later in the process, so if you are one of our generous program committee members, you can see who perhaps wrote the excellent review you saw, and also who left the blank one—on submissions you also reviewed.

It’s established that signed reviews—that is, those including the reviewer’s name—are generally found by authors to be of better quality and tone. We gave an option to reviewers to sign their reviews. This time, 121 reviewers used this, out of 1020 active review authors (11.9%).

On the topic of anonymity, there have been a few rejections due to poor or absent anonymization. To help future authors, here are some ways anonymity can be broken.

  • Linking to a personal or institutional github account and making it clear in the prose it is the authors’ (e.g. “We make this available at”).
  • Describing and citing prior work as “we showed”, “our previous work”, and so on
  • Leaving names and affiliations on the front page
  • Including unpublished papers in the bibliography

Some of these can be avoided by simply only referring to one’s past literature in the camera-ready copy, and holding back for review, which is a strategy we recommend. Of course it’s not always possible, but in most of cases we saw, refraining from self-citing would not have damaged the narrative and would have left the paper compliant.

The final step in the review process, from the author side, is author response to chairs. Please remember to keep yourself anonymous here—the chairs know neither author nor reviewer identities, which helps them be impartial.

3 thoughts on “Anonymity and Review

  1. Not allowing area chairs to see the author names might be one of the best changes being made at COLING this year. It makes the decision process much simpler and likely fairer from area chair’s perspectives.

    Another good change is adding the explicit categories of papers (survey, reproduction, etc.). It definitely helps to remind the reviewers to be more open-minded of various type of work.

    • Yes, this was disallowed by the ACL policy, which we are following (see the call for papers, for example). We extracted the 30 days prior preprints from various places and ran this through a secure private instance of some duplicate detection software alongside the submitted manuscripts.

Leave a Reply to Leon Derczynski Cancel reply

Your email address will not be published. Required fields are marked *