Best paper categories and requirements

Recognition of excellent work is very important.  In particular, we see the role of best/outstanding paper awards as being two-fold: On the one hand, it is a chance for the conference program committee to highlight papers it found particularly compelling and promote them to a broader audience.  On the other hand, it provides recognition to the authors and may help advance their careers.

From the perspective of both of these functions we think it is critical that different kinds of excellent work be recognized.  Accordingly, we have established an expanded set of categories in which an award will be given for COLING 2018. The categories are:

  • Best linguistic analysis
  • Best NLP engineering experiment
  • Best reproduction paper
  • Best resource paper
  • Best position paper
  • Best survey paper
  • Best evaluation, for a paper that does their eval very well
  • Most reproducible, where the paper’s work is highly reproducible
  • Best challenge, for a paper that sets a new challenge
  • Best error analysis, where the linguistic analysis of failures is exemplary

The first six of these correspond to our paper types.  The last cross-cut those categories, at least to a certain extent.  We hope that ‘Best evaluation’ and ‘Most reproducible’ in particular will provide motivation for raising the bar in best practice in these areas.

A winner will be selected for each category by a best paper committee. However, while there are more opportunities for recognition, we’ve also raised the minimum requirements for winning a prize. Namely, any work with associated code or other resources must make that openly available, and do so before the best paper committee finished selecting works.

We’ve taken this step to provide a solid reward for those who share their work and help advance our field (see e.g. “Sharing is Caring”, Nissim et al. 2017, Computational Linguistics), without excluding others (e.g. industrial authors) who cannot easily share work from participating in COLING 2018’s many tracks.

We look forward with great anticipation to this collection of papers!

7 thoughts on “Best paper categories and requirements

  1. “any work with associated code or other resources must make that openly available, and do so before the best paper committee comes to select works.”

    At what stage are the best papers selected? Should the code be public already during reviewing? Or by the camera ready deadline?

    • Seconded!

      This is a great development, but the community hasn’t yet converged on a good way to share large codebases anonymously. If that’s something COLING is encouraging, it’d be good to have some guidance on how it should be implemented.

      • Thanks Sam! We agree have this has been the subject of long discussions. On the one had, giving people a toolkit for reproducibility makes it easier – on the other hand, we don’t have the resources to support that, and haven’t found anyone yet to commit to it. As a best-effort measure, expect detailed blog posts addressing exactly this topic, before christmas.

    • Great question, thanks Marek! A firm date for this hasn’t been set, but it’s currently expected to be shortly – perhaps a couple of weeks – after notifications and reviews are sent out. This gives a window for those who haven’t yet uploaded code etc. to do so, and for those with organizational constraints to argue their case (internally) for making artifacts openly available using their strong reviews as support.

    • We only have one paper length (approx “long”), but some will end up presented as oral presentations and some as posters. There’s also a demo session, organized/reviewed separately, for which we hope to have the call up soon.

Leave a Reply

Your email address will not be published. Required fields are marked *