Comments on: COLING 2018 Best papers

By: Leon Derczynski

Leon Derczynski — Mon, 02 Jul 2018 09:04:08 +0000

Thanks Jason. We tried to capture community feedback last year when structuring the awards, as you link to, and this did impact the final configuration. I share your impression about the theoretical papers – though based on the discussion in comments on the page about paper types, as well as discussions on mailing lists and direct emails, it quickly became evident that interpretation of this category was wildly diverse, and so we couldn’t create recognition under this label and then award it in a consistent manner.

While awards are named after paper types, ACs were briefed to award them as they saw fit. The unconscious bias given by the naming suited well – recognition must be broader than just NLPEE papers.

Regarding changes at the next COLING – the best bet for implementing these suggestions is to run that event!

By: Leon Derczynski

Leon Derczynski — Thu, 28 Jun 2018 21:55:56 +0000

The notifications were sent out today.

By: Ravin

Ravin — Thu, 21 Jun 2018 02:17:12 +0000

When will we know that whether the paper is oral or poster？

By: Jason Eisner

Jason Eisner — Sat, 16 Jun 2018 14:42:56 +0000

Thanks – I like the use of multiple categories. I can’t help noticing that this year’s awards (at least in name) seem to focus on the evaluation or facilitation of existing ideas. Some award categories you might consider in the future:

* Best mathematical model [generative probability model, grammar formalism, etc.]
* Best theoretical result
* Best new algorithm
* Best new problem

as well as other kinds of insight:

* Best generalization, synthesis, or illumination of previous work
* Best exposition of a difficult concept

Followup: The award names may have been misleading (to me, anyway). One thing that I liked about the new awards is that they seemed to be calling out particular noteworthy *elements* of the paper, such as whether the linguistic analysis of the data was exemplary (regardless of the rest of the paper). However, I’m now doubting this interpretation, since I see from https://coling2018.org/best-paper-categories-and-requirements/ that several of the 2018 awards were actually named after the paper types.

* So “Best linguistic analysis” probably just means “Best paper in the category ‘Computationally aided linguistic analysis'”. So the paper’s key contribution might actually have been a new mathematical model. But the award title sounds like it’s rewarding a good manual analysis of linguistic data (as in a linguistics journal) or a good linguistic error analysis of system output.
* Similarly, “Best NLP engineering experiment” probably just means “Best paper in the category ‘NLP engineering experiment paper'”. So the paper’s key contribution might have been a new task or a new algorithm. But the award title sounds like the award is given for carefully comparing the performance of two methods in an engineering setting.
* It’s a pity that there are no categories for theoretical papers (formal language theory, computational complexity, algorithms, etc.), as some commenters already noted at http://coling2018.org/call-for-input-paper-types-and-associated-review-forms/ . One of the NAACL 2018 outstanding paper awards was given to such a paper.