Review statistics

So far, there have been many things to measure of our review process at COLING. Here are a few.

Firstly, it’s interesting to see how many reviewers recommend the authors cite them. We can’t evaluate how appropriate this is, but it happened in 68 out of 2806 reviews (2.4%).

Best paper nominations are quite rare in general. This gives very little signal for the best paper committee to work with. To gain more information, in addition to asking whether a paper warranted further recognition, we asked reviewers to say if a given paper was the best out of those they had reviewed. This worked well for 747 reviewers, but 274 reviewers (26.8%) said no paper they reviewed was the best of their reviewing allocation.

Mean scores and confidence can be broken down by type, as follows.

Score Confidence
Computationally-aided linguistic analysis 2.85 3.42
NLP engineering experiment paper 2.86 3.51
Position paper 2.41 3.36
Reproduction paper 2.92 3.54
Resource paper 2.76 3.50
Survey paper 2.93 3.58

We can see that reviewers were least confident with position papers, and were both most confident and most pleased with survey papers—though reproduction papers came in a close second in regard to mean score. This fits the general expectation that position papers are hard to evaluate.

The overall distribution of scores follows.

29 thoughts on “Review statistics

  1. What’s the graph about? How come more than 900 submissions have a mean score of 5 (only 1000 submissions in total) ?

  2. Thank you for providing the interesting information. Is mean score mean of overall recommendation scores by three reviews??

  3. I just want to thank you, Emily, Leon, for all these great initiatives:
    – Having this “reproducibility track” is really really great !
    – Anonymity at area chairs level is also nice
    – “Minimum author responses” to area chairs looks the right way to go – let’s see afterwards if it went well or not 😉
    Even this blog post (although not so original 😉 ) live during rush time is so nice

    At this pace, I hope next year we’ll have mandatory attached source code to every submission, that would be another nice progress I think 🙂

    Thanks a lot !

    • Thank you!

      We don’t want to lock out those working on unreleasable data (e.g. clinical records), so we can’t really enforce this. What we can do so far – and have done – is to only give best paper awards to papers who have already made code/resource already available, and to run a reproducibility track.

Leave a Reply to Leon Derczynski Cancel reply

Your email address will not be published. Required fields are marked *