“The numbers should be part of the decision in my view.”

Definitely.

However, papers with high variance (as already mentioned 4-3-1 or 4-2-2) or papers in the very middle range (e.g. 3-3-3) should be inspected thoroughly by ACs: exact reviewers’ comments, reviewers’ confidence, other reviews by the involved reviewers, and the paper itself.

Simply rejecting a 4-2-2 or a 4-4-1 paper based on their ranks according to the average score is not really a good approach.

Best,
Maja

I don’t see how the argument you gave can also be applied to 4-2-2, though, but also here: Yes, when given conflicting scores, an AC should have a closer look.

Again, I don’t wanna argue at all for that it’s only about scores. Of course, an informed decision is always better than just following three numbers. Rather, my main point is:

The numbers should be part of the decision in my view.

Here is my reason (exaggerated): If we don’t believe in our reviewers, then why should the reviewers put effort into their job? Especially why should they think about reasonable scores? Or the other way round: If we believe in what they write in their reviews, why not in their scores? And if the tendency of the scores is clear (5-4-4, 4-4-4, 3-2-2, 2-2-2, …), why not just follow it then?

Best,
Henning

PS:
Side thing, from an author’s perspective: I like the idea of responding only to the AC, partly because it avoids the stressing worry that a reviewer reduces his or her score afterwards. But for the scores: Before I knew that certain scores most likely mean acceptance (say, 4-4-4) or rejection (say, 2-2-2). Even the latter can help, thinking of re-submission. Now, it seems like I need to worry until the final decision.

PS. I am not an author whose paper got accepted with a score of less than 3 🙂

It is not the case that we asked the area chairs to ignore the reviewers. Nor is it the case that the area chairs were ‘overruling’ anything. The reviewers make recommendations and record their opinion both in the form of text comments and in the form of numerical scores. The area chairs look at all of that, with the perspective not necessarily of greater expertise but rather of the context of their whole area and make recommendations to the PCs. The ultimate responsibility for decisions rests with the PCs, who again look things over (in our case, just for the borderline papers) with the broader context in view. (And still without the author names in view, it should be stressed!)

The assumption here is not that the ACs have more relevant expertise than any given reviewer (though they will in some cases and not in others), but that they have more information. They can see: All the reviews for a paper, other reviews that that same reviewer wrote (were they just generally negative? did they tend to give high scores across the board?), the author response, and the same for all of the papers in their area. Furthermore, we didn’t tell the ACs not to look at the scores, but rather not to start with the papers ranked by score.

I hope this response contributes to the discussion you are aiming to start!

first of all, let me also thank you for all these great insights into the COLING organization process and many thoughful decisions you made within the process. As others have said before, some ideas (such as writing the author response to the area chairs) hopefully stay with the CL community over time.

However, I’d also like to say a word of criticism, because I think it should at least be discussed. My criticism refers to the treatment of the reviewers’ scores suggested to the area chairs, as described in the “window into the decision process” (sorry that this comes a bit late):

– In my view, one main idea of peer-reviewing is that the decision about a submission is shared over multiple people, thereby making it at least a bit more objective. An initial filtering/sorting of papers based on their scores actually supports that this idea is followed.

– Yes, it’s not only about scores. Yes, of course several papers with medium overall scores should be looked at in more detail. And yes, scores depend on the subjective opinions of the reviewers. But after all, also an area chair makes a subjective decision. Agreed, he or she even may be more expert for the whole area – but maybe also not for the topic of the paper at hand.

– I know the area chairs are asked to use the reviews as evidence, but still for me the guidelines you gave on http://coling2018.org/a-window-into-the-decision-process/ sound like you counter the idea of a shared decision, giving the responsibility only to the area chair.

– Naturally, area chairs can generally ignore the reviewers, but now they are somewhat encouraged to do so. And when I read that 27 papers with average score lower than 3 made it (so, 4-2-2, 4-3-1, …), I’m happy for the authors, but I have second thoughts that it’s good to overrule the reviewers. Besides, from a reviewer’s perspective, how much should I care about the given reasonable scores then?

Please notice that this is not meant to complain, but rather to trigger further discussion. It might be that I’m missing something, also seeing that the conducted process was based on experience of others. But I would be glad to hear whether you thought about these things!

Thanks and best,
Henning