Comments on: A window into the decision process

By: Emily M. Bender

Emily M. Bender — Sun, 13 May 2018 04:18:38 +0000

Thanks for the reminder. They should be visible to co-reviewers now.

By: Arne Köhn

Arne Köhn — Fri, 11 May 2018 12:20:10 +0000

“Reviewer identities revealed to co-reviewers” should have been a week ago but I can’t see other reviewer’s names. It is not that important but should I look at some unexpected place for the names?

By: Emily M. Bender

Emily M. Bender — Tue, 08 May 2018 22:38:56 +0000

That seems to me like an unnecessary additional burden to put on the ACs, who are already doing a lot of work. The point here is, in the first instance, to run a conference. Yes, we are also looking to learn from what is and isn’t effective in the various approaches we are taking, but we can’t instrument everything and hope to get it done effectively.

By: COLING Author and Reviewer

COLING Author and Reviewer — Tue, 08 May 2018 22:04:14 +0000

I firmly believe that ACs had considered author response into their recommendation. However, I am not convinced that there is no way to quantify the role of author response into final decision. I wish ACs had made & recorded preliminary decisions (solely based on the recommendation by the reviewers ) and then read author responses and updated them.

By: Emily M. Bender

Emily M. Bender — Tue, 08 May 2018 14:47:29 +0000

Going through the AC reports, it is absolutely clear that the ACs took author response into consideration. There’s no way to quantify which were accepted because of that however—it’s not like the ACs made & recorded preliminary decisions and then read author responses and updated them. Also (as noted in this post), there’s no sensible way to turn reviewer scores into a proxy for preliminary decisions.

By: COLING Author and Reviewer

COLING Author and Reviewer — Tue, 08 May 2018 13:09:27 +0000

It is a good initiative and I very much appreciate the move by the organizing committee! (After acceptance notification) I personally feel that COLING should release statistics of papers which are accepted because of AC’s consideration of the author responses (otherwise would have been rejected). It will make the initiative more credible..Otherwise, most of authors will believe that the process is no better than ACL, NAACL (especially this year ACL) where most of the responses (and also direct comments to ACs) go unnoticed.

By: Just another author

Just another author — Tue, 08 May 2018 02:46:29 +0000

That being said, I really like COLING, and I think the organizers are doing really good job this time. Keep it up! Thanks!!

By: Just another author

Just another author — Tue, 08 May 2018 02:44:55 +0000

Well, honestly, it was just weird if reviewers can see each other’s names and scores. Why do they need to see other’s names and scores in the first place? Does the paper’s quality depend on reviewers’ names and scores? It only makes reviews more suspicious and biased.

I think the only thing that needs be revealed to everyone who is involved in the decision making process is the paper itself. Nobody (including AC, reviewers, etc) knows nothing but the paper.

By: COLING Author

COLING Author — Sun, 06 May 2018 15:13:24 +0000

Yes, I agree wit this. When the responses are gone to the ACs, not to the reviewers, the authors are sure that their response will be at least read (I guess reviewers also will pay more attention in this type of process). It is very rare the reviewers read the response, let alone change their mind. This approach should be adopted by other conferences too, before it becomes acceptance in an NLP conferences become a sole random process. I personally also appreciate 1) reviewers will not know each other and (do not see others review and score) as it is in the other conferences. 2) Authors should write the the response before seeing the overall recommendation score. Thank you very much !

By: COLING reviewer/author

COLING reviewer/author — Sat, 05 May 2018 23:51:51 +0000

In my personal opinion, author response to ACs is a really good idea and I hope this would work. So I guess, does a research paper have more chances to be accepted when its overall scores are 5, 4, and 1, particularly 1 with too short and inappropriate review, for instance? I have seen a bunch of these cases from ACL, NAACL, etc., and reviewers have never changed their scores even if authors clearly respond to their points.