A citable paper is not anonymous.

Based on this, the logical conclusion is that non-anonymous review is incompatible with acceptance of papers posted on arXiv and other social media etc. Please think logically, making wrong decision on this will seriously endanger CL’s reputation as a field.

I imagine that our proposal (reviewers’ names revealed to each other at the end) won’t help in this case, because someone who feels that way will likely worry that the person they’re disagreeing with is powerful. Would you agree?

Re benefitting reviewers — in a sense we are trying to figure out ways to benefit reviewers, to entice better reviewing/more effort out of them, because they get ‘more’ for it. But I can see that maybe this isn’t an effective move in that direction.

I strongly disagree with one of your premises, though — while there are overt/conscious acts of bias, there is also unconscious bias (which tends to favor well-known researchers, well-known labs, and people in dominant demographics). While we can, at least to a certain extent, rely on the scruples of reviewers to behave accordingly if we ask them not to go looking for preprints on arXiv or elsewhere, even the most well-meaning reviewers can’t effectively account for unconscious bias.

Maybe edited reviews can be useful, but this puts yet another burden on reviewers.

I think Pullum’s point about publishing the names of accepting referees is to put some pressure against giving favorable reviews to shoddy work. This might be less of a problem at a conference like COLING, where historical acceptance rates are fairly low to begin with, and the problem is more rejection of good work than acceptance of bad work.

I agree that reviewer stress and time compression is a big problem (I definitely suffer from it), and I think the only solution is less reviewing. This requires a community-wide effort and cannot be handled on a conference-by-conference basis. Unfortunately the trend is for each conference to place more and more burdens on reviewers, so a person’s only recourse is to opt out of program committees.

Re reviewers blind from each other: I’ve heard from several people (at least four) that their willingness to disagree with a powerful reviewer (eg someone senior who might review their grant proposals at some point) is often quite close to zero when they know the powerful reviewer will know who they are. FWIW, All those who have mentioned this to me come from underrepresented/historically-excluded populations in the NLP/CL/ML community.

Re signing reviews: I’m not entirely sure what the motivation is here. It’s true I’m more likely to sign my review if I’m confident in it (and probably if it’s positive [and probably also if I’m in some position of authority]) but the general argument seems to get the causality backwards here. In general I’m not a huge fan of “opt in” things because people will naturally opt in iff it benefits them in some way, and I don’t think benefiting reviewers is what you’re trying to solve here.

One related question, not exactly on blindness, is the question of whether reviews are made public for accepted paper. I personally think this is a really nice practice.

One incentive to consider: Promise to recognize excellent reviewers, and stipulate that reviewers must sign at least one review in order to qualify.

Another experiment that would be interesting (not sure if it’s been done): ask authors when they submit reviews to self-assess thoroughness. (This is not necessarily the same thing as confidence: one could have low confidence because a paper is not within one’s area of expertise, yet still read it carefully and give thoughtful feedback.) I would guess that a) stressed reviewers are aware that their reviews aren’t particularly thorough, and b) knowing that self-assessed thoroughness will be taken into account by ACs will incentivize some reviewers to put in more effort.