IMHO, there’ll come a point where eventually all code will go to a point where “Sometimes provided code did not even run”. It’s a natural evolutionary thing where programming languages and dependency libraries gets upgraded faster than scientific code. And most times, our code goes into abyss when we completed our presentation at a conference and the incentive to publish other code for a new project is more attractive than maintaining old code. I guess that’s the nature of science.

When code grows stale then reimplementation is necessary for replication if we think it’s an important code to keep for NLP advancement. And I think documenting the nuggets of knowledge gained from the replication is truly essential since we know the same code would surely grow stale again someday.

I remember ever so clearly one of my first experiences of science, and that was confirming that objects of different masses fall at the same rate. I must have been about 10 years old, and I was absolutely convinced this could not be true. Yet when I dropped a pebble and a much heavier rock they hit the ground at the same time. Over and over again. It was weirdly thrilling, and made what I realize now was a very profound impression on me.

And indeed the experience of working with Antske and colleagues on reproducing some earlier results for the ACL 2013 paper was an exciting experience – at first the fact we couldn’t do that was mysterious and frustrating, but then all of a sudden we realized “ah ha, that’s it…” and I think it was a particularly satisfying moment all around, and hopefully gave us all a bit of confidence not just in ourselves but in our field as a whole.

There is also a related issue what it means to approach someone about reproducing their results (especially if you are having some trouble with that, and even more so if it isn’t someone you know). I think sometimes this is interpreted as an adversarial move, but in reality I think it should be seen as a very high complement that someone is interested enough in whatever you might have done to try and repeat whatever that was. And indeed the fact that Anstske and colleagues contacted me asking these kinds of questions led not just to a satisfying scientific experience, but also to a paper that I am quite proud to have been a part of. And so I think that’s how these efforts can and should go, and I’m very happy the COLING chairs have created paper categories and awards that might encourage more of this kind of work.

I keep hearing it’s a small world, so I’m sure our paths will cross. 🙂 I finally met one of our other colleagues on the ACL paper not too long ago (Marten Postma) and I was very pleased to see a recent paper of his that addresses a reproduction issue in WSD and reaches a very interesting and useful conclusion, so I will close by encouraging anyone interested in these issues to take a look at that (url below), and to share their own examples of papers of this genre that they have enjoyed.

https://arxiv.org/abs/1712.03376

Cordially,
Ted