In this episode of EJIL: Live! the Editor-in-Chief of the Journal, Professor Joseph Weiler, speaks with Yahli Shereshevsky, Michigan Grotius Research Scholar at the University of Michigan Law School, whose article “Does Exposure to Preparatory Work Affect Treaty Interpretation? An Experimental Study on International Law Students and Experts”, co-authored with Tom Noah, PhD candidate in the Department of Psychology of the Hebrew University of Jerusalem, appears in issue 4 of volume 28 of the Journal.
The conversation takes viewers behind the scenes of this experimental study, one of the first of its kind in the international law field, to provide a deeper understanding of the motivation behind the study and the methodology used by the authors. The conversation highlights the importance of the study, not only for its results but principally for its methodology and the potential it reveals for future studies. The interview was recorded at New York University.
New EJIL: Live! Interview with Yahli Shereshevsky on his Article “Does Exposure to Preparatory Work Affect Treaty Interpretation? An Experimental Study on International Law Students and Experts”
Written by Mary GuestIn this episode of EJIL: Live! the Editor-in-Chief of the Journal, Professor Joseph Weiler, speaks with Yahli Shereshevsky, Michigan Grotius Research Scholar at the University of Michigan Law School, whose article “Does Exposure to Preparatory Work Affect Treaty Interpretation? An Experimental Study on International Law Students and Experts”, co-authored with Tom Noah, PhD candidate in the Department of Psychology of the Hebrew University of Jerusalem, appears in issue 4 of volume 28 of the Journal.
The conversation takes viewers behind the scenes of this experimental study, one of the first of its kind in the international law field, to provide a deeper understanding of the motivation behind the study and the methodology used by the authors. The conversation highlights the importance of the study, not only for its results but principally for its methodology and the potential it reveals for future studies. The interview was recorded at New York University.
Share this:
Related
Categories
Tags
No tags available
Leave a Comment
Comments for this post are closed
Comments
Gabriel M. Lentner says
March 11, 2018
Thank you for this!
I find this approach very interesting and I’m grateful for this interview.
However, I think it would be important to discuss two (interrelated) points further (maybe in future works):
How should we assess this study (and similar ones that might follow) given the limitations the authors themselves concede and those identified by the commentators regarding internal and external validity of experimental studies? The commentators raised ‘concerns regarding the internal validity of Shereshevsky and Noah’s experiments’ (pp. 1336-37); they also raise two important issues regarding external validity (pp. 1337-39): can we really generalize the results to real-world situations, ie would the ‘international law experts’ also behave like that outside the experimental setting (and that even though we already identified issues with internal validity)? If yes, under what conditions etc. Then, if we aren’t even sure whether we can generalize to that population, how can we then assume that the real-world population of interest (ie the international judges, arbitrators) behave just like that? In other words, how far is too far?
At what point do these (and other) limitations make a study inconclusive?
This leads me to a second point. Most (if not almost all) international lawyers are not trained in understanding (and interpreting) the (limitations of) findings of experimental studies. There might be a danger that findings are taken as ‘scientific facts’, are not seriously questioned and no one will be interested in trying to reproduce such studies (would journals publish that?).
On a more general note, I thought the problems with external validity was among the reasons why scholarship instead turns to quantitative empirical legal research (see Epstein/Martin, Quantitative Approaches to Empirical Legal Research in Cane/Kritzer (eds) Oxford Handbook of Empirical Legal Research (OUP 2010) 902-5). As they point out, it is simply infeasible to make experiments with those individuals that we want to study (the one exception they cite is an experimental study that recruited 113 bankruptcy court judges to do an experiment to find out whether the race of a party affected the judges' decisions- it found that it did not, Rachlinski, J.A., Guthrie, C., and Wistrich, A.J., ‘Inside the Bankruptcy Judge's Mind’ (2006) 86 Boston University Law Review 1227–65.) The limitations of using proxies seem to be very significant and should not be downplayed. But maybe the publication in EJIL will help to get international judges and arbitrators (and other decision-makers) to participate…
Rossana Deplano says
March 11, 2018
Many congratulations on this fantastic project.
Thanks Gabriel for the insightful remarks on the generalizability of the findings.
Yahli Shereshesvky says
April 11, 2018
Gabriel and Rossana, Thanks. I am sorry that I missed the comments and am replying so late.
Gabriel, we addressed much of the concerns that you raised in our reply to Dunoff and Pollack - https://www.ejiltalk.org/repy-to-dunoff-and-pollack-experimenting-with-international-law/. Specifically, with regard to the 'population of interest', our study was conducted on such population – international law experts. As Dunoff and Pollack recognize, our study is 'an example of ‘best practice’ in which investigators carefully test their hypotheses not only on samples of convenience but also on more realistic elite samples'. We address in our paper and in the reply to Dunoff and Pollack the potential differences between our experts and international judges and arbitrators. In addition, while it is challenging, it is clearly not infeasible to conduct experiments on 'individuals that we want to study'. You can find in our study, as well as in Dunoff and Pollack's reply, references to many other studies that were conducted on relevant expert populations in addition to the study that is mentioned in your comment. I hope that this is sufficient to answer the first point.
As to your second point, we acknowledge the danger of relying on the main findings of a paper, on its “sound-bites”, without taking a proper account of its limitations. Nonetheless, this problem is not unique to experimental work. Almost every scholarly work has weaknesses and limitations. We do not believe that there is a real danger that legal scholars will disregard the limitations of experimental studies (or other empirical work) more than other types of literature that does not adequately discuss its limitations. Numerous international law studies make empirical assumptions or are based on “case studies” without sufficiently supporting these assumptions or explicitly recognizing their limitations. Yet, we do not find much scholarly work that discusses the general merits and pitfalls of other types of international law scholarship. Thus, it seems that the transparency and structured methodology of quantitative studies enables a serious discussion of its limitations and promises and should be applauded for it. We strongly believe in the ability of international law scholars to critically read all types of legal research, and hope that they will address the strengths and weaknesses of all such scholarship with similar rigor.