New opportunities for replications: Guest post by Abel Brodeur

• Email
• Share
• Tweet
• Share
• Comment

This is a guest post byAbel Brodeur, Associate Professor of Economics at the University of Ottawa.

We are excited to announce the official launch of theInstitute for Replication (I4R), an institute working to improve the credibility of science by systematically reproducing and replicating research findings in leading academic journals. Here are three things we think might be of particular interest to readers of this blog.

1. New opportunities for doing and publishing replications

We are actively looking for replicators and have an ongoinglist of studieswe're looking to be replicated. We have already checked that the codes and data provided by the selected studies are sufficient to reproduce their results (i.e., run the codes). This allows replicators to focus on conducting sensitivity analysis and/or replicating the results using the raw data. Our collaborators provide detailed instructions on how to conduct a replication in theGuide for Accelerating Computational Reproducibility in the Social Sciences. We also developed a template for writing replications which is availablehere. This template provides examples of robustness checks and how to report the replication results.

Replicating a paper could be useful for instructors looking for class projects, and also to junior and senior researchers who want to improve their knowledge of some leading research papers by going through them in detail. Of note, replicators may decide to remain anonymous. The decision to remain anonymous can be made at any point during the process; initially, once completed or once the original author(s) provided an answer.

We will provide assistance for helping replicators publish their work through, for example, special issues. Replicators will also be invited to co-author a large meta-analysis paper which will combine the work of all replicators and answer questions such as which type of studies replicate and what characterizes results that replicate.

2. New resources for teaching replication

If you want to teach replication in class assignments, our team has developed some resources that might be of interest. A list of educational resources is availablehere.

A very useful resource is theSocial Science Reproduction Platform (SSRP)which was developed by our collaborators at the Berkeley Initiative for Transparency in the Social Sciences in collaboration with the AEA Data Editor. This is a platform for systematically conducting and recording reproductions of published social science research. The SSRP can be easily incorporated as a module in applied social science courses at graduate and undergraduate levels. Students can use the platform and materials with little to no supervision, covering learning activities such as assessing and improving the reproducibility of published work and applying good coding and data management practices. Guidance for instructors such as how to select a paper, timelines and grading strategy is availablehere.

Reach out to us if you want to learn more about the SSRP and other teaching resources. We are here to help!

3. A survey of journal editors about replication

Our collaborators (Jörg Peters and Nathan Fiala) and the Chair, Abel Brodeur, surveyed over 50 journal editors in economics, finance and political science about their interest in replications and report their full answershere. Our survey consisted of asking whether they publish comments that discusses and potentially challenge the empirical results from another paper, for example, based on a re-analysis or robustness check. We also asked if these comments were only on papers published in their journal or whether they would also consider comments on papers published elsewhere.

We find the majority of editors answer Yes to the first question, and many also answer Yes also to the second question. However, they often note that such comments are rare, and would need to make a substantive or general interest contribution, which would go through the regular review process. Development economists may be particularly interested to hear the thoughts of Andrew Foster, editor of the JDE:

"I'm afraid I don't have a clear answer that will fit in a box but we do have a philosophy. We do not publish comments per se but we do sometimes publish papers that are submitted as comments. Our point is that comments are a lot of work, create delicate politics of powerful versus less powerful, and generally do little to advance the literature. But every paper is in a way is a comment on the literature and in our experience most submitted comments can be reframed as a stand alone paper that makes a broader point that will be of interest to people other than the two sets of authors. Such papers would go through a reasonably normal referee process. We do not necessarily seek the input (as a regular referee) of the first papers' authors although we will sometimes do that. We would impose the same standard if someone wanted to publish a response to a reframed comment. This approach also tends to dovetail nicely with our new "short paper option". Given this philosophy we are willing to consider these modified "comments" even if they primarily target a paper published in another journal. We see our role as supporting the field of development economics so if a reframed submitted comment would be informative to researchers in the development field then we will consider publishing it after an appropriate referee review. I in fact just accepted such a modified comment"

A second point of interest to development economists is that there is a new companion journal to World Development, calledWorld Development Perspectives. The journal accepts submissions of replications on a rolling basis and a special issue (online) will be compiled once a reasonable number of papers is published.

We will continue developing new and exciting features based on input from the community. Do not hesitate to reach out to us!

https://i4replication.org/contact.html

Twitter @I4Replication