PNNL

An Open Experiment PNNL’s ‘open experiment’ required every aspect of the research to be documented online, in real time for transparency in every step of the process and reproducibility. The automated analytical pipeline in an open repository offered significant advantages and the process has been increasingly encouraged and adopted by funders, journals, and governments. Enlarge Image.

Results: They broke new ground—and not just for earth science. Using their work on the chemical and biological properties of soil, scientists at Pacific Northwest National Laboratory (PNNL) conducted an ‘open experiment,' in which every aspect of their work was documented online, in real time. Using software tools such as version control, issue tracking, and open-source statistical software, they improved data integrity, accelerated the team's communication and productivity, and ensured transparency in every step of the research process.

Opening the code and data to the world (see sidebar: Transparency and reproducibiity in science), in real time, pushed the team to write higher-quality analysis code and made for a stronger manuscript that was then sent to Environmental Research Letters for publication.

Why It Matters: Researchers in every field of science are being pressed—by funders, journals, governments, and their peers—to increase transparency and reproducibility of their work. A key part of this effort is a move toward open data. The idea is that research data should be freely available for everyone to use and a way to combat post-publication data loss, improve data and code quality, and enable powerful meta- and cross-disciplinary analyses. It is also designed to increase public trust in, and the efficiency of, publicly-funded research. The approach described by PNNL researchers is way to help achieve these goals, and may serve as a model for interested researchers.

Methods: In early 2015, PNNL researchers working in Richland, WA and at the lab's Joint Global Change Research Institute in Maryland, planned a laboratory incubation experiment to characterize the chemical and biological properties of sub-Arctic, active layer soils subjected to changes in temperature and moisture.

This ‘open-experiment' required several elements:

a multidisciplinary team that was not located in one time zone; integrating a variety of different data; performing quality control and diagnostics rapidly, so if, for instance, instrument problems arose the team would lose only the minimum amount of time and data; tightly integrating data, statistical analyses, and manuscript results.

The team designed a data processing and analytical system, written in an open-source and widely used language for statistical computing and graphics, placed in a publicly-available ‘repository' that stored all code and data, making them available in real time.

What's Next? Using an automated analytical pipeline in an open repository offered significant advantages, although the costs of such an approach and investments required should also be considered. This is only one example of many avenues to improve scientific reproducibility and data availability. Elements of this case study might be adopted in isolation, but the researchers offer the entire experiment as an example of individual scientists' decisions and practices having a larger impact.

Acknowledgments

Sponsor: This research was supported by the US Department of Energy's Office of Science, Office of Biological and Environmental Research as part of the Terrestrial Ecosystem Sciences Program

Research Team: Ben Bond-Lamberty, Peyton Smith and Vanessa Bailey, PNNL. The Joint Global Change Research Institute is a collaboration between PNNL and the University of Maryland.

Research Area: Climate and Earth Systems Science

Reference: Bond-Lamberty B, AP Smith and VL Bailey. 2016. "Running an Open Experiment: Transparency and Reproducibility in Soil and Ecosystem Science." Environmental Research Letters 11(8): 084004. DOI: 10.1088/1748-9326/11/8/084004