PNNL

The Science 

High-resolution environmental data are crucial to understand and predict biogeochemical and ecological changes, but this generates vast amounts of data. The challenge is in rapidly and consistently processing these vast data streams to make them useful. 

The COMPASS-FME team built a sophisticated open-source processing pipeline that automates data cleaning, restructuring, and metadata generation. This approach enables rapid availability of quality-controlled data at different levels of refinement. It is streamlined for efficiency and performs robustly across large datasets, enabling rapid processing on standard computing infrastructure. This pipeline has generated half a billion observations to date; it enhances transparency, reproducibility, and scientific collaboration, enabling faster feedback between observational efforts and predictive modeling frameworks.

The Impact 

This lightweight software pipeline processes biogeochemical sensor data crucial for ecosystem monitoring and advancing scientific understanding. Robust, transparent and reproducible handling of such data is more crucial than ever to support the new, data-hungry models powered by artificial intelligence and machine learning. The open availability of the code on GitHub aims to increase scientific transparency, be a resource for other groups looking to solve similar problems, and accelerate scientific projects. To understand coastal landscapes and predict the impacts of environmental changes to these landscapes researchers need accessible and open software. A successful pipeline must produce observational data to monitor ecosystems, understand drivers of change, and predict future biogeochemical and ecological shifts in the Earth system. This system empowers research teams to focus on analysis and insight, rather than data wrangling—enhancing both scientific rigor and productivity.

Summary 

This work introduces a reliable, open-source system for processing and sharing high-frequency environmental sensor data from coastal ecosystems. The pipeline is designed to support large-scale monitoring efforts by automating time-consuming tasks like data cleaning, formatting, quality flagging, and metadata creation. Built in R, the system is modular and flexible, using automated checks and version-controlled code on GitHub to ensure transparency and reproducibility. As part of its implementation, the team processed over 500 million data points that had been collected every 15 minutes across multiple coastal sites. The pipeline successfully flagged anomalies, standardized outputs, and enabled long-term public archiving of the resulting datasets. By reducing manual labor and improving documentation, the pipeline accelerates model–data integration and scientific discovery. These improvements significantly lower the barrier to using complex environmental data, allowing research teams to collaborate more efficiently and accelerating insights into ecosystem dynamics and change.

Contact 

Daniel Stover, Environmental System Science Program, daniel.stover@science.doe.gov, (301) 903-0289  

Vanessa L. Bailey, COMPASS-FME Data Management Lead, Pacific Northwest National Laboratory, vanessa.bailey@pnnl.gov, (509) 375-6640

Stephanie C. Pennington, PI, COMPASS-FME, Pacific Northwest National Laboratory, stephanie.pennington@pnnl.gov, (509) 371-6965

Funding 

This research was supported by Coastal Observations, Mechanisms, and Predictions Across Systems and Scales, Field, Measurements, and Experiments (COMPASS-FME), a multi-institutional project supported by the U.S. Department of Energy (DOE), Office of Science, Biological and Environmental Research as part of the Environmental System Science Program (https://compass.pnnl.gov/FME/COMPASSFME). The Pacific Northwest National Laboratory (PNNL) leads this project, operating under contract DE-AC05-76RL01830 through Battelle Memorial Institute for the DOE. This work was also supported by the Smithsonian Environmental Research Center and the University of Toledo.