We propose a novel approach for efficient execution of workflows on distributed resources. The key components of this framework include: performance modeling to quantitatively predict workflow component behavior; optimization-based scheduling such as choosing an optimal subset of resources to meet demand and assignment of tasks to resources; distributed I/O optimizations such as prefetching; and provenance methods for collecting performance data. In preliminary results, these techniques improve throughput on a small Belle II workflow by 20%.
Revised: September 23, 2020 |
Published: November 23, 2017
Citation
Schram M., V. Bansal, R.D. Friese, N.R. Tallent, J. Yin, K.J. Barker, and E.G. Stephan, et al. 2017.Integrating prediction, provenance, and optimization into high energy workflows.Journal of Physics: Conference Series 898, no. 6:Article No. 062052.PNNL-SA-129007.doi:10.1088/1742-6596/898/6/062052