PNNL

Abstract

In production high-performance computing environments, data transfer times and queue times on live systems are highly variable. As the ubiquity of high-performance computing environments increases and the compute capacity of these systems continues to grow, there is an increasing need for more flexible scientific workflow implementations that do not require users to learn the details of high-performance computing but also can create applications that can better utilize the high-performance computing resources and flexible enough to provide best time-to-solution results. In this paper, we show the design and implementation of an adaptive scientific workflow that will dispatch batch jobs on a highly utilized cluster and yet provide the best possible batch scheduling decision dynamically.