Computing testbeds offer experimental platforms for testing advanced technologies, driving innovation in computing performance and scalability.

Table of Contents

• What are advanced computing testbeds?
• Role of advanced computing testbeds in science
• Advanced computing testbeds at PNNL
• History of advanced computing testbeds
• Why do advanced computing testbeds matter?
• Limitations of advanced computing testbeds
• Future applications of advanced computing testbeds

What are advanced computing testbeds?

Computing testbeds are the proving grounds of new technologies, a critical process for developing next-generation supercomputers before they become available in the marketplace. Such advanced computing has helped us understand and protect our environment and secure our nation, transforming the way we live and touching nearly every aspect of our lives. It is so fundamental to our relationship with the natural world that scientists consider it to be the third pillar of research, alongside theory and experimentation.

Pacific Northwest National Laboratory (PNNL) is a key player in Department of Energy (DOE) investments in advanced computing testbeds, bringing extensive expertise to frontier research and the computing challenges facing the nation today.

Role of advanced computing testbeds in science

Testbeds allow researchers to explore complex design spaces and to evaluate new computing technologies for performance and efficiency on critical scientific workloads. Testbeds give computational scientists an opportunity to refine their algorithms and to develop new tools. They also allow commercial partners to tailor their designs. The value of testbeds is enhanced if they are available early in the design cycle before designs are committed to manufacture. Having a chance to collaborate on future designs helps to bring more efficient and higher performing designs to the scientific community. These “laboratories of machines,” in which multiple components are available for experimentation, are crucial for the next great advancements in computation.

Advanced computers, which cost hundreds of millions of dollars and take years to plan and build, can be constructed and programmed in countless ways. The most effective among them require fundamentally new answers to questions involving application algorithms, programming models, system architecture, component and device technology, resilience, power, and cost.

This “building the machine” requires both theory and experimentation, making advanced computing testbeds ever more essential. They give scientists, engineers, and designers an opportunity to evaluate their theories and see whether their vision of its programming and architecture will deliver reasonable performance and efficiency. This type of experimentation is akin to the push and pull between theoretical and experimental physics: a scientist imagines what type of system might work, but there is distance between theory and reality. 

Advanced computing testbeds at PNNL

Critical to this effort was the Center for Advanced Technology Evaluation (CENATE) at PNNL. CENATE’s vision was to serve as a “neutral broker” for the evaluation of emerging computing, networking, and memory technologies, allowing the laboratory to independently assess the applicability of these technologies to key scientific computing workloads. Funded by the DOE’s Office of Science, CENATE evaluated both full systems and system components, including pre-production technologies that could pave the way to larger-scale production systems using metrics of interest such as performance, efficiency, and security. Through CENATE, PNNL and the DOE were able to strengthen technical acuity in emerging computing paradigms and build key relationships with commercial technology developers. This led to the deployment of production-scale systems serving the computational science community.

The experience gained from CENATE led directly to the following projects: 

• Advanced Memory to Support Artificial Intelligence for Science (AMAIS)
• End-to-End Co-design for Performance, Energy Efficiency, and Security in AI-Enabled Computational Science (ENCODE)
• Democratization of Co-design for Energy-Efficient Heterogeneous Computing (DeCoDe).

PNNL is now evaluating computing testbeds using dataflow architectures, analog computing, in-memory and near-memory computing, and reversible computing. These point to potential future directions that high-performance computers may follow to address bottlenecks in performance and efficiency. 

History of advanced computing testbeds

PNNL is leading the next generation of computing for scientific discovery.       
Explore our Computing & AI story

The research and development of even the oldest “modern” computers—those built during World War II for complex mathematical calculations related to ballistics for artillery—required testbeds.

This continued through the next iteration of computers in the 1950s and ’60s as these machines moved from mechanical to electronic devices. It was around this time that computers expanded their reach into the business world, where they were used to forecast business models. Early devices were programmed machines that relied on stacks of punch cards. Later innovations in software—specifically, the creation of programming languages—spurred rapid change, drastically expanding computers’ capabilities.

The modern era brought about Seymour Cray and his vector machines, which made their debut in the late ’70s and early ’80s. These highly specialized computers were particularly valuable in scientific discovery because they could efficiently “vectorize” loops of instructions—rather than applying a set of instructions to each data point individually, sets of instructions could be applied to sets of values. This allowed for rapid processing over data vectors and proved useful in the area of fluid dynamics and, later, climate modeling.

The ’90s brought a new idea in computing: linking multiple devices to one another, increasing their power. The idea was realized through the development of petascale computers, those capable of calculating at least one quadrillion calculations per second (technically known as floating point operations per second [FLOPS]), or 1 petaFLOPS. The first “petascale” computer was the Roadrunner machine developed by IBM and deployed at Los Alamos National Laboratory in 2008.

Why do advanced computing testbeds matter?

Advanced computing has proven essential to the development of more accurate predictive models, allowing for a far better understanding of local weather patterns. This new information is improving our ability to predict severe weather and manage the impacts.

It has also been used to accelerate computational chemistry calculations, discovering new types of catalysis for energy storage. In some cases, this means refining already known processes, while in others, it means looking for more revolutionary practices.

Advanced computing has also greatly enhanced our understanding of nuclear weapons. No physical testing occurs anymore—all is completed through simulations. It has also proven critical to protecting major infrastructure, including the national power grid, which consists of more than 7,300 power plants and 160,000 lines of high-voltage power lines. Models allow us to anticipate demand and prevent cascading blackouts.

Advanced computing’s success and expanded capabilities will only continue to shape our future, and it remains an essential focus of governments and institutions around the world.

The evolution of advanced computing testbeds

One way to envision the effort needed to improve computing power and efficiency from one device to the next is to imagine a freeway: when we change a two-lane road to an eight-lane highway, we have to do more than simply quadruple the amount of material we use. At some point, it would become unwieldy—a driver might have to cross six lanes to make an exit.

Likewise, continually linking computers together will cause similar problems. At some point, it would be too difficult to connect them efficiently, effectively, and sustainably. At least one of the modules or network cables would likely fail, causing the application to crash.

Advanced computers have a million components. There are roughly a billion threads—a stream of instructions that tell the computer what calculations to perform—in each of them, meaning many opportunities for breakdown. One or two might fail every hour. Sometimes, these breakdowns are silent, leading to undetectable corruption. At the other end is a fatal error that causes a program to crash or some part of the machine to shut down, all of which can greatly interrupt scientific experimentation.

New devices must account for all of these concerns. Not surprisingly, the leap to exascale computing—systems that can perform at least one quintillion floating-point operations per second—moved away from simply stringing devices together. Exascale looks back to the earlier notion of creating highly specialized machines born to complete a specific set of tasks, with each looking and operating differently with special components and strengths.

Advanced computing testbeds give system architects and domain scientists a vehicle for codesigning the hardware and software components included in next-generation systems. Once one of these large-scale systems is deployed, the only way to make sure the delivered performance matches expectations is for software engineers to modify their code or algorithms to conform to the capabilities of the machine. At that point, there are not many degrees of freedom left. 

Testbeds and computational science

With testbeds, architects and domain scientists can work together to make sure the machine delivers the capabilities the algorithms need, while algorithmic changes can be completed simultaneously to better utilize the machine. The design of the software and hardware is thus made “in concert.”

Testbeds can be small scale, so numerous options and configurations can be explored. This includes using specialized hardware, such as GPUs. (GPUs, which were primarily used to develop games, have since been adapted for scientific computing.) Testbeds also provide the ability to explore novel and disruptive technologies without major cost, so designers can consider multiple generations of computing technologies in their designs.

No matter how a new computer is constructed and programmed, energy consumption is a major issue. Some of the world’s most advanced computers use 30 megawatts of power, and a single megawatt would be enough to power 400 to 900 homes.

Of course, energy costs are not the only consideration. They are in addition to the price tag for the device itself plus that of operational staff, application development, and other expenses. Testbeds help scientists account for these costs. 

Limitations of advanced computing testbeds

There are many kinds of advanced computing testbeds. Some explore different processing and methods, while others focus on the computer’s memory and its role in computation. Some memory technologies, for example, can store data when the power is off. Certain memory technologies favor speed (low latency), others high bandwidth, high capacity, and persistence (data remains when powered off).

Networking is another critical issue—namely, how will the data be moved? Data movement is both slow and power intensive and is widely considered one of the most important problems to address. It calls for an incredible amount of hardware—wires and transistors—that take up a large amount of physical space. Setting aside quantum computers, there is only a finite amount of space for these materials.

Methods for moving data have improved incrementally, but the best techniques are reaching the speed of light. As a result, questions about data movement also immediately raise questions about machine architecture, programming models, and algorithms. Can we mix compute and memory? Can we write programs in a way that minimizes data movement? What is the best way to reason about data movement?

Addressing data storage problems

Data storage is another sticking point and comes with trade-offs relating to the computer’s speed, capacity, persistence, resilience, and overall agility. And this is where testbeds come into play.

A processing testbed is essentially a machine room with perhaps 20 different processors upon which scientists can search for strengths and weaknesses. Multiple components, including processors, memory, storage, networking, and system control mechanisms such as operating system modules, execution runtimes, programming models, algorithms, and workloads, are among the variables.

Though they have enabled significant progress, advanced computing testbeds are not without limitations and challenges. There is always a danger in extrapolating any experimental data collected beyond the conditions in which it was mined.

This could be, for instance, collecting data on a hardware/software testbed that has significant changes from a final product, or executing an application workload that is different from what will occur in the real world.

This is particularly relevant for high-performance computing (i.e., large-scale computing) because the testbeds are typically “small scale”—often too small (either in terms of available memory or processing power) to perform the full-scale computation.

In advanced architectural testbeds, it is often the case that both the software running on the system and the system itself are evolving and being developed concurrently. Thus, understanding where the boundaries or limitations lie is a current challenge.

For more uncharted technologies, such as quantum computing, scientists must “simulate” the system because there are likely no existing machines with which to experiment. In these cases, scientists must take great care to assure that the simulation tools they develop capture all salient characteristics of the final machine.

Future applications of advanced computing testbeds

Today, there is no consensus on post-exascale advanced computing. The increasing importance of very large datasets is changing the problems that advanced computers solve.

New applications frequently combine traditional scientific computing (simulating physical systems with numerical methods), large data analytics, and machine learning (ML), a branch of artificial intelligence (AI) and computer science focusing on the use of data and algorithms to imitate the way humans learn.

ML represents a new class of computation different from scientific computing. Scientists are not just accelerating the performance of scientific computing, they are converging it with ML.

One possibility for advanced computing is that rather than emphasizing zettascale (exascale x 1,000), the solution space becomes more fragmented and involves customization for different problem domains. Common categories include ML, data analytics, or quantum.

There is much interest in customizing compute units, making advanced computing testbeds ever more in demand.