PNNL

Inspired by the human brain, researchers use neural networks as models core to machine learning (ML). There are two primary types of neural networks: artificial and spiking. While they require more computing power and time, artificial neural networks are the standard for approaching most scientific problems because they excel at regression tasks.

But what about spiking neural networks (SNNs)? SNNs are more efficient than artificial neural networks—imagine a camera that is motion-activated versus constantly recording—but fail to produce accurate results during regression. A collaborative team of Pacific Northwest National Laboratory (PNNL) and Brown University researchers in the Scalable, Efficient and Accelerated Causal Reasoning Operators, Graphs and Spikes for Earth and Embedded Systems (SEA-CROGS) Center found a way to increase the accuracy of SNNs for regression.

“This is the first real proof-of-concept showing SNNs can be effective in regression,” said Panos Stinis, a mathematician at PNNL. “Previously, people were reluctant to use SNNs for scientific ML. But I think this work shows the promise for novel approaches to using SNNs.”

The paper, published in npj Artificial Intelligence, evaluated the quadratic integrate-and-fire (QIF) neuron model for SNN frameworks. QIF neurons have different spiking dynamics from the standard leaky integrate-and-fire (LIF) neurons, which allowed the team to use different training and optimization techniques. While LIF-based frameworks can be efficient, their predictions often exhibit unwanted jagged and oscillatory behavior that reduces their overall accuracy for regression tasks. This is because spikes are highly sensitive to small changes in the synaptic weights of the neural network or the input spike times. On the other hand, for QIF neurons, these types of small changes only affect the precise timing of a spike rather than whether it occurs or not. As a result, the output of a QIF neuron is much smoother than the input.

When they tested the QIF-based frameworks, the team found that the regression predictions were indeed both accurate and smooth. However, there was a trade-off for the dramatically increased accuracy and smoothness. The framework had lower efficiency than an LIF-based SNN approach.

“We’re optimistic about the next steps,” said Stinis. “What we’ve done is shown that, fundamentally, SNNs can perform accurate regression. Now we are in the process of finding ways to get back efficiency.”

In addition to Stinis, the team included Ruyin Wan and George Karniadakis of Brown University. SEA-CROGS is supported by the Department of Energy, Office of Science, Advanced Scientific Computing Research program.