Neuromorphic computing is a brain-inspired computing paradigm, generally referring to the operation of Spiking Neural Networks (SNN) on neuromorphic chips.

Essentially, neuromorphic computing is a design paradigm driven by algorithms and hardware. With the advantage of low power consumption, neuromorphic computing is also considered a "potential stock" to replace traditional AI.

The understanding of neuromorphic computing should be carried out at the system level, and should not be isolated to only looking at algorithms or only looking at hardware.

The "brain-like" aspect in neuromorphic computing refers to the ability of spiking neurons to simulate the structure and function of biological neurons.

Advertisement

Through this simulation: on the one hand, neuromorphic computing has biomimetic complex spatiotemporal dynamics, and on the other hand, neuromorphic computing can use spike signals to transmit information.The former allows the expressive power of the spiking neuron model to theoretically surpass that of artificial neuron models based on traditional artificial neural networks (ANN); the latter endows spiking neurons with pulse-driven computational characteristics.

When spiking neural networks operate on neuromorphic chips, sparse computation is triggered only when input pulse signals are present. Otherwise, the neurons remain in a resting state. Therefore, to achieve low-power neuromorphic computing, pulse driving is an essential element.

Currently, the field of neuromorphic computing faces such a harsh reality: compared to traditional artificial neural network algorithms, spiking neural network algorithms are far from meeting the performance of the former, and it is difficult to meet the needs of various complex scenarios.

For edge computing scenarios, it is often necessary to meet the requirements of low power consumption and low latency. Once the performance issues of spiking neural networks at the algorithm level are resolved, combined with the advantages of neuromorphic chips, the advantages of neuromorphic computing will be greatly highlighted.

Academician Li Guoqi from the Institute of Automation, Chinese Academy of Sciences, and his team believe that the performance potential of neuromorphic computing has not yet been fully explored.For instance, in terms of neural network architecture, the vast majority of applications in neuromorphic computing currently revolve around the Spiking Convolutional Neural Network (CNN), and the neuromorphic chips available now can only support Spiking CNNs.

In contrast, traditional deep learning has long achieved significant breakthroughs in various tasks with the help of the Transformer architecture. It was not until this team proposed a series of Spike-driven Transformer models that the field of neuromorphic computing integrated the spike-driven paradigm into the Transformer architecture.

How should Spiking Neural Networks and Transformers be combined?

For Li Guoqi, the work around Spiking Neural Networks can be traced back to the work published in 2018. At that time, he was still working in Professor Shi Luping's team at the Brain-like Computing Center of Tsinghua University.

He said: "Professor Shi's team proposed a backpropagation algorithm that replaces the gradient in space-time, solving the basic training problem in the field of Spiking Neural Networks."However, due to the lack of basic programming frameworks, the non-differentiability of binary spikes, and the degradation of deep network spikes, the depth of spiking neural networks was limited to only a dozen layers or so before 2021.

Such a small scale resulted in the performance of spiking neural networks being far behind that of traditional deep learning.

Later, spiking neural networks also began to develop in the direction of increasing depth.

For example, in 2021, the team led by Li Guoqi published a paper at the Association for the Advancement of Artificial Intelligence (AAAI) conference, which solved the deep training problem of spiking neural networks.

After joining the Institute of Automation, Chinese Academy of Sciences, Li Guoqi and Professor Tian Yonghong from Peking University co-published a paper on the open-source training framework for spiking neural networks, SpikingJelly, in Science Advance.

This paper resolved the issue of the absence of training frameworks in the field, greatly reducing the learning threshold for spiking neural networks.At the same time, the team of Li Guoqi and the team of Tian Yonghong respectively proposed two different residual depth spiking neural networks, which have now become the common residual architecture in the field.

These two architectures enable the spiking neural network to reach a depth of hundreds of layers and avoid the problem of spike degeneration, solving the technical bottleneck of large-scale spiking neural network training in depth and scale.

Although the performance gap between spiking neural networks and artificial neural networks has been greatly narrowed, it is still far from enough. The Transformer architecture is a milestone in deep learning and has also attracted the interest of scholars in the field of spiking neural networks.

Since about 2022, research on spiking Transformers has been emerging. These studies basically replace some artificial neurons in the Transformer architecture with spiking neurons.

Some key operations, such as the self-attention operator, are preserved to ensure task performance.These early works have inspired the work of Li Guoqi's team. However, they felt that it was more like a heterogeneous form of artificial neural networks/spike neural networks.

Thus, the research group posed such a question: "How should the combination of spiking neural networks and Transformers be approached to leverage the advantages of both simultaneously?"

After repeated contemplation and continuous discussion, the research group ultimately chose the "spike-driven self-attention operator" as the breakthrough for the problem.

The reason is that: within the field of spiking neural networks, there are currently only two types of spike-driven operators, which are convolution and full connection.

The self-attention mechanism is the key to the success of Transformers, so is it possible to modify the self-attention mechanism to be driven by spikes?After determining this approach, they conducted repeated experiments and ultimately designed some pulse-driven self-attention operators that could function normally.

It was found that the pulse-driven self-attention operators possess many excellent characteristics, such as being inherently linear operators and performing sparse computations.

Once the pulse-driven Transformer could operate normally, they attempted to further enhance performance by improving the architecture.

However, there are so many variants of the Transformer architecture that it is even dazzling.

Thus, they began to ponder: could they design a meta-architecture for spiking neural networks? In this way, the gap between the architecture of spiking neural networks and artificial neural networks could be significantly reduced immediately.Later, the team mainly divided this series of works into two steps:

First step: Proposing a pulse-driven self-attention operator. This is also the third type of operator in the field of spiking neural networks, which allows only sparse addition in the entire Spike-driven Transformer.

Second step: Exploring the meta-architecture of spiking neural networks. Through this, the gap between the architectural design of spiking neural networks and traditional artificial neural networks can be narrowed.

After completing the above steps, they successfully introduced new operators and new architectures to the field of spiking neural networks, allowing neuromorphic computing to take a step forward in task performance while having the advantage of low energy consumption.

The research group believes that if progress continues in this direction within two years, the performance of spiking neural networks will be fully comparable to that of artificial neural networks, and the former's energy efficiency advantage will be more prominent.In the current mainstream visual tasks, natural language processing tasks, and generative tasks, if neuromorphic computing can solve performance bottlenecks at the algorithmic level, it will certainly inspire the design of neuromorphic chips based on new spiking operators and new spiking neural network architectures. At the same time, it is also of great significance for the implementation of low-power artificial intelligence.

Not long ago, the relevant research paper on the above topic was included in the 2024 International Conference on Learning Representations (ICLR 2024) under the title "Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips."

Yao Man, a research assistant at the Institute of Automation, Chinese Academy of Sciences, is the first author of the paper, and Researcher Li Guoqi is the corresponding author.

On the one hand, the results of this study can be used in edge neuromorphic computing scenarios, such as adopting the combination of "neuromorphic vision + neuromorphic computing."

The neuromorphic vision here refers to a biomimetic perception paradigm that perceives changes in brightness in the visual scene through a dynamic vision sensor (DVS) and then only outputs asynchronous sparse event streams.For neuromorphic computing, it inherently possesses the characteristic of event-driven computation, making it highly suitable for processing such sparse event streams.

Recently, the team has also collaborated with a brain-inspired startup to deploy spiking neural networks onto asynchronous sense-compute-integrated chips.

The static power consumption of the chip's processor part is only 0.42mW, and the power consumption in typical neuromorphic vision task scenarios is also below 10mW.

This gives the chip the characteristic of being "always-on," which offers significant advantages in some edge low-power computing scenarios.

If the Spike-driven Transformer architecture can be integrated into the asynchronous brain-like chip, it will not only continue to maintain its low power consumption characteristics. At the same time, with the enhancement of the model's expressive power, it can also be used in more scenarios.On the other hand, this achievement provides technical support for the design of ultra-large-scale networks based on neuromorphic computing.

At present, most large models based on artificial neural networks are designed based on the Transformer architecture. This work integrates the spiking-driven paradigm into the Transformer architecture, bringing a Transformer that operates purely with addition.

At the same time, the operators designed in this work are linear with both the number of input tokens and the feature dimensions. Therefore, the larger the model scale, the more significant the energy consumption advantage of the model.

As is well known, artificial intelligence has now entered the era of large models, and large models are also expected to become the basic service facilities of future human society.

However, with the increase in the number of users and usage frequency, the high energy consumption problem of AI will become an issue that cannot be ignored.Under such circumstances, the exploration of a new generation of linear pulse neural network architectures that integrate brain-like spatiotemporal dynamics becomes particularly important. This also means that this achievement can provide technical support for low-power brain-like pulse large models.

The field of neuromorphic computing is expected to experience significant development.

Throughout the journey, Li Guoqi has felt the difficulty. He said: "No matter for outsiders or insiders, the field of pulse neural networks has always been questioned. Even some of our group members are not very confident because they often see netizens' doubts about this direction."

In response, he also expressed understanding. Although pulse neural networks have the advantages of being brain-like and low power consumption, these advantages can only be reflected at the system level.

As mentioned earlier, compared to the already mature artificial neural networks, pulse neural networks still have certain gaps in all aspects. Therefore, the direction in which the field of pulse neural networks can go has always been unclear.Fortunately, in recent years, the field of spiking neural networks has made significant progress, and his confidence in this area is growing stronger.

He stated: "I personally have an optimistic attitude towards the development of the neuromorphic computing field, and I anticipate that the field of neuromorphic computing will experience significant growth in the next few years."

Especially with the advent of the era of large models, if AI wants to become the underlying infrastructure of human society, it cannot ignore the issue of huge energy consumption.

Therefore, he and his team are very hopeful that this achievement can promote the practical application of spiking neural networks and provide inspiration for the design of the next generation of neuromorphic chips.

Overall, there are still many challenges to be overcome in the field of neuromorphic computing, which requires the joint efforts of the entire field.Based on the results of this study, they will continue to work around the following aspects:

1. Achieving larger-scale spiking neural network models. Due to the complex spatiotemporal dynamics of spiking neural networks, they are more difficult to train than artificial neural networks, which necessitates the design of new training methods to achieve efficient training.

2. Promoting spiking neural networks to more types of tasks. This work mainly focuses on computer vision tasks, and in the future, they also want to try using the designed structures for more tasks, such as long-term sequence tasks.

3. Proposing a brain-like pulse large model architecture based on spiking neural networks. It can be anticipated that this will be a challenging task, requiring systematic breakthroughs in current spiking neural networks in terms of training speed, architectural design, model scale, task performance, and long-distance dependency modeling.

4. Designing hardware computing architectures suitable for brain-like pulse large models. At present, the team has already carried out some explorations around the hardware implementation of this work.If the efficient pulse-driven self-attention operator can be implemented in hardware, combined with the sparsity of large-scale spiking neural networks, more functions will be realized.