Learning connection delays with backpropagation in spiking neural networks

Spiking neural networks (SNNs) are appealing for AI because they are particularly energy-efficient when implemented on event-driven neuromorphic chips. Interest in them has surged recently due to a major breakthrough, known as surrogate gradient learning (SGL), which enables training SNNs via backpropagation, thereby solving real-world problems.

We recently demonstrated that SGL enables learning not only connection weights but also connection delays. These delays represent the time needed for a spike to travel from the emitting to the receiving neurons. Delays matter because they shift the spike arrival times, whose synchrony is often required to elicit an output spike. Learning these delays thus significantly enhances the expressivity of SNNs. Although this fact is well established theoretically, efficient algorithms for learning connection delays have been lacking. We proposed a family of delay-learning algorithms based on differentiable interpolation techniques that outperform previous proposals. Our results show that jointly learning weights and delays in fully connected, convolutional, and recurrent spiking layers substantially improves accuracy on several temporal vision and audio tasks, achieving new state-of-the-art performance.

Our approach is already highly relevant to the neuromorphic engineering community, as most existing digital neuromorphic chips (e.g., Intel Loihi, IBM True North, Spinnaker, SENECA) feature programmable delays, and many analog chips can also implement delays through their underlying circuit physics (e.g., BrainScaleS-2, DYNAP-SE2, DenRAM). In the long term, our research should also shed light on the role of myelin plasticity in the brain, which tunes conduction velocities and thus the delays.

Timothée Masquelier
Research Director - AI & Computational Neuroscience