From ReLU to Spikes

If you have followed the Computer Vision posts up to Vision Transformers, you already know almost everything needed to understand spiking neural networks (SNNs). This series takes that knowledge and changes one thing at a time:

From ReLU to Spikes (this post): replace the activation with a neuron that has memory and fires spikes.
Seeing in Events: replace the frame camera with a sensor that only reports changes.
Training Spiking Networks: make backpropagation work through spikes.
Counting Energy: why spikes can be cheaper, and how to measure it honestly.
Hopfield Networks and Spiking Attention: attention, seen as associative memory, without multiplications.
Fusing Sensors Without Burning the Battery: putting it together in an always-on wearable that combines an event camera with motion sensors.

1. Why Spikes?

Your brain runs on roughly 20 watts. Training one large language model like GPT-3 was estimated to take about 1,300 MWh (Patterson et al., 2021), the energy a brain uses in about 7,000 years. Brains and GPUs compute very differently, and three differences stand out (Eshraghian et al., 2023):

Spikes. Biological neurons communicate with all-or-nothing pulses, about 100 mV high and 1 ms long. A spike is one bit: “something happened, now.” Multiplying a weight by a spike is not a multiplication at all, just “add the weight, or don’t”.
Sparsity. Most neurons are silent most of the time. A silent neuron sends nothing and, on suitable hardware, costs almost nothing.
Static suppression. Sensory neurons respond to change. An unchanging scene produces few spikes. (Post 2 builds a camera that works this way.)

A conventional network does the opposite: every neuron computes a 32-bit number on every input, whether it matters or not. Spiking neural networks keep the structure of the networks you know (convolutions, linear layers, even attention) but replace the neuron. This is often called the third generation of neural networks, after threshold units (perceptrons) and continuous activations like ReLU (Maass, 1997).

2. The Neuron You Know, and the One That Spikes

An artificial neuron is a function: weighted sum, nonlinearity, done. It has no memory, so the same input always produces the same output, immediately. A spiking neuron is a small dynamical system: it has a state (the membrane potential) that builds up and decays over time, and its output is a sequence of events rather than a number.

3. The Leaky Integrate-and-Fire Neuron

The most widely used spiking neuron goes back to Louis Lapicque in 1907, who modelled a nerve as a resistor–capacitor circuit (Brunel & van Rossum, 2007). The cell membrane is an insulator separating charges (a capacitor $C$ ), and ion channels let charge leak through (a resistor $R$ ). An input current $I (t)$ charges the membrane:

τ \frac{d U}{d t} = - U (t) + R I (t), τ = RC

$U$ is the membrane potential and $τ$ the time constant, typically 1–100 ms. Without input, $U$ decays exponentially back to rest: the leak. Add a rule that the neuron emits a spike when $U$ crosses a threshold $θ$ and then resets, and you have the leaky integrate-and-fire (LIF) neuron.

For a computer we step time in discrete increments $Δ t$ . Over one step without input, the exact solution multiplies $U$ by $β = e^{- Δ t / τ}$ (the simple forward-Euler approximation gives $β = 1 - Δ t / τ$ , which agrees only when $Δ t ≪ τ$ ). Folding the resistance and input scaling into a learnable weight $W$ gives the form used in deep learning:

U [t] = leak β U [t - 1] + input W x [t] - reset θ S [t - 1], S [t] = {10 U [t] > θ otherwise

The reset subtracts $θ$ after a spike (soft reset), which keeps any charge above threshold. Setting $U$ back to zero (hard reset) is the other common choice. Try both, and try removing the leak ( $β = 1$ ):

Three behaviours to notice:

Integrate. Weak input still fires eventually if there is no leak; with a leak, it may never reach threshold.
Leak. With constant input $I$ , the membrane settles at $I / (1 - β)$ . If that is below $θ$ , the neuron stays silent forever, even though a ReLU would already output a positive number.
Coincidence. With input spikes, the neuron fires only when several arrive close together. Timing matters, not just the total.

4. A Spiking Neuron Is a Recurrent Neural Network

Look at the update rule again: $U [t]$ depends on $U [t - 1]$ . That is a recurrent network, with a very simple cell:

This is the most useful idea in this post. Everything you know about RNNs carries over: a spiking network is unrolled over time steps, its weights are shared across steps, it is trained with backpropagation through time, and it can suffer from vanishing gradients over long sequences (see Attention from Scratch). The differences are that the recurrence is a fixed decay $β$ rather than a learned matrix, and that the output is binary. That binary output is the one real obstacle to training, and post 3 is about how to get around it.

5. Where ReLU Comes From

If a neuron only outputs 0 or 1, how does it represent a value like 0.73? By how often it fires. Feed a constant input and count spikes:

Without a leak, the integrate-and-fire neuron’s firing rate is exactly a ReLU: zero for negative input, proportional to the input above zero, saturating at one spike per step, and quantised to multiples of $1/ T$ . This is not a coincidence. It is the basis of ANN-to-SNN conversion: train an ordinary ReLU network, copy its weights into integrate-and-fire neurons, rescale the thresholds, and run it for enough time steps (Cao et al., 2015; Diehl et al., 2015; Rueckauer et al., 2017). The price is time: to resolve a value to 1% you need on the order of 100 steps. Adding the leak shifts the curve: small inputs are filtered out entirely.

Representing a value by a firing rate is rate coding. It is robust but slow and spends many spikes. The alternatives, putting information into when a spike happens (latency coding) or emitting spikes only on change (delta coding), come back in the next post.

6. The CNN You Know, Running on Spikes

Now take a small CNN of the kind used in From Neural Networks to CNNs: two convolution + max-pool layers and a linear layer, about 15,000 parameters. Replace every ReLU with an LIF neuron ( $β = 0.9$ , $θ = 1$ , soft reset) and run the network for $T = 25$ time steps:

I trained this network and the identical ReLU network on MNIST with the same optimiser and three epochs each (the spiking one with surrogate gradients, explained in post 3), then exported both to run in your browser.

A spiking network needs an input at every time step, and the demo lets you choose what that input is:

The image (direct). The same grey-level image is fed at every step. The first layer turns it into a constant input current, and the spiking starts there.
Random spikes (rate). Each step gets a fresh random 0/1 image in which every pixel fires with probability equal to its brightness.
Events (saccades). This is what an event camera would send. The digit is shown to a simulated event camera that makes three small saccades, exactly how the N-MNIST dataset was recorded with a real sensor (Orchard et al., 2015). Every step then carries only the ON/OFF events of the edges that moved in that 10 ms bin, never the full image. This needs its own network, trained on such events (two input channels, 34 × 34, 30 steps).

Draw a digit, pick an input, then press Play and watch the input panel change step by step:

The browser code reproduces PyTorch’s output spike counts exactly for the two image networks. For the event network the browser generates about 0.5% fewer events (floating-point rounding when a pixel’s change lands exactly on a multiple of the threshold), so a spike count occasionally differs by one or two; the predicted digit was the same on every test digit I checked. What the numbers show:

	ReLU CNN	Spiking CNN, image input ( $T = 25$ )	Spiking CNN, saccade events ( $T = 30$ )
MNIST test accuracy	98.83%	98.84%	98.26%
Input	the image, once	the image, at every step	ON/OFF events, about 4,200 per digit
Output	10 real numbers, one pass	10 spike counts after 25 steps	10 spike counts after 30 steps
Neurons active per step (hidden layers)	all computed	about 5–6% spike	about 6% and 4%
Training time per epoch (laptop CPU)	10 s	259 s	536 s

With rate-coded (Poisson) input instead of the image, the image network reaches 98.74%. The event network sees only moving edges and loses just 0.6 points against the image networks. Watch the input panel during the second saccade on a “1”: the sensor then moves along the stroke, so almost no pixel changes and few events fire. Edges parallel to the motion are invisible to an event camera (the aperture problem), which is one reason N-MNIST uses three saccades in different directions.

Two lessons are already visible.

The spiking network matches the ReLU network, with only a small fraction of neurons active at any moment. That sparsity is the whole case for SNNs.

But time is not free. The networks need several steps before they can answer at all:

With 1 or 2 steps the output neurons have not yet charged up, and accuracy is at chance (10%). From 8 steps on it is within 0.6 points of the ReLU network, and from 15 steps on the two are level. The event network reaches 85% after 3 steps and 97% after 10, that is, after the first of its three saccades; the remaining two add the last point. On a CPU or GPU, which have no notion of “silent neurons”, the spiking network simply costs $T$ forward passes, which is why training took 26× longer here. The energy advantage only appears on hardware that skips silent neurons (post 4).

How the demo was built

The whole pipeline is a normal PyTorch training loop with one extra dimension, time:

Encode each image into $T$ inputs, one per time step: the image itself (direct), random spikes (rate) or saccade events.
Run the network for $T$ steps. Every LIF layer keeps its membrane potential from one step to the next.
Count the output spikes per class. The counts act as the logits.
Compute an ordinary cross-entropy loss on those counts.
Backpropagate through all $T$ steps (backpropagation through time), with a surrogate gradient for the spike, then update the weights with Adam.

The LIF neuron. The spike is a step function in the forward pass. In the backward pass its derivative, which is zero almost everywhere, is replaced by a smooth bump. Why that works is the topic of post 3.

import math, torch, torch.nn as nn, torch.nn.functional as F
 
class Spike(torch.autograd.Function):
    @staticmethod
    def forward(ctx, u):                 # spike where the membrane is above threshold
        ctx.save_for_backward(u)
        return (u > 0).float()
 
    @staticmethod
    def backward(ctx, grad):             # surrogate gradient: a smooth bump instead of 0 / infinity
        u, = ctx.saved_tensors
        return grad / (1 + (math.pi * u) ** 2)
 
def lif(current, u, beta=0.9, theta=1.0):
    s_prev = (u > theta).float().detach()   # did the neuron spike on the previous step? (reset)
    u = beta * u + current - s_prev * theta  # leak + input - reset
    return Spike.apply(u - theta), u         # spikes of this step, new membrane potential

The network. The same layers as the ReLU CNN; only the activation changes, and the forward pass loops over time:

class SpikingCNN(nn.Module):
    def __init__(self, in_channels=1, size=28):          # events: in_channels=2, size=34
        super().__init__()
        self.c1 = nn.Conv2d(in_channels, 12, 5)
        self.c2 = nn.Conv2d(12, 32, 5)
        side = ((size - 4) // 2 - 4) // 2
        self.fc = nn.Linear(32 * side * side, 10)
 
    def forward(self, frames):                            # frames: (T, batch, C, H, W)
        u1 = u2 = u3 = torch.zeros(())                    # membrane potentials, start at rest
        counts = 0
        for x in frames:                                  # one time step at a time
            s1, u1 = lif(F.max_pool2d(self.c1(x), 2), u1)
            s2, u2 = lif(F.max_pool2d(self.c2(s1), 2), u2)
            s3, u3 = lif(self.fc(s2.flatten(1)), u3)
            counts = counts + s3                          # spikes per output neuron
        return counts                                     # used as logits

The three inputs. Direct and rate encoding are one line each. The events come from moving the digit along a small triangle (three saccades) and running the event-camera pixel of Seeing in Events on every pixel:

T = 25                                              # img: (batch, 1, 28, 28), values in [0, 1]
direct = img.expand(T, *img.shape)                  # the same image at every step
rate   = (torch.rand(T, *img.shape) < img).float()  # pixel fires with probability = brightness
 
def make_events(img, T=30, sub=4, C=0.3):
    """img: (batch, 28, 28) -> saccade events, N-MNIST style: (T, batch, 2, 34, 34) ON/OFF counts."""
    out = torch.zeros(T, img.shape[0], 2, 34, 34)
    ref = torch.log(scene(img, *saccade(0)))           # each pixel's reference log brightness
    for k in range(1, T * sub + 1):                     # 4 sub-steps per 10 ms bin
        d = torch.log(scene(img, *saccade(k))) - ref    # change since the pixel's last event
        n = torch.floor(d.abs() / C) * torch.sign(d)    # threshold crossings, signed
        ref = ref + n * C
        out[(k - 1) // sub, :, 0] += n.clamp(min=0)     # ON events
        out[(k - 1) // sub, :, 1] += (-n).clamp(min=0)  # OFF events
    return out

Here scene draws the digit on a dark 34 × 34 screen, shifted with sub-pixel (bilinear) precision, and saccade(k) gives the sensor’s offset along the triangle at sub-step k.

Training. Exactly the loop you would write for a ReLU CNN. The only difference is that loss.backward() now flows back through every time step:

net = SpikingCNN()                                   # SpikingCNN(2, 34) for events
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
for epoch in range(3):
    for img, label in train_loader:                  # batches of 128
        counts = net(encode(img))                    # encode = direct, rate or make_events
        loss = F.cross_entropy(counts, label)        # spike counts as logits
        opt.zero_grad()
        loss.backward()                              # backpropagation through time
        opt.step()

After training, the weights were exported to JSON. The browser re-implements the same forward pass (and the event generator) in plain JavaScript and was checked against PyTorch, as described above. The full scripts are in the repository: train_spiking_cnn.py for the image networks, and events.py with train_events.py for the event network.

7. A Small Zoo of Spiking Neurons

The LIF neuron is the workhorse, but several variants appear in practice and later in this series:

Neuron	What changes	Why use it
Integrate-and-fire (IF)	no leak, $β = 1$	simplest; its rate equals ReLU; common in low-power chips and ANN conversion
LIF	leak $β < 1$	forgets old input; the default in deep SNNs
Parametric LIF (Fang et al., 2021)	$β$ (the time constant) is learned per layer	lets each layer pick its own memory length
Adaptive LIF (Bellec et al., 2018; Baronig et al., 2025)	a second state variable raises the threshold or feeds back after each spike	longer memory and richer temporal patterns; fires less after bursts
Recurrent LIF	output spikes fed back through weights	sequence tasks
Event-based GRU (Subramoney et al., 2023)	a GRU whose units transmit only when their state crosses a learned threshold	deep-learning-style memory with spike-like sparse communication
Hodgkin–Huxley, Izhikevich	detailed ion-channel dynamics	neuroscience models; too stiff and expensive for deep learning

8. Common Claims, Checked

Claim	Verdict
”A spiking network is just a binarised neural network.”	No. Each step’s activation is binary, but information is also carried by when and how often a neuron fires, and the membrane state carries real-valued memory between steps. Binarised networks have no time axis.
”Spiking networks are always more energy-efficient.”	Only on hardware that exploits sparsity. On a CPU or GPU, a spiking network costs about $T$ times a normal forward pass: the demo network above trained 26× slower than its ReLU twin. The advantage needs event-driven hardware and sparse activity.
”The LIF neuron models a real neuron.”	Coarsely. It keeps integration, leak and threshold but drops ion-channel dynamics, dendrites and the spike’s shape. The Hodgkin–Huxley model (1952) describes the biophysics; LIF is the cheap abstraction that keeps the input–output behaviour useful for computation.
” $β = e^{- Δ t / τ}$ comes from the forward Euler method.”	Forward Euler gives $β = 1 - Δ t / τ$ . The exponential is the exact solution of the decay over one step, and is the better choice when $Δ t$ is not much smaller than $τ$ .
”Spike if `S > threshold`.”	A pitfall in some published example code: the comparison must be on the membrane potential, $S [t] = U [t] > θ$ , and the reset must use the previous spike. Comparing the spike variable with the threshold means the neuron never fires.
”Third-generation networks are more powerful than ANNs.”	In theory, networks of spiking neurons can compute some functions with fewer units (Maass, 1997). In practice, on static benchmarks like the one above, they match a ReLU network at best; their strengths are efficiency and time-varying data.

Summary

Idea	In one line
LIF neuron	$U [t] = β U [t - 1] + W x [t] - θS [t - 1]$ : integrate, leak, fire, reset
Spiking neuron = RNN cell	unrolled over $T$ steps, trained like an RNN
Firing rate ≈ ReLU	the IF neuron’s rate is a clipped, quantised ReLU, which is why ReLU networks can be converted
Spiking CNN	same layers, activation replaced, run for $T$ steps, predict from spike counts
The trade	sparse, binary activity in exchange for time steps and a harder training problem

Next: Seeing in Events: a camera that does not take pictures, and how to turn ordinary video into the event streams spiking networks are built for.

References

Spiking neurons

Lapicque, L. (1907). Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarisation. Journal de Physiologie et de Pathologie Générale, 9, 620–635. Discussed in: Brunel, N., & van Rossum, M. C. W. (2007). Lapicque’s 1907 paper: from frogs to integrate-and-fire. Biological Cybernetics, 97, 337–339.
Hodgkin, A. L., & Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve. Journal of Physiology, 117(4), 500–544.
Izhikevich, E. M. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6), 1569–1572.
Gerstner, W., & Kistler, W. M. (2002). Spiking Neuron Models. Cambridge University Press.
Maass, W. (1997). Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9), 1659–1671.

Neuron variants

Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., & Tian, Y. (2021). Incorporating learnable membrane time constant to enhance learning of spiking neural networks. ICCV.
Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., & Maass, W. (2018). Long short-term memory and learning-to-learn in networks of spiking neurons. NeurIPS.
Baronig, M., Ferrand, R., Sabathiel, S., & Legenstein, R. (2025). Advancing spatio-temporal processing in spiking neural networks through adaptation. Nature Communications.
Subramoney, A., Nazeer, K. K., Schöne, M., Mayr, C., & Kappel, D. (2023). Efficient recurrent architectures through activity sparsity and sparse back-propagation through time. ICLR.

From ANNs to SNNs

Cao, Y., Chen, Y., & Khosla, D. (2015). Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113, 54–66.
Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S.-C., & Pfeiffer, M. (2015). Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. IJCNN.
Rueckauer, B., Lungu, I.-A., Hu, Y., Pfeiffer, M., & Liu, S.-C. (2017). Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience, 11, 682.
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015). Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 9, 437.
Eshraghian, J. K., Ward, M., Neftci, E. O., et al. (2023). Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 111(9), 1016–1054.

Context

Mead, C. (1990). Neuromorphic electronic systems. Proceedings of the IEEE, 78(10), 1629–1636.
Patterson, D., Gonzalez, J., Le, Q., et al. (2021). Carbon emissions and large neural network training. arXiv:2104.10350.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.

Notes

Explorer