From ReLU to Spikes replaced the neuron. This post replaces the camera. A spiking network fed with ordinary video frames has to turn every frame into spikes first, and spends most of its effort on pixels that did not change. An event camera produces something much closer to what a spiking network wants: a stream of pixel-level “something changed here, now” signals.
1. A Camera That Does Not Take Pictures
Every camera in the Computer Vision posts, and every image dataset, works the same way: a shutter opens at a fixed rate and every pixel reports its brightness, whether anything changed or not. An event camera, also called a dynamic vision sensor (DVS) or “silicon retina”, has no shutter and no frames. Each pixel works on its own and reports only when its brightness changes (Lichtsteiner et al., 2008; Gallego et al., 2022).
The output is a stream of events, each a tuple : pixel position, a timestamp with microsecond resolution, and a polarity (ON, brighter) or (OFF, darker).
2. Inside One Pixel
Each pixel stores the logarithm of the brightness at its last event. When the current log brightness has moved more than a contrast threshold away from that reference, the pixel emits an event and moves the reference by :
This is the same delta modulation idea as the integrate-and-fire neuron’s threshold, and it has two important consequences: the pixel reports relative changes, so it works in bright sunlight and in dim rooms alike, and a large change produces a burst of events, one per threshold step:
| Frame camera | Event camera | |
|---|---|---|
| Output | full images at a fixed rate (30–120 fps typical) | asynchronous events |
| Time resolution | 8–33 ms between frames | microseconds |
| Dynamic range | about 60 dB | 120 dB and more |
| Motion blur | yes, during the exposure | no exposure, no blur |
| Static scene | full data rate | almost nothing (noise only) |
| Absolute brightness, colour | yes | no (only changes; most sensors are monochrome) |
The first DVS (Lichtsteiner et al., 2008) had 128 × 128 pixels, 120 dB dynamic range and 15 µs latency at about 23 mW. Later sensors added a frame readout in the same pixel (DAVIS240: 240 × 180, Brandli et al., 2014), and current industrial sensors reach 1280 × 720 (Finateu et al., 2020).
3. Watch It Work
The simulator below runs the pixel model above on every pixel of a 128 × 128 scene, at 1 ms steps. The scenes are synthetic so that nothing needs to be downloaded; the webcam option runs entirely in your browser.
Things to try:
- Moving ball. Only the edges fire: ON at the leading edge (the bright ball arrives), OFF at the trailing edge. About 2% of the pixels are active in any 40 ms window, and the event data rate is around a tenth of what 30 fps frames would need.
- Still scene. Almost silent, apart from noise events: real sensors produce background events from leakage currents and photon shot noise, on the order of 0.1 per pixel per second in good light, more in the dark.
- Flickering lamp. A light flickering at 10 Hz fires on every cycle without any motion. Mains-powered lights (100–120 Hz) and screens are a real nuisance for event cameras.
- Camera moves. Tick the box. Now the whole textured room moves across the sensor, and the event rate jumps by an order of magnitude, to over 100,000 events per second with a third or more of the pixels active. For a head-mounted camera this is the normal situation: the wearer’s own head motion, not the activity in front of them, decides how many events there are.
- 15 fps video. The scene is now sampled as a slow video and the brightness between frames is interpolated. Events still appear, but their timing between frames is made up, and fast edges skip pixels. That is the limitation of the conversion tools in Section 5.
The same events in space and time
A useful way to picture an event stream is as a cloud of points in . A rotating bar draws a continuous spiral surface; a frame camera would cut this volume into a handful of slices:
4. Not Every Sensor Is a Camera: Three Spike Encodings
The same question comes up for any signal fed to a spiking network: how to turn a value into spikes. Three codes are common (Eshraghian et al., 2023):
- Rate coding: the value sets the probability of a spike at each step. Robust, but spends many spikes, even when nothing changes.
- Latency coding: one spike per time window, earlier for larger values. Very few spikes, but sensitive to timing noise.
- Delta modulation: a spike only when the value has changed by more than a threshold, ON or OFF. Exactly what the event camera pixel does.
There is a fourth option, used in many state-of-the-art networks: no encoding at all. Feed the real-valued signal into the first layer at every step (direct encoding, as in the spiking CNN of the previous post) and let the first layer of spiking neurons decide when to fire. It loses nothing in conversion. The price is that the first layer multiplies real numbers instead of adding weights. For a low-dimensional signal, such as a wrist-worn accelerometer with three channels at 50 Hz, that price is tiny, which is why motion sensors are usually fed to spiking networks directly.
5. Turning Ordinary Video into Events
Event cameras are still rare, and most labelled datasets were recorded with frame cameras. Simulators convert video into events (Rebecq et al., 2018; Hu et al., 2021):
v2e (Hu et al., 2021) is the most widely used. It first increases the frame rate with a learned frame interpolator (Super SloMo, Jiang et al., 2018) until objects move less than about a pixel between frames, then applies a realistic pixel model: logarithmic response, contrast thresholds that vary from pixel to pixel, a photoreceptor bandwidth that drops in dim light, shot noise, leak events and a refractory period.
Converted events are very useful for training, but they inherit the limits of the source video: the timing between frames is interpolated, not measured; the dynamic range is that of the frame camera, not 120 dB; and any motion blur in the video becomes blur in the events. Results on converted data are best treated as comparisons between models on the same data, not as predictions of performance on a real sensor.
6. From an Event Stream to a Tensor
A neural network needs a tensor. The simplest and most common representation counts events per polarity, pixel and time bin:
A spiking network then processes one time bin per step, exactly like the steps of the spiking CNN in the previous post:
Two things to notice. The tensor is very sparse (here about 1% non-zero), which is good for event-driven hardware. But its entries are counts, not spikes: a high-contrast edge passing a pixel changes its log brightness by several thresholds at once and produces a burst, so even narrow bins hold counts of 2 to 12. Choosing the bin width trades timing resolution (narrow bins, many steps) against the number of network steps. Other representations keep more of the timing: time surfaces store the time of each pixel’s latest event (Lagorce et al., 2017), and voxel grids spread each event over the two nearest bins (Zhu et al., 2019).
7. Event Datasets
| Dataset | Content | Sensor |
|---|---|---|
| N-MNIST, N-Caltech101 (Orchard et al., 2015) | MNIST and Caltech101 images shown on a screen, recorded during three small camera “saccades” | ATIS, real |
| DVS Gesture (Amir et al., 2017) | 11 hand and arm gestures, 29 people, 3 lighting conditions | DVS128, real |
| DSEC (Gehrig et al., 2021) | driving scenes, stereo event cameras and frame cameras | real |
| SHD, SSC (Cramer et al., 2022) | spoken digits and commands as spikes from a simulated cochlea | simulated |
| any video dataset + v2e | whatever the video shows | converted |
N-MNIST is a cautionary example: the “motion” comes from moving the sensor in front of a static image, so the event data contains little that the original image did not. Iyer et al. (2021) showed that classifiers which ignore spike timing do about as well on it as spiking networks that use it. A benchmark only tests temporal processing if the information is actually in the timing.
8. Common Claims, Checked
| Claim | Verdict |
|---|---|
| ”An event camera records images, just faster.” | It records changes in log brightness. A perfectly static scene, seen by a static sensor, is invisible to it. Sensors that also need images (DAVIS) contain a separate frame readout. |
| ”No motion means no data.” | Almost: real sensors produce background noise events (leakage and shot noise), and flickering lights produce events without any motion. |
| ”Event data is always sparse.” | Only when little changes. A moving camera makes the whole textured scene fire; in the simulator above, turning the camera raises the event rate more than tenfold. For head-mounted cameras this is the common case. |
| ”An event tensor is a spike train.” | Binned event tensors hold counts, and edges produce bursts of several events per pixel. A network reading them multiplies real values in its first layer, like any CNN. |
| ”Events converted from video are as good as real events.” | They are realistic, but the timing between frames is interpolated, and the dynamic range and motion blur of the source video carry over. Compare models on converted data with each other, not with results on real sensors. |
| ”Event cameras have no latency.” | Latency is in the microseconds in good light, but grows in dim light as the photoreceptor slows down, and a busy scene can saturate the readout bandwidth. |
| ”Rate-coding an image turns it into event data.” | Random Poisson spikes add noise and no temporal information. An image converted this way still contains only the information of the image. |
Summary
| Idea | In one line |
|---|---|
| Event camera | each pixel reports changes in log brightness as , asynchronously |
| Contrast threshold | an event per change of in log brightness: relative, high dynamic range, bursts on strong edges |
| Sparsity | high for a static camera and a mostly static scene; much lower when the camera itself moves |
| Encodings | rate, latency, delta, or direct real-valued input |
| Video to events | v2e-style conversion: interpolate frames, model the pixel; timing is synthesised, not measured |
| Event tensor | counts per (polarity, pixel, time bin), shape , one bin per network step |
Next: Training Spiking Networks: why a spike has no useful gradient, and the trick (surrogate gradients) that makes backpropagation work anyway.
References
Event cameras
- Lichtsteiner, P., Posch, C., & Delbruck, T. (2008). A 128 × 128 120 dB 15 µs latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2), 566–576.
- Brandli, C., Berner, R., Yang, M., Liu, S.-C., & Delbruck, T. (2014). A 240 × 180 130 dB 3 µs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10), 2333–2341.
- Finateu, T., Niwa, A., Matolin, D., et al. (2020). A 1280 × 720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 µm pixels, 1.066 GEPS readout, programmable event-rate controller and compressive data-formatting pipeline. ISSCC.
- Gallego, G., Delbrück, T., Orchard, G., et al. (2022). Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1), 154–180.
Simulation and representation
- Hu, Y., Liu, S.-C., & Delbruck, T. (2021). v2e: From video frames to realistic DVS events. CVPR Workshops.
- Rebecq, H., Gehrig, D., & Scaramuzza, D. (2018). ESIM: An open event camera simulator. Conference on Robot Learning (CoRL).
- Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned-Miller, E., & Kautz, J. (2018). Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. CVPR.
- Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2017). HOTS: A hierarchy of event-based time-surfaces for pattern recognition. IEEE TPAMI, 39(7), 1346–1359.
- Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2019). Unsupervised event-based learning of optical flow, depth, and egomotion. CVPR.
- Eshraghian, J. K., Ward, M., Neftci, E. O., et al. (2023). Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 111(9), 1016–1054.
Datasets
- Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015). Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 9, 437.
- Amir, A., Taba, B., Berg, D., et al. (2017). A low power, fully event-based gesture recognition system. CVPR.
- Gehrig, M., Aarents, W., Gehrig, D., & Scaramuzza, D. (2021). DSEC: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3), 4947–4954.
- Iyer, L. R., Chua, Y., & Li, H. (2021). Is neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain. Frontiers in Neuroscience, 15, 608567.
- Cramer, B., Stradmann, Y., Schemmel, J., & Zenke, F. (2022). The Heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(7), 2744–2757.