This series has followed two lines that looked unrelated. One went from CNNs to attention, Vision Transformers and self-supervised features. The other built the geometry of cameras: image formation, feature matching and 3D from two views. This post is where they meet. Over the last few years, networks have taken over one step of the geometric pipeline after another: detecting keypoints, matching them with attention, matching without keypoints at all, and finally predicting 3D structure directly from images.
To see what that buys, the learned methods run on exactly the benchmark from the feature-matching post: the same four photos, the same known transformations, the same ground truth. And a look inside LightGlue shows attention doing what the ratio test did by hand.
1. Step by Step, the Pipeline Gets Learned
Every row keeps the last steps. Even the end-to-end networks are followed by an optimisation that makes their outputs geometrically consistent. The geometry of the previous posts does not disappear; it moves to the end of the pipeline and checks what the networks propose.
2. Learned Keypoints: SuperPoint
SuperPoint (DeTone et al., 2018) detects keypoints and computes 256-d descriptors with one small CNN, in a single pass over the image. Its training is a neat piece of self-supervision. First, a detector is trained on synthetic shapes (lines, triangles, cubes), where the corners are known exactly. Then it is applied to real photos under many random homographies, and the detections are mapped back and aggregated (homographic adaptation), producing pseudo-labels for real images. The descriptors are trained so that the same point under a known homography gets the same vector: the label comes from the geometry, not from a person. DISK (Tyszkiewicz et al., 2020) and ALIKED (Zhao et al., 2023) learn detection and description end to end, directly for matching.
3. Matching with Attention: SuperGlue and LightGlue
The ratio test from the feature-matching post decides each match on its own, from two distances. But matching is a joint problem: a point’s correct partner depends on where its neighbours go, and each point can be matched at most once. SuperGlue (Sarlin et al., 2020) turned matching into a Transformer problem. The keypoints of both images, with their positions and descriptors, pass through layers of self-attention (within an image) and cross-attention (across images), and an optimal-transport step (Sinkhorn) with an extra “no match” slot produces the assignment. LightGlue (Lindenberger et al., 2023) keeps the idea and makes it adaptive:
- Rotary positional encoding of the keypoint positions, so attention can use relative positions.
- Early stopping: after each layer, a small classifier estimates whether the predictions are final; easy image pairs stop after a few layers.
- Pruning: points that are confidently unmatchable (not visible in the other image) are dropped from further layers.
- A simpler assignment: a similarity score times a learned matchability per point, instead of Sinkhorn iterations.
What does the cross-attention actually do? Here is SuperPoint + LightGlue on the photo pair from the feature-matching post, with early stopping and pruning switched off so that all 9 layers can be followed:
Averaged over all keypoints of this pair (704 in the first image, 531 in the second) that have a known true match, the cross-attention a keypoint pays to its true match is 22% in the first layer. That is already far above the 0.19% of an even spread, because SuperPoint’s descriptors alone are informative. By layer 3 it is 92%, and it peaks at 99% in layer 7. In the last layer it spreads out again (78%); the final matches are computed from the updated descriptors after the last layer, not from the attention weights, so nothing forces the last layer’s attention to stay sharp. This is the joint reasoning the ratio test cannot do: each layer refines every descriptor using what the other image contains, and repetitive structures such as windows or paving stones, which confuse the ratio test, are told apart by their context.
4. Without Keypoints: LoFTR
Keypoint detectors find nothing on low-texture surfaces: a white wall, a road, the sky. LoFTR (Sun et al., 2021) skips detection. A CNN computes features on a coarse grid (1/8 of the resolution), a Transformer with self- and cross-attention (linear attention, to keep the cost manageable for thousands of positions) transforms the features of both images together, and every coarse cell is matched by a dual softmax over the similarity matrix with a mutual-nearest-neighbour check. Matches are then refined to sub-pixel accuracy on a finer grid. Because every cell can take part, LoFTR also matches regions that a detector would leave empty, which is why it typically returns thousands of matches. EfficientLoFTR (Wang et al., 2024) makes it several times faster.
A note on tools: I first used the EfficientLoFTR model as packaged in Hugging Face Transformers 5.18. On my tests it matched shifted images perfectly but returned almost no correct matches once the second image was rotated, even by 5°, while the original LoFTR in Kornia matched the same rotated pairs correctly (100% precision at 5°, 99.7% at 30°). So the benchmark below uses Kornia’s LoFTR. I did not track down the cause, but it is a reminder to test any model on a case with known ground truth before trusting it.
5. The Benchmark Again
The five methods, on the four photos and the transformations of the feature-matching post, plus a new family that combines everything at once: rotation, scaling, viewpoint change, darkening and blur, increasing with the level. As before, a match counts as correct within 3 px of its true position, and “homography found” means RANSAC’s estimate is within 5 px of the truth at the image corners. All methods run on the same laptop CPU (4 threads); SIFT and ORB use the ratio test at 0.8 and 2,000 keypoints, the learned matchers their confidence threshold of 0.2.
The results do not fit “learned beats classical” or the reverse. Each family tells a different story:
| Change | What happened |
|---|---|
| Rotation | SIFT and ORB found the homography at every angle. SuperPoint + SuperGlue failed from 90°, SuperPoint + LightGlue from 60° (3 of 4) and 90° (0 of 4), LoFTR from 60°. The networks were trained on mostly upright photos, and their descriptors are not rotation invariant. |
| Scaling | All five handled 0.4× to 2×. At 2.8×, only SIFT still found the homography (4 of 4); the learned methods managed 1 of 4 at best. |
| Viewpoint | All five succeeded throughout; LoFTR returned about 4 times as many correct matches as SIFT (1,379 vs 368 at the strongest change). |
| Darkening to 30% | SIFT fell to 253 correct matches and failed on 1 photo; SuperPoint + LightGlue kept 521, LoFTR 4,250, all with 4 of 4. |
| Blur | The learned methods held up much better: at σ = 2 px LoFTR kept 4,203 correct matches against SIFT’s 204. At σ = 6 px everything degraded; SIFT still found 3 of 4, LoFTR 2. |
| Noise, gamma | No problem for any method; LoFTR kept the most matches. |
| Everything at once | At level 1 (20° rotation, 0.83× scale, a small viewpoint change, slightly darker and blurred) all succeeded; at level 2 (40°) LoFTR failed on one photo. From level 3, which includes a 60° rotation, the learned methods failed (SuperGlue still 2 of 4), while SIFT found 3 of 4 even at level 4. |
| Time per pair (CPU) | SIFT 0.04 s, ORB 0.02 s, SuperPoint + SuperGlue 1.5 s, SuperPoint + LightGlue 1.3 s, LoFTR 1.5 s. On a GPU the learned methods take milliseconds, but on a CPU they are about 35–40 times slower than SIFT and 75–90 times slower than ORB. |
The pattern is consistent: the learned methods win wherever appearance changes (lighting, blur, low texture) and get many more matches, but they inherit the biases of their training data. They were trained on photo pairs from the internet and from driving and indoor scenes, which are almost always upright, so a camera rolled by 60° or more is outside their experience. Rotation-equivariant variants and test-time rotation augmentation exist for exactly this reason. SIFT’s hand-designed orientation assignment, on the other hand, makes rotation simply irrelevant.
6. From Images Straight to 3D
The last step removes even the matching. DUSt3R (Wang et al., 2024) takes two images with no calibration and outputs, for every pixel of each, a 3D point, both expressed in the first camera’s frame (a “pointmap”), plus a confidence:
From the two pointmaps, everything in the two-view post can be read off directly: the relative pose by aligning the points, the focal length from how pixels relate to 3D points, and dense matches by finding pixels with the same 3D point. The network is a ViT encoder–decoder in which the two decoders cross-attend to each other, and it was initialised from CroCo (Weinzaepfel et al., 2022), a masked-image pre-training like MAE, except that the hidden patches are reconstructed with the help of a second view of the same scene. To predict a hidden patch from another viewpoint, the network has to learn geometry. DUSt3R itself is then trained with real 3D supervision, on several million image pairs whose ground truth comes from classical reconstruction and 3D scanning.
MASt3R (Leroy et al., 2024) adds a head for local descriptors to make the matches more precise. VGGT (Wang et al., 2025) goes from two views to many: a 1.2-billion-parameter Transformer that alternates attention within each frame and across all frames, and predicts cameras, depth maps, pointmaps and point tracks for dozens of images in a single forward pass of about a second on a GPU. For many inputs its raw output already rivals classical pipelines, and a short bundle adjustment on top improves it further.
Depth from a single image
With one image, there is no parallax at all. Yet networks predict depth from a single photo surprisingly well, by learning how scenes usually look. Depth Anything V2 (Yang et al., 2024) uses the DINOv2 encoder from the self-supervised post and a DPT decoder (Ranftl et al., 2021) that turns the patch tokens back into a dense map. It was trained on precise synthetic depth and then on 62 million real images with pseudo-labels from a larger teacher model. The small version (25 M parameters) runs on a laptop CPU:
On these four photos, the depth maps separate people, dogs, cars and the cat from the background with sharp outlines, and the ground plane gets the smooth gradient one would expect, in 0.22 s per image on a laptop CPU. The output is relative inverse depth, though: it orders the scene from near to far, but the scale and offset are unknown. That is the scale ambiguity of the two-view post, at its most extreme; with a single image, nothing but learned experience about how scenes look decides the depth.
7. What Is Still Geometry
It is tempting to conclude that geometry is obsolete. The opposite is true:
- Every learned matcher is followed by RANSAC. The benchmark’s “homography found” numbers all come from
findHomographywith RANSAC on the networks’ matches. - The end-to-end networks are refined by optimisation. DUSt3R aligns its pairwise pointmaps globally, and VGGT’s predictions improve with bundle adjustment.
- The supervision comes from geometry. SuperPoint learns from homographies, the matchers from camera poses and depth computed by structure from motion, and DUSt3R and VGGT from 3D reconstructions of real scenes.
- The limits are geometric. No network removes the scale ambiguity of images; monocular depth is relative unless something in the scene, or the training data, fixes the scale.
What the networks bring is priors: what objects look like, how scenes are usually laid out, which points tend to match. That helps exactly where classical methods have nothing to work with: textureless regions, large changes of viewpoint, few images.
8. Common Claims, Checked
| Claim | Verdict |
|---|---|
| ”Learned features are always better than SIFT.” | Not across the board. On the benchmark here, they were far better under darkening and blur and found many more matches, but failed under large rotations (60° and more) and extreme scaling (2.8×), where SIFT kept working. On a CPU they are also 35–90 times slower. |
| ”LightGlue replaces RANSAC.” | It replaces the ratio test: it decides which keypoints correspond. Its matches still contain errors, and a robust estimate of the geometry (RANSAC) is still applied afterwards. |
| ”Attention in a matcher just compares descriptors.” | Cross-attention also updates each descriptor with what it sees in the other image, and self-attention with its neighbours, layer by layer; the attention on the correct partner grows across the layers (§3). |
| ”Detector-free matching finds more correct matches.” | Usually far more matches, including in weakly textured regions; whether more of them are correct depends on the change (§5). It is also slower than keypoint matching. |
| ”DUSt3R and VGGT make structure from motion obsolete.” | They replace much of the pipeline and work where classical methods fail (few views, little overlap), but their outputs are still refined geometrically, and they are trained on data produced by classical reconstruction. |
| ”Monocular depth networks measure distance.” | Most predict relative depth, correct up to scale and offset; metric versions need training data with known scale and still guess it from priors. |
| ”A pre-trained model from a library works as documented.” | Usually, but not always: the EfficientLoFTR port tested here failed on rotated images. A test with known ground truth catches this in minutes. |
Summary
| Idea | What it learns, or replaces |
|---|---|
| SuperPoint, DISK, ALIKED | Detection and description, from homographies instead of human labels |
| SuperGlue, LightGlue | The ratio test: self- and cross-attention between the two keypoint sets |
| LoFTR, EfficientLoFTR | The detector: dense Transformer features, matched everywhere |
| DUSt3R, MASt3R, VGGT | Matching, pose and triangulation: images in, 3D points out |
| Depth Anything | Depth from one image: DINOv2 features plus a dense decoder; relative, not metric |
| RANSAC, bundle adjustment | Still applied after all of them |
This closes the geometry part of the series, and with it the arc from a single convolution to networks that turn photos into 3D scenes.
References
Learned features and matching
- DeTone, D., Malisiewicz, T., & Rabinovich, A. (2018). SuperPoint: Self-supervised interest point detection and description. CVPR Workshops.
- Tyszkiewicz, M. J., Fua, P., & Trulls, E. (2020). DISK: Learning local features with policy gradient. NeurIPS.
- Sarlin, P.-E., DeTone, D., Malisiewicz, T., & Rabinovich, A. (2020). SuperGlue: Learning feature matching with graph neural networks. CVPR.
- Sun, J., Shen, Z., Wang, Y., Bao, H., & Zhou, X. (2021). LoFTR: Detector-free local feature matching with Transformers. CVPR.
- Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local feature matching at light speed. ICCV.
- Zhao, X., Wu, X., Chen, W., Chen, P. C. Y., Xu, Q., & Li, Z. (2023). ALIKED: A lighter keypoint and descriptor extraction network via deformable transformation. IEEE Transactions on Instrumentation and Measurement.
- Wang, Y., He, X., Peng, S., Tan, D., & Zhou, X. (2024). Efficient LoFTR: Semi-dense local feature matching with sparse-like speed. CVPR.
3D from images
- Weinzaepfel, P., Leroy, V., Lucas, T., Brégier, R., Cabon, Y., Arora, V., Antsfeld, L., Chidlovskii, B., Csurka, G., & Revaud, J. (2022). CroCo: Self-supervised pre-training for 3D vision tasks by cross-view completion. NeurIPS.
- Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., & Revaud, J. (2024). DUSt3R: Geometric 3D vision made easy. CVPR.
- Leroy, V., Cabon, Y., & Revaud, J. (2024). Grounding image matching in 3D with MASt3R. ECCV.
- Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., & Novotny, D. (2025). VGGT: Visual geometry grounded transformer. CVPR.
- Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021). Vision Transformers for dense prediction. ICCV.
- Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., & Zhao, H. (2024). Depth Anything V2. NeurIPS.
Tools
- Hugging Face Transformers 5.18 (SuperPoint, SuperGlue, LightGlue, Depth Anything V2) and Kornia 0.8 (LoFTR); the script runs everything on a CPU.