From Two Views to 3D: Epipolar Geometry, Triangulation and Bundle Adjustment

Image Formation ended with a problem: as soon as a camera moves instead of only turning, no single homography maps one image onto the other, because how far a point shifts depends on its depth. Finding the Same Point Twice gave us matched points between two images. This post puts the two together: from matched points alone, recover how the camera moved and where the points are in 3D.

The tools are epipolar geometry (what one image says about where to look in the other), the essential and fundamental matrices, triangulation, and bundle adjustment, which refines everything together. Together they make up structure from motion, the method behind 3D reconstructions from photo collections and the camera poses that NeRF and Gaussian splatting start from. The claims along the way are tested on synthetic scenes with exact ground truth (script, 15 seconds on a laptop CPU).

---

1. Epipolar Geometry

A pixel ${\mathbf{x}}_1$ in the first image does not determine a 3D point, only a ray: every point along it projects to the same pixel. Seen from the second camera, this ray is a line in the second image. So the match of ${\mathbf{x}}_1$, whatever its depth, must lie on that line, the epipolar line:

• The two camera centres and the 3D point span a plane, the epipolar plane. It cuts each image in an epipolar line.
• The line through the two centres (the baseline) meets each image in the epipole: the image of the other camera’s centre. Every epipolar line passes through it.
• For a camera that moves forward, the epipole lies inside the image and the epipolar lines radiate from it (the “focus of expansion”). For a camera that moves sideways, it lies far outside and the lines are almost parallel.

Click a point in the left image, then move the second camera:

This is what makes matching between two views of a scene efficient and reliable: instead of the whole image, the match is searched along one line, and a candidate far from the line can be rejected even if its descriptor looks similar.

---

2. The Essential and Fundamental Matrices

The epipolar constraint is one equation. Take the second camera’s frame relative to the first, ${\mathbf{X}}_2=\mathbf{R}{\mathbf{X}}_1+\mathbf{t}$. The vectors $\mathbf{t}$, ${\mathbf{X}}_2$ and $\mathbf{R}{\mathbf{X}}_1$ all lie in the epipolar plane, so ${\mathbf{X}}_2$ is perpendicular to $\mathbf{t}\times \mathbf{R}{\mathbf{X}}_1$. Writing the cross product as a matrix, $[\mathbf{t}{]}_{\times }$, and using normalised image coordinates $\hat{\mathbf{x}}={\mathbf{K}}^{-1}\tilde{\mathbf{x}}$:

$${\hat{\mathbf{x}}}_2^{\top }\mathbf{E}{\hat{\mathbf{x}}}_1=0,\mathbf{E}=[\mathbf{t}{]}_{\times }\mathbf{R}$$

This is the essential matrix (Longuet-Higgins, 1981). In pixel coordinates the same constraint holds with the fundamental matrix:

$${\tilde{\mathbf{x}}}_2^{\top }\mathbf{F}{\tilde{\mathbf{x}}}_1=0,\mathbf{F}={\mathbf{K}}_2^{-\top }\mathbf{E}{\mathbf{K}}_1^{-1}$$

The epipolar line of ${\mathbf{x}}_1$ in the second image is simply ${\mathbf{l}}_2=\mathbf{F}{\tilde{\mathbf{x}}}_1$.

	Essential matrix $\mathbf{E}$	Fundamental matrix $\mathbf{F}$
Needs	the intrinsics $\mathbf{K}$ (calibrated cameras)	nothing but point matches
Encodes	$\mathbf{R}$ and the direction of $\mathbf{t}$	the same plus both cameras’ intrinsics
Degrees of freedom	5 (3 rotation + 2 translation direction)	7 (9 entries − scale − $\det \mathbf{F}=0$)
Structure	two equal singular values, one zero	rank 2
Minimal estimation	5 point pairs (Nistér, 2004)	7 pairs; 8 for the linear algorithm

So epipolar geometry does not need to know where the cameras are: $\mathbf{F}$ can be estimated from point matches alone, and the camera motion is then derived from it. The lab above uses the true $\mathbf{F}$ of its scene; it agrees with OpenCV’s 8-point estimate from exact matches to within $2\times 10^{-6}$ (relative).

The 8-point algorithm, and why normalisation matters

Each match gives one equation that is linear in the 9 entries of $\mathbf{F}$, so 8 matches (or more, by least squares) determine it up to scale: stack the equations into $\mathbf{Af}=0$, take the singular vector with the smallest singular value, and finally set the smallest singular value of $\mathbf{F}$ to zero to make it rank 2.

In raw pixel coordinates this is badly conditioned: the entries of $\mathbf{A}$ range from 1 to about $10^5$ (products of pixel coordinates). Hartley (1997) showed that simply shifting each point set to its centroid and scaling it to an average distance of $\sqrt{2}$ before solving, then undoing the scaling, makes the 8-point algorithm accurate. Measured on 200 random trials per setting:

With 100 matches and 1 px noise, the epipolar error is 0.31 px with normalisation and 2.72 px without, almost 9 times worse, for the price of two 3 × 3 matrices.

---

3. From E to the Camera Motion

The essential matrix can be decomposed with an SVD into a rotation and a translation, but not uniquely. There are two possible rotations and two signs of the translation, so four candidate poses. Only one of them puts the triangulated points in front of both cameras, and testing that (the cheirality check) picks it:

In practice the motion is estimated with the 5-point algorithm inside RANSAC (Nistér, 2004; OpenCV’s findEssentialMat) and decomposed with recoverPose. How accurate is it? The experiment compares three ways on 50 matches:

Three lessons:

1. Wrong matches decide everything. With only 5% wrong matches, the least-squares 8-point estimate of the translation direction is 67° off, essentially random. The RANSAC variants barely notice: with local optimisation, 3.7° with no wrong matches and 5.8° with 40% of them.
2. RANSAC’s best minimal sample is not the final answer. Plain RANSAC keeps the model of 5 points and is the least accurate on clean data (8.4° translation-direction error at 1 px noise); refining on the inliers (USAC with local optimisation) brings it to 3.5°.
3. Short baselines are hard. With 1 px noise and a 5 cm baseline the rotation is still found to within a degree, but the translation direction is 58–74° off: the parallax that reveals it is buried in the noise.

The scale is unknown

The essential matrix contains $\mathbf{t}$ only up to scale, and this is not a weakness of the algorithm. Two images cannot distinguish a small scene close by from a large scene far away, if the camera motion is scaled by the same factor:

A two-view reconstruction is therefore correct up to an unknown scale (with calibrated cameras; with uncalibrated ones, only up to a projective transformation). Something outside the images has to fix the scale: a known distance between two cameras (a stereo rig), an object of known size, an IMU, GPS or wheel odometry.

---

4. Triangulation

With both camera poses known, each match gives a 3D point: intersect the two viewing rays. With noise, the rays do not meet:

The linear method writes $\tilde{\mathbf{x}}\times \mathbf{P}\tilde{\mathbf{X}}=0$ for both views, 4 independent equations in $\tilde{\mathbf{X}}$, and solves them with an SVD (OpenCV’s triangulatePoints). The optimal point minimises the reprojection error, the distance between the observed pixels and the projections of $\mathbf{X}$ (Hartley & Sturm, 1997). The midpoint method is the simplest, but it is biased when the two cameras see the point from very different distances.

How precise is the result? For a rectified pair (two cameras side by side, a baseline $b$ apart), depth follows from the disparity $d$, the horizontal shift of the point between the images: $Z=fb/d$. A small disparity error $\delta$ gives a depth error of about

$$\Delta Z\approx \frac{Z^2}{fb}\delta$$

It grows with the square of the depth and shrinks with the baseline and focal length:

Measured with OpenCV’s linear triangulation (2,000 points per setting, 0.5 px noise, $f$ = 800 px), the formula fits closely as long as the disparity is large compared with the noise: with a 30 cm baseline, the RMS depth error at 16 m is 0.76 m against 0.75 m predicted. At 32 m with a 10 cm baseline, the disparity is only 2.5 px, the error distribution grows a long tail, and the RMS error (23.7 m) is far above the first-order prediction (9.1 m).

Stereo: the special case

A stereo camera fixes the geometry: two cameras with a known baseline, calibrated once. The images are rectified, warped by a homography each so that epipolar lines become the same image rows, and the match of every pixel is searched along its row. The result is a dense disparity map and, through $Z=fb/d$, a depth map. Semi-global matching (Hirschmüller, 2008) was the standard for a decade; today, deep networks estimate disparity, and monocular networks estimate depth from a single image. The $Z^2$ law still limits how far a stereo rig can see accurately.

---

5. Adding More Cameras: PnP

Once some 3D points exist, a new image can be placed by matching its features to those points and solving for its pose from 2D–3D correspondences: the Perspective-n-Point (PnP) problem.

• With known intrinsics, 3 points give a finite number of solutions (P3P, up to 4; Gao et al., 2003), and a fourth point picks the right one. EPnP (Lepetit et al., 2009) handles any $n\ge 4$ efficiently. In practice PnP runs inside RANSAC too.
• With unknown intrinsics, the whole $3\times 4$ projection matrix $\mathbf{P}$ can be estimated linearly (DLT): 11 degrees of freedom, 2 equations per point, so at least 6 points, and they must not all lie on one plane. $\mathbf{K}$ and $\mathbf{R}$ are then separated by an RQ decomposition of the left $3\times 3$ block of $\mathbf{P}$ (an upper-triangular times an orthogonal matrix; OpenCV’s decomposeProjectionMatrix).

---

6. Bundle Adjustment

Each step so far adds a little error, and the errors accumulate as more images are added. Bundle adjustment fixes this by refining all camera poses and all 3D points together, minimising the total reprojection error:

$$\underset{\{{\mathbf{R}}_i,{\mathbf{t}}_i\},\{{\mathbf{X}}_j\}}{\min }\sum _{(i,j)\text{observed}}\Vert {\mathbf{x}}_{ij}-\pi ({\mathbf{K}}_i,{\mathbf{R}}_i,{\mathbf{t}}_i,{\mathbf{X}}_j){\Vert }^2$$

where $\pi$ is the projection, including lens distortion. The name refers to the “bundles” of rays from each camera, which are adjusted until they agree. It is a non-linear least-squares problem, solved with Levenberg–Marquardt, and it stays tractable for thousands of images because each observation involves only one camera and one point: the Jacobian is very sparse, and the Schur complement eliminates the points to leave a small system in the cameras (Triggs et al., 2000).

A reconstruction from images alone can be moved, rotated and scaled without changing any reprojection error: 7 degrees of freedom (the gauge) must be fixed, for example by holding the first camera and the distance to the second. Here is a scene with 6 cameras and 300 points, started from a poor estimate:

From a start with cameras up to a metre off, bundle adjustment (SciPy’s sparse trust-region solver) converges in 4 steps: the reprojection error drops from 124 px to 1.21 px and the camera position error from 1.3 m to 1.4 cm. The final 1.21 px is not a failure to reach zero: with 1 px of noise per coordinate and 930 parameters fitted to 3,600 numbers, the expected residual is $\sqrt{2}\sqrt{1-930/3600}\approx 1.22$ px. A residual far below the noise would be a sign of overfitting.

---

7. Calibration: Where K Comes From

Everything above with an $\mathbf{E}$ assumes known intrinsics. The standard method (Zhang, 2000) photographs a flat checkerboard of known square size from several angles. Each view’s detected corners are related to the board by a homography; each homography gives two constraints on $\mathbf{K}$, so a closed-form estimate needs at least 3 views (2 if the skew is assumed zero). A non-linear refinement, a bundle adjustment over the board poses, then estimates $\mathbf{K}$, the distortion coefficients, and one $(\mathbf{R},\mathbf{t})$ per view, which is the board’s pose in that image, not a property of the camera.

How many views are enough, and does it matter which? Simulated with OpenCV’s calibrateCamera on a synthetic camera:

The focal length error falls from 13.5 px with 2 views to 1.9 px with 10 and about 1 px with 20; beyond that, more views barely help. The variety of the views matters more than their number. Ten views that all face the board head-on leave the focal length off by 292 px, because moving a fronto-parallel board closer and increasing $f$ look the same. Yet the reprojection error (0.41 px) is just as small as for a good calibration. A low reprojection error does not prove a correct calibration; tilted views, covering the image corners where distortion is strongest, do.

---

8. The Whole Pipeline

This is incremental structure from motion, as implemented in COLMAP (Schönberger & Frahm, 2016) and used, at a much larger scale, to reconstruct cities from internet photos (Snavely et al., 2006; Agarwal et al., 2011). Two steps are easy to forget when the pipeline is summarised: matching with RANSAC, which removes the wrong matches before anything is estimated, and repeated bundle adjustment as images are added, without which errors accumulate. The sparse result, camera poses plus a point cloud, is the input to dense methods: multi-view stereo, meshing, and today NeRF and 3D Gaussian splatting, which take COLMAP’s camera poses as their starting point.

Recent work replaces much of this pipeline with a single network. DUSt3R (Wang et al., 2024) and MASt3R (Leroy et al., 2024) regress 3D points for every pixel directly from an image pair with a Vision Transformer, and VGGT (Wang et al., 2025) predicts cameras, depth and points for many views in one forward pass. They are the subject of the next post, which connects this geometry with the attention and self-supervised learning of the earlier posts.

---

9. Common Claims, Checked

Claim	Verdict
”Epipolar geometry needs the positions and calibration of the cameras.”	It can be estimated from point matches alone: $\mathbf{F}$ from 7–8 matches, without any calibration. The camera motion is derived from it, not required for it.
”Use $\mathbf{E}$ for calibrated cameras, $\mathbf{F}$ for uncalibrated ones.”	Right, and they are related by $\mathbf{F}={\mathbf{K}}_2^{-\top }\mathbf{E}{\mathbf{K}}_1^{-1}$. $\mathbf{E}$ has 5 degrees of freedom, $\mathbf{F}$ has 7.
”The 8-point algorithm is numerically unstable.”	Only in raw pixel coordinates. With Hartley’s normalisation it is accurate: 0.31 px epipolar error instead of 2.72 px (100 matches, 1 px noise).
”Two images give the 3D structure of a scene.”	Up to an unknown scale (calibrated cameras) or a projective transformation (uncalibrated). The scale needs outside information.
”Decomposing $\mathbf{E}$ gives the camera motion.”	It gives four candidates; the one that puts triangulated points in front of both cameras is correct.
”The two viewing rays intersect at the 3D point.”	With noise they do not. Use the linear (DLT) or the reprojection-optimal solution.
”Depth from stereo is equally precise at all distances.”	The error grows with $Z^2$: in the experiment, 0.76 m at 16 m with a 30 cm baseline, and much worse than the formula once the disparity nears the noise.
”P3P needs 3 points.”	3 points give up to 4 solutions; a fourth point is needed to choose. With unknown intrinsics, the DLT needs at least 6 points, not all on one plane.
”K and R are separated from P by a QR decomposition.”	By an RQ decomposition of the left 3 × 3 block (upper-triangular $\mathbf{K}$ times orthogonal $\mathbf{R}$).
”Calibration gives the camera’s R and t.”	It gives the camera’s $\mathbf{K}$ and distortion, plus an $(\mathbf{R},\mathbf{t})$ for each view: the pose of the board in that image.
”More calibration images are always better.”	Up to about 10–20 varied views. Variety matters more: 10 head-on views gave a focal length 292 px off with a small reprojection error.
”Bundle adjustment is an optional final polish.”	It is what keeps the errors of incremental reconstruction from accumulating, and it is run repeatedly as images are added.

---

Summary

Idea	Why
Epipolar constraint ${\tilde{\mathbf{x}}}_2^{\top }\mathbf{F}{\tilde{\mathbf{x}}}_1=0$	A match lies on a line; F from matches alone
$\mathbf{E}=[\mathbf{t}{]}_{\times }\mathbf{R}$, 5-point + RANSAC	The camera motion, robust to wrong matches
Normalised 8-point	Linear estimation without numerical trouble
4 solutions + cheirality; scale unknown	What two views can and cannot determine
Triangulation, $\Delta Z\approx Z^2\delta /(fb)$	3D points, and why far points are imprecise
PnP (P3P, EPnP), DLT + RQ	Adding cameras to an existing reconstruction
Bundle adjustment	Refine all cameras and points together; fix the gauge
Checkerboard calibration (Zhang)	K and distortion; varied views matter more than many

Next: When Attention Meets Geometry: Learned Matching and 3D from Images: learned matchers like SuperGlue and LightGlue, and networks like DUSt3R and VGGT that predict 3D structure directly, built on the Transformers and self-supervised features of the earlier posts.

---

References

Two-view geometry

• Longuet-Higgins, H. C. (1981). A computer algorithm for reconstructing a scene from two projections. Nature, 293.
• Hartley, R. I. (1997). In defense of the eight-point algorithm. IEEE TPAMI, 19(6).
• Hartley, R. I., & Sturm, P. (1997). Triangulation. CVIU, 68(2).
• Nistér, D. (2004). An efficient solution to the five-point relative pose problem. IEEE TPAMI, 26(6).
• Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press.

Pose, calibration, stereo

• Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE TPAMI, 22(11).
• Gao, X.-S., Hou, X.-R., Tang, J., & Cheng, H.-F. (2003). Complete solution classification for the perspective-three-point problem. IEEE TPAMI, 25(8).
• Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. IJCV, 81(2).
• Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. IEEE TPAMI, 30(2).

Bundle adjustment and structure from motion

• Triggs, B., McLauchlan, P. F., Hartley, R. I., & Fitzgibbon, A. W. (2000). Bundle adjustment: A modern synthesis. Vision Algorithms: Theory and Practice, LNCS 1883.
• Snavely, N., Seitz, S. M., & Szeliski, R. (2006). Photo tourism: Exploring photo collections in 3D. ACM SIGGRAPH.
• Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., & Szeliski, R. (2011). Building Rome in a day. Communications of the ACM, 54(10).
• Schönberger, J. L., & Frahm, J.-M. (2016). Structure-from-motion revisited. CVPR.

Learned 3D reconstruction

• Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., & Revaud, J. (2024). DUSt3R: Geometric 3D vision made easy. CVPR.
• Leroy, V., Cabon, Y., & Revaud, J. (2024). Grounding image matching in 3D with MASt3R. ECCV.
• Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., & Novotny, D. (2025). VGGT: Visual geometry grounded transformer. CVPR.