From a 3D Point to a Pixel: Image Formation

Every network in this series so far has treated an image as a grid of numbers. But an image is also a measurement of the 3D world: light from a point in a scene passes through a lens and lands on one pixel. Where it lands follows a few lines of geometry, and that geometry is what makes it possible to stitch panoramas, place virtual objects in a camera view, measure distances from photos, or rebuild a scene in 3D.

It also shows up inside deep learning more often than one might think. The random crops, flips and rotations used for augmentation in Learning Without Labels are 2D transformations from this post, and the training code there applied them with affine_grid and grid_sample, which is exactly the inverse warping described in §3. Neural radiance fields and 3D Gaussian splatting start from the camera model in §5–6.

This post builds that model step by step, from homogeneous coordinates to the full projection $\tilde{\mathbf{x}}=\mathbf{K}[\mathbf{R}\mid \mathbf{t}]\tilde{\mathbf{X}}$, with an interactive lab for each step. The lab’s maths is checked against OpenCV (§10). The following posts use it to match features between images and to reconstruct 3D scenes.

---

1. Homogeneous Coordinates

A point in an image is normally written $(x,y)$. Homogeneous coordinates add a third number: $(x,y,w)$, with the rule that two vectors describe the same point if one is a multiple of the other, so $(x,y,w)\sim (kx,ky,kw)$ for any $k\ne 0$. To get the ordinary coordinates back, divide by the last entry: $(x/w,y/w)$. A tilde marks a homogeneous vector: $\tilde{\mathbf{x}}=(x,y,1)$.

This looks like bookkeeping, but it has a geometric meaning. Each image point corresponds to a ray through the origin, and the image is where the rays cross the plane $w=1$:

Three things become simple with this extra coordinate:

1. Translation becomes a matrix product. $x^{\prime }=x+t_x$ is not linear in $(x,y)$, but in homogeneous coordinates it is: ${\tilde{\mathbf{x}}}^{\prime }=\left[\begin{array}{ccc}1& 0& t_x\\ 0& 1& t_y\\ 0& 0& 1\end{array}\right]\tilde{\mathbf{x}}$. So every transformation in this post, including the camera itself, is one matrix, and a chain of them is one matrix product.
2. Lines are vectors too. The line $ax+by+c=0$ is $\tilde{\mathbf{l}}=(a,b,c)$, and a point lies on it if ${\tilde{\mathbf{l}}}^{\top }\tilde{\mathbf{x}}=0$. The intersection of two lines is their cross product, $\tilde{\mathbf{x}}={\tilde{\mathbf{l}}}_1\times {\tilde{\mathbf{l}}}_2$, and the line through two points is ${\tilde{\mathbf{x}}}_1\times {\tilde{\mathbf{x}}}_2$.
3. Points at infinity are ordinary vectors. Two parallel lines, for example $x=-1$ and $x=1$, are $(1,0,1)$ and $(1,0,-1)$; their cross product is $(0,2,0)$. This vector has $w=0$: it is the point at infinity in the direction $(0,1)$, where the two parallel lines “meet”.

The third point is what a camera makes visible. Parallel lines on a road meet at a vanishing point in a photo, because the camera’s projective transformation maps their point at infinity to an ordinary point:

---

2. The 2D Transformations

Every 2D transformation in this post is a 3 × 3 matrix acting on homogeneous coordinates. They form a hierarchy: each class adds freedom and preserves less.

Transformation (top two rows of the matrix)	DOF	Preserves
Translation $[\mathbf{I}\mid \mathbf{t}]$	2	orientation, lengths, angles
Euclidean (rigid) $[\mathbf{R}\mid \mathbf{t}]$	3	lengths, angles
Similarity $[s\mathbf{R}\mid \mathbf{t}]$	4	angles, ratios of lengths
Affine $[\mathbf{A}\mid \mathbf{t}]$, any 2 × 2 $\mathbf{A}$	6	parallel lines, ratios of areas
Projective (homography) $\mathbf{H}$, all 9 entries, up to scale	8	straight lines, cross-ratios

For the first four the last row is $(0,0,1)$, so a point at infinity ($w=0$) stays at infinity: parallel lines stay parallel. Only a homography, whose last row can be anything, moves points at infinity into the image, which is why it can model perspective. A homography has 9 entries but only 8 degrees of freedom, since $\mathbf{H}$ and $k\mathbf{H}$ give the same mapping.

The degrees of freedom tell you how many point correspondences are needed to estimate a transformation. Each point pair gives 2 equations, so a translation needs 1 pair, a similarity 2, an affine transformation 3, and a homography 4 (no three of them on a line). With more pairs than that, the parameters are fitted by least squares. Drag the corners and switch between the models:

Estimating a homography from 4 points

For a pair $(x,y)\to (u,v)$, the equation $\tilde{\mathbf{u}}\sim \mathbf{H}\tilde{\mathbf{x}}$ says $u=\frac{h_{11}x+h_{12}y+h_{13}}{h_{31}x+h_{32}y+h_{33}}$, and similarly for $v$. Multiplying out the denominator gives two equations that are linear in the 9 entries of $\mathbf{H}$:

$$h_{11}x+h_{12}y+h_{13}-u(h_{31}x+h_{32}y+h_{33})=0$$ $$h_{21}x+h_{22}y+h_{23}-v(h_{31}x+h_{32}y+h_{33})=0$$

Stacking these for all pairs gives $\mathbf{Ah}=0$, and the solution is the right singular vector of $\mathbf{A}$ with the smallest singular value: the direct linear transform (DLT). With exactly 4 pairs it is exact; with more noisy pairs it minimises an algebraic error, and it is much more accurate if the points are first normalised (shifted to their centroid and scaled to an average distance of $\sqrt{2}$, Hartley 1997). With wrong matches among the pairs, least squares breaks down completely; that is the job of RANSAC, in the next post.

---

3. Warping an Image: Backwards, Not Forwards

A transformation matrix multiplies coordinates, not images. To produce a warped image, there are two options:

• Forward mapping: send every source pixel to $\mathbf{H}\tilde{\mathbf{x}}$ and round. Where the image is enlarged, several target pixels receive nothing, leaving holes (tick “map pixels forward” in the lab above to see them), and where it shrinks, several source pixels land on the same target.
• Inverse mapping: for every target pixel, compute where it comes from, ${\mathbf{H}}^{-1}\tilde{\mathbf{u}}$, and interpolate the source image there (bilinear interpolation uses the 4 nearest pixels). Every target pixel gets exactly one value.

Image libraries (OpenCV’s warpPerspective, PyTorch’s grid_sample) all use inverse mapping. It is also what makes warping differentiable: in a spatial transformer network (Jaderberg et al., 2015), a small network predicts the 6 parameters of an affine transformation, affine_grid computes the source location of every output pixel, and grid_sample interpolates there, so the gradient flows back into the predicted parameters.

---

4. Transformations in 3D

The same hierarchy exists in 3D, with 4 × 4 matrices acting on $\tilde{\mathbf{X}}=(X,Y,Z,1)$: translation (3 degrees of freedom), Euclidean (6), similarity (7), affine (12) and projective (15).

The important one for cameras is the rigid (Euclidean) transformation ${\mathbf{X}}^{\prime }=\mathbf{RX}+\mathbf{t}$. A 3D rotation matrix has 9 entries but only 3 degrees of freedom, because it must satisfy ${\mathbf{R}}^{\top }\mathbf{R}=\mathbf{I}$ and $\det \mathbf{R}=1$. It can be written in several ways:

• Euler angles (yaw, pitch, roll): intuitive, but the order matters, and at certain angles two of them describe the same motion (gimbal lock).
• Axis–angle (a rotation of $\theta$ about a unit axis $\hat{\mathbf{n}}$, stored as the 3-vector $\theta \hat{\mathbf{n}}$): what OpenCV calls rvec, converted to a matrix by Rodrigues’ formula.
• Unit quaternions (4 numbers with norm 1): no singularities, easy to interpolate, common in robotics and graphics.

---

5. The Pinhole Camera

Light from a scene point passes through a tiny hole and hits the sensor behind it. The image on the sensor is upside down; mathematically it is easier to put a virtual image plane in front of the pinhole, at the same distance $f$:

By similar triangles, a point at $(X,Y,Z)$ in the camera’s coordinate frame (with $Z$ along the optical axis) appears at

$$x=f\frac{X}{Z},y=f\frac{Y}{Z}$$

The division by $Z$ is perspective: an object twice as far away appears half as large. It is also where depth is lost: every point on the same ray through the pinhole lands on the same pixel. In homogeneous coordinates the division becomes a matrix product followed by the usual division by the last entry:

$$\left[\begin{array}{c}fX\\ fY\\ Z\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]$$

The projection matrix is 3 × 4: it maps 3D (4 homogeneous coordinates) to 2D (3 homogeneous coordinates). Computer graphics pipelines use a 4 × 4 version that keeps a depth value for the z-buffer, but for forming an image, one row is dropped.

Orthographic projection drops the depth instead of dividing by it: $(x,y)=(X,Y)$, with matrix rows $(1,0,0,0)$, $(0,1,0,0)$, $(0,0,0,1)$. It is a good approximation (as scaled orthographic or weak perspective, $x=sX$) when an object’s depth range is small compared with its distance, for example a telephoto shot.

---

6. Intrinsics: From the Image Plane to Pixels

The pinhole equation gives positions in the image plane, measured from the optical axis. Pixel coordinates start at the top-left corner, and pixels have a size. The intrinsic matrix $\mathbf{K}$ converts one into the other:

$$\mathbf{K}=\left[\begin{array}{ccc}{f}_x& s& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$$

• $f_x,f_y$: the focal length in pixels: the physical focal length divided by the pixel width and height. They differ only for non-square pixels.
• $c_x,c_y$: the principal point, where the optical axis meets the sensor, usually close to the image centre but not exactly at it.
• $s$: skew, non-zero only if the pixel grid is not rectangular; zero for practically every modern camera.

Focal length from the field of view

If only the field of view (FOV) is known, the focal length in pixels follows from the triangle formed by half the image and half the angle:

$$f=\frac{\text{extent}/2}{\tan (\text{FOV}/2)}$$

The catch is the word extent. A field of view of 100° on a 640 × 480 image gives $f=335.6$ px if the 100° is measured across the diagonal (800 px), $268.5$ px across the width, and $201.4$ px across the height. Specifications are given in all three ways:

---

7. Extrinsics and the Full Camera Matrix

The pinhole equation needs points in the camera’s coordinate frame. Points in a scene are usually given in a world frame, so first they are moved into the camera frame with a rigid transformation, the extrinsics $[\mathbf{R}\mid \mathbf{t}]$. Then the whole chain is one 3 × 4 matrix:

$$\tilde{\mathbf{x}}=\mathbf{K}[\mathbf{R}\mid \mathbf{t}]\tilde{\mathbf{X}}=\mathbf{P}\tilde{\mathbf{X}}$$

Two details trip up almost everyone:

• $\mathbf{t}$ is not the camera’s position. $[\mathbf{R}\mid \mathbf{t}]$ maps world points into the camera frame, ${\mathbf{X}}_c=\mathbf{R}{\mathbf{X}}_w+\mathbf{t}$. The camera centre $\mathbf{C}$ is the world point that lands at the origin, so $\mathbf{RC}+\mathbf{t}=0$, which gives $\mathbf{C}=-{\mathbf{R}}^{\top }\mathbf{t}$, and $\mathbf{t}=-\mathbf{RC}$.
• Axis conventions differ. OpenCV uses $x$ right, $y$ down, $z$ forward (into the scene). A world frame often has $z$ up. The rows of $\mathbf{R}$ are the camera’s right, down and forward directions written in world coordinates; for a camera looking straight along the world $y$ axis they are $(1,0,0)$, $(0,0,-1)$ and $(0,1,0)$. OpenGL and many graphics tools use $y$ up and $z$ backwards instead, a common source of upside-down or mirrored results.

$\mathbf{P}$ has 12 entries but 11 degrees of freedom (scale does not matter): 5 intrinsic, 3 for the rotation and 3 for the translation. Move the camera and change its intrinsics, and follow one corner of the blue box through every step of the chain:

---

8. From One Camera to Another

Given a pixel in one image, where does the same scene point appear in a second image? Take the second camera’s frame relative to the first: ${\mathbf{X}}_1=\mathbf{R}{\mathbf{X}}_0+\mathbf{t}$. A pixel ${\tilde{\mathbf{x}}}_0$ only fixes the ray ${\mathbf{X}}_0=Z_0{\mathbf{K}}_0^{-1}{\tilde{\mathbf{x}}}_0$; the point on it depends on its depth $Z_0$. Projecting into camera 1:

$${\tilde{\mathbf{x}}}_1\sim {\mathbf{K}}_1\mathbf{R}{\mathbf{K}}_0^{-1}{\tilde{\mathbf{x}}}_0+\frac{1}{Z_0}{\mathbf{K}}_1\mathbf{t}$$

So in general you need each pixel’s depth (or its inverse depth $1/Z_0$, which is how many depth-estimation and SLAM methods store it). The second term is the parallax: it shifts a point by about $fb/Z$ pixels for a sideways baseline $b$, a lot for near points and little for far ones. That is how two eyes, or two cameras, perceive depth.

There are two cases where the depth drops out and one 3 × 3 homography maps one image onto the other:

• Pure rotation ($\mathbf{t}=0$, the same camera centre): ${\tilde{\mathbf{x}}}_1\sim {\mathbf{K}}_1\mathbf{R}{\mathbf{K}}_0^{-1}{\tilde{\mathbf{x}}}_0$ for every point at every depth. This is why a panorama shot by turning a camera on a tripod can be stitched perfectly, and why hand-held panoramas show ghosting on nearby objects. The condition is the same centre, ${\mathbf{C}}_0={\mathbf{C}}_1$; equal translation vectors ${\mathbf{t}}_0={\mathbf{t}}_1$ are not the same thing, since $\mathbf{C}=-{\mathbf{R}}^{\top }\mathbf{t}$ also depends on $\mathbf{R}$.
• A planar scene: for points on the plane ${\mathbf{n}}^{\top }{\mathbf{X}}_0=d$, the factor ${\mathbf{n}}^{\top }{\mathbf{X}}_0/d$ equals 1, so $\mathbf{t}=\mathbf{t}{\mathbf{n}}^{\top }{\mathbf{X}}_0/d$ and ${\tilde{\mathbf{x}}}_1\sim {\mathbf{K}}_1(\mathbf{R}+\mathbf{t}{\mathbf{n}}^{\top }/d){\mathbf{K}}_0^{-1}{\tilde{\mathbf{x}}}_0$. This is the homography behind document scanning and the checkerboard used for calibration.

Turn the second camera, then move it sideways:

With rotation only, the predicted rings sit exactly on the true points (the error is about $10^{-13}$ px, rounding). Once the camera moves, no single homography fits both the near box and the far wall: with a 0.8 m baseline, the plane homography is exact on the wall and about 80 px off on the box. Recovering the depth that is missing here is the subject of 3D reconstruction.

---

9. Real Lenses: Distortion

A pinhole lets through too little light, so real cameras use lenses, and lenses bend straight lines, most visibly near the image border. The standard model (Brown, 1966; used by OpenCV) acts on the normalised coordinates $x=X/Z$, $y=Y/Z$ before $\mathbf{K}$. With $r^2=x^2+y^2$:

$$x_d=x(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)+2p_{1}xy+p_2(r^2+2x^2)$$ $$y_d=y(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)+p_1(r^2+2y^2)+2p_{2}xy$$

The $k$ terms are radial distortion: negative $k_1$ gives barrel distortion (wide-angle lenses), positive $k_1$ gives pincushion (telephoto lenses), and a mix gives a wavy “mustache” shape. The $p$ terms are tangential distortion, from a lens that is not exactly parallel to the sensor.

Distortion is not a matrix: no 3 × 3 transformation can bend a straight line. So in practice the coefficients are estimated together with $\mathbf{K}$ by calibration: photographing a checkerboard of known size from several angles and fitting all parameters so that the projected corners match the detected ones (Zhang, 2000). Images are then undistorted before the pinhole equations are applied. Fisheye lenses (field of view near or beyond 180°) need a different model altogether, since $\tan (\text{FOV}/2)$ goes to infinity at 90° (Kannala & Brandt, 2006).

---

10. Checking the Lab Against OpenCV

Every formula in the lab is reimplemented in plain JavaScript. To make sure it matches the standard tools, a Python script computes the same quantities with OpenCV 5.0 and numpy, and the lab’s functions are compared with them in a browser:

What	Reference	Largest difference
homography from 4 corners	cv2.getPerspectiveTransform	7 × 10⁻¹²
affine and similarity least-squares fits	numpy.linalg.lstsq	1 × 10⁻¹²
Euclidean fit	Procrustes (SVD) in numpy	1 × 10⁻⁶ (float32 input)
projection of 3D points with K, R, t	cv2.projectPoints	1 × 10⁻¹³ px
the same with lens distortion (k₁, k₂, p₁, p₂)	cv2.projectPoints with distCoeffs	9 × 10⁻¹⁴ px
rotation-only homography at zero baseline	exact projection	1 × 10⁻¹³ px
plane homography on the far wall	exact projection	1 × 10⁻¹³ px

---

11. Common Claims, Checked

Claim	Verdict
”Homogeneous coordinates just add a 1.”	The 1 is only the default. The point is the scale equivalence $(x,y,w)\sim (kx,ky,kw)$: it turns translation and perspective division into matrix products, and it gives points at infinity ($w=0$) a normal representation.
”Under affine transformations, parallel lines remain parallel.”	True. The last row of an affine matrix is $(0,0,1)$, so points at infinity stay at infinity. Under a homography they need not, and parallel lines can meet at a vanishing point.
”All 2D transformations are done by multiplying the image with a matrix.”	The matrix multiplies coordinates. The image itself is produced by inverse warping and interpolation (§3), and lens distortion is not a matrix at all.
”In an affine matrix, $a_{11}$ and $a_{22}$ are the scale and $a_{12}$, $a_{21}$ the shear.”	Only if there is no rotation. In general rotation, scaling and shear mix in all four entries; to separate them, decompose $\mathbf{A}$ (for example $\mathbf{A}={\mathbf{R}}_1\text{diag}(s_1,s_2){\mathbf{R}}_2$ by the SVD).
”The perspective projection is a 4 × 4 matrix.”	For image formation it is 3 × 4: $[\mathbf{I}\mid 0]$ (times $\mathbf{K}$). A 4 × 4 version, with a zero or depth row, only appears when a depth value has to be kept, as in graphics pipelines.
”To get pixel coordinates, it is enough to transform world points into the camera frame.”	No, that gives metres in the camera frame. Dividing by the depth and applying $\mathbf{K}$ are still needed.
”The camera matrix is $\mathbf{P}=[\mathbf{R}\mid \mathbf{t}]$.”	$[\mathbf{R}\mid \mathbf{t}]$ alone gives normalised coordinates (a camera with $f=1$ and the principal point at 0). The pixel projection is $\mathbf{P}=\mathbf{K}[\mathbf{R}\mid \mathbf{t}]$.
”$\mathbf{t}$ is the position of the camera.”	The camera centre is $\mathbf{C}=-{\mathbf{R}}^{\top }\mathbf{t}$. $\mathbf{t}$ is the world origin expressed in camera coordinates.
”A 100° camera at 640 × 480 has $f=335.6$ px.”	Only if the 100° is the diagonal field of view. Horizontal gives 268.5 px, vertical 201.4 px.
”Two cameras with the same $\mathbf{t}$ are related by a homography ${\mathbf{K}}_1\mathbf{R}{\mathbf{K}}_0^{-1}$.”	The condition is the same camera centre. With different rotations, equal $\mathbf{t}$ vectors mean different centres.
”4 point pairs are enough to estimate a homography.”	4 pairs in general position (no 3 collinear) determine it exactly. Real matches are noisy and some are wrong, so in practice many pairs, normalisation and RANSAC are used.

---

Summary

Idea	Why
Homogeneous coordinates	Translation and projection become matrix products; points at infinity are ordinary vectors
2D hierarchy: 2, 3, 4, 6, 8 degrees of freedom	Tells you what is preserved and how many correspondences are needed (1, 2, 2, 3, 4)
DLT, with normalisation	Estimates a homography from point pairs by linear algebra
Inverse warping + interpolation	How images (and spatial transformer networks) are actually warped
Pinhole: $x=fX/Z$	Perspective, and the loss of depth
$\mathbf{P}=\mathbf{K}[\mathbf{R}\mid \mathbf{t}]$, 11 degrees of freedom	The whole camera in one 3 × 4 matrix; $\mathbf{C}=-{\mathbf{R}}^{\top }\mathbf{t}$
Rotation-only or planar scene → homography	Panoramas, document scans; otherwise parallax needs depth
Lens distortion + calibration	Real lenses bend lines; estimate $\mathbf{K}$ and distortion, then undistort

Next: Finding the Same Point Twice: SIFT, the Ratio Test and RANSAC: detecting and describing keypoints with SIFT, matching them between images, and estimating a homography robustly with RANSAC, even when many matches are wrong.

---

References

Books

• Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.). Springer. Free at szeliski.org/Book.

Papers

• Hartley, R. I. (1997). In defense of the eight-point algorithm. IEEE TPAMI, 19(6).
• Brown, D. C. (1966). Decentering distortion of lenses. Photogrammetric Engineering, 32(3).
• Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE TPAMI, 22(11).
• Kannala, J., & Brandt, S. S. (2006). A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE TPAMI, 28(8).
• Jaderberg, M., Simonyan, K., Zisserman, A., & Kavukcuoglu, K. (2015). Spatial transformer networks. NeurIPS.
• Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Representing scenes as neural radiance fields for view synthesis. ECCV.
• Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian splatting for real-time radiance field rendering. ACM TOG (SIGGRAPH).

Tools

• OpenCV camera calibration and 3D reconstruction module (calib3d), whose conventions this post follows.