Optical Flow from First Principles

1What optical flow actually is

The intuition

Put a finger on a pixel in frame 1, say a freckle on someone's cheek. Now find that same freckle in frame 2. It has moved four pixels right and one pixel down. Write that down as a little arrow, (+4, +1), and pin it to the original location.

Do that for every pixel in the image. What you get is optical flow: a two-channel image, the same width and height as the input, where instead of storing a color at each location you store a displacement.

So an RGB image is a function from pixel coordinates to color, and a flow field is a function from pixel coordinates to motion. Same grid, different payload.

What lives in memory

Concretely, for one Sintel frame pair at 1024×436, here is every array involved and its exact shape:

The file on disk

Flow is shipped in the Middlebury .flo format, which is about as simple as a binary format gets, and worth seeing, because it tells you exactly what a "ground truth" file is:

Invalid or unknown pixels are stored as any value with magnitude > 1e9. KITTI instead uses a 16-bit PNG: u = (R − 2¹⁵)/64, v = (G − 2¹⁵)/64, and the blue channel is a {0,1} validity flag, which is how a benchmark encodes "we don't know what happened here."

The mathematics

The single assumption underneath all of classical optical flow is brightness constancy: a point in the world keeps the same intensity as it moves.

That equation is exact but unusable: it relates a pixel to an unknown location. So take a first-order Taylor expansion of the right side around (x, y, t), assuming the motion is small:

First, the notation, because it is genuinely misleading. I is the intensity, the brightness of a pixel on some scale like 0 to 255. But the subscript in I_x is not multiplication and not a coordinate. It means rate of change with respect to, so I_x is shorthand for ∂I/∂x: how much the brightness changes when you step one pixel to the right. Nobody is adding an x coordinate to a brightness.

Once you read the subscripts that way, the units work out. Each term is a brightness change per frame:

Every term is a brightness change per frame, so they are the same kind of thing and adding them is legitimate. Read aloud, the equation says: the brightness change you would expect from sliding this pixel through the image, plus the brightness change actually observed, comes to zero. The two cancel, which is another way of saying the pixel kept its brightness while it moved.

The two unknowns are u and v, the horizontal and vertical displacement of this one pixel, which is exactly what we are trying to find. Everything else is measured straight off the two images.

Here is the whole problem in one worked pixel. Take a 3×3 patch sitting on a vertical edge, where brightness climbs by 20 per pixel to the right and does not change at all going down:

Now measure the three quantities, using nothing but those numbers:

Check the units on those numbers before going on. If the true motion is (2, 2), then moving 2 px right through a gradient of 20 intensity per pixel should darken this pixel by 20 × 2 = 40 intensity per frame. Moving 2 px down through a gradient of 0 changes nothing: 0 × 2 = 0. And the brightness actually measured did drop by 40 per frame. The predicted change and the observed change cancel, which is the whole content of the equation.

That is the one equation, and u and v are the two unknowns. The equation fixes u = 2, and then it has nothing left to say: because I_y is zero, the v term vanishes, so v can be anything at all. These all satisfy it exactly:

And they are not just algebraically valid, they are visually indistinguishable. Slide that patch straight down and the numbers do not change, because every row is identical. The image genuinely does not record how far it moved vertically. If the true motion was (2, 2), nothing in this pixel can tell you the 2.

The fix is to find a second, different equation. A nearby pixel sitting on a horizontal edge gives one, and its blind spot is the opposite:

Put the two together and you get u = 2 from the first and v = 2 from the second: a single answer, (2, 2). Two equations, two unknowns, solved. That is the entire idea behind Lucas–Kanade in Section 5, which gathers one equation per pixel over a whole window and solves them together.

What the vector (u, v) actually represents

It is worth being pedantic here, because the units trip people up constantly.

Four things follow that are easy to get wrong:

• The arrow is anchored in frame 1, not frame 2. F is indexed by frame-1 coordinates. This is forward flow. Backward flow, indexed by frame 2 and pointing back, is a different array, and the two are not simply negatives of each other wherever anything is occluded.
• Positive v is down, because image rows increase downward. Plotting libraries that assume y-up will render your entire flow field upside down.
• The units are pixels per frame, not pixels per second and definitely not metres per second. Double the frame rate and every number in the array halves. Halve the image resolution and every number halves again. A flow field is meaningless without knowing the resolution and frame interval it was computed at.
• The values are continuous. u = 3.72 is a normal, correct answer. Sub-pixel precision is what separates the top of the leaderboards: first and tenth place on Sintel are about half a pixel apart.

Typical magnitudes, so you have a feel for the scale: on Sintel about 60% of pixels move less than 10 px/frame, roughly 10% move more than 40 px/frame, and the maximum displacement in the dataset exceeds 400 px. On KITTI, driving at 50 km/h, road pixels near the bottom of the frame move 20–30 px/frame while pixels near the horizon move less than 1.

Dense vs. sparse

Same problem, two different output data structures, and that difference alone changes which algorithms are even applicable.

Flow ≠ tracking ≠ physical velocity

These three get conflated constantly, and the distinctions are load-bearing.

The bridge from flow to physics is scene flow: 3D motion per pixel, in metres. Getting there requires depth and camera intrinsics, because a small nearby object and a large distant one moving proportionally produce identical optical flow. This is also why KITTI and Spring are jointly stereo, flow, and scene-flow benchmarks. They carry the depth needed to make the conversion.

And point tracking (TAP-Vid, CoTracker) is the modern middle ground: track a set of points across hundreds of frames, through occlusion, with an explicit visibility flag. As of 2026 the flow and point-tracking literatures have substantially merged. See Section 5.

Why this is hard: the six failure modes

Every one of these is a case where the question "where did this pixel go?" is ill-posed, not merely difficult.

• Occlusion. A pixel visible in frame 1 is hidden in frame 2. There is no correct answer in the image data; the ground truth still specifies one (where the surface went), so the network must hallucinate it from context. On Sintel, error on occluded pixels runs roughly 10× higher than on visible ones, 0.69 vs 7.89 EPE for the current leader. Occlusions are perhaps 5–10% of pixels and dominate the score.
• Motion blur. Brightness constancy assumes a point keeps its intensity. A blurred point is smeared across dozens of pixels and its intensity is a mixture. This is exactly what Sintel's "final" pass adds over "clean," and it costs the leaders about 0.5 EPE.
• Large motion. Displacement beyond the algorithm's search radius. Coarse-to-fine methods handle it by shrinking the image until the motion is small, but see the next item.
• Small, fast objects. The pathological interaction: coarse-to-fine downsampling makes large motion tractable, but a thin fast object disappears at coarse levels, so the pyramid propagates the background's motion onto it and later levels cannot recover. This failure is the one RAFT was designed to eliminate.
• Textureless regions. Sky, walls, road surface. The structure tensor is singular; there is no local evidence whatsoever. The answer must be interpolated from distant boundaries.
• Camera motion. When the camera moves, every pixel moves, and the field is dominated by the ego-motion pattern: expansion from a focus of expansion when driving forward, near-uniform translation when panning. Flow magnitude then encodes depth, not object motion, which is why "is that car moving?" is not answerable from flow alone.

2How to read a flow field

Section 1 gave the short version of the colour code. This is the full one, including the exact convention, so you can hold the figures here up against the ones in any paper.

The practical reading skill: flat regions of uniform color are rigid surfaces moving together; sharp color boundaries are motion boundaries, usually object silhouettes; smooth gradients are surfaces slanting away from the camera or rotating. A good flow field ends up looking a lot like a segmentation of the scene.

The same motion at different frame rates

This is the clearest way to see why "large motion" is an algorithmic problem rather than a physical one. The ball's velocity never changes. Only the sampling interval does, and the numbers in the array scale linearly with it.

A real-world case: ego-motion versus object motion

The most common practical use of flow is separating "the camera moved" from "something moved." Because forward camera motion produces a highly structured expansion field, anything that doesn't fit that pattern is an independently moving object. This is the basis of motion segmentation, and of KITTI's foreground/background metric split.

3How benchmarks know the truth

To score a method you need the true displacement of every pixel. Nobody can label that by hand. A single Sintel frame would need 446,464 sub-pixel-accurate annotations. So the field has four tricks, and each one shapes the dataset built on it.

1. Render it. In a 3D renderer you know where every surface point is at both times, so you project both positions and subtract. Blender's vector pass does exactly this. dense, exact, free, unlimited synthetic
   Used by: Sintel, FlyingThings3D, Spring, AutoFlow, Monkaa, Driving.
2. Warp it. Take a real photo and apply a known transform to it. The flow is the transform, analytically. real textures, exact no real 3D motion, no real occlusion
   Used by: FlyingChairs (2D affine), and every data-augmentation pipeline.
3. Measure it with a laser. Mount a LiDAR and a GPS/IMU on a car. Accumulate scans into a 3D point cloud, register consecutive poses, and project. Moving objects break the static-world assumption, so KITTI 2015 additionally fits 3D CAD models to every moving vehicle. genuinely real imagery sparse (~19–50% of pixels), no sky, no thin structures
   Used by: KITTI 2012, KITTI 2015.
4. Hide the texture. The Middlebury trick, and still the most elegant idea in the literature: paint the scene in fluorescent paint that is invisible under normal light, then alternate between visible-light and UV illumination under computer control. The UV frames are covered in dense high-contrast random texture that makes matching trivial and near-exact; the visible-light frames are what you hand to the algorithm. real, dense, non-rigid lab-only, tiny, slow motion
   Used by: Middlebury 2007/2011.

Endpoint Error

The primary metric. It is exactly what it sounds like: the Euclidean distance between where you said the pixel went and where it actually went.

A worked example on four pixels:

Look at what happened: three of the four pixels are decent, one is catastrophic, and the mean is dominated entirely by the catastrophe. EPE is a mean of an extremely heavy-tailed distribution. That single property explains most of the metric design that follows, and why the leaderboards also report robust alternatives.

The other metrics, and why they exist

The KITTI outlier condition is a conjunction, and that matters: a pixel counts as an outlier only if the error exceeds 3 px and exceeds 5% of the true displacement's magnitude. So for a pixel that truly moved 80 px, an error of 3.5 px is forgiven. Fl-all is reported as a percentage, and the current best on KITTI 2015 is around 2.84%. That is: on 97 of every 100 pixels the leading method is essentially right, and the entire competition happens in the remaining three.

Spring went the opposite direction. At 1920×1080 with ground truth rendered at 4× super-resolution, a 3 px threshold is far too coarse to be informative, so its headline metric is the 1 px outlier rate, and the leaders sit around 3.3%, meaning roughly one pixel in thirty is not yet sub-pixel accurate.

What makes a benchmark hard

Difficulty is engineered, not incidental. The levers:

• Occlusion fraction. Sintel's unmatched-pixel EPE is ~11× its matched EPE. Adding fast foreground objects to a scene raises the difficulty faster than anything else.
• Displacement distribution. Not the maximum but the tail. A dataset where 5% of pixels move over 40 px punishes coarse-to-fine methods far more than one where everything moves 8 px.
• Thin structures and small objects. Hair, foliage, fences, wires. This is Spring's entire thesis, and why it renders ground truth at 4× resolution. Sub-pixel detail is invisible at 1× and methods were being scored on a blurred version of the truth.
• Nuisance imagery. Motion blur, defocus, atmospheric haze, fog, rain, sensor noise, low light. Sintel's clean/final split isolates this variable exactly; RobustSpring adds 20 corruption types on top of Spring.
• Non-rigid and non-Lambertian content. Water, smoke, fire, cloth, reflections, transparency. Brightness constancy fails outright and there is no correct pixel correspondence even in principle.
• Domain gap. The hardest property to design for. Nearly all training data is synthetic; the test set that matters may be real. A method can win Sintel and fall apart on KITTI, which is why generalisation results (train on Chairs+Things, test on Sintel/KITTI without fine-tuning) are reported separately and taken seriously.
• A withheld test set with a submission limit. The unglamorous but decisive one. Sintel, KITTI, and Spring all keep test ground truth secret and cap submissions to prevent tuning against the leaderboard.

4The five datasets that define the field

Two of them are training sets nobody evaluates on, three are benchmarks nobody trains on, and the standard recipe uses all five in a fixed order.

MPI Sintel 2012 · benchmark

Built from Sintel, an open-source animated short by the Blender Foundation. The authors took the actual production movie files and re-rendered them with the motion vector pass enabled, which is why the imagery is dramatically more complex than anything purpose-built: dragons, flowing capes, blowing snow, water, hair, characters running toward camera.

• Size: 35 sequences, 1628 frames, 1024×436. 23 sequences train / 12 test.
• Two render passes from the same geometry: clean (full shading, no blur or atmosphere) and final (adds motion blur, depth of field, and atmospheric effects). Same ground truth for both, a controlled experiment in how much nuisance imagery costs you.
• Ground truth: Blender render pass. Dense, exact, plus occlusion masks and invalid masks.
• Motion tested: very large displacement (>400 px), non-rigid deformation, severe occlusion, motion blur, specular and transparent materials.
• Metric: AEPE, split into matched/unmatched, plus s0-10/s10-40/s40+ and d0-10/d10-60/d60-140.
• Where it stands: the current final-pass leader, FreeFlow-L, is at 1.480 EPE overall, 0.689 on matched pixels but 7.894 on unmatched. Fourteen years in, occlusion is still where the error lives. Official leaderboard →

KITTI 2012 / 2015 · benchmark

The only one of the five with real photographs. A station wagon drove around Karlsruhe with two stereo camera pairs, a Velodyne HDL-64E laser scanner, and a GPS/IMU unit. Ground truth comes from registering accumulated laser point clouds using the measured vehicle pose and projecting them into the image.

• Size: KITTI 2015 has 200 training and 200 test pairs, roughly 1242×375.
• Ground truth: semi-dense. Laser returns cover only part of the frame: no sky, nothing beyond ~80 m, nothing on reflective or transparent surfaces. Roughly 19–50% of pixels have labels; the rest are excluded from scoring.
• The 2015 upgrade: KITTI 2012 assumed a completely static world, so moving cars had no valid ground truth at all. KITTI 2015 fixed this by fitting 3D CAD models from a library of car shapes to every moving vehicle, recovering their motion, which is exactly why the leaderboard splits Fl-bg from Fl-fg.
• Motion tested: forward camera motion (large expansion fields), independently moving vehicles, real sensor noise, real lighting, shadows, saturated sky.
• Metric: Fl-all. A pixel is an outlier if EPE > 3 px and > 5% of the true magnitude. Current leaders sit at Fl-all ≈ 2.84–2.94% (GeoMFlow, MS-RAFT-3D+, MEMFOF). Official leaderboard →

FlyingChairs 2015 · training only

Created for FlowNet because no training set of the required size existed. Deliberately, almost aggressively unrealistic: take a Flickr photo as a background, composite renders of 3D chair models on top, and apply independently sampled 2D affine transforms to the background and to each chair.

• Size: 22,872 image pairs at 512×384. 809 chair models × 62 viewpoints.
• Ground truth: analytic. The affine transforms are chosen, not measured, so the flow is exact by construction and costs nothing to produce.
• Motion tested: 2D translation, rotation, and scale only. No 3D structure, no perspective, no realistic occlusion relationships. Chairs simply slide over a background.
• Why it still matters: it demonstrated that a network trained on obviously fake data generalises to real video, which is the premise the whole deep-flow literature rests on. It remains stage one of the standard training curriculum, because its simple motions teach matching before the harder data teaches 3D.

FlyingThings3D 2016 · training only

The 3D successor, part of the Scene Flow Datasets (with Monkaa and Driving). Random ShapeNet objects are launched along randomised 3D trajectories through a scene with a moving camera, rendered in Blender.

• Size: ~22,000 stereo frame pairs at 960×540, ~2,250 sequences.
• Ground truth: Blender render passes: optical flow, disparity, disparity change, object segmentation, camera pose. Enough for full scene flow.
• Motion tested: genuine 3D motion with perspective, real occlusion and disocclusion, very large displacements, objects entering and leaving frame.
• Why it still matters: stage two of the curriculum. The canonical schedule, C → T → S/K fine-tune, meaning Chairs, then Things, then Sintel or KITTI, was established by FlowNet 2.0 and is still what nearly every 2026 paper reports. Training on Things first actively hurts; the easy-to-hard order is load-bearing.

Spring 2023 · benchmark

The modern benchmark, and a direct response to a specific complaint: Sintel is only 1024×436, so sub-pixel structure (hair, foliage, thin branches) is not even representable in the ground truth, and methods were being penalised for being more accurate than the labels.

• Size: 6,000 frame pairs from 47 sequences of the Blender open movie Spring, at 1920×1080. Roughly 60× more annotated pixels than Sintel.
• Ground truth: Blender, rendered at 4× super-resolution (3840×2160) specifically so that sub-pixel detail survives. Provides forward and backward flow for both stereo views, plus disparity and disparity change: four flow maps per frame, enabling stereo, flow, and scene flow on one dataset.
• Motion tested: high-detail thin structures, high resolution (so large pixel displacements even for modest real motion), and via RobustSpring, 20 corruption types (blur, noise, weather, compression) applied temporally consistently.
• Metric: 1 px outlier rate as the headline, with detail-focused and matched/unmatched breakdowns. The current leader MEMFOF is at 3.289%. Official benchmark →

5The algorithms

Every method here answers the same question, where do I get the second equation?, and the history is a steady march from hand-written assumptions toward learned ones, with one structural idea (iterative refinement at a single resolution) turning out to matter more than any of them.

Lucas–Kanade (1981)

The core idea: one pixel gives one equation. So take a 5×5 window, assume all 25 pixels share the same motion, and you have 25 equations for 2 unknowns. Solve by least squares.

A is the structure tensor, the same matrix as the Harris corner detector, which is not a coincidence. Its eigenvalues tell you whether the window contains enough structure to solve for both components:

min(λ₁, λ₂) is the Shi–Tomasi "good feature to track" score. This is why sparse flow tracks corners: they are precisely the pixels where the 2×2 system is well-conditioned. It is also why plain LK handles only small motion. The Taylor expansion is only valid for displacements under about a pixel. The fix is the pyramid: downsample until the motion is sub-pixel, solve, upsample the estimate, warp, and solve for the residual. Four levels turn a 1 px capability into roughly 16 px.

Farnebäck (2003)

The core idea: stop linearising the image and fit it to a quadratic instead. Approximate the neighbourhood of every pixel by a 2D polynomial:

Now the trick. If the image is displaced by d, substitute x → x − d and expand. The quadratic term is unchanged, and the linear term shifts in a way that isolates d exactly:

In practice A is averaged over a neighbourhood for stability, and the whole thing runs coarse-to-fine. The result is fully dense, runs in real time on a CPU, and was the default in OpenCV (calcOpticalFlowFarneback) for a decade. Its signature weakness: the quadratic model is a smoothness assumption in disguise, so motion boundaries get blurred and small objects are absorbed into the background.

FlowNet (2015): flow as supervised learning

The core idea: stop designing the matching function; learn it. Two variants shipped in the same paper:

• FlowNetS ("simple"): stack the two frames into a 6-channel input and hand it to a plain encoder–decoder CNN. It works, which was itself the surprising result.
• FlowNetC ("correlation"): process each frame through its own weight-shared tower, then insert an explicit correlation layer that computes the dot product between a feature in frame 1 and features in a neighbourhood of frame 2. This bakes "matching" into the architecture rather than hoping the network invents it.

FlowNet 2.0 (2017) made it competitive by stacking multiple FlowNets, each one warping frame 2 by the previous estimate and predicting a residual, plus a dedicated small-displacement branch and a fusion network. It also established the Chairs → Things3D training curriculum that everyone still uses.

PWC-Net (2018): three classical principles, made learnable

The core idea: take the three things classical methods always did (, Warping, cost volume) and make each one a learned module. At every pyramid level, from coarse to fine:

1. Warp frame 2's feature map (not the image) by the upsampled flow from the level above.
2. Build a cost volume between frame 1's features and the warped features, but only over a ±4 pixel search radius, because after warping the remaining motion is small. That's 81 channels instead of a full all-pairs volume.
3. Run a small CNN to predict the residual flow, then pass to the next level.

The payoff was efficiency: PWC-Net is 17× smaller than FlowNet2 and more accurate. The cost is structural, and it is the flaw RAFT was built to fix. Coarse-to-fine cannot recover an object that vanished at a coarse level. A thin, fast-moving limb is gone by level 4, the pyramid assigns it the background's motion, and the ±4 search at fine levels can never find it again.

RAFT (2020): the architecture that reset the field

Best paper at ECCV 2020, and still the backbone that most subsequent methods modify. It has three parts.

1. Feature encoders. One weight-shared encoder maps both frames to 256-channel features at 1/8 resolution. A separate context encoder runs on frame 1 only.
2. An all-pairs correlation volume. Compute the dot product between every feature in frame 1 and every feature in frame 2, a full 4D tensor (H/8, W/8, H/8, W/8). No search radius, no coarse-to-fine, no assumption about how far anything moved. It is then average-pooled into a 4-level pyramid over the last two dimensions only, so large displacements are covered without ever downsampling frame 1.
3. A recurrent update operator. Start from flow = 0. Repeat 12–32 times: look up correlation values in a small window around the current flow estimate, feed them plus context into a GRU, and get an increment Δf. Add it. The weights are shared across all iterations. This is a learned optimiser rather than a deeper network.

The key structural difference: RAFT maintains a single flow field at a single resolution and refines it repeatedly, instead of building it up across a pyramid. Nothing ever disappears, so small fast objects are recoverable, because the entire correlation volume was computed at full 1/8 resolution before iteration began.

2021 → 2026: what happened after RAFT

Five threads, all still active. Almost everything below keeps RAFT's iterative-refinement skeleton and changes what feeds it.

The through-line: the twelve years from FlowNet to MegaFlow moved the source of the "second equation" from a hand-written smoothness term, to a task-specific network trained on synthetic chairs, to a general-purpose visual representation trained on everything. What stayed constant is RAFT's insight that you should refine one full-resolution estimate iteratively rather than build it up a pyramid.

6End to end, with real numbers

Two frames in, a benchmark score out. Everything below runs for real in your browser: the scene is rendered as separate layers with exactly known motion, so the ground truth is analytic, and a genuine pyramidal Lucas–Kanade solver produces the prediction.

Two things are worth taking away from the numbers above. First, the errors are not spread evenly. They concentrate on occlusion boundaries, on the fastest-moving layer, and in the textureless road and sky where the structure tensor is singular. Second, this classical solver would place nowhere near a modern leaderboard, and the gap is almost entirely in exactly those regions. Every architectural idea in Section 5 (the pyramid, the all-pairs volume, global aggregation, multi-frame context, borrowed ViT features) is aimed at one of the bright patches in that error map.

Where to go next

• Run the real thing: torchvision.models.optical_flow.raft_large ships with pretrained weights and takes about ten lines. Compare its output on your own video against cv2.calcOpticalFlowFarneback. The difference at motion boundaries is startling.
• Download Sintel's training split (it includes ground truth) and compute EPE yourself. Then compute it separately on the occluded and non-occluded masks, and you'll feel the 10× gap directly.
• Read the RAFT paper before any of the newer ones; nearly everything since is a modification of it.

Sources

Benchmark figures were checked against the official leaderboards while writing this page (August 2026); they change frequently.

• MPI Sintel optical flow leaderboard. Butler, Wulff, Stanley & Black, ECCV 2012
• KITTI 2015 optical flow leaderboard. Menze & Geiger, CVPR 2015
• The Spring dataset & benchmark. Mehl, Schmalfuss, Jahedi, Nalivayko & Bruhn, CVPR 2023 (paper)
• Middlebury optical flow and Baker et al., IJCV 2011. The fluorescent-texture ground-truth method, and the origin of EPE and the color code
• MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow. Current Spring / KITTI / Sintel-clean leader
• MegaFlow: Zero-Shot Large Displacement Optical Flow. Zhang, Wang, Pollefeys & Xu, 2026 (project page)
• SEA-RAFT: Simple, Efficient, Accurate RAFT. Wang, Lipson & Deng, ECCV 2024
• MemFlow: Optical Flow Estimation and Prediction with Memory. CVPR 2024

Every figure on this page is generated procedurally at load time: the scenes, the flow fields, the cost volumes, the solver, and the scores. Dataset panels in Section 4 are schematic recreations, not real dataset frames.

7Practice problems

Twenty problems, all doable on paper in under five minutes each. Work them before opening the solution. The arithmetic is the point, because the quantities are exactly the ones you'll be printing to a console when something goes wrong.