Test-time geometry constraints improve vision models

Self-Geometry, a test-time add-on, lifts depth and pose scores of vision foundation models by up to 37.3 % for depth and 9.2 % for pose on the ETH3D benchmark.

Vision foundation models such as VGGT predict scene depth and camera pose in a single forward pass. The convenience comes at a cost: they treat each view independently and ignore the epipolar constraints that bind multiple images of the same scene. Existing “self-consistency” tricks try to spot contradictions in a model’s own outputs, but when the initial prediction is far off, the correction collapses.

Self-Geometry avoids that flaw. It first extracts 2-D point correspondences between overlapping images and treats those matches as pseudo ground truth. During inference it runs a lightweight optimizer that adjusts the model’s depth and pose outputs to satisfy two loss terms:

  1. Multi-view consistency – the same 3-D point must project correctly into every view.
  2. Epipolar consistency – the classic line-of-sight rule of stereo geometry. No gradients flow back into the backbone, so the original weights stay untouched.

On the ETH3D benchmark, the method raises VGGT’s pose accuracy by 9.2 % and its depth accuracy by 37.3 %. Other models see gains of 5.0 % and 25.1 %. The improvement holds across six architectures and four public datasets, showing the approach is not tied to a single network design.

Trade-offs

The technique relies on reliable 2-D matches. Texture-less surfaces, repetitive patterns, or moving objects can corrupt the correspondence set and weaken the correction. Runtime is another hurdle: a LoRA-based (low-rank adaptation) implementation finishes in under two minutes per scene on an RTX PRO 6000 GPU—far from the frame-rate needs of live SLAM or AR.

Looking ahead

If the community adopts geometric adaptation as a standard post-processing step, many existing models could gain immediate performance without costly retraining. Benchmark suites would also need to include a Self-Geometry baseline to reflect realistic deployment where test-time refinement is feasible.

Takeaway: A modest, test-time geometric sanity check can extract significantly more accuracy from off-the-shelf vision models, but its dependence on clean matches and non-real-time speeds limits immediate adoption in latency-critical systems.