Augmenting head-mounted displays requires combining fast inertial data with rich but heavier camera or depth measurements. Sensor fusion techniques reduce jitter and drift while keeping latency low and pose accuracy high. Evidence from research groups shows two families of approaches dominate practical systems: filter-based real-time estimators and optimization-based smoothing.
Model-based filters
Filter approaches such as the Extended Kalman Filter and Complementary Filter run very efficiently on-device by propagating an inertial measurement unit state and correcting it with camera or depth updates. Andrew J. Davison Imperial College London demonstrated how visual measurements can stabilise pose estimates in real time with MonoSLAM, establishing the value of tight visual–inertial coupling for head-mounted use. Filter solutions excel at low-latency feedback needed for comfortable AR but can accumulate bias if visual observability is intermittent, for example in textureless indoor spaces or in culturally significant sites where preserving artifacts limits sensor placement.
Optimization and graph methods
Optimization-based methods build a spatiotemporal graph of poses and measurements and solve a nonlinear least-squares problem periodically to reduce drift. Frank Dellaert Georgia Institute of Technology and collaborators developed smoothing frameworks that underpin modern visual–inertial odometry and simultaneous localization and mapping systems. Christian Forster University of Zurich and Davide Scaramuzza ETH Zurich explored tightly-coupled visual–inertial algorithms that trade more computation for improved global accuracy. These methods reduce long-term drift and handle loop closures, which matters for territorial mapping or environmental monitoring where consistent reconstructions across sessions are required. The cost is higher compute and potentially greater latency unless parts of the pipeline run on separate hardware.
Practical AR systems often blend methods: a low-latency filter provides immediate pose for rendering, while an optimizer runs in the background to refine the trajectory and correct accumulated errors. Tightly-coupled visual-inertial odometry outperforms loosely-coupled designs when camera and IMU timing are well calibrated, but magnetic disturbances, reflective surfaces, and occlusions still degrade performance and must be handled by robust outlier rejection or fallback sensors such as depth or ultrasonic range.
Choosing the right fusion strategy affects user comfort, power draw, and the cultural or environmental usability of AR: systems intended for dense urban canyons rely more on vision and map reuse, whereas devices used in remote or heritage environments benefit from conservative, drift-minimising smoothing. The best results come from combining well-understood filters for responsiveness with graph-based optimizers for long-term fidelity.