BudVLN: Retrospective Rectification for Robust Vision-Language Navigation

Figure 1: Illustration of instruction-state misalignment. Standard DAgger forces a recovery from the error state, causing backward-correcting actions that fail to establish a semantic connection with the instruction. In contrast, BudVLN employs retrospective rectification to synthesize a forward-looking demonstration that remains strictly aligned with the natural language instruction.

Abstract

Vision-Language Navigation (VLN) requires embodied agents to interpret natural language instructions and navigate through complex continuous 3D environments. However, the dominant imitation learning paradigm suffers from exposure bias, where minor deviations during inference lead to compounding errors.

While DAgger-style approaches attempt to mitigate this by correcting error states, we identify a critical limitation: Instruction-State Misalignment. Forcing an agent to learn recovery actions from off-track states often creates supervision signals that semantically conflict with the original instruction.

In response to these challenges, we introduce BudVLN, an online framework that learns from on-policy rollouts by constructing supervision to match the current state distribution. BudVLN performs retrospective rectification via counterfactual re-anchoring and decision-conditioned supervision synthesis, using a geodesic oracle to synthesize corrective trajectories that originate from valid historical states, ensuring semantic consistency. Experiments on the standard R2R-CE and RxR-CE benchmarks demonstrate that BudVLN consistently mitigates distribution shift and achieves state-of-the-art performance in both Success Rate and SPL.

Methodology: The BudVLN Framework

To address the challenges of exploration efficiency and adversarial supervision, BudVLN employs an Adaptive Mutual Exclusion Strategy to dynamically bifurcate the training process based on the agent's real-time proficiency.

Figure 2: Overview of the BudVLN training framework. For a given instruction, a greedy probe first evaluates the agent's proficiency. When proficient, the framework routes to the optimality seeking pathway via GRPO. Conversely, upon failure, it triggers the rectification pathway, synthesizing alignment-preserving supervision via valid historical states.

Optimality-Seeking Exploration via GRPO: When the probe confirms proficiency, we optimize for shorter, more efficient paths using Group Relative Policy Optimization, bypassing Supervised Fine-Tuning.
Retrospective Rectification (SFT): For hard samples, we perform a History-Aware Rollback. We identify the best waypoint achieved, retain the valid visual history prefix, and use an oracle planner to generate the optimal action sequence to the goal. This teaches the agent how to avert errors from the outset without Instruction-State Misalignment.

Qualitative Results

Compared to baseline methods that fail to ground instructions and get stuck during deviations, BudVLN remains robust at critical turning points, effectively avoiding failure modes and preserving history-action consistency.

Qualitative comparison between Baseline and BudVLN

Figure 3: Qualitative comparison between the Baseline and BudVLN. The Baseline agent fails to ground the instruction, leading to a deviation. BudVLN successfully navigates the complex environment, demonstrating superior robustness.

Navigation Demos

Here we present several successful navigation trajectories performed by our BudVLN agent in unseen environments. Guided by our retrospective rectification mechanism, the agent robustly follows complex natural language instructions and reaches the target.

Instruction 1: "Exit the room and turn right to find the fire extinguisher."

Instruction 2: "Walk to the end of the hallway, turn right, and stop in front of the fire extinguisher."

Instruction 3: "Walk straight, then turn right to find the orange box."

BibTeX

@misc{he2026nipping,
    title={Nipping the Drift in the Bud: Retrospective Rectification for Robust Vision-Language Navigation},
    author={Gang He and Zhenyang Liu and Kepeng Xu and Li Xu and Tong Qiao and Wenxin Yu and Chang Wu and Weiying Xie},
    year={2026},
    eprint={2602.06356},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}