ReliableNav: Uncertainty-Aware Zero-Shot Vision-Language Navigation in Continuous Environments

Submitted to IEEE Transactions on Multimedia, 2026


Zhanbo Shi1, Lin Zhang1*, Shengjie Zhao1, Yicong Zhou2

1School of Computer Science and Technology, Tongji University, Shanghai, China

2Department of Computer and Information Science, University of Macau, China

Code

Abstract

In recent years, numerous studies have attempted to solve the Vision-Language Navigation in Continuous Environments (VLN-CE) task in a zero-shot manner, leveraging Large Language Models (LLMs) to generate navigation decisions. However, all of these methods overlook the unreliability of LLMs' outputs, which significantly reduces the success rate of the VLN-CE task. This unreliability can be attributed to semantic uncertainty arising from the ambiguous inputs of LLMs and sampling uncertainty caused by the inherent randomness of LLMs. Taking the aforementioned uncertainty into consideration, we propose ReliableNav, the first uncertainty-aware zero-shot navigation framework for the VLN-CE task. To cope with semantic uncertainty, ReliableNav is equipped with the Chain-of-Atomic-Actions (CoA2), which is designed to improve the semantic equivalence of textual and visual inputs in the reasoning process of LLMs. In addition, we introduce the Reliability Check Map (RCM) to necessitate replanning for unreliable decisions, thus mitigating the impact of sampling uncertainty on navigation performance. Experimental results demonstrate that our ReliableNav significantly surpasses its counterparts on two VLN-CE benchmarks, R2R-CE and RxR-CE. Furthermore, a detailed analysis was conducted on 100 episodes from the validation unseen split of the R2R-CE dataset to validate the effectiveness of our ReliableNav.

Method Overview

MY ALT TEXT

The architecture overview of our ReliableNav. At each decision step in the VLN-CE task, the next subgoal is generated by an LLM-based planner with the assistance of our CoA2. Subsequently, the reliability of this subgoal is assessed by our RCM. It should be noted that only reliable subgoals are utilized to update CoA2 and guide the navigation process of the agent.

Subgoal Planning Process

MY ALT TEXT

The process of the subgoal planning stage. At each decision step, an LLM-based Waypoint Predictor is first employed to identify potential waypoints from the observations. Subsequently, the next subgoal of the agent is predicted by the CoA2-based Subgoal Planner. It is important to note that the subgoals' actions are restricted to >Atomic Actions, which effectively reduces the semantic uncertainty due to the semantic ambiguity of textual and visual inputs of LLMs in the VLN-CE task.

Subgoal Reliability Assessment

MY ALT TEXT

The workflow of the subgoal reliability assessment. After obtaining the subgoal, its reliability is evaluated using our RCM in the following perspectives, navigability, visitation status, existence, and uniqueness. In situations where multiple landmarks with similar semantics are around the agent, additional visual prompts are employed, as shown in the red rectangles.

Experiments