{"id":"edd1167b-918f-40ab-8f22-d8fd4949c1b3","arxiv_id":"2505.20897","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A dual-branch text-imagination system, with one LLM branch for state estimation and one for candidate-direction description, improves R2R navigation success over prior LLM-based VLN methods.","lead":"The paper introduces Adaptive Text Dreamer (ATD), a navigation system that uses two language model branches to estimate where the agent is and to imagine, in words, what lies in candidate directions, then feeds that imagined text into a graph-based navigation policy. It reports state-of-the-art results on the R2R benchmark with fewer parameters than several prior LLM-based agents.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The right brain's training target is internally inconsistent: Fig. 6 suggests a single stitched-panorama caption, while Eq. 5 sums cross-entropy over N candidate captions; the actual objective is ambiguous and may not train genuine future imagination.","rationale":"The reader's weakest assumption — that the right brain's ground-truth captions are of candidate views already visible from the current node — is a reasonable reading of Fig. 6, and if true it would collapse the 'future imagination' story into current-view summarization. However, the manuscript contains evidence against that reading: Sec. 3.1 says captions are for candidate nodes one step ahead, and the Limitation in Sec. 9 explicitly states that data is limited to 'candidate nodes one step ahead of the current node.' So the more precise, and more defensible, concern is that the paper never specifies a coherent training objective for the right brain. Eq. 5 defines a sum over N candidate captions, while the collection description in Fig. 6 describes a single caption of a stitched panorama. These cannot both be correct. This mismatch is load-bearing because the central claim attributes the performance gain to this 'adaptive text dreamer' module; if the loss is undefined as written, or if the implementation differs from the text, the mechanism is unverified. A code inspection would settle it. This does not move the verdict: CONDITIONAL remains appropriate, since the issue is concrete and checkable, and the reported ablations (Table 4) suggest the branch does contribute empirically. Secondary issues — no error bars, unreported lambda, and the parameter-count ambiguity in Sec. 7.2 relative to the abstract — also warrant attention but are less load-bearing than the undefined training target.","tokens_in":22808,"tokens_out":13299,"duration_ms":133279,"concrete_test":"Inspect the released code and data-collection scripts (or, if unavailable, contact the authors) to determine the exact ground-truth format for the right brain: (a) one caption per candidate node, as Eq. 5 implies, or (b) a single caption for the panorama stitched from all candidate images, as Fig. 6 states. Then verify the loss implementation: if (b), L_rightbrain should be a single cross-entropy term; if (a), the collection pipeline must be re-described and Fig. 6 corrected. Also check whether the candidate images are taken from the candidate nodes' own panoramic positions (future) or from the current node's directional views (visible). This single check settles whether the method actually trains future imagination and whether Eq. 5 is implementable as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the right-brain Imagination LLM learns to predict 'future key environmental semantics,' and that this drives the state-of-the-art gains. For that claim to hold, the training objective must be well-defined and must supervise the model with information beyond the current observation. The paper does not provide a consistent description. Sec. 3.1 says captions {C^i_candidate_t}_{i=1}^N are collected for N candidate nodes, and Eq. 5 defines L_rightbrain = -Σ_t Σ_i {C^i_candidate_t} log(I_t), implying N distinct ground-truth captions per step. But the supplementary collection procedure (Fig. 6) says images from all candidate nodes are stitched into a single panoramic view, and Qwen2.5-VL generates 'a caption' for that panoramic image. If there is one stitched caption, the loss over i is undefined (or N reduces to 1), and the model is trained on a description of a panorama composed of candidate views, which may already be visible from the current node — reducing 'imagination' to selective summarization. If instead there are N separate captions taken at candidate nodes, then the method is genuinely predicting future observations, but the collection description and Fig. 6 are wrong, and the loss/prompt mismatch remains: the prompt asks for per-candidate imagination plus a choice, while the target is described as a caption of the stitched panorama. Without the released code or a clarified data-collection specification, the reader cannot determine which objective was actually optimized, so the mechanism responsible for the reported gains is unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ATD, a dual-branch language-imagination system for vision-and-language navigation. A left 'state estimation' branch uses a frozen InstructBLIP/FlanT5-XL with a tunable Q-Former to describe the current navigation state, while a right 'imagination' branch is trained on captions of candidate-node views to predict future scene semantics in text form. The two branches interact through a State-Grounded Cross-Attention (SGCA) layer, and the resulting embedding is injected into a DUET-style graph transformer policy. Experiments on R2R report state-of-the-art results among LLM-based methods with 1.5B parameters, together with ablations of the two branches, the SGCA layer count, and zero-shot cross-dataset evaluations on REVERIE and R4R.","tokens_in":23011,"tokens_out":13619,"duration_ms":137569,"significance":"If the claims hold, the paper is a meaningful step: it shows that language-form 'imagination' can replace expensive pixel-level future-scene generation in VLN, and that a parameter-efficient dual-branch design can outperform larger LLM-based navigators. The paper is clearly organized, the architecture is described in enough detail to be re-implemented in outline, and the ablations are useful: they show that both branches contribute and that the SGCA direction (state grounding imagination) beats reversed and parallel variants. The honest limitation statement in Sec. 9 that the imagination data covers only one step ahead is a point in favor of transparency. However, the central 'future imagination' claim rests on a data-collection description that is internally inconsistent, and the quantitative support lacks variance estimates and uses a weak ablation baseline; these issues need to be resolved before the headline claims can be fully accepted.","major_comments":[{"comment":"The training objective for the Imagination LLM is defined at the level of N per-candidate captions, but the collection procedure describes a single stitched-panorama caption. Eq. (5) writes L_rightbrain = -Σ_t Σ_i {C^i_candidate_t} log(I_t), which requires N separate caption sequences for the N candidate nodes at each step. Supplementary Sec. 6 instead says that images from all candidate nodes are stitched into one panoramic view and that Qwen2.5-VL is used to generate 'a caption' for that panoramic image, and Fig. 6 shows a single caption. If the actual target is one stitched caption, the sum over i is undefined (or N collapses to 1); if there are N per-candidate captions, the supplementary description and Fig. 6 are wrong. This ambiguity is load-bearing because the paper's core claim is that the right brain learns to predict future key semantics. The authors should specify the exact protocol, including whether candidate-node images are obtained by visiting those nodes and whether those images are visible from the current node, and then align Eq. (5), the prompt in Fig. 7, and the collection description with that protocol.","section":"Sec. 3.1, Eq. (5); Supplementary Sec. 6, Fig. 6"},{"comment":"The attribution of the reported gains to the imagination branch is not established because the ablation baseline is not a strong matched model. The text states that removing both SELLM and IMLLM yields 'DUET without the local branch and without BERT pretraining', while the abstract reports gains of 8%/12% SR and 5%/11% SPL. Such a weakened baseline confounds the effect of the new branches with the removal of standard DUET components. The most informative comparison in the paper is against NavGPT2 (Table 1), where the gains are 2-4 points in SR and 3-4 points in SPL on the validation/test splits. Please add an ablation that removes only the right-brain (Imagination) branch from the full ATD, or starts from a matched single-branch NavGPT2-style policy, so that the contribution of the text dreamer can be isolated.","section":"Sec. 4.5, Table 4; Abstract"},{"comment":"All reported numbers are single runs without error bars or multiple seeds. Several conclusions rest on very small differences: for example, in Table 5 the val-unseen SR values for 3 and 4 SGCA layers are 74.93 and 74.63, and the text nonetheless concludes that 3 or 4 layers are best; Table 3 shows SR differences of 1-2 points between FlanT5-XL and FlanT5-XXL. Given the 2-4 point gaps that separate ATD from NavGPT2 on some splits, the claim of state-of-the-art performance needs to be accompanied by variance estimates. Please report means and standard deviations over at least three seeds for the main results and for the key ablations.","section":"Tables 1, 3, 4, 5, 7; Sec. 4.5"},{"comment":"The parameter-efficiency claim should be made precise. The abstract states that ATD uses 1.5B parameters, but Supplementary Sec. 7.2 says 'the parameter size of our model is 1.5B because only the encoder was used during policy training, resulting in half of the parameters of the Flant5-XL model.' InstructBLIP with FlanT5-XL also includes a ViT, a Q-Former, and the decoder side of the LM, so the total number of weights in the full ATD system at inference is not simply 1.5B. Please specify the exact total and trainable parameter counts for the full ATD system (both VLM branches plus the graph policy) and compare those numbers consistently with NavGPT2 and NaviLLM.","section":"Sec. 4.2; Supplementary Sec. 7.2"}],"minor_comments":[{"comment":"The text says 'with V_vis_t as the query and V_ATD_t as both the key and value', but Eq. (9) writes V_fusion_t = MCA(V_ATD_t, V_vis_t), which reverses the order under the standard MCA(query, key, value) convention. Please correct either the text or the equation.","section":"Sec. 3.2, Eq. (9)"},{"comment":"The zero-shot REVERIE numbers are inconsistent across tables: Table 2 lists OSR/SR/SPL of 33/27/23 for ATD(F lanT5-XXL), while Table 6 lists 32.52/27.04/22.51 for the same model. Please align the values and state the split and whether PREVALENT data was used.","section":"Table 2 and Table 6"},{"comment":"The abstract's 'gains in SR of 8.0% and 12.0%' and 'SPL of 5.0% and 11.0%' are taken from the weak ablation baseline in Table 4, not from the comparison against the previous state of the art. Please clarify in the abstract that these gains are relative to a DUET-based baseline without the proposed branches.","section":"Abstract; Sec. 4.5"},{"comment":"The notation {C^i_candidate_t} in Eq. (5) is informal and could be read as a set; please write the cross-entropy over token sequences explicitly and specify the masking over different caption lengths.","section":"Sec. 3.1, Eq. (5)"},{"comment":"The Imagination System Prompt asks the model to 'choose the one that best matches the instructions', but Eq. (5) and the caption-collection description do not explain how this choice is supervised or how it is used in the SGCA layer. Please clarify whether the choice token is part of the imagination output I_t.","section":"Sec. 3.1 and Fig. 7"},{"comment":"Please state explicitly which components are frozen and which are trainable during the second-stage policy training: are the Q-Formers and LLMs from the two branches frozen when the graph policy is trained?","section":"Sec. 3.2 and Sec. 4.2"},{"comment":"The attention visualization in Fig. 3 is qualitative, and several conclusions in Sec. 4.5 are based on differences of under one point (e.g., Table 5). Please either provide a quantitative measure of attention quality or soften the corresponding claims.","section":"Sec. 4.4 and Sec. 4.5"},{"comment":"The claim 'state-of-the-art performance with fewer parameters' should be qualified: in Table 1, BEVBert achieves comparable or higher val-unseen SR/SPL (75/64 vs 75/63) while using additional depth information, so the 'state-of-the-art' claim is best restricted to LLM-based methods or to methods without extra depth input.","section":"Sec. 4.3"},{"comment":"The conclusion states that ATD achieves 'lower computational costs' than visual-imagination methods, but no runtime, GPU memory, or inference-time measurements are reported. Please add a quantitative efficiency comparison.","section":"Sec. 2 and Sec. 5"},{"comment":"There is a typo in the Table 1 caption ('ooutperforms'), and references [89] and [90] appear to be the same paper; please correct both.","section":"Table 1 and References"},{"comment":"The statement that PREVALENT data is 'vital' to prevent overfitting is important, but the paper does not report results without PREVALENT for the final ATD model. Please include these numbers so the reader can see the sensitivity to synthetic data.","section":"Supplementary Sec. 7.2"},{"comment":"Please report the number of training episodes/samples used for the GPT-4V state-estimation labels and the Qwen2.5-VL imagination captions, as well as any filtering criteria, since these are the core training data for the two branches.","section":"Sec. 4.2"},{"comment":"The hyperparameter λ in the loss L = λL_BC + L_PID and the number of SGCA layers are free parameters; please state whether these were selected on the validation splits and include an ablation over λ.","section":"Sec. 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is not critically flawed: I see no evidence of circular evaluation or benchmark leakage beyond the ambiguity in the right-brain data collection. The main risks are that the reported gains may not be statistically robust and that the 'imagination' framing may be stronger than what the data actually supports. Asking for code or the data-collection scripts as part of the revision would allow the key ambiguity in Eq. (5) vs. Fig. 6 to be resolved definitively."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible engineering paper that does something modest and useful—injecting textual 'imagination' into a graph-based VLN policy—but the central narrative overstates what the right brain actually does. The gains on R2R are small but consistent, and the ablations support the design choices. The main soft spot is that the imagination branch is trained on captions of candidate nodes one step ahead, which in the discrete R2R setting are largely visible from the current viewpoint. That is selective summarization of current perception more than genuine future imagination. The paper's own limitation section (Sec. 9) admits this. So the partial-observability motivation is not really tested.\n\nWhat's new: The dual-branch state-estimation + imagination LLM with SGCA (state-grounded cross-attention) and latent injection into a DUET-style graph policy is a new combination. Each ingredient exists (NavGPT2's Q-former tuning, DUET's graph, NavCoT's text imagination), but the architecture and the SGCA filtering are a reasonable contribution. The ablations are informative: both branches help, and swapping the SGCA orientation or using parallel fusion hurts. That is real evidence the interaction design matters.\n\nWeaknesses, in order of severity:\n\n1. The loss ambiguity. Eq. 5 sums cross-entropy over N candidate captions, but the supplementary collection (Fig. 6) describes stitching candidate views into one panorama and generating a single caption. These cannot both be right. This needs clarifying before the mechanism is trustworthy. It is not fatal—likely the text in one place is wrong—but it is exactly the kind of detail that should be caught in review.\n\n2. No error bars, no multiple seeds, and the loss trade-off lambda is unreported. The gains over NavGPT2 are 2–3 points on some splits, so without variance estimates the headline 'state-of-the-art' is fragile.\n\n3. The baseline for the main ablation is DUET stripped of its local branch and BERT pretraining, not the full DUET. So the 8-point gain is over a weaker baseline, not a direct comparison.\n\n4. Best numbers depend on PREVALENT synthetic data, which the paper notes is vital. That is fine, but it should be prominent.\n\nOn the positive side, the paper is clearly written, the related work is thorough, and the authors are candid about the one-step-ahead limitation and the broader-impact issues. The code link exists; if the code ships and the objective is clarified, this becomes a useful reference for the VLN community.\n\nMy recommendation: send it to review. The concerns are addressable via clarifications and a few more runs. The core idea—language as a cheap, selective form of future imagination—is worth testing properly. I would not cite it yet for the 'future imagination' claim, but I would cite it for the SGCA design and the dual-branch latent injection once the authors clarify the data collection.","headline":"Solid engineering contribution with a coherent dual-branch design, but the 'future imagination' claim is weaker than advertised and the training objective is underspecified.","tokens_in":23686,"tokens_out":3068,"would_cite":false,"duration_ms":31965,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that imagining future scenes in language rather than rendering them as images lets a 1.5B-parameter navigation agent beat larger vision-based and LLM baselines on the R2R benchmark.","keywords":["vision-and-language navigation","text imagination","large language models","state estimation","Q-Former tuning","graph-based navigation policy","partial observability","R2R benchmark"],"falsifier":"Run ATD on R2R with candidate-node images withheld from both the current observation and the LLM prompts, keeping only the current panorama; if the success-rate gain over the no-imagination baseline disappears, the effect is selective summarization of visible candidates, not imagination of unseen future scenes. Alternatively, feed the actual candidate captions into the policy in place of the imagination branch at inference; if SR does not change, the imaginative generation itself is not the source of the gain.","tokens_in":22512,"feed_emoji":"🧠","tokens_out":5970,"duration_ms":57651,"temperature":0.7,"pith_summary":"This paper argues that a navigation agent does not need to render future scenes pixel by pixel; it can imagine the key upcoming objects and rooms in words. It introduces Adaptive Text Dreamer (ATD), a dual-branch system built on a frozen large language model, where one branch estimates how far along the instruction the agent has progressed and the other writes short descriptions of what lies in each candidate direction. The imagined text is filtered by the state estimate and injected into a graph-based navigation policy. On the R2R benchmark the authors report state-of-the-art success rates with only 1.5B parameters, outperforming the 7B NaviLLM and the 1.5B NavGPT2. If true, this suggests that selective language-based imagination can replace expensive visual future synthesis in instruction-following navigation.","feed_headline":"Text dreamer boosts navigation success 8-12% with fewer parameters","feed_subtitle":"Two LLM branches—state estimation and future imagination—beat a 7B navigator on R2R with only 1.5B parameters.","key_machinery":"The load-bearing mechanism is the paired left-right brain of frozen LLMs, each adapted by fine-tuning only the Q-Former on top of frozen InstructBLIP. The left brain is trained with cross-entropy loss to reproduce GPT-4V state-estimation reasoning, and the right brain is trained to reproduce Qwen2.5-VL captions of candidate-node panoramic stitches, giving it a text dreamer target. State Grounded Cross-Attention (SGCA) computes $A=\\mathrm{SoftMax}(\\mathrm{Sim}_{\\cos}(Q_S,K_I))$ with state-estimation embeddings as queries and imagination embeddings as keys and values, then feeds the attended imagination into the graph policy through multi-head cross-attention. The navigation policy is a DUET-style graph transformer trained with behavior cloning and pseudo-interactive-demonstrator losses.","core_discovery":"ATD is a dual-branch self-guided imagination policy: a left-brain State Estimation LLM reasons about which part of the instruction has been completed and what the agent currently sees, while a right-brain Imagination LLM generates candidate-direction descriptions of future key environmental semantics. The right brain is constrained by the left brain through State Grounded Cross-Attention (SGCA), and the resulting latent is fused into the node embeddings of a graph-based navigation expert. On the R2R benchmark the paper reports gains of 8.0% and 12.0% in Success Rate and 5.0% and 11.0% in SPL over its NavGPT2-style baseline on val seen and val unseen, respectively, with a 1.5B-parameter model beating the fully fine-tuned 7B NaviLLM. The paper frames this as evidence that abstract textual imagination is more reliable and efficient than visual synthesis for handling partial observability.","pith_inferences":["Because the imagination ground truth is captions of one-step-ahead candidate nodes, a charitable reading is that ATD learns selective summarization; a strict reading is that the partial-observability motivation is not tested. One could withhold all candidate images from input and see whether gains survive.","The same dual-branch recipe could be extended to multi-step-ahead imagination targets; the authors note the current data only covers one step ahead, so the ceiling of text dreaming is likely understated.","The SGCA design is a state-conditioned latent filter; any LLM-policy hybrid that suffers from stale or irrelevant text could reuse this cross-attention grounding.","A simple testable extension: at inference, replace the imagination branch's generated text with the true candidate captions; if performance is unchanged, the contribution is in fusion and grounding rather than in generation."],"forward_implications":["Language-form imagination can substitute for pixel-level future rendering in discrete VLN, cutting the cost of imagination while improving success.","Keeping the LLM frozen and tuning only the Q-Former is enough to activate navigation-specific reasoning, so a 1.5B model can outperform a fully fine-tuned 7B action generator.","The state-estimation branch acts as an adaptive filter: SGCA attention visualizations show completed instruction steps being suppressed as navigation proceeds.","The trained policy transfers zero-shot to REVERIE and R4R with consistent gains over the DUET baseline, indicating the imagined text helps rather than overfits to R2R.","Synthetic PREVALENT data remains necessary to avoid premature validation-loss plateaus, so the reported gains depend on that pre-training data as well as on the imagination branches."],"supporting_citations":[{"why":"NavGPT2 is the direct LLM-policy baseline and the source of the two-stage latent-space training and PREVALENT recipe that ATD builds on and compares against.","marker":"[110]"},{"why":"DUET supplies the graph-based navigation policy backbone; the ablation baseline is DUET without the local branch and without BERT pretraining.","marker":"[18]"},{"why":"InstructBLIP is the frozen vision-language backbone whose Q-Former is fine-tuned for both the state-estimation and imagination branches.","marker":"[22]"},{"why":"The Q-Former mechanism from BLIP-2 is the parameter-efficient adapter that activates domain-specific knowledge in the frozen LLM.","marker":"[50]"},{"why":"GPT-4V generates the state-estimation reasoning texts used as ground truth to distill the left-brain capability.","marker":"[1]"},{"why":"Qwen2.5-VL produces the candidate-node panoramic captions that serve as ground truth for the right-brain imagination LLM.","marker":"[5]"},{"why":"PREVALENT synthetic data is used in the main experiments and is reported as vital for preventing premature overfitting during policy training.","marker":"[31]"},{"why":"The R2R benchmark defines the task, the discrete graph environment, and the success metrics on which all comparisons are made.","marker":"[4]"}],"fun_headline_variants":["Language dreamer shrinks model, boosts navigation 8-12%","Left brain checks, right brain invents: VLN improved","Textual imagination beats vision in navigation task","1.5B text-dreaming model tops 7B navigator","Adaptive text dreamer: state-of-the-art VLN with fewer params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The right brain is trained on captions of candidate viewpoints that are one step ahead and are stitched from images reachable from the current position; the method assumes that learning from these captions teaches the agent to imagine genuinely unseen scenes rather than to summarize what is already visible.","fun_headline_variants_meta":{"raw":{"variants":["Language dreamer shrinks model, boosts navigation 8-12%","Left brain checks, right brain invents: VLN improved","Textual imagination beats vision in navigation task","1.5B text-dreaming model tops 7B navigator","Adaptive text dreamer: state-of-the-art VLN with fewer params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3435,"prompt_tokens":1010,"completion_tokens":2425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2335}},"tokens_in":626,"tokens_out":2425,"duration_ms":19310,"temperature":1.0,"reasoning_tokens":2335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:44:52.994305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ATD on R2R with candidate-node images withheld from both the current observation and the LLM prompts, keeping only the current panorama; if the success-rate gain over the no-imagination baseline disappears, the effect is selective summarization of visible candidates, not imagination of unseen future scenes. Alternatively, feed the actual candidate captions into the policy in place of the imagination branch at inference; if SR does not change, the imaginative generation itself is not the source of the gain.","supporting_citations":[{"cited_title":"Navgpt-2: Unleashing navigational reasoning capability for large vision-language models","cited_arxiv_id":null,"evidence_quote":"NavGPT2 is the direct LLM-policy baseline and the source of the two-stage latent-space training and PREVALENT recipe that ATD builds on and compares against."},{"cited_title":"Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"The Q-Former mechanism from BLIP-2 is the parameter-efficient adapter that activates domain-specific knowledge in the frozen LLM."}],"review_version":1}