{"id":"7d7482f1-14f0-4293-ace9-e3be2217f20e","arxiv_id":"2507.10894","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A modular pipeline generates navigation instructions from egocentric video by extracting actions, scenes, and objects and synthesizing them with LLMs, plus a metric suite that evaluates the result without human reference text.","lead":"NavComposer automatically writes navigation instructions for robot and agent trajectories by breaking a trip into actions, scenes, and objects, then using a large language model to stitch them into natural language. It also introduces an evaluation system that scores those instructions without needing human-written reference sentences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NavComposer's headline superiority is unsupported: the semantic-consistency metrics in Table VI score all methods against NavComposer's own extracted entities, so the comparison is circular (NavComposer writes instructions from the same entity lists used to judge consistency).","rationale":"The paper sets out to show that a modular, zero-training pipeline produces better navigation instructions than end-to-end speakers and video LLMs. The empirical core is Table VI, and the largest margins are in semantic consistency. The evaluation's reference entities are NavComposer's own outputs (Section IV-A). Since NavComposer's instructions are generated from exactly these entity lists, the semantic consistency dimension rewards NavComposer for prompt adherence rather than for correct description. Baselines are judged against the same variable reference, so valid alternative descriptions are penalized. This is a circularity in the measurement, not a stylistic disagreement. The ablation rows #8–#10 also show score redistribution (e.g., w/o scene raises ASC and OSC), which indicates the scores are sensitive to the reference composition rather than grounded in an external truth. The module selection and final comparison are also on the same validation split, and no error bars are reported, compounding the risk. An independent re-scoring against fixed gold entities would settle whether the claimed superiority is real. The engineering is plausible, but the central claim as evidenced is unsupported; major revision is needed.","tokens_in":19616,"tokens_out":4573,"duration_ms":55763,"concrete_test":"Recompute Table VI ASC/SSC/OSC/MSC on 100–200 VLN-CE validation trajectories using a fixed reference entity set: human-annotated or gold action/scene/object labels (e.g., derived from existing R2R/RxR annotations) instead of NavComposer's extractions. Score every method, including #1 baseline and #6 Qwen2.5-VL, with the same Qwen-14B prompt against this fixed reference. If NavComposer's MSC lead over #1 and #6 shrinks or reverses, the circular evaluation is the cause; if the lead persists, the concern is empirically refuted.","verdict_should_be":"REJECT","load_bearing_attack":"Section IV-A states that for methods without explicit entity extraction, semantic consistency uses \"unified results from our best-performing variant\"—i.e., NavComposer's action/scene/object extractions—as the reference. Section III-A then constructs NavComposer's instruction I by feeding exactly those A, S, O entity lists to an LLM (Algorithm 1, lines 27–31). Consequently, ASC/SSC/OSC/MSC in Table VI largely measure whether NavComposer's synthesis prompt was followed, not whether the entities are correct or match the trajectory; the method is judged on the same entities it generates from. A baseline that describes a trajectory validly but with different landmarks, scene granularity, or action phrasing is penalized against NavComposer's extraction, so the large MSC gap (5.112 vs 2.768–3.049 for methods #1–#6) is an artifact of the evaluation protocol. The paper's central sentence in Section IV-C1, \"NavComposer outperforms all other methods across nearly all metrics, demonstrating its superiority,\" rests on this circular comparison. The contrastive matching scores are less obviously circular, but they also use a matcher trained on VLN-CE and do not correct for the semantic-consistency bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NavComposer, a modular pipeline that generates natural-language navigation instructions from egocentric video by first extracting action, scene, and object entities and then prompting an LLM to compose them into an instruction. It also introduces NavInstrCritic, an annotation-free evaluation system covering contrastive matching, semantic consistency (ASC/SSC/OSC/MSC), and linguistic diversity. Experiments on VLN-CE and ten additional datasets compare NavComposer with an end-to-end CoCa baseline, classical speaker models, and video LLMs, reporting superiority on most metrics and demonstrating adaptation to diverse data sources.","tokens_in":19874,"tokens_out":4445,"duration_ms":54127,"significance":"If the claims were supported, the paper would make a practical contribution: modular zero-shot instruction generation for continuous navigation, an annotation-free evaluation suite, and a broad adaptability study across 10 datasets. The modular design is flexible, the module comparison is systematic, and the diversity metrics are standard. However, the headline comparison is currently undermined by the semantic-consistency protocol, which evaluates all methods against entities produced by NavComposer itself. The core superiority result therefore needs to be re-established with an unbiased reference before the contribution can be assessed.","major_comments":[{"comment":"The semantic-consistency evaluation is circular. Section IV-A states that for methods without explicit entity extraction, the evaluator uses 'unified results from our best-performing variant,' i.e., NavComposer's A, S, O lists. Algorithm 1 (lines 27-31) constructs the instruction I from exactly those same lists via the LLM. Consequently ASC/SSC/OSC/MSC in Table VI measure whether a baseline happens to reproduce NavComposer's entity choices, not whether the instruction is faithful to the trajectory. A baseline using different but valid landmarks or scene granularity is penalized. The MSC gap in Table VI (5.112 vs 2.768-3.049) is therefore an artifact of the protocol, and the claim in Section IV-C1 that NavComposer 'outperforms all other methods across nearly all metrics' is unsupported by this metric.","section":"IV-A and Algorithm 1"},{"comment":"The module selection procedure inflates the comparison. The best-performing variant is chosen on the same NavInstrCritic metrics and the same validation set that is later used for the headline comparison in Table VI. Since the semantic-consistency metrics are biased toward NavComposer's own entities, selecting the variant that maximizes these metrics guarantees an advantage over methods that do not generate from those entity lists. A separate validation split or an unbiased semantic metric is needed before the comparison can be interpreted.","section":"IV-B"},{"comment":"The semantic-consistency scores are not shown to correlate with any external notion of instruction quality. The paper's ablation in rows #8-#10 shows that removing one entity type can raise the scores for the other two (e.g., removing scenes raises ASC and OSC), which is consistent with score redistribution rather than absolute quality. Without validation against human judgments, downstream navigation success, or an independent entity annotation set, the MSC scores cannot carry the weight of the paper's central claim.","section":"III-B2 and Table VI"}],"minor_comments":[{"comment":"There is a typo: 'the genralization of instructions' should be 'the generalization of instructions.'","section":"II-B"},{"comment":"The generated examples contain typographical artifacts such as 'Ve e rat', 'V eerleft', and 'Cease at desk', which conflict with the claimed instruction quality; please clean up or regenerate these examples.","section":"Figure 6"},{"comment":"The phrase 'unified results from our best-performing variant' needs a precise definition: which variant was used, how its entities were extracted for each baseline, and whether those entities were available to the baselines.","section":"IV-A"},{"comment":"The sampling logic for 'move forward' actions is not self-explanatory: line 8 gates on Z ≤ 1/2 and then splits into 1/6, 1/3, and 1/3; please clarify the intended probabilities in the text.","section":"Algorithm 1"},{"comment":"The paper honestly notes that the contrastive matcher is trained on VLN-CE and may suffer domain mismatch, but this also means the MAP scores in Table VII should be interpreted as measurements of matcher domain transfer rather than of instruction quality alone.","section":"IV-D"}],"recommendation":"major_revision","confidential_remarks":"The circularity issue is real and central. The NavInstrCritic framing is attractive, but the semantic-consistency dimension needs an independent reference, such as human entity annotations or a fixed external entity extractor applied uniformly to all methods. If this cannot be provided, the paper's main superiority claim should be withdrawn in favor of a system description with ablations only."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. The paper's central claim—that NavComposer beats all baselines on instruction quality—is not supported as written, because the semantic-consistency evaluator scores every method against NavComposer's own extracted entities. Section IV-A says methods without explicit entity extraction are judged using 'unified results from our best-performing variant,' i.e., the entity lists NavComposer itself generates from. So the MSC gap in Table VI (5.112 vs 2.768–3.049 for the other methods) largely measures whether the LLM followed the prompt, not whether the instructions are correct. The reader's circularity charge lands.\n\nThat said, this is not a paper without merit. The action-scene-object modularization is a sensible way to inject control into instruction generation, and the data-agnostic pipeline plus the adaptability study across 10 datasets is genuinely useful. NavInstrCritic's three-axis design (contrastive matching, semantic consistency, linguistic diversity) is a reasonable attempt at annotation-free evaluation, and the contrastive matching part is not circular. There, NavComposer does lead, but the margin over Qwen2.5-VL is thin: 0.802 vs 0.799 HR, 0.695 vs 0.685 MRR, 0.685 vs 0.675 MAP. So even the non-circular evidence is not overwhelming.\n\nOther soft spots: module selection and the final comparison use the same validation split, and no error bars are reported. The threshold probabilities for key-frame downsampling and the temporal correction rules are hand-set; the paper doesn't show sensitivity to them. The contrastive matcher is trained on VLN-CE and applied cross-domain, which the paper honestly flags as a limitation, but it still undercuts the cross-dataset numbers.\n\nI'd want a major revision before trusting the superiority claim: held-out entity annotations or human evaluation for semantic consistency, or at minimum scoring baselines against their own extracted entities, plus a separate validation split for model selection. The framework and evaluation system deserve serious refereeing; the current evidence for the headline claim does not.\n\nNet: worth a round of review, but with a clear request to fix the evaluation protocol. I wouldn't cite the current numbers in my own work.","headline":"The paper's headline superiority claim is undermined by a circular semantic-consistency evaluation, but the modular generation framework and the annotation-free evaluation system are real contributions worth a careful revision.","tokens_in":20384,"tokens_out":1984,"would_cite":false,"duration_ms":21995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NavComposer claims that decomposing a navigation trajectory into actions, scenes, and objects before writing the instruction produces higher-quality language than end-to-end generators, with no training on the target dataset.","keywords":["language-guided navigation","vision-and-language navigation","instruction generation","modular framework","semantic entity decomposition","annotation-free evaluation","egocentric video captioning","VLN-CE"],"falsifier":"Run a human rating study in which annotators judge NavComposer and baseline instructions for correctness, completeness, and naturalness without being shown NavComposer's entity lists; if humans do not rank NavComposer first, the reported superiority is an artifact of its own evaluator. A complementary check is to train a navigation agent on instructions from each generator and compare success rate and SPL on unseen environments.","tokens_in":19433,"feed_emoji":"🧭","tokens_out":6970,"duration_ms":76211,"temperature":0.7,"pith_summary":"NavComposer claims that the best way to write navigation instructions for a trajectory is to split the problem: first extract three kinds of semantic entities from the egocentric video—actions, scenes, and objects—then let a large language model stitch those entities into fluent instructions. The paper argues this modular decomposition produces instructions that are more accurate, richer, and more varied than those from end-to-end trained generators or from video-language models that see the whole video at once, and that it does so without training on the target dataset. To make the claim testable without expert annotations, the paper also introduces NavInstrCritic, a three-part evaluation covering contrastive matching, semantic consistency, and linguistic diversity. If the claim holds, data-hungry vision-and-language navigation could be trained at scale from automatically generated instructions, including from real robot and internet videos.","feed_headline":"A modular recipe beats trained models at writing navigation instructions","feed_subtitle":"Training-free pipeline that splits video into actions, scenes, and objects tops 872-trajectory benchmark.","key_machinery":"The central object is the semantic-entity tuple of actions, scenes, and objects extracted per trajectory segment, together with the synthesis protocol that converts that tuple into language. Action classification uses frame-pair motion analysis with temporal correction rules; scene recognition and object detection can be unimodal or multimodal-LLM based; and the synthesis module downsamples repeated actions to key frames, applies a double random synonym replacement for diversity, and prompts an LLM to phrase the temporal entity sequence. The evaluation is carried by NavInstrCritic, whose contrastive matcher maps trajectory and instruction into one space, whose LLM grader scores action, scene, and object consistency, and whose diversity metrics measure vocabulary and compression properties.","core_discovery":"On the paper's own terms, the central discovery is that explicit entity-based decomposition is the load-bearing ingredient for instruction quality. On 872 continuous-navigation trajectories, the best NavComposer variant reaches a MAP of 0.685, a mean semantic consistency of 5.112, and a compression-ratio diversity score of 4.478, beating the end-to-end baseline, two classical speaker-model baselines, and three video-language models on nearly every metric without ever being trained on the target dataset. Ablations removing actions, scenes, or objects each lower the mean semantic consistency, and the object entity is the largest contributor to contrastive matching and diversity. The framework is also applied to ten trajectory sources spanning simulators, handheld cameras, driving footage, and a wheeled robot, generating 232,569 instructions in total.","pith_inferences":["Beyond the paper: the reported semantic-consistency scores partly measure agreement with NavComposer's own entity extraction, because baselines without explicit extraction are scored against the unified entity lists from NavComposer's best variant; a human reference or a fixed third-party entity extractor would make the comparison less self-referential.","Beyond the paper: the contrastive matcher is trained on the continuous-navigation benchmark, so cross-source MAP scores reflect domain mismatch as much as instruction quality; a matcher trained on broader video-text data would be needed to separate the two.","Beyond the paper: a direct downstream test—training a navigation policy on NavComposer-generated instructions and measuring success rate and SPL on unseen environments—would show whether the instruction quality improvements translate into agent performance, which the paper does not run.","Beyond the paper: because the entity lists are explicit, the same pipeline could be pointed backwards to debug why an instruction fails: the action, scene, or object that caused the error is identifiable by construction."],"forward_implications":["Large-scale instruction corpora can be generated for any trajectory video, including domains where no expert annotations exist.","The modular design means future improvements in action classification, scene recognition, object detection, or language synthesis can be swapped in without retraining the whole pipeline.","Synonym replacement and prompt control allow instruction style, length, and vocabulary to be shaped to match a downstream agent or user preference.","Generated instructions are roughly twice as long as classical speaker outputs (58 vs 27 words on average), carrying more landmarks and scene context for an agent to ground on.","The same generator can cover simulated, handheld-camera, driving, and robot-camera trajectories after only light preprocessing."],"supporting_citations":[{"why":"Supplies the continuous-navigation dataset, action space, and training trajectories used in all main experiments.","marker":"[42]"},{"why":"Classical speaker model that serves as a baseline instruction generator in the quality comparison.","marker":"[8]"},{"why":"Classical speaker model with environmental dropout; second baseline instruction generator in the comparison.","marker":"[9]"},{"why":"The vision-to-text model adapted as the end-to-end baseline generator trained on the target dataset.","marker":"[67]"},{"why":"The pretrained vision-language encoder used to train the contrastive matcher for instruction–trajectory alignment.","marker":"[51]"},{"why":"Multimodal video model used both as a video-language baseline and as the best offline module backend.","marker":"[60]"},{"why":"Supplies the keypoint extraction that underpins the visual-odometry action classification module.","marker":"[44]"},{"why":"Transformer-based detector used as a unimodal object module baseline, showing the advantage of multimodal modules.","marker":"[48]"}],"fun_headline_variants":["Modular decomposition upstages trained nav instruction models","No training needed: entity split writes better nav instructions","Action-scene-object modularity crafts stronger navigation language","Training-free NavComposer beats baselines on instruction quality","Entity-based recipe tops classic and video-language baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The semantic-consistency evaluation assumes that the actions, scenes, and objects that NavComposer itself extracts are the correct and complete checklist for judging any instruction for a trajectory, so a baseline that describes the same path through different landmarks or wording can be unfairly marked as wrong.","fun_headline_variants_meta":{"raw":{"variants":["Modular decomposition upstages trained nav instruction models","No training needed: entity split writes better nav instructions","Action-scene-object modularity crafts stronger navigation language","Training-free NavComposer beats baselines on instruction quality","Entity-based recipe tops classic and video-language baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1223,"prompt_tokens":932,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":548,"tokens_out":291,"duration_ms":4261,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:22:09.109356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human rating study in which annotators judge NavComposer and baseline instructions for correctness, completeness, and naturalness without being shown NavComposer's entity lists; if humans do not rank NavComposer first, the reported superiority is an artifact of its own evaluator. A complementary check is to train a navigation agent on instructions from each generator and compare success rate and SPL on unseen environments.","supporting_citations":[{"cited_title":"Beyond the nav-graph: Vision-and-language navigation in continuous environments,","cited_arxiv_id":null,"evidence_quote":"Supplies the continuous-navigation dataset, action space, and training trajectories used in all main experiments."},{"cited_title":"Speaker-follower models for vision-and-language nav- igation,","cited_arxiv_id":null,"evidence_quote":"Classical speaker model that serves as a baseline instruction generator in the quality comparison."},{"cited_title":"Learning to navigate unseen environments: Back trans- lation with environmental dropout,","cited_arxiv_id":null,"evidence_quote":"Classical speaker model with environmental dropout; second baseline instruction generator in the comparison."},{"cited_title":"Coca: Contrastive captioners are image-text foundation models,","cited_arxiv_id":null,"evidence_quote":"The vision-to-text model adapted as the end-to-end baseline generator trained on the target dataset."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"The pretrained vision-language encoder used to train the contrastive matcher for instruction–trajectory alignment."},{"cited_title":"Object recognition from local scale-invariant features,","cited_arxiv_id":null,"evidence_quote":"Supplies the keypoint extraction that underpins the visual-odometry action classification module."},{"cited_title":"End-to-end object detection with transformers,","cited_arxiv_id":null,"evidence_quote":"Transformer-based detector used as a unimodal object module baseline, showing the advantage of multimodal modules."}],"review_version":1}