{"id":"46db8d0f-e031-4ac8-bc9e-6b8fcf178b4e","arxiv_id":"2411.11394","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A ChatGPT-4V-based pipeline generates multi-granularity navigation instructions from YouTube house tour videos and improves VLN agent generalization on R2R and RxR benchmarks.","lead":"InstruGen automatically creates navigation instructions for house-tour videos by asking a large multimodal model to describe sampled walking paths, then checking the outputs for consistency. The paper reports that agents pre-trained on these generated instruction-path pairs outperform prior methods on the R2R and RxR navigation benchmarks, especially in unseen environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is confounded by the R2R* fine-tuning protocol: the best model is trained on R2R plus the added R2R# instructions, so its advantage over Lily and Airbert may reflect doubled supervision rather than instruction quality.","rationale":"The paper's central claim is that agents trained on InstruGen-generated path-instruction pairs achieve state-of-the-art results on R2R and RxR. For that claim to be sound, the experimental comparison must isolate the effect of generated instruction quality from other variables. The strongest comparison in Table III uses row #6 of Table II, whose fine-tuning data are R2R*: the union of R2R and R2R#. This roughly doubles the number of fine-tuning instructions relative to the baselines, so the observed SOTA margin can be explained by a trivial data-volume effect. The internal ablation supports this reading: row #4, which uses the same pretraining but only R2R fine-tuning, obtains 69.93 on Val-Unseen, essentially tied with Lily's 70.00. The improvement to 71.10 appears only when the extra R2R# instructions are added. Without a matched-data comparison, the headline claim remains plausible but unproven. The reader's formal weakest assumption was the action inverse model, but their rationale also noted the R2R* protocol; I agree with the latter as the more decisive threat. I also acknowledge the paper's positive contributions: a reproducible pipeline, a verification mechanism with an ablation, and public code. The concern is not that the method is invalid, but that the SOTA claim is not yet isolated from a training-data confound. A conditional verdict, requiring an equal-budget comparison, remains appropriate.","tokens_in":11218,"tokens_out":3999,"duration_ms":43546,"concrete_test":"Re-run the R2R comparison with a matched data budget: keep the identical coarse-pretraining checkpoint used for Table II row #4, then fine-tune on (a) R2R only, (b) R2R* as in row #6, and (c) R2R with a second copy of the original R2R instructions, so (b) and (c) have identical instruction count. If (c) matches (b) on Val-Unseen SR, the gain is data volume, not generated-instruction quality; if (b) clearly beats (c), the claim survives. Repeat with template-generated or Marky-generated extra instructions as a quality control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table III ('Ours' 71.10 SR val-unseen) is taken from Table II row #6, whose fine-tuning set is R2R* = R2R ∪ R2R#. R2R# is an InstruGen rewrite of the R2R trajectories; hence the model sees the original R2R instructions plus an equal-sized generated set during fine-tuning. Baselines (Lily, Airbert, Speaker-Follower) are fine-tuned on R2R only, so they receive roughly half as many fine-tuning instruction examples. The ablation shows the jump from row #4 (coarse pretraining + R2R only, 69.93) to row #6 (same pretraining + R2R*, 71.10) is +1.17 SR, and row #4 is statistically indistinguishable from Lily (70.00). Thus the SOTA margin is created by adding R2R# to fine-tuning, not by the pretraining corpus on YouTube videos. Since baselines do not receive an equivalent quantity of extra fine-tuning instructions, the abstract's claim that agents trained with InstruGen path-instruction pairs achieve SOTA is not established: the comparison conflates data quantity with data quality. This is a load-bearing fairness issue independent of the action-inverse-model correctness raised by the reader.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents InstruGen, a pipeline that uses ChatGPT-4V to generate vision-and-language navigation (VLN) path-instruction pairs from trajectories sampled in YouTube house-tour videos. Trajectory nodes are labeled with CLIP, actions between nodes are estimated with an inverse action model, and the generated instructions are passed through a consistency module and regular-expression rules to reduce hallucinations. The authors pre-train and fine-tune a ViLBERT-style agent with the generated data and report state-of-the-art success rates on the R2R and RxR benchmarks, supported by ablations on the verification mechanism and on coarse- versus fine-grained instructions. The paper also acknowledges in Section V a limitation: discrete trajectory sampling may restrict applicability to continuous navigation scenes.","tokens_in":11464,"tokens_out":8161,"duration_ms":81486,"significance":"If the central claim were established, the paper would be a useful contribution: it replaces template-based instruction generation with LMM-based generation, proposes a multi-stage verification mechanism, and shows that YouTube house-tour videos can serve as weakly supervised navigation data that transfers to Matterport-style benchmarks. The provision of code is a positive step for reproducibility. However, the current significance is limited by three issues that are load-bearing for the SOTA claim: the headline comparison is confounded by fine-tuning on R2R* (R2R plus an equal-sized InstruGen rewrite of R2R trajectories), the reported gains are small and lack error bars or significance tests, and the generated dataset itself is not released. The self-referential consistency check is partly mitigated by the manual-review row in Table I, but that mitigation is itself only a single run and is not quantified in detail.","major_comments":[{"comment":"The central SOTA claim is not currently supported because the 'Ours' row in Table III is identical to row #6 of Table II, whose fine-tuning set is R2R* = R2R ∪ R2R#. The baselines in Table III (Speaker-Follower, VLNbert, Airbert, Lily) are fine-tuned on R2R only. Row #4 of Table II, which keeps the InstruGen YouTube pretraining but fine-tunes on R2R only, achieves 69.93 SR on Val-Unseen, essentially the same as Lily's 70.00. The +1.17 SR difference between row #4 and row #6 therefore measures the effect of doubling the number of fine-tuning instructions, not the quality of InstruGen-generated instructions per se. A controlled comparison is needed: for example, fine-tune the same architecture on R2R plus a same-sized set of template-generated or speaker-generated rewrites, or compare the full InstruGen pipeline against baselines that also receive an equal-sized additional fine-tuning set, with multiple seeds. The same confound affects Table V, where the zero-shot RxR model is described as using row #6 from Table II.","section":"§IV-C, Table III (with Table II)"},{"comment":"All conclusions about the verification mechanism and instruction granularity rest on single-run SR values with no error bars or significance tests. The differences at stake are small: in Table I, rows #3 and #4 differ by 0.2 SR (70.9 vs 71.10), and Table II contains several 1-point-scale differences. The text uses phrases such as 'significantly better' and 'significant effect,' but no statistical evidence is provided. Please report means and standard deviations over multiple seeds, and where feasible paired tests or confidence intervals; if the training cost makes this prohibitive, state that explicitly and provide the best-of-N or a similar protocol.","section":"§IV-B, Tables I–II"},{"comment":"The action inverse model borrowed from [12] is not evaluated independently. The actions produced by this model are used both as input to ChatGPT-4V for instruction generation and as ground truth in the consistency verification module, so any systematic error in the action model will propagate directly into the generated instructions and cannot be caught by the verification step. Please include a validation of the action model on a sample of manually annotated action sequences, or at least a sensitivity analysis that replaces it with alternative action estimates, so that the reliability of the generated path-instruction pairs can be assessed.","section":"§III-A"},{"comment":"The paper lists a dataset as one of its contributions and provides a code URL, but the generated YouTube path-instruction pairs and, in particular, the R2R# and R2R* sets are not released. Because the central experiments depend on this generated data, the inability to download or inspect the data prevents independent verification of the pipeline and of the main empirical claims. Please release the dataset, or at least a representative sample, with the revised version.","section":"§IV-A and Data Availability"}],"minor_comments":[{"comment":"The negative-sample construction contains unresolved placeholders: 'see Figure ??' appears twice in the description of the two methods for constructing trajectory negative samples. These references should be fixed or removed.","section":"Appendix A"},{"comment":"The table header has typos and inconsistent spacing, including 'Finetuing' and 'R2R #' versus 'R2R#' in the text. The column alignment should be cleaned so that the pretraining, fine-tuning, and testing columns are unambiguous.","section":"§IV-A, Table II"},{"comment":"The displayed prompt appears to contain stray line numbers ('451', '452', etc.) at the beginning of lines in the code block. If these are not part of the actual prompt, they should be removed for clarity.","section":"§III-B"},{"comment":"The consistency module asks ChatGPT-4V to judge outputs that the same model generated, so it is not an independent verifier. Although Table I row #4 shows that manual review gives a similar result (71.10 vs 70.9), the paper should state this self-referential limitation explicitly and report the agreement rate between the automated consistency check and manual review.","section":"§III-B and Table I"},{"comment":"The paper does not compare with Marky [2] or other recent LMM-based instruction generation methods in the experimental section. Even if those methods require different training protocols, a discussion of why they are not included as baselines would help position the contribution.","section":"§II-B and §IV-C"}],"recommendation":"major_revision","confidential_remarks":"The core issue for me is the confound between R2R and R2R* in the fine-tuning protocol: the paper's central SOTA comparison does not isolate instruction quality from data quantity. The fix is feasible within the manuscript's scope (add a same-size extra-data control, report seed variance, and release the dataset), so I recommend major revision rather than rejection. I would also ask the editor to ensure that the authors validate the external action inverse model, since both generation and verification depend on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, InstruGen is a real pipeline: YouTube house tours, CLIP node labels, an action inverse model, ChatGPT-4V instruction generation with a consistency/regular-expression verification loop, and a path-instruction judgment training task. That combination is new relative to template-based generators like Lily's. Second, the headline SOTA claim is weaker than it looks. The model that gets 71.10 SR on R2R val-unseen is fine-tuned on R2R* (original R2R plus an equal-sized InstruGen rewrite of those same trajectories). Baselines are fine-tuned on R2R only. Your ablation shows row 4 (coarse pretraining + R2R only) lands at 69.93, statistically indistinguishable from Lily's 70.00; the +1.17 jump to row 6 comes from adding R2R#. So the paper conflates data quantity with data quality, and the abstract's 'state-of-the-art' claim isn't established.\n\nWhat I liked: the verification ablation is honest and useful—Table I shows the consistency check and regular expressions each add a couple points, and the manual-review row suggests the generated instructions get close to human quality. The granularity comparison (coarse vs fine instructions) is a nice angle, and the one-shot/zero-shot results are suggestive, though also not error-bared. The paper also has a limitations paragraph, which is rare.\n\nSoft spots beyond the R2R* confound: no error bars or significance tests anywhere; the generated dataset is not released (code is, but the data isn't, despite claiming to 'propose a dataset'); the pipeline depends on a closed commercial LMM (ChatGPT-4V) and on an external action inverse model [12] that isn't independently evaluated—if that model mislabels actions, the instructions inherit the error. Marky, which generates synthetic instructions with imitation learning, isn't compared against; that's a relevant baseline for the data-generation claim.\n\nNet: it's a credible data-centric VLN paper with a clear pipeline and a genuine verification contribution. The central empirical claim needs a fair comparison—give baselines equal fine-tuning data, report variances, release the data, and add Marky and a couple of recent baselines. I'd send it to review, with the expectation of major revision.","headline":"A solid data-generation pipeline with an honest ablation, but the SOTA claim is confounded by unequal fine-tuning data.","tokens_in":11972,"tokens_out":3166,"would_cite":false,"duration_ms":76994,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstruGen claims that automatically generated path-instruction pairs from YouTube house tours, produced by ChatGPT-4V with a multi-stage verification mechanism, give vision-and-language navigation agents state-of-the-art performance on…","keywords":["vision-and-language navigation","instruction generation","large multimodal models","ChatGPT-4V","YouTube house tour videos","path-instruction pairs","R2R benchmark","RxR benchmark"],"falsifier":"Collect a test batch of sampled YouTube trajectories, deliberately flip a random subset of action labels (left to right, straight to turn), run the full InstruGen generation and verification pipeline, and measure how many corrupted instructions pass verification and how far a follower agent trained on them drifts on R2R unseen splits. If corrupted labels pass verification at a high rate, the consistency module is not actually checking against the path, and the claimed alignment is not the source of the gains.","tokens_in":11024,"feed_emoji":"🧭","tokens_out":6025,"duration_ms":57674,"temperature":0.7,"pith_summary":"InstruGen sets out to solve a data bottleneck in vision-and-language navigation: agents trained on the standard R2R/RxR pairs generalize poorly to unseen homes because the training scenes are few and the instructions are templated. The paper's claim is that a fully automatic pipeline can mine house-tour videos from YouTube, sample navigation trajectories, and use a large multimodal model (an AI system that reads images and text and writes text) to write fresh instructions for each path, with a multi-stage verification mechanism filtering out hallucinated or inconsistent outputs. If that claim is right, the expensive manual collection of path-instruction pairs can be replaced by an endlessly renewable source: any walking-through-a-home video becomes usable training data. The reported experiments support the claim by showing that agents pre-trained on InstruGen data beat prior methods on R2R and RxR, with the largest gains on unseen environments.","feed_headline":"AI-written tour directions outdo template routes in unseen homes","feed_subtitle":"Pre-training on LMM-generated path-instruction pairs from YouTube videos raises success on R2R and RxR unseen splits.","key_machinery":"The load-bearing object is the trajectory-as-triplets representation combined with the verification mechanism. Each sampled path becomes a sequence such as $\\{..., (I_{\\mathrm{room}}^t, C_t, a_t), (I_{\\mathrm{trans}}, \\mathrm{None}, \\mathrm{None}), ...\\}$, where $C_t$ is a CLIP label of the room and its key objects and $a_t$ is an action estimated by the inverse model of [12]; these triplets are what ChatGPT-4V is prompted to turn into prose. The verification mechanism then extracts $(\\text{node label}, \\text{action})$ pairs from the generated instruction using the proximity principle and compares them against the ground-truth triplet sequence, regenerating the instruction on any mismatch and applying regular expressions to remove artifacts. This mechanism is what converts a stochastic, sometimes hallucinating LMM into a reliable generator of aligned path-instruction pairs, and its ablation accounts for most of the reported quality gain.","core_discovery":"The core discovery is that the bottleneck in VLN generalization is not the navigation model itself but the scarcity and rigidity of training instructions. InstruGen demonstrates that by representing a video-sampled trajectory as a sequence of triplets—image, CLIP-derived room-and-object label, and an estimated turn-or-straight action—a multimodal LLM can be prompted to produce instructions at either coarse or fine granularity that stay aligned with the visual path. The multi-stage verification mechanism (extract the room-and-action pairs implied by the generated text, compare them against the ground-truth triplet sequence, regenerate on mismatch, then clean with regular expressions) is what makes the pipeline reliable enough to use for pre-training. Across R2R and RxR, agents trained with these pairs reach the reported state-of-the-art success rates, particularly on unseen validation splits, and the zero-shot and one-shot settings also improve.","pith_inferences":["If the gains survive replication, the same triplet-and-verify recipe could be applied to other instruction-following domains where raw video is plentiful but labeled demonstrations are scarce, such as tabletop manipulation, driving, or service robots, provided an action estimator exists.","The method's ceiling is set by the action inverse model: since actions are estimated rather than ground truth, instruction quality cannot exceed label accuracy; a testable consequence is that replacing the inverse model with a stronger one should further improve R2R unseen success.","The paper compares against template-generated instructions (Lily) but not against another LMM-based instruction generator; a like-for-like comparison of generation cost and downstream performance would separate the contribution of the LMM itself from the verification pipeline.","An auxiliary metric on instruction quality itself, such as human-rated instruction validity or text-to-path consistency measured by a trained discriminator, would make the mechanism's contribution more directly observable than downstream navigation metrics alone."],"forward_implications":["Pre-training on InstruGen pairs raises success rate on R2R unseen validation from 70.00 (Lily) to 71.10, with larger relative gains on the harder RxR zero-shot setting (29.35 vs. 27.20), so the value concentrates exactly where VLN agents historically fail: never-seen environments.","Because the pipeline accepts any house-tour video and any prompt granularity, VLN training data can expand without manual annotation or simulator rendering, removing a major scaling constraint on instruction diversity.","The verification module makes LMM-generated instructions nearly as reliable as manually reviewed ones, as the manual-review row in Table I differs only slightly from the automatic verification row, suggesting that expensive human inspection can be largely automated.","Coarse-grained instructions during pre-training outperformed fine-grained ones, while mixing rewritten R2R instructions with originals (R2R*) gave the best fine-tuning, implying instruction diversity matters more than instruction length during pre-training."],"supporting_citations":[{"why":"Supplies the YouTube house-tour video source and the trajectory-sampling scheme InstruGen builds on; also the main baseline (Lily) that InstruGen outperforms.","marker":"[14]"},{"why":"Supplies the action inverse model that estimates turn/straight actions between trajectory nodes; these action labels are part of the triplets fed to ChatGPT-4V.","marker":"[12]"},{"why":"Supplies ChatGPT-4V, the LMM used for instruction generation; the paper's novelty is prompting and verifying it rather than training it.","marker":"[16]"},{"why":"Supplies the in-domain pretraining methodology (MLM/MVM and path ranking) and is a baseline; InstruGen follows Airbert's two-stage fine-tuning recipe.","marker":"[11]"},{"why":"Supplies CLIP, used to label each room node with its room type and key objects, preventing visual hallucinations by the LMM.","marker":"[32]"},{"why":"Supplies the ViLBERT-style backbone architecture on which the pretext tasks and path-instruction judgment are built.","marker":"[33]"},{"why":"Supplies the R2R benchmark and the definition of the navigation task; the main evaluation suite.","marker":"[4]"},{"why":"Supplies the RxR benchmark used for zero-shot evaluation; the harder multilingual test set.","marker":"[20]"}],"fun_headline_variants":["YouTube videos train VLN agents via AI-generated instructions","LMM-generated navigation instructions boost unseen-home success","InstruGen: AI instructions from YouTube tours lift VLN scores","Using house tour videos, AI writes route instructions for VLN","AI-crafted path instructions from videos improve VLN agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline presupposes that the external action-inverse model correctly identifies whether the camera turned left, turned right, or went straight between consecutive room nodes; a wrong action label produces a plausible-sounding instruction that describes a different route than the actual trajectory.","fun_headline_variants_meta":{"raw":{"variants":["YouTube videos train VLN agents via AI-generated instructions","LMM-generated navigation instructions boost unseen-home success","InstruGen: AI instructions from YouTube tours lift VLN scores","Using house tour videos, AI writes route instructions for VLN","AI-crafted path instructions from videos improve VLN agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1288,"prompt_tokens":936,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":552,"tokens_out":352,"duration_ms":4035,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:33:06.217574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a test batch of sampled YouTube trajectories, deliberately flip a random subset of action labels (left to right, straight to turn), run the full InstruGen generation and verification pipeline, and measure how many corrupted instructions pass verification and how far a follower agent trained on them drifts on R2R unseen splits. If corrupted labels pass verification at a high rate, the consistency module is not actually checking against the path, and the claimed alignment is not the source of the gains.","supporting_citations":[{"cited_title":"Learning vision-and-language navigation from youtube videos,","cited_arxiv_id":null,"evidence_quote":"Supplies the YouTube house-tour video source and the trajectory-sampling scheme InstruGen builds on; also the main baseline (Lily) that InstruGen outperforms."},{"cited_title":"Learning Navigation Subroutines from Egocentric Videos","cited_arxiv_id":"1905.12612","evidence_quote":"Supplies the action inverse model that estimates turn/straight actions between trajectory nodes; these action labels are part of the triplets fed to ChatGPT-4V."},{"cited_title":"Air- bert: In-domain pretraining for vision-and-language navigation,","cited_arxiv_id":null,"evidence_quote":"Supplies the in-domain pretraining methodology (MLM/MVM and path ranking) and is a baseline; InstruGen follows Airbert's two-stage fine-tuning recipe."},{"cited_title":"Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the ViLBERT-style backbone architecture on which the pretext tasks and path-instruction judgment are built."},{"cited_title":"Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,","cited_arxiv_id":null,"evidence_quote":"Supplies the RxR benchmark used for zero-shot evaluation; the harder multilingual test set."}],"review_version":1}