{"id":"ca30cf29-1682-41ce-b5fd-350048b02d1d","arxiv_id":"2607.17806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PGN adapts OpenPangu-7B with Q-Former alignment and LoRA to offline vision-language navigation action prediction, reaching 62.29% normalized action match on 500 held-out expert trajectories.","lead":"The authors adapted a 7-billion-parameter multimodal AI model to predict navigation actions from a language instruction and five camera views, reporting 62% agreement with expert actions in an offline one-step test. The test does not execute actions in a simulator, so the result shows feasibility of offline action matching rather than real navigation success.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 62.29% NAM headline is not independently checkable: NAM extraction/normalization rules are unspecified, no majority-action baseline is reported, and V9 per-step logs were not retained, so a parser or class-imbalance artifact cannot be ruled out.","rationale":"The reader's weakest_assumption correctly identifies the NAM extraction and missing baseline. My independent read reaches the same place: the load-bearing link in the chain is metric validity, not the model architecture. The paper is honest about scope (open-loop only), internally consistent in the counts (V8/V9 interval sums are 500), and appropriately cautious in §5.3, and none of those observations removes the fact that V8/V9 aggregate numbers cannot be recomputed. I do not see a separate architectural objection that would change the verdict, and disagreement with published VLN systems is not relevant because the authors explicitly disclaim comparison. The one concrete improvement that would settle the issue is releasing the parser and per-step logs; until then CONDITIONAL is the right verdict, so I leave the reader's verdict unchanged.","tokens_in":5776,"tokens_out":2889,"duration_ms":26145,"concrete_test":"Release the exact action-extraction/normalization code (or a formal grammar covering reasoning-then-action outputs) and the retained per-step V9 predictions for the 500 held-out trajectories. Then rerun the reported NAM computation and also compute the NAM of the trivial policy that always predicts the most frequent expert action on the same split. If the fixed parser moves V9's NAM materially (e.g., by more than 2–3 points) or the majority baseline reaches or exceeds 62.29%, the central feasibility claim should be downgraded; if both checks pass, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the V9 teacher-forced NAM of 62.29% and NER of 100% on 500 held-out trajectories. For that claim to be meaningful, NAM must faithfully measure agreement between the model's generated response and the expert action. That condition is not secured. Section 5.1 defines NAM only as 'an action is extracted from the response after normalizing forward, left, right, and stop expressions,' with no extraction grammar, no examples, and no handling of reasoning text. Because V8/V9 emit reasoning text before the action, the parser is doing real work: EM is 0.00%, so NAM is entirely determined by an unspecified extraction step. Separately, the paper says action-aware sampling was introduced to rebalance a 'forward-dominated selection' (§4.3), so the expert action distribution is plausibly imbalanced, yet no baseline such as always predicting the modal action is reported. The per-trajectory distribution (Table 1, V9: 133 trajectories in 50–60, 160 in 60–70, etc.) is consistent with many trajectories being largely forward, which makes a majority-action baseline especially important. The strongest version of the concern: with no parser specification, no baseline, and no retained V8/V9 per-step logs (explicitly acknowledged in §5.3), the 62.29% figure cannot be verified or falsified from the manuscript as written. This is a verifiability gap, not an allegation of misconduct; it is the single most load-bearing issue because the entire feasibility conclusion rests on the metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PGN, an offline Vision-Language Navigation action-prediction system built on the OpenPangu-7B multimodal language model. Training proceeds in two stages: first a visual-language alignment stage using a frozen EVA-ViT-G/14 encoder, a Q-Former, and a two-layer MLP projector; then a navigation-adaptation stage on expert trajectories using five-observation windows, epoch-dependent temporal sampling, and LoRA adapters, with all visual and language backbones frozen. The evaluation is teacher-forced and open-loop: on 500 held-out expert trajectories, the V9 iteration reports a Normalized Action Match (NAM) of 62.29% and a Non-empty Rate (NER) of 100.00%. The authors are explicit that these numbers measure offline expert-action alignment and do not constitute closed-loop navigation success, path efficiency, or goal completion. The paper also documents a development history across versions V3 through V9, including raw trajectory-level distributions, and explicitly acknowledges that per-action logs were retained only for V6.","tokens_in":6152,"tokens_out":4699,"duration_ms":40241,"significance":"If the reported evaluation were fully specified and independently reproducible, the paper would provide a useful engineering recipe for adapting a 7B-scale multimodal foundation model to offline VLN action prediction on Ascend NPUs. The authors deserve credit for several strengths: they transparently separate offline teacher-forced alignment from closed-loop navigation success; they report raw per-trajectory counts rather than only aggregate numbers; they retain and reproduce V6 per-action predictions; and they explicitly decline to claim causal attribution for individual design components. The feasibility claim—that OpenPangu-7B can be adapted to produce non-empty actions that align with expert actions on a held-out trajectory split—is modest and largely consistent with the reported NER=100.00%. However, the central metric NAM is underspecified and uncalibrated, which currently prevents an independent reader from verifying whether 62.29% reflects genuine instruction-following. Because the paper is framed as a technical report whose main conclusion rests on this number, closing that gap is necessary before the result can be accepted as a reliable evidence point.","major_comments":[{"comment":"The definition of Normalized Action Match is not operational. The text says only that \"an action is extracted from the response after normalizing forward, left, right, and stop expressions,\" but it does not specify the extraction grammar, the handling of reasoning text that precedes the action, the treatment of multi-action or ambiguous responses, or the exact normalization rules (e.g., case, synonyms, punctuation). Since V8 and V9 have EM=0.00%, NAM is entirely determined by this unspecified parser. Without a precise parser specification, example inputs and outputs, or released parsing code, the headline 62.29% cannot be verified or falsified. Please provide the full extraction and normalization procedure, including at least three worked examples from V9 responses, and state how the parser behaves when no canonical action is present in the response.","section":"Section 5.1, NAM definition"},{"comment":"No baseline calibrates the NAM numbers. The action space has only four symbols, and Section 4.3 itself states that action-aware sampling was introduced because of a \"forward-dominated selection,\" indicating strong class imbalance. Under such imbalance, a trivial policy that always predicts the modal action can achieve nontrivial action-match accuracy. Please report on the same 500 held-out trajectories the NAM of (a) always predicting the majority action, (b) a uniform random policy over the four actions, and (c) an instruction-blind policy that always predicts an action based only on the observation window. Also report the per-class distribution of expert actions in the held-out split. This baseline is necessary to interpret whether 62.29% exceeds what can be achieved without language understanding.","section":"Section 5.2 and Section 4.3, baselines"},{"comment":"The paper does not describe how the 500 held-out trajectories were selected from the 19,699 collected trajectories. Was the split random, stratified by environment, or stratified by instruction length or trajectory length? Are the held-out trajectories drawn from the same environments as the training trajectories, and is there any risk of near-duplicate instructions across the split? Without this information, the generality of the 62.29% figure is unclear, and the possibility of accidental distribution shift between training and held-out sets cannot be assessed. Please specify the exact split procedure and report environment-level overlap statistics.","section":"Section 5.1, validation split construction"},{"comment":"The V3 row reports action-level percentages but its per-trajectory interval counts sum to 56 rather than 500. The paper acknowledges this in a footnote but does not state the denominator used for V3's Strict, Fuzzy, and Non-blank percentages. If those percentages are computed over a different subset, V3 should be excluded from the version comparison entirely, or its true denominator should be given. As written, the reader cannot determine whether V3's 8.40% is computed over 56 trajectories or over a different set, which undermines its use as a development baseline. Please clarify the V3 denominator and, if necessary, remove V3 from the discussion of version-level trends.","section":"Table 1, V3 row"}],"minor_comments":[{"comment":"The phrase \"DAgger-compatible simulation pipeline\" is misleading because the recorded configuration fixes beta=1, which makes the procedure expert-only data collection followed by supervised behavior cloning. Suggest renaming this subsection to \"Expert Trajectory Collection\" and describing the data collection as behavior cloning rather than DAgger.","section":"Section 4.2"},{"comment":"The sentence \"Figure 4 retains the evaluation flow from the prior manuscript\" is a leftover from a previous version and should be removed; the figure should be described on its own terms. Similarly, the phrase \"Although the Habitat simulator is shown as the source of the recorded trajectory\" is confusing because the data collection section does not mention Habitat; please state whether Habitat was used to generate the trajectories or remove the reference.","section":"Figure 4 caption and Section 5.1"},{"comment":"Reference [3] is listed as \"Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition,\" but the paper uses \"OpenPangu-7B\" as the backbone. Please clarify whether this is the correct source for OpenPangu-7B or provide the correct citation, since the current reference appears unrelated to the model name used in the abstract.","section":"References"},{"comment":"The sentence \"Their EM is zero because these versions generate reasoning text before the action\" is a reasonable explanation, but since EM is defined as whole-response exact match, reporting EM for V8/V9 alongside versions that emit only an action phrase may mislead readers. Consider reporting EM only for versions where the output format is a single action string, or explicitly restating that EM is reported for completeness and is not comparable across formats.","section":"Section 5.2"},{"comment":"There are inconsistent spellings of the vision encoder name: \"EV A-ViT-G/14\" appears in several places and \"EVA-ViT-G/14\" in others. Please standardize. Also, \"LLaV A\" in Section 2 should be \"LLaVA.\"","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case between major revision and reject. The central feasibility claim is modest and the authors are unusually transparent about the offline nature of the evaluation and the lack of per-step logs for V8/V9. The main obstacle is the underspecified NAM metric and the absence of any baseline, both of which are fixable within the manuscript's scope by adding a parser specification and simple baselines. The manuscript also contains several artifacts suggesting it is a resubmission or extension of a previous report (e.g., \"retains the evaluation flow from the prior manuscript\"); for a journal submission, the authors should ensure all such artifacts are removed and that the relationship to any prior version or technical report is disclosed in the acknowledgments or a footnote. The citation mismatch for OpenPangu-7B should also be verified, as it may indicate a substantive attribution issue rather than a simple typo."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is not a VLN breakthrough paper; it is an honest development record for adapting OpenPangu-7B to offline expert-action matching on Ascend NPUs. Second, the 62.29% Normalized Action Match figure should be read with caution: the extraction/normalization rules for NAM are not specified, no majority-action baseline is reported, and the authors acknowledge they did not retain per-step logs for the final versions. That does not mean the number is wrong; it means it is not independently verifiable from the manuscript.\n\nWhat the paper does well is scope itself. The abstract and Section 5.3 are explicit that teacher-forced NAM is not closed-loop navigation success, that the numbers are not Success Rate or SPL, and that the version-to-version changes are not an ablation. They also kept the V6 per-action logs and reproduce V6's metrics from them, which is more than most technical reports do. The integration story is genuinely useful for practitioners: two-stage training with a frozen EVA-ViT-G, Q-Former, MLP projector, LoRA, and three structural-token embeddings, running on eight Ascend 910B NPUs with selective FP32 for stability. None of these components are new, but the specific combination and hardware notes have practical value.\n\nThe soft spots are real but narrow. The biggest one is NAM itself. Since V8/V9 generate reasoning text before the action, whole-response exact match is zero, so NAM is entirely determined by an unspecified parser that must extract the action from the reasoning text. The paper says 'normalizing forward, left, right, and stop expressions' but does not define how that works or how it handles the reasoning text. The per-trajectory distribution shows many trajectories in the 50–80% NAM range, and the paper notes action-aware sampling was added because forward-dominated selection was an issue, so a majority-class baseline (always predict forward) is exactly the calibration missing here. Also missing: a description of how the 500 held-out trajectories were split from the 19,699 collected ones, and error bars or comparable metrics across runs. These are all fixable in a revision, not fatal flaws.\n\nWho gets value from this paper: engineers adapting 7B-scale MLLMs to action prediction on Ascend hardware, and researchers who want a candid example of why open-loop teacher-forced metrics are insufficient. It deserves a serious referee, but the referee should require the parser specification, a majority baseline, and the raw logs or code before treating the headline number as meaningful.","headline":"An honest, clearly scoped engineering report on adapting OpenPangu-7B to offline VLN action matching, whose headline NAM number is plausible but not independently checkable as written because the extraction rule and baseline are missing.","tokens_in":6650,"tokens_out":1464,"would_cite":false,"duration_ms":14826,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An offline vision-language navigation system matches 62.29% of expert actions on held-out trajectories, measured as teacher-forced action alignment rather than closed-loop navigation success.","keywords":["vision-language navigation","action prediction","multimodal large language model","Q-Former","LoRA","teacher-forced evaluation","behavior cloning","offline evaluation"],"falsifier":"Re-run the V9 model on the same 500 trajectories with the extraction rule made explicit and compare Normalized Action Match against an always-forward baseline and against a random-action baseline; if the gap shrinks to near zero, the claimed alignment is not instruction-following. Alternatively, execute V9's predicted actions in a Habitat environment and measure success rate and SPL, since the paper itself identifies closed-loop evaluation as the test its offline numbers cannot replace.","tokens_in":5507,"feed_emoji":"🧭","tokens_out":10714,"duration_ms":84816,"temperature":0.7,"pith_summary":"The paper tries to show that a 7-billion-parameter multimodal language model can be turned into a vision-language navigation action predictor through a two-stage, parameter-efficient recipe rather than full fine-tuning. It first aligns a frozen vision encoder with a frozen language backbone using a trainable Q-Former and a two-layer projector, then adapts the aligned model to expert navigation trajectories with five-observation windows, reasoning-then-action outputs, structural tokens, and LoRA. On 500 held-out expert trajectories, the best version achieves a 62.29% Normalized Action Match and a 100% Non-empty Rate under teacher-forced, open-loop evaluation. The authors are careful to state that these numbers measure offline expert-action agreement, not navigation success, and that closed-loop metrics remain future work. A reader should care because the result offers evidence that a general multimodal language model can be steered toward embodied action prediction with modest trainable parameters, while the protocol itself is a development tool rather than a benchmark for embodied performance.","feed_headline":"62% of expert navigation actions matched by offline 7B model","feed_subtitle":"Teacher-forced test on 500 trajectories shows a frozen 7B backbone can be adapted with LoRA.","key_machinery":"The load-bearing mechanism is a two-stage training scheme that separates visual-language alignment from navigation-task adaptation. In Stage 1, 32 learnable Q-Former queries compress each vision-transformer patch representation into a fixed 768-dimensional vector, which a two-layer MLP projects into the language model's embedding space. In Stage 2, five chronologically ordered observations are packed into one autoregressive context using learned image-start, image-end, and frame-boundary tokens, and the model outputs reasoning text followed by a canonical action from $\\mathcal{A}=\\{\\text{forward},\\text{left},\\text{right},\\text{stop}\\}$. The factorization $P(r_t,a_t\\mid I,O_t)=P(r_t\\mid I,O_t)\\,P(a_t\\mid r_t,I,O_t)$ fixes the output format, while epoch-dependent starting offsets and an action-aware sampler vary the temporal windows and rebalance forward-dominated action selection. Only the structural-token embeddings and LoRA adapters are updated in Stage 2, keeping the aligned visual pathway and language backbone frozen.","core_discovery":"The central claim is that PGN, built on a frozen 7B language backbone with a frozen vision encoder, can learn to reproduce expert navigation actions in an offline setting. After Stage 1 trains the Q-Former and MLP projector on image-caption pairs, Stage 2 freezes the visual pathway and trains only three structural-token embeddings and LoRA adapters on expert trajectories. V9 then reports Normalized Action Match of 62.29% and Non-empty Rate of 100.00% across 500 held-out trajectories under teacher forcing, with the per-trajectory distribution concentrated in the 50–80% action-match range. The paper explicitly frames this as evidence for feasible expert-action alignment, not as a closed-loop navigation result; success rate, SPL, navigation error, and error-recovery behavior are explicitly deferred.","pith_inferences":["A cheap audit would be to compare Normalized Action Match against an always-forward baseline on the same 500 trajectories; because the data are forward-dominated, a high baseline would reveal how much of 62.29% is genuine instruction-following rather than action-distribution bias.","The sharp Non-empty Rate jump from 33.98% in V5 to 99.17% in V6 while Normalized Action Match stayed roughly flat suggests the model first learned to emit a non-empty action and only later learned to choose it correctly; inspecting retained V6 samples would test this.","The reported numbers come from one training run with several changes introduced together, so an ablation varying reasoning-then-action output, temporal sampling, action-aware sampling, and identity prompting would identify which component carries the improvement.","If a closed-loop test later shows low navigation success, the offline alignment could still be useful as an action-prior that narrows the action space for a planner or as a cheap filter before expensive simulation evaluation."],"forward_implications":["If the teacher-forced numbers hold, a 7B-scale multimodal language model can be adapted to a four-action navigation space with only LoRA and a handful of embeddings, which sharply lowers the cost of exploring vision-language navigation with large backbones.","The reasoning-then-action format yields zero exact-string match in V8 and V9 because the action is embedded in longer text, so evaluation must use action extraction, and any comparison across model versions must account for output-format changes.","The per-trajectory shift from the 10–30% action-match band in V5 and V6 to the 50–80% band in V8 and V9 suggests that the combined recipe of identity prompting, temporal sampling, action-aware sampling, and reasoning output drives improvement, though the paper states these components were not ablated separately.","A 100% Non-empty Rate means every teacher-forced step produced a response, but it says nothing about whether the chosen action is correct; the 62.29% Normalized Action Match is the only content-bearing alignment number.","Closed-loop evaluation remains the deciding test: the paper's protocol measures local agreement under expert histories, not the ability to recover after mistakes, so success rate and SPL cannot be inferred from Normalized Action Match alone."],"supporting_citations":[{"why":"Defines the R2R vision-and-language navigation task with discrete navigation graphs that PGN's offline action prediction builds on.","marker":"[1]"},{"why":"Provides the language backbone that PGN adapts for navigation action generation.","marker":"[3]"},{"why":"Supplies the LoRA low-rank adaptation technique used for Stage 2 navigation-specific updates.","marker":"[6]"},{"why":"Defines continuous-environment vision-and-language navigation with forward and turning actions, motivating the canonical four-action space.","marker":"[8]"},{"why":"Provides paired image-caption data used in Stage 1 visual-language alignment.","marker":"[9]"},{"why":"Supplies the Q-Former technique for connecting a frozen vision encoder and a frozen language model.","marker":"[11]"},{"why":"Provides paired image-caption data used in Stage 1 visual-language alignment alongside the Visual Genome data.","marker":"[12]"},{"why":"Motivates the reasoning-before-action output format that PGN adopts for navigation action prediction.","marker":"[14]"}],"fun_headline_variants":["Frozen 7B vision-language model hits 62% action match offline","LoRA adapters on a frozen 7B backbone match 62% of expert actions","Offline two-stage tuning lets a frozen 7B model navigate by text","62% action match on 500 trajectories with a frozen 7B model","Frozen vision and language align via LoRA to predict navigation actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported 62.29% Normalized Action Match score reflects genuine action following; the paper never defines how an action is extracted from the model's response or how forward, left, right, and stop expressions are normalized, so the score could be an artifact of extraction rules or of a forward-dominated action distribution.","fun_headline_variants_meta":{"raw":{"variants":["Frozen 7B vision-language model hits 62% action match offline","LoRA adapters on a frozen 7B backbone match 62% of expert actions","Offline two-stage tuning lets a frozen 7B model navigate by text","62% action match on 500 trajectories with a frozen 7B model","Frozen vision and language align via LoRA to predict navigation actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001153,"raw_usage":{"total_tokens":4795,"prompt_tokens":979,"completion_tokens":3816,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":3714}},"tokens_in":595,"tokens_out":3816,"duration_ms":23217,"temperature":1.0,"reasoning_tokens":3714,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:33:25.864592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the V9 model on the same 500 trajectories with the extraction rule made explicit and compare Normalized Action Match against an always-forward baseline and against a random-action baseline; if the gap shrinks to near zero, the claimed alignment is not instruction-following. Alternatively, execute V9's predicted actions in a Habitat environment and measure success rate and SPL, since the paper itself identifies closed-loop evaluation as the test its offline numbers cannot replace.","supporting_citations":[{"cited_title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments","cited_arxiv_id":null,"evidence_quote":"Defines the R2R vision-and-language navigation task with discrete navigation graphs that PGN's offline action prediction builds on."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, et al","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA low-rank adaptation technique used for Stage 2 navigation-specific updates."},{"cited_title":"Beyond the nav- graph: Vision-and-language navigation in continuous environments","cited_arxiv_id":null,"evidence_quote":"Defines continuous-environment vision-and-language navigation with forward and turning actions, motivating the canonical four-action space."},{"cited_title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations.International Journal of Computer Vision, 2017","cited_arxiv_id":null,"evidence_quote":"Provides paired image-caption data used in Stage 1 visual-language alignment."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"Provides paired image-caption data used in Stage 1 visual-language alignment alongside the Visual Genome data."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 2022","cited_arxiv_id":null,"evidence_quote":"Motivates the reasoning-before-action output format that PGN adopts for navigation action prediction."}],"review_version":2}