{"id":"44050208-7b0c-4989-9c0b-2acd3799f44c","arxiv_id":"2411.14811","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Bayesian Optimization-based adversarial framework that selects frames to replace in positive trajectories creates harder contrastive vision negatives and modestly improves VLN navigation performance.","lead":"This paper proposes a Bayesian Optimization-based adversarial training method that creates hard fine-grained vision negatives for training vision-and-language navigation models. It reports modest gains over prior work on the R2R and REVERIE benchmarks, mainly in unseen environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own best configuration (out-domain replacement, Table V index 7) contradicts the 'fine-grained' mechanism: replacing a frame with one from a different room makes the negative semantically coarse, so the reported gains do not establish fine-grained alignment.","rationale":"Reader's weakest assumption is correct and is the most load-bearing point. The paper's own ablation shows the winning configuration uses out-domain replacement; this directly undermines the semantic premise of 'fine-grained' negatives. The central claim has two parts: (1) BO selects impactful frames, and (2) the resulting negatives are fine-grained and improve VLN. Part (1) is plausibly supported by the random-mask comparison, but part (2) is not, because the chosen replacement frames are semantically far. The large variance in Table I makes the embedding-distance evidence weak. I did not focus on the lack of held-out test results or error bars because, while important, those would affect the strength of the performance claim; the semantic mismatch attacks the mechanism itself. If the mechanism is wrong, the paper is at best a contribution on hard-negative mining by BO, not on fine-grained alignment. The conditional verdict remains appropriate: the empirical gains are modest and the mechanism needs verification.","tokens_in":12530,"tokens_out":4312,"duration_ms":42646,"concrete_test":"Compute an encoder-independent semantic distance between each positive trajectory and its corresponding FGN negative under the deployed out-domain setting—e.g., mean cosine distance in CLIP or ImageNet feature space between the replaced frame and the original frame it replaced, and/or agreement of predicted room labels. Compare these distances with random-shuffle and beam-search negatives. If out-domain FGN negatives are statistically indistinguishable from coarse negatives (similar distance, low room-label agreement), the 'fine-grained' interpretation fails. Also rerun FGVLN with in-domain replacement plus BO-selected masks: if out-domain still gives higher val-unseen SR, the gain is attributable to coarse adversarial augmentation, not to fine-grained negatives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BO identifies the frames whose replacement creates *fine-grained* vision negatives that improve cross-modal alignment. The generation function F(v+, M, xr) in Eq. 4 replaces masked frames with a single replacement frame xr. Section V-D and Table V show that the best configuration (FGVLN-add-iter, index 7) uses out-domain xr, sampled from a different room, and that this beats all in-domain configurations (indices 3-4). A path with one frame swapped for a frame from a different room is not semantically close to the positive path: it breaks visual continuity and likely shifts the trajectory into a different scene context. That is a coarse negative, not a fine-grained one. The evidence offered in Table I does not resolve this: the L2 distances for 'Fine-grained' negatives are 4.72 ± 95.79 (Lily) and 7.64 ± 47.35 (Ours), with enormous variance, no significance test, and no direct semantic-similarity measurement. Therefore the stated mechanism—teaching fine-grained alignment—is not the only (or best) explanation for the gains; the improvement could come from generic adversarial augmentation or harder negatives. The outcome may be real, but the central claim about *fine-grained* negatives is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FGVLN, a Bayesian-optimization-based adversarial training framework for Vision-and-Language Navigation (VLN). The key idea is to use a TPE-based Bayesian optimizer in an inner maximization loop to select frames in a positive trajectory whose replacement produces contrastive 'fine-grained' vision negatives, then to train the model on these negatives via an outer minimization loop. The authors evaluate the learned embeddings on R2R and REVERIE in both discriminative and generative settings, reporting modest gains over the Lily baseline (e.g., R2R validation unseen SR 67.69 vs. 66.70) and providing an ablation study of BO configurations. The paper includes source code and trained models.","tokens_in":12793,"tokens_out":4743,"duration_ms":52674,"significance":"If the central claim is supported, the contribution is potentially useful: it introduces a principled, BO-driven mechanism for generating vision negatives in VLN contrastive learning, with a clear separation of inner and outer optimization and a delayed-update strategy. The empirical study covers two benchmarks and both discriminative and generative settings, and the release of code and models is a strength. However, the current evidence does not fully establish that the improvement is due to 'fine-grained' negatives: the paper's own best configuration uses out-domain replacement frames, and the statistical support for the headline improvements is weak, with large variances and no significance tests. The significance of the work is therefore contingent on additional validation that the proposed negatives are indeed semantically fine-grained and that the gains are not attributable to generic hard-negative augmentation or selection on the validation split.","major_comments":[{"comment":"The central mechanism is not established by the reported experiments. Table V identifies index 7 (FGVLN-add-iter) as the best configuration, and this configuration uses out-domain replacement frames, i.e., frames sampled from a different room. The generation function in Eq. (4) replaces masked frames with such a frame, producing a trajectory that contains one or more observations from a different scene context. The paper does not provide any measure of semantic distance between the original and replaced paths, so it is unclear why these negatives should be called 'fine-grained' rather than 'hard' or 'coarse' negatives. The L2 distances in Table I have enormous standard deviations (e.g., 4.72 ± 95.79 and 7.64 ± 47.35), so they do not resolve this. To support the central claim, the authors should either define and measure fine-grainedness directly (e.g., through per-frame semantic similarity or a perceptual distance metric) or compare against an alternative hard-negative generator (e.g., random out-domain replacement without BO) to show that the BO-selected masks, rather than the out-domain operation, drive the gains.","section":"Section V-D, Table V, Eq. (4)"},{"comment":"The headline navigation improvements are small and are reported without statistical confidence. On R2R validation unseen, FGVLN improves SR by 0.99 absolute (67.69 vs. 66.70) and SPL by 0.02 absolute (0.64 vs. 0.62); the paper does not report the number of seeds, standard deviations, or significance tests for any navigation metric. Table I, which is used to argue that the learned encoder better distinguishes fine-grained negatives, reports L2 means with standard deviations that exceed the mean differences (e.g., 4.72 ± 95.79 for Lily and 7.64 ± 47.35 for FGVLN). With such variance, the claim of better distinguishability is not statistically supported. Please provide multiple runs, confidence intervals, and a statistical test (e.g., bootstrap or paired test) for both the embedding distances and the navigation metrics.","section":"Table I and Tables II–III"},{"comment":"The configuration is selected on the validation unseen split and the same split is then used to report the final performance in Tables II and III. The text states: 'we select the model setting with the best performance in the unseen dataset for all our experiments.' This is a form of selection on the evaluation split, which can inflate the reported gains. The authors should either report results on a held-out test unseen split (as used in the R2R/REVERIE benchmarks) or justify why using validation unseen for both model selection and final reporting does not bias the comparison.","section":"Section V-D and experimental protocol"}],"minor_comments":[{"comment":"The ablation text refers to 'Table 4 of the main manuscript' when presenting Table V, and the reference to 'row 5' is inconsistent with the numbering: index 7 has a higher validation-seen SR (61.98) than index 5 (61.25), contrary to the statement that 'the model with three iterations (row 5) performed better.' Please correct the cross-references and the data interpretation.","section":"Section V-D"},{"comment":"The generation function F(v+, M, xr) and the mask M are not fully specified: the paper does not state the number of frames that can be replaced (i.e., the cardinality of M) nor how xr is sampled in the in-domain versus out-domain cases. Please define the sampling procedure precisely, as this is central to the definition of a 'fine-grained' negative.","section":"Section IV-A, Eq. (4)"},{"comment":"The embedding visualization in Figure 3 is qualitative and lacks axes, a dimensionality reduction method (e.g., t-SNE or PCA), and a color legend that maps dots to the three negative types. Please specify the visualization procedure so the reader can interpret the claimed overlap.","section":"Section V-B, Figure 3"},{"comment":"The abstract contains grammatical errors and awkward phrasings (e.g., 'experiments on the them demonstrate'), which should be corrected. The paper would also benefit from a final proofread throughout (e.g., 'Airbert' vs. 'AirBert', 'Baysian' in the title of Section V-D).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an interesting problem and includes a reproducible framework, but the core claim about fine-grained negatives is currently not well supported. The out-domain replacement finding and the lack of statistical validation are the main obstacles. If the authors can provide a direct measure of fine-grainedness, significance tests, and results on a held-out test split, the paper could become acceptable. I would not recommend rejection because the proposed method is novel and the empirical direction is promising, but the evidence needs substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is using a TPE-based Bayesian optimizer to select which trajectory frames to replace when constructing vision negatives for path-ranking contrastive learning. That specific mechanism is not in the cited literature, and the ablations separating BO from random mask selection show a real effect (e.g., indices 2 vs 6–7 in Table V). The paper also shows consistent, if modest, gains over Lily on R2R, and the generative adaptation on REVERIE gives a clear SPL improvement. The delayed-update idea is a sensible guard against online–target feedback and is tested directly. Credit is also due for promising to release code and for doing a fairly systematic design-space exploration.\n\nThe soft spots are in the conceptual framing. The paper calls these 'fine-grained' negatives, but the best configuration (index 7) uses out-domain replacement frames sampled from a different room, and the authors' own text says that in-domain frames—the ones that would actually be semantically close—overfit and are 'not sufficiently informative.' A frame from another room is a coarse visual edit, so the claim that the gains come from teaching fine-grained alignment is not established. The Table I L2-distance evidence is too noisy to help: standard deviations are 40 to 130, with no significance tests. Also, the best BO configuration is selected on the validation unseen split and then reported on that same split; the ablation is run on a subset, which softens the peeking, but it is still a holdout leak. The discrepancy between the large ablation gains (+5 SR on the subset) and the small full-benchmark gain (+1 SR) is unexplained and could indicate variance or a different training regime.\n\nFor whom is this? VLN researchers working on contrastive pretraining and hard-negative mining will find the BO-for-frame-selection idea worth borrowing, and the generative-setting adaptation is a useful data point. The central mechanism, however, needs a cleaner test: a semantic-similarity measure for the generated negatives, a direct in-domain vs out-domain comparison on the full benchmark, and error bars across seeds.\n\nMy verdict: this deserves a serious referee, not a desk rejection, because the method is concrete and the distinction between 'hard' and 'fine-grained' negatives is an interesting question the field could use. I would send it to review with the expectation of a heavy-revision request.","headline":"A plausible adversarial hard-negative mining method for VLN whose 'fine-grained' framing outruns the evidence; worth refereeing, but the authors should be pushed on mechanism and selection bias.","tokens_in":13328,"tokens_out":3205,"would_cite":true,"duration_ms":40015,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that replacing only the frames a Bayesian optimizer flags in a positive navigation trajectory produces fine-grained vision negatives that improve vision-language alignment.","keywords":["Vision-and-Language Navigation","Contrastive Learning","Bayesian Optimization","Fine-grained Negative Sampling","Adversarial Training","R2R Benchmark","REVERIE Benchmark","Cross-modal Embeddings"],"falsifier":"A concrete test would be to run the full-scale R2R discriminative experiment with a random mask selector in place of the Bayesian optimizer, using identical out-domain replacement frames and the same number of additional negatives; if the random selector matches FGVLN's 67.69% unseen success within noise, the BO search is not the operative ingredient. A second test would measure the semantic distance between original and replaced paths with a vision-language embedding model or human ratings; if out-domain replacements are far from the positives, the fine-grained mechanism is not what the data show.","tokens_in":12336,"feed_emoji":"🧭","tokens_out":6005,"duration_ms":58755,"temperature":0.7,"pith_summary":"This paper sets out to show that the quality of negative path-instruction pairs, not just their quantity, is what limits how well vision-and-language navigation models align visual trajectories with language. It introduces FGVLN, a Bayesian-optimization-based adversarial training loop that searches a positive path for the one or two frames whose replacement most increases the contrastive loss, then uses those replacements to build fine-grained vision negatives. On R2R in the discriminative setting, the method reaches 67.69% success on unseen scenes versus 66.70% for the Lily baseline, with a higher SPL, and it also improves SPL and grounding metrics on REVERIE. The authors argue that the resulting embeddings separate fine-grained negatives from positives far better than the baseline encoder, and that this separation is what drives the navigation gains.","feed_headline":"Frame-swap training lifts unseen-scene navigation success to 67.7%","feed_subtitle":"A Bayesian optimizer picks which frames to replace, creating harder vision negatives that improve path-ranking and generative navigation.","key_machinery":"The load-bearing object is a binary frame mask $M$ over the $K$ frames of a positive trajectory, together with a replacement generator $F(v^+, M, x_r)$ that swaps the flagged frames for a replacement frame $x_r$, normally sampled from a different room. The inner maximization uses a TPE-based Bayesian optimizer to find the set of masks that maximize the target model's path-ranking loss; the outer minimization trains the online model against a batch augmented with the resulting fine-grained negatives. The delayed-update rule, which refreshes the target parameters every $J$ steps, keeps the inner search from chasing a moving target, and the ablation credits it with about 3% success on the unseen validation split.","core_discovery":"On the paper's own terms, the central discovery is a training procedure that turns a positive trajectory into a set of hard, fine-grained vision negatives. A Tree-structured Parzen Estimator Bayesian optimizer scores candidate binary masks over the frames of a positive trajectory, selecting masks that maximize the path-ranking loss of a frozen target model. The masked frames are replaced with a frame drawn from a different room, producing negative paths that share most of their visual content with the positive path. These negatives are then fed to an online model in a minimax-style loop, and the target model's weights are refreshed only every J steps to stabilise the search. The paper reports that this procedure yields a vision encoder whose embeddings separate fine-grained negatives from positives more cleanly than the Lily encoder does, and that the encoder transfers to both the discriminative path-ranking setting and a recurrent generative navigation policy.","pith_inferences":["Editorial inference: An implicit claim worth testing is that the hardness of the negative, rather than the specific identity of the replaced frame, is what drives the gain; a direct test would compare BO-selected masks against a random mask selector on the full dataset with identical replacement frames and the same number of negatives.","Editorial inference: Because out-domain replacement frames come from a different room, the label 'fine-grained' depends on the negative staying semantically close to the original path; measuring path-level semantic distance with a separate vision-language model would show whether the mechanism is truly fine-grained or partly a domain-mixing regularizer.","Editorial inference: The same mask-search-plus-replacement recipe could transfer to other contrastive sequence-alignment tasks, such as video-text retrieval or embodied instruction following, wherever a sequence can be decomposed into frames that can be masked and swapped.","Editorial inference: The paper's observation of more uniform attention across sequences predicts that FGVLN encoders should be more robust to single-frame perturbations, such as occlusions or noise, than the Lily encoder; this could be tested directly on the R2R validation set."],"forward_implications":["In the discriminative R2R setting, FGVLN reaches 67.69% success and 0.64 SPL on unseen scenes, against 66.70% and 0.62 for Lily, while keeping trajectory length comparable.","In the generative R2R setting, FGVLN reports the best success rate and SPL among the compared models that are fine-tuned without the extra augmented data used by EnvDrop.","On REVERIE unseen scenes, FGVLN improves navigation SPL to 22.09% and remote grounding RGSPL to 19.10%, indicating gains beyond path selection.","Ablations show that out-domain replacement frames outperform in-domain ones, and that more Bayesian-optimizer iterations improve unseen-scene success, whereas a random mask selector with the same replacement frames does not.","The learned embeddings keep small L2 distances between positives and fine-grained negatives yet still separate them, implying that the encoder captures subtle visual differences that the baseline misses."],"supporting_citations":[{"why":"Lily supplies the pre-trained backbone and the main baseline whose embeddings and R2R numbers FGVLN is compared against.","marker":"[8]"},{"why":"ViLBERT provides the two-stream Transformer encoder architecture and the path-ranking formulation used for contrastive training.","marker":"[33]"},{"why":"VLN-BERT introduces the path selection formulation, image-text pre-training, and the beam-search negatives that this work extends.","marker":"[6]"},{"why":"AirBERT supplies the in-domain pre-training recipe, the MLM/MVM losses, and the adaptation protocol used for the generative setting.","marker":"[7]"},{"why":"The Tree-structured Parzen Estimator is the Bayesian optimization algorithm used for mask selection in the inner maximization.","marker":"[42]"},{"why":"The sparse adversarial video attack work provides the adversarial-example inspiration for locating frames that most impact model predictions.","marker":"[14]"},{"why":"The recurrent VLN-BERT architecture is the generative policy that the trained FGVLN backbone is adapted into for R2R and REVERIE.","marker":"[41]"},{"why":"The R2R benchmark supplies the dataset and the SR, SPL, NE, and TL metrics on which the main comparisons are made.","marker":"[2]"}],"fun_headline_variants":["Bayesian frame-swap creates harder vision negatives for VLN","Hard negatives via Bayesian optimization improve VLN alignment","Frame-swap training with Bayesian optimizer sharpens VLN","Optimized frame replacement boosts vision-language navigation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the assumption that replacing one or two frames of a positive trajectory with frames from a different room produces a negative that is still semantically close to the original path; if those replacements actually create a coarse, obviously different trajectory, then the reported gains are not caused by fine-grained alignment.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian frame-swap creates harder vision negatives for VLN","Hard negatives via Bayesian optimization improve VLN alignment","Frame-swap training with Bayesian optimizer sharpens VLN","Optimized frame replacement boosts vision-language navigation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3045,"prompt_tokens":859,"completion_tokens":2186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":2135}},"tokens_in":475,"tokens_out":2186,"duration_ms":15920,"temperature":1.0,"reasoning_tokens":2135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:51:20.312238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to run the full-scale R2R discriminative experiment with a random mask selector in place of the Bayesian optimizer, using identical out-domain replacement frames and the same number of additional negatives; if the random selector matches FGVLN's 67.69% unseen success within noise, the BO search is not the operative ingredient. A second test would measure the semantic distance between original and replaced paths with a vision-language embedding model or human ratings; if out-domain replacements are far from the positives, the fine-grained mechanism is not what the data show.","supporting_citations":[{"cited_title":"Learning vision-and-language navigation from youtube videos,","cited_arxiv_id":null,"evidence_quote":"Lily supplies the pre-trained backbone and the main baseline whose embeddings and R2R numbers FGVLN is compared against."},{"cited_title":"Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,","cited_arxiv_id":null,"evidence_quote":"ViLBERT provides the two-stream Transformer encoder architecture and the path-ranking formulation used for contrastive training."},{"cited_title":"Improving vision-and-language navigation with image-text pairs from the web,","cited_arxiv_id":null,"evidence_quote":"VLN-BERT introduces the path selection formulation, image-text pre-training, and the beam-search negatives that this work extends."},{"cited_title":"Airbert: In-domain pretraining for vision-and-language navigation,","cited_arxiv_id":null,"evidence_quote":"AirBERT supplies the in-domain pre-training recipe, the MLM/MVM losses, and the adaptation protocol used for the generative setting."},{"cited_title":"Vln-bert: A recurrent vision-and-language bert for navigation,","cited_arxiv_id":null,"evidence_quote":"The recurrent VLN-BERT architecture is the generative policy that the trained FGVLN backbone is adapted into for R2R and REVERIE."},{"cited_title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,","cited_arxiv_id":null,"evidence_quote":"The R2R benchmark supplies the dataset and the SR, SPL, NE, and TL metrics on which the main comparisons are made."}],"review_version":1}