{"id":"e8136387-1a17-4477-8853-2ba3dd4f6f45","arxiv_id":"2506.18535","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning all-MiniLM-L6-v2 on MS MARCO with triplet loss degrades MRR@10 by 13.5 to 32.3 percent relative to the base model.","lead":"This paper reports that fine-tuning a popular sentence-embedding model on an information-retrieval benchmark made its ranking performance worse, and uses visualizations to suggest why. The result may matter because it questions a common assumption that fine-tuning usually helps, though the experiments cover only one model and one training setup.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Universal 'fine-tuning fails' claim rests on a single non-standard loss and no validation-based selection; the paper's own §VI-F.4 concedes a different loss could change the outcome.","rationale":"The reader's verdict identifies the same weakest assumption: the fine-tuning protocol is not representative of fine-tuning approaches in general. I agree, and the strongest evidence for this concern is internal. The paper's limitation section concedes that the exclusive use of triplet loss is a 'significant limitation' and that MultipleNegativesRankingLoss 'could significantly impact' the results. The central claim in the abstract is universal — 'all fine-tuning approaches underperform' — but the experiments only cover one loss, one margin, three epochs for most variants, no early stopping, no seed variation, and a 1M-subsample of the available training triplets. The claimed saturation hypothesis is a post hoc explanation rather than a tested mechanism, and the paper explicitly redirects to future work the question of whether alternative losses could alter the outcome. A single successful run with a standard loss and validation-based selection would falsify the universal framing. The concrete test is therefore the minimal experiment that separates 'fine-tuning fails on MS MARCO' from 'this particular pipeline failed on MS MARCO.' Because the paper itself provides the relevant concession, the REJECT verdict is appropriate and I see no reason to manufacture a different attack; the internal limitation is sufficient.","tokens_in":7041,"tokens_out":3300,"duration_ms":35233,"concrete_test":"Re-run the Full FT (Random) and LoRA FT (Random) configurations with MultipleNegativesRankingLoss instead of triplet loss, using the same 1M triplet sample, the same LoRA rank, and a dev-based early-stopping rule (e.g., stop when dev MRR@10 on qrels.dev.tsv stops improving). Compare the best checkpoint's MRR@10 against the base model's 0.3026. If either variant's best checkpoint exceeds 0.3026, the universal claim in the abstract and conclusion is false; if neither does, the claim survives this test.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline claim is that 'all fine-tuning approaches underperform the base model' (Abstract; §V-A, Table I). The evidence base, however, consists of four runs that share a single training objective — triplet loss with margin 0.2 (§III-C) — and fixed epoch counts with no dev-based early stopping (§IV-B). The manuscript itself flags this in §VI-F.4: 'Our exclusive reliance on triplet loss represents a significant limitation... alternative loss functions such as MultipleNegativesRankingLoss [14] could significantly impact the preservation-enhancement balance.' That admission is not a minor caveat; it concedes that the central claim is conditional on a non-standard protocol. Standard SBERT retrieval fine-tuning typically uses MultipleNegativesRankingLoss with validation-based checkpoint selection, and the paper tests neither. If those standard choices produce any variant that beats MRR@10 0.3026, the 'fine-tuning fails' conclusion collapses from a general result to an artifact of this specific recipe. The hard-negative variants also converge poorly (Table III: final loss 2.26 and 3.10), so the 'universal' pattern may be dominated by under-training rather than by a fundamental saturation effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes sentence-transformers/all-MiniLM-L6-v2 on the MS MARCO passage-ranking task using triplet loss with margin 0.2, in four configurations: full parameter fine-tuning and LoRA, each with random and hard negatives. It reports that all four variants underperform the base model (MRR@10 0.3026, degradations of 13.5% to 32.3%), attributes the failures to a saturation effect caused by the base model's heavy MS MARCO pretraining, supports this with UMAP visualizations of alleged embedding-space flattening, and additionally reports that LoRA incurs a roughly 2x inference slowdown. The paper concludes that fine-tuning can actively degrade billion-scale-pretrained representations and that architectural change, rather than further parameter tuning, is needed.","tokens_in":7227,"tokens_out":6121,"duration_ms":70735,"significance":"If the broad claim were established, the paper would be a valuable cautionary counterexample to the default assumption that fine-tuning a strong retrieval model on task data improves it. The paper is clearly organized, uses a standard benchmark, and provides code, data, and model links, which are welcome reproducibility assets. However, the evidence supports only a narrow statement about the specific protocol tested: triplet loss, fixed epoch counts, no dev-based early stopping, and unmerged LoRA adapters during inference. The universal claims in the abstract and conclusion are not supported by the experiments as reported; the saturation hypothesis is plausible but is not tested by any distinguishing experiment.","major_comments":[{"comment":"The central claim that 'all fine-tuning approaches underperform' is not established because every fine-tuned variant uses the same training objective, triplet loss with margin α=0.2 (§III-C), and no validation-based early stopping (§IV-B). The paper does not test MultipleNegativesRankingLoss or ContrastiveLoss, which are standard for Sentence-BERT retrieval fine-tuning, and §VI-F.4 explicitly concedes that an alternative loss 'could significantly impact the preservation-enhancement balance'. Since even one positive result with a standard loss would invalidate the universal conclusion, the headline should be narrowed to a claim about this particular triplet-loss protocol.","section":"Abstract; §V-A, Table I"},{"comment":"All MRR values come from a single run per configuration, with no standard deviation, confidence intervals, or multiple seeds. The differences among Full FT (Hard) (0.2536), LoRA FT (Random) (0.2557), and Full FT (Random) (0.2619) are small enough that run-to-run variance could change the ranking of these variants; the strongest quantitative statement that survives is that these four runs did not beat the base model, not that fine-tuning universally fails.","section":"§IV-B; §V-A, Table I"},{"comment":"The reported ~2x inference slowdown for LoRA models appears to assume unmerged adapters, but the paper never states whether the adapters were merged into the base weights before timing. With LoRA, the adapter matrices can be folded into W_q and W_v so that inference cost equals the base model. As written, Table II may measure a particular implementation choice rather than a property of LoRA; the 'hidden computational costs' conclusion requires a merged-weights timing comparison and a precise description of the inference code path.","section":"§V-B, Table II"},{"comment":"The hard-negative variants end training with final losses of 2.26 (Full FT Hard) and 3.10 (LoRA FT Hard), far above the random-negative variants (0.79 and 1.42), and the LoRA hard curve is described as unstable. These numbers indicate that the hard-negative models are under-optimized, so their poor MRR cannot be attributed to a fundamental saturation or hard-negative paradox; it may be an artifact of insufficient training or training instability. A converged hard-negative run, or a matched-convergence comparison, is needed before drawing conclusions about negative sampling.","section":"§V-D, Table III"},{"comment":"The 'embedding space flattening' explanation is based on visual inspection of UMAP projections of 1,000 query-passage pairs. No quantitative geometry measure (e.g., average pairwise cosine distance, isotropy, intrinsic dimensionality, or cluster separation) is computed, no UMAP hyperparameters or random seed are reported, and the visual correlation is not tested against chance. The explanation is therefore a post hoc restatement of the observed performance drop rather than an independent test of the proposed mechanism.","section":"§V-C, Fig. 2; §VI-B"}],"minor_comments":[{"comment":"There is a typo in 'V ocabulary size'; it should read 'Vocabulary size'.","section":"§III-B"},{"comment":"The 'random negatives' dataset is described as sampled from triples.train.small.tsv, but the paper does not clarify whether the negatives in that file are truly random or BM25-selected. Please define the source distribution of the negatives before labeling them 'random'.","section":"§III-D"},{"comment":"The 'Eval Cosine Accuracy' column in Table III is never defined; specify whether it is computed on the training set, a held-out set, or the dev qrels, and state the exact formula.","section":"§IV-C; Table III"},{"comment":"MultipleNegativesRankingLoss is cited to a documentation page; please cite the original method (e.g., Henderson et al., 2017, or Karpukhin et al., 2020) in addition to, or instead of, the documentation.","section":"§VI-F.4; Reference [14]"},{"comment":"The UMAP figure caption and text do not report the UMAP hyperparameters (n_neighbors, min_dist, metric) or the random seed, and it is unclear whether the same 1,000 pairs are used across all variants; without these details the visual comparison is difficult to reproduce.","section":"§V-C, Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"I concur with the reader that the current framing is too broad for the evidence. The paper would be acceptable only if the authors substantially narrow the claims to the specific protocol tested, add at least one standard-loss baseline and dev-based model selection, merge LoRA weights for a fair inference-time comparison, and report multiple seeds or other uncertainty information. The existing self-acknowledged limitations in §VI-F.4 and §VII already undercut the universal conclusions, so the revision path is clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a concrete, reproducible core: four fine-tuned variants of all-MiniLM-L6-v2 (full and LoRA, random and hard negatives) all underperform the base model on MS MARCO MRR@10, and they've released the code, data, and model checkpoints. That specific comparison is new as far as I can tell, and the numbers look like they could be genuine for the exact configurations described.\n\nWhat's not new, and what the paper overclaims, is the lesson. All four runs share one recipe: triplet loss with margin 0.2, fixed epoch counts, final checkpoints without dev-based early stopping, and a 1M-triplet subsample. The manuscript's own limitations section (VI-F.4) concedes that switching to MultipleNegativesRankingLoss could change the outcome. That is not a footnote; it collapses the universal \"fine-tuning fails\" statement. The LoRA inference slowdown (2x) is most likely an artifact of unmerged adapters, so the \"hidden cost\" conclusion is probably wrong. The UMAP flattening is descriptive, not mechanistic, and the hard-negative runs converge so poorly (final losses 2.26 and 3.10) that the overall pattern may just reflect under-training, not a fundamental saturation effect.\n\nI think the reader's REJECT verdict is correct on substance: the central generalization is unsupported because two standard controls are missing – a MultipleNegativesRankingLoss run with dev-based checkpoint selection, and a run with a larger or better balanced sample. Without at least the first, the paper is a cautionary anecdote about one training recipe, not a general result about fine-tuning on saturated benchmarks.\n\nOn the other hand, the authors are transparent about their setup and list the loss choice as a limitation. That honesty earns them a serious referee rather than a desk reject. A conscientious reviewer could push them to add the standard baseline, merge the LoRA adapters for inference, and reframe the conclusion as a narrow negative result. The paper is most useful to people working on sentence embedding fine-tuning and negative sampling, as a warning about protocol choices.\n\nMy recommendation: send it out for review, but expect major revision. If the authors add the missing baseline and drop the universal framing, this could be a worthwhile published negative result.","headline":"A reproducible negative result with an overbroad conclusion; the standard-loss baseline is missing.","tokens_in":7796,"tokens_out":2887,"would_cite":false,"duration_ms":30652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning an already domain-adapted dual-encoder model consistently degrades MS MARCO passage ranking, with every tested variant falling below the base model's MRR@10 of 0.3026.","keywords":["fine-tuning","passage ranking","MS MARCO","sentence embeddings","LoRA","negative sampling","embedding space analysis","saturation hypothesis"],"falsifier":"Run the same base model with a standard protocol the paper itself names as untested—for example MultipleNegativesRankingLoss, a larger or better balanced training sample, or development-based early stopping—and check whether MRR@10 on the MS MARCO dev qrels exceeds the base model's 0.3026. Any such protocol that surpasses 0.3026 would falsify the claim that fine-tuning consistently degrades this model.","tokens_in":6776,"feed_emoji":"📉","tokens_out":8520,"duration_ms":82521,"temperature":0.7,"pith_summary":"This paper tries to establish that fine-tuning an already domain-adapted sentence-embedding model can actively hurt passage ranking on MS MARCO, rather than improve it. Across five variants—full parameter fine-tuning and LoRA adaptation, each trained on random or hard negatives—every result lands below the unmodified base model, whose MRR@10 is 0.3026; degradations range from 13.5% to 32.3%. The authors attribute this to saturation: the base model was pre-trained on over one billion sentence pairs, including 9.1 million MS MARCO samples, so additional task-specific training disrupts the embedding-space structure that made it strong. If true, the result challenges the default assumption that more task data and fine-tuning yield better retrieval, and it redirects effort toward architectural change rather than parameter updates.","feed_headline":"Fine-tuning fails on MS MARCO passage ranking, losing up to 32%","feed_subtitle":"All full and LoRA variants underperform the base all-MiniLM-L6-v2 model, whose MRR@10 stays at 0.3026.","key_machinery":"The load-bearing object is the embedding space of the base dual-encoder model, a Siamese MiniLM-L6 architecture producing 384-dimensional query and passage vectors compared by cosine similarity. The paper's saturation hypothesis says that billion-scale contrastive pre-training, with heavy MS MARCO exposure, has already positioned this space so that optimizing the triplet loss $L_{\\text{triplet}} = \\max(0, \\cos(q, p^-) - \\cos(q, p^+) + 0.2)$ on a one-million-triplet sample can only distort it. UMAP projections of query-passage pairs serve as the diagnostic that ties geometric flattening to retrieval loss, and LoRA (rank 16, $\\alpha$ 32), which trains small injected matrices on attention query and value projections, is the parameter-efficient intervention whose adapters are left unmerged during inference.","core_discovery":"The paper's central claim is that on a saturated benchmark, fine-tuning degrades rather than refines a well-optimized dual-encoder model. Concretely, the base all-MiniLM-L6-v2 model, already exposed to 9,144,553 MS MARCO pairs during billion-scale contrastive pre-training, scores MRR@10 = 0.3026 on the dev queries; full fine-tuning with random negatives scores 0.2619, with hard negatives 0.2536, and the LoRA variants score 0.2557 and 0.2050. The authors interpret the uniform decline, together with UMAP projections showing progressive flattening of the embedding cloud, as evidence that fine-tuning overwrites the carefully learned geometry instead of preserving it. They also report that hard negatives hurt more than random negatives and that LoRA suffers the largest drop, alongside roughly twice as slow inference, contradicting the expectation that parameter-efficient adapters are cheap to deploy.","pith_inferences":["A testable extension the paper leaves open is whether the same failure appears with MultipleNegativesRankingLoss and development-based checkpoint selection; the authors list these as untried, so their absence leaves the universal conclusion provisional.","The scale mismatch between one million new triplets and 9.1 million MS MARCO pairs seen during pre-training suggests the result may be a data-volume effect rather than a fundamental property of fine-tuning; training on the full 39.7-million-triplet set is the obvious experiment that would separate these.","If the saturation account is right, the same pattern should appear on other heavily domain-pretrained embedding models and should weaken on less saturated domains where pre-training exposure is small.","The slower unmerged-LoRA inference hints that adapter efficiency claims should be evaluated end-to-end rather than by parameter count; merging the adapters before timing is a direct check the paper did not run."],"forward_implications":["Fine-tuning an already domain-adapted dual encoder is not a reliable route to better passage ranking on MS MARCO; every tested variant loses to the untouched base model.","Hard negatives mined by the base model itself can be counterproductive on a saturated model, producing worse results than random negatives.","Parameter-efficient LoRA is not necessarily deployment-friendly: in these experiments it roughly doubles inference time while degrading accuracy more than full fine-tuning on hard negatives.","Embedding-space visualization can expose degradation that scalar metrics alone summarize, making it a candidate diagnostic for future retrieval experiments.","If saturation is real, gains on such benchmarks are more likely to come from architectural changes, such as cross-encoders or hybrid sparse-dense systems, than from additional parameter updates."],"supporting_citations":[{"why":"Supplies the MS MARCO passage ranking dataset, training queries, collection, and dev qrels used in all experiments.","marker":"[1]"},{"why":"Documents the base model's billion-pair pre-training, including 9,144,553 MS MARCO samples, which grounds the saturation hypothesis.","marker":"[2]"},{"why":"Provides the dual-encoder Siamese architecture that the base model and fine-tuned variants use for query-passage encoding.","marker":"[3]"},{"why":"Introduces LoRA, the parameter-efficient adaptation method whose retrieval behavior and inference cost the paper evaluates.","marker":"[7]"},{"why":"Defines the siamese dual-encoder sentence embedding approach and cosine similarity scoring that the evaluation relies on.","marker":"[8]"},{"why":"Supplies the UMAP projection method used to visualize embedding-space flattening across model variants.","marker":"[9]"},{"why":"Introduces the triplet loss with margin ranking that is the paper's training objective.","marker":"[11]"}],"fun_headline_variants":["Fine-tuning all-MiniLM-L6-v2 on MS MARCO drops MRR by up to 32%","Base sentence-transformers model beats every fine-tuned variant on MS MARCO","LoRA fine-tuning underperforms full fine-tuning on passage ranking","When fine-tuning fails: MS MARCO base model wins over all tuned versions","Hard negatives hurt more than random ones in MS MARCO fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The broad conclusion rests on the assumption that the specific fine-tuning recipe tested here—triplet loss with margin 0.2, fixed epoch counts, final checkpoints without development-set selection, and a one-million-triplet training sample—is representative of fine-tuning approaches in general.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning all-MiniLM-L6-v2 on MS MARCO drops MRR by up to 32%","Base sentence-transformers model beats every fine-tuned variant on MS MARCO","LoRA fine-tuning underperforms full fine-tuning on passage ranking","When fine-tuning fails: MS MARCO base model wins over all tuned versions","Hard negatives hurt more than random ones in MS MARCO fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000974,"raw_usage":{"total_tokens":4116,"prompt_tokens":895,"completion_tokens":3221,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":3116}},"tokens_in":511,"tokens_out":3221,"duration_ms":24621,"temperature":1.0,"reasoning_tokens":3116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:15:39.634876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same base model with a standard protocol the paper itself names as untested—for example MultipleNegativesRankingLoss, a larger or better balanced training sample, or development-based early stopping—and check whether MRR@10 on the MS MARCO dev qrels exceeds the base model's 0.3026. Any such protocol that surpasses 0.3026 would falsify the claim that fine-tuning consistently degrades this model.","supporting_citations":[{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Documents the base model's billion-pair pre-training, including 9,144,553 MS MARCO samples, which grounds the saturation hypothesis."},{"cited_title":"Dong et al., ”Exploring Dual Encoder Architectures for Question An- swering,” in Proc","cited_arxiv_id":null,"evidence_quote":"Provides the dual-encoder Siamese architecture that the base model and fine-tuned variants use for query-passage encoding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces LoRA, the parameter-efficient adaptation method whose retrieval behavior and inference cost the paper evaluates."},{"cited_title":"Reimers and I","cited_arxiv_id":null,"evidence_quote":"Defines the siamese dual-encoder sentence embedding approach and cosine similarity scoring that the evaluation relies on."},{"cited_title":"Schroff, D","cited_arxiv_id":null,"evidence_quote":"Introduces the triplet loss with margin ranking that is the paper's training objective."}],"review_version":1}