Pith. sign in

REVIEW 3 major objections 4 minor 40 references

This paper claims that a dynamic data-pruning strategy that adjusts sampling probabilities at both query and document levels during training resolves the quality-coverage tradeoff of dense-retriever finetuning, improving both ranking and re

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:28 UTC pith:DEDR6O56

load-bearing objection Useful empirical paper on two-level dynamic data pruning for dense retriever finetuning; the theory appendix is largely decorative and should be pruned, but the experiments deserve a serious look. the 3 major comments →

arxiv 2603.17205 v3 pith:DEDR6O56 submitted 2026-03-17 cs.IR cs.AIcs.CLcs.LG

OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation

Haoyang Fang , Shuai Zhang , Yifei Ma , Hengyi Wang , Cuixiong Hu , Katrin Kirchhoff , Bernie Wang , George Karypis This is my paper
classification cs.IR cs.AIcs.CLcs.LG
keywords dense retrievaldata pruningdynamic pruningdomain adaptationquality-coverage tradeoffcontrastive finetuningtraining efficiencysampling probability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard finetuning of dense retrievers treats all query-document pairs equally, but pairs differ in quality. The paper first shows that statically pruning to keep only high-similarity pairs improves ranking (NDCG) but hurts retrieval coverage (Recall), a tradeoff intrinsic to the two-stage query-then-document sampling used in retrieval training. It then proposes dynamic pruning (DP), which keeps the full training set but continuously re-weights sampling probabilities at both query and document levels, becoming more selective as the model improves. Across eight datasets and two retriever architectures, DP is claimed to improve ranking by about one NDCG point and recall by about 0.4 points while requiring less than half the training time. The paper also proves a condition under which pruning beats finetuning: the scoring function must identify true positives at a rate above the noise rate.

Core claim

The central discovery is a quality-coverage tradeoff in dense-retriever finetuning: keeping only the top-similarity training pairs (static pruning, SP) consistently improves NDCG but degrades Recall, because pruning disproportionately excludes queries with few high-quality documents and breaks the balanced query sampling that underpins retrieval coverage. The paper's resolution is dynamic pruning (DP), which assigns every pair a nonzero sampling probability but modulates those weights with cosine-scheduled strengths, so early training keeps broad coverage while later training concentrates on examples the evolving model finds informative. In the paper's experiments DP achieves the best averag

What carries the argument

Two-level sampling modulation: at each step, the model's forward pass produces query scores (mean contrastive loss offset to be positive) and document scores (cosine similarity between query and positive-document embeddings). A fixed virtual dataset size keeps the training loop compatible with standard schedulers; cosine scheduling of the strength parameters sharpens the sampling distribution as training progresses, so the model becomes more selective over time while never fully excluding any pair. Static pruning serves as a control to expose the tradeoff, and Theorem 1 formalizes the condition under which selecting above the noise rate improves learned representations.

Load-bearing premise

The whole method rests on the assumption that cosine similarity between query and document embeddings — from the frozen pretrained model for SP, or the evolving finetuned model for DP — ranks true positives above noise at a rate better than random; if that ranking is wrong, pruning removes informative hard positives and DP up-weights misleading pairs.

What would settle it

Run DP on a dataset where the pretrained model's cosine similarity is deliberately inverted (or known to be anti-correlated with relevance); if DP then underperforms standard finetuning, the claim that selection quality beats the noise rate is falsified. More directly, on any existing dataset one could measure the fraction of correctly labeled examples among the pruned-in pairs and check whether it exceeds the noise rate; Theorem 1 predicts a reversal below that threshold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If DP's claims hold, practitioners can cut the training cost of domain-adapting a dense retriever in half without sacrificing final ranking or recall.
  • SP alone offers a cheap option for ranking-focused or time-critical deployments, since it can improve NDCG while discarding the majority of training pairs.
  • In noisy-label settings, the two-stage SP+DP pipeline should be preferred, as it outperforms all baselines on both metrics in the paper's denoising experiment.
  • The pattern transfers to LLM-based retrievers, suggesting the mechanism is architecture-agnostic; larger LLM retrievers are a natural next test.
  • The quality-coverage tradeoff appears intrinsic to two-stage contrastive sampling, so the pruning framework may generalize beyond the eight datasets evaluated.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because DP relies on cosine similarity as the quality signal, its advantage may shrink in domains where the pretrained model's similarity ranking is poorly calibrated; a testable extension is to replace the score with a calibrated relevance model or hard-negative-aware scores.
  • Inference: Since DP modulates rather than excludes, it could be combined with curriculum schedules that change the definition of 'informative' over training — for instance, shifting from diversity-based to difficulty-based selection — without altering the framework's core.
  • Inference: The same two-level sampling modulation could apply to other two-stage contrastive learning problems, such as recommendation or entailment training, where quality-coverage tension is likely to appear.
  • Inference: The theorem suggests a practical diagnostic: measuring the selection-quality rate on a validation sample could tell practitioners in advance whether pruning will help or hurt, before committing to a full training run.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces OPERA, a data-pruning framework for dense-retriever finetuning. Static Pruning (SP) uses pretrained-model cosine similarity to retain a top-k fraction of query-document pairs; it is shown to improve NDCG@10 but degrade Recall@20, exposing a quality-coverage tradeoff. Dynamic Pruning (DP) replaces hard exclusion with soft, scheduled reweighting of query and document sampling, maintaining full coverage. Experiments on eight datasets spanning six domains with bge-large-en-v1.5 and Qwen3-Embedding-0.6B show that DP achieves the best average NDCG@10 and Recall@20, and that both SP and DP converge faster than standard finetuning. Appendix B states a theorem giving a condition under which pruning beats finetuning when some positives are noisy.

Significance. If the empirical results are robust, OPERA is a practical, architecture-agnostic method for efficient retrieval adaptation, and the quality-coverage tradeoff is a useful conceptual contribution. The paper is commendable for fixing pruning hyperparameters across datasets, comparing against InfoBatch and random pruning, and including both encoder-only and LLM-based retrievers. However, the stated theoretical guarantee applies only to a noisy-label setting, not to the main clean-data experiments, and the absence of variance estimates leaves the magnitude and reliability of the reported gains uncertain.

major comments (3)
  1. [Appendix B, Theorem 1 (Eq. 7)] The pruning condition γ > m+_q/m_q is vacuous for the main experiments. In the clean training sets used for Tables 2 and 3, m+_q = m_q, so the condition becomes γ > 1, impossible for any pruning rate. The paragraph 'When Does Pruning Help?' in Section 2.3 invokes this theorem to explain why DP outperforms SP, but the theorem cannot support that explanation on clean data. It only applies to the deliberately noisy denoising experiment (Section 3.5). Please restrict the theoretical claim to the noisy setting or provide a separate analysis for clean data.
  2. [Appendix B, Eqs. (9)-(11)] The proof optimizes the sum of cosine similarities to positive documents only. The actual training loss (Eq. 1) is a contrastive loss with hard negatives; the optimal query direction under that loss is not the mean of positive directions, so the inequalities in Eq. (11) do not automatically transfer to the training procedure used in experiments. Additionally, Eq. (8) asserts ESP > EDP whenever γ=ρ, while Eq. (12) states ESP > EDP iff γ > m+_q/m_q; these are inconsistent unless m+_q/m_q < γ always holds, which is not stated. The theorem needs to be corrected and stated under the loss actually optimized.
  3. [Tables 2-6 and Appendix G] No variance across seeds is reported, and the acknowledged limitation in Appendix G is a genuine concern for the central empirical claim. The headline gains are modest (BGE average NDCG@10 0.545 vs 0.535 for FT; Recall@20 0.534 vs 0.530; Qwen average NDCG@10 0.501 vs 0.488). Without standard deviations or at least 3-seed means for the main comparisons, it is difficult to assess whether these differences are meaningful. Consistency across eight datasets is encouraging, but variance estimates for the main tables (or a representative subset) are needed to support the strength of the claims.
minor comments (4)
  1. [Abstract] The reported gains are inconsistent: the abstract states 'NDCG@10 +1.0 points' and 'Recall@20 +0.4 points', while the full-text abstract states 'NDCG@10 +1.9%' and 'Recall@20 +0.7%'. Please harmonize the numbers.
  2. [Algorithm 1 / Section 2.3] The variables n0, r, and qtop are not fully defined. In particular, the formula n0 = ⌊n·(1−r_s)/α_s + r_s·n⌋ is unclear; please define all variables and explain the relationship between n0 and the original dataset size n.
  3. [References] He et al. 2024a and 2024b are the same publication (Large-scale dataset pruning with dynamic uncertainty) and should be collapsed into a single reference.
  4. [Section 3.4.1] The 'less than 50% training time' claim is based on comparing DP at 8,000 iterations to FT at 16,000 on two datasets. Please clarify whether this is a post-hoc observation or a prespecified early-stopping criterion, and report the per-iteration overhead more directly in the convergence comparison.

Circularity Check

0 steps flagged

No significant circularity: the empirical comparisons are external, DP/SP hyperparameters are not fit to test metrics, and Theorem 1 is a conditional statement with explicit assumptions rather than a disguised input.

full rationale

OPERA's central claims are empirical: SP and DP are compared against FT, InfoBatch, and random pruning on held-out test sets of eight datasets. The pruning scores are cosine similarities from the pretrained or evolving model, which is the intended self-paced mechanism, not a quantity defined in terms of the evaluation metric. Theorem 1 (Appendix B) is a conditional result: it states that if a selector's true-positive rate γ exceeds the base rate m+/mq, then the expected query embedding is closer to the true-positive mean. This is a mathematical lemma (proved via Lemma 1) and it is not used to set any hyperparameter or to compute any reported number; the reported NDCG/Recall numbers come from actual training runs. The paper explicitly states that DP hyperparameters were not tuned (Appendix G), and the FT baseline hyperparameters were optimized separately, so no fitted parameter is renamed as a prediction. The 'quality-coverage tradeoff' is an empirical observation supported by the SP results and Appendix D.2, not an equation that reduces to its inputs. The only concerns the skeptic raises — Theorem 1's objective is a sum of cosine similarities rather than the actual contrastive loss, and the guarantee is vacuous when m+=mq — are correctness/applicability limitations, not circularity: they do not make a reported result equal to an input by construction. There are no load-bearing self-citations: all cited methods (BGE, Qwen3, InfoBatch, CBS) are external prior work with no author overlap. Therefore the derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

8 free parameters · 3 axioms · 0 invented entities

The method introduces no new entities. It does rely on hand-set sampling-strength and ratio parameters (α, β, r, v), and on the assumption that similarity/loss scores are reliable quality signals. The theoretical analysis is conditional on the scoring function beating random, which is a domain assumption about the pretrained model's embeddings.

free parameters (8)
  • α_s (query strength start) = 2
    Initial sampling strength for query-level weighting; chosen by hand, not tuned.
  • α_e (query strength end) = 5
    Terminal sampling strength; fixed cosine schedule endpoint.
  • β (document strength) = 5
    Constant document-level sampling strength; chosen by hand.
  • r_s (query ratio start) = 0.25
    Initial fraction of high-quality queries retained; set a priori.
  • v_s (document ratio start) = 0.25
    Initial document cutoff threshold fraction.
  • v_e (document ratio end) = 0.5
    Terminal document cutoff threshold fraction.
  • I_u (query update interval)
    Not reported in main experiments; ablation shows 10 or 100. Affects performance/overhead tradeoff.
  • k (SP retention rate) = 0.25 (0.5 for FiQA)
    Fraction of top-similarity pairs retained in static pruning.
axioms (3)
  • domain assumption Scoring function (cosine similarity or contrastive loss) identifies true positives at a rate higher than the noise rate.
    Assumed in Theorem 1 (Appendix B) and implicitly in the SP/DP design; if false, pruning could remove useful hard examples.
  • domain assumption The two-stage sampling structure of dense retrieval finetuning is accurately described by the stated probabilistic model.
    Used in Section 2.1 to formalize FT and in Theorem 1's proof; may not capture all practical data distributions.
  • ad hoc to paper The DP hyperparameters (α, β, ratios, update interval) generalize across datasets without tuning.
    The paper fixes them for all datasets; robustness is empirically shown but not theoretically justified.

pith-pipeline@v1.3.0-alltime-deepseek · 2904 in / 3502 out tokens · 105349 ms · 2026-08-03T02:28:24.504934+00:00 · methodology

0 comments
read the original abstract

Domain-specific finetuning is essential for dense retrievers, yet not all data pairs contribute equally to the learning process. We introduce OPERA, a data pruning framework that exploits this heterogeneity to improve both the effectiveness and efficiency of retrieval model adaptation. We first investigate static pruning (SP), which retains only high-similarity query-document pairs, revealing an intrinsic quality-coverage tradeoff: ranking (NDCG) improves while retrieval (Recall) can degrade due to reduced query diversity. To resolve this tradeoff, we propose a two-stage dynamic pruning (DP) strategy that adaptively modulates sampling probabilities at both query and document levels throughout training, prioritizing high-quality examples while maintaining access to the full training set. Evaluations across eight datasets spanning six domains demonstrate the effectiveness of both approaches: SP improves ranking over standard finetuning (NDCG@10 +0.2 points), while DP achieves the strongest performance on both ranking (NDCG@10 +1.0 points) and retrieval (Recall@20 +0.4 points), with an average rank of 1.38 across all methods. These findings scale to Qwen3-Embedding, an LLM-based dense retriever, confirming architecture-agnostic benefits. Notably, DP reaches comparable performance in less than 50\% of the training time required by standard finetuning.

Figures

Figures reproduced from arXiv: 2603.17205 by Bernie Wang, Cuixiong Hu, George Karypis, Haoyang Fang, Hengyi Wang, Katrin Kirchhoff, Shuai Zhang, Yifei Ma.

Figure 1
Figure 1. Figure 1: Comparison of sampling probability distributions across training strategies. Left three panels: Standard finetuning (FT) samples all data pairs uniformly, while static pruning (SP) discards the lowest-similarity ones and up-weights the rest, improving ranking but reducing query coverage. Dynamic pruning (DP) at the start of training retains all pairs with non-zero probability, assigning higher sampling wei… view at source ↗
Figure 2
Figure 2. Figure 2: Training efficiency on ANTIQUE (unseen) and FEVER (seen). RP and SP use [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) shows the probability distribution grouped by query. Notably, nearly all queries maintain nonzero sampling probabilities throughout training, demonstrat￾ing that DP preserves broad query coverage unlike SP, which would exclude many queries entirely. The vari￾ation in intensity reflects quality-aware up-weighting while maintaining the diversity needed to preserve and even improve recall [PITH_FULL_IMAG… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of SP data retention rate k on NFCorpus. SP improves NDCG@10 over FT at all retention rates. Recall@20 shows a monotonic decrease as the retention rate reduces from 100% to 5%, which aligns with the reduced data visibility. These findings highlight SP’s data efficiency and indicate the potential for computational resource optimization while maintaining or enhancing ranking performance. E Detailed bg… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 23 linked inside Pith

  1. [1]

    Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540,

    Amro Abbas, Kushal Tirumala, D ´aniel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540,

    Pith/arXiv arXiv

  2. [9]

    Pruning then reweighting: Towards data- efficient training of diffusion models.arXiv preprint arXiv:2409.19128,

    Yize Li, Yihua Zhang, Sijia Liu, and Xue Lin. Pruning then reweighting: Towards data- efficient training of diffusion models.arXiv preprint arXiv:2409.19128,

    Pith/arXiv arXiv

  3. [10]

    Handschuh, A

    Macedo Maia, S. Handschuh, A. Freitas, Brian Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: Financial opinion mining and question answering. Companion Proceedings of the The Web Conference 2018,

    2018

  4. [11]

    When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    Max Marion, Ahmet ¨Ust ¨un, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    Pith/arXiv arXiv

  5. [12]

    Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316,

    Niklas Muennighoff, Nouamane Tazi, Lo¨ıc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316,

    Pith/arXiv arXiv

  6. [14]

    Competence-based curriculum learning for neural machine translation

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848,

    Pith/arXiv arXiv 1903

  7. [19]

    Nandan Thakur, Nils Reimers, and Jimmy Lin

    URLhttps://arxiv.org/abs/2104.08663. Nandan Thakur, Nils Reimers, and Jimmy Lin. Domain adaptation for memory-efficient dense retrieval.arXiv preprint arXiv:2205.11498,

    Pith/arXiv arXiv

  8. [20]

    FEVER: a large-scale dataset for fact extraction and VERification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809–819, New Orleans, Louisiana, June

    2018

  9. [21]

    doi: 10.18653/v1/N18-1074

    Association for Computational Linguistics. doi: 10.18653/v1/N18-1074. URL https://www.aclweb.org/anthology/N18-1074. Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving llm pretraining via document de-duplication and diversification.Advances in Neural Information Processing Systems, 36,

    doi:10.18653/v1/n18-1074

  10. [22]

    An empirical study of example forgetting during deep neural network learning.arXiv preprint arXiv:1812.05159,

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning.arXiv preprint arXiv:1812.05159,

    Pith/arXiv arXiv

  11. [23]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  12. [24]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533,

    Pith/arXiv arXiv

  13. [25]

    Dynamic data pruning for automatic speech recognition.arXiv preprint arXiv:2406.18373,

    Qiao Xiao, Pingchuan Ma, Adriana Fernandez-Lopez, Boqian Wu, Lu Yin, Stavros Petridis, Mykola Pechenizkiy, Maja Pantic, Decebal Constantin Mocanu, and Shiwei Liu. Dynamic data pruning for automatic speech recognition.arXiv preprint arXiv:2406.18373,

    Pith/arXiv arXiv

  14. [26]

    Dataset prun- ing: Reducing training data by examining generalization influence.arXiv preprint arXiv:2205.09329,

    Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset prun- ing: Reducing training data by examining generalization influence.arXiv preprint arXiv:2205.09329,

    Pith/arXiv arXiv

  15. [27]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium, October-November

    2018

  16. [28]

    doi: 10.18653/v1/D18-1259

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://www.aclweb.org/anthology/D18-1259. 12 Preprint. Under review. Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. InProceedings of the 44th International ACM SIGIR Conference on Research...

    doi:10.18653/v1/d18-1259

  17. [29]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Pith/arXiv arXiv

  18. [31]

    While models like NV-Embed- v1 (Lee et al.,

    to generate robust embeddings in various retrieval tasks (Muennighoff et al., 2022; Thakur et al., 2021). While models like NV-Embed- v1 (Lee et al.,

    2022

  19. [32]

    with 7B parameters have pushed performance boundaries, a more compact model family such as BGE (Xiao et al., 2023; Chen et al.,

    2023

  20. [33]

    offers an attractive balance between computational efficiency and effectiveness. Our work is based on the bge-large-en-v1.5 model (Xiao et al., 2023), for its favorable balance between computational efficiency and strong performance, as well as its open source training and evaluation process, which enables reproducibility and practical deployment. This ch...

    2023

  21. [34]

    have further advanced the state of the art by adapting large language models directly for embedding generation, achieving strong retrieval performance across diverse tasks, though at substantially higher computational cost. A.2 Data Pruning in Neural Networks Data pruning has emerged as a promising approach to improve training efficiency while maintaining...

    2022

  22. [35]

    Our work extends these ideas to the domain adaptation setting, where different considerations apply due to the distinct nature of the training data

    introduced the consistency-based filter for pretraining, which retains only high-quality text pairs based on their ranking against random documents. Our work extends these ideas to the domain adaptation setting, where different considerations apply due to the distinct nature of the training data. Dynamic PruningMore recent approaches have explored dynamic...

    2021

  23. [36]

    A.3 Curriculum Learning and Adaptive Training Curriculum learning has emerged as a promising approach to train neural networks by presenting training examples in a meaningful order

    which prunes on the query level. A.3 Curriculum Learning and Adaptive Training Curriculum learning has emerged as a promising approach to train neural networks by presenting training examples in a meaningful order. The idea of curriculum learning was formalized in (Bengio et al., 2009), where it showed that gradually increasing the difficulty of training ...

    2009

  24. [39]

    • Training set:Documents with relevance levels of 2, 3, and 4 were treated as positive samples

    defines relevance levels as follows: • Level 1:Completely out of context or does not make any sense (4.6% of the training data) • Level 2:Does not answer the question, or provides an unreasonable answer, but is not out of context (23.1% of the training data) • Level 3:Can be an answer to the question, but is not sufficiently convincing (29.5% of the train...

    2020

  25. [40]

    as a quality control mechanism for large-scale pretraining. The core insight is that high-quality training pairs should maintain relevance compared to random documents: the method ranks each positive pair against random negatives and retains only those that consistently rank highly. We adapt this to our finetuning context with three modifications: (1) we ...

    2016

  26. [41]

    The results, presented in Figure 4, demonstrate that SP consistently surpasses FT in terms of NDCG@10 across all retention rates

    and SP with k= 0.75, 0.5, 0.25, 0.1, 0.05. The results, presented in Figure 4, demonstrate that SP consistently surpasses FT in terms of NDCG@10 across all retention rates. The optimal performance was achieved atk= 0.25, achieving a peak NDCG of 0.491. Notably, even with minimal data retention (5%), SP main- tains superior performance with an NDCG of 0.48...

    2000

  27. [64]

    The temperature is set to 0.02, and the system leverages cross-device negatives during training

    Training uses FP16 precision with a maximum gradient norm of 1.0, no warmup, and no weight decay. The temperature is set to 0.02, and the system leverages cross-device negatives during training. The maximum query and passage length are set to 128 and 512 tokens, respectively. All embeddings are normalized. FiQA (Maia et al., 2018)’s data retention ratio k...

    2018

  28. [2002]

    Mistral 7b.arXiv preprint arXiv:2310.06825,

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825,

    Pith/arXiv arXiv

  29. [2003]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    Pith/arXiv arXiv 1908

  30. [2010]

    Other works have explored the use of teacher-student frameworks (Matiisen et al., 2019), where a teacher model determines the curriculum for a student model

    allows the model to automatically select its own curriculum based on the loss of training examples. Other works have explored the use of teacher-student frameworks (Matiisen et al., 2019), where a teacher model determines the curriculum for a student model. Related paradigms include active learning, which selects informative samples in discrete select- re...

    2019

  31. [2016]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216,

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216,

    Pith/arXiv arXiv

  32. [2017]

    Dense passage retrieval for open-domain question answer- ing.arXiv preprint arXiv:2004.04906,

    Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing.arXiv preprint arXiv:2004.04906,

    Pith/arXiv arXiv 2004

  33. [2018]

    How abilities in large lan- guage models are affected by supervised fine-tuning data composition.arXiv preprint arXiv:2310.05492,

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large lan- guage models are affected by supervised fine-tuning data composition.arXiv preprint arXiv:2310.05492,

    Pith/arXiv arXiv

  34. [2019]

    Infobatch: Lossless training speed up by unbiased dynamic data pruning.arXiv preprint arXiv:2303.04947,

    Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning.arXiv preprint arXiv:2303.04947,

    Pith/arXiv arXiv

  35. [2020]

    Large-scale dataset pruning with dynamic uncertainty

    Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7713–7722, 2024a. Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. InProceedings of the IEEE/CVF Conference on ...

    Pith/arXiv arXiv

  36. [2021]

    Kalervo J¨arvelin and Jaana Kek¨al¨ainen

    URLhttps://arxiv.org/abs/2112.09118. Kalervo J¨arvelin and Jaana Kek¨al¨ainen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446,

    Pith/arXiv arXiv

  37. [2022]

    Generative representational instruction tuning.arXiv preprint arXiv:2402.09906,

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning.arXiv preprint arXiv:2402.09906,

    Pith/arXiv arXiv

  38. [2023]

    Accelerating deep learning with dynamic data pruning.arXiv preprint arXiv:2111.12621,

    Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning.arXiv preprint arXiv:2111.12621,

    Pith/arXiv arXiv

  39. [2024]

    Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    Pith/arXiv arXiv

  40. [2025]

    Under review

    13 Preprint. Under review. Contents 1 Introduction 1 2 Methodology 3 2.1 Preliminary: Standard Finetuning (FT) . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Static Pruning (SP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3 Dynamic Pruning (DP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 Experiments 5 3.1...

    2018