{"id":"bbd0ebd7-b49a-4cdc-ad0e-31d33fdb5224","arxiv_id":"2505.12250","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EQUAL iteratively selects document clusters via multi-armed bandit and optimal transport scores to extract only the instruction-tuning QA pairs that matter, saving 5-10x cost with small accuracy gains.","lead":"This paper introduces EQUAL, a system that picks which web documents are worth turning into question-answer training data for language models, instead of processing everything. It reports cutting the computing cost of data extraction by 5 to 10 times while slightly improving accuracy on math and code benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Math gains risk being inflated by AutoMathText contamination: it is the only corpus not n-gram filtered (Section 4.1), while the OT reward targets GSM8K-train embeddings; an overlap check is needed before the 2.5% accuracy claim is accepted.","rationale":"The reader's first weakest assumption is whether OT embedding distance tracks fine-tuning utility; the reader's second premise already flags AutoMathText contamination. I make contamination the primary attack because it is more decision-relevant: the paper's own no-OT ablation provides some direct evidence that the OT reward helps, whereas contamination would invalidate both the outcome numbers and the mechanism evidence at once. The manuscript's explicit asymmetry in filtering—StackOverflow filtered, AutoMathText not—is an observable, concrete confound rather than a speculative concern. The MAB framing is coherent, code and data are promised, and the cost-saving framing is plausible, so I do not recommend rejection. I would keep the reader's CONDITIONAL verdict: the math-domain accuracy claims should be accepted only after the contamination check is run and reported. If the check comes back clean, this objection is resolved; if not, the reported GSM8K/MATH gains should not be cited as evidence for EQUAL's accuracy advantage.","tokens_in":21835,"tokens_out":7896,"duration_ms":83449,"concrete_test":"Compute exact and 13-gram overlap (following Guo et al., 2024) between the QA pairs extracted by EQUAL from AutoMathText and the GSM8K train/test and MATH train/test sets; report the number and fraction of selected pairs that match benchmark items or near-duplicate documents. Then remove all such documents from the candidate pool Dc, rerun EQUAL (and the no-OT ablation from Table 3) on the cleaned corpus with the same hyperparameters, and check whether the GSM8K/MATH accuracies and the OT-versus-no-OT gap persist. If they shrink substantially, the accuracy and OT-reward portions of the central claim are not established for the math domain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 explicitly n-gram filters StackOverflow \"to ensure that our training data is not contaminated by information from the downstream tasks,\" but applies no analogous filter to AutoMathText, even though GSM8K-train is the reference distribution Dr used in the OT reward (Equation 4) and GSM8K/MATH test sets are the reported math benchmarks. AutoMathText is a broad crawler-derived math corpus, so it can contain benchmark questions or near-duplicates. Because EQUAL's cluster arm selection maximizes OT proximity to Dr, contaminated documents are exactly the ones the MAB tends to pull; the 5%-extracted pool can therefore contain near-duplicates of evaluation questions. The reported gains—for example EQUAL Full at 73.01 versus Random Full at 67.40 on GSM8K, and the OT-versus-no-OT gap in Table 3—could then reflect memorization of benchmark items rather than the claimed property that embedding-space OT tracks instruction-data utility. This is a correctness risk and an unresolved confound for the math half of the central claim. It can be resolved only by overlap analysis, not by appealing to the method's internal consistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the problem of constructing instruction-tuning QA pairs from large document corpora. EQUAL first warms up an embedding model with contrastive learning on sampled documents and their extracted QA pairs, clusters the candidate corpus, and then iteratively selects clusters with a multi-armed bandit whose reward is the optimal-transport distance between the embeddings of extracted QA pairs and a reference set Dr (GSM8K-train or MBPP-train). Extracted pairs are pooled for fine-tuning. Experiments on AutoMathText and StackOverflow with LLaMA-3-8B and Mistral-7B, in full and LoRA settings, compare against random selection, extraction-from-all, rewriting, perplexity- and influence-based selection, and MAB variants, reporting accuracy on GSM8K/MATH and HumanEval/MBPP plus FLOPs. The paper claims a 5-10x cost reduction and about 2.5% accuracy improvement.","tokens_in":22014,"tokens_out":6717,"duration_ms":64783,"significance":"If the claims hold, EQUAL would be a practical contribution: interleaving document selection and QA extraction is a sensible way to avoid the cost of extracting all pairs, and the contrastive warm-up addresses a real distribution mismatch between documents and QA pairs. Strengths include a clear problem formulation, a complete algorithmic description, ablations for each component (CL, MAB, OT), experiments across two domains and four benchmarks, and a release of code and data. However, the central empirical claims are currently under-supported by the measurement and contamination issues detailed below; the claimed gains are plausible but not yet established.","major_comments":[{"comment":"Section 4.1 states that StackOverflow is n-gram filtered 'to ensure that our training data is not contaminated by information from the downstream tasks,' but no analogous statement is made for AutoMathText. Because Dr for math is GSM8K-train and the MAB reward is OT distance to Dr (Eq. (4)), any near-duplicate of GSM8K questions in AutoMathText would be precisely the content EQUAL tends to select, so the GSM8K gains in Table 1 and the OT ablations in Table 3 could reflect memorization rather than a learned property of useful instruction data. Please provide overlap analysis (n-gram and/or embedding-level) between AutoMathText documents / extracted pairs and GSM8K train+test and MATH, and rerun the math experiments after removing flagged near-duplicates.","section":"Section 4.1, Eq. (4)"},{"comment":"The FLOPs measurement in Appendix B uses torch.profiler on a dummy nn.Linear(1024, 512) with random input, which does not measure the FLOPs of the actual pipeline (Qwen2.5-72B extraction calls, embedding inference, clustering, MAB scoring, and fine-tuning). Since the headline '5-10x reduction' is computed from these numbers, this is load-bearing. Please report the actual profiled operations or state the estimation protocol explicitly; also report FLOPs for the All(Mammoth) baseline against which the 5-10x factor is claimed.","section":"Appendix B, Table 1"},{"comment":"Algorithm 1's stopping condition is |De| < gamma |Dc|, where |De| counts extracted QA pairs and |Dc| counts documents. Section 4.2 alternately says 'we acquire 5% documents' and 'extracting just 5% of QA pairs.' If each document yields multiple QA pairs, the two interpretations differ by a large factor, and the comparison with a 'Random(5%)' baseline that samples 5% of documents is not apples-to-apples. Please clarify the stopping rule and report both the fraction of documents processed and the number of QA pairs extracted.","section":"Algorithm 1, Section 4.2"},{"comment":"Table 1 reports averages over three runs without error bars, while Table 2 lists Random(5%) Full GSM8K as 67.40 and Table 1 lists Random FULL GSM8K as 68.92 for what appears to be the same setting. The claimed ~2.5% accuracy improvement is of the same magnitude as typical run-to-run variance; please provide standard deviations or confidence intervals and reconcile the two tables.","section":"Table 1 and Table 2"},{"comment":"The OT reward is optimized against Dr, which is the training split of the evaluation benchmark (GSM8K-train for math, MBPP-train for code), while the reported accuracies are on GSM8K/MATH and HumanEval/MBPP test splits. This makes GSM8K-test performance partly a measure of alignment with the selection target rather than of general instruction-data utility. Please add a transfer experiment with a reference set disjoint from all evaluation benchmarks, or evaluate on an additional benchmark not used in Dr, to support the general claim that OT-to-Dr predicts downstream accuracy.","section":"Section 3.3, Section 4.1"}],"minor_comments":[{"comment":"The model name is inconsistent: the abstract and Section 4.1 refer to LLaMA-3.1-8B while Table 1 and parts of Section 4.1 say LLAMA-3-8B; please standardize the name.","section":"Abstract, Section 4.1"},{"comment":"Figure 3 would benefit from axis labels and a legend; as printed, it is hard to verify the claimed cluster alignment from the two panels.","section":"Figure 3"},{"comment":"Table 2 compares EQUAL and Random at multiple ratios but omits the FLOPs column, making it impossible to see the cost of the All(Mammoth) baseline in the same table; please include FLOPs for the All baseline in Table 1 as well.","section":"Table 2"},{"comment":"Section 4.1 says the number of clusters is 'automatically determined by the Elbow method' and later reports k=1000 as the selected value; please state whether k=1000 is the outcome of the method for both datasets or a manually fixed value.","section":"Section 4.1, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the core idea is timely, but the math-domain contamination risk and the unverified FLOPs measurement are serious enough that the main quantitative claims should not be accepted as-is. The code/data release is a strong point and should make the requested overlap and measurement clarifications feasible. I also recommend asking the authors to reconcile the Random FULL numbers between Tables 1 and 2 before the revision is reviewed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EQUAL is worth a serious look: it interleaves cluster-level MAB selection with contrastive embedding alignment and an optimal-transport reward, and the integration is more than a label. The warm-up alignment is a sensible response to a real problem—documents and extracted QA pairs sit in different feature spaces—and the ablations (no-CL, no-MAB, no-OT) each pull in the expected direction. The StackOverflow half of the evaluation is also reasonably clean, since that corpus gets n-gram filtered against the downstream tasks.\n\nThe soft spots are real, and they cluster around the math side. AutoMathText is the one corpus that did not get n-gram filtered, and the OT reward is computed against the GSM8K training split. That is exactly the configuration where the bandit will favor clusters whose QA-pair embeddings look like GSM8K train, so if AutoMathText contains benchmark material or near-duplicates, the reported math gains could be inflated by memorization. The authors need an overlap analysis at the document and n-gram level before the GSM8K/MATH numbers can be trusted. This is not a nitpick; Section 4.1 makes the asymmetry explicit.\n\nTwo smaller issues. The FLOPs appendix shows a PyTorch profiler snippet but not the actual cost accounting, so the 5-10x claim is not independently auditable as written. And Table 1 reports three-run averages without variance; the headline gains are small enough that error bars matter. It is also worth noting that on MATH, EQUAL at 5% (35.10 Full) is below All/Mammoth (40.02 Full) in Table 2, so the accuracy advantage over extract-all is not uniform across benchmarks. The abstract's \"2.5% improvement\" is really against Influence, not against all.\n\nOverall, the framework is plausible, the engineering is careful, and the paper is clearly a contribution to the data-curation subfield. But the math claims currently have a load-bearing confound. I would send it to peer review—this is exactly the kind of empirical paper a good referee can fix—and make acceptance conditional on a contamination check, error bars, and a real FLOPs breakdown. I would not cite the math numbers until that check is done.","headline":"EQUAL is a credible framework for cutting the cost of turning web documents into instruction-tuning data, but its headline math gains are not yet trustworthy because AutoMathText was not contamination-filtered against GSM8K even though the selection reward targets GSM8K-train.","tokens_in":22642,"tokens_out":3774,"would_cite":false,"duration_ms":37321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EQUAL builds instruction-tuning data from only the useful documents, cutting compute 5-10x while gaining about 2.5% accuracy.","keywords":["instruction tuning","data extraction","document selection","multi-armed bandit","optimal transport","contrastive learning","question-answer pairs","web corpus"],"falsifier":"Take the cluster that EQUAL scores highest and the cluster it scores lowest on the same corpus, fine-tune the base model separately on QA pairs extracted from each with equal pair counts and equal extraction budgets, and compare accuracy on GSM8K/MATH or HumanEval/MBPP; if the low-scoring cluster performs as well as the high-scoring one, the optimal-transport reward is not carrying the claimed signal.","tokens_in":21588,"feed_emoji":"🎯","tokens_out":9875,"duration_ms":88907,"temperature":0.7,"pith_summary":"The paper claims that instruction-tuning data can be extracted from large web corpora without paying the cost of reading every document. Its method, EQUAL, clusters documents, samples a small fraction of documents from a chosen cluster, extracts question-answer (QA) pairs from those documents with an LLM, and repeats, using a multi-armed bandit to balance exploiting clusters that already look useful against exploring clusters that have not been sampled. Across math and code corpora with more than a million documents, this cuts computation by 5-10x while improving accuracy by about 2.5% relative to the selection baselines it is compared with. If the claim holds, open web corpora become a practical source of diverse instruction-tuning data rather than a compute sink.","feed_headline":"Extract QA pairs from 5% of documents, not all — EQUAL cuts cost 5-10x","feed_subtitle":"The bandit-chosen clusters also lift accuracy by about 2.5 percent on math and code benchmarks.","key_machinery":"The machinery is an iterative loop built on three pieces. First, a warm-up stage uses supervised contrastive learning to fine-tune an embedding model so that documents and the QA pairs extracted from them land close together in the same space. Second, the candidate corpus is clustered in that aligned space. Third, each cluster is an arm in a multi-armed bandit: a document-selection score $\\mathrm{DS}_j = \\widehat{\\mathrm{OT}}_j + \\alpha \\sqrt{2 \\ln T_{\\mathrm{total}} / T(C_j)}$ combines the estimated optimal transport score $\\widehat{\\mathrm{OT}}_j$ between the cluster's sampled QA pairs and the reference set with an exploration bonus for under-sampled clusters. The loop extracts QA pairs only from the selected cluster, then updates that cluster's $\\widehat{\\mathrm{OT}}_j$, so the reward estimate sharpens as more pairs are seen.","core_discovery":"The paper's central claim is that the document is the wrong unit of selection for instruction-tuning data, and so is the individual QA pair; the right unit is a cluster of documents whose extractable QA pairs resemble the distribution of a reference set such as the GSM8K or MBPP training split. EQUAL operationalizes this by aligning document and QA-pair embeddings with a contrastive-learning warm-up, clustering the candidate corpus, and treating each cluster as an arm of a multi-armed bandit whose reward is the optimal transport distance between sampled QA pairs and the reference distribution. Each round selects a cluster, extracts a batch of QA pairs, and updates that cluster's reward estimate, so document selection and data extraction reinforce each other. The reported experiments on AutoMathText and StackOverflow, with over a million documents each, show that using 5% of documents through this loop beats exhaustive all-document extraction on three of four math and code benchmarks and beats the evaluated selection baselines on all of them, at roughly 5-10x lower FLOPs; at 20% of documents it matches or exceeds exhaustive extraction on all four.","pith_inferences":["Inference: because the machinery needs only an embedding space and a reference set, the same interleaving should apply to expensive non-LLM extractors, such as multimodal captioning pipelines, where the cost per extracted item is even higher.","The paper does not test how the optimal transport reward behaves with very small reference sets; a natural extension is to replace the GSM8K and MBPP training splits with a few dozen seed questions and measure whether selection accuracy degrades.","Inference: the optimal transport estimate accumulated over rounds could itself serve as a training label for a cheap classifier, potentially removing the need for sampled extraction during selection in later rounds.","Inference: because exploration is explicit, EQUAL may be more robust in streaming settings where new document clusters arrive over time and the bandit must keep sampling unknown regions."],"forward_implications":["A practitioner can skip exhaustive LLM extraction over an entire web corpus; iteratively sampling a few percent of documents yields comparable or better downstream accuracy on math and code benchmarks.","Corpora too large for full extraction, such as general web crawls, become feasible instruction-tuning sources instead of being filtered out by budget.","Data selection no longer requires computing influence functions or perplexity over every extracted pair; the cost moves to a cheap embedding-space reward computed on sampled clusters.","Because the reward is distributional, the method transfers to a new downstream task whenever a small reference set of questions exists, with experiments covering both mathematical and code domains."],"supporting_citations":[{"why":"Provides the Mammoth baseline that extracts QA pairs from every retrieved document, the cost and quality comparison point.","marker":"Yue et al., 2024"},{"why":"Supplies the multi-armed bandit formulation that EQUAL maps cluster selection onto.","marker":"Vermorel & Mohri, 2005"},{"why":"Defines optimal transport, which EQUAL uses as the cluster reward to compare extracted QA distributions with the reference distribution.","marker":"Villani et al., 2009"},{"why":"Provides the supervised contrastive loss used in the warm-up to align document and QA-pair embeddings before clustering.","marker":"Khosla et al., 2020"},{"why":"Provides the GSM8K reference set and math evaluation task.","marker":"Cobbe et al., 2021a"},{"why":"Provides the MBPP reference set and code evaluation task.","marker":"Austin et al., 2021"},{"why":"Provides the AutoMathText candidate corpus for math-domain extraction.","marker":"Zhang et al., 2024"},{"why":"Provides the influence-function selection baseline and frames the irrelevant-data problem EQUAL addresses.","marker":"Xia et al., 2024"},{"why":"Provides the perplexity-based selection baseline used as a comparison.","marker":"Li et al., 2024b"}],"fun_headline_variants":["EQUAL picks 5% of clusters to slash QA extraction cost 5-10x","Bandit-chosen doc clusters beat full-corpus extraction at 5% data","Not all docs: EQUAL's cluster bandit cuts cost 5-10x, lifts accuracy","Skip 95% of web docs: EQUAL finds the QA gold with a bandit","Cluster-armed bandit extracts instruction QA from 5% of documents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop assumes that the optimal transport distance between embeddings of extracted QA pairs and the reference training set (GSM8K or MBPP) reliably predicts how much those pairs improve the target benchmark, and it assumes that the math corpus is not already contaminated with near-duplicates of the reference questions, since AutoMathText is not n-gram filtered while StackOverflow is.","fun_headline_variants_meta":{"raw":{"variants":["EQUAL picks 5% of clusters to slash QA extraction cost 5-10x","Bandit-chosen doc clusters beat full-corpus extraction at 5% data","Not all docs: EQUAL's cluster bandit cuts cost 5-10x, lifts accuracy","Skip 95% of web docs: EQUAL finds the QA gold with a bandit","Cluster-armed bandit extracts instruction QA from 5% of documents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1582,"prompt_tokens":1044,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":660,"tokens_out":538,"duration_ms":5389,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:45.712690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the cluster that EQUAL scores highest and the cluster it scores lowest on the same corpus, fine-tune the base model separately on QA pairs extracted from each with equal pair counts and equal extraction budgets, and compare accuracy on GSM8K/MATH or HumanEval/MBPP; if the low-scoring cluster performs as well as the high-scoring one, the optimal-transport reward is not carrying the claimed signal.","supporting_citations":[],"review_version":1}