{"id":"4757e393-5dc3-4db6-b69e-dc512fd412ed","arxiv_id":"2509.06524","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.","lead":"This paper introduces LAMDAS, a way to pick the most useful training examples for adapting a language model to a specific domain. It uses a small language model to score each candidate example by comparing its likelihood with and without a domain-specific prefix learned from a small reference set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection threshold τ is tuned on the same evaluation benchmarks used to report final results, so the headline gains may be inflated by selection bias; a held-out evaluation is needed.","rationale":"The reader's stated weakest assumption is that the learned domain prefix faithfully represents the target domain. That is a legitimate theoretical concern, and the paper offers only empirical illustrations (t-SNE, Table 6) rather than guarantees. However, the more immediately load-bearing issue for the paper's central empirical claim is the tuning of the selection threshold on the evaluation benchmarks themselves. The ablation in Section 4.4 / Figure 4 explicitly varies τ and reports downstream scores on the same benchmarks (HE, HE+, MBPP, MBPP+, GSM8K, MATH500, AIME, AMC) used in Tables 1, 2, and 7. The method's only real hyperparameter (besides prefix length, which is tuned on reference likelihood, not on benchmarks) is therefore chosen to maximize the exact metrics used to claim superiority. This is a form of test-set selection bias: the reported numbers are the maximum over the threshold grid, which inflates the apparent advantage over baselines that did not receive such tuning. The concern is concrete and falsifiable: re-run the comparison with τ fixed by principle at 1.0, or with τ selected on a disjoint validation split, and see whether the margins hold. The paper is otherwise well-structured: it compares against nine baselines, provides efficiency measurements, and includes ablation studies. The central method is intuitive and the computational cost is reasonable. The conditional verdict is appropriate; my concern reinforces it rather than changing it, hence UNCHANGED.","tokens_in":22121,"tokens_out":3720,"duration_ms":43822,"concrete_test":"Split the evaluation benchmarks into a tuning set and a held-out set: for example, tune τ using only HE, MBPP, GSM8K, and MATH500, then evaluate on HE+, MBPP+, AIME, and AMC. Alternatively, fix τ=1 by the stated design principle without any test-benchmark feedback and re-run the comparisons. If LAMDAS's margin over the best baseline (LESS/DsDm) persists on the untouched held-out set, the claim survives; if the margin shrinks or reverses, the headline gains are partly an artifact of test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that LAMDAS outperforms nine baselines and full-data training at a fixed selection ratio. The paper's own ablation (Section 4.4, Figure 4) tunes the selection threshold τ on the same benchmarks used to report final performance (MATH500, AIME, GSM8K, HumanEval, MBPP, etc.), then declares τ=1.0 optimal and uses it for all headline numbers. Because τ directly controls the selected subset (retention ratios ~50%, 40%, 30%, 10%, 5% for τ=0.8..1.2), selecting τ by maximizing performance on the test benchmarks means the reported results are the best of several runs, not an unbiased estimate. No baseline is given the same test-set tuning. This selection bias can inflate the reported improvements; the true advantage of LAMDAS over, e.g., LESS or DsDm at a fairly chosen threshold is unknown. The 'Full' baseline is additionally confounded by catastrophic forgetting (the paper admits Random also beats Full), but the benchmark-tuning issue is more load-bearing because it affects all comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LAMDAS, a data-selection method for domain-specific continual pretraining (CPT) and supervised fine-tuning (SFT). A soft prefix is tuned on a small reference dataset (Eq. 1); each candidate sample y is scored by the likelihood ratio s(y) = p(y|C)/p(y) under a pretrained LLM with and without this prefix, and samples above threshold τ are selected. The paper reports that LAMDAS outperforms nine baselines and full-data training across code and math benchmarks, with average relative gains up to 24.7% over the Full baseline for CPT on Qwen2.5-7B and 19.6% for SFT code on Qwen2.5-7B-Instruct, while being substantially faster than direct-optimization baselines. Experiments cover model sizes 0.5B, 1.5B, 7B, and 32B, with ablations on τ, prefix length, classifier size, and classifier type.","tokens_in":22391,"tokens_out":4936,"duration_ms":60193,"significance":"If the empirical claims hold, LAMDAS is a simple, efficient, and reasonably general recipe for domain-specific data selection: it needs only a small reference set and uses the pretrained LLM itself as a scorer, avoiding gradient computation and extra classifier training. The paper includes a broad set of comparisons, a concrete efficiency analysis, and ablations over several design choices. Its strengths are the simplicity of the method and the breadth of the evaluation across CPT/SFT, code/math, and four model sizes. However, the theoretical framing (Neyman-Pearson optimality, mutual-information maximization) is asserted rather than established, and the empirical evaluation is weakened by the use of the test benchmarks to select the threshold τ. With a clean held-out evaluation and a more measured theoretical presentation, this would be a useful contribution to the data-selection literature.","major_comments":[{"comment":"The threshold τ is tuned on the same evaluation benchmarks that are later used to report the headline numbers. Fig. 4 varies τ ∈ {0.8, 0.9, 1.0, 1.1, 1.2} over MATH500, AIME, GSM8K, HE, HE+, MBPP, and MBPP+ for all four model sizes and then selects τ = 1.0 as 'optimal'. Tables 2 and 7 then report results with this threshold as LAMDAS's performance. Because τ directly controls retention ratios (≈50%, 40%, 30%, 10%, 5%), the reported numbers are the best among several runs, not an unbiased estimate, and no baseline receives a comparable test-set tuning. This can inflate the claimed improvements. The authors should choose τ on a held-out validation split, or report results for all thresholds for all methods; the main comparisons should be re-run accordingly.","section":"§4.4 / Fig. 4"},{"comment":"The Neyman-Pearson claim is not supported. Optimal likelihood-ratio testing requires two simple hypotheses with known class-conditional densities. Here p(y|C) and p(y) are both text probabilities produced by the same LLM, with and without a learned prefix; they are not the conditional densities of 'in-domain' vs 'out-of-domain' candidate data. The proposition is therefore an analogy or heuristic, not a theorem. Additionally, Eq. (2) treats p(C) as a prior over the domain, but C is a point-estimated prefix parameter vector, so p(C) is not a well-defined prior. The statement in §3.1.1 that Eq. (1) 'maximizes the mutual information between the reference data and C' is also unproven: maximizing log p(x_{1:N}|C) over C does not by itself maximize MI between the text and a parameter vector. Please either add the missing assumptions/derivation or explicitly present these as motivations rather t","section":"§3.1.2, Proposition 3.1"},{"comment":"The efficiency comparison omits the cost of learning the domain prefix. Table 3 lists LAMDAS's time complexity as 2×C_forward, but Appendix A.5 says prefix tuning runs for 10 epochs over the reference set, consuming about one hour on a single A100 GPU. This is a fixed training overhead before any candidate is scored. For a fair complexity accounting, this prefix-training cost should be included in the comparison, or justified as negligible with an amortized calculation. Without this, Figure 1's 'most favorable trade-off' claim is overstated relative to methods that do not train on the reference set.","section":"Table 3 / Appendix A.5"},{"comment":"The 'Full' baseline is not a meaningful upper bound in the CPT setting: the paper itself reports that Random beats Full (e.g., 29.6 vs 28.7 average for Qwen2.5-7B) and attributes this to catastrophic forgetting. Since the paper's headline is 'LAMDAS outperforms full-data training,' the comparison with Full is confounded by training budget. The paper should consistently frame the result as 'LAMDAS outperforms Random and other selection methods at the same budget,' and report a matched-budget control (same token count, random selection) as the primary baseline, rather than emphasizing the Full comparison.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"All results are single runs without standard errors or multiple seeds. Given the relatively small benchmark sizes (e.g., AIME has 30 problems), percentage differences of a few points can be within noise. Reporting seeds, repeats, or confidence intervals would substantially strengthen the empirical claims.","section":"Tables 1, 2, 7"},{"comment":"The abstract says 'nine SOTA baselines,' while the introduction says seven baselines for CPT and nine for SFT. Make the count consistent and specify which baselines are used where.","section":"Abstract / §1"},{"comment":"Typo: 'Qwen2.5 serious models' should be 'Qwen2.5 series models.' Similar typos appear in Table 7 and elsewhere.","section":"Table 2"},{"comment":"Units are inconsistent: Table 3 reports 'Run Time s/100' and Table 8 reports 'Speed s/k'. Use a single convention, e.g., seconds per 100 samples, and clarify whether the times include model loading.","section":"Table 3 / Table 8"},{"comment":"Engstrom et al. are cited as both 2024a and 2024b for what appears to be the same DsDm paper. Please check and merge duplicate entries.","section":"References"},{"comment":"The footnote says 'Code will be available upon publication.' For a method whose main value is practical data selection, code release is important; please make the code and configuration public, or at least provide detailed pseudocode and hyperparameter settings.","section":"§1 / Footnote a"},{"comment":"The caption sentence is duplicated ('suggests a more favorable trade-off' appears twice). Also, the regression line has no error or stability analysis, so the perpendicular-distance ranking should be treated as illustrative.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is attractive and the empirical scope is broad, but the threshold-tuning protocol on the test benchmarks is a serious correctable flaw. The theoretical section is more decorative than substantive; the paper would be stronger if it explicitly presented the likelihood ratio as a heuristic motivated by OCC rather than as a Neyman-Pearson-optimal test. I do not see a fundamental novelty issue: the combination of prefix tuning and likelihood-ratio scoring is simple enough to be useful, but the comparisons need to be re-run with honest validation. I would also encourage the editor to request code release as a condition of acceptance, given the paper's practical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LAMDAS is a genuinely new and elegant data-selection trick: learn a soft prefix on the reference set, then score candidates by the likelihood ratio p(y|C)/p(y) and keep everything above tau=1. The idea is simple, well-motivated, and the empirical sweep is broad—multiple Qwen2.5 sizes, CPT and SFT, code and math, nine baselines. The method is efficient, and the paper's framing as one-class classification is fair. That part deserves credit.\n\nThe soft spots are real but not fatal. Proposition 3.1 just invokes the Neyman-Pearson lemma for plug-in likelihoods from an LLM; those likelihoods are not true class-conditional densities, so the 'optimality' is rhetorical. The mutual-information claim in Section 3.1.1 is asserted, not proved. More load-bearing: the threshold tau=1 is chosen from Figure 4, an ablation run on the same benchmarks used to report the headline numbers (MATH500, AIME, GSM8K, HE, MBPP). With retention ratios from 50% down to 5%, that is a real search over subsets, and none of the baselines get the same test-set tuning. The reported margins over baselines are likely inflated by selection bias; we don't know how LAMDAS would fare with tau chosen on a held-out set. The CPT 'Full' baseline is also admitted to suffer catastrophic forgetting—Random beats Full—so the 'beats full data' headline there is not a fair comparison.\n\nThe core score is not circular; it is a fixed pretrained LLM's likelihood ratio, not a quantity fit to the evaluation metric. That is honest design. What's missing is reproducibility: no code, no error bars, no multiple seeds. The related work is careful and the citations check out; the efficiency results are genuinely useful.\n\nWho this is for: anyone working on data selection for domain adaptation. The idea is worth knowing and likely to be built on. It deserves a serious referee. I would send it to review, with the requirement that the authors validate tau on a genuinely held-out split and release code. If that holds up, this is a solid contribution.","headline":"Clever and efficient likelihood-ratio data selection via a learned domain prefix, but threshold tuning on the test benchmarks inflates the reported gains.","tokens_in":22868,"tokens_out":4564,"would_cite":true,"duration_ms":39201,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LAMDAS claims that a likelihood-ratio score from a small domain prefix selects data that outperforms full-data training and nine baselines, with only two forward passes per candidate.","keywords":["data selection","likelihood ratio","prefix tuning","one-class classification","domain adaptation","continual pre-training","supervised fine-tuning","code and math reasoning"],"falsifier":"Build a reference set from one narrow slice of a domain, such as only Python function-definition problems, and a candidate pool spanning many code categories; then train on the LAMDAS-selected subset and evaluate on held-out tasks from the other categories. If the selected-subset model scores below a random subset of equal size on those held-out tasks, the likelihood-ratio score fails to capture the domain as a whole.","tokens_in":22034,"feed_emoji":"🎯","tokens_out":4703,"duration_ms":52044,"temperature":0.7,"pith_summary":"The paper claims that the problem of selecting training data for a specific domain can be solved by asking the language model itself to act as a one-class classifier: learn a soft prefix that maximizes the likelihood of a small reference set, then score each candidate by the likelihood ratio p(y|C)/p(y). Candidates whose likelihood rises when the prefix is present—score above 1—are kept. The paper reports that models continually pre-trained or fine-tuned on LAMDAS-selected subsets outperform models trained on the full data, and also outperform nine existing selection baselines, while requiring only two model forward passes per candidate. A sympathetic reader would care because it promises accurate domain adaptation with a fraction of the data and modest compute, without hand-crafted features or per-candidate optimization.","feed_headline":"LLM likelihood ratio selects data that beats full-data training","feed_subtitle":"Learn a prefix on a small reference set, score candidates by p(y|C)/p(y), and keep those above 1 to beat full-data training.","key_machinery":"The domain prefix—a learnable soft prefix of fixed length (about 30 tokens) tuned only on the reference data—is the object that carries the argument. It condenses the reference set into a representation that interacts with each candidate through the LLM's cross-attention, and the likelihood ratio s(y)=p(y|C)/p(y) against the prefix-free likelihood turns the LLM into a one-class classifier. The score's optimality rests on the Neyman-Pearson lemma, which is what justifies the threshold of 1.","core_discovery":"LAMDAS's central claim is that a likelihood-ratio score computed from a pretrained LLM, without updating the LLM's weights, is a sufficient and efficient relevance signal for domain-specific data selection. Formally, the paper learns a domain prefix C by maximizing log p(x|C) over a small reference set, then scores a candidate y as s(y)=p(y|C)/p(y) and keeps it when s(y)>1. By the Neyman-Pearson lemma, this score is the optimal discriminant between the reference-conditioned and unconditional distributions. Experiments on Qwen2.5 models show, for example, a 24.7% average improvement over full-data continual pre-training for the 7B model using 15% of the data, and a 19.6% average improvement o","pith_inferences":["The likelihood-ratio framing suggests a direct test of the domain-representation assumption: if the reference set is a mixture of two coherent subdomains, the learned prefix should behave like a mixture model, and the score should rank pooled candidates by their subdomain membership probabilities.","When the reference set is noisy, skewed, or multi-modal, the prefix may collapse onto one mode, causing LAMDAS to systematically filter out samples from other modes; the paper does not explore this failure regime.","The same implicit-classifier idea could be extended to multiple reference domains, producing a soft routing score rather than a binary keep/drop decision and turning data selection into multi-way classification.","The reported gains over full-data training may partly reflect the well-known risk of catastrophic forgetting when continually pretraining on pure code; a longer-horizon comparison measuring retention of general NLP ability would clarify how much of the gain is selection rather than forgetting avoidance."],"forward_implications":["Using roughly 15% of candidate tokens for CPT and 30–40% for SFT, models trained on LAMDAS-selected data outperform full-data training on code benchmarks, with gains persisting across model sizes.","Selection requires only two forward passes per candidate, so switching domains means retraining a short prefix rather than retraining the LLM.","Because the LLM's weights are untouched, small LLMs (0.5B parameters) can serve as the implicit classifier, further reducing selection cost.","The same likelihood-ratio score can be combined with quality, diversity, or novelty filters in a pipeline, since it targets domain relevance rather than data quality alone.","The Neyman-Pearson justification gives the score a principled decision threshold, making the method nearly hyperparameter-free beyond prefix length and model choice."],"supporting_citations":[{"why":"Supplies the Neyman-Pearson lemma used to prove that the likelihood-ratio score is an optimal discriminant.","marker":"(Casella & Berger, 2024)"},{"why":"Defines the DSIR n-gram importance-resampling baseline that LAMDAS must outperform.","marker":"(Xie et al., 2023)"},{"why":"Defines the TSDS optimal-transport embedding baseline, the strongest similarity-based comparison.","marker":"(Liu et al., 2024c)"},{"why":"Defines the DsDm direct-optimization baseline whose simplified proxy model motivates LAMDAS's condensed-prefix alternative.","marker":"(Engstrom et al., 2024b)"},{"why":"Defines the DEITA quality-and-complexity baseline that LAMDAS outperforms on domain-relevance benchmarks.","marker":"(Liu et al., 2024a)"},{"why":"Defines the LESS gradient-based influence baseline, a key SFT comparison that is computationally heavier.","marker":"(Xia et al., 2024)"},{"why":"Defines the Nuggets pairwise likelihood-comparison method whose per-pair cost motivates the compact domain prefix.","marker":"(Li et al., 2023)"},{"why":"Supplies the LIMO math reference data that defines the target domain in the SFT math experiments.","marker":"(Ye et al., 2025)"},{"why":"Supplies the O1-journey math reference data complementing LIMO in the SFT math experiments.","marker":"(Qin et al., 2024)"}],"fun_headline_variants":["Implicit LLM classifier picks data that beats full training","LLM likelihood ratio: optimal data selection without weight updates","One-class LLM data selection beats full-data training","Score by LLM likelihood ratio, beat full-data training"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The learned domain prefix faithfully represents the target domain, so the likelihood ratio p(y|C)/p(y) is a reliable relevance score; if the small reference set is noisy, skewed, or unrepresentative, the prefix cannot separate useful candidates from useless ones.","fun_headline_variants_meta":{"raw":{"variants":["Implicit LLM classifier picks data that beats full training","LLM likelihood ratio: optimal data selection without weight updates","One-class LLM data selection beats full-data training","Score by LLM likelihood ratio, beat full-data training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":3894,"prompt_tokens":780,"completion_tokens":3114,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":3060}},"tokens_in":524,"tokens_out":3114,"duration_ms":24126,"temperature":1.0,"reasoning_tokens":3060,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:29:04.175756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a reference set from one narrow slice of a domain, such as only Python function-definition problems, and a candidate pool spanning many code categories; then train on the LAMDAS-selected subset and evaluate on held-out tasks from the other categories. If the selected-subset model scores below a random subset of equal size on those held-out tasks, the likelihood-ratio score fails to capture the domain as a whole.","supporting_citations":[{"cited_title":"Data selection for language models via importance resampling","cited_arxiv_id":null,"evidence_quote":"Defines the DSIR n-gram importance-resampling baseline that LAMDAS must outperform."}],"review_version":1}