{"id":"5b0d7587-ffa7-4f71-a6ba-3d802391e93e","arxiv_id":"2411.10545","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Alignment performance plateaus quickly with data, and a GMM-plus-entropy sampling method (ISA) can allegedly match full-dataset alignment with under 10% of the data, though the supporting evidence is weak.","lead":"The authors propose a data-sampling method, ISA, that selects a small high-quality subset of human preference data for LLM alignment, claiming it matches full-data alignment using less than 10% of the data. They also report that alignment performance follows an exponential plateau as data grows, suggesting much alignment data is wasted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim ('ISA performs comparable to full-data alignment') is contradicted by the paper's own Tables 1–2: on OpenAssistant and Ultrafeedback the full-data model leads by 2.9 and 1.6 points with non-overlapping 95% CIs.","rationale":"The reader's weakest_assumption focused on GPT-4o judge validity, which is a legitimate concern. But the more immediate and load-bearing problem is internal: the paper's own Tables 1 and 2 show that on two of three datasets the full-data model beats ISA by margins larger than the reported 95% confidence intervals. This contradiction does not depend on any outside assumption about judge quality; it is visible in the reported numbers themselves. The reader's rationale did mention this contradiction, so there is partial agreement, but the weakest_assumption field did not identify it as the primary issue. The paper does report three-seed means and confidence intervals, which is a useful level of transparency, and the idea of data-efficient alignment is worth exploring. However, the central empirical claim is not supported by the paper's own evidence, and the appendix's negative results on five other models are attributed to data leakage without supporting analysis. The recommended verdict stays REJECT, so no change from the reader's verdict is needed.","tokens_in":15098,"tokens_out":2940,"duration_ms":30861,"concrete_test":"Re-run the KTO alignment for OpenAssistant and Ultrafeedback using the ISA 10% subsample and the full dataset, evaluating on a held-out set of at least 200 prompts with GPT-4o (or another judge) and computing bootstrap 95% confidence intervals for the full-data-minus-ISA winrate difference. If the one-sided lower confidence bound exceeds 0, the paper's 'comparable performance' claim is rejected by its own evaluation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4 claim that ISA achieves 'comparable' performance to full-data alignment at under 10% data, but the paper's own reported win rates contradict this. From Table 2 (full data) and Table 1 (ISA): OpenAssistant full-data 24.6601 ± 0.5427 vs ISA 21.73 ± 0.6329; Ultrafeedback full-data 26.869 ± 0.1756 vs ISA 25.2252 ± 0.7007. In both cases the 95% confidence intervals do not overlap, so under the paper's own stated statistics the full-data model is significantly better. Only the Anthropic Golden dataset shows ISA numerically ahead (84.9306 vs 82.7093), but the claimed general pattern of 'comparable or better' is not supported. This issue is independent of whether GPT-4o is a biased judge: if the judge is valid, the headline claim is false; if the judge is biased, the claim is unestablished. In addition, the scaling-law fit in Eq. 1 (Table 3) sets r equal to the observed 75%-data winrate and a equal to the unaligned-model winrate, so the exponential-plateau pattern is substantially imposed by the fit endpoints rather than discovered independently. The most load-bearing problem, however, remains the internally contradictory comparison between ISA and full-data alignment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses data-efficient alignment of LLMs. It first claims an empirical scaling law R(x) = r - (r-a)e^{-bx} (Eq. 1) for alignment performance as a function of data fraction, and then proposes Information Sampling for Alignment (ISA), which models alignment data as a 2-component Gaussian mixture and uses entropy-based sampling to select a small subset. The authors claim that ISA outperforms random, density, and LLM-based sampling baselines and is comparable to full-data alignment while using less than 10% of the data, yielding over 90% resource savings.","tokens_in":15430,"tokens_out":5819,"duration_ms":52466,"significance":"If the claims were supported, the paper would make a practically valuable contribution by showing that alignment can be performed with a fraction of the data, and the proposed ISA method is a creative combination of GMM-based diversity and information-theoretic selection. However, the central 'comparable to full data' claim is contradicted by the paper's own winrate tables on two of three datasets, and the scaling-law fit is partly circular because its asymptote and initial value are fixed to observed endpoints. The appendix also reports that the scaling pattern did not appear on five other models. These issues mean the contributions are not yet established; the paper's significance is therefore currently limited.","major_comments":[{"comment":"The abstract and Section 4 claim that ISA performs 'comparable' to full-data alignment. This is contradicted by the reported 95% confidence intervals. On OpenAssistant, full-data KTO wins 24.6601 ± 0.5427 vs ISA 21.73 ± 0.6329; on Ultrafeedback, full-data wins 26.869 ± 0.1756 vs ISA 25.2252 ± 0.7007. In both cases the intervals do not overlap, indicating a statistically significant deficit for ISA under the paper's own statistics. Only the Anthropic Golden dataset shows ISA numerically ahead, but there the full-data result (82.7093) is below the 75% checkpoint (83.1681), so the scaling pattern is non-monotonic. The paper must either weaken the central claim to 'slightly worse but with large resource savings' or provide a pre-specified equivalence margin and show the differences fall within it.","section":"Section 4 / Tables 1 and 2"},{"comment":"The scaling-law fit in Eq. (1) is not an independent discovery: in Table 3, the asymptote r is set equal to the observed 75%-data winrate (e.g., 83.1681 for Anthropic, 24.6601 for OpenAssistant, 27.4849 for Ultrafeedback) and a is set equal to the 0% baseline from Table 2. Thus only the rate parameter b is fitted to the remaining data, so the exponential-plateau shape is substantially imposed by construction. The paper reports no goodness-of-fit, no uncertainty on b, and no comparison against alternative functional forms such as the hump-shaped scaling found by Rafailov et al. (2024). The non-monotonic Anthropic data (75% vs 100%) further suggests that the exponential plateau is not an adequate description. Please re-fit with all parameters free, report model comparison, and test predictive accuracy on held-out fractions.","section":"Section 3 / Table 3"},{"comment":"The appendix states that experiments on Alpaca-7B, Pythia-6.9B, LLaMa-2-7B-Chat, Dolly-6B, and Vicuna-7B 'did not reveal significant performance improvements for alignment.' This is a direct counterexample to the general claim in Section 3 that alignment performance follows the exponential plateau, and to the conclusion that data-efficient alignment is broadly feasible. The paper cannot simply speculate about data leakage without verification; either the claims must be restricted to models where the pattern holds, or the failure must be analyzed to show that the absence of improvement is not due to an incompatible scaling law.","section":"Appendix A.6.2"},{"comment":"All empirical conclusions rest on GPT-4o judge winrates, but the paper does not report the evaluation set size, the number of judge calls, judge-human agreement, or any variance due to judge sampling. The reference to 'the same setting as the KTO study' is insufficient for reproducibility, and using the same judge for both fitting the scaling law and evaluating ISA creates a risk that systematic judge bias shapes the comparison. Please provide a detailed evaluation protocol, report judge agreement, and include at least one non-judge metric (e.g., standard benchmark scores) to corroborate the winrate findings.","section":"Section 3 / Evaluation protocol"}],"minor_comments":[{"comment":"The dataset name 'OpenAssitant1' should be 'OpenAssistant1'.","section":"Section 2"},{"comment":"The word 'intial' should be 'initial'.","section":"Section 3"},{"comment":"The values of a in Table 3 (e.g., 18.2681 for Anthropic) differ slightly from the N/A rows in Table 2 (18.2663). Please ensure consistency.","section":"Table 3 / Table 2"},{"comment":"Algorithm 1 recomputes the entropy over the whole dataset for every candidate removed, giving O(N^2) complexity. Since efficiency is a core claim, a discussion of runtime or a scalable approximation would be helpful.","section":"Algorithm 1"},{"comment":"The LLM sampling prompt contains the typo 'reposne' instead of 'response'.","section":"Figure 5"},{"comment":"The 'Limitations' section is only in the appendix; the main text should explicitly reference it when making general claims about the scaling law and the feasibility of efficient alignment.","section":"Appendix A.6.2"}],"recommendation":"reject","confidential_remarks":"The central claim of the paper is directly contradicted by its own reported results on two of the three datasets, with non-overlapping confidence intervals. The scaling-law fit is also largely predetermined by the choice of endpoints, and the appendix admits failures on five additional models. These are load-bearing problems that cannot be resolved by minor edits or re-interpretation within the current scope. The ISA method itself is interesting, but the paper would require new experiments, revised claims, and a more rigorous fitting methodology before it could be considered for publication. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, short version: the abstract's headline—ISA aligns as well as the full dataset at under 10% of the data—is not supported by the paper's own Table 1 versus Table 2. On OpenAssistant, full data wins 24.66 vs 21.73; on Ultrafeedback, 26.87 vs 25.23. The error bars don't overlap. Only Anthropic Golden shows ISA ahead, and the paper generalizes from that one dataset.\n\nCredit where due: the question is worth asking—does KTO alignment need full data?—and the paper adds a real empirical component: three datasets, three seeds, CIs, and comparisons against random, density, and LLM-based sampling. ISA is a new sampler and it does beat those baselines in the reported numbers. That part of the paper is honest, legible work.\n\nBut the soft spots are load-bearing. The scaling-law plateau in Eq. 1 is essentially forced: parameters a and r are taken directly from the observed unaligned and maximum winrates, leaving b as the only fitted parameter. That is a two-parameter identity, not an independent discovery. Table 2 also shows non-monotonic behavior (OpenAssistant 10% -> 15.2, 25% -> 10.4) that a smooth exponential plateau cannot explain without a lot of noise. The ISA math is under-specified: p(x) = exp(l'(x)) with min-max normalized log-likelihood is not a normalized distribution, so the 'entropy' in Eq. 3 isn't a true entropy, and the leave-one-out algorithm is O(N^2) without any complexity discussion. Finally, the appendix explains away null results on five other models as data leakage, with no evidence.\n\nWho gets value? Someone benchmarking sampling strategies for KTO might look at the ISA-vs-baseline numbers, but not as evidence for the 90% cost-savings claim. This is workshop-level work: the idea has some novelty, the execution has real gaps. A referee should not desk-reject it, because the question is timely and the authors have put in enough structure to engage with. But I'd expect the final verdict would be reject: the central claim doesn't survive, and the scaling law needs a proper held-out test to be credible.\n\nNet: worth a careful look, not worth believing as is.","headline":"The central claim that ISA matches full-data alignment is contradicted by the paper's own tables on two of three datasets; the scaling law is a two-point fit.","tokens_in":15957,"tokens_out":3521,"would_cite":false,"duration_ms":34307,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM alignment quality plateaus exponentially with data, so a 10% subset can match the full dataset.","keywords":["LLM alignment","data-efficient alignment","exponential scaling law","KTO","entropy sampling","Gaussian mixture model","data subsampling","GPT-4o judge"],"falsifier":"Run the same KTO alignment on three additional datasets at 40%, 70%, and 100% data fractions and score the outputs with human annotators; if human win rates keep rising past 40% or ISA's sampled models do not match full-data models under human judgment, the plateau law and the >90% savings claim fail.","tokens_in":14863,"feed_emoji":"📉","tokens_out":8246,"duration_ms":72160,"temperature":0.7,"pith_summary":"The paper sets out to show that LLM alignment is data-hungry only in its early phase: alignment performance rises quickly with the fraction of training data and then plateaus, following an exponential law $R(x) = r - (r-a)e^{-bx}$. From that law it concludes that a small, carefully chosen subset of a preference dataset can align a model about as well as the full dataset. To choose that subset it introduces ISA, which fits a two-component Gaussian mixture to embeddings of the data and keeps the points whose removal changes the dataset's entropy the most. The reported experiments on three datasets claim that ISA beats random, density-based, and LLM-based sampling and is comparable to full-data alignment while using 3.5--10% of the data, a >90% saving. The paper also reports that extending the study to several other open models gave no clear gains, which it attributes to possible data leakage.","feed_headline":"10% of alignment data reaches the full-data quality plateau","feed_subtitle":"Exponential scaling law says KTO alignment gains stop early, so an entropy-based sampler can cut costs by 90%.","key_machinery":"The machinery is the combination of the plateau law and the ISA sampler. ISA first models all alignment examples as a two-component Gaussian mixture over embeddings from a Llama-3-8B-Instruct encoder, interpreting the two components as desired and undesired responses. It computes each point's log-likelihood under the mixture, converts it to a probability, computes the dataset entropy $H(X)$, and then scores every point by the drop in entropy its removal would cause. Keeping the highest-drop points yields a subset that is meant to be both diverse, because the mixture separates clusters, and high-quality, because points that most affect the distribution are retained. The law, in turn, tells the user how little data is enough: it is the value of $x$ at which $R(x)$ stops changing.","core_discovery":"The central discovery is a scaling law for direct alignment with KTO: win rate versus data fraction follows $R(x) = r - (r-a)e^{-bx}$, where $r$ is the ceiling, $a$ the unaligned model's win rate, and $b$ a growth rate tied to dataset complexity. Empirically the curve rises sharply and then flattens, meaning the last 50--90% of the dataset contributes almost nothing to the judge-measured win rate. The paper presents this as evidence that alignment is over-optimized on full data and that KTO is more robust to over-optimization than earlier direct algorithms, which show a hump-shaped rather than plateau pattern.","pith_inferences":["The reported numbers already contain a tension: on OpenAssistant and Ultrafeedback the full-data model beats ISA by 2.9 and 1.6 win-rate points with non-overlapping error bars, so 'comparable' holds only at moderate tolerance; a stricter equivalence test would likely reject equality.","The judge is used both to fit the curve and to compare samplers, which can create a feedback loop: if GPT-4o prefers a certain style, the sampler that mimics that style will appear to work best; human evaluation would be needed to rule this out.","The exponential fit has only three parameters and five data points per dataset; a model with more flexible curves might fit equally well, so the functional form is a hypothesis to test, not a law.","The ISA sampler's reliance on a specific embedding model means its benefit may not transfer to other encoders; a sensitivity analysis across embedding models would clarify which component of the method carries the gain."],"forward_implications":["Alignment runs can be cut to 10% or less of the preference data without a measurable drop in judge-reported win rate, saving most of the compute, memory, and time.","The plateau law gives a concrete rule for choosing subset size: stop adding data once $R(x)$ flattens, rather than training on the full set.","If the plateau generalizes to other direct alignment algorithms, current RLHF-style pipelines are systematically over-optimized.","High-quality subsets can be identified without labels: a two-component Gaussian mixture plus entropy scoring supplies both diversity and quality.","The same scaling law could be used to predict the data quantity needed before running a new alignment experiment."],"supporting_citations":[{"why":"Defines the KTO alignment algorithm and the GPT-4o judge setting used for all win-rate measurements.","marker":"[10]"},{"why":"Supplies the Density and LLM sampling baselines that ISA is compared against.","marker":"[24]"},{"why":"Establishes the reward-overoptimization scaling framework that the paper extends to direct alignment algorithms.","marker":"[12]"},{"why":"Shows hump-shaped scaling for prior direct alignment algorithms, the contrast that motivates the plateau claim.","marker":"[22]"},{"why":"Supplies the Ultrafeedback-binarized dataset and the Zephyr recipe used in the experiments.","marker":"[30]"},{"why":"Supplies the OpenAssistant dataset used for the scaling experiments.","marker":"[17]"},{"why":"Supplies the Anthropic Golden HH-RLHF dataset used for the scaling experiments.","marker":"[5]"},{"why":"Shows that small high-quality datasets are sufficient in the fine-tuning paradigm, the prior result the paper transfers to alignment.","marker":"[37]"}],"fun_headline_variants":["Exponential plateau: 10% of data matches full alignment","Entropy-based sampling cuts alignment cost by 90%","Align LLMs with 10% data, keep plateau quality","KTO alignment scales exponentially, then flattens early"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All measured win rates, including the exponential plateau curve and the ISA advantage, come from a single GPT-4o judge with no reported agreement check or alternative metric, so a systematic bias in that judge would reshape every experimental conclusion.","fun_headline_variants_meta":{"raw":{"variants":["Exponential plateau: 10% of data matches full alignment","Entropy-based sampling cuts alignment cost by 90%","Align LLMs with 10% data, keep plateau quality","KTO alignment scales exponentially, then flattens early"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1275,"prompt_tokens":910,"completion_tokens":365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":526,"tokens_out":365,"duration_ms":4219,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:34:50.930931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same KTO alignment on three additional datasets at 40%, 70%, and 100% data fractions and score the outputs with human annotators; if human win rates keep rising past 40% or ISA's sampled models do not match full-data models under human judgment, the plateau law and the >90% savings claim fail.","supporting_citations":[{"cited_title":"Yes\" if the interaction contains an informative signal for alignment and","cited_arxiv_id":null,"evidence_quote":"Shows that small high-quality datasets are sufficient in the fine-tuning paradigm, the prior result the paper transfers to alignment."}],"review_version":1}