{"id":"976dcf99-1ded-4d7b-a252-6d2106e59faf","arxiv_id":"2411.18126","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A demonstration selection method that samples one example per difficulty tier improves few-shot language model performance by small and sometimes inconsistent margins.","lead":"The paper proposes Curriculum Demonstration Selection (CDS), a method that picks few-shot examples for language models by taking one sample from each difficulty level of the training set. Tests on nine models and three benchmarks show small, often noisy gains over a similarity-based baseline and random selection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported results contradict 'consistently outperforms': several CDS cells are within noise or below KATE, and no paired significance testing is provided, so the central claim is not established as stated.","rationale":"The reader's CONDITIONAL verdict already identifies the core risk: the evaluation does not establish the central claim. My stress-test sharpens this to a concrete, table-level contradiction: several reported cells show CDS at or below KATE, and no significance testing is supplied, so the abstract's 'consistently outperforms' is an overstatement of the evidence. This is load-bearing because the paper's contribution is an empirical recipe; if the gains are not statistically distinguishable from noise, the recipe has no demonstrated value. The missing random-stratified baseline is a second, related weakness: it prevents attributing any real gain to the difficulty metadata, which is the paper's stated novelty. Neither concern requires new theory; both can be settled by re-analysis of the existing predictions or by adding one baseline. The paper does have independent value in a simple, clearly specified algorithm and a reasonable suite of models and benchmarks, and the code-generation results on Pass are largely consistent; however, the overclaim and lack of significance testing justify the reader's conditional stance rather than outright rejection. I therefore keep the verdict unchanged and recommend the authors run the proposed significance check and stratified baseline before claiming consistent, notable improvements.","tokens_in":13298,"tokens_out":7799,"duration_ms":68318,"concrete_test":"Re-run the 17 model/benchmark cells (5 math/ARC models plus 4 Mercury models with Pass and Beyond) using the authors' own three-seed setup, but report per-instance predictions and compute paired bootstrap or permutation 95% confidence intervals for CDS minus KATE. The 'consistently outperforms' claim survives only if a preregistered majority of cells (e.g., at least 15 of 17) have confidence intervals excluding zero in CDS's favor; under the current data, several cells would fail this check. Additionally, add a random-stratified baseline that selects one random demonstration from each difficulty partition; if it matches CDS within noise, the curriculum-specific mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the Abstract and Section 4.2 is that CDS 'consistently outperforms baseline methods' across nine LLMs on three benchmarks. The paper's own tables do not support that wording. In Table 2 (Mercury), CodeLlama-7b Pass is 36.98±1.93 for CDS vs 36.85±0.90 for KATE, a difference of 0.13 points with heavily overlapping errors, and DeepSeek-Coder-6.7b Beyond is 51.51±1.38 for CDS vs 52.24±0.11 for KATE, i.e., CDS is worse. In Table 1, Llama-3-8B ARC-c average is 81.0±0.15 for CDS vs 81.11±0.20 for KATE, again worse, and Mistral-7B Algebra is 14.89±0.03 vs 14.90±0.30. These are not isolated: with only three random seeds and no paired significance test, the claimed 'consistent' improvements are mostly point estimates within one standard error. Even where CDS does appear better, the design does not isolate the mechanism: the only baselines are uniform random and KATE, with no random-stratified baseline that draws one demonstration per difficulty partition. Consequently, the observed gains could be due to simple diversity or stratification rather than to the difficulty curriculum that the method is named for. The paper's Section 6 Limitations concedes that the difficulty metadata assumption is unvalidated, but the more immediate problem is that the headline empirical claim is not established by the reported statistics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Curriculum Demonstration Selection (CDS), a method for in-context learning that partitions the training set by human-annotated difficulty levels and selects one demonstration per level, optionally using similarity-based retrieval within each level. The authors evaluate CDS against random selection and KATE on MATH, ARC-Challenge, and Mercury across five LLMs for math/commonsense and four code-specific LLMs, reporting mean and standard deviation over three seeds. The paper claims that CDS consistently outperforms both baselines and is especially effective on hard problems. However, the reported results do not support the 'consistently outperforms' claim: several CDS scores are within noise or below KATE, and no significance testing is provided. The experimental design also lacks a control that isolates the difficulty-stratification mechanism, and the reliance on unvalidated difficulty metadata raises concerns about the curriculum-based explanation.","tokens_in":13644,"tokens_out":8255,"duration_ms":67615,"significance":"The paper addresses a practical and timely question in in-context learning: how to select demonstrations to improve LLM performance. If CDS's effects are real, the method is attractive because it leverages metadata that is often available and adds little computational cost. The breadth of the evaluation (nine LLMs, three task types) is a strength, as are the ablations on retrieval function and demonstration ordering. However, the empirical evidence as presented is not sufficient to support the headline claim of consistent improvement; the missing control and lack of significance testing leave the magnitude and source of the reported gains unclear. The idea is still potentially valuable and the identified issues are addressable in a revision.","major_comments":[{"comment":"The abstract and Section 4.2 state that CDS 'consistently outperforms' the baselines, but several cells in the paper's own tables contradict this. For example, Table 1 (Llama-3-8B ARC-c Avg) gives 81.0±0.15 for CDS versus 81.11±0.20 for KATE, and Table 2 (DeepSeek-Coder Beyond) gives 51.51±1.38 for CDS versus 52.24±0.11 for KATE; Mistral-7B Algebra in Table 1 is also lower for CDS (14.89±0.03 vs 14.90±0.30). Since all results are means over only three seeds and no paired significance test is reported, differences of 0.1-1.5 points with overlapping standard errors do not establish consistent improvement. Please provide a per-dataset and per-model significance analysis (e.g., paired bootstrap or permutation test) and revise the 'consistently outperforms' claim to match the evidence.","section":"4.2, Tables 1-2"},{"comment":"The experimental design confounds the difficulty-stratification mechanism with the effect of simply diversifying the demonstration set. CDS differs from KATE not only by partitioning by difficulty but also by selecting exactly one demonstration per partition; a method that randomly samples one demonstration from each of k random partitions of the training set would control for the coverage/diversity effect. Without such a control, the gains attributed to the curriculum could be driven by any type of stratification. Please add a random-stratified baseline (one demonstration per partition, partitions defined by difficulty or by random grouping) and a KATE variant that selects one similar example per difficulty partition, to isolate the contribution of the difficulty-based curriculum.","section":"4.1.4, 4.2"},{"comment":"CDS relies on human-annotated difficulty metadata (MATH levels, ARC grade levels, LeetCode labels plus acceptance rates), but the paper does not validate that these metadata reflect the difficulty experienced by the target LLMs. This is particularly problematic for the claim in Section 4.2.4 and Figure 2 that CDS 'especially' improves hard problems, because the same metadata are used both to construct the curriculum and to define the difficulty bins in the evaluation. Please report per-item correlations between the metadata and model solve rates (or use a model-based difficulty measure as a robustness check) to rule out circularity.","section":"3.2, 4.2.4"},{"comment":"The relationship between the number of difficulty partitions and the number of demonstrations k is underspecified. Section 3.2 states that k is determined by the distribution of complexity scores, but Section 4.1.3 fixes k=5 for all experiments. For ARC-Challenge (grade levels 3-9, i.e., seven levels) and Mercury (Easy/Medium/Hard plus acceptance rates), it is not explained how exactly five partitions are formed. Please specify the binning procedure explicitly, including how grade levels or acceptance rates are mapped to five partitions, so that the method is reproducible.","section":"3.2, 4.1.3"},{"comment":"The related work section discusses In-Context Curriculum Learning (ICCL, [27]) and other curriculum-based demonstration selection methods, but the experiments compare CDS only against Uniform random and KATE. Since ICCL is the closest method that also applies curriculum principles to ICL, the absence of a comparison weakens the claim that CDS is a novel and effective curriculum-based selection method. Please add ICCL (or a faithful reimplementation) as a baseline, or justify why it is not applicable.","section":"2.2, 4.1.4"}],"minor_comments":[{"comment":"The prompt examples contain several LaTeX/formatting errors (e.g., 'äab¨', '\\textbfOutput', 'lcm' without the operator formatting in Table 4, and a garbled character in Table 5). These should be fixed for readability.","section":"Appendix A, Tables 4-5"},{"comment":"The caption 'Result on MATH and ARC-c dataset' does not clarify that the 'Avg' column refers to ARC-c accuracy while the first five columns are MATH topic accuracies. Please rename the columns or split the table to avoid ambiguity.","section":"Table 1"},{"comment":"The statement that 'KATE underperforms the random baseline by approximately 9% with Qwen 7B' is inaccurate: the difference in Table 1 is 48.66-44.62 = 4.04 points, which is about 8% relative, not 9%. Please either give absolute differences or compute the relative change correctly.","section":"4.2.1"},{"comment":"The E2H row reports values without standard deviations, while the Rand row includes them; please report errors for both rows or note that E2H is a single run.","section":"Table 3"},{"comment":"The paper does not provide a link to code or data, which makes the exact partition and retrieval procedures hard to reproduce; consider releasing a public implementation.","section":"General"},{"comment":"The keywords 'ACM proceedings, LATEX, text tagging' appear to be template placeholders and should be replaced with actual keywords for this work.","section":"Abstract/Keywords"}],"recommendation":"major_revision","confidential_remarks":"The Mercury benchmark (reference [8]) was created by one of the co-authors of this manuscript, and the paper uses it without acknowledging this overlap or comparing to an independently constructed code benchmark. This is not necessarily disqualifying, but the editor may want to ensure that the benchmark results are not a source of bias. Also, the paper is submitted to SAC (not a top NLP venue), so the novelty bar may be lower, but the statistical rigor concern still applies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nShort version: the paper proposes CDS, which picks one demonstration per difficulty bucket, and claims it consistently beats random selection and KATE across nine LLMs and three benchmarks. The specific combination is new and easy to apply, and the experiments cover a respectable spread of models. But the headline claim is not supported by the reported numbers. Several cells show CDS at parity or worse than KATE — e.g., Llama-3-8B on ARC-c (81.0 vs 81.11), DeepSeek-Coder on Beyond (51.51 vs 52.24) — and the error bars on the rest overlap heavily. There's no paired significance testing, so 'consistently outperforms' is a stretch.\n\nWhat's genuinely new: selecting exactly one example from each difficulty level, rather than only the hardest or only the most similar, and comparing random vs similarity retrieval inside that scheme. The order ablation (Table 3) is also a nice negative result — it tells you the 'curriculum' ordering doesn't matter, which quietly undermines the curriculum-learning framing. The real mechanism seems to be stratified diversity, not easy-to-hard progression.\n\nThe soft spots are mostly in the evaluation. Only two baselines, and the two nearest curriculum-based methods (complexity-based prompting and ICCL) are cited but never compared. There's no control that draws uniformly at random from each difficulty bucket, so you can't tell whether the gains come from difficulty coverage or just from having one example per bucket. The difficulty labels themselves are borrowed from human annotations (MATH levels, grade levels, LeetCode difficulty) and the paper concedes in the Limitations that they may not track model difficulty — but that caveat is buried and the main text still claims consistent gains. The Mercury benchmark is from the same group, and no code is released, which makes the numbers hard to verify.\n\nThe paper is clearly written, the method is trivial to implement, and the authors are upfront about some limitations. It's a reasonable workshop paper or a short conference submission after revision. As it stands, the evidence doesn't justify the abstract's claim. I'd send it to peer review, but the revision needs significance testing, a random-stratified control, comparison to the missing baselines, and a softer central claim.\n\nFor your reading group: maybe — good for a discussion of how easy it is to overclaim with a few random seeds.","headline":"A simple, cheap demonstration-selection idea with an overclaimed headline; the paper's own tables do not support 'consistently outperforms'.","tokens_in":14173,"tokens_out":4459,"would_cite":false,"duration_ms":39647,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Difficulty-stratified demonstrations improve in-context learning across nine LLMs.","keywords":["in-context learning","demonstration selection","curriculum learning","difficulty stratification","few-shot prompting","large language models","MATH benchmark","code generation"],"falsifier":"Take a fixed training pool and a fixed model, and run CDS twice: once with true difficulty labels and once with difficulty labels randomly permuted across examples while keeping the same bucket sizes. If random-label CDS matches true-label CDS on the same test set, the curriculum signal is not doing the work. A second check is to replace human labels with model-computed difficulty, such as the model's own zero-shot accuracy on each candidate example, and require that CDS performance tracks the model-based partition rather than the human-based one.","tokens_in":13095,"feed_emoji":"🎯","tokens_out":6039,"duration_ms":52748,"temperature":0.7,"pith_summary":"Large language models learn from examples placed in the prompt, and picking which examples matters. This paper argues that choosing one demonstration from each difficulty level, easy through hard, works better than choosing the most similar examples or a random set. The proposed Curriculum Demonstration Selection (CDS) splits the training data by human-annotated complexity, retrieves one sample per level for each test question, and shuffles their order. Across nine open-weight LLMs on math, commonsense, and code benchmarks, CDS consistently beats uniform random and KATE similarity selection, with the largest gains on the hardest problems. A sympathetic reading treats CDS as evidence that difficulty coverage, not just example quality or similarity, is what makes few-shot demonstrations useful.","feed_headline":"One example per difficulty tier improves few-shot LLM accuracy","feed_subtitle":"CDS samples one demonstration per complexity level, beating random and similarity baselines on math, commonsense, and code.","key_machinery":"The load-bearing object is the difficulty partition: the training set is divided into k strata by human-annotated complexity labels, and the retrieval function, either random or CLS-embedding similarity as in KATE, is constrained to return exactly one example per stratum. This forces the prompt to span the full difficulty range instead of concentrating on the most similar or most complex items. The shuffle at the end makes clear that ordering is not part of the mechanism.","core_discovery":"The central claim is that a balanced difficulty curriculum in the prompt, rather than the hardest or the most similar examples, elicits better in-context performance. Concretely, CDS partitions the training set into k difficulty groups using dataset metadata, such as the five MATH complexity levels, ARC grade levels, and LeetCode difficulty labels plus acceptance rates, then retrieves exactly one demonstration from each group and shuffles them. The paper reports that this recipe outperforms uniform random selection and KATE on all nine tested models, that similarity retrieval within levels beats random retrieval, that easy-to-hard ordering does not matter, and that the improvement grows with problem difficulty, from about 2% on easy MATH problems to 6% on hard ones. If correct, CDS is a lightweight, metadata-driven upgrade to few-shot prompting that needs no model training.","pith_inferences":["Beyond the paper's claims: if these results hold up, any benchmark that ships with grade levels, contest tiers, or acceptance rates can be plugged into CDS without retraining, and datasets lacking such labels could be partitioned with model-based difficulty estimates, an extension the paper itself names as future work.","A natural falsification experiment the paper does not run is assigning random difficulty labels while keeping the same bucket structure; if CDS still beats KATE, the measured gains would come from stratification itself rather than from the meaning of the labels.","The pattern that hardest problems benefit most suggests CDS acts less like a curriculum in the training sense and more like a coverage regularizer that prevents the prompt from being dominated by easy, high-similarity examples; that hypothesis could be tested by comparing CDS to selecting k diverse examples purely by embedding distance.","A general consequence the paper leaves implicit is that difficulty-stratified prompts could transfer to few-shot settings where the test distribution is unknown or changing, because coverage provides robustness that pure similarity cannot."],"forward_implications":["On MATH and ARC-Challenge, CDS outperforms both uniform random selection and KATE across Llama-2 7B/13B, Llama-3 8B, Mistral-7B, and Qwen-7B.","On Mercury code generation, CDS improves the Pass metric over both baselines for CodeLlama-7B/13B, DeepSeek-Coder-6.7B, and StarCoder2-3B, and matches or slightly trails KATE on the Beyond efficiency metric.","The performance advantage of CDS grows with problem difficulty, from roughly 2% on easy MATH problems to 6% on hard ones, and ARC-Challenge shows a similar but smaller trend.","Presenting demonstrations easy-to-hard versus shuffled order makes no significant difference, so the benefit comes from the composition of the demonstration set, not its ordering.","Using similarity-based retrieval inside each difficulty level is better than random retrieval within levels, so difficulty coverage and query relevance combine."],"supporting_citations":[{"why":"Supplies the MATH dataset with its five human-annotated complexity levels, the primary math benchmark and curriculum source.","marker":"[15]"},{"why":"Supplies the ARC-Challenge dataset with grade-level metadata used to build the commonsense curriculum.","marker":"[5]"},{"why":"Supplies the Mercury benchmark whose LeetCode difficulty labels and acceptance rates define the code-generation curriculum.","marker":"[8]"},{"why":"Provides the KATE similarity-retrieval baseline and the CLS-embedding retrieval function that CDS uses within each difficulty bucket.","marker":"[25]"},{"why":"Introduces curriculum learning, the conceptual basis for presenting demonstrations from easy to difficult.","marker":"[2]"},{"why":"Prior curriculum-based in-context learning method that orders selected demonstrations, which CDS compares with and contrasts against.","marker":"[27]"},{"why":"Prior work selecting the hardest demonstrations via demonstration-query likelihood, the opposite curriculum choice that CDS positions against.","marker":"[7]"},{"why":"Chain-of-thought prompting is the prompt format used in the math and commonsense experiments.","marker":"[46]"}],"fun_headline_variants":["One demo per difficulty tier: curriculum selection beats random","Easy-to-hard demos: one per complexity tier boosts few-shot LLM","Difficulty-tier balanced prompts lift LLM accuracy on hard problems","CDS: one example per difficulty class improves few-shot performance","Balanced curriculum demos: easy-to-hard beats similarity in ICL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that human-annotated difficulty labels, such as MATH complexity levels, ARC grade levels, and LeetCode difficulty plus acceptance rates, track what each LLM actually finds hard, and that a flat one-example-per-level sample is more useful than the hardest or the most similar examples.","fun_headline_variants_meta":{"raw":{"variants":["One demo per difficulty tier: curriculum selection beats random","Easy-to-hard demos: one per complexity tier boosts few-shot LLM","Difficulty-tier balanced prompts lift LLM accuracy on hard problems","CDS: one example per difficulty class improves few-shot performance","Balanced curriculum demos: easy-to-hard beats similarity in ICL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3577,"prompt_tokens":837,"completion_tokens":2740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":2652}},"tokens_in":453,"tokens_out":2740,"duration_ms":16428,"temperature":1.0,"reasoning_tokens":2652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:29:12.426585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed training pool and a fixed model, and run CDS twice: once with true difficulty labels and once with difficulty labels randomly permuted across examples while keeping the same bucket sizes. If random-label CDS matches true-label CDS on the same test set, the curriculum signal is not doing the work. A second check is to replace human labels with model-computed difficulty, such as the model's own zero-shot accuracy on each candidate example, and require that CDS performance tracks the model-based partition rather than the human-based one.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the KATE similarity-retrieval baseline and the CLS-embedding retrieval function that CDS uses within each difficulty bucket."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces curriculum learning, the conceptual basis for presenting demonstrations from easy to difficult."}],"review_version":1}