{"id":"6e1c8af1-27b9-45b2-9b2d-c006b1afd4e2","arxiv_id":"2502.01683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"BenchMaker automatically builds multiple-choice benchmarks from assessment demands and matches human benchmarks' ranking power (0.967 Pearson correlation with MMLU-Pro across 12 LLMs) at $0.005 per item.","lead":"This paper introduces BenchMaker, a system that automatically generates multiple-choice benchmark questions from a short description of the skills to test, using several LLM-based quality checks. It also proposes a ten-criterion evaluation framework and reports that BenchMaker-built tests rank 12 LLMs nearly identically to the human-written MMLU-Pro (0.967 Pearson correlation) at about half a cent per question.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generated benchmarks' overlap with MMLU-Pro/MATH/HellaSwag is never measured; if GPT-4o mini reproduces memorized benchmark content, the 0.967 correlation is not evidence of generic generation.","rationale":"The reader's weakest assumption concerns the LLM-judge debiasing claim in Section 3.1. That concern is valid for the faithfulness and alignment scores in Table 2, but those scores are not the primary quantified claim. The abstract's headline evidence is the 0.967 Pearson correlation between BenchMaker and MMLU-Pro across 12 LLMs, computed from objective multiple-choice accuracy, not from LLM judges. The most load-bearing condition for that claim is that the generated benchmark is genuinely new rather than a regurgitation of the human benchmark from pretraining. The paper's only safeguard is that assessment demands do not mention MMLU-Pro; this is insufficient because GPT-4o mini has almost certainly seen MMLU-Pro during training. The authors provide no quantitative overlap analysis, even though they already use embeddings and t-SNE for other analyses. The examples in Appendix H look novel, but two selected examples per task do not establish the absence of near-duplicates in thousands of samples. If overlap is high, the effectiveness correlation is not evidence for the proposed pipeline and the paper's central 'generic and reliable' claims are unsubstantiated. This is a concrete, testable risk, and the existing CONDITIONAL verdict is appropriate pending such an overlap audit. I therefore keep the reader's verdict unchanged but disagree with the identified weakest assumption, since the contamination risk is more directly load-bearing for the headline result.","tokens_in":31561,"tokens_out":12238,"duration_ms":132914,"concrete_test":"Compute overlap between the generated benchmark (e.g., 500 samples per MMLU-Pro subset) and the corresponding human benchmark (MMLU-Pro) after normalizing text: (a) exact match rate, (b) maximum 13-gram overlap ratio per generated sample, and (c) cosine similarity of text-embedding-ada-002 embeddings to the nearest MMLU-Pro sample. Calibrate against a control of held-out MMLU-Pro questions (e.g., random train/test splits) to establish expected chance overlap. If the generated samples show near-duplicate rates significantly above control (e.g., >1-2%), remove all near-duplicate generated samples and recompute the 12-model Pearson correlation with MMLU-Pro; if the correlation drops below roughly 0.9, the headline effectiveness result is not robust to contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central effectiveness claim requires that BenchMaker outputs are newly generated from assessment demands, not near-duplicates of the human benchmarks used for validation. Section 5.3 asserts that 'the input demands do not mention any information related to MMLU-Pro, effectively preventing the model from achieving a high degree of alignment by memorizing and replicating samples from MMLU-Pro,' but this does not prevent GPT-4o mini from regenerating benchmark questions seen during pretraining. The paper reports no overlap statistics, no n-gram or embedding duplicate checks, and no membership tests against MMLU-Pro, MATH, or HellaSwag. The t-SNE and word-cloud analyses show global distributional differences but cannot rule out a substantial subset of near-duplicates. If a nontrivial fraction of generated samples are memorized or lightly rephrased versions of the comparison benchmarks, then the 0.967 Pearson correlation is inflated and the 'generic' claim collapses. This concern is more directly threatening to the headline result than the LLM-judge debiasing issue, which affects auxiliary faithfulness/alignment metrics rather than the accuracy-based effectiveness correlation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"BenchMaker is a pipeline that generates multiple-choice benchmarks solely from written assessment demands. The paper first defines a ten-criteria evaluation framework for benchmark generators, with LLM-judge scores debiased via regression on judge length. It then analyzes a direct-prompt baseline, adds modules for faithfulness, difficulty, and diversity, and evaluates the resulting BenchMaker on math (MATH), language understanding (MMLU-Pro), and commonsense (HellaSwag). The headline result is a 0.967 Pearson correlation between model accuracies on the generated MMLU-Pro-style benchmark and on MMLU-Pro across 12 LLMs, at a cost of about $0.005 and 0.4 minutes per sample.","tokens_in":31721,"tokens_out":5823,"duration_ms":51507,"significance":"The claimed result is potentially important: if a cheap, generic generator can produce benchmarks whose model rankings match human-curated benchmarks, it would lower the cost of customized evaluation and help mitigate contamination. The paper's use of external human benchmarks for validation is the right methodology, and the per-sample cost estimate is compelling. The framework also usefully separates credibility, diversity, difficulty, and benchmark-level criteria. However, the headline claim is only as strong as the exclusion of memorization and the validity of the debiasing adjustment; both need additional evidence before the 'reliable, generic' label is warranted. The paper ships code and includes a manual check and a reliability formula, which are positive features.","major_comments":[{"comment":"The effectiveness claim (0.967 Pearson against MMLU-Pro) is vulnerable to memorization contamination. Section 5.3 argues that because the input demands do not mention MMLU-Pro, the model cannot achieve high alignment by 'memorizing and replicating samples from MMLU-Pro,' but this does not rule out GPT-4o mini reproducing or lightly paraphrasing benchmark items seen during pretraining. The t-SNE and word-cloud analyses show global distributional differences but cannot detect a substantial subset of near-duplicates. Please report quantitative overlap statistics between the generated benchmarks and MMLU-Pro, MATH, and HellaSwag (e.g., n-gram overlap, embedding-similarity percentiles, or membership tests). Without such analysis, the 'generic generation' interpretation of the 0.967 correlation is not established.","section":"§5.3, Table 2"},{"comment":"The framework's 'unbiased' metrics rely on the claim that all LLM-judge biases are mediated by judgment length. This is supported by partial correlations on a single dataset (MATH) with two judges (Qwen-Plus and GPT-4o mini). Unmeasured confounders (e.g., sample topic, rationale style, or position bias) could invalidate the regression adjustment in Eq. (1), and the faithfulness/alignment scores in Table 2 depend on it. Please validate the mediation assumption on at least one additional dataset and provide a sensitivity analysis with alternative covariate sets.","section":"§3.1"},{"comment":"Difficulty controllability and knowledge diversity are computed using the error rates and correctness patterns of the same 12-model set M1:|M| that is later used for the effectiveness correlation. This creates a risk of self-reference: the difficulty labels are defined by the models' behavior, and the benchmark is then evaluated on those same models. Please report whether the difficulty and effectiveness results are stable when M1:|M| is split into separate sets for label construction and evaluation, or otherwise justify why this circularity does not affect the claims.","section":"§3.3–§3.4, Appendix E"},{"comment":"The abstract states that BenchMaker 'achieves superior or comparable performance to human-annotated benchmarks on all metrics,' but Table 2 shows faithfulness of 0.930 (MATH) and 0.918 (BenchMaker4o) against a human benchmark score of 1.000, and the manual check in §5.3 reports a 7.5% error rate. These point estimates are below the human benchmark. Please replace the blanket claim with a metric-by-metric comparison that includes confidence intervals or a statement of statistical significance.","section":"Abstract, §5.1, Table 2"}],"minor_comments":[{"comment":"The method name 'BENCH MAKER' appears with and without a space (e.g., 'BENCHMAKER', 'BenchMaker', 'BENCH MAKER'); please unify the notation.","section":"Throughout"},{"comment":"The variance formula for ¯a − ¯b ignores the covariance between the two models' accuracies because both are scored on the same benchmark items; please state this approximation explicitly.","section":"Appendix A, Eq. (8)"},{"comment":"The 'Human Benchmark' rows show '-' for effectiveness and robustness; adding a one-sentence explanation would help readers understand why these cells are empty.","section":"Table 2"},{"comment":"The phrase 'The most relevant path' is not defined; please specify the selection rule (e.g., highest absolute correlation along a directed path).","section":"Figure 2 caption"},{"comment":"The manual check of 80 samples should describe the sampling procedure and whether the multiple authors annotated independently, as this affects the reported 7.5% error rate.","section":"§5.3"},{"comment":"References to 'Appendix I' are ambiguous because the appendix contains several prompt templates; please cite the specific prompt names (e.g., 'LLM as Faithfulness Judge').","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The memorization concern is the main risk for the headline result; I recommend that the editor ask the authors to release the generated benchmark corpora along with the code so that reviewers and the community can independently run overlap checks. The debiasing claim in §3.1 is also stronger than the evidence, as it is tested on a single dataset. The paper is otherwise within scope for an NLP evaluation venue, and the external validation against human benchmarks is a real strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, BenchMaker is a genuinely well-engineered system: it takes assessment demands as sole input, generates MCQs, and reports a 0.967 Pearson correlation between model rankings on its own benchmark and on MMLU-Pro across 12 LLMs. Second, the paper never checks whether its generated questions overlap with the human benchmarks used for validation. If GPT-4o mini is reproducing memorized items, that correlation says nothing about generic generation. The stress-test concern lands.\n\nWhat's new: the ten-criterion evaluation framework (faithfulness, alignment, diversity, difficulty controllability, boundary, effectiveness, robustness, efficiency) is a useful organizing schema, and the causal-debiasing attempt for LLM-as-a-judge is a thoughtful step, even if incomplete. The integrated pipeline—stepwise self-correction, conflict-guided contrastive discrimination, difficulty diffusion, in-batch diversity boosting—is a reasonable combination of existing ideas plus some new mechanisms. The paper is honest: it reports a 7.5% error rate from a manual check of 80 samples, and it describes unsuccessful attempts in Appendix B. Code is promised.\n\nSoft spots. The overlap issue is the load-bearing one. The paper's argument that the input demands don't mention MMLU-Pro does not address memorization during pretraining. t-SNE and word clouds show global distributional differences but cannot rule out a substantial subset of near-duplicates. There are no n-gram, embedding, or membership statistics against MMLU-Pro, MATH, or HellaSwag. This is more threatening than the judge-debiasing limitation, which affects auxiliary faithfulness/alignment metrics rather than the accuracy-based effectiveness correlation.\n\nThe debiasing assumption—that all LLM-judge biases are mediated by judgment length—is tested on one dataset (MATH) with partial correlations for two factors. That's thin support for 'unbiased' metrics used throughout Table 2. Also, the same 12-model set supplies difficulty ground truth and effectiveness evaluation; that's partially circular, though the central effectiveness against human benchmarks is external. Minor: no error bars or confidence intervals in Table 2, and hyperparameters like reference number and in-batch candidates lack sensitivity analysis.\n\nFor whom: anyone working on synthetic data, benchmark generation, or LLM evaluation methodology. It deserves a serious referee. The authors should be asked to add overlap analysis, extend the debiasing validation to at least one more task and judge, and report variance. If the overlap check comes back clean, the paper is a solid contribution; if it shows substantial overlap, the central claim weakens significantly.","headline":"Well-engineered benchmark generator with a strong external-validity claim, but missing overlap checks against the human benchmarks it validates on—fix that before trusting the 0.967 correlation.","tokens_in":32327,"tokens_out":3197,"would_cite":true,"duration_ms":31158,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BenchMaker, a purely LLM-driven pipeline, generates a custom multiple-choice benchmark from a written assessment demand, and the paper reports that its model rankings reproduce MMLU-Pro's ordering of 12 LLMs with 0.967 Pearson correlation…","keywords":["benchmark generation","synthetic data","LLM-as-a-judge","multiple-choice questions","evaluation framework","difficulty control","data contamination","BenchMaker"],"falsifier":"Run the Figure 2 partial-correlation analysis on a dataset with a known judge bias unrelated to length, such as a preference for a particular answer position or phrasing style, and check whether faithfulness and alignment scores still agree with human labels after controlling for judge length; if any such factor remains significant, the mediation-by-length assumption fails.","tokens_in":31288,"feed_emoji":"📊","tokens_out":9670,"duration_ms":81069,"temperature":0.7,"pith_summary":"The paper claims that a single LLM pipeline, BenchMaker, can generate a custom multiple-choice benchmark from nothing more than a written assessment demand, and that the resulting benchmark matches or beats human-annotated benchmarks on the authors' ten evaluation criteria. The headline result is that model rankings produced by BenchMaker's benchmark correlate with MMLU-Pro at 0.967 Pearson across 12 LLMs, at a cost of about $0.005 and 0.38 minutes per sample. To make this credible, the paper builds a four-dimension, ten-criteria evaluation framework and argues that LLM-judge biases in this scenario are mediated entirely by judgment length, so regression on judge length yields unbiased faithfulness and alignment scores. If true, this would let practitioners produce tailored, contamination-resistant, difficulty-controllable benchmarks on demand.","feed_headline":"Machine-made benchmarks match human rankings at 0.967","feed_subtitle":"BenchMaker turns a one-line assessment demand into a test at $0.005 per question and matches MMLU-Pro's 12-model ordering.","key_machinery":"The load-bearing mechanism is the automated evaluation framework itself, a four-dimension, ten-criteria scoring system: credibility (faithfulness and alignment), diversity (lexical, semantic, and knowledge), difficulty (controllability and boundary), and benchmark-level properties (effectiveness, robustness, and efficiency). Inside it, the debiasing step fits a multiple-regression model $f(i)=\\beta_i+\\beta_{\\text{len}}\\cdot \\text{judge\\_length}+\\epsilon$, treating judge length as a covariate so that the coefficient $\\beta_i$ becomes the debiased faithfulness or alignment score. On the generation side, the difficulty diffusion mechanism carries the argument: the hardest generated samples are fed back as references so that sample difficulty rises continuously, while conflict-guided contrastive discrimination treats the generator as a test-taker and uses self-consistency prediction conflicts to correct questionable rationales and labels.","core_discovery":"The central claim is that a generic benchmark generator, not a task-specific augmenter, can be made reliable enough to substitute for human-annotated benchmarks. With only assessment demands as input, BenchMaker generates multiple-choice samples containing a question, rationale, options, and label, and it patches the weaknesses of directly prompting an LLM: stepwise self-correction and conflict-guided contrastive discrimination raise faithfulness, difficulty strategy guidance and a difficulty diffusion mechanism extend difficulty controllability and boundary, and AttrPrompt plus in-batch redundancy filtering raise diversity. Under the proposed framework, BenchMaker is reported to achieve faithfulness comparable to human benchmarks, higher alignment, knowledge diversity, difficulty controllability and boundary, and an average of 0.953 Pearson and 0.966 Spearman correlation with the model rankings of human benchmarks, including 0.967 Pearson against MMLU-Pro across 12 LLMs. The paper also derives a formula intended to show that unbiased label noise does not change the statistical significance of a model ranking.","pith_inferences":["Editorial inference: the 0.967 correlation is measured with MMLU-Pro as the ground truth, so the strongest version of the claim is about reproducing an existing human benchmark rather than about discovering a correct ranking for a brand-new domain.","Editorial inference: a natural testable extension is to apply BenchMaker to an assessment demand outside MATH, MMLU-Pro, and HellaSwag with an independent gold standard, such as a professional licensing exam, and check whether the effectiveness correlation stays above 0.9.","Editorial inference: the debiasing assumption implies that judge biases not channeled through response length, such as option-position preference or topic aversion, would escape the regression correction, so the unbiasedness claim is only as broad as that assumption.","Editorial inference: the paper's own format-conversion experiment shows that open-ended versions score lower on knowledge diversity and effectiveness than the MCQ versions, suggesting the headline numbers may not transfer directly to free-form generative benchmarks."],"forward_implications":["If BenchMaker is right, producing a tailored benchmark for a new assessment demand requires only a written demand, not an existing seed benchmark.","The reported 0.967 Pearson correlation with MMLU-Pro across 12 LLMs implies that benchmark effectiveness can be bought for roughly $0.005 per sample, making large custom evaluation suites affordable.","The improved difficulty controllability and boundary imply that generated benchmarks can include hardest subsets capable of differentiating advanced models, addressing benchmark saturation.","The reliability formula implies that a known fraction of incorrect labels does not by itself weaken the statistical significance of a model ranking, provided the errors are unbiased.","Because samples are generated on demand rather than drawn from a static pool, the approach could serve as a dynamic benchmark generator to mitigate data contamination."],"supporting_citations":[{"why":"Provides MMLU-Pro, the human-annotated benchmark used as the effectiveness ground truth and the target of the 12-LLM ranking comparison.","marker":"(Wang et al., 2024b)"},{"why":"Provides MATH, the dataset used for the judge-bias analysis, the sequential ablations, and one of the three effectiveness comparisons.","marker":"(Hendrycks et al., 2021b)"},{"why":"Provides HellaSwag, the third human-annotated benchmark used to show generalization across task types.","marker":"(Zellers et al., 2019)"},{"why":"AttrPrompt is the diversity technique BenchMaker adopts for lexical and semantic diversification.","marker":"(Yu et al., 2023)"},{"why":"Underpins the choice of comparison-based judging and motivates mitigating self-enhancement bias in the conflict-guided contrastive discrimination step.","marker":"(Zheng et al., 2023)"},{"why":"Self-consistency majority voting supplies the conflict signal used to detect questionable labels.","marker":"(Wang et al., 2023a)"},{"why":"Supports the claim that LLMs struggle to correctly judge their prior answers, motivating the test-taker formulation.","marker":"(Huang et al., 2024)"},{"why":"Provides the partial-correlation computation used to test whether judge biases remain after controlling for judge length.","marker":"(Vallat, 2018)"}],"fun_headline_variants":["AI benchmarks match human rankings at 0.967 for under a cent","BenchMaker: reliable AI-generated benchmarks, 0.967 correlation","Generic AI benchmark generator hits human-level fidelity at $0.005","0.967 correlation with human benchmarks, made for $0.005"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation framework is only as trustworthy as the claim that every LLM-judge bias in this scenario is mediated by judgment length, so any bias that does not run through response length survives the debiasing regression.","fun_headline_variants_meta":{"raw":{"variants":["AI benchmarks match human rankings at 0.967 for under a cent","BenchMaker: reliable AI-generated benchmarks, 0.967 correlation","Generic AI benchmark generator hits human-level fidelity at $0.005","0.967 correlation with human benchmarks, made for $0.005"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3733,"prompt_tokens":948,"completion_tokens":2785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2708}},"tokens_in":564,"tokens_out":2785,"duration_ms":16435,"temperature":1.0,"reasoning_tokens":2708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:02:22.516473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Figure 2 partial-correlation analysis on a dataset with a known judge bias unrelated to length, such as a preference for a particular answer position or phrasing style, and check whether faithfulness and alignment scores still agree with human labels after controlling for judge length; if any such factor remains significant, the mediation-by-length assumption fails.","supporting_citations":[],"review_version":1}