{"id":"5841da2f-fd32-4c67-a605-af2a1a9ae38c","arxiv_id":"2504.14039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"MEQA scores eight cybersecurity QA benchmarks against a 44-sub-criteria rubric, finding strengths in reproducibility and comparability and weaknesses in prompt robustness and reliability.","lead":"The paper proposes MEQA, a rubric for scoring the quality of question-and-answer LLM benchmarks, and applies it to eight cybersecurity benchmarks using three human evaluators and GPT-4o. The scores suggest current benchmarks are strong on reproducibility and comparability but weak on prompt robustness and reliability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scorecard rests on unvalidated evaluator judgments: three author-evaluators, no kappa/ICC, and GPT-4o few-shot prompting that included human scores, so cross-benchmark rankings may be noise.","rationale":"The reader's weakest assumption is that the evaluators' scores are reliable and unbiased ground truth for benchmark quality. I agree this is the core issue, and the paper's own text supports the concern: Appendix D concedes that GPT-4o 'sometimes struggles to provide correct, accurate scores' and may be influenced by author tone, while Appendix C reveals that the LLM-evaluator results were generated with few-shot examples that included human scores, making the reported LLM-human agreement partly circular. I considered two other possible concerns: how N/A subcriteria are excluded from means, which could affect cross-benchmark comparability, and the equal weighting of 44 subcriteria, which is arbitrary but standard for rubric proposals. Neither is as load-bearing as the unestablished reliability of the scores themselves, because even perfect aggregation rules cannot repair ratings that are noisy or systematically influenced. The proposed check would use data the authors already possess and would distinguish a stable instrument from a procedure whose rankings are evaluator-dependent. This reinforces the reader's conditional verdict rather than changing it.","tokens_in":10167,"tokens_out":5838,"duration_ms":56423,"concrete_test":"Release the individual evaluator-by-subcriterion score matrix and compute Fleiss' kappa or ICC (two-way random, absolute agreement) from the three human evaluators' ratings. If kappa/ICC is below 0.75, or if the standard error of measurement exceeds the smallest reported benchmark difference (e.g., 0.1 between CyberSecEval 2 and WMDP-Cyber), the Table 2 rankings are within measurement noise. As a second component, re-run GPT-4o evaluation zero-shot without any human-scored examples and compare the resulting benchmark rankings to the few-shot rankings; if they differ materially, the reported LLM-human agreement is a calibration artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MEQA yields standardized, quantifiable, meaningfully comparable scores depends on the reliability of the evaluators' 1-5 sub-criterion ratings, but that reliability is not established. Section 2.1 reports only that the three human evaluators 'agreed exactly on the majority (over 80%) of the sub-criteria' and that disagreements were at most 1 point; no kappa, ICC, or per-sub-criterion score distribution is given. On a 5-point scale, 80% exact agreement can occur with strong central tendency even when raters are not distinguishing benchmarks. Appendix C states that the reported LLM-evaluator results were obtained with few-shot prompting that included scores for benchmarks already evaluated by humans, so the later claim that GPT-4o 'tended to match human evaluators' is partly produced by the calibration examples, not an independent confirmation. If these judgments are noisy or systematically influenced by the evaluators' prior expectations, then differences such as the 3.6 vs 2.7 means in Table 2 cannot be read as differences in benchmark quality, and the framework's core purpose of enabling meaningful intra-benchmark comparisons is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEQA, a meta-evaluation framework for question-answering (QA) benchmarks, organized around eight criteria (memorization robustness, prompt robustness, evaluation design, evaluator design, reproducibility, comparability, validity, reliability) and 44 sub-criteria, each scored on a 1–5 scale. The framework is demonstrated on nine cybersecurity QA benchmarks using three human evaluators (the authors) and GPT-4o as an LLM evaluator. The paper reports overall mean scores per benchmark (Table 2), per-criterion scores (Figure 1), and per-sub-criterion scores (Appendix B), and concludes that most benchmarks are strong in reproducibility and comparability but weak in prompt robustness and reliability. The central claim is that MEQA provides standardized, quantifiable, and comparable assessments that enable meaningful intra-benchmark comparisons and gap analysis for benchmark developers.","tokens_in":10376,"tokens_out":5280,"duration_ms":45268,"significance":"If MEQA were shown to be reliable and valid, it would address a genuine gap in the literature: prior meta-evaluation work has largely focused on isolated aspects such as safety-washing, reproducibility, or prompt sensitivity. The framework's 44 sub-criteria operationalize a broad set of considerations from prior work, and the paper is transparent about its proof-of-concept status in Appendix D. The framework has face validity as a structured checklist for benchmark auditing. However, the demonstration in this manuscript does not yet establish that MEQA produces stable or meaningful measurements: human-evaluator reliability is unreported, the LLM evaluation is calibrated on the same human scores, and the aggregation procedure is unspecified. These gaps directly weaken the central claim of 'meaningful intra-benchmark comparisons'; the current results are best interpreted as a pilot illustration rather than a validated instrument.","major_comments":[{"comment":"The reliability of the human evaluator scores is not established. The paper reports only that the three human evaluators 'agreed exactly on the majority (over 80%) of the sub-criteria' with disagreements at most 1 point, but it provides no chance-corrected agreement measure (e.g., Cohen's or Fleiss' kappa, or ICC) and no per-sub-criterion score distributions. On a 5-point scale, high exact-agreement rates can coexist with strong central tendency and little discriminative power between benchmarks, so the reported agreement does not rule out that the mean differences in Table 2 (e.g., 3.6 vs 2.7) reflect rater noise rather than genuine differences in benchmark quality. Since the central claim of meaningful intra-benchmark comparisons rests on these scores, this is a load-bearing omission.","section":"Section 2.1, Table 2, Appendix D"},{"comment":"The statement that GPT-4o 'tended to match human evaluators' is not evidence of independent agreement. Appendix C explains that the reported LLM results were obtained with few-shot prompting that included scores for benchmarks already evaluated by humans, so the LLM's agreement is partly a consequence of the calibration examples. The paper should report the zero-shot results separately, specify how many and which few-shot examples were used, and compute agreement statistics (e.g., Cohen's kappa or ICC) on benchmarks that were excluded from the few-shot set. Without this, the LLM-evaluator results cannot be interpreted as validating the framework or as evidence that automated meta-evaluation is scalable.","section":"Appendix C, Section 2.1"},{"comment":"The aggregation of sub-criterion scores into criterion scores and overall means is unspecified. The text states that sub-criterion scores are 'aggregated to create Figure 1, Table 2 and Table 11' (Appendix B), but no formula is given: it is unclear whether criteria are equally weighted, whether N/A sub-criteria are excluded or imputed, and whether the overall mean is the mean of criteria or of sub-criteria. This makes the reported scores non-reproducible and undercuts the claims of 'quantifiable scores' and 'standardized assessments.' The aggregation rule should be stated explicitly and, ideally, implemented in the released code so that the mapping from raw sub-criterion ratings to the reported tables can be verified.","section":"Section 2.1, Appendix B, Table 2"},{"comment":"The validation is self-referential. The human evaluators are the paper's authors, and the LLM evaluator is calibrated against those same evaluators. The manuscript provides no external anchor for the scores—no independent expert panel, no comparison with known benchmark properties, and no correlation with downstream performance or independently measured benchmark quality. The limitations in Appendix D acknowledge the proof-of-concept nature, but the abstract and conclusions nevertheless state that MEQA 'highlights the benchmarks' strengths and weaknesses' and 'enable[s] meaningful intra-benchmark comparisons.' These claims should be tempered until the instrument is validated against an external criterion, or the paper should be reframed as a proposal with pilot results.","section":"Appendix D, Abstract, Conclusions"}],"minor_comments":[{"comment":"The word 'bechmarks' is a typo; it should read 'benchmarks.'","section":"Section 3"},{"comment":"The row 'Memorization Detection' contains a stray LaTeX command 'textit' in the text; this should be removed or rendered properly.","section":"Table 3"},{"comment":"The row 'Cross-Model Consistency' contains the typo 'appraches'; it should read 'approaches.'","section":"Table 8"},{"comment":"The row 'Representative data' contains the typo 'vulnerabilty'; it should read 'vulnerability.'","section":"Table 9"},{"comment":"The row 'Annotator Expertise' contains the typo 'Prefereable'; it should read 'Preferable.'","section":"Table 6"},{"comment":"The code is described as 'available anonymously,' but the manuscript gives no repository identifier, URL, or anonymized link; please provide a way for readers to access the code.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a concise proof-of-concept, likely from a hackathon context. The framework is sensible, but the validation is thin and the central claims in the abstract and conclusions exceed what the evidence supports. I recommend major revision: the authors should either substantially expand the validation with independent evaluators, reliability statistics, and an external calibration point, or clearly reframe the paper as a proposal/position piece with the current results labeled as pilot data. I see no indication of misconduct; the limitations are openly acknowledged in Appendix D. The main risk is that the current form will be cited as evidence that MEQA is a validated instrument, which would be premature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MEQA is a proposed meta-evaluation framework for QA benchmarks, distilled into eight criteria and 44 sub-criteria, demonstrated on nine cybersecurity benchmarks. The rubric is the real contribution: it organizes a scattered meta-evaluation literature into a practical checklist for benchmark developers. The paper is clear and honest about its limitations, even including the LLM prompt template in an appendix. That said, the scorecard itself is not yet trustworthy. The three human evaluators are the authors, their agreement is reported only as 'over 80%' without kappa or ICC, and the GPT-4o few-shot examples included human scores, so its 'tended to match' is partly circular. There is no external ground truth for benchmark quality, and the aggregation from sub-criteria to criteria to overall means is not specified. On a 5-point scale, 80% exact agreement can coincide with poor discrimination, and the reported standard deviations are large enough that the mean differences in Table 2 (3.6 vs 2.7) may be noise. So the central claim of 'meaningful intra-benchmark comparisons' is unsupported by the evidence presented. None of this is fatal to the framework as a proposal. The criteria are well-grounded, the use case is timely, and the authors are appropriately cautious in their conclusions. The paper would benefit from a serious referee and major revision: add inter-rater reliability statistics, specify the aggregation weighting, and either drop the comparative ranking claims or validate them with independent evaluators. As it stands, I'd recommend engaging with the rubric but not with the specific scores. It's a useful starting point for a meta-evaluation checklist, and a serious editor should send it out rather than desk-reject.","headline":"Useful rubric, unvalidated scores: the MEQA framework is a solid contribution to benchmark meta-evaluation, but its demonstration on cybersecurity benchmarks doesn't yet support the comparative claims.","tokens_in":10908,"tokens_out":2786,"would_cite":false,"duration_ms":24080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEQA proposes a standardized scorecard for judging question-answering benchmarks and applies it to nine cybersecurity benchmarks.","keywords":["meta-evaluation","question-answering benchmarks","LLM evaluation","benchmark quality","cybersecurity benchmarks","LLM-as-judge","evaluation rubric","prompt robustness"],"falsifier":"Take the same 44 sub-criteria and the same nine benchmarks to an independent panel of cybersecurity and evaluation experts who have not seen the paper's scores, and compute inter-rater agreement (e.g., Fleiss' kappa or ICC) plus the resulting benchmark rankings. If kappa or ICC falls below conventional thresholds or the rankings change materially, the MEQA scores are an artifact of its own raters rather than a stable property of the benchmarks.","tokens_in":9964,"feed_emoji":"📊","tokens_out":7121,"duration_ms":61530,"temperature":0.7,"pith_summary":"The paper sets out to fill a gap: there are many benchmarks for question-answering LLMs, but no standard way to judge the quality of the benchmarks themselves. It proposes MEQA, a meta-evaluation framework that turns that judgment into a scorecard: eight criteria, 44 concrete sub-criteria, each scored from 1 to 5 by human or LLM evaluators. Applied to nine cybersecurity QA benchmarks, the scorecard yields mean scores from 2.7 to 3.6 and exposes a pattern—strong reproducibility and comparability, weak prompt robustness and reliability. The paper offers MEQA as a reusable blueprint for benchmark developers to audit and improve their benchmarks.","feed_headline":"MEQA grades 9 cybersecurity benchmarks on 44 quality checks","feed_subtitle":"A rubric-based meta-evaluation finds shared weaknesses in prompt robustness and reliability across all nine benchmarks.","key_machinery":"The load-bearing object is the MEQA scorecard: eight named criteria—memorization robustness, prompt robustness, evaluation design, evaluator design, reproducibility, comparability, validity, and reliability—each expanded into sub-criteria rubrics, for a total of 44. Each sub-criterion carries a 1-5 description of what a low, medium, or high score looks like, plus an N/A option for criteria that do not apply to a benchmark. The rubric is what carries the argument: it converts qualitative judgments about a benchmark into numbers that can be averaged, compared, and used for gap analysis. A second mechanism is the evaluator protocol, in which human raters and, optionally, an LLM are given the sub-criterion definition and few-shot examples before scoring.","core_discovery":"The central claim is that benchmark quality can be meaningfully decomposed, quantified, and compared: MEQA's eight criteria and 44 sub-criteria are meant to cover the main failure modes of QA benchmarks, and the 1-5 scores are meant to produce comparable numbers rather than vague impressions. The demonstration on nine cybersecurity benchmarks reports that HarmBench-Cyber and WMDP-Cyber score highest (3.6 and 3.5), SecQA and SECURE lowest (2.7), and that per-benchmark variability is large (standard deviations around 1.0-1.6). It also reports that few-shot LLM scoring tended to match the human evaluators, especially at extreme scores, which the paper takes as evidence that automated meta-evaluation can scale. The paper concludes that most cybersecurity benchmarks already do well on reproducibility and comparability but need work on prompt robustness and reliability.","pith_inferences":["If the pattern of low prompt robustness and reliability generalizes beyond these nine benchmarks, then many current QA leaderboards may be prompt-format artifacts; a quick check would re-run the same models on rephrased prompts and see whether rankings move.","The paper reports over 80% exact agreement among its three human raters but no chance-corrected statistic; computing Fleiss' kappa or ICC from the sub-criteria scores would separate the rubric's clarity from the raters' shared leniency.","Because MEQA averages sub-criteria into a mean, correlated sub-criteria could inflate or dilute the signal; a profile of per-criterion scores or a weighted aggregation would be a more conservative reading of the same data."],"forward_implications":["Benchmark developers can run MEQA before releasing a new QA benchmark and get a concrete list of weak sub-criteria to fix, instead of relying on intuition.","Automated meta-evaluation becomes practical: the paper reports that few-shot LLM scores tend to match human scores, so the 44-item rubric can be applied to many benchmarks at low cost.","The published scores give a baseline for cybersecurity benchmarks: HarmBench-Cyber at 3.6 and WMDP-Cyber at 3.5 are the current top of the set, and SecQA and SECURE at 2.7 sit at the bottom.","Large within-benchmark standard deviations imply that a single mean score hides uneven quality, so consumers of benchmark results should look at sub-criteria profiles rather than only the headline number."],"supporting_citations":[{"why":"supplies the reproducible-evaluation lessons that motivate the reproducibility criterion.","marker":"[5]"},{"why":"provides prior meta-evaluation of AI safety benchmarks that MEQA extends to QA benchmarks.","marker":"[17]"},{"why":"provides evidence that prompt formatting changes model behavior, motivating the prompt robustness criterion.","marker":"[20]"},{"why":"defines the multi-prompt evaluation metrics used as high-score anchors in prompt robustness.","marker":"[16]"},{"why":"documents evaluator inconsistency and bias, motivating evaluator design and scoring-granularity sub-criteria.","marker":"[21]"},{"why":"supplies the validity framework of face, substantive, discriminant, and convergent validity.","marker":"[22]"},{"why":"provides measurement-theory foundations and the test-retest, internal consistency, and inter-rater reliability concepts used for reliability.","marker":"[24]"},{"why":"offers the standardized evaluation harness referenced in the comparability criterion.","marker":"[8]"},{"why":"grounds the validity criterion's emphasis on real-world applicability.","marker":"[9]"}],"fun_headline_variants":["MEQA reveals shared weaknesses in cybersecurity QA benchmarks","Benchmark quality quantified: MEQA ranks 9 cyber QA tests","9 cybersecurity benchmarks graded by new MEQA framework","MEQA: a rubric to grade QA benchmarks, tested on cyber"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scorecard is only as trustworthy as the evaluators who fill it in, and the paper's evidence for that trust comes from three human raters who are also the authors, with no chance-corrected agreement measure and no outside validation of their scores.","fun_headline_variants_meta":{"raw":{"variants":["MEQA reveals shared weaknesses in cybersecurity QA benchmarks","Benchmark quality quantified: MEQA ranks 9 cyber QA tests","9 cybersecurity benchmarks graded by new MEQA framework","MEQA: a rubric to grade QA benchmarks, tested on cyber"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3011,"prompt_tokens":836,"completion_tokens":2175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":452,"tokens_out":2175,"duration_ms":15757,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:57:24.403249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 44 sub-criteria and the same nine benchmarks to an independent panel of cybersecurity and evaluation experts who have not seen the paper's scores, and compute inter-rater agreement (e.g., Fleiss' kappa or ICC) plus the resulting benchmark rankings. If kappa or ICC falls below conventional thresholds or the rankings change materially, the MEQA scores are an artifact of its own raters rather than a stable property of the benchmarks.","supporting_citations":[{"cited_title":"Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt for- matting, 2024","cited_arxiv_id":null,"evidence_quote":"provides evidence that prompt formatting changes model behavior, motivating the prompt robustness criterion."}],"review_version":1}