{"id":"7518e1fc-9edf-45a2-8d96-d1b17609e915","arxiv_id":"2508.07353","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors argue that data scaling alone is suboptimal for domain-specific benchmark construction and introduce Comp-Comp, an iterative comprehensiveness and compactness framework, demonstrated through the new academic benchmark PolyBench.","lead":"This paper proposes Comp-Comp, a framework for building domain-specific AI benchmarks that balances full domain coverage against compactness, and applies it to create an academic benchmark called PolyBench. A generalist reader should care because benchmark design steers how foundation models are judged, and this work argues that more data is not always better.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that Comp-Comp generalizes across domains rests on an unverified, likely circular operationalization of 'semantic recall' from a single institution's corpus.","rationale":"The reader's verdict is UNVERDICTED, and my stress-test supports that: the abstract alone provides insufficient evidence to evaluate the central claims. I do not shift the verdict because my concern is not a demonstrated error but an absence of necessary support—specifically, an external gold standard for 'semantic recall' and cross-domain replication. The GitHub release of code and data is a positive signal and makes the proposed test feasible. I partially agree with the reader's weakest assumption: the single-institution concern is real, but I would sharpen it as a potential circularity in how 'domain breadth' is defined and measured. If the full text reveals an external taxonomy and held-out validation, the concern would dissolve. Until then, the manuscript remains unverdictable based on the abstract.","tokens_in":899,"tokens_out":4173,"duration_ms":41612,"concrete_test":"Choose a second, independent academic corpus from a different university (e.g., MIT OpenCourseWare syllabi) as an external gold standard for 'semantic breadth.' Run Comp-Comp on the original university's corpus, then measure semantic recall as the fraction of external gold-standard course topics matched by the resulting benchmark, using expert annotators. Compare against a size-matched random subset and a frequency-ranked subset. If the Comp-Comp benchmark does not beat these baselines on the external topics, or if recall degrades substantially on the external corpus, then the single-corpus construction encodes institutional bias and the domain-agnosticism claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Comp-Comp procedure, by enforcing comprehensiveness and compactness, produces a benchmark that outperforms raw scaling and is domain-agnostic. For this to hold, the procedure needs a non-circular target: a definition of the domain's full breadth that is independent of the corpus being mined. The abstract provides no such definition. The case study builds PolyBench from 'a well-renowned university's' academic corpus. If the topic taxonomy and ground-truth recall labels are derived from that same corpus, then 'semantic recall' measures coverage of one institution's curriculum, not academia. 'Precision' gains from compactness are likewise relative to whatever noise definition is chosen. The framework and benchmark are co-introduced, with no external validation or cross-domain replication reported. Thus the phrase 'domain-agnostic and readily adaptable' is an assertion, not evidence. The load-bearing premise is not merely that one university represents academia; it is that the framework has a well-defined, externally anchored notion of the domain it compresses. Absent that, the comparison against data-scaling baselines is uninterpretable. This is a measurement/generalization risk, not an internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that data scaling is not always the optimal principle for domain-specific LLM benchmark construction. It introduces Comp-Comp, an iterative framework grounded in comprehensiveness (to ensure semantic recall) and compactness (to improve precision). A case study at a single 'well-renowned university' yields PolyBench, described as a large-scale, high-quality academic benchmark. The framework is asserted to be domain-agnostic and readily adaptable to other specialized fields. This assessment is based solely on the abstract; the full manuscript was not available for review.","tokens_in":1059,"tokens_out":4083,"duration_ms":39113,"significance":"If substantiated, the paper would make a valuable, falsifiable contribution: a principled alternative to raw data scaling for domain-specific LLM evaluation, with a concrete artifact (PolyBench) and public code/data. The claim that balanced coverage can beat scale is important for benchmark design, and the proposed iterative procedure could transfer to other fields. However, the abstract alone contains no benchmark statistics, no baseline comparison, no LLM evaluation, and no cross-domain validation. The significance is therefore entirely conditional on evidence that is not visible in this manuscript.","major_comments":[{"comment":"The paper asserts that comprehensiveness ensures semantic recall and compactness improves precision, and that Comp-Comp outperforms data scaling. No benchmark statistics, baseline comparison, LLM evaluation, or formal definitions are provided. The central empirical claim is therefore unsupported. A revision must include the actual construction algorithm, explicit metrics for precision and recall, and at least one comparison against a scaled benchmark with a matched construction budget.","section":"Abstract, central claim"},{"comment":"PolyBench is built using Comp-Comp, and its quality is then used to demonstrate Comp-Comp's effectiveness. If the topic taxonomy and ground-truth recall labels are derived from the same single-university corpus, the evaluation is circular: 'semantic recall' would measure coverage of that corpus rather than of academia. The paper needs an external anchor, such as an independently defined domain ontology or validation on external exam/standardized tasks, and must explicitly state where the taxonomy and labels come from.","section":"Abstract, case study"},{"comment":"The claim that Comp-Comp is 'domain-agnostic and readily adaptable' is asserted without cross-domain validation. A single-institution academic corpus may encode institutional or disciplinary biases. At minimum, the authors should report a second-domain replication or provide a non-circular procedure for defining a domain's 'full breadth' that is independent of the source corpus. Without this, the generality claim is not evidence-based.","section":"Abstract, generalization claim"},{"comment":"The phrase 'data scaling is not always the optimal principle' is vague. What exactly is the scaled baseline—number of questions, corpus size, or compute? The comparison must control for budget and evaluation protocol; otherwise the trade-off claim is not meaningful. The abstract should specify the scaling dimension and the matched resources used in the PolyBench case study.","section":"Abstract, baseline definition"}],"minor_comments":[{"comment":"Define 'comprehensiveness' and 'compactness' more precisely: are they cardinalities, diversity measures, or something else? The current phrasing is evocative but not operational.","section":"Abstract"},{"comment":"Specify PolyBench's scale and composition (e.g., number of QA pairs, subjects, question types) rather than the vague term 'large-scale.'","section":"Abstract"},{"comment":"Identify the specific university or explain why it is anonymized; reproducibility and assessment of disciplinary bias benefit from specificity.","section":"Abstract"},{"comment":"Provide a versioned identifier (e.g., DOI or commit) and license for the linked code and datasets to support reproducibility.","section":"Abstract, GitHub link"}],"recommendation":"uncertain","confidential_remarks":"This submission appears to be abstract-only; no full text was available for review. The major comments identify the specific evidence that would be required to evaluate the central claims. I recommend either requesting the full manuscript or desk-rejecting if the abstract is the entire submission, as the current form cannot be meaningfully assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nQuick take on arXiv:2508.07353. I only have the abstract, so this is a provisional read, but here's where I land.\n\nThe core idea is clearly stated: for domain-specific LLM benchmarks, data scaling isn't automatically the best strategy; you can do better by explicitly optimizing for comprehensiveness (semantic recall) and compactness (precision). That's a plausible reframing, and if the authors can show it works, it's a useful contribution to benchmark design. The paper also ships code and data via GitHub, which is a point in its favor.\n\nWhat I can't tell from the abstract is whether the claim is actually supported. The case study builds PolyBench from a single university's corpus. The worry — and the stress-test note puts it well — is that if the notion of \"domain breadth\" and the ground-truth recall labels are derived from that same corpus, then \"semantic recall\" just measures coverage of that institution's curriculum, not academia as a whole. That would make the comparison against data-scaling baselines largely circular. The abstract says the framework is domain-agnostic but gives no cross-domain validation, so that part is asserted, not demonstrated.\n\nI want to be fair: this is a risk, not a proven flaw. With only the abstract, I can't tell whether the authors have an independent definition of domain coverage or whether they tested on something external. The same could be said for the precision gains: they depend on the noise definition chosen. If the full paper includes a non-circular evaluation — say, a baseline built from multiple corpora, or a transfer test to a different field — then the framework would be a lot more convincing.\n\nSo my honest position: the idea is worth taking seriously, and the paper deserves a proper peer review to see whether the evidence matches the ambition. I would not desk-reject it. But I also wouldn't cite it or put it in a reading group until I've seen the full text and the claims hold up. The stress-test's concern is legitimate, and it's the load-bearing question the referees need to push on.\n\nIf you're deciding whether to engage, I'd say: get the full version, check how they define recall and precision, and see if there's any external validation. If that's there, this could be a solid methods paper for the evaluation community. If not, it's a well-written framework with a circular proof.\n\nBest,\n[Your name]","headline":"Plausible benchmark-construction idea, but abstract-only review means the circularity risk is unresolved; deserves peer review to test the load-bearing generalization claim.","tokens_in":1607,"tokens_out":2383,"would_cite":false,"duration_ms":23095,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that piling more data into domain-specific LLM benchmarks is not always the best route; a comprehensiveness-and-compactness procedure yields a better academic benchmark and is claimed to transfer to any field.","keywords":["LLM evaluation","domain-specific benchmarks","benchmark construction","data scaling","comprehensiveness","compactness","PolyBench","Comp-Comp"],"falsifier":"Apply Comp-Comp to a second domain, such as legal or medical text, and compare its benchmark against a much larger, scale-built baseline on held-out expert judgments. If the larger baseline yields equal or better precision and recall, or if the compressed benchmark fails to predict model performance on the domain's full breadth, the domain-agnostic claim is falsified. Alternatively, if a randomly sampled subset of the same university corpus matches PolyBench's predictive power, the compactness step is doing no work.","tokens_in":735,"feed_emoji":"📊","tokens_out":5623,"duration_ms":49562,"temperature":0.7,"pith_summary":"This paper challenges the default principle of data scaling in building benchmarks for specialized LLM domains. It argues that merely enlarging corpora or question sets does not reliably improve how well a benchmark measures a model's true capabilities, and may add redundancy and noise. The proposed Comp-Comp framework instead balances comprehensiveness, defined as covering the full semantic breadth of the domain, against compactness, defined as removing redundancy to sharpen precision. Applied to a case study at a single university, the framework produced PolyBench, a large-scale academic benchmark. The authors contend that the procedure is domain-agnostic and can be ported to other specialized fields.","feed_headline":"Balanced benchmarks beat raw data scaling for LLMs","feed_subtitle":"A comprehensiveness-and-compactness framework builds academic benchmarks with less data.","key_machinery":"Comp-Comp, an iterative benchmarking framework built on two principles. Comprehensiveness ensures semantic recall by forcing coverage of the domain's full breadth; compactness improves precision by reducing redundancy and noise. The framework's work is to balance these two pressures during benchmark construction, rather than simply accumulating more data.","core_discovery":"The central claim is that benchmark quality for domain-specific LLM evaluation is governed by the precision–recall trade-off of semantic coverage, not by raw scale. Comprehensiveness is the recall side: the benchmark should span the full breadth of the domain so that no important capability is left untested. Compactness is the precision side: redundant or noisy items should be pruned so that the benchmark's scores reflect genuine domain ability rather than repeated or trivial material. The authors implement this as Comp-Comp, an iterative construction procedure, and instantiate it as PolyBench for academia. They assert that the same procedure is domain-agnostic and readily adaptable to other","pith_inferences":["The paper does not validate the domain-agnostic claim with a second domain; a natural test is to run Comp-Comp on, say, legal or medical text and compare precision and recall against a scale-built benchmark.","Reading comprehensiveness and compactness as the two axes of a precision–recall curve suggests a quantitative way to rank benchmarks that the paper leaves implicit.","If the framework transfers, the practical payoff is that domain-specific benchmark construction shifts from data collection to iterative curation, potentially lowering the cost of reliable evaluation."],"forward_implications":["Benchmark builders should no longer treat data scaling as the default for domain-specific evaluation; balanced coverage is the controlling factor.","PolyBench becomes available as a large-scale academic benchmark with reduced redundancy, for evaluating LLMs across university-level subjects.","The precision–recall framing gives a common language for comparing the quality of any two domain benchmarks.","If the domain-agnostic claim holds, the same iterative procedure could yield compact high-precision benchmarks in legal, medical, and other specialized fields."],"supporting_citations":[],"fun_headline_variants":["Benchmark smarter, not bigger: Comp-Comp's precision-recall balance","Comp-Comp: Precision and recall over raw scale for LLM benchmarks","LLM benchmark quality: Less data, more comprehensiveness and compactness","PolyBench: Building academic benchmarks with precision and recall","Comp-Comp: A domain-agnostic way to build LLM benchmarks"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole approach depends on the assumption that one university's academic corpus represents the full semantic breadth of academia, and that the same iterative recipe will transfer to any specialized field; if that corpus carries hidden institutional or disciplinary biases, the measured precision and recall gains would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark smarter, not bigger: Comp-Comp's precision-recall balance","Comp-Comp: Precision and recall over raw scale for LLM benchmarks","LLM benchmark quality: Less data, more comprehensiveness and compactness","PolyBench: Building academic benchmarks with precision and recall","Comp-Comp: A domain-agnostic way to build LLM benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000894,"raw_usage":{"total_tokens":3679,"prompt_tokens":723,"completion_tokens":2956,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":2872}},"tokens_in":467,"tokens_out":2956,"duration_ms":20777,"temperature":1.0,"reasoning_tokens":2872,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:10:12.369419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply Comp-Comp to a second domain, such as legal or medical text, and compare its benchmark against a much larger, scale-built baseline on held-out expert judgments. If the larger baseline yields equal or better precision and recall, or if the compressed benchmark fails to predict model performance on the domain's full breadth, the domain-agnostic claim is falsified. Alternatively, if a randomly sampled subset of the same university corpus matches PolyBench's predictive power, the compactness step is doing no work.","supporting_citations":[],"review_version":1}