{"id":"089e80b1-f657-4dd6-8b1b-2e403223096d","arxiv_id":"2507.21419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A new Chinese government-domain benchmark uses a ModernBERT model trained on subjectively assigned, Beta-diffused relevance labels to score LLM responses.","lead":"GovRelBench adds 50 Chinese government-themed prompts plus an automated scorer, GovRelBERT, that rates an AI model's answer on a 0 to 1 scale of government relevance. The creators want a quick, reproducible way to compare government AI assistants, but the scoring labels are hand-picked by the authors and have not been checked against human judges yet.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GovRelBERT's 'accurate' relevance scores inherit their ground truth from the authors' own subjective category-to-score mapping (Eq.","rationale":"The reader's weakest assumption is exactly the load-bearing fragility I find: the hand-assigned hard scores and Beta diffusion define the ground truth, and no independent human validation is provided. My attack adds specificity by pointing to Eq. (4) and Table 4 as the point where subjectivity enters, and by clarifying that the human panel in Section 4.3.3 only compares two model variants and does not validate the target labels. I also emphasize the distribution shift between long source documents used for training and shorter LLM responses used at benchmark inference, which makes the missing validation more serious. I agree with the CONDITIONAL verdict: the dataset, code, and internal consistency are promising, but the central claim about accurate relevance scoring is not yet supported by external evidence. The recommended concrete test is direct human scoring of GovRelBench responses, which would either validate or falsify the benchmark's quantitative use. No change to the reader's verdict is needed; the same conditional posture remains appropriate.","tokens_in":10828,"tokens_out":3294,"duration_ms":43648,"concrete_test":"Release a validation set of roughly 400 GovRelBench responses (50 prompts x 8 models from Table 3). Have at least five independent annotators with government-domain expertise rate each response on a 0-100 relevance scale using the paper's relevance definition, without seeing GovRelBERT scores. Pre-register thresholds: inter-annotator ICC >= 0.7; Spearman rho >= 0.8 between mean human ratings and GovRelBERT scores; Kendall tau >= 0.7 between model rankings from human mean ratings and Table 3. If the thresholds are not met, the quantitative metric is not validated for LLM outputs and should be reported as dependent on the authors' scoring convention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GovRelBERT 'accurately compute[s]' government-domain relevance and that GovRelBench provides a quantitative metric for LLM evaluation. For that claim to hold, the target scores used in training must be a valid operationalization of relevance. But the entire regression target is fixed by Eq. (4): E[Beta] = alpha/(alpha+beta) = H_L, where H_L is the hand-assigned hard-score mapping in Appendix A.2/Table 4 (Government=1.00, Law_A=0.90, News=0.70, Soy Milk=0.10, Others=0.00), with additional undocumented boundary adjustments mentioned in Section 4.2. There is no inter-annotator agreement, no external expert elicitation for the 18 category scores, and no human rating of the actual artifacts the benchmark scores, namely free-form LLM responses. The only human panel in Section 4.3.3 compares GovRelBERT A versus B on selected high- and low-error samples; it does not check whether the target scores themselves are correct. Consequently, the high accuracy/F1 values in Table 1 (71% and 93% at tolerance 0.2) mostly show that GovRelBERT can reproduce the authors' labeling convention on held-out in-distribution documents. They do not establish that scores assigned to responses on the 50 GovRelBench prompts measure relevance. Because every Table 3 model ranking is produced by this scorer, any error in the mapping propagates into all benchmark comparisons. The paper's own Limitations section concedes the subjectivity and lack of universal applicability. This is an external-validity gap rather than an internal inconsistency, but it is load-bearing because the reusable benchmark requires third parties to accept the label mapping.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GovRelBench, a benchmark for evaluating LLMs in the Chinese government domain, together with GovRelBERT, a ModernBERT-based regression model that assigns a continuous \"government domain relevance\" score to text. The key methodological contribution is SoftGovScore, a two-step procedure that first maps 18 domain categories to hand-assigned hard relevance scores (e.g., Government=1.0, News=0.7, Soy Milk=0.1) and then diffuses these hard scores into soft training targets by sampling from Beta distributions whose mean equals the hard score (Eq. 4). GovRelBERT is trained with MSE loss on these soft targets. The paper reports that GovRelBERT substantially outperforms traditional classifiers and larger LLMs on the in-distribution test set (Table 1), and applies the model to score responses of several LLMs to 50 government-domain prompts (Table 3). The central claim is that GovRelBERT accurately computes government-domain relevance and that GovRelBench provides a quantitative metric for LLM evaluation in this domain.","tokens_in":11097,"tokens_out":4831,"duration_ms":54828,"significance":"If the relevance scores were externally validated, the paper would offer a useful and inexpensive evaluation tool for a relatively underserved domain: a reproducible 50-prompt test set, a fast 149M-parameter scorer, a clear training pipeline, and public code/data. The authors also provide an ablation (GovRelBERT B) and report the random seed, which are praiseworthy. However, the significance is currently limited by the fact that the entire evaluation loop is anchored to the authors' own subjective hard-score mapping, and none of the headline numbers are checked against independent human judgments of relevance. The contribution is therefore more a demonstration of a plausible training procedure than a validated benchmark.","major_comments":[{"comment":"The entire training-target distribution is anchored to the authors' hand-assigned category scores in Table 4, with small boundary adjustments described in §4.2, through Eq. (4), which fixes the Beta mean to the hard score. There is no inter-annotator agreement, no external expert elicitation protocol, and no validation against human judgments of actual LLM responses. Because Table 1's accuracy/F1 numbers are computed against these soft targets and Table 3's model rankings are produced by the resulting scorer, any error in this mapping propagates to every headline result. The paper's own Section 5 concedes that the mapping relies heavily on subjective judgment. This is load-bearing for the abstract's claim that GovRelBERT \"accurately computes\" relevance and for §3.3's claim that GovRelBench provides a \"quantitative metric.\" Please add external human relevance ratings on a sample of training texts and on the actual LLM responses, report agreement/correlation with GovRelBERT scores, and ideally include a sensitivity analysis over plausible alternative mappings.","section":"§4.2 / Appendix A.2 / Eq. (4)"},{"comment":"The evaluation of GovRelBERT is performed on a held-out subset of the same dataset whose soft labels were generated by the authors' Beta diffusion from their own hard scores. Thus the high accuracy/F1 values (e.g., 93.58% at tolerance 0.2) primarily show that GovRelBERT can reproduce the authors' labeling convention on in-distribution documents; they do not establish accuracy on arbitrary government-domain text, and they certainly do not validate the relevance scores assigned to free-form LLM responses. In addition, all values in Table 1 are single-run point estimates with no confidence intervals, repeated seeds, or significance tests, which makes the claimed superiority over much larger LLMs difficult to assess. Please provide multiple seeds/error bars and, more importantly, an independently annotated evaluation set with human relevance scores.","section":"§4.3.2 / Table 1"},{"comment":"The benchmark application in §4.4 scores free-form LLM responses with GovRelBERT, but no human rating of those responses is reported. The human panel in §4.3.3 compared GovRelBERT A and B on selected high- and low-error samples; it did not check whether the target scores themselves are correct. The LLM-judge panel in Table 2 is also weak evidence: five judges split 2-2-1, the per-instance vote count is 11 vs 9, and no judge prompts, sampling details, or agreement statistics are provided. Consequently, Table 3's model rankings are not grounded in an independent measure of government-domain relevance. Please add human judgments on the actual 50-prompt responses, with inter-annotator agreement and correlation between GovRelBERT scores and human scores, and report at least a few qualitative examples of scores assigned to generated responses.","section":"§4.4 / Table 3 / §4.3.3"},{"comment":"GovRelBench currently consists of only 50 self-authored prompts (45 authored plus 5 news-like), with no stated selection criteria, no analysis of coverage across government tasks, and no stability check. For a reusable benchmark, the authors should describe how prompts were constructed and selected, characterize the covered government scenarios, and test whether model rankings are stable under prompt rewording or repeated sampling. Without this, the benchmark's reliability as a quantitative evaluation instrument is unestablished.","section":"§3.3"}],"minor_comments":[{"comment":"The notation E[beta_L(α, β)] in Eq. (4) is confusing because β is both the distribution parameter and part of the distribution name; please use E[S_soft | α, β] or similar.","section":"§3.1 / Eq. (4)"},{"comment":"The boundary adjustments ('Government' to approximately 0.95 and 'Other' to approximately 0.15) are described only in prose; please provide the final hard-score mapping table so that the training targets are fully specified.","section":"§4.2"},{"comment":"Please define how Accuracy and F1 are computed for a continuous scorer with tolerance thresholds, and clarify that 'GovRelBERT A (Error Tolerance 0.1)' is an evaluation setting rather than a separate model.","section":"Table 1"},{"comment":"There are naming inconsistencies: the text mentions Qwen1.5-72B-Instruct and Claude-3 Opus while Table 3 lists Qwen2.5-72B-Instruct and claude-3-7-sonnet; please align names and versions. The speculative sentence about a hypothetical 115B TeleChat variant should be removed or clearly labeled as speculation.","section":"§4.4 / Table 3"},{"comment":"The optimal model being reached at approximately 0.7 epochs is unusual; please explain the early-stopping configuration, the patience setting, and why the optimum occurs before one full epoch.","section":"§4.3.1"},{"comment":"The text contains formatting artifacts such as 'M odernBERTA' and 'M odernBERTB'; please fix these. Also provide the exact prompts and protocol used for the LLM judges.","section":"§4.3.3"}],"recommendation":"major_revision","confidential_remarks":"The self-referential validation loop is the central obstacle: the hard-score mapping is both the training target and the evaluation reference, and no external human validation is provided. The core idea is reasonable and the paper is clearly written, but the benchmark's validity claim needs substantial additional evidence. I would be willing to accept after external human relevance ratings, error bars, and a stability analysis of the 50-prompt benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GovRelBench is a usable, honestly-scoped benchmark resource for Chinese government-domain LLM evaluation, but its core metric inherits the authors' subjective category-to-score mapping. The paper's own Limitations section concedes this, and the stress-test note is right: the load-bearing labels in Table 4 are not externally validated, so the high accuracy numbers show internal consistency, not that the scores measure relevance for third parties.\n\nWhat's actually new: the 50 Chinese government prompts plus the ModernBERT-based scorer are a real bundle for a niche but active deployment area. Prior government-domain evaluation work leans safety-focused, so a relevance-oriented benchmark is a reasonable gap to fill. SoftGovScore itself is not methodologically novel—Beta-diffused soft labels with fixed α+β are a known technique—but the paper does a decent job showing the direct-regression variant beats a classify-then-score ablation, and the small human/LLM panels, though tiny, support that internal comparison.\n\nThe soft spots are in the validity claims, not the execution. Every headline number in Table 1 is measured against targets sampled from the authors' own hard-score assignments (with minor boundary adjustments in Section 4.2 that are not fully specified). No inter-annotator agreement, no external elicitation for the 18 category scores, and no human ratings of the actual LLM responses scored in Section 4.4. The Table 1 comparison against binary classifiers is also a bit apples-to-oranges: regression with tolerance versus hard classification. None of this is fatal for a resource paper, but it makes the abstract's claim that GovRelBERT \"accurately computes\" relevance stronger than the evidence supports.\n\nProportionately: the central flaw is real and load-bearing for the benchmark's external validity, but it is honestly flagged and addressable. The math is simple and sound; the data collection is described well enough to reproduce; the promised code and dataset are a plus.\n\nWho this is for: teams building or evaluating Chinese government-domain LLM deployments who need a cheap, fast, automated relevance filter. A serious referee should send it back with requests for human-validated labels, error bars on benchmark scores, and a comparison with the existing GovEval framework. I'd take the resource with a grain of salt until the external validation arrives, but it deserves peer review rather than a desk reject.","headline":"A useful niche benchmark whose central relevance scores are the authors' own subjective mapping, honestly acknowledged in the paper's limitations.","tokens_in":11689,"tokens_out":1985,"would_cite":false,"duration_ms":25190,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that government-domain relevance is a continuous spectrum that a small encoder can score precisely, making it a reusable evaluation tool for LLMs.","keywords":["government domain relevance","LLM evaluation benchmark","soft labels","Beta distribution diffusion","ModernBERT","relevance scoring","Chinese government domain","regression evaluation"],"falsifier":"Give a fresh panel of domain experts, who had no role in Table 4, a new set of 50 government-domain prompts, have them rank LLM outputs by relevance, and check whether GovRelBERT's continuous scores order the outputs the same way; a near-zero correlation, or experts judging highly scored texts as off-topic, would refute the claim.","tokens_in":10561,"feed_emoji":"🏛️","tokens_out":7537,"duration_ms":75857,"temperature":0.7,"pith_summary":"The paper aims to make government-domain relevance a measurable, comparable quantity for large language models rather than a vague impression. It proposes GovRelBench, a set of 50 prompts that probe how well an LLM stays on government topics, plus GovRelBERT, a 149M-parameter encoder trained to score any text on a 0-to-1 government-relevance scale. The training recipe, SoftGovScore, turns coarse category labels into hand-set scores such as Government=1.00, News=0.70, and Others=0.00, then diffuses each score through a Beta distribution so the model learns graded relevance instead of crisp classes. If those scores are valid, GovRelBench gives researchers a fast, reproducible way to rank and compare LLMs for government-domain work, filling a gap left by safety-focused evaluations.","feed_headline":"149M relevance scorer outranks 32B LLM on government grading","feed_subtitle":"GovRelBench scores LLM responses on a 0-to-1 government-relevance scale built from hand-scored categories and Beta-diffused soft targets.","key_machinery":"The machinery is SoftGovScore's two-step label transform, pinned by Equation (4): each category label L is mapped to a hard score H_L in [0,1], then soft scores are sampled from a Beta distribution whose mean equals H_L because alpha/(alpha+beta)=H_L, with alpha+beta=20 to keep prior confidence uniform while letting variance shrink near 0 and 1 and grow near 0.5. GovRelBERT is a ModernBERT encoder (long bidirectional context, 8192-token window, 149M parameters) with a single regression head trained by mean squared error on those soft targets. The Beta diffusion is the load-bearing step: it turns discrete hand-set reference points into a smooth, uncertainty-aware training distribution.","core_discovery":"On the paper's own terms, the central discovery is that direct regression onto soft relevance scores works where classification fails: binary classifiers and embedding-based baselines stay near 24-50% accuracy at separating government from non-government text, while GovRelBERT reaches 71.5% accuracy within a 0.1 error tolerance and 93.6% within 0.2 on the held-out test set. The same scorer then ranks LLMs: Chinese-focused models such as deepseek-chat (0.7959 mean relevance) and Qwen2.5-72B-Instruct (0.7967) score well above Claude-3-7-sonnet (0.4533), with GPT-4o in between (0.7359). The authors conclude that domain relevance is best treated as a continuous spectrum and that a small encoder trained with SoftGovScore can serve as the evaluation tool for a government-domain benchmark.","pith_inferences":["SoftGovScore should transfer to other fuzzy domains such as law, finance, or medicine, since the recipe only assumes that anchor scores can be set; the paper has validated it only for government, so this is an extrapolation.","The variance column in Table 3 can double as a consistency diagnostic: a model with low variance stays on-topic across all 50 prompts, while high variance flags prompts where the model drifts, a use the paper does not develop.","A natural stress test would be to apply GovRelBERT to outputs from prompts outside the 50-set and compare against expert relevance ratings; strong agreement would show the label premise generalizes beyond curated test prompts."],"forward_implications":["GovRelBench supplies a reusable 50-prompt test: any LLM can be scored on government-domain relevance by feeding its responses to GovRelBERT, with no additional annotation required.","GovRelBERT's small size and speed make the benchmark practical to run repeatedly, for example during model selection or fine-tuning, on a single consumer GPU.","The reported rankings show the benchmark can separate model families: Qwen2.5-72B-Instruct and deepseek-chat lead with means near 0.80, while Claude-3-7-sonnet trails at 0.45.","Direct score regression (GovRelBERT A) is the recommended evaluation configuration, beating the classify-then-score variant in the 0.1-tolerance comparison and the human panel's 3:2 preference."],"supporting_citations":[{"why":"Supplies the ModernBERT encoder (149M parameters, 8192-token window) that GovRelBERT is trained from.","marker":"Warner et al., 2024"},{"why":"Defines soft labels, the conceptual basis for representing relevance as graded scores rather than binary classes.","marker":"Galstyan and Cohen, 2007"},{"why":"Establishes the BERT encoder architecture from which ModernBERT and the regression approach descend.","marker":"Devlin et al., 2019"},{"why":"Provides the m3e embedding-plus-classifier baseline that GovRelBERT is compared against in Table 1.","marker":"Chen et al., 2024"},{"why":"Represents the existing government-domain evaluation work focused on safety, the gap GovRelBench is designed to fill.","marker":"Liu et al., 2025"}],"fun_headline_variants":["Tiny 149M encoder beats 32B LLM on government relevance scoring","Small model wins: GovRelBench scores LLM government know-how","GovRelBench: soft scoring outperforms hard labels for government AI","149M GovRelBERT outranks 32B LLM on government-domain relevance","GovRelBench ranks DeepSeek and Qwen above GPT-4o on government relevance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-assigned hard scores in Table 4 (Government=1.00, Law_A=0.90, News=0.70, Soy Milk=0.10, Others=0.00) together with Beta diffusion at alpha+beta=20 are a valid ground truth for government-domain relevance; if they do not reflect expert judgment on free-form LLM outputs, every score in Tables 1 and 3 inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["Tiny 149M encoder beats 32B LLM on government relevance scoring","Small model wins: GovRelBench scores LLM government know-how","GovRelBench: soft scoring outperforms hard labels for government AI","149M GovRelBERT outranks 32B LLM on government-domain relevance","GovRelBench ranks DeepSeek and Qwen above GPT-4o on government relevance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000875,"raw_usage":{"total_tokens":3757,"prompt_tokens":888,"completion_tokens":2869,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":2766}},"tokens_in":504,"tokens_out":2869,"duration_ms":23883,"temperature":1.0,"reasoning_tokens":2766,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:47:17.859116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a fresh panel of domain experts, who had no role in Table 4, a new set of 50 government-domain prompts, have them rank LLM outputs by relevance, and check whether GovRelBERT's continuous scores order the outputs the same way; a near-zero correlation, or experts judging highly scored texts as off-topic, would refute the claim.","supporting_citations":[{"cited_title":"hard” and “soft","cited_arxiv_id":null,"evidence_quote":"Defines soft labels, the conceptual basis for representing relevance as graded scores rather than binary classes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the existing government-domain evaluation work focused on safety, the gap GovRelBench is designed to fill."}],"review_version":1}