{"id":"0650e3d3-f191-400b-9ef6-fed8177c4f2a","arxiv_id":"2502.09316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A deterministic n-gram benchmark for Japanese open-ended QA, built from LLM-generated reference answer sets, that reports a 0.9896 correlation with GPT-4o judge scores.","lead":"This paper introduces a Japanese benchmark that scores LLM answers with n-gram statistics and rules instead of human or LLM judges. It reports a 0.9896 correlation with GPT-4o-based judging, suggesting a cheap deterministic alternative for open-ended QA evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 50-model validation set includes the three LLMs used to construct the reference answer sets, so the reported r=0.9896 may be inflated by self-similarity.","rationale":"The reader's weakest assumption identifies the reference answer sets as the key risk: if they are biased or unrepresentative, the benchmark does not measure quality. I agree, but the most load-bearing and concretely testable form of this concern is the validation leak. The paper validates the benchmark by correlating with GPT-4o on 50 models, and the same three models used to generate the reference sets are among those 50. The benchmark's three metrics are all defined against these reference sets, so the reference models are not independent from the thing being measured. This is not an accusation of fraud; it is a standard methodological issue that can be quantified by recomputing the correlation on held-out models. The stability analysis in Figure 3, which shows high agreement among the three reference models, actually underscores the problem: it confirms the reference distribution is anchored to these three models, but it says nothing about how a model outside that small family would score. The GPT-4o sample answer used in the judge prompt is a human-written answer, not the reference set, so the correlation between the two measures is not trivially circular, but including the reference models can still inflate the apparent agreement. The proposed test is cheap, requires only the already-promised data, and would directly reveal whether the headline correlation is an artifact of the construction. In the meantime, the paper's other evidence, such as the diversity of questions and the deterministic low-cost nature of the metrics, supports a conditional acceptance. The reader's conditional verdict is appropriate; the specific test described here should be added as a condition. I do not see a reason to change the verdict level, so I mark it UNCHANGED.","tokens_in":17136,"tokens_out":5089,"duration_ms":55509,"concrete_test":"Using the per-model scores in the GitHub repository, recompute Pearson and Spearman correlations between the proposed benchmark and the GPT-4o judge after removing the three reference-generating models (stockmark-100b, PLaMo-100b, Swallow-MX-8x7b-NVE-v0.1). Report the change in r and a 95% bootstrap confidence interval for both the 50-model and 47-model sets. If the held-out correlation falls below 0.95 or its CI overlaps the correlations reported for MT-Bench and Nejumi (about 0.7), the headline r=0.9896 is not robust. Also perform leave-one-out analysis on these three models to identify influence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that deterministic n-gram metrics computed against LLM-generated reference answer sets approximate LLM-as-a-judge evaluation, with correlation r=0.9896 against GPT-4o on 50 models. However, the reference answer sets are constructed from outputs of stockmark-100b, PLaMo-100b, and Swallow-MX-8x7b-NVE-v0.1, and these three models appear in the evaluated set (e.g., Table 1). Fluency, Truthfulness, and Helpfulness all measure overlap with this reference distribution, so the three reference models necessarily score higher than held-out models that are not represented in the reference set. Including them in the validation correlation means the benchmark is partly measuring agreement with its own construction, which can inflate the headline r. The stability check in Figure 3 does not address this: it compares the three generating models with each other, showing only that the reference distribution is stable with respect to these three models, not that it represents the space of desirable answers or that the high correlation survives on truly held-out models. The reader's concern about reference-set representativeness is well founded, and the sharpest testable version of that concern is the contamination of the validation set by the reference-generating models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deterministic, judge-free benchmark for evaluating open-ended Japanese QA responses of LLMs. For each of 50 questions, a reference answer set is constructed by sampling one million responses from each of three Japanese LLMs (stockmark-100b, PLaMo-100b, Swallow-MX-8x7b-NVE-v0.1), followed by rule-based and 5-gram-frequency filtering and refinement to 1,000 responses. Three metrics are defined: Fluency (Eqs. 1-3), Truthfulness (Eqs. 6-9), and Helpfulness (manually defined rules), all computed against the reference set. The authors report a correlation of r=0.9896 between the benchmark and GPT-4o-as-a-judge across 50 models, and correlations above 0.7 with Japanese MT-Bench and Nejumi, concluding that the n-gram-based approach approximates LLM-judge evaluation at much lower computational cost.","tokens_in":17427,"tokens_out":6782,"duration_ms":61851,"significance":"If the claimed correlation holds on genuinely held-out models, this would be a useful contribution: deterministic, reproducible, low-cost evaluation for Japanese short-form QA, with public code and data. The paper is also transparent in releasing full results and in performing a reference-stability analysis (Fig. 3). However, the headline validation is compromised by the fact that the three reference-generating models are part of the 50-model correlation set, which can inflate r because the metrics measure overlap with the very distribution those models created. Consequently, the empirical support for the central claim is currently weaker than stated, and the Truthfulness metric is not grounded in factual ground truth.","major_comments":[{"comment":"The validation set used to compute r=0.9896 includes the three models that generated the reference answer sets: stockmark-100b, PLaMo-100b, and Swallow-MX-8x7b-NVE-v0.1 appear in Table 1, and Section 3.2 states that these three models produced the reference sets. Because Fluency (Eqs. 1-3) and Truthfulness (Eqs. 6-9) are calculated as n-gram overlap with those reference sets, the three generating models receive a constructional advantage over held-out models. Recomputing the GPT-4o correlation without these three models is necessary to establish that the benchmark predicts judge scores on models it was not built from; the paper should report both correlations and discuss any difference. The stability analysis in Figure 3 does not remedy this, since it compares scores obtained from reference sets built by the same three models rather than validating against held-out models or human judgments.","section":"5.2, Table 1, Section 3.2"},{"comment":"Truthfulness is operationalized as the proportion of character 3-grams whose frequency in the reference set exceeds 0.5%, under the assumption that rare n-grams indicate hallucination. This is not validated against factual ground truth; the filtering in Section 3.2.3 removes only 5-grams that appear once, so any incorrect statement that is shared and frequent across the three generating LLMs will survive and be scored as truthful. The paper itself acknowledges in Section 4 that the validity of the metrics was not demonstrated mathematically or theoretically. A spot-check of the reference sets by human annotators, or a comparison of Truthfulness scores against a factual QA benchmark for a subset of questions, is needed before the metric can be called a measure of truthfulness.","section":"3.3.2, Eqs. (6)-(9), Section 3.2.3"},{"comment":"The reported correlation r=0.9896 is given without a confidence interval or any measure of uncertainty, and the number of models is only 50. The GPT-4o judge scores themselves are stochastic; the paper does not state how many judge runs were averaged. Reporting a 95% confidence interval (e.g., via Fisher z-transformation or bootstrap) and, ideally, a bootstrap analysis that excludes the three reference-generating models would make the strength of the correlation interpretable.","section":"5.2"},{"comment":"The benchmark contains several free parameters whose values are chosen without reported sensitivity analysis: the target length of 100 characters and discount to zero at 150, the 3-gram frequency threshold of 0.5% in Eq. (6), the n-gram order range 1-10 in Eq. (3), and the reference-set size of 1,000 per question. If the correlation with LLM-as-a-judge is robust to reasonable variations in these parameters, the claim is stronger; if not, the metric is tied to the particular choices. A sensitivity analysis over at least the threshold and length parameters should be added.","section":"3.3, Eqs. (2), (6), (9)"}],"minor_comments":[{"comment":"The text states that 1.5 billion responses were generated across 50 questions; with one million responses per question from each of three models, the total is 150 million, not 1.5 billion. Please correct the number or clarify the intended meaning.","section":"3.2.1"},{"comment":"The description says 'proportion of 3-grams with a frequency of at least 0.5%' but Eq. (6) uses min(L*,0.005)/0.0005, which is not exactly a proportion; the paper's own text acknowledges 'almost the same'. Please replace the prose with the exact formula.","section":"3.3.2, Eq. (6)"},{"comment":"The correlation above 0.999 between single-model and ensemble reference sets is reported as a stability indicator, but a rank correlation or a comparison of per-question score differences would be more informative for whether individual model rankings change.","section":"Figure 3"},{"comment":"Some citations are incomplete or informal (e.g., 'Bard (Google, 2023)' and 'Vicuna (2023)'); please provide full references with authors and venues.","section":"Section 2 and References"},{"comment":"There is a typo 'muti-turn' for 'multi-turn'.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is transparent and the idea is promising, but the headline correlation is likely inflated by the inclusion of the reference-set generators in the validation set. The authors should also consider whether the benchmark measures 'truthfulness' at all without human-verified reference sets. If the reanalysis on held-out models still yields high correlation, the paper could be publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a practical, cheap, deterministic evaluation for Japanese open-ended QA, and the recipe is specific enough to reproduce. The headline r=0.9896 against GPT-4o-as-judge should not be taken at face value, though—the validation set includes at least three of the models that generated the reference answer sets, so some of that correlation may be self-similarity.\n\nWhat's actually new: the three-metric n-gram recipe (Fluency/Truthfulness/Helpfulness), the reference-set construction pipeline (1M sampled responses per question, rule-based and 5-gram-frequency filtering, hill-climbing reduction to 1000), and the 50-question Japanese benchmark. The underlying move—compare generated text to a reference distribution—goes back to MAUVE and n-gram evaluation generally; they cite MAUVE but not the fuller lineage. So it's a new combination and engineering effort, not a conceptual breakthrough. The paper is honest in Section 4 that the metrics' validity was not demonstrated mathematically or theoretically.\n\nWhat it does well: scoring is deterministic and cheap, appropriate for Japanese character-level analysis, and details including prompts are specified. The reference-set stability check (r>0.999 across the three construction models) is useful as an internal sanity check, though it does not show representativeness. Also worth noting: in Table 1, several held-out models (Claude, GPT-4o) score higher than the reference-set builders, so the benchmark is not simply ranking its own builders on top.\n\nSoft spots: the reference answers are LLM-generated with no human validation beyond one sample answer per question and rule filters; a hallucination that survives the filters becomes part of what 'truthful' means. Truthfulness itself is never checked against factual ground truth. The validation against GPT-4o has no confidence intervals or sensitivity analysis, and contaminating the 50-model set with the reference-generating models is a real concern—report the correlation excluding those three. The distributional-hypothesis framing is evocative but thin; their own conclusion admits stronger theory is future work.\n\nWho this is for: people doing Japanese LLM development and evaluation who want a fast regression signal. It deserves a serious referee and a request for sensitivity analysis rather than desk rejection. I'd bring it to reading group and would probably cite it as related work.","headline":"A practical, cheap, deterministic Japanese open-ended QA benchmark with a concrete recipe, but the headline correlation with LLM-as-judge needs a sensitivity analysis because the validation set includes the reference-set-building models.","tokens_in":17935,"tokens_out":3288,"would_cite":true,"duration_ms":34979,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deterministic n-gram benchmark for Japanese open-ended QA reproduces a frontier LLM judge's rankings at correlation $r = 0.9896$ across 50 models, offering a cheap, judge-free evaluation alternative.","keywords":["LLM evaluation","open-ended generation","n-gram metrics","distributional hypothesis","reference answer set","LLM-as-a-judge","Japanese benchmark","deterministic evaluation"],"falsifier":"Take the 50 questions and write high-quality answers that deliberately avoid every 3-gram present in the reference sets; if a panel of human judges (or a frontier judge model) rates them as correct and fluent while the Truthfulness score collapses, the central claim is refuted. A cheaper variant: insert one plausible wrong fact that happens to use frequent n-grams, such as answering the clock-overlap question with '23' instead of '22', and check whether the benchmark or the judge detects the error first.","tokens_in":16970,"feed_emoji":"📊","tokens_out":7507,"duration_ms":66274,"temperature":0.7,"pith_summary":"This paper proposes a benchmark that evaluates open-ended Japanese question answering without any human or LLM judge. The authors' claim is that three deterministic metrics—Fluency, Truthfulness, Helpfulness—computed from character-level n-gram statistics against a curated reference answer set, reproduce what a frontier LLM judge would say, with correlation $r = 0.9896$ across 50 models. If that claim holds, LLM evaluation for this class of tasks becomes cheap, deterministic, and usable on models that cannot follow instruction prompts. The paper positions this as evidence that the distributional hypothesis, that meaning correlates with word distributions, is still a workable tool for judging generated text, at least for short-answer tasks. The reference sets are built once per question from 1 million responses per generating model, then shrunk to 1,000 representative answers.","feed_headline":"Judge-free n-gram benchmark matches LLM judging at r = 0.99","feed_subtitle":"Deterministic Japanese QA scoring for 50 LLMs, no judge model or human rating needed.","key_machinery":"The load-bearing object is the reference answer set: for each question, a distribution of 1,000 representative Japanese answers, generated by three large Japanese LLMs, normalized to about 100 characters, and filtered to remove hallucinated or atypical phrasing. The scoring machinery is character-level n-gram matching against this distribution: Fluency sums the reference-set probability of the response's 1-10 grams, Truthfulness thresholds the frequency of the response's 3-grams at 0.5%, and Helpfulness checks manually defined key terms. Length normalization matters throughout, with a linear discount that peaks at 100 characters and reaches zero at 150 characters, so that longer but padded answers do not inflate scores. The n-gram frequencies act as a proxy for the full answer distribution, and the distributional hypothesis is what justifies treating overlap with that proxy as agreement with 'good' answers.","core_discovery":"The central discovery is that the statistical overlap between a generated response and a precomputed reference answer set captures most of what an LLM judge rewards. For each of 50 Japanese questions, the authors assemble a reference set of 1,000 answers by generating 1 million responses from each of three Japanese LLMs, filtering out hallucinations with question-specific rules and a 5-gram frequency filter, then keeping length-typical and diverse answers. A test response is scored by (1) Fluency, the summed occurrence likelihood of its character-level 1- through 10-grams in the reference set; (2) Truthfulness, the fraction of its character-level 3-grams that appear in the reference set with at least 0.5% frequency, treating rarer n-grams as hallucination signals; and (3) Helpfulness, hand-built keyword rules per question. The average of the three metrics correlates at $r = 0.9896$ with a frontier LLM judge's scores on 50 models and at about 0.74 with two established Japanese LLM leaderboards. The reference set itself is stable: swapping any one of the three generating models leaves benchmark scores correlated above $r = 0.999$. The authors read this as confirmation that the distributional hypothesis, the idea that words that appear in similar contexts carry similar meaning, remains sufficient for evaluating open-ended generation when the answer space is tight.","pith_inferences":["A testable extension is to build the same reference-set construction for English questions and check whether the $r = 0.99$ correlation with a judge model survives; the character-level n-gram choice would likely need to become token- or word-level.","Because the correlation target is the judge model, the benchmark inherits whatever biases that judge has; a stronger claim would compare the n-gram scores directly against human ratings.","The near-1.0 correlation with a judge that supposedly reads for meaning suggests that much of judge reward is driven by local text-surface regularity on short, constrained answers; if so, this benchmark is also a probe of judge behavior, not just of the models being scored.","Applying the same construction to multi-turn conversation or idea generation, where the answer space is wider, would stress the distributional assumption; the authors note this as future work, and the approach would likely need sentence- or discourse-level units."],"forward_implications":["Open-ended Japanese QA can be scored deterministically on a laptop, without paying for judge-model API calls.","The metrics work on base (non-instruction-tuned) models, which LLM-as-a-judge cannot meaningfully assess.","Scores are stable with respect to which LLMs produced the reference set, so the benchmark can be rebuilt with newer models without re-calibrating.","Some frontier models already outscore the reference set, indicating the benchmark tracks a manifold of acceptable answers rather than a single gold answer.","The paper expects the method to cover moderate future improvements but acknowledges that qualitatively different future outputs may escape n-gram detection."],"supporting_citations":[{"why":"Supplies the distributional hypothesis, the theoretical grounding for measuring answer quality through agreement of word distributions.","marker":"(Harris, 1954)"},{"why":"Provides the distributional evaluation framing and the notion of Type I errors from rare n-grams, motivating the frequency-based hallucination filtering.","marker":"(Pillutla et al., 2021)"},{"why":"Defines LLM-as-a-judge and MT-Bench, the evaluation methodology that this benchmark's scores are compared against as the baseline it must approximate.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["N-gram stats beat LLM judge: r=0.99, zero cost","Judge-free eval: n-grams match GPT-4o at r=0.99","Distributional hypothesis scores LLMs for free","N-gram fluency, truth, help: r=0.99 vs LLM judge","Cheap n-gram benchmark rivals LLM-as-a-judge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reference answer sets, built from three specific LLMs and filtered by rules and n-gram frequency, faithfully represent the distribution of correct and desirable Japanese answers; if they do not, the n-gram scores measure agreement with a possibly biased sample rather than quality.","fun_headline_variants_meta":{"raw":{"variants":["N-gram stats beat LLM judge: r=0.99, zero cost","Judge-free eval: n-grams match GPT-4o at r=0.99","Distributional hypothesis scores LLMs for free","N-gram fluency, truth, help: r=0.99 vs LLM judge","Cheap n-gram benchmark rivals LLM-as-a-judge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1344,"prompt_tokens":943,"completion_tokens":401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":559,"tokens_out":401,"duration_ms":3543,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:54:09.201141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 50 questions and write high-quality answers that deliberately avoid every 3-gram present in the reference sets; if a panel of human judges (or a frontier judge model) rates them as correct and fluent while the Truthfulness score collapses, the central claim is refuted. A cheaper variant: insert one plausible wrong fact that happens to use frequent n-grams, such as answering the clock-overlap question with '23' instead of '22', and check whether the benchmark or the judge detects the error first.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the distributional evaluation framing and the notion of Type I errors from rare n-grams, motivating the frequency-based hallucination filtering."}],"review_version":1}