{"id":"9d716e6c-3c5d-49ec-851f-2300c0881eab","arxiv_id":"2611.14861","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Jev is a cost-effective multilingual judge competitive with generative LLMs on most tasks but falls behind on knowledge-intensive answer verification, with lower consistency in low-resource languages.","lead":"This paper tests whether Jev, a cheap decision model that outputs judgment probabilities, works as a multilingual evaluator. It finds Jev competitive with expensive generative models on most tasks but weaker at verifying knowledge-intensive answers, especially in low-resource languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified: the paper's central claim is empirically supported and limitations are transparently documented.","rationale":"The reader's verdict is ACCEPT with high confidence. My stress-test finds no significant objection. The paper's methodology is sound: it compares Jev against two generative judges on four established multilingual benchmarks, with clear metrics and cost accounting. Limitations are explicitly discussed, including variability across tasks, lower cross-lingual consistency in low-resource languages, and limited gains from confidence routing. The central claim that Jev is competitive on most multilingual judging tasks at 1/31 to 1/61 of GPT-6.1's cost is directly supported by Tables 2–5 and the per-language breakdowns. The reader's weakest assumption (annotation quality) is a general benchmark limitation, not a paper-specific flaw; the paper does not rely on any novel annotations. The analysis of cross-lingual inconsistency includes a decomposition that addresses potential confounds. Thus I concur with the ACCEPT verdict.","tokens_in":19203,"tokens_out":3228,"duration_ms":25942,"concrete_test":"Recompute the accuracy and cost figures for Global-MMLU-Lite and MENLO using the same API versions and prompts but on a held-out subset of samples not used in the paper (e.g., 10% random sample from each benchmark) to verify that the relative ordering and cost ratios replicate within statistical error. This would confirm that the reported results are not artifacts of a particular sample draw or API transient behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"After careful review, I find no load-bearing concern that undermines the central claim that Jev achieves competitive multilingual judging performance at substantially lower cost than GPT-6.1. The paper conducts a systematic evaluation across four benchmarks, three judgment formats, and up to 47 language varieties. Cost comparisons are clearly stated with recorded token usage and API prices. Performance differences are reported in full per-language detail, and the authors explicitly note where Jev falls behind (knowledge-intensive answer verification) and where gains are modest (confidence routing on most tasks). The reader flagged annotation quality as a weakest assumption, but this is a standard limitation of benchmark evaluation that the paper acknowledges implicitly by using established human-annotated datasets with prior validation. The analysis of cross-lingual inconsistency includes a decomposition into new errors vs. corrections of English errors, addressing the natural concern that disagreement with English might reflect improved judgments. The paper provides reproducible prompt templates and inference settings. I see no internal inconsistency, hidden assumption that would flip the conclusions, or omitted validation that is clearly required. The claim holds as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper evaluates Jev, a decision model that directly outputs judgment probabilities, as a multilingual judge across four benchmarks (MEMERAG, Global-MMLU-Lite, MENLO, METAL) covering up to 47 language varieties and three evaluation formats (binary decisions, pairwise comparisons, graded ratings). Against generative LLM judges Qwen3.8-27B and GPT-6.1, Jev achieves competitive performance on most tasks at approximately 1/31 to 1/61 of GPT-6.1's cost, but falls behind on knowledge-intensive answer verification (Global-MMLU-Lite). Further analyses examine cross-lingual consistency, the sources of disagreement, confidence-based error detection, and practical recommendations on instruction language, confidence thresholds, and model routing.","tokens_in":19519,"tokens_out":2661,"duration_ms":20712,"significance":"If the results hold, the paper provides strong empirical evidence that a decision model can serve as a cost-effective multilingual judge for many evaluation tasks, potentially reducing the computational expense of automated evaluation. The thoroughness across 47 language varieties and multiple judgment formats, combined with explicit cost accounting and practical recommendations, makes this a valuable resource for practitioners. The analysis of cross-lingual inconsistency and confidence-based error detection is methodologically sound and offers actionable insights.","major_comments":[{"comment":"The paper describes Jev as 'competitive on most multilingual judging tasks' (Abstract, §1, §8). However, on MEMERAG (Table 3) Jev's BAcc (69.0) is 4-5 points below both Qwen and GPT-6.1, and on Global-MMLU-Lite it is 6.5 points below GPT-6.1. The term 'competitive' is not operationally defined (e.g., within a specified margin or statistical test). Given the cost advantage, the claim may be justified, but the paper would benefit from an explicit statement of what constitutes 'competitive' for each task, perhaps with respect to task difficulty or baseline variability. Without this, a reader could interpret the phrase as overclaiming.","section":"§4.2, Tables 3-5","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The cost comparison uses API prices for Jev and GPT-6.1 and an estimated API-equivalent cost for Qwen. The paper should note whether the Jev price is as of the date of the experiments and that pricing may change, as this directly affects the cost ratios.","section":"§3.2, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The results are reported without confidence intervals or significance tests. Given that comparisons are based on fixed benchmark samples, reporting bootstrap intervals or McNemar tests for binary tasks would strengthen the claims of differences.","section":"§4.2, Tables 3-5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The decomposition of cross-lingual judgment changes is informative. The paper should clarify whether the error categories (false acceptances, false rejections, etc.) are computed from Jev's binary decisions (threshold 0.5) or from its probability outputs. This is implied but not explicitly stated.","section":"§5, Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 8 reports accuracy and coverage at fixed confidence thresholds. For reproducibility, the paper should specify the exact set of thresholds used and whether confidence is defined as max(p,1-p) for binary tasks (as in Appendix G.2). The current text is clear but could cross-reference the appendix more directly.","section":"§7.2, Table 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The cost calculation for Qwen uses Alibaba API pricing, but the paper does not state the date of query or whether it reflects batch vs. on-demand pricing. Adding a note would improve replicability.","section":"Appendix F","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"There are a few minor typographical errors (e.g., 'judgment' vs. 'judgement' is inconsistent with the style; the use of 'evaluation cost' in the abstract is split across lines in the preprint version). These do not affect comprehension but should be corrected in the final version.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper gives a clear, honest answer to a practical question: how well does Jev work as a multilingual judge compared to expensive generative models? The answer is “pretty well, with caveats,” and the evidence supports it.\n\nWhat’s new: this is the first systematic multilingual evaluation of Jev. Prior work focused on English. The paper covers 47 languages, three judgment formats, includes cross-lingual consistency analysis and confidence-based error detection, and provides per-language results that let you see where the model works and where it doesn’t. The experimental design is thorough, the cost comparison is transparent ($2.64 total for Jev vs. $97.45 for GPT-6.1), and the authors are upfront about the limitations: Jev falls behind on knowledge-intensive answer verification and degrades in low-resource languages, mostly through new false acceptances. The practical recommendations (use English instructions, raise confidence thresholds when accuracy matters, route low-confidence samples to a stronger judge) are well-supported.\n\nSoft spots: the novelty is incremental. The paper applies existing evaluation methodology to a new model and language set—no new method or analytical tool. It relies entirely on benchmark annotations without independent quality checks; this is standard practice but limits depth. The confidence-based routing gains are modest (0.4–3.2 percentage points), which the authors acknowledge. The cost figures depend on current API pricing and may shift, but the relative ratios are what matter. Also, readers unfamiliar with Jev’s architecture will need to go elsewhere for background, which is fine for a short empirical report but worth noting.\n\nThe paper is for practitioners who need a cost-effective multilingual judge and want guidance on when Jev works and when it does not. Researchers will appreciate the per-language breakdowns. It is not a theoretical breakthrough, but it is a well-executed, honest evaluation of an important practical question.\n\nRecommendation: accept for peer review. The paper is solid, the claims are supported, and the limitations are transparent. Minor revision could add a short note on annotation quality or a sensitivity check on cost assumptions. No fatal flaws.","headline":"Competent practical evaluation of Jev across languages: the central claim holds, novelty is incremental, and the paper deserves a serious referee.","tokens_in":19969,"tokens_out":1581,"duration_ms":12898},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A decision model matches GPT-class judges on multilingual text evaluation at a fraction of the cost.","keywords":["multilingual evaluation","LLM-as-judge","decision models","probability outputs","cost efficiency","cross-lingual consistency","confidence calibration","automated evaluation"],"falsifier":"If an independent evaluation across the same four benchmarks (or a broader set) found that Jev's balanced accuracy or preference accuracy is consistently lower than a simple baseline (e.g., always predicting majority class), or that its cost advantage vanishes when including API call failures or retry overhead, then the paper's central claim of competitive performance at low cost would be invalidated.","tokens_in":20737,"feed_emoji":"🌐","tokens_out":2725,"duration_ms":21174,"temperature":0.7,"pith_summary":"The paper asks whether Jev, a decision model that directly outputs judgment probabilities, can serve as a multilingual judge instead of generative LLMs like GPT-6.1. Across four benchmarks covering up to 47 language varieties and three evaluation formats, Jev achieves competitive performance on most tasks at roughly 1/31 to 1/61 of GPT-6.1's cost. However, it falls behind on knowledge-intensive answer verification, particularly in low-resource languages where it tends to accept incorrect answers that it correctly rejects in English. The analysis also shows that Jev's probability outputs help distinguish correct from incorrect judgments, especially for binary tasks, offering a practical confidence signal for filtering or routing decisions.","feed_headline":"Jev matches GPT-class judges on multilingual text at 1/60 cost","feed_subtitle":"Decision model offers competitive evaluations across 47 languages but lags on knowledge verification.","key_machinery":"The central object is Jev, a decision model that outputs probability distributions over predefined judgment categories (e.g., Yes/No, rating levels) rather than generating free-form text. This probability output enables confidence-based error detection and makes inference cheap, as it avoids autoregressive token generation. The paper compares Jev against generative models using the same prompts and benchmarks, leveraging its structured API and probability outputs for downstream analysis.","core_discovery":"Jev, a decision model, is competitive with generative LLM judges (Qwen, GPT-6.1) on most multilingual judging tasks at approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, but trails GPT-6.1 on knowledge-intensive answer verification. Across binary decisions, pairwise comparisons, and graded ratings in up to 47 languages, Jev achieves the highest balanced accuracy on METAL hallucination detection (85.9%) and the highest macro-F1 on MENLO ratings (33.3%), while its cross-lingual consistency (93.4% agreement with English) exceeds Qwen but is lower than GPT-6.1, especially in low-resource languages. The primary driver of inconsistency in low-resource languages is false acceptance of incorrect answers that Jev correctly rejects in English.","pith_inferences":["The paper's results suggest a practical cascade: use Jev for low-cost initial judgments and escalate only low-confidence samples to a generative model. This could preserve cost savings while mitigating the knowledge-intensive weakness, but the limited gains reported indicate that the models share similar failure patterns, reducing the benefit.","The analysis of cross-lingual judgment changes reveals that Jev's errors in low-resource languages are predominantly false acceptances, hinting that the model's calibration or prior probabilities may be skewed toward acceptance in unfamiliar language contexts. Testing this hypothesis with synthetic data could inform targeted fine-tuning.","The confidence signal is weaker for graded ratings (AUROC 0.550–0.599) than for binary tasks, suggesting that probability-based error detection may require reformulation (e.g., using distributional distance metrics) for multi-class judgments.","The paper's recommendation to use English instructions by default, combined with the observed cross-lingual inconsistency, implies that the model's understanding of evaluation criteria is language-independent but its application to content varies; this could be leveraged to design language-agnostic evaluation templates."],"forward_implications":["If Jev is used as a multilingual judge, organizations can perform large-scale evaluation across many languages at a fraction of the cost of generative LLMs, making automated quality assessment accessible for lower-budget settings.","On tasks where Jev competes with GPT-6.1 (e.g., pairwise preference, hallucination detection), it can serve as a drop-in replacement, freeing resources for more complex evaluations.","Confidence-based filtering can improve accuracy on binary verification tasks (e.g., Global-MMLU-Lite accuracy rises from 89.57% to 99.18% when retaining only samples with confidence >= 0.95), though coverage drops substantially.","Routing low-confidence Jev judgments to a stronger model like GPT-6.1 yields modest gains on most tasks, suggesting limited room for improvement when errors are shared across models.","The finding that Jev accepts incorrect answers more often in low-resource languages implies that multilingual robustness requires attention to language-specific biases, even in models that are otherwise consistent."],"supporting_citations":[{"why":"Defines the Jev model (jev-1.13.0) that is the central object of study, supplying the model's architecture and API for probability-based judgments.","marker":"TypeSafe AI, 2026"},{"why":"Provides prior evidence that Jev can serve as a competitive judge in English, motivating the extension to multilingual settings and informing the methodology for confidence-based escalation.","marker":"Li et al., 2026b"},{"why":"Compares decision models with generative judges under shared rubric evaluation, providing a baseline for Jev's performance and highlighting shared failure modes.","marker":"Rao and Callison-Burch, 2026"},{"why":"Supplies the MEMERAG benchmark for multilingual RAG faithfulness evaluation, used to assess Jev's binary sentence-level judgments across five languages.","marker":"Cruz Blandón et al., 2025"},{"why":"Provides the Global-MMLU-Lite benchmark with parallel multilingual multiple-choice questions, enabling cross-lingual consistency analysis and answer verification evaluation.","marker":"Singh et al., 2025"},{"why":"Supplies the MENLO benchmark for response quality across 47 language varieties, used for pairwise comparison and graded rating tasks.","marker":"Whitehouse et al., 2026"},{"why":"Provides the METAL benchmark for summary evaluation in ten languages, supporting hallucination detection and three-dimensional rating tasks.","marker":"Hada et al., 2024"}],"fun_headline_variants":["Jev rivals GPT-6.1 for multilingual judging at 1/60 cost","Jev matches most GPT-6.1 judging tasks across 47 languages","Jev: decision model matches generative judges on multilingual tasks, lags on knowledge","At 1/60 cost, Jev competes with GPT-6.1 on multilingual judging but lags on knowledge"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The human annotations used as ground truth across all four benchmarks are assumed to be correct and representative, but the paper does not audit annotation quality, inter-annotator agreement, or cultural representativeness beyond the benchmarks' original validation, so any noise or bias in gold labels directly propagates into all reported accuracy metrics.","fun_headline_variants_meta":{"raw":{"variants":["Jev rivals GPT-6.1 for multilingual judging at 1/60 cost","Jev matches most GPT-6.1 judging tasks across 47 languages","Jev: decision model matches generative judges on multilingual tasks, lags on knowledge","At 1/60 cost, Jev competes with GPT-6.1 on multilingual judging but lags on knowledge"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"cost":0.000325388,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000325388,"upstream_inference_prompt_cost":9.1868e-05,"upstream_inference_completions_cost":0.00023352},"total_tokens":1695,"prompt_tokens":861,"completion_tokens":834,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":737}},"tokens_in":861,"tokens_out":834,"cost_source":"openrouter_usage","duration_ms":7015,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":737,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T21:33:38.511305+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If an independent evaluation across the same four benchmarks (or a broader set) found that Jev's balanced accuracy or preference accuracy is consistently lower than a simple baseline (e.g., always predicting majority class), or that its cost advantage vanishes when including API call failures or retry overhead, then the paper's central claim of competitive performance at low cost would be invalidated.","supporting_citations":[{"cited_title":"Proceedings of the 63rd Annual Meeting of the","cited_arxiv_id":null,"evidence_quote":"Supplies the MEMERAG benchmark for multilingual RAG faithfulness evaluation, used to assess Jev's binary sentence-level judgments across five languages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Global-MMLU-Lite benchmark with parallel multilingual multiple-choice questions, enabling cross-lingual consistency analysis and answer verification evaluation."},{"cited_title":"METAL: Towards Multilingual Meta-Evaluation","cited_arxiv_id":null,"evidence_quote":"Provides the METAL benchmark for summary evaluation in ten languages, supporting hallucination detection and three-dimensional rating tasks."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}