{"id":"099b7825-4ae5-4704-ab71-1c0b64eb69d3","arxiv_id":"2411.17793","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A constitution-based, search-driven development framework for AI judge systems improves judged accuracy by up to 6.2% on commit message generation, with about 58% of general principles reused across five languages.","lead":"Researchers at Huawei and Queen's University propose a four-stage, constitution-based framework for engineering AI judge systems that grade AI-generated software outputs, and test it on commit message generation. The framework judge agrees with automatic quality metrics up to 6.2% more often than a simple holistic judge, but the experiment uses a proxy ground truth and a baseline that may not isolate the framework's value.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6.2% accuracy claim rests entirely on MetricVote (Eq. 1), an unvalidated majority vote of automatic metrics as ground truth; if MetricVote does not track human judgments, Table III measures agreement with reference-overlap metrics, not judging quality.","rationale":"This concern is load-bearing because the paper's only quantitative evidence for contribution #3 is the accuracy comparison in Section IV-C. If MetricVote is not a valid measure of which commit message is better, the reported accuracy values in Table III and the 6.2% gain do not establish that the framework improves judgment quality. The paper even invokes the known limitations of automatic metrics in Section II-C, and BLEURT is not calibrated for code-commit messages. The absence of any human validation or correlation analysis makes Eq. 3 a proxy-validity assumption rather than a measurement. In addition, the w/ vs w/o comparison varies scoring granularity at the same time as the constitution, so even under MetricVote the causal attribution to the framework is unclear. I agree with the reader's weakest assumption and see the CONDITIONAL verdict as appropriate: the framework proposal and challenge discussion have value, and the reuse percentages plus the user quote provide some qualitative support, but the central accuracy claim needs either human ground-truth validation or a sharply scoped claim about agreement with automatic metrics. A human annotation study on a few hundred pairs would settle whether the 6.2% reflects real judgment quality.","tokens_in":16814,"tokens_out":7956,"duration_ms":72299,"concrete_test":"Take a stratified random sample of about 300 pairs from P across the five languages and have three developers with commit-message expertise independently label which message in each pair is better, or tie, given the diff. Then (i) measure agreement between MetricVote and the human majority (e.g., Cohen's kappa), and (ii) recompute the accuracy of the w/ and w/o judges against the human labels instead of MetricVote. If MetricVote-human agreement is low, or the w/ advantage shrinks or disappears under human labels, the headline 6.2% accuracy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the framework-built judge 'outperforms' the baseline by up to 6.2% is operationalized in Eq. 3 as agreement between JudgeVote and MetricVote. MetricVote (Eq. 1) treats the original MCMD commit message as the better half of every pair via majority vote over BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT. The paper gives no evidence that this proxy tracks human judgments of commit-message quality. That omission matters because the paper itself argues in Sec. II-C that automatic metrics 'primarily focus on surface-level similarities' and can miss contextual relevance, coherence, and creativity; BLEURT is trained for machine-translation evaluation, not commit messages. A judge that simply correlates better with n-gram overlap would receive a higher 'accuracy' without being more accurate. The comparison is also confounded: the w/ condition changes both the content (constitution principles) and the scoring format (sum of per-principle scores), while the w/o baseline is one holistic 0-to-n score. Thus the 6.2% cannot be cleanly attributed to the framework's constitution or reuse. Without human labels, Table III is evidence only of agreement with a particular automatic-metric vote.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on industrial experience developing AI judge systems for Foundation Model-powered software (FMware). It catalogs nine challenges across defining judging requirements, developing judges, and evolving judges, and proposes a search-driven constitution-based framework with four stages: creation of a general constitution, specialization to contextualized constitutions, search for cognitive architectures, and evolution of the judge. The empirical evaluation is a case study on judging commit message generation, comparing an AI judge system built with the framework (per-principle scores summed) against one built without it (a single holistic score), using agreement with an automatic-metric majority vote as the accuracy measure. The paper claims up to 6.2% accuracy improvement and about 58% average reuse of general principles across five programming languages.","tokens_in":17201,"tokens_out":7278,"duration_ms":62749,"significance":"The challenge catalog and the idea of reusable, constitutional judging principles are potentially useful for practitioners building LLM-as-a-judge systems. The paper makes an explicit falsifiable claim that its framework improves judge accuracy and reduces development effort. However, the significance hinges entirely on the validity of the evaluation: the ground truth is an unvalidated majority vote of automatic metrics, the comparison is confounded with scoring granularity, and the reported accuracy values are below the standard chance level for binary comparisons. If these issues are addressed, the framework could be a meaningful contribution to the emerging practice of AI judge engineering; as presented, the empirical evidence does not establish the claimed benefits.","major_comments":[{"comment":"The ground-truth heuristic MetricVote is a majority vote over BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT, but the paper provides no evidence that this vote corresponds to human judgments of commit-message quality. This is especially problematic because Section II-C explicitly argues that automatic metrics 'primarily focus on surface-level similarities' and can miss contextual relevance, coherence, and creativity, and because BLEURT is trained for machine-translation evaluation rather than for commit messages. Without a human-validated ground truth, Eq. (3) measures only agreement with a particular automatic-metric vote, not judging accuracy.","section":"Section IV-B, Eq. (1)"},{"comment":"All reported accuracy values (37.0% to 46.1%) are below 50%, which is the expected agreement level for a two-alternative forced choice when ties are rare. The paper compares against a 'randomly assigning a class in a three-categorie classification (i.e., 33%)' baseline, but this is only appropriate if MetricVote and JudgeVote frequently return the tie value 0; the paper does not report the frequency of ties. If ties are rare, both judge systems are actually worse than random with respect to the chosen ground truth, which undermines the interpretation of these numbers as 'accuracy'. The authors should report the outcome distribution of Eqs. (1) and (2) and use the correct chance baseline.","section":"Section IV-C, Table III"},{"comment":"The comparison between the w/ and w/o systems is confounded. The w/ system scores each principle on a 0-1 scale and sums the scores, while the w/o system assigns a single holistic 0-to-n score. Thus the treatment differs in both the content (constitution principles) and the scoring format (per-principle aggregation vs. holistic). The paper's own result heading attributes the gain to 'fine-grained evaluation methods and scoring,' which suggests the 6.2% improvement cannot be attributed to the constitution framework per se. An ablation that holds the scoring format fixed (e.g., a holistic score using the same principles) is needed to isolate the framework's contribution.","section":"Section IV-B"},{"comment":"The reuse percentages in Table II are computed over the authors' own LLM-generated general and contextualized principles and do not measure development effort. No baseline for creating principles from scratch, no time or cost data, and no controlled comparison are provided; the single internal user quote is anecdotal. The abstract's claim of 'significant reduction in development effort' is therefore not supported by the reported evidence.","section":"Section IV-C, 'Productivity of developing AI judge systems'"}],"minor_comments":[{"comment":"'Claude 3.5 Sonnect' appears to be a typo for 'Sonnet'.","section":"Section II-C, first paragraph"},{"comment":"The description of the MCMD dataset says 'We leverage the MCMD dataset obtained by Shi et al. [58] that the authors added...' The antecedent of 'the authors' is ambiguous.","section":"Section IV-A"},{"comment":"The replication of Wu et al. [67] (16-shot result) is described only briefly; details such as prompt templates, model version, temperature, and sample selection would be needed for reproducibility.","section":"Section IV-B"},{"comment":"The paper states the accuracy improvement is 'up to 6.2%', but no confidence intervals or significance tests are reported for the differences in Table III.","section":"Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as an industrial experience report rather than a controlled empirical study. The central claims require a human-validated ground truth and a fair baseline; the current evidence is not sufficient for acceptance. The authors should also reconsider the chance-level comparison, as the reported accuracy values are below 50%."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but read the evaluation with care. The strength is the framework and the challenge taxonomy, drawn from real industrial experience. The four-stage lifecycle (general constitution, specialization, architecture search, evolution) is a sensible synthesis of existing ideas — constitutional AI, DSPy-style search, TDD — and the paper is honest that Stages III and IV are future work. The reuse numbers (53–64% of general principles reused across five languages) are a plausible productivity signal, and the five-language commit-message case study is real work: ~377 sampled diffs per language, ~71K pairwise comparisons.\n\nThe soft spot is exactly what the stress-test note says. The headline accuracy gain (up to 6.2%) is agreement with MetricVote, a majority vote of BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT. The paper gives no evidence that MetricVote tracks human judgments, and it even admits that automatic metrics focus on surface-level similarity. BLEURT is trained for MT evaluation, not commit messages. On top of that, the w/ vs w/o comparison changes two things at once: the constitution content and the scoring format (sum of per-principle scores vs a single holistic score). So the reported gain could be a granularity effect rather than a framework effect. No error bars, no human validation, no released artifacts.\n\nStill, I'd send it to peer review. The problem is real — building LLM judges today is ad hoc and expensive — and the framework gives practitioners a structured vocabulary for a messy process. The challenge taxonomy alone is a useful contribution. But the evaluation section needs major revision: either validate MetricVote against human labels, or reframe the result as \"agreement with automatic metrics\" and stop calling it accuracy. Ideally add a second task and at least one human-annotated set.\n\nFor your reading group: maybe. It is a good discussion piece on evaluation methodology. I would cite the challenge list in future work on LLM-as-judge engineering. A serious referee should engage with it, expecting heavy revision.","headline":"A useful industrial framework paper with a confounded evaluation: the 6.2% accuracy claim rests on an unvalidated automatic-metric vote and a comparison that changes scoring format alongside content.","tokens_in":102,"tokens_out":1814,"would_cite":true,"duration_ms":57701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A search-driven constitution-based framework for building AI judges turns judging requirements into reusable principles, and in a commit-message case study improves judgment accuracy by up to 6.2% while cutting development effort through…","keywords":["AI judge systems","LLM-as-a-judge","foundation model software (FMware)","constitution-based framework","commit message generation","evaluation metrics","knowledge-driven evaluation","software engineering"],"falsifier":"Take a random sample of the commit-message pairs from the case study, have human experts label which message is better, and compute whether the framework-built judge agrees with humans more often than the baseline judge does. If the framework's advantage over the baseline disappears under human labels, the central claim about judgment quality is not supported. A second check: run the framework judge and the baseline judge with the same scoring granularity (both per-principle or both holistic) to see whether the 6.2% gain persists.","tokens_in":16656,"feed_emoji":"⚖️","tokens_out":4421,"duration_ms":35351,"temperature":0.7,"pith_summary":"The paper argues that building AI judge systems—automated evaluators for software powered by foundation models—is an engineering problem with its own life cycle, and that current practice is ad hoc and costly. It proposes a search-driven, constitution-based framework that turns abstract judging requirements into reusable, context-specific principles, then uses those principles to construct and evolve the judge. In a case study on commit message generation spanning five programming languages, the judge built with the framework agreed with the paper's ground-truth heuristic more often than a judge built without it, by up to 6.2%, while reusing an average of 58% of general principles. A sympathetic reading takes this as evidence that organizing judging requirements into explicit, reusable criteria improves both the productivity and the quality of AI judge development.","feed_headline":"Reusable 'constitution' principles lift AI judge accuracy by 6.2%","feed_subtitle":"A four-stage framework turns judging requirements into reusable principles and beats ad-hoc judges on commit messages.","key_machinery":"The central object is the constitution: a set of generic judging principles derived from high-level requirements by an FM, refined through at least four rounds of critique and revision, then specialized into a contextualized constitution for a specific FMware context. Four stages carry the argument: (I) creating the general constitution, (II) specializing it, (III) searching for cognitive architectures (jury FMs, heuristics, metrics, and their interactions) using the contextualized constitution, and (IV) evolving the judge by tracing judgment flaws back to requirement bugs. The case study also relies on MetricVote (Eq. 1), a majority vote among BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT that serves as the ground-truth heuristic for which of two commit messages is better; the paper measures judge accuracy as agreement with this vote.","core_discovery":"The central claim is that transforming judging requirements into a reusable 'constitution' of generic principles, then specializing those principles to a target context, yields AI judge systems that are both cheaper to build and more accurate. In the paper's commit-message case study, the framework's judge scores each principle separately and sums the scores, while a baseline judge assigns one holistic 0-to-n score; across C++, C#, Java, Python, and JavaScript, the framework's judge matches the MetricVote ground truth more often, with gains from 1.0% (Java) to 6.2% (C#). The authors attribute the improvement to fine-grained, principle-based evaluation and to the reuse of roughly 58% of general principles across languages, which reduces the manual work of defining judging criteria.","pith_inferences":["The reported accuracy gain may partly come from the scoring granularity change (per-principle vs holistic) rather than from the constitution's content; a controlled comparison that keeps granularity fixed would isolate the contribution of reusable principles.","Because MetricVote is itself a majority of automatic metrics, the 6.2% improvement measures agreement with those metrics. A human-judgment validation on a sample of pairs would show whether the framework also improves agreement with people.","The 58% reuse figure comes from one commit-message domain; reuse rates could differ sharply for other FMware types, so the productivity claim is strongest for similar text-generation tasks.","The framework's search and evolution stages (III and IV) are not evaluated in the case study, so the 'search-driven' part of the framework is currently a proposal rather than a demonstrated result."],"forward_implications":["Judging requirements for similar FMware can be reused across teams and contexts, so AI judge development shifts from writing criteria from scratch to customizing a shared constitution.","Decomposing a judgment into per-principle scores (a fine-grained metric per criterion) can improve agreement with reference metrics compared with a single holistic score.","Because principles are traceable back to requirements, judgment errors can be debugged as requirement bugs rather than as isolated prompt or model issues.","New judging techniques can be added to the search space in Stage III, letting the framework automatically reconstruct a judging architecture without manual redesign.","If Stages III and IV work as designed, the framework would support continuous evolution of judges as foundation models and FMware change."],"supporting_citations":[{"why":"Supplies the constitution concept and the practice of refining principles through multiple rounds of critique and revision.","marker":"[11]"},{"why":"Provides the multi-language commit message dataset (MCMD) used as the evaluation corpus.","marker":"[59]"},{"why":"Defines the replicated commit message generation FMware and the four automatic metrics (BLEU, ROUGE-L, CiDEr, METEOR) that MetricVote builds on.","marker":"[67]"},{"why":"Adds BLEURT to the metric set, which the paper says reduces ties in the majority-vote ground truth.","marker":"[55]"},{"why":"Provides the fine-grained code change actions added to MCMD data points used in the case study.","marker":"[58]"},{"why":"Inspires the test-driven search approach in Stage III, where generated data points verify the constructed judging architecture.","marker":"[40]"}],"fun_headline_variants":["Constitution-based AI judges: up to 6.2% more accurate","Reusable principles lift AI judge accuracy by 6.2%","AI judges get 6.2% more accurate with reusable principles","Constitution-style judge rules: 6.2% accuracy gain, 58% reuse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on treating MetricVote—a majority vote among five automatic text metrics—as the correct answer for which commit message is better; if that proxy does not track what human reviewers judge to be good commit messages, the reported accuracy gain is agreement with metrics, not evidence of better judging.","fun_headline_variants_meta":{"raw":{"variants":["Constitution-based AI judges: up to 6.2% more accurate","Reusable principles lift AI judge accuracy by 6.2%","AI judges get 6.2% more accurate with reusable principles","Constitution-style judge rules: 6.2% accuracy gain, 58% reuse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2235,"prompt_tokens":836,"completion_tokens":1399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":1317}},"tokens_in":452,"tokens_out":1399,"duration_ms":9094,"temperature":1.0,"reasoning_tokens":1317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:54:02.765477+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the commit-message pairs from the case study, have human experts label which message is better, and compute whether the framework-built judge agrees with humans more often than the baseline judge does. If the framework's advantage over the baseline disappears under human labels, the central claim about judgment quality is not supported. A second check: run the framework judge and the baseline judge with the same scoring granularity (both per-principle or both holistic) to see whether the 6.2% gain persists.","supporting_citations":[{"cited_title":"RACE: Retrieval-augmented commit message generation,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-language commit message dataset (MCMD) used as the evaluation corpus."},{"cited_title":"Fake it till you make it: face analysis in the wild using synthetic data alone,","cited_arxiv_id":null,"evidence_quote":"Defines the replicated commit message generation FMware and the four automatic metrics (BLEU, ROUGE-L, CiDEr, METEOR) that MetricVote builds on."},{"cited_title":"On the evaluation of neural code summarization,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-grained code change actions added to MCMD data points used in the case study."}],"review_version":1}