{"id":"464168f3-d30f-4b61-b404-0b7cebd89bbe","arxiv_id":"2607.28636","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.","lead":"A second AI that reads the first AI's reasoning can fix biased answers, but only if the second AI is chosen for the specific bias at play. The authors' selection rule reaches 0.884 accuracy on biased test sets, beating any single fixed auditor.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix K.3's audit prompt contradicts G.3: clean prompt plus bias advisory vs. biased prompt without warning. If K.3 reflects the actual protocol, the selector's reported gains are confounded.","rationale":"The reader flagged the K.3/G.3 conflict in the rationale but placed the weakest assumption on known bias type and calibration noise. My read makes the conflict the load-bearing concern: if K.3 reflects the actual protocol, the comparison between chain accuracy and no-audit baseline is unfair because the auditor receives a different (cleaner) input and an explicit bias hint. This directly threatens the headline 0.884 vs. 0.824 vs. 0.805 result, not just the uncertainty around it. The paper deserves credit for its calibration/test split, the gamma=0 ablation showing the e-term matters, the released code, and the honest limitation about the +0.4 pp authority effect being indistinguishable from zero. Those do not resolve the protocol contradiction. The concrete test is straightforward and within the authors' existing setup: rerun the same chains under G.3-prompts and report both. I keep the reader's conditional verdict, but the condition should be protocol alignment and re-estimation of e under that protocol, not merely confidence intervals or inference-time bias detection.","tokens_in":22460,"tokens_out":5026,"duration_ms":45694,"concrete_test":"Re-run the four biased slices of Tables 5 and 7 using exactly the G.3 audit prompt: original (biased) prompt plus M1's full trace, with no bias advisory, on the same 25/25 calibration/test splits and temperature. If under G.3 the selector's sycophancy accuracy drops from 0.870 toward the always-on GPT-4o value (~0.630), or the overall 0.884 falls below 0.824, the central claim fails. Also run the K.3-style clean-prompt-plus-advisory condition and report both numbers, so the protocol discrepancy is resolved empirically.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—per-bias cross-family auditing beats fixed auditors—rests on the auditor receiving the original biased prompt plus M1's trace, with no bias warning. Section 2.1 states \"the auditor is not told that bias may be present,\" and Appendix G.3 says the original prompt \"including any injected bias cue—is passed verbatim to Mi.\" Appendix K.3 documents the opposite condition for a key correction: the auditor \"receives M1's trace labeled 'Advisory: Another analyst's reasoning (may contain biases)', but sees the clean prompt without bandwagon cue.\" That removes the bias cue from M2's input and explicitly flags bias. If K.3 reflects the actual experimental condition, chain accuracy is not measuring an auditor resisting a cue that M1 fell for; it is measuring an auditor answering an uncleaned question while being told the prior reasoning may be biased. The empirical-effectiveness term e would then be inflated for auditors that benefit from cue removal and bias flagging, and Table 7's sycophancy result (selector picks GLM-5, 0.870) could reflect input-protocol differences rather than audit skill. The paper presents K.3 as verbatim model output and calls it \"the key mechanism,\" so this is not a minor wording issue: two core protocol statements cannot both describe the same experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chain-of-Models (CoM), a sequential auditing pipeline in which a second LLM receives the first model's reasoning trace and produces a final judgment. Fixing the generator as Qwen2.5-72B-Instruct, the authors evaluate five cross-family auditors and one same-family ladder across four cognitive biases on MMLU-Pro and a subjective DPO sycophancy track. They report two main empirical findings: standalone bias resistance does not predict audit effectiveness (e.g., Kimi-K2.5 is most standalone-resistant but the weakest auditor on bandwagon and authority), and the best auditor is bias-specific (GPT-4o for bandwagon/authority/distraction; GLM-5 for factual sycophancy). They then propose a per-bias selector scoring auditors by functional DNA distance, standalone resistance, and calibration-split audit effectiveness, and claim 0.884 held-out accuracy across biased slices versus 0.824 for always-on GPT-4o and 0.805 for no audit.","tokens_in":22824,"tokens_out":7056,"duration_ms":65171,"significance":"If substantiated, the paper would make a useful contribution: it treats auditor identity as a design variable, provides a concrete selection rule, and releases code and configurations. The calibration/test split is a sound way to avoid selection-on-test, and the two headline findings are practically relevant. However, the protocol inconsistency in Appendix K.3 and the lack of uncertainty quantification currently prevent acceptance.","major_comments":[{"comment":"The stated protocol and Appendix K.3 contradict each other. Section 2.1 says the auditor 'is not told that bias may be present' and G.3 says 'the original prompt—including any injected bias cue—is passed verbatim to Mi.' Appendix K.3, described as 'the key mechanism,' instead shows M2 receiving the clean prompt without the bandwagon cue and with the trace labeled 'Advisory: Another analyst's reasoning (may contain biases).' If K.3 reflects the actual experimental condition, the reported chain accuracies are confounded by cue removal and explicit bias flagging, and the central claim that a cross-family auditor resists the same cue M1 fell for is not supported. The authors must reconcile this: either K.3 is outside the evaluation protocol and must be relabeled/replaced with a verbatim G.3 trace, or G.3 is wrong and the experiments must be redone.","section":"§2.1 / §G.3 vs. Appendix K.3"},{"comment":"The paper claims the selector combines functional diversity, standalone resistance, and empirical audit effectiveness, but the evidence shows the empirical-effectiveness term alone reproduces the result. Table 12's 'effectiveness-only' weighting (0,0,1) selects GPT-4o with the same 0.830 authority accuracy as the default (0.2,0.3,0.5); the diversity-only and resistance-only weightings collapse. No ablation tests whether removing d or r from the full score changes the selection. The three-axis claim is therefore not supported by the reported data. Please either demonstrate that d and r contribute beyond e (e.g., ablations that drop each term, or cases where e is unavailable) or reframe the selector as primarily e-based.","section":"§3.3, Eq. (4), Table 12"},{"comment":"The calibration split uses only n=25 examples per bias–dataset cell to estimate e(M1, Mi, b), and all reported accuracies are point estimates without confidence intervals. With 25 calibration examples, the auditor ranking can easily flip under sampling noise; the overall 0.884 vs. 0.824 improvement could be inflated by a lucky calibration draw, especially since the entire gain is concentrated in the sycophancy slice. The paper acknowledges this in the limitations but does not quantify the risk. Please provide bootstrap confidence intervals, a calibration-size sensitivity analysis, or a larger calibration set.","section":"§3.3, Table 7; Appendix F"},{"comment":"The statement that audit effectiveness 'cannot be predicted' from standalone bias resistance is stronger than the evidence supports. The paper demonstrates two counterexamples (Kimi-K2.5 and DeepSeek-V3) and shows that the best standalone model is not the best auditor, but this does not establish a general impossibility of any prediction from standalone resistance. A correlational analysis or a more tempered wording—e.g., 'standalone resistance is not a reliable predictor'—would match the data. This matters because the finding is one of the two headline claims motivating the selector.","section":"§3.2"}],"minor_comments":[{"comment":"Appendix C references 'the D-6 collapse on bandwagon (§3.2),' but Section 3.2 does not mention D-6 chains. Either add the D-6 results to the main text or fix the cross-reference.","section":"Appendix C"},{"comment":"Appendix F says 'An earlier trace-only detector pilot reported in §I had much lower recall on bandwagon and distraction,' but Appendix I does not describe such a pilot. The cross-reference appears to point to nonexistent content.","section":"Appendix F vs. Appendix I"},{"comment":"Several references are incomplete or placeholder-style, e.g., 'Jane Li and Others. 2024' and 'Nurit Cohen Inger and 1 others. 2026 ... Co-author list pending verification before camera-ready.' These need full author lists and verified details for a journal submission.","section":"References"},{"comment":"The K.3 trace uses GPT-4o-mini as M1, whereas the main experiments fix M1 as Qwen2.5-72B-Instruct. If this trace is intended as an illustrative example rather than a main-protocol run, it should be explicitly labeled as such.","section":"Appendix K.3"},{"comment":"The text says 'over 500 experiments (∼100,000 API calls),' but the stated design (9 models × 4 biases × 4 datasets × 50 plus chains) corresponds to a different count. Please clarify the arithmetic or describe how the count was derived.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The K.3 contradiction is the decisive issue. If the authors confirm that K.3 reflects the actual protocol, the main results are confounded and the manuscript should be rejected; if G.3 is correct, K.3 must be corrected and its provenance clarified. The remaining concerns—calibration noise, the overstrong 'cannot be predicted' claim, and the redundancy of the d and r terms—are fixable but should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the CoM paper. There is a genuinely useful result here: for Qwen-72B as the judge, the auditor that best corrects its biased traces is bias-specific, and the most bias-resistant standalone auditor isn't the best auditor. The Kimi-K2.5 inversion is a real, non-obvious finding. The per-bias selection rule with a calibration/test split is a sensible operationalization, and the ablations show the empirical-effectiveness term earns its place. That part holds up.\n\nBut there is a problem you need to know about before taking the numbers at face value. The stated protocol (Section 2.1, Appendix G.3) says the auditor receives the original biased prompt verbatim and is not told bias may be present. Appendix K.3, presented as the key mechanism and labeled verbatim model output, shows the opposite: the auditor sees the clean prompt without the bandwagon cue and an explicit \"Advisory: Another analyst's reasoning (may contain biases)\" label. You cannot read both as describing the same experiment. If K.3 reflects what was actually run, the headline 0.884 selector accuracy is measuring something different -- an auditor answering an unbias-cued question while told the prior analyst may be biased -- and the empirical-effectiveness term e would be inflated for auditors that benefit from cue removal and bias flagging. The paper must resolve this, either by correcting the protocol description or by re-running the experiment under the stated condition.\n\nSecondary points, smaller but worth saying. The \"standalone resistance cannot be predicted\" claim rests on two counterexamples; that's evidence against the simple heuristic, not a proof of unpredictability. The n=50 cells have no confidence intervals; the paper admits this in the limitations, but the macro comparisons in Table 7 are drawn from 25 test examples per cell, so the 0.884 vs 0.824 gap needs some uncertainty quantification. And the routing rule assumes the bias type is known at inference time; the paper is upfront about this, but it is a real constraint on deployment.\n\nWho gets value: researchers and engineers working on LLM-as-judge debiasing or multi-model inference. The idea of trace-level cross-family auditing is worth engaging, but the current version needs a serious revision before its numbers are usable.\n\nRecommendation: send it to review, but flag the protocol contradiction as a mandatory fix. If that is resolved, the core insight is worth publishing.","headline":"Real insight about cross-family auditors and the limits of standalone resistance, but a protocol contradiction in the appendix and an overstrong claim need fixing.","tokens_in":23249,"tokens_out":2039,"would_cite":false,"duration_ms":19151,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A different-family second model that reads a judge's reasoning trace corrects LLM bias better than any single fixed auditor.","keywords":["LLM-as-judge","cognitive bias","cross-model auditing","reasoning trace","auditor selection","functional diversity","sycophancy","bias mitigation"],"falsifier":"Withhold the bias label and infer it with a learned classifier, or bootstrap the 25 calibration examples per cell: if the chosen auditor frequently flips or the selector's margin over always-on GPT-4o disappears, the central claim fails.","tokens_in":22391,"feed_emoji":"⚖️","tokens_out":8681,"duration_ms":70444,"temperature":0.7,"pith_summary":"The paper tries to establish that the best way to correct a biased LLM judge is not to prompt it harder or swap in the most bias-resistant model, but to add a second model from a different training family that reads the judge's full reasoning trace and issues the final judgment. Across nine models from six families, four cognitive biases, and four factual datasets, it argues that this auditor must be chosen per bias type: the model most resistant to a bias when answering alone is often the worst at correcting another model's biased reasoning, and no single auditor wins on every bias. The paper operationalizes this as Chain-of-Models (CoM), a routing rule that scores candidate auditors by functional diversity, standalone bias resistance, and a calibrated audit-effectiveness estimate, and shows under a calibration/test split that the rule reaches 0.884 accuracy across four biased slices, beating the strongest fixed auditor (0.824) and the no-audit baseline (0.805). A sympathetic reader would care because this offers a deployable, weight-free way to improve LLM-as-judge pipelines in high-stakes domains where human evaluation does not scale.","feed_headline":"Per-bias auditor choice beats fixed LLM judge audits","feed_subtitle":"Routing biased queries to the right second model lifts accuracy to 0.884, beating fixed-auditor and no-audit baselines.","key_machinery":"The load-bearing mechanism is the two-model Chain-of-Models pipeline: M1 produces a reasoning trace and answer, then M2 receives the same prompt plus M1's full trace and re-judges, with no bias-specific instructions. Auditor selection is driven by a weighted score for each candidate: functional diversity (cosine distance between the generator's and candidate's behavioral fingerprint vectors), per-bias standalone resistance (candidate's own biased accuracy), and calibrated audit effectiveness e(M1, M2, b), the chain accuracy estimated on a held-out calibration split. The e term is what rules out the highly standalone-resistant Kimi-K2.5; conditioning on bias type b is what routes sycophancy t","core_discovery":"Auditor identity, not standalone quality, decides whether a second model corrects a biased judge. With generator fixed at Qwen2.5-72B-Instruct, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5, most resistant standalone on three biases, is the weakest auditor on bandwagon (0.400) and authority (0.500). The best auditor is bias-specific: GPT-4o wins bandwagon, authority, and distraction; GLM-5 wins sycophancy, where GPT-4o auditing is worse than no audit. A per-bias rule combining functional diversity, standalone resistance, and calibrated audit effectiveness reaches 0.884 test accuracy versus 0.824 for the best fixed auditor and 0.805 for no audit.","pith_inferences":["The resistance/effectiveness inversion suggests audit quality is about complementary failure modes between generator and auditor, not individual capability; a cheap trace-level disagreement score might predict it without costly calibration runs.","The rule assumes the bias type is known per query; a learned bias detector tested end-to-end would determine whether the method works on unlabeled in-the-wild prompts.","With only 25 calibration examples per bias–dataset cell, the effectiveness estimate is noisy; bootstrapping those cells would show how often the auditor ranking flips, and larger calibration sets could justify learning the scoring weights instead of fixing them.","The best auditor differing between factual and subjective sycophancy implies auditor rankings can be task-family-labile, so transferring a routing policy across domains without recalibration is risky."],"forward_implications":["A deployed LLM judge can be made more accurate on biased inputs without retraining or prompt engineering, by routing flagged queries to the auditor calibrated for that bias.","Standalone bias-resistance leaderboards should not be used to choose auditors; a model's own robustness says little about how well it corrects another model's biased reasoning.","Same-model or same-family self-audit is insufficient; the auditor's training lineage should differ from the judge's to catch correlated blind spots.","Per-bias routing beats always-on auditing at the same per-query cost, with the gain concentrated on the bias where the fixed auditor is weakest.","The selection rule transfers to subjective preference judging, but audit-effectiveness estimates must be re-estimated on the target domain."],"fun_headline_variants":["Auditor choice beats model strength in LLM bias audits","Bias-specific auditor beats any single LLM judge","Pick second model per bias: 0.884 audit accuracy","Not the best model, the right auditor for each bias","Matching auditor to bias lifts LLM judge accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The selection advantage depends on knowing each query's bias type at inference time and on the 25-example calibration estimate of audit effectiveness being accurate enough to rank candidate auditors reliably.","fun_headline_variants_meta":{"raw":{"variants":["Auditor choice beats model strength in LLM bias audits","Bias-specific auditor beats any single LLM judge","Pick second model per bias: 0.884 audit accuracy","Not the best model, the right auditor for each bias","Matching auditor to bias lifts LLM judge accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1187,"prompt_tokens":866,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":610,"tokens_out":321,"duration_ms":3392,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:50:04.110270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Withhold the bias label and infer it with a learned classifier, or bootstrap the 25 calibration examples per cell: if the chosen auditor frequently flips or the selector's margin over always-on GPT-4o disappears, the central claim fails.","supporting_citations":[],"review_version":1}