{"id":"a9609d5d-c6de-4394-8c80-7c2bfb33e95c","arxiv_id":"2508.02029","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A risk score built from confidence and vote entropy is proposed for triaging LLM qualitative coding, but the central R-squared=0.979 result is inflated by the definitions of agreement and diversity.","lead":"Eight LLMs coded interview segments, and the paper uses their confidence and vote diversity to triage which coding decisions need human review. The headline result, that these two signals explain 97.9% of inter-model agreement, is largely mechanical because diversity and agreement are computed from the same binary votes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that diversity added to confidence explains agreement with R²=0.979 is largely tautological: with binary votes, agreement A=max(p,1-p) and diversity d (binary entropy) are both deterministic functions of vote share p, so d already determines A at the item level.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing flaw. With binary votes, diversity (normalised Shannon entropy) and agreement (max of vote share and its complement) are both symmetric functions of the vote share p. Since binary entropy is strictly decreasing as p moves from 0.5 toward 0 or 1, while agreement strictly increases, the mapping between d and A is bijective on the support. Thus any claim that adding diversity to confidence explains agreement almost perfectly is not a statement about LLM behaviour but a restatement of how the variables are defined. The authors' own Methods section reports that an exhaustive grid search over the risk-score weight found the in-sample optimum at w*=0, i.e., pure diversity; the 0.6/0.4 weight is justified only by external transfer, not by the primary corpus. This is additional evidence that the 'dual signal' is really a single, definitional signal. There is also a secondary issue: the headline regression is fit on ten category-level points (df=7), and although bootstrap and item-level checks are provided, they inherit the same functional dependence and cannot rescue the interpretation. The cross-domain validation is similarly affected because the enhanced R² gains come from adding d, which is again a transform of the agreement being predicted. None of this implies the triage workflow is useless as a heuristic: routing on vote entropy is a reasonable way to find contested items. But the paper's central scientific claim—that confidence plus diversity calibrates AI judgement and achieves R²=0.979—is not supported as stated. The verdict should remain REJECT; a revision could reframe the contribution as an ensemble-triage heuristic and validate it against expert-adjudicated error on harder tasks.","tokens_in":12178,"tokens_out":5751,"duration_ms":70528,"concrete_test":"Use the public replication data to reconstruct p=n_yes/8 for all 710 item-level points and compute item-level A and d from the published vote counts. (1) Plot A against d for every item; if every point lies on the deterministic binary-entropy curve, the functional dependence is confirmed. (2) Refit the main regressions with d as the only predictor and also with a flexible spline in d; if d-only R² matches the reported 0.979 or the spline reaches near 1, the dual-signal result is tautological. (3) Re-run the triage evaluation with the risk score replaced by S=d alone (pure diversity) and compare auto-accept error rates against the 0.6/0.4 weighting. If the d-only version performs equally well or better, confidence contributes nothing beyond the vote distribution, and the headline claim requires reframing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's headline R²=0.979 is not an empirical discovery about LLM calibration; it is a mathematical consequence of how agreement and diversity are defined. For each coding point, A=max(p,1-p) and d=-p log2(p)-(1-p)log2(1-p), with p=n_yes/8. Because binary entropy is strictly monotone in |p-0.5| and A is also strictly monotone in |p-0.5|, d is a one-to-one deterministic function of A at the item level: knowing the vote distribution (or d) determines A exactly. The item-level regressions therefore regress a quantity on its own transform; the so-called dual signal contains no independent error information beyond the votes themselves. At the category level the same shared dependence on the underlying vote shares produces the near-perfect fit. The paper's own Methods section makes this concrete: an exhaustive grid search over risk-score weights found the in-sample optimum at w*=0, i.e., pure diversity, and the adopted 0.6/0.4 weight is justified only by cross-domain stability, not by the primary data. A proper test of the framework would need to show that d adds predictive power beyond the known functional relation, for example by comparing R² for Agreement~d alone against the reported dual regression, or by predicting expert-adjudicated error rather than A. As written, the central calibration claim is circular, and the three-tier workflow inherits this problem because it routes items using d, which is the same information as the agreement it purports to predict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a confidence-diversity calibration framework for LLM-based qualitative coding. Using 5,680 binary coding decisions from eight LLMs across ten thematic categories, it defines inter-model agreement A = max(p, 1-p) with p = n_yes/8 and model diversity d as the normalized Shannon entropy of the binary vote distribution. The authors report that mean self-confidence correlates with A (r = 0.82) and that a dual-signal regression of A on mean confidence and d achieves R^2 = 0.979, enabling a three-tier workflow that auto-accepts roughly 35% of coding points with reported residual error below 5% and reduces manual effort by 65%. Cross-domain validation on six benchmark datasets is presented as evidence of transferability, with kappa improvements of 0.20-0.78.","tokens_in":12536,"tokens_out":4638,"duration_ms":53588,"significance":"If the dual-signal relationship were an empirical discovery about LLM calibration, the framework would be practically valuable for qualitative research at scale: it would let analysts estimate inter-model agreement without ground truth and route low-risk items away from human review. The paper's reproducibility package (Harvard Dataverse DOI, run_all.sh, raw model outputs) is a clear strength. However, the central claim is undermined by a definitional circularity: for binary votes, both A and d are deterministic functions of the same vote share p, and d is strictly monotone in |p - 0.5|, so d is a one-to-one transform of A. The reported R^2 = 0.979 largely reflects this identity rather than any behavioral property of LLMs. The practical workflow routes on d, which is essentially the same information as the agreement it purports to predict, so the error-concentration results do not establish an independent error signal.","major_comments":[{"comment":"The definitions A = max(p, 1-p) and d = -[p log2 p + (1-p) log2(1-p)]/log2, with p = n_yes/8, imply a deterministic one-to-one relationship between A and d at the item level: both are strictly monotone functions of |p - 0.5|. Therefore the item-level 'dual-signal' regression regresses a quantity on a transform of itself, and the category-level R^2 = 0.979 largely inherits this identity. The paper never reports the d-only regression, so the claimed incremental contribution of confidence is unestablished. This is load-bearing because the central claim of the paper is that confidence plus diversity produces a dual signal that explains agreement almost completely; as written, the result is a mathematical consequence of the definitions, not an empirical finding.","section":"Results, 'Dual-signal mechanism eliminates residual disagreement'; Methods, 'Dual-signal calibration framework'"},{"comment":"The grid search over w in S = w(1 - c_bar) + (1 - w)d reports an in-sample optimum at w* = 0 (pure diversity, MAE = 0.076), and the adopted 0.6/0.4 weight is justified only by cross-domain stability (external delta MAE = 0.02). Combined with the identity between d and A, this means the primary-corpus evidence does not support the 'confidence-diversity' framing: the risk score S is effectively a function of d, i.e., of the same vote share that defines the target agreement A. A proper test of the framework would need to show that d adds predictive power beyond the known functional relation, for example by comparing R^2 for Agreement ~ d alone against the reported dual regression, or by predicting expert-adjudicated error rather than A.","section":"Methods, 'Dual-signal calibration framework'"},{"comment":"The category-level regression is fitted on n = 10 observations with two predictors (df = 7), which the paper itself acknowledges as low-powered. More seriously, the near-perfect fit is expected under the deterministic d-A relationship, so the reported R^2 = 0.979 and the bootstrap intervals in Supplementary Table 11 do not address the circularity. The claim that 'diversity corrects systematic biases left by confidence alone' (Results, 'Enhanced regression') requires a baseline that isolates d's contribution; no such baseline is reported in the main text, and the in-sample w* = 0 result suggests confidence contributes little beyond d.","section":"Results, 'Enhanced regression unifies confidence and diversity'; Limitations and Boundary Conditions"}],"minor_comments":[{"comment":"The audit sample sizes are internally inconsistent: 20% of 5,680 decisions is 1,136, not 1,242 as stated, and the green-zone audit of n = 242 does not match any clearly derived fraction of the 35% auto-accepted segments (20% of 35% of 5,680 is about 398). Please clarify the sampling scheme and reconcile these numbers.","section":"Methods, 'Human validation procedure'"},{"comment":"The text refers to a 'selected threshold of 4.5' in the efficiency-accuracy trade-off curve, but the risk score S in Eq. (1) lies in [0, 1]; presumably this is a typo for S < 0.45 or for a different quantity. Please correct the threshold value.","section":"Results, 'Three-tier workflow reduces manual review'; Figure 2D"},{"comment":"The caption for Figure 2C states that the horizontal line indicates the proportion of points selected for expert audit as 4.7%, which conflicts with the three-tier assignment of 50% of points to full review and 15% to light audit. Please clarify whether 4.7% refers to a stratified audit subsample rather than the review workload.","section":"Results, 'Three-tier workflow reduces manual review'; Figure 2C"},{"comment":"The definitions of A and d are given as inline formulas; numbering these equations would improve precision, since the paper later refers to 'Eq. (1)' for the risk score S but the agreement and diversity definitions are not numbered.","section":"Methods, 'Dual-signal calibration framework'"},{"comment":"References [38]-[41] appear in the bibliography but are not visibly cited in the main text; please reconcile the reference list with the in-text citations.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is circular: because d is a deterministic one-to-one function of A for binary votes, the high R^2 is not evidence about LLM calibration. The authors' own grid search showing w* = 0 reinforces this concern. A revision could potentially reframe the contribution as 'vote entropy as a triage heuristic,' but that would change the paper's central claim and the current framing is misleading. I would also note that several reporting inconsistencies (audit sample sizes, threshold of 4.5, Figure 2C caption) suggest the manuscript needs more careful checking before any resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the central result—mean confidence plus vote entropy explains 98% of variance in agreement—is not an empirical finding about LLMs. With binary votes, agreement A=max(p,1-p) and normalized Shannon entropy d are both deterministic functions of the same vote share p. d is a one-to-one transform of A, so regressing A on d is predicting A from itself. The paper's own grid search finding w*=0 (pure diversity best in-sample) is the tell. Second, the practical triage workflow—routing items by a risk score to auto-accept, light review, or full review—has heuristic value that is independent of the inflated regression claim.\n\nWhat the paper does well: the authors report a full reproducibility package on Dataverse, with code and data; the cross-domain validation on six public datasets is a serious attempt at transferability; and the limitations section is unusually candid, noting the n=10 category-level regression and the ceiling effect in one dataset.\n\nThe soft spots are substantial. The headline regression is circular, and the paper never reports the d-only baseline that would expose this. The category-level fit has n=10 with two predictors; the 0.6/0.4 weight was selected after inspecting the data, with the in-sample optimum at w*=0. The 65% effort-saving figure is built on the same circular signal—d is disagreement, so it is unsurprising that high-disagreement items concentrate errors. The item-level analysis in the supplement, if it includes the same structural identity, would be equally circular.\n\nThis is a useful heuristic for practitioners who want a cheap way to triage ensemble coding, but it is not a calibration method, and the claim 'diversity plus confidence predicts agreement' is not a discovery. A revised version that reframes the contribution as an ensemble-disagreement triage rule, adds a d-only baseline, and tests on tasks where the LLM is not already near ceiling could deserve publication in an applied venue.\n\nWho should read it: methodologists in qualitative research and anyone building LLM-assisted coding pipelines. It deserves a serious referee only if reframed; the current central claim would not survive review. I'd recommend the authors be told to fix the framing and the missing baseline.","headline":"The R²=0.979 claim is a mathematical tautology for binary votes; the practical triage heuristic is salvageable but the paper overstates the calibration result.","tokens_in":13034,"tokens_out":3620,"would_cite":false,"duration_ms":41884,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Confidence plus vote spread predicts AI coding agreement at 98%","keywords":["confidence calibration","model diversity","qualitative coding","inter-model agreement","ensemble uncertainty","LLM reliability","Shannon entropy","human-AI workflow"],"falsifier":"On the same 710 item-level decisions, fit agreement $A$ to the diversity index $d$ alone; if $R^2$ already approaches 0.979 with no confidence term, the dual-signal gain is an artifact of the shared vote share $p$; then repeat on a three-or-more-option coding scheme where $d$ and $A$ are no longer one-to-one, and an $R^2$ far below 0.979 would show the original result depended on the binary-vote identity.","tokens_in":11928,"feed_emoji":"✅","tokens_out":8786,"duration_ms":86372,"temperature":0.7,"pith_summary":"This paper asks when a label produced by a large language model can be trusted in qualitative coding tasks where human coders themselves often disagree. The authors propose that two observable signals—the average self-confidence of a panel of eight models and the spread, or diversity, of their votes, measured as normalized Shannon entropy—together predict how well the models agree with one another. Across 5,680 coding decisions in ten categories, the two signals explain 97.9% of the variance in inter-model agreement, compared with 87.5% for confidence alone. If this holds, researchers can estimate coding reliability without ground truth and route low-risk items away from human review: the proposed three-tier workflow auto-accepts about 35% of segments with under 5% error and cuts manual effort by about 65%.","feed_headline":"Confidence plus vote spread predicts AI coding agreement at 98%","feed_subtitle":"Without any ground truth, a panel's confidence and vote diversity let researchers auto-accept a third of segments under 5% error.","key_machinery":"The load-bearing object is the diversity index $d$, the normalized Shannon entropy of the binary vote distribution over the eight-model panel: $d=-\\sum_j p_j\\log p_j/\\log 2$, with perfect consensus at $d=0$ and a 4–4 split at $d=1$. Combined with mean confidence $\\bar{c}$, it forms a two-dimensional signal whose linear regression predicts agreement $A=\\max(p,1-p)$ almost exactly. The mechanism the paper identifies is error concentration: high-diversity, low-confidence categories concentrate 87% of all disagreements in just 50% of the data, while green-zone items carry under 5% error, which is what makes a three-tier risk workflow efficient.","core_discovery":"The central claim is that in accessible qualitative coding tasks, the agreement between LLMs can be read off from two cheap signals: the panel's mean self-confidence $\\bar{c}$ and the vote diversity $d$, defined as the normalized Shannon entropy of the binary vote distribution. With agreement defined as $A=\\max(p,1-p)$, where $p=n_{\\text{yes}}/8$ is the share of affirmative votes among eight models, the authors fit Agreement% $=30.24\\bar{c}-39.41d-54.63$, which explains $R^2=0.979$ of the variance at the category level, up from 0.875 for confidence alone. They interpret the residual pattern—underprediction in high-diversity categories and overprediction in near-consensus ones ($r=-0.94$)—as evidence that diversity corrects the systematic bias of overconfident unanimous panels. The paper then converts this regression into a risk score $S=0.6(1-\\bar{c})+0.4d$, partitions coding points into auto-accept, light-audit, and full-review tiers, and validates the dual-signal principle across six external datasets, reporting Cohen's $\\kappa$ improvements of 0.20–0.78. The authors emphasize that the near-perfect fit likely reflects the relative simplicity of these coding tasks for current LLMs, and that the method's real value is as a foundation for harder scenarios.","pith_inferences":["With exactly two vote options, the diversity index $d$ and the agreement $A$ are both deterministic functions of the same vote share $p$, so regressing $A$ on $d$ is to a large extent predicting a quantity from itself; the reported residual correlation ($r=-0.94$) and the high $R^2$ may be inflated by this operational identity rather than by a genuine metacognitive signal.","The paper's own grid search found the best internal cross-validated risk score at $w^*=0$ (pure diversity), which suggests the confidence term may contribute little once diversity is known; the 0.6/0.4 weighting is chosen for interpretability and cross-domain stability, not because confidence is the stronger empirical signal.","A decisive test of the framework is to apply it to coding schemes with more than two categories or to open-ended multi-label coding, where entropy and max-agreement are no longer one-to-one transforms; a sharp drop in explained variance would confirm that the binary-vote identity, not LLM behavior, drove the $R^2=0.979$ result.","If the identity effect dominates, the workflow's efficiency numbers should be reproducible with any vote source—including a single model perturbed by random prompts or even random labelers—which would reframe the contribution as an aggregation-statistics result rather than an LLM calibration result."],"forward_implications":["In coding tasks of comparable accessibility, researchers can estimate inter-model agreement from mean confidence and vote entropy without ground truth, reducing the need for extensive two-coder validation.","The three-tier workflow (risk score $S<0.25$ auto-accept, $0.25\\le S<0.45$ light audit, $S\\ge0.45$ full review) would auto-accept roughly one-third of segments with under 5% error and cut expert review load by about 65%, saving days on thousand-segment studies.","Across six public datasets spanning finance, medicine, law, moral reasoning, machine-translation quality, and natural-language inference, the dual-signal model raises Cohen's $\\kappa$ by 0.20 to 0.78 (mean +0.66), indicating transfer beyond the original interview corpus.","Because the dual signal is computed entirely from model outputs, it gives qualitative researchers a ground-truth-free calibration audit that can be tightened (green-zone cut-off $S<0.35$) for high-stakes domains such as medicine and law.","Expert adjudication of contested segments indicates that 87.6% of AI-majority decisions were correct against expert consensus, implying the original human codes—not the AI—harbored the errors; if this generalizes, LLM coding may exceed typical human-coder reliability."],"supporting_citations":[{"why":"Supplies the premise that human annotations are imperfect gold standards, which justifies using inter-model agreement as the calibration target.","marker":"[1]"},{"why":"Provides Cohen's kappa, the chance-corrected agreement metric that the paper uses for both human and AI reliability.","marker":"[11]"},{"why":"Documents that neural-network confidence is miscalibrated, motivating the need for a second error signal beyond self-confidence.","marker":"[12]"},{"why":"Extends the miscalibration finding to modern neural-network architectures, supporting the paper's overconfidence premise.","marker":"[13]"},{"why":"Gives the ensemble-diversity error-bound rationale that justifies using vote spread as an error cue.","marker":"[14]"},{"why":"Supplies the kappa interpretation bands the paper uses to characterize agreement levels as 'almost perfect'.","marker":"[16]"},{"why":"Provides quantitative diversity measures for classifier ensembles, the basis for the normalized Shannon entropy index $d$.","marker":"[25]"},{"why":"Supplies Fleiss' kappa, used for the three-expert adjudication of contested coding segments.","marker":"[26]"}],"fun_headline_variants":["AI self-confidence and vote spread predict coding agreement at 98%","Confidence and vote diversity from AI panels calibrate coding accuracy","Confidence plus disagreement lets AI-coding panels auto-accept 35% of segments","Dual AI signal cuts manual coding review by 65% while staying accurate","AI panel's own confidence and divergence gauge coding trust without ground truth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework treats vote diversity $d$ as an independent error signal, but with only two answer options $d$ is a purely mathematical transformation of the agreement $A$ it is used to predict, so the near-perfect fit partly measures the signal against itself.","fun_headline_variants_meta":{"raw":{"variants":["AI self-confidence and vote spread predict coding agreement at 98%","Confidence and vote diversity from AI panels calibrate coding accuracy","Confidence plus disagreement lets AI-coding panels auto-accept 35% of segments","Dual AI signal cuts manual coding review by 65% while staying accurate","AI panel's own confidence and divergence gauge coding trust without ground truth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3446,"prompt_tokens":1031,"completion_tokens":2415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":2320}},"tokens_in":647,"tokens_out":2415,"duration_ms":20177,"temperature":1.0,"reasoning_tokens":2320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:13:25.535705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same 710 item-level decisions, fit agreement $A$ to the diversity index $d$ alone; if $R^2$ already approaches 0.979 with no confidence term, the dual-signal gain is an artifact of the shared vote share $p$; then repeat on a three-or-more-option coding scheme where $d$ and $A$ are no longer one-to-one, and an $R^2$ far below 0.979 would show the original result depended on the binary-vote identity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the premise that human annotations are imperfect gold standards, which justifies using inter-model agreement as the calibration target."},{"cited_title":"A coefficient of agreement for nominal scales","cited_arxiv_id":null,"evidence_quote":"Provides Cohen's kappa, the chance-corrected agreement metric that the paper uses for both human and AI reliability."},{"cited_title":"& Weinberger, K","cited_arxiv_id":null,"evidence_quote":"Documents that neural-network confidence is miscalibrated, motivating the need for a second error signal beyond self-confidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Extends the miscalibration finding to modern neural-network architectures, supporting the paper's overconfidence premise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the ensemble-diversity error-bound rationale that justifies using vote spread as an error cue."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides quantitative diversity measures for classifier ensembles, the basis for the normalized Shannon entropy index $d$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Fleiss' kappa, used for the three-expert adjudication of contested coding segments."}],"review_version":1}