{"id":"22e00ce9-9571-4fa0-9eb3-fd2bd9605282","arxiv_id":"2608.10315","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Answers that stay stable under topic-neutral context changes predict correctness better than self-reported confidence or self-consistency across reasoning, factuality, and code benchmarks.","lead":"The paper defines a score, C3, that measures whether a language model gives the same answer when a neutral, unrelated sentence is added in front of the question. Across six benchmarks and many models, answers that stay stable under this added context are noticeably more likely to be correct, suggesting a lightweight way to spot untrustworthy model outputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Perturbation neutrality is checked only as aggregate accuracy delta on one benchmark; per-instance semantic shifts could make C3 measure prompt sensitivity, not credibility.","rationale":"I read the paper as making an empirical claim: C3, defined as inverse normalized MMD between original and perturbed output distributions, is a reference-free proxy for answer correctness because stable answers are better supported. The most load-bearing condition is that perturbations are answer-neutral and topic-aligned at the instance level, since otherwise the metric's interpretation as 'credibility' is unwarranted. The paper verifies neutrality only through aggregate Δ on a single multiple-choice benchmark, and its topic-alignment check is a coarse automatic classification. Both are explicitly approximate, and the paper itself states the neutrality check does not prove per-perturbation neutrality. This is a validity threat, not an internal inconsistency: Appendix H's normalization is coherent under the bounded-kernel assumption, and the Appendix G ablation supports MMD-agnosticism. But the whole setup—C3 as a credibility signal—requires that the perturbation does not change the task. The reader's conditional verdict already captures this concern along with missing artifacts, missing confidence intervals, and the mismatch between '26 models' and '16 models'; my independent read agrees that the neutrality validation is the weakest load-bearing point. A targeted re-analysis controlling for per-instance perturbation effect would adjudicate. Until then, the conditional verdict stands: no change in verdict is needed, but the paper should not be accepted without addressing this validity check.","tokens_in":30922,"tokens_out":5872,"duration_ms":64657,"concrete_test":"Recompute the SVAMP and SimpleQA C3 AUROC after restricting to instances where the per-instance performance delta Δ_i = P(correct|perturbed) − P(correct|original), measured over the existing 30-trials samples, satisfies |Δ_i| ≤ 0.1; also have independent annotators (or a stronger judge) flag perturbations that 'introduce answer-relevant information or change the question' and drop flagged instances. If the AUROC advantage over self-consistency collapses (e.g., from 0.917 toward the 0.86 range on SVAMP), the correlation is driven by non-neutral perturbations rather than cross-contextual consistency. If the advantage persists after filtering, the neutrality concern is not the primary driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—C3 predicts correctness because it measures stability under answer-neutral contextual variation—rests on the perturbations being content-neutral at the instance level (Section 3). The neutrality check in Appendix A.3 computes Δ = #Correct_perturbed − #Correct_original on MMLU High School Statistics only and shows the distribution centered near zero. Aggregate near-zero Δ is compatible with large per-instance semantic shifts: on SVAMP, for example, an added premise introducing a distractor quantity or changing the referent of 'that day' alters the task itself, so the original-versus-perturbed divergence reflects task change rather than fragility of knowledge. The paper explicitly concedes that the check 'does not prove that every individual perturbation is perfectly neutral' (A.3), and the topic-alignment check (A.4) uses a coarse Qwen3-8B judge that can pass perturbations that are topically relevant yet semantically answer-relevant. Under this concession, high C3 could arise because the perturbation systematically shifted both correct and incorrect answers into a different but stable response pattern, or low C3 could arise because the perturbation changed the question. The headline AUROC numbers (0.917 on SVAMP, 0.823 on SimpleQA) would then be measuring prompt sensitivity, not credibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cross-Contextual Consistency (C3), a black-box behavioral metric that measures how stable an LLM's output distribution is when the same question is prefixed with topic-aligned, content-neutral contextual perturbations. C3 is operationalized as one minus half the squared Maximum Mean Discrepancy between original and perturbed output distributions. The authors evaluate C3 on 26 models and six benchmarks spanning math reasoning, multiple-choice reasoning, commonsense, short- and long-form factuality, and code generation. They report that C3 aligns with correctness and factuality better than vanilla baselines (self-report, self-consistency, paraphrasing consistency) on aggregate metrics such as AUROC and ECE, and they propose C3 as a benchmark diagnostic that separates 'mastered', 'brittle', 'biased', and 'unlearned' instances.","tokens_in":31139,"tokens_out":4054,"duration_ms":43501,"significance":"If the central claim holds, C3 is a practically valuable reference-free, label-free, black-box proxy for answer credibility, applicable to closed models and free-form generation where token-probability methods fail. The paper's strengths include very extensive per-model tables, ablations on perturbation source (GPT-4.1, Qwen3-8B, web noise) and on the distance metric (MMD vs. cross-comparison), and a clear presentation of the method's computational cost. The claim is falsifiable and the protocol is reproducible in principle. However, the interpretation of C3 as a credibility measure depends critically on the perturbations being content-neutral at the instance level, and the current neutrality evidence is aggregate and explicitly conceded to be incomplete. The reported aggregate wins also lack paired significance testing, and several per-model cells show C3 below baselines, so the headline 'consistently outperforms' is not yet statistically grounded.","major_comments":[{"comment":"The load-bearing assumption that perturbations are content-neutral is verified only as an aggregate correctness delta on MMLU High School Statistics, and the manuscript itself states that this 'does not prove that every individual perturbation is perfectly neutral'. Aggregate near-zero delta is compatible with large per-instance semantic shifts: on SVAMP, an added premise can introduce a new quantity or change the referent of 'that day', thereby altering the task rather than the context. Under that possibility, high C3 could reflect stability under a changed task and low C3 could reflect task change, so the reported AUROC gains (e.g., 0.917 on SVAMP in Table 1) would be measuring prompt sensitivity rather than credibility. I ask for a per-instance neutrality audit on a sampled subset of each benchmark (human annotation or a stronger judge with explicit answer-relevance criteria), and a demonstration that the C3-correctness correlation survives when restricted to instances whose perturbations are verified neutral. Without this, the central interpretive claim is not established.","section":"§3, Appendix A.3"},{"comment":"The aggregate scores in Table 1 are averages over 16 models, but no confidence intervals, bootstrap estimates, or paired significance tests are reported. The per-model tables show that C3 is not uniformly better than baselines: for example, on MMLU High School Stats, llama-3.1-70b-instruct has C3 AUROC 0.549 versus self-consistency 0.598, and on SimpleQA, gemini-2.5-flash has C3 AUROC 0.827 versus self-consistency 0.838. Because the paper's headline claim is that C3 'consistently outperforms' baselines, the authors should report paired tests (e.g., Wilcoxon signed-rank over models) or bootstrap confidence intervals for the AUROC/ECE differences, and should quantify how many models show improvement per benchmark.","section":"Tables 1, 2, 6"},{"comment":"The normalization C3 = 1 - 0.5 * MMD^2 relies on the assumptions that the kernel is bounded with k(y,y)=1 and 0 <= k(y,y') <= 1, giving MMD^2 <= 2. For open-ended generations the paper suggests using cosine similarity, which can take negative values and is not necessarily nonnegative; in that case the stated upper bound and the [0,1] interpretation of C3 do not follow. The authors should either restrict the kernel to a nonnegative normalized similarity (e.g., ReLU-cosine or a nonnegative kernel) or derive the correct normalization for the cosine kernel actually used in the experiments.","section":"Appendix H"}],"minor_comments":[{"comment":"The heading 'A closer inspection on facutality' contains a typo; it should read 'factuality'.","section":"§5.2"},{"comment":"In the related work section, 'when when paraphrased the quesiton' contains a duplicated word and a typo; it should read 'when the question is paraphrased'.","section":"§6"},{"comment":"The benchmark name is written inconsistently as both 'SVAMP' and 'SV AMP'; please standardize.","section":"Throughout"},{"comment":"The distance notation is inconsistent: the main text uses eD, while Appendix H uses D and MMD^2; please unify.","section":"§3, Appendix H"},{"comment":"The judge prompt shown in D.5 is specific to code completions ('Compare two code completions'), but the text says this judge is used for all benchmarks; please clarify whether the same prompt was reused for non-code tasks or provide the task-adapted versions.","section":"Appendix D.5"},{"comment":"The four-region partition is based on medians of the same data used to interpret the regions; this is acceptable as a visualization, but the text should avoid causal language such as 'supports the hypothesis of benchmark leakage' without a more direct contamination measure.","section":"§5.3 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a substantial empirical study and the correlational signal appears real, but the credibility interpretation currently outruns the neutrality evidence. The revision should either narrow the claims to 'stability under a particular perturbation family' or add the per-instance neutrality verification and paired significance tests requested. The manuscript is likely salvageable, but the current framing overstates what the aggregate checks establish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious empirical paper. The core idea—measure distributional shift between original and content-neutral perturbed prompts with MMD, call it C3—isn't brand new in spirit (self-consistency and paraphrase consistency are close neighbors), but the specific package is: topic-aligned prefix perturbations, MMD on answer distributions, and a four-quadrant benchmark diagnosis. And the evidence base is real: 16 models across six benchmarks, detailed per-model tables, ablations replacing GPT-4.1 with Qwen3-8B and with random web noise, and replacing MMD with a simple cross-comparison score. The SimpleQA and SVAMP results are noticeably better than the vanilla baselines. Credit where due: this is a lot of work, and it mostly supports the claim that cross-contextual stability tracks correctness better than self-consistency or self-report.\n\nThe soft spots are in interpretation more than execution. The stress-test concern about per-instance neutrality is legitimate. Appendix A.3 checks neutrality only as an aggregate correctness delta on MMLU High School Stats, and the paper says in plain words that this doesn't prove every perturbation is neutral. That means on a specific SVAMP problem, the added premise could change the referent of 'that day' or introduce a distractor quantity—making the divergence reflect task change, not fragility. The topic-alignment check using a Qwen3-8B judge on coarse domain labels doesn't catch answer-relevant semantic shifts. So the headline claim that C3 measures credibility is one step ahead of the evidence; at minimum it measures sensitivity to a perturbation class that is approximately neutral on average.\n\nThat said, the random-web-noise ablation partially softens the concern. If the signal depended on subtle answer-relevant semantic shifts, random noise should wash it out, yet C3 with random noise still beats the baselines. So I'd frame it as: the correlation is robust; the mechanism the authors attribute to it is not fully pinned down.\n\nOther issues are fixable: no released code, data, or perturbation pools; no confidence intervals or significance tests; the abstract says 26 models across six benchmarks but only 16 ran all six (the other 10 are an MMLU-stats-only case study); and the judge prompt in D.5 is written for code completions but used for all tasks. These don't sink the paper, but they matter for reproducibility.\n\nVerdict: worth a serious referee. I'd send it out, with instructions to release artifacts, add per-instance neutrality checks or soften the causal language, and fix the 26/16 mismatch. If those land, this could be a useful reference-free credibility signal.","headline":"C3 is a promising, well-tested black-box consistency signal, but the paper's 'credibility' interpretation leans on a neutrality assumption it explicitly concedes it doesn't fully verify.","tokens_in":31731,"tokens_out":2281,"would_cite":false,"duration_ms":23248,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM answer's stability under neutral, topic-aligned rewording reliably signals whether it is correct or factual.","keywords":["cross-contextual consistency","LLM credibility","maximum mean discrepancy","prompt perturbation","calibration","factuality detection","benchmark saturation","black-box evaluation"],"falsifier":"A decisive test would build a set of items in which each perturbed prompt has been independently verified to preserve the original correct answer, then measure C3's AUROC on that set; if wrong answers shift as little as right answers there, the reported correlation is an artifact of perturbations quietly changing the question.","tokens_in":30675,"feed_emoji":"🎯","tokens_out":8852,"duration_ms":72276,"temperature":0.7,"pith_summary":"The paper claims that a language model answer is credible to the degree that it stays stable when the same question is surrounded by neutral, topic-aligned extra context, and that this stability can be measured from the outside. It introduces Cross-Contextual Consistency (C3), a black-box score that compares the distribution of outputs sampled from the original prompt with the distribution under perturbed prompts, and reports that across 26 models and six benchmarks, higher C3 tracks correctness and factuality better than self-reported confidence, self-consistency, or paraphrasing consistency. If true, C3 gives practitioners a reference-free, label-free way to flag doubtful outputs without touching model internals, and it gives evaluators a second axis for spotting benchmark questions that are saturated, brittle, or systematically biased.","feed_headline":"Stable answers under neutral rewording flag correct LLM outputs","feed_subtitle":"A label-free consistency score beats self-report and self-consistency on reasoning, factuality, and code benchmarks.","key_machinery":"The central object is C3, a normalized inverse distance between two empirical output distributions: $C3(x;E)=1-\\frac{1}{2}\\widehat{\\mathrm{MMD}}^2(Y,Y^E)$, where $Y$ are samples from the original prompt, $Y^E$ are samples from prompts prefixed with topic-aligned, content-neutral perturbations, and MMD is a kernel-based two-sample distance with task-adaptive feature maps and kernels. The work it does is to translate the behavioral principle that well-supported answers are stable under cross-examination into a single number, and to give one comparison axis across multiple-choice, short-answer, long-form, and code tasks; an ablation replacing MMD with a simple cross-comparison agreement score preserves the signal, so the comparison itself, not the specific kernel, carries the result.","core_discovery":"The paper's discovery is that cross-contextual consistency works as a proxy for truthfulness: an LLM that internally supports an answer should reproduce that answer when the prompt is prefixed with content-neutral contextual variation, whereas a hallucinated or weakly grounded answer should shift. Operationalizing this as $C3=1-\\frac{1}{2}\\widehat{\\mathrm{MMD}}^2$ between original and perturbed output distributions, the paper reports that answers with smaller shifts are more likely to be correct across arithmetic reasoning, multiple-choice knowledge, short- and long-form factuality, and code generation, with AUROCs of 0.917 on SVAMP and 0.823 on SimpleQA. The paper also claims C3 doubles as a benchmark diagnostic: instances where models agree on the correct answer but have low C3 are labeled brittle and suspected of surface pattern matching or contamination, while low-accuracy, perturbation-invariant instances are labeled biased and attributed to shared misconceptions.","pith_inferences":["Editorial inference: because C3 needs no labels, it could be used during data curation or fine-tuning to rank answers by stability, a training-side use the paper does not explore.","Editorial inference: the aggregate neutrality check leaves open per-instance semantic drift, so the strongest version of the claim would require verifying that each perturbed prompt preserves the original correct answer.","Editorial inference: the brittle-region pattern is a testable contamination detector: questions many models answer correctly but unstably should show larger accuracy drops under paraphrasing, which the paper's interpretation predicts.","Editorial inference: the corpus-noise result suggests a fully offline, low-cost deployment without any frontier model, which the ablation supports but does not develop into a protocol."],"forward_implications":["A user can rank individual answers by fragility using only API sampling, with no ground-truth labels and no access to token probabilities.","Benchmark curators can mark question subsets as saturated, brittle, biased, or unlearned from the joint C3-and-performance scatter, giving a diagnostic that aggregate accuracy hides.","On hard factuality questions, C3 gives much better calibration than self-report (ECE 0.166 vs 0.778 on SimpleQA) and better discrimination than self-consistency (AUROC 0.823 vs 0.792).","The signal survives weaker perturbation sources: C3 with Qwen3-8B-generated noise reaches AUROC 0.833 on SimpleQA, and random web noise stays competitive, so the method does not depend on a frontier model to write the perturbations.","C3's alignment with correctness grows with model scale within a family, so residual cross-contextual fragility becomes a sharper marker of wrong answers as models get stronger."],"supporting_citations":[{"why":"Supplies the kernel-based two-sample distance used to define the C3 distributional comparison.","marker":"[Gretton et al., 2012]"},{"why":"Provides SVAMP, the arithmetic benchmark where C3 achieves its strongest AUROC of 0.917.","marker":"[Patel et al., 2021]"},{"why":"Provides SimpleQA Verified, the hard factuality benchmark on which C3 reaches AUROC 0.823 and low ECE.","marker":"[Haas et al., 2025]"},{"why":"Provides MMLU High School Statistics, used both as a main benchmark and for the content-neutrality verification.","marker":"[Hendrycks et al., 2021b,a]"},{"why":"Provides FActScore, the long-form factuality benchmark that C3 correlates with at 0.624 Spearman.","marker":"[Min et al., 2023]"},{"why":"Provides HumanEval, the code-generation benchmark showing C3 applies to open-ended outputs.","marker":"[Chen et al., 2021]"},{"why":"Provides CommonsenseQA, the commonsense reasoning benchmark in the main six-benchmark comparison.","marker":"[Talmor et al., 2019]"},{"why":"Defines self-consistency, the strongest vanilla baseline that C3 outperforms on most metrics.","marker":"[Wang et al., 2023]"},{"why":"Defines self-reported confidence, the baseline that C3 shows to be near-random in discriminative power.","marker":"[Lin et al., 2022]"},{"why":"Defines paraphrasing consistency, another baseline that C3 surpasses across the six benchmarks.","marker":"[Portillo Wightman et al., 2023]"}],"fun_headline_variants":["Consistent answers under neutral rewording predict LLM correctness","Cross-context consistency flags reliable LLM answers","Stable answers under topic-neutral rewording signal truth","Rewording invariance: a label-free check for LLM factuality","Cross-contextual consistency predicts correct LLM answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every perturbation prefix is genuinely content-neutral and topic-aligned for the specific question it is attached to, so that any output shift reflects answer fragility rather than a changed task.","fun_headline_variants_meta":{"raw":{"variants":["Consistent answers under neutral rewording predict LLM correctness","Cross-context consistency flags reliable LLM answers","Stable answers under topic-neutral rewording signal truth","Rewording invariance: a label-free check for LLM factuality","Cross-contextual consistency predicts correct LLM answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000858,"raw_usage":{"total_tokens":3697,"prompt_tokens":891,"completion_tokens":2806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2727}},"tokens_in":507,"tokens_out":2806,"duration_ms":19487,"temperature":1.0,"reasoning_tokens":2727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:49.894540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would build a set of items in which each perturbed prompt has been independently verified to preserve the original correct answer, then measure C3's AUROC on that set; if wrong answers shift as little as right answers there, the reported correlation is an artifact of perturbations quietly changing the question.","supporting_citations":[],"review_version":1}