Pith.

Artificial Intelligence · arXiv:2605.30335 v1 · submitted 28 May 2026 · 3 references

Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agents

Anany Kotawala

The paper says

“Multi-component LLM agents assemble probabilistic claims from components that each see only part of a joint problem; the composition can violate basic probability axioms even when every component is locally coherent.”

more of the abstract

We formalise this locally coherent, globally incoherent failure via the compositional residual eps*, the L2 distance from the composed quote to the joint coherent polytope, computable at runtime from system output and the declared cross-component coupling constraints. A product-structure dichotomy characterises when local coherence suffices, and a Rayleigh-quotient prediction matches the observed residual within 7% on three of four relation classes. A hierarchical Boyle-Dykstra projection repairs the composition deterministically; an anytime-valid e-process gives sequential coherence monitoring. Across 1,876 ensemble cliques on a four-LLM mid-tier panel (frontier-panel rerun in Section 5.5), eps* > 0 on 33-94% of cliques, translating to +0.115 nats per bet of regret on 1,770 resolved bets under the proportional allocation rule (the gain collapses to +0.006 under bettors that themselves coherentise). Three intuitive LLM-side mitigations(retrieval, partition-aware prompting, aggregator-LLM) each fail or regress.

from the abstract

Pith says

Machine review · grok-4.3 · 29 Jun 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

No verdict was given on this paper.

2 major objections · 1 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 3 references say.

The paper says, at Abstract (definition of eps* and Rayleigh-quotient prediction); Section 5.5 (frontier-panel rerun)

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The definition of eps* as the L2 distance to the joint coherent polytope (stated in the abstract and used for the Rayleigh-quotient predictor and Boyle-Dykstra repair) assumes the declared cross-component coupling constraints are complete and fully specify the feasible set.

more of objection 1

Unstated logical implications or emergent dependencies between LLM components would shrink the true polytope, making reported eps* values underestimates and invalidating both the 7% match and the 33-94% prevalence figures. This assumption is load-bearing for the central empirical claims and the product-structure dichotomy.

The paper says, at Abstract (Rayleigh-quotient prediction); results on relation classes

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The Rayleigh-quotient prediction is reported to match observed residuals within 7% on three of four relation classes, but the manuscript provides no explicit derivation or equation showing whether this is an a priori prediction from the polytope geometry or a post-hoc fit to the same data used for the 1,876-clique experiment. Without the underlying equations, it is impossible to assess whether the match supports the claimed predictive power or reflects circularity.

1 minor objection

The paper says, at Abstract

Pith says: minor objection 1 of 1

The abstract contains a missing space: 'mitigations(retrieval' should be 'mitigations (retrieval'.

What would settle it

For 50 randomly sampled cliques from the 1,876, augment the declared constraints with two plausible implicit ones (e.g., transitivity on overlapping events and consistency with a shared external fact mentioned in the prompts); recompute eps* and the projection. If the median |Δeps*| exceeds 0.05 or the sign of incoherence flips on >20% of cases, the declared-constraint assumption does not hold.

from the skeptic's reading

How the review went

5 readings by grok-4.3 · 29 Jun 2026

  1. Reader no verdict given confidence low
  2. Skeptic agrees 20 s · 1,737 tokens read, 357 written
  3. Editor wrote the review in brief 20 s · 2,356 tokens read, 393 written
  4. Referee major revision: 2 major objections, 1 minor 25 s · 1,884 tokens read, 620 written
  5. Strongest defence (simulated, not the authors) 22 s · 1,572 tokens read, 528 written

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The core observation is that LLM components can each output locally coherent probabilities yet produce a joint assignment that violates basic axioms, and the paper measures this gap directly as eps*, the L2 distance from the composed output to the coherent polytope defined by the runtime coupling constraints.

What is new is the eps* construction itself, the product-structure dichotomy that tells when local coherence is enough, the Rayleigh-quotient predictor that reportedly matches residuals within 7% on three of four classes, and the deterministic Boyle-Dykstra projection for repair. The sequential e-process for monitoring is also a practical addition. The experiments on 1,876 cliques from a four-LLM panel give concrete prevalence numbers (33-94%) and a regret figure (+0.115 nats under proportional betting), which is more than most papers supply.

The soft spot is the completeness assumption on the declared constraints. If the LLMs carry unstated background knowledge or emergent dependencies, the true feasible set shrinks and the reported eps* becomes an underestimate; that would also weaken the predictor and the repair guarantee. The 7% match looks good on the surface but needs the underlying equations to confirm it is not a post-hoc fit. The three mitigations failing is useful negative evidence, though the details of how they were implemented matter.

This is aimed at people building modular LLM systems for joint probabilistic tasks. It deserves peer review because it names a measurable failure mode with numbers and a fix, even if the constraint-completeness issue needs tighter handling.

The strongest objection

The central construction defines eps* as the L2 distance from the composed output to the joint coherent polytope, with the polytope specified solely by the runtime-declared cross-component coupling constraints. This requires that those constraints are complete: any unstated logical implications, shared background knowledge, or emergent dependencies between the LLM components would shrink the true feasible set, making the reported eps* an underestimate of actual incoherence and invalidating both the Rayleigh-quotient predictor and the Boyle-Dykstra repair. The product-structure dichotomy and the 7% match on three of four relation classes are derived under the same completeness assumption; if it fails, the empirical prevalence (33-94%) and regret figures (+0.115 nats) no longer quantify the claimed phenomenon.

The reader's weakest assumption is exactly the load-bearing point; the full-text description of runtime computation from declared constraints confirms this is the unverified step on which the formalization, predictor, and empirical claims rest. No other internal inconsistency appears in the abstract or described results.

The referee's report

If the central claims hold, the work supplies a runtime-computable metric, a predictive bound, a repair algorithm, and monitoring tool for a practically relevant failure mode in LLM agents. Strengths include the scale of the 1,876-clique experiment, the specific 7% match on the Rayleigh-quotient predictor, the deterministic repair method, and the regret quantification under proportional allocation. These elements would be useful for building more reliable multi-component systems if the polytope construction and predictor derivation are robust.

Pith's reading of the paper (a machine's summary, not the authors')

The paper establishes that LLM agents built from multiple components, each seeing only part of a problem, can assemble claims that violate probability axioms despite each part being coherent on its own. It introduces the compositional residual eps* as the L2 distance between the composed output and the nearest joint distribution that satisfies all coherence constraints. This residual can be computed at runtime using the declared coupling constraints between components. The work shows that this incoherence occurs frequently in practice and leads to measurable regret in betting tasks, while several intuitive fixes fail to resolve it.

What carries the argument. the compositional residual eps*, defined as the L2 distance from the composed quote to the joint coherent polytope, which quantifies the global incoherence arising from partial views

Core claim. Multi-component LLM agents assemble probabilistic claims from components that each see only part of a joint problem; the composition can violate basic probability axioms even when every component is locally coherent, quantified by the compositional residual eps*, the L2 distance from the composed quote to the joint coherent polytope, computable at runtime from system output and the declared cross-component coupling constraints.

The premise it rests on. The declared cross-component coupling constraints supplied at runtime are sufficient to define the joint coherent polytope and to compute the L2 projection without additional unstated dependencies.

If the paper is right (Pith's inferences, not the authors')

eps* exceeds zero in 33-94% of ensemble cliques on a four-LLM panel

the incoherence produces +0.115 nats per bet of regret under proportional allocation on resolved bets

the regret gain collapses to +0.006 when bettors apply their own coherence repairs

a hierarchical Boyle-Dykstra projection repairs the composition deterministically

an anytime-valid e-process supports sequential coherence monitoring

Designers of multi-component agents may need to embed joint coherence checks at the system level rather than depending solely on component-level outputs.

The observed failure mode implies that modular AI systems could accumulate decision errors as the number of components grows without explicit coupling enforcement.

Runtime calculation of eps* could function as a practical diagnostic for detecting incoherence in deployed agent ensembles before decisions are made.

Strongest defence (simulated, not the authors)

We thank the referee for these detailed comments. We address each major point below, clarifying the scope of eps* with respect to declared constraints and committing to include the explicit derivation of the Rayleigh-quotient predictor.

  1. Referee: [Abstract (definition of eps* and Rayleigh-quotient prediction); Section 5.5 (frontier-panel rerun)] The definition of eps* as the L2 distance to the joint coherent polytope (stated in the abstract and used for the Rayleigh-quotient predictor and Boyle-Dykstra repair) assumes the declared cross-component coupling constraints are complete and fully specify the feasible set. Unstated logical implications or emergent dependencies between LLM components would shrink the true polytope, making reported eps* values underestimates and invalidating both the 7% match and the 33-94% prevalence figures. This assumption is load-bearing for the central empirical claims and the product-structure dichotomy.

    Simulated defence: eps* is defined with respect to the runtime-declared cross-component coupling constraints, which are the only constraints available to the system at composition time. The metric and all derived claims (prevalence, regret, dichotomy) quantify observable incoherence relative to those declarations. Unstated dependencies would indeed define a smaller polytope, but this does not invalidate the reported results because they are conditioned on the declared constraints. We will add a clarifying sentence in the abstract and Section 3 to make this scope explicit.

  2. Referee: [Abstract (Rayleigh-quotient prediction); results on relation classes] The Rayleigh-quotient prediction is reported to match observed residuals within 7% on three of four relation classes, but the manuscript provides no explicit derivation or equation showing whether this is an a priori prediction from the polytope geometry or a post-hoc fit to the same data used for the 1,876-clique experiment. Without the underlying equations, it is impossible to assess whether the match supports the claimed predictive power or reflects circularity.

    Simulated defence: The Rayleigh-quotient predictor is derived a priori from the spectral geometry of the constraint matrix under the product-structure assumption and is not fitted to the experimental data. The 7% match is an out-of-sample observation on the 1,876-clique results. We will add the full derivation and governing equation in the revised Section 4 to demonstrate that the predictor is not post-hoc.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by grok-4.3, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Multi-component LLM agents assemble probabilistic claims from components that each see only part of a joint problem; the composition can violate basic probability axioms even when every component is locally coherent. We formalise this locally coherent, globally incoherent failure via the compositional residual eps*, the L2 distance from the composed quote to the joint coherent polytope, computable at runtime from system output and the declared cross-component coupling constraints. A product-structure dichotomy characterises when local coherence suffices, and a Rayleigh-quotient prediction matches the observed residual within 7% on three of four relation classes. A hierarchical Boyle-Dykstra projection repairs the composition deterministically; an anytime-valid e-process gives sequential coherence monitoring. Across 1,876 ensemble cliques on a four-LLM mid-tier panel (frontier-panel rerun in Section 5.5), eps* > 0 on 33-94% of cliques, translating to +0.115 nats per bet of regret on 1,770 resolved bets under the proportional allocation rule (the gain collapses to +0.006 under bettors that themselves coherentise). Three intuitive LLM-side mitigations(retrieval, partition-aware prompting, aggregator-LLM) each fail or regress.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.74

“Multi-component LLM agents assemble probabilistic claims from components that each see only part of a joint problem; the composition can violate basic probability axioms even when every component is locally coherent.”

How the main claim is stated: asserted 0.78

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.64 · human participants 0.10 · controlled comparison 0.66 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.78 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Multi-component LLM agents assemble probabilistic claims from components that each see only part of a joint problem; the composition can violate basic probability axioms even when every component is locally coherent. background 0.98 0.19 0.49 asserted 0.78
We formalise this locally coherent, globally incoherent failure via the compositional residual eps*, the L2 distance from the composed quote to the joint coherent polytope, computable at runtime from system output and the declared cross-component coupling constraints. contribution 0.60 0.93 0.20 asserted 1.00
A product-structure dichotomy characterises when local coherence suffices, and a Rayleigh-quotient prediction matches the observed residual within 7% on three of four relation classes. result 0.89 0.90 0.14 asserted 1.00
A hierarchical Boyle-Dykstra projection repairs the composition deterministically; an anytime-valid e-process gives sequential coherence monitoring. method 0.82 0.82 0.11 asserted 1.00
Across 1,876 ensemble cliques on a four-LLM mid-tier panel (frontier-panel rerun in Section 5.5), eps* > 0 on 33-94% of cliques, translating to +0.115 nats per bet of regret on 1,770 resolved bets under the proportional allocation rule (the gain collapses to +0.006 under bettors that themselves coherentise). result 1.00 0.92 0.27 asserted 1.00
Three intuitive LLM-side mitigations(retrieval, partition-aware prompting, aggregator-LLM) each fail or regress. result 0.99 0.84 0.16 asserted 1.00

What kind of paper the classifier reports this is: theoretical result 0.48 · evidence from observation or experiment 0.76

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,253 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. A locally coherent, globally incoherent failure. Four specialists are routed one outcome each of the partition “Largest US AI startup IPO 2026” (infrastructure, model labs, applications, other; from §5.3). Each specialist sees only its assigned sector as a single Bernoulli question, quotes a locally well-calibrated probability, and the agent assembles the four quotes. No specialist saw that the sectors tile the field; the assembled mass is 2.50 and ε ⋆ = 0.749 certifies the failure. scarce. In multi-component agents, planners route retrieval, arithmetic, and probability assessment…
Figure 2 of the paper
Figure 2. Compositional residuals and the exposure bound. (a) ECDF of ε ⋆ across N=1,876 ensemble cliques, stratified by relation. Each component is per-call JCD-projected, so any positive ε ⋆ is cross-component incoherence introduced by composition. (b) ECDF of the exposure bound √ m⋆ε ⋆ (Cor. 3.5) under three regimes: naive composition (mean 0.137), hierarchical JCD (at the OSQP floor), and a single-LLM no-composition reference. Shaded bands are 95% bootstrap CIs (B=1000). 4.5, N=80 cliques per relation, 40 partitions). Mean ε ⋆ falls to 0.025/0.019/0.040/0.058 on neg/and/or/partition:…
Figure 3 of the paper
Figure 3. Bounded-difference e-process Et on per-component residual streams (Theorem D.2). Crossing 1/α = 104 at t=16 (Llama), t=177 (GPT-nano), t=422 (GPT-5.4-mini); Claude-Haiku stays below 1. E. Clique-mining pipeline and reproducibility Paleka. Tuples are pre-mined by the original authors: each JSONL record is a single (checker, {Qi},relation) entry, parsed directly. Polymarket. We query gamma-api at the event level, retain events with ≥2 resolved markets, and emit one of two relation types: (i) a partition relation P i pi = 1 when an event has multi-candidate markets that resolve with exa
Figure 4 of the paper
Figure 4. Polymarket cross-platform replication of the per-call result (Theorem A.1): the incoherence-conditional gain pattern of Theorem A.1 replicates on resolved Polymarket events, with the largest mean gain in the highest-ε quartile at m≥4 (note the smaller per-call absolute magnitudes; the lower panel’s cumulative Hanson–LMSR exposure reduction to the QP floor matches Corollary A.2)
Figure 5 of the paper
Figure 5. K-sweep on Anthropic Claude-Haiku-4.5 NegChecker forecasts. Non-vanishing ∆Brier asymptote: the empirical signature of structural incoherence in pF . Compositional layer. We also sweep K ∈ {8, 16, 32} at the compositional layer (the residual ε ⋆ in §4). K = 8 uses the existing 4-LLM panel and K = 16, 32 extends Anthropic Claude-Haiku-4.5, GPT-5.4-mini, and GPT-5.4-nano to additional samples on a 40-clique-per-relation, 20-partition subset (Groq Llama-3.3-70b excluded after exhausting its tokens-per-day quota mid-sweep). The 3-model panel ensemble is computed identically to the 4-LLM…
Figure 6 of the paper
Figure 6. Structured-routing replication of
Figure 7 of the paper
Figure 7. Tool augmentation does not eliminate the compositional residual. Each point is one of the 30 matched partitions; x-axis is ε ⋆ in the no-tool baseline, y-axis is ε ⋆ when each specialist receives top-five DuckDuckGo snippets for its assigned outcome before quoting. Most points lie on or above y=x: tool augmentation tightens within-specialist variance but cross-specialist disagreement persists, so the assembled partition quote is no more coherent with retrieval than without. 23

Figures are the authors', reproduced from the paper.

References (3)
  1. [1] doi: 10.1214/20-AOS1991. McDiarmid, C. On the method of bounded differences. In Siemons, J. (ed.),Surveys in Combinatorics, 1989, volume 141 ofLondon Mathematical Society Lecture Note Series, pp. 148–188. Cambridge University Press, on Pithdoi:10.1214/20-aos1991
  2. [2] doi: 10.1017/CBO9781107359949.008. Paleka, D., Sudhir, A. P., Alvarez, A., Bhat, V ., Shen, A., Wang, E., and Tram`er, F. Consistency checks for language model forecasters. InInternational Conference on Learn- ing Representations (ICLR), 2025. arXiv:2412.18544. Platt, J. C. Pr... on Pithdoi:10.1017/cbo9781107359949.008
  3. [3] The slack ∥ˆp−Π(ˆp)∥2 2 is the squared distance from coherence; Brier improvement is largest where pre-projection incoherence is largest. Proof. Hilbert projection theorem (Bauschke & Combettes, 2017, Thm. 3.16) gives ⟨ˆp−Π(ˆp), p∗ −Π(ˆp)⟩ ≤0. Expand ∥ˆp−p∗∥2 aroundΠ(ˆp)and dr... on Pith
Cited by 1 Pith paper
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 5 sentences without a confident label

Uses the method or tool of 1 paper
Uses data or values from 1 paper
  • Empirically, we evaluate $1,876$ ensemble cliques drawn from Paleka [ref] and Polymarket across four contemporary foundation models, stratified by four logical-relation classes.

    Consistency Checks for Language Model Forecasters · Introduction · 0.94

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment