Recognition: no theorem link
Token-Efficient Change Detection in LLM APIs
Pith reviewed 2026-05-16 02:36 UTC · model grok-4.3
The pith
Border inputs enable black-box LLM change detection at 30x lower cost while matching grey-box performance.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Border inputs, defined as inputs for which more than one token ranks among the top outputs, enable powerful change detection in a strict black-box setting by exploiting the model's Jacobian and Fisher information of the output distribution in low-temperature regimes, allowing the B3IT scheme to achieve grey-box level performance at 30 times lower token cost.
What carries the argument
Border inputs, which are prompts having multiple top tokens in the output distribution, carrying the statistical power for change detection through Jacobian and Fisher information quantities.
If this is right
- Change detection works using only output tokens with no need for log probabilities or weights.
- Token costs drop by a factor of 30 compared with existing approaches.
- Detection accuracy reaches parity with the strongest grey-box methods on non-reasoning endpoints.
- Border inputs prove easy to locate for the tested non-reasoning models.
Where Pith is reading between the lines
- Routine auditing of commercial LLM services becomes feasible at scale because of the cost reduction.
- The same principle could track other forms of model drift such as fine-tuning effects.
- Similar border-input ideas might apply to change detection in non-LLM probabilistic APIs.
Load-bearing premise
Border inputs with multiple top tokens exist in sufficient quantity and can be located efficiently for the target models.
What would settle it
An experiment on a major public LLM API showing either that border inputs are too rare to find in reasonable queries or that the resulting detection power falls well below grey-box baselines.
read the original abstract
Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model's Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and achieve performance on par with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Border Inputs—prompts for which an LLM has multiple top tokens—and uses them for strict black-box change detection in LLM APIs via the B3IT scheme. It provides a statistical analysis based on the model's Jacobian and Fisher information in low-temperature regimes to motivate why these inputs enable powerful detection tests. Extensive experiments claim that such inputs are easily located for non-reasoning models and that B3IT matches grey-box performance while reducing costs by 30×.
Significance. If validated, the result would be significant for practical deployment of LLM monitoring at scale, as it removes the need for log-probability access or high query volumes. The principled statistical motivation and the reported cost reduction represent a meaningful advance over existing methods that either require grey-box access or are prohibitively expensive.
major comments (3)
- [§3] §3: The description of the border-input search procedure in the strict black-box setting (observing only output tokens) is insufficiently detailed; it is unclear how the method locates inputs with multiple top tokens with few queries without access to probabilities.
- [§4.2] §4.2: The in-vivo experiments report performance parity with grey-box approaches but do not include error bars, confidence intervals, or the exact number of trials, making it impossible to assess whether the 30× cost reduction is statistically reliable.
- [§2.3] §2.3: The low-temperature Jacobian/Fisher analysis is invoked to justify detection power, yet the experiments use standard sampling temperatures; no ablation shows that the theoretical advantage persists at the temperatures actually used by the tested endpoints.
minor comments (2)
- [Abstract] Abstract: The term 'in-vivo and in-vitro experiments' is used without defining what distinguishes the two settings.
- [§1] §1: Some citations to prior grey-box methods are missing specific page or equation references.
Simulated Author's Rebuttal
We thank the referee for the constructive review and for recognizing the potential significance of B3IT for scalable black-box LLM monitoring. We address each major comment below and commit to revisions that strengthen the manuscript's clarity and statistical rigor without altering its core claims.
read point-by-point responses
-
Referee: [§3] §3: The description of the border-input search procedure in the strict black-box setting (observing only output tokens) is insufficiently detailed; it is unclear how the method locates inputs with multiple top tokens with few queries without access to probabilities.
Authors: We agree that the procedure in §3 requires more explicit detail for reproducibility. In the revised manuscript we will expand the description to clarify that the search proceeds by controlled, low-cost perturbations of the input (e.g., token-level edits or suffix appendages) while monitoring only the emitted top token. A binary-search-style traversal over perturbation magnitude identifies inputs at which the top token flips, thereby locating border inputs with O(log N) queries where N is the effective search granularity. This relies solely on token observation and does not require probability values. revision: yes
-
Referee: [§4.2] §4.2: The in-vivo experiments report performance parity with grey-box approaches but do not include error bars, confidence intervals, or the exact number of trials, making it impossible to assess whether the 30× cost reduction is statistically reliable.
Authors: We acknowledge the omission. The revised §4.2 will report results over 50 independent trials per endpoint, include 95% confidence intervals and error bars on all performance and cost metrics, and add a statistical comparison confirming that the observed 30× cost reduction is significant relative to the grey-box baselines. revision: yes
-
Referee: [§2.3] §2.3: The low-temperature Jacobian/Fisher analysis is invoked to justify detection power, yet the experiments use standard sampling temperatures; no ablation shows that the theoretical advantage persists at the temperatures actually used by the tested endpoints.
Authors: The low-temperature analysis supplies intuition for why border inputs concentrate Fisher information; however, we recognize the need to bridge theory and practice. The revised manuscript will include a new ablation subsection that evaluates B3IT detection power across a range of sampling temperatures (including the default values used by the tested APIs) and shows that the performance advantage over non-border baselines remains consistent. revision: yes
Circularity Check
No significant circularity; derivation grounded in external statistical analysis
full rationale
The paper's core derivation analyzes the Jacobian and Fisher information of the output distribution in low-temperature regimes to establish that border inputs enable powerful change detection tests. This relies on standard statistical properties rather than any parameter fitted to the detection task or self-referential definitions. The B3IT scheme is constructed from this analysis, with experimental validation that border inputs are readily located in black-box settings. No steps reduce by construction to fitted inputs, self-citations, or renamed known results; the cost and performance claims follow from empirical query efficiency without circular reduction. The approach is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Low-temperature regime approximation for output token distributions
invented entities (1)
-
Border Inputs
no independent evidence
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.