Pith. sign in

REVIEW 2 major objections 4 minor 28 references

Mergeable Model-Side Aggregation States for Long-Context Language Models

T0 review · 2 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed 2 KiB HyperLogLog state, carried beside a frozen language model, gives long-context reasoning set statistics that nearly match exact aggregation and avoid external code execution.

desk verdict Solid, honest engineering paper: a model-side HLL aggregation interface for LLMs, carefully validated on structured streams, with the extractor explicitly left out of scope. read the letter →

arxiv 2607.26448 v1 pith:F5KVT75H submitted 2026-07-29 cs.CL cs.AI

Dachuan Song , Junyu Yin , Zechen Hu , Xuan Wang This is my paper
classification cs.CLcs.AI
keywords long-contextlanguagemodelsset-basedaggregationHyperLogmergeablesketchescardinalityestimationapproximatenumbersenseaggregate-then-reasonfixed-budgetstate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the unreliable set-based aggregation of long-context language models—counting distinct items, comparing and grouping sets—can be remedied by maintaining a small, explicit, mergeable aggregation state beside a frozen model, built from a HyperLogLog sketch and updated while the context is read. The state is fixed at 2 KiB regardless of context length or distinct-item count, and its readouts (distinct count, union, Jaccard similarity, containment, group counts) are handed to the model as plain-text evidence, so no generate–execute–return cycle is needed. Across 3,969 aggregate-then-reason tasks from 174 source windows, the fixed-budget interface reached 99.2% accuracy on one model versus 100.0% for exact aggregation and 89.0% versus 93.0% on another, while far outperforming full-context reading and chain-of-thought. The point of the design is that set statistics become immediately available to the reasoner with predictable error following HyperLogLog's 1.04/√m law, rather than being recovered by averaging attention over long contexts.

What carries the argument

The central object is the HyperLogLog (HLL) sketch: a fixed array of registers (2,048 one-byte registers in the main runs) updated by hashing each canonical identity to a register and keeping the maximum of a rank value. Because the hash is deterministic, repeated occurrences of the same identity leave the state unchanged, so the state encodes support, not multiplicity. Segment merge is registerwise maximum, which provably reproduces the single-pass state; cardinality readout uses the standard HLL estimator with small-range correction; union readout merges two states temporarily; and Jaccard similarity and containment are estimated by joint maximum-likelihood estimation on paired registers.

What would settle it

Take the same aggregate-then-reason tasks but run the extractor on raw logs or prose rather than the released structured records, and compare end-to-end accuracy with exact aggregation over human-verified identities. If accuracy falls toward the full-context baseline as identity-extraction noise increases—or if a controlled perturbation that merges or splits identities moves downstream answers by more than the sketch's own error—the claim that a fixed-budget state suffices for long-context aggregation is falsified for realistic inputs.

Watch

Extended reading notes

Core claim

The central claim is that set-based aggregation over very long contexts can be performed model-side with a fixed-size, mergeable state that a frozen language model can use as evidence, without external execution. Concretely, a 2,048-register HyperLogLog state per stream tracks distinct identities with about 1.6% mean relative error on streams of one million records; states built from up to 256 segments merge into exactly the same register array as a single pass; and readouts appended to the prompt preserve most of the accuracy of exact aggregation on aggregate-then-reason tasks. The same states support distinct count, union, Jaccard similarity, containment, and grouped distinct count, with C

Load-bearing premise

The load-bearing premise is that an extractor can correctly map the relevant records in a context to canonical identities and route them to the right stream; all experiments start from already-structured records with precompiled operator specifications, so noisy or incomplete identity extraction from real prose would be inherited by the sketch and could shrink the reported gains.

Editorial extensions

If this is right

  • A fixed 2 KiB state per stream can summarize arbitrarily long identity streams for set queries, with relative error that stays flat as records grow (about 1.3–1.5% from 8K to 200K records in the reported runs).
  • Aggregation states computed independently from separate context segments merge exactly, which lets a long context be processed in chunks without re-reading and without any drift between the merged state and a single pass.
  • Supplying HLL readouts as explicit evidence recovers most of the accuracy of exact aggregation on aggregate-then-reason tasks (a 0.8-point gap on one model and 4.0 on the other) while using roughly 2 output tokens per task instead of hundreds-to-thousands for code generation.
  • The counterfactual evidence-use test indicates the models' final decisions track the supplied aggregate, so the fixed-budget state is functioning as decision evidence, not as an ignored side channel.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would infer that the practical ceiling is set by the extractor, not the sketch: if identity extraction from raw logs or prose is clean, the 2 KiB state carries nearly all the aggregation information needed; if extraction is noisy, those errors enter the state directly and the 99%-class numbers should be read as upper bounds.
  • A testable extension suggested by the design is to run the same pipeline with a learned extractor over raw text and measure end-to-end accuracy against human-verified exact aggregates; the paper's own boundary statement suggests this is the next open question.
  • The per-stream fixed budget means total memory still scales with the number of concurrently maintained groups; for high-cardinality group-bys, an indexed array of HLL states would be the natural follow-up, though it trades memory for grouping parallelism.
  • The same interface could be used to keep aggregation state across turns of a conversation or across a tool-use loop, since states are mergeable and do not require re-reading prior context—this is implied by the merge property but not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes SketchOps, an interface that equips a frozen large language model with fixed-size HyperLogLog (HLL) aggregation states. Section 3 formalizes identity-aware aggregation: an extractor E(x1:L, q) produces operand streams of canonical identities, each stream maintains an HLL state of m registers, updates are idempotent, and segment states merge by register-wise maximum. Readouts support distinct count, union, Jaccard similarity, containment, group-by, and (via Count-Min) point frequency. The empirical evaluation uses aggregate-then-reason tasks built from Oolong-Synth structured records: on 3,969 tasks, SketchOps reaches 89.0% (Qwen) and 99.2% (Gemma) versus 93.0% and 100.0% with exact aggregation; on 174 matched tasks it outperforms full-context baselines by large margins; on 1,200 tasks it uses far fewer output tokens than code execution at a small accuracy cost on Qwen. State-level experiments confirm that HLL errors follow the 1.04/sqrt(m) law, remain flat with stream length, and that segment merging is exact.

Significance. The paper is methodologically careful: it pins dataset/model revisions, preselects experimental settings on validation data, uses source-window-cluster bootstraps and randomization tests, includes an explicit counterfactual evidence-use check, and states its scope limitations clearly. At the state level, the HLL error measurements calibrate against the known 1.04/sqrt(m) law, and Proposition 4 is stated with a proof. If the extractor problem is set aside, the paper provides a clean demonstration that classical mergeable sketches can supply reliable set statistics to LLM reasoning with fixed memory and negligible output-token overhead. The practical significance, however, depends on the unevaluated extractor: as an end-to-end system, the current evidence is incomplete, and the contribution is better characterized as an aggregation interface for already-extracted streams.

major comments (2)
  1. [§3, Eq. (1); §5.1; Appendix G.2] The problem formulation and method assume an extractor E(x1:L, q) that maps raw context to canonical identity streams, but no experiment exercises extraction from raw prose, logs, or tables. All main evaluations consume released structured records with stable canonical identities and precompiled operator specifications, as Appendix G.2 explicitly acknowledges. The abstract, however, presents E as a functioning component ('an extractor maps each relevant record to a canonical identity'). This is load-bearing: extraction noise or canonicalization errors enter the HLL registers before the sketch can contribute, and the reported 0.8–4.0 percentage-point gaps versus exact aggregation are optimistic lower bounds conditional on perfect extraction. I recommend either adding an end-to-end evaluation with raw input and a noisy-extractor sensitivity analysis, or explicitly re-scoping the paper's cl
  2. [§4; Appendix E.1; Appendix G.2] The claim that the aggregation state is updated 'in parallel with the model's forward computation' is not operationalized or measured. The experimental protocol in Appendix E.1 states that the structured records and precompiled operator specifications are fixed before model inference, meaning the aggregates are precomputed rather than produced by a live model-side update. Because the extractor E is not implemented, there is no evidence that the interface avoids a separate extraction pass or an additional generate–execute–return cycle in practice. The comparison with code execution reports output tokens, not wall-clock time or total model calls. This is closely tied to the extractor gap and should be addressed either by implementing the online update or by qualifying the 'model-side operation' and 'no additional cycle' claims.
minor comments (4)
  1. [Abstract; Table 12; Appendix F.3] The abstract's 'mean relative error was 1.6%' uses a different error functional and replicate pool than Table 12, which reports 2.34% RMSE at m=2048. The appendix explains this, but the abstract alone is misleading. Please state the error functional and replication axis in the abstract or main text.
  2. [§4.1] The hash definition Hash(z) = (j(z), r(z)) is introduced without saying how many hash bits are used or that the rank is capped for a finite hash. The details are in Appendix C.1; moving a one-sentence summary to the main text would improve readability.
  3. [Figure 2a; Table 1] The bar label 'full ctx' in Figure 2a lumps together direct, CoT, and guided-choice baselines, which have different accuracies in Table 1. Please label the exact baseline used in the figure.
  4. [§4.1; Appendix D] The Count–Min extension for point-frequency queries is described but not evaluated in the main experiments. Consider labeling it explicitly as an extension not covered by the empirical validation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the HLL core and error laws are external; the evaluation is independent of the test set; the only self-citation is peripheral.

full rationale

The derivation chain is self-contained. The sketch update (Eq. 3), merge (Eq. 4), and readouts (Eqs. 5-6) are standard HLL/JMLE/Count-Min algorithms cited to external work (Flajolet et al. 2007; Ertl 2017; Cormode and Muthukrishnan 2005); the paper does not derive them from its own assumptions. Proposition 4 only verifies that the register-wise max used to define merge equals a single pass, which is an algebraic identity of max, not a fitted prediction. State-level errors (Tables 4-5, Appendix F.3) are measured against the external 1.04/sqrt(m) law and against UMBC on public data; the 2,048-register budget was fixed before test evaluation (Appendix E.2-E.3), and task answers were not used to select generator settings. The reasoning evaluations use the public Oolong-Synth test set, and the code-execution answers are independently re-derived by a CPU verifier. The only self-citation (Song and Wang 2026) appears in a related-work list of learned Bloom filters (Section 2.3) and is not used to justify any load-bearing premise, uniqueness claim, or ansatz. The paper explicitly limits its evaluation to released structured identities and precompiled operator specifications: Appendix G.2 states 'It does not establish the accuracy of extracting those identities from unrestricted prose.' That is an acknowledged assumption about the extractor E in Eq. 1, not a result obtained from it, so it affects external validity but is not circular. I find no equation defined in terms of its own output and no fitted parameter renamed as a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The core machinery is standard streaming-sketch mathematics; the only hand-set design knob affecting the claim is state size (and hash seed). The main unvalidated assumption is the extractor, which is a scoping gap rather than a new entity. No new physical or model-level inventions are introduced.

free parameters (2)
  • HLL register count m = 2048 (2 KiB)
    User-set budget controlling the accuracy/storage trade-off; main experiments use m=2048 and controlled sweeps vary m=256..8192. Not fitted, but central to the fixed-budget claim.
  • Hash seed = 0
    Chosen by hand; required for deterministic merge across segments. Arbitrary in principle and not fitted to data, but a fixed seed is an input assumption of the implementation.
assumptions (4)
  • standard math Standard HLL estimator correctness and asymptotic error ~1.04/sqrt(m)
    Used for all cardinality readouts (§4.2 and Appendix C.3); the experiment validates this against measured error.
  • domain assumption The fixed 64-bit keyed hash behaves like a random hash for the tested identity streams
    HLL error bounds and register-max merge rely on near-uniform register assignment. The paper uses a fixed seed (C.1) but does not test hash independence on the specific data.
  • domain assumption Existence of a reliable extractor E that yields canonical identities from context
    Eq. (1) assumes E; experiments bypass this by consuming released structured records (E.2, G.2). If this fails on real text, the interface is not end-to-end.
  • standard math JMLE joint-likelihood model for HLL pairs correctly estimates region counts
    Used for Jaccard and containment readouts (C.4); cited to Ertl 2017 and implemented as a deterministic optimizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mergeable Model-Side Aggregation States for Long-Context Language Models." pith.science (2026). https://pith.science/paper/F5KVT75H

@misc{pith2026260726448,
  author       = {Pith},
  title        = {Pith review of: Mergeable Model-Side Aggregation States for Long-Context Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5KVT75H}},
  note         = {Machine review of arXiv:2607.26448}
}
read the original abstract

A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include cardinality estimation, set relationships, and grouped statistics, which widely exist in logs, program outputs, tables, and multi-turn conversations. To provide the aggregation state required by these tasks, we introduce a model-side aggregation interface that maintains compact Hash-based HyperLogLog (HLL) sketch states alongside a frozen language model. While the model processes the context, an extractor maps each relevant record to a canonical identity. The identity is then hashed and updates the HLL state. These states can be merged across context segments and/or read out directly for downstream reasoning, avoiding an additional generate-execute-return cycle. We validate the proposed approach by setting the HLL state size as 2 KiB (2,048 registers), which does not increase with context length or set cardinality. In a distinct-count experiment involving one million records, the mean relative error was 1.6%. In a separate merge test, states built from as many as 256 segments produced exactly the same readout as a single pass over the same stream. On 3,969 aggregate-then-reason tasks from 174 source windows, the fixed-budget interface reached 99.2% accuracy on Gemma 4 (31B, BF16), compared with 100.0% under exact aggregation; the paired gap was 0.8 percentage points (95% window-cluster CI: 0.5-1.3 points). On a matched set of 174 items, our method improved over direct full-context reasoning by 63.2 points on Qwen and 56.3 points on Gemma. The corresponding gains over chain-of-thought (CoT) reasoning were 60.9 and 63.2 points, respectively. On a fixed 1,200-task Oolong-Synth subset, our method reached 91.1% on Qwen and 99.3% on Gemma. Code is available at https://github.com/songdc98/sketchops.

Figures

Figures reproduced from arXiv: 2607.26448 by the authors.

Figure 1
Figure 1. Model-side aggregation workflow. As the frozen language model processes the context, the extractor [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Main results. Colour is the model (blue Qwen 3.6 (35B), vermillion Gemma 4 (31B)); solid fill is SketchOps, pale fill the comparison method. (a) 174 matched tasks against the strongest full-context baseline. (b) All 3,969 tasks: the black tick is exact aggregation and the whisker the 95% window-cluster CI of the paired gap. (c) Mean output tokens per task against external code execution. (d, e) State level, no langu… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 8 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2511.02817 , year=

    Oolong: Evaluating long context reasoning and aggregation capabilities , author=. arXiv preprint arXiv:2511.02817 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2601.22488 , year=

    Elastic Spectral State Space Models for Budgeted Inference , author=. arXiv preprint arXiv:2601.22488 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2006.03176 , year=

    Partitioned learned bloom filter , author=. arXiv preprint arXiv:2006.03176 , year=

    arXiv 2006

  4. [4]

    arXiv preprint arXiv:2404.06654 , year=

    RULER: What's the real context size of your long-context language models? , author=. arXiv preprint arXiv:2404.06654 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2511.05722 , year=

    OckBench: Measuring the Efficiency of LLM Reasoning , author=. arXiv preprint arXiv:2511.05722 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2603.20432 , year=

    Coding agents are effective long-context processors , author=. arXiv preprint arXiv:2603.20432 , year=

    arXiv

  7. [7]

    International conference on machine learning , pages=

    Pal: Program-aided language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  8. [8]

    arXiv preprint arXiv:2211.12588 , year=

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks , author=. arXiv preprint arXiv:2211.12588 , year=

    arXiv

Show all 28 references
  1. [9]

    How Good Are

    Kate, Kiran and Rizk, Yara and Ghosh, Poulami and Gulati, Ashu and Chakraborti, Tathagata and Wright, Zidane and Agarwal, Mayank , booktitle =. How Good Are. 2026 , doi =

    2026

  2. [10]

    arXiv preprint arXiv:2606.17094 , year=

    LogCopilot: Automating Log Aggregation Analysis through Large Language Models , author=. arXiv preprint arXiv:2606.17094 , year=

    arXiv

  3. [11]

    Conference on Analysis of Algorithms (AofA) , pages =

    Flajolet, Philippe and Fusy,. Conference on Analysis of Algorithms (AofA) , pages =. 2007 , doi =

    2007

  4. [12]

    Proceedings

    On the resemblance and containment of documents , author=. Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171) , pages=. 1997 , organization=

    1997

  5. [13]

    International Workshop on Randomization and Approximation Techniques in Computer Science , pages=

    Counting distinct elements in a data stream , author=. International Workshop on Randomization and Approximation Techniques in Computer Science , pages=. 2002 , organization=

    2002

  6. [14]

    arXiv preprint arXiv:2101.00314 , year=

    SetSketch: filling the gap between MinHash and HyperLogLog , author=. arXiv preprint arXiv:2101.00314 , year=

    arXiv

  7. [15]

    Journal of Algorithms , volume=

    An improved data stream summary: the count-min sketch and its applications , author=. Journal of Algorithms , volume=. 2005 , publisher=

    2005

  8. [16]

    Advances in neural information processing systems , volume=

    Deep sets , author=. Advances in neural information processing systems , volume=

  9. [17]

    International conference on machine learning , pages=

    Set transformer: A framework for attention-based permutation-invariant neural networks , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  10. [18]

    International Conference on Machine Learning , pages=

    Scalable set encoding with universal mini-batch consistency and unbiased full set gradient approximation , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  11. [19]

    International conference on machine learning , pages=

    On the limitations of representing functions on sets , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  12. [20]

    International conference on machine learning , pages=

    Meta-learning neural bloom filters , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  13. [21]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Meta-sketch: A neural data structure for estimating item frequencies of data streams , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  14. [22]

    arXiv preprint arXiv:2605.15571 , year=

    MaxSketch: Robust Distinct Counting in Streams via Random Projections , author=. arXiv preprint arXiv:2605.15571 , year=

    arXiv

  15. [23]

    Communications of the ACM , volume=

    A relational model of data for large shared data banks , author=. Communications of the ACM , volume=. 1970 , publisher=

    1970

  16. [24]

    arXiv preprint arXiv:1702.01284 , year=

    New cardinality estimation algorithms for HyperLogLog sketches , author=. arXiv preprint arXiv:1702.01284 , year=

    arXiv

  17. [25]

    IJCAI: proceedings of the conference , volume=

    Improving attention mechanism in graph neural networks via cardinality preservation , author=. IJCAI: proceedings of the conference , volume=

  18. [26]

    Advances in neural information processing systems , volume=

    Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=

  19. [27]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  20. [28]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Grammar-constrained decoding for structured NLP tasks without finetuning , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.