{"id":"582519c4-d8c1-40ea-99a6-6b71f2b90e53","arxiv_id":"2505.04153","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A purpose-built concurrent hash table using ticket-based indirection matches or beats partitioned group-by aggregation in morsel-driven databases on tested synthetic workloads.","lead":"The paper tests a simpler alternative to the usual way databases split up group-by work: a shared hash table that all CPU threads use at once. It finds that a purpose-built version of this old idea can match or beat the standard partitioned approach on several synthetic workloads, which could change how database engines implement aggregation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance claim depends on perfect cardinality estimation; the one forced-resize test in Section 4.5 shows a 5.2x latency hit for unique keys, so practical viability as stated is not established.","rationale":"I agree with the reader's weakest-assumption analysis; the perfect-cardinality-estimation condition in Section 2.4 is the most load-bearing assumption. The microbenchmark findings (Folklore* beating general-purpose maps, thread-local vs. atomic update tradeoffs) are internally consistent and supported by multi-platform measurements and an available artifact, so I would not weaken the verdict to REJECT. The concern is that the headline practical claim is stated more broadly than the experiments support: exact sizing is not available to a real optimizer, and Section 4.5 shows the fully concurrent method is highly sensitive to forced resizing (5.2x for unique keys with atomic updates). Because the paper does not compare against partitioning under the resized condition, the central advantage could vanish under a realistic 2x estimation error. A targeted experiment measuring the throughput crossover under controlled mis-sizing would settle this. This keeps the verdict CONDITIONAL, unchanged from the reader.","tokens_in":17277,"tokens_out":8423,"duration_ms":89232,"concrete_test":"Re-run the Figure 7 end-to-end comparison with the fully concurrent ticketing table and aggregate vectors sized at 50%, 75%, 125%, and 200% of true cardinality (one and multiple forced resizes), while keeping the partitioned baseline at its best configuration. Record the sizing-error crossover point for unique-key and high-cardinality workloads. If fully concurrent throughput drops below partitioned throughput at any error below 2x, the practical claim does not survive realistic estimator error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 states that all end-to-end experiments except Section 4.5 assume perfect cardinality estimation and therefore perfectly sized hash tables and partial aggregate vectors. This is the condition on which the central claim rests. Section 4.5 is the only robustness test: it forces a single resize by allocating at half the required capacity. Even that single resize produces a 5.2x latency increase for unique keys with atomic updates and a 2.0x increase for thread-local updates at 48 threads. The paper does not report the resized configuration against the partitioned baseline, so the unique-key advantage visible in Figure 7 is not known to survive a 5.2x penalty. Real cardinality estimates are often off by more than 2x after filters, joins, and on skewed data, and larger underestimates would require multiple resizes that Section 4.5 does not test. The paper honestly labels resizing as future work, but the stated conclusion that fully concurrent aggregation is a viable alternative to partitioning therefore requires accurate cardinality estimates or resize costs that have not been demonstrated. This is a load-bearing condition, not a cosmetic caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits fully concurrent hash-based GROUP BY aggregation, where a shared global hash table maps each key to a ticket that indexes a vector of partial aggregates, and compares it against the partitioned aggregation approach with local preaggregation used in morsel-driven engines. The authors evaluate several concurrent hash table designs for the ticketing step, compare atomic, locking, and thread-local update methods, and then report end-to-end scaling experiments on AMD, ARM, and Intel platforms across low, high, and unique cardinality datasets with skew variants, along with profiling, resizing, memory, and tuple-size analyses. The main claim is that a simple purpose-built linear-probing hash table with a customized atomic get-or-insert operation makes fully concurrent aggregation competitive with, and often faster than, partitioning-based aggregation, except in the acknowledged heavy-hitter regime where the partitioned baseline retains an advantage.","tokens_in":17495,"tokens_out":3910,"duration_ms":41082,"significance":"If the stated results hold, the paper provides a concrete, implementable alternative to partitioned GROUP BY aggregation, with useful practical guidance: linear probing suffices for the ticketing workload, lockless fast paths are essential, a fuzzy ticket counter avoids atomic-counter contention, and thread-local updates are robust under skew but memory-hungry at high thread counts. The study is extensive and well-designed in several respects: it covers three architectures, uses top-down profiling and performance counters, includes resizing and peak-memory analyses, and makes the source code and artifacts available. The paper is also careful to acknowledge exceptions (e.g., heavy hitters at high thread counts) and to present its conclusions in a balanced way. The main weakness is that the overarching practical-viability claim depends on an assumption of perfect cardinality estimation, and the paper's single robustness test for this assumption shows a large performance penalty in the worst case.","major_comments":[{"comment":"The central claim that fully concurrent aggregation is a viable alternative to partitioning rests on the assumption, stated in Section 2.4, that all end-to-end experiments except Section 4.5 assume perfect cardinality estimation and therefore perfectly sized hash tables and partial aggregate vectors. Section 4.5, the only robustness test, forces a single resize by allocating the ticketing table at half the required capacity. The resulting latency increase is substantial: a 5.2x slowdown for unique keys with atomic updates and 2.0x for thread-local updates at 48 threads (Figure 13). Because the paper does not report the resized configuration against the partitioned baseline, the unique-key advantage visible in Figure 7 is not known to survive this penalty. Real cardinality estimates after filters, joins, and on skewed data are often off by more than a factor of two, and larger underestimates would trigger multiple resizes, which Section 4.5 does not test. The manuscript labels resizing as future work, but the stated conclusion that fully concurrent aggregation is a viable alternative requires either experimentally demonstrated resize costs or a scoping of the claim to workloads with accurate cardinality estimates. This is a load-bearing condition, not a cosmetic caveat.","section":"§2.4, §4.5"},{"comment":"The paper's recommendation to choose the update method adaptively based on the number of unique keys (Section 3.2, Table 2) is derived from experiments that assume exact cardinality knowledge: the isolated update benchmarks in Section 3.2 use a perfect hash function (ticket = key), and the end-to-end experiments in Section 4.1 assume perfectly sized structures. The practical guidelines in the 'Recommendations' paragraph of Section 4.1 therefore presume that the optimizer's cardinality estimate is reliable. The paper does not evaluate how estimation error affects the choice between atomic and thread-local updates, nor how the 5.2x resize penalty interacts with the switch between methods. Without such an evaluation, the adaptive-guidance claim is only as strong as the perfect-estimation assumption. I would ask the authors to either test the adaptive policy under noisy cardinality estimates or explicitly restrict the guidelines to settings where cardinality estimates are known to be accurate.","section":"§3.2, §4.1"}],"minor_comments":[{"comment":"In the pseudocode, the expression 'table.[idx].k.load()' appears to contain a stray dot; it should likely read 'table[idx].k.load()'.","section":"Algorithm 1"},{"comment":"The sentence 'the fully concurrent workload is does display significant performance degradation' contains a verb duplication; also, the final sentence 'Resizing performance and should be an important dimension' is grammatically incomplete.","section":"§4.5"},{"comment":"The term 'zero-allocation' is used to describe calloc-style zero-initialized memory that enables copy-on-write; 'zero-initialized allocation' would be less ambiguous and would avoid implying that no allocation occurs.","section":"§3.1"},{"comment":"The comparison baseline is an in-house implementation of partitioned aggregation, not a production engine's aggregation path; a note about how this choice affects generalizability would be helpful, since real engines may have additional optimizations or overheads.","section":"§4"},{"comment":"Reference [16] is cited without a full title; listing the actual paper title would improve completeness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid experimental study and the authors are honest about many limitations, but the central 'viable alternative' claim is conditioned on perfect cardinality estimation in a way that the single resize experiment (§4.5) does not fully validate. I would not reject; a major revision that either adds resize-robustness experiments against the partitioned baseline or explicitly scopes the claim to perfectly estimated cardinalities would address the concern. The self-implemented baseline is a secondary concern; a comparison with at least one production engine's aggregation operator would strengthen external validity but is not strictly required for the paper's core argument."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Daniel, here's my take on Xue and Marcus (arXiv:2505.04153). The headline claim—that a purpose-built shared hash table with ticket indirection can match or beat partitioned aggregation for GROUP BY in morsel-driven engines—is supported by the measurements. But the paper's own resize experiment shows the result depends on near-perfect cardinality estimates, so the practical viability claim is conditional.\n\nWhat's genuinely new: the observation that ticketing reduces the concurrent hash table workload to lookup/insert-only semantics, and the resulting Folklore* design (single-word CAS linear probing with a fuzzy ticketer). The paper is honest about the failure cases: heavy hitters at high thread counts still favor partitioned aggregation, and the update-step tradeoffs are laid out clearly. The experimental breadth is a real strength: three architectures, multiple cardinalities and skew distributions, top-down profiling, and memory/resizing analyses. The measurements look careful, the artifact is public, and there are no fitted parameters masquerading as predictions.\n\nThe soft spot is load-bearing: all end-to-end results except Section 4.5 assume perfect cardinality estimation. When a resize is forced (half the required capacity), unique keys with atomic updates take a 5.2x latency hit at 48 threads; thread-local takes 2.0x. The paper doesn't show the resized concurrent configuration against the partitioned baseline, so we don't know whether the central advantage survives a realistic estimation error. Real estimates are often off by more than 2x after filters and joins, and larger errors would trigger multiple resizes, which aren't tested. The paper flags resizing as future work, which is honest, but it does bound the strength of the conclusion.\n\nOne more moderate concern: the partitioned baseline is an in-house implementation. That's standard for this type of controlled comparison, but a cross-check against a real system's aggregation path (e.g., DuckDB's) would make the parity claim more convincing. This is a fairness note, not a dismissal.\n\nOverall: the paper is a solid, readable systems study with a real negative lesson too—there's no single best method. The perfect-estimation caveat makes me hesitant to say 'deploy this in production tomorrow,' but the analysis is rigorous enough to take seriously. I'd send it to a serious referee. For a reading group, it's worth a slot.","headline":"A well-executed experimental study showing purpose-built shared hash tables can match partitioned GROUP BY, but the perfect-cardinality-estimation assumption makes the practical claim conditional.","tokens_in":17987,"tokens_out":2740,"would_cite":true,"duration_ms":26714,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fully concurrent GROUP BY using a single purpose-built hash table can match or beat partitioning-based aggregation in morsel-driven engines when the table is specialized for the lookup-and-insert ticketing workload.","keywords":["group by aggregation","parallel query processing","concurrent hash table","linear probing","ticketing","partitioned aggregation","morsel-driven execution","OLAP"],"falsifier":"Run an end-to-end comparison on the paper's own high-cardinality dataset (10 million unique keys, SUM) at 48 threads while forcing the fully concurrent implementation to start with a hash table sized at one-quarter of the true cardinality, forcing multiple resizes, and measure whether the fully concurrent method still matches partitioned aggregation; a decisive slowdown would show the central claim holds only under the paper's perfect-sizing assumption.","tokens_in":17075,"feed_emoji":"🗃️","tokens_out":8187,"duration_ms":73105,"temperature":0.7,"pith_summary":"Parallel GROUP BY in modern analytic databases is usually done by partitioning the input so each thread processes only its own keys, which adds a second aggregation pass and more memory. This paper argues that a simpler alternative—a single shared hash table used by all threads—has been wrongly dismissed because it was implemented with general-purpose concurrent hash tables, which carry deletes, shrinking, and mixed-workload support that aggregation never needs. The authors build a purpose-built, lock-free linear-probing table with an atomic get-or-insert 'ticketing' operation and show that, in a morsel-driven execution engine, it matches or beats partitioning across synthetic workloads of different cardinality and skew, with a heavy-hitter extreme as the main exception. They also provide a practical characterisation of the remaining design choices: atomic partial-aggregate updates for high-cardinality inputs, thread-local updates for skewed inputs, capped thread counts for the thread-local variant, and the continued importance of accurate cardinality estimation to avoid costly resizing.","feed_headline":"Shared hash table matches partitioning when built for GROUP BY","feed_subtitle":"Specializing the table to lookups and inserts removes the contention that doomed earlier shared-table aggregation.","key_machinery":"The central object is the Folklore* hash table: a lock-free linear-probing hash table that supports only a single atomic get-or-insert operation, in which a key is looked up and, if absent, assigned an integer ticket via a one-word compare-and-swap on a pre-reserved slot state. A 'fuzzy ticketer' gives each thread a private range of ticket values so the shared counter is only touched when a range is exhausted, and zero-allocation (calloc-style) defers table initialization so allocation leaves the single-threaded critical path. Tickets index a dense vector of partial aggregates, which are updated either atomically or in per-thread vectors; the table also stores a copy of keys in ticket order for final materialization. This combination reduces the concurrent data structure's job to the minimal lookup-and-insert workload that group aggregation actually requires.","core_discovery":"The paper's central claim is that the long-standing verdict against shared-hash-table aggregation stems from using general-purpose concurrent hash tables, which must support deletes, shrinking, and mixed workloads; a table written only for the lookup-and-insert pattern of group aggregation removes the scalability barrier. Concretely, the authors show that their Folklore* linear-probing table, paired with a fuzzy ticket counter and zero-allocation, achieves a 37.6x speedup on low-cardinality workloads at 48 threads and, end-to-end, a fully concurrent implementation reaches parity or beats the partitioned baseline on synthetic workloads, with a heavy-hitter extreme as the notable exception. They also document the operational tradeoffs: atomic updates win when keys are unique, thread-local updates win under skew, and resizing remains the weak spot.","pith_inferences":["Viewing ticketing as incremental perfect hashing suggests that offline perfect hash functions, already proposed for OLAP aggregation, could eliminate ticketing contention entirely—an extension the paper names but does not implement.","A hybrid update strategy, using thread-local aggregation for heavy hitters and atomic updates for the rest, would likely close the heavy-hitter gap that is the paper's clearest exception; such hybrids already exist in the aggregation literature.","The paper's results were obtained under perfect cardinality estimation; feeding the same implementation with the estimation errors of a real optimizer would be a natural stress test and could change the recommended operating region.","Because the benchmark uses only morsel-driven execution, the conclusions may not transfer to operator-level or pipeline-parallel engines, where the thread-to-data assignment and synchronization patterns differ."],"forward_implications":["A database engine can implement parallel GROUP BY with a single shared hash table instead of partitioning, eliminating the second aggregation pass and the associated spilling and memory overhead in many workloads.","Per-query choice of update method matters: atomic updates win on high-cardinality, unique-key inputs, while thread-local updates win under skew and low cardinality; the paper's measurements quantify when each should be selected.","Implementers should cap the thread count for thread-local updates to avoid inverse scaling from the materialization merge, and should prefer zero-allocation for large hash tables to keep allocation off the single-threaded path.","Resizing is the main remaining weakness: with tables sized at half the required capacity, end-to-end latency rises by up to 5.2x for unique keys, so accurate cardinality estimation is a prerequisite for the approach's practical viability."],"supporting_citations":[{"why":"Supplies the ticketing-and-indirection model: keys map to integer tickets that index partial aggregate vectors.","marker":"[4]"},{"why":"Defines morsel-driven parallel execution, the assumed execution model, and the partitioned aggregation baseline.","marker":"[20]"},{"why":"Introduces the partitioning-with-preaggregation strategy that serves as the baseline for comparison.","marker":"[29]"},{"why":"Provides the lock-free Folklore hash table that the paper adapts and the resize-migration approach used in the resizing experiment.","marker":"[23]"},{"why":"Introduces local preaggregation for skew resistance, the key mechanism of the partitioned baseline.","marker":"[35]"},{"why":"Shows a real analytic system adopting the partitioned approach, motivating the claim that the comparison targets current practice.","marker":"[19]"},{"why":"Supplies a state-of-the-art concurrent cuckoo hash table baseline whose scaling problems motivate the purpose-built design.","marker":"[21]"},{"why":"Documents partitioned aggregation in a production morsel-driven system, grounding the relevance of the comparison.","marker":"[17]"},{"why":"Iceberg hashing, a modern high-performance hash table, is benchmarked as a ticketing candidate alongside the authors' table.","marker":"[26]"}],"fun_headline_variants":["Specialized hash table overturns partitioning's edge for GROUP BY","Concurrent hash table matches partitioning when tailored to GROUP BY","Custom hash table for GROUP BY rivals partitioning at 48 threads","Fully concurrent GROUP BY: shared table beats partitioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed advantage assumes the database knows the number of distinct keys ahead of time so the hash table and partial aggregate vectors are allocated at exactly the right size; in the forced-resize test at half capacity, latency rises by up to 5.2x in the unique-keys case, so the practical viability claim depends on accurate cardinality estimates.","fun_headline_variants_meta":{"raw":{"variants":["Specialized hash table overturns partitioning's edge for GROUP BY","Concurrent hash table matches partitioning when tailored to GROUP BY","Custom hash table for GROUP BY rivals partitioning at 48 threads","Fully concurrent GROUP BY: shared table beats partitioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1490,"prompt_tokens":887,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":503,"tokens_out":603,"duration_ms":6935,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:35:32.993737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an end-to-end comparison on the paper's own high-cardinality dataset (10 million unique keys, SUM) at 48 threads while forcing the fully concurrent implementation to start with a hash table sized at one-quarter of the true cardinality, forcing multiple resizes, and measure whether the fully concurrent method still matches partitioned aggregation; a decisive slowdown would show the central claim holds only under the paper's perfect-sizing assumption.","supporting_citations":[{"cited_title":"Boncz, Marcin Zukowski, and Niels Nes","cited_arxiv_id":null,"evidence_quote":"Supplies the ticketing-and-indirection model: keys map to integer tickets that index partial aggregate vectors."},{"cited_title":"Ross, and Norases Vesdapunt","cited_arxiv_id":null,"evidence_quote":"Introduces local preaggregation for skew resistance, the key mechanism of the partitioned baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows a real analytic system adopting the partitioned approach, motivating the claim that the comparison targets current practice."},{"cited_title":"Andersen, Michael Kaminsky, and Michael J","cited_arxiv_id":null,"evidence_quote":"Supplies a state-of-the-art concurrent cuckoo hash table baseline whose scaling problems motivate the purpose-built design."},{"cited_title":"Bender, Alex Conway, Martin Farach-Colton, William Kuszmaul, Guido Tagliavini, and Rob Johnson","cited_arxiv_id":null,"evidence_quote":"Iceberg hashing, a modern high-performance hash table, is benchmarked as a ticketing candidate alongside the authors' table."}],"review_version":1}