Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines

T0 review · 2 major / 1 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Puffin files attached to Iceberg snapshots can carry full Vamana graphs at billion-vector scale.

desk verdict The paper gives a concrete pattern for binding sharded Vamana indexes to Iceberg snapshots via Puffin, but the performance claims rest on projections rather than measurements. read the letter →

arxiv 2606.04196 v1 pith:UANVZCO5 submitted 2026-06-02 cs.DB

Artur Borycki This is my paper · ORCID
classification cs.DB
keywords ApacheIcebergPuffinformatapproximatenearestneighborVamanagraphsvectorsimilaritysearchcomputedisaggregationDiskANNsnapshotmanagement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows how to embed distributed approximate nearest neighbor indexes inside Apache Iceberg tables by storing them in Puffin sidecar files and binding them through the existing snapshot summary property. This turns index creation, refresh, and deletion into ordinary Iceberg snapshot operations, so the indexes inherit atomicity, time travel, and garbage collection without any new consistency layer. The approach targets compute-disaggregated engines where executors are stateless and read everything from object storage, avoiding the need for a separate dedicated index store. A sharded layout places small centroid indexes on the coordinator and large DiskANN graphs on executor SSDs, with protocols for distributed build and probe that fit inside the catalog's optimistic commit path.

What carries the argument

Puffin sidecar files bound to Iceberg snapshots via the statistics file snapshot summary property, holding sharded Vamana graphs with a binary layout for coordinator-executor probe.

What would settle it

Measure recall at fixed latency on a 10^9-vector table built and probed with the described sharded protocol; if the numbers fall below the projected trade-off curve, the design does not hold.

Watch

Extended reading notes

Core claim

The Puffin format, originally introduced for portable level statistics and deletion vectors, is sufficient to carry full Vamana graphs at billion vector scale, and that linking these blobs through the existing statistics file snapshot summary property reduces ANN index management to standard Iceberg snapshot operations.

Load-bearing premise

The independent-shard design and tiered probe strategy will deliver acceptable recall and latency without introducing consistency or performance problems that break the disaggregated model.

Editorial extensions

If this is right

  • ANN index management collapses to standard Iceberg snapshot commit and orphan-file cleanup.
  • Atomicity, time travel, and multi-engine readability apply to the vector indexes at zero extra cost.
  • The build and probe protocols integrate directly into the optimistic-concurrency path of an Iceberg REST catalog.
  • Projected query and build performance is quantified for tables up to 10^9 vectors under the independent-shard model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same binding pattern could attach other index structures if they fit the Puffin binary layout.
  • Executor-local SSD capacity becomes a first-class resource for the large-graph tier in disaggregated deployments.
  • The approach may generalize to any table format that already supports sidecar files and mutable snapshot metadata.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper claims that Puffin files (originally for statistics and deletion vectors) can store full Vamana/DiskANN graphs at billion-vector scale, and that binding these via the existing Iceberg snapshot-summary property reduces ANN index management to standard Iceberg snapshot operations. It presents a binary layout for sharded graph indexes inside Puffin, a coordinator-executor protocol for build/probe/refresh, integration with Iceberg's optimistic-concurrency commit path, a tiered probe strategy (small centroid indexes on coordinator, large graphs on executor SSDs), and projected recall/latency/build numbers for tables up to 10^9 vectors. The implementation extends FlockDB and inherits atomicity, time travel, and orphan-file GC from Iceberg.

Significance. If the design and performance projections hold, the work enables vector similarity search inside compute-disaggregated analytical engines without a separate index storage layer or consistency model, at essentially zero additional implementation cost. The explicit reuse of Puffin and snapshot-summary mechanisms is a concrete strength that avoids new lifecycle or operational surfaces.

major comments (2)
  1. [Abstract / Evaluation] Abstract and evaluation sections: the manuscript supplies only projected performance numbers rather than measured results from the FlockDB implementation. This leaves the recall/latency claims for the independent-shard tiered-probe design unverified, which is load-bearing for the central assertion that the pattern delivers acceptable performance while preserving disaggregation.
  2. [Tiered probe strategy] Tiered probe strategy description: the independent-shard design (small centroid indexes on coordinator, full Vamana/DiskANN graphs on executor SSDs) plus the extra network hop and coordinator summary requirement are asserted to produce acceptable recall without partition-induced loss or load imbalance, but no concrete analysis, bounds, or fallback mechanisms are supplied to support this at the claimed scale.
minor comments (1)
  1. [Binary layout / Coordinator protocol] The binary layout and coordinator protocol sections would benefit from an explicit diagram or pseudocode for the commit-path integration to make the optimistic-concurrency handling clearer.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the potential of reusing Puffin and snapshot-summary mechanisms. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract / Evaluation] Abstract and evaluation sections: the manuscript supplies only projected performance numbers rather than measured results from the FlockDB implementation. This leaves the recall/latency claims for the independent-shard tiered-probe design unverified, which is load-bearing for the central assertion that the pattern delivers acceptable performance while preserving disaggregation.

    Authors: We agree the evaluation section relies on projected numbers derived from published Vamana/DiskANN scaling behavior and micro-benchmarks of the Puffin I/O path rather than end-to-end measurements on the FlockDB prototype. This is a genuine limitation of the current draft. In the revised manuscript we will (1) explicitly label all numbers as projections with their derivation sources and (2) add a short subsection describing the measurement plan once the tiered-probe implementation is complete. We do not claim the projections constitute verification; they serve only to illustrate order-of-magnitude feasibility. revision: yes

  2. Referee: [Tiered probe strategy] Tiered probe strategy description: the independent-shard design (small centroid indexes on coordinator, full Vamana/DiskANN graphs on executor SSDs) plus the extra network hop and coordinator summary requirement are asserted to produce acceptable recall without partition-induced loss or load imbalance, but no concrete analysis, bounds, or fallback mechanisms are supplied to support this at the claimed scale.

    Authors: The manuscript contains a qualitative discussion of recall/latency trade-offs for independent shards, but we acknowledge it supplies neither quantitative bounds on recall degradation nor explicit fallback mechanisms. In revision we will add (a) a short probabilistic bound on routing error given centroid sampling density and (b) a description of a simple coordinator-side load threshold that triggers fallback to a full-shard probe. These additions will directly address the request for concrete analysis. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: design pattern on existing Iceberg/Puffin mechanisms

full rationale

The paper describes an engineering integration that attaches Vamana/DiskANN graphs to Puffin sidecar files and binds them via the existing snapshot-summary property. All claimed properties (atomicity, time-travel, orphan-file GC) are inherited directly from Iceberg’s commit path and file catalog; no equations, fitted parameters, or self-referential definitions appear. The independent-shard tiered-probe strategy is presented as an explicit design choice whose recall/latency behavior is left as an engineering assumption rather than derived from the paper’s own inputs. No self-citations are load-bearing, and no result is renamed or smuggled in via prior author work. The derivation chain is therefore self-contained against external Iceberg semantics.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new free parameters, invented entities, or ad-hoc axioms beyond standard assumptions about the Iceberg table format and Puffin sidecar capabilities.

assumptions (2)
  • domain assumption Puffin sidecar files can store full Vamana graphs at billion-vector scale without format or performance limitations
    Invoked when claiming sufficiency for carrying the indexes.
  • domain assumption Linking indexes through snapshot summary properties preserves Iceberg atomicity and lifecycle semantics
    Central to the claim that management reduces to standard snapshot operations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines." pith.science (2026). https://pith.science/paper/UANVZCO5

@misc{pith2026260604196,
  author       = {Pith},
  title        = {Pith review of: Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UANVZCO5}},
  note         = {Machine review of arXiv:2606.04196}
}
read the original abstract

We describe a design pattern and concrete implementation for embedding distributed approximate nearest neighbor indexes inside the Apache Iceberg table format, using the Puffin sidecar file as the storage container and the snapshot summary as the binding mechanism. Modern analytical query engines increasingly adopt a compute disaggregated architecture: executors are stateless, scale elastically, and read all data from object storage. Adding vector similarity search to such an engine traditionally requires a dedicated index storage layer with its own lifecycle, consistency model, and operational surface breaking the disaggregation in variant. We show that the Puffin format, originally introduced portable level statistics and deletion vectors, is sufficient to carry full Vamana graphs at billion vector scale, and that linking these blobs through the existing statistics file snapshot summary property reduces ANN index management to standard Iceberg snapshot operations. We present a binary layout for sharded graph indexes inside Puffin, a coordinator executor protocol for distributed index build, probe, and incremental refresh, the integration into the existing optimistic-concurrency commit path of an Iceberg REST catalog, and a tiered probe strategy that places small centroid indexes on the coordinator and large DiskANN graphs on executor SSDs. The pattern inherits atomicity, time travel, multi engine read ability, and orphan file garbage collection from the table format at zero implementation cost. We discuss the recall/latency trade-offs introduced by the independent-shard design and quantify projected query and build performance for tables up to 109 vectors. Our implementation extends FlockDB, a distributed MPP engine built on DuckDB.

Figures

Figures reproduced from arXiv: 2606.04196 by the authors.

Figure 1
Figure 1. System architecture. Compute (client, coordinator, executors) is stateless and elastic; [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Puffin file layout for a sharded Vamana index. The Iceberg snapshot binds to the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Three-stage distributed ANN probe. Stage A runs an approximate Vamana beam [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN

    cs.DB 2026-08 conditional novelty 6.0 of 10

    Per-file IVF indexes inside Parquet footers, combined with Iceberg file pruning, provide fast filtered ANN search in a disaggregated lakehouse when the filter column is file-local.

Reference graph

Works this paper leans on

21 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Apache Hudi: Streaming data lake platform.https://hudi

    Apache Software Foundation. Apache Hudi: Streaming data lake platform.https://hudi. apache.org. Accessed 2026-04

    2026

  2. [2]

    Apache Iceberg Puffin format specification.https:// iceberg.apache.org/puffin-spec/

    Apache Software Foundation. Apache Iceberg Puffin format specification.https:// iceberg.apache.org/puffin-spec/. Accessed 2026-04

    2026

  3. [3]

    Apache Iceberg table specification.https://iceberg

    Apache Software Foundation. Apache Iceberg table specification.https://iceberg. apache.org/spec/. Accessed 2026-04

    2026

  4. [4]

    ApacheParquet: Columnarstorageforthehadoopecosystem

    ApacheSoftwareFoundation. ApacheParquet: Columnarstorageforthehadoopecosystem. https://parquet.apache.org. Accessed 2026-04

    2026

  5. [5]

    Delta lake: High- performance ACID table storage over cloud object stores.Proceedings of the VLDB En- dowment, 13(12):3411–3424, 2020

    Michael Armbrust, Tathagata Das, Liwen Sun, Burak Yavuz, Shixiong Zhu, Mukul Murthy, Joseph Torres, Herman van Hovell, Adrian Ionescu, Alicja Łuszczak, Michal Switakowski, Michal Szafranski, Xiao Li, Takuya Ueshin, Mostafa Mokhtar, Peter Boncz, Ali Ghodsi, Sameer Paranjpye, Pieter Senster, Reynold Xin, and Matei Zaharia. Delta lake: High- performance ACID...

    2020

  6. [6]

    Lakehouse: A new gener- ation of open platforms that unify data warehousing and advanced analytics

    Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia. Lakehouse: A new gener- ation of open platforms that unify data warehousing and advanced analytics. InProceedings of the Conference on Innovative Data Systems Research (CIDR), 2021

    2021

  7. [7]

    SPANN: Highly-efficient billion-scale approximate nearest neighborhood search

    Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zheng Li, Mao Yang, and Jingdong Wang. SPANN: Highly-efficient billion-scale approximate nearest neighborhood search. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  8. [8]

    Lee, Ashish Motivala, Abdul Q

    Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Alli- son W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, and Philipp Unterbrunner. The snowflake elastic data warehouse. InProceedings...

    2016

Show all 21 references
  1. [9]

    The Faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. arXiv preprint arXiv:2401.08281, 2024

    2024 arXiv

  2. [10]

    Product quantization for nearest neigh- bor search.IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117– 128, 2011

    Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neigh- bor search.IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117– 128, 2011

    2011

  3. [11]

    pgvector: Open-source vector similarity search for Postgres.https:// github.com/pgvector/pgvector

    Andrew Kane. pgvector: Open-source vector similarity search for Postgres.https:// github.com/pgvector/pgvector. Accessed 2026-04. 13

    2026

  4. [12]

    Chi, Jeffrey Dean, and Neoklis Polyzotis

    Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. InProceedings of the ACM SIGMOD International Conference on Management of Data, 2018

    2018

  5. [13]

    YuryA.MalkovandDmitryA.Yashunin. Efficientandrobustapproximatenearestneighbor search using hierarchical navigable small world graphs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2018

    2018

  6. [14]

    DiskANN: Graph-structured indices for scalable, fast, fresh and filtered approximate nearest neighbor search.https://github.com/microsoft/DiskANN

    Microsoft Research. DiskANN: Graph-structured indices for scalable, fast, fresh and filtered approximate nearest neighbor search.https://github.com/microsoft/DiskANN. Accessed 2026-04

    2026

  7. [15]

    DuckDB: An embeddable analytical database

    Mark Raasveldt and Hannes Mühleisen. DuckDB: An embeddable analytical database. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pages 1981–1984, 2019

    1981

  8. [16]

    DiskANN: Fast accurate billion-point nearest neighbor search on a single node

    Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krish- naswamy, and Rohan Kadekodi. DiskANN: Fast accurate billion-point nearest neighbor search on a single node. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  9. [17]

    Vespa: The open big data serving engine.https://vespa.ai

    Verizon Media. Vespa: The open big data serving engine.https://vespa.ai. Accessed 2026-04

    2026

  10. [18]

    Building an elastic query engine on disaggregated storage

    Midhul Vuppalapati, Justin Miron, Rachit Agarwal, Dan Truong, Ashish Motivala, and Thierry Cruanes. Building an elastic query engine on disaggregated storage. InProceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2020

    2020

  11. [19]

    Milvus: A purpose-built vector data management system

    Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, Kun Yu, Yuxing Yuan, Yinghao Zou, Jiquan Long, Yudong Cai, Zhenxiang Li, Zhifeng Zhang, Yihua Mo, Jun Gu, Ruiyi Jiang, Yi Wei, and Charles Xie. Milvus:...

    2021

  12. [20]

    AnalyticDB-V: A hybrid analytical engine towards query fusion for structured and unstructured data.Proceedings of the VLDB Endowment, 13(12):3152–3165, 2020

    Chuangxian Wei, Bin Wu, Sheng Wang, Renjie Lou, Chaoqun Zhan, Feifei Li, and Yuanzhe Cai. AnalyticDB-V: A hybrid analytical engine towards query fusion for structured and unstructured data.Proceedings of the VLDB Endowment, 13(12):3152–3165, 2020

    2020

  13. [21]

    PASE: PostgreSQL ultra-high-dimensional approximate nearest neighbor search extension

    Wen Yang, Tao Li, Gai Fang, and Hong Wei. PASE: PostgreSQL ultra-high-dimensional approximate nearest neighbor search extension. InProceedings of the ACM SIGMOD In- ternational Conference on Management of Data, pages 2241–2253, 2020. 14

    2020

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.