pith. sign in

arxiv: 2606.27979 · v1 · pith:UFEP2BWXnew · submitted 2026-06-26 · 💻 cs.DB · cs.DC· cs.PF

DiStash: A Disaggregated Multi-Stash Transactional Key-Value Store

Pith reviewed 2026-06-29 02:10 UTC · model grok-4.3

classification 💻 cs.DB cs.DCcs.PF
keywords disaggregated storagetransactional key-value storemulti-stashFoundationDBdata consistencystorage tiersephemeral storagedurable storage
0
0 comments X

The pith

DiStash enables a single transaction to read and write copies of key-value pairs across multiple disaggregated stash pools.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DiStash is a transactional key-value store that governs data across pools of different storage media, referred to as stashes, such as DRAM, SSD, HDD, or NVM. It allows an application to issue one transaction that operates on different copies of the same key-value pairs whether those stashes are set up as inclusive replicas or exclusive tiers. The design removes the need for applications to manage race conditions or handle partial failures that could leave inconsistent values or lost data. Stash pools can be designated ephemeral or durable at the application's direction. The implementation extends FoundationDB and measures the resulting tradeoffs on microbenchmarks plus eBay's production workload.

Core claim

DiStash governs KVs across pools of stash types and enables an application to use a single transaction to read and write different copies of one or more key-value pairs across the different pools of stashes, simplifying application logic by preventing undesirable race conditions that may cause copies of data to reflect different values and failures that may result in loss of key-value pairs.

What carries the argument

The multi-stash transactional coordination layer, built by extending FoundationDB, that enforces atomicity across disaggregated inclusive or exclusive stash pools.

If this is right

  • Applications can issue one atomic operation instead of separate reads and writes that risk inconsistency across storage types.
  • Stash pools can be configured as either replicated copies or tiered exclusive storage without changing the transaction interface.
  • Stashes can be treated as ephemeral or durable storage depending on the durability requirements set by the application.
  • Performance and consistency tradeoffs become measurable through standard microbenchmarks and production traces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same multi-stash pattern could be added to other transactional key-value systems to support flexible storage hierarchies.
  • Independent scaling of individual stash pools becomes possible while still presenting a unified transactional view.
  • Applications that previously required custom synchronization code for multi-tier storage may be able to simplify their logic further.

Load-bearing premise

Extending FoundationDB with multi-stash logic preserves its original transactional guarantees and failure semantics when stashes are disaggregated and configured as inclusive or exclusive.

What would settle it

A transaction that commits yet leaves different stash pools holding inconsistent values for the same keys, or that loses data under a failure scenario that FoundationDB would have survived, would show the extension does not preserve the original guarantees.

Figures

Figures reproduced from arXiv: 2606.27979 by Hieu Nguyen, Jun Li, Shahram Ghandeharizadeh, Yiming Gao.

Figure 1
Figure 1. Figure 1: DiStash with three pools of stashes: DRAM, SSD, and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A taxonomy of stash types and their usage. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Write-Around. In the presence of writes to the data in the durable storage, the application must maintain the cache entries in the ephemeral stor￾age consistent. It may use different write policies [15, 21] [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the write-through policy. It requires the applica￾tion to issue a transaction to update all copies of a cache entry and the original data across all ℎ stash pools. The update to a stash pool might be in the form of an SQL update command, a memcached append command, or Read-Modify-Write (RMW). With the latter, the transaction may read the impacted data from a stash, modify its value, and write it back… view at source ↗
Figure 5
Figure 5. Figure 5: Read and write transactions with DiStash. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Read and write transactions with different stashes being managed by different storage managers. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An undesirable race condition results in inconsistent data across DRAM and SSD. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An implementation of DiStash. See [38] for definition of arrows. • A new Storage Server (fdbserver/KeyValueStoreCache.actor.cpp) that extends FDB’s existing in-memory implementation with the LRU replacement technique in support of DRAM ephemeral storage. • Extensions of the configuration file to specify the ℎ stashe pools in the system and whether each is ephemeral or durable. • Initialization of the FDB t… view at source ↗
Figure 9
Figure 9. Figure 9: Throughput of an ephemeral DRAM stash when [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Latency of an ephemeral DRAM stash when com [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: SSD Utilization, moderate workload. utilization with 2Stash, see [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Failure and recovery of an ephemeral volatile [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Query result lookup (cache hit) latency of 1Stash [PITH_FULL_IMAGE:figures/full_fig_p009_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Utilization of 40 SSDs with epochs of Figure [PITH_FULL_IMAGE:figures/full_fig_p009_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Query result lookup (cache hit) latency of 2Stash, [PITH_FULL_IMAGE:figures/full_fig_p010_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: CPU utilization of 5 DRAMs with epochs of Fig [PITH_FULL_IMAGE:figures/full_fig_p010_16.png] view at source ↗
read the original abstract

A stash is a storage medium such as Dynamic Random Access Memory (DRAM), Solid State Disk (SSD), Hard Disk Drive (HDD), or Non-Volatile Memory (NVM). This paper presents a disaggregated transactional key-value (KV) store, DiStash, that governs KVs cross pools of stash types. It enables an application to use a single transaction to read and write different copies of one or more key-value pair across the different pools of stashes. It simplifies the application logic by (a) preventing undesirable race conditions that may cause copies of data across different stash pools to reflect different values and/or (b) failures that may result in loss of key-value pairs. A configuration of DiStash may use a pool of stashes as either ephemeral or durable storage. The application dictates whether the content of its participating stashes are inclusive (replicated) or exclusive (tiered). We implement a DiStash by extending FoundationDB. We quantify the tradeoffs with its design decisions using microbenchmarks and eBay's production workload. We open source our implementation at https://github.com/ebay-USC/DiStash.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper presents DiStash, a disaggregated transactional key-value store built by extending FoundationDB. It allows a single transaction to read and write different copies of one or more key-value pairs across pools of heterogeneous stashes (DRAM, SSD, HDD, NVM), with pools configurable as inclusive (replicated) or exclusive (tiered) and as ephemeral or durable storage. The design aims to simplify applications by avoiding cross-stash races and data loss. The authors report an implementation, microbenchmarks, and evaluation on eBay's production workload, and release the code at https://github.com/ebay-USC/DiStash.

Significance. If the extension to FoundationDB correctly preserves ACID properties and failure semantics across disaggregated stash pools, the work would offer a practical mechanism for transactional access to tiered heterogeneous storage, reducing the need for application-level coordination. The open-source release supports reproducibility and further experimentation.

major comments (2)
  1. [Abstract] Abstract: the central claim requires that the multi-stash extension preserves FoundationDB's original transactional guarantees and failure semantics (including cross-stash conflict detection, commit coordination, and recovery) when stashes are disaggregated and configured as inclusive or exclusive. No description of the modified transaction protocol, logging, or recovery paths is supplied, leaving the preservation assumption unverified.
  2. [Abstract and Evaluation section] Abstract and Evaluation section: the abstract states that microbenchmarks and an eBay workload were used to quantify design tradeoffs, yet supplies no performance data, error bars, throughput/latency numbers, or comparison baselines, preventing assessment of whether the claimed simplifications are achieved in practice.
minor comments (1)
  1. [Abstract] Abstract: 'one or more key-value pair' should read 'pairs'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on DiStash. The comments highlight areas where additional detail would strengthen the presentation of transactional guarantees and empirical results. We address each point below and commit to revisions that directly incorporate the requested information without altering the core claims.

read point-by-point responses
  1. Referee: [Abstract] Abstract: the central claim requires that the multi-stash extension preserves FoundationDB's original transactional guarantees and failure semantics (including cross-stash conflict detection, commit coordination, and recovery) when stashes are disaggregated and configured as inclusive or exclusive. No description of the modified transaction protocol, logging, or recovery paths is supplied, leaving the preservation assumption unverified.

    Authors: We agree that the manuscript does not supply a description of the modified transaction protocol, logging, or recovery paths. The current text focuses on the high-level design and implementation overview. In the revised version we will add a dedicated section that explains the extensions to FoundationDB's transaction protocol, including how cross-stash conflict detection, commit coordination, and recovery are handled for both inclusive and exclusive configurations while preserving the original ACID and failure semantics. revision: yes

  2. Referee: [Abstract and Evaluation section] Abstract and Evaluation section: the abstract states that microbenchmarks and an eBay workload were used to quantify design tradeoffs, yet supplies no performance data, error bars, throughput/latency numbers, or comparison baselines, preventing assessment of whether the claimed simplifications are achieved in practice.

    Authors: We agree that the evaluation section currently lacks the quantitative results, error bars, throughput/latency numbers, and baseline comparisons referenced in the abstract. The revised manuscript will expand the evaluation section to include these specific measurements from the microbenchmarks and eBay production workload, enabling direct assessment of the design tradeoffs. revision: yes

Circularity Check

0 steps flagged

No circularity: systems implementation paper with no derivations or equations

full rationale

The paper is a description of a disaggregated KV store implementation extending FoundationDB. It contains no equations, fitted parameters, predictions, or derivation chains. The central contribution is an engineering design evaluated via microbenchmarks and workload traces, which is self-contained and externally verifiable through the open-sourced code and described configuration options. No load-bearing steps reduce to self-definition or self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a systems implementation paper. No free parameters, mathematical axioms, or invented theoretical entities are introduced or required by the abstract description.

pith-pipeline@v0.9.1-grok · 5750 in / 1001 out tokens · 25567 ms · 2026-06-29T02:10:59.937424+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 13 canonical work pages

  1. [1]

    Steve Byan, James Lentini, Anshul Madan, Luis Pabon, Michael Condict, Jeff Kimmel, Steve Kleiman, Christopher Small, and Mark Storer. 2012. Mercury: Host-Side Flash Caching for the Data Center. In2012 IEEE 28th symposium on mass storage systems and technologies (MSST). IEEE, 1–12

  2. [2]

    2013.Redis in Action

    Josiah Carlson. 2013.Redis in Action. Simon and Schuster

  3. [3]

    Ronald P Cenker, Donald G Clemons, William R Huber, Joseph B Petrizzi, Frank J Procyk, and George M Trout. 1979. A Fault-Tolerant 64K Dynamic Random- Access Memory.IEEE Transactions on Electron Devices26, 6 (1979), 853–860

  4. [4]

    Feng Chen, Binbing Hou, and Rubao Lee. 2016. Internal Parallelism of Flash Memory-Based Solid-State Drives.ACM Transactions on Storage (TOS)12, 3 (2016), 1–39

  5. [5]

    Pierre Clouzet, Clément Foyer, Brice Goglin, Emmanuel Jeannot, Jannis Klinken- berg, Christian Terboven, and Anara Kozhokanova. 2024. H2M: Heuristics for DiStash: A Disaggregated Multi-Stash Transactional Key-Value Store Heterogeneous Memory. InHMEM 2024-5th Workshop on Heterogeneity and Memory Systems

  6. [6]

    Biplob Debnath, Sudipta Sengupta, and Jin Li. 2010. FlashStore: High Throughput Persistent Key-Value Store.Proc. VLDB Endow.3, 1–2 (Sept. 2010), 1414–1425. doi:10.14778/1920841.1921015

  7. [7]

    Cagdas Dirik and Bruce Jacob. 2009. The Performance of PC Solid-State Disks (SSDs) as a Function of Bandwidth, Concurrency, Device Architecture, and System Organization.ACM SIGARCH Computer Architecture News37, 3 (2009), 279–289

  8. [8]

    Siying Dong, Mark Callaghan, Leonidas Galanis, Dhruba Borthakur, Tony Savor, and Michael Strum. 2017. Optimizing Space Amplification in RocksDB.. InCIDR, Vol. 3. 3

  9. [9]

    Andersen, and Michael Kaminsky

    Bin Fan, Hyeontaek Lim, David G. Andersen, and Michael Kaminsky. 2011. Small Cache, Big Effect: Provable Load Balancing for Randomly Partitioned Cluster Services. InProceedings of the 2nd ACM Symposium on Cloud Computing(Cascais, Portugal)(SOCC ’11). Association for Computing Machinery, New York, NY, USA, Article 23, 12 pages. doi:10.1145/2038916.2038939

  10. [10]

    Dashti, and Cyrus Shahabi

    Shahram Ghandeharizadeh, Ali E. Dashti, and Cyrus Shahabi. 1995. Pipelining Mechanism to Minimize the Latency Time in Hierarchical Multimedia Storage Managers.Comput. Commun.18, 3 (1995), 170–184

  11. [11]

    Shahram Ghandeharizadeh, Connor Gorman, Sandy Irani, Shiva Jahangiri, Jenny Lam, Hieu Nguyen, Ryan Tani, and Jason Yap. 2014. A Demonstration of KOSAR: An Elastic, Scalable, Highly Available SQL Middleware. InProceedings of the Middleware ’14 Posters & Demos Session, Bordeaux, France, December 8-12, 2014, Romain Rouvoy (Ed.). ACM, 23–24. doi:10.1145/26785...

  12. [12]

    Shahram Ghandeharizadeh, Douglas Ierardi, Dongho Kim, and Roger Zimmer- mann. 1996. Placement of Data in Multi-zone Disk Drives. InSecond International Baltic Workshop on DB and IS. Citeseer

  13. [13]

    Shahram Ghandeharizadeh, Sandy Irani, and Jenny Lam. 2018. On Configuring a Hierarchy of Storage Media in the Age of NVM. In34th IEEE International Conference on Data Engineering, ICDE 2018, Paris, France, April 16-19, 2018. IEEE Computer Society, 1380–1383. doi:10.1109/ICDE.2018.00155

  14. [14]

    Shahram Ghandeharizadeh, Sandy Irani, and Jenny Lam. 2025. Memory Hierar- chy Design for Caching Middleware in the Age of NVM. arXiv:2506.05071 [cs.DB] https://arxiv.org/abs/2506.05071

  15. [15]

    Shahram Ghandeharizadeh and Hieu Nguyen. 2019. Design, Implementation, and Evaluation of Write-Back Policy with Cache Augmented Data Stores.Proceedings of the VLDB Endowment12, 8 (2019), 836–849

  16. [16]

    Shahram Ghandeharizadeh and Jason Yap. 2017. SQL Query to Trigger Transla- tion: A Novel Transparent Consistency Technique for Cache Augmented SQL Systems. In28th International Workshop on Database and Expert Systems Applica- tions, DEXA 2017 Workshops, Lyon, France, August 28-31, 2017. IEEE Computer Society, 37–41. doi:10.1109/DEXA.2017.24

  17. [17]

    Shahram Ghandeharizadeh, Jason Yap, and Hieu Nguyen. 2014. Strong Con- sistency in Cache Augmented SQL Systems. InProceedings of the 15th Inter- national Middleware Conference, Bordeaux, France, December 8-12, 2014, Lau- rent Réveillère, Lucy Cherkasova, and François Taïani (Eds.). ACM, 181–192. doi:10.1145/2663165.2663318

  18. [18]

    Jim Gray and Franco Putzolu. 1987. The 5 Minute Rule for Trading Memory for Disc Accesses and the 10 Byte Rule for Trading Memory for CPU Time. In Proceedings of the 1987 ACM SIGMOD international conference on Management of data. 395–398

  19. [19]

    Haoyu Huang and Shahram Ghandeharizadeh. 2021. Nova-LSM: A Distributed, Component-based LSM-tree Key-value Store. InSIGMOD ’21: International Con- ference on Management of Data, Virtual Event, China, June 20-25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 749–763. doi:10.1145/3448016.3457297

  20. [20]

    Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate Foster, Changhoon Kim, and Ion Stoica. 2017. NetCache: Balancing Key-Value Stores with Fast In-Network Caching. InProceedings of the 26th Symposium on Operating Systems Principles(Shanghai, China)(SOSP ’17). Association for Computing Machinery, New York, NY, USA, 121–136. doi:10.1145/313...

  21. [21]

    Norman P Jouppi. 1993. Cache Write Policies and Performance.ACM SIGARCH Computer Architecture News21, 2 (1993), 191–201

  22. [22]

    Bohyun Lee, Seongjae Moon, Jonghyeok Park, and Sang-Won Lee. 2025. Boosting OLTP Performance with Per-Page Logging on NVDIMM.Proceedings of the ACM on Management of Data3, 1 (2025), 1–28

  23. [23]

    Changmin Lee, Wonjae Shin, Dae Jeong Kim, Yongjun Yu, Sung-Joon Kim, Taekyeong Ko, Deokho Seo, Jongmin Park, Kwanghee Lee, Seongho Choi, et al

  24. [24]

    In2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)

    NVDIMM-C: A Byte-Addressable Non-Volatile Memory Module for Com- patibility with Standard DDR Memory Interfaces. In2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 502–514

  25. [25]

    Jacques Lenfant. 1977. Fast Random and Sequential Access to Dynamic Memories of any Size.IEEE Trans. Comput.100, 9 (1977), 847–855

  26. [26]

    memcached. [n. d.]. Memcached, http://www.memcached.org/. http://www. memcached.org/

  27. [27]

    Hieu Nguyen, Jun Li, and Shahram Ghandeharizadeh. 2023. Graph Stores with Application-Level Query Result Caches. InPerformance Evaluation and Benchmarking - 15th TPC Technology Conference, TPCTC 2023, Vancouver, British Columbia, Canada, August 28, 2023 (Lecture Notes in Computer Science, Vol. 14247), Raghunath Nambiar and Meikel Poess (Eds.). Springer

  28. [28]

    Hieu Nguyen, Jun Li, and Shahram Ghandeharizadeh. 2024. One-Hop Sub- Query Result Caches for Graph Database Systems. arXiv:2412.04698 [cs.DB] https://arxiv.org/abs/2412.04698

  29. [29]

    Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, and Venkateshwaran Venkataramani

    Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, and Venkateshwaran Venkataramani. 2013. Scaling Memcache at Facebook. InProceedings of the 10th USENIX Conference on Networked Systems Design and Implementation(Lombard, IL)(nsdi’13). USENIX ...

  30. [30]

    Michael Norman, Vince Kellen, Shava Smallen, Brian DeMeulle, Shawn Strande, Ed Lazowska, Naomi Alterman, Rob Fatland, Sarah Stone, Amanda Tan, Katherine Yelick, Eric Van Dusen, and James Mitchell. 2021. CloudBank: Managed Services to Simplify Cloud Access for Computer Science Research and Education. In Practice and Experience in Advanced Research Computin...

  31. [31]

    John Ousterhout, Arjun Gopalan, Ashish Gupta, Ankita Kejriwal, Collin Lee, Behnam Montazeri, Diego Ongaro, Seo Jin Park, Henry Qin, Mendel Rosenblum, Stephen Rumble, Ryan Stutsman, and Stephen Yang. 2015. The RAMCloud Storage System.ACM Trans. Comput. Syst.33, 3, Article 7 (Aug. 2015), 55 pages. doi:10.1145/2806887

  32. [32]

    Onkar Patil, Latchesar Ionkov, Jason Lee, Frank Mueller, and Michael Lang

  33. [33]

    InProceedings of the International Symposium on Memory Systems

    NVM-based Energy and Cost Efficient HPC Clusters. InProceedings of the International Symposium on Memory Systems. 1–14

  34. [34]

    David A Patterson, Garth Gibson, and Randy H Katz. 1988. A Case for Redundant Arrays of Inexpensive Disks (RAID). InProceedings of the 1988 ACM SIGMOD international conference on Management of data. 109–116

  35. [35]

    Dan R. K. Ports, Austin T. Clements, Irene Zhang, Samuel Madden, and Barbara Liskov. 2010. Transactional Consistency and Automatic Management in an Ap- plication Data Cache. In9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010, October 4-6, 2010, Vancouver, BC, Canada, Proceedings, Remzi H. Arpaci-Dusseau and Brad Chen (Eds.). ...

  36. [36]

    Guoyu Wang, Xilong Che, Haoyang Wei, Shuo Chen, Puyi He, and Juncheng Hu

  37. [37]

    In23rd USENIX Conference on File and Storage Technologies (FAST 25)

    Boosting File Systems Elegantly: A Transparent {NVM} Write-ahead Log for Disk File Systems. In23rd USENIX Conference on File and Storage Technologies (FAST 25). 19–34

  38. [38]

    Brian White, Jay Lepreau, Leigh Stoller, Robert Ricci, Shashi Guruprasad, Mac Newbold, Mike Hibler, Chad Barb, and Abhijeet Joglekar. 2002. An Integrated Experimental Environment for Distributed Systems and Networks.SIGOPS Oper. Syst. Rev.36, SI, 255–270. doi:10.1145/844128.844152

  39. [39]

    Arpaci-Dusseau, and Remzi H

    Kan Wu, Zhihan Guo, Guanzhou Hu, Kaiwei Tu, Ramnatthan Alagappan, Rathijit Sen, Kwanghyun Park, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau

  40. [40]

    In19th USENIX Conference on File and Storage Technologies (FAST 21)

    The Storage Hierarchy is Not a Hierarchy: Optimizing Caching on Modern Storage Devices with Orthus. In19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, 307–323. https://www.usenix.org/ conference/fast21/presentation/wu-kan

  41. [42]

    InProceedings of the 2021 International Conference on Management of Data

    FoundationDB: A Distributed Unbundled Transactional Key Value Store. InProceedings of the 2021 International Conference on Management of Data. 2653– 2666

  42. [43]

    Jingyu Zhou, Meng Xu, Alexander Shraer, Bala Namasivayam, Alex Miller, Evan Tschannen, Steve Atherton, Andrew J Beamon, Rusty Sears, John Leach, et al

  43. [44]

    ACM66, 6 (2023), 97–105

    FoundationDB: A Distributed Key-Value Store.Commun. ACM66, 6 (2023), 97–105