REVIEW 3 major objections 4 minor 12 references
From Block to Byte: Transforming PCIe SSDs with CXL Memory Protocol and Instruction Annotation
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read CXL can turn PCIe block storage into cacheable working memory, with Type 3 CXL-SSDs outperforming PCIe memory expanders by 10.9x.
desk verdict A credible FPGA prototype and a genuinely useful annotation idea, but the headline speedups are simulator projections that need calibration evidence before they are quoted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the cacheable system mapping provided by CXL.mem for a Type 3 endpoint: once a device's HDM is mapped into the host's physical address space, loads and stores are coherent with CPU caches, so repeated accesses never reach the storage. The annotations, Determinism (DT/ND) and Bufferability (BF/NB), are placed in the reserved fields of CXL.mem request and response messages, letting the SSD controller decide whether to defer internal tasks and whether to cache data in its internal DRAM. The evaluation machinery is a full-system simulation built by coupling a CPU simulator with an SSD simulator and adjusting both with cycle counts observed on a custom FPGA prototype; the projected configuration is a 4 GHz, four-core CPU with a CXL 3.1 endpoint.
What would settle it
Run the same workload mix on a real CXL Type 3 SSD built from commodity flash (or a fully released simulation configuration) and measure execution time versus a PCIe memory expander using the same media; the paper's claims are settled by whether the 10.9x improvement and the additional 5.4x gain from annotations reproduce outside the calibrated model.
Extended reading notes
Core claim
The central claim is that cacheability, not raw bandwidth, is what lets block storage act as working memory, so the right CXL device type is a Type 3 endpoint. Under this design, the host maps the device's host-managed device memory (HDM) into system memory and issues ordinary load/store instructions; CPU cache hits then bypass the storage device entirely, eliminating the per-access latency of flash. The two annotations, Determinism and Bufferability, ride in reserved fields of CXL.mem request messages and tell the SSD controller whether to suppress background tasks and whether to buffer data in its internal DRAM or force it to flash. In the calibrated simulation, this combination reduces storage access frequency by 72.1% on average, improves execution time by 10.9x over a non-cacheable PCIe memory expander, and the annotations contribute an additional 5.4x, with bandwidth microbenchmarks showing up to 14.6x gains at low thread counts.
Load-bearing premise
The performance results depend entirely on the fidelity of the calibrated full-system simulation; if the cycle counts observed on the FPGA prototype do not transfer to a real 4 GHz CXL 3.1 endpoint, the 10.9x, 5.4x, and DRAM-like claims are not supported.
Editorial extensions
If this is right
- CXL-SSDs could supply NAND-class capacity as a lower-cost memory tier, with CPU caches absorbing most accesses in locality-heavy workloads.
- The DT/ND and BF/NB annotations give software a way to enforce persistence without issuing a global flush, reducing the cost of transaction commit in databases.
- Because the SSD-side changes are described as small (extending the PCIe endpoint logic and simplifying the NVMe controller), most current PCIe SSDs could be reworked into Type 3 CXL endpoints.
- With CXL switches and logical device partitioning, CXL-SSDs can be disaggregated and shared across multiple hosts, allowing fine-grained memory pooling.
- For latency-sensitive access patterns the annotations give the larger gains (up to 14.6x at one thread), while bandwidth-bound multi-threaded access still improves 4.2x, indicating the internal DRAM of the SSD is the key resource.
Reading between the lines
- If the reserved annotation fields are adopted by the CXL specification, compilers and profilers could tag loads and stores automatically, removing the need for hand-annotating library code.
- The paper does not address OS page placement: using CXL-SSD as working memory would require memory-tiering policies that decide which pages live on flash-backed CXL memory, so real-world gains depend on the OS policy as much as the hardware.
- The DRAM-like results for high-locality workloads suggest CXL-SSD is best deployed as a large, slow tier below a small DRAM tier, rather than as an outright DRAM replacement.
- A cheap way to test the sensitivity of the results is to vary the SSD's internal DRAM size and the flash program/read latencies in the calibrated simulator, since the annotation gains depend heavily on internal DRAM absorbing flash latency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes using CXL Type 3 endpoints to transform PCIe-based block storage into byte-addressable, cacheable working memory (CXL-SSDs). The authors prototype a CXL-SSD on a custom FPGA platform and propose two instruction annotations, Determinism (DT) and Bufferability (BF), meant to improve latency and persistence behavior when CXL memory requests interact with SSD internals. The evaluation combines FPGA-prototype measurements (Apex-Map latency) with a gem5+SimpleSSD full-system simulator, reporting that CXL-SSD outperforms a PCIe-based memory expander by 10.9x and that the annotation-augmented CXL-ASSD improves performance over CXL-SSD by an additional 5.4x, with DRAM-like performance claimed for high-locality workloads.
Significance. If the quantitative claims are reliable, the paper makes a strong case for a practical storage-memory convergence path that reuses standard CXL infrastructure. The FPGA prototype is a concrete hardware artifact, and the proposed DT/BF annotations are a novel idea for letting host-aided hints control SSD-internal scheduling and persistence in a memory semantics. The paper also ships a useful system-integration and disaggregation discussion. However, all headline numbers come from a simulator calibrated only against narrow Apex-Map measurements on a dual-core RISC-V prototype and projected to a 4-core 4GHz target; no code, configuration files, or workload-level hardware validation are provided. The quantitative evidence therefore does not yet meet the standard needed to support the abstract's strongest claims.
major comments (3)
- [Evaluation Setup and Preliminary Performance Model of CXL-SSD] The full-system simulation is calibrated only against Apex-Map latency measurements from the FPGA prototype (Fig. 4b), whose host is a dual-core RISC-V CPU with 128KB L1 and 4MB L2. The simulated target in Table 1a is a 4-core 4GHz CPU with 64KB L1 and 2MB L2, connected through CXL 3.1/PCIe 6.0. Because the reported speedups (10.9x vs. PCIe, 5.4x from annotations, and DRAM-like performance for high-locality workloads) are driven largely by cache hit rates, and because cache hit rates depend on core count, cache capacity, and memory-level parallelism, the transfer of calibration from the prototype to the simulated target is not self-evident. The paper provides no calibration error, no comparison of simulated versus measured execution time for any real workload, and no sensitivity analysis with respect to core count or cache capacity. The headline numbers are therefore projections of unknown accuracy.
- [Instruction Annotation and Performance Analysis] The claimed 5.4x improvement of CXL-ASSD over CXL-SSD in Fig. 6b is not tied to a stated annotation coverage. The sensitivity test in Fig. 6d shows that execution time varies substantially with the fraction of annotated functions/instructions (e.g., annotating 25% of functions yields a 50.1% average reduction, but per-workload results vary widely). The DT policy is described qualitatively as applying when the proportion of load instructions in the instruction queue and reorder buffer 'exceeds a certain threshold,' and the STREAM evaluation in Fig. 6f uses a p75 setting. Without a precise specification of how the annotation coverage and the DT threshold are chosen, the 5.4x result is not reproducible and may depend on favorable tuning.
- [Instruction Annotation, first paragraph] The proposed Determinism and Bufferability annotations are carried in the 10-bit reserved fields of CXL.mem M2S Req and S2M NDR messages. The paper asserts that these fields 'can support' the annotations without additional transmission costs or data-payload modification, but it does not cite the relevant CXL 3.1 specification clauses or demonstrate compliance. If the CXL specification requires reserved bits to be transmitted as zero and to be ignored by the receiver, or if existing hosts/endpoints treat non-zero reserved bits as errors, the entire annotation mechanism cannot be deployed on standard CXL hardware. This assumption is load-bearing for the main contribution.
minor comments (4)
- [Table 1b and Evaluation Setup] The workload description does not specify the benchmark versions (e.g., SPEC CPU 2006 vs. SPEC CPU 2017) or the exact set of RV8 benchmarks used; providing this information would improve reproducibility.
- [Performance Analysis] The abstract's claim that CXL-SSD 'approaches DRAM-like performance' is not quantified; the STREAM results in Fig. 6f show CXL-ASSD is still 4.7x slower than CXL-DRAM, so the abstract would be more precise if it qualified the claim as applying only to latency-sensitive, high-locality workloads.
- [Why CXL Memory for PCIe Storage?] The statement that Intel and AMD ISAs enforce non-cacheability of PCIe-related memory requests is presented without a citation; a reference to an architectural manual would be helpful.
- [Disaggregation Discussion] The sentence 'it does need careful network and storage designs' is grammatically awkward and should be rewritten for clarity.
Circularity Check
No significant circularity: the 10.9x/5.4x speedups are simulator projections with explicit parameters and external workloads, not fitted or definitionally forced results.
full rationale
The paper's derivation chain is: FPGA prototype measurements (Apex-Map) -> calibrated gem5+SimpleSSD full-system model (Table 1a, workloads in Table 1b) -> simulated execution-time comparisons. The headline '10.9x better than PCIe' and '5.4x with annotations' are outputs of that simulation, not quantities recovered from a fitted equation or from the definitions of the configurations. Cacheability is an architectural property implemented in the model, and the simulated speedup is a consequence of cache hit rates and memory-latency parameters; it is not equivalent by construction to 'CXL is cacheable.' Similarly, the 'actual cycles observed' in the Preliminary Performance Model are input parameters for calibration, and no reported target number is used to fit those parameters. The annotation results (CXL-ASSD) are obtained by enabling different controller behaviors in the same simulator; the measured improvement is a model prediction, not a renamed fit. The main limitation is simulator fidelity and generalization from a 2-core FPGA RISC-V prototype to a 4-core, 4 GHz target; this is a correctness/validation risk, not circularity. The SimpleSSD citation (ref [8]) is a self-citation by the authors, but it is a tool citation for the simulator platform, and the paper's claims are not justified by an unverified theorem from that prior work. No circular step is exhibited.
Assumptions & free parameters
free parameters (2)
- DT annotation coverage threshold =
p75 (approximately 75% of instructions annotated)
- BF annotation fraction =
p75 (75% of commonly used functions annotated as bufferable)
assumptions (4)
- domain assumption CXL root port maps HDM into host's cacheable system memory, making CXL.mem requests cacheable and coherent.
- domain assumption PCIe BAR memory regions are non-cacheable by x86 hosts, so PCIe-based memory expanders cannot benefit from CPU caches.
- ad hoc to paper The 10-bit reserved fields in CXL.mem M2S Req and S2M NDR messages can carry Determinism and Bufferability hints without protocol violation or performance cost.
- ad hoc to paper The gem5 and SimpleSSD simulation, calibrated with FPGA prototype cycles, accurately projects real 4 GHz multi-core CXL-SSD performance.
Cite this review
Pith. "Pith review of From Block to Byte: Transforming PCIe SSDs with CXL Memory Protocol and Instruction Annotation." pith.science (2026). https://pith.science/paper/IY7PBU3K
@misc{pith2026250615613,
author = {Pith},
title = {Pith review of: From Block to Byte: Transforming PCIe SSDs with CXL Memory Protocol and Instruction Annotation},
year = {2026},
howpublished = {\url{https://pith.science/paper/IY7PBU3K}},
note = {Machine review of arXiv:2506.15613}
}
read the original abstract
This paper explores how Compute Express Link (CXL) can transform PCIe-based block storage into a scalable, byte-addressable working memory. We address the challenges of adapting block storage to CXL's memory-centric model by emphasizing cacheability as a key enabler and advocating for Type 3 endpoint devices, referred to as CXL-SSDs. To validate our approach, we prototype a CXL-SSD on a custom FPGA platform and propose annotation mechanisms, Determinism and Bufferability, to enhance performance while preserving data persistency. Our simulation-based evaluation demonstrates that CXL-SSD achieves 10.9x better performance than PCIe-based memory expanders and further reduces latency by 5.4x with annotation enhancements. In workloads with high locality, CXL-SSD approaches DRAM-like performance due to efficient on-chip caching. This work highlights the feasibility of integrating block storage into CXL's ecosystem and provides a foundation for future memory-storage convergence.
Reference graph
Works this paper leans on
-
[1]
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 9 12 #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockconfadjsp...
-
[2]
`` Gen-Z .'' https://genzconsortium.org
-
[3]
`` CCIX .'' https://www.ccixconsortium.com/
- [4]
-
[5]
``Compute express link specification revision 3.1.'' https://computeexpresslink.org/cxl-specification/
-
[6]
UDON: A case for offloading to general purpose compute on CXL memory
J. Hermes, J. Minor, M. Wu, A. Patil, and E. Van Hensbergen, ``Udon: A case for offloading to general purpose compute on cxl memory,'' arXiv preprint arXiv:2404.02868 , 2024
work page Pith review arXiv 2024
-
[7]
E. Strohmaier and H. Shan, ``Apex-map: A global data access benchmark to analyze hpc systems and parallel programming paradigms,'' in SC'05: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing , pp. 49--49, IEEE, 2005
work page 2005
-
[8]
N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, et al. , ``The gem5 simulator,'' ACM SIGARCH computer architecture news , vol. 39, no. 2, pp. 1--7, 2011
work page 2011
Show all 12 references
-
[9]
D. Gouk, M. Kwon, J. Zhang, S. Koh, W. Choi, N. S. Kim, M. Kandemir, and M. Jung, ``Amber: Enabling precise full-system simulation with detailed modeling of all ssd resources,'' in 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , pp. 469--481, I...
2018
-
[10]
J. L. Henning, ``Spec cpu2006 benchmark descriptions,'' ACM SIGARCH Computer Architecture News , vol. 34, no. 4, pp. 1--17, 2006
2006
-
[11]
``rv8-bench.'' https://github.com/michaeljclark/rv8-bench
-
[12]
J. D. McCalpin, ``Memory bandwidth and machine balance in current high performance computers,'' IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter , pp. 19--25, Dec. 1995
1995
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.