The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

Daming Li; Jialu Zhang; Yuan Si; Yufeng Lin

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

T0 review · 0 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cached checkpoint pages can feed a GPU at resident speed if frameworks stop copying them first.

desk verdict Careful empirical systems work showing mapped file-backed weights can match resident bandwidth on unified/coherent memory at zero rate cost; the missing artifact is the main obstacle to unqualified acceptance. read the letter →

arxiv 2608.12114 v1 pith:56TCRUPW submitted 2026-08-12 cs.OS

Yuan Si , Yufeng Lin , Daming Li , Jialu Zhang This is my paper · ORCID

classification cs.OS

keywords file-backedweightsingestiontaxmemorymappingpagecacheGPUinferenceunifiedtopologytensorframeworks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-weight models often sit in a middle regime: the active weights fit in DRAM as cached file pages, but a second framework-owned copy does not fit or must be refilled every token. The paper isolates a cost it calls the ingestion tax—file-backed bytes the OS already holds in GPU-readable memory being copied into framework-owned allocations before each use—and argues that the tax is entirely avoidable on unified and coherent-memory systems. It presents file-backed weight adoption, a producer that maps each tensor with MAP_SHARED, wraps the pages as a no-copy GPU buffer, and exports an interchange capsule that tensor frameworks import as ordinary storage. Under a three-condition execution contract, the mapped path reaches 516 GB/s where default constructors reach 53–82 GB/s, and a matched 72B model decodes at essentially the same rate as a load-once resident control. The reason to care is that parity buys ownership: the weights stay one shared, evictable, reclaimable file-backed copy, which changes multi-process serving, startup latency, and the role of the page cache.

What carries the argument

The mechanism is a framework-independent producer: it opens each page-aligned tensor file, maps it read-only with MAP_SHARED, wraps the range as a no-copy GPU buffer, and returns a versioned DLPack capsule—an open in-memory tensor interchange format—that a tensor framework imports as ordinary storage. The argument's second pillar is the three-condition execution contract: C1 adopt the mapped pages with no per-use copy, C2 keep activations in the accelerator-visible domain, and C3 establish dependencies on the GPU so the host never drains in-flight work. The paper prices each condition with a randomized same-harness factorial—removing C1, C2, or C3 costs 12.3, 5.6, or 3.4 ms per pass—which is what converts zero-copy import from a mechanism into a correct fast path; a deployment-time endpoint probe then chooses among map, resident-copy, and per-use stream-copy based on full-working-set bandwidth.

What would settle it

On a unified-memory machine, run the paper's matched protocol with a fully cached checkpoint and a single kernel over the same bytes in two forms: imported mapped pages and one-time resident copies; the paper's central claim predicts the mapped arm stays within roughly one percent of the resident arm (the paper's interval is [-0.66%, +0.48%]), so an observed gap beyond that interval under the stated protocol would refute the no-rate-cost parity claim. A complementary control—forcing page-cache eviction mid-decode and watching the spine stage return to storage-bound time—would confirm the residency boundary rather than refute the mechanism.

Watch

Extended reading notes

Core claim

The central claim is that on systems where file pages already occupy a GPU-readable memory domain, the only thing standing between a kernel and resident-class bandwidth is the framework's ownership model: constructor and transfer paths allocate a second representation and copy file-backed bytes into it, a pure software cost with no faster placement behind it. File-backed weight adoption removes that copy, and the paper shows the removal costs nothing in steady-state decode rate when three conditions hold: the kernel reads the mapped pages, activations stay accelerator-resident, and ordering happens on the GPU. The measured endpoint is 516 GB/s through the public import route against 53–82 GB/s for default constructors, with mapped-minus-resident bandwidth bounded by [-0.66%, +0.48%] and a matched 72B model at 7.14 versus 7.23 tokens/s relative to a load-once resident control. The same parity argument carries a deployment boundary: on a discrete PCIe GPU, mapped reads are interconnect-bound and adoption loses by 39x and should be refused, so memory topology, not the API, decides the byte path.

Load-bearing premise

The active mapped weight set must remain resident in the page cache for the whole decode; once the OS evicts those pages, storage traffic dominates and the in-memory parity result no longer applies.

Editorial extensions

If this is right

On unified and coherent-memory systems, a checkpoint can be served as one clean, shared, evictable file-backed copy, so N concurrent decoders read the same pages instead of materialising N private copies.
Time-to-first-token for capacity-scale checkpoints improves sharply because module construction and shard parsing are replaced by mapping and first-touch wiring; the paper measures a 6.4x faster first token for a 65 GB checkpoint.
The same adoption mechanism improves an integrated-GPU inference runtime by 1.21x at about half the footprint, ties overlapped streaming on a coherent-link system whose model exceeds fast memory, and is rejected on a discrete PCIe GPU, so the deployment rule is topology- and working-set-aware.
Tensor frameworks need a first-class read-only mapped-storage constructor that handles alignment, lifetime, immutability, and placement; no new scheduler or intermediate representation is required.
The page cache becomes a reclaimable, shared accelerator-storage tier, so capacity checks should measure hot-set residency and representation count rather than model size against installed memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: a natural extension is a residency-aware loading policy that combines adoption with cache-pinning or prefault hints, so the page-cache precondition is actively maintained instead of assumed.
Beyond the paper: the measured scaling with activation columns suggests the advantage shrinks as weight reuse grows, so fusing multiple tokens or heads into one kernel could preserve more of the benefit for batched or prefill regimes than the paper's low-batch setting demonstrates.
Beyond the paper: the read-only enforcement gap—a GPU store to the mapping is silently discarded on one substrate, faults on another, and rejected on a third—points toward a framework-level immutable external storage type; testing whether consumers reject in-place stores once immutability is propagated is a direct next experiment.
Beyond the paper: with the byte path decided by memory topology, the same endpoint probe could apply to emerging coherent-memory and CXL-style tiers, where mapped file pages might become the default accelerator placement the way they are on integrated parts.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a careful empirical systems study: the central claim—mapped checkpoint pages can be adopted as ordinary framework storage and read at resident-class bandwidth under a three-condition execution contract—is measured with unusual discipline and scoped honestly. The components are prior art, but the producer-plus-contract-plus-deployment-rule package is new, and the evidence is strong.

The causal decomposition is the real contribution. They separate adoption (C1), activation residency (C2), and GPU-side ordering (C3), then price each with a randomized same-harness factorial. The finding that a host-visible drain costs an order of magnitude more than an on-GPU event, and that ordering only matters when activations are resident, is the kind of result that changes how loaders are built. The cross-platform work is also credible: the same mechanism speeds up llama.cpp on an APU, ties overlapped streaming on GH200, and loses 39x on PCIe, so they don't overclaim. The held-out M3 sessions are genuine predictions, and the appendices are full of disclosed measurement traps that most systems papers skip. The citation pattern is fair; they explicitly disclaim novelty for the components and engage with the closest prior MLX prototype.

Soft spots, in proportion. First, no code ships: the producer is called 'public' but there's no artifact, commit hash, or reproduction script. That alone prevents a straight accept. Second, the page-cache residency precondition is load-bearing—if the OS evicts the mapped pages mid-decode, the path falls to storage speed and the parity result is void. They gate on it at deployment time but don't pin or continuously monitor, so memory pressure in multi-tenant settings could silently nullify the benefit. It's disclosed, but it limits the deployment claim. Third, the GH200 sample is small and the Kimi K3 headline uses proprietary weights, so the most impressive number isn't independently reproducible. These are limitations, not errors.

This paper is for systems researchers who care about where bytes live in LLM inference on integrated or coherent-memory hardware. It deserves a serious referee. Recommendation: send to peer review, and make the artifact a condition of acceptance—the methodology is strong enough that the claims deserve independent reproduction.

Referee Report

0 major / 6 minor

Summary. The paper identifies an 'ingestion tax' on unified and coherent-memory systems: tensor frameworks copy page-cache-resident checkpoint bytes into framework-owned allocations even though the original file pages are already GPU-readable. The authors propose file-backed weight adoption, in which a framework-independent producer maps each tensor with MAP_SHARED, wraps the pages as a no-copy Metal buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. They establish a three-condition execution contract (adopt the mapped pages, keep activations accelerator-resident, order on the GPU) via a randomized same-harness factorial, and show that under the contract adoption reaches resident-class bandwidth: 516 GB/s where default constructors reach 53–82 GB/s, with a mapped-minus-resident difference bounded by [-0.66%, +0.48%]. End-to-end results include Qwen2.5-72B at 7.14 tok/s versus 0.94 tok/s for per-use ingestion and 7.23 tok/s for a load-once resident control, a 6.4x first-token speedup for a 65 GB checkpoint, a 1.21x improvement in llama.cpp on an AMD APU, a tie with overlapped streaming on a capacity-exceeding GH200, and a 39x loss on a discrete PCIe GPU. The deployment rule (Algorithm 2) chooses among map, resident, and stream-copy endpoints based on full-working-set probes. The paper explicitly states that the central claim is conditional on the active mapped set remaining in the page cache, and it measures and gates this precondition.

Significance. If the results hold, this is a practically important and unusually well-measured mechanism for eliminating a real inefficiency in local LLM inference on integrated and coherent-memory systems. The empirical methodology is disciplined: a randomized factorial with per-process arm rotation and twelve-pass warm-up discard, an instrument chain covering residency probes, mutation visibility, checksums, and output oracles, paired bootstrap intervals, disclosed measurement traps, and a prospective held-out machine that tests the deployment rule before the end-to-end runs. The central claim is conditional on page-cache residency, but that condition is explicit, measured, and gated, so the paper does not overclaim. The cross-architecture results (APU, GH200, PCIe) ground the topology-dependent placement rule, and the deployment heuristic yields falsifiable predictions. The paper also contributes a clear decomposition of the three execution requirements and shows that ordering costs are tied to host visibility rather than queue ownership, which is a useful systems insight.

minor comments (6)

[§6.4] The text states that 'the storage change contributes 1.6× on identical files', but the reported TTFT values (rebind 6.97 s vs. copy control 8.42 s) give a ratio of about 1.21×, not 1.6×. Please correct the number or clarify what comparison yields 1.6×.
[§8.1 (Algorithm 2)] The residency check in Algorithm 2 is performed only at deployment time. The paper does not specify a runtime mechanism to detect later eviction of the mapped set; given the RQ3 results showing that hot competitors can evict mapped pages, a deployment on a shared or multi-tenant machine could silently fall back to storage-bound throughput. Consider adding a runtime guard or explicitly stating that the rule assumes no subsequent memory pressure.
[Table 13] The row 'production spine, live 508–524' is labeled 'live', but Section 6.1 and Table 4 report a live spine stage of 0.35 s, which is about 162 GB/s for the 52.8 GiB spine. Please clarify whether this row refers to the replay harness rather than the live system, or explain the discrepancy.
[Abstract / §6.4] The abstract and Finding (RQ2) say a 65 GB checkpoint reaches its first token '6.4× sooner (1.6× from storage alone under identical files)'. As noted above, the 1.6× figure appears inconsistent with the reported copy-control ratio; this needs correction because the decomposition is a quantitative claim in a headline finding.
[§8.1] The statement 'sweeping the band θ from 2% to 20% changes no verdict in Table 9; only below the GH200 pair's own 1.4% gap does that tie resolve to stream-copy' is slightly misleading: because the sweep starts at 2%, it never tests the region below 1.4% where the GH200 verdict would change. Please report the sensitivity boundary explicitly or rephrase the 'not load-bearing' claim.
[§5.1 / §7.1] The paper repeatedly calls the adoption path 'public' and states that the mechanism is framework-independent, but it does not provide a repository URL or an artifact-availability statement. Please state where the 226-line producer extension and the llama.cpp patch will be made available for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central rate-parity claims are measured against external baselines, the one operational threshold is explicitly swept, and the held-out predictions were made prospectively.

full rationale

The paper's derivation chain is empirical rather than self-referential. The central claim that adoption removes the ingestion tax at no rate cost rests on a paired mapped-minus-resident measurement campaign with bootstrap intervals ([-0.66%, +0.48%]) and on the public route's measured 516 GB/s versus 53-82 GB/s for default constructors; no fitted value is renamed as a prediction. The three-condition execution contract is established by a randomized same-harness factorial with process-level contrasts and IQRs, and the ablation costs are measured, not imposed by the definitions of C1-C3. Algorithm 2's 5% tie band is explicitly described as an operational reporting threshold that is not load-bearing, and the paper reports sweeping it from 2% to 20% with no verdict change other than the GH200 pair's own sub-band gap; the 75-80% eviction boundary is labeled descriptive, not fitted. The held-out M3 machine sessions were predictions recorded before token generation, making them prospective validations rather than post-hoc fits. No load-bearing step relies on the authors' own prior work, and no uniqueness theorem or ansatz is imported by self-citation. The page-cache residency precondition is stated as a scope condition and checked by mincore before deployment, which is a documented limitation, not a circular input. The paper is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions, all explicitly stated or measured in the paper, plus one hand-chosen operational threshold (the 5% tie band) that the authors sweep and show to be non-load-bearing. No free parameters are fitted to data to produce the headline results. No new entities are posited.

free parameters (1)

theta (tie band in Algorithm 2) = 5% (swept 2-20%, no verdict change)
Chosen by hand as an operational reporting threshold for when map versus resident endpoints are considered tied. The paper states it is not load-bearing and sweeping it does not alter Table 9 decisions. It is included for completeness; it does not appear to be fitted to outcomes.

assumptions (4)

domain assumption On integrated and coherent-memory systems, clean file pages cached in DRAM are directly readable by the GPU at resident-class bandwidth.
Central premise behind the ingestion tax and the adoption mechanism; supported by measurements on Apple M5 Max, AMD APU, and GH200, but not a general theorem.
domain assumption A MAP_SHARED file mapping wrapped by Metal's newBufferWithBytesNoCopy remains shared and file-backed when the GPU wires it; MAP_PRIVATE would be converted to anonymous pages.
Stated in Section 5.1 and Appendix B. Measured on macOS and platform-specific, but load-bearing for the Apple path.
domain assumption DLPack capsule import into PyTorch and MLX treats the wrapped buffer as ordinary zero-copy storage, and downstream operators inherit the framework's stream and ordering.
Verified by mutation visibility, checksums, and paired campaigns in Sections 3-5, but depends on consumer framework behavior.
domain assumption The mapped working set must stay in the page cache during execution; once residency fails, storage traffic dominates and the in-memory comparisons no longer apply.
Explicit precondition in Section 10 and Algorithm 2 lines 2-4. The paper tests eviction and shows transparent reclaim, but the precondition bounds all headline claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks." pith.science (2026). https://pith.science/paper/56TCRUPW

@misc{pith2026260812114,
  author       = {Pith},
  title        = {Pith review of: The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/56TCRUPW}},
  note         = {Machine review of arXiv:2608.12114}
}

read the original abstract

Open-weight models can occupy a middle capacity regime: active weights fit in DRAM as cached file pages, but a second framework-owned representation does not fit or must be refilled as layers run, so low-batch decode rereads the weights every token. On integrated and coherent-memory systems the checkpoint's file pages already occupy a GPU-readable domain, yet accelerator-loading paths still copy them into framework-owned allocations. We call this the ingestion tax: the OS holds the bytes as clean, evictable file pages, the architecture makes them GPU-readable, and only the framework's ownership model stands between the two. We present file-backed weight adoption: a framework-independent producer maps each tensor with MAP_SHARED, wraps the pages as a no-copy GPU buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. Zero-copy import alone is not enough: a randomized factorial establishes a three-part execution contract - read the mapping, keep activations accelerator-resident, order on the GPU; a bolt-on extension forfeiting the last two runs 2.3x slower than stock. Under the contract, adoption removes the tax at no rate cost: 516 GB/s where default constructors reach 53-82, at parity with the same kernel over resident storage ([-0.66%, +0.48%]; Qwen2.5-72B 7.14 vs 7.23 tok/s). Parity buys ownership: N processes decode one mapped copy where resident loading pays N (at capacity, 5.5 vs 0.08 tok/s); a 65 GB checkpoint reaches its first token 6.4x sooner; the Kimi K3 spine stage falls from 2.62 to 0.35 s per token, 3.8x from storage alone. The same mechanism improves llama.cpp 1.21x at half the footprint on an AMD APU, ties overlapped streaming on a capacity-exceeding GH200, and loses 39x across PCIe: memory topology, not the API, decides the byte path. The deployment rule treats the page cache as a first-class, reclaimable accelerator-storage tier.

Figures

Figures reproduced from arXiv: 2608.12114 by the authors.

**Figure 1.** Two byte paths from the same clean file pages: per-use ingestion builds a second, framework-owned representation; adoption republishes the pages as ordinary framework storage. Condition Requirement C1: adopt The kernel reads the mapped file pages; no per-use framework copy is created. C2: retain Inputs and outputs remain acceleratorvisible across layer calls; activations do not round-trip through host arrays. C3: o… view at source ↗

**Figure 2.** The public mapped path at scale: (a) the working-set sweep, (b) dispatch granularity. Threat Control (claim it protects) clock ramp discard 12 passes, rotate arms (rates) cache-sized endpoint hold the full deployed set (policy) kernel confound one mapped/resident GEMV (attribution) format denominator count tensors read per token (rates) page-cache eviction residency queried before/after (regime) dead lazy graph forc… view at source ↗

**Figure 3.** The same three byte strategies on both memory architectures (log scale). On discrete hardware, mapping is not competitive (48 vs. 815: the interconnect decides) and staging into VRAM is the right policy. On unified memory, mapping reaches resident-class bandwidth, while the frameworks’ default copy path (61 GB/s) restages almost exactly the discrete bus rate (57) on hardware that has no bus. 7.3 Discrete PCIe: reje… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The ordering hierarchy; each excess is the median of process medians across 8 independent processes, ten timed passes each (process-level 95% intervals: in-stream [0.24, 0.47], event [0.40, 0.56], shared [3.14, 3.67], drain [5.66, 6.02]). In-stream work largely fills a…

**Figure 5.** Figure 5: The four measured working-set points use the standalone producer and the public kernel. Throughput rises from 548 to 578 GB/s across 2–16 GB, above the survey protocol’s 517 GB/s pure-read reference (dashed): a 16 GB pass amortizes the dispatch and synchronization bou…

**Figure 6.** Figure 6: Controlled same-kernel sweep. Adoption’s advantage narrows from 𝑇=1, passes through 3.66×, 1.88×, and 1.50×, and remains 1.25× at 𝑇=16. The dashed line is parity. Let𝐴(𝑇 ) denote the adopted arm’s arithmetic and dispatch time and𝐶(𝑇 ) the stock arm’s additional ingest…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

[1]

Abacus Noir. 2026. Zero-copy GPU inference from WebAssembly on Apple silicon.https://abacusnoir.com/2026/04/18/zero-copy-gpu- inference-from-webassembly-on-apple-silicon/

work page 2026
[2]

Gulavani, Alexey Tumanov, and Ramachandran Ramjee

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. InUSENIX Symposium on Operating Systems Design and Implementation (OSDI)

work page 2024
[3]

Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatami- fard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient large language model inference with limited memory. InProceedings of ACL

work page 2024
[4]

Tyler Allen and Rong Ge. 2021. In-Depth Analyses of Unified Virtual Memory System for GPU Accelerated Computing.Proceedings of SC (2021)

work page 2021
[5]

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Min- jia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. InProceedings of SC

work page 2022
[6]

Bennett Cooper, Thomas R. W. Scogland, and Rong Ge. 2024. Shared Virtual Memory: Its design and performance implications for diverse applications. InProceedings of ICS. 26–37. doi:10.1145/3650200.3656608

arXiv 2024
[7]

Andrew Crotty, Viktor Leis, and Andrew Pavlo. 2022. Are You Sure You Want to Use MMAP in Your Database Management System?. In Conference on Innovative Data Systems Research (CIDR)

work page 2022
[8]

Mohammad Dashti and Alexandra Fedorova. 2017. Analyzing memory management methods on integrated CPU-GPU systems. InProceedings of ISMM

work page 2017

Show all 39 references

[9]

DeepSpeed team. 2022. ZeRO-Inference: Democratizing massive model inference.https://www.deepspeed.ai/2022/09/09/zero- inference.html

2022
[10]

DLPack contributors. 2023. DLPack: an open in-memory tensor struc- ture.https://dmlc.github.io/dlpack/latest/

2023
[11]

Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low-latency serverless inference for large language models. InProceedings of OSDI

2024
[12]

Luigi Fusco, Mikhail Khalilov, Marcin Chrapek, Giridhar Chukkapalli, Thomas Schulthess, and Torsten Hoefler. 2024. Understanding data movement in tightly coupled heterogeneous systems: A case study with the Grace Hopper superchip. arXiv:2408.11556

2024 arXiv
[13]

Debashis Ganguly, Ziyu Zhang, Jun Yang, and Rami Melhem. 2019. Interplay Between Hardware Prefetcher and Page Eviction Policy in CPU-GPU Unified Virtual Memory. InInternational Symposium on Computer Architecture (ISCA)

2019
[14]

Debashis Ganguly, Ziyu Zhang, Jun Yang, and Rami Melhem. 2020. Adaptive Page Migration for Irregular Data-intensive Applications under GPU Memory Oversubscription. InIEEE International Parallel and Distributed Processing Symposium (IPDPS)

2020
[15]

ggml-org/llama.cpp contributors. 2023. llama.cpp: LLM inference in C/C++.https://github.com/ggml-org/llama.cpp

2023
[16]

Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Col- lobert. 2023. MLX: Efficient and flexible machine learning on Apple silicon.https://github.com/ml-explore/mlx

2023
[17]

Jiong He, Mian Lu, and Bingsheng He. 2013. Revisiting co-processing for hash joins on the coupled CPU-GPU architecture.Proceedings of the VLDB Endowment(2013)

2013
[18]

Intel. 2014. Getting the most from OpenCL 1.2: increasing perfor- mance by minimizing buffer copies on Intel processor graphics. Intel Developer Zone whitepaper

2014
[19]

Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fid- dler: CPU-GPU orchestration for fast inference of mixture-of-experts models. arXiv:2402.07033

2024 arXiv
[20]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica
[21]

Coskun, and Mar- tin Herbordt

Raphael Landaverde, Tiansheng Zhang, Ayse K. Coskun, and Mar- tin Herbordt. 2014. An investigation of unified memory access per- formance in CUDA. InIEEE High Performance Extreme Computing Conference (HPEC)

2014
[22]

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. 2023. Deja vu: Contextual sparsity for efficient LLMs at inference time. InProceedings of ICML

2023
[23]

ml-explore/mlx contributors. 2024. Discussion #615: Loading models with mmap.https://github.com/ml-explore/mlx/discussions/615

2024
[24]

ml-explore/mlx contributors. 2025. Issue #2878: On-disk (out-of-core) streaming of model weights for low-RAM inference.https://github.c om/ml-explore/mlx/issues/2878

2025
[25]

ml-explore/mlx contributors. 2026. MLX documentation: Conversion to NumPy and other frameworks (DLPack interchange).https://ml- explore.github.io/mlx/build/html/usage/numpy.html

2026
[26]

Anastasios Papagiannis, Giorgos Xanthakis, Giorgos Saloustros, Mano- lis Marazakis, and Angelos Bilas. 2020. Optimizing Memory-mapped I/O for Fast Storage Devices. InUSENIX Annual Technical Conference

2020
[27]

Anastasios Papagiannis, Giorgos Xanthakis, Giorgos Saloustros, Mano- lis Marazakis, and Angelos Bilas. 2021. FastMap: Revisiting the Memory-mapped I/O Path for Fast Storage Devices. InUSENIX Annual Technical Conference

2021
[28]

PyTorch contributors. 2026. PyTorch documentation: torch.from_file.https://docs.pytorch.org/docs/stable/gen erated/torch.from_file.html

2026
[29]

Prabod Rathnayaka, Fabian Waschkowski, and Lukas Wesemann. 2026. BaseRT: Best-in-class LLM inference on Apple silicon via native Metal. arXiv:2607.00501

2026 arXiv
[30]

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-throughput generative inference of large language models with a single GPU. InProceedings of ICML

2023
[31]

Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of SOSP

2024
[32]

Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE-Infinity: Activation-aware expert offloading for efficient MoE serving. arXiv:2401.14361

2024 arXiv
[33]

Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast large language model inference on a smartphone. arXiv:2406.06282

2024 arXiv
[34]

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InUSENIX Symposium on Op- erating Systems Design and Implementation (OSDI)

2022
[35]

Jiahuan Yu, Mingtao Hu, Zichao Lin, and Minjia Zhang. 2026. Super- Infer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips. InProceedings of Machine Learning and Systems (MLSys)

2026
[36]

Qi Yu, Bruce Childers, Libo Huang, Cheng Qian, and Zhiying Wang
[37]

≥55 GB free

Feng Zhang, Jidong Zhai, Bingsheng He, Shuhao Zhang, and Wen- guang Chen. 2017. Understanding Co-Running Behaviors on Inte- grated CPU/GPU Architectures.IEEE Transactions on Parallel and Distributed Systems(2017). 12 The Ingestion Tax: Adopting File-Backed Weights in Tensor Fr...

2017
[2020]

InThe Journal of Supercomputing

A Quantitative Evaluation of Unified Memory in GPUs. InThe Journal of Supercomputing
[2023]

InProceedings of SOSP

Efficient memory management for large language model serving with PagedAttention. InProceedings of SOSP

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Abacus Noir. 2026. Zero-copy GPU inference from WebAssembly on Apple silicon.https://abacusnoir.com/2026/04/18/zero-copy-gpu- inference-from-webassembly-on-apple-silicon/

work page 2026

[2] [2]

Gulavani, Alexey Tumanov, and Ramachandran Ramjee

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. InUSENIX Symposium on Operating Systems Design and Implementation (OSDI)

work page 2024

[3] [3]

Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatami- fard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient large language model inference with limited memory. InProceedings of ACL

work page 2024

[4] [4]

Tyler Allen and Rong Ge. 2021. In-Depth Analyses of Unified Virtual Memory System for GPU Accelerated Computing.Proceedings of SC (2021)

work page 2021

[5] [5]

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Min- jia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. InProceedings of SC

work page 2022

[6] [6]

Bennett Cooper, Thomas R. W. Scogland, and Rong Ge. 2024. Shared Virtual Memory: Its design and performance implications for diverse applications. InProceedings of ICS. 26–37. doi:10.1145/3650200.3656608

arXiv 2024

[7] [7]

Andrew Crotty, Viktor Leis, and Andrew Pavlo. 2022. Are You Sure You Want to Use MMAP in Your Database Management System?. In Conference on Innovative Data Systems Research (CIDR)

work page 2022

[8] [8]

Mohammad Dashti and Alexandra Fedorova. 2017. Analyzing memory management methods on integrated CPU-GPU systems. InProceedings of ISMM

work page 2017