Pith. sign in

REVIEW 4 major objections 4 minor 18 references

omni-macos: On-Device Omni-Modal Search on Apple Silicon

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read omni-macos runs a full omni-modal search engine—encoder, index, and store—in one process on a Mac, bound by a user-set memory cap, with no file, query, or vector leaving the device.

desk verdict A well-measured on-device omni-modal search engine with an honest but unproven memory-cap guarantee; deserves refereeing. read the letter →

arxiv 2608.05543 v1 pith:YWMOLMQM submitted 2026-08-06 cs.IR

Han Xiao This is my paper · ORCID
classification cs.IR
keywords on-devicesearchomni-modalembeddingsApplesiliconunifiedmemorybudgetvectorchunk-hashreusererankfunnel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that semantic search over all local file types—text, code, documents, images, audio, and video—can be delivered entirely on the Mac that holds the files, with no server component and no data leaving the machine. The load-bearing design target is not the encoder, which fits, but the coexistence of a continuous background indexer and an interactive search-as-you-type box inside one memory ceiling the user sets. The engine keeps to that ceiling by re-encoding only the chunks an edit changes, handing the GPU smaller units while the user types, answering scans from a 4-bit replica with exact rescoring of a shortlist, and propagating the cap to every allocator it can find that draws on unified memory. The paper reports measurements of each mechanism on five Macs spanning an eightfold range of accelerator width and a thirty-twofold range of memory, and argues the system holds to the budget while keeping the device usable for other work.

What carries the argument

The central mechanism is a one-process, one-encoder architecture with two paths—index and query—meeting at a non-preemptive two-class gate on Apple silicon. Around it sit four load-bearing devices: fixed-grid chunking with per-chunk content hashes, so an edit re-encodes only chunks whose text or embedding parameters changed; anticipatory shaping, which uses the debounce interval between keystrokes to make the indexer hand the gate smaller units, bounding the longest wait a query can inherit; a rerank funnel holding a resident 4-bit replica of every vector, scanned in full, with a fixed-size shortlist rescorred against exact vectors; and cap propagation, which routes the user's one memory number to every allocator that draws on unified memory, including framework buffer caches and compaction transients that no single setting names.

What would settle it

Run omni-macos on an Apple-silicon machine with a small pinned cap, index a large corpus, then trigger a compaction and a scan-matrix conversion while the encoder and the resident base are loaded; if any allocation draws on unified memory outside the enumerated allocators, peak resident memory will exceed the cap.

Watch

Extended reading notes

Core claim

On the paper's own terms, omni-macos is the first engine to run an omni-modal encoder in the application process on consumer Apple silicon while a continuous indexer and an interactive query path share the device. The entire stack—crawler, chunker, encoder, vector store, query path, and local service—lives in one process with no Python and no separate server, so nothing indexed or queried is ever transmitted and the system runs air-gapped. The user sets a single memory ceiling; the system propagates that number to all allocators that draw on Apple's unified memory, and it demonstrates on five machines that indexing keeps pace with edits while queries stay responsive. The measurements show the accelerator is busy 97.7–99.7% of a fresh indexing pass, that chunk-hash reuse removes 86–88% of append re-indexing time and 40–43% of mid-file re-indexing time, that shaping cuts the 99th-percentile query wait by 45–93%, and that the coarse-then-exact funnel scans a quarter of the bytes while matching or exceeding the exact bf16 scan on the same encoder.

Load-bearing premise

The user-set memory cap is guaranteed only if the empirically discovered list of allocators that draw on unified memory is complete, and the paper states in Section 3.5 that no interface enumerates them, so a missed allocator could let the process exceed the cap under a usage the list did not anticipate.

Editorial extensions

If this is right

  • A user can search across every file type on a Mac without uploading anything: the engine indexes and answers with the network disconnected, so privacy follows from where the code runs rather than from access-control or encryption layering.
  • Because queries scan a quarter-size 4-bit replica and rescore only 4096 candidates, per-query cost stays nearly flat with corpus size, so a laptop with 16 GB can answer over millions of chunks in single-digit milliseconds.
  • Rapid file editing no longer forces full re-indexing: appends recover about 86–88% of re-encoding time and mid-file edits about 40–43%, making continuous indexing affordable on the machine being used.
  • Interactive search remains responsive during background indexing: the 99th-percentile query latency falls by 45–93% when anticipatory shaping is armed, at a median-latency cost only on accelerators that are already saturated.
  • The same engine can serve other local agents as a file-search service over local HTTP, exposing MCP tools and an embeddings route so an agent can use it without configuration and no data leaves the machine.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Chunk-hash reuse on a fixed grid should transfer to any live index of locally edited files: the hash has to key every parameter that changes the meaning of a chunk, and the lookup must stay scoped to one file's history to avoid aliasing across duplicate text.
  • The admission-control principle generalizes: on any device with non-preemptive execution, interactive latency can be bounded by shrinking the size of queued work during a known pre-query signal, without needing cancellation or priority inversion.
  • A testable hardening of the memory guarantee would be to enumerate all unified-memory allocators automatically at runtime, turning the empirically discovered list into a checkable invariant.
  • If the budget holds in all conditions, the engine could serve as a memory-accounted local retrieval backend for on-device agents, where the cap becomes a composable resource contract rather than a single-app setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. omni-macos presents a fully on-device, single-process search engine for Apple Silicon that embeds text, code, documents, images, audio and video into one representation space, keeps a continuously updated index, and exposes an interactive search box, all under a user-set memory cap. The paper's contributions are (i) a Swift/Metal implementation of the jina-embeddings-v5-omni encoder with per-tensor precision, merged kernel launches, and tail-row narrowing; (ii) chunk-hash reuse across edits; (iii) anticipatory shaping of GPU admission; (iv) a 4-bit coarse-plus-exact rerank funnel; and (v) propagation of the memory cap to the allocators that draw on unified memory. The evaluation covers five Macs spanning a wide hardware range, with real per-machine corpora, a pinned synthetic corpus, per-mechanism ablations, and detailed appendices on peak transients, funnel crossover, and selection. The paper is honest about costs and failures, including shaping's median regression on saturated accelerators and the can't-win prune's occasional loss.

Significance. If the central claims hold, this is a valuable systems contribution: it demonstrates that omni-modal semantic search with continuous indexing can run entirely on consumer hardware in one process, and it ships reproducible measurements (pinned corpus hash, generated corpus, seeded vectors) and open-source code. The decision to treat the encoder as consumed prior work, citing its published retrieval quality rather than re-deriving it, is appropriate and keeps the paper's contribution focused on the systems layer. The ablations are carefully designed against the same binary, and several predictions are quantitative enough to be checked, such as tail-row narrowing landing near the derived 6.25%. The main weakness is that the most distinctive guarantee — the memory cap — is supported only by an empirically assembled list of allocators that the paper itself states cannot be proven exhaustive; the headline claim needs either a stronger enforcement argument or an explicit qualification. The shaping and funnel-accuracy claims also need a few missing baselines and dispersion measures before the quantitative conclusions can be taken at face value.

major comments (4)
  1. [3.5 / Abstract] The memory-cap guarantee is stated unconditionally in the Abstract ('inside one memory budget the user sets') and in Section 6 ('the budget reaches every allocator'), but Section 3.5 concedes that 'no interface enumerates the allocators that draw on the pool, so the list can only be assembled empirically' and that the cap is propagated to 'each allocator we could find.' A missed allocator — a Metal driver cache, a future code path, or a third-party library such as the local MCP server — would let the process exceed the user-set ceiling, violating the guarantee as written. Because this is the central user-facing promise, the revision must either provide a systematic argument or OS-level mechanism that bounds every unified-memory consumer, or explicitly restrict the claim to the tested allocators and workloads and adjust the Abstract and Conclusion accordingly.
  2. [4.2 / Table 2] The text claims that shaping reduces the 99th percentile of search-while-indexing latency by 78–93% on four machines and 45% on the fifth, and that unshaped queries could wait more than a second, but no unshaped baseline appears in Table 2 or in the surrounding text; the 'Search while indexing' row reports only one condition. Please report the full p50/p95/p99 distributions for both the shaped and unshaped conditions on all five machines, with the same sample sizes, so the claimed reduction can be verified.
  3. [4.3 / Table 3] Each mechanism cell in Table 3 is a single percentage per machine with no repetition count, standard error, or confidence interval. This matters because some effects are small (tail-row narrowing, 5.3–6.6%) or highly variable (can't-win prune, −9.5 to +32.7%), and the paper itself acknowledges in Appendix A.1 that sub-millisecond kernels are not separable from run-to-run variation. Please add repeat counts and dispersion measures per cell, or state explicitly that the table records one run and interpret the between-machine regularities accordingly.
  4. [A.3 / Table 6] The recall comparison at 2M and 4M chunks rests entirely on synthetic rows and queries fitted with a single anisotropy parameter α=0.249, and no sensitivity analysis is provided. Since the measured recall gap between the bf16 scan and the funnel is at most about 0.01, plausible differences in tie density between the synthetic distribution and real corpora could change the conclusion. Please either validate the ranking at 4M rows on a real corpus or vary α and the query-generation procedure and report the range of recall differences; a sentence describing how α=0.249 was estimated would also aid reproducibility.
minor comments (4)
  1. [1] In the Introduction, 'is aworseproduct than' is a typo for 'is a worse product than'.
  2. [4.3 / Table 3] The table header says 'so each percentage is a gain,' which is inconsistent with the negative can't-win prune entry of −9.5 on the M4; clarify that negative values are losses.
  3. [3.3 / Figure 2] The label '180 ms 2 s activity window' is ambiguous; please state explicitly which constant is the debounce interval and which is the activity window.
  4. [A.4 / Table 7] The memory guard that declines larger rungs on smaller machines is described in the text but not in the caption; adding it to the caption would make the table stand alone.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system's mechanisms are measured against their own ablated baselines, and the encoder-quality self-citations are not load-bearing.

full rationale

The paper's derivation chain is self-contained: each mechanism (reuse, shaping, rerank funnel, cap propagation) is implemented, ablated against the same binary, and measured on real corpora and a pinned synthetic corpus. The only fitted scalar, the anisotropy alpha = 0.249 in Appendix A.3, is used to synthesize query directions for scale-testing the funnel, not to set any reported latency, recall, or memory number; the funnel's accuracy is judged against an exact fp32 top-10 from the same encoder, so the comparison is internal and not circular. The encoder itself is prior published work by the same group, and the paper explicitly treats retrieval quality as a property of the consumed model ('reported with it'), so citing its benchmark papers is not load-bearing. The Section 3.5 admission that the allocator list 'can only be assembled empirically' is an honest completeness limitation of the memory-cap guarantee, not a reduction of the result to its input; no prediction is forced by construction. Accordingly no circular step is identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; it is a software system. The central claim draws on Apple hardware behavior, prior encoder work, and standard retrieval theory as axioms. The only fitted parameter in the evaluation pipeline is the anisotropy alpha used for synthetic queries; the shortlist size and debounce constants are design choices, not fitted to the target result.

free parameters (4)
  • Shortlist size C = 4096
    Fixed constant for coarse-then-exact rescoring, independent of N; determines recall and query cost (Section 3.4, Table 6).
  • Anisotropy alpha for synthetic queries = 0.249
    Fitted to real index vectors to generate synthetic query directions in Appendix A.3; affects the funnel accuracy and crossover measurements.
  • Debounce interval and activity window = 180 ms and 2 s
    Chosen timing constants for search-as-you-type shaping (Figure 2, Section 3.3); affect shaping effectiveness.
  • Pinned memory cap for ablations = 6 GB
    Ablation setting applied across all five machines (Section 4.3); affects crossover and memory results.
assumptions (5)
  • domain assumption Metal command buffers are non-preemptive and cannot be cancelled or reprioritized once committed
    Section 3.3 cites Apple documentation; the entire anticipatory shaping design rests on this property.
  • domain assumption macOS memory-pressure reporting is advisory rather than enforced
    Section 3.5 cites Apple documentation; this motivates the custom cap propagation mechanism.
  • domain assumption Cosine similarity of the Swift/Metal encoder to the reference implementation implies retrieval equivalence
    Section 3.1 checks cosine similarity 'to rounding level' but no downstream retrieval benchmark is run.
  • domain assumption GPU and CPU share unified memory such that all allocators draw on one pool
    Sections 1 and 3.5 rely on Apple's unified memory model for the central memory-cap argument.
  • standard math Coarse-then-exact retrieval and top-k selection preserve ranking
    Section 3.4 and Appendix A.3 rely on standard retrieval theory (Jegou et al.) for the rerank funnel.

how reviews work

0 comments
Cite this review

Pith. "Pith review of omni-macos: On-Device Omni-Modal Search on Apple Silicon." pith.science (2026). https://pith.science/paper/YWMOLMQM

@misc{pith2026260805543,
  author       = {Pith},
  title        = {Pith review of: omni-macos: On-Device Omni-Modal Search on Apple Silicon},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWMOLMQM}},
  note         = {Machine review of arXiv:2608.05543}
}
read the original abstract

A search engine that embeds text, code, documents, images, audio and video into the same representation space has to run its encoder and keep its index somewhere, and almost every component built for the purpose assumes a server. We present omni-macos, which runs that whole engine, encoder, index and store, on the Mac the files are already on, so no file, query or vector ever leaves the machine. It keeps a background indexer and an interactive search box inside one memory budget the user sets: it re-encodes only the chunks an edit changes, hands the GPU smaller units while the user is typing, answers queries from a quantized replica with exact rescoring, and propagates that budget to the allocators that draw on unified memory. We measure every mechanism on five Macs spanning an eightfold range of accelerator width and a thirty-twofold range of memory, each one indexing its own local files.

Figures

Figures reproduced from arXiv: 2608.05543 by the authors.

Figure 1
Figure 1. omni-macos answering porsche sports car over one real corpus of local files. One query returns one ranked list whose members are a PDF, rendered HTML, XML, source code, Mark￾down and photographs, because every file type shares one embedding space [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The two paths and the one encoder they share. A query enters from the search box, where [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Anticipatory shaping. Time runs left to right; both lanes carry the same indexing work and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The rerank funnel. Bar width is bytes per row and bar height is rows, so the replica is a [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Fused attention throughput against sequence length at the attention shape of the vision [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

  1. [1]

    Quantization for Vector Search under Streaming Updates.arXiv preprint arXiv:2512.18335,

    Ishaq Aden-Ali, Hakan Ferhatosmanoglu, Alexander Greaves-Tunnell, Nina Mishra, and Tal Wag- ner. Quantization for Vector Search under Streaming Updates.arXiv preprint arXiv:2512.18335,

    arXiv

  2. [5]

    Native LLM and MLLM Inference at Scale on Apple Silicon.arXiv preprint arXiv:2601.19139,

    Wayner Barrios. Native LLM and MLLM Inference at Scale on Apple Silicon.arXiv preprint arXiv:2601.19139,

    arXiv

  3. [10]

    doi: 10.1145/3600006.3613165. Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836,

    work page arXiv

  4. [14]

    A faster generalized two-stage approximate top-k.Transactions on Machine Learning Research, 2026,

    Yashas Samaga, Varun Yerram, Spandana Raj Babbula, Prateek Jain, and Praneeth Netrapalli. A faster generalized two-stage approximate top-k.Transactions on Machine Learning Research, 2026,

    work page 2026

  5. [16]

    Omni-Embed-Nemotron: A unified multimodal retrieval model for text, image, audio, and video.arXiv preprint arXiv:2510.03458,

    Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. Omni-Embed-Nemotron: A unified multimodal retrieval model for text, image, audio, and video.arXiv preprint arXiv:2510.03458,

    arXiv

  6. [17]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan

    doi: 10.1145/3805713.3820402. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment. In ICLR,

    work page arXiv

  7. [346]

    Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS.arXiv preprint arXiv:2511.05502,

    Varun Rajesh, Om Jodhpurkar, Pooja Anbuselvan, Mantinder Singh, Ashok Jallepali, Shantanu God- bole, Pradeep Kumar Sharma, and Hritvik Shrivastava. Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS.arXiv preprint arXiv:2511.05502,

    arXiv

  8. [1272]

    At the short rungs on the M3 Ultra the kernel runs in half a millisecond, where the arms are not separable from run-to-run variation. M3 Ultra 2025 M4 Pro 2024 M4 2024 M3 Pro 2023 M2 2022 Patch count 256 1.11 1.21 1.17 1.25 1.08 512 0.97 1.09 1.12 1.10 1.09 1000 1.10 1.15 1.23 1.23 1.17 1272 1.00 1.05 0.79 1.20 0.86 2000 1.12 1.23 1.52 1.26 1.21 4888 1.23...

    work page 2025

Show all 18 references
  1. [2003]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra

    doi: 10.1145/95 6863.956944. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. InCVPR, pages 15180–15190,

  2. [2011]

    Taehee Jeong

    doi: 10.1109/TPAMI.2010.57. Taehee Jeong. 4bit-Quantization in Vector-Embedding for RAG.arXiv preprint arXiv:2501.10534,

    2010 arXiv

  3. [2014]

    Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md

    doi: 10.1109/ISCA.2014.6853208. Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham, and Arman Luthra. Fu- sion Embedding: A unified embedding space for text, image, video, and audio.arXiv preprint arXiv:2607.18666,

    2014

  4. [2015]

    doi: 10.1145/2694344.2694

  5. [2020]

    12 Leland McInnes, John Healy, and James Melville

    doi: 10.1109/TPAMI.2018.2889473. 12 Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426,

    2018

  6. [2021]

    Darya Kaviani, Alp Eren Ozdarendeli, Jinhao Zhu, Yu Ding, and Raluca Ada Popa

    doi: 10.1109/TBDATA.2019.2921572. Darya Kaviani, Alp Eren Ozdarendeli, Jinhao Zhu, Yu Ding, and Raluca Ada Popa. Opal: Private memory for personal AI.arXiv preprint arXiv:2604.02522,

    2019

  7. [2023]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace

    doi: 10.1109/CVPR52729.2023.01457. Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. InOSDI, pages 443–462,

    2023

  8. [2024]

    Apple Inc

    doi: 10.186 53/v1/2024.acl-long.678. Apple Inc. MTLCommandBuffer. Metal Developer Documentation, 2026a.https://develope r.apple.com/documentation/metal/mtlcommandbuffer. Apple Inc. MTLDevice:hasUnifiedMemoryandrecommendedMaxWorkingSetSize. Metal Developer Documentation, 2026b....

    2024 arXiv

  9. [2025]

    Locally-adaptive quantization for streaming vector search.arXiv preprint arXiv:2402.02044,

    Cecilia Aguerrebere, Mark Hildebrand, Ishwar Singh Bhati, Theodore Willke, and Mariano Tep- per. Locally-adaptive quantization for streaming vector search.arXiv preprint arXiv:2402.02044,

    arXiv

  10. [2026]

    Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S

    doi: 10.1145/380571 2.3808455. Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. InACL, pages 12562–12584,

    doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.