{"id":"b872ac5a-6dc3-4d98-bc7b-df0ce15157d1","arxiv_id":"2507.01079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A fully on-device RAG pipeline using a partitioned, partially disk-loaded graph index and selective sentence-window reduction claims 1.72-8.89x faster vector search and up to 40.2% lower power than baselines, with comparable accuracy.","lead":"MobileRAG is an on-device retrieval-augmented generation pipeline that combines a disk-based vector index (EcoVector) with a sentence-selection method (SCR) to cut memory, latency, and power use on phones. The authors report large speedups and power savings on a Galaxy S24 while keeping question-answering accuracy roughly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'maintaining accuracy' claim rests on SCR's sentence-similarity proxy; without an answer-containment check, the measured accuracy maintenance could reflect sLM memorization rather than retrieval quality.","rationale":"The reader identified the SCR similarity proxy as the weakest assumption, and I agree. The central claim has two parts: MobileRAG is faster and cheaper, and it maintains accuracy. The speed, memory, and power improvements are backed by measured on-device experiments on a Galaxy S24, so those parts are reasonably supported even if the analytical formulas in Section 3.4 contain issues such as the missing N/N_c factor in the IVF-DISK and EcoVector memory expressions. The accuracy part is different: it depends on SCR discarding content that the sLM does not need, and the paper provides no direct evidence that the discarded content is actually irrelevant. The experiments in Table 5 show that accuracy does not drop, but without an answer-containment analysis it is unclear whether this is because the right sentences are kept or because the small LMs answer from memory. The additional problem of selecting SCR hyperparameters on the same test sets used for the final accuracy numbers weakens the claim of generalizability. These concerns do not refute the measured results, but they justify the CONDITIONAL verdict: the authors should provide the containment analysis, freeze hyperparameters on a development split, and ideally release code and exact configurations so the accuracy-maintenance claim can be independently reproduced.","tokens_in":19956,"tokens_out":5098,"duration_ms":58832,"concrete_test":"Re-run the SCR component on SQuAD, HotpotQA, and TriviaQA with an oracle answer-containment metric: for each query, after SCR selects its window(s), check whether the gold answer span (SQuAD/TriviaQA) or both supporting sentences (HotpotQA) appear in the retained prompt. Report P(answer in SCR output | retrieval hit) and compare against a control that keeps a random window of the same length. If the containment rate is not far above random, or if accuracy drops less than containment predicts, then the measured accuracy maintenance is at least partly due to parametric memorization, and the SCR proxy is unsupported. To remove test-set tuning, fit (sliding_window_size, overlap_size, context_extension_size) on a development split and freeze them before computing the Table 5 accuracy numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that GTE-Small sentence-level cosine similarity to the query (Section 4, Step 1) identifies the sentences the sLM actually needs. SCR keeps only the top-1 sliding window per retrieved document, plus context_extension_size sentences on each side, and discards the rest. If this proxy is wrong, the prompt loses the gold answer and the 'maintaining accuracy' half of the central claim collapses. The paper reports accuracy approximately equal to Naive-RAG in Table 5, but it never measures how often the retained window contains the gold answer span or, for HotpotQA, both supporting sentences. On SQuAD and TriviaQA, small LMs may answer from parametric memory even when the relevant evidence is discarded, and HotpotQA's multi-hop questions could be answered from one retained hop while the other is dropped. The SCR hyperparameters (sliding_window_size=3, overlap_size=2, context_extension_size=1) are selected by inspecting Figure 12 on the same test benchmarks later used for Table 5, so the reported accuracy is not a blind test of the proxy. This makes the 'without degrading accuracy' part of the central claim empirically fragile even though the measured numbers are internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MobileRAG, a fully on-device RAG pipeline that combines EcoVector, a disk-partitioned graph vector index, with Selective Content Reduction (SCR), a post-retrieval method that scores sentence windows against the query embedding and keeps only the most relevant content. The authors derive analytical memory, latency, and power models, and report experiments on a Galaxy S24 using SIFT and NYTimes for ANNS and SQuAD, HotpotQA, and TriviaQA with three small language models. The central claims are 1.72–8.89x search latency improvements, 10.7–54.5% memory reductions, 24.4–40.2% power reductions, and accuracy maintained relative to Naive-RAG and other baselines.","tokens_in":20179,"tokens_out":7629,"duration_ms":85700,"significance":"The problem is timely and the evaluation is genuinely on-device, with multiple datasets and small language models, which is a strength. If the claims hold, the EcoVector design of keeping a small centroid HNSW graph in RAM while paging per-cluster graphs from disk is a practical recipe for mobile vector search, and the SCR token-reduction idea would be useful for on-device RAG. However, three load-bearing points currently need additional support: the power model rests on current draws that are not independently sourced, the SCR accuracy-preservation claim is not verified by an answer-containment analysis and its hyperparameters are selected on the test benchmarks, and the EcoVector memory expression appears to undercount the loaded-cluster graph size. These issues do not invalidate the system concept, but they materially affect the strength of the central claims.","major_comments":[{"comment":"The power model uses I(t_s)=2300 μA and I(t_d)=800 μA, described only as \"determined\" with citations to general smartphone power analyses rather than to a measurement protocol on the Galaxy S24/Exynos 2400. Because these two constants enter the theoretical power values in Figure 9 and the retrieval-power discussion in Figure 13, the power advantage is partially fitted to the same experiments it is used to explain. Please provide an independent source or a direct measurement protocol for these current draws, and validate predicted vs. measured energy on a held-out configuration.","section":"Section 3.4.3; Figures 9 and 13"},{"comment":"SCR keeps only the top-1 sliding window per retrieved document based on GTE-Small cosine similarity to the query. The paper never reports how often the retained window contains the gold answer span, or both supporting sentences for HotpotQA, so the \"maintaining accuracy\" claim could partly reflect the small LM answering from parametric memory rather than from the preserved evidence. In addition, the SCR hyperparameters (sliding_window_size, overlap_size, context_extension_size) are selected by inspecting Figure 12 on the same three test benchmarks that are later used for Table 5; this is tuning on the test set, not a blind test of the sentence-similarity proxy. Please add an answer-containment analysis and a proper train/validation/test split for SCR hyperparameter selection.","section":"Section 4, Steps 1–2; Section 5.3.1; Table 5"},{"comment":"The loaded-cluster memory term 4(d + M'/(1-p0)) has the units of a single graph node, not a cluster. Since each inverted-list graph contains N/N_c vectors, this term should be (N/N_c)·4(d + M'/(1-p0)) (or an equivalent justification should be given). As written, the expression understates EcoVector's RAM footprint by roughly a factor of N/N_c in the graph term, which affects the theoretical memory curves in Figure 6 and the memory-reduction percentages stated in the abstract and conclusion.","section":"Table 1, EcoVector row"},{"comment":"The text defines T_search = t_s + t_d, but Table 2 lists only CPU-side search-time expressions such as ef_c·M' + n_P·ef_L·M' for EcoVector, with no corresponding disk-I/O term. Since the analytic power values in Section 3.4.3 require t_d, the end-to-end search-time expression should be stated explicitly and used consistently in the theoretical comparisons; otherwise the analytic latency and power contributions are incomplete.","section":"Section 3.4.2 and Table 2"}],"minor_comments":[{"comment":"The accuracy metric is not defined; please state whether it is exact match, F1, or another measure, and describe the evaluation protocol for the three QA datasets.","section":"Section 5.3"},{"comment":"The description alternates between scoring \"sliding windows\" and scoring \"chunks\" (e.g., the example assigns scores to Chunk1–Chunk5), which makes it unclear whether the window score is the max, mean, or some other aggregate of sentence scores. Please align the terminology and define the scoring precisely.","section":"Section 4, Steps 1–2"},{"comment":"The caption reads \"N1, N0.9, N0.7, N0.6 denote Naive-RAG at chunk ratios 1, 0.9, 0.8, 0.6,\" which mismatches N0.7 and the ratio 0.8. Please correct this inconsistency.","section":"Figure 13 caption"},{"comment":"The figures mix actual measured values and \"Theoretical Value\" markers without a clear legend; please distinguish measured bars from analytically computed values so the reader can see how well the models predict the measurements.","section":"Figures 6 and 9"},{"comment":"No variance or number of runs is reported. Please add standard deviations or confidence intervals, or at least state that each number is a single run, so the magnitude of the reported differences can be assessed.","section":"Table 5"},{"comment":"The summary line \"improving search latency by 1.72–8.89 times (at 0.93 recall@10 for SIFT)\" should specify the dataset, recall level, and whether this is retrieval-only or end-to-end latency, because the abstract's latency claim is broader than the evidence shown.","section":"Section 7, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper would benefit from releasing code and configuration files for reproducibility, since no artifacts are mentioned. If the EcoVector memory formula is corrected, the memory-efficiency advantage may shrink; the authors should re-check the constants in Table 1 before the next revision. The power-model concern is the more serious one: without independent current-draw measurements, the theoretical power analysis is not a prediction. The SCR accuracy concern is fixable with an answer-containment analysis, but it is essential before the 'maintaining accuracy' part of the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering paper with genuine on-device measurements. EcoVector is a combination of known pieces—k-means partitioning, HNSW over centroids, per-cluster graphs on disk—but the specific configuration and the SCR post-retrieval filter are a working pipeline that gets consistent latency, memory, and power improvements on a Galaxy S24 across three QA datasets and three small LMs. That is real evidence, and I trust the direction more than the reader's 4/10 soundness suggests.\n\nWhat the paper does well: the experiments are on actual hardware, the trends are consistent (1.7-8.9x search latency, 1.2-1.4x TTFT, 10-54% memory, 24-40% power), and the accuracy largely holds or improves. The SCR method is the most interesting piece—sentence-window selection plus reordering is cheap and clearly helps TTFT. The prototype chat app is a nice extra.\n\nSoft spots, in proportion. The biggest is that the SCR hyperparameters (sliding_window_size, overlap_size, context_extension_size) are chosen by looking at Figure 12 on the same test datasets used for the headline Table 5. That is tuning on the test set, and it inflates confidence in the accuracy-maintenance claim. Related: the paper never checks whether the selected window actually contains the gold answer or both supporting sentences for HotpotQA. The stress-test worry about small-LM memorization is fair—SQuAD/TriviaQA answers can come from parametric memory. But note: the measured accuracy is roughly maintained, so this is an unverified mechanism, not a refuted result. A validation split and an answer-containment statistic would settle it.\n\nThe analytical models are rougher than the experiments. The EcoVector memory expression in Table 1 looks like it drops the N/N_c factor in the loaded-cluster term; Table 2 lists search-time expressions without the disk I/O component defined in 3.4.2; and the power model's current draws (2300uA, 800uA) are 'determined' but not independently sourced. These are fixable, but they weaken the 'theoretical analysis' contribution. No code or exact hyperparameters are released, which hurts reproducibility.\n\nIf I were editing: send to peer review. The problem is relevant, the measurements are real, and the flaws are addressable in revision. Require code release, validation-based hyperparameter selection, answer-containment analysis, and corrected formulas. I'd bring it to a reading group—there's a good discussion about what counts as 'maintaining accuracy' in small-LM RAG.","headline":"Real on-device measurements make this a useful systems contribution; the accuracy-maintenance claim and the analytical models need tightening, but the paper deserves serious refereeing.","tokens_in":20800,"tokens_out":3377,"would_cite":true,"duration_ms":36087,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fully on-device RAG pipeline can cut search latency, memory, and power use while holding accuracy by pairing a cluster-partitioned vector index with selective pruning of retrieved text.","keywords":["retrieval-augmented generation","on-device RAG","approximate nearest neighbor search","graph-based vector index","selective content reduction","mobile large language models","memory-efficient indexing","power-efficient search"],"falsifier":"Take the three QA benchmarks used in the paper and, for each query with a known gold answer, record whether the SCR-selected window actually contains the gold answer sentence. If that containment rate is far below the reported accuracy, or if answer accuracy on the excluded sentences is near chance, then the claim that SCR maintains accuracy is falsified.","tokens_in":19660,"feed_emoji":"📱","tokens_out":7069,"duration_ms":72077,"temperature":0.7,"pith_summary":"MobileRAG sets out to make retrieval-augmented generation practical on a phone, where RAM, battery, and heat budgets are tight. It combines EcoVector, a vector index that keeps a small graph of cluster centers in memory and loads per-cluster graphs from disk only when needed, with Selective Content Reduction (SCR), which re-chunks retrieved documents and keeps only the sentences most similar to the query before sending them to a small language model. The paper reports that on a Galaxy S24 this beats standard vector-search and RAG pipelines by 1.72–8.89x in search latency, reduces time to first token by 1.18–1.41x, cuts memory by 10.7–54.5%, and cuts power by 24.4–40.2%, with accuracy matching or exceeding baselines. The payoff, if true, is that private personal data can be searched, summarized, and questioned entirely on-device.","feed_headline":"On-device RAG: 1.72-8.89x faster search, same accuracy","feed_subtitle":"A phone-local index and a sentence-level filter cut memory, power, and time-to-first-token on a Galaxy S24.","key_machinery":"The two load-bearing mechanisms are EcoVector and Selective Content Reduction. EcoVector is a two-level graph index: a small HNSW graph (a hierarchical navigable small-world graph) over cluster centroids lives in RAM, while each cluster's own HNSW graph is stored on disk and loaded cluster-by-cluster during a query, so distance computations—the dominant power cost—are cut while memory stays near disk-based IVF. SCR is a post-retrieval filter: it splits each retrieved document into overlapping sentence windows, scores every window against the query with the same embedding model used for retrieval, keeps the top window plus a fixed number of neighboring sentences, merges the kept text across documents, and reorders documents by their best window score so that the prompt fed to the small language model is both shorter and better ordered.","core_discovery":"The central claim is that the two perceived bottlenecks of on-device RAG—the vector index and the language-model input—can both be attacked without surrendering accuracy, by restructuring where computation happens. EcoVector partitions embeddings into clusters, builds a hierarchical navigable small-world graph over the few thousand centroids that stays in RAM, and stores separate small graphs for each cluster's vectors on flash storage, loading and unloading them per query. This trades some disk I/O for far fewer CPU distance computations, which the paper argues is a net win because CPU draw dominates mobile power. SCR then re-embeds sliding windows of the retrieved documents, keeps the single most query-similar window per document plus a small context extension, merges the survivors, and reorders documents by their top window score. Together the two components shrink both the search cost and the token count handed to the small language model; the experiments claim accuracy is maintained or improved while latency, memory, and energy all drop.","pith_inferences":["The paper does not measure how often SCR's top-1 window actually contains the gold answer; an obvious extension is an answer-aware or trained selector that could make the accuracy guarantee robust rather than benchmark-tuned.","EcoVector's two-level design suggests a natural hierarchy for very large personal corpora: cluster again within each disk-resident cluster to keep per-query I/O bounded as the collection grows past what one level of partitioning handles.","The power argument depends on phone flash being fast enough that disk reads are cheap relative to CPU cycles; on phones with slower eMMC storage the balance could reverse, so the reported gains may not transfer to the lowest-end devices.","The reported SCR parameters (window size 3, overlap 2, context extension 1) are fixed across all experiments and tuned on the evaluation benchmarks; a parameter-free or per-query adaptive window size is a testable way to tell whether the method's accuracy hold depends on that tuning."],"forward_implications":["On the reported hardware, EcoVector reaches 0.93 recall@10 on a million-vector SIFT set with 1.72–8.89x lower search latency than IVF, HNSW, and their disk-based variants.","SCR trims average per-document input tokens by 42% on SQuAD, 31% on TriviaQA, and 7% on HotpotQA at unchanged accuracy, which translates into 10.4–41.6% lower time-to-first-token depending on the small language model.","Across the full RAG pipeline the method lowers power by 24.4–40.2% and memory by 10.7–54.5% compared with Naive-RAG, EdgeRAG, and Advanced RAG, with the gains growing for larger on-device models.","Because the index, retrieval, and generation all run locally, a phone can answer queries about personal documents with no network round trip, which is the privacy argument the paper makes.","SCR's document reordering step behaves like a reranker without a separate model, letting MobileRAG roughly match Advanced RAG accuracy without the extra inference cost."],"supporting_citations":[{"why":"Supplies the HNSW graph construction and search that EcoVector reuses for its centroids graph and per-cluster inverted-list graphs.","marker":"[39]"},{"why":"Defines the Naive-RAG retrieval-then-generation pipeline that MobileRAG replaces and uses as a primary accuracy and latency baseline.","marker":"[31]"},{"why":"EdgeRAG is the closest on-device RAG baseline, using IVF-DISK and embedding caching; MobileRAG measures its memory, latency, and power gains against it.","marker":"[48]"},{"why":"GTE-Small is the embedding model that produces query and sentence-window similarities in SCR, so the reduction's correctness depends on it.","marker":"[32]"},{"why":"BERTSUM is the compressor baseline that SCR is compared against in the accuracy-versus-input-size experiments.","marker":"[34]"},{"why":"ANN-Benchmarks provides the SIFT and NYTimes datasets and the recall/QPS measurement protocol used to evaluate EcoVector.","marker":"[2]"},{"why":"Qwen2.5 0.5B and 1.5B are the small language models whose time-to-first-token and power consumption are measured across all RAG variants.","marker":"[58]"},{"why":"DeepSeek-R1 1.5B is the third small language model used to show that SCR's token savings grow with model size.","marker":"[15]"}],"fun_headline_variants":["MobileRAG: 1.72–8.89x faster on-device search, accuracy same","On-device RAG: EcoVector and SCR cut memory, power, and latency","MobileRAG: slice the index, trim the prompt, run RAG offline","Phone-friendly RAG: 1.72–8.89x faster search, no accuracy hit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that SCR keeps accuracy rests on sentence-level embedding similarity being a reliable proxy for the sentences the language model actually needs, yet the paper never measures how often the selected window contains the gold answer.","fun_headline_variants_meta":{"raw":{"variants":["MobileRAG: 1.72–8.89x faster on-device search, accuracy same","On-device RAG: EcoVector and SCR cut memory, power, and latency","MobileRAG: slice the index, trim the prompt, run RAG offline","Phone-friendly RAG: 1.72–8.89x faster search, no accuracy hit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3267,"prompt_tokens":926,"completion_tokens":2341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2244}},"tokens_in":542,"tokens_out":2341,"duration_ms":21489,"temperature":1.0,"reasoning_tokens":2244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:05:26.748904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the three QA benchmarks used in the paper and, for each query with a known gold answer, record whether the SCR-selected window actually contains the gold answer sentence. If that containment rate is far below the reported accuracy, or if answer accuracy on the excluded sentences is near chance, then the claim that SCR maintains accuracy is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Naive-RAG retrieval-then-generation pipeline that MobileRAG replaces and uses as a primary accuracy and latency baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ANN-Benchmarks provides the SIFT and NYTimes datasets and the recall/QPS measurement protocol used to evaluate EcoVector."}],"review_version":1}