{"id":"32716ab5-e43a-4060-b8a1-1bdcb04b847c","arxiv_id":"2411.08244","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An NVCiM-assisted prompt tuning framework stores per-domain virtual tokens in non-volatile memory, retrieves them via a multi-scale search, and improves edge LLM accuracy and speed.","lead":"This paper describes a prompt-tuning framework for edge large language models that uses non-volatile computing-in-memory (NVCiM) hardware to store and retrieve per-user virtual tokens. It reports accuracy and latency improvements over several baselines, positioning NVCiM as a way to make on-device LLM personalization more efficient.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SSA retrieval is the load-bearing assumption: WMSDP (Eq. 5) is never validated against oracle or random OVT selection, so the headline gains might not come from the retrieval contribution.","rationale":"The reader's weakest-assumption analysis correctly identifies SSA retrieval as the most load-bearing component. The central claim depends on the framework retrieving the right OVT from NVM; if WMSDP does not rank OVTs by usefulness, the framework degenerates to near-random prompt selection and the empirical gains would need a different explanation. The paper does present substantial supporting evidence: a broad evaluation across three LLMs, five LaMP datasets, five NVM devices (Table I), buffer-size and noise-variation sweeps (Tables III, IV), and NeuroSim-based latency/energy comparisons. These are real and relevant. However, none of these experiments validate the retrieval ranking itself, because every reported result confounds SSA with the other components. The missing oracle control is precisely what would settle whether the retrieval assumption holds. This does not require changing the reader's conditional verdict: the paper should remain conditional until the authors provide the oracle comparison, disclose the missing hyperparameters, and ideally release code. My concern is a strengthening reason for the same conditionality, not a rejection of the paper's plausibility.","tokens_in":15316,"tokens_out":5571,"duration_ms":60023,"concrete_test":"Add an oracle-retrieval control to the Table I evaluation: for each test sample, replace the SSA-retrieved OVT with the OVT trained on that same sample (or on the same user's data, for per-user tasks), keeping every other component identical. Also add a random-OVT lower bound. If NVCiM-PT's end-task accuracy is close to the oracle upper bound and clearly above random selection, SSA is validated; if it is closer to random selection, the retrieval component is not reliably selecting the optimal OVT and the headline performance gains are not attributable to SSA. Run this check on at least one LLM-dataset pair from Table I, e.g., Phi-2 on LaMP-5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central novelty is the store-and-scaled-search algorithm (SSA), yet Sec. III-D2 justifies it on an acknowledged representational gap: word embeddings and prompt-tuned virtual tokens \"are not compatible.\" SSA's WMSDP (Eq. 5) tries to bridge this gap with multi-scale average pooling, but the paper provides no evidence that WMSDP ranks OVTs by their actual benefit to the LLM. The experiments compare SSA against MIPS and noise-mitigation baselines, but they omit the two controls that would validate retrieval: (i) a random-OVT lower bound and (ii) an oracle-retrieval upper bound in which the OVT known to correspond to the test sample/user is used. Without these, the gains in Tables I, III, and IV could be produced entirely by noise-aware training and representative selection, while SSA may be retrieving near-random prompts. Since the paper's title contribution is NVCiM-assisted retrieval, this unvalidated ranking assumption is load-bearing. A secondary internal issue is Eq. 3, which selects a representative sample by argmin of cosine similarity; taken literally that selects the least similar sample to the centroid, though this may be a typographical error for argmax. The primary unresolved risk remains the SSA ranking property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NVCiM-PT, a prompt-tuning framework for edge LLMs that leverages non-volatile computing-in-memory (NVCiM) hardware. The framework has three components: representative selection (RS) to reduce the data buffer via k-means clustering and cosine-similarity selection; noise-aware training (NT) that injects Gaussian device-variation noise into virtual tokens during prompt tuning; and a store-and-scaled-search algorithm (SSA) that encodes virtual tokens with an autoencoder, stores them in NVM crossbars, and retrieves an OVT for each user input using a weighted multi-scale dot product. Experiments across three LLMs, five LaMP datasets, five NVM devices, and five baselines report consistent improvements, with up to 36.7% accuracy/ROUGE gains, 120x lower search latency, and 60x lower energy than a Jetson Orin CPU. The central claim is that NVCiM-PT, and in particular the SSA retrieval, enables effective use of optimal virtual tokens (OVTs) for domain-shift adaptation under edge resource constraints.","tokens_in":15647,"tokens_out":3029,"duration_ms":30663,"significance":"If the retrieval and noise-mitigation effects are real, NVCiM-PT addresses a genuine problem: improving edge LLM personalization under domain shift with limited resources by storing per-domain prompts in NVM. The empirical scope is broad: three LLMs, five task datasets, five NVM devices, and multiple baselines, with consistent gains in Tables I, III, and IV. The framework is a plausible co-design of prompt tuning and CiM architecture. However, the paper's main novelty, SSA, is only compared against MIPS and not against random or oracle retrieval, so the attribution of gains to SSA's ranking quality is not established. The missing hyperparameter values and lack of variance reporting further weaken the reproducibility and statistical confidence. The latency/energy results are promising but preliminary because they rely on NeuroSim simulations without a clear end-to-end accounting. With additional controls and specifications, the contribution could be solid.","major_comments":[{"comment":"The SSA retrieval ranking is load-bearing for the paper's central claim, but it is never validated in isolation. The experiments compare SSA against MIPS and noise-mitigation baselines, yet they omit two essential controls: a random-OVT baseline (lower bound) and an oracle-retrieval baseline where the OVT is known to correspond to the test user/sample (upper bound). Without these, the gains in Tables I, III, and IV could be produced entirely by the representative selection and noise-aware training, while WMSDP might be retrieving near-random prompts. Since Section III-D2 itself acknowledges that word embeddings and virtual tokens are not compatible, the pooling assumption in Eq. (5) needs direct validation. Please add experiments showing that WMSDP ranks OVTs by their actual benefit to the LLM, e.g., by comparing the LLM performance under SSA-selected OVTs with random and oracle OVTs across several datasets and devices.","section":"§III-D2, Eq. (5)"},{"comment":"Equation (3) uses argmin of cosine similarity to select the most representative sample e*_i in each cluster. Taken literally, argmin selects the sample least similar to the cluster centroid, which is the opposite of representativeness. This appears to be a typographical error for argmax, but the discrepancy must be resolved because the RS component is the first step of the pipeline and directly affects which OVTs are trained. If argmin was intended, the authors should justify why outlier samples are beneficial.","section":"§III-B, Eq. (3)"},{"comment":"Key hyperparameters in the proposed method are not specified, making the results irreproducible. Equation (2) requires values for nmin, nmax, b0, and s, while Eq. (4) requires the noise factors f1, f2, f3, f4 and the relationship between the global σ (described as 0.1 in Section IV-A2) and the level-dependent standard deviations in Table II. None of these values are reported in the experimental setup or in the default settings. Please provide the exact values used for all reported experiments, or include a sensitivity study showing how performance varies with these parameters.","section":"§III-B, Eq. (2) and §III-C, Eq. (4)"},{"comment":"No repeated-run variance or confidence intervals are reported. The experiments use small buffer sizes (10–60 samples) and stochastic training, so differences such as 0.207 vs. 0.215 in Table III or 0.215 vs. 0.206 in Table IV may not be statistically significant. Please report the mean and standard deviation (or confidence intervals) across at least three independent runs, and indicate whether the observed improvements over the best baseline are statistically significant.","section":"Tables III and IV"},{"comment":"The latency and energy claims (up to 120x and 60x) are based on NeuroSim simulations for NVCiM crossbars compared with a Jetson Orin CPU. It is unclear whether the simulation and the CPU measurement account for the same end-to-end retrieval workload, including the overhead of encoding/decoding the query, data movement, and the LLM inference itself. Please clarify the scope of the measurement and report the speedup/energy ratio for the complete search pipeline rather than only the crossbar-array operation; otherwise the headline speedup numbers may overstate the benefit.","section":"§IV-B2, Fig. 5"}],"minor_comments":[{"comment":"The equation for WMSDP is malformed: the denominator 'P i ∈Lwi' is written inline and the fraction is ambiguous. Please rewrite it as a properly typeset fraction and clarify the indexing over the scale set L.","section":"§III-D2, Eq. (5)"},{"comment":"The definition 'Pool_i(x) = 1/i \\sum_{j=1}^{i} x_k : k + i' is hard to parse. Please provide a precise definition with explicit index ranges, e.g., average pooling over non-overlapping windows of size i.","section":"§III-D2, pooling definition"},{"comment":"In Section IV-A2 the text refers to 'TABE II' (a typo for Table II) and states that the device variation σ is 0.1, but Table II gives per-level variation values that differ across devices. Clarify how the global σ=0.1 relates to the per-device, per-level values used in the experiments.","section":"§IV-A2 and Table II"},{"comment":"The framework overview in Fig. 3 is dense and the three components (RS, NT, SSA) are not clearly separated in the caption or the figure labels. Adding named bounding boxes or distinct colors with a legend would improve readability.","section":"Fig. 3"},{"comment":"The phrase 'NVCiM' is introduced as 'non-volatile computing-in-memory' but later used as a noun ('NVCiM devices', 'NVCiM-PT'). Consider defining the abbreviation once at first use and using it consistently throughout.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical breadth is a strength, but the core SSA retrieval claim is under-supported. I would be willing to accept after the authors add random/oracle retrieval controls, specify all hyperparameters, report variance, and clarify the latency/energy measurement scope. The heavy self-citation pattern is not disqualifying but worth noting to the editor; it does not affect my technical assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take on arXiv:2411.08244. The paper is a decent system contribution: it's the first to put prompt-tuned virtual tokens for edge LLMs into NVCiM, with a store-and-search pipeline. The empirical sweep is broad—three LLMs, five LaMP datasets, five NVM device models, five baselines—and the reported gains (up to ~36.7% on some tasks) are consistent across the tables. That gives me reasonable confidence that the framework does something real, even if the components are known: k-means selection, noise-injection training, and in-memory matrix multiplication.\n\nWhat I'd want fixed before I trust the details: the SSA retrieval is the load-bearing novelty, but WMSDP (Eq. 5) is never validated against a random-OVT lower bound or an oracle upper bound. The paper compares SSA against MIPS and shows it wins, but MIPS is a weak baseline given the acknowledged incompatibility between word embeddings and virtual tokens. Without knowing whether SSA retrieves anything close to the correct OVT, the title contribution is not actually confirmed. The stress-test note is right about that. Also, Eq. 3 has argmin where it almost certainly means argmax—selecting the least similar sample to the centroid can't be intended. Minor but needs fixing.\n\nThe other soft spots are the usual ones: key hyperparameters in Eqs. 2 and 4 are not given (nmin, nmax, b0, s, f1–f4, sigma), there are no error bars or repeated-run stats, and neither code nor data is released. For a paper whose gains are empirical measurements, that's a reproducibility problem, not a fatal one.\n\nThe central claim—that NVCiM can help with OVT storage and retrieval for edge prompt tuning—is plausible and mostly supported. The paper is not a fundamental breakthrough; it's an engineering contribution with a plausible hardware angle. I'd send it to review, but with a clear request for the missing controls and hyperparameters. A reader working on edge LLM personalization or CiM-based accelerators would get value from it; I'd probably cite it if I were in that space. Worth a serious referee.","headline":"A plausible but under-validated system paper: the NVCiM-assisted prompt tuning idea is real and the empirical sweep is broad, but SSA retrieval needs oracle/random controls before the headline claim is proven.","tokens_in":16189,"tokens_out":1800,"would_cite":true,"duration_ms":16936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Edge LLMs can personalize to each user by storing per-domain prompt vectors in non-volatile computing-in-memory arrays and retrieving the right prompt with an in-memory similarity search, a scheme the paper reports lifts accuracy by up to…","keywords":["prompt tuning","edge LLM","non-volatile computing-in-memory","virtual tokens","domain shift","device variation","personalized LLM","in-memory search"],"falsifier":"Take one held-out user, train an OVT on that user's own data, and compare the LLM's accuracy with that oracle prompt against the accuracy with the prompt that SSA retrieves for the same user; if the SSA-retrieved prompt's accuracy is no better than a randomly chosen prompt's, the retrieval score is not measuring prompt quality.","tokens_in":15117,"feed_emoji":"⚡","tokens_out":10700,"duration_ms":93415,"temperature":0.7,"pith_summary":"This paper argues that the domain-shift problem that cripples prompt tuning on edge LLMs can be solved by storing many small, specialized prompt vectors—one per user-data cluster—inside non-volatile computing-in-memory (NVCiM) arrays and retrieving the right one for each new input with an in-memory similarity search. The proposed framework, NVCiM-PT, contributes three pieces: a k-means-based representative selection that decides which user samples deserve their own prompt, a noise-aware training scheme that makes prompts robust to the conductance variations of real memory devices, and a scaled-search algorithm that pools user inputs and stored prompts at multiple granularities and computes a weighted multi-scale dot product to pick the best prompt. The paper reports that this raises accuracy by up to 36.7 percent over existing noise-mitigation and retrieval baselines across five datasets, three edge-scale LLMs, and five memory-device models, while cutting retrieval latency by up to 120x and energy by up to 60x compared with a typical edge CPU. If true, this would make per-user personalization of on-device LLMs practical under severe memory and power budgets.","feed_headline":"Stored prompts boost edge LLMs by up to 36.7 percent","feed_subtitle":"Each user's custom prompt lives in memory chips, so retrieval skips data movement and cuts latency 120x, energy 60x.","key_machinery":"The load-bearing object is the optimal set of virtual tokens (OVT), a small learned prompt vector specialized to a particular user-data domain rather than shared across all inputs. The mechanism that carries the argument is the store-and-scaled-search algorithm (SSA) on NVCiM: each OVT is encoded via a pretrained autoencoder, stored in a crossbar array, and retrieved by evaluating Eq. 5, the weighted multi-scale dot product (WMSDP), which average-pools both the user input embedding and the stored prompt at scales 1, 2, and 4, weights the three dot products, and selects the OVT with the highest score. The same crossbar array that stores the prompts also performs the matrix-matrix multiplications for the search, which is what turns a memory-size bottleneck into a latency and energy win. Around this sit the representative-selection step (k-means clustering plus cosine-similarity sample choice) and noise-aware training (Gaussian noise injection scaled by prompt magnitude) that make the OVTs worth storing in the first place.","core_discovery":"The paper's central claim is that the optimal set of virtual tokens (OVT) idea—previously restricted to cloud settings—can be brought to edge LLMs if the per-sample prompts are stored and retrieved inside NVCiM. Rather than training one 'one4all' prompt on all users, NVCiM-PT trains separate prompts on representative samples from each detected data domain, stores them encoded as low-precision vectors in non-volatile crossbar arrays, and at inference time retrieves the most appropriate prompt by running a weighted multi-scale dot product between pooled user-input embeddings and pooled stored prompts as in-situ matrix-matrix multiplications. Noise-aware training injects device-variation noise into the prompts during optimization so the stored prompts tolerate the conductance fluctuations of RRAM and FeFET devices. The upshot is that prompt tuning's accuracy advantage and its low trainable-parameter count are retained, while the storage and retrieval costs that once made per-user prompts impractical are absorbed by the memory hardware itself.","pith_inferences":["A natural extension: if the multi-scale pooling truly bridges word-embedding and virtual-token spaces, the same retrieval scheme could choose among other parameter-efficient fine-tuning artifacts such as LoRA adapters stored in memory.","The buffer-size sweep's intermediate optimum suggests an adaptive policy—grow the prompt library only while marginal retrieval quality keeps improving—but the paper does not derive such a policy.","The reported 120x and 60x gains cover the retrieval search itself; an end-to-end measurement that includes writing new prompts and updating the autoencoder would show how much of the gain survives a full personalization cycle."],"forward_implications":["Per-user personalization no longer requires retraining or large adapter storage: the edge LLM stays frozen and only the prompt changes.","The prompt library for a device is bounded by NVM capacity, not DRAM, because every stored prompt is searched by the same in-memory matrix multiplication.","The reported 36.7 percent accuracy gain indicates that a large share of prompt tuning's domain-shift loss is a retrieval problem rather than a training-capacity ceiling.","All prompts in the library are searched in parallel in the crossbar, so retrieval latency grows only weakly (if at all) as the number of stored user domains increases."],"supporting_citations":[{"why":"Prefix-tuning: establishes the continuous-prompt paradigm the framework builds on.","marker":"[11]"},{"why":"Supplies the premise that each data sample can have an optimal set of virtual tokens.","marker":"[13]"},{"why":"The prompt-tuning procedure used to train virtual tokens for representative samples.","marker":"[14]"},{"why":"Selective write-verify: a noise-mitigation baseline and a device-variation mitigation method.","marker":"[20]"},{"why":"Gives the Gaussian device-variation model used in noise-aware training.","marker":"[21]"},{"why":"Deep Compression: basis of the autoencoder that encodes prompts for low-precision NVM storage.","marker":"[24]"},{"why":"Pooling mechanisms for word embeddings, the basis of the multi-scale pooling in SSA.","marker":"[28]"},{"why":"MIPS: the maximum inner product search baseline that SSA is compared against.","marker":"[35]"},{"why":"The compute-in-memory simulator used to estimate retrieval latency and energy.","marker":"[36]"}],"fun_headline_variants":["Edge LLMs get 120x faster prompt tuning via in-memory storage","Per-user prompts stored in RRAM cut edge LLM latency 120x","In-memory prompt retrieval boosts edge LLM accuracy 36.7%","Stored per-user prompts in crossbar arrays: edge LLM speedup 120x","Noise-aware in-memory prompts: edge LLM tuning with 60x less energy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that pooling the user input and a stored prompt at a few scales and taking their weighted dot product reliably identifies the prompt that will actually improve the LLM, even though word embeddings and virtual tokens live in different representational spaces; if pooling does not bridge that gap, the retrieved prompt is no better than a random one.","fun_headline_variants_meta":{"raw":{"variants":["Edge LLMs get 120x faster prompt tuning via in-memory storage","Per-user prompts stored in RRAM cut edge LLM latency 120x","In-memory prompt retrieval boosts edge LLM accuracy 36.7%","Stored per-user prompts in crossbar arrays: edge LLM speedup 120x","Noise-aware in-memory prompts: edge LLM tuning with 60x less energy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3603,"prompt_tokens":989,"completion_tokens":2614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2507}},"tokens_in":605,"tokens_out":2614,"duration_ms":16709,"temperature":1.0,"reasoning_tokens":2507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:47:41.738271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one held-out user, train an OVT on that user's own data, and compare the LLM's accuracy with that oracle prompt against the accuracy with the prompt that SSA retrieves for the same user; if the SSA-retrieved prompt's accuracy is no better than a randomly chosen prompt's, the retrieval score is not measuring prompt quality.","supporting_citations":[{"cited_title":"Swim: Selective write-verify for computing-in-memory neural accelerators","cited_arxiv_id":null,"evidence_quote":"Selective write-verify: a noise-mitigation baseline and a device-variation mitigation method."},{"cited_title":"Uncertainty modeling of emerging device based computing- in-memory neural accelerators with application to neural architecture search","cited_arxiv_id":null,"evidence_quote":"Gives the Gaussian device-variation model used in noise-aware training."},{"cited_title":"Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms","cited_arxiv_id":"1805.09843","evidence_quote":"Pooling mechanisms for word embeddings, the basis of the multi-scale pooling in SSA."},{"cited_title":"Learning binary codes for maximum inner product search","cited_arxiv_id":null,"evidence_quote":"MIPS: the maximum inner product search baseline that SSA is compared against."},{"cited_title":"Dnn+ neurosim v2","cited_arxiv_id":null,"evidence_quote":"The compute-in-memory simulator used to estimate retrieval latency and energy."}],"review_version":1}