{"id":"543d35b3-2f57-49dd-8d53-effb12037ceb","arxiv_id":"2508.18736","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.","lead":"SISO is a semantic caching system for LLM serving that stores cluster centroids instead of individual queries, and lowers its similarity threshold during heavy load to hit the cache more often. It reports up to 1.71x higher hit ratios and steadier SLO attainment than GPTCache and vLLM across several query datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hit-ratio gains may be inflated by unvalidated semantic hits: at theta_ret=0.60, non-duplicate pairs (median cosine sim 0.62) will often match, so the 1.71x hit-ratio claim is not yet evidence of useful hits.","rationale":"The reader's weakest assumption (Observation #4: similar inputs yield similar outputs) is real, but the more load-bearing issue is that the paper's headline metric itself does not test it. SISO's dynamic threshold can drop to 0.60, below the median cosine similarity of non-duplicate pairs (0.62) reported in the paper's own Fig. 2. At that operating point, the hit condition is satisfied by many queries whose outputs would not be acceptable, so raw hit ratio is not a valid proxy for useful cache hits. The separate F1/win-rate evaluation is encouraging but is aggregated across RPS and datasets and does not condition on the thresholds actually used in the headline runs. This does not make the paper's central contribution fraudulent or vacuous; centroid caching and locality-aware replacement are plausible and the authors disclose limitations in Sec. 6. However, the 1.71x hit-ratio and 'consistently stronger SLO attainment' claims should be re-expressed as quality-adjusted goodput, or accompanied by a validation of hit correctness, before the numbers are taken at face value. Since the reader already issued CONDITIONAL, this concern reinforces that verdict rather than changing it; code release and the gated-hit check are the natural conditions to lift it.","tokens_in":25787,"tokens_out":7804,"duration_ms":82839,"concrete_test":"Take the actual high-RPS runs from Sec. 5.2 (e.g., Quora/MSMARCO) where SISO selects low theta_ret values (about 0.60-0.70), sample the cache-hit requests, and have GPT-4 or human annotators label whether the returned cached output correctly/sufficiently answers the original query. Compute quality-gated hit ratios (correct hits / total queries) for SISO and GPTCache. If SISO's gated hit-ratio advantage is materially below the raw 1.54-1.71x, the headline claim is an artifact of an unvalidated similarity threshold and should be restated as a goodput/latency tradeoff, not a hit-ratio improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparisons use raw cache hit ratios, where a hit is any query whose embedding is within theta_ret of a cached vector/centroid (Sec. 3.3, Fig. 7). SISO's dynamic controller is allowed to lower theta_ret to 0.60 under heavy load, yet Sec. 3.1/Fig. 2 shows non-duplicate query pairs have median cosine similarity 0.62 across QQP/MRPC/MQP. Thus at theta_ret=0.60 many semantically unrelated queries will be counted as hits, and the stored answer may not answer the query. The paper's accuracy checks (Alpaca-eval win rate, F1 vs GPT-4) are computed separately from the hit-ratio experiments and are averaged over all RPS/datasets; they do not gate or validate the hits counted in the headline 1.54-1.71x numbers. The reported win-rate drop (~8 points at high RPS) already signals non-trivial quality loss. Because the dynamic-threshold mechanism is exactly what produces the SLO and hit-ratio gains, the central claim depends on a hit definition that has not been validated as corresponding to correct or acceptable outputs. This is a correctness risk, not merely the Sec. 6 scope limitation to multi-turn/coding tasks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SISO, a semantic caching layer for LLM serving systems. SISO clusters historical queries into centroids and caches one representative output per centroid, replaces centroids based on semantic locality rather than recency/frequency, and dynamically adjusts the retrieval similarity threshold theta_t to balance response quality and SLO attainment under varying load. The evaluation compares SISO with vLLM and GPTCache on MSMARCO, NQ, Quora, Reddit, and ShareGPT using LLaMa-3.1-8B and 70B, reporting SLO attainment, E2E latency, cache hit ratios, and response quality measured by Alpaca-eval win rates and F1 against GPT-4. The paper claims up to 1.71x higher hit ratios, stronger SLO attainment, and only a marginal accuracy drop, while acknowledging that the approach is limited for multi-turn and coding/debugging queries.","tokens_in":26058,"tokens_out":7251,"duration_ms":84878,"significance":"Semantic caching is a promising orthogonal optimization for LLM serving, and SISO makes a useful conceptual step by caching centroids instead of individual query vectors and by making the similarity threshold workload-aware. The authors honestly report poor performance on ShareGPT and complex query categories, include two model sizes, and provide an ablation that isolates dynamic threshold adjustment. However, the central quantitative claims are not yet established. The hit-ratio gains are measured at thresholds where, by the paper's own similarity statistics, many non-duplicate query pairs would be counted as hits, and the quality metrics are aggregated separately from the hit-ratio experiments. If the authors can directly validate that low-threshold hits return acceptable outputs, the contribution is solid; with the current evidence the headline hit-ratio and SLO-improvement figures are not trustworthy.","major_comments":[{"comment":"The central hit-ratio claim is not validated as semantically useful. In Sec. 3.1/Fig. 2, non-duplicate query pairs have a median cosine similarity of 0.62, while Sec. 3.3/Fig. 7 explicitly considers theta_t = 0.60 and the controller in Sec. 4.3 lowers theta_t under heavy load. At that threshold a large fraction of non-duplicate pairs are within the match radius, so a measured 'hit' does not imply that the cached answer addresses the query. The headline 1.71x/1.54x hit ratios (abstract, Sec. 3.1) therefore conflate true and false hits. The quality checks in Sec. 5.2.7 (Figs. 14-15) are aggregate over all RPS and over all hits and misses, so they do not gate the hits counted in Figs. 9-13. The Sec. 6 limitation about multi-turn and coding tasks does not address this issue, since the false-hit problem also affects single-turn information-seeking queries at low theta_t. Please report hit rat","section":"Sec. 3.3, Fig. 7; Sec. 5.2.1, Fig. 18"},{"comment":"The choice of theta_c = 0.86 and the 'Impact on accuracy' analysis are circular. The threshold is selected from the duplicate/non-duplicate cosine-similarity distributions in Fig. 2, and the accuracy of centroid caching is then measured by the cosine similarity between the cached answer and the original answer (Fig. 3(a)). This uses the same type of metric as the design decision, so it cannot independently establish that cached centroids produce correct or acceptable outputs. The paper should validate with an independent quality metric, such as exact/semantic answer equivalence or human/LLM-judged correctness, and include a sensitivity analysis over theta_c and theta_t.","section":"Sec. 3.1, Fig. 3(a); Sec. 4.1"},{"comment":"The M/D/1 model in Eq. (2) does not match the actual service-time distribution. With semantic caching, each request's service time is bimodal: near-zero for a cache hit and a full LLM generation for a miss. The derivation E[T] = S(1 - h(theta_t)) uses only the mean service time, but the M/D/1 waiting-time formula assumes deterministic service times. Feeding a mean into a deterministic-formula model can materially misestimate the waiting time, especially as h(theta_t) changes. Since dynamic thresholding is one of the three core contributions, the controller should either use an M/G/1 Pollaczek-Khinchine formula with measured service-time variance, or the paper should justify the deterministic assumption with empirical service-time distributions and a comparison of predicted vs. actual latency across RPS.","section":"Sec. 4.3, Eq. (2)"},{"comment":"The evaluation reports single-run point estimates without error bars, confidence intervals, or significance tests, and no code is released. The workloads use randomized Poisson arrivals and dataset splits, so the claim that SISO provides 'consistently' stronger SLO attainment (e.g., Figs. 9-10) needs variance across multiple seeds or runs. The absence of code also prevents checking details that the paper states but does not fully specify, such as the HNSW hierarchy placement, T2H table construction, and the re-clustering trigger. Adding reproducibility material or at least multi-run statistics is necessary to support the quantitative strength of the central claims.","section":"Sec. 5.1, Figs. 9-13"}],"minor_comments":[{"comment":"Several figures have garbled axis labels and legend text after rendering (e.g., Fig. 14 y-axis appears as 't in rate', Fig. 9 legends appear as 'MaMa-3.1-8', etc.). Please regenerate the figures so all labels are legible and unambiguous.","section":"Figures throughout"},{"comment":"The symbols theta_c and theta_t are introduced with different levels of formality. Please define both explicitly at first use and use them consistently in all equations, figures, and table captions.","section":"Sec. 3.1 and Sec. 4.1"},{"comment":"Reference [41] appears to duplicate reference [40] (AlpaServe) with a different proceedings name; please verify and merge or correct. Other reference URLs should be checked for consistency and accessibility.","section":"References"},{"comment":"The introduction states a 'marginal accuracy drop of 6.9%' but the results section does not derive this number directly. Please define the calculation (e.g., relative F1 drop, win-rate drop) and point to the specific figure or table that supports it.","section":"Sec. 5.2.7 and Introduction"},{"comment":"The statement that SISO deliberately lowers hit ratio under light load to improve output quality should be stated in the main text near Fig. 18, since readers may otherwise interpret the hit-ratio drop as an inconsistency with the headline hit-ratio improvement.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"This is a worthwhile systems idea with an honest discussion of limitations, but the current evaluation overclaims. The decisive issue is that the dynamic thresholding that produces the hit-ratio and SLO gains also creates hits at similarity levels where non-duplicate queries are common, and the accuracy measurements are not conditioned on those hits. If the authors can directly show that low-threshold hits are semantically acceptable, the paper could become a solid contribution to the semantic-caching literature; otherwise the headline hit-ratio numbers should be substantially qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real systems paper. What's new in the LLM-serving context is caching cluster centroids instead of individual query vectors, plus semantic-locality-aware replacement and a load-driven similarity threshold. The evaluation covers five datasets, two model sizes, and the authors honestly report where it fails — multi-turn queries, coding and debugging. It deserves a serious referee.\n\nWhat I like: the design is clean and the motivating observations are tested. Centroid caching is a genuine departure from GPTCache's per-vector LRU, and the memory-efficiency argument (1.27–1.14x hit-ratio gains at equal cache size, or ~1.85x less memory for equal hits) is plausible from Fig. 3. The dynamic threshold controller built on an M/D/1 queue plus a T2H table is practical and implemented, not just simulated. The stability analysis of semantic locality gives a principled reason to replace centroids slowly rather than with LRU/LFU. And the limitation section is refreshing: they explicitly say similar inputs do not always yield similar outputs, and they verify the consequence with category-level SLO results.\n\nWhere I get nervous: the headline hit-ratio gains are only as good as the hit definition. At theta_ret = 0.60, which their controller uses under load, the median cosine similarity for non-duplicate pairs is 0.62. So many of those extra hits are not semantically equivalent queries; they are just close in embedding space. The paper's accuracy checks (Alpaca-eval win rate, F1) are averaged over thresholds and request rates, so they don't tell us how many low-threshold hits are actually useful. The reported win-rate drop (~8 points at high RPS) signals real quality loss. That doesn't kill the paper, but it means \"1.71x higher hit ratio\" is not yet evidence of \"1.71x more useful answers.\" Also, the abstract says \"up to 1.71x\" while the conclusion says \"1.71x on average\"; those are different claims and the paper should pin one down.\n\nNo code, no error bars. For a systems paper whose evidence is measurements, that is a real reproducibility problem. The design thresholds are tuned on the same embedding statistics used for accuracy evaluation, so the circularity is mild but worth acknowledging.\n\nBottom line: if you work on LLM serving or semantic caching, this is a useful contribution and the core design argument holds up. But treat the headline numbers as conditional until code and quality-gated hit analysis are available. I'd send it to peer review and ask for a major revision: release code, add error bars, and break down hit quality by threshold.","headline":"SISO is a solid, well-evaluated semantic-caching design for single-turn LLM workloads, but the headline hit-ratio and accuracy claims need tighter validation before I'd trust them.","tokens_in":26640,"tokens_out":4500,"would_cite":true,"duration_ms":51383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SISO claims that caching one answer per cluster of similar queries, evicting by long-term semantic popularity, and loosening the similarity threshold under overload lifts LLM cache hit ratios up to 1.71x over GPTCache.","keywords":["semantic caching","LLM serving","centroid-based caching","cache replacement","semantic locality","dynamic threshold","SLO attainment","embedding similarity"],"falsifier":"A controlled replay would settle it: take a corpus of multi-turn or code-editing queries where near-identical prompts legally have different answers (for example, fix-this-error prompts with different stack traces), run SISO with the threshold tuned to meet a given SLO, and measure how often the cached centroid answer is wrong, say with F1 below a fixed bar against a reference completion. If the wrong-answer rate exceeds what a deployment tolerates at the threshold that delivers the advertised hit-ratio gain, the core observation fails; the paper's own category split in §6, where coding and de","tokens_in":25640,"feed_emoji":"⚡","tokens_out":7810,"duration_ms":82389,"temperature":0.7,"pith_summary":"SISO is a semantic caching layer for LLM serving that argues traditional caching instincts — store each query separately, evict by recency or frequency, always serve a hit — are wrong for LLM workloads. Instead it stores one centroid vector per cluster of semantically similar questions, keeps the centroids that represent the most future queries, and dynamically relaxes the similarity threshold when the serving system is overloaded so more requests are answered from cache instead of hitting the model. The payoff claimed is concrete: up to 1.71x higher cache hit ratios than GPTCache, better SLO attainment under heavy and bursty load, and roughly 1.85x less memory to reach the same hit ratio. The authors are explicit that the benefit is scoped to single-turn, meaning-based queries (information and advice seeking), and that the underlying similar-input-yields-similar-output assumption breaks on multi-turn queries and on coding and debugging tasks.","feed_headline":"Caching by meaning lifts LLM hit ratios 1.71x","feed_subtitle":"Storing one answer per cluster of similar questions keeps latency targets under heavy load, with a small accuracy cost.","key_machinery":"The central object is the centroid of a query cluster: a single embedding vector — carrying a stored answer, a cluster_size, and an access_count — that stands in for many semantically similar queries. Around it two further mechanisms do their work: replacement is driven by cluster_size interpreted as semantic locality, which the paper measures as stable over weeks (96.1% of centroids change rank by no more than 10%), so re-clustering is triggered only when about 10% new queries accumulate; and SLO-aware serving uses a threshold-to-hit-ratio (T2H) table plus an M/D/1 queueing estimate of expected waiting time to pick the loosest similarity threshold that still meets the target SLO. The load-b","core_discovery":"SISO's central claim is that semantic caching should cache answers to questions, not answers to query strings. The paper establishes Observation #1 (high cosine similarity between queries strongly indicates duplicate meaning, with median similarities of 0.82 for duplicate pairs versus 0.62 for non-duplicates on QQP, MRPC, and MQP) and Observation #4 (input similarity correlates with output similarity), then builds a system in which historical query logs are clustered into centroids, each carrying one generated answer, a cluster_size measuring semantic locality, and an access counter. Cache replacement is triggered rarely, on a long window, swapping in centroids with large cluster_size; acces","pith_inferences":["If centroid caching works as claimed, the same collapse of repeated work into one representative applies beyond LLM caches — RAG retrievals, code-assist completions, and database result caches could store cluster representatives instead of individual entries; the measured stability of semantic locality (96% of centroids shift rank by under 10%) is what would make periodic re-clustering cheap enoug","A testable extension: apply the dynamic threshold per query category, since the paper's own category analysis shows the input-output similarity correlation is much stronger for advice/information seeking than for coding/brainstorming; routing only high-locality categories through the semantic cache could keep the SLO gains while removing the concentrated accuracy loss on code tasks.","The 6.9% average accuracy drop hides a concentrated error distribution: on tasks where the core assumption fails, mistakes cluster, so production deployments would need a guard such as detecting multi-turn or context-heavy queries and bypassing the cache — an extension of the repeated-query bypass SISO already implements.","An economic consequence left implicit: lowering the threshold under load turns cached answers into an admission-control mechanism, letting a provider absorb spikes without GPU over-provisioning; with the paper's estimate that clustering roughly a year of queries costs under $300, the break-even point is reachable at moderate query volumes."],"forward_implications":["Semantic caching becomes substantially more memory-efficient: at equal hit ratios on Quora and Reddit, GPTCache needs roughly 1.85x more cache memory, and centroid caching reaches a given hit ratio with far smaller capacity.","SLO attainment survives heavy and bursty load: where vLLM and GPTCache fall below the target SLO as requests-per-second or the coefficient of variation rises, SISO keeps attainment high by converting load into cache hits instead of computation.","The approach is a drop-in complement to the serving stack: it sits alongside vLLM with no architecture-level changes, unlike scheduling, kernel-level, and KV-cache memory optimizations.","Quality is traded only when necessary: under light load the threshold rises so output quality matches uncached serving, while under heavy load the average win rate against vLLM stays near 42%.","The gains concentrate in single-turn, meaning-based queries (information seeking, advice seeking, editing, reasoning), which the paper argues dominate real workloads: 99% of API calls and 67% of chatbot interactions are single-turn."],"supporting_citations":[{"why":"The state-of-the-art semantic cache (per-query embeddings with LRU) that SISO is measured against for hit ratio, latency, and F1 scores.","marker":"[6]"},{"why":"vLLM, the underlying LLM serving system and the no-cache baseline for SLO attainment, latency, and response-quality comparisons.","marker":"[36]"},{"why":"The community-detection clustering routine used to build centroids from query logs, chosen after benchmarking OPTICS, DBSCAN, and HDBSCAN.","marker":"[54]"},{"why":"Sentence-BERT, the basis of the paraphrase-albert-small-v2 embedding model used for all query similarity computations.","marker":"[55]"},{"why":"HNSW approximate nearest-neighbor search used for cache lookup; SISO re-orders its hierarchy levels by semantic locality to speed retrieval.","marker":"[43]"},{"why":"Field characterization data showing single-turn queries dominate API (99%) and chatbot (67%) workloads, bounding the impact of SISO's multi-turn limitation.","marker":"[67]"},{"why":"Reddit dataset, used alongside Quora, QQP, MRPC, and MQP to establish duplicate-pair similarity distributions and long-term centroid popularity stability.","marker":"[24]"},{"why":"Alpaca-eval benchmark used to measure output-quality win rates of SISO against the reference model, vLLM.","marker":"[19]"},{"why":"Source of the practice of scoring SLO-violating requests as dropped (HTTP 429) in quality evaluation, and evidence of the limits of prefix-only caching.","marker":"[53]"}],"fun_headline_variants":["Semantic caching lifts LLM hit ratios 1.71x","Cache by meaning: 1.71x better LLM hits","SISO: semantic caching for LLM, 1.71x hit ratio gain","Why LLM caches should store meanings, not strings"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole system leans on the premise that semantically similar questions get semantically similar answers; where that premise fails — multi-turn context, coding, debugging — the cached answer can be wrong, so the hit-ratio gain turns directly into an accuracy loss.","fun_headline_variants_meta":{"raw":{"variants":["Semantic caching lifts LLM hit ratios 1.71x","Cache by meaning: 1.71x better LLM hits","SISO: semantic caching for LLM, 1.71x hit ratio gain","Why LLM caches should store meanings, not strings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3186,"prompt_tokens":656,"completion_tokens":2530,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":2462}},"tokens_in":400,"tokens_out":2530,"duration_ms":21374,"temperature":1.0,"reasoning_tokens":2462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:14:36.780642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled replay would settle it: take a corpus of multi-turn or code-editing queries where near-identical prompts legally have different answers (for example, fix-this-error prompts with different stack traces), run SISO with the threshold tuned to meet a given SLO, and measure how often the cached centroid answer is wrong, say with F1 below a fixed bar against a reference completion. If the wrong-answer rate exceeds what a deployment tolerates at the threshold that delivers the advertised hit-ratio gain, the core observation fails; the paper's own category split in §6, where coding and de","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The state-of-the-art semantic cache (per-query embeddings with LRU) that SISO is measured against for hit ratio, latency, and F1 scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The community-detection clustering routine used to build centroids from query logs, chosen after benchmarking OPTICS, DBSCAN, and HDBSCAN."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sentence-BERT, the basis of the paraphrase-albert-small-v2 embedding model used for all query similarity computations."},{"cited_title":"Malkov and D","cited_arxiv_id":null,"evidence_quote":"HNSW approximate nearest-neighbor search used for cache lookup; SISO re-orders its hierarchy levels by semantic locality to speed retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Field characterization data showing single-turn queries dominate API (99%) and chatbot (67%) workloads, bounding the impact of SISO's multi-turn limitation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reddit dataset, used alongside Quora, QQP, MRPC, and MQP to establish duplicate-pair similarity distributions and long-term centroid popularity stability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the practice of scoring SLO-violating requests as dropped (HTTP 429) in quality evaluation, and evidence of the limits of prefix-only caching."}],"review_version":1}