{"id":"d7234acf-a26c-4776-a197-cee3b21588e0","arxiv_id":"2607.23815","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Query-aware LLM serving with adaptive memory-aware pipelining reuses cross-operator KV-cache state and speeds semantic queries by up to 4.57×.","lead":"Kalypso makes LLM serving query-aware so semantic operators can pipeline and reuse KV-cache prefixes instead of recomputing them. That can cut end-to-end semantic-query latency by up to about 4.6× versus request-centric serving.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The 4.57× headline is baseline-sensitive: the paper's own blocking ablation attributes only 1.2–1.83× to pipelining+prefix reuse, and on ContractNLI the two request-centric baselines differ by ~2× on identical LLM work, indicating baseline-internal overhead in the comparison target.","rationale":"The paper is transparent — it publishes the blocking ablation (Fig 9) that reveals the mechanism-isolated gain — so this is a question of claim calibration, not misconduct. The contribution (memory-aware pipelined scheduling; 1.2–1.83× mechanism gain; ~2.2× vs the strongest baseline; robustness under memory pressure) survives my concern intact, which is why I do not recommend lowering the verdict. But the abstract's \"up to 4.57×\" is the number that will be cited, and on the workload that produces it, two request-centric baselines running near-identical LLM call counts on the same engine differ by 2×, which is direct internal evidence that baseline implementation quality — not query-aware serving — accounts for a large share of that figure. A secondary configuration ambiguity reinforces the attribution worry: the 8B proxy is said to run on a separate vLLM instance on one GPU while the 70B model occupies all 4 GPUs at 90% memory utilization, and it is unclear where the proxy physically fits, so baseline configurations may be constrained in ways that are not reported. The reader conditioned on reproducibility artifacts and tighter experimental controls; my read agrees and sharpens which control matters most — a framework-free request-centric baseline that isolates the serving layer. I agree only partially with the reader's weakest assumption: prefix retention is a genuine deployment condition, but the paper covers it with ablations, whereas headline-number attribution is both less secure and more consequential for the strongest claim. Verdict remains CONDITIONAL; the proposed test would settle whether the conditions should specifically include restating the headline as mechanism-attributable speedup.","tokens_in":21912,"tokens_out":7521,"duration_ms":207029,"concrete_test":"On ContractNLI (and MEDEC), implement a minimal request-centric baseline: a hand-written script that issues the same prompts operator-at-a-time with materialization directly to vLLM, no SQPS framework, sweeping max concurrency (64/256/unbounded); separately rerun stock Lotus/Palimpzest with max batch size raised from 64 and instrument where wall-clock time goes (inside vLLM vs framework code). If the lean baseline lands near Kalypso-blocking's 1940s, the 4.57× mostly reflects baseline overhead and the serving-layer headline should be restated as ~1.8–2.2×; if it also sits near 4854s, the headline stands as mechanism-plus-integration effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on end-to-end comparisons against full SQPS frameworks (Lotus, Palimpzest), but the paper's own numbers let us decompose the headline, and the decomposition does not fully support its magnitude. On ContractNLI: Lotus 4854s, Palimpzest 2373s, Kalypso 1062s (Fig 7, §7.3), and Kalypso in blocking mode 1940s (Fig 9, §7.4). The mechanism-isolated gain — pipelined vs blocking Kalypso, same code, same engine — is 1.83×. The residual gap between Kalypso-blocking and Lotus (~2.5×) cannot be caused by pipelining or cross-operator KV reuse, because blocking mode has neither; yet Table 2 shows near-identical oracle call counts (17,445 vs 17,342), the same engine, plan, and token bound, and Lotus even gets a proxy cascade on the first filter. Decisively, Palimpzest — also request-centric, operator-at-a-time, oracle-only — is only ~1.22× slower than Kalypso-blocking, so the two request-centric baselines differ by 2× on essentially identical LLM work. That 2× is baseline-internal overhead (framework orchestration, the default max batch size 64, request-issue patterns), not serving-layer phenomena. The \"up to 4.57×\" figure therefore mixes the actual mechanism (~1.8×) with the implementation quality of one baseline on one workload. If a lean request-centric orchestrator recovers most of the residual, the durable serving-layer claim is ~1.2–1.83× mechanism gain and ~2.2× end-to-end vs the strongest baseline — real, but well below the headline. By contrast, the reader's flagged assumption (prefix formatting/retention) is a transparent deployment condition (§4 puts prompt formatting on the SQPS) with direct supporting evidence in the paper (virtual≈explicit pinning, Fig 11; memory sweep, Fig 8), so it is less load-bearing than attribution of the headline number.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper introduces relational LLM serving and Kalypso, a layer between semantic query processing systems and request-centric engines such as vLLM. Query plans are divided into pipelines and stages; stages run as tuple-level tasks, with dependent tasks launched promptly so downstream operators can reuse upstream KV-cache prefixes. An adaptive scheduler assigns and rebalances per-stage memory budgets to avoid starvation and saturation, while token-bound estimation, explicit or virtual pinning, retry handling, and deadlock recovery address uncertain memory demand. Experiments on FEVER, MEDEC, BioDEX, and ContractNLI compare Kalypso with Lotus and Palimpzest over vLLM and Llama-3.3-70B. Kalypso reports speedups up to 4.57× with broadly similar oracle-call counts, supported by ablations for pipelining, adaptive budgeting, pinning, token bounds, and memory pressure.","tokens_in":22430,"tokens_out":3519,"duration_ms":77231,"significance":"Making LLM serving query-aware is a timely and useful systems direction. Kalypso’s stage/task abstraction, adaptive budgets, and virtual pinning could be adopted beneath multiple semantic-query systems, particularly because the approach is intended to complement rather than replace cascades and query optimization. The evaluation has substantial strengths: four distinct workloads, a common vLLM/model configuration, LLM-call counts, repeated runs, blocking and static-budget ablations, virtual-versus-explicit pinning, token-budget sensitivity, and memory-pressure sweeps. If the gains are shown to preserve outputs and are properly attributed to the serving layer, the work would be a significant contribution. At present, the magnitude and mechanism of the largest end-to-end claim remain insufficiently isolated.","major_comments":[{"comment":"§7.3/Fig. 7 versus §7.4/Fig. 9 and Table 2: the 4.57× ContractNLI headline is not fully attributable to relational serving. Lotus takes 4,854s, Palimpzest 2,373.4s, Kalypso 1,062.4s, and blocking Kalypso 1,940s. Thus the pipelined/blocking ablation shows 1.83×, while Kalypso is 2.23× faster than the stronger request-centric baseline. Lotus and Palimpzest differ by about 2× despite similar oracle-call counts. Please add a lean request-centric control using the same UDFs, prompts, engine, and batching, or otherwise decompose orchestration, batching, and serving effects; report the strongest-baseline speedup alongside the headline.","section":"§7.3, Figure 7; §7.4, Figure 9; Table 2"},{"comment":"The central mechanism assumes exact shared prompt prefixes and timely retention (§2.1, §4, §5.2, §6). However, Fig. 9’s blocking-versus-pipelined comparison confounds operator overlap with KV reuse, and the evaluation does not report cache-hit rates, reusable-prefix lengths, recomputed prefill tokens, premature evictions, or retries caused by eviction. Since virtual pinning depends on LRU behavior and prompt formatting is delegated to SQPS UDFs, please measure these quantities directly and include a sensitivity test in which prompt layouts reduce prefix sharing or eviction timing violates the assumed retention window.","section":"§2.1, §4, §5.2, §6, §7.4/Figure 9"},{"comment":"The abstract and §1 claim that query semantics and output accuracy are preserved, but §7 reports latency and call counts rather than output equivalence or task quality. This matters because prompts may be reformatted for prefix sharing, predicate calls are capped at 8 tokens, and cascade use differs: FEVER has 183.3 Lotus oracle fallbacks versus 243.6 for Kalypso, while cascading is disabled for Kalypso on MEDEC and ContractNLI. Please compare final outputs under deterministic settings and report workload accuracy/F1, with matched oracle-only and matched-cascade configurations.","section":"Abstract; §1; §7.1–§7.3; Table 2"},{"comment":"§7.1 states that all systems use the same operator implementations and manually optimized plans, but it is unclear whether Palimpzest’s optimizer was disabled, how its default batch limit of 64 interacts with vLLM admission, and where the large ContractNLI gap between the two request-centric systems arises. Per-operator timelines and baseline orchestration/batching settings are needed to establish that the end-to-end comparison isolates the serving layer rather than framework overhead.","section":"§7.1; §7.3/Figure 7"}],"minor_comments":[{"comment":"§5.1/Figure 4 contains an apparent unresolved editorial note: the figure is said to place outputs in the queue, while the text and Algorithm 1 place new tasks there. Please make the queue semantics consistent.","section":"§5.1, Figure 4"},{"comment":"Algorithm 1 and §7.1 use α=1 and β=0.5 without a sensitivity study. A small sweep or explanation of why the results are insensitive to these thresholds would strengthen the adaptive-scheduling claims.","section":"§5.3, Algorithm 1; §7.1"},{"comment":"The static 8-token bound for predicate calls should be justified empirically. Please report whether truncation or retries occurred and how often the bound was reached.","section":"§7.1"},{"comment":"The evaluation uses one model, one GPU platform, and left-deep static plans. This is reasonable for a prototype, but the conclusion should state these scope limits more explicitly, especially for operators with variable fanout or non-prefix-compatible prompts.","section":"§7; §9"},{"comment":"Several typographical errors should be corrected, including “dowstream,” “runing,” “Speeups,” “in is a separate pipeline,” and “we the small LLM.”","section":"§4, §5.1, §7.2, §7.4, §8"}],"recommendation":"major_revision","confidential_remarks":"The central unresolved question is attribution: a substantial part of the largest speedup may reflect baseline framework internals rather than the proposed serving layer. If the venue supports artifact review, I recommend requiring the modified vLLM code, workload UDFs, and run scripts, because these are important for assessing that attribution."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core here is not “another semantic query system.” It is a serving-layer problem: admit pipelined operator tasks under KV pressure so dependent operators reuse prefixes before eviction, and adapt stage memory budgets when the pipeline starves or saturates. That framing is new relative to Lotus/Palimpzest (plan/operator side) and vLLM/SGLang (request-centric). The API contract (pipelining / predicate / CP), stage-task model, and starve/saturate rebalancer are concrete and readable.\n\nWhat they do well is the evaluation structure. Blocking vs pipelined Kalypso isolates the mechanism at 1.20–1.83×. Static stage splits lose to adaptive budgeting on both multi-stage workloads. Virtual ≈ explicit pinning is a practical result. Memory-pressure sweeps stay flat while baselines degrade. LLM call counts are roughly matched, so they are not buying latency with fewer oracle calls. Complementary to cascades without accuracy tradeoffs is the right positioning.\n\nThe soft spot is attribution of the headline, not the idea. On ContractNLI, Lotus is ~2× slower than Palimpzest on essentially the same LLM work; Kalypso-blocking is already near Palimpzest. So 4.57× vs Lotus mixes real pipelining/reuse with framework orchestration overhead in one baseline. Durable claim is closer to ~1.2–1.8× from the mechanism and ~2× end-to-end vs the stronger request-centric baseline. Prefix-sharing is a deployment condition they put on the SQPS UDF—honest, not hidden. Minor: cascade fairness tweaks, non-determinism, α/β and 99th-percentile token knobs, no artifact release.\n\nMath is scheduling logic, not theorem-heavy; citations cover the right serving and SQPS lines. For anyone building AI analytics runtimes or LLM serving for tabular semantic ops, this is worth reading. I would send it to peer review; ask referees to force a lean request-centric orchestrator baseline and lead with the blocking ablation, not 4.57×.","headline":"Real systems contribution on query-aware KV scheduling; the 4.57× headline oversells a cleaner ~1.2–1.8× mechanism gain plus baseline noise.","tokens_in":23288,"tokens_out":534,"would_cite":true,"duration_ms":17060,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Making LLM serving aware of semantic query plans lets intermediate KV-cache state be reused across operators, cutting query time without changing results.","keywords":["relational LLM serving","semantic operators","KV-cache reuse","pipelined query execution","memory-aware scheduling","prefix caching","semantic query processing"],"falsifier":"Run the same multi-operator plans with prompts rewritten so shared system-plus-tuple prefixes disappear, or force immediate eviction of every completed request’s cache: if Kalypso’s speedups over operator-at-a-time baselines vanish while call counts stay similar, the central claim fails.","tokens_in":22939,"feed_emoji":"⚡","tokens_out":888,"duration_ms":20089,"temperature":0.7,"pith_summary":"Semantic queries treat large language models as relational operators over unstructured text—filters, maps, joins, and so on—but today’s stacks hand each operator call to a request-centric serving engine that never sees the plan. Prefill work on shared tuple prefixes is therefore often thrown away and recomputed. This paper argues that a relational serving layer between the query system and the engine can pipeline operators so a tuple’s cached attention state is consumed by the next operator before eviction, while still preserving exact query semantics and accuracy. Kalypso implements that idea with an API for operator plans and an adaptive scheduler that assigns GPU memory budgets per pipeline stage, rebalancing online to avoid both starving downstream work and saturating the cache with upstream tasks. Across fact verification, medical notes, biomedical matching, and contract entailment, the system reports end-to-end speedups up to about 4.6× over operator-at-a-time baselines that issue a similar number of model calls.","feed_headline":"Query-aware LLM serving cuts semantic query time up to 4.6×","feed_subtitle":"Pipelining operators reuses tuple KV-cache state before eviction, without changing answers.","key_machinery":"Adaptive memory-aware scheduling of pipeline stages and tasks: each stage gets a dynamic KV-cache budget; the scheduler admits tasks only when budget remains, tracks waiting-queue pressure to detect starvation versus saturation, and rebalances memory between upstream and downstream stages so reusable prefixes are consumed before eviction.","core_discovery":"Relational LLM serving—query-plan-aware admission and memory control over pipelined semantic operators—can substantially reduce end-to-end query completion time by raising cross-operator KV-cache hit rates, without changing query semantics or output accuracy. Kalypso shows this with speedups up to 4.57× versus request-centric, materializing baselines on diverse workloads.","pith_inferences":["If prompt templates stay prefix-aligned by convention, relational serving could become a default substrate for multi-step agent and RAG pipelines, not only declarative SQPS plans.","The same starvation/saturation tradeoff likely appears whenever any multi-stage LLM workflow shares long context across steps under a finite paged KV pool.","Virtual pinning nearly matching explicit pinning suggests many engines could gain most of the benefit without new pinning APIs, if higher-level schedulers control admission tightly enough."],"forward_implications":["Semantic query systems can treat serving-layer pipelining as a free efficiency win complementary to cascades, proxy models, and plan rewrites.","GPU memory for KV cache becomes a first-class query-execution resource that must be budgeted across stages, not only across independent requests.","Operator APIs need to declare pipelining, predicate pruning, and join structure so the serving layer can form stages and dependent tasks.","Under tight KV-cache budgets, plan-aware admission control can avoid the sharp latency cliffs that materializing intermediate tables causes when tables exceed cache capacity."],"fun_headline_variants":["Kalypso: relational LLM serving reuses KV-cache across operators","Query-plan-aware serving pipelines semantic ops, cuts time to 4.57×","Pipelined operators reuse tuple KV state before eviction","Memory-aware scheduler lifts KV hits without changing answers","Relational serving beats request-centric baselines on semantic queries"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Consecutive operators must share a long stable prompt prefix for each tuple, and that prefix must stay in GPU memory until the next operator runs—either by pinning or by careful launch timing under ordinary cache eviction.","fun_headline_variants_meta":{"raw":{"variants":["Kalypso: relational LLM serving reuses KV-cache across operators","Query-plan-aware serving pipelines semantic ops, cuts time to 4.57×","Pipelined operators reuse tuple KV state before eviction","Memory-aware scheduler lifts KV hits without changing answers","Relational serving beats request-centric baselines on semantic queries"]},"model":"grok-4.5","effort":"low","cost_usd":0.001874,"raw_usage":{"total_tokens":862,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":18744000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":72,"duration_ms":2228,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:34:44.587206+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same multi-operator plans with prompts rewritten so shared system-plus-tuple prefixes disappear, or force immediate eviction of every completed request’s cache: if Kalypso’s speedups over operator-at-a-time baselines vanish while call counts stay similar, the central claim fails.","supporting_citations":[],"review_version":1}