{"id":"7c1d75cf-7424-4bf9-87a5-f85f42ccda1e","arxiv_id":"2505.07833","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An end-to-end RAG serving framework that uses component-level batching, resource allocation, and runtime prioritization to improve throughput and reduce SLO violations.","lead":"This paper describes a serving system for retrieval-augmented generation pipelines that automatically allocates CPUs and GPUs across pipeline stages and prioritizes at-risk requests at runtime. It reports roughly 1.48x higher throughput than a commercial RAG framework and about 24% fewer deadline misses, though the submission abstract claims substantially larger gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (1) defines throughput without the resource-allocation variable, so the MILP may not be maximizing what it claims; the optimizer's role in the reported gains is unsubstantiated.","rationale":"The reader's weakest assumption points at the concurrent-flow and piecewise-linear model; I agree that model fidelity is a risk, but I found a more direct issue in the same section. The optimization variables are defined per node type, yet neither the objective nor the flow-conservation constraint uses a_{i,k} multiplicatively. Unless T_i is defined as total time for the whole batch spread across all replicas (which the text does not state and which conflicts with per-node batch-size constraints), Eq. (1) has the wrong units. This means the claimed \"throughput-maximizing\" allocations in the evaluation are not actually implied by the formulation. The empirical batching/pipelining ablation in Fig. 12b is real evidence that component-level batching helps, and the scheduler latency/auto-scaling sections are plausible; I am not disputing that the system may work. But the central algorithmic contribution needs a corrected derivation and re-evaluation before the 1.48x/24% numbers can be attributed to the optimizer. The abstract/body number and name mismatch and the weak baselines reinforce the need for revision, so I keep the reader's conditional verdict rather than accepting the paper as is.","tokens_in":20912,"tokens_out":11153,"duration_ms":123004,"concrete_test":"Re-derive Eq. (1) from the definitions in §3.3 and test a single component with R=2 identical nodes, T_i=1, b_i=1. The stated objective is 1 query/s for both a_i=1 and a_i=2, while actual throughput is 1 and 2 query/s; if so, the objective is not throughput. Then recompute the CRAG configuration in Table 3 with the corrected objective a_{i,k}·b_{i,k}/T_i and flow constraint Σ_k b_{i,k} ≤ Σ_k Σ_g a_{g,k}·b_{g,k}, and compare the resulting allocation and measured throughput against the paper's reported allocation; if the allocation changes, the reported gains are not evidence for the claimed optimizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is the offline scheduler's throughput-maximizing allocation, but Section 3.3's formulation does not close. In Eq. (1), the objective is max min_i Σ_k b_{i,k}/T_i(b_{i,k},a_{i,k}), with T_i defined as \"the time to process a single query\" and a_{i,k} as the number of resources allocated. If each of a_{i,k} nodes processes a batch of size b_{i,k}, component throughput is a_{i,k}·b_{i,k}/T_i (or a_{i,k}/T_i if T_i is per-query latency), not b_{i,k}/T_i. Constraint (3) likewise conserves Σ_k b_{i,k} without weighting predecessor outputs by a_{g,k}, so adding replicas to a stage does not relax the bottleneck constraint unless T_i is silently made a-dependent. As written, the optimizer can be indifferent among very different allocations, and Table 3's \"optimal\" batch sizes and resource splits are not derived from the stated objective. This is more specific than the concurrent-flow approximation: it is a unit/quantification problem in the core model. The manuscript also needs to reconcile the abstract (Harmonia, 2.04x, 78.4%) with the body (Patchwork, 1.48x, ~24%) and the fact that three of four baselines are author-provided single-node implementations, but the Eq. (1) issue is the most load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper, titled 'Patchwork: A Unified Framework for RAG Serving' in the body but announced as 'Harmonia' in the abstract, proposes an end-to-end serving framework for Retrieval-Augmented Generation pipelines. The system has three main components: a lightweight Python interface that captures user-defined RAG pipelines via PEP 523 frame evaluation; an offline scheduler that formulates component-level resource allocation and batch-size selection as a mixed-integer linear program with piecewise-linear profiling of component runtimes; and an online scheduler that predicts SLO violations and mitigates them through prioritization, admission control, and autoscaling. The evaluation covers four RAG applications (CRAG, MemoRAG, IRCoT, HippoRAG) and reports throughput improvements and SLO-violation reductions relative to a LangGraph baseline and author-provided single-node baselines. The abstract claims a 2.04x throughput improvement and up to 78.4% SLO-violation reduction, while the body reports 1.48x and about 24%.","tokens_in":21152,"tokens_out":3334,"duration_ms":34952,"significance":"If the central claims hold, the paper would be a useful contribution to systems support for compound AI workloads: it addresses a real problem (heterogeneous, unpredictable RAG pipelines), introduces a plausible end-to-end design, and provides an ablation separating batching, pipelining, and resource-allocation effects. The paper's strengths include a concrete implementation, measurements of scheduler and optimizer scalability, a stated one-time profiling cost, and an explicit discussion of the throughput cost of SLO mitigation. However, the significance is currently limited by three issues: the core optimization objective in Section 3.3 does not, as written, quantify the effect of replica count on throughput; the abstract and body report conflicting headline numbers and even conflicting system names; and the SLO metric is defined relative to the system's own low-load latency, making the reported violation reduction difficult to interpret. These are load-bearing for the paper's main performance claims rather than cosmetic concerns.","major_comments":[{"comment":"The SLO definition in Section 4.3 is self-referential: the threshold is set to 2x the average request latency of Patchwork under low load. Violation rates are therefore measured against a target derived from the system being evaluated, not from an application requirement. When comparing Patchwork with a baseline, the same numeric threshold is applied, but the baseline may have a very different low-load latency, so the comparison is not apples-to-apples. The authors should either fix a single SLO value independently of any system's measured latency, or report the low-load latencies of both systems and show that the chosen threshold is meaningful for each.","section":"Section 3.3, Eq. (1)"},{"comment":"The paper's headline results are internally inconsistent. The abstract states that 'Harmonia' achieves throughput gains of 'more than 2.04x' and SLO-violation reductions of 'up to 78.4 percent,' while the body and conclusion report that 'Patchwork' achieves a 1.48x speedup and about 24% SLO-violation reduction. Section 4.2 further reports speedups of 15x, 14x, and 22x for MemoRAG, HippoRAG, and IRCoT relative to the respective author-provided baselines. The central claim of the paper cannot be assessed until the authors state which numbers are the definitive ones, reconcile the naming inconsistency, and make the abstract match the experimental evidence in the body.","section":"Abstract and Section 4.2"},{"comment":"Three of the four baselines (MemoRAG, HippoRAG, IRCoT) are author-provided single-node implementations that lack component-level batching, pipelining, and resource allocation. The large speedups of 14-22x are thus improvements over unoptimized reference implementations, not over commercial serving systems. Only CRAG is compared with LangGraph, and there the reported gain is 1.48x at high load. The paper should make this distinction precise in the abstract and conclusion, and the evaluation should include stronger baselines, such as an optimized pipeline with per-component batching and pipelining but without Patchwork's resource-allocation and SLO mechanisms, so that the incremental contribution of each component is clear. Without this, the claim of being the first end-to-end RAG serving system with the stated benefits is overstated.","section":"Section 4.2 and Table 1"},{"comment":"All throughput and SLO-violation plots lack error bars, confidence intervals, or multiple-run statistics, even though the system involves Poisson arrivals, gRPC networking, and variable LLM latencies. The reported differences, particularly the 1.48x LangGraph comparison and the roughly 24% SLO-violation reduction, are not established to be statistically distinguishable from noise. The authors should report means and variances over repeated runs, or at least state the number of runs and the observed variability.","section":"Section 4.2, Figure 11 and Figure 13"}],"minor_comments":[{"comment":"The system is called 'Harmonia' in the abstract and in the caption of Figure 8, but 'Patchwork' in the body, headings, tables, and conclusion. The manuscript must use a single name consistently.","section":"Throughout"},{"comment":"Constraint (5) uses the symbol r_{i,k}, which was not defined; the resource allocation variable is a_{i,k} elsewhere. The notation should be unified.","section":"Section 3.3, constraints (4)-(5)"},{"comment":"The phrase 'profiling improvement threshold' is introduced as a fixed threshold in the binary-search profiling procedure, but the threshold value is never specified or varied in the sensitivity analysis. The authors should state the value and, ideally, test its sensitivity.","section":"Section 3.3, estimating T_i"},{"comment":"The comparison between the running-average estimator and XGBoost is shown for a single workload (HippoRAG) without error bars or confidence intervals, so the claim that the two are 'largely similar' is not strongly supported.","section":"Section 4.3, Figure 10"},{"comment":"The auto-scaling experiment appears to be a single trace, and the text does not state how many violations were detected or how often autoscaling triggered. Reporting aggregate statistics would strengthen the claim that autoscaling works with zero downtime.","section":"Section 4.4, Figure 15"},{"comment":"The problem is described as a 'max-flow' formulation, but the constraints as written do not model flow on edges, capacity per edge, or routing; they only conserve batch sizes across adjacent components. The term is misleading and should either be justified or replaced with a more standard throughput-maximization description.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early submission with two conflicting system names and conflicting abstract/body numbers; the authors should be asked to reconcile these before the paper can be evaluated fairly. The Eq. (1) issue is the most serious technical concern: as written, the optimizer may not actually maximize the throughput that the experiments attribute to it. If the formulation can be corrected without changing the experimental results, the paper could become a solid systems contribution; if the corrected optimizer changes the recommended configurations and the reported gains, the experimental claims will need substantial rework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I've read the manuscript carefully, and I have a mixed take. The system-level idea is genuinely solid: capture the RAG pipeline's control flow and compute nodes with PEP 523 frame evaluation, then use a MILP to choose component-level batch sizes and resource allocations, and add a runtime controller that prioritizes requests predicted to miss SLO. As far as I can tell, this combination is new. The evaluation includes four realistic RAG apps, and the 1.48x gain over LangGraph is a real result, even if the 15-22x gains over author-provided baselines mostly reflect how unoptimized those references are. The ablation that separates batching, pipelining, and resource allocation is useful and done honestly.\n\nThat said, there's a load-bearing flaw in the optimization model. Eq. (1) maximizes min_i Σ_k b_{i,k}/T_i(b_{i,k}, a_{i,k}) with T_i the per-query processing time. If a_{i,k} is the number of replicas, each processing a batch of b_{i,k}, the component's throughput should include a factor of a_{i,k} in the numerator. As written, the objective is indifferent to adding replicas, unless T_i implicitly depends on a. The same issue appears in the batch-flow constraint (3), which doesn't weight predecessor output by replica count. The 'optimal' allocations in Table 3 are therefore not derivable from the stated formulation. This is fixable, but it's central.\n\nThe abstract/body mismatch is also hard to ignore: the arXiv abstract (Harmonia) claims 2.04x throughput gain and 78.4% SLO violation reduction, while the full text (Patchwork) reports 1.48x and ~24%. These are materially different claims. The paper needs one consistent set of numbers.\n\nMore minor: the SLO is defined as 2x Patchwork's own low-load latency, which is partly circular, and there are no error bars or code/data artifacts. None of these are disqualifying on their own, but they add to the sense that the evaluation needs another pass.\n\nI would still send this to peer review. The core approach and the engineering are worth engaging with. But I'd expect a major revision: fix the formulation, reconcile the claims, and ideally release the code so the optimizer behavior can be checked.","headline":"A useful RAG-serving system with a novel graph-capture idea and real gains, but the optimizer has a unit error and the abstract overclaims results.","tokens_in":21738,"tokens_out":6729,"would_cite":true,"duration_ms":65224,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims the first end-to-end serving system for RAG pipelines, reporting a 1.48x throughput gain over the best commercial alternative and about 24% fewer SLO violations across four applications.","keywords":["retrieval-augmented generation","RAG serving","resource allocation","batching","service-level objectives","distributed inference","throughput optimization","pipeline scheduling"],"falsifier":"Run the four reported RAG applications on the same cluster with the same Poisson arrivals and compare end-to-end throughput and SLO violations against the best commercial engine: if the throughput gain does not reach roughly 1.48x or the SLO reduction falls short of about 24%, the central claim fails. A sharper check is to profile one component whose latency is not piecewise-linear in batch size—such as a network-bound retriever—and verify whether the optimizer's recommended batch size still maximizes measured throughput.","tokens_in":20667,"feed_emoji":"⚙️","tokens_out":8332,"duration_ms":68965,"temperature":0.7,"pith_summary":"This paper claims that serving retrieval-augmented generation (RAG) pipelines can be optimized end to end instead of tuning each component in isolation. It presents a system that lets developers write a RAG pipeline in ordinary Python, then automatically captures the compute graph, profiles how each component's latency scales with batch size, and solves a throughput-maximizing resource-allocation problem to decide batch sizes and CPU/GPU splits per component. At runtime, the system tracks each request's progress and prioritizes or autoscales to keep requests inside their latency deadlines. The reported payoff is a 1.48x throughput gain over the best commercial serving system tested and about 24% fewer SLO violations across four RAG applications, with much larger gains over unoptimized reference implementations. This matters because RAG serving mixes GPU-bound generation with CPU-bound retrieval, so the component that bottlenecks the pipeline can shift with workload, component choice, and configuration.","feed_headline":"RAG serving platform lifts throughput 1.48x over commercial engines","feed_subtitle":"Component-level batching and resource split cut deadline misses by about 24 percent.","key_machinery":"The carrier of the argument is the maximizing-concurrent-flow formulation: each RAG component is a node that can simultaneously send and receive batches, and the scheduler maximizes the minimum per-component throughput subject to resource limits, batch-flow conservation from predecessors, and per-node memory bounds on batch size. Because real component runtimes are not known in closed form, the system profiles each component and fits a piecewise-linear curve $T_i(b_{i,k}, a_{i,k})$, which turns the optimization into a mixed-integer linear program solvable in seconds even for hundreds of nodes. The second mechanism is the runtime controller: a running-average estimator of remaining completion time, paired with out-of-order prioritization, temporary admission pause, and autoscaling, all coordinated by a scheduler whose per-request latency stays near two milliseconds in the reported measurements. Together these two mechanisms let the system adapt as the bottleneck moves from the retriever to the generator when components or workloads change.","core_discovery":"The central claim is that a developer-specified RAG pipeline can be deployed as a distributed inference system in which every component gets its own batch size and resource share, chosen by solving a max-concurrent-flow problem over the compute graph rather than by per-component heuristics. The objective is to maximize the minimum throughput across components, written as $\\max \\min_i \\sum_k b_{i,k}/T_i(b_{i,k}, a_{i,k})$ where $b_{i,k}$ is the batch size for component $i$ on node type $k$, $a_{i,k}$ the resources allocated, and $T_i$ the profiled per-query processing time. Profiling is made scalable by modeling $T_i$ as a piecewise-linear function of batch size, which keeps the optimization a mixed-integer linear program. At runtime a lightweight estimator flags requests likely to miss their latency deadlines, and the scheduler responds by prioritizing those requests, pausing new admissions, and autoscaling the bottleneck component. The paper reports that this combination achieves a 1.48x throughput gain over the best commercial engine tested and up to 24% fewer SLO violations, with component-level batching as the largest single source of gain.","pith_inferences":["The body of the manuscript reports 1.48x throughput gain and ~24% SLO reduction against one commercial engine, while the abstract's larger 2.04x and 78.4% figures do not correspond to the evaluation described in the body; a careful reader should treat the body's numbers as the measured claim.","The same concurrent-flow optimization could plausibly extend to other compound AI pipelines beyond RAG—tool-calling agents, multi-model chains, or database-backed LLM applications—wherever components scale differently with batch size and resources.","The SLO controller's simplicity (a running average rivals a trained tree model) suggests that deadline-miss prediction is not the main bottleneck; richer per-stage features or online learning could tighten admission control further and reduce the measured 16% throughput cost of mitigation.","A direct test of transferability would be to apply the piecewise-linear profiling to components with strongly superlinear or network-bound scaling, where the fitted $T_i$ may deviate enough that the chosen batch sizes cease to be throughput-optimal."],"forward_implications":["A developer can write a RAG pipeline as ordinary single-node Python and deploy it across heterogeneous CPUs and GPUs without writing communication or orchestration code.","Because batch sizes and resource splits are recomputed during autoscaling, the system can re-balance the pipeline when the bottleneck component shifts under load.","Component-level batching delivers the largest share of the throughput gain (about 9.6x in one ablation), with pipelining and resource allocation contributing further multiplicative gains.","Enabling SLO mitigation reduces deadline misses by up to 24% but can cost up to 16% end-to-end throughput, a tradeoff the paper measures explicitly."],"supporting_citations":[{"why":"Defines the retrieval-augmented generation paradigm that the serving system targets.","marker":"[37]"},{"why":"Provides the Python frame-evaluation mechanism used to capture the user's pipeline graph automatically.","marker":"[56]"},{"why":"Supplies the mixed-integer linear program solver used to compute optimal batch sizes and resource allocations.","marker":"[3]"},{"why":"Is the commercial serving engine used as the strongest baseline in the throughput comparisons.","marker":"[35]"},{"why":"Supplies the corrective-RAG application used as one of the four evaluation workloads.","marker":"[79]"},{"why":"Establishes the predictable-latency assumption of prior serving systems that RAG workloads are shown to violate.","marker":"[22]"},{"why":"Supplies a memory-based RAG reference implementation used as a baseline in the ablation and throughput studies.","marker":"[58]"},{"why":"Provides the LLM serving engine whose scaling behavior motivates component-level batching and resource allocation.","marker":"[34]"}],"fun_headline_variants":["Harmonia: 1.48x RAG throughput, 24% fewer SLO misses","RAG serving up 1.48x, SLO violations down 24% with Harmonia","Harmonia end-to-end RAG: 1.48x throughput, 24% fewer deadline misses","RAG pipeline optimization: Harmonia boosts throughput 1.48x, cuts SLO 24%","Outperforms commercial RAG engines: 1.48x throughput, 24% fewer SLO misses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole optimization assumes that each pipeline component can concurrently send and receive batches and that its per-query runtime is accurately captured by piecewise-linear profiling, so if real components behave differently under queueing, network, or straggler effects, the chosen batch sizes and resource splits will not be the throughput-maximizing ones.","fun_headline_variants_meta":{"raw":{"variants":["Harmonia: 1.48x RAG throughput, 24% fewer SLO misses","RAG serving up 1.48x, SLO violations down 24% with Harmonia","Harmonia end-to-end RAG: 1.48x throughput, 24% fewer deadline misses","RAG pipeline optimization: Harmonia boosts throughput 1.48x, cuts SLO 24%","Outperforms commercial RAG engines: 1.48x throughput, 24% fewer SLO misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3727,"prompt_tokens":917,"completion_tokens":2810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2681}},"tokens_in":533,"tokens_out":2810,"duration_ms":17735,"temperature":1.0,"reasoning_tokens":2681,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:34:05.977436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the four reported RAG applications on the same cluster with the same Poisson arrivals and compare end-to-end throughput and SLO violations against the best commercial engine: if the throughput gain does not reach roughly 1.48x or the SLO reduction falls short of about 24%, the central claim fails. A sharper check is to profile one component whose latency is not piecewise-linear in batch size—such as a network-bound retriever—and verify whether the optimizer's recommended batch size still maximizes measured throughput.","supporting_citations":[{"cited_title":"https://peps","cited_arxiv_id":null,"evidence_quote":"Provides the Python frame-evaluation mechanism used to capture the user's pipeline graph automatically."},{"cited_title":"https: //www.langchain.com/langgraph","cited_arxiv_id":null,"evidence_quote":"Is the commercial serving engine used as the strongest baseline in the throughput comparisons."},{"cited_title":"Corrective retrieval augmented generation","cited_arxiv_id":null,"evidence_quote":"Supplies the corrective-RAG application used as one of the four evaluation workloads."},{"cited_title":"Serving {DNNs} like clockwork: Performance predictability from the bottom up","cited_arxiv_id":null,"evidence_quote":"Establishes the predictable-latency assumption of prior serving systems that RAG workloads are shown to violate."},{"cited_title":"Efficient memory management for large language model serving with pagedattention","cited_arxiv_id":null,"evidence_quote":"Provides the LLM serving engine whose scaling behavior motivates component-level batching and resource allocation."}],"review_version":1}