{"id":"129e1629-0631-45aa-b682-b94a05989d6c","arxiv_id":"2608.09185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An agentic system for data warehouse delivery at Tencent reached 87.2% end-to-end success and 73.5% autonomous submission across 18,240 production sessions.","lead":"Tencent built SiriusDeliver, an LLM-based agent that automates data warehouse task delivery, including code generation, configuration, pre-submission checks, failure diagnosis, and skill updates from past sessions. In production tests it cut median delivery time from 228 to 23 minutes and engineer effort from 95 to 11 minutes while keeping final success close to the manual baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A/B test lacks task-level comparability: treatment users may route simpler tasks through SiriusDeliver, confounding both cost savings and the claim that final success is comparable.","rationale":"The reader's weakest assumption targets measurement comparability of engineer effort and delivery time in the A/B test. That is a real and important concern, but an even more fundamental threat is task comparability: the A/B test randomizes users, not tasks, and treatment-group users can decide which requests to route through SiriusDeliver. If treatment tasks are systematically simpler, then both the cost reductions and the 'comparable final success' result are artifacts of workload composition, not system capability. This confound affects every headline number from Section 4.3, including final delivery success parity, not just the effort metric. The paper's stratification by task type and business team is helpful but insufficient because complexity varies within strata and no complexity covariates are reported. I therefore identify task-level comparability, rather than measurement instrumentation alone, as the single most load-bearing concern. The reader's verdict of CONDITIONAL remains appropriate: the authors can address this by providing per-session task complexity and a re-analysis that controls for it. I do not move the verdict because the concern is addressable and the offline results plus the large effect sizes leave room for a genuine, if smaller, improvement. The agreement is partial because the reader focused on measurement, while I stress the selection/confounding channel that also undermines the quality-parity claim.","tokens_in":18540,"tokens_out":8261,"duration_ms":89122,"concrete_test":"Request the authors to release, for each A/B session, task-level complexity covariates such as number of input tables, artifact length, dependency depth, and pre-registered difficulty labels, plus a historical manual delivery time for the same or equivalent request. Then re-estimate the treatment effect using coarsened exact matching or inverse-probability weighting on these covariates. If the adjusted median delivery time and engineer effort are no longer near 23 and 11 minutes for the treatment arm, or if the final-success gap widens materially, the headline reductions are confounded by task selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 randomizes 70 engineers at user level (35 per arm), not delivery tasks, so the 718 treatment tasks and 682 control tasks are not necessarily comparable in difficulty. Because treatment-group engineers interact with a web interface and can choose which requests to submit, they may selectively route simpler or more standardized tasks through SiriusDeliver while handling complex ones manually. The paper reports no task-complexity covariates for the two arms (e.g., number of dependencies, artifact length, table count, historical manual delivery time for equivalent requests). If the task mix differs, the 89.9% delivery-time reduction and 88.4% effort reduction reflect workload composition rather than SiriusDeliver's causal effect, and the final success parity (97.6% vs 98.1%) no longer supports the claim that automation preserves final quality. Stratification by task type and business team mitigates but does not eliminate this confound, because difficulty varies within each stratum. The central claim—'substantially more automated and cheaper without sacrificing final quality'—therefore rests on an A/B test whose internal validity is not established as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SiriusDeliver is an end-to-end agent for automating production data warehouse task delivery at Tencent. The paper proposes three components: a hierarchical delivery agent that orchestrates warehouse skills, an artifact lifecycle control module that verifies and revises artifacts before and after platform execution, and a trace-driven skill evolution mechanism that converts delivery trajectories into bounded skill updates. The evaluation has three parts: an offline benchmark of 200 real production cases across four warehouse task types, a two-month production deployment serving 3,600 monthly active users and 18,240 delivery sessions, and a one-month A/B test with 70 engineers. The paper claims a 14.5-point improvement in offline end-to-end delivery success over the strongest skill-augmented baseline (71.5% to 86.0%), an 87.2% end-to-end success rate and 73.5% autonomous submission rate in deployment, and, from the A/B test, reductions in median delivery time from 228 to 23 minutes, engineer effort from 95 to 11 minutes, and manual intervention from 100% to 21%, while final delivery success remains comparable (97.6% vs. 98.1%).","tokens_in":18714,"tokens_out":4976,"duration_ms":49845,"significance":"If the reported results hold, SiriusDeliver would provide strong evidence that production data warehouse delivery can be largely automated without sacrificing final quality. The offline evaluation is unusually well controlled: within each product pair the skill library, platform tools, execution environment, and refinement budget are fixed; evaluation snapshots are frozen; skill-evolution trajectories are disjoint from evaluation cases; and success is adjudicated by the platform and manual verification. The ablation study and the multi-backbone paired evaluation strengthen the internal validity of the offline claims. The production deployment numbers are large and the A/B test addresses a relevant practical question. However, the internal validity of the A/B test is not established as reported, and this A/B test is the load-bearing evidence for the headline cost-reduction claims. The paper also provides a public companion benchmark, though it is explicitly not used to produce the reported results, which limits external reproducibility. On balance, the contribution is significant if the A/B test concerns can be resolved.","major_comments":[{"comment":"The A/B test is randomized at the user level, not at the task level, and treatment-group engineers choose which requests to route through the SiriusDeliver web interface. The 718 treatment tasks and 682 control tasks are therefore not shown to be comparable in difficulty; no task-complexity covariates (e.g., number of dependencies, table count, artifact length, or historical manual delivery time for equivalent requests) are reported for either arm. Under these conditions, the reported 89.9% reduction in median delivery time and 88.4% reduction in engineer effort, as well as the near-equal final success rates (97.6% vs. 98.1%), could reflect systematic differences in workload composition rather than a causal effect of SiriusDeliver. Please provide evidence that the task mix is balanced across arms, or analyze the outcomes within predefined task-difficulty strata, or redesign the experiment so that tasks themselves are randomly assigned.","section":"§4.3 (Online Deployment, A/B test, Figure 8)"},{"comment":"The 'engineer effort' metric is not defined. The paper reports median engineer effort of 95 minutes in the control group and 11 minutes in the treatment group, but it does not state how effort was collected for engineers performing manual delivery (e.g., self-reported estimates, activity logs, or time tracking) or what 'net' excludes. If the two arms used different measurement instruments, the 88.4% reduction is not interpretable. The same ambiguity applies to 'time to first executable artifact' and 'interaction rounds.' Please provide precise definitions and the data-collection protocol for each cost metric, ideally with a demonstration that the measurement methods are consistent across arms.","section":"§4.3 (Online Deployment, Figure 8)"}],"minor_comments":[{"comment":"The 'manual intervention rate' is 100% in the control group by construction, so the reduction to 21% is not informative; please clarify what counts as an intervention and report absolute rates for each intervention type (clarification, artifact edit, manual submission, manual repair, and so on).","section":"§4.3 (Online Deployment)"},{"comment":"The public companion benchmark DataClawEval is explicitly not used for any reported result and does not model WeData-specific workflows; the paper should state more prominently that the main offline results are not externally reproducible with the released artifact, or provide additional released components that can reproduce a subset of the claims.","section":"§4.1 (Dataset) and Appendix A.2"},{"comment":"The per-backbone success rates in Table 2 are reported as integers without confidence intervals; given the sample size of 200 cases, the 12-to-16-point gains should be accompanied by uncertainty estimates or a note that the comparisons are descriptive rather than inferential.","section":"§4.2 (End-to-end performance, Table 2)"},{"comment":"The skill-evolution evaluation reports token usage and reasoning turns for the original and evolved skill libraries, but it does not report end-to-end success rates on the held-out set for both conditions; please include success rates (or an explicit statement that they are unchanged) so the efficiency gains can be interpreted without concern that evolution achieves lower cost by failing more often.","section":"Figure 6 and §4.2 (Effect of skill evolution)"},{"comment":"The LLM-as-judge protocol for artifact quality uses three judges and position swapping, but the paper does not report the per-judge agreement rate or the number of 'inconsistent' outcomes; a high inconsistency rate would weaken the claim that evolved artifacts are preferred or comparable in 79.2% of cases, so please report these quantities.","section":"Appendix A.3 (LLM-as-Judge Protocol)"},{"comment":"There are several small presentation issues: 'RT Sync' and 'Off. Sync' are not expanded in Table 1; the sentence in §4.2 beginning 'these results show that lifecycle control Artifact lifecycle control improves' contains a duplicated phrase; and the phrase 'one-month A/B test' in the abstract is not consistent with the deployment timeline described in §4.3, so please clarify the relationship between the two.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The A/B test internal-validity issue is the primary reason for the major revision. The authors should be asked to provide task-level comparability evidence or an alternative analysis for the cost-reduction claims. The offline evaluation is strong and well-controlled, and I would be willing to accept the paper after the A/B test metrics are defined and the confound is addressed. The paper is a good fit for the journal's systems track, though the external reproducibility of the central offline results is inherently limited by the proprietary nature of the benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid industrial systems paper, and the strongest evidence is the offline paired evaluation, not the A/B test. The headline production numbers are believable but less rigorously supported than they look.\n\nWhat's new: applying agent techniques—hierarchical skill orchestration, pre/post-execution artifact validation, trace-driven skill evolution—to the warehouse delivery problem at Tencent scale. Individually the components resemble existing agent patterns (Reflexion, Voyager, etc.), but the combination for DW delivery, with a concrete skill hierarchy and artifact lifecycle control, is genuinely engineered and evaluated at a scale that matters. The offline evaluation is well controlled: shared skill library, frozen snapshots, disjoint evolution trajectories, per-product paired comparisons, and ablations that isolate each component. The 14.5-point success gain over the strongest baseline is credible. The backbone study showing consistent gains across different models is a nice touch.\n\nThe soft spots are real but concentrated. First, the A/B test randomizes 70 engineers, not tasks. Treatment users interact with a web interface and can pick which requests to route through the agent, so task difficulty may differ between arms. The paper reports no task-complexity covariates, and stratified randomization by team and task type doesn't fix that. Second, engineer effort in the control arm is never defined: is it self-estimated, logged, or derived from timestamps? Without that, the 95-to-11 minute effort reduction is hard to interpret. These don't invalidate the offline results, but they weaken the production cost claims. Third, the benchmark is internal and proprietary; the public DataClawEval companion is useful but doesn't test the actual system. Success adjudication is human verification, which is reasonable.\n\nI'd send this to peer review. The architecture is coherent, the offline methodology is above the usual bar for systems papers, and the production deployment gives the community real grounding. The A/B test needs a major revision—task comparability analysis, effort measurement definition, ideally a causal or at least stratified analysis—but that's fixable, not fatal.","headline":"Credible industrial agent system for warehouse delivery; offline evidence is well controlled, but the production A/B test needs tighter measurement before the cost-reduction claims are accepted.","tokens_in":19274,"tokens_out":1882,"would_cite":false,"duration_ms":18281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SiriusDeliver claims full automation of data warehouse delivery, raising offline success from 71.5% to 86.0% and cutting median delivery time from 228 to 23 minutes in production, all while keeping final delivery quality steady.","keywords":["data warehouse task delivery","delivery automation agent","LLM agents","hierarchical skill orchestration","artifact lifecycle control","trace-driven skill evolution","production deployment","DataOps"],"falsifier":"Instrument the manual-workflow control arm with the same automated session-timestamp logging used for the agent arm—start-of-request through final submission—and log engineer effort by the same interaction-tracking method across a comparable set of delivery tasks. If matched-instrumentation medians come out well below 228 minutes for delivery time or well below 95 minutes for effort, the headline A/B reductions would not survive; a complementary check is evaluating the three components on the paper's public companion benchmark, which omits the scheduling, permission, and lifecycle workflows of the private platform.","tokens_in":18354,"feed_emoji":"🤖","tokens_out":10437,"duration_ms":83270,"temperature":0.7,"pith_summary":"SiriusDeliver is a system that attempts to automate the entire enterprise data warehouse delivery process—turning a business request into a validated, submittable bundle of workflow definition, code, and configuration—rather than merely generating SQL. The paper argues that production delivery needs three elements that general coding agents lack: a hierarchy of reusable warehouse skills composed step by step by an agent, a lifecycle control stage that verifies artifacts before submission and diagnoses failures after execution, and an offline mechanism that converts past delivery trajectories into bounded, human-reviewed skill updates. The reported evidence is a 14.5-point offline gain in end-to-end delivery success over the strongest skill-augmented coding-agent baseline, an 87.2% success rate and 73.5% autonomous submission rate across 18,240 production sessions, and an A/B test in which median delivery time fell from 228 to 23 minutes and engineer effort from 95 to 11 minutes with final success essentially unchanged. If true, the most labor-intensive part of data engineering—coordinating context, dependencies, permissions, and schedules—could be handed to an agent that keeps improving with use.","feed_headline":"Agent cuts warehouse delivery time from 228 to 23 minutes","feed_subtitle":"Production agent automates end-to-end warehouse delivery, cutting engineer effort 88% at equal success rates.","key_machinery":"The load-bearing object is the artifact bundle $\\langle G, C, P \\rangle$—workflow specification, executable artifacts, and task configurations treated as jointly validated units rather than free-standing code—together with a four-layer skill hierarchy (scenario, context, artifact, and platform skills) that the delivery agent composes in a closed loop with a manager, an executor, and a memory. Lifecycle control carries the reliability argument: pre-execution diagnosis combines deterministic platform evidence (metadata, dependencies, pre-compilation results) with LLM reasoning over defined diagnosis items, and post-execution diagnosis converts platform feedback into extracted failure signals plus retrieved diagnostic knowledge before any revision is generated. Trace-driven skill evolution carries the adaptation argument: trajectories grouped by skill and outcome are compressed into structured evidence, LLM updates are bounded by an update budget, validated by offline replay, and gated by human review with rollback.","core_discovery":"On its own terms, the paper's central claim is that warehouse delivery automation fails when it is modelled as code generation and succeeds when it is modelled as a closed loop over four layers of skills—scenario, context, artifact, and platform—governed by a delivery state, verified before and after platform execution, and maintained over time from recorded trajectories. The unit of delivery is the artifact bundle $\\langle G, C, P \\rangle$ (workflow specification, executable artifacts, task configurations), which is produced by the orchestration loop, passes through pre-execution diagnosis (evidence grounding, hybrid verification, candidate revision), is submitted, and, if it fails, is repaired by post-execution diagnosis that extracts failure signals, retrieves diagnostic knowledge, and generates a grounded revision. Trace-driven skill evolution then groups trajectories by skill and outcome, applies bounded LLM updates for failure repair and success compression, validates them on held-out trajectories, and releases them only after human review with rollback. The reported result is that offline end-to-end delivery success rises from 71.5% to 86.0% over the strongest baseline while token use falls 30%, and in production the system attains an 87.2% end-to-end success rate with 73.5% of submissions fully autonomous, with the A/B test showing delivery time down 89.9% and engineer effort down 88.4% at comparable final quality (97.6% versus 98.1%).","pith_inferences":["The same three-component recipe—hierarchical skill orchestration, pre- and post-artifact verification, and trace-driven skill updates—should transfer to other approval-gated production workflows such as CI/CD pipeline configuration, infrastructure-as-code changes, and ML model deployment, which share the generate-submit-verify-repair shape; this is an extrapolation, not a claim the paper makes.","The backbone results suggest that structured orchestration and deterministic verification act as a capability floor, so a testable corollary is that the per-backbone gain should persist on newer models released after this evaluation.","The A/B test's 21% residual manual intervention rate and the offline 78% success on offline computation tasks jointly suggest that the remaining automation ceiling is the hardest task class; whether skill evolution erodes that ceiling over many months is an open question the two-month window cannot settle.","Because the private 200-case benchmark cannot be released, the portability of the diagnosis components cannot be verified publicly; building the pre-execution partition-scan and post-execution log-diagnosis checks on a public warehouse schema would test whether the gain generalizes beyond the system's home platform."],"forward_implications":["Warehouse delivery can be largely automated without sacrificing final quality: the paper's production figures put end-to-end success at 87.2% with 73.5% of sessions submitted without manual artifact modification, and the A/B test holds final success at 97.6% versus 98.1%.","The gains are architectural rather than model-specific: paired tests across four LLM backbones show 12.0 to 16.0 percentage-point success gains, with the weakest backbone gaining the most, implying structured orchestration substitutes for raw reasoning ability.","Lifecycle control converts most execution failures into single-shot repairs: 73.0% of failed executions were repaired within one diagnosis-guided revision and 75.9% of real artifact issues were caught before submission, at latencies of 24.2 seconds and 6.1 seconds respectively.","The system becomes cheaper as it is used: trace-driven skill evolution cut token use by 25.5% and reasoning turns by 17.5% on held-out cases, and produced 128 human-approved skill updates during two months of production.","The largest single lever is hierarchical skill orchestration: removing it drops offline success by 11.5 points, more than removing lifecycle control or skill evolution."],"supporting_citations":[{"why":"Claude Code overview: the strongest skill-augmented baseline (71.5% success) that +SiriusDeliver must beat in the offline paired comparison.","marker":"[2]"},{"why":"OpenAI Codex: one of the three coding-product baselines whose +SiriusDeliver comparison isolates the framework's incremental gain.","marker":"[15]"},{"why":"Tencent Cloud CodeBuddy: the third baseline and the agent runtime used for the backbone-generalization study.","marker":"[25]"},{"why":"SWE-agent: the representative code-agent paradigm that repairs reactively from logs, which artifact lifecycle control is designed to surpass.","marker":"[28]"},{"why":"AutoCodeRover: representative of post-execution repair from logs alone, the limitation motivating proactive pre-submission verification.","marker":"[30]"},{"why":"Chen et al. 2024: log-grounded root cause analysis for cloud incidents that post-execution diagnosis builds its failure extraction on.","marker":"[4]"},{"why":"SiriusHelper: the platform operations assistant whose log-grounded diagnosis approach is cited as the basis for post-execution diagnosis.","marker":"[21]"},{"why":"Reflexion: the verbal-reinforcement lifelong learning approach contrasted with bounded, reviewable trace-driven skill evolution.","marker":"[22]"},{"why":"ExpeL: experiential learning from trajectories, the prior self-improvement paradigm the paper contrasts with auditable skill updates.","marker":"[31]"}],"fun_headline_variants":["Agent cuts warehouse delivery time from 228 to 23 min","Engineer effort for warehouse delivery down 88%","SiriusDeliver automates 73.5% of warehouse submissions","Warehouse delivery agent hits 87.2% end-to-end success","Tencent's warehouse agent: 228 to 23 min, 88% less effort"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The A/B test's headline reductions assume that 'engineer effort' (95 to 11 minutes) and 'delivery time' (228 to 23 minutes) were measured with the same method in both arms, but Section 4.3 reports the manual-workflow control figures without defining how they were logged, so if manual effort was self-estimated or tracked differently from the agent's session timestamps, the 89.9% and 88.4% reductions could be measurement artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Agent cuts warehouse delivery time from 228 to 23 min","Engineer effort for warehouse delivery down 88%","SiriusDeliver automates 73.5% of warehouse submissions","Warehouse delivery agent hits 87.2% end-to-end success","Tencent's warehouse agent: 228 to 23 min, 88% less effort"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001401,"raw_usage":{"total_tokens":5748,"prompt_tokens":1109,"completion_tokens":4639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":4545}},"tokens_in":725,"tokens_out":4639,"duration_ms":28703,"temperature":1.0,"reasoning_tokens":4545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:50:00.907306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the manual-workflow control arm with the same automated session-timestamp logging used for the agent arm—start-of-request through final submission—and log engineer effort by the same interaction-tracking method across a comparable set of delivery tasks. If matched-instrumentation medians come out well below 228 minutes for delivery time or well below 95 minutes for effort, the headline A/B reductions would not survive; a complementary check is evaluating the three components on the paper's public companion benchmark, which omits the scheduling, permission, and lifecycle workflows of the private platform.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude Code overview: the strongest skill-augmented baseline (71.5% success) that +SiriusDeliver must beat in the offline paired comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OpenAI Codex: one of the three coding-product baselines whose +SiriusDeliver comparison isolates the framework's incremental gain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tencent Cloud CodeBuddy: the third baseline and the agent runtime used for the backbone-generalization study."},{"cited_title":"Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press","cited_arxiv_id":null,"evidence_quote":"SWE-agent: the representative code-agent paradigm that repairs reactively from logs, which artifact lifecycle control is designed to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AutoCodeRover: representative of post-execution repair from logs alone, the limitation motivating proactive pre-submission verification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chen et al. 2024: log-grounded root cause analysis for cloud incidents that post-execution diagnosis builds its failure extraction on."},{"cited_title":"SiriusHelper: An LLM Agent-Based Operations Assistant for Big Data Platforms","cited_arxiv_id":"2605.00043","evidence_quote":"SiriusHelper: the platform operations assistant whose log-grounded diagnosis approach is cited as the basis for post-execution diagnosis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reflexion: the verbal-reinforcement lifelong learning approach contrasted with bounded, reviewable trace-driven skill evolution."},{"cited_title":"etl_task.py","cited_arxiv_id":null,"evidence_quote":"ExpeL: experiential learning from trajectories, the prior self-improvement paradigm the paper contrasts with auditable skill updates."}],"review_version":1}