{"id":"e1dbc2e7-4e2c-47b6-9030-40a8e3279846","arxiv_id":"2608.10224","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A production customer-support agent that continuously evolves its prompts and retrieval in a closed evaluation loop improved live self-serve resolution and routing accuracy at LinkedIn.","lead":"LinkedIn's customer-support agent rewrites its own prompts every week using an evolutionary search, checks the new behavior with automated judges, and only then rolls the changes out to live users. In a two-week production experiment, the self-evolving system raised self-service resolution by 9 percentage points for questions and 4.8 points for cancellations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The routing-accuracy result in Table 5 is presented as part of the user-randomized online A/B, but the footnote says it was measured on a fixed labeled evaluation set; the largest reported lift may not be a live-traffic result.","rationale":"Read in good faith, the paper is a systems and empirical paper whose central claim is that a self-evolving workflow improved live support outcomes. The QA and cancellation results are enormous in scale (z = 27.6 and z = 10.0), internally consistent with the reported sample sizes, and robust to the paper's stated cluster adjustment; the disclosed limitations are unusually honest. The reader's weakest assumption, LLM-as-judge alignment, is a real concern but is partially mitigated by the fact that the online experiment measures user-facing outcomes rather than judge scores: if the judge were grossly misaligned, the same evolution loop would likely not produce large improvements in QA self-serve and cancellation self-serve. By contrast, the routing row in Table 5 creates a direct factual inconsistency in the main empirical evidence. It is presented as part of a user-randomized online A/B, yet the footnote says it was measured on a fixed labeled evaluation set. Unless the authors clarify that the fixed labels were applied to decisions from randomized live users, the largest headline effect is not an online A/B result, and the two-proportion z-test used in Table 5 is misspecified for paired replay. This does not overturn the QA and cancellation evidence, so the CONDITIONAL verdict is unchanged, but the paper should resolve the routing discrepancy before the strongest claim is taken at face value. The proposed check is a direct request for the routing data-collection protocol and, if needed, a McNemar re-analysis.","tokens_in":8896,"tokens_out":12248,"duration_ms":122856,"concrete_test":"Ask the authors for the routing evaluation protocol: (a) were the 356 control and 356 treatment routing decisions disjoint subsets of live randomized user traffic, or (b) were the same 356 pre-labeled conversations replayed through both arms? If (b), reclassify the routing metric as an offline paired evaluation, re-run the significance analysis with McNemar's test, report the discordant-pair counts, and remove the routing metric from the user-randomized online A/B claim in the abstract and Section 5. Also state how the 356-case labeled set was sampled from production routing traffic and whether the control arm's 38.2% accuracy reflects the true deployed production router on that sample.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 frames the study as a two-week user-randomized A/B on production support traffic, and the abstract repeats this framing for all three headline metrics. However, Table 5's routing row reports '356 / 356 decisions (control / treatment) on a fixed labeled evaluation set.' The natural reading is that the same 356 labeled routing inputs were replayed through the control and treatment configurations, not that routing decisions were collected from two independently randomized live user populations. If that is what happened, then the 30.6 pp routing lift is an offline paired benchmark, not an online A/B outcome. The two-proportion z-test in Table 5 treats the 356 decisions per arm as independent samples, which is misspecified for paired replay; the 'user is the inference unit' statement also does not apply to this row. This matters because routing accuracy is the largest of the three reported effects and the abstract's 'user-randomized A/B test on production support traffic' claim explicitly covers it. The QA and cancellation lifts (z = 27.6 and z = 10.0) are large enough to survive reasonable concerns about clustering and early stopping, so the overall integrated-system claim may still hold; but the routing claim, as stated, is not supported by the procedure described unless the footnote means something narrower, such as a fixed label set applied to decisions collected from randomized live traffic. The text needs to say which.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes LinkedIn's production customer-support agent that closes the loop between prompting, retrieval, and evaluation: an evolutionary auto-prompt engine (Algorithm 1) optimizes system prompts against fitness signals from a modular LLM-judge evaluation framework, while an agentic RAG layer retrieves versioned support content. The authors report offline simulations (Tables 1–4) showing reduced hallucination, improved response completeness, and gains from crossover and mutation in prompt evolution, and a two-week production A/B test (Section 5, Table 5) in which the integrated workflow raises QA self-serve by 9.0 pp, cancellation self-serve by 4.8 pp, and routing accuracy by 30.6 pp. The paper's central claim is that the closed-loop system improves live support outcomes relative to the handcrafted production agent, and the Limitations section candidly enumerates threats to generalizability and evaluator dependence.","tokens_in":9147,"tokens_out":4153,"duration_ms":43795,"significance":"If the production results are as reported, this is a valuable industrial validation of closed-loop prompt evolution and agentic RAG, with large treatment effects and cluster-robust checks for the QA and cancellation outcomes. The paper also gives credit for its explicit operational framing: versioned artifacts, staged rollout, rollback, and a candid limitations section. However, the largest reported effect—routing accuracy, 30.6 pp—is, on the manuscript's own description, measured on a fixed labeled evaluation set of 356 decisions per arm, which is inconsistent with the abstract's claim that all three outcomes come from a user-randomized A/B test on production support traffic. This discrepancy is load-bearing because it changes the nature of the headline result. The offline tables also lack uncertainty quantification on small samples, and the self-evolution loop rests on LLM-as-judge alignment measured on 100 chats. These issues are fixable within a revision, so the paper merits a major revision rather than rejection.","major_comments":[{"comment":"The routing-accuracy row reports '356 / 356 decisions (control / treatment) on a fixed labeled evaluation set,' which is inconsistent with the section's statement that 'The three metrics use separate randomized subpopulations' and with the abstract's claim that all three outcomes come from a user-randomized A/B test on production support traffic. The natural reading is that the same 356 labeled routing inputs were replayed through the control and treatment configurations. If so, the 30.6 pp lift is a paired offline comparison, not an online A/B result; the two-proportion z-test (z = 8.2) treats 356 independent decisions per arm and is misspecified for paired replay, and the 'user is the inference unit' statement does not apply to this row. Please state explicitly whether the 356 decisions were sampled from randomized live traffic or replayed from a fixed set, and if the latter, report McNemar's test or a paired bootstrap and revise the abstract and Section 5 framing accordingly.","section":"Section 5, Table 5"},{"comment":"The offline simulations report point estimates without error bars, confidence intervals, or significance tests, on small samples: Table 1 uses 100 interactions, Table 2 uses N = 30 routing prompts, Table 3 uses 100 chats, and Table 4 uses 300 chats. For example, Table 2 shows average-accuracy improvements of 1.4–5.4 percentage points across generations, and Table 3 shows alignment differences of 1–20 points, but no uncertainty is attached to any of these values. Since these tables support the component-level claims (reduced hallucination, prompt-evolution gains, evaluator design), please provide bootstrap confidence intervals or equivalent measures and specify the number of independent inference runs per interaction.","section":"Tables 1–4"},{"comment":"The outer optimization loop selects prompts by fitness scores from an LLM judge, and the calibration evidence in Table 3 is based on 100 chats with 87% alignment to human labels. This sample size is small relative to the number of generated prompts and generations, so the results do not fully rule out judge-specific overfitting that the evolutionary search could exploit. The paper's Limitations section acknowledges this risk, but the central claim that the system evolves genuine quality improvements depends on the judge's alignment. Please provide a larger held-out human audit of the final evolved prompts, per-generation judge-versus-human correlation, or an explicit analysis of judge–prompt interaction and reward-hacking resistance.","section":"Section 2.2 and Section 4.3"}],"minor_comments":[{"comment":"Please revise the wording so the routing metric is not described as part of the user-randomized A/B unless it is in fact measured on live traffic; the current phrasing overstates the result.","section":"Abstract and Section 5"},{"comment":"The operators SEMANTICBLEND and MUTATE are not formally defined, and the population size N, elite size K, and number of generations G used in the routing-prompt simulation are not reported; please provide these values or point to a supplement.","section":"Algorithm 1 and Section 4.2"},{"comment":"Please define how 'average prompt accuracy' is computed across the prompt population and state how many random seeds or independent runs the reported numbers are averaged over.","section":"Table 2"},{"comment":"Since the production generator (GPT-4o-mini) differs from the offline judge (GPT-4.1), please clarify whether hallucination labels on the 100-interaction set were also checked by human annotators and report the human–judge agreement for the hallucination dimension specifically.","section":"Section 4.1"},{"comment":"Several references are truncated to 'and 1 others' (e.g., Asai et al., Zhou et al., Fernando et al.), which prevents verification of the cited work; please use complete author lists.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The routing-metric discrepancy is a publication-integrity issue: as written, a headline number in the abstract is not backed by the described experiment. The QA and cancellation results are large enough that they may well stand, so the fix is feasible within a revision; the authors should either reclassify the routing result as an offline paired benchmark or clearly document how the 356 routing decisions were sampled from live randomized traffic. I would also encourage the editor to ask for a small reproducibility appendix or artifact release, as the paper's contribution is partly engineering-oriented and the current text gives no code or configuration details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper from LinkedIn about a self-evolving agentic support system. The genuinely new thing is not any single component—prompt evolution, agentic RAG, LLM-as-judge are all known—but the integrated, versioned closed loop running in production with A/B results. If the QA and cancellation lifts hold, that is a meaningful template for applied agent deployments.\n\nWhat the paper does well: the architecture is described concretely with versioned artifacts, staged rollout, and rollback. The limitations section is honest about component attribution, closed-source dependencies, and judge dependence. The A/B for QA and cancellation is large and the clustered standard error check is a good touch. The offline ablations, though small, point in the right direction.\n\nThe biggest soft spot is the routing accuracy result. It is presented in Section 5 and the abstract as part of the user-randomized A/B, but Table 5's footnote says it was measured on 356 fixed labeled decisions per arm. That reads like an offline paired replay, not a randomized live-traffic comparison, and the z-test treats the two arms as independent samples, which is misspecified for paired data. The paper needs to state explicitly whether routing decisions were collected from the randomized live populations or replayed on a fixed set. As it stands, the 30.6 pp routing lift is not supported by the procedure described. It may well be real, but the abstract overstates it.\n\nOther issues are secondary. The offline simulations have no error bars and small N (30 routing prompts, 100 RAG chats). The LLM-as-judge alignment is measured on 100 chats at 87%, and the whole evolution loop depends on that; the paper acknowledges this but doesn't stress-test judge drift or overfitting. The online treatment bundles all components, so marginal credit is unknown—acknowledged in Limitations. No code or data release, so reproducibility relies on description alone.\n\nThe citation pattern looks fine; the relevant prompt-optimization and agentic RAG literature is cited, and self-citation is not an issue here.\n\nThis is for applied researchers and engineers building production LLM agents. It deserves a serious referee; the systems claims are concrete and the QA/cancellation results are large enough to matter. I would send it to review with a request to fix the routing claim and add error bars or explicit caveats on offline tables. My own verdict would be conditional until the routing description is clarified.","headline":"A genuinely useful production-systems paper whose headline routing result may not be a live A/B outcome; QA and cancellation lifts look solid, but the routing claim needs clarification.","tokens_in":9747,"tokens_out":1734,"would_cite":true,"duration_ms":16222,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a two-week production A/B test, LinkedIn's self-evolving support agent—combining evolutionary prompt search, agent-invoked retrieval, and modular LLM-judge evaluation—raised QA self-serve by 9.0 percentage points and routing accuracy…","keywords":["self-evolving agents","customer support","evolutionary prompt optimization","retrieval-augmented generation","LLM-as-a-judge","production A/B test","enterprise AI deployment","versioned workflows"],"falsifier":"Take fresh human-labeled support chats from each generation of the production loop and recompute the automated judge's agreement with blind human reviewers; if judge fitness rises while judge-human agreement falls below the 87% calibration level, the loop is optimizing the judge rather than service quality.","tokens_in":8653,"feed_emoji":"🤖","tokens_out":9595,"duration_ms":88021,"temperature":0.7,"pith_summary":"This paper claims that enterprise customer-support agents can be made to improve continuously—without retraining the underlying model—by closing the loop between prompts, retrieval, and evaluation. The authors built and deployed at LinkedIn a support agent whose system prompt is evolved by a genetic algorithm, whose retrieval-augmented generation is invoked as an explicit tool, and whose outputs are scored by a modular multi-signal evaluator; fitness scores drive the next round of prompt evolution. In offline simulations the full agent reduced hallucination from 4.8–6.2% to under 0.1%, and in a two-week randomized production test the integrated workflow raised question-answering self-serve by 9.0 percentage points, cancellation self-serve by 4.8 points, and routing accuracy by 30.6 points. The sympathetic reading is that a carefully guarded closed loop can make a live agent measurably better under real-world policy and knowledge drift.","feed_headline":"Self-evolving support AI lifts self-serve by 9 points","feed_subtitle":"A live A/B test shows the gains come from a closed loop of prompt evolution, agentic RAG, and LLM judges.","key_machinery":"The load-bearing mechanism is the outer optimization loop that connects three versioned layers. An Automatic Prompt Engineering Engine runs a genetic algorithm over system-prompt text—initializing candidates with an LLM under business rules, then repeatedly selecting elite prompts by evaluator fitness, recombining them with semantic crossover, and mutating them, while filtering any prompt that violates an immutable policy constraint. At inference time the agent invokes retrieval as a tool over a versioned content lake, and a modular evaluation framework scores each response on groundedness, relevance, completeness, and multilingual fidelity, aggregating those signals into the fitness that drives the next generation. Everything—prompts, retrieval snapshots, scores, and candidate artifacts—is versioned and gated through regression checks and staged rollout, so the loop can improve or roll back without redeploying runtime code.","core_discovery":"The paper's central claim is that a production support agent can improve itself by treating its system prompt, its retrieval layer, and its evaluation layer as one versioned closed loop. The authors argue that rather than retraining or hand-editing the prompt when support knowledge changes, an evolutionary search over prompt text—constrained by immutable business rules and scored by a modular LLM-judge evaluator—can find better behavior automatically. They report that in a two-week user-randomized production test the integrated self-evolved workflow raised QA self-serve from 33.7% to 42.7%, cancellation self-serve from 61.9% to 66.6%, and routing accuracy from 38.2% to 68.8%, with all three significant after Holm correction; offline ablations attribute parts of the gain to agentic RAG, to having both crossover and mutation in prompt search, and to keeping groundedness as a first-class evaluation signal.","pith_inferences":["Because the production test bundles auto-prompt, agentic RAG, and evaluator-driven iteration, the reported lifts are joint effects; a factorial deployment would be required to see which component carries the gains.","The same loop could plausibly be adapted to other fast-changing enterprise workflows—sales triage, compliance Q&A, policy assistance—wherever a stable judge and a versioned content store exist; the paper does not test that transfer.","A natural experiment is to hold the judge fixed across generations and re-measure judge-human agreement on fresh human-labeled chats; rising judge scores with falling agreement would show the loop gaming its evaluator rather than improving service.","Swapping the closed-source judge and retriever for open-weight equivalents, which the authors call feasible but unverified, would tell whether the loop's gains depend on the specific GPT-4.1 judge."],"forward_implications":["Deploying the integrated workflow on live support traffic increased QA self-serve from 33.7% to 42.7% and cancellation self-serve from 61.9% to 66.6% over the handcrafted control.","Routing accuracy rose from 38.2% to 68.8%, measured on a fixed labeled set of 356 routing decisions.","Both crossover and mutation are needed for sustained prompt improvements: with both, average prompt accuracy rose from 62.6% to 68.0% in two generations, while removing either operator stalled early.","The full agentic RAG configuration reduced hallucination to below 0.1% and raised completeness to 87.8% in offline simulation, against 4.8–6.2% hallucination for vanilla and agentic baselines.","Multi-signal evaluation is load-bearing: dropping groundedness lowers evaluator-human alignment by 11 points, and collapsing all signals into a single overall score lowers it by 20 points."],"supporting_citations":[{"why":"Supplies the score-guided prompt-optimization baseline (APE) that Auto-Prompt extends into population search.","marker":"Zhou et al., 2023"},{"why":"Provides the LLM-as-optimizer baseline (OPRO) against which the evolutionary approach is positioned.","marker":"Yang et al., 2024"},{"why":"Supplies the PromptBreeder evolutionary prompt-search lineage that the Auto-Prompt engine implements.","marker":"Fernando et al., 2023"},{"why":"Introduces DSPy's declarative, metric-compiled prompts, the alternative the paper contrasts with constrained evolution.","marker":"Khattab et al., 2023"},{"why":"Grounds responses in retrieved content, the RAG mechanism the agent constrains itself to.","marker":"Lewis et al., 2020b"},{"why":"Provides the reasoning-and-acting paradigm that justifies exposing retrieval as an agent-invoked tool.","marker":"Yao et al., 2023"},{"why":"Documents LLM-as-judge behavior and biases, motivating the modular multi-signal evaluator.","marker":"Zheng et al., 2023"},{"why":"Shows how length control debiases automatic evaluators, informing the evaluator design.","marker":"Dubois et al., 2024"},{"why":"Supplies the COMET baseline the modular multi-agent evaluator is measured against for translation quality.","marker":"Rei et al., 2020"}],"fun_headline_variants":["Closed-loop prompt evolution: +9 self-serve, +30.6 routing","Self-evolving support AI: routing accuracy +30.6 in A/B","No retraining: LinkedIn support AI evolves prompts, boosts self-serve 9 pts","Agentic loop: self-evolving prompts +30.6 routing in production"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop's fitness signal is an automated LLM judge, and the paper calibrates that judge against human labels on only 100 chats (87% agreement); if the judge is biased in a way the evolutionary search can exploit, prompts could please the judge without actually improving customer service.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop prompt evolution: +9 self-serve, +30.6 routing","Self-evolving support AI: routing accuracy +30.6 in A/B","No retraining: LinkedIn support AI evolves prompts, boosts self-serve 9 pts","Agentic loop: self-evolving prompts +30.6 routing in production"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001734,"raw_usage":{"total_tokens":6833,"prompt_tokens":904,"completion_tokens":5929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":5845}},"tokens_in":520,"tokens_out":5929,"duration_ms":41942,"temperature":1.0,"reasoning_tokens":5845,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:48.607169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take fresh human-labeled support chats from each generation of the production loop and recompute the automated judge's agreement with blind human reviewers; if judge fitness rises while judge-human agreement falls below the 87% calibration level, the loop is optimizing the judge rather than service quality.","supporting_citations":[],"review_version":1}