{"id":"3e91395c-d2f1-47e8-9eeb-e67eee8fd1a8","arxiv_id":"2505.17716","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.","lead":"This paper proposes AgentRR, a record-and-replay framework that captures successful agent or human task traces, summarizes them into multi-level experiences, and replays them with check functions to guide LLM agents. It argues this lowers cost, improves reliability, preserves privacy, and speeds up execution, but provides no quantitative evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The check-function TCB is asserted, not constructed: Section 3.2.2's 'guarantee' is contradicted by Section 6's admission that 100% reliable replay remains elusive, so the central reliability claim lacks support.","rationale":"The reader's weakest_assumption identifies the check-function TCB, and I agree that this is the load-bearing point. AgentRR's claim to improve reliability depends on the ability of check functions to catch every unsafe divergence during replay; without that, the framework is just an LLM agent with a prompt-level memory system. The paper gives no method to make check functions complete, and in fact permits ML-generated check functions, which are subject to the same hallucination and incompleteness issues the paper attributes to LLMs. The paper's own limitations section admits replay fidelity and robustness are unsolved, so the 'guarantee' language in Section 3.2.2 is not merely unproven; it is internally contradicted. The privacy, cost, and performance claims also rest on the assumption that low-level replay succeeds often enough to avoid falling back to expensive high-level LLM reasoning; if replay is unreliable or check functions are too conservative, those benefits also erode. The paper is coherent as a vision and correctly identifies real limitations of existing approaches, which should be credited. However, because the central claim is empirical and the only mechanism that supplies safety is an unverified TCB, I would maintain the reader's REJECT verdict. The concrete test above would convert the paper's assertion into a falsifiable claim.","tokens_in":21508,"tokens_out":4713,"duration_ms":38382,"concrete_test":"Build a small formal environment (e.g., a web booking flow with an explicit state machine enumerating all legal and illegal transitions). Record two traces, summarize them into a low-level experience with a check function per Section 4.2, and then run an exhaustive/adversarial replay suite that attempts every illegal transition (wrong order, missing precondition, out-of-range parameter, unsafe payment action). Measure whether the check function blocks 100% of illegal actions. Separately measure replay success rate, API cost, and latency against a pure LLM agent on the same benchmark. If any illegal action is not blocked, the Section 3.2.2 guarantee is false and the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 3.2.2: once a user audits and trusts the check functions, \"it guarantees that the agent will adhere to the user's safety requirements throughout the replay process.\" The check functions are the only proposed TCB separating AgentRR from an ordinary LLM agent, so the reliability and safety half of the central claim (Section 6) depends entirely on this guarantee. Yet the paper provides no construction, formal semantics, or verification method for a complete check function. It explicitly allows check functions to be \"generated by the user-provided description and ML-based summary\" (Section 3.2.2), so they inherit the very hallucination and incompleteness problems the framework is supposed to remove. Section 4.2 suggests manual audit and bug bounties, which are heuristics, not guarantees. The paper's own Section 6 concedes that \"achieving 100% reliable replay remains elusive\" and that summarization can be incomplete, directly contradicting the guarantee. If even one illegal transition, bad precondition, or unsafe parameter escapes the check function, the replay is unsafe and the claimed reliability benefit collapses to that of an unvalidated LLM agent with a partial safety filter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentRR, a record-and-replay paradigm for LLM agents in which an agent's interaction trace is recorded, summarized into multi-level 'experiences' (low-level concrete operations and high-level procedural knowledge), and replayed under the supervision of 'check functions' that are claimed to act as a trusted computing base. The authors argue that this decouples intelligence from execution and thereby improves reliability, privacy, cost, and performance compared with both pure LLM agents and traditional record-and-replay tools. The paper also sketches several application modes (human-record/model-replay, large-model-record/small-model-replay, and untrusted-model-record/trusted-model-replay) and an Experience Store, and concludes that AgentRR effectively addresses the four challenges. No quantitative evaluation, implementation, or formal correctness analysis is reported.","tokens_in":21709,"tokens_out":4336,"duration_ms":47925,"significance":"If the proposed mechanisms worked as claimed, AgentRR would be a useful conceptual contribution to the emerging area of dependable LLM agents: the taxonomy of record/replay modes is clear, the related-work survey is broad, and the discussion of the reliability-privacy-cost-performance trade-off space is thoughtful. However, the central claims are not supported by any measurement, implementation, or formal argument. The paper itself, in Section 6, concedes that 100% reliable replay remains elusive and that summarization can be incomplete, which directly undermines the safety guarantee advertised in Section 3.2.2. At present the contribution is therefore a position paper, not a validated system; its significance depends entirely on a future implementation and evaluation.","major_comments":[{"comment":"The sentence 'Our design and evaluation demonstrate that AgentRR effectively addresses the challenges of agent reliability, privacy, and performance through its innovative record-and-replay mechanism' is unsupported by the contents of the manuscript. Section 5 is an anecdotal case study: it reports that OpenAI's CUA takes approximately three minutes and still fails, and that Chrome Recorder requires manual parameter input, but it gives no AgentRR success rate, no latency or cost measurements, no privacy metric, and no controlled comparison with the baselines. The central claim of the paper is therefore an assertion, not a demonstrated result.","section":"Section 6"},{"comment":"The guarantee that once a user audits and trusts the check functions, 'it guarantees that the agent will adhere to the user's safety requirements throughout the replay process' is load-bearing but unsubstantiated. The check functions are called a trusted computing base, yet the paper provides no construction, formal semantics, completeness condition, or verification method. Section 4.2 allows check functions to be 'generated by the user-provided description and ML-based summary,' so they can inherit the very hallucination and incompleteness problems the framework is meant to remove. Section 6 then concedes that 'achieving 100% reliable replay remains elusive' and that summarization can be incomplete. This internal contradiction removes the support for the reliability and safety half of the central claim.","section":"Section 3.2.2"},{"comment":"The entries for AgentRR in Table 2—'High (Exceeds human speed)', 'High' accuracy, and 'High (Generalized for repetitive tasks)'—are presented as comparative facts, but no experiment or quantitative model in the paper justifies them. A comparison table requires either measurements or a clearly stated analytical derivation; neither is provided. In particular, 'Exceeds human speed' is a quantitative claim with no supporting data.","section":"Table 2"},{"comment":"The state-transition-diagram formalism is introduced as a foundation for the replay guarantee, but it is underspecified: the paper does not define when a check function is complete with respect to a state-transition diagram, nor does it give an algorithm or proof that the replay phase produces a valid trajectory. Section 4.3 similarly describes selecting 'the lowest-level experience that still maintains the highest success rate' without defining a selection criterion. These omissions make the core replay mechanism difficult to assess or reproduce.","section":"Section 3.4"}],"minor_comments":[{"comment":"In the paragraph comparing AgentRR with traditional R&R, the word 'Experienc' should be 'Experience'.","section":"Section 2.3"},{"comment":"The figure contains the typo 'informantion'; it should be 'information'.","section":"Figure 2"},{"comment":"In the Summary Phase description, 'Vaild' should be 'Valid'.","section":"Section 5"},{"comment":"Reference [75] is cited for RnR-Safe, but the listed title, 'Density-Aware Routing in Highly Dynamic DTNs: The RollerNet Case', appears unrelated to the cited security-hardening mechanism; please verify and correct this citation.","section":"References"},{"comment":"The row 'LLM Agent | Human' with primary use case 'Understanding agent behavior, teaching' is confusing; clarify whether the human is the replayer in the record-and-replay sense or the end user of the replayed experience.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript is a vision/position paper whose central Section 6 claim of demonstrated effectiveness is not backed by any evaluation, and whose Section 3.2.2 safety guarantee is contradicted by its own Section 6 limitations. I do not see a set of local revisions that would make the central claim defensible; a substantially new empirical study and a re-framing of the safety guarantee would be required. The conceptual taxonomy of record/replay modes could be valuable in a future position-paper venue, but it does not meet the bar for a research paper as submitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid vision paper, not a results paper. What's new is the packaging: multi-level experience abstraction (low-level precise actions vs high-level procedural plans) plus check functions as a TCB, and the record/replay matrix (who records, who replays) that opens up modes like large-model-record/small-model-replay and untrusted-record/trusted-replay. The authors cite the nearby work (MobileGPT, UFO2, Workflow Use, browser-use) and are honest that the direction exists; their contribution is the synthesis and the explicit design space. That's worth having on record.\n\nWhere it falls short: there is no implementation or measurement anywhere. The Section 6 claim that AgentRR 'effectively addresses' reliability, privacy, and performance is not supported by the qualitative case study. The check-function TCB is the load-bearing piece for reliability, but Section 3.2.2 only asserts that a user-audited check function 'guarantees' adherence to safety requirements; there is no construction, no completeness argument, and the paper itself concedes in Section 6 that 100% reliable replay is elusive and summarization can be incomplete. That internal tension means the central reliability claim is currently an act of faith. The novelty is incremental: record-and-replay for agents already exists; the specific combination is interesting but not empirically tested.\n\nWhat it does well: the honest limitations section, the state-transition framing, and the application modes. The authors are clear about what they haven't solved.\n\nWho this is for: anyone scoping research on agent reliability/efficiency, or looking for a framework to attack. If it crossed my desk as a full paper, I'd ask for experiments or a formal treatment of check functions before accepting. As a vision/position paper, it deserves a serious referee — the ideas are coherent and the open problems are real. I'd encourage you to engage with it if you're working in this area, but don't cite it as evidence that AgentRR works.","headline":"A coherent and honestly scoped vision paper for record-replay in LLM agents, but the reliability claims rest on an unconstructed check-function TCB and there is no empirical support.","tokens_in":22273,"tokens_out":2497,"would_cite":false,"duration_ms":26185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replay recorded runs to make LLM agents reliable","keywords":["record and replay","LLM agents","multi-level experience","check functions","trusted computing base","agent reliability","cost reduction","privacy"],"falsifier":"A replay session in which the agent performs an action that violates a user-defined safety rule (for example, submitting a payment without confirmation) and the check function does not flag it would directly refute the claim that check functions guarantee adherence to safety requirements throughout replay.","tokens_in":1334,"feed_emoji":"🤖","tokens_out":2297,"duration_ms":67286,"temperature":0.7,"pith_summary":"The paper argues that the main problems of LLM-based agents—unreliable outputs, privacy exposure, high cost, and slow execution—are inherent to letting a large model reason from scratch on every task. It proposes AgentRR, which records a proven interaction trace, summarizes it into a structured multi-level experience, and replays that experience on similar future tasks. Check functions act as a trusted boundary during replay, confining the agent to actions that stay within the validated experience. The paper claims this preserves the generalization of LLMs while adding the determinism and efficiency of a scripted procedure, and it outlines deployment modes such as a large model recording for a small local model to replay.","feed_headline":"Replay proven traces to make LLM agents reliable","feed_subtitle":"AgentRR records a successful run, distills it into multi-level experience, and reuses it under safety checks.","key_machinery":"The central mechanism is the multi-level experience: a summarized collection of trajectories from similar tasks, stored at low and high levels of abstraction, together with check functions that serve as the system's trusted computing base (TCB). Low-level experiences capture concrete action sequences such as clicks and API calls; high-level experiences capture abstract plans like 'select city, select hotel, choose dates, query.' Check functions verify execution-flow integrity, state preconditions, data and parameter constraints, and safety invariants during replay, thereby bounding the agent's behavior. The paper also models executions as state-transition diagrams, where experience reduces the space of trajectories the model must search.","core_discovery":"AgentRR's central claim is that agent intelligence and agent execution can be decoupled: the expensive, fallible reasoning that plans a task needs to happen only once, during recording, and can then be distilled into structured experience that a cheaper and more controlled replay process executes. The discovery is that experience must live at multiple levels of abstraction—low-level experiences preserve exact action sequences for fast, faithful replay, while high-level experiences capture procedural knowledge that a local model can adapt to new contexts—and that each level needs a check function as a trust anchor. The paper contends that this combination addresses reliability by bounding actions with validated traces, privacy by keeping raw data local during replay, cost by reducing large-model calls, and performance by making replay closer to scripted execution than to step-by-step reasoning.","pith_inferences":["If check functions are the practical trust anchor, the value of AgentRR will hinge on how much of a task's safety requirements can be expressed in them; a testable extension would measure how often hand-written checks miss edge cases that a recorded trace happens to avoid.","The multi-level idea suggests a continuum between RPA-style scripting and autonomous agents, so AgentRR could be compared against incremental improvements in prompt caching and workflow engines to see where the boundary of 'replay' and 'reasoning' actually lies.","Experiences could be used as fine-tuning data: after a small model replays a task successfully several times, the recorded experience might be distilled into the model's weights, gradually reducing even the local model's reliance on high-level reasoning.","The experience repository could become a learning system if replays that diverge from an experience are recorded and analyzed, refining the experience's check functions over time; the paper does not explore this feedback loop."],"forward_implications":["Agents can solve repetitive tasks at near-scripted speed and with greatly reduced LLM API cost, because most steps are replayed rather than re-reasoned.","Privacy improves because the replay phase can run locally with a small model, and raw user data need not be sent to a cloud LLM for planning.","Reliability improves because hallucinations and unrecoverable error states are caught by check functions that confine the agent to validated behavior.","The record-and-replay paradigm opens a design space where recording and replaying can be performed by different entities, such as a large model recording and a small edge model replaying, letting expensive planning be done once and executed many times.","An experience repository could let users share vetted task experiences, so that effective agents do not require every user to perform expensive reasoning from scratch."],"supporting_citations":[{"why":"Supplies the claim that LLMs will always hallucinate, the core reliability problem AgentRR is designed to bound.","marker":"[5]"},{"why":"Provides the classical record-and-replay mechanism from systems that AgentRR imports and adapts to agent tasks.","marker":"[62]"},{"why":"Playwright is the recording and replay tool used in the case study, representing the traditional R&R approach with high fidelity but limited generalization.","marker":"[52]"},{"why":"Chrome Recorder is used to capture user interaction traces in the case study, grounding the record phase in an existing tool.","marker":"[26]"},{"why":"MobileGPT is an earlier LLM-agent R&R system with hierarchical memory, a direct precursor that AgentRR generalizes into multi-level experience.","marker":"[88]"},{"why":"UFO2's agent operating system with record-and-replay is compared as a related but less general approach, positioning AgentRR's contribution.","marker":"[86]"},{"why":"Workflow Use applies record and replay to browser agents but focuses on identical workflow scripts, contrasting with AgentRR's multi-level experience.","marker":"[1]"}],"fun_headline_variants":["AgentRR: record, distill, replay for reliable agents","Record once, replay many: LLM agents get cheaper","Decouple reasoning from execution with AgentRR","Replay experiences to fix LLM agent reliability","Record & replay: a new paradigm for LLM agents"],"cache_read_input_tokens":24448,"weakest_assumption_plain":"The load-bearing assumption is that the check function can act as a complete and trustworthy safety boundary during replay, catching every illegal action, bad precondition, and unsafe parameter.","fun_headline_variants_meta":{"raw":{"variants":["AgentRR: record, distill, replay for reliable agents","Record once, replay many: LLM agents get cheaper","Decouple reasoning from execution with AgentRR","Replay experiences to fix LLM agent reliability","Record & replay: a new paradigm for LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1181,"prompt_tokens":964,"completion_tokens":217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":155}},"tokens_in":580,"tokens_out":217,"duration_ms":2576,"temperature":1.0,"reasoning_tokens":155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:41:03.768109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replay session in which the agent performs an action that violates a user-defined safety rule (for example, submitting a payment without confirmation) and the check function does not flag it would directly refute the claim that check functions guarantee adherence to safety requirements throughout replay.","supporting_citations":[],"review_version":1}