{"id":"cedafc31-5a8d-4a60-944a-7118a551fc8a","arxiv_id":"2607.28629","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A review-plus-demo claiming agentic capabilities emerge from system integration, backed by a 15-task benchmark whose C1→C3 performance gap is largely built into the test design.","lead":"This paper pairs a review of agentic AI architecture with a small benchmark experiment using the Ollama LLM server and the OpenClaw agent runtime. It reports that adding tools and memory improves task success, but the experiment's design makes most of that improvement a foregone conclusion.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The C1 baseline is denied file content that C2/C3 receive via the harness, so the reactive-task improvement is an input artifact, not evidence for OpenClaw.","rationale":"The reader's weakest_assumption correctly identifies the central experimental confound: C1 is denied file content that C2/C3 receive through the harness, and C2 wipes memory between paired store/recall tasks. This makes the claimed monotonic improvement a designed property of the protocol rather than evidence of architectural emergence. My reading of §§4.1–4.4 confirms this precisely. The paper's own description in §4.1 and §4.2 makes the asymmetry explicit, so the concern is not an uncharitable interpretation. I also note the paper's internal contradiction: the abstract and conclusion claim \"performance improving consistently as architectural complexity increases,\" while Table 4 shows tool-use accuracy decreasing from C2 to C3 for both models. This strengthens the case that the headline result is fragile. The proposed concrete test would settle the issue: if giving C1 the same document text removes the reactive gap, the experiment cannot support the architectural attribution. I concur with the reader's REJECT verdict; no adjustment is needed.","tokens_in":32350,"tokens_out":2187,"duration_ms":22424,"concrete_test":"Re-run the full protocol with a variant C1* that receives the identical file-content-injected prompt text used in C2/C3, while still giving the raw LLM no tools and no memory. If C1* reactive success rises to 1.000 (and overall success rises proportionally), the C1→C2 reactive improvement is purely an artifact of input enrichment. As a secondary check, run C2 without the per-task stateless reset; if T10 then succeeds, the C2 persistent-memory failure is an artifact of the reset policy, not of missing memory architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — strict monotonic success C1<C2<C3, with capabilities \"emerging from system-level integration\" — rests on an unfair comparison. In §4.1, C1 is defined as receiving \"the raw relative path string\" for local file references, while \"the evaluation harness applies file-content enrichment only when the OpenClaw orchestration layer is active (C2 and C3).\" In §4.2, reactive tasks explicitly state: \"Under C1, the model receives only the raw file path as a string; ... Under C2 and C3, the harness injects the full document text into the prompt prior to submission.\" Thus the reactive-category jump from 0.000 (C1) to 1.000 (C2/C3) is guaranteed by the difference in input content, not by any capability of OpenClaw. A raw Chat-GPT-style prompt with the document text pasted in would likely also answer the three reactive questions. Similarly, the persistent-category gap is partly designed: under C2, the harness performs a \"full stateless reset\" between every task, so T10 (recall of the alias stored in T9) cannot succeed regardless of the model or runtime. The paper itself calls this a \"controlled memory-isolation negative test,\" but then uses the resulting low C2 score as evidence that only C3's persistent memory enables recall. This is circular. The claim of \"consistently improving\" as complexity increases is also contradicted by the paper's own Table 4: Qwen tool accuracy drops from 0.944 (C2) to 0.833 (C3), and Gemma tool accuracy drops from 1.000 to 0.944. The architectural thesis may be true, but this experiment does not test it because the configurations differ in input information and memory-reset policy, not only in architectural layer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a layered architecture for Agentic AI in which Ollama provides the LLM inference layer and OpenClaw provides the agent runtime layer (orchestration, tools, memory). After a broad architectural review, it presents a prototype evaluation with three configurations: C1 (Ollama-only), C2 (OpenClaw stateless), and C3 (OpenClaw with persistent memory). The central empirical claim is a strictly monotonic improvement C1 < C2 < C3 in task success rate across two open-weight models on 15 tasks, taken as evidence that autonomous capabilities such as tool use and persistent memory emerge from system-level integration rather than from the model alone.","tokens_in":32781,"tokens_out":4782,"duration_ms":49936,"significance":"If the empirical claim were sound, the paper would provide a useful, reproducible demonstration of how an agent runtime plus persistent memory can be layered over a local LLM. The released code, task definitions, and raw logs are a strength, as is the attempt to compare two different base models. However, the experimental protocol confounds the contribution of the architecture with changes in the input content and with the deliberate wiping of memory in C2. The headline monotonicity is therefore partly encoded in the configuration definitions rather than discovered. The surrounding survey is competent but does not by itself establish the paper's primary conclusion. The tool-use sub-metric also contradicts the claimed 'consistent' improvement, as accuracy drops from C2 to C3 for both models. The central empirical contribution needs a substantially redesigned, properly controlled comparison before it can support the stated conclusions.","major_comments":[{"comment":"C1 receives only the raw relative path string for reactive_notes.md, while C2 and C3 receive the full document text injected by the evaluation harness. The reactive-category jump from 0.000 (C1) to 1.000 (C2/C3) is therefore guaranteed by the input-enrichment policy, not by anything OpenClaw does. A fair baseline for isolating the orchestration layer would provide the same file content to C1 via the prompt. As written, the paper's conclusion that 'reactive capability emerges from the enriched prompt pipeline that the OpenClaw runtime enables' overstates the result: the enrichment is a harness action, and a raw LLM prompted with the pasted text would likely also solve the three tasks.","section":"§4.1, §4.2 (reactive tasks)"},{"comment":"In C2 the harness performs a 'full stateless reset' between each task, so the paired store/recall tasks T9 and T10 cannot both succeed: T10's alias recall is guaranteed to fail regardless of model, runtime, or tool access. The paper itself calls this a 'controlled memory-isolation negative test,' yet it then uses this designed failure to argue that persistent memory is necessary. This is circular for the C2→C3 comparison. The design makes the C2 persistent-category score artificially low, so the observed ordering C1<C2<C3 in the persistent category is not empirical evidence of the architecture's contribution.","section":"§4.2, §4.4 (persistent tasks, C2 reset)"},{"comment":"Tool-use accuracy decreases from C2 to C3 for both models: Qwen3.5:4b goes from 0.944 to 0.833 and Gemma4:e4b from 1.000 to 0.944. This contradicts the abstract's claim of 'performance improving consistently as architectural complexity increases' and Finding 1's 'strict C1<C2<C3 ordering'. The authors attribute the C3 drop to a T11 tool-selection routing issue, but that means the more complex configuration is less reliable on tool use, so the per-category data do not support a strict monotonic architectural benefit. The headline claim should be restricted to the overall success rate and honestly discussed as a trade-off rather than as uniform improvement.","section":"Table 4; §4.4; Finding 1"},{"comment":"All conclusions rest on n=3 repetitions, with no significance tests, confidence intervals (beyond means and standard deviations), or effect-size analysis. Given the small sample and the deterministic design failures discussed above, the assertion of a 'strict' ordering and especially the 'zero-variance perfection' for C3 memory recall needs a much stronger statistical and experimental foundation. At minimum, the number of independent runs and task instances must be large enough to distinguish architectural effects from stochastic variation and design artifacts.","section":"§4.3, Tables 4–5 (statistical basis)"}],"minor_comments":[{"comment":"The paper uses 'OpenClaw' and 'Openclaw' inconsistently. Also, several citations refer to arXiv:260x.xxxxx papers from 2026 that the reader cannot verify; the authors should confirm all references, especially the model documentation links, and include access dates consistently.","section":"Throughout"},{"comment":"The claim that the H100 81GB GPU is used for 4-bit 4B models is relevant, but the paper does not report GPU memory usage, model quantization details, or inference-time parameters beyond temperature and top-p. These details would help reproducibility.","section":"§4.1"},{"comment":"The reactive-category result is described as 'emerging from the enriched prompt pipeline that the OpenClaw runtime enables,' but the harness itself performs the enrichment. The wording should be corrected to avoid attributing a harness action to OpenClaw.","section":"§4.4"},{"comment":"The figure shows only the overall success rate; given that tool-use accuracy drops from C2 to C3, a figure with per-category error bars and individual task outcomes would be more informative and less likely to be read as supporting uniform monotonic improvement.","section":"Figure 8"},{"comment":"The future-directions discussion is useful but largely speculative and not tied to the experimental results. Consider trimming or grounding it in the specific failure modes identified in the trace logs.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The empirical protocol's confounds are severe enough that the current results do not support the paper's central claim. The authors can redesign the experiment to hold input content constant across configurations and to avoid the designed failure of C2 persistent recall, then re-run and report. Because the architectural taxonomy and the released tooling are valuable, I do not recommend rejection outright. However, the revised paper must either present a properly controlled comparison or substantially weaken the 'consistent monotonic improvement' claim. The discrepancy between the abstract's 'consistently' and Table 4's tool-use decline should also be addressed head-on."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical artifact — a 15-task benchmark for the OpenClaw+Ollama stack, with code and data released, run on two 4B models — is genuinely new and the paper is transparent about its protocol. Second, the headline claim that capabilities \"emerge from system-level integration\" with \"performance improving consistently\" is not supported by the experiment as designed, and the paper's own tables show it.\n\nWhat is actually good: the task suite is sensible (reactive, tool-use, persistent memory), the scoring criteria are explicit, and per-category results are reported with standard deviations. The paper openly acknowledges the tool-accuracy drop from C2 to C3 (0.944→0.833 for Qwen, 1.000→0.944 for Gemma in Table 4) and explains the T4/T11 failures. That is honest reporting.\n\nThe soft spots are in the experimental logic. C1 receives only an unresolvable path for reactive tasks, while C2/C3 get the full document text injected by the harness. So the reactive 0→1 jump is an input artifact, not evidence for OpenClaw. Similarly, C2 deliberately wipes all memory between tasks, so the paired store/recall task T10 cannot possibly succeed under C2; using that as evidence that only C3 enables recall is circular. The paper is upfront about these choices — it calls them \"controlled\" negative tests — but the central monotonicity claim is largely encoded in the configuration definitions. The abstract's \"consistently improving\" is also contradicted by the tool-use metrics within the same table.\n\nMinor issues: n=3 with no significance tests, a placeholder \"Figure Suggestion\" left in Section 5.1, and an arXiv ID versus submission date mismatch. None are fatal alone, but they add to the impression of an unfinished draft.\n\nThe architectural thesis — that tool use and persistent memory matter — is probably true and widely accepted. But this experiment does not test it, because the configurations differ in input information and memory-reset policy, not only in architectural layer. The benchmark itself could be reused with a fairer C1 baseline (file content without orchestration) and a C2 that preserves memory between store/recall pairs; then the comparison would actually isolate layers.\n\nWho this is for: researchers working on agentic evaluation will find the task definitions and released artifacts useful, but the conclusions should be read skeptically. I would not cite the central result, but I would send the paper to a serious referee — the benchmark and the transparency deserve constructive feedback, and a major revision could turn this into a modest systems paper.\n\nRecommendation: send to peer review, expecting heavy revision.","headline":"The new 15-task benchmark and released artifacts are real, but the central claim of monotonic improvement is largely designed into the protocol rather than demonstrated by it.","tokens_in":33266,"tokens_out":2845,"would_cite":false,"duration_ms":28021,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that autonomous-agent capabilities—tool use, file access, and cross-task recall—emerge from the layered integration of a local inference model and a persistent agent runtime, not from the language model alone.","keywords":["Agentic AI","layered architecture","persistent memory","tool use","agent runtime","Ollama","OpenClaw","system-level evaluation"],"falsifier":"Re-run the 15-task benchmark with file-content enrichment applied to the C1 baseline as well; if reactive-task success rises from 0.000 to 1.000 and C1 overall success approaches or exceeds C2's, the reported C1<C2 jump is an artifact of prompt enrichment. Likewise, if removing the stateless reset in C2 makes T10 recall succeed, then 'persistent' capability is not uniquely attributable to the C3 memory layer.","tokens_in":1586,"feed_emoji":"🤖","tokens_out":3945,"duration_ms":92620,"temperature":0.7,"pith_summary":"This paper argues that 'agentic' behavior is a property of the full system architecture, not of the language model alone. To support that, it runs a 15-task benchmark in three configurations: a raw local LLM, the same LLM behind an agent runtime with tools but no memory, and the full runtime with persistent on-disk memory. Both of two 4-billion-parameter models follow the same strictly monotonic path in overall task success, from 0.467 to roughly 0.91–0.96 to roughly 0.93–0.98, with perfect memory recall in the full configuration. The authors read this as evidence that each architectural layer contributes independently and that orchestration and memory layers largely erase model-level differences. A reader should care because the result shifts design attention from model size to runtime and memory engineering, and argues for system-level evaluation rather than model-only benchmarks.","feed_headline":"Adding runtime and memory lifts agent success from 47% to 98%","feed_subtitle":"Two small local models improve the same way, suggesting agentic skill is architecture, not model size.","key_machinery":"The load-bearing comparison is a three-configuration experiment. C1 is Ollama, a local LLM inference server, queried directly with no tools and no memory. C2 routes the same model through OpenClaw, an agent runtime that orchestrates reasoning, tool invocation, and memory, but wipes session files, the SQLite store, and workspace memory after every task. C3 is identical to C2 except that the on-disk memory layer accumulates, enabling long-horizon recall. The 15-task benchmark covers reactive document lookup, tool use (CSV arithmetic, file writing, API calls, cron), and three store–recall pairs plus memory probes. The central identity is the strict ordering C1 < C2 < C3 in overall task success,","core_discovery":"The central discovery is a strictly monotonic capability ordering across three configurations: raw local LLM only (C1) gives 0.467 success for both models; adding an OpenClaw runtime with tools but wiping memory between tasks (C2) raises it to 0.911 and 0.955; enabling persistent memory (C3) raises it to 0.933 and 0.978. The full-stack configuration achieves memory recall of 1.000 ± 0.000 across all six store–recall tasks, including correct override of a stale stored alias. Identical C1 baselines and near-convergence at C3 indicate an architectural rather than model-specific gradient. Tool-use accuracy dips slightly from C2 to C3, attributed to tool-selection disambiguation.","pith_inferences":["Beyond the paper, the same architecture claim could be stress-tested by keeping C1's prompts identical to C2/C3 (injecting document text) and adding a much larger cloud model as a fourth configuration; if the gradient compresses or reverses, model capability still matters at scale.","Beyond the paper, the security discussion implies a testable failure mode: a single adversarial prompt stored in persistent memory could poison all later tasks under C3 but not under C2, which would make memory a liability as well as a capability.","Beyond the paper, the task suite is small (15 tasks, 3 repetitions) and the latency comparison mixes tool-bound tasks with memory-bound tasks; a scaled version with longer horizons and open-ended goals would show whether the monotonic ordering persists.","Beyond the paper, the slight C3 dip in tool-use accuracy suggests a concrete optimization target: better tool-selection disambiguation in the orchestration layer, which the paper identifies but does not solve."],"forward_implications":["If the ordering holds, agentic capability is an emergent property of the layered stack: the same model behaves reactively alone and agentically once an orchestration layer and persistent memory are added.","Different small models converge to similar full-stack performance, so architecture and memory design can matter more than which open-weight model is chosen, at least at this task complexity.","Persistent memory makes cross-task recall deterministic in this benchmark (1.000 ± 0.000), including correct overwrite of stale values, implying on-disk memory is a reliable substrate for long-horizon behavior.","The measured 11–13x orchestration latency and its partial offset by accumulated memory imply that agentic systems trade latency for capability, and that memory context can reduce reasoning iterations over time.","Evaluation of agentic systems should move from model-centric benchmarks to system-level metrics—task success, tool accuracy, memory accuracy, latency, reasoning depth—as the paper's protocol demonstrates."],"fun_headline_variants":["Architecture, not model size, drives agent success from 47% to 98%","OpenClaw+Ollama: runtime and memory boost agent success to 98%","Agentic skill emerges from system integration, not model size","Full-stack agent architecture lifts success from 47% to 98%","Memory and tools turn 47% agent success into 98%"],"cache_read_input_tokens":34432,"weakest_assumption_plain":"The ordering rests on the harness design: under C1 the reactive prompts receive an unresolvable file path while C2/C3 receive the document text, and under C2 memory is wiped between every paired store and recall task, so the measured gradient partly reflects what the experiment intentionally withholds rather than what the runtime adds.","fun_headline_variants_meta":{"raw":{"variants":["Architecture, not model size, drives agent success from 47% to 98%","OpenClaw+Ollama: runtime and memory boost agent success to 98%","Agentic skill emerges from system integration, not model size","Full-stack agent architecture lifts success from 47% to 98%","Memory and tools turn 47% agent success into 98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1165,"prompt_tokens":838,"completion_tokens":327,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":582,"tokens_out":327,"duration_ms":3625,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:51:36.742391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 15-task benchmark with file-content enrichment applied to the C1 baseline as well; if reactive-task success rises from 0.000 to 1.000 and C1 overall success approaches or exceeds C2's, the reported C1<C2 jump is an artifact of prompt enrichment. Likewise, if removing the stateless reset in C2 makes T10 recall succeed, then 'persistent' capability is not uniquely attributable to the C3 memory layer.","supporting_citations":[],"review_version":1}