{"id":"d2839b1a-c155-4b93-b8c1-3fe2cb7946fa","arxiv_id":"2607.20709","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A Python object-oriented agent framework (NOOA) lets methods, state, docstrings and type annotations serve as the LLM interface, and achieves strong results on agentic benchmarks.","lead":"This paper presents NOOA, a Python framework where an AI agent is a regular Python object: methods are tools, fields are state, docstrings are prompts. It reports that current models use the interface fluently and beat several open harnesses on SWE-bench, Terminal-Bench, and ARC-AGI-3.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ARC-AGI-3 '6.4× harness effect' is not a controlled comparison: the 13.3% baseline is ARC Prize's raw-model eval with different budgets, and no non-OO harness with the same skill is tested, so the interface-specific claim is unproven.","rationale":"The paper is an honest engineering contribution with strong supporting evidence: the capability suite (4,400 records, 97.9%) is broad, the memory ablation (+11.8 RHAE) is internally controlled, and the appendices document containment and reproduction. My concern is not with the existence of an effect but with its attribution. The central claim — that the agent-as-Python-object surface itself lets a compact agent replace multi-component stacks — is most strongly advanced by the ARC-AGI-3 result, yet that result is the least controlled. The 13.3% baseline is not another harness with the same skill; it is a raw-model evaluation with different budgets (footnote 2). The skill content (world-model encoding, predict/retrodiction, search) is substantial and could plausibly account for most of the gap in any interactive harness. Without a non-OO harness running the identical skill under identical constraints, we cannot distinguish interface value from skill/harness-loop value. The reader's weakest assumption identifies the same apples-to-apples problem and the missing uncertainty quantification in Tables 3–5; I agree. The appropriate disposition remains conditional — the paper should not be rejected, but the strongest empirical pillar needs the matched comparison before the interface-specific claim is accepted.","tokens_in":39358,"tokens_out":9375,"duration_ms":74141,"concrete_test":"Run the same 50-line ARC-AGI-3 world-model skill in smolagents (a non-OO CodeAct harness) under the same GPT-5.6-sol backend, same 25 games, same two-hour per-game wall-clock cap, and same per-game dollar budget, with NOOA memory disabled in both arms. If the non-OO harness reaches RHAE within a few points of 85.1%, the OO surface is not the driver of the claimed harness effect; if it stays near the 13–50% range, the interface-specific claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4's headline result — 85.1% vs. 13.3% RHAE, called a '6.4× harness effect' — carries the paper's strongest claim that the OO surface advances the Pareto frontier. As footnote 2 concedes, the comparison is against ARC Prize's raw GPT-5.6-sol eval with different budgets, not against an agent harness. A raw model is not an agent loop, so this conflates 'using a harness at all' with 'using this OO harness.' The other curves in Figure 7 are all NOOA variants (baseline skill, markdown-file memory), so none isolates the OO interface from the 50-line world-model skill or the memory subsystem. The same issue appears in Tables 3–5: single runs without repeated-seed intervals, leaving the 3–5 point leads over OpenCode/PI within plausible noise. The paper is transparent about the budget mismatch, which is a credit, but the result that would prove the interface itself is the cause of the gain is missing. A matched run of the same skill in a non-OO CodeAct harness (e.g., smolagents) or a JSON-tool harness would settle it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"NOOA is a Python agent framework built on the idea that an agent is a Python object: methods are actions, fields are state, docstrings are prompts, and type annotations are contracts. Methods whose body is '...' are executed by an LLM loop under Predict or CodeAct strategies; context, events, and an optional long-term memory subsystem are exposed as Python APIs. The paper's contributions are the programming model, a claim to be the first to combine six model-facing capabilities on one surface, and an empirical evaluation: 97.9% on 88 self-authored capability tests (4,400 records across 10 models), 82.2% on SWE-bench Verified with GPT-5.5 xhigh, 73.0% on Terminal-Bench 2.0, 86.8% on CyberGym L1, and an ARC-AGI-3 fleet at 85.1% RHAE vs 13.3% for raw GPT-5.6-sol, described as a '6.4x harness effect'. A 14-framework comparison and detailed appendices on harness scoring, memory design, and the ARC containment audit are included.","tokens_in":39618,"tokens_out":7670,"duration_ms":63978,"significance":"The paper is a serious systems contribution with a substantial and largely transparent evaluation on the capability side. If the benchmark claims held, the contribution would be important: a single compact object-oriented surface would let developers and models share one programming model while matching or beating specialized agent stacks. The paper deserves credit for shipping the source, pinning framework snapshots, reporting 4,400 capability records, and including a red-team audit for the ARC runs. However, the strongest interface-specific claims — the '6.4x harness effect' and the Pareto-frontier advantage over OpenCode/PI — rest on comparisons that are either not controlled (ARC baseline) or reported without uncertainty (SWE-bench, Terminal-Bench, CyberGym). The central empirical case is defensible but needs revision before those stronger claims can be accepted.","major_comments":[{"comment":"The headline '6.4x harness effect' (85.1% vs 13.3% RHAE) is not a controlled comparison. The 13.3% baseline is ARC Prize's raw-model evaluation, which has no agent loop, no tools, and no skill; footnote 2 concedes that evaluation budgets differ. The other curves in Fig. 7 are all NOOA variants, so none isolates the OO interface from the 50-line world-model skill or the memory subsystem. As reported, the result shows that a harness plus this skill plus memory reaches 85.1%, not that the OO interface itself advances the Pareto frontier. A matched run of the same skill in a non-OO CodeAct harness (e.g., smolagents) or in a JSON-tool harness is needed; without it, the paper's strongest interface-specific claim is not established.","section":"Sec 4.4, Fig. 7, footnote 2"},{"comment":"Benchmark scores are reported as single point estimates with no confidence intervals or repeated-seed variance. On SWE-bench (Table 3) NOOA's advantage over OpenCode/PI is 3-5 points; on Terminal-Bench (Table 4) it ranges from -2.3 to +12.3 points depending on configuration; the ARC ablations (Fig. 7) are single 25-game fleets. Known run-to-run variability on these benchmarks is comparable to or larger than several of these gaps, so the claims that NOOA 'defines most of the observed accuracy-cost frontier' and that memory contributes +11.8 RHAE points are not yet supported at the reported precision. Please provide repeated runs (at least 3-5 seeds or fleets) with intervals for the key comparisons, or explicitly state that only single runs were performed and soften the frontier claims.","section":"Sec 4.2-4.4, Tables 3-5, Fig. 6-7"},{"comment":"The stress-test analysis rests on 300 records total, with only 20 per cell for the small/efficient group (e.g., sentiment_batch 8/20, refinement 11/20). The conclusion that large models show only intermittent failures while small models exhibit systematic 0/5 failures is based on very few observations per model family. The qualitative reading of Appendix B is plausible and well illustrated, but the quantitative claims ('scale gap widens to 23 points', '12.5% of small-model stress pairs at 0/5') should be accompanied by uncertainty estimates or additional runs before being used to support design conclusions.","section":"Sec 4.1, Table 2"}],"minor_comments":[{"comment":"The duplicate values in the GPT-5.5 row (73.0 for both high and xhigh) and the Opus row (64.0 and 65.2) should be verified; if deliberate, a note would avoid the appearance of a formatting artifact.","section":"Table 4"},{"comment":"The 'MDASH' placeholders in system names (e.g., 'Microsoft MDASHv2', 'MDASHunknown') are formatting artifacts and should be replaced with actual em-dashes or labels.","section":"Table 5"},{"comment":"The repository link '/githubnvidia-nemo/labs-OO-Agents' lacks a URL scheme; it should be 'https://github.com/nvidia-nemo/labs-OO-Agents' or similar.","section":"Header before abstract"},{"comment":"The phrase 'completely remove the interface friction' overstates the evidence: the stress-test pass rate is 84.7%, and even frontier models show intermittent failures. Suggest softening to 'substantially reduce interface friction'.","section":"Sec 4.1, final sentence"},{"comment":"The claim that models perform well 'despite never being trained on it' cannot be verified from the paper; models are trained on Python, CodeAct patterns, and similar tool-use formats. Suggest rephrasing to 'not trained on this specific interface' and pointing to the zero-shot capability results as evidence.","section":"Sec 7, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The ARC comparison should be reframed as an ablation of the full system rather than a 'harness effect'; the phrase in Sec 4.4 is otherwise likely to be read as an interface-specific claim the paper does not yet support. No concerns about attribution or citation ethics; the companion-work connections are explicit. The capability suite and framework comparison are strong enough that the paper is worth a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NOOA is a real idea and the paper is mostly as good as it looks: an agent is a Python object, methods are actions, fields are state, docstrings are prompts, annotations are contracts. The capability evidence is solid -- 88 tests, ten models, five runs each, 97.9% overall -- and the stress-test breakdown (large models fail intermittently, small models fail outright) is a genuinely useful observation, not a dashboard. The memory ablation on ARC-AGI-3 (+11.8 RHAE over markdown files) is a controlled within-fleet comparison, and the red-team audit of the runs is a nice, rare touch. The survey of fourteen harnesses is careful and reproducible: pinned commits, explicit rubrics, and they credit the obvious precedents (DSPy, CodeAct, MemGPT, smolagents) without pretending the pieces are new. The code and appendices are shipped.\n\nThe soft spot is the headline ARC-AGI-3 number. Calling 85.1% vs 13.3% a '6.4x harness effect' is not supported: the 13.3% is ARC Prize's raw-model eval with different budgets, not a non-OO harness with the same world-model skill. The paper admits this in a footnote, which is a credit, but the framing still overclaims. As it stands, the ARC result shows a full NOOA agent plus skill plus memory does very well; it does not isolate the OO interface. A matched run in a non-OO CodeAct harness (smolagents) or a JSON-tool harness would settle it, and I'd want to see that before believing the interface itself is the cause. The SWE-bench and Terminal-Bench tables also lack confidence intervals; the 3-5 point leads over OpenCode and PI could be noise, though the token-efficiency differences are more concrete. The 'first to combine' claim is a self-assessed survey, but the pinned-snapshot method makes it auditable.\n\nBottom line: I agree with the conditional verdict. The central design is credible, the paper is honest about many limitations, and the engineering is substantial. This deserves a serious referee -- someone who will push on the ARC comparison, ask for matched-budget or cross-harness runs, and request intervals on the benchmark tables. I'd accept it with that expectation.","headline":"Genuine engineering with solid capability evidence; the ARC-AGI-3 '6.4x harness effect' is an uncontrolled comparison and should be reframed before publication.","tokens_in":40203,"tokens_out":4478,"would_cite":true,"duration_ms":36016,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims an agent is a Python object—methods as actions, docstrings as prompts, type annotations as contracts—and shows current models use it fluently enough to replace multi-component agent stacks without losing benchmark performan","keywords":["agent-as-a-python-object","object-oriented agents","code-as-action","pass-by-reference","typed contracts","agentic loops","long-term memory","ARC-AGI-3"],"falsifier":"Rerun the same 25 ARC-AGI-3 games for the raw frontier model at the NOOA fleet's exact budget—same 2-hour wall clock, same tokens for input/output/cache, same per-turn retry allowance—and record RHAE; if the raw model approaches 85.1%, the 'harness effect' collapses. Separately, run SWE-bench and Terminal-Bench with 5–10 repeated seeds per harness and report confidence intervals; if the NOOA margin over the two open general-purpose agents falls inside the overlap, the reported advantage is sampling noise.","tokens_in":39202,"feed_emoji":"🐍","tokens_out":7235,"duration_ms":54960,"temperature":0.7,"pith_summary":"This paper tries to establish that agent development has overcomplicated itself: instead of prompt templates, tool schemas, callback stacks, and workflow graphs, an agent can be a single Python class. The paper's central claim is that language models already know this language, so the same object can serve as source code, prompt, type contract, tool interface, and state boundary. The authors support it with three results: 97.9% average pass rate on 88 targeted capability tests across ten models, 82.2% on SWE-bench Verified with a frontier model, and 85.1% on the interactive reasoning benchmark ARC-AGI-3 with a single agent plus a 50-line skill, compressing a previous multi-agent world-model system. If right, this removes the learning curve of agent frameworks and makes agent behavior testable, traceable, and refactorable like ordinary software.","feed_headline":"One Python object is the whole agent","feed_subtitle":"Methods are actions, docstrings are prompts, type hints are contracts — and models score 97.9% on 88 capability tests.","key_machinery":"The load-bearing object is the Agent class itself, whose two built-in strategies define the loop: PredictStrategy makes a single typed LLM call with output validation and local retry, and CodeActStrategy runs an iterative Python REPL in which the model writes and executes code, inspects live objects, and must finish with a validated return. Around this, bounded-variable previews implement pass-by-reference at scale, ContextManager and EventManager expose static/dynamic context and typed event history as model-callable APIs, and the optional MemoryManager lets the agent curate its own SQLite store, retrieved by a relevance/recency/importance activation score. The claim is that these six model","core_discovery":"The discovery is the agent-as-a-Python-object programming model. A class mixes deterministic methods, whose bodies run as normal Python, with 'agentic' methods, whose ellipsis bodies are completed at runtime by an LLM-driven loop. The signature supplies typed inputs and a validated return; the docstring becomes the prompt; methods and state on self become callable tools; and large arguments are passed by reference with bounded previews instead of being serialized into context. On this surface the paper reports that current models are already fluent—97.9% on the capability suite, with failures concentrated in long-horizon stress cases rather than interface understanding—and that this ordinary","pith_inferences":["The 6.4x ARC-AGI-3 harness effect is computed against a raw-model evaluation with different budgets; a matched-budget rerun (same wall-clock, tokens, and turn allowance) would tell whether the interface itself, rather than the evaluation scale, drives the gain.","The stress-test failure pattern—models transcribing results instead of returning the live variable—points to a fixable mismatch between model habits and the interface; trajectory-level RL or preference data over this surface could plausibly close most of the remaining 2% capability gap.","If self-curated SQLite memory generalizes beyond ARC-AGI-3, it opens a route to cross-task transfer for frozen-weight agents, effectively accumulating skill without parameter updates—an implication the paper gestures at but does not establish.","The six-capability taxonomy could serve as a neutral checklist for comparing future agent harnesses, since the paper's own survey shows the field converging on these features piecemeal."],"forward_implications":["Agent behavior becomes unit-testable, traceable, refactorable software: docstrings, signatures, and deterministic helpers replace prompt engineering, so the same CI practices apply to agents.","Scaling past the context window: because arguments are live objects with bounded previews, the prompt carries fixed-size views while the agent processes full multi-million-row tables in code.","Fewer false completions: validated typed returns turn termination into a programmatic contract, reducing premature 'done' responses in interactive benchmarks.","The ARC-AGI-3 compression result suggests multi-agent orchestration can sometimes be replaced by a single agent with a skill and memory, changing how world-model systems are built."],"fun_headline_variants":["Agent as a Python object: methods are actions, docstrings are prompts","LLM completes dotted methods: Python object as agent loop","Typed contracts, live objects: agent-as-Python-object model","97.9% on tests: agents that are just Python objects","From prompt templates to Python: object-oriented agents"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline ARC-AGI-3 comparison pits the NOOA agent at a large interactive budget against the raw model at a different, smaller evaluation budget, so if unequal budgets explain the 6.4x gap, the claim that the interface itself advances the score–cost Pareto frontier does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Agent as a Python object: methods are actions, docstrings are prompts","LLM completes dotted methods: Python object as agent loop","Typed contracts, live objects: agent-as-Python-object model","97.9% on tests: agents that are just Python objects","From prompt templates to Python: object-oriented agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000899,"raw_usage":{"total_tokens":3767,"prompt_tokens":862,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2819}},"tokens_in":606,"tokens_out":2905,"duration_ms":17165,"temperature":1.0,"reasoning_tokens":2819,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:34:42.103842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same 25 ARC-AGI-3 games for the raw frontier model at the NOOA fleet's exact budget—same 2-hour wall clock, same tokens for input/output/cache, same per-turn retry allowance—and record RHAE; if the raw model approaches 85.1%, the 'harness effect' collapses. Separately, run SWE-bench and Terminal-Bench with 5–10 repeated seeds per harness and report confidence intervals; if the NOOA margin over the two open general-purpose agents falls inside the overlap, the reported advantage is sampling noise.","supporting_citations":[],"review_version":1}