Pith.

Artificial Intelligence · arXiv:2610.22289 v3 · submitted 15 Oct 2026

Improving Environment-Agnostic Agents: The Role of Scaffolding

Sergey Rodionov

The paper says

“Agents built on stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering.”

more of the abstract

We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. The reference prescribes organization and control rather than supplying new game rules or solutions. To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. This small change recovers part of the specialized agent's advantage. A complementary AGI Maze experiment shows that additional tutorial experience can reduce performance. Together, these findings highlight the distinction between having information and tools available and using them effectively. Supplied procedures still matter for the models tested here. The performance gains also come with higher token-based cost. The experiments do not separate better use of information and procedures from additional or differently allocated inference. We provide the EnvTraverse framework and an evaluated baseline agent for testing whether further environment-independent changes improve performance in some environments without degrading it in others.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Honest, well-scoped empirical paper with a real cross-environment prompt result, but the headline gain is partly entangled with extra recovery restarts and extra tokens, and the paper says so without quantifying it. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 4.2 (recovery supervisor); Sections 8.1-8.2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.69):

“The experiments do not separate better use of information and procedures from additional or differently allocated inference.”

Pith says: Objection 1 of 3 · major

Section 4.2 specifies that a controller exit after the stuck protocol triggers the recovery supervisor, which restarts the controller up to ten times per environment run. Because Stop-and-report is instructed to return control whenever it is uncertain, it systematically enters the stuck-exit-recovery path, whereas Baseline, told to continue working, rarely does.

more of objection 1

The two arms therefore differ not only in the text of continuation_string.txt but in the expected number of sessions per run, with the treatment arm able to receive up to ten additional 'keep working' restarts. The observed ARC-AGI-3 gains (1.37-7.29 points) and AGI Maze gains (0.37-4.95 points) come with 2-52% and 7-39% higher token costs, a pattern consistent with the gains being carried by extra inference or restart opportunities rather than by the instruction's content. The paper discloses the mechanism and disclaims the broader information-use/inference separation (Sections 9.2 and 10), and I do not read this as concealment; however, the exposure is never quantified: no per-arm counts of normal continuations, stuck-protocol exits, or recovery events appear in the tables, the appendix, or the reported artifacts, so the reader cannot tell how much of the headline contrast flows through the recovery channel. Because the abstract's claim that 'changing only an instruction... improves mean scores' is the paper's central empirical contribution, this unquantified structural asymmetry is load-bearing. I recommend reporting stuck-protocol and recovery counts per arm and per repetition, and either an ablation with the recovery supervisor disabled or an analysis restricted to runs with zero recovery events; failing that, the causal wording should be revised to describe the effect of the instruction combined with the existing continuation and recovery machinery.

The paper says, at Section 7.1-7.2; Tables 1-2, 4-5

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

With only three block repetitions, several headline gains are small relative to the run-to-run variation visible in Appendix C, and the direction is not stable repetition-by-repetition. For ARC-AGI-3 at GPT-5.6-sol max (mean gain 1.37 points), the individual runs are Baseline 87.68/92.21/88.87 versus Stop-and-report 92.35/87.29/93.23 (Table 4): Stop-and-report is lower in the second repetition.

more of objection 2

For AGI Maze at GPT-5.5 high in the single setting, Stop-and-report is below Baseline in two of the three 0005 batches (52.93/51.91/58.24 versus 57.34/54.31/53.67, Table 5), and the AGI Maze evaluation sample itself is two arbitrarily chosen labyrinth identifiers (Section 7.1). The paper states plainly in Section 7.2 that contrasts are descriptive and do not establish significance, and this honesty is commendable; the difficulty is that the abstract and Section 8.2 nevertheless present the all-four and all-eight improvement pattern as the paper's cross-environment result. I ask for per-repetition paired differences and effect sizes for every configuration, additional repetitions for the small-gain cells, or a correspondingly softened statement of the claim. As written, the smallest reported gains are within the spread of the individual runs, and the cross-environment generalization rests on means of three.

The paper says, at Section 7.2 (Eq. 1); Appendix D

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

RHAE70 is an author-defined score whose reference factor was adopted during analysis after examining the results, informed by informal internal playthroughs of eight labyrinths by one practiced player (Appendix D). The paper discloses this and notes that the ordering is not invariant to the factor: RHAE80 slightly favors Baseline at GPT-5.5 xhigh with curriculum (67.14% versus 66.54%).

more of objection 3

Because Section 8.2's eight-comparison claim is the main evidence for improvement beyond ARC-AGI-3, a single sensitivity sentence is not sufficient: I recommend a systematic table of RHAE60/RHAE70/RHAE80 for all eight configurations, or a pre-specified justification of the 0.7 factor, so the reader can see how much of the cross-environment claim survives the metric choice. The disclosure is honest; what remains unresolved is the robustness of the headline claim to the chosen normalization.

4 minor objections

The paper says, at Equation (2)

Pith says: minor objection 1 of 4

Equation (2) is typeset ambiguously: C = 10^-6( N_cached/60 + Ninput - N_cached6 + N_output ). Please make the parenthesization explicit (presumably (N_input - N_cached)/6) and define all variables at first use.

The paper says, at Figure 3

Pith says: minor objection 2 of 4

Figure 3 omits the runtime-recovery supervisor, and the caption notes the omission; given the supervisor's role in interpreting the Stop-and-report results (Section 4.2), either include the recovery path in the diagram or add a pointer in the caption to the description of the recovery mechanism.

show 2 more

The paper says, at Table 2

Pith says: minor objection 3 of 4

In Table 2, RHAE70 and the x1-solved count can move in opposite directions (for example, GPT-5.5 high single: Baseline solves 54 targets within the original budget versus 50 for Stop-and-report, while RHAE70 still favors Stop-and-report because it credits efficiency within 70% of the reference budget); a sentence explaining the relationship between the two columns would prevent confusion.

The paper says, at Section 8.3

Pith says: minor objection 4 of 4

The run-level spread in Table 6 is comparable to the reported Terminal-Bench differences (for example, GPT-5.5 xhigh Stop-and-report runs of 79.78/79.78/87.64 versus Baseline runs of 85.39/82.02/80.90); a sentence noting this would help readers calibrate the summary that the changes are small and mixed.

What would settle it

Run an ablation of Stop-and-report with the recovery supervisor's post-stuck restart allowance set to zero (controller exits permanently after the second no-action return), at the reported settings, on the four ARC-AGI-3 configurations and the eight AGI Maze comparisons, with three blocks each and all other settings unchanged; compare mean RHAE/RHAE70 and costs to Tables 1 and 2. If the gains from Stop-and-report shrink to within the run-to-run spread or reverse, the reported benefit is carried by recovery restarts rather than by the instruction's content; if they persist, the causal attribution to the instruction survives.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 37 s · 28,616 tokens read, 11,237 written
  3. Editor wrote the verdict above 33 s · 29,597 tokens read, 3,292 written
  4. Referee major revision: 3 major objections, 4 minor 2 min 24 s · 28,918 tokens read, 18,588 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this deserves a serious referee, and the reader's conditional verdict is about right. The genuinely useful new things are the clean prompt contrast, the Controlled_RESET diagnostic, and the released EnvTraverse framework with run artifacts. The paper is unusually candid: it scopes claims, reports individual runs in appendices, describes the failure-replacement policy, and states plainly that information-use effects are not separated from inference effects. That candor is real and should be credited.

The biggest soft spot is the one the stress-test names, and it holds up on reading. Stop-and-report is designed to return control whenever Codex is uncertain. Every early return produces an additional continued session, and repeated non-action can trigger the supervisor's restart path, with up to ten recovery attempts per run. Baseline rarely enters that path. The treatment therefore differs not only in the text of continuation_string.txt but in the number of sessions, restarts, and tokens spent; costs are 2-52% higher on ARC-AGI-3 and 7-39% higher on AGI Maze. The paper acknowledges in Sections 9.2 and 10 that the effects are not separated, but the contribution of extra restart attempts is left unquantified. For the central causal reading, that is load-bearing, not a footnote.

Two smaller soft spots. First, there is no significance testing, and the smallest headline gains (1.37 points on ARC-AGI-3, 0.37 and 2.49 points on AGI Maze) sit inside the run-to-run spread visible in Appendix C. The authors call their contrasts descriptive, which is honest, but the abstract's "improves" will be read causally. Second, the AGI Maze headline depends on an author-chosen RHAE70 factor, selected after inspecting results, and the disclosed sensitivity check shows one reversal at RHAE80. That is disclosed too, but it caps how strongly the cross-environment claim can be stated.

What holds up: the monotone ordering Baseline < Stop-and-report < Controlled_RESET < twma_v1.61 in every ARC-AGI-3 configuration, the Controlled_RESET result recovering 34-60% of the specialization gap using a convention already in the environment description, and the careful all-or-nothing replacement policy. Those are solid descriptive findings. The paper does not pretend the prompt alone explains everything.

Recommendation: send it to peer review. The referee should ask for per-variant recovery and restart counts, a comparison that blocks or matches recovery opportunities, block-level bootstrap intervals or more repetitions, and RHAE80 alongside RHAE70. With those, the causal claim can be tested. Without them, the descriptive results still stand as useful evidence that scaffolding and control matter.

The strongest objection

The central interpretation is that the modified instruction's content improves performance. Section 4.2 describes a recovery supervisor that restarts the controller after a stuck-protocol exit, up to ten times per environment run. Stop-and-report is designed to return control whenever Codex is uncertain, so it systematically enters the path that leads to the stuck protocol and supervisor restarts; Baseline is instructed to continue and effectively does not. The two variants therefore differ not only in the text of continuation_string.txt but in the expected number of sessions per task: Stop-and-report can receive several additional resumed sessions per run, each with a fresh 'keep working' prompt. The observed score gains on ARC-AGI-3 (1.37-7.29 RHAE) and AGI Maze (0.37-4.95 RHAE70) are accompanied by 2-52% and 7-39% higher token costs, and Section 9.2 notes that the score/cost ordering tracks. The paper states in Sections 9.2 and 10 that information-use and inference effects are not separated (b13, b66), so this is acknowledged rather than concealed; but precisely because the headline cross-environment improvement is this contrast, the unquantified contribution of extra recovery attempts is load-bearing. If most of the gain comes from restarts rather than from the instruction's content, the claim that a small guidance change improves the agent is weakened, even though the descriptive mean differences in the tables are real.

The paper is unusually candid, and the descriptive claims in the tables are supported by the reported runs; I am not proposing rejection. The reader's weakest assumption identifies the right fault line: the causal reading of Stop-and-report depends on the unquantified interaction between early returns and the recovery supervisor. Section 4.2 creates a structural asymmetry: the treatment is designed to return control, which is the precondition for the stuck protocol and supervisor restart, while Baseline is told to continue and rarely triggers it. The cost data in Tables 1 and 2, and the score-cost ordering noted in Section 9.2, make extra inference and restarts a plausible carrier of the effect. The paper states in Sections 9.2 and 10 that these effects are not separated, which is honest but does not resolve the attribution. The ablation with recovery disabled would settle it. If gains persist, the 'small guidance change' reading is supported; if not, the correct interpretation is that the change buys additional attempts or compute, and the stronger claim about supplied procedures would need to rest on the Controlled_RESET and twma_v1.61 comparisons alone. I keep the reader's CONDITIONAL verdict because the concern is real, testable, and the paper has disclosed the interaction without quantifying it.

The referee's report

If the reported effects are reliable, the paper contributes three useful things. First, it demonstrates a small, environment-agnostic guidance change that transfers across two game environments, scored on external benchmarks with verifier-defined rewards. Second, the Controlled_RESET intervention isolates one concrete mechanism - moving an already-stated RESET convention from the agent to the controller - and shows it accounts for roughly one-third to three-fifths of the specialized agent's advantage at each configuration; this directly supports the paper's central conclusion that supplied procedures still matter for GPT-5.6-sol, and this part of the evidence is not threatened by the concerns below. Third, the paper is exemplary in disclosure: individual runs are fully tabulated in Appendix C, the all-or-nothing replacement policy is documented in Section 7.5, the post-hoc selection and sensitivity of the RHAE70 factor are stated in Appendix D, and the authors repeatedly disclaim causal separation between better information use and additional inference.

Pith's reading of the paper (a machine's summary, not the authors')

This paper asks whether developer-supplied scaffolding still earns its keep now that strong models may construct their own procedures, and whether an environment-agnostic agent can be improved without per-environment engineering. It claims that a one-sentence instruction change, telling the agent to stop and report difficulties instead of pushing on, improves mean scores on ARC-AGI-3 and AGI Maze in every tested model-effort configuration while changing no code, tools, or other prompts. The same general agent still trails an ARC-AGI-3-specific reference by about seven points even at the strongest model setting, and scripting a reset rule already stated in the environment description recovers roughly one-third to three-fifths of that gap. The paper's conclusion is that having information and tools available is not the same as using them effectively: supplied procedures still matter for the models tested here. The gains come with higher token-based cost, and the paper states that the experiments do not separate better use of information from additional or differently allocated inference.

What carries the argument. The load-bearing object is a pair of nearly identical prompt variants that share one agent implementation: Baseline and Stop-and-report differ only in continuation_string.txt, a short instruction included at initialization and in every continuation. The Stop-and-report wording, continue as long as you can proceed confidently, otherwise stop and report back, changes when the coding agent returns control to the scripted controller, which then resumes the same conversation with a goal reminder and a suggestion to reread the environment description; the controller, the generic client and workspace, and the recovery supervisor (which restarts a stalled run up to ten times) are identical across variants. The targeted intervention Controlled_RESET adds a second mechanism: Codex is forbidden to call RESET, and the controller issues it at GAME_OVER, testing whether a convention already stated in the environment description becomes more useful when implemented explicitly. The scoring apparatus, RHAE (Relative Human Action Efficiency) for ARC-AGI-3 and the RHAE70 variant for AGI Maze, measures completion weighted by action efficiency, so the reported gains reflect solving tasks with fewer wasted actions rather than merely finishing more of them.

Core claim. The central claim is that scaffolding, the prompts, control logic, and procedures wrapped around a language model, still carries measurable value even for a strong model, and that part of that value can be captured without environment-specific engineering. The core evidence is a single-instruction intervention: replacing the continuation text 'Continue working until you have completed your task' with a stop-and-report instruction raised mean ARC-AGI-3 RHAE by 1.37-7.29 points in all four model-effort configurations and mean AGI Maze RHAE70 by 0.37-4.95 points in all eight comparisons, with small, mixed changes on Terminal-Bench. On ARC-AGI-3, the specialized reference twma_v1.61, which prescribes organization and control rather than new game rules or solutions, still exceeded Stop-and-report by about seven RHAE points at GPT-5.6-sol max, and scripted RESET management (Controlled_RESET) recovered roughly one-third to three-fifths of that gap by implementing a convention the environment description already states. A complementary AGI Maze experiment found that additional tutorial experience can reduce rather than help performance, depending on the model and effort setting. The paper concludes that supplied procedures still matter for the models tested here, while acknowledging that the gains also come with higher token-based cost and that the experiments do not separate better information use from additional or differently allocated inference.

The premise it rests on. The load-bearing premise is that the instruction's content, rather than the extra restart opportunities and extra inference it triggers, drives the measured gains; a return with no new action makes the recovery supervisor restart the controller up to ten times per run, and the paper states that better use of information, additional inference, and different allocation are not separated.

If the paper is right (Pith's inferences, not the authors')

A single environment-independent instruction can improve an unchanged agent beyond the environment that motivated it: the stop-and-report gain transferred from ARC-AGI-3, where it was developed, to AGI Maze, which played no role in selecting the instruction.

Developer-supplied procedures retain measurable value at the strongest model setting tested: the specialized reference still beats the general agent by about seven RHAE points with GPT-5.6-sol at max reasoning effort.

Implementing a convention already written in the environment description, the controller issuing RESET at GAME_OVER, recovers roughly one-third to three-fifths of the specialization gap without adding any new game information.

Performance and token cost move together: within every ARC-AGI-3 configuration the ordering of mean score matches the ordering of mean token-based cost, so the gains hold under the tested execution policies rather than at matched inference cost.

Additional relevant experience is not automatically beneficial: tutorials lowered AGI Maze scores for both GPT-5.5-based variants at both effort settings, and their effect with GPT-5.6-sol depended on the effort level.

The unresolved question the paper leaves open is whether scaffolding helps mainly through more effective use of information and tools, additional inference, or both, since the experiments do not separate these effects.

If the stop-and-report gains are partly an artifact of the recovery-supervisor interaction, where a return with no new action triggers a restart and up to ten restarts are allowed per run, then the instruction may be functioning as an indirect inference-budget control rather than purely as guidance; a direct test would disable the stuck-protocol restarts and see whether the gains survive.

The result suggests a cheap, ordered recipe for improving environment-agnostic agents: first shift when control returns through guidance, then script the control decisions the environment already documents; the same two steps could be tried on any benchmark whose description states a reset-like convention.

An extension not tested here is a matched-token comparison: because cost scales with score, comparing variants at equal token budgets rather than equal execution policies could reveal whether the instruction's advantage is mostly 'think longer' rather than 'think differently.'

The tutorial result hints that curriculum design for coding agents may need to be configuration-aware, since the same experience sequence helped or hurt depending on the model and effort setting.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agents built on stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering. We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. The reference prescribes organization and control rather than supplying new game rules or solutions. To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. This small change recovers part of the specialized agent's advantage. A complementary AGI Maze experiment shows that additional tutorial experience can reduce performance. Together, these findings highlight the distinction between having information and tools available and using them effectively. Supplied procedures still matter for the models tested here. The performance gains also come with higher token-based cost. The experiments do not separate better use of information and procedures from additional or differently allocated inference. We provide the EnvTraverse framework and an evaluated baseline agent for testing whether further environment-independent changes improve performance in some environments without degrading it in others.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.99

“Supplied procedures still matter for the models tested here.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.97 · human participants 0.03 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.12 · compares with prior results 0.95 · mentions code or data availability 0.91

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. background 0.99 0.06 0.06 hedged possibility 1.00
We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering. contribution 0.46 0.22 0.03 unclear 0.69
We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. method 0.96 0.95 0.03 asserted 0.98
Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. method 1.00 0.89 0.03 asserted 0.99
Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. result 1.00 0.96 0.13 asserted 1.00
Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. result 0.96 0.81 0.10 asserted 1.00
The reference prescribes organization and control rather than supplying new game rules or solutions. interpretation 0.62 0.26 0.05 asserted 1.00
To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. method 0.92 0.88 0.04 asserted 0.87
This small change recovers part of the specialized agent's advantage. result 1.00 0.95 0.09 asserted 1.00
A complementary AGI Maze experiment shows that additional tutorial experience can reduce performance. result 1.00 0.94 0.04 asserted 0.96
Together, these findings highlight the distinction between having information and tools available and using them effectively. interpretation 1.00 0.78 0.31 asserted 0.98
Supplied procedures still matter for the models tested here. interpretation 0.77 0.85 0.76 asserted 1.00
The performance gains also come with higher token-based cost. result 0.89 0.88 0.04 asserted 1.00
The experiments do not separate better use of information and procedures from additional or differently allocated inference. limitation 1.00 0.69 0.04 asserted 0.99
We provide the EnvTraverse framework and an evaluated baseline agent for testing whether further environment-independent changes improve performance in some environments without degrading it in others. contribution 0.90 0.88 0.07 asserted 0.47

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 10,551 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Common agent–environment interaction contract. At initialization, the environment supplies a textual description of the task and interaction interface, optionally accompanied by files. The agent then submits JSON actions and receives observation files and terminal status. The agent remains unchanged across environments, while each environment defines its own actions, observations, and termination conditions. 3.2 Implementation The implementation runs the agent and the environment server in separate Docker containers. Actions are JSON objects exchanged over HTTP, and responses carry…
Figure 2 of the paper
Figure 2. Architecture shared by the two prompt variants. The scripted controller sends prompts and checks generic session state. Codex performs the task-solving work and uses the provided client to interact with the environment. The client is part of the agent and saves the environment description, actions, and observations. All components remain unchanged across environments. The main prompt (Appendix B.1.1) asks Codex to read the environment description and accomplish the goal stated there. It explains how to invoke the client with a JSON action and where to find the resulting…
Figure 3 of the paper
Figure 3. Scripted controller used by both prompt variants. The initial prompt includes the selected continuation instruction. State is inspected after Codex returns; “No progress” means no new action since the preceding continuation. “End” denotes controller exit after environment termination or a return with no new action following the stuck reminder. The runtime-recovery supervisor is omitted; its handling of controller exits is described in the text. Normal protocol. The controller resumes the same Codex conversation with a reminder to keep working toward the goal in the environment…
Figure 4 of the paper
Figure 4. Mean ARC-AGI-3 RHAE over three runs. Solid lines show the environment-agnostic variants Baseline and Stop-and-report; dashed lines show Controlled_RESET and twma_v1.61. The four horizontal positions are discrete model–effort configurations, not a calibrated capability scale; connecting lines only aid comparison. Individual runs are reported in Appendix C.1. 17
Figure 5 of the paper
Figure 5. AGI Maze mean RHAE70; higher is better. Solid lines show single-labyrinth results, and dotted lines show the tutorial-set curriculum. Blue circles and triangles denote Baseline; orange squares and diamonds denote Stop-and-report. Each point averages three complete 106-target evaluation blocks. Model–effort configurations are discrete. Individual execution batches are given in Appendix C.2. Tutorial experience is not uniformly beneficial. Preceding tutorials reduce mean RHAE70 for both GPT-5.5-based agents at both effort settings. With GPT-5.6-sol xhigh, tutorials improve both…

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment