Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Self-Evolving Code-with-Image Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free reflection loop turns a frozen model's failed code into reusable plain-text skills that substantially improve Code-with-Image reasoning.

desk verdict CwI-Bench is a real contribution and the skill-evolution results are consistent, but the 'beyond inspection' premise needs a human baseline before the benchmark's central claim is airtight. read the letter →

arxiv 2608.11292 v1 pith:4MQOPMK6 submitted 2026-08-11 cs.CV

Tianze Yang , Liang Wu , Ruitong Sun , Yucheng Shi , Yanqiao Wang , Mayank Darbari , Ninghao Liu , Jin Sun
show 1 more author
Liangjie Hong
This is my paper · ORCID
classification cs.CV
keywords Code-with-Imagevisualreasoningmultimodalagentsskillevolutiontraining-freereflectionCwI-Benchexecutableprogram-aided
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that some visual questions are not answerable by looking, no matter how many tools zoom, crop, or brighten, because the target is a precise quantity that only a multi-step computation over pixel arrays can recover. It argues that in this 'Code-with-Image' regime the model's program is the reasoning, and that a frozen model can improve itself by reflecting on why its own programs fail. The proposed loop, Self-Reflection over Executable Reasoning, studies failed trajectories, re-executes them in a sandbox to expose silent implementation faults, and saves verified repairs as short plain-text skills. On CwI-Bench, the paper reports bare Code-with-Image accuracy of 32.6% for Qwen3.5-27B and 43.0% for GPT-5.6-luna, rising to 55.9% and 66.6% with self-evolved skills, with the skills transferring across model scales and families. If right, this is a practical route to accumulating visual-reasoning procedures without weight updates.

What carries the argument

The load-bearing object is the Code-with-Image episode, in which the model writes a program, a sandboxed interpreter executes it, and the program's final output is the answer, so the model never carries the intermediate values in language. On top of that, the skill library is a capped set of plain-text skills with three fields—When, Procedure, and Verify—and the learning loop, Self-Reflection over Executable Reasoning, alternates between observational reflection (a single read-only call over failed trajectories) and executable reflection (a sandboxed debugging session that re-runs the failed code, renders intermediate arrays as images, and admits a repair only after it restores the gold answer on at least two failing cases). The library is the only mutable state: the model stays frozen, and edits are gated by a validation-side delivery margin so that a skill is deployed only if it beats bare Code-with-Image by at least three percentage points.

What would settle it

Show CwI-Bench images to expert humans with unlimited zoom and search but no computation, and ask for answers at the scored tolerances; if a meaningful fraction land within tolerance, the beyond-inspection premise is wrong and the gap between language reasoning and Code-with-Image would be overstated.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in hard visual questions is not perception but computation, and that giving a frozen multimodal model a bare Python interpreter converts language-side description into executable reasoning. Once execution is available, the remaining failures are silent implementation faults that hide in intermediate state, and these can be found by re-running the failed computation, rendering intermediates, and testing repairs against constructive ground truth. The paper's evidence is CwI-Bench: 30 task families (transformations, perception, localization, color) whose targets are beyond inspection precision, with disjoint train/validation/test splits, and a training-free skill-evolution loop that edits a small library of When/Procedure/Verify skills. On this benchmark the loop raises bare Code-with-Image accuracy from 32.6% to 55.9% on Qwen3.5-27B and from 43.0% to 66.6% on GPT-5.6-luna, and the delivered text skills transfer: injecting the 27B model's skills into a 9B solver outperforms the 9B's own self-evolved skills (34.5% versus 30.6%), while cross-family transfer lifts gemma-4-26B from 27.7% to 45.0%.

Load-bearing premise

The benchmark's premise that every target is truly beyond human inspection at the scored precision, enforced by generator screens and examples rather than human-subject validation, is load-bearing; if many items were guessable by eye, the tool-free failure rates would not isolate computation.

Editorial extensions

If this is right

  • A frozen model can improve its own visual reasoning without any weight update, as long as it can run and debug programs; this applies even to closed models that expose only inference APIs.
  • Plain-text skills can be copied across model scales and families: a 27B model's procedures lift a 9B solver above the 9B's own self-evolved skills, and the same library improves a solver from a different model family.
  • Tool-free chain of thought, even with thinking mode enabled, stays below 30% on CwI-Bench, suggesting that language-only reasoning has a hard ceiling when the answer is a computed quantity rather than a described one.
  • The learning recipe is not tied to a strong reasoner: the same loop works on open 9B/27B models and on a frontier model, and removing executable reflection removes roughly a quarter to a third of the total gain, concentrated on tasks where faults hide in intermediate state.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these skills are truly portable plain text, the same loop could be run once on a strong model and the resulting skill libraries shared as a lightweight curriculum, letting weaker models inherit algorithms they could not discover on their own; the paper's co-adaptation results suggest such transfer should be tuned to the recipient's execution ability.
  • CwI-Bench's 'beyond inspection' premise rests on generator screens rather than human-subject testing, so a direct human study at the scored tolerances would reveal whether any residual successes actually come from inspection rather than computation.
  • The executable-reflection mechanism—reload failed code, render intermediates, test repairs against gold—could apply to any program-native reasoning domain where intermediate state hides faults, such as numerical data analysis or scientific computing, not just images.
  • The paper evolves one library per task with bounded training splits; a natural next test is whether skills evolve further when the loop sees a broader distribution or multiple related families within a single library.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CwI-Bench, a 30-task benchmark for a regime the authors call Code-with-Image, in which a model is given only a Python interpreter and must author and execute a program that computes the answer from the pixels. It further proposes Self-Reflection over Executable Reasoning, a training-free loop in which a frozen model reflects on its own failed programs, repairs them in a sandbox against constructive ground truth, and accumulates portable plain-text skills. The reported results are large: on GPT-5.6-luna, test accuracy rises from 12.9% in tool-free Instruct mode to 25.0% in Thinking mode, 43.0% with a bare interpreter, and 66.6% with self-evolved skills; on Qwen3.5-27B the corresponding ladder is 8.6% to 14.4% to 32.6% to 55.9%. The paper also reports transfer of skills across model scales and families, and an ablation in which removing executable reflection reduces mean validation uplift by about 6 of 19.7 percentage points on Qwen3.5-27B.

Significance. If the claims hold, CwI-Bench is a valuable benchmark for program-native visual reasoning, and the self-reflection loop is a practical, weight-update-free mechanism for accumulating and transferring procedural knowledge across frozen LVLMs. The paper has notable strengths: constructive ground-truth generation with disjoint train/val/test splits, a sealed test set, verbatim reflection prompts, per-task worked examples, archived raw trajectories, and multi-executor consistency for both the benchmark evaluation and the ablation. The main risks are the unvalidated 'beyond inspection' premise that underlies the benchmark's interpretation, the absence of uncertainty quantification on 100-item test sets, and some over-strong claims about the safety of delivered skills.

major comments (3)
  1. [§3.3, Appendix B] Section 3.3 asserts that every CwI-Bench target lies 'beyond inspection' and that human annotation is 'not only unnecessary but impossible,' with enforcement only through generator-side screens against an undefined 'closest visually plausible look-alike.' The protocol notes in Appendix B, however, state that think-with-image agents 'concentrate on the few tasks where inspection suffices (counting colors, shapes, or stripes),' which directly contradicts the universal beyond-inspection claim. Because the benchmark's central interpretation—that tool-free failure isolates missing computation rather than perception—depends on this premise, please either provide a human-subject or perceptual baseline on a representative sample of test items, or explicitly identify and handle the tasks where inspection can suffice.
  2. [Tables 2, 3, 5–8] All headline results are point estimates on 100-item test sets with no confidence intervals or significance tests. Per-task differences as small as +1 to +5 points and cross-condition comparisons such as the 9B transferred (34.5) versus self-evolved (30.6) overall accuracies may be within sampling error. Please report bootstrap confidence intervals or equivalent uncertainty measures, at least for the aggregate and transfer claims, before asserting that one regime outperforms another on every family.
  3. [§4.3, Fig. 2, Table 5] The delivery gate in §4.3 and Eq. (6) is described as a validation-side safeguard that is 'not a guarantee against test regression,' yet Figure 2 labels the delivery rule as 'skills can never hurt,' and §5.3 emphasizes that no delivered luna skill regresses on test. In the open-model results, delivered skills do regress on specific test tasks (Table 5, film color: 39.0 to 29.0; Table 8, restore count: 18.0 to 14.0). Please remove the 'never hurt' framing, report the full distribution of per-task test deltas, or provide a statistical guarantee at the aggregate level.
minor comments (5)
  1. [§4.2, Eq. (4)] The use of '⇐⇒' to equate classical zeroth-order ascent with the reflective update is misleading, since the right-hand side is not an equivalent formalization; I suggest replacing the equivalence symbol with a correspondence arrow.
  2. [§5.1, Appendix A.4] The paper says 'one shared recipe on every task with no per-task tuning' but luna uses a single restart and 8 rounds while the open models use 3 restarts and 12 rounds; this asymmetry should be acknowledged in the protocol comparison.
  3. [Table 1] The 'beyond human' checkmark for CwI-Bench repeats the unvalidated premise; consider marking it as asserted rather than validated, or providing the human baseline that would justify it.
  4. [§3.1, Appendix E.1] The phrase 'nothing to call' is overstated given that the Python interpreter includes rich libraries such as OpenCV and the delivered skill example uses SIFT and RANSAC; clarify that the distinction is the absence of a task-specific tool vocabulary, not the absence of any pre-existing functions.
  5. [Figure 2] Figure 2 is dense and the small text and multiple arrows are hard to read in print; consider simplifying the diagram or enlarging the font.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: skills are gated on validation and scored on held-out test items; no step reduces to its input.

full rationale

The claimed derivation chain is empirical rather than definitional. The skill-evolution loop (Eqs. 2-6) consumes training-split rollouts and gold labels, selects libraries on a probe and a held-out validation split, and only then reports test-100 accuracy; the test split is never used for selection, so the headline gains (32.6 to 55.9 on Qwen3.5-27B and 43.0 to 66.6 on GPT-5.6-luna) are measurements, not fitted predictions. Equation (6) is a delivery gate, not a generative model of the result. The 'beyond inspection' premise in Section 3.3 is asserted by construction (generator screens discard eye-guessable samples: 'if an eyeball guess would land within tolerance, the sample never enters the benchmark') and is not independently human-validated; this is a validity and correctness risk for the benchmark's interpretation, but it is not a circular step because no equation or cited result defines the conclusion in terms of the premise. I flag this explicitly per the reviewing rule: 'human annotation is not only unnecessary but impossible' (Section 3.2) is a missing-support assertion and should be weighed as a benchmark-validity concern, not as circularity. The related-work audits [8,10] are external to the present authors and are not load-bearing for the self-evolution result. The same model writing and using skills is a design property, not a logical loop. No self-definitional, fitted-input-as-prediction, or self-citation-load-bearing pattern is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method has no mathematical derivation; the central empirical claims depend on the benchmark's construct validity, the i.i.d. split, and several hand-set hyperparameters listed above. No new physical entities are introduced.

free parameters (5)
  • delivery margin gamma = 3 pp
    A delivered skill must beat bare Code-with-Image by at least 3 percentage points on validation (eq. 6). This threshold determines which tasks receive skills and shapes reported test gains.
  • patience p = 5 rounds
    Number of consecutive stalled probe rounds before switching from observational to executable reflection; controls how often the expensive sandbox mode is used.
  • library capacity = 6 skills x 250 words
    Capacity cap on the skill library, intended as a regularizer to force compression; affects generalization.
  • executable reflection step budget T_max = 10 steps
    Maximum investigation steps per executable-reflection session; bounds the depth of debugging.
  • probe size / D_probe = 32 items (8 per tier)
    Difficulty-balanced probe used to steer the zeroth-order search; chosen to balance signal and cost.
assumptions (4)
  • domain assumption Generator labels are correct by construction.
    Each benchmark instance is rendered from a known target, so the gold answer is exact. This is standard for synthetic benchmarks and is not independently audited.
  • ad hoc to paper CwI-Bench targets are beyond human inspection at the scored precision.
    The benchmark's load-bearing premise. The paper provides generator screens that reject eye-guessable samples, but no human-subject validation. This is the paper's weakest assumption.
  • domain assumption A skill that generalizes on training and validation will also generalize to test items from the same generator.
    The held-out test is drawn from the same task distributions, so this is a standard i.i.d. assumption; the paper relies on it when delivering skills based on validation.
  • domain assumption The sandbox interpreter executes code faithfully and deterministically for both bare and skilled solvers.
    All accuracy comparisons assume identical execution environments; no environment-specific failure mode is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Evolving Code-with-Image Reasoning." pith.science (2026). https://pith.science/paper/4MQOPMK6

@misc{pith2026260811292,
  author       = {Pith},
  title        = {Pith review of: Self-Evolving Code-with-Image Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4MQOPMK6}},
  note         = {Machine review of arXiv:2608.11292}
}
abstract

Multimodal models increasingly reach for tools when solving visual tasks (crop, zoom, rotate, brighten), a paradigm known as thinking-with-images. The central challenge is one of perception: tools mostly serve to expose visual evidence, reasoning over that evidence stays in language, and most targets are ones a human could in principle determine by inspection. Some visual questions, however, are not bottlenecked by perception: recovering their answers requires executing a multi-step visual algorithm over the pixels. On such questions a model often names the correct algorithm at once yet still answers wrong, because language can describe an algorithm without being able to run one. Code-with-Image crosses that line: given nothing but a Python interpreter, the model must implement a genuine visual algorithm in code to solve the task; the program itself becomes the reasoning. The bottleneck then shifts from executing code to deciding which algorithm to implement. So we let the model teach itself: a training-free reflection loop studies its own failed programs, tests repairs against constructive ground truth, and keeps what survives as portable skills. On our Code-with-Image Bench (CwI-Bench), thirty task families induced by hidden visual computations with disjoint learning and evaluation splits, even GPT-5.6-luna stays below 30% with tool-free chain of thought; given a bare interpreter it reaches 43%, and with skills evolved through its own executable reflection, 67%. The open 27B model climbs the same ladder (9% $\rightarrow$ 33% $\rightarrow$ 56%), and the skills are plain text, transferable across scales and families. When code carries the reasoning, debugging code becomes debugging reasoning.

Figures

Figures reproduced from arXiv: 2608.11292 by the authors.

Figure 1
Figure 1. Benchmark overview. Each row pairs two representative held-out examples from one semantic family (difficulty controls, question, and constructive ground-truth answer); the waist cards curve around the central wheel. Wheel: all 30 task distributions, named and colored by family (inner ring = the four families, angle ∝ task count); the hub reports the key statistics — 99.9% of bare-Code-with-Image test trajectories ex… view at source ↗
Figure 2
Figure 2. Overview. A frozen LVLM solves each task by Code-with-Image: the image enters a persistent sandbox as data, and an authored program computes the answer. Failed trajectories drive two reflection modes: observational reflection reads complete multimodal trajectories and proposes bounded edits to a small skill library; once progress stalls, executable reflection re-executes the failures in the sandbox, renders intermed… view at source ↗
Figure 3
Figure 3. Self-evolved skills. Overall test accuracy for the four executors that ran the evolution loop: tool-free CoT (Instruct, Thinking), bare Code-with-Image, and Code-with-Image with delivered skills (tasks without one fall back to bare Code-with-Image). Per-task detail: Appendix B [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 42 canonical work pages

  1. [1]

    Gepa: Reflective prompt evolution can outperform reinforcement learning

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning. In C. V ondrick, B. Hariharan,...

    work page 2026

  2. [2]

    Evoskill: Auto- mated skill discovery for multi-agent systems.arXiv preprint arXiv:2603.02766, 2026

    Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Weiyuan Chen, and Tu Vu. Evoskill: Auto- mated skill discovery for multi-agent systems.arXiv preprint arXiv:2603.02766, 2026

    arXiv 2026

  3. [3]

    Are we on the right way for evalu- ating large vision-language models? In A

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evalu- ating large vision-language models? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37...

    work page 2024

  4. [4]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URLhttps://openreview.net/forum?id= YfZ4ZPt8zd

    work page 2023

  5. [5]

    Teaching large language mod- els to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou. Teaching large language mod- els to self-debug. In B. Kim, Y . Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y . Sun, editors,International Conference on Learning Representations, volume 2024, pages 8746–8825,

    work page 2024

  6. [6]

    PAL: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of...

    work page 2023

  7. [7]

    Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming.SIAM journal on optimization, 23(4):2341–2368, 2013

    Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming.SIAM journal on optimization, 23(4):2341–2368, 2013

    work page 2013

  8. [8]

    Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains

    Garvin Guo, Donglei Yu, Yu Chen, Xiang Wang, Shuai Li, Xinpei Zhao, Huaxing Liu, Qinghao Wang, and Minpeng Liao. Do multimodal agents really benefit from tool use? a systematic study of capability gains.arXiv preprint arXiv:2606.02357, 2026

    work page Pith review arXiv 2026

Show all 59 references
  1. [9]

    Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning.arXiv preprint arXiv:2510.12712, 2025

    Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Jayeon Park, Ernesto Gabriel Hern´andez Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, et al. Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning.arXiv...

  2. [10]

    Thinking with programming vision: Towards a unified view for thinking with images

    Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, and Tao Jin. Thinking with programming vision: Towards a unified view for thinking with images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 33467–33476, June 2026

    2026

  3. [11]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2023. doi: 10.1109/CVPR52729.2023.01436

    2023

  4. [12]

    Deepeyesv2: Toward agentic multimodal model

    Jack Hong, Chenxiao Zhao, ChengLIn Zhu, Weiheng Lu, Guohai Xu, and XingYu. Deepeyesv2: Toward agentic multimodal model. In C. V ondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors,International Conference on Learning Representations, volume 2026, pages 1...

    2026

  5. [13]

    Codev: Code with images for faithful visual reasoning via tool-aware policy optimization

    Xinhai Hou, Shaoyuan Xu, Manan Biyani, Moyan Li, Jia Liu, Todd C Hollon, and Bryan Wang. Codev: Code with images for faithful visual reasoning via tool-aware policy optimization. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2026

  6. [14]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Pa- quet, J. Tomczak, and C...

    2024

  7. [15]

    Xskill: Continual learning from experience and skills in multimodal agents.arXiv preprint arXiv:2603.12056, 2026

    Guanyu Jiang, Zhaochen Su, Xiaoye Qu, and Yi R Fung. Xskill: Continual learning from experience and skills in multimodal agents.arXiv preprint arXiv:2603.12056, 2026

    2026 arXiv

  8. [16]

    Tir-bench: A comprehensive benchmark for agentic thinking- with-images reasoning.arXiv preprint arXiv:2511.01833, 2025

    Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Kon- stantinos Psounis, and Kaipeng Zhang. Tir-bench: A comprehensive benchmark for agentic thinking- with-images reasoning.arXiv preprint arXiv:2511.01833, 2025

    2025

  9. [17]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean confer- ence on computer vision, pages 740–755. Springer, 2014

    2014

  10. [18]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In B. Kim, Y . Yue, S. Chaudhuri, K. Fragkiadaki, M...

    2024

  11. [19]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Pro- cessi...

    2023

  12. [20]

    Random gradient-free minimization of convex functions

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017

    2017

  13. [21]

    Hypothesis-driven skill optimization for llm agents.arXiv preprint arXiv:2606.22330, 2026

    Fangxin Shang and Yehui Yang. Hypothesis-driven skill optimization for llm agents.arXiv preprint arXiv:2606.22330, 2026

    2026 arXiv

  14. [22]

    Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026

    Yifei Shen, Bo Li, and Xinjie Zhang. Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026

    2026 arXiv

  15. [23]

    Re- flexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Process- ing Sys...

    2023

  16. [24]

    Codedance: A dynamic tool-integrated mllm for executable visual reasoning

    Qi Song, Honglin Li, Yingchen Yu, Haoyi Zhou, Lin Yang, Song Bai, Qi She, Zilong Huang, and Yunqing Zhao. Codedance: A dynamic tool-integrated mllm for executable visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pag...

    2026

  17. [25]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation.IEEE transactions on automatic control, 37(3):332–341, 1992

    James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation.IEEE transactions on automatic control, 37(3):332–341, 1992

    1992

  18. [26]

    Pixel reasoner: Incentiviz- ing pixel space reasoning via curiosity-driven reinforcement learning

    Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentiviz- ing pixel space reasoning via curiosity-driven reinforcement learning. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural In...

    2025

  19. [27]

    Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers.arXiv preprint arXiv:2506.23918, 2025

    Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, et al. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers.arXiv preprint arXiv:2506.23918, 2025

    2025 arXiv

  20. [28]

    Agentvista: Evaluating multimodal agents in ultra-challenging realistic visual scenarios.arXiv preprint arXiv:2602.23166, 2026

    Zhaochen Su, Jincheng Gao, Hangyu Guo, Zhenhua Liu, Lueyang Zhang, Xinyu Geng, Shijue Huang, Peng Xia, Guanyu Jiang, Cheng Wang, et al. Agentvista: Evaluating multimodal agents in ultra-challenging realistic visual scenarios.arXiv preprint arXiv:2602.23166, 2026

    2026

  21. [29]

    Vipergpt: Visual inference via python execution for reasoning

    D ´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11854– 11864, 2023. doi: 10.1109/ICCV51070.2023.01092

    2023

  22. [30]

    V oyager: An open-ended embodied agent with large language mod- els.Transactions on Machine Learning Research, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language mod- els.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URLhttps: //openreview.net/forum?...

    2024

  23. [31]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. InProceedings of the AAAI Conference on Artificial In...

    2025

  24. [32]

    Agentic-mme: What agentic capability really brings to multimodal intelligence?arXiv preprint arXiv:2604.03016, 2026

    Qianshan Wei, Yishan Yang, Siyi Wang, Jinglin Chen, Binyu Wang, Jiaming Wang, Shuang Chen, Zechen Li, Yang Shi, Yuqi Tang, et al. Agentic-mme: What agentic capability really brings to multimodal intelligence?arXiv preprint arXiv:2604.03016, 2026

    2026 arXiv

  25. [33]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084–13094, June 2024

    2024

  26. [34]

    Skillopt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904, 2026

    Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. Skillopt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904, 2026

    2026 arXiv

  27. [35]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  28. [36]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  29. [37]

    Spyce: Skill-policy co-evolution for multimodal agents.arXiv preprint arXiv:2607.13854, 2026

    Ru Zhang and Weijie Qiu. Spyce: Skill-policy co-evolution for multimodal agents.arXiv preprint arXiv:2607.13854, 2026

    2026 arXiv

  30. [38]

    Thyme: Think beyond images

    YiFan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Haonan fan, Kaibing Chen, Jiankang Chen, Haojie Ding, Kaiyu Tang, Zhang Zhang, Liang Wang, Fan Yang, Tingting Gao, and Guorui Zhou. Thyme: Think beyond images. ...

    2026

  31. [39]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    2024

  32. [40]

    Pyvision: Agentic vision with dynamic tooling

    Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei. Pyvision: Agentic vision with dynamic tooling. InFirst Workshop on Multi-Turn Inter- actions in Large Language Models, 2025. URLhttps://openreview.net/forum?id= l7NYIYODoQ

    2025

  33. [41]

    Pyvision-RL: Forging open agentic vision models via RL

    Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, and Chen Wei. Pyvision-RL: Forging open agentic vision models via RL. InForty-third Interna- tional Conference on Machine Learning, 2026. URLhttps://openreview.net/forum? id=EFq5Ml9rqv. 17

    2026

  34. [42]

    Deepeyes: Incentivizing ”thinking with images” via reinforcement learn- ing

    Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and XingYu. Deepeyes: Incentivizing ”thinking with images” via reinforcement learn- ing. In C. V ondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Confer...

    2026

  35. [43]

    No skill

    Xuanyu Zhu, Yuhao Dong, Rundong Wang, Yang Shi, Zhipeng Wu, Yinlun Peng, YiFan Zhang, Yihang Lou, Yuanxing Zhang, Ziwei Liu, et al. Vtc-bench: Evaluating agentic multimodal models via compositional visual tool chaining.arXiv preprint arXiv:2603.15030, 2026. 18 Appendix Table o...

    2026

  36. [46]

    replace" name=

    Verify: ...</edit> <edit op="replace" name="existing-name">...new full body...</edit> <edit op="delete" name="existing-name"></edit> - At most 3 edit operations per reflection. Keep the library at <= 6 skills, each under 250 words (the Verify section does not count toward the ...

  37. [47]

    Compare, line by line, how your successful and failing attempts compute the same quantity

    INVESTIGATE with code: re-run suspect computations from your failing code on the actual images, print intermediate values AND dtypes, render intermediate arrays as images and look at them, and test hypotheses by checking against the known gold. Compare, line by line, how your ...

  38. [48]

    Verify the fix: show with code that the corrected computation recovers the gold on at least 2 of the failing cases

  39. [49]

    add" name=

    Only then, output skill-library edits in EXACTLY this format (and nothing after them): <edit op="add" name="short-skill-name">When: ... Procedure:

  40. [50]

    Verify: ...</edit> (ops: add / replace / delete; at most 3 edits; library <= 6 skills; each skill <= 250 words; every skill must have When/Procedure/Verify; Verify = which intermediate to RENDER and what correct vs broken looks like. Skills must be GENERAL for this task type |...

  41. [51]

    Convert both input images to grayscale using OpenCV (`cv2.cvtColor` with`COLOR_RGB2GRAY`).,→

  42. [52]

    SIFT is preferred for robustness against scale and rotation changes often present in crops

    Detect keypoints and compute descriptors in both images using SIFT (`cv2.SIFT_create()`) if available; otherwise use ORB (`cv2.ORB_create(nfeatures=500)`). SIFT is preferred for robustness against scale and rotation changes often present in crops. ,→ ,→ ,→

  43. [53]

    For SIFT use`NORM_L2`, for ORB use`NORM_HAMMING`

    Match descriptors using`cv2.BFMatcher`. For SIFT use`NORM_L2`, for ORB use`NORM_HAMMING`. Use`knnMatch(k=2)`.,→

  44. [54]

    Apply Lowe's ratio test: keep match m if`m.distance < 0.75 * n.distance`where n is the second-best match.,→

  45. [55]

    If fewer than 4 matches pass the ratio test, fall back to `crossCheck=True`matching.,→

  46. [56]

    Ensure at least 4 inliers remain.,→

    Use RANSAC (`cv2.findHomography`with`cv2.RANSAC`) to filter outliers from the good matches. Ensure at least 4 inliers remain.,→

  47. [57]

    Calculate coordinate differences for each RANSAC-filtered match:`dx = pt2.x - pt1.x`,`dy = pt2.y - pt1.y`.,→

  48. [58]

    Round to nearest integer.,→

    Compute the **median** of all`dx`values and all`dy`values to determine the final shift. Round to nearest integer.,→

  49. [59]

    Total cases: {len(cases)}

    Render a visualization: Create copies of original images. Draw red circles on Image 1 at matched keypoint locations from Image 1. Draw green circles on Image 2 at matched keypoint locations from Image 2. Display them side-by-side. ,→ ,→ ,→ Verify: The rendered image must show ...

  50. [2024]

    URLhttps://proceedings.iclr.cc/paper_files/paper/2024/file/ 2460396f2d0d421885997dd1612ac56b-Paper-Conference.pdf

    2024

  51. [2026]

    URLhttps://proceedings.iclr.cc/paper_files/paper/2026/file/ cdb347d7516d52a9280ea9d6708911f6-Paper-Conference.pdf

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.