Pith. sign in

REVIEW 2 major objections 5 minor 44 references

This paper introduces MobileForge, a project-level benchmark that evaluates six frontier multimodal LLMs on generating complete multi-screen mobile apps, finding that all build successfully but navigation correctness ranges from 58% to 92%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:48 UTC pith:G22DYMOX

load-bearing objection A genuinely useful new benchmark for project-level design-to-code, but its headline navigation metric conflates interaction correctness with guessing the annotators' URL slugs. the 2 major comments →

arxiv 2607.28645 v1 pith:G22DYMOX submitted 2026-05-29 cs.HC cs.AI

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

Fan Wu , Cuiyun Gao , Yiming Huang , Yang Xiao , Yujia Chen , Qing Liao This is my paper
classification cs.HC cs.AI
keywords multi-screen app generationdesign-to-code benchmarkmultimodal LLM evaluationnavigation testingvisual fidelity evaluationcode maintainabilityVLM-as-judgestate-isolated testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that design-to-code evaluation must move from single screenshots to full multi-screen projects, and introduces MobileForge to do that. MobileForge gives each model a folder of screenshots from a real app plus a page-relationship description, and scores the generated React/TypeScript project on five axes: build, navigation, visual fidelity, maintainability, and cost. Across 174 runs on six frontier models, every project compiles, but interactive navigation is the weak point—the best model passes 92% of navigation tests, the worst 58%—and even the visual leader scores only 2.99/5 on point-wise visual fidelity. The paper also finds that code maintainability is not correlated with visual quality: the visually best model writes 2.2x more code with more dead components than the maintainability leader. Taken together, these results indicate that single-page benchmarks miss the dominant bottlenecks of cross-page consistency and interaction wiring.

Core claim

MobileForge, with 29 real apps, 309 human-reviewed screens, and 701 navigation test specifications, is the first benchmark to treat multi-screen app generation as a project-level task. Its five-axis protocol separates build success from navigation correctness, visual fidelity, maintainability, and efficiency, with two new evaluation techniques: state-isolated navigation testing, which launches each test from a fixed source route so failures are independent and attributable, and anchor-reference list-wise visual evaluation, which mixes the original screenshot into the candidate set as a hidden anchor so every judge call is self-validating. Under this protocol, six frontier multimodal LLMs all

What carries the argument

The central object is MobileForge itself: a benchmark of 29 real consumer apps, each represented by a folder of screenshots, a page-relationship description (tab structure plus parent–child screen relations), and 701 navigation test specifications in three categories (page existence, tab navigation, parent–child navigation). The evaluation rests on two mechanisms: state-isolated navigation testing, which drives the browser directly to each source route so that each test failure is independent and can be attributed to a single interaction, and anchor-reference list-wise visual evaluation, in which the reference screenshot is hidden in the candidate set and a judge's ranking is accepted only w

Load-bearing premise

The load-bearing premise is that one run per app–model pair gives a fair basis for ranking the models; the benchmark acknowledges this and leaves multi-trial replication to future work, and stochastic agent runs can vary enough to change the reported orderings.

What would settle it

Run the same 174-cell benchmark with, say, five trials per (app, model) cell, keeping the harness and judge fixed. If per-model NavPass or Borda rankings change materially—for example, if Claude Opus 4.6 and GPT-5 swap leadership on visual fidelity, or GPT-5 no longer dominates Claude Haiku 4.5 and Gemini 2.5 Pro on cost-quality—the paper's comparative conclusions are not stable. The paper itself notes that a multi-trial replication is future work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-page design-to-code benchmarks overstate model readiness; project-level evaluation reveals navigation and cross-page consistency as the limiting factors.
  • Build success is no longer a discriminating axis when models can iterate with a verify-repair loop; the meaningful axes are navigation, visual fidelity, and maintainability.
  • The dissociation between visual fidelity and maintainability means users must choose an operating point: a model that produces prettier screens may also produce harder-to-maintain code.
  • State-isolated navigation testing can be reused as a general protocol for evaluating multi-screen or multi-route applications without cascading failure artifacts.
  • The anchor-reference list-wise judging protocol offers a way to make VLM-as-Judge evaluations self-validating, discarding rankings where the hidden reference is not ranked first.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A multi-trial replication would likely show that individual model rankings (e.g., which model wins visual fidelity on a given app) are noisy; the paper's single-trial design means its headline rankings should be read as directional, not definitive.
  • The anchor mechanism could generalize beyond app generation to any visual generation benchmark where reference images exist, giving every judge call an inexpensive sanity check.
  • Because the output target is a web stack (React+TypeScript+Tailwind), the findings about navigation reliability may transfer only partially to native mobile development, where routing and event handling differ.
  • The failure taxonomy (blank page, wrong route, missing target, dead click) could be turned into automated lint-style checks on generated component trees, allowing early detection of the dominant failure modes before runtime.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. MobileForge is a benchmark for project-level multi-screen mobile app generation from screenshots. It consists of 29 real consumer apps, 309 human-reviewed screens, structured page-relationship annotations, and 701 navigation test specifications. The paper proposes two evaluation techniques: state-isolated navigation testing and an anchor-reference list-wise visual evaluation protocol, and evaluates six frontier multimodal LLMs over 174 end-to-end runs on five axes: build, navigation, visual fidelity, code maintainability, and efficiency. The headline results are that all models produce buildable projects (100% build success), navigation correctness ranges from 58% to 92%, visual fidelity tops out at 2.99/5 on a point-wise scale, maintainability is dissociated from visual quality, and navigation failures concentrate in three or four predictable modes.

Significance. If the navigation-validity concern is resolved, MobileForge is a timely and valuable benchmark. It addresses a real gap: single-page design-to-code benchmarks cannot measure cross-page consistency, shared-component reuse, or interaction wiring. The paper is unusually transparent: it releases the benchmark, harness, and run artifacts; explicitly attributes build-success saturation to the verify-repair loop (Appendix A); calibrates the visual judge against humans (strict Kendall tau = 0.716) and cross-checks with a second judge (inter-judge tau = 0.867); and hand-labels a failure sample with Wilson confidence intervals. Those are genuine strengths. However, the central navigation-axis claim rests on an evaluation protocol that may penalize route-naming choices that are never communicated to the model, and the comparative ranking is based on one stochastic run per cell. These issues need to be fixed before the benchmark's headline conclusions can be taken at face value.

major comments (2)
  1. [§3.3, §4.2, §7] The navigation axis is not a valid measure of interaction correctness unless the model is given the URL routes used by the evaluator. The model input in §3.3 is screenshots plus natural-language page-relationship descriptions (screen names, tab structure, parent-child trigger statements); route paths are not supplied. Yet §4.2's state-isolated tests drive the browser directly to 'the route of the source page' and declare a pass only when 'the URL matches the expected page.' A generated project that wires all clicks and tabs correctly but names its routes /search-page instead of /search fails page-existence tests and cannot even initialize source routes for tab/parent-child cases. The paper's own C2 definition in §7 ('the agent declared the route, but the URL resolves to a different page... the model has correctly produced the page') shows that the protocol treats a route-naming mismatch
  2. [§5.2, §6.3, Limitations] The comparative model claims are based on a single trial per (app, model) cell. Table 2 reports NavPass and Borda without trial-level variance estimates; the error bars in Figure 3 are standard deviations across the 29 apps, not across repeated runs. Since agentic LLM generation is stochastic, and the paper does not state temperature or decoding parameters, the distinctions between GPT-5 (NavPass 89%, Borda 0.542) and Gemini 2.5 Pro (82%, 0.513) are within the range that a few additional runs could reverse. The statement in §6.3 that 'GPT-5 dominates two mid-tier models on cost and quality jointly' is a point estimate, not a statistically supported dominance. The Limitation paragraph acknowledges the single-trial choice but does not temper the Results claims. I recommend adding a multi-trial subset (e.g., 3-5 repetitions on 5-10 apps) to bound run-to-run variance, or rephrasing the ranki
minor comments (5)
  1. [Figure 3] The vertical axis label reads 'VisBefore (1--5, higher is better)', but the text says the plotted quantity is the list-wise Borda score in [0,1]. The label and axis range should be corrected.
  2. [§4.1] The command 'npx tsc –noEmit' uses an en dash; it should be '--noEmit'.
  3. [§5.2] Please state the sampling temperature and any other decoding parameters for each model (or point to the repository). This is relevant both to reproducibility and to the single-trial variance discussion.
  4. [§3.3] It would help to state explicitly whether the 'screen names' in the page-relationship description are intended to be used as route slugs. If they are, the route-naming concern in the major comments is partly mitigated; if they are not, the mismatch should be clarified.
  5. [Appendix C] The phrase 'unrankable-set IoU, meaning agreement on which candidates are unrankable, of 0.968' would read better as 'unrankable-set IoU of 0.968, meaning agreement on which candidates are unrankable.'

Circularity Check

0 steps flagged

No significant circularity: all five axes are measured against human-reviewed ground truth or build/type-check output, with judge calibration and cross-checks.

full rationale

This paper constructs a benchmark and runs an evaluation; there is no derivation chain that predicts a quantity from fitted inputs. The five axes are directly measured: build success comes from tsc/vite output (§4.1); navigation correctness is scored against 701 human-reviewed test specifications (§3.3, §4.2); visual fidelity is produced by a fixed VLM judge, calibrated against human raters (Appendix C, strict Kendall τ=0.716, top-1 accuracy 83.6%) and cross-checked with a second judge (Appendix D, inter-judge τ=0.867); maintainability and efficiency are direct code/cost metrics (§4.1). The anchor-reference list-wise protocol is a quality-control mechanism, not a fitted prediction: it discards judge calls that fail to rank the pixel-identical anchor first, and the same judge is fixed across all models, so no model's score is constructed to equal the judge's identity. The only self-citation with overlapping authorship (Anonymous, 2026) is used in Related Work to position the benchmark against a concurrent single-page interactive-webpage study and is not load-bearing for any result. The reviewer-flagged concerns — single trial per (app, model) cell and the fact that navigation tests drive the browser directly to benchmark route slugs that were not supplied to the model — are reproducibility and external-validity threats, not circularity: they do not make any measured result equal to an input by construction. Under the requested circularity criteria, no specific reduction can be quoted, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No fitted parameters appear in the benchmark; the values above are hand-chosen protocol constants. The central results rest on the four domain assumptions listed, all of which are acknowledged in the paper. No new theoretical entities (forces, particles, dimensions, etc.) are introduced.

free parameters (2)
  • Point-wise rubric weights (Layout/Style/Content/Consistency) = 30/25/25/20 (%)
    Hand-chosen weights for the auditable reference point-wise visual baseline in §4.3; the primary Borda metric does not use these weights, so the central visual-fidelity ranking does not depend on them.
  • List-wise k-shuffle parameter = 3
    Chosen by hand for the full-scale list-wise judge run (Appendix C). Reported ranking stability is tested across judges and scenario types but not across different values of k, so the Borda scores inherit this protocol choice.
axioms (4)
  • domain assumption React+TypeScript+Tailwind web rendering is an adequate proxy for mobile app generation.
    Limitations: “The output target is web (React+Tailwind), not native iOS/Android… at the cost of not exercising native APIs.” All navigation and visual claims are measured on this web proxy.
  • domain assumption Screenshots plus page-relationship descriptions suffice as a project specification.
    §3.3 and §5.1 feed models only screenshots and a natural-language page-relationship description; any app behavior not visible in these (APIs, auth, dynamic data) is absent, so generated projects are judged only against what the spec encodes.
  • domain assumption Gemini 2.5 Pro list-wise rankings approximate human visual fidelity.
    §4.3 and Appendix C; calibrated on 55 scenarios (Kendall τ=0.716) and cross-checked with Claude Sonnet 4.5 (inter-judge τ=0.867), but VLM-as-judge remains an approximate proxy that can still be biased.
  • domain assumption State-isolated navigation testing measures user-visible navigation correctness.
    §4.2 directly routes the browser to the source route and executes one action; this decouples reachability from operability but assumes route-level rendering and the test-spec selectors are faithful to real user paths, and it bypasses chained state-dependent navigation.

pith-pipeline@v1.3.0-alltime-deepseek · 18508 in / 12879 out tokens · 119044 ms · 2026-08-03T00:48:57.285547+00:00 · methodology

0 comments
read the original abstract

Recent multimodal large language models can convert visual designs directly into executable code, but real mobile products require multiple screenshots to become a buildable codebase with shared components and working navigation. This project-level setting exposes three limits of existing design-to-code benchmarks: they focus on single-page generation rather than complete codebases, cannot evaluate cross-page navigation, and do not measure project-wide maintainability. We introduce MobileForge, the first benchmark for project-level multi-screen mobile app generation, comprising real mobile apps, human-reviewed screens, structured page-relationship annotations, and navigation test specifications. MobileForge supports five-axis evaluation of build, navigation, visual fidelity, code maintainability, and efficiency. We also propose state-isolated navigation testing to avoid cascading failures in navigation evaluation and an anchor-referenced list-wise visual evaluation protocol to improve visual-judge reliability. Across end-to-end runs on six frontier multimodal LLMs, current models can build mobile-app projects that compile and reach the correct pages, but interactive navigation remains unreliable and visual fidelity and maintainability still lag. The benchmark and supporting materials are available at https://github.com/anoa12159-hue/mobileforge_eval.

Figures

Figures reproduced from arXiv: 2607.28645 by Cuiyun Gao, Fan Wu, Qing Liao, Yang Xiao, Yiming Huang, Yujia Chen.

Figure 1
Figure 1. Figure 1: Prior design-to-code benchmarks take a single screenshot as input and score the resulting HTML page by visual similarity alone (top). MobileForge takes the full set of screenshots of a real mobile app and a page￾relationship description as input, asks an agentic harness to produce a runnable multi-page project, and scores it on five orthogonal axes: build, navigation, visual fidelity, code maintainability,… view at source ↗
Figure 2
Figure 2. Figure 2: MobileForge pipeline. (1) Dataset construction starts from three design principles: realism, project granularity, and runnable evaluation. It collects 29 in-market apps across 14 category buckets with coherent multi-page screenshot sets (309 screens, mean 10.7 per app). (2) Annotation runs a VLM auto-draft over the screenshots, and then a human reviewer edits the page-relationship description and the navig… view at source ↗
Figure 3
Figure 3. Figure 3: Capability-cost frontier on MobileForge. Each [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Annotation review interface, per-page pane (§ [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Annotation review interface, page-relationship pane. Parent–child and tab pairs are surfaced with [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Annotation review interface, navigation test-case pane. Page-existence, tab-navigation, and parent–child [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: C1: blank starting page (1/2). (a) Messenger / Gemini 2.5 Flash, calls_history → chats; (b) Reddit / Gemini 2.5 Flash, home_feed → inbox. The source page renders empty or as a stub placeholder with no interaction target, so no navigation action can be dispatched (§7). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: C1: blank starting page (2/2). (c) Reddit / Gemini 2.5 Flash, inbox → chat_channels_welcome. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: C2: route mapping error. (a) Revolut / GPT-5 Mini, crypto_portfolio → send_money; (b) Uber / Claude Haiku 4.5, uber_eats_home → eats_home. The agent declared the route, but the URL resolves to a different page than the one referenced by the test; the model has correctly produced the page, but the global routing table maps the navigation action to the wrong page (§7). 20 [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 10
Figure 10. Figure 10: C3: target unreachable or occluded (1/3). (a) Discord / GPT-5, notifications → messages; (b) Fiverr / Claude Haiku 4.5, inbox → manage_orders_active. The action target is not present in the rendered viewport, either because the layout omits it entirely or because another element covers it. This is structurally distinct from C4: here the page is malformed, whereas in C4 it is well-formed but inert (§7). 21… view at source ↗
Figure 11
Figure 11. Figure 11: C3: target unreachable or occluded (2/3). (c) Spotify / Gemini 2.5 Pro, create_playlist_options → home; (d) Uber Eats / GPT-5 Mini, profile → uber_one_membership. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: C3: target unreachable or occluded (3/3). (e) WhatsApp / Gemini 2.5 Flash, settings → account_settings. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: C4: target clickable but unresponsive (1/3). (a) Airbnb / GPT-5, profile → profile_details; (b) Coinbase / Gemini 2.5 Pro, portfolio_overview → notifications_page. The page renders, the affordance is in the right place, but the click registers no effect, consistent with a missing or mis-wired onClick handler (§7). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: C4: target clickable but unresponsive (2/3). (c) Etsy / Claude Opus 4.6, favorites → favorites_collections; (d) Notion / Claude Opus 4.6, home → document_editor_block_selection_overlay. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: C4: target clickable but unresponsive (3/3). (e) TikTok / Gemini 2.5 Flash, profile → following_list. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 9 linked inside Pith

  1. [1]

    Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) , year=

    pix2code: Generating Code from a Graphical User Interface Screenshot , author=. Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) , year=. doi:10.1145/3220134.3220135 , eprint=

    arXiv

  2. [2]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=. 2403.03163 , archivePrefix=

    Pith/arXiv arXiv

  3. [3]

    Unlocking the conversion of Web Screenshots into HTML Code with the

    Lauren. Unlocking the conversion of Web Screenshots into HTML Code with the. arXiv preprint arXiv:2403.09029 , year=

    Pith/arXiv arXiv

  4. [4]

    2025 , doi=

    Gui, Yi and Li, Zhen and Wan, Yao and Shi, Yemin and Zhang, Hongyu and Chen, Bohua and Su, Yi and Chen, Dongping and Wu, Siyuan and Zhou, Xing and Jiang, Wenbin and Jin, Hai and Zhang, Xiangliang , booktitle=. 2025 , doi=

    2025

  5. [5]

    Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST) , pages=

    Rico: A Mobile App Dataset for Building Data-Driven Design Applications , author=. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST) , pages=. 2017 , doi=

    2017

  6. [6]

    Screen2Words: Automatic Mobile

    Wang, Bryan and Li, Gang and Zhou, Xin and Chen, Zhourong and Grossman, Tovi and Li, Yang , booktitle=. Screen2Words: Automatic Mobile. 2021 , doi=

    2021

  7. [7]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle=

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle=. 2024 , eprint=

    2024

  8. [8]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  9. [9]

    2025 , doi=

    Zhang, Chi and Yang, Zhao and Liu, Jiaxuan and Li, Yanda and Han, Yucheng and Chen, Xin and Huang, Zebiao and Fu, Bin and Yu, Gang , booktitle=. 2025 , doi=

    2025

  10. [10]

    2024 , eprint=

    Cheng, Kanzhi and Sun, Qiushi and Chu, Yougang and Xu, Fangzhi and Li, Yantao and Zhang, Jianbing and Wu, Zhiyong , booktitle=. 2024 , eprint=

    2024

  11. [11]

    arXiv preprint arXiv:2401.16158 , year=

    Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception , author=. arXiv preprint arXiv:2401.16158 , year=

    Pith/arXiv arXiv

  12. [12]

    2024 , doi=

    Baechler, Gilles and Sunkara, Srinivas and Wang, Maria and Zubach, Fedir and Mansoor, Hassan and Etter, Vincent and Cărbune, Victor and Lin, Jason and Chen, Jindong and Sharma, Abhanshu , booktitle=. 2024 , doi=

    2024

  13. [13]

    You, Keen and Zhang, Haotian and Schoop, Eldon and Weers, Floris and Swearngin, Amanda and Nichols, Jeffrey and Yang, Yinfei and Gan, Zhe , booktitle=. Ferret-

  14. [14]

    Xie, Tianbao and Zhang, Danyang and Chen, Jixuan and Li, Xiaochuan and Zhao, Siheng and Cao, Ruisheng and Hua, Toh Jing and Cheng, Zhoujun and Shin, Dongchan and Lei, Fangyu and Liu, Yitao and Xu, Yiheng and Zhou, Shuyan and Savarese, Silvio and Xiong, Caiming and Zhong, Victor and Yu, Tao , booktitle=

  15. [15]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle=

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle=

  16. [16]

    Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements , author=. Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

  17. [17]

    Liu, Xiao and Yu, Hao and Zhang, Hanchen and Xu, Yifan and Lei, Xuanyu and Lai, Hanyu and Gu, Yu and Ding, Hangliang and Men, Kaiwen and Yang, Kejuan and Zhang, Shudan and Deng, Xiang and Zeng, Aohan and Du, Zhengxiao and Zhang, Chenhui and Shen, Sheng and Zhang, Tianjun and Su, Yu and Sun, Huan and Huang, Minlie and Dong, Yuxiao and Tang, Jie , booktitle=

  18. [18]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  19. [19]

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle=

  20. [20]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  21. [21]

    Liu, Junpeng and Song, Yifan and Lin, Bill Yuchen and Lam, Wai and Neubig, Graham and Li, Yuanzhi and Yue, Xiang , booktitle=

  22. [22]

    , booktitle=

    Gui, Yi and Zhang, Jiawan and Wang, Yina and Ma, Tianran and Wan, Yao and He, Shilin and Chen, Dongping and Zhao, Zhou and Jiang, Wenbin and Shi, Xuanhua and Jin, Hai and Yu, Philip S. , booktitle=. 2026 , eprint=

    2026

  23. [23]

    2025 , eprint=

    Ge, Tong and Liu, Yashu and Ye, Jieping and Li, Tianyi and Wang, Chao , journal=. 2025 , eprint=

    2025

  24. [24]

    and Yue, Xiangyu , journal=

    Jiang, Yilei and Zheng, Yaozhi and Wan, Yuxuan and Han, Jiaming and Wang, Qunzhong and Lyu, Michael R. and Yue, Xiangyu , journal=. 2025 , eprint=

    2025

  25. [25]

    2025 , eprint=

    Yang, Zhen and Hong, Wenyi and Xu, Mingde and Fan, Xinyue and Wang, Weihan and Cheng, Jiale and Gu, Xiaotao and Tang, Jie , journal=. 2025 , eprint=

    2025

  26. [26]

    2024 , eprint=

    Hong, Wenyi and Wang, Weihan and Lv, Qingsong and Xu, Jiazheng and Yu, Wenmeng and Ji, Junhui and Wang, Yan and Wang, Zihan and Zhang, Yuxuan and Li, Juanzi and Xu, Bin and Dong, Yuxiao and Ding, Ming and Tang, Jie , booktitle=. 2024 , eprint=

    2024

  27. [27]

    2024 , eprint=

    Lu, Yadong and Yang, Jianwei and Shen, Yelong and Awadallah, Ahmed , journal=. 2024 , eprint=

    2024

  28. [28]

    2026 , eprint=

    Tian, Muxin and Wang, Zhe and Yang, Blair and Tang, Zhenwei and Zhu, Kunlun and Dong, Honghua and Li, Hanchen and Xie, Xinni and Wang, Guangjing and You, Jiaxuan , journal=. 2026 , eprint=

    2026

  29. [29]

    2024 , eprint=

    Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Yiming and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Pasari, Nitin and Rane, Chetan and Sampath, Karmini and Krishnan, Maya and Kundurthy, Srivatsa and Hendryx, Sean and Wang, Zifan and Bharadwaj, Vijay and Holm, Jeff and Aluri, Raja and Zhang, Chen Bo Calvin and Jacobson, ...

    2024

  30. [30]

    Ma, Chang and Zhang, Junlei and Zhu, Zhihao and Yang, Cheng and Yang, Yujiu and Jin, Yaohui and Lan, Zhenzhong and Kong, Lingpeng and He, Junxian , booktitle=

  31. [31]

    Executable Code Actions Elicit Better

    Wang, Xingyao and Chen, Yangyi and Yuan, Lifan and Zhang, Yizhe and Li, Yunzhu and Peng, Hao and Ji, Heng , booktitle=. Executable Code Actions Elicit Better. 2024 , eprint=

    2024

  32. [32]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=. 2303.17651 , archivePrefix=

    Pith/arXiv arXiv

  33. [33]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=. 2305.10601 , archivePrefix=

    Pith/arXiv arXiv

  34. [34]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  35. [35]

    Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming , journal=

  36. [36]

    and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H

    Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H. and Li, Fuqiang and Ma, Ren and Zheng, Mingzhang and Qian, Bill and Shao, Yanjun and Muennighoff, Niklas and Zhang, Yizhe and Hui, Binyuan and Lin, Junyang and Brennan, Robe...

  37. [37]

    International Conference on Learning Representations (ICLR) , year=

    Hong, Sirui and Zhuge, Mingchen and Chen, Jiaqi and Zheng, Xiawu and Cheng, Yuheng and Zhang, Ceyao and Wang, Jinlin and Wang, Zili and Yau, Steven Ka Shing and Lin, Zijuan and Zhou, Liyang and Ran, Chenyu and Xiao, Lingfeng and Wu, Chenglin and Schmidhuber, J. International Conference on Learning Representations (ICLR) , year=. 2308.00352 , archivePrefix=

    Pith/arXiv arXiv

  38. [38]

    2024 , eprint=

    Qian, Chen and Liu, Wei and Liu, Hongzhang and Chen, Nuo and Dang, Yufan and Li, Jiahao and Yang, Cheng and Chen, Weize and Su, Yusheng and Cong, Xin and Xu, Juyuan and Li, Dahai and Liu, Zhiyuan and Sun, Maosong , booktitle=. 2024 , eprint=

    2024

  39. [39]

    Chen, Weize and Su, Yusheng and Zuo, Jingwei and Yang, Cheng and Yuan, Chenfei and Chan, Chi-Min and Yu, Heyang and Lu, Yaxi and Hung, Yi-Hsin and Qian, Chen and Qin, Yujia and Cong, Xin and Xie, Ruobing and Liu, Zhiyuan and Sun, Maosong and Zhou, Jie , booktitle=

  40. [40]

    International Conference on Machine Learning (ICML) , year=

    Multi-Agent Architecture Search via Agentic Supernet , author=. International Conference on Machine Learning (ICML) , year=

  41. [41]

    2026 , eprint=

    Li, Chunyang and Zheng, Yilun and Huang, Xinting and Fang, Tianqing and Xu, Jiahao and Song, Yangqiu and Chen, Lihui and Hu, Han , booktitle=. 2026 , eprint=

    2026

  42. [42]

    Lee, Seongyun and Kim, Seungone and Park, Sue Hyun and Kim, Geewook and Seo, Minjoon , booktitle=

  43. [43]

    arXiv preprint arXiv:2502.15210 , year=

    Feizi, Aarash and Rajeswar, Sai and Romero-Soriano, Adriana and Rabbany, Reihaneh and Zantedeschi, Valentina and Gella, Spandana and Monteiro, Jo. arXiv preprint arXiv:2502.15210 , year=. 2502.15210 , archivePrefix=

    Pith/arXiv arXiv

  44. [44]

    2026 , note=

    Anonymous , title=. 2026 , note=

    2026