Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Forcing agents to write a behavioral specification before coding improves from-scratch program reimplementation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:02 UTC pith:2XPDFTZM

load-bearing objection Solid multi-model ProgramBench result for a two-stage “spec then code” agent; the win is real, the causal story for SPEC.md quality is only partly locked down. the 3 major comments →

arxiv 2607.27167 v1 pith:2XPDFTZM submitted 2026-07-29 cs.SE cs.CL

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Yihao Chen , Shi Chang , Feng Lin , Khaled Chawa , Boyuan Chen , Shaowei Wang , Ahmed E. Hassan This is my paper
classification cs.SE cs.CL
keywords LLM agentsprogram synthesis from scratchspecification elicitationProgramBenchrequirements engineeringblack-box probingcode agentsbehavioral specification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Building a program from scratch with an LLM agent is much harder than editing an existing codebase. On ProgramBench, agents receive only natural-language documentation and an execute-only binary; even strong models fully solve under 1% of tasks. Single-loop agents mix probing, reading, and coding in one budget, so they under-explore the binary, lose behavioral intent as context grows, and lock in early misreadings. SpecFirst splits the work: a dedicated spec agent first probes the binary and writes a structured SPEC.md; only then does a code-synthesis agent implement against that artifact. Across all 200 instances and four models, this raises average test pass rates by 6.9%–21.3% and binary line coverage by 9.4%–18.5%. With a prior specification, the coder also starts writing earlier and builds larger final codebases. The paper’s claim is that classical requirements elicitation should be a first-class phase for agentic from-scratch construction.

Core claim

An explicit two-stage pipeline that first elicits a structured behavioral specification from documentation plus black-box binary probing, then hands that artifact to a code-synthesis agent, consistently beats the single-loop baseline on ProgramBench: higher average test pass rates (6.9%–21.3%), higher probing coverage (9.4%–18.5%), and earlier, more sustained code construction across four models.

What carries the argument

SpecFirst: a dedicated spec agent that systematically black-box probes the binary (boundary, error-path, flag-combination, and output-format probes), writes a six-section SPEC.md (Overview, Flags, Input & stdin, Output format, Error patterns, Edge cases), and only then lets a separate code-synthesis agent implement against that persistent artifact.

Load-bearing premise

The gains come from the quality of the written specification itself, not merely from giving the system a second agent phase and more total inference budget.

What would settle it

A same-total-budget control that runs two agent phases but withholds or randomizes the SPEC.md content: if that control matches SpecFirst’s pass-rate and coverage lifts on the 200 ProgramBench instances, the causal claim for specification-first collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent scaffolds for from-scratch synthesis should treat behavioral elicitation as a separate, forced phase rather than an optional interleaving with coding.
  • A persistent external SPEC.md reduces context-drift and early-error compounding that single-loop agents suffer over long horizons.
  • Dedicated probing raises exercised binary line coverage, which in turn correlates with larger, higher-pass final implementations.
  • The same split can be applied whenever any queryable behavioral oracle (binary, API, service) exists even if source is unavailable.
  • Failure modes shift: many remaining errors become execution-stage mismatches against a correct spec rather than pure discovery failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dominant residual failure is faithful implementation of a correct spec, the next leverage is likely test-driven repair and self-consistency against SPEC.md, not only better probing.
  • The approach suggests a general pattern for other long-horizon agent tasks where written requirements are incomplete: insert an active elicitation stage before commitment to a solution artifact.
  • Cost rises with the extra phase, so practical deployments may need adaptive early-stop rules that halt elicitation once marginal coverage gains flatten.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that from-scratch program construction under ProgramBench-style conditions (NL documentation plus an execute-only binary) fails in single-loop agents because elicitation, exploration, and coding are entangled. It proposes SpecFirst: a dedicated spec agent that black-box probes the binary and writes a structured SPEC.md (Sections format), followed by a code-synthesis agent (mini-SWE-agent) that implements against documentation, binary, and SPEC.md. On all 200 ProgramBench instances and four models (Qwen3.5-397B, Qwen3.6-35B, GPT-5.5-high, GPT-5.4-mini), SpecFirst raises mean test pass rate by 6.9%–21.3% and probing coverage by 9.4%–18.5% versus Direct-Synthesis (same scaffold, no prior elicitation), with paired Wilcoxon significance, win/loss/tie counts, difficulty slices, LOC-vs-progress curves, a small format ablation, cost breakdown, and a 50-case failure taxonomy (F1–F5).

Significance. From-scratch behavioral re-implementation is a genuine open problem; ProgramBench’s near-zero full-resolve rates make partial pass-rate and coverage gains scientifically useful. The work’s main strengths are experimental: full 200-instance suite, two model families and a wide capability range, identical synthesis scaffold, paired non-parametric tests, language-specific coverage instrumentation, difficulty stratification, and behavioral process measures (when code is written, final LOC). The RE-inspired two-stage decomposition is a clear, reusable systems idea, and the failure analysis honestly shows that execution-stage faults (F4, 52%) dominate even when the spec is adequate. If the causal story holds, SpecFirst is a practical paradigm shift for agent pipelines that currently conflate discovery and implementation.

major comments (3)
  1. [§V RQ1/RQ4; §VII-D Internal Validity] Central causal claim vs. controls (§I, RQ1/Table II, RQ4, §VII-D): The manuscript credits the elicited behavioral specification (and “requirements-engineering phase”) for the gains, but the only internal-validity defense against extra phase/budget is that Direct-Synthesis never hits the 1000-turn cap and self-terminates at median 22–177 turns. That rules out simple turn starvation in the single loop; it does not rule out (a) forced explore-then-build structure, (b) fresh context between phases, or (c) a second independent attempt. §VII-B only ablate SPEC.md shape (Sections/Freeform/OpenSpec) inside SpecFirst on 50 instances—not a matched-budget sham first phase (empty SPEC.md, doc-only summary, or raw probe transcript). Without that control, or a clearly softened claim limited to “two-stage elicitation+synthesis beats single-loop,” the load-bearing “specification artifact first” interpre
  2. [§VII-A; Abstract; §VIII] What the pass-rate lift actually measures relative to the paradigm claim (§IV-C Eq. 2, §VII-A): Average test pass rate awards partial credit and is appropriate given <1% full resolve, but failure taxonomy on 50 cases attributes 52% of failures to F4 (correct Spec.md, wrong implementation) and only 40% combined to spec omission/wrong/inaccurate description (F1–F3). The paper notes this, yet the abstract/conclusion still present “explicit requirements-engineering phase” as the effective paradigm for construction. Please either (i) quantify how much of the Table II delta is explained by coverage/spec quality vs. synthesis behavior change, or (ii) rebalance claims so that SpecFirst is framed as necessary scaffolding whose residual bottleneck is execution-stage fidelity—consistent with your own F4 dominance.
  3. [§V RQ3; Table V] Probing coverage as evidence for the spec agent (§IV-C Eq. 3, Table V): Union coverage gains of 9.4%–18.5% and “spec agent alone exceeds synthesis agent” are important, but coverage is line coverage of the reference binary under the authors’ instrumentation, not a direct measure of SPEC.md completeness or fidelity. Given F2/F3 (wrong/inaccurate specs that still “cover” lines), please report at least a coarse link between per-instance coverage (or spec length/sections filled) and pass-rate delta, or human/LLM judged spec-quality scores on a subsample, so RQ3 supports the mechanism rather than only showing more probing occurred in a dedicated phase.
minor comments (6)
  1. [Abstract; §I; passim] Typographical glitches from macro spacing appear throughout (e.g., “SPECFIRSTon”, “SPECFIRSTconsistently”, “PROGRAMBENCH[5]”). Normalize name macros so PDF spacing is correct.
  2. [Fig. 2] Figure 2 y-axes differ substantially across models; state explicitly in the caption that scales are not comparable across panels and that curves are medians with IQR.
  3. [§V RQ2] Table IV is referenced as difficulty results but the in-text “Table” numbering and the probing table (Table V) should be cross-checked for consistency with the difficulty discussion in RQ2.
  4. [§V RQ1; Listing 2] Listing 1/2 and the gomplate running example are helpful; add a one-line pointer in RQ1 to which model/run produced Listing 2 so readers can align anecdote with Table II.
  5. [§III-D] §III-D termination: “self-declared completion” is acknowledged as a research variable; a short distribution (turns-to-stop, fraction hitting 1000/6h) in an appendix would strengthen reproducibility claims.
  6. [§VI-B] Related work correctly separates protocol RE and invariant mining; a brief note on how SPEC.md differs from executable contracts/tests (you produce NL for the synth agent, not a test suite) would sharpen the BDD contrast.

Circularity Check

0 steps flagged

No circularity: empirical agent comparison on external hidden tests and instrumented binaries; measured deltas are not forced by definition or fit.

full rationale

SpecFirst is an empirical systems paper. Its load-bearing claims are measured outcomes on ProgramBench (average test pass rate vs a hidden suite T; probing coverage via language-specific instrumentation of binary B), not algebraic predictions derived from fitted parameters. The baseline contrast (Direct-Synthesis = same mini-SWE-agent scaffold without a prior SPEC.md) and the reported lifts (Tables II–V: +6.9%–21.3% pass rate; +9.4%–18.5% coverage; Wilcoxon tests) are external evaluations: the agent does not fit coefficients to the scored tests and then restate them as predictions. Design choices (Sections template, 1000-turn/6h caps, deliverable gate, ReAct scaffold) constrain the method but do not make the scored metrics equal the inputs by construction. Self-citations are ordinary related-work framing (SWE-agent, ProgramBench, OpenSpec), not uniqueness theorems that force the result. Causal incompleteness of the internal-validity defense (no sham-spec matched-budget control) is a correctness/identification concern, not circularity under this rubric. No self-definitional loop, fitted-input-as-prediction, or renaming of a known closed-form result appears in the derivation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly experimental design and domain assumptions about ProgramBench-style oracles, not free physical constants. The claim rests on the benchmark’s hidden tests as ground truth, black-box-only interaction rules, the Sections SPEC.md schema, termination/self-judgment of the spec agent, and the assertion that early voluntary stopping means extra baseline compute would not close the gap.

free parameters (4)
  • spec_agent_step_limit = 1000 turns
    Hard cap of 1000 agent–environment turns for specification elicitation; safety net that can truncate incomplete specs.
  • spec_agent_wall_clock_limit = 6 hours
    Six-hour wall-clock cutoff for the spec phase; another completeness/cost tradeoff chosen by authors.
  • deliverable_rejection_cap = 8
    Up to 8 rejections if SPEC.md is missing before recording failure; procedural hyperparameter affecting measured success.
  • Sections_heading_schema = 6 named headings
    Six fixed SPEC.md headings (Overview; Flags; Input & stdin; Output format; Error patterns; Edge cases) chosen as ‘minimal structure’; format ablation shows sensitivity though Sections wins on a 50-instance subset.
axioms (5)
  • domain assumption Hidden ProgramBench tests and instrumented line coverage are valid proxies for behavioral fidelity and exploration quality.
    Central metrics (Eqs. 1–3, §IV-C) treat test pass rate and |C_probed|/|C| as the scientific outcomes.
  • domain assumption Faithful from-scratch reconstruction may be studied under execute-only binary + natural-language documentation without source, network, or disassembly.
    Problem formulation §II-A and shortcut prohibitions §III-B define the allowed information channel.
  • ad hoc to paper Self-declared specification completeness by the LLM spec agent is an acceptable primary termination criterion.
    §III-D admits no semantic oracle for SPEC.md completeness; completeness is the agent’s own judgment.
  • ad hoc to paper Baseline agents’ early voluntary termination implies additional turn budget would not raise Direct-Synthesis pass rates to SpecFirst levels.
    Internal-validity argument in RQ4 and §VII-D; not directly tested with a matched-budget single-agent control that continues probing.
  • standard math Statistical comparisons use paired two-sided Wilcoxon signed-rank tests at α=0.05 without multiple-comparison correction across models/metrics.
    RQ1/RQ3 significance claims; standard non-parametric paired test on instance-level scores.
invented entities (2)
  • SpecFirst two-stage pipeline (spec agent → SPEC.md → code synthesis agent) independent evidence
    purpose: Force behavioral elicitation before implementation and provide a persistent external specification anchor.
    Core proposed method (§III, Fig. 1); evaluated entity rather than a physical postulate.
  • Sections-format SPEC.md behavioral contract independent evidence
    purpose: Standardize what the spec agent must externalize for the exec agent.
    Deliverable format in §III-C; compared to Freeform and OpenSpec in §VII-B.

pith-pipeline@v1.2.0-daily-grok45 · 22224 in / 3686 out tokens · 89620 ms · 2026-07-30T11:02:33.914751+00:00 · methodology

0 comments
read the original abstract

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

Figures

Figures reproduced from arXiv: 2607.27167 by Ahmed E. Hassan, Boyuan Chen, Feng Lin, Khaled Chawa, Shaowei Wang, Shi Chang, Yihao Chen.

Figure 1
Figure 1. Figure 1: Framework of SPECFIRST. On top of the default code synthesis pipeline in the blue box, SPECFIRST introducesspec agent to elicit the specification before implementation in the green box. how a human engineer would interact with an unfamiliar tool, and because it allows the agent to chain commands (e.g., constructing input files, running the binary, and inspecting its output in one step). For interactive or … view at source ↗
Figure 2
Figure 2. Figure 2: Median reconstructed codebase size (lines of code) as a function of normalised run progress (0 = start, 1 = termination), [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of the code synthesis phase between [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 11 linked inside Pith

  1. [1]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    2024

  2. [2]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  3. [3]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    GitHub Copilot: Your AI Pair Programmer,

    GitHub, “GitHub Copilot: Your AI Pair Programmer,” https://github. com/features/copilot, 2021

    2021

  5. [5]

    Programbench: Can language models rebuild programs from scratch?

    J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeveet al., “Programbench: Can language models rebuild programs from scratch?”arXiv preprint arXiv:2605.03546, 2026

    Pith/arXiv arXiv 2026

  6. [7]

    Drift no more? context equilibria in multi-turn llm interactions,

    V . Dongre, R. A. Rossi, V . D. Lai, D. S. Yoon, D. Hakkani-T ¨ur, and T. Bui, “Drift no more? context equilibria in multi-turn llm interactions,” arXiv preprint arXiv:2510.07777, 2025

    arXiv 2025

  7. [8]

    Llms get lost in multi- turn conversation,

    P. Laban, H. Hayashi, Y . Zhou, and J. Neville, “Llms get lost in multi- turn conversation,”arXiv preprint arXiv:2505.06120, 2025

    Pith/arXiv arXiv 2025

  8. [9]

    Solving context window overflow in ai agents,

    A. B. Labate, V . M. de Sousa, S. R. Fiorini, L. G. Azevedo, R. M. Thiago, and V . T. da Silva, “Solving context window overflow in ai agents,”arXiv preprint arXiv:2511.22729, 2025

    arXiv 2025

  9. [10]

    Quantifying laziness, decoding suboptimality, and context degradation in large language models,

    Y . Ma and J.-H. Liu, “Quantifying laziness, decoding suboptimality, and context degradation in large language models,”arXiv preprint arXiv:2512.20662, 2025

    arXiv 2025

  10. [11]

    Requirements engineering: a roadmap,

    B. Nuseibeh and S. Easterbrook, “Requirements engineering: a roadmap,” inProceedings of the Conference on the Future of Software Engineering, 2000, pp. 35–46

    2000

  11. [12]

    Sommerville,Software engineering, 9/E

    I. Sommerville,Software engineering, 9/E. Pearson Education India, 2011

    2011

  12. [13]

    Wiegers and J

    K. Wiegers and J. Beatty,Software requirements. Pearson Education, 2013

    2013

  13. [14]

    Requirements elicitation: A survey of tech- niques, approaches, and tools,

    D. Zowghi and C. Coulin, “Requirements elicitation: A survey of tech- niques, approaches, and tools,” inEngineering and managing software requirements. Springer, 2005, pp. 19–46

    2005

  14. [15]

    Openspec: A lightweight spec-driven framework for ai- assisted development,

    Fission AI, “Openspec: A lightweight spec-driven framework for ai- assisted development,” 2026, accessed: 2026-06-10. [Online]. Available: https://openspec.dev

    2026

  15. [16]

    Github spec kit: Toolkit to help you get started with spec- driven development,

    GitHub, “Github spec kit: Toolkit to help you get started with spec- driven development,” 2026, accessed: 2026-06-29. [Online]. Available: https://github.github.com/spec-kit/

    2026

  16. [17]

    From contract drafting to software specification: Linguistic sources of ambiguity,

    D. M. Berry, E. Kamsties, and M. M. Krieger, “From contract drafting to software specification: Linguistic sources of ambiguity,” 2003

    2003

  17. [18]

    Pragmatic ambiguity detection in natural language requirements,

    A. Ferrari, G. Lipari, S. Gnesi, and G. O. Spagnolo, “Pragmatic ambiguity detection in natural language requirements,” in2014 IEEE 1st International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). IEEE, 2014, pp. 1–8

    2014

  18. [19]

    Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling,

    X. Yang, J. Zhou, M. Pacheco, W. Zhu, P. He, S. Wang, K. Liu, and R. Pan, “Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling,”arXiv preprint arXiv:2510.11838, 2025

    arXiv 2025

  19. [20]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  20. [21]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  21. [22]

    (2026, 4) Introducing gpt-5.5

    OpenAI. (2026, 4) Introducing gpt-5.5. [Online]. Available: https: //openai.com/index/introducing-gpt-5-5/

    2026

  22. [23]

    (2026, 3) Introducing gpt-5.4 mini and nano

    ——. (2026, 3) Introducing gpt-5.4 mini and nano. [Online]. Available: https://openai.com/index/introducing-gpt-5-4-mini-and-nano/

    2026

  23. [24]

    Litellm: Call 100+ llms using the openai input/output format,

    BerriAI, “Litellm: Call 100+ llms using the openai input/output format,” https://github.com/BerriAI/litellm, Aug. 2023. [Online]. Available: https://github.com/BerriAI/litellm

    2023

  24. [25]

    Program synthesis,

    S. Gulwani, O. Polozov, and R. Singh, “Program synthesis,”Foundations and Trends in Programming Languages, vol. 4, no. 1-2, pp. 1–119, 2017

    2017

  25. [26]

    Robustfill: Neural program learning under noisy i/o,

    J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-r. Mohamed, and P. Kohli, “Robustfill: Neural program learning under noisy i/o,” in International conference on machine learning. PMLR, 2017, pp. 990– 998

    2017

  26. [27]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,”arXiv preprint arXiv:2203.13474, 2022

    Pith/arXiv arXiv 2022

  27. [28]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  28. [29]

    Deepseek-coder: when the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: when the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

    Pith/arXiv arXiv 2024

  29. [30]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  30. [31]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  31. [32]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 65 882–65 919

    2025

  32. [33]

    Trae agent: An llm-based agent for software engineering with test-time scaling,

    P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y . Xiao, Y . Liu, Z. Zhang, J. Chen, C. Gaoet al., “Trae agent: An llm-based agent for software engineering with test-time scaling,”arXiv preprint arXiv:2507.23370, 2025

    Pith/arXiv arXiv 2025

  33. [34]

    Creativity and design thinking as facilitators in requirements elicitation,

    E. D. Canedo, A. T. S. Calazans, G. R. S. Silva, P. H. T. Costa, R. P. de Mesquita, and E. T. S. Masson, “Creativity and design thinking as facilitators in requirements elicitation,”International Journal of Software Engineering and Knowledge Engineering, vol. 32, no. 10, pp. 1527– 1558, 2022

    2022

  34. [35]

    Advancing candidate link generation for requirements tracing: The study of methods,

    J. H. Hayes, A. Dekhtyar, and S. K. Sundaram, “Advancing candidate link generation for requirements tracing: The study of methods,”IEEE Transactions on Software Engineering, vol. 32, no. 1, pp. 4–19, 2006

    2006

  35. [36]

    Requirements ambiguity detection and ex- planation with llms: An industrial study,

    S. Bashir, A. Ferrari, A. Khan, P. E. Strandberg, Z. Haider, M. Saa- datmand, and M. Bohlin, “Requirements ambiguity detection and ex- planation with llms: An industrial study,” in2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2025, pp. 620–631

    2025

  36. [37]

    J. F. Smart and J. Molak,BDD in Action: Behavior-driven development for the whole software lifecycle. Simon and Schuster, 2023

    2023

  37. [38]

    Empirical observa- tions on software testing automation,

    K. Karhu, T. Repo, O. Taipale, and K. Smolander, “Empirical observa- tions on software testing automation,” in2009 International Conference on Software Testing Verification and Validation. IEEE, 2009, pp. 201– 209

    2009

  38. [39]

    Polyglot: Automatic extraction of protocol message format using dynamic binary analysis,

    J. Caballero, H. Yin, Z. Liang, and D. Song, “Polyglot: Automatic extraction of protocol message format using dynamic binary analysis,” inProceedings of the 14th ACM conference on Computer and commu- nications security, 2007, pp. 317–329

    2007

  39. [40]

    Prospex: Protocol specification extraction,

    P. M. Comparetti, G. Wondracek, C. Kruegel, and E. Kirda, “Prospex: Protocol specification extraction,” in2009 30th IEEE Symposium on Security and Privacy. IEEE, 2009, pp. 110–125

    2009

  40. [41]

    Mining specifications,

    G. Ammons, R. Bod ´ık, and J. R. Larus, “Mining specifications,”ACM Sigplan Notices, vol. 37, no. 1, pp. 4–16, 2002

    2002

  41. [42]

    The daikon system for dynamic detection of likely invariants,

    M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, “The daikon system for dynamic detection of likely invariants,”Science of computer programming, vol. 69, no. 1-3, pp. 35–45, 2007

    2007

  42. [43]

    Llm4decompile: Decompiling binary code with large language models,

    H. Tan, Q. Luo, J. Li, and Y . Zhang, “Llm4decompile: Decompiling binary code with large language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 3473–3487

    2024

  43. [44]

    Unsu- pervised translation of programming languages,

    B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsu- pervised translation of programming languages,”Advances in neural information processing systems, vol. 33, pp. 20 601–20 611, 2020

    2020