{"id":"e0e1487b-9708-4ebb-99c2-f62bd6561d91","arxiv_id":"2607.11042","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Agentic LLMs can implement many local backend API behaviors, but under a hardened HTTP oracle the best model fully solves only 28.6% of 56 deployable service tasks.","lead":"BackendForge is a 56-task benchmark that asks agentic LLMs to build Dockerized backend services from a written spec and OpenAPI contract, then scores them only with black-box HTTP tests. It shows top models often pass local API checks but still fail most tasks once harder, contract-grounded service tests are added.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The final-oracle gap may partly measure residual oracle/reference co-construction bias rather than pure model inability to realize the visible contract.","rationale":"The reader correctly identifies the load-bearing assumption: co-evolution admits only specification-grounded requirements. The manuscript’s strongest empirical claim is not merely that final SR is low, but that the large Base\to Final attrition shows models struggle with complete services rather than that the hardened oracle is a construction artifact. That claim is well-motivated and carefully designed (§3.4 failure-first filter + review + regression), and the paper is transparent about 113 filters and residual incompleteness. Still, without an external grounding audit, the causal leap from oracle attrition to model incompleteness remains the single most important soft spot. I agree with the reader’s weakest_assumption and with CONDITIONAL: the contribution is accept-shaped if tasks/oracle/harness are released and readers treat the co-evolved oracle as a strong but imperfect contract approximation. No stronger rejection is warranted—the staged multi-model results, black-box HTTP evaluation, and failure taxonomy still support a real difficulty signal even if some fraction of the gap is oracle-side.","tokens_in":18069,"tokens_out":609,"duration_ms":7020,"concrete_test":"Independently sample 50 admitted co-evolved pytest items and 50 filtered candidates; have two blinded human backend engineers label each as uniquely supported / ambiguous / unsupported by only that task’s released spec.md+openapi.yaml (no reference). If >15% of admitted items are not uniquely supported, or KEEP/FILTER agreement with the paper’s review agent is <0.7 Cohen’s κ, the Base\to Final gap cannot be cleanly attributed to model inability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim treats the Base\to Final SR drop (GPT-5.5 55.4%\to28.6%; larger drops for other models) as evidence that models implement local API behaviors but fail complete deployable services. That inference requires that admitted co-evolved items are uniquely entailed by the visible contract and independent of the construction agents. §3.4 admits a candidate only if it fails the current reference, the review agent judges it contract-supported, and the code agent repairs the reference until full regression passes. §5.2 reports 113 filtered candidates, but provides no independent human audit of KEEP/FILTER decisions, no inter-rater agreement, and no hold-out of co-evolved items against a human-written oracle. Because the same LLM-agent stack proposes tests, judges grounding, and repairs the reference, systematic over-admission of strict-but-plausible behaviors (or under-admission of true gaps) can inflate the final-oracle gap without corresponding to pure model inability on the released contract. The staged tables and failure categories are real; the causal attribution to model incompleteness is the soft spot.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"BackendForge introduces a 56-task benchmark for agentic end-to-end backend service generation. Each task supplies a visible natural-language specification and OpenAPI contract; models must produce a Dockerized Python service that is scored only via black-box HTTP tests after build and deploy. The authors construct a base oracle (7,250 pytest items) and then strengthen it via multi-agent co-evolution (test, review, and code agents), admitting 640 additional items only when they fail the current reference, pass contract-grounding review, and survive full regression after reference repair. Under the base oracle the best model (GPT-5.5) fully solves 55.4% of tasks; under the final oracle success falls to 28.6% (16/56). Failures concentrate on authorization, validation, state consistency, side effects, and multi-step workflows rather than missing endpoints. A limited self-verification protocol recovers some tasks but leaves most unsolved. The paper positions BackendForge as combining rigorous deterministic evaluation, agentic development, and end-to-end deployable output.","tokens_in":18370,"tokens_out":1552,"duration_ms":20661,"significance":"If the construction and evaluation hold, this is a timely and useful contribution to agentic code evaluation. Prior benchmarks emphasize functions, repository patches, or full-stack apps with entangled GUI signals; BackendForge isolates contract realization for deployable services with deterministic HTTP oracles, which is a clean and realistic substrate. Strengths include: a fixed 56-task denominator with substantial API surface (2,345 operations; 24,798 HTTP calls); staged oracles that make the base-to-final attrition explicit (Table 2); explicit review filters against hidden requirements; and failure-mode breakdowns (Table 3, Figs. 3–5) that go beyond pass/fail. The co-evolution methodology is a concrete attempt to harden oracles without silent over-specification. These results would give the community a harder, more service-level signal than unit-test or patch benchmarks alone.","major_comments":[{"comment":"§3.4 and §5.2: The central causal claim—that the Base→Final SR drop (e.g., GPT-5.5 55.4%→28.6%) measures model inability to realize the visible contract—depends on admitted co-evolved items being uniquely entailed by spec.md/OpenAPI and free of construction bias. The paper reports that the review agent filtered 113 candidates and describes KEEP/FILTER criteria, but provides no independent human audit of a sample of KEEP and FILTER decisions, no inter-rater agreement, and no hold-out comparison against a human-written oracle. Because the same agent stack proposes tests, judges grounding, and repairs the reference, residual over-admission of strict-but-plausible behaviors (or under-admission of true gaps) remains a load-bearing threat. A modest audited sample (e.g., 50–100 KEEP/FILTER cases with human labels and disagreement analysis) or a human-oracle subset would substantially strengthen","section":"§3.4 Multi-Agent Oracle Co-Evolution; §5.2"},{"comment":"§4.1–§4.2 and Table 2: Agentic coding runs are known to be high-variance (tool use, exploration order, early stopping). The main results appear to report a single trajectory per model–task pair, with no seed/run variance, confidence intervals, or pass@k-style aggregation. Given that task-level success is all-or-nothing over thousands of pytest items, a single unlucky trajectory can flip a near-pass into a fail and inflate model ranking differences (e.g., GPT-5.5 16/56 vs. Claude Opus 4.7 10/56). At minimum, the paper should state the number of runs explicitly and, where cost allows, report multi-run stability for the top models or a stratified task subset; otherwise the comparative claims should be qualified as single-run point estimates.","section":"§4.1 Experimental Setup; Table 2"},{"comment":"§3.1–§3.2 and Tables 4–7: Tasks are rewritten from real open-source backends to reduce recoverability, which is appropriate, but the manuscript does not quantify residual contamination risk (name/domain leakage, structural similarity to popular repos, or whether models emit framework idioms that match common public implementations). A short contamination check—e.g., n-gram/API-path overlap with upstream sources, or a closed-book probe of whether models can name the source project from the rewritten contract—would better support the claim that success reflects contract realization rather than partial memorization of public backends.","section":"§3.1 Task Selection and Controlled Rewriting"}],"minor_comments":[{"comment":"Table 1: “Rigorous Eval.” / “Agentic” / “End-to-End” are binary checkmarks that compress nuanced differences (e.g., BaxBench vs. BackendForge). A short footnote clarifying the criteria used for each check would reduce ambiguity.","section":"Table 1"},{"comment":"Figure 1 caption and pipeline text: “7,890 PyTest Items / 24,798 HTTP Calls” are aggregate oracle statistics; stating that these are oracle-side totals (not per-submission agent self-tests) would avoid reader confusion.","section":"Figure 1"},{"comment":"§5.3 / Figure 7: The self-verification protocol is valuable but only reported for DeepSeek V4 Pro. A sentence on why other models were not included (cost, API limits) would help interpret generalizability of the +5 recovery result.","section":"§5.3 Hidden Self-Verification"},{"comment":"Appendix A prompts are useful; ensuring that any released artifact includes the exact harness configuration (timeouts, max turns, tool set) would improve reproducibility of the agentic results in Table 2.","section":"Appendix A; §4.1"},{"comment":"Limitations correctly note Python-only scope; consider also noting that five tasks use mock external services, so integration difficulty is only partially represented.","section":"Limitations; §3.2"}],"recommendation":"major_revision","confidential_remarks":"The contribution is real and the empirical setup is carefully designed; I would not reject. The main editorial risk is over-claiming that the final-oracle gap is purely a model capability result without independent validation of co-evolved items. If the authors add a human audit sample of KEEP/FILTER decisions and clarify single-run variance, this could become a strong accept for a systems/SE venue. Model names and dates (GPT-5.5, Claude Opus 4.7, arXiv 2607) are consistent with a 2026 timeline and are not themselves a problem, but the editor may want to confirm that reported model cards and costs are accurately cited in camera-ready."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: BackendForge is a useful, carefully scoped benchmark for whether agentic LLMs can ship a deployable backend that actually satisfies a visible OpenAPI+spec contract under black-box HTTP tests. The headline number is real on the paper’s terms—GPT-5.5 goes from 55.4% base-oracle all-pass to 28.6% on the final oracle—and the failure mass sits where backend work actually hurts: auth isolation, validation, state, side effects, workflows.\n\nWhat is new is not “REST testing” or “agent harnesses” alone. It is the combination: from-scratch Dockerized services (not repo repair), deterministic service-boundary scoring, rewritten contracts so upstream code is not a free pass, and a multi-agent co-evolution loop that only admits tests after failure-on-reference, contract review, and full regression repair. Table 1’s three-axis framing is fair. The construction pipeline (Figs. 1–2), staged oracles, multi-model table, and category breakdowns are coherent and inspectable. Restricting to Python and black-box HTTP is a deliberate control, not a dodge.\n\nSoft spot, in proportion: the stress-test concern lands partially. The final-oracle gap is only as clean as the review agent’s KEEP/FILTER judgments. They report 113 filtered candidates and good process language, but no independent human audit, inter-rater numbers, or hold-out against a human-written oracle. So some of the Base→Final drop could be residual construction strictness rather than pure model inability on the released contract. That weakens causal attribution a bit; it does not erase the staged evidence or make the base-oracle numbers the right success story. Artifact release is promised rather than shown here—standard for this venue stage, still load-bearing for trust.\n\nWho it is for: people building or evaluating coding agents, API/backend synthesis, and SE benchmarks that care about deployable behavior rather than unit-test theater. Math is not the point; the data design is. Citations look appropriate.\n\nI would send this to peer review. Engage with it, use the tasks if they ship cleanly, and treat the final oracle as a strong but still incomplete contract approximation—not as ground truth about all backend competence.","headline":"Solid agentic-coding benchmark with a real empirical gap (55%→29% all-pass); the co-evolved oracle is the contribution and the soft spot, not a reason to ignore the work.","tokens_in":19028,"tokens_out":582,"would_cite":true,"duration_ms":8915,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Agentic LLMs can implement many local API behaviors, but still fail to produce complete deployable backend services on most contract-defined tasks.","keywords":["agentic coding","backend generation","OpenAPI","HTTP oracle","code generation benchmark","LLM evaluation","multi-agent oracle co-evolution","software engineering"],"falsifier":"Have independent auditors re-label a large sample of the 640 admitted co-evolved tests against the released specs: if many enforce behavior not uniquely required by those materials, or if stronger models close most of the base-to-final success gap without those tests being wrong, the claim that models fail at complete services (rather than at an over-specified oracle) would not hold.","tokens_in":18901,"feed_emoji":"⚙️","tokens_out":980,"duration_ms":22197,"temperature":0.7,"pith_summary":"This paper asks whether an agentic LLM can generate a full backend service that builds, deploys, and behaves correctly under execution—not just whether it can write local functions or patches. BackendForge isolates that problem: given a visible natural-language specification and OpenAPI contract, the model must produce a Dockerized service scored only by black-box HTTP tests. The benchmark has 56 tasks rewritten from real open-source applications, plus a multi-agent process that hardens the test oracle while requiring every added test to be grounded in the visible contract. The central result is a sharp drop: the best model succeeds on 55.4% of tasks under the base oracle but only 28.6% under the final oracle. Failures concentrate on service-level semantics—authorization, validation, state consistency, side effects, and multi-step workflows—showing that endpoint-level progress is real but incomplete for complete backend services.","feed_headline":"Best model fully solves only 29% of backend tasks","feed_subtitle":"Hardened HTTP oracles cut success from 55% to 29% by catching auth, state, and workflow bugs.","key_machinery":"Multi-agent oracle co-evolution: a test agent proposes black-box HTTP tests from the visible specification and OpenAPI contract; only candidates that fail the current reference are reviewed; a review agent admits a candidate only if it is uniquely supported by that contract; a code agent then repairs the reference until full regression passes. This process adds 640 admitted items (about 8.8% more tests) that sharply reduce all-pass success without introducing hidden requirements.","core_discovery":"Current agentic LLMs implement many local API behaviors but rarely construct fully correct deployable backend services. Across 56 contract-defined tasks, the strongest model fully passes the final HTTP oracle on only 16 tasks (28.6%), down from 55.4% under the base oracle. The remaining failures are not mainly syntax errors or missing endpoints; they are failures of authorization isolation, validation, state consistency, side-effect propagation, and multi-step workflows.","pith_inferences":["If review filtering is imperfect, part of the base-to-final drop may reflect oracle construction choices rather than pure model inability—so independent re-audit of admitted items would tighten the claim.","Failure concentration on ownership, isolation, and cascades suggests agent scaffolding that models resource graphs and side-effect rules may help more than raw longer context alone.","Python-only Dockerized scoring may understate difficulty or transfer for multi-language production backends with real external dependencies.","The same co-evolution pattern could harden oracles for other contract-rich surfaces (gRPC, database schemas, event buses) without inventing requirements."],"forward_implications":["Endpoint-level progress can look strong while a single failed invariant still blocks full service success.","A modest number of contract-grounded, co-evolved HTTP tests can expose large correctness gaps that base endpoint suites miss.","Sparse hidden self-verification can recover some tasks but does not make the benchmark collapse: most tasks still fail.","Deterministic black-box HTTP scoring against OpenAPI contracts is a workable way to evaluate end-to-end service generation without GUI entanglement.","Deployable backend generation under explicit contracts remains a hard target for agentic coding workflows."],"fun_headline_variants":["Top model fully solves only 29% of backend tasks","Hardened oracles drop backend success from 55% to 29%","Best agentic LLM passes final HTTP oracle on 16 of 56 tasks","Local API wins mask failures on deployable backends","Agentic models rarely ship fully correct backend services"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The hardened oracle only adds requirements uniquely required by the visible specification and OpenAPI contract, which depends on the review agent correctly rejecting unsupported candidates.","fun_headline_variants_meta":{"raw":{"variants":["Top model fully solves only 29% of backend tasks","Hardened oracles drop backend success from 55% to 29%","Best agentic LLM passes final HTTP oracle on 16 of 56 tasks","Local API wins mask failures on deployable backends","Agentic models rarely ship fully correct backend services"]},"model":"grok-4.5","effort":"low","cost_usd":0.005474,"raw_usage":{"total_tokens":1497,"prompt_tokens":829,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":54740000,"prompt_tokens_details":{"text_tokens":829,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":600,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":829,"tokens_out":68,"duration_ms":4549,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T07:26:25.380796+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Have independent auditors re-label a large sample of the 640 admitted co-evolved tests against the released specs: if many enforce behavior not uniquely required by those materials, or if stronger models close most of the base-to-final success gap without those tests being wrong, the claim that models fail at complete services (rather than at an over-specified oracle) would not hold.","supporting_citations":[],"review_version":1}