{"id":"f02d569f-034c-4e6a-8435-5c3d2705401c","arxiv_id":"2505.14978","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.","lead":"JARVIS is a multi-agent system that uses a domain-adapted language model plus a custom code compiler to generate scripts for Nvidia's private chip-design tools, reporting 92%, 93%, and 81% accuracy on three internal benchmarks it defined. It demonstrates how combining synthetic data, rule enforcement, retrieval, and agent feedback can lift LLM code generation in a niche engineering domain like EDA.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported pass@1 may be conditional on Guardrail acceptance and on benchmark questions generated from the same API graph as the training data; refusal counts and benchmark provenance are missing.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the self-constructed evaluation may not be a valid and unbiased measure of script-generation quality. My concern sharpens this into two concrete, testable threats: (1) the Guardrail Agent's refusal path is unmeasured, so pass@1 may be conditional on accepted outputs, and (2) the evaluation questions and the synthetic training data are both derived from the same tool API graph, so the easy benchmark in particular may reward memorization rather than generalization. These are standard validity concerns for an industrial LLM evaluation, not evidence of fraud, and the paper's internal pipeline is otherwise coherent and reproducible in principle. The conditional verdict should remain, now explicitly conditioned on reporting refusals and on a held-out contamination probe, rather than on accepting the current numbers at face value.","tokens_in":10575,"tokens_out":4904,"duration_ms":48558,"concrete_test":"Re-run the full JARVIS flow on a held-out set of 100 queries about the same tool that are independently authored by engineers not involved in the paper, with guardrail refusals counted as failures and with B-medium/B-hard graded by two independent raters against a pre-registered rubric; if pass@1 drops materially below the Table II values, the headline accuracy is an artifact of the current evaluation protocol, whereas if it stays within a few points the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, as stated in the abstract and Table II, is that the full JARVIS flow achieves 92/93/81 pass@1 on the three benchmarks. For that claim to hold, pass@1 must be an unbiased measure over all benchmark queries, not only over outputs the system chooses to submit. Algorithm 2 (line 17) routes every final answer through the Guardrail Agent (Section IV-6), which replaces low-scoring outputs with the message 'Sorry, I don't have enough knowledge...'. The paper never reports how often this refusal path is taken on each benchmark. If the reported pass@1 figures are computed only over accepted outputs, they overstate task-level accuracy and are not comparable to the single-model baselines in Table I, which always emit a code answer. This is a selection-bias threat, not an allegation of misconduct. A second threat is benchmark-test overlap: the 35,000 synthetic training snippets in Section III were generated from the same tool API graph used to construct the evaluation, and B-easy is scored by string-matching 'desired APIs' (Section V). Without evidence that the evaluation questions were authored independently of the SDG pipeline, the B-easy gain in particular could reflect memorization of the training distribution rather than general script-generation ability. The B-medium and B-hard human evaluation additionally lacks a published rubric and inter-rater statistics, making the 93/81 numbers hard to audit independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents JARVIS, a multi-agent framework for generating Python scripts for an internal, custom VLSI EDA tool. The authors construct a synthetic training set of 35,000 code snippets by randomly traversing the tool's API graph and using an LLM to add comments and questions; they then fine-tune ChipNeMo with this data (DAPT+DSFT). The inference pipeline is a multi-agent ReAct flow that combines a code generator, a RuleEnforce tool, an AST-based code compiler that simulates code and returns corrective feedback, RAG over tool documentation, a code-fixing agent, and a guardrail agent that filters the final output. The system is evaluated on three self-constructed benchmarks (B-easy: 150 questions, B-medium: 30, B-hard: 20) using pass@1 with temperature 0. The full flow achieves 92%, 93%, and 81% on B-easy, B-medium, and B-hard, respectively, compared to best single-model rates of 67%, 62%, and 43% (Tables I and II).","tokens_in":10924,"tokens_out":4361,"duration_ms":37209,"significance":"If the reported results hold, the paper makes a useful engineering contribution: it demonstrates a concrete recipe for adapting LLMs to a data-scarce, domain-specific code-generation setting by combining synthetic data derived from tool documentation with a multi-agent feedback loop that provides structured, compiler- and rule-based corrections. The experimental design has notable strengths: deterministic decoding (temperature 0), pass@1 as the metric, and ablation studies that isolate the contributions of RuleEnforce, RAG, the code-fixing agent, and the initial model. However, the validity of the central claim — that the full JARVIS flow delivers 92/93/81 pass@1 and that these gains represent general improvement rather than measurement artifacts — is currently undermined by three load-bearing evaluation issues: unreported guardrail refusals, benchmark construction that overlaps with the training-data pipeline, and human evaluation without a published rubric or inter-rater statistics. These issues are fixable with additional reporting and analysis, but they must be addressed before the quantitative claims can be accepted.","major_comments":[{"comment":"The Guardrail Agent can decline to answer a query by emitting a standardized message ('Sorry, I don't have enough knowledge...') instead of code. The paper never reports how often this refusal path is taken on each benchmark. If the reported pass@1 figures are computed only over outputs that pass the guardrail, they are not comparable to the single-model baselines in Table I, which always emit a code answer, and they would overstate task-level accuracy. The authors should report the refusal counts per benchmark, and compute pass@1 with refusals counted as failures (or report both conditional and unconditional pass@1).","section":"Section IV-6 and Algorithm 2 (line 17)"},{"comment":"The evaluation benchmarks appear to be constructed from the same tool API graph (man-page/API documentation) that was used to generate the 35,000 synthetic training snippets in Section III. In particular, B-easy is scored by string-matching 'desired APIs', which may reward recall of the training distribution rather than genuine script-generation ability. This is a circularity risk that affects the central claim. The authors should either provide evidence that the evaluation questions were authored independently of the SDG pipeline, or quantify the overlap between the training snippets and the benchmark questions (e.g., API-ngram overlap, question semantic similarity), and report performance on a held-out set of questions that are not derived from the same graph.","section":"Sections III and V"},{"comment":"The B-medium and B-hard results (93% and 81%) rest entirely on human evaluation, but the paper gives no scoring rubric, no description of how many annotators evaluated each answer, and no inter-rater agreement statistics. These numbers are central to the claimed improvement and are not independently auditable. Please provide the evaluation rubric (e.g., what counts as a correct script, how partial credit is handled), the number of independent annotators, and inter-rater agreement (e.g., Cohen's kappa) for both benchmarks.","section":"Section V (B-medium and B-hard)"},{"comment":"The abstract claims 'significant improvements over state-of-the-art domain-specific models,' but the experimental comparison set is limited to GPT-4o, LLaMa 3.1, and three in-house ChipNeMo variants, all evaluated on the authors' own benchmarks. The related work discusses TOOLLLM and CodeAgent, but these existing domain-specific or agent-based approaches are not compared against. Either add comparisons to prior EDA LLM agents that can run on this task, or revise the 'state-of-the-art' claim to be scoped to the systems actually compared.","section":"Abstract and Section II/V"}],"minor_comments":[{"comment":"The configuration values for TimeLimit, ItrLimit, and the number of multi-episode iterations are not reported; please provide these hyperparameters for reproducibility.","section":"Algorithm 2"},{"comment":"The caption contains a typo: 'Linsting 1' should be 'Listing 1'.","section":"Listing 2 caption"},{"comment":"The benchmark sizes are small (B-easy: 150, B-medium: 30, B-hard: 20) and the reported differences between configurations are often a few percentage points; the paper should report exact counts or confidence intervals so that the magnitude of the gains can be assessed.","section":"Section V-A"},{"comment":"The description of the B-easy auto-evaluation is ambiguous: please specify whether 'string match for the desired APIs' means exact API-name matching, whether the order of API calls matters, and how partial matches are scored.","section":"Section V"},{"comment":"Several references have formatting or typographical issues, including [4] with an extra space, [12] with a malformed author entry, and [14] with an incomplete author list; a careful pass over the bibliography is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industrial-systems paper built on a proprietary internal EDA tool; this explains the lack of public benchmarks and code release, but it also means the evaluation is unusually difficult for an outside reader to audit. The guardrail-refusal issue is the most serious: if refusals are excluded from pass@1, the 92/93/81 headline numbers are not an honest task-level accuracy. The benchmark-construction overlap with the SDG pipeline is also a genuine methodological concern, but it is addressable by reporting overlap statistics and held-out results. I would not reject the paper, but the authors need to supply the missing measurement details before the claims are publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's my read on JARVIS. It's a solid industrial application of multi-agent LLM systems to a real bottleneck: generating scripts for a proprietary VLSI EDA tool. The central claim, 92/93/81% pass@1 on the authors' own B-easy/medium/hard benchmarks, is probably too good to take at face value because of how the evaluation is set up, but the underlying engineering is real and the paper deserves a serious look.\n\nThe genuinely new pieces are the random-AST synthetic data generator, which builds syntactically valid code snippets from the tool's API graph and then uses an LLM to add comments and questions, and the AST-based compiler that gives structural feedback on hallucinated attributes. The multi-agent flow is a sensible integration of known components: DAPT/DSFT from ChipNeMo, RAG with BM25+FAISS, ReAct agents, RuleEnforce, and a Guardrail agent. The ablations are clean, and the deterministic decoding (temperature 0) is the right call. The paper is honest about off-the-shelf models doing near zero.\n\nThe soft spots are real and cluster in the evaluation. Two issues stand out. First, the Guardrail agent can refuse to output code, leaving a generic 'Sorry...' message. The paper never reports how often this happens. If pass@1 is computed only over accepted outputs, the numbers are not comparable to single-model baselines that always emit code. That is a selection-bias threat. Second, the evaluation questions and the training data both derive from the same API graph. B-easy is scored by string-matching desired APIs, so a high score there could partly reflect memorization of the training distribution rather than general script-generation ability. The human evaluation on B-medium and B-hard lacks a published rubric and inter-rater statistics, which makes the 93/81 numbers hard to audit.\n\nThese are not signs of misconduct — they are standard limitations of industrial ML work on proprietary tools. But they mean the headline numbers should be treated as upper bounds until refusal rates and benchmark provenance are clarified. The paper would benefit from releasing benchmarks (or an independent eval), reporting guardrail refusal counts, and adding error bars.\n\nWho is this for? Researchers and engineers working on domain-specific code generation, especially with custom tools, will get value from the system design and the ablation study. It deserves a serious referee, even though the evaluation needs substantial work before the numbers can be trusted.","headline":"Credible industrial multi-agent code generation paper with clean ablations, but the headline pass@1 numbers are likely overstated until guardrail refusals and benchmark provenance are reported.","tokens_in":11503,"tokens_out":2093,"would_cite":true,"duration_ms":17337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JARVIS claims that coupling a domain-adapted LLM with an AST-based compiler and rule enforcement in a multi-agent loop reaches 92/93/81 pass@1 on EDA script benchmarks.","keywords":["EDA script generation","multi-agent LLM","synthetic data generation","AST-based compiler","retrieval-augmented generation","domain adaptation","chip design automation","code generation"],"falsifier":"Independently re-running the three benchmarks with a blinded scoring rubric and counting guardrail refusals in the denominator would either reproduce the 92/93/81 figures or reveal that selective submission and permissive scoring drive them.","tokens_in":10372,"feed_emoji":"⚙️","tokens_out":5635,"duration_ms":45000,"temperature":0.7,"pith_summary":"The paper tries to show that a domain-specific code assistant for a custom VLSI EDA tool can be built by combining a language model fine-tuned on synthetic data with a set of specialized tools: an AST-based compiler that checks object-attribute relationships, a rule-enforcement module, retrieval, a code-fixing agent, and a guardrail, arranged in a multi-agent refinement loop. If correct, the framework would overcome the data scarcity and hallucination problems that make general LLMs near-useless on proprietary EDA APIs. The central evidence is a set of three internal benchmarks on which the full flow scores 92%, 93%, and 81% pass@1, versus 67%, 62%, and 43% for the best single model without the loop. The authors argue this shows LLMs can be made reliable in specialized engineering domains through tool-assisted self-correction.","feed_headline":"Multi-agent loop hits 92% on EDA script benchmarks","feed_subtitle":"Domain-tuned model plus compiler, rules, and retrieval outdoes single models on a custom VLSI tool.","key_machinery":"The load-bearing mechanism is the AST-based custom compiler and the API graph built from the tool's man page. The compiler converts generated code into an abstract syntax tree, walks the graph checking every object-attribute relationship against the known tool API, and, when a hallucinated attribute appears, returns the list of valid attributes or the shortest path from the object to the desired attribute. This structured feedback is fed into a ReAct-style top agent that iterates over episodes; each episode ends with a progress summary that seeds the next, which prevents the infinite-loop failure mode of naive agent loops. A second mechanism is the synthetic data pipeline: a random code generator traverses the same API graph to produce syntactically valid snippets, and an LLM adds line-by-line comments and generates questions, producing 35,000 extra fine-tuning examples.","core_discovery":"The paper's central claim is that EDA script generation for a proprietary VLSI tool becomes reliable only when a domain-adapted LLM is embedded in a multi-agent refinement loop with tool feedback. The authors report that the full JARVIS flow achieves 92% pass@1 on B-easy, 93% on B-medium, and 81% on B-hard, compared with 67%, 62%, and 43% for the best-performing single model (domain-adapted with synthetic data but no agent loop), and near 0% for off-the-shelf models. The improvement is attributed to each component: synthetic fine-tuning data from an AST-based random code generator plus LLM comments, a custom compiler that simulates code against a tool API graph and returns valid attribute paths, rule enforcement that encodes domain recipes, retrieval from tool documentation, a code-fixing agent that supplies example snippets, and a guardrail that filters final output. The ablations show removing any component lowers accuracy, with RuleEnforce contributing the largest gain on hard questions.","pith_inferences":["The authors do not report how often the Guardrail Agent refuses to answer; since the pass@1 denominator presumably excludes refusals, the 81% B-hard figure may overstate raw generation ability. Publishing refusal counts would make the headline number interpretable.","The B-medium and B-hard scores rely on the authors' own human evaluation with no published rubric or inter-rater statistics; an independent blind re-scoring could shift the numbers.","The same framework should transfer to other data-scarce scripting domains, such as FPGA tooling or EDA physical-design flows, provided an API graph can be extracted from documentation."],"forward_implications":["If the results hold, any proprietary engineering tool with parseable documentation can get a similar assistant: parse the man page into an API graph, generate synthetic snippets, fine-tune a domain model, and wrap it in the agent loop.","The 25-point gain on B-hard from RuleEnforce suggests that for complex, multi-step scripts, explicit domain rules matter more than additional retrieval or examples.","The near-zero scores of off-the-shelf models imply that general LLM coding assistants should not be expected to handle custom tool APIs without domain adaptation and verification tools.","The multi-episode termination summary, by preventing repeated tool calls, makes agent-based code generation more predictable for production use."],"supporting_citations":[{"why":"Supplies the base domain-adapted LLM that JARVIS fine-tunes with synthetic data and uses as its Code Generator.","marker":"[14]"},{"why":"Provides the reasoning-and-acting agent loop that JARVIS modifies into its multi-episode refinement flow.","marker":"[24]"},{"why":"Generates the line-by-line comments and questions that turn random AST code into synthetic SFT training data.","marker":"[19]"},{"why":"Serves as the prior LLM-tool integration baseline that the related work contrasts with the multi-agent approach.","marker":"[21]"},{"why":"Supports the paper's premise that synthetic data generation can boost LLM performance in data-scarce settings.","marker":"[8]"}],"fun_headline_variants":["JARVIS: multi-agent loop lifts EDA script pass@1 to 92%","Multi-agent JARVIS hits 92% pass@1 on EDA scripts","JARVIS: domain LLM plus compiler achieves 92% pass@1 on EDA","92% pass@1: JARVIS multi-agent flow for EDA script generation","JARVIS: agent loop with compiler and retrieval beats single LLMs in EDA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported pass rates assume that the authors' self-constructed benchmarks and their human evaluation of medium and hard outputs are an unbiased measure of script-generation quality.","fun_headline_variants_meta":{"raw":{"variants":["JARVIS: multi-agent loop lifts EDA script pass@1 to 92%","Multi-agent JARVIS hits 92% pass@1 on EDA scripts","JARVIS: domain LLM plus compiler achieves 92% pass@1 on EDA","92% pass@1: JARVIS multi-agent flow for EDA script generation","JARVIS: agent loop with compiler and retrieval beats single LLMs in EDA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000887,"raw_usage":{"total_tokens":3802,"prompt_tokens":893,"completion_tokens":2909,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2796}},"tokens_in":509,"tokens_out":2909,"duration_ms":18859,"temperature":1.0,"reasoning_tokens":2796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:26:02.948730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-running the three benchmarks with a blinded scoring rubric and counting guardrail refusals in the denominator would either reproduce the 92/93/81 figures or reveal that selective submission and permissive scoring drive them.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reasoning-and-acting agent loop that JARVIS modifies into its multi-episode refinement flow."}],"review_version":1}