{"id":"0b8abdb3-15bf-49f8-8b9e-1cd49a9069ee","arxiv_id":"2608.04032","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EDATracer is a knowledge-graph-backed AI agent framework for answering questions about chip design artifacts, claiming better accuracy and up to 3.2x lower token cost than Cursor and Claude Code on a new 90-question benchmark.","lead":"This paper presents EDATracer, an AI agent system that searches chip-design files such as source code, logs, netlists, and timing reports to answer engineering questions. It also contributes a large open-source chip design dataset and a 90-question benchmark, reporting better accuracy and 2.0 to 3.2x fewer tokens than Cursor and Claude Code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pass@1 superiority rests on one small author-created benchmark with no confidence intervals; the paper's own LLM-council scoring flips the ranking, so the headline comparison is not yet robust.","rationale":"EDATracer is a substantial systems contribution: the 18.9 GB dataset with 2,787 curated designs, the automated synthesis artifact pipeline, and the KG-versus-RAG ablation give real evidence for the framework's value, and the human-scoring protocol is at least blinded and randomized. My concern is narrower than the reader's but related. The reader flags potential benchmark/scoring bias favoring EDATracer's graph retrieval; I would sharpen that into a statistical and protocol-sensitivity problem. The reported margins are small relative to benchmark size, no confidence intervals or significance tests are given, and the paper's own alternative scoring protocol reverses the pass@1 ranking in a majority of settings. That is direct internal evidence that the result depends on the scoring choice, not merely on an external possibility of bias. I also note the token-efficiency claim has a concrete measurement flaw: Cursor usage is taken from website aggregate statistics rather than from the same benchmark runs, so the 2.0-3.2x reduction is not a controlled comparison. These issues do not refute the framework; they mean the paper supports a conditional acceptance with release of artifacts and independent evaluation, not a definitive superiority claim. The reader's CONDITIONAL verdict therefore remains appropriate, so no verdict change is needed.","tokens_in":25297,"tokens_out":7280,"duration_ms":93069,"concrete_test":"Release the 90 benchmark QA pairs, the per-question pass/fail decisions for every agent/model configuration, and the scoring rubric; have two external EDA engineers score all responses blind to system identity, and compute paired bootstrap 95% confidence intervals for EDATracer minus Cursor and EDATracer minus Claude Code pass@1 under both human and LLM-council rubrics. If either confidence interval includes zero, or if the external scores do not reproduce a positive average margin in at least four of the five shared settings, the headline 'best pass@1' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline pass@1 claim is not yet supported by the reported evaluation. The benchmark is author-created: the appendix states that 'all final benchmark content determined by the authors' and that golden answers were 'manually re-extracted by the authors.' With only 90 questions, the average 6.4 pp pass@1 margin over Cursor corresponds to about 5.8 questions; for a single 90-question setting the standard error of the difference is roughly 6 pp, so several individual comparisons are within sampling noise. More tellingly, the paper's own LLM-council evaluation (Tables 10-11) changes the result: EDATracer leads pass@1 in only 2 of 5 settings, and for Claude Haiku 4.5 and GPT-5.4-mini, Cursor/Claude Code have higher or comparable LLM-council overall averages. The paper attributes this to the council being more lenient about missing/wrong units, but it does not report inter-rater reliability for the human scoring, nor does it validate that the human rubric is insensitive to the same ambiguity. The benchmark was built by the same group that designed the framework, so the combination of small sample, author-controlled gold answers, and scoring-protocol sensitivity means the reported plus signs could shift under an independent evaluation. The token-efficiency half of the central claim is similarly under-controlled: Cursor token counts come from aggregate statistics on the Cursor website, not from measurements on the same 90-question benchmark, so the 2.0-3.2x reduction is not an apples-to-apples comparison. Neither issue proves EDATracer is weak; both show the central superiority claim needs an independent evaluation before it can be taken as established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents EDATracer, an agentic framework that converts EDA artifacts (RTL source, synthesis scripts, logs, netlists, PPA reports) into a knowledge graph plus a semantic vector index, and then uses retrieval and reasoning agents to answer questions that require evidence distributed across multiple artifact types. The authors curate an 18.9 GB dataset of 2,787 open-source chip designs, generate synthesis artifacts for them, and introduce a 90-question benchmark spanning factual, statistical, and reasoning questions. They evaluate EDATracer against Cursor and Claude Code across several LLMs, reporting that EDATracer achieves the best pass@1 accuracy (outperforming Cursor and Claude Code by 6.4 and 7.2 percentage points on average) while using 2.0–3.2x fewer tokens. The paper also includes a RAG-vs-KG ablation, an LLM-council evaluation, and a local open-weight-model experiment with Qwen 3.5-9B.","tokens_in":25619,"tokens_out":6780,"duration_ms":71739,"significance":"If the accuracy and efficiency claims were robustly established, this paper would be a solid contribution to EDA-oriented LLM agents: it provides a large-scale cross-artifact EDA artifact dataset with synthesis logs, netlists, and PPA reports; a graph-and-vector retrieval framework that grounds answers in artifact provenance; and evidence that a small open-weight model can perform reasonably on factual and statistical questions. The paper's strengths include a detailed, largely reproducible dataset curation pipeline, an explicit iterative repair algorithm, a controlled RAG-vs-KG ablation, and candid reporting of the LLM-council divergence. These features are valuable regardless of the exact rank ordering, and the benchmark could become a useful community resource. The main weakness is that the headline numerical claims rest on a small, author-controlled evaluation with no uncertainty quantification and with internal inconsistencies in the token-usage reporting.","major_comments":[{"comment":"The central pass@1 claim is not yet supported by the reported evaluation. The benchmark is only 90 questions (30 per category), all final content and golden answers were determined and verified by the authors (Appendix 'Creation of EDA Evaluation Benchmark'), and no confidence intervals or inter-annotator agreement are reported. For a single 90-question setting the standard error of a difference in pass@1 between two agents with pass rates near 80% is roughly 6 percentage points, so the headline average margin of 6.4 pp over Cursor is about one standard error; several individual comparisons in Table 1 are within sampling noise. The authors should report bootstrap confidence intervals, per-question variance, and ideally independent human scoring or a second annotator with agreement statistics.","section":"Appendix 'Creation of EDA Evaluation Benchmark'; §Evaluation metrics"},{"comment":"The paper's own alternative evaluation method flips the central ranking. Under LLM-council scoring (Tables 10-11), EDATracer leads pass@1 in only 2 of 5 settings, and for Claude Haiku 4.5 and GPT-5.4-mini, Cursor or Claude Code have higher or comparable council averages. The manuscript attributes the discrepancy to the council being more lenient about missing or wrong units, but it does not provide inter-rater reliability for the human scoring nor evidence that the human rubric is insensitive to the same ambiguity. Because human-expert scoring is selected as primary after observing these divergences, the scoring-protocol choice itself needs a principled defense (e.g., pre-registration, blind dual scoring, or unit-error-specific validation).","section":"§LLM Benchmark Evaluation Process; Tables 10-11"},{"comment":"The token-efficiency claims are internally inconsistent and not measured on the same benchmark. RQ2 states that EDATracer 'consistently uses fewer tokens across all evaluated LLMs, using only 20M–36M tokens', but Table 10 reports 64.1M tokens for EDATracer with Qwen 3.5-9B. Even if that row is excluded, the comparison against Cursor is not apples-to-apples: footnote 3 says Cursor token counts come from aggregate statistics on the Cursor website, not from measurements on the same 90-question benchmark. Report measured token usage for every configuration on the same benchmark, and clarify whether the counts are per run or across five repeated runs.","section":"§RQ2: Efficiency of Token Usage; Figure 5; Table 10; footnote 3"}],"minor_comments":[{"comment":"The RAG baseline scores are extremely low (2.58 and 1.47 overall, Table 9), which could indicate an underpowered baseline rather than a genuine advantage of the KG; consider strengthening the RAG baseline (e.g., multi-hop retrieval or re-ranking) and reporting retrieval-recall-oriented metrics.","section":"§Ablation Studies / Database Formulation"},{"comment":"The artifact tree in Figure 4 contains the typo 'READMNE.md'; it should be 'README.md'.","section":"Figure 4"},{"comment":"The text 'Opus 4,7' should read 'Opus 4.7'.","section":"Paragraph above Table 10"},{"comment":"This section does not specify the number of human evaluators, their expertise criteria, or whether any evaluator is an author; please report these details.","section":"§Human Expert Benchmark Evaluation Process"},{"comment":"The definition of pass@k says a question passes if at least one score is 8 or higher, but it does not explain how the five repeated runs are used to compute pass@1 (e.g., best-of-run or majority) or whether pass@5 is computed from the five runs; please clarify.","section":"§Evaluation metrics"},{"comment":"The paper would be easier to reproduce if the benchmark, dataset manifest, and code were released under a persistent identifier; as written, the appendix describes protocols but no public artifact link is given.","section":"Appendix and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be honest and well-scoped, but the central evaluation is not yet rigorous enough for the headline claims. My main concern is the post-hoc choice of human-expert scoring after observing that the LLM-council evaluation flips the ranking, combined with the external Cursor token counts and the internal inconsistency in EDATracer's reported token usage. These issues are fixable within the scope of the manuscript by adding confidence intervals, independent or dual human scoring, and same-benchmark token measurements for all agents. I would not reject the paper, but I would require these changes before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Don't take the headline numbers at face value. The real contribution of this paper is the dataset and the graph-retrieval design, not the pass@1 comparison. The curation funnel—23,115 candidates down to 2,787 synthesizable designs with explicitly listed filter thresholds and a deterministic repair loop—is careful and, in principle, reproducible. The 90-question benchmark is small and author-created, but it fills a real gap: cross-artifact, evidence-grounded EDA QA. The KG-vs-RAG ablation is the strongest evidence in the paper: with LLM, prompts, and rubric held fixed, the graph formulation gives large accuracy gains, especially for smaller models. That is a clean result.\n\nSoft spots are exactly where the stress-test note puts them. The headline accuracy advantage over Cursor and Claude Code is 6–7 percentage points on 90 questions, which is roughly five or six questions, and no confidence intervals are reported. The appendix states that all final benchmark content and gold answers were determined by the authors, so the evaluation is not independent. The paper's own LLM-council scoring flips the ranking for two of five model settings; the authors explain why they trust human scoring, but they don't report inter-rater reliability or validate the rubric against the ambiguity the council exposed. The token comparison is also not apples-to-apples: Cursor token counts come from aggregate website statistics, not from the same 90-question runs, so the 2.0–3.2x reduction is not a measured quantity. These are fixable flaws, but they mean the central claim is not yet robust.\n\nAlso, no code or data is released, which hurts both reuse and verification. The limitations section honestly acknowledges sensitivity to KG quality and offline preprocessing cost, which I credit.\n\nWho is this for? Anyone building LLM agents for hardware design flows or studying structured retrieval for technical document QA. It deserves a serious referee, but the realistic outcome is major revision: release the dataset and code, report confidence intervals and inter-rater reliability, and measure all agents' tokens on the same benchmark. I'd take the dataset seriously even if I remain skeptical of the accuracy comparison.","headline":"Don't take the headline numbers at face value: the dataset and graph-retrieval design are the real contributions, but the accuracy and token claims need independent evaluation before they are established.","tokens_in":26223,"tokens_out":2636,"would_cite":true,"duration_ms":31085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structuring chip-design artifacts as a knowledge graph lets an LLM agent answer EDA artifact questions more accurately than Cursor or Claude Code while using 2.0-3.2x fewer tokens.","keywords":["EDA artifact analysis","knowledge graph","LLM agents","retrieval-augmented generation","chip design","synthesis artifacts","question answering benchmark","token efficiency"],"falsifier":"Have external chip-design engineers, blind to which framework produced each answer, score the same 90 questions with the same rubric: if EDATracer's pass@1 margin over Cursor and Claude Code does not reproduce under blinded scoring, the claimed accuracy advantage fails. A complementary check is to generate a second question set from an independent pool of repositories using the same pipeline and see whether EDATracer's lead survives questions the authors did not write.","tokens_in":25075,"feed_emoji":"⚙️","tokens_out":13741,"duration_ms":126820,"temperature":0.7,"pith_summary":"EDATracer claims that the right retrieval structure, not a bigger model, is the key to making LLM agents reliable at analyzing chip-design artifacts. The paper builds a typed knowledge graph from source files, synthesis scripts, logs, netlists, and PPA reports, pairs it with a semantic vector index, and lets a reasoning agent pull grounded evidence through that structure to answer questions. On a new 90-question benchmark over 2,787 synthesized open-source designs, EDATracer achieves the best pass@1 accuracy, outperforming Cursor and Claude Code by 6.4 and 7.2 percentage points on average while using 2.0-3.2x fewer tokens. The largest gains appear on statistical questions that require gathering and aggregating evidence across many files, and a local open-weight model becomes competitive on factual and statistical questions when paired with the framework. If the claim holds, cheaper, grounded EDA debugging assistance is achievable with retrieval structure rather than raw context.","feed_headline":"Graph agent tops Cursor, Claude Code on chip-design QA","feed_subtitle":"Structuring synthesis artifacts as a knowledge graph grounds LLM answers and cuts token use by 2.0-3.2x.","key_machinery":"The load-bearing object is the knowledge graph $G=(V,E)$ paired with a graph-associated semantic vector index $Z$. Each node $v$ carries metadata $\\phi(v)$ such as artifact type, file path, line span, extracted metrics, and warning labels, and each typed edge $(u,\\rho,v)$ records artifact hierarchy, provenance, or structural relation, so every retrieved chunk resolves back to its design and synthesis run. Retrieval runs in two modes: semantic search over $Z$ by cosine similarity, and structured Cypher queries (a graph query language) over $G$; a reasoning agent inspects the accumulated evidence $E_q^{(i)}$, requests more when it is insufficient, and returns an answer only when the graph context suffices. The offline ingestion pipeline of artifact parsing, provenance linking, and semantic indexing is what turns raw heterogeneous files into this resolvable structure.","core_discovery":"The paper's central claim is that organizing EDA artifacts into a knowledge graph with provenance makes LLM agents answer cross-artifact questions both more accurately and more cheaply than general-purpose coding agents. EDATracer parses each design's ingestion set into an attributed, typed graph $G=(V,E)$ whose nodes are designs, artifact files, text chunks, RTL modules, and netlist elements, with edges encoding hierarchy, provenance, and dependencies, plus a semantic vector index $Z$ over text-bearing nodes. An online loop alternates a reasoning agent, which issues information requests until evidence is sufficient, and a retrieval agent, which answers requests through cosine-similarity search over $Z$ or structured Cypher queries over $G$. In the paper's evaluation, EDATracer achieves the best pass@1 accuracy, outperforming Cursor and Claude Code by 6.4 and 7.2 percentage points on average while using 2.0-3.2x fewer tokens, and an ablation against flat retrieval-augmented generation reports 3.36x and 5.03x improvements in average human-expert score on the two LLM backbones tested.","pith_inferences":["My inference: the graph-plus-vector recipe should transfer to other tool-generated artifact domains, such as software build and test logs, infrastructure telemetry, or simulation output, wherever evidence is scattered across heterogeneous file types tied to a single run; the paper demonstrates the mechanism only on synthesis-stage artifacts.","A natural test of generalizability: the appendix's alternative LLM-council scoring has EDATracer leading pass@1 in only 2 of 5 model settings, so an independently authored benchmark with blind expert scoring would show whether the 6.4- and 7.2-point margins persist.","The paper notes that weaker models benefit less on reasoning questions, which suggests the knowledge graph compensates for retrieval and aggregation failures but not for composing a final explanation; adding an explicit answer-drafting step to the reasoning agent is a concrete next lever.","The future-work sketch of floorplanning, placement, routing, timing-closure, and sign-off artifacts would let the same schema trace timing violations to their sources across the full design flow, since the provenance edges already model design-to-artifact lineage."],"forward_implications":["If the knowledge-graph representation is the cause of the gains, flat retrieval-augmented generation is insufficient for cross-artifact EDA question answering: the paper's ablation reports 3.36x and 5.03x improvements in average human-expert score over RAG on the same LLMs.","Token consumption drops from 54M-115M across the two commercial baselines to 20M-36M for EDATracer on the five LLM settings tested, so per-question cost falls by roughly the same factor.","Local open-weight models become viable for many artifact questions: Qwen 3.5-9B with EDATracer approaches GPT-5.4-mini on factual and statistical questions, though it still lags on reasoning-heavy questions.","The 18.9 GB dataset of 2,787 synthesizable designs with generated synthesis logs, netlists, and PPA reports becomes a reusable public resource for future EDA agent research.","The largest category-level gains are on statistical questions, up to +1.86 points with Claude Haiku 4.5, indicating that multi-artifact aggregation is where grounding matters most."],"supporting_citations":[{"why":"Design Compiler is the synthesis engine whose logs, netlists, and PPA reports form the dataset's generated EDA artifacts.","marker":"[Synopsys 2026]"},{"why":"The commercial coding-agent baseline that EDATracer must beat on accuracy and token usage.","marker":"[Cursor 2026]"},{"why":"Claude Code, the second commercial agentic baseline, is the other head-to-head comparison.","marker":"[Anthropic 2026]"},{"why":"Defines pass@k, the metric from which the paper's pass@1 and pass@5 accuracies are computed.","marker":"[Chen et al. 2021]"},{"why":"Defines retrieval-augmented generation, the flat-retrieval paradigm the ablation uses as the non-graph baseline.","marker":"[Lewis et al. 2020]"},{"why":"Provides the vector-index search and Cypher query language the retrieval agent executes over the knowledge graph.","marker":"[Neo4j 2026]"}],"fun_headline_variants":["Graph-grounded agent tops Cursor, Claude Code on chip QA","Chip-design QA: EDATracer beats Cursor and Claude Code","Knowledge graph agent improves chip QA, cuts tokens 2-3x","EDATracer: graph-indexed AI answers chip design queries better","Chip design: graph agent boosts LLM accuracy, slashes tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy comparison rests on the assumption that the authors' 90-question benchmark and their human-expert scoring are unbiased measures of EDA artifact analysis ability, since the authors wrote and grounded every question themselves and manually re-extracted the gold answers from the artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Graph-grounded agent tops Cursor, Claude Code on chip QA","Chip-design QA: EDATracer beats Cursor and Claude Code","Knowledge graph agent improves chip QA, cuts tokens 2-3x","EDATracer: graph-indexed AI answers chip design queries better","Chip design: graph agent boosts LLM accuracy, slashes tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1487,"prompt_tokens":1002,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":618,"tokens_out":485,"duration_ms":5340,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:12:33.458892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have external chip-design engineers, blind to which framework produced each answer, score the same 90 questions with the same rubric: if EDATracer's pass@1 margin over Cursor and Claude Code does not reproduce under blinded scoring, the claimed accuracy advantage fails. A complementary check is to generate a second question set from an independent pool of repositories using the same pipeline and see whether EDATracer's lead survives questions the authors did not write.","supporting_citations":[],"review_version":1}