{"id":"07430e44-b6ab-44bd-9b7e-c6178c29e340","arxiv_id":"2411.13899","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned LLM converts circuit netlists to LTSpice .asc schematics with higher compilation and structural similarity than GPT-4o and Llama baselines.","lead":"Schemato is a fine-tuned Llama-3.1-8B model that converts SPICE netlists into LTSpice schematic files, reaching 76% compilation success versus 63% for GPT-4o on a filtered test set. The work aims to make machine-generated circuit topologies interpretable to human analog designers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Only human-drawn netlists are used for evaluation; the stated target input (ML-generated netlists) is never tested, so the paper's practical-utility claim remains an extrapolation.","rationale":"The reader's weakest assumption identifies exactly the concern I find most load-bearing: the evaluation distribution differs from the stated target distribution. The paper's experimental design is internally consistent, and the reported metrics are computed plausibly (the CSR scaling correctly converts the compilable-only averages into a mean over all test samples assuming score 0 for non-compilables). No computational error or internal inconsistency is apparent. However, the external validity of the central claim depends on the unstated assumption that human-derived netlists resemble ML-generated ones. The paper provides no evidence for this, and its own limitations section highlights sensitivity to unfamiliar components. A direct test on ML-generated netlists would settle the issue. Since the reader already marked the paper CONDITIONAL based on this gap, my stress-test does not change the verdict; it reinforces the same condition.","tokens_in":12777,"tokens_out":7152,"duration_ms":69338,"concrete_test":"Obtain a set of netlists produced by an analog circuit generation ML model (e.g., the companion GraCo system, arXiv:2411.13890), apply the exact preprocessing of Sec. IV-A, and run Schemato and GPT-4o (with Prompt 5) on them. Compare CSR and CSR-scaled GED/MSSIM against Table I. If Schemato's CSR falls below roughly 50% or its advantage over GPT-4o disappears on this out-of-distribution set, the claimed practical utility for ML-generated netlists is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper motivates Schemato by the need to translate ML-generated netlists into interpretable schematics (Sec. I, Fig. 1), but the evaluation pipeline (Sec. IV-A) builds all inputs by running LTSpice on human-drawn .asc files. The preprocessing removes .lib/.model lines, replaces SpiceModel/ModelFile with InstName, and normalizes coordinates, yielding a simplified format that ML-generated netlists may not follow. ML generators often emit subcircuits, non-default symbols, or arbitrary node/component naming; these characteristics are absent from the 117-circuit test set (Circuits-LTSpice), which is composed of generic components. The paper's own conclusion admits that Schemato 'cannot generate schematics containing less familiar or unfamiliar components' and that over half of the training samples contain low-frequency components. No evaluation is performed on netlists from an ML circuit generator, so distribution shift in naming, topology, and scale is unmeasured. If Schemato's performance degrades on such inputs, the central claim of enabling human interpretability of ML-generated circuits is not established. This is the most load-bearing gap because it targets the stated purpose of the system rather than an internal metric artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Schemato, a fine-tuned Llama-3.1-8B model that converts SPICE netlists into LTSpice .asc schematic files. The authors create a training set from publicly available LTSpice schematics, preprocess and augment it, and evaluate Schemato against pretrained Llama-3.1-8B and GPT-4o on 117 filtered circuits from the Circuits-LTSpice repository. Using four metrics (GED score, MSSIM, compilation success rate, and BLEU), the paper reports that Schemato achieves 76% CSR and higher GED/MSSIM scores scaled by CSR than the best pretrained models (1.8x and 4.3x, respectively). The paper concludes that Schemato can generate interpretable schematics from netlists and discusses limitations related to component familiarity and circuit size.","tokens_in":13132,"tokens_out":4059,"duration_ms":40098,"significance":"If the reported results hold, the paper presents a potentially useful application of LLMs to an underexplored EDA task: translating netlists into human-readable schematic files. The systematic comparison of prompting strategies, the construction of a curated training set, and the use of multiple evaluation metrics are strengths. However, the paper's central claim—that Schemato supports interpretation of ML-generated netlists—is not directly tested, and the evaluation methodology has load-bearing weaknesses (test-set-based model selection, no confidence intervals, metric computation on compilable outputs only) that affect the reliability of the quantitative claims. The contribution is timely given growing interest in LLMs for analog circuit design, but the evidence as presented is not yet conclusive.","major_comments":[{"comment":"The final model is selected using test-set metrics: the prompt variant is chosen based on Table I results, and the fine-tuning epoch (epoch 6) is chosen because it 'demonstrated the best combination of scores in Fig. 4 and Fig. 5,' where both figures are computed on the test set. This means the reported test performance is the result of selecting the best model on the same 117 circuits, which inflates the apparent advantage over baselines and makes the exact margins (1.8x, 4.3x, 76% vs 63%) unreliable. The authors should use the validation set for all model-selection decisions and report test performance only for the final selected model.","section":"Sec. V, Figs. 4-5"},{"comment":"The motivating use case is translating ML-generated netlists into schematics, yet the evaluation uses only netlists derived from human-drawn LTSpice schematics (Circuits-LTSpice). The preprocessing in Sec. IV-A removes .lib/.model lines, renames components, and normalizes coordinates, so the resulting netlists are in a simplified format that may not match the output of ML circuit generators, which often emit subcircuits, non-default symbols, or arbitrary naming. No experiment uses netlists from an ML-based generator, so distribution shift in naming, topology, and scale is unmeasured. The paper's own conclusion (Sec. VI) states that Schemato 'cannot generate schematics containing less familiar or unfamiliar components,' which further limits the generality of the practical-utility claim. The authors should either evaluate on ML-generated netlists or explicitly restrict their claims to the tested distribution.","section":"Sec. I, Sec. IV-A"},{"comment":"The GED and MSSIM scores are computed only for compilable outputs, and the headline numbers are 'scaled by CSR' (raw score multiplied by CSR). This composite is a nonstandard heuristic: multiplying a quality score by a compilation rate conflates syntactic correctness with structural/visual similarity, and the direction of the bias is unclear. For example, a model that compiles only easy circuits could obtain a high raw score, while a model that compiles many hard circuits could obtain a lower raw score despite better overall performance. The paper does not report confidence intervals for any metric, which is especially problematic given the small test set (117 circuits). The authors should report raw scores on compilable outputs, provide confidence intervals or error bars, and justify the use of CSR-scaled scores as a primary comparison.","section":"Sec. IV-D, Table I"},{"comment":"The limitations section explicitly acknowledges that over half of the training samples (5,227 of 9,907) contain components appearing fewer than 10 times, and that the model struggles with larger circuits and unfamiliar components. These limitations are central to the claimed generalization: the test set is small and composed of 'generic components,' so the favorable results do not establish that Schemato would succeed on the diverse, large, or ML-generated netlists that the introduction motivates. The paper should either provide evidence on such challenging inputs or temper the abstract's claim that Schemato demonstrates 'the potential of Schemato in reliably generating human-intuitive schematics from netlists' to the specific evaluated setting.","section":"Sec. VI"}],"minor_comments":[{"comment":"The phrase 'average graph edit distance score and mean structural similarity index measure, scaled by the compilation success rate that are 1.8x and 4.3x higher' is grammatically awkward and could be misread as comparing raw scores; it should clearly state that the comparison is on CSR-scaled scores.","section":"Abstract"},{"comment":"The caption says 'Validations losses are computed after every training epoch,' but the figure shows training and validation loss curves; consider labeling the curves explicitly and correcting the typo 'Validations' to 'Validation'.","section":"Fig. 3"},{"comment":"The sentence 'The number of weight updates per epoch is: Ntrain/(Niter*N_GPU*N_batch) = 44,995/(8*8*1) ≈ 703' is missing a division sign in the text; the equation is clear but should be typeset correctly.","section":"Sec. IV-C"},{"comment":"The preprocessing description states 'Remove the lines starting with *, TEXT, RECTANGLE, WINDOW, LINE, and CIRCLE.' It would be helpful to clarify whether this includes lines starting with a semicolon or other comment markers used in SPICE netlists.","section":"Sec. IV-A"},{"comment":"Reference [24] has a typo: 'https//github.com/pytorch/torchtune' is missing a colon after 'https'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The test-set-based model selection is the most serious methodological issue and may require re-running the experiments with proper validation-based selection; if the authors can do that and provide confidence intervals, the paper could become acceptable. The distribution-shift concern is also significant but could be addressed with an additional evaluation or a clearly scoped claim. I would not recommend rejection, as the core idea and dataset are valuable, but the current evidence does not fully support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSchemato is a straightforward, honest paper: fine-tune Llama-3.1-8B (LoRA, 10 epochs) on a few thousand SPICE netlist-to-LTSpice .asc pairs, then compare against GPT-4o and raw Llama on a 117-circuit test set. The result is credible: 76% compilation success versus 63% for GPT-4o with the best prompt, and GED/MSSIM scaled by CSR are 1.8x and 4.3x better. As far as I can tell, this is the first LLM applied to netlist-to-schematic translation; prior work used symmetry heuristics or RL. That is a real, if modest, novelty.\n\nWhat the paper does well: the data pipeline is legible (preprocessing, augmentation via line shuffling, overlap removal), the metrics are defined carefully, and the authors show plots of GED/MSSIM per number of components across epochs. They also admit the hard limit: more than half of the training samples contain components appearing fewer than ten times, and the model cannot produce unfamiliar components. That honesty helps.\n\nThe soft spots are real but not fatal. First, the model selection actually uses the test set: prompt and epoch are chosen from Figs. 4-5, which are computed on the test circuits. The paper states this openly, but it means the quoted margins are optimistic, and with 117 filtered circuits and no error bars I would not take the exact 1.8x/4.3x numbers too seriously. Second, GED and MSSIM are averaged only over compilable outputs, which biases the pretrained baselines downward since they compile less often; the CSR-scaled version helps but is still a chosen correction. Third, and most important, the paper motivates Schemato as a way to make ML-generated netlists interpretable, but every test input is derived from human-drawn LTSpice schematics. ML-generated netlists with subcircuits, odd component names, or non-default symbols never appear. So the central practical-utility claim is an extrapolation, and the gap is unmeasured.\n\nAlso no code or data is released, so reproducibility is limited to the public test set.\n\nWho is this for? People in analog EDA who want a baseline for LLM-based schematic generation, and anyone studying what fine-tuning does to structured file-format translation. It deserves a serious referee; the core result is plausible and the flaws are addressable (train/val split for selection, release artifacts, add an ML-generated-netlist eval set). Send it to review with a request for revision.\n\nRegards,","headline":"A plausible first LLM netlist-to-schematic converter, modestly new and honestly reported, but the headline numbers are weakened by test-set selection and the evaluation never touches ML-generated netlists.","tokens_in":13570,"tokens_out":2898,"would_cite":true,"duration_ms":25969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 8-billion-parameter LLM converts circuit netlists into compilable LTSpice schematics, beating GPT-4o.","keywords":["netlist-to-schematic conversion","large language models","analog circuit design","LTSpice .asc format","fine-tuning","graph edit distance","structural similarity","electronic design automation"],"falsifier":"Run Schemato on a batch of netlists produced by an ML-based circuit generator, not derived from human schematics, and measure its compilation success rate and CSR-scaled GED and MSSIM; if those numbers fall back to pretrained-LLM levels, the claim that Schemato is ready for ML-generated designs would be contradicted.","tokens_in":12576,"feed_emoji":"⚡","tokens_out":10350,"duration_ms":90641,"temperature":0.7,"pith_summary":"The paper aims to show that translating a circuit netlist -- a plain-text list of components and connections -- into a human-readable schematic can be automated by fine-tuning a large language model, rather than by writing heuristic placement rules or relying on a specific circuit topology. It introduces Schemato, an 8-billion-parameter LLM adapted on human-drawn LTSpice schematic pairs, and reports that on 117 test circuits it generates .asc files that compile 76.07% of the time, compared with 63.25% for the best pretrained LLM. When scores are adjusted for compilation success, its connectivity and visual-similarity scores are 1.8x and 4.3x the best pretrained values. The reason this matters is that ML-generated analog circuits are often returned as netlists that human designers cannot inspect directly, so fast schematic recovery would let designer knowledge feed back into ML design loops.","feed_headline":"LLM turns netlists into schematics at 76% compile rate","feed_subtitle":"Schemato fine-tunes an 8B model on human-drawn LTSpice circuits, beating GPT-4o 76.07% to 63.25% on compilation success.","key_machinery":"The machinery is instruction fine-tuning of an 8-billion-parameter language model on paired netlist and .asc data, where the schematic file is treated as the model's target translation. Prompt design carries part of the load: one prompt variant tells the model to begin with 'Version 4' and a SHEET line carrying the reference schematic's sheet size, and that single addition has a large effect on visual similarity in the fine-tuned models. Data augmentation randomly shuffles the order of SYMBOL/SYMATTR lines so the model learns that line ordering is irrelevant to the circuit. The evaluation stack -- graph edit distance for connectivity, SSIM for visual similarity, compilation success rate for syntax, and BLEU for text fidelity -- determines which prompt and training epoch become Schemato.","core_discovery":"The central claim is that netlist-to-schematic conversion can be treated as a text-to-text translation task and solved by supervised fine-tuning of a general LLM, without heuristic layout rules, topology-specific templates, or an explicit aesthetic measure. Schemato starts from the Llama-3.1-8B instruct model and is trained on roughly 45,000 netlist-to-.asc pairs derived from human-drawn LTSpice schematics. On a held-out test set of 117 circuits, it reaches a 76.07% compilation success rate, compared with 63.25% for the best pretrained baseline, and its compilation-scaled average graph-edit-distance score and mean-structural-similarity score are 0.27 and 0.17, 1.8x and 4.3x the best pretrained values. The paper takes this as evidence that a fine-tuned LLM can output schematics that are syntactically valid, topologically faithful, and visually closer to the human reference than any pretrained model.","pith_inferences":["The obvious untested scenario is a netlist produced by an ML circuit generator; the dataset contains only human-drawn schematics, so distribution shift in component naming, scale, or topology could lower the reported 76.07% CSR.","The paper's own analysis shows that more than half of the training samples contain components seen fewer than ten times, suggesting that rare component geometry is a hidden bottleneck and that a larger or more constrained component library would push reliable conversion beyond five-component circuits.","Since GED and MSSIM measure different things, a practical deployment could combine Schemato with a deterministic placement pass to fix geometric layout while keeping the model's topological correctness.","The same fine-tuning recipe could be used to learn a specific designer's layout style by training on that designer's own schematics, turning the model into a personalized drafting aid."],"forward_implications":["A designer working with an ML-generated netlist could receive a compilable LTSpice schematic instead of having to draw one by hand, making human review of ML-designed analog circuits practical.","Because the conversion is framed as translation and the model is fine-tuned on many topologies, the same trained model applies to diverse circuit types without per-topology templates.","The gap in compilation success between fine-tuned and pretrained models shows that most of the improvement comes from task-specific training, not from stronger base-model reasoning.","The approach extends to other text-based schematic formats, since the learning procedure does not depend on LTSpice-specific syntax beyond the training data."],"supporting_citations":[{"why":"Defines the LTSpice .asc file format and supplies the tool used to compile generated schematics for the compilation success rate.","marker":"[9]"},{"why":"Supplies the GPT-4 family baseline that serves as the strongest pretrained comparison for Schemato.","marker":"[11]"},{"why":"Provides the Llama-3.1-8B base model that is fine-tuned into Schemato.","marker":"[12]"},{"why":"Provides the fine-tuning recipe, including chunked loss and low-rank adaptation, used to adapt the base model.","marker":"[24]"},{"why":"Supplies the graph construction and graph edit distance routines used to compare connectivity between generated and reference schematics.","marker":"[27]"},{"why":"Supplies the exact depth-first graph edit distance algorithm on which the GED score computation relies.","marker":"[28]"},{"why":"Defines SSIM, the basis of the visual similarity metric MSSIM used to compare schematic images.","marker":"[29]"},{"why":"Defines BLEU, the text-based N-gram fidelity metric used for evaluation.","marker":"[30]"}],"fun_headline_variants":["Schemato: 76% success turning netlists into readable schematics","LLM beats GPT-4o on netlist-to-schematic: 76% compile rate","Netlist to schematic: Schemato LLM hits 76% success","Fine-tuned 8B LLM converts netlists to schematics 76% of time","Schemato: LLM for netlist-to-schematic conversion reaches 76%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that netlists extracted from human-drawn LTSpice schematics are representative of the netlists that machine-learning circuit generators will hand to Schemato, because the training and test sets contain only human-drawn examples while the motivation is ML-generated designs.","fun_headline_variants_meta":{"raw":{"variants":["Schemato: 76% success turning netlists into readable schematics","LLM beats GPT-4o on netlist-to-schematic: 76% compile rate","Netlist to schematic: Schemato LLM hits 76% success","Fine-tuned 8B LLM converts netlists to schematics 76% of time","Schemato: LLM for netlist-to-schematic conversion reaches 76%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2673,"prompt_tokens":991,"completion_tokens":1682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1569}},"tokens_in":607,"tokens_out":1682,"duration_ms":13732,"temperature":1.0,"reasoning_tokens":1569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:45:17.904068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Schemato on a batch of netlists produced by an ML-based circuit generator, not derived from human schematics, and measure its compilation success rate and CSR-scaled GED and MSSIM; if those numbers fall back to pretrained-LLM levels, the claim that Schemato is ready for ML-generated designs would be contradicted.","supporting_citations":[{"cited_title":"Ltspice simulator,","cited_arxiv_id":null,"evidence_quote":"Defines the LTSpice .asc file format and supplies the tool used to compile generated schematics for the compilation success rate."},{"cited_title":"Automatic analog schematic diagram generation based on building block classification and reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuning recipe, including chunked loss and low-rank adaptation, used to adapt the base model."},{"cited_title":"Exploring network structure, dynamics, and function using networkx,","cited_arxiv_id":null,"evidence_quote":"Supplies the graph construction and graph edit distance routines used to compare connectivity between generated and reference schematics."},{"cited_title":"An Exact Graph Edit Distance Algorithm for Solving Pattern Recognition Problems,","cited_arxiv_id":null,"evidence_quote":"Supplies the exact depth-first graph edit distance algorithm on which the GED score computation relies."},{"cited_title":"String edit distance, random walks and graph matching,","cited_arxiv_id":null,"evidence_quote":"Defines SSIM, the basis of the visual similarity metric MSSIM used to compare schematic images."}],"review_version":1}