{"id":"04ed3a65-a83b-4871-9475-9b5c803b9784","arxiv_id":"2505.17040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The dissertation shows that text-centric prompting, inference-time optimization, and correct-by-construction synthetic data can improve LLM robustness and Verilog code generation under resource constraints.","lead":"This doctoral dissertation bundles several methods for making large language models work when data, compute, or input modalities are scarce: text-based multimodal alignment, inference-time optimization, and synthetic data for Verilog code generation. A generalist reader might care because the work tests whether clever prompting and data curation can substitute for model scaling.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verilog SOTA claim rests on decontamination and fair re-evaluation; no artifacts shipped to verify.","rationale":"I agree with the reader that modality-to-text information loss is a real assumption for Chapter 3, but the more load-bearing, falsifiable point is the Verilog SOTA claim: it is a specific numerical claim that can be invalidated by data leakage or evaluation protocol differences. The paper's own text flags decontamination as a concern and confines details to an appendix that is not fully supplied here. The reader's conditional verdict already anticipates this by requesting artifacts; my test specifies the decisive check. Hence no change to the verdict, but the condition should explicitly include a contamination audit and a shared-harness re-evaluation.","tokens_in":47422,"tokens_out":5803,"duration_ms":62120,"concrete_test":"Release the full fine-tuning corpora (SDG, CC, Repair) and the model checkpoint; run a near-duplicate detection (e.g., 13-gram Jaccard overlap after normalizing whitespace and identifiers) between every fine-tuning example and the VerilogEval-Machine/Human and RTLLM prompts, solutions, and testbenches. Also independently recompute pass@1 for the released checkpoint and for each prior baseline using one shared harness (same temperature 0.8, n=20, same iverilog/Quartus simulator and testbench). If any benchmark-derived content appears in training data, or if the re-evaluated baseline numbers differ by more than a rounding error, the SOTA claim is void.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's strongest quantitative evidence is the Starcoder2-15B pass@1 gains over prior SOTA (3.8/10.9/6.6 points, Section 5.3.1). This claim depends on two conditions: (i) the synthetic fine-tuning data (SDG-CC-Repair) contains no near-duplicates of VerilogEval/RTLLM problem statements, reference solutions, or testbenches; (ii) the re-evaluation of prior baselines uses the identical sampling protocol (same temperature, n, pass@k estimator, simulator). The manuscript states decontamination for CC data (\"To prevent data contamination, entries that duplicate the data representations of benchmark problems were excluded\", Section 5.3.5) and claims \"All models are re-evaluated (see Appendix I.1)\", but the appendix is not fully included and no code or dataset is released. Moreover, the targeted Repair data (Section 5.3.5, Figure 5.7) is explicitly constructed from error reports on benchmark problems, so a subtle contamination path exists if any benchmark-derived prompt or testbench survives into the fine-tuning file or the validation step (\"generated code fix will be evaluated for functional correctness\" — against what oracle?). If either condition fails, the SOTA numbers are not trustworthy, and the \"significantly broaden usability\" claim loses its strongest quantitative support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This doctoral dissertation argues that principled methods for alignment, optimization, and synthetic data generation can broadly improve LLM usability under resource constraints. It develops a text-centric multimodal alignment pipeline (TAMML) that converts images, tables, and text into natural language and uses in-context learning to handle modality mismatch and robustness, with adversarial prompting as a robustness mechanism. It also studies inference-time optimization with uncertainty quantification, and addresses Verilog code generation through agent-based repair (RTLFixer), correct-by-construction synthetic data, targeted code-repair data, and data pruning. The most specific load-bearing empirical claims are that TAMML outperforms embedding-based cross-modality baselines under modality mismatch (Tables 3.2 and 3.3), and that a fine-tuned Starcoder2-15B model surpasses prior state-of-the-art pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM by 3.8%, 10.9%, and 6.6% respectively (Section 5.3.1).","tokens_in":47629,"tokens_out":3705,"duration_ms":43646,"significance":"If the claims hold, the dissertation makes several useful contributions. The correct-by-construction synthetic data generation for Karnaugh maps, FSMs, and waveforms is a genuinely valuable idea for low-resource hardware-language modeling, and the use of external benchmarks such as VerilogEval, RTLLM, GSM8K, and StrategyQA means the central comparisons are not definitionally circular. The TAMML framework's modular text-centric design, with its interpretable intermediate representations, is also a sensible alternative to embedding-fusion approaches in dynamic modality settings. However, the manuscript is a compilation of separate studies, and the unified 'usability across resource-constrained settings' claim is only partially tested because the chapters use different tasks, datasets, and baselines. The strongest quantitative result, the Verilog SOTA claim, currently rests on decontamination and re-evaluation procedures that are described only by reference to an appendix that is not present in the submitted text, and no code or data artifacts are provided. Reproducibility and decontamination evidence are therefore essential before the headline claims can be accepted.","major_comments":[{"comment":"The Verilog state-of-the-art claim depends on two conditions that are not verifiable from the submitted manuscript. First, Section 5.3.5 says that CC data excludes entries duplicating benchmark data representations, but the Repair data is explicitly constructed from error reports on benchmark problems (Figure 5.7), and the validation step in Appendix I.49 states that 'the generated code fix will be evaluated for functional correctness' without specifying the oracle or whether any benchmark-derived function is used as an oracle. If any benchmark problem statement, reference solution, or testbench survives into the fine-tuning data or validation filter, the reported pass@1 gains of 3.8%, 10.9%, and 6.6% could be inflated. The table captions state 'All models are re-evaluated (see Appendix I.1)', but Appendix I.1 is not included in the submitted text, and no code or dataset is released. Please provide the full re-evaluation protocol, a precise decontamination description covering the Repair pipeline, and release the fine-tuning data and evaluation harness.","section":"Section 5.3.1 and Tables 5.12-5.13"},{"comment":"The headline cross-modality comparisons are not fully controlled. The embedding-based baselines (SDEdit, DDRM, Idinvert) involve trained generative translation plus fine-tuned downstream models, while TAMML uses a frozen LLM, so the comparison conflates representation choice with training protocol. In addition, the reported numbers are single-point estimates without error bars or significance tests, even though the text states that LLM outputs can vary under identical prompts and temperature (Section 5.2.4). The claimed 21% accuracy improvement on PetFinder and 54% MSE reduction on Airbnb are computed from these single averages. Reporting variance across at least several repeated evaluation runs, and ideally matching the downstream fine-tuning budget or reporting it explicitly, is necessary to support the comparative claims.","section":"Section 3.4.2, Tables 3.2-3.3"},{"comment":"The robustness claims for adversarial prompting are partially self-referential. The adversarial perturbations are generated by an LLM of the same general class used both for the text-centric alignment and for the downstream reasoning, guided by labels and instructions (LLM(x', inst, label, T)). Demonstrating robustness against self-generated perturbations does not necessarily imply robustness against independently constructed or human-authored adversarial inputs. This is a correctness-risk concern rather than a circularity claim about the benchmarks themselves; a concrete test would be to evaluate the trained text-centric pipeline on perturbations produced by a different model family (or by human annotators) and show that the robustness advantage persists.","section":"Section 3.3.1, Eq. (3.2)"},{"comment":"The central premise that modality-to-text transformation preserves task-relevant information is tested only on PetFinder, Airbnb, and Avito, which are all social-adoption or e-commerce datasets with tabular, image, and text inputs. The abstract's claim of extending to 'any modalities' and the inclusion of waveforms and FSM diagrams in the pipeline are not supported by experimental evidence in Chapter 3. At minimum, the text should restrict the generalization claim to the modality types and datasets actually evaluated, or add experiments on at least one structurally different modality, such as audio or time-series, where captioning/serialization loss is known to be more severe.","section":"Section 3.2.3, Eq. (3.1)"}],"minor_comments":[{"comment":"The chapter outline skips Chapter 4 entirely, moving from Chapter 3 to Chapter 5; please correct the outline to include the inference-time optimization chapter.","section":"Section 1.4"},{"comment":"The two sections both carry the title 'Modality Robustness Baselines' and contain overlapping text about MLLMs, robust training, and text-centric strategies; merge them or rename one to avoid duplication.","section":"Sections 3.4.6 and 3.4.7"},{"comment":"The row labels such as 'text+image | tabular' are visually ambiguous in the typeset table; clarifying the notation (e.g., 'train: text+image, test: tabular') would substantially improve readability.","section":"Tables 3.2 and 3.3"},{"comment":"The full text references Appendices I.1 and Appendix H for evaluation details, but these appendices are not included in the submitted manuscript; since the SOTA claim depends on them, at least a complete evaluation-protocol appendix should be part of the manuscript.","section":"References and Appendix I.1"},{"comment":"The model name is written inconsistently as 'Starcoder2', 'StarCoder2', and 'starcoder2-15B'; please unify the spelling and the hyphenation in all tables and text.","section":"Notation throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a compilation of several independent studies, some already published, and the unified narrative is persuasive in structure but not fully demonstrated by the evidence as submitted. The Verilog SOTA claim is the strongest quantitative anchor, but it is exactly the one most vulnerable to contamination concerns, and the missing appendix plus absent artifacts make it impossible to verify. I would encourage the editor to treat artifact release and a complete re-evaluation/decontamination appendix as a hard requirement for revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know: this is a PhD dissertation that repackages the author's published papers — TAMML, RTLFixer, the correct-by-construction Verilog data work, plus data pruning and a short RL chapter — under a \"resource-constrained generalization\" umbrella. If you're expecting a single new technical result, you won't find one. But if you treat it as a unified record of a coherent research program, it is a reasonably careful one.\n\nWhat it does well: the empirical backbone is broad and mostly uses external benchmarks (PetFinder, Airbnb, Avito, VerilogEval, RTLLM, GSM8K, HumanEval/MBPP). TAMML's text-centric alignment is tested against diffusion/GAN translation baselines and against fine-tuned LanguageBind-style alignment, which is a fair comparison set. The Verilog synthetic-data chapters give a clear analysis of why naive SDG fails — non-textual representations and checkpoint variability — and the correct-by-construction pipeline (KMaps, FSMs, waveforms) plus the targeted repair data is a sensible, believable fix. The author also states limitations in the conclusion, which I respect.\n\nSoft spots, in order of importance. The headline SOTA numbers for Starcoder2-15B on VerilogEval and RTLLM rest on decontamination and re-evaluation claims that are not verifiable from the preprint. The appendix that supposedly details re-evaluation of baselines is not fully included, and no code or data is released. The Repair-data generation (Figure 5.7) starts from error reports on benchmark problems; the decontamination sentence in Section 5.3.5 is about the CC data, not the Repair data. If any benchmark-derived prompt or testbench leaks into the fine-tuning set, the 3.8/10.9/6.6 pass@1 gains are inflated. Second, many headline tables report single-point estimates with no error bars or significance tests. Third, the \"significantly broaden usability\" thesis is a meta-claim that is never directly tested — it is an arrangement of results across separate domains, not a single experiment.\n\nNone of this kills the paper. The individual components were peer-reviewed separately, and the framing is honest. But the dissertation as submitted doesn't yet support the strongest quantitative claim in its abstract.\n\nWho's it for: someone entering RTL code generation or modality-mismatch robustness who wants a map of what worked and what didn't. It deserves a serious referee if the artifacts are provided; without them, I'd condition acceptance on releasing data and doing an explicit contamination audit of the Repair set. I'd send it out, but I'd ask for those revisions.","headline":"A solid thesis-style compilation of prior wins; the headline Verilog SOTA needs artifacts and a contamination audit before I'd bank on it.","tokens_in":48160,"tokens_out":2103,"would_cite":true,"duration_ms":21633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This dissertation claims that text-centric representation, inference-time optimization, and correct-by-construction synthetic data — not parameter scale — can make LLMs generalize across modalities, resource constraints, and low-resource…","keywords":["large language models","multimodal alignment","modality mismatch","prompt optimization","uncertainty quantification","RTL code generation","Verilog","correct-by-construction synthetic data"],"falsifier":"Run the TAMML pipeline on a dataset whose label depends on a precise continuous table value (a price, a timing delay) and corrupt the serialization so that value is rounded or dropped; if accuracy collapses toward a constant predictor while an embedding-based model trained on raw values holds its performance, the 'text preserves task-relevant information' premise is falsified. A complementary check for the Verilog claim: score Starcoder2-15B-CC-Repair on freshly written, uncontaminated non-textual RTL problems; if pass@1 falls back to the base model's level, the reported gains came from benchmark overlap rather than new capability.","tokens_in":47195,"feed_emoji":"📝","tokens_out":19700,"duration_ms":162517,"temperature":0.7,"pith_summary":"This dissertation argues that the practical ceiling on large language models in constrained settings is set less by model scale than by how inputs are represented and how training data is built. Its central demonstration is that translating every input modality — images, tables, waveforms — into natural language lets a single frozen LLM handle unseen, noisy, or missing modality combinations without retraining, and that this text-centric pipeline (TAMML) outperforms embedding-based cross-modality translation baselines on three datasets. A second thread shows LLMs can act as optimizers through iterative prompting, and that standard uncertainty metrics measure answer diversity rather than correctness, making them unfit to guide such search. A third thread addresses Verilog, a data-scarce hardware language, with correct-by-construction synthetic problems (Karnaugh maps, finite-state machines, waveforms) and targeted code-repair data, lifting a fine-tuned Starcoder2-15B above prior state-of-the-art pass@1 by 3.8%, 10.9%, and 6.6% on VerilogEval-Machine, VerilogEval-Human, and RTLLM. If these results hold, users facing limited data or compute can buy LLM capability with data curation and prompt engineering instead of scaling models.","feed_headline":"Turn every input into text, and the LLM beats embedding models","feed_subtitle":"Unseen or missing modalities are handled in-context; curated synthetic data lifts Verilog scores by up to 10.9%.","key_machinery":"The load-bearing device is the modality-to-text transformation $F_m(x)$ of Eq. (3.1): a function that converts any modality input — image, table, waveform, or text — into a natural-language summary. Together with the three stages that follow it (text-style translation across modalities, cross-modality summarization, and chain-of-thought reasoning augmentation), this transformation collapses all input heterogeneity into one linguistic space, so a frozen LLM can do the reasoning in-context and the model never needs new weights for new modalities. The same 'turn heterogeneous input into a uniformly verifiable format' move recurs on the code side in the correct-by-construction generators: programs that sample random Boolean minterms, finite-state-machine transition graphs, and waveform timing patterns, then emit problem statements and Verilog solutions whose consistency is guaranteed by construction rather than by the LLM's own verification. A companion repair-data loop writes detailed LLM-generated error reports for the model's own mistakes and injects those same errors into open-source code, producing training pairs aimed at the 'minor' bugs that cause pass-rate volatility.","core_discovery":"The dissertation's central claim is that principled methods in three areas — alignment, optimization, and synthetic data generation — can significantly broaden LLM usability across modalities, resource regimes, and application domains without large-scale retraining or parameter scaling. Concretely, it claims that a frozen LLM prompted with structured natural-language summaries of each input modality beats embedding-based zero-shot cross-modality translation methods (SDEdit, DDRM, Idinvert) under train/test modality mismatch, with roughly 21% relative accuracy gain over the strongest baseline on PetFinder and 54% lower mean squared error on Airbnb, and stays competitive even when train and test modality sets match. It further claims that prompt-space adversarial perturbation, rather than gradient-based robust training, is the effective way to harden this pipeline against noisy, missing, or reordered modalities. On the code side, it claims that LLM-generated synthetic Verilog data is unreliable precisely where hardware problems are hardest — non-textual representations like Karnaugh maps and waveforms — and that replacing it with generator-verified correct-by-construction data plus LLM-injected repair data is what allows a 15B model to surpass prior state of the art on VerilogEval and RTLLM. Finally, it claims that the uncertainty metrics commonly used for LLM reasoning track answer diversity, not correctness uncertainty, so they should not guide prompt optimization; a correctness-aligned metric would hit 50% accuracy at maximum uncertainty on binary tasks.","pith_inferences":["If the text-centric premise holds, the framework's ceiling tracks the quality of the captioners and serializers it leans on, so the pipeline should improve automatically as off-the-shelf captioning improves; the paper's own four-captioner comparison suggests the downstream model is not the bottleneck.","The correct-by-construction recipe should transfer to any machine-checkable low-resource language — VHDL, SystemVerilog assertions, assembly, formal specifications — since it only needs an invertible generator that samples problems with certified solutions; testing it on VHDL would be a direct check of the generality claim.","The paper stops short of building the optimizer its uncertainty analysis points to; a natural next step is an iterative search that asks the LLM for its own correctness probability (via self-consistency over perturbed questions) and uses that to decide when to stop, with the 50%-accuracy-at-max-uncertainty criterion as the acceptance test.","The same adversarial-prompting machinery could double as an automated robustness test-suite for multimodal systems: let the LLM propose semantically plausible corruptions and use the drop in a correctness-aligned uncertainty metric as the pass/fail gate."],"forward_implications":["A deployment that adopts text-centric alignment can add a new modality (audio, sensor streams, video) by writing one captioner-stage prompt and rerunning inference, with no weight updates and no paired data for the new modality.","Prompt-space adversarial perturbation becomes a viable alternative to gradient-based robust training for multimodal inputs, with the extra property that every perturbation is human-readable and attributable.","Correct-by-construction and repair data make fine-tuning effective in low-resource symbolic domains where LLM self-verification is unreliable, because solution correctness is guaranteed by the generator, not by the model.","Because standard uncertainty metrics do not track correctness, iterative prompting guided by a correctness-aligned uncertainty signal is the paper's prescribed — and benchmarked — direction for inference-time optimization.","Extreme data pruning (1% of the MBPP training set retains near-full-data pass@1) implies fine-tuning budgets can be cut by orders of magnitude when clustering plus diversity metrics select the examples."],"supporting_citations":[{"why":"Supplies the text-centric alignment idea and the modality-mismatch taxonomy (fully seen, fully unseen, mixture) that Chapter 3 builds into a full transformation-translation-summarization-reasoning pipeline.","marker":"[138]"},{"why":"SDEdit, a diffusion-based zero-shot cross-modality translation method that serves as a primary embedding-based baseline TAMML must beat in Tables 3.2 and 3.3.","marker":"[88]"},{"why":"DDRM, the second diffusion-based cross-modality translation baseline in the modality-mismatch comparisons.","marker":"[54]"},{"why":"Idinvert, the GAN-inversion translation baseline that completes the zero-shot embedding-based comparison set.","marker":"[177]"},{"why":"MULTIBENCH, whose noisy/missing/dynamic-modality evaluation protocol frames the robustness experiments of Section 3.4.8.","marker":"[66]"},{"why":"VerilogEval, the benchmark (Machine and Human splits) on which the fine-tuned Starcoder2-15B records state-of-the-art pass@1 gains of 3.8% and 10.9%.","marker":"[71]"},{"why":"RTLLM v1.1, the second benchmark where the model gains 6.6% pass@1 and whose released code samples seed the OSS-Instruct data generation.","marker":"[80]"},{"why":"StarCoder2, the open 15B code model that the dissertation fine-tunes on correct-by-construction and repair data to reach state of the art.","marker":"[77]"},{"why":"Self-Instruct, the synthetic-instruction paradigm adapted to Verilog in the SDG pipeline, whose failure on non-textual problems motivates the correct-by-construction data.","marker":"[147]"},{"why":"OSS-Instruct, the open-source-to-instruction method that generates a large fraction of the synthetic Verilog examples and is likewise found insufficient without CC-Repair.","marker":"[152]"}],"fun_headline_variants":["Text-centric alignment lets frozen LLMs handle unseen modalities","Adversarial prompting hardens LLMs against missing and noisy inputs","Correct-by-construction data beats LLM-generated for Verilog code","Uncertainty metrics track answer diversity, not correctness uncertainty","Generalize LLM usability across modalities and resources without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole multimodal argument rests on one premise: that turning each input modality into a written description via the transformation $F_m(x)$ at Eq. (3.1) loses none of the information the task depends on, because if the captioner or table serializer drops exact values, timing relations, or visual details, the frozen text-only LLM has no way to recover them and the claimed advantage over embedding-based systems collapses; the premise is tested on only three datasets (PetFinder, Airbnb, Avito).","fun_headline_variants_meta":{"raw":{"variants":["Text-centric alignment lets frozen LLMs handle unseen modalities","Adversarial prompting hardens LLMs against missing and noisy inputs","Correct-by-construction data beats LLM-generated for Verilog code","Uncertainty metrics track answer diversity, not correctness uncertainty","Generalize LLM usability across modalities and resources without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2313,"prompt_tokens":1093,"completion_tokens":1220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":1135}},"tokens_in":709,"tokens_out":1220,"duration_ms":10949,"temperature":1.0,"reasoning_tokens":1135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:01:50.347283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the TAMML pipeline on a dataset whose label depends on a precise continuous table value (a price, a timing delay) and corrupt the serialization so that value is rounded or dropped; if accuracy collapses toward a constant predictor while an embedding-based model trained on raw values holds its performance, the 'text preserves task-relevant information' premise is falsified. A complementary check for the Verilog claim: score Starcoder2-15B-CC-Repair on freshly written, uncontaminated non-textual RTL problems; if pass@1 falls back to the base model's level, the reported gains came from benchmark overlap rather than new capability.","supporting_citations":[],"review_version":1}