{"id":"6cf2f5fe-6d6b-47cc-997f-f1e631ee9a13","arxiv_id":"2608.03078","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LDU-Bench evaluates multimodal LLMs on four lithography review tasks and finds a systematic drop-off from defect triage to morphology, localization, and cause analysis.","lead":"This paper introduces LDU-Bench, a benchmark that tests multimodal AI models on four steps of lithography defect review: spotting defects, naming their shape, locating them, and explaining likely causes. Results show current models can spot defects but fail at the later reasoning steps, revealing where industrial AI review tools break down.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'does not stably transfer' claim rests on aggregate task-score gaps without image-conditional transfer analysis or task-difficulty controls; the observed drop may reflect task difficulty, not a semantic capability break.","rationale":"The paper has real strengths: deterministic scorers, multiple diagnostics, a consistent pattern across six models, and honest reporting of annotation reliability limits. My concern is not that results are fabricated or that the direction is implausible; it is that the central claim's strongest form—'does not stably transfer' and 'systematic capability-structure gap'—requires a comparison that is absent. The reader's weakest assumption (taxonomy validity and sample selection) is related and important, but it is narrower: even with perfect labels and unbiased selection, aggregate task scores across different metrics cannot establish transfer failure. This is why I view the transfer/difficulty confound as the most load-bearing concern. A conditional analysis or a difficulty-matched baseline would settle it. The missing data and code make independent verification impossible now, so the conditional verdict remains appropriate. If the authors release the manifest and show conditional gaps, the claim would be much stronger; if not, the paper should be read as reporting task-level observations rather than a demonstrated capability break.","tokens_in":9934,"tokens_out":4739,"duration_ms":46772,"concrete_test":"Using the released manifest, compute image-conditional transfer statistics: for each image with both Task A and Task B labels, estimate P(B correct | A correct) and compare it with P(B correct | A incorrect) and with marginal P(B correct); repeat for Task C/D on their labeled subsets using DICU > 0.5 as correctness. If conditional accuracy is statistically indistinguishable from marginal (with bootstrap confidence intervals), the 'does not transfer' claim is unsupported. As a second check, construct a difficulty-matched baseline—same class count for B, random/center boxes for C, majority-cause rubric for D—and test whether the observed A-to-B/C/D decline exceeds the baseline decline by a pre-registered margin. If it does not, the semantic capability-structure interpretation should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 interprets the drop from Task A macro-F1 (0.856–0.932) to Task B (0.201–0.408), Task C DICU (≤0.492), and Task D rubric (≤0.409) as evidence that triage ability 'does not stably transfer' to downstream review stages (Abstract; Sec. 4). This inference is not supported by the reported analyses. Task A is a binary decision, Task B is 12-way classification, Task C requires exact xyxy coordinates scored by DICU, and Task D uses a rubric with semantic and keyword components; metric scales and chance baselines differ across tasks. A monotone score decline can arise purely from increasing task difficulty and stricter metrics, without any 'capability-structure gap across semantic levels.' The paper does not report any image-conditional transfer statistic—for example, morphology/localization/cause accuracy restricted to images where triage was correct—even though Tasks A and B share the same 1,761 images and Tasks C/D are subsets. The LCS in Sec. 2.3 is a power mean of four heterogeneous scores; its ranking stability across p (Appendix C) says nothing about whether the A-to-C/D gap reflects semantic structure. The Sec. 3.5 annotation result (79% agreement, kappa 0.727) is relevant but narrower: it bears mainly on Task B label ambiguity, not on the absence of conditional transfer evidence for C/D. The central claim may be true, but the current analysis does not distinguish it from a task-difficulty confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LDU-Bench, a four-task multimodal benchmark for evaluating large language models on lithography defect review. The benchmark uses real IC-SEM images and decomposes review into defect triage (Task A), morphology recognition (Task B), coarse localization (Task C), and image-conditioned cause analysis (Task D), with task-level metrics, diagnostic readouts (CMD, EFS), and a power-mean Lithography Closure Score (LCS). Six MLLMs plus AnomalyGPT are evaluated. The central empirical claim is that current MLLMs achieve relatively high triage performance but this ability does not stably transfer to morphology recognition, coarse localization, or cause attribution, which the paper interprets as a systematic capability-structure gap across semantic levels.","tokens_in":10232,"tokens_out":4380,"duration_ms":41921,"significance":"If the central claim were established, LDU-Bench would be a useful diagnostic resource for industrial MLLM evaluation: it targets a real engineering workflow, uses deterministic scorers rather than LLM judges for the main metrics, evaluates a consistent set of models on shared images, and makes LCS an explicit non-fitted function of task scores, which avoids the circularity that often plagues benchmark-derived conclusions. The diagnostic readouts for morphology prompting, localization utility, and evidence faithfulness are sensible additions. However, the current evidence does not distinguish the claimed semantic capability break from a task-difficulty confound, and several reproducibility-critical pieces (artifact release, appendix tables, ground-truth reliability details) are missing. The contribution is potentially significant, but the load-bearing interpretation is not yet supported.","major_comments":[{"comment":"The central claim that defect-triage ability \"does not stably transfer\" to downstream review stages is not supported by the aggregate task scores as reported. Task A is a binary macro-F1, Task B is a 12-way macro-F1, Task C is DICU on xyxy boxes, and Task D is a rubric score; these have different chance baselines, metric scales, and intrinsic difficulty levels. A monotone score decline can arise simply because the downstream tasks are harder and use stricter metrics, without implying any \"capability-structure gap across semantic levels.\" The paper does not report image-conditional transfer statistics, such as Task B/C/D performance restricted to images where Task A was correct, even though Tasks A and B share the same 1,761 images and Tasks C and D are subsets. I recommend adding conditional analyses, per-task chance baselines, and per-image score correlations to separate a semantic capability break from a task-difficulty confound.","section":"Section 3.4 / Table 1"},{"comment":"The LCS power-mean aggregation cannot by itself establish that the A-to-C/D gap reflects semantic structure. LCS is an explicit function of four heterogeneous task scores, and the reported stability of the model ranking across p in {0.25, 0.5, 0.75, 1.0} only shows robustness to the aggregation exponent; it says nothing about whether the level differences between tasks are due to model capability rather than task properties. The Task D rubric weight (0.7/0.3) is another free parameter with no sensitivity analysis. The paper should report score distributions, chance-level baselines per task, and ideally a matched-image transfer analysis before interpreting the LCS-driven ranking as evidence for a systematic break.","section":"Section 2.3 / Appendix C"},{"comment":"Ground-truth validity is under-supported. Section 2.2 does not describe how the 1,761 images were sampled, how the morphology taxonomy was derived, how the expert cause references were curated, or how the mask annotations and cause labels were verified. Section 3.5 reports only 79.0% exact agreement and Cohen's kappa 0.727 on 100 blind-labeled samples for Task B, without confidence intervals, and without any reliability analysis for Task C masks or Task D cause references. Given that the paper attributes cross-task score gaps to model understanding, the benchmark labels themselves must be shown to be consistent and representative; otherwise the observed gaps could partly reflect taxonomy ambiguity or sample selection.","section":"Sections 2.2 and 3.5"},{"comment":"The manuscript states that detailed output-validity results are reported in \"Appendix Table X\" and that LCS hyperparameter stability is shown in \"Appendix Table Y,\" but those tables are not included in the appendix; only placeholder captions appear. These missing data are load-bearing for the reproducibility claims of 99.88% output validity and ranking stability across p. In addition, the benchmark artifacts are not released or linked anywhere in the paper, even though the Limitations section describes the dataset as \"a publicly releasable lithography-review benchmark.\" The authors should complete the appendix tables and provide a release plan or repository URL, or explicitly state the conditions under which the benchmark will be made available.","section":"Section 3.3 / Appendix Tables X and Y"}],"minor_comments":[{"comment":"The diagnostic figure is referenced as Figure 3 in the text but its caption says \"Figure 2. Diagnostic readouts...\"; the figure numbering should be corrected.","section":"Figure 3"},{"comment":"For AnomalyGPT, Task C requires an xyxy bounding box, but the text only mentions a mask-level AD-Seg summary; the conversion rule from mask predictions to boxes is not specified, which makes the AnomalyGPT DICU comparison less transparent.","section":"Section 3.3 / Table 3"},{"comment":"The Task D rubric defines semantic and keyword-level components but the exact computation of KeyF1 is not specified; please define the keyword matching procedure and the alias/supergroup rules precisely.","section":"Section 2.3 / Task D rubric"},{"comment":"The EFS diagnostic is based on 250 manually scored responses, but no inter-annotator agreement or scoring instruction details are provided for this human judgment; at minimum, report the number of annotators and their agreement.","section":"Section 3.4 / Diagnostic readouts"},{"comment":"Appendix headings A.2 and A.3 appear without introductory sentences; adding one or two sentences linking each heading to the following table or procedure would improve readability.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the benchmark itself is a reasonable contribution and the deterministic-scoring design is a strength. The main issue is that the paper's headline conclusion — a semantic capability-structure gap — is not yet separated from a task-difficulty confound, and the missing appendix tables and artifact release further weaken the reproducibility case. I would condition acceptance on the requested conditional analyses, reliability expansions, and completion of the appendix tables. This is a major revision rather than a rejection because the central claim is defensible and can in principle be supported with additional analyses within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is LDU-Bench itself: four review-chain tasks (triage, morphology, coarse localization, cause analysis) built on real IC-SEM images, with deterministic scorers, diagnostic readouts, and a sensible summary metric. That is new. Existing MLLM anomaly benchmarks like MMAD do not target lithography review or the triage-to-downstream transfer question. The paper also does several things right: no LLM judge for the main metrics, controlled decoding, seeded diagnostics, output validity counted as part of end-to-end reliability, and honest limitations (cross-fab taxonomy, no process logs). The reliability experiment on the morphology taxonomy (79% agreement, kappa 0.727) is a real, if small, check on label quality. The geometric-baseline comparison for localization is also a good idea and shows that large boxes are not enough.\n\nNow the soft spots, in proportion. The biggest one is that the headline claim of a 'systematic capability-structure gap across semantic levels' is not actually established by the reported numbers. Task A is binary, Task B is 12-way, Task C requires an xyxy box scored by DICU, and Task D uses a rubric with semantic and keyword components. The metric scales and chance baselines all differ, so a monotone decline from high triage to low morphology/localization/cause scores could simply reflect increasing task difficulty and stricter scoring. The paper never reports the obvious image-conditional statistic—for instance, morphology or cause accuracy restricted to images where triage was correct—even though Tasks A and B share the same images. I agree with the stress-test note on this point. The diagnostics (CMD, EFS) are informative and do show that definitions help and that visible evidence is present even when cause attribution fails, but they do not fill the missing conditional-transfer analysis.\n\nOther issues are more mundane but real for a benchmark paper: the artifacts (data, code, prompts) are not released; Appendix Table X and Table Y are placeholders; annotation reliability is measured on only 100 samples with no confidence bounds; and no statistical significance tests accompany any of the model comparisons. These are fixable, but they matter for a paper whose value is a reusable evaluation tool.\n\nFor whom is this paper? Industrial practitioners and researchers evaluating MLLMs for semiconductor defect review. They will find the task decomposition and metrics useful even if the headline claim needs tempering. I would send it to peer review—it deserves referee time—but with a clear expectation of major revision: release the artifacts, fill the placeholders, add difficulty controls or conditional analyses, and soften the 'systematic gap' language until it is supported. The core benchmark is a good idea; the evidence for the grander claim is not there yet.","headline":"A genuinely new lithography-review benchmark with deterministic scoring, but the central 'capability-structure gap' claim is not yet cleanly separated from task-difficulty confounds.","tokens_in":10747,"tokens_out":1689,"would_cite":false,"duration_ms":18265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal LLMs can reliably flag a lithography defect yet fail to recognize its morphology, locate it compactly, or attribute an image-supported cause—a systematic capability-structure gap, not noise in one metric.","keywords":["lithography defect understanding","multimodal large language models","benchmark","defect triage","morphology recognition","coarse localization","cause analysis","semiconductor review"],"falsifier":"A fresh annotation study using the same taxonomy on a larger, independently selected sample: if exact agreement among reviewers falls well below 79% (or kappa below roughly 0.5), then low model morphology scores could reflect label ambiguity rather than a model capability gap. Alternatively, if a model fine-tuned on the benchmark's training split reaches near-ceiling scores on B/C/D without changes to Task A, the claim that the gap is systematic and semantic-level-specific would be weakened.","tokens_in":9733,"feed_emoji":"🔬","tokens_out":6395,"duration_ms":52700,"temperature":0.7,"pith_summary":"This paper builds LDU-Bench, a benchmark of real lithography and integrated-circuit scanning-electron-microscope review images, to test whether multimodal large language models can do more than detect a defect. It splits the review workflow into four independent tasks—defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis—scored by deterministic metrics and summarized by a Lithography Closure Score. Across the evaluated commercial and open-weight models, triage is relatively reliable (best macro-F1 0.932) while the downstream stages lag far behind (best morphology macro-F1 0.408, best localization utility 0.492, best cause score 0.409). The paper argues this is not a fluctuation of one metric but a systematic capability-structure gap: models can say 'there is a defect' but cannot stably say what it looks like, where it is, or what visible evidence points to as its cause. The finding matters because lithography engineering review needs an auditable evidence chain, not just a detection flag.","feed_headline":"AI can spot chip defects but can't explain them","feed_subtitle":"On real wafer images, top models detect defects well but stumble at morphology, location, and cause.","key_machinery":"The load-bearing device is the four-task decomposition of the lithography review chain over one shared manifest of real IC-SEM review images, so that each task is scored independently and cascade error is avoided. The overall summary is the Lithography Closure Score (LCS), a power mean of the four task scores (with $p=0.5$). Three diagnostic instruments carry the interpretation: CMD compares label-only versus definition-guided prompts on confusable morphology pairs to separate visual from semantic errors; DICU (Defect Intersection-Coverage Utility) combines IoU and ground-truth coverage so that large boxes are penalized for covering irrelevant background; and EFS (Evidence-Faithfulness Score) audits whether Task D rationales actually cite visible image evidence. The argument works by showing high triage scores coexisting with low downstream scores, and by showing the ranking under LCS is stable across different $p$ values.","core_discovery":"The central discovery is a triage-to-understanding break in current multimodal LLMs. On the same image set, a model that reliably decides whether a defect is present does not reliably recognize the defect's morphology under the benchmark's 11-class taxonomy, does not localize it with a compact bounding box, and does not map visible cues to expert-reviewed cause references. The paper shows the break persists across model families and providers, and diagnostics locate its sources: morphology errors are partly label-semantics confusion, because short definitions improve accuracy; localization failures are about box compactness, because full-image boxes achieve perfect coverage but near-zero utility; and cause-analysis outputs are visually faithful yet still miss the controlled cause labels, so seeing evidence is not the same as attributing cause.","pith_inferences":["If the triage-to-understanding gap is real, one testable prediction is that targeted fine-tuning on morphology labels would raise Task B scores without changing Task A, with the largest gains on confusable pairs such as bump_protrusion versus buried_anomaly.","The 79% exact agreement and kappa 0.727 on 100 blind-labeled samples sets a practical ceiling for morphology recognition; a model's low Task B F1 should be read against that human ceiling rather than against 100%.","A natural stress test is to stratify Task A/B/C scores by circuit layout density or design layer; if the downstream gap widens as backgrounds become denser, that would support the paper's layout-varying framing and point to layout-aware training as a remedy.","Task D deliberately excludes process logs and recipe metadata; extending the cause task with such non-visual evidence would test whether the cause-attribution bottleneck is a visual-evidence limitation or a deeper reasoning limitation."],"forward_implications":["A high defect-triage score should not be treated as evidence that an MLLM is ready for lithography review; evaluation of industrial MLLMs should include morphology, localization, and cause-attribution tasks.","Providing short label definitions alongside morphology categories materially raises accuracy for several models, so label semantics—not only visual discrimination—is a correctable bottleneck.","Specialized anomaly-detection models can beat general MLLMs at coarse localization but cannot complete morphology or cause analysis, so assembling a full review chain currently requires either specialized modules or stronger general vision-language models.","Because LCS rankings are insensitive to the chosen power-mean exponent, the overall model ordering reflects downstream capability accumulation rather than a metric artifact.","Improving triage alone would not improve the review chain; the spread among models is dominated by performance on morphology, localization, and cause attribution."],"supporting_citations":[{"why":"Supplies MMAD, the multimodal-QA benchmark for industrial anomaly detection that LDU-Bench extends from single detection to a four-stage review chain.","marker":"Jiang et al. (2025)"},{"why":"Supplies AnomalyGPT, the anomaly-detection vision-language baseline whose Task A and Task C results anchor the comparison.","marker":"Gu et al. (2024)"},{"why":"Supplies MVTec AD, the standard anomaly-detection dataset whose task framing LDU-Bench moves beyond.","marker":"Bergmann et al. (2019)"},{"why":"Supplies CLIP-style image-text alignment on which the evaluated vision-language anomaly models build.","marker":"Radford et al. (2021)"},{"why":"Supplies the grounded-vision-language evaluation paradigm (Kosmos-2) that Task C's bounding-box protocol follows.","marker":"Peng et al. (2023)"},{"why":"Supplies HallusionBench, the diagnostic approach behind the EFS evidence-faithfulness audit for Task D rationales.","marker":"Guan et al. (2024)"},{"why":"Supplies the object-hallucination evaluation lens used to detect non-visual inference in generated explanations.","marker":"Li et al. (2023)"},{"why":"Provides the kappa coefficient used in the Task B blind-label reliability analysis.","marker":"Cohen (1960)"}],"fun_headline_variants":["MLLMs detect chip defects, but fail to understand them","Defect detection easy, explanation hard for multimodal LLMs","LDU-Bench exposes gap between defect triage and diagnosis","See chip defects, can't explain them: LDU-Bench verdict"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's ground truth—in particular the frozen 11-class morphology taxonomy and the expert-reviewed cause references—is a valid and unbiased standard for review understanding; the paper's own blind-label check reaches only 79% exact agreement (kappa 0.727), and the image and cause selection process is not described.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs detect chip defects, but fail to understand them","Defect detection easy, explanation hard for multimodal LLMs","LDU-Bench exposes gap between defect triage and diagnosis","See chip defects, can't explain them: LDU-Bench verdict"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1421,"prompt_tokens":924,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":540,"tokens_out":497,"duration_ms":4716,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:52:18.360527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A fresh annotation study using the same taxonomy on a larger, independently selected sample: if exact agreement among reviewers falls well below 79% (or kappa below roughly 0.5), then low model morphology scores could reflect label ambiguity rather than a model capability gap. Alternatively, if a model fine-tuned on the benchmark's training split reaches near-ceiling scores on B/C/D without changes to Task A, the claim that the gap is systematic and semantic-level-specific would be weakened.","supporting_citations":[],"review_version":1}