{"id":"e94df9d4-40a4-49ac-b76e-b718f8f40a56","arxiv_id":"2507.14430","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 32B domain-tuned reasoning model beats DeepSeek-R1-671B on proprietary semiconductor display benchmarks, with an LLM-based evaluation framework and domain RAG.","lead":"TCL built X-Intelligence 3.0, a 32B reasoning model fine-tuned for semiconductor display questions. On in-house benchmarks it scores above DeepSeek-R1-671B, but the evaluation data, code, and model are not public.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing decontamination analysis for the in-house test sets; if test questions overlap with training data, the reported margins over R1-671B reflect memorization rather than generalization.","rationale":"The paper's central claim is that a 32B model, fine-tuned for semiconductor display reasoning, outperforms DeepSeek-R1-671B. That claim stands or falls on the validity of the proprietary evaluation sets. The reader identified the in-house evaluation design as the weakest assumption; I agree, but sharpen the point: no quantitative decontamination is reported. The training and test data are built from overlapping internal sources, and the paper's own dedup machinery (BGE-M3, cosine >0.9) is never applied to the train/test boundary. If overlap exists, the human evaluation tables (2, 3, 5) could be measuring memorization, and the automated evaluation (Sec 5.3) is doubly vulnerable because it scores against R1-generated references while the model was trained on R1-distilled outputs. I do not accuse the authors of dishonesty; the omission may be an oversight, and the 'parallel development' statement suggests intent to avoid contamination. But intent is not evidence. The concrete test is feasible and decisive: apply the same similarity threshold the authors already use for dedup and re-evaluate. This does not change the reader's CONDITIONAL verdict, because the evidence as published remains insufficient to verify the claim, but the concern is specific and testable. I credit the paper for a detailed training pipeline, blind human review, and a clearly described evaluation protocol; these are real strengths. The missing decontamination check is the single most load-bearing gap.","tokens_in":12418,"tokens_out":9879,"duration_ms":119242,"concrete_test":"Run the paper's own deduplication criterion (Sec 2.1: BGE-M3 cosine similarity > 0.9) between every question in the 100/400/800 evaluation sets and every training sample (SFT, RL preference, RAG-SFT). Remove flagged near-duplicates and recompute the human and automated scores. If the margins over DeepSeek-R1-671B shrink or disappear, the central claim fails; if the margins persist, the contamination concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim—a 32B model outperforming DeepSeek-R1-671B on semiconductor-display reasoning—the 100/400/800-question evaluations must measure generalization. The paper never reports a train/test overlap check. Section 2.1 describes training data drawn from the same internal sources (user questions from internal systems, expert-written questions, LLM-extracted questions from internal documents) that plausibly feed the 'parallel' evaluation set (Sec 4). The training pipeline already uses BGE-M3 with a 0.9 cosine-similarity dedup threshold; no analogous threshold is reported between the evaluation questions and the SFT/RL/RAG-SFT corpora. If near-duplicates exist, the small margins (Table 2: 5–10 acceptable-rate points; Table 3: 13.3 points; Table 5: 4.9 points) could be generated by recall of memorized content. The concern is compounded by Section 5.3.1, where the 100-question set is explicitly used for iterative model selection before being reported in Table 2; that result is not an independent estimate. The 400/800 results are less affected by selection but still lack any reported contamination screen. This is the most load-bearing concern because it threatens all three human-evaluation tables, which are the primary evidence for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents X-Intelligence 3.0, a 32B-parameter reasoning LLM for the semiconductor display industry, trained via supervised fine-tuning on domain-specific chain-of-thought data distilled from DeepSeek-R1, followed by DPO-based reinforcement learning and a domain-adapted RAG pipeline. The authors also propose an LLM-based automated evaluation framework that decomposes responses into statements and computes precision/recall against reference answers. The central claim is that, despite being far smaller, X-Intelligence 3.0 outperforms DeepSeek-R1-671B on in-house evaluation sets of 100, 400, and 800 domain questions, as measured by expert human scores and an Acceptable Rate metric. Supporting experiments report RAG component ablations, automated-evaluation agreement with human rankings, and iterative model selection results.","tokens_in":12685,"tokens_out":3587,"duration_ms":40461,"significance":"If the headline claim holds, a compact domain-tuned 32B model beating a 671B generalist on semiconductor-display reasoning would be practically valuable, and an automated evaluator that tracks human rankings could substantially reduce expert review costs. The paper deserves credit for a transparent description of the training pipeline, the use of a blind-review protocol, and for reporting both human and automated evaluation rather than relying on a single metric. However, the scientific significance currently hinges on the validity of proprietary, small-margin comparisons that lack contamination analysis, variance reporting, and independence from the teacher model family. The manuscript is best read as a technical report; as a journal contribution, the evaluation methodology needs substantially more rigor before the central comparative claim can be accepted.","major_comments":[{"comment":"No train-test contamination analysis is reported for the evaluation sets. The training data in Sec. 2.1 are drawn from internal-system user questions, expert-written questions, and LLM-extracted questions from internal documents, which overlap in provenance with the 'parallel' evaluation sets described in Sec. 4. The BGE-M3 0.9 cosine-similarity deduplication threshold is applied within training data, but no analogous screen is reported between the evaluation questions and the SFT, RL, or RAG-SFT corpora. Given that the reported margins over DeepSeek-R1-671B are 0.02-0.14 points on a 3-point scale and 5-13 percentage points in Acceptable Rate (Tables 2, 3, and 5), near-duplicate recall could account for the advantage. Please add a deduplication check between every evaluation question and the training corpora, and report results after removing any flagged near-duplicates.","section":"Sec. 2.1, Sec. 4, Sec. 5.1, Tables 2-5"},{"comment":"The 100-question set is first used for iterative model selection ('we selected the aforementioned test set of 100 questions for quick validation') and is then reported in Table 2 as a headline human-evaluation result. Model selection on the same set means the 100-question comparison is an overfit estimate rather than an independent evaluation; the paper should either label Table 2 as a development-set result or select final checkpoints by another criterion and re-evaluate on a held-out set. The 400- and 800-question sets are less affected by selection, but they still lack contamination screens.","section":"Sec. 5.3.1, Table 6, Table 2"},{"comment":"The automated evaluation uses reference answers generated by DeepSeek-R1, which is the same model family as the SFT teacher and the main comparison baseline (R1-671B). The agreement with human rankings reported in Tables 6-8 does not establish that the metric is neutral: key-point inference from R1-generated references may systematically favor R1-style reasoning and phrasing, inflating the candidate's score relative to a non-R1 baseline. Please provide evidence of metric neutrality, for example by running the automated evaluator on a set with expert-written reference answers, or by probing with correct but deliberately non-R1-style responses.","section":"Sec. 5.3.1, Sec. 5.3.2, Eqs. (5)-(7)"},{"comment":"The human evaluation is reported without variance or reliability statistics. Three doctoral experts rate the 100- and 400-question sets and company employees rate the 800-question set, but the paper gives only mean scores and Acceptable Rates, with no inter-annotator agreement, standard errors, or significance tests. Because the headline margins are roughly 0.1 points on a 3-point scale (e.g., Table 2: 2.316 vs. 2.44; Table 5: 2.373 vs. 2.431), the reported differences cannot be distinguished from evaluator noise without such statistics. Please report per-item distributions or confidence intervals and an inter-annotator agreement measure for at least one subset.","section":"Sec. 4.1, Sec. 5.2"}],"minor_comments":[{"comment":"The phrase 'the mode’s fundamental language abilities' should be 'the model’s fundamental language abilities'.","section":"Sec. 4.1"},{"comment":"The table header contains the typo 'Obiective Evaluation Rank'; it should read 'Objective Evaluation Rank'.","section":"Table 6"},{"comment":"The conclusion refers to 'DeepSeek-R1-67B', which appears to be a typo for 'DeepSeek-R1-671B'.","section":"Sec. 6"},{"comment":"The notation 'Nacc≥2, comp≥2, prac≥2' in Eq. (4) is ambiguous; it should clearly define the count as the number of responses whose Accuracy, Comprehensiveness, and Practicality scores are all at least 2.","section":"Eq. (4)"},{"comment":"Figure 1 is labeled 'Benchmark performance of X-Intelligence 3.0' but the axes, dataset names, and error information are not described in the text; please either provide a full caption with data details or remove the figure.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially a technical report with proprietary evaluations and no release of data, code, or models, which limits independent verification. The most important fix for the journal version is to address evaluation independence and statistical rigor; if the authors can add contamination checks, error bars, and an external or independently authored reference set, the paper could be suitable. There is also a scope question: the training and RAG engineering contributions are incremental, so the evaluation methodology is the main scientific core and should be treated accordingly in review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent technical report on building a 32B reasoning model for semiconductor-display QA. The methods are standard—SFT, DPO, RAG with tuned embedding/reranker, LLM-as-judge—but the paper is honest about the pipeline, and the human evaluation protocol is decent. The claim that it beats DeepSeek-R1-671B is plausible, but the evidence is weaker than the abstract implies.\n\nCredit where it is earned: Table 9 shows monotonic gains from each RAG component, and Table 6 shows the automated evaluator correctly ranks 6 of 7 checkpoints against human scoring. The paper documents its data construction with dedup thresholds and difficulty filters. That is more than most industry reports do.\n\nSoft spots. First, the stress-test note is right: there is no decontamination analysis between the in-house test sets and the training corpora. The 100-question set is used for iterative model selection and then reported in Table 2; that number is not an independent estimate. The 400 and 800 sets are better, but still no overlap check. With acceptable-rate margins of 5–13 points, memorization could produce exactly these numbers. Second, the ground-truth references for the automated eval come from DeepSeek-R1, the same family as the teacher and the baseline. The paper mitigates this by validating references with human judges (2.78/3) and showing human/auto rank agreement, so it is not fully circular, but it is a soft spot. Third, there are no error bars or inter-annotator agreement on the 0–3 human scores; the score gaps of about 0.1 points could be noise.\n\nNone of this is disqualifying. The central argument—that a compact domain-tuned model can beat a huge generalist on a narrow vertical QA task—holds up as a plausible engineering result. The paper is a credible industrial technical report, not a broad scientific claim.\n\nWho is it for? Practitioners building vertical LLMs, especially in proprietary manufacturing settings. They will get useful pipeline details. Researchers will want more rigor: error bars, contamination checks, release of eval sets.\n\nRecommendation: send it to peer review with a major-revision expectation. The empirical claim is worth referee time, and the paper is honest enough to improve. Ask for contamination analysis, variance reporting, and clearer separation of the 100-question development set from final evaluation.","headline":"Plausible, honest industrial report on a 32B domain model beating R1-671B, but the evidence is thinner than the abstract and the missing contamination analysis is the most fixable gap.","tokens_in":13430,"tokens_out":2138,"would_cite":false,"duration_ms":24176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A domain-tuned 32-billion-parameter model beats a 671-billion-parameter generalist on expert-reviewed display reasoning benchmarks.","keywords":["semiconductor display","domain-specific reasoning LLM","domain-specific post-training","direct preference optimization","retrieval-augmented generation","automated evaluation","compact LLM"],"falsifier":"Have independent display-industry engineers at a different company write 400 fresh questions and their own reference answers, then score both models blind; if the 32-billion-parameter model does not match or beat the 671-billion-parameter model on acceptable rate or mean score, the central claim fails.","tokens_in":12229,"feed_emoji":"🖥️","tokens_out":6602,"duration_ms":535339,"temperature":0.7,"pith_summary":"X-Intelligence 3.0 is a reasoning language model built for the semiconductor display industry. The paper reports that after supervised fine-tuning, preference-based reinforcement learning, and a domain-adapted retrieval pipeline, a 32-billion-parameter model outperforms a 671-billion-parameter general-purpose reasoning model on the company's own expert-reviewed benchmarks. Scores rise by roughly 0.1 points on a 3-point scale and acceptable rates by about 5 to 13 percentage points. The paper also proposes an automated evaluator that ranks model checkpoints in close agreement with human expert ranking. A sympathetic reading is that compact domain-tuned models can beat much larger generalists on narrow industrial reasoning tasks.","feed_headline":"32B display model beats 671B generalist on domain tests","feed_subtitle":"Domain-tuned 32B model outscores the 671B flagship in expert-reviewed display reasoning tests.","key_machinery":"The argument is carried by three components. A two-stage post-training pipeline: stage one distills chain-of-thought reasoning into the model from a larger teacher, and stage two uses direct preference optimization on preference pairs built by sampling multiple responses and scoring them against reference answers. A domain-specific RAG stack: embedding and reranker models fine-tuned with hard negatives mined from the display literature, combined with iterative retrieval that generates supplementary queries until coverage is complete. An automated evaluation protocol: an LLM decomposes answers into statements and computes answer precision and recall against reference answers, combined as 0.3P plus 0.7R, validated against blind expert scoring. The evaluation protocol is what makes the performance comparison measurable at all, since the domain had no public benchmark.","core_discovery":"The central claim is that domain-specific post-training is enough to make a compact model the best available reasoner for semiconductor display problems. Starting from 32-billion-parameter base checkpoints, the authors add chain-of-thought supervised fine-tuning on curated domain questions, then direct preference optimization on automatically generated preference pairs, and finally a semiconductor-display retrieval-augmented generation module with fine-tuned embeddings and iterative query expansion. On proprietary test sets of 100, 400, and 800 questions, the resulting model scores 2.44, 2.40, and 2.43 respectively against 2.32, 2.26, and 2.37 for the 671-billion-parameter generalist, with acceptable rates of 82 to 88 percent versus 69 to 81 percent. The authors additionally claim that the automated evaluation framework reproduces human ranking of model checkpoints closely enough to replace expert review during iteration.","pith_inferences":["The evaluation design computes ground truth from the same generalist family that serves as baseline, so a head-to-head test with independently written reference answers and third-party domain experts would show whether the margin is a real capability gap or partly an artifact of reference-answer style.","Because acceptable rate jumps more than mean score, the main advantage may be fewer partially correct or unusable answers, not uniformly stronger reasoning; a per-question difficulty breakdown would reveal where the wins concentrate.","One testable extension: run the same training recipe on a second 32-billion-parameter base model from a different family; if gains persist, the pipeline, not the base checkpoint, is the cause.","An external benchmark built from recent display patents rather than textbooks and papers would stress generalization to unseen knowledge, since the paper's broad subdomain taxonomy still draws its evaluation from the same literature used to build training data."],"forward_implications":["If the reported margins hold, semiconductor display companies can get expert-level reasoning assistance from a model small enough to deploy internally without sharing proprietary data with a large hosted model.","The automated evaluation framework lets the developers screen many training checkpoints quickly, so data and training decisions can be made on objective scores rather than slow expert panels.","The RAG gains imply that linking a compact model to a curated domain knowledge base is a reliable route to fewer hallucinations in specialized technical Q&A.","The same SFT plus preference optimization plus domain RAG recipe should transfer to other vertical industries that have proprietary knowledge but no public reasoning benchmark.","The comparison suggests model size is not the dominant factor in narrow expert domains; data relevance and reasoning structure can outweigh raw parameter count."],"supporting_citations":[{"why":"DeepSeek-R1 is both the 671B baseline the paper must beat and the teacher used to distill chain-of-thought answers and generate evaluation reference answers.","marker":"Guo et al. [2025]"},{"why":"Qwen3-32B is one of the base checkpoints the model is fine-tuned from, supplying the starting weights for SFT and RL.","marker":"Yang et al. [2025]"},{"why":"Direct Preference Optimization is the training objective used in the reinforcement-learning stage on domain preference pairs.","marker":"Rafailov et al. [2023]"},{"why":"BGE-M3 computes semantic vectors for question deduplication, and the BGE embedding and rerank family is fine-tuned for domain retrieval.","marker":"Chen et al. [2024]"},{"why":"Ragas supplies the faithfulness-style statement-decomposition idea behind Answer Precision and Answer Recall in the automated evaluator.","marker":"Es et al. [2025]"},{"why":"Periodic hard-negative mining during embedding training is the mechanism that sharpens retrieval discrimination on ambiguous vertical-domain terms.","marker":"Li et al. [2024]"}],"fun_headline_variants":["32B domain model beats 671B generalist on display tests","Tuned 32B outranks 671B generalist in display reasoning","Compact 32B wins display reasoning vs 671B flagship","32B beats 671B: domain tuning wins on display tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the in-house test questions and the reference answers used for scoring are unbiased and representative of real semiconductor display reasoning work; if the tests overlap training data or the references favor one style of answer, the reported edge over the larger baseline would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["32B domain model beats 671B generalist on display tests","Tuned 32B outranks 671B generalist in display reasoning","Compact 32B wins display reasoning vs 671B flagship","32B beats 671B: domain tuning wins on display tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1481,"prompt_tokens":930,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":546,"tokens_out":551,"duration_ms":5280,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:56:19.772677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent display-industry engineers at a different company write 400 fresh questions and their own reference answers, then score both models blind; if the 32-billion-parameter model does not match or beat the 671-billion-parameter model on acceptable rate or mean score, the central claim fails.","supporting_citations":[],"review_version":1}