{"id":"1a0c4a09-e209-4a81-a64f-9085b2cf995f","arxiv_id":"2411.10172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-stage sequence tagging method adapted from prior work extracts causal relations from semiconductor FMEA documents at 93% F1 and from presentation slides at 73% F1, on a private dataset.","lead":"This paper tests two natural-language methods for pulling cause-and-effect statements out of semiconductor manufacturing documents, and reports 93% F1 on structured FMEA tables and 73% on slides. It matters because industrial causal knowledge is usually locked in human-written tables and slides, and automating extraction could feed risk analysis and root-cause tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 93% F1 is token-level, not relation-level, so the practical-suitability claim is not yet supported.","rationale":"The reader's weakest assumption about annotation reliability is valid but secondary. Even if the gold standard were perfect, the reported F1 would still not establish the central claim because it measures token classification, not causal relation extraction. The pipeline's value proposition is the structured output of relations; therefore the evaluation should be relation-centric. The paper is otherwise honest about limitations (OCR errors, explicit-trigger-only, intra-text scope, annotation dependence), which supports a conditional rather than unconditional verdict. Requiring relation-level evaluation and reporting split sizes would turn this from a promising case study into a defensible one. I therefore keep the reader's CONDITIONAL verdict unchanged but emphasize that the missing relation-level metric is a load-bearing gap that should be addressed before deployment claims are made.","tokens_in":14397,"tokens_out":5455,"duration_ms":53975,"concrete_test":"Evaluate the best MST model (e.g., MatBERT+PMI as reported) on the FMEA test set with relation-level scoring: a prediction is correct only if the predicted cause span, trigger span, and effect span exactly match a gold relation (or overlap by a specified criterion). Report precision, recall, and F1 for triples, together with the number of test instances. If relation-level F1 is substantially below the token-level macro F1 (e.g., more than 10 points lower), the practical-suitability claim in the abstract should be downgraded or reworded to refer to token tagging rather than 'extracting causal information.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states that the evaluation covers each component (trigger detection, trigger grouping, cause/effect detection), and Table 2 reports token-level F1 per label. The abstract and conclusion interpret the resulting macro-average F1 (93% on FMEA, 73% on slides) as evidence that the MST method is 'suitable for practical applications.' The missing check is end-to-end relation extraction quality: a causal relation is only useful if the cause, trigger, and effect spans are correctly paired. The MST pipeline is cascaded, so a token can be correctly tagged while the trigger-grouping or argument-assignment step still produces wrong triples. No relation-level precision/recall, and no exact-match or head-matching metric, is reported anywhere. The practical claim therefore rests on a metric that does not measure the actual output (cause-trigger-effect triples) used for downstream knowledge graphs and risk assessment. This is a correctness-risk concern independent of annotation quality; even with a perfect gold standard, the headline number can be inflated relative to the real extraction task. The absence of reported test-set size and training/validation split sizes further limits the interpretability of the 93±3 figure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a case study in which two sequence-tagging approaches — a single-stage sequence tagger (SST) and a multi-stage sequence tagger (MST) adapted from a co-author's master's thesis — are used to extract explicit causal relations (cause, trigger, effect triples) from industrial documents in the semiconductor manufacturing domain. The authors introduce annotation guidelines, use them to label 495 FMEA texts and 440 presentation-slide texts (plus 481 additional slide texts annotated by one expert), and report inter-annotator agreement, per-component token-level F1 scores, and results for different BERT-based language models with and without domain-specific fine-tuning using two masking strategies (UM and PMI). The headline result is a macro-averaged token-level F1 of 93% for MST on FMEA texts and 73% on slides, leading to the claim that the MST method is 'suitable for practical applications.'","tokens_in":14451,"tokens_out":4438,"duration_ms":43937,"significance":"If the reported performance is trustworthy, the paper contributes a useful industrial application report: it demonstrates that existing causal IE machinery can be transferred to a private industrial corpus with relatively high token-level F1, and it provides annotation guidelines plus an empirical study of domain-adaptive pre-training for this setting. The inter-annotator agreement analysis is a positive feature, as is the use of five-fold cross-validation with standard deviations. However, the central 'practical suitability' claim rests on token-level component F1 scores rather than on an evaluation of the final causal relation triples, and no external baseline is included; both aspects limit the significance of the findings as an advance in causal information extraction. The lack of public data is understandable but makes a stringent in-paper evaluation all the more important.","major_comments":[{"comment":"The headline '93% F1 score' is a token-level macro-averaged F1 over individual label classes (Cause, Effect, Trigger) and trigger-grouping, not an evaluation of the complete causal relation extraction output. The MST pipeline is cascaded: trigger detection, trigger grouping, and argument detection are measured separately, and no relation-level precision/recall, exact-match, or head-matching metric is reported. A token can be tagged correctly while the trigger-grouping or argument-assignment step yields an incorrect cause-trigger-effect triple. Thus the abstract's and Conclusion's claim that MST is 'suitable for practical applications' is not supported by the reported measurements. I request an end-to-end evaluation of the extracted triples, at minimum a relation-level F1 with exact argument span matching (or a well-defined partial matching scheme), together with the test-set size and the label distribution.","section":"Abstract, Section 3.4, Table 2"},{"comment":"The first research question, 'How effective are existing causal information extraction methods on different types of industrial documents?', is not answered because no existing causal IE method is implemented or evaluated as a baseline. The paper compares only the proposed SST and the MST method adapted from Gärber (2022), both of which are sequence-tagging variants in the same family. Adding an external baseline (for example, a published span-based or pattern-based system, or a standard generative extraction model) would contextualize the absolute F1 values and substantiate the claim that the authors are 'extending causal information extraction methods to industrial documents.'","section":"Section 1 (Research Questions) and Section 4"},{"comment":"The aggregation rule states that 'when there is a disagreement between the annotators, the annotations are aggregated,' meaning both conflicting annotations are kept as parallel gold labels. Since the two annotators are both NLP experts rather than domain experts, and since the guidelines are introduced in this paper, the resulting gold standard may contain substantial label noise. The paper does not report the proportion of disagreed instances in the test set, nor does it analyze performance on agreed versus disagreed items. This makes it difficult to interpret the reported F1 scores as estimates of extraction quality on a reliable gold standard. I ask the authors to report disagreement statistics and to discuss or quantify the impact of the aggregation rule on the reported numbers.","section":"Section 4 (Annotation Aggregation)"}],"minor_comments":[{"comment":"In the description of the MST argument-detection step, 'devices them as an input to a multi-class calcification' contains two typos: 'devices' should be 'uses' and 'calcification' should be 'classification'.","section":"Section 3.3"},{"comment":"The caption begins with 'Automated Causal Information Extraction from T ext' — 'T ext' should be 'Text'.","section":"Figure 2 caption"},{"comment":"Several formatting inconsistencies appear: 'T able' in Table 1's caption, inconsistent spacing in values such as '82 ± 4' versus '82±4', and incomplete alignment of the 'F 1%' header. These should be corrected for readability.","section":"Tables 1 and 2"},{"comment":"The text says 'we executed a performance evaluation of each component of the MST pipeline,' but the description of the evaluation does not specify the exact evaluation unit (e.g., is the token-level F1 computed with BIO tagging or a simpler token classification scheme?) or how the SST and MST outputs are converted to the labels in Table 2. A few sentences clarifying the evaluation protocol would strengthen reproducibility.","section":"Section 3.4"},{"comment":"The contribution mentions addressing 'data consistency issues commonly found in semi structured documents, like the merged cells in FMEA,' but the evaluation does not explicitly measure how well the proposed pipeline handles merged-cell texts. The paper should either include a specific analysis of such texts or soften this contribution claim.","section":"Introduction and Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable industrial case study, but the gap between the token-level evaluation and the claimed practical suitability is substantial. I would not recommend rejection, since the missing relation-level evaluation and baselines are addressable in revision. The method's origin in a co-author's thesis (Gärber 2022) should be made more explicit in the contribution statement to avoid any appearance of dual publication; currently the abstract credits 'proposed MST methods' to this paper, which is misleading. The private data availability is a known constraint for industry studies, but it further raises the burden on the internal evaluation to be complete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I think this is a solid, useful case study, but the main claim is overstated. The 93% F1 on FMEA texts is a token-level macro average across trigger, cause, and effect labels, not a metric for the actual output of the pipeline—cause-trigger-effect triples. The MST method cascades trigger detection, trigger grouping, and argument detection, and the paper reports each component's F1 separately, but never reports end-to-end relation-level precision/recall or exact-match accuracy. So the headline number can be high even when the model mispairs causes and effects. The stress-test note is right: this is a correctness risk independent of annotation quality.\n\nWhat's genuinely good: the annotation guidelines are detailed and clearly tailored to industrial text—merged FMEA cells, enchained relations, disrupted entities. The inter-annotator agreement analysis is informative: 85% Cohen's kappa overall, with triggers easiest and effects hardest. The comparison of SST vs. MST across two document types and the study of in-domain fine-tuning with two masking strategies are reasonable, and the tables are internally consistent. The limitations section is candid about OCR errors and the loss of visual information from slides.\n\nThe soft spots are the missing pieces. First, no external baseline: RQ1 asks about existing methods, but only SST and MST are compared. Second, the test set size is not reported, so the ±3 confidence intervals are hard to interpret. Third, the aggregation rule—keeping both annotators' labels on disagreement—is unusual and could inflate scores, since the model only needs to match one of two gold labels. Each of these is addressable, and I would not call any of them fatal, but together they make the practical-suitability claim premature.\n\nWho is this for? Practitioners working on causal extraction in industrial or semi-structured document settings. They'll get value from the annotation guidelines, the model comparison, and the honest account of what worked and what didn't. It is not a methodological breakthrough, but it is a legitimate applied contribution.\n\nMy recommendation: send it to peer review, not desk-reject. The application is real and the guidelines are useful. The authors should add relation-level evaluation metrics, run at least one existing causal extraction system as a baseline, report split sizes, and revise the deployment claim to match what the evidence supports. With those changes, this would be a solid case study.","headline":"A competent, honest applied case study whose 'practical suitability' claim outruns the evidence because the metrics are token-level, not relation-level.","tokens_in":15109,"tokens_out":2994,"would_cite":false,"duration_ms":29356,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-stage sequence tagging pipeline extracts causal relations from semiconductor FMEA documents at 93% F1, making industrial causal knowledge machine-readable.","keywords":["causal information extraction","sequence tagging","FMEA","semiconductor manufacturing","presentation slides","domain-specific language models","inter-annotator agreement","industrial NLP"],"falsifier":"Take a random sample of the 495 FMEA texts, have a third expert annotate them with the same guidelines, adjudicate every disagreement with the original annotators, and recompute macro F1 for the best MST configuration; if the adjudicated F1 falls well below 93%, the practical-suitability claim fails.","tokens_in":14065,"feed_emoji":"⚙️","tokens_out":5773,"duration_ms":50456,"temperature":0.7,"pith_summary":"The paper asks whether automated causal information extraction can make the causal knowledge locked in industrial documents machine-readable, and tests it on real semiconductor manufacturing records. It proposes two sequence-tagging approaches: a single-stage multi-label tagger and a multi-stage pipeline that finds causal trigger words first and then identifies the causes and effects attached to each trigger. On FMEA table cells the multi-stage method reaches a mean macro F1 of 93%, which the authors argue is suitable for practical use; on presentation slides it reaches 73%. The authors also show that starting from a materials-aware language model and fine-tuning it on in-domain text, especially with pointwise mutual information masking, tends to improve extraction. A sympathetic reader would take the central claim to be that a previously lab-proposed pipeline transfers to real industrial documents, with the caveat that full relations are only extracted when an explicit causal trigger is present.","feed_headline":"Causal text mining hits 93% F1 on semiconductor failure-mode tables","feed_subtitle":"A multi-stage sequence tagger turns cell-level failure-mode text into cause-effect pairs, aided by a materials-aware language model.","key_machinery":"The load-bearing mechanism is the MST cascade: a binary token classifier detects causal triggers (words like \"due to\" or \"leads to\"), a grouping classifier merges tokens belonging to the same interrupted trigger, an attention network aggregates the trigger's token embeddings, and a final multi-label classifier assigns each token as cause or effect relative to that trigger. Because cause/effect detection is conditioned per trigger, the same token can be cause for one relation and effect for another, which is how enchained relations are captured. The pipeline runs on BERT-style embeddings, and the paper compares BERT with MatBERT, with and without in-domain masked-language-model fine-tuning under uniform and PMI masking. Annotation follows the paper's own guidelines defining cause, effect, and trigger, with explicit-trigger-only, nested/chained/disrupted entity rules.","core_discovery":"The central discovery, stated on the paper's own terms, is that the multi-stage sequence tagging (MST) method — detect trigger tokens, group tokens into trigger entities, aggregate trigger embeddings, then label cause and effect tokens for each trigger — extracts causal relations from FMEA documents at a mean macro F1 of 93% (MatBERT with PMI masking) and from presentation slides at 73%. This makes MST, originally developed for historical texts, the stronger of the two evaluated methods for industrial use: it handles enchained relations, nested relations, and disrupted entities, which plain single-stage tagging cannot. The same experiments show that swapping the general BERT base model for MatBERT, a model pretrained on materials-science text, and then fine-tuning in-domain, improves results in most configurations. The paper presents these results as a case study demonstrating practical applicability for semi-structured FMEA documents in particular.","pith_inferences":["If the 93% figure holds under adjudicated gold labels, the same MST recipe should transfer to adjacent semi-structured documents such as 8D reports, CAPA records, or equipment maintenance logs, where cause–effect entries are similarly cell-structured.","The gap between trigger agreement and effect agreement suggests that part of the model's remaining error on effect boundaries may be inherited from annotation subjectivity; a reconciled gold standard would separate label noise from model error.","The explicit-trigger constraint means the method is best framed as a recall-limited structure extractor rather than a full causal reader, since implicit causal language is invisible to it.","A natural extension is to use the annotation guidelines as prompts for instruction-tuned language models, which the paper mentions as future potential; if those models can also resolve coreference across cells, inter-cell relations would become accessible."],"forward_implications":["FMEA cells containing several merged causal relations can be automatically split into separate cause–effect pairs, feeding a more connected knowledge graph.","Practitioners with similar semi-structured quality documents have a concrete recipe: a domain-aligned pretrained model, in-domain fine-tuning, and the MST tagging cascade.","Presentation slides remain harder: extraction at 73% F1 is useful for triage but not full automation, and OCR plus explicit-trigger constraints lose visual and implicit causal information.","The annotation guidelines provide a starting template for other industrial domains, with triggers proving easier to annotate consistently than causes and effects."],"supporting_citations":[{"why":"Supplies the original MST cascade that the paper adapts, including trigger grouping and attention aggregation.","marker":"Gärber, 2022"},{"why":"Represents the span-based and sequence tagging causal extraction approach whose limitations motivate the MST design.","marker":"Saha et al., 2022"},{"why":"Provides a related deep-learning causal extraction framework from business text that struggles with detailed causal relations.","marker":"Gopalakrishnan et al., 2023"},{"why":"Establishes the FMEA cell extraction approach and the merged-cell problem that the paper builds on.","marker":"Razouk and Kern, 2022"},{"why":"Shows how causal relations in FMEA text are structured into a knowledge graph, the downstream target of this pipeline.","marker":"Razouk et al., 2023"},{"why":"Provides MatBERT, the domain-aligned language model whose gains are a main result.","marker":"Walker et al., 2021"},{"why":"Supplies the semiconductor in-domain fine-tuning data set and pipeline used for the UM and PMI experiments.","marker":"Tosone, 2022"},{"why":"Defines PMI masking, one of the two fine-tuning objectives compared in the study.","marker":"Levine et al., 2020"},{"why":"Sets the explicit-trigger and lexical-causative exclusion rules that shape the annotation guidelines.","marker":"Dunietz et al., 2017"}],"fun_headline_variants":["MST tagger yields 93% F1 on causal extraction from FMEAs","Causal IE from FMEAs: multi-stage tagging scores 93% F1","Domain-tuned MatBERT lifts causal extraction to 93% F1 on FMEAs","93% F1 on FMEAs: MST captures cause-effect from failure text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the two annotators' aggregated labels as the correct gold standard, even though the guidelines are new and disagreements are kept as parallel annotations rather than resolved, so if those labels are biased the F1 scores overstate how well the method extracts true causal relations.","fun_headline_variants_meta":{"raw":{"variants":["MST tagger yields 93% F1 on causal extraction from FMEAs","Causal IE from FMEAs: multi-stage tagging scores 93% F1","Domain-tuned MatBERT lifts causal extraction to 93% F1 on FMEAs","93% F1 on FMEAs: MST captures cause-effect from failure text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001026,"raw_usage":{"total_tokens":4316,"prompt_tokens":924,"completion_tokens":3392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":3301}},"tokens_in":540,"tokens_out":3392,"duration_ms":26020,"temperature":1.0,"reasoning_tokens":3301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:52:43.605185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 495 FMEA texts, have a third expert annotate them with the same guidelines, adjudicate every disagreement with the original annotators, and recompute macro F1 for the best MST configuration; if the adjudicated F1 falls well below 93%, the practical-suitability claim fails.","supporting_citations":[{"cited_title":", Chen , V.Z","cited_arxiv_id":null,"evidence_quote":"Provides a related deep-learning causal extraction framework from business text that struggles with detailed causal relations."},{"cited_title":", Kern , R","cited_arxiv_id":null,"evidence_quote":"Establishes the FMEA cell extraction approach and the merged-cell problem that the paper builds on."},{"cited_title":", Trewartha , A","cited_arxiv_id":null,"evidence_quote":"Provides MatBERT, the domain-aligned language model whose gains are a main result."},{"cited_title":": Improving fmea consistency in semiconductor manufacturing through text classification","cited_arxiv_id":null,"evidence_quote":"Supplies the semiconductor in-domain fine-tuning data set and pipeline used for the UM and PMI experiments."},{"cited_title":", Levin , L","cited_arxiv_id":null,"evidence_quote":"Sets the explicit-trigger and lexical-causative exclusion rules that shape the annotation guidelines."}],"review_version":1}