{"id":"29c6e8ba-d4b5-4493-b9c0-d9ab6353219f","arxiv_id":"1908.06606","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A question answering formulation with BERT plus clinical named entity features improves extraction of tumor size and resection margins from Chinese pathology reports.","lead":"The paper recasts clinical text structuring as a question answering task and trains a BERT model with clinical named entity features to answer questions about Chinese pathology reports. On a private dataset from one hospital, the model reports higher span extraction scores than BERT-Base and QANet, though without error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's clinical structuring claims rest on span-extraction metrics, but Section III itself notes final answers require post-processing; end-to-end accuracy is never measured.","rationale":"The reader's weakest assumption identifies exactly the load-bearing weakness: evaluation measures span extraction, while the stated task is clinical text structuring, whose final output may require conversion and negation handling. The paper itself acknowledges this gap in Section III, so the concern is internal to the paper's framing rather than an external disagreement. This is more consequential than the absence of error bars or code release, because even perfect span extraction may not yield correct structured values. The data-integration analysis also shows a non-uniform benefit, but that is a secondary qualification. I recommend keeping the reader's CONDITIONAL verdict: the current evidence supports the QA-CTS span-extraction formulation and model performance on this dataset, but acceptance of the clinical-structuring claim should require end-to-end final-answer evaluation or a released post-processing module that can be independently checked.","tokens_in":7384,"tokens_out":2213,"duration_ms":25832,"concrete_test":"Take the 815 test instances and define the ground-truth final structured field for each query type (e.g., margin distance in cm plus a negation flag, tumor size value). Apply the post-processing steps described in Section III (entity-name conversion and negative-word recognition) to the predicted spans from QANet, BERT-Base, and the proposed model, and compute final-field exact-match and macro-F1. If the proposed model's advantage over the baselines shrinks or reverses at this end-to-end level, the current span-level evaluation does not support the central CTS claim; if its advantage persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that QA-CTS improves clinical text structuring, yet the only evaluated outputs are answer-related substrings, scored with EM and F1. This is a proxy, not the clinical structuring result. Section III explicitly states that for many cases, 'several steps such as entity names conversion and negative words recognition' are required to obtain the final answer. Since the paper reports no end-to-end evaluation of the final structured fields, the headline 91.84% EM / 93.75% F1 does not establish that the proposed model improves actual CTS. In particular, margin reports require negation recognition (e.g., 'no tumor cells at margin') and unit/value conversion; errors in these post-processing steps could dominate the 5.64-point EM gap over BERT-Base and the 0.13-point F1 gap over QANet. Without quantifying post-processing errors, the reported gains may not translate to real structuring accuracy. Additionally, mixed-data training does not uniformly help: Tables II/III show tumor-size EM falls from 96.27% (pure) to 95.10% (mixed), so the claim that shared data improves task-specific performance needs qualification. The load-bearing issue, however, is the span-vs-final-answer gap: the paper's evaluation stops before the stated goal of clinical text structuring is achieved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QA-CTS, a reading-comprehension-style formulation of clinical text structuring in which a model extracts an answer-related substring from a clinical paragraph given a query. The proposed model augments BERT-Base with one-hot clinical named-entity tags (from an external CNER system) and uses a feed-forward layer to predict start and end positions, trained with a two-stage fine-tuning procedure. On a Chinese pathology-report dataset from Ruijin Hospital with three question types (tumor size, proximal resection margin, distal resection margin), the model is reported to achieve EM 91.84 and F1 93.75, outperforming BERT-Base and QANet on EM while being comparable to QANet on F1. Data-integration experiments show that training on mixed query types often helps, with some exceptions. The paper concludes that the shared QA-CTS task improves individual CTS tasks.","tokens_in":7651,"tokens_out":5714,"duration_ms":55036,"significance":"The shared QA formulation is a reasonable and potentially practical way to pool annotations across heterogeneous CTS tasks, and the dataset of 2,714 clinician-annotated QA pairs is a useful resource. The architecture is clearly specified, and the comparison with strong QA baselines is appropriate. However, the reported results only measure span extraction, not the final structured fields, and no significance testing or ablations are provided; therefore the central performance claims are not fully established. If the additional experiments requested below confirm the results, the paper would make a modest but useful contribution to clinical NLP.","major_comments":[{"comment":"The evaluation is on the answer-related substring, not on the final structured value. Section III states that 'several steps such as entity names conversion and negative words recognition' are needed to obtain the final answer, so the reported EM/F1 scores do not measure clinical text structuring as claimed in the abstract. Please either evaluate the full post-processing chain end-to-end or, at minimum, quantify how often post-processing is required and measure its error contribution; otherwise the headline 'clinical text structuring' claim overstates the evidence.","section":"Section III; Tables II-IV"},{"comment":"There are no multiple runs, standard deviations, or significance tests reported. The F1 advantage over QANet is only 0.13%, which is well within run-to-run variation for neural reading-comprehension models, so the statement that the proposed model 'significantly outperformed' QANet (Section V-C) is unsupported. Report repeated-run statistics with confidence intervals or significance tests for all main comparisons.","section":"Section V-C, Table IV"},{"comment":"The mixed-data conclusion is not uniformly supported by the tables. For Tumor Size, the pure model achieves EM 96.27/F1 96.08 while the Mixed Data model achieves EM 95.10/F1 94.51, so shared training actually hurts this task. The recommendation that 'pre-training the model in multiple datasets and then fine tuning the model on the specific dataset is the best way' needs to be qualified with a discussion of when negative transfer occurs.","section":"Section V-D, Tables II and III"},{"comment":"The central novelty is the integration of clinical named-entity information into a pre-trained language model, yet no ablation isolates this component. The reported comparison with BERT-Base lacks a BERT+two-stage baseline without CNER features, and a CNER-features baseline without two-stage training. Add such ablations to substantiate the claim that the CNER feature integration is responsible for the improvement.","section":"Section IV and V-C"}],"minor_comments":[{"comment":"'Ruijing Hospital' should be 'Ruijin Hospital' to match the affiliation.","section":"Abstract"},{"comment":"The notation for the CNER tag sequences (Inq, Int, In) is undefined and visually confusing; define them in equations or a table.","section":"Section IV"},{"comment":"The text says 'the best results in tables are in bold,' but no numbers are bolded in Tables I-IV.","section":"Section V"},{"comment":"The F1 measure is described as macro-averaged in Section V-A, but the table headers just say 'F1-score'; state the averaging explicitly in the captions.","section":"Tables II and III"},{"comment":"The two-stage training description is vague about which layers are frozen during the first stage; specify the freezing schedule.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"The paper would be better positioned as a clinical NLP application paper with the required statistical rigor. The lack of significance testing and the span-vs-final-answer gap are the main concerns; if the authors can add post-processing evaluation and ablations, the contribution is defensible. I also note that the dataset is not released, which limits reproducibility of the claimed data-integration benefits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper's central trick — recasting clinical text structuring as a question-answering span-extraction task so data can be shared across tasks — is a real idea, and the mixed-data transfer results roughly support it. But the evaluation is done one step short of the actual clinical claim. The authors measure EM/F1 on the extracted substring, while Section III states plainly that final answers often need extra steps like entity name conversion and negative word recognition. So the 91.84 EM and 93.75 F1 are about span finding, not about whether 'proximal margin: no tumor cells' is correctly structured. That gap isn't a side issue; it's the difference between the title and the experiment.\n\nWhat's actually new: framing CTS as QA with shareable data, and concatenating clinical NER label embeddings into BERT. Both are modest but useful. The two-stage training trick is borrowed from fine-grained visual recognition and seems to help. The paper is honest about its scope — three question types, one hospital — and it doesn't oversell the model architecture.\n\nThe soft spots, in order of weight. First, the span-vs-final-answer problem is load-bearing. The reported gains might vanish once you add the post-processing the authors themselves say is required. At minimum, they should quantify errors introduced by those steps. Second, the 0.13% F1 edge over QANet is noise without error bars or significance tests, and the word 'significant' appears in the abstract without support. Third, mixed-data training doesn't uniformly help: tumor-size EM drops from 96.27 to 95.10 in Table II/III. That doesn't kill the transfer claim — most tasks do benefit — but the abstract's 'very effective' is too strong. Fourth, no code or data, single institution, so external validation is impossible from the paper alone.\n\nThe citation pattern is fine; self-citations to the CNER work are appropriate, and they cite standard baselines. No circularity — the test set is held out.\n\nWho this is for: anyone working in clinical NLP or QA for EHRs will get value from the task formulation, even if the empirical case is incomplete. I'd bring it to a reading group to discuss the proxy-evaluation issue. I would not cite the numbers, but I might cite the QA-CTS formulation if I worked in that area.\n\nRecommendation: this deserves a serious referee, not a desk reject. A competent reviewer could push the authors to add an end-to-end evaluation (or at least a post-processing error analysis), error bars, and a more cautious abstract. As it stands, it's a conditional accept at best.","headline":"A useful task formulation that deserves review, but the headline numbers are about span extraction, not the clinical structuring the title promises.","tokens_in":8149,"tokens_out":2636,"would_cite":true,"duration_ms":26214,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By recasting clinical text structuring as question answering over the original report text, this paper shows that a pre-trained language model augmented with clinical named-entity tags reaches an exact-match score of 91.84 percent on…","keywords":["clinical text structuring","question answering","pre-trained language model","clinical named entity recognition","electronic health records","Chinese pathology reports","span extraction","transfer learning"],"falsifier":"Compute end-to-end accuracy on the final structured values (after entity-name conversion and negative-word recognition) for both the proposed model and the plain pre-trained language model; if the plain model's final-answer accuracy is close to or better than the proposed model's despite its lower span-extraction EM, then the claimed advantage of the QA-CTS model does not translate to real structuring accuracy.","tokens_in":7219,"feed_emoji":"🩺","tokens_out":6827,"duration_ms":62531,"temperature":0.7,"pith_summary":"Clinical text structuring is usually tackled task by task, each with its own output format, its own model, and its own labeled data. This paper argues that most structuring problems share a common core: given a query such as \"proximal resection margin,\" find the answer-related span inside the report text. Recasting structuring as question answering over the original text lets different tasks reuse one training set, and the paper shows this shared-data setup improves performance on task-specific test sets. The paper's model, which feeds clinical named-entity tag sequences into a pre-trained language model and then classifies start and end positions, reaches an exact-match score of 91.84 percent and an F1 score of 93.75 percent on Chinese pathology reports. That beats the plain pre-trained language model (EM 86.20, F1 90.06) and a strong reading-comprehension baseline (EM 85.45, F1 93.62).","feed_headline":"Clinical text structuring as QA lifts exact match to 91.8%","feed_subtitle":"A question-answering model with clinical entity tags beats two strong baselines on Chinese pathology reports.","key_machinery":"The load-bearing object is the QA-CTS task itself: given a report paragraph $X$ and a query $Q$, predict the start and end index of the answer-related substring $X_s$. The model couples two input streams: a pre-trained language model encodes $[CLS] Q [SEP] X [SEP]$ into a contextualized vector, while a clinical named-entity recognizer labels every character with BIEOS tags (Begin, Inside, End, Outside, Single) for 44 entity types; the two representations are concatenated and passed through a feed-forward network that scores each position as the start or end of the answer. The paper also uses a two-stage training schedule, first fine-tuning the language model with a prediction layer and then adding the entity-information layers and retraining, which it credits with improving performance.","core_discovery":"The central claim is that unifying clinical text structuring under a question-answering format makes the underlying task learnable from shared data and improves accuracy on individual structuring tasks. Rather than outputting a final structured value directly, the model is asked to locate the substring of the paragraph that answers a clinical query; later steps such as entity-name conversion and negative-word recognition are handled separately. On three query types from gastric pathology reports (tumor size, proximal resection margin, and distal resection margin), the authors find that a model trained on all three types performs better on each than a model trained on that type alone, and that mixed-data pretrained parameters lift accuracy on query types not seen in final training from near zero to roughly 60-70 percent. Adding clinical named-entity information yields a further gain, giving the best exact-match and F1 scores.","pith_inferences":["The span-extraction framing should transfer most directly to electronic-health-record fields whose values appear verbatim in text, such as lab results, measurements, and dates; testing it on fields requiring negation handling or inference would show how far the unification extends.","Because the model relies on a clinical named-entity tagger trained on Chinese EHRs, a natural extension is to replace that tagger with a multilingual or cross-institution tagger and measure whether the transfer gains persist across hospitals or languages.","The near-zero to 60-70 percent jump on unseen query types suggests the QA-CTS task could serve as a pretraining objective for clinical NLP, analogous to general-domain reading-comprehension pretraining, though the paper does not test this directly."],"forward_implications":["Training data from different clinical structuring tasks can be pooled, so a query type with few examples can borrow signal from more abundant query types.","On the three tested query types, the shared-task model reaches EM 91.84 and F1 93.75, surpassing both the plain pre-trained language model and the stronger baseline on exact match.","Mixed-data pretraining converts near-zero transfer performance on untrained query types into 60-70 percent accuracy, evidence of commonality across tasks.","The two-stage training mechanism is reported as a practical way to adapt large pre-trained models when computational resources are limited."],"supporting_citations":[{"why":"It supplies the pre-trained language model whose contextualized representations carry the query-and-paragraph encoding.","marker":"[12]"},{"why":"It supplies the clinical named-entity recognition model that produces the BIEOS tag sequences used as domain features.","marker":"[16]"},{"why":"It is the strong reading-comprehension baseline whose exact-match and F1 scores the proposed model is compared against.","marker":"[26]"},{"why":"It defines the exact-match score used to evaluate the model's output.","marker":"[21]"}],"fun_headline_variants":["Clinical text structuring as QA outperforms baselines","QA-based clinical structuring gains from entity tags","Turning clinical structuring into QA improves accuracy","Unified QA task boosts clinical text structuring accuracy","Entity-aware QA model sharpens clinical text extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that extracting the answer-related substring is a faithful proxy for clinical text structuring, since the paper measures only span extraction and states that final answers still need steps such as entity-name conversion and negative-word recognition.","fun_headline_variants_meta":{"raw":{"variants":["Clinical text structuring as QA outperforms baselines","QA-based clinical structuring gains from entity tags","Turning clinical structuring into QA improves accuracy","Unified QA task boosts clinical text structuring accuracy","Entity-aware QA model sharpens clinical text extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1457,"prompt_tokens":825,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":441,"tokens_out":632,"duration_ms":6516,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:38:55.319906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute end-to-end accuracy on the final structured values (after entity-name conversion and negative-word recognition) for both the proposed model and the plain pre-trained language model; if the plain model's final-answer accuracy is close to or better than the proposed model's despite its lower span-extraction EM, then the claimed advantage of the QA-CTS model does not translate to real structuring accuracy.","supporting_citations":[{"cited_title":"Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field,","cited_arxiv_id":null,"evidence_quote":"It supplies the clinical named-entity recognition model that produces the BIEOS tag sequences used as domain features."},{"cited_title":"SQuAD: 100,000+ questions for machine comprehension of text,","cited_arxiv_id":null,"evidence_quote":"It defines the exact-match score used to evaluate the model's output."}],"review_version":1}