REVIEW 4 major objections 5 minor 26 references
Question Answering based Clinical Text Structuring Using Pre-trained Language Model
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read By recasting clinical text structuring as question answering over the original report text, this paper shows that a pre-trained language model augmented with clinical named-entity tags reaches an exact-match score of 91.84 percent on…
desk verdict A useful task formulation that deserves review, but the headline numbers are about span extraction, not the clinical structuring the title promises. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the QA-CTS task itself: given a report paragraph $X$ and a query $Q$, predict the start and end index of the answer-related substring $X_s$. The model couples two input streams: a pre-trained language model encodes $[CLS] Q [SEP] X [SEP]$ into a contextualized vector, while a clinical named-entity recognizer labels every character with BIEOS tags (Begin, Inside, End, Outside, Single) for 44 entity types; the two representations are concatenated and passed through a feed-forward network that scores each position as the start or end of the answer. The paper also uses a two-stage training schedule, first fine-tuning the language model with a prediction layer and then adding the entity-information layers and retraining, which it credits with improving performance.
What would settle it
Compute end-to-end accuracy on the final structured values (after entity-name conversion and negative-word recognition) for both the proposed model and the plain pre-trained language model; if the plain model's final-answer accuracy is close to or better than the proposed model's despite its lower span-extraction EM, then the claimed advantage of the QA-CTS model does not translate to real structuring accuracy.
Extended reading notes
Core claim
The central claim is that unifying clinical text structuring under a question-answering format makes the underlying task learnable from shared data and improves accuracy on individual structuring tasks. Rather than outputting a final structured value directly, the model is asked to locate the substring of the paragraph that answers a clinical query; later steps such as entity-name conversion and negative-word recognition are handled separately. On three query types from gastric pathology reports (tumor size, proximal resection margin, and distal resection margin), the authors find that a model trained on all three types performs better on each than a model trained on that type alone, and that mixed-data pretrained parameters lift accuracy on query types not seen in final training from near zero to roughly 60-70 percent. Adding clinical named-entity information yields a further gain, giving the best exact-match and F1 scores.
Load-bearing premise
The evaluation assumes that extracting the answer-related substring is a faithful proxy for clinical text structuring, since the paper measures only span extraction and states that final answers still need steps such as entity-name conversion and negative-word recognition.
Editorial extensions
If this is right
- Training data from different clinical structuring tasks can be pooled, so a query type with few examples can borrow signal from more abundant query types.
- On the three tested query types, the shared-task model reaches EM 91.84 and F1 93.75, surpassing both the plain pre-trained language model and the stronger baseline on exact match.
- Mixed-data pretraining converts near-zero transfer performance on untrained query types into 60-70 percent accuracy, evidence of commonality across tasks.
- The two-stage training mechanism is reported as a practical way to adapt large pre-trained models when computational resources are limited.
Reading between the lines
- The span-extraction framing should transfer most directly to electronic-health-record fields whose values appear verbatim in text, such as lab results, measurements, and dates; testing it on fields requiring negation handling or inference would show how far the unification extends.
- Because the model relies on a clinical named-entity tagger trained on Chinese EHRs, a natural extension is to replace that tagger with a multilingual or cross-institution tagger and measure whether the transfer gains persist across hospitals or languages.
- The near-zero to 60-70 percent jump on unseen query types suggests the QA-CTS task could serve as a pretraining objective for clinical NLP, analogous to general-domain reading-comprehension pretraining, though the paper does not test this directly.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces QA-CTS, a reading-comprehension-style formulation of clinical text structuring in which a model extracts an answer-related substring from a clinical paragraph given a query. The proposed model augments BERT-Base with one-hot clinical named-entity tags (from an external CNER system) and uses a feed-forward layer to predict start and end positions, trained with a two-stage fine-tuning procedure. On a Chinese pathology-report dataset from Ruijin Hospital with three question types (tumor size, proximal resection margin, distal resection margin), the model is reported to achieve EM 91.84 and F1 93.75, outperforming BERT-Base and QANet on EM while being comparable to QANet on F1. Data-integration experiments show that training on mixed query types often helps, with some exceptions. The paper concludes that the shared QA-CTS task improves individual CTS tasks.
Significance. The shared QA formulation is a reasonable and potentially practical way to pool annotations across heterogeneous CTS tasks, and the dataset of 2,714 clinician-annotated QA pairs is a useful resource. The architecture is clearly specified, and the comparison with strong QA baselines is appropriate. However, the reported results only measure span extraction, not the final structured fields, and no significance testing or ablations are provided; therefore the central performance claims are not fully established. If the additional experiments requested below confirm the results, the paper would make a modest but useful contribution to clinical NLP.
major comments (4)
- [Section III; Tables II-IV] The evaluation is on the answer-related substring, not on the final structured value. Section III states that 'several steps such as entity names conversion and negative words recognition' are needed to obtain the final answer, so the reported EM/F1 scores do not measure clinical text structuring as claimed in the abstract. Please either evaluate the full post-processing chain end-to-end or, at minimum, quantify how often post-processing is required and measure its error contribution; otherwise the headline 'clinical text structuring' claim overstates the evidence.
- [Section V-C, Table IV] There are no multiple runs, standard deviations, or significance tests reported. The F1 advantage over QANet is only 0.13%, which is well within run-to-run variation for neural reading-comprehension models, so the statement that the proposed model 'significantly outperformed' QANet (Section V-C) is unsupported. Report repeated-run statistics with confidence intervals or significance tests for all main comparisons.
- [Section V-D, Tables II and III] The mixed-data conclusion is not uniformly supported by the tables. For Tumor Size, the pure model achieves EM 96.27/F1 96.08 while the Mixed Data model achieves EM 95.10/F1 94.51, so shared training actually hurts this task. The recommendation that 'pre-training the model in multiple datasets and then fine tuning the model on the specific dataset is the best way' needs to be qualified with a discussion of when negative transfer occurs.
- [Section IV and V-C] The central novelty is the integration of clinical named-entity information into a pre-trained language model, yet no ablation isolates this component. The reported comparison with BERT-Base lacks a BERT+two-stage baseline without CNER features, and a CNER-features baseline without two-stage training. Add such ablations to substantiate the claim that the CNER feature integration is responsible for the improvement.
minor comments (5)
- [Abstract] 'Ruijing Hospital' should be 'Ruijin Hospital' to match the affiliation.
- [Section IV] The notation for the CNER tag sequences (Inq, Int, In) is undefined and visually confusing; define them in equations or a table.
- [Section V] The text says 'the best results in tables are in bold,' but no numbers are bolded in Tables I-IV.
- [Tables II and III] The F1 measure is described as macro-averaged in Section V-A, but the table headers just say 'F1-score'; state the averaging explicitly in the captions.
- [Section IV-D] The two-stage training description is vague about which layers are frozen during the first stage; specify the freezing schedule.
Circularity Check
No circularity: the QA-CTS model is trained on a held-out split with externally pretrained BERT and a task-independent CNER feature extractor; the self-citation is not load-bearing.
full rationale
The paper's central empirical claim is that the proposed BERT-plus-CNER model reaches EM 91.84% and F1 93.75% on held-out QA-CTS test instances, exceeding BERT-Base and QANet baselines. This result is not constrained by construction: BERT weights are externally pretrained on a general Chinese corpus, and the CNER tagger from the authors' prior work [16] is trained on a separate corpus with 44 entity types and is not fine-tuned on the QA-CTS training split. The mixed-data benefit (Tables II and III) is measured by training on pure versus shared question-type datasets and testing on the same held-out split, so the comparison is empirical rather than definitional. The only notable weakness is a scope limitation stated by the authors in Section III: for many instances, the answer-related substring is not the final structured value, and further steps such as entity-name conversion and negative-word recognition are required. Because the paper evaluates only span extraction with EM and F1, the advertised clinical-text-structuring improvement may be overstated, and tumor-size EM slightly drops under mixed training (96.27% to 95.10%). These are validity concerns, not circular reductions: no fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and the one self-citation is used as a feature extractor rather than as evidence for the target result. Accordingly, no specific circular step can be exhibited, and the circularity score is low.
Assumptions & free parameters
free parameters (3)
- BERT fine-tuning learning rate =
5e-5
- Batch size =
3 or 4
- CNER entity type set =
44 types
assumptions (4)
- domain assumption BERT pre-trained on general Chinese corpus transfers to clinical Chinese text.
- domain assumption The CNER model of the authors' previous work produces useful features on pathology reports.
- domain assumption The answer-related substring is sufficient for the final structured answer.
- domain assumption The three question types share enough common structure for mixed-data training to transfer.
Cite this review
Pith. "Pith review of Question Answering based Clinical Text Structuring Using Pre-trained Language Model." pith.science (2026). https://pith.science/paper/PCFY3MBQ
@misc{pith2026190806606,
author = {Pith},
title = {Pith review of: Question Answering based Clinical Text Structuring Using Pre-trained Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/PCFY3MBQ}},
note = {Machine review of arXiv:1908.06606}
}
read the original abstract
Clinical text structuring is a critical and fundamental task for clinical research. Traditional methods such as taskspecific end-to-end models and pipeline models usually suffer from the lack of dataset and error propagation. In this paper, we present a question answering based clinical text structuring (QA-CTS) task to unify different specific tasks and make dataset shareable. A novel model that aims to introduce domain-specific features (e.g., clinical named entity information) into pre-trained language model is also proposed for QA-CTS task. Experimental results on Chinese pathology reports collected from Ruijing Hospital demonstrate our presented QA-CTS task is very effective to improve the performance on specific tasks. Our proposed model also competes favorably with strong baseline models in specific tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Toward infor- mation extraction: identifying protein names from biological papers,
K.-i. Fukuda, T. Tsunoda, A. Tamura, T. Takagi et al. , “Toward infor- mation extraction: identifying protein names from biological papers,” in Pacific Symposium on Biocomputing , vol. 707, no. 18, 1998, pp. 707– 718
work page 1998
-
[2]
Linguistic mapping of terminologies to SNOMED CT,
Y . Wang, J. Patrick, G. Miller, and J. O’Halloran, “Linguistic mapping of terminologies to SNOMED CT,” in Semantic Mining Conference on SNOMED CT. Citeseer, 2006
work page 2006
-
[3]
Developing a hybrid dictionary- based bio-entity recognition technique,
M. Song, H. Yu, and W.-S. Han, “Developing a hybrid dictionary- based bio-entity recognition technique,” BMC Medical Informatics and Decision Making, vol. 15, no. 1, p. S9, 2015
work page 2015
-
[4]
M. Topaz, K. Lai, D. Dowding, V . J. Lei, A. Zisberg, K. H. Bowles, and L. Zhou, “Automated identification of wound information in clinical notes of patients with heart diseases: Developing and validating a nat- ural language processing application,” International Journal of Nursing Studies, vol. 64, pp. 25–31, 2016
work page 2016
-
[5]
H.-J. Tan, R. Clarke, K. Chamie, A. Kaplan, A. Chin, M. S. Litwin, C. Saigal, and A. S. Hackbarth, “Development and validation of an au- tomated method for identifying patients undergoing radical cystectomy for bladder cancer using natural language processing,” Urology Practice, vol. 4, 11 2016
work page 2016
-
[6]
J. T. Senders, A. V . Karhade, D. J. Cote, A. Mehrtash, N. Lamba, A. DiRisio, I. S. Muskens, W. B. Gormley, T. R. Smith, M. L. Broekman et al., “Natural language processing for automated quantification of brain metastases reported in free-text radiology reports,” JCO Clinical Cancer Informatics, vol. 3, pp. 1–9, 2019
work page 2019
-
[7]
Automated extraction of family history information from clinical notes,
R. Bill, S. Pakhomov, E. S. Chen, T. J. Winden, E. W. Carter, and G. B. Melton, “Automated extraction of family history information from clinical notes,” in AMIA Annual Symposium Proceedings , vol. 2014. American Medical Informatics Association, 2014, p. 1709
work page 2014
-
[8]
E. Iqbal, R. Mallah, D. Rhodes, H. Wu, A. Romero, N. Chang, O. Dzahini, C. Pandey, M. Broadbent, R. Stewart et al. , “ADEPt, a semantically-enriched pipeline for extracting adverse drug events from free-text electronic health records,” PloS one , vol. 12, no. 11, p. e0187121, 2017
work page 2017
Show all 26 references
-
[9]
Fonferko-Shadrach, A
B. Fonferko-Shadrach, A. S. Lacey, A. Roberts, A. Akbari, S. Thomp- son, D. V . Ford, R. A. Lyons, M. I. Rees, and W. O. Pickrell, “Using natural language processing to extract structured epilepsy data from unstructured clinic letters: development and validation of the ExECT (...
2019
-
[10]
Improving language understanding by generative pre- training,
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre- training,” URL https://s3-us-west-2. amazonaws. com/openai- assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018
2018
-
[11]
Deep contextualized word representations,
M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...
2018
-
[12]
BERT: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018
2018 arXiv
-
[13]
XLNet: Generalized autoregressive pretraining for language under- standing,
Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. Salakhutdinov, and Q. V . Le, “XLNet: Generalized autoregressive pretraining for language under- standing,” arXiv preprint arXiv:1906.08237 , 2019
1906 arXiv
-
[14]
Biobert: pre-trained biomedical language representation model for biomedical text mining,
J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: pre-trained biomedical language representation model for biomedical text mining,” arXiv preprint arXiv:1901.08746 , 2019
1901 arXiv
-
[15]
A neural named entity recognition and multi- type normalization tool for biomedical text mining,
D. Kim, J. Lee, C. H. So, H. Jeon, M. Jeong, Y . Choi, W. Yoon, M. Sung, and J. Kang, “A neural named entity recognition and multi- type normalization tool for biomedical text mining,”IEEE Access, vol. 7, pp. 73 729–73 740, 2019
2019
-
[16]
Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field,
J. Qiu, Y . Zhou, Q. Wang, T. Ruan, and J. Gao, “Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field,” IEEE Transactions on NanoBioscience , vol. 18, no. 3, pp. 306–315, July 2019
2019
-
[17]
Incorporating dictionaries into deep neural networks for the chinese clinical named entity recognition,
Q. Wang, Y . Zhou, T. Ruan, D. Gao, Y . Xia, and P. He, “Incorporating dictionaries into deep neural networks for the chinese clinical named entity recognition,” Journal of Biomedical Informatics , vol. 92, p. 103133, 2019
2019
-
[18]
Bilinear cnn models for fine- grained visual recognition,
T.-Y . Lin, A. RoyChowdhury, and S. Maji, “Bilinear cnn models for fine- grained visual recognition,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 1449–1457
2015
-
[19]
Image style transfer using convolutional neural networks,
L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 2414–2423
2016
-
[20]
Boosted convolutional neural networks
M. Moghimi, S. J. Belongie, M. J. Saberian, J. Yang, N. Vasconcelos, and L.-J. Li, “Boosted convolutional neural networks.” in BMVC, 2016, pp. 24–1
2016
-
[21]
SQuAD: 100,000+ questions for machine comprehension of text,
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. ...
2016
-
[22]
A strategy on selecting performance metrics for classifier evaluation,
Y . Liu, Y . Zhou, S. Wen, and C. Tang, “A strategy on selecting performance metrics for classifier evaluation,” International Journal of Mobile Computing and Multimedia Communications (IJMCMC) , vol. 6, no. 4, pp. 20–35, 2014
2014
-
[23]
Correlation analysis of performance metrics for classifier,
Y . Zhou and Y . Liu, “Correlation analysis of performance metrics for classifier,” in Decision Making and Soft Computing: Proceedings of the 11th International FLINS Conference. World Scientific, 2014, pp. 487– 492
2014
-
[24]
Chollet et al., “Keras,” https://keras.io, 2015
F. Chollet et al., “Keras,” https://keras.io, 2015
2015
-
[25]
Tensorflow: a system for large- scale machine learning
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: a system for large- scale machine learning.” in OSDI, vol. 16, 2016, pp. 265–283
2016
-
[26]
Qanet: Combining local convolution with global self-attention for reading comprehension,
A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V . Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541 , 2018
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.