Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Question Answering based Clinical Text Structuring Using Pre-trained Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read By recasting clinical text structuring as question answering over the original report text, this paper shows that a pre-trained language model augmented with clinical named-entity tags reaches an exact-match score of 91.84 percent on…

desk verdict A useful task formulation that deserves review, but the headline numbers are about span extraction, not the clinical structuring the title promises. read the letter →

arxiv 1908.06606 v2 pith:PCFY3MBQ submitted 2019-08-19 cs.CL

classification cs.CL
keywords clinicaltextstructuringquestionansweringpre-trainedlanguagemodelnamedentityrecognitionelectronichealthrecordsChinesepathologyreportsspanextractiontransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clinical text structuring is usually tackled task by task, each with its own output format, its own model, and its own labeled data. This paper argues that most structuring problems share a common core: given a query such as "proximal resection margin," find the answer-related span inside the report text. Recasting structuring as question answering over the original text lets different tasks reuse one training set, and the paper shows this shared-data setup improves performance on task-specific test sets. The paper's model, which feeds clinical named-entity tag sequences into a pre-trained language model and then classifies start and end positions, reaches an exact-match score of 91.84 percent and an F1 score of 93.75 percent on Chinese pathology reports. That beats the plain pre-trained language model (EM 86.20, F1 90.06) and a strong reading-comprehension baseline (EM 85.45, F1 93.62).

What carries the argument

The load-bearing object is the QA-CTS task itself: given a report paragraph $X$ and a query $Q$, predict the start and end index of the answer-related substring $X_s$. The model couples two input streams: a pre-trained language model encodes $[CLS] Q [SEP] X [SEP]$ into a contextualized vector, while a clinical named-entity recognizer labels every character with BIEOS tags (Begin, Inside, End, Outside, Single) for 44 entity types; the two representations are concatenated and passed through a feed-forward network that scores each position as the start or end of the answer. The paper also uses a two-stage training schedule, first fine-tuning the language model with a prediction layer and then adding the entity-information layers and retraining, which it credits with improving performance.

What would settle it

Compute end-to-end accuracy on the final structured values (after entity-name conversion and negative-word recognition) for both the proposed model and the plain pre-trained language model; if the plain model's final-answer accuracy is close to or better than the proposed model's despite its lower span-extraction EM, then the claimed advantage of the QA-CTS model does not translate to real structuring accuracy.

Watch

Extended reading notes

Core claim

The central claim is that unifying clinical text structuring under a question-answering format makes the underlying task learnable from shared data and improves accuracy on individual structuring tasks. Rather than outputting a final structured value directly, the model is asked to locate the substring of the paragraph that answers a clinical query; later steps such as entity-name conversion and negative-word recognition are handled separately. On three query types from gastric pathology reports (tumor size, proximal resection margin, and distal resection margin), the authors find that a model trained on all three types performs better on each than a model trained on that type alone, and that mixed-data pretrained parameters lift accuracy on query types not seen in final training from near zero to roughly 60-70 percent. Adding clinical named-entity information yields a further gain, giving the best exact-match and F1 scores.

Load-bearing premise

The evaluation assumes that extracting the answer-related substring is a faithful proxy for clinical text structuring, since the paper measures only span extraction and states that final answers still need steps such as entity-name conversion and negative-word recognition.

Editorial extensions

If this is right

  • Training data from different clinical structuring tasks can be pooled, so a query type with few examples can borrow signal from more abundant query types.
  • On the three tested query types, the shared-task model reaches EM 91.84 and F1 93.75, surpassing both the plain pre-trained language model and the stronger baseline on exact match.
  • Mixed-data pretraining converts near-zero transfer performance on untrained query types into 60-70 percent accuracy, evidence of commonality across tasks.
  • The two-stage training mechanism is reported as a practical way to adapt large pre-trained models when computational resources are limited.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The span-extraction framing should transfer most directly to electronic-health-record fields whose values appear verbatim in text, such as lab results, measurements, and dates; testing it on fields requiring negation handling or inference would show how far the unification extends.
  • Because the model relies on a clinical named-entity tagger trained on Chinese EHRs, a natural extension is to replace that tagger with a multilingual or cross-institution tagger and measure whether the transfer gains persist across hospitals or languages.
  • The near-zero to 60-70 percent jump on unseen query types suggests the QA-CTS task could serve as a pretraining objective for clinical NLP, analogous to general-domain reading-comprehension pretraining, though the paper does not test this directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces QA-CTS, a reading-comprehension-style formulation of clinical text structuring in which a model extracts an answer-related substring from a clinical paragraph given a query. The proposed model augments BERT-Base with one-hot clinical named-entity tags (from an external CNER system) and uses a feed-forward layer to predict start and end positions, trained with a two-stage fine-tuning procedure. On a Chinese pathology-report dataset from Ruijin Hospital with three question types (tumor size, proximal resection margin, distal resection margin), the model is reported to achieve EM 91.84 and F1 93.75, outperforming BERT-Base and QANet on EM while being comparable to QANet on F1. Data-integration experiments show that training on mixed query types often helps, with some exceptions. The paper concludes that the shared QA-CTS task improves individual CTS tasks.

Significance. The shared QA formulation is a reasonable and potentially practical way to pool annotations across heterogeneous CTS tasks, and the dataset of 2,714 clinician-annotated QA pairs is a useful resource. The architecture is clearly specified, and the comparison with strong QA baselines is appropriate. However, the reported results only measure span extraction, not the final structured fields, and no significance testing or ablations are provided; therefore the central performance claims are not fully established. If the additional experiments requested below confirm the results, the paper would make a modest but useful contribution to clinical NLP.

major comments (4)
  1. [Section III; Tables II-IV] The evaluation is on the answer-related substring, not on the final structured value. Section III states that 'several steps such as entity names conversion and negative words recognition' are needed to obtain the final answer, so the reported EM/F1 scores do not measure clinical text structuring as claimed in the abstract. Please either evaluate the full post-processing chain end-to-end or, at minimum, quantify how often post-processing is required and measure its error contribution; otherwise the headline 'clinical text structuring' claim overstates the evidence.
  2. [Section V-C, Table IV] There are no multiple runs, standard deviations, or significance tests reported. The F1 advantage over QANet is only 0.13%, which is well within run-to-run variation for neural reading-comprehension models, so the statement that the proposed model 'significantly outperformed' QANet (Section V-C) is unsupported. Report repeated-run statistics with confidence intervals or significance tests for all main comparisons.
  3. [Section V-D, Tables II and III] The mixed-data conclusion is not uniformly supported by the tables. For Tumor Size, the pure model achieves EM 96.27/F1 96.08 while the Mixed Data model achieves EM 95.10/F1 94.51, so shared training actually hurts this task. The recommendation that 'pre-training the model in multiple datasets and then fine tuning the model on the specific dataset is the best way' needs to be qualified with a discussion of when negative transfer occurs.
  4. [Section IV and V-C] The central novelty is the integration of clinical named-entity information into a pre-trained language model, yet no ablation isolates this component. The reported comparison with BERT-Base lacks a BERT+two-stage baseline without CNER features, and a CNER-features baseline without two-stage training. Add such ablations to substantiate the claim that the CNER feature integration is responsible for the improvement.
minor comments (5)
  1. [Abstract] 'Ruijing Hospital' should be 'Ruijin Hospital' to match the affiliation.
  2. [Section IV] The notation for the CNER tag sequences (Inq, Int, In) is undefined and visually confusing; define them in equations or a table.
  3. [Section V] The text says 'the best results in tables are in bold,' but no numbers are bolded in Tables I-IV.
  4. [Tables II and III] The F1 measure is described as macro-averaged in Section V-A, but the table headers just say 'F1-score'; state the averaging explicitly in the captions.
  5. [Section IV-D] The two-stage training description is vague about which layers are frozen during the first stage; specify the freezing schedule.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the QA-CTS model is trained on a held-out split with externally pretrained BERT and a task-independent CNER feature extractor; the self-citation is not load-bearing.

full rationale

The paper's central empirical claim is that the proposed BERT-plus-CNER model reaches EM 91.84% and F1 93.75% on held-out QA-CTS test instances, exceeding BERT-Base and QANet baselines. This result is not constrained by construction: BERT weights are externally pretrained on a general Chinese corpus, and the CNER tagger from the authors' prior work [16] is trained on a separate corpus with 44 entity types and is not fine-tuned on the QA-CTS training split. The mixed-data benefit (Tables II and III) is measured by training on pure versus shared question-type datasets and testing on the same held-out split, so the comparison is empirical rather than definitional. The only notable weakness is a scope limitation stated by the authors in Section III: for many instances, the answer-related substring is not the final structured value, and further steps such as entity-name conversion and negative-word recognition are required. Because the paper evaluates only span extraction with EM and F1, the advertised clinical-text-structuring improvement may be overstated, and tumor-size EM slightly drops under mixed training (96.27% to 95.10%). These are validity concerns, not circular reductions: no fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and the one self-citation is used as a feature extractor rather than as evidence for the target result. Accordingly, no specific circular step can be exhibited, and the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The listed free parameters are standard hyperparameters or inherited from prior work, not fitted to the target result. The axioms are domain assumptions about transferability and task sufficiency that the empirical study depends on.

free parameters (3)
  • BERT fine-tuning learning rate = 5e-5
    Set to the default Adam learning rate, not varied or fitted to the test set, but a hyperparameter that could affect results.
  • Batch size = 3 or 4
    Chosen due to GPU memory constraints, not optimized for performance.
  • CNER entity type set = 44 types
    The authors' previous CNER model was trained on another corpus with 44 entity types; this choice may influence the quality of the features but is not tested.
assumptions (4)
  • domain assumption BERT pre-trained on general Chinese corpus transfers to clinical Chinese text.
    Section IV.A directly adopts Google's BERT-base Chinese parameters without further clinical pre-training.
  • domain assumption The CNER model of the authors' previous work produces useful features on pathology reports.
    Section IV.B applies a CNER model trained on another corpus with 44 entity types to both the query and the paragraph.
  • domain assumption The answer-related substring is sufficient for the final structured answer.
    Section III defines the task as finding the substring, and Section V evaluates only span extraction; the paper states that some cases require extra steps like negative word recognition, yet those steps are not evaluated.
  • domain assumption The three question types share enough common structure for mixed-data training to transfer.
    Section V.D interprets cross-task improvements as evidence of commonality, but this is an empirical assumption not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Question Answering based Clinical Text Structuring Using Pre-trained Language Model." pith.science (2026). https://pith.science/paper/PCFY3MBQ

@misc{pith2026190806606,
  author       = {Pith},
  title        = {Pith review of: Question Answering based Clinical Text Structuring Using Pre-trained Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCFY3MBQ}},
  note         = {Machine review of arXiv:1908.06606}
}
read the original abstract

Clinical text structuring is a critical and fundamental task for clinical research. Traditional methods such as taskspecific end-to-end models and pipeline models usually suffer from the lack of dataset and error propagation. In this paper, we present a question answering based clinical text structuring (QA-CTS) task to unify different specific tasks and make dataset shareable. A novel model that aims to introduce domain-specific features (e.g., clinical named entity information) into pre-trained language model is also proposed for QA-CTS task. Experimental results on Chinese pathology reports collected from Ruijing Hospital demonstrate our presented QA-CTS task is very effective to improve the performance on specific tasks. Our proposed model also competes favorably with strong baseline models in specific tasks.

Figures

Figures reproduced from arXiv: 1908.06606 by the authors.

Figure 1
Figure 1. An illustrative example of QA-CTS task. To reduce the pipeline depth and break the barrier of non￾uniform output formats, we present a question answering based clinical text structuring (QA-CTS) task (see [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our proposed model for QA-CTS task [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    Toward infor- mation extraction: identifying protein names from biological papers,

    K.-i. Fukuda, T. Tsunoda, A. Tamura, T. Takagi et al. , “Toward infor- mation extraction: identifying protein names from biological papers,” in Pacific Symposium on Biocomputing , vol. 707, no. 18, 1998, pp. 707– 718

  2. [2]

    Linguistic mapping of terminologies to SNOMED CT,

    Y . Wang, J. Patrick, G. Miller, and J. O’Halloran, “Linguistic mapping of terminologies to SNOMED CT,” in Semantic Mining Conference on SNOMED CT. Citeseer, 2006

  3. [3]

    Developing a hybrid dictionary- based bio-entity recognition technique,

    M. Song, H. Yu, and W.-S. Han, “Developing a hybrid dictionary- based bio-entity recognition technique,” BMC Medical Informatics and Decision Making, vol. 15, no. 1, p. S9, 2015

  4. [4]

    Automated identification of wound information in clinical notes of patients with heart diseases: Developing and validating a nat- ural language processing application,

    M. Topaz, K. Lai, D. Dowding, V . J. Lei, A. Zisberg, K. H. Bowles, and L. Zhou, “Automated identification of wound information in clinical notes of patients with heart diseases: Developing and validating a nat- ural language processing application,” International Journal of Nursing Studies, vol. 64, pp. 25–31, 2016

  5. [5]

    Development and validation of an au- tomated method for identifying patients undergoing radical cystectomy for bladder cancer using natural language processing,

    H.-J. Tan, R. Clarke, K. Chamie, A. Kaplan, A. Chin, M. S. Litwin, C. Saigal, and A. S. Hackbarth, “Development and validation of an au- tomated method for identifying patients undergoing radical cystectomy for bladder cancer using natural language processing,” Urology Practice, vol. 4, 11 2016

  6. [6]

    Natural language processing for automated quantification of brain metastases reported in free-text radiology reports,

    J. T. Senders, A. V . Karhade, D. J. Cote, A. Mehrtash, N. Lamba, A. DiRisio, I. S. Muskens, W. B. Gormley, T. R. Smith, M. L. Broekman et al., “Natural language processing for automated quantification of brain metastases reported in free-text radiology reports,” JCO Clinical Cancer Informatics, vol. 3, pp. 1–9, 2019

  7. [7]

    Automated extraction of family history information from clinical notes,

    R. Bill, S. Pakhomov, E. S. Chen, T. J. Winden, E. W. Carter, and G. B. Melton, “Automated extraction of family history information from clinical notes,” in AMIA Annual Symposium Proceedings , vol. 2014. American Medical Informatics Association, 2014, p. 1709

  8. [8]

    ADEPt, a semantically-enriched pipeline for extracting adverse drug events from free-text electronic health records,

    E. Iqbal, R. Mallah, D. Rhodes, H. Wu, A. Romero, N. Chang, O. Dzahini, C. Pandey, M. Broadbent, R. Stewart et al. , “ADEPt, a semantically-enriched pipeline for extracting adverse drug events from free-text electronic health records,” PloS one , vol. 12, no. 11, p. e0187121, 2017

Show all 26 references
  1. [9]

    Fonferko-Shadrach, A

    B. Fonferko-Shadrach, A. S. Lacey, A. Roberts, A. Akbari, S. Thomp- son, D. V . Ford, R. A. Lyons, M. I. Rees, and W. O. Pickrell, “Using natural language processing to extract structured epilepsy data from unstructured clinic letters: development and validation of the ExECT (...

  2. [10]

    Improving language understanding by generative pre- training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre- training,” URL https://s3-us-west-2. amazonaws. com/openai- assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018

  3. [11]

    Deep contextualized word representations,

    M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

  4. [12]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

  5. [13]

    XLNet: Generalized autoregressive pretraining for language under- standing,

    Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. Salakhutdinov, and Q. V . Le, “XLNet: Generalized autoregressive pretraining for language under- standing,” arXiv preprint arXiv:1906.08237 , 2019

  6. [14]

    Biobert: pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: pre-trained biomedical language representation model for biomedical text mining,” arXiv preprint arXiv:1901.08746 , 2019

  7. [15]

    A neural named entity recognition and multi- type normalization tool for biomedical text mining,

    D. Kim, J. Lee, C. H. So, H. Jeon, M. Jeong, Y . Choi, W. Yoon, M. Sung, and J. Kang, “A neural named entity recognition and multi- type normalization tool for biomedical text mining,”IEEE Access, vol. 7, pp. 73 729–73 740, 2019

  8. [16]

    Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field,

    J. Qiu, Y . Zhou, Q. Wang, T. Ruan, and J. Gao, “Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field,” IEEE Transactions on NanoBioscience , vol. 18, no. 3, pp. 306–315, July 2019

  9. [17]

    Incorporating dictionaries into deep neural networks for the chinese clinical named entity recognition,

    Q. Wang, Y . Zhou, T. Ruan, D. Gao, Y . Xia, and P. He, “Incorporating dictionaries into deep neural networks for the chinese clinical named entity recognition,” Journal of Biomedical Informatics , vol. 92, p. 103133, 2019

  10. [18]

    Bilinear cnn models for fine- grained visual recognition,

    T.-Y . Lin, A. RoyChowdhury, and S. Maji, “Bilinear cnn models for fine- grained visual recognition,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 1449–1457

  11. [19]

    Image style transfer using convolutional neural networks,

    L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 2414–2423

  12. [20]

    Boosted convolutional neural networks

    M. Moghimi, S. J. Belongie, M. J. Saberian, J. Yang, N. Vasconcelos, and L.-J. Li, “Boosted convolutional neural networks.” in BMVC, 2016, pp. 24–1

  13. [21]

    SQuAD: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. ...

  14. [22]

    A strategy on selecting performance metrics for classifier evaluation,

    Y . Liu, Y . Zhou, S. Wen, and C. Tang, “A strategy on selecting performance metrics for classifier evaluation,” International Journal of Mobile Computing and Multimedia Communications (IJMCMC) , vol. 6, no. 4, pp. 20–35, 2014

  15. [23]

    Correlation analysis of performance metrics for classifier,

    Y . Zhou and Y . Liu, “Correlation analysis of performance metrics for classifier,” in Decision Making and Soft Computing: Proceedings of the 11th International FLINS Conference. World Scientific, 2014, pp. 487– 492

  16. [24]

    Chollet et al., “Keras,” https://keras.io, 2015

    F. Chollet et al., “Keras,” https://keras.io, 2015

  17. [25]

    Tensorflow: a system for large- scale machine learning

    M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: a system for large- scale machine learning.” in OSDI, vol. 16, 2016, pp. 265–283

  18. [26]

    Qanet: Combining local convolution with global self-attention for reading comprehension,

    A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V . Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541 , 2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.