Pith. sign in

REVIEW 3 major objections 4 minor 20 references

Addressing Data Bias Problems for Chest X-ray Image Report Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Routing each chest X-ray report sentence to a dedicated abnormal or normal word LSTM reduces normal-case bias and produces more distinct reports without sacrificing relevance.

desk verdict Useful architecture and a real metric critique, but the distinctiveness benefit is not demonstrated in the selected model and the routing predictor is unvalidated. read the letter →

arxiv 1908.02123 v1 pith:NBKJP2EF submitted 2019-08-06 cs.CV

classification cs.CV
keywords chestX-rayreportgenerationdatabiashierarchicalLSTMdualwordabnormalsentencepredictiondistinctivenessBLEUevaluationmedicalimagecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the normal-case bias in chest X-ray report generation can be reduced by splitting sentence generation into two decoders: one word LSTM trained on sentences describing abnormalities and one trained on normal findings. A hierarchical LSTM proposes each sentence's topic, an abnormal-sentence predictor decides which decoder should write it, and the final report is assembled from both. The authors report that this dual-decoder model generates a larger number of distinct sentences per paragraph position while keeping, and in several cases improving, BLEU, METEOR, ROUGE-L and CIDEr scores relative to single-decoder hierarchical baselines. They also show that BLEU alone is a misleading selection signal: a model can reach its highest BLEU score while emitting the same paragraph for every input. The practical point of the proposal is that automated reports should vary with the image rather than collapsing to the most frequent normal template.

What carries the argument

The load-bearing mechanism is a hierarchical LSTM with a dual word LSTM. A sentence LSTM produces a topic vector $t_m$ for each sentence from attended image features; a stop predictor decides when the paragraph ends; and two separate word LSTMs, one trained only on abnormal sentences and one only on normal sentences, write the actual words. A fully-connected abnormal-sentence predictor $\tau_m$ reads the topic vector and, at inference, routes each sentence to the abnormal or normal word LSTM depending on whether its probability exceeds 0.5. The full objective combines the hierarchical captioning loss, the stop loss, the abnormality-prediction loss, and a multi-task loss for medical-tag (MTI) prediction. The pair of decoders is what lets the model move away from the most probable normal template while keeping the sentences grammatical.

What would settle it

Compute the abnormal-sentence predictor's accuracy on held-out sentences with clinician labels, then compare predicted routing against oracle routing that uses ground-truth labels: if oracle routing does not beat predicted routing, the routing module, not the dual decoders, carries the reported gains, and if predicted routing is near chance, the central benefit is not realized.

Watch

Extended reading notes

Core claim

The central claim is that separating generation of abnormal and normal sentences, rather than training one shared word LSTM for all sentences, counteracts the data imbalance in chest X-ray reports. On the public chest X-ray report corpus, the model with attention and dual word LSTMs is consistently better than its single-decoder hierarchical baselines on multi-gram BLEU, METEOR and ROUGE-L, and best on CIDEr, while also increasing the number of distinct sentences at each sentence position within the paragraph. The paper states that this indicates 'the relevance is not sacrificed while distinctiveness is increased.' A second, related claim is that BLEU-style evaluation alone is inadequate for this setting: the paper documents a training checkpoint with a higher BLEU-4 score than its selected model even though the model generated the very same paragraph for every validation sample, and therefore proposes selecting models with a minimum-distinctiveness constraint alongside BLEU-4.

Load-bearing premise

The inference-time routing relies on the abnormal sentence predictor choosing the right decoder for each sentence, but the paper does not report that predictor's accuracy or the agreement of the human labels it was trained on; if routing is often wrong, the separate decoders cannot deliver their benefit.

Editorial extensions

If this is right

  • If the dual-decoder result holds, report generators for imbalanced clinical corpora can raise output diversity without sacrificing n-gram overlap with reference reports.
  • The documented case of peak BLEU-4 from a model that repeats one paragraph implies that BLEU-style rankings on this dataset should be paired with a distinctiveness measure.
  • On the held-out abnormal- and normal-only test slices, a dual model is best for both, so the diversity gain is not concentrated in one class.
  • The proposed checkpoint selection rule—at least four distinct first sentences on the validation set, then highest BLEU-4—is a reproducible recipe that the paper shows outperforms BLEU-only selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the abnormal-sentence predictor is the operational gatekeeper: a clinical deployment would need its accuracy reported and calibrated, because a misrouted sentence would be written by a decoder trained on the wrong distribution.
  • The same routing idea transfers to other imbalanced text-generation settings where individual sentences carry the minority class, such as pathology or follow-up radiology notes, at the cost of per-sentence annotation.
  • A testable extension is to compare the dual-decoder model against a single decoder whose topic vector is concatenated with the predicted abnormality probability, isolating whether separate parameters or the routing signal drive the gain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses the bias of chest X-ray report generation toward normal findings by proposing a hierarchical LSTM with dual word LSTMs, one for abnormal and one for normal sentences, plus an abnormal-sentence predictor that routes each sentence at inference time. The authors manually annotate sentences in the IU chest X-ray dataset as abnormal or normal, use MTI tags for multitask learning, and evaluate BLEU, CIDEr, METEOR, and ROUGE-L on validation and held-out test splits. They also analyze the trade-off between BLEU scores and the number of distinct generated sentences and propose a model-selection criterion that includes a minimum distinctness threshold.

Significance. The dual-decoder idea and the analysis of distinctiveness versus BLEU are potentially useful for medical report generation, where overly generic reports are a known failure mode. The paper contributes a sentence-level abnormality annotation of a public dataset and a concrete model-selection heuristic. If the central claims were supported, the architecture would be a simple and interesting step toward more clinically informative report variability. However, the current evidence does not establish the main claims: the paper's own distinctness table appears to contradict the distinctiveness claim, the routing predictor is not evaluated, and the quality improvements are reported without statistical significance.

major comments (3)
  1. [Section 5.2, Table 3, Section 5.3, Section 6] The central claim that the dual word LSTM increases distinctiveness without sacrificing relevance is contradicted by Table 3 for the selected checkpoints. HLSTM+att+Dual generates 5, 10, 7, 8, 8, 4, 1, 0, 0, 0 distinct validation sentences for sentence indices m=0..9, whereas HLSTM+att generates 5, 24, 24, 33, 25, 31, 23, 14, 9, 3, and HLSTM+Dual generates 8, 28, 36, 45, 32, 17, 2, 0, 0, 0. Thus the model for which the main quality claim is made has the lowest distinctiveness among the hierarchical baselines at nearly every sentence position. This directly undermines the statement in Section 5.2 that 'far more distinct sentences are generated' and the conclusion in Section 6 that the dual word LSTM 'can help to increase the number of distinct sentences faster.' If the claim refers to training dynamics rather than the final selected checkpoints, that distinction is not made, and Tables 2 and 4 report scores for the final selected models only. The authors should report distinctiveness for the actual selected models, or provide a checkpoint-matched comparison with a BLEU-distinctiveness trade-off curve that supports the claim.
  2. [Section 4.2] The abnormal sentence predictor τ_m that routes each sentence to the abnormal or normal word LSTM at inference is never evaluated. The paper does not report its held-out accuracy, F1, confusion matrix, or any qualitative example of routing behavior. Since the entire benefit of the dual-LSTM architecture depends on correct routing, the gains in Tables 2 and 4 cannot be attributed to the proposed separation of abnormal and normal generation. The training of the two word LSTMs uses ground-truth labels, so the inference-time routing is a train/test mismatch that must be validated directly. Please report routing performance on the test set and, ideally, an analysis of how routing errors affect generated report quality.
  3. [Tables 2 and 4, Section 5.3] No confidence intervals, error bars, or significance tests are provided for any of the reported metrics. The test set is only 5% of roughly 3,955 reports, so differences such as BLEU-4 of 12.6 versus 11.7 (HLSTM+att+Dual vs. HLSTM+att) or CIDEr of 49.5 versus 42.6 on the normal subset may be within run-to-run or test-sampling noise. The claim that 'HLSTM+att+Dual is consistently better than other baselines' therefore needs bootstrap confidence intervals or paired tests across multiple random seeds and checkpoint selections.
minor comments (4)
  1. [Figure 3] The solid and dashed vertical lines are described only in the text; the caption should define them explicitly. Also, the y-axis label '# distinct sentences' appears twice; one of them should be labeled to indicate the sentence index m.
  2. [Table 2 note] The footnote that CoAtt and KERP results use a different dataset split is important, but the table lists them in the same column as directly evaluated models. Consider separating them with a different table or a clear horizontal rule so the comparison is not misleading.
  3. [Section 4.2] The manual annotation procedure is described in one sentence with no inter-annotator agreement statistic. Reporting Cohen's kappa or a similar measure would strengthen the claimed label quality.
  4. [Typos] There are minor typographical issues, e.g., 'hierachical' in Section 4.1 and 'HARZIG ET AL.: DA TA BIAS' in the running header. These do not affect the science but should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical train/test evaluation with held-out test set; no claim reduces to its own inputs.

full rationale

This is an empirical systems paper. The central contribution—separate abnormal and normal word LSTMs selected by an abnormal sentence predictor—is trained on manually annotated sentence labels and evaluated on a held-out test split. No 'prediction' is algebraically equivalent to a fitted value: the reported BLEU, CIDEr, METEOR, Rouge-L, and distinct-sentence counts are measured outputs of trained models on validation/test data, not constraints that define the model. The model selection rule in Section 5.1 (at least 4 distinct sentences at m=0, then highest BLEU-4) is applied uniformly to all baselines and does not by construction favor any architecture. Comparisons against external methods are either marked as not directly comparable or replaced by self-implemented hierarchical baselines, so no load-bearing claim rests on a self-citation chain. The paper's own references to previous work are limited to standard architectural building blocks and do not smuggle in the dual-LSTM result. Although there is an apparent tension between the qualitative claim that dual word LSTMs produce 'far more distinct sentences' and the numeric Table 3 results, that is a potential factual or checkpoint-selection inconsistency affecting correctness, not circularity: it does not reduce a claimed result to an input by construction. Therefore no significant circularity is found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

This paper is an empirical machine-learning study, so the numbers it depends on are learned weights plus hand-picked hyperparameters. I list the externally chosen hyperparameters and the data/annotation assumptions. No new physical or mathematical entities are introduced, so invented_entities is empty. The important caveat is that the manual abnormal/normal labels and the accuracy of the predictor are assumed but not independently verified.

free parameters (3)
  • loss weighting factors = lambda_MTI=10; lambda_stop=lambda_hierarchical=lambda_abnormal=1
    Equation 4 defines the total loss; these weights are set by hand and control the balance between captioning, stop prediction, abnormal prediction, and MTI classification.
  • minimum distinct first-sentence threshold = 4
    Section 5.1 uses 'at least 4 distinct sentences' at sentence index 0 as one model-selection criterion. The value is arbitrary and influences which checkpoint is reported.
  • training hyperparameters = lr=5e-4, batch size=16, 250 epochs, hidden/embedding dim=512
    Adam with a fixed learning rate and no decay; these choices affect convergence and final scores.
assumptions (5)
  • domain assumption The manual abnormal/normal annotations are correct and consistently applied.
    Two annotators labeled every training sentence, but no inter-annotator agreement or validation of the labels is reported (Section 4.2).
  • domain assumption The abnormal sentence predictor accurately maps a topic vector to the correct decoder at inference.
    The paper uses the predictor to route each sentence to the abnormal or normal LSTM but never reports the predictor's accuracy (Sections 4.2 and 5).
  • domain assumption ResNet-152 features pretrained on ImageNet are useful for chest X-ray images.
    Section 4 uses the res4b35 feature map as the image representation; transfer from natural images to X-rays is assumed.
  • domain assumption Automatic metrics and distinct-sentence counts measure report quality and diversity well enough to support the conclusions.
    No clinical or human evaluation is performed; the conclusions rest on BLEU, CIDEr, Meteor, Rouge-L, and a hand-defined distinctiveness count.
  • domain assumption The IU chest X-ray reports are correctly extracted and the train/validation/test split is unbiased.
    Section 5.1; the paper concatenates impression and findings and splits by patient, but does not verify the parser or annotation quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing Data Bias Problems for Chest X-ray Image Report Generation." pith.science (2026). https://pith.science/paper/NBKJP2EF

@misc{pith2026190802123,
  author       = {Pith},
  title        = {Pith review of: Addressing Data Bias Problems for Chest X-ray Image Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBKJP2EF}},
  note         = {Machine review of arXiv:1908.02123}
}
read the original abstract

Automatic medical report generation from chest X-ray images is one possibility for assisting doctors to reduce their workload. However, the different patterns and data distribution of normal and abnormal cases can bias machine learning models. Previous attempts did not focus on isolating the generation of the abnormal and normal sentences in order to increase the variability of generated paragraphs. To address this, we propose to separate abnormal and normal sentence generation by using two different word LSTMs in a hierarchical LSTM model. We conduct an analysis on the distinctiveness of generated sentences compared to the BLEU score, which increases when less distinct reports are generated. We hope our findings will help to encourage the development of new metrics to better verify methods of automatic medical report generation.

Figures

Figures reproduced from arXiv: 1908.02123 by the authors.

Figure 1
Figure 1. An example from the IU chest X-ray dataset, which shows an abnormal case with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our dual word LSTM model. The green boxes show the CNN image encoder, the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The number of distinct sentences of sentence [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of generated paragraphs with our model [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

  1. [1]

    Preparing a collection of radiology examinations for distribution and retrieval

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310, 2015

  2. [2]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  3. [3]

    On the automatic generation of medical imaging reports

    Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2577–2586. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/ P18-1240

  4. [4]

    Densecap: Fully convolutional lo- calization networks for dense captioning

    Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional lo- calization networks for dense captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016

  5. [5]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015

  6. [6]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

  7. [7]

    A hierarchical ap- proach for generating descriptive image paragraphs

    Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical ap- proach for generating descriptive image paragraphs. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 3337–3345. IEEE, 2017

  8. [8]

    Hybrid retrieval- generation reinforced agent for medical image report generation

    Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. Hybrid retrieval- generation reinforced agent for medical image report generation. arXiv preprint arXiv:1805.08298, 2018. HARZIG ET AL.: DA TA BIAS PROBLEMS FOR CXR MEDICAL REPORT GENERA TION 11

Show all 20 references
  1. [9]

    Knowledge-driven encode, retrieve, paraphrase for medical image report generation.AAAI Conference on Artificial Intelligence, 2019

    Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. Knowledge-driven encode, retrieve, paraphrase for medical image report generation.AAAI Conference on Artificial Intelligence, 2019

  2. [10]

    Tho- racic disease identification and localization with limited supervision

    Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and F Li. Tho- racic disease identification and localization with limited supervision. arXiv preprint arXiv:1711.06373, 2017

  3. [11]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. Proceedings of Workshop at ICLR, 2013, 2013

  4. [12]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. InAdvances in neural information processing systems, pages 3111–3119, 2013

  5. [13]

    Distributional semantics re- sources for biomedical text processing

    SPFGH Moen and Tapio Salakoski2 Sophia Ananiadou. Distributional semantics re- sources for biomedical text processing. In Proceedings of the 5th International Sympo- sium on Languages in Biology and Medicine, Tokyo, Japan, pages 39–43, 2013

  6. [14]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics , pages 311–318. Association for Computational Linguistics, 2002

  7. [15]

    Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning

    Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017

  8. [16]

    Cider: Consensus- based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 4566–4575, 2015

  9. [17]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015

  10. [18]

    Chestx-ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localization of common thorax diseases. In Computer Vision and Pattern Recognition...

  11. [19]

    Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9049–9058, 2018

  12. [20]

    Multimodal recurrent model with attention for automated radiol- ogy report generation

    Yuan Xue, Tao Xu, L Rodney Long, Zhiyun Xue, Sameer Antani, George R Thoma, and Xiaolei Huang. Multimodal recurrent model with attention for automated radiol- ogy report generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, page...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.