REVIEW 3 major objections 4 minor 20 references
Addressing Data Bias Problems for Chest X-ray Image Report Generation
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Routing each chest X-ray report sentence to a dedicated abnormal or normal word LSTM reduces normal-case bias and produces more distinct reports without sacrificing relevance.
desk verdict Useful architecture and a real metric critique, but the distinctiveness benefit is not demonstrated in the selected model and the routing predictor is unvalidated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a hierarchical LSTM with a dual word LSTM. A sentence LSTM produces a topic vector $t_m$ for each sentence from attended image features; a stop predictor decides when the paragraph ends; and two separate word LSTMs, one trained only on abnormal sentences and one only on normal sentences, write the actual words. A fully-connected abnormal-sentence predictor $\tau_m$ reads the topic vector and, at inference, routes each sentence to the abnormal or normal word LSTM depending on whether its probability exceeds 0.5. The full objective combines the hierarchical captioning loss, the stop loss, the abnormality-prediction loss, and a multi-task loss for medical-tag (MTI) prediction. The pair of decoders is what lets the model move away from the most probable normal template while keeping the sentences grammatical.
What would settle it
Compute the abnormal-sentence predictor's accuracy on held-out sentences with clinician labels, then compare predicted routing against oracle routing that uses ground-truth labels: if oracle routing does not beat predicted routing, the routing module, not the dual decoders, carries the reported gains, and if predicted routing is near chance, the central benefit is not realized.
Extended reading notes
Core claim
The central claim is that separating generation of abnormal and normal sentences, rather than training one shared word LSTM for all sentences, counteracts the data imbalance in chest X-ray reports. On the public chest X-ray report corpus, the model with attention and dual word LSTMs is consistently better than its single-decoder hierarchical baselines on multi-gram BLEU, METEOR and ROUGE-L, and best on CIDEr, while also increasing the number of distinct sentences at each sentence position within the paragraph. The paper states that this indicates 'the relevance is not sacrificed while distinctiveness is increased.' A second, related claim is that BLEU-style evaluation alone is inadequate for this setting: the paper documents a training checkpoint with a higher BLEU-4 score than its selected model even though the model generated the very same paragraph for every validation sample, and therefore proposes selecting models with a minimum-distinctiveness constraint alongside BLEU-4.
Load-bearing premise
The inference-time routing relies on the abnormal sentence predictor choosing the right decoder for each sentence, but the paper does not report that predictor's accuracy or the agreement of the human labels it was trained on; if routing is often wrong, the separate decoders cannot deliver their benefit.
Editorial extensions
If this is right
- If the dual-decoder result holds, report generators for imbalanced clinical corpora can raise output diversity without sacrificing n-gram overlap with reference reports.
- The documented case of peak BLEU-4 from a model that repeats one paragraph implies that BLEU-style rankings on this dataset should be paired with a distinctiveness measure.
- On the held-out abnormal- and normal-only test slices, a dual model is best for both, so the diversity gain is not concentrated in one class.
- The proposed checkpoint selection rule—at least four distinct first sentences on the validation set, then highest BLEU-4—is a reproducible recipe that the paper shows outperforms BLEU-only selection.
Reading between the lines
- An implication the authors leave implicit is that the abnormal-sentence predictor is the operational gatekeeper: a clinical deployment would need its accuracy reported and calibrated, because a misrouted sentence would be written by a decoder trained on the wrong distribution.
- The same routing idea transfers to other imbalanced text-generation settings where individual sentences carry the minority class, such as pathology or follow-up radiology notes, at the cost of per-sentence annotation.
- A testable extension is to compare the dual-decoder model against a single decoder whose topic vector is concatenated with the predicted abnormality probability, isolating whether separate parameters or the routing signal drive the gain.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses the bias of chest X-ray report generation toward normal findings by proposing a hierarchical LSTM with dual word LSTMs, one for abnormal and one for normal sentences, plus an abnormal-sentence predictor that routes each sentence at inference time. The authors manually annotate sentences in the IU chest X-ray dataset as abnormal or normal, use MTI tags for multitask learning, and evaluate BLEU, CIDEr, METEOR, and ROUGE-L on validation and held-out test splits. They also analyze the trade-off between BLEU scores and the number of distinct generated sentences and propose a model-selection criterion that includes a minimum distinctness threshold.
Significance. The dual-decoder idea and the analysis of distinctiveness versus BLEU are potentially useful for medical report generation, where overly generic reports are a known failure mode. The paper contributes a sentence-level abnormality annotation of a public dataset and a concrete model-selection heuristic. If the central claims were supported, the architecture would be a simple and interesting step toward more clinically informative report variability. However, the current evidence does not establish the main claims: the paper's own distinctness table appears to contradict the distinctiveness claim, the routing predictor is not evaluated, and the quality improvements are reported without statistical significance.
major comments (3)
- [Section 5.2, Table 3, Section 5.3, Section 6] The central claim that the dual word LSTM increases distinctiveness without sacrificing relevance is contradicted by Table 3 for the selected checkpoints. HLSTM+att+Dual generates 5, 10, 7, 8, 8, 4, 1, 0, 0, 0 distinct validation sentences for sentence indices m=0..9, whereas HLSTM+att generates 5, 24, 24, 33, 25, 31, 23, 14, 9, 3, and HLSTM+Dual generates 8, 28, 36, 45, 32, 17, 2, 0, 0, 0. Thus the model for which the main quality claim is made has the lowest distinctiveness among the hierarchical baselines at nearly every sentence position. This directly undermines the statement in Section 5.2 that 'far more distinct sentences are generated' and the conclusion in Section 6 that the dual word LSTM 'can help to increase the number of distinct sentences faster.' If the claim refers to training dynamics rather than the final selected checkpoints, that distinction is not made, and Tables 2 and 4 report scores for the final selected models only. The authors should report distinctiveness for the actual selected models, or provide a checkpoint-matched comparison with a BLEU-distinctiveness trade-off curve that supports the claim.
- [Section 4.2] The abnormal sentence predictor τ_m that routes each sentence to the abnormal or normal word LSTM at inference is never evaluated. The paper does not report its held-out accuracy, F1, confusion matrix, or any qualitative example of routing behavior. Since the entire benefit of the dual-LSTM architecture depends on correct routing, the gains in Tables 2 and 4 cannot be attributed to the proposed separation of abnormal and normal generation. The training of the two word LSTMs uses ground-truth labels, so the inference-time routing is a train/test mismatch that must be validated directly. Please report routing performance on the test set and, ideally, an analysis of how routing errors affect generated report quality.
- [Tables 2 and 4, Section 5.3] No confidence intervals, error bars, or significance tests are provided for any of the reported metrics. The test set is only 5% of roughly 3,955 reports, so differences such as BLEU-4 of 12.6 versus 11.7 (HLSTM+att+Dual vs. HLSTM+att) or CIDEr of 49.5 versus 42.6 on the normal subset may be within run-to-run or test-sampling noise. The claim that 'HLSTM+att+Dual is consistently better than other baselines' therefore needs bootstrap confidence intervals or paired tests across multiple random seeds and checkpoint selections.
minor comments (4)
- [Figure 3] The solid and dashed vertical lines are described only in the text; the caption should define them explicitly. Also, the y-axis label '# distinct sentences' appears twice; one of them should be labeled to indicate the sentence index m.
- [Table 2 note] The footnote that CoAtt and KERP results use a different dataset split is important, but the table lists them in the same column as directly evaluated models. Consider separating them with a different table or a clear horizontal rule so the comparison is not misleading.
- [Section 4.2] The manual annotation procedure is described in one sentence with no inter-annotator agreement statistic. Reporting Cohen's kappa or a similar measure would strengthen the claimed label quality.
- [Typos] There are minor typographical issues, e.g., 'hierachical' in Section 4.1 and 'HARZIG ET AL.: DA TA BIAS' in the running header. These do not affect the science but should be corrected.
Circularity Check
No significant circularity: empirical train/test evaluation with held-out test set; no claim reduces to its own inputs.
full rationale
This is an empirical systems paper. The central contribution—separate abnormal and normal word LSTMs selected by an abnormal sentence predictor—is trained on manually annotated sentence labels and evaluated on a held-out test split. No 'prediction' is algebraically equivalent to a fitted value: the reported BLEU, CIDEr, METEOR, Rouge-L, and distinct-sentence counts are measured outputs of trained models on validation/test data, not constraints that define the model. The model selection rule in Section 5.1 (at least 4 distinct sentences at m=0, then highest BLEU-4) is applied uniformly to all baselines and does not by construction favor any architecture. Comparisons against external methods are either marked as not directly comparable or replaced by self-implemented hierarchical baselines, so no load-bearing claim rests on a self-citation chain. The paper's own references to previous work are limited to standard architectural building blocks and do not smuggle in the dual-LSTM result. Although there is an apparent tension between the qualitative claim that dual word LSTMs produce 'far more distinct sentences' and the numeric Table 3 results, that is a potential factual or checkpoint-selection inconsistency affecting correctness, not circularity: it does not reduce a claimed result to an input by construction. Therefore no significant circularity is found.
Assumptions & free parameters
free parameters (3)
- loss weighting factors =
lambda_MTI=10; lambda_stop=lambda_hierarchical=lambda_abnormal=1
- minimum distinct first-sentence threshold =
4
- training hyperparameters =
lr=5e-4, batch size=16, 250 epochs, hidden/embedding dim=512
assumptions (5)
- domain assumption The manual abnormal/normal annotations are correct and consistently applied.
- domain assumption The abnormal sentence predictor accurately maps a topic vector to the correct decoder at inference.
- domain assumption ResNet-152 features pretrained on ImageNet are useful for chest X-ray images.
- domain assumption Automatic metrics and distinct-sentence counts measure report quality and diversity well enough to support the conclusions.
- domain assumption The IU chest X-ray reports are correctly extracted and the train/validation/test split is unbiased.
Cite this review
Pith. "Pith review of Addressing Data Bias Problems for Chest X-ray Image Report Generation." pith.science (2026). https://pith.science/paper/NBKJP2EF
@misc{pith2026190802123,
author = {Pith},
title = {Pith review of: Addressing Data Bias Problems for Chest X-ray Image Report Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/NBKJP2EF}},
note = {Machine review of arXiv:1908.02123}
}
read the original abstract
Automatic medical report generation from chest X-ray images is one possibility for assisting doctors to reduce their workload. However, the different patterns and data distribution of normal and abnormal cases can bias machine learning models. Previous attempts did not focus on isolating the generation of the abnormal and normal sentences in order to increase the variability of generated paragraphs. To address this, we propose to separate abnormal and normal sentence generation by using two different word LSTMs in a hierarchical LSTM model. We conduct an analysis on the distinctiveness of generated sentences compared to the BLEU score, which increases when less distinct reports are generated. We hope our findings will help to encourage the development of new metrics to better verify methods of automatic medical report generation.
Figures
Reference graph
Works this paper leans on
-
[1]
Preparing a collection of radiology examinations for distribution and retrieval
Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310, 2015
work page 2015
-
[2]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[3]
On the automatic generation of medical imaging reports
Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2577–2586. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/ P18-1240
work page 2018
-
[4]
Densecap: Fully convolutional lo- calization networks for dense captioning
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional lo- calization networks for dense captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016
work page 2016
-
[5]
Deep visual-semantic alignments for generating image descriptions
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015
2015
-
[6]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
arXiv 2014
-
[7]
A hierarchical ap- proach for generating descriptive image paragraphs
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical ap- proach for generating descriptive image paragraphs. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 3337–3345. IEEE, 2017
work page 2017
-
[8]
Hybrid retrieval- generation reinforced agent for medical image report generation
Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. Hybrid retrieval- generation reinforced agent for medical image report generation. arXiv preprint arXiv:1805.08298, 2018. HARZIG ET AL.: DA TA BIAS PROBLEMS FOR CXR MEDICAL REPORT GENERA TION 11
arXiv 2018
Show all 20 references
-
[9]
Knowledge-driven encode, retrieve, paraphrase for medical image report generation.AAAI Conference on Artificial Intelligence, 2019
Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. Knowledge-driven encode, retrieve, paraphrase for medical image report generation.AAAI Conference on Artificial Intelligence, 2019
2019
-
[10]
Tho- racic disease identification and localization with limited supervision
Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and F Li. Tho- racic disease identification and localization with limited supervision. arXiv preprint arXiv:1711.06373, 2017
2017 arXiv
-
[11]
Efficient estimation of word representations in vector space
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. Proceedings of Workshop at ICLR, 2013, 2013
2013
-
[12]
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. InAdvances in neural information processing systems, pages 3111–3119, 2013
2013
-
[13]
Distributional semantics re- sources for biomedical text processing
SPFGH Moen and Tapio Salakoski2 Sophia Ananiadou. Distributional semantics re- sources for biomedical text processing. In Proceedings of the 5th International Sympo- sium on Languages in Biology and Medicine, Tokyo, Japan, pages 39–43, 2013
2013
-
[14]
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics , pages 311–318. Association for Computational Linguistics, 2002
2002
-
[15]
Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017
2017 arXiv
-
[16]
Cider: Consensus- based image description evaluation
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 4566–4575, 2015
2015
-
[17]
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015
2015
-
[18]
Chestx-ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localization of common thorax diseases
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and bench- marks on weakly-supervised classification and localization of common thorax diseases. In Computer Vision and Pattern Recognition...
2017
-
[19]
Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9049–9058, 2018
2018
-
[20]
Multimodal recurrent model with attention for automated radiol- ogy report generation
Yuan Xue, Tao Xu, L Rodney Long, Zhiyun Xue, Sameer Antani, George R Thoma, and Xiaolei Huang. Multimodal recurrent model with attention for automated radiol- ogy report generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, page...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.