{"id":"d5715641-0d41-4d9a-beb3-1ae6bc110ee9","arxiv_id":"1908.02123","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A dual word LSTM that separates normal from abnormal sentence generation modestly improves diverse chest X-ray report generation and reveals that BLEU rewards repetitive reports.","lead":"An image-captioning system for chest X-rays is modified to generate normal and abnormal sentences with two separate word-level LSTMs, so that rare abnormal sentences are not drowned out by common normal ones. The paper also shows that BLEU scores can rise when a model repeats the same generic paragraph, and proposes a model-selection rule that prefers more varied reports.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dual-LSTM distinctiveness claim is contradicted by Table 3 and depends on an unvalidated routing predictor; the central benefit is not established.","rationale":"The reader's weakest assumption about the unvalidated abnormal-sentence predictor is genuine and load-bearing, but it is secondary to an even more direct problem already visible in the paper: the distinctiveness table contradicts the prose claim. If the dual model actually produces fewer distinct sentences than the single-LSTM baseline at the selected checkpoint, the 'distinctiveness increased' half of the central claim is false as stated, regardless of how well the router works. The predictor accuracy issue matters because the architecture's proposed mechanism is routing-based, but the Table 3 contradiction is the stronger, evidence-based concern. I agree with the reader that the paper makes useful observations about BLEU rewarding repetitive output and describes a plausible architecture, but the reported evidence does not establish the headline benefit. The CONDITIONAL verdict remains appropriate because the issues are addressable: the authors could clarify the checkpoint selection, provide predictor accuracy, and rerun the distinctiveness analysis with oracle routing. I did not find a separate flaw in the multi-task objective or the attention mechanism that would change the verdict.","tokens_in":9693,"tokens_out":10810,"duration_ms":119895,"concrete_test":"Ask the authors to recompute the Table 3 distinct-sentence counts at the exact checkpoints used for the Table 2 test-set scores and to report the abnormal sentence predictor's accuracy, precision, recall, and confusion matrix on the held-out test set against the manual labels. If HLSTM+att+Dual still has lower distinct counts than HLSTM+att at positions m>=1, or if the predictor is near the majority baseline, the claimed distinctiveness and routing benefits are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sections 5.3 and 6) is that the attention-augmented dual word LSTM increases distinctiveness without sacrificing relevance, and that the dual word LSTM increases the number of distinct sentences faster. However, Table 3, which reports the number of distinct sentences per sentence index for the selected models, shows HLSTM+att+Dual generating 5,10,7,8,8,4,1,0,0,0 distinct sentences for m=0..9 on the validation split, versus 5,24,24,33,25,31,23,14,9,3 for HLSTM+att and 8,28,36,45,32,17,2,0,0,0 for HLSTM+Dual. Thus the model for which the main claim is made has the lowest distinctiveness among the hierarchical baselines at nearly every sentence position. This directly contradicts the statement in Section 5.2 that 'far more distinct sentences are generated' and the conclusion's claim that the dual word LSTM 'can help to increase the number of distinct sentences faster.' The selection rule in Section 5.1 was applied to all models, so it does not explain the discrepancy. If 'faster' refers to training dynamics rather than final selected checkpoints, the paper does not make that distinction, and the reported scores in Tables 2 and 4 are for final selected models. Additionally, the abnormal sentence predictor tau_m (Section 4.2) that routes each sentence to the abnormal or normal word LSTM has no reported accuracy, F1, or confusion matrix on held-out data. Without it, the gains in Tables 2 and 4 cannot be attributed to correct abnormal/normal routing; a near-chance router would reduce the architecture to an uncontrolled mixture of two decoders.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the bias of chest X-ray report generation toward normal findings by proposing a hierarchical LSTM with dual word LSTMs, one for abnormal and one for normal sentences, plus an abnormal-sentence predictor that routes each sentence at inference time. The authors manually annotate sentences in the IU chest X-ray dataset as abnormal or normal, use MTI tags for multitask learning, and evaluate BLEU, CIDEr, METEOR, and ROUGE-L on validation and held-out test splits. They also analyze the trade-off between BLEU scores and the number of distinct generated sentences and propose a model-selection criterion that includes a minimum distinctness threshold.","tokens_in":9975,"tokens_out":3885,"duration_ms":38859,"significance":"The dual-decoder idea and the analysis of distinctiveness versus BLEU are potentially useful for medical report generation, where overly generic reports are a known failure mode. The paper contributes a sentence-level abnormality annotation of a public dataset and a concrete model-selection heuristic. If the central claims were supported, the architecture would be a simple and interesting step toward more clinically informative report variability. However, the current evidence does not establish the main claims: the paper's own distinctness table appears to contradict the distinctiveness claim, the routing predictor is not evaluated, and the quality improvements are reported without statistical significance.","major_comments":[{"comment":"The central claim that the dual word LSTM increases distinctiveness without sacrificing relevance is contradicted by Table 3 for the selected checkpoints. HLSTM+att+Dual generates 5, 10, 7, 8, 8, 4, 1, 0, 0, 0 distinct validation sentences for sentence indices m=0..9, whereas HLSTM+att generates 5, 24, 24, 33, 25, 31, 23, 14, 9, 3, and HLSTM+Dual generates 8, 28, 36, 45, 32, 17, 2, 0, 0, 0. Thus the model for which the main quality claim is made has the lowest distinctiveness among the hierarchical baselines at nearly every sentence position. This directly undermines the statement in Section 5.2 that 'far more distinct sentences are generated' and the conclusion in Section 6 that the dual word LSTM 'can help to increase the number of distinct sentences faster.' If the claim refers to training dynamics rather than the final selected checkpoints, that distinction is not made, and Tables 2 and 4 report scores for the final selected models only. The authors should report distinctiveness for the actual selected models, or provide a checkpoint-matched comparison with a BLEU-distinctiveness trade-off curve that supports the claim.","section":"Section 5.2, Table 3, Section 5.3, Section 6"},{"comment":"The abnormal sentence predictor τ_m that routes each sentence to the abnormal or normal word LSTM at inference is never evaluated. The paper does not report its held-out accuracy, F1, confusion matrix, or any qualitative example of routing behavior. Since the entire benefit of the dual-LSTM architecture depends on correct routing, the gains in Tables 2 and 4 cannot be attributed to the proposed separation of abnormal and normal generation. The training of the two word LSTMs uses ground-truth labels, so the inference-time routing is a train/test mismatch that must be validated directly. Please report routing performance on the test set and, ideally, an analysis of how routing errors affect generated report quality.","section":"Section 4.2"},{"comment":"No confidence intervals, error bars, or significance tests are provided for any of the reported metrics. The test set is only 5% of roughly 3,955 reports, so differences such as BLEU-4 of 12.6 versus 11.7 (HLSTM+att+Dual vs. HLSTM+att) or CIDEr of 49.5 versus 42.6 on the normal subset may be within run-to-run or test-sampling noise. The claim that 'HLSTM+att+Dual is consistently better than other baselines' therefore needs bootstrap confidence intervals or paired tests across multiple random seeds and checkpoint selections.","section":"Tables 2 and 4, Section 5.3"}],"minor_comments":[{"comment":"The solid and dashed vertical lines are described only in the text; the caption should define them explicitly. Also, the y-axis label '# distinct sentences' appears twice; one of them should be labeled to indicate the sentence index m.","section":"Figure 3"},{"comment":"The footnote that CoAtt and KERP results use a different dataset split is important, but the table lists them in the same column as directly evaluated models. Consider separating them with a different table or a clear horizontal rule so the comparison is not misleading.","section":"Table 2 note"},{"comment":"The manual annotation procedure is described in one sentence with no inter-annotator agreement statistic. Reporting Cohen's kappa or a similar measure would strengthen the claimed label quality.","section":"Section 4.2"},{"comment":"There are minor typographical issues, e.g., 'hierachical' in Section 4.1 and 'HARZIG ET AL.: DA TA BIAS' in the running header. These do not affect the science but should be corrected.","section":"Typos"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the venue and the distinctiveness analysis is a worthwhile contribution. However, the paper's own Table 3 undercuts the main claim, and the routing predictor is a black box. Both issues are fixable with additional experiments and careful rephrasing, so I do not recommend rejection, but the revision must address them directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper has a genuinely new architectural idea: split the hierarchical LSTM's word decoder into two LSTMs, one for abnormal and one for normal sentences, chosen by a learned sentence-level predictor. That is a sensible response to the normal-case bias in the IU chest X-ray data, and I haven't seen it in the cited work. The second contribution, an empirical look at how BLEU can be inflated by low-distinctness reports, is also worth something. They show a concrete case where a model with the highest BLEU-4 produces the same paragraph for every validation image. That is a useful caution for the field.\n\nWhat the paper does well: the experiments are on a held-out test split, they compare against their own hierarchical baselines plus cited numbers from CoAtt and KERP, and they make the evaluation protocol explicit (selection criterion, hyperparameters). Table 2's test scores are consistent with a modest quality improvement for the attention-augmented dual model. The manual abnormal/normal sentence annotation is a resource, though it is not released. The citation pattern is fine; the external comparisons are flagged as non-comparable due to different splits.\n\nThe soft spots. The biggest one is that the headline claim about distinctiveness is not actually demonstrated in the model they advertise. In Table 3, HLSTM+att+Dual has 5,10,7,8,8,4,1,0,0,0 distinct sentences by position, while HLSTM+att has 5,24,24,33,25,31,23,14,9,3. The training curves in Figure 3 may show more variability for the dual model during training, but the selected checkpoint—the one used for all reported scores—is the least distinct of the hierarchical variants. The selection rule (BLEU-4 subject to at least 4 distinct first sentences) explains why: it optimizes the metric their own analysis says rewards repetition. So the conclusion 'distinctness is increased' is at best true under a different stopping criterion than the one they used. That needs to be fixed or qualified.\n\nSecond, the abnormal-sentence predictor tau_m is load-bearing. During inference it routes every sentence to one of the two decoders, but there is no accuracy, F1, or confusion matrix for it anywhere. If it is near chance, the architecture is basically an uncontrolled mixture of two decoders and the test-score gains could be noise or something else. This is easy to report and should have been reported.\n\nThird, no error bars or significance tests. The margins in Table 2 are small and some comparisons are a point or less. Without variance estimates I can't tell which differences are real. The unreleased code and annotations also make the manual labels' details impossible to check.\n\nI don't think the Table 3 mismatch makes the paper incoherent. The architecture and the BLEU-versus-distinctness observation are real contributions, and the flaws are addressable. A serious referee should see it. I'd recommend conditional acceptance with a request to report the predictor accuracy, add variance estimates, and align the distinctiveness claim with the actual selection rule.","headline":"Useful architecture and a real metric critique, but the distinctiveness benefit is not demonstrated in the selected model and the routing predictor is unvalidated.","tokens_in":10555,"tokens_out":4044,"would_cite":true,"duration_ms":43284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routing each chest X-ray report sentence to a dedicated abnormal or normal word LSTM reduces normal-case bias and produces more distinct reports without sacrificing relevance.","keywords":["chest X-ray report generation","data bias","hierarchical LSTM","dual word LSTM","abnormal sentence prediction","sentence distinctiveness","BLEU evaluation","medical image captioning"],"falsifier":"Compute the abnormal-sentence predictor's accuracy on held-out sentences with clinician labels, then compare predicted routing against oracle routing that uses ground-truth labels: if oracle routing does not beat predicted routing, the routing module, not the dual decoders, carries the reported gains, and if predicted routing is near chance, the central benefit is not realized.","tokens_in":9445,"feed_emoji":"🩻","tokens_out":7793,"duration_ms":75476,"temperature":0.7,"pith_summary":"This paper argues that the normal-case bias in chest X-ray report generation can be reduced by splitting sentence generation into two decoders: one word LSTM trained on sentences describing abnormalities and one trained on normal findings. A hierarchical LSTM proposes each sentence's topic, an abnormal-sentence predictor decides which decoder should write it, and the final report is assembled from both. The authors report that this dual-decoder model generates a larger number of distinct sentences per paragraph position while keeping, and in several cases improving, BLEU, METEOR, ROUGE-L and CIDEr scores relative to single-decoder hierarchical baselines. They also show that BLEU alone is a misleading selection signal: a model can reach its highest BLEU score while emitting the same paragraph for every input. The practical point of the proposal is that automated reports should vary with the image rather than collapsing to the most frequent normal template.","feed_headline":"Two LSTM writers produce more varied chest X-ray reports","feed_subtitle":"Routing each sentence to an abnormal or normal decoder raises sentence diversity without sacrificing relevance.","key_machinery":"The load-bearing mechanism is a hierarchical LSTM with a dual word LSTM. A sentence LSTM produces a topic vector $t_m$ for each sentence from attended image features; a stop predictor decides when the paragraph ends; and two separate word LSTMs, one trained only on abnormal sentences and one only on normal sentences, write the actual words. A fully-connected abnormal-sentence predictor $\\tau_m$ reads the topic vector and, at inference, routes each sentence to the abnormal or normal word LSTM depending on whether its probability exceeds 0.5. The full objective combines the hierarchical captioning loss, the stop loss, the abnormality-prediction loss, and a multi-task loss for medical-tag (MTI) prediction. The pair of decoders is what lets the model move away from the most probable normal template while keeping the sentences grammatical.","core_discovery":"The central claim is that separating generation of abnormal and normal sentences, rather than training one shared word LSTM for all sentences, counteracts the data imbalance in chest X-ray reports. On the public chest X-ray report corpus, the model with attention and dual word LSTMs is consistently better than its single-decoder hierarchical baselines on multi-gram BLEU, METEOR and ROUGE-L, and best on CIDEr, while also increasing the number of distinct sentences at each sentence position within the paragraph. The paper states that this indicates 'the relevance is not sacrificed while distinctiveness is increased.' A second, related claim is that BLEU-style evaluation alone is inadequate for this setting: the paper documents a training checkpoint with a higher BLEU-4 score than its selected model even though the model generated the very same paragraph for every validation sample, and therefore proposes selecting models with a minimum-distinctiveness constraint alongside BLEU-4.","pith_inferences":["An implication the authors leave implicit is that the abnormal-sentence predictor is the operational gatekeeper: a clinical deployment would need its accuracy reported and calibrated, because a misrouted sentence would be written by a decoder trained on the wrong distribution.","The same routing idea transfers to other imbalanced text-generation settings where individual sentences carry the minority class, such as pathology or follow-up radiology notes, at the cost of per-sentence annotation.","A testable extension is to compare the dual-decoder model against a single decoder whose topic vector is concatenated with the predicted abnormality probability, isolating whether separate parameters or the routing signal drive the gain."],"forward_implications":["If the dual-decoder result holds, report generators for imbalanced clinical corpora can raise output diversity without sacrificing n-gram overlap with reference reports.","The documented case of peak BLEU-4 from a model that repeats one paragraph implies that BLEU-style rankings on this dataset should be paired with a distinctiveness measure.","On the held-out abnormal- and normal-only test slices, a dual model is best for both, so the diversity gain is not concentrated in one class.","The proposed checkpoint selection rule—at least four distinct first sentences on the validation set, then highest BLEU-4—is a reproducible recipe that the paper shows outperforms BLEU-only selection."],"supporting_citations":[{"why":"Supplies the chest X-ray images, doctors’ reports, MTI tags and MeSH labels that define the dataset and the abnormality annotation task.","marker":"[1]"},{"why":"Provides the hierarchical LSTM approach for medical report generation that the paper extends with attention and dual word LSTMs.","marker":"[3]"},{"why":"Introduces the hierarchical LSTM paragraph-generation structure (sentence LSTM plus word LSTM) on which the architecture is built.","marker":"[7]"},{"why":"Defines the BLEU metric used throughout the evaluation and the distinctiveness-versus-BLEU analysis.","marker":"[14]"},{"why":"Supplies the CIDEr metric, on which the dual model shows its largest relative improvement.","marker":"[16]"},{"why":"Provides the CNN-RNN image-captioning baseline whose scores are compared against the hierarchical models.","marker":"[17]"}],"fun_headline_variants":["Dual LSTM decoders boost chest X-ray report diversity","Separate abnormal, normal sentence writers improve report variety","Chest X-ray reports: splitting sentence generation raises diversity","Two decoders for abnormal vs normal produce more distinct reports","BLEU alone biased: dual-decoder model writes more varied X-ray reports"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The inference-time routing relies on the abnormal sentence predictor choosing the right decoder for each sentence, but the paper does not report that predictor's accuracy or the agreement of the human labels it was trained on; if routing is often wrong, the separate decoders cannot deliver their benefit.","fun_headline_variants_meta":{"raw":{"variants":["Dual LSTM decoders boost chest X-ray report diversity","Separate abnormal, normal sentence writers improve report variety","Chest X-ray reports: splitting sentence generation raises diversity","Two decoders for abnormal vs normal produce more distinct reports","BLEU alone biased: dual-decoder model writes more varied X-ray reports"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2866,"prompt_tokens":839,"completion_tokens":2027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1942}},"tokens_in":455,"tokens_out":2027,"duration_ms":15102,"temperature":1.0,"reasoning_tokens":1942,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:53:02.428407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the abnormal-sentence predictor's accuracy on held-out sentences with clinician labels, then compare predicted routing against oracle routing that uses ground-truth labels: if oracle routing does not beat predicted routing, the routing module, not the dual decoders, carries the reported gains, and if predicted routing is near chance, the central benefit is not realized.","supporting_citations":[{"cited_title":"Preparing a collection of radiology examinations for distribution and retrieval","cited_arxiv_id":null,"evidence_quote":"Supplies the chest X-ray images, doctors’ reports, MTI tags and MeSH labels that define the dataset and the abnormality annotation task."},{"cited_title":"On the automatic generation of medical imaging reports","cited_arxiv_id":null,"evidence_quote":"Provides the hierarchical LSTM approach for medical report generation that the paper extends with attention and dual word LSTMs."},{"cited_title":"A hierarchical ap- proach for generating descriptive image paragraphs","cited_arxiv_id":null,"evidence_quote":"Introduces the hierarchical LSTM paragraph-generation structure (sentence LSTM plus word LSTM) on which the architecture is built."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"Defines the BLEU metric used throughout the evaluation and the distinctiveness-versus-BLEU analysis."}],"review_version":1}