{"id":"a0b40c54-a32a-4b17-993a-f34a65b2bec1","arxiv_id":"2411.15413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FG-CXR aligns radiologist gaze with seven anatomical regions and reports; Gen-XAI uses this alignment to generate more accurate and interpretable chest X-ray reports.","lead":"Radiologist eye-tracking on chest X-rays is paired with per-anatomy report sentences in a new curated dataset, FG-CXR, and a model, Gen-XAI, is trained to predict both the gaze heatmaps and the reports. The authors report that this explicit gaze supervision improves both the accuracy and the interpretability of automated report generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Report ground truth is partly synthetic (Sec. 3.3): fill-in templates from MIMIC-CXR labels mean the claimed accuracy gains may reflect template learning, not gaze.","rationale":"The reader identified the temporal alignment and mask accuracy as the weakest assumption. I agree those matter, but the more load-bearing issue is the synthetic report supervision in Section 3.3. The central claim that gaze-conditioned generation yields more accurate reports rests on NLG/CE scores computed against reference reports that are, for many anatomies, auto-generated from MIMIC-CXR labels. If those templates dominate, the strong BLEU/CIDEr and clinical efficacy results may simply indicate that the model learns the template distribution; the explicit anatomical intention tokens make this easier. The internal ablation (Table 9) cannot settle this because all variants share the same synthetic targets. A targeted test on original-text-only reports would disambiguate. This does not invalidate the dataset's potential value, but it means the headline 'outperforms' claim is conditional on the synthetic labels being a valid proxy for radiologist diagnoses.","tokens_in":13821,"tokens_out":4926,"duration_ms":44963,"concrete_test":"Compute the fraction of region-report pairs in FG-CXR that were generated by the Section 3.3 template (i.e., regions absent from the original EGD/REFLACX report). Then re-train or re-evaluate Gen-XAI and the strongest baseline (e.g., M2 Transformer) using only test instances where the reference report is original radiologist text, and separately on only synthetic-template instances. If the performance gap (e.g., BLEU-4, F1ex, fwIoU) narrows or disappears on the original-text subset, the claimed advantage is an artifact of the synthetic template distribution rather than gaze alignment.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that gaze-conditioned generation produces more accurate and interpretable reports than black-box models. But the report supervision in FG-CXR is not fully radiologist-authored. Section 3.3 states that for any anatomy not mentioned in the original REFLACX/EGD report, the authors generate a template sentence using MIMIC-CXR labels: 'the {area} is possibly normal' or 'the patient is possibly suffering from {findings} in the {area}'. Thus an unknown fraction of the 20,657 region-report pairs are synthetic, derived from automated labels. The NLG and CE metrics are computed against these synthetic references. A model trained on this data can memorize the template patterns, and the proposed decoder explicitly conditions on anatomical intention tokens that align with the template structure, potentially inflating BLEU/CIDEr and CE scores relative to baselines. The paper does not report the proportion of synthetic sentences, nor does it validate the MIMIC-CXR labels for the specific EGD/REFLACX images. Therefore the head-to-head improvement over baselines is confounded: it may reflect the synthetic report distribution, not the value of radiologist gaze. The internal ablation (Table 9) does not address this, since all settings are trained and evaluated on the same synthetic ground truth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FG-CXR, a curated dataset that aligns radiologist eye-tracking data from EGD and REFLACX with per-anatomy reports and attention heatmaps for seven chest X-ray regions. It also proposes Gen-XAI, a three-module system consisting of a gaze attention predictor built on BiomedCLIP, a spatial-aware attended encoder, and a GPT-2 decoder, trained with heatmap and report losses plus dynamic penalty coefficients. Experiments on a 70/10/20 split of 2,951 images report state-of-the-art results on natural language generation, clinical efficacy, and attention similarity metrics. The dataset and a checkpoint are publicly released.","tokens_in":14076,"tokens_out":4692,"duration_ms":41906,"significance":"If the curation pipeline is sound, FG-CXR is a valuable resource for gaze-interpretable report generation and for benchmarking anatomical segmentation and report generation in chest X-ray analysis, and Gen-XAI is a reasonable first architecture for this new task. The paper ships a public dataset and checkpoint, and the ablations in Tables 7-9 are informative, particularly the comparison between pixel-level and feature-level attention and the penalty-term study. However, the synthetic template reports and the uncontrolled attention comparison currently leave the central claims of improved accuracy and interpretability conditional on additional validation.","major_comments":[{"comment":"The report ground truth is partly synthetic. Section 3.3 states that for any anatomy not mentioned in the original REFLACX or EGD report, the authors generate a template sentence from MIMIC-CXR labels, e.g., 'the {area} is possibly normal' or 'the patient is possibly suffering from {findings} in the {area}'. The paper does not report the proportion of the 20,657 region-report pairs that are synthetic, nor does it validate the MIMIC-CXR labels for the specific EGD/REFLACX images. Since the proposed decoder is explicitly conditioned on anatomical intention tokens that mirror the template structure, the NLG and CE gains in Tables 4 and 5 may reflect template memorization rather than the value of radiologist gaze. This is load-bearing for the central claim of more accurate report generation, and the internal ablation in Table 9 does not address it because all settings are trained and evaluated on the same synthetic ground truth.","section":"Section 3.3, Tables 4-5"},{"comment":"The attention similarity comparison is not controlled. Gen-XAI is explicitly trained to minimize an L2 loss against radiologist gaze heatmaps, while the baseline models are never trained to predict gaze and are evaluated on gaze metrics using their internal attention scores or bounding boxes. The large fwIoU advantage (+20.47 over the runner-up) is therefore structurally expected and does not by itself establish that Gen-XAI is more interpretable. The claim that the method 'outperforms other methods across all criteria' conflates being trained to match gaze with being more interpretable. I recommend either training baselines with the same gaze supervision, or evaluating interpretability through a radiologist study or a task-based evaluation that does not use the training objective as the metric.","section":"Section 5.2, Table 6"},{"comment":"The temporal alignment assumption is not validated. The curation pipeline removes all gaze points after the latest end time of sentences containing anatomy-related keywords, which assumes that a radiologist finishes examining a region before dictating the corresponding sentence and never returns to that region. It also assumes that the automatically generated anatomical masks from [34] are accurate, since gaze points falling outside a mask are discarded. If either assumption fails, the resulting 'fine-grained' gaze heatmaps are noisy, and the reported gains from gaze attention could be artifacts of the curation pipeline. The paper should provide a sensitivity analysis or validate the alignment on a subset of cases, for example by comparing the automatically aligned heatmaps with manually aligned ones.","section":"Section 3.2"},{"comment":"All quantitative conclusions rest on a single random 70/10/20 split of 582 test images with point estimates and no confidence intervals or multiple-seed results. Given the relatively small test set and the modest margins on some clinical efficacy metrics (e.g., F1ex 0.497 vs. 0.440 in Table 5), the paper should report standard deviations across at least three runs or use a statistical significance test to show that the improvements are robust. Without this, the headline comparisons in Tables 4-6 are not yet fully supported.","section":"Section 3.4, Section 5.2"}],"minor_comments":[{"comment":"The notation '10.5' in Eq. (1) is confusing; replace it with the indicator function \\mathbb{1} and clarify that the threshold is 0.5. Also, in the sentence 'we want the model the model explicitly predict directions' there is a duplicated phrase 'the model'.","section":"Section 4.4, Eq. (1)"},{"comment":"The metric R@4 is reported in Table 4 but is not defined in the metrics list in Section 5.1; please provide a definition or citation. Similarly, the Div@2 definition is cited but not stated.","section":"Section 5.1, Table 4"},{"comment":"The paper says GPT2 is initialized with DistillGPT2 and cites [39] (Sanh et al.), but reference [39] is the DistilBERT paper; please cite the correct DistilGPT2 source.","section":"Section 5.1, Section 4.3"},{"comment":"The last column of Table 1, 'Method', contains entries such as 'Semi-automated' and 'Automated' that appear to describe how annotations were created, but the column header is unclear. Please rename the column to 'Annotation Method' and ensure the row entries are aligned with the correct columns.","section":"Table 1"},{"comment":"The paper filters out images with 'extreme brightness' but does not define the threshold; since this is a free parameter in the curation pipeline, please state the criterion or cite the procedure.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the dataset release is a tangible contribution. The main risk is that the synthetic template reports in Section 3.3, combined with the uncontrolled attention comparison, could lead to overclaiming. I recommend major revision rather than rejection because the central idea is defensible and the issues are addressable with additional experiments and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the FG-CXR dataset is a genuinely useful construction: per-anatomy gaze-heatmap/report pairs curated from EGD and REFLACX. The raw datasets carry gaze and text but not aligned at the region level, so this fills a real gap. Second, the central claim — that gaze supervision makes report generation both more accurate and more interpretable — is only half-supported. The interpretability half holds; the accuracy half is confounded.\n\nThe cleanest evidence is the Table 9 ablation: gaze attention beats anatomical segmentation masks and traditional attention on identical ground truth. That is a fair internal comparison, and it suggests the radiologist gaze signal carries information beyond which anatomy is being examined. The paper also deserves credit for stating plainly that its large attention-metric lead over baselines is structurally expected, since baselines are not trained to predict gaze.\n\nThe load-bearing weakness is Section 3.3. For any anatomy not mentioned in the original report, the authors synthesize a template sentence from MIMIC-CXR labels: 'the {area} is possibly normal' or 'the patient is possibly suffering from {findings} in the {area}'. The paper never reports what fraction of the 20,657 region-report pairs are template-generated, and it does not validate the MIMIC-CXR labels against these specific EGD/REFLACX images. The decoder is explicitly structured per region with anatomical intention tokens, which matches the template structure. So the BLEU/CIDEr/CE gains over whole-report baselines may partly reflect learning the template distribution, not the value of gaze. The stress-test note pointing at this is right. Separately, Section 3.5 claims the reports are 'validated by experts,' which sits uneasily next to the automated template filling.\n\nTwo smaller soft spots. The temporal alignment in Section 3.2 assumes the radiologist never returns to a region after dictating its sentence; reasonable heuristic, but it adds noise to the fine-grained supervision. And the evaluation uses a single random split with no error bars, so we cannot tell whether the smaller differences are meaningful. The brightness filter threshold is also unspecified.\n\nWho this is for: people building interpretable report generation or gaze-conditioned medical image models. The dataset is the contribution; Gen-XAI is a serviceable baseline rather than the main event.\n\nRecommendation: send it to review. A serious referee should ask for the synthetic fraction, a sanity check on the fully human-authored subset, and error bars. Desk rejection would be wrong; acceptance without those fixes would also be wrong. This is a conditional-accept trajectory.","headline":"A useful per-region gaze-report dataset, but the accuracy gains over baselines are confounded by template-generated report ground truth.","tokens_in":14620,"tokens_out":5199,"would_cite":true,"duration_ms":44236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces FG-CXR, a chest X-ray gaze dataset with per-anatomy alignment between report sentences and radiologist attention heatmaps, and Gen-XAI, a generator that it reports surpasses all baselines on language, clinical, and…","keywords":["chest X-ray report generation","gaze attention","interpretability","eye-tracking dataset","radiology report generation","anatomical heatmaps","explainable AI","medical imaging"],"falsifier":"Re-record eye tracking and transcripts under a protocol that marks each anatomy's examination interval explicitly, or have a radiologist manually segment existing gaze streams in a random subset; if a model trained on those manually aligned heatmaps does not reproduce Gen-XAI's margin over baselines, the temporal trimming in Section 3.2 is the source of the reported advantage.","tokens_in":13629,"feed_emoji":"🫁","tokens_out":7791,"duration_ms":70481,"temperature":0.7,"pith_summary":"This paper is trying to establish that radiologists' eye movements can be turned into structured supervision that makes automated chest X-ray report generation both more accurate and more interpretable. It introduces FG-CXR, a curated dataset of 2,951 chest X-rays with 20,657 anatomy-level pairs of gaze attention heatmaps and report sentences, aligned by using sentence timing to trim gaze streams and anatomical masks to discard off-target gaze. It then proposes Gen-XAI, a generator with a gaze attention predictor that outputs seven anatomy heatmaps and a decoder that conditions on those heatmaps. In experiments on FG-CXR, Gen-XAI reportedly outperforms five strong captioning baselines on all natural-language, clinical-efficacy, and attention-similarity metrics. If correct, the paper shows that explicit gaze supervision can replace implicit attention in radiology report generation, giving clinicians both a better report and a visual record of what the model looked at.","feed_headline":"Gaze-guided model beats black-box baselines on chest X-ray reports","feed_subtitle":"A new dataset ties each report sentence to the anatomy the radiologist was viewing; a gaze-conditioned model wins on every metric.","key_machinery":"The load-bearing mechanism is the anatomical-aware gaze attention pipeline used to build FG-CXR and the Gaze Attention Predictor (GAP) used to train on it. The curation pipeline takes a raw eye-tracking stream plus a report, finds the latest end time of sentences mentioning each anatomy via keyword matching, removes gaze points after that time, then drops points outside the automatically generated anatomical mask; the surviving points are rendered into a Gaussian-blurred heatmap. During training, the GAP predicts seven such heatmaps from the image and seven anatomical intention tokens, a spatial-aware attended encoder multiplies latent visual features by the predicted maps, and an autoregressive report decoder cross-attends to the reweighted features. This forces the model to ground each generated sentence in a region a radiologist actually looked at, and the two dynamic penalty terms push it to look at the correct anatomy and mention the correct direction.","core_discovery":"The central claim, stated on the paper's own terms, is that a report-generation model explicitly constrained to predict and attend to radiologist gaze heatmaps produces reports that are linguistically closer to radiologist transcripts and clinically more correct than black-box captioning models, while remaining interpretable because its attention is anchored to expert behavior. Anchoring this claim is the FG-CXR dataset, which re-pairs gaze with reports at the level of individual anatomies: each of seven regions (heart, left, right, upper and lower left and right lungs) has its own gaze heatmap and its own sentence or sentences. On this benchmark, the paper reports that Gen-XAI surpasses all compared methods on every reported metric, with its largest margin in attention similarity. The paper interprets this as direct evidence that mimicking radiologists' viewing focus is better than letting the model discover its own attention for both accuracy and explainability.","pith_inferences":["Inference: A direct extension would be to apply the same sentence-timing and mask-filtering curation to other eye-tracking corpora, producing per-anatomy heatmaps without new gaze recordings; the value of that extension depends entirely on the revisit assumption named below.","Inference: The architecture suggests a testable variant where predicted heatmaps are optimized jointly with the report through a differentiable attention-similarity reward rather than fixed penalty coefficients, which could tighten the coupling between gaze and text.","Inference: Because missing anatomies are filled with templated sentences from external label sets, the report distribution of FG-CXR is partly synthetic; comparing against a subset containing only original dictated sentences would clarify how much of the reported report-quality gain comes from gaze supervision versus the templates."],"forward_implications":["Radiologist gaze can serve as a direct supervision signal for report generation, not just an evaluation tool, so future systems can be trained to look where experts look.","Per-anatomy heatmaps give clinicians a visual audit trail: each generated sentence is tied to a region the model was looking at, which makes errors in reports traceable to errors in attention.","The FG-CXR benchmark allows interpretability to be quantified as agreement with expert gaze, alongside traditional language and clinical metrics.","The model's gaze attention module can be repurposed for anatomical segmentation, since the same predictor trained on segmentation masks performs competitively."],"supporting_citations":[{"why":"Supplies the anatomical mask generation and the pretrained-encoder gaze predictor design that FG-CXR and GAP build on.","marker":"[34]"},{"why":"Source of the chest X-ray images and raw gaze streams used to derive FG-CXR's heart and lung heatmaps.","marker":"[17]"},{"why":"Source of reports and eye-tracking data that FG-CXR re-aligns per anatomy.","marker":"[1]"},{"why":"Region-guided interpretable report-generation baseline that Gen-XAI is compared against and outperforms.","marker":"[44]"},{"why":"Baseline report-generation model whose natural-language and clinical metrics anchor the comparison for the central claim.","marker":"[4]"},{"why":"Cross-modal memory report generator baseline that Gen-XAI surpasses on language and clinical metrics.","marker":"[3]"},{"why":"Transformer captioning baseline that is the runner-up on several metrics and a key comparator for attention similarity.","marker":"[6]"},{"why":"Pretrained vision-language encoder that supplies visual and text features for the gaze attention predictor.","marker":"[54]"},{"why":"Convolutional vision transformer that produces the latent features the predicted heatmaps reweight.","marker":"[50]"},{"why":"Autoregressive text decoder that generates the report conditioned on spatial-aware attended features.","marker":"[36]"}],"fun_headline_variants":["Gaze-locked chest X-ray model sees like a radiologist","New dataset pairs each CXR sentence with the anatomy radiologists view","Gen-XAI beats black-box captioning by mimicking radiologist gaze","FG-CXR aligns gaze and report text at the region level","Chest X-ray reports gain interpretability from gaze-conditioned generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a radiologist never returns to an anatomy after dictating the sentence about it, so trimming each gaze stream at the latest sentence end produces clean per-anatomy attention; if radiologists do circle back, the dataset's gaze labels are noisy and the reported gains may come from the curation pipeline rather than from gaze itself.","fun_headline_variants_meta":{"raw":{"variants":["Gaze-locked chest X-ray model sees like a radiologist","New dataset pairs each CXR sentence with the anatomy radiologists view","Gen-XAI beats black-box captioning by mimicking radiologist gaze","FG-CXR aligns gaze and report text at the region level","Chest X-ray reports gain interpretability from gaze-conditioned generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1294,"prompt_tokens":997,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":207}},"tokens_in":613,"tokens_out":297,"duration_ms":3108,"temperature":1.0,"reasoning_tokens":207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:19:29.591883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-record eye tracking and transcripts under a protocol that marks each anatomy's examination interval explicitly, or have a radiologist manually segment existing gaze streams in a random subset; if a model trained on those manually aligned heatmaps does not reproduce Gen-XAI's margin over baselines, the temporal trimming in Section 3.2 is the source of the reported advantage.","supporting_citations":[{"cited_title":"In: WACV (2024) 2, 3, 5, 8, 9, 10, 11","cited_arxiv_id":null,"evidence_quote":"Supplies the anatomical mask generation and the pretrained-encoder gaze predictor design that FG-CXR and GAP build on."},{"cited_title":"PhysioNet (2020) 2, 4, 5, 7 16 Pham et al","cited_arxiv_id":null,"evidence_quote":"Source of the chest X-ray images and raw gaze streams used to derive FG-CXR's heart and lung heatmaps."},{"cited_title":"Scientific data (2022) 2, 4, 5, 7","cited_arxiv_id":null,"evidence_quote":"Source of reports and eye-tracking data that FG-CXR re-aligns per anatomy."},{"cited_title":"In: CVPR (2023) 10, 11","cited_arxiv_id":null,"evidence_quote":"Region-guided interpretable report-generation baseline that Gen-XAI is compared against and outperforms."},{"cited_title":"In: CVPR (2020) 10, 11, 12","cited_arxiv_id":null,"evidence_quote":"Transformer captioning baseline that is the runner-up on several metrics and a key comparator for attention similarity."},{"cited_title":"In: ICCV (2021) 9, 10","cited_arxiv_id":null,"evidence_quote":"Convolutional vision transformer that produces the latent features the predicted heatmaps reweight."},{"cited_title":"OpenAI blog (2019) 8, 9, 10","cited_arxiv_id":null,"evidence_quote":"Autoregressive text decoder that generates the report conditioned on spatial-aware attended features."}],"review_version":1}