{"id":"19b04d15-bde8-4591-8cdf-8cf98f63c999","arxiv_id":"2508.13068","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A gaze-supervised multimodal chest X-ray classifier and LLM-based report generator that reports accuracy gains partly caused by fine-tuning rather than gaze.","lead":"This paper combines radiologist eye-tracking with contrastive learning to classify chest X-rays and generate region-aware radiology reports. It reports modest classification gains, but the headline numbers mix in a fine-tuning step, and the report-generation gains lack a baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline classification gain is confounded: abstract's F1/AUC improvement compares transcription baseline (0.597/0.822) to a gaze model fine-tuned on 493 validation samples (0.631/0.849); isolated enhanced-gaze model shows no gain (0.597/0.823).","rationale":"The paper's method is described in detail and the ablation table is thoughtfully structured, and I take the reported numbers as genuine. The load-bearing problem is that the abstract and the Section 4.2 discussion select two columns that differ by more than one variable. The enhanced gaze-attention loss, in isolation, shows no F1 improvement and only +0.001 AUC over the transcription baseline (Table 1). Thus the central claim 'gaze improves classification' depends on the fine-tuning pass, not on gaze. This is an internal-validity error, not a disagreement with external consensus. The Pearson-threshold inconsistency is a second, smaller internal error: r=0.237 is reported as meeting r≥0.30, and the uncertainty interval makes the alignment evidence weak; this undercuts the interpretability sub-claim independently. I considered the reader's weakest_assumption about YOLO-completed REFLACX boxes; it is a real and self-acknowledged limitation, but it affects the region-grounding stage and would not rescue the classification claim even if the boxes were perfect. The report-generation comparison (Table 3) also lacks a no-grounding baseline, but that is secondary to the classification confound. On balance the central claims as stated are not supported, so I keep the reader's REJECT verdict.","tokens_in":22046,"tokens_out":5221,"duration_ms":46555,"concrete_test":"Re-run the Table 1 ablation with a matched protocol: take the image+bbox+transcription baseline and the +Fixation Enhanced model, and either apply the same fine-tuning pass on the 493 validation samples to both models or to neither, then evaluate both on the identical 400-case test set. If the fine-tuned enhanced-gaze model does not exceed the fine-tuned transcription baseline by roughly the claimed 0.027 AUC and 0.034 F1, the abstract's attribution of the gains to gaze supervision is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 has five columns: baseline (images+labels+bbox), +Transcription, +Fixations, +Fixation Enhanced, and +Fine Tune (Val→Train, n=493). The abstract attributes 'Incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%)' by pairing the +Transcription column with the +Fine Tune column. Those two columns differ by two interventions: the proposed gaze-attention loss and an extra fine-tuning pass on the 493-case validation split merged into training. The +Fixation Enhanced column, which isolates the contribution of the proposed multi-term gaze-attention loss, reports F1 0.597 and AUC 0.823, essentially identical to the +Transcription baseline (F1 0.597, AUC 0.822). Therefore the abstract's central quantitative claim is not attributable to gaze supervision; it is attributable to the fine-tuning protocol. The same issue undercuts the paper's conclusion that gaze improves classification and interpretability together. Separately, Section 4.2 says Pearson r = 0.237 ± 0.191 'meets Cohen's moderate threshold (r ≥ 0.30)', which is internally inconsistent: 0.237 is below 0.30, and the ±0.191 interval extends below 0.05. The self-acknowledged YOLO bounding-box completion for REFLACX (Section 4.1 and Appendix A.2) is a real additional risk for the report-generation stage's region grounding, but the classification claim is already unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage multimodal framework for chest X-ray classification and report generation using the MIMIC-Eye dataset. Stage 1 fuses image, bounding-box, text, and gaze modalities through contrastive learning with a newly proposed multi-term gaze-attention loss combining MSE, KL divergence, correlation, and center-of-mass alignment. Stage 2 extracts diagnostic keywords, maps them to a 17-region anatomical dictionary, and prompts LLMs to generate region-aligned reports. The authors claim that incorporating fixations improves classification F1 from 0.597 to 0.631 and AUC from 0.821 to 0.849, and that the pipeline improves report quality as measured by clinical keyword recall and ROUGE overlap.","tokens_in":22443,"tokens_out":5466,"duration_ms":51420,"significance":"If the claims were supported, gaze-supervised contrastive learning with interpretable attention maps would be a useful contribution to medical multimodal learning, and a fully public-data pipeline with intermediate outputs and LLM comparisons would aid reproducibility. The manuscript has genuine strengths: it uses publicly available data, compares five LLMs, documents hyperparameters and preprocessing in appendices, acknowledges its own limitations, and frames the gaze-attention loss as a testable design. However, the headline classification improvement is confounded in Table 1, and the report-generation evaluation is circular because the prompt injects keywords extracted from the reference reports and then measures lexical overlap against those same references. These are load-bearing issues for both central claims, so the significance as presented is not established.","major_comments":[{"comment":"The abstract's headline improvement (F1 0.597 to 0.631, AUC 0.821 to 0.849) pairs the '+Transcription' column with the '+Fine Tune' column. These two columns differ by two interventions: the proposed multi-term gaze-attention loss and the addition of 493 validation samples to training. The '+Fixation Enhanced' column, which isolates the proposed gaze-attention loss, reports F1 0.597 and AUC 0.823, essentially identical to the '+Transcription' baseline (F1 0.597, AUC 0.822). The claim that incorporating fixations improves classification is therefore not supported by Table 1; the gain is attributable to the fine-tuning protocol. The authors should compare models under an identical training data and protocol, or revise the abstract and conclusions to report the isolated effect of gaze supervision.","section":"Section 4.2, Table 1"},{"comment":"The statement that Pearson r = 0.237 ± 0.191 \"meets Cohen's moderate threshold (r ≥ 0.30)\" is internally inconsistent: 0.237 is below 0.30, and the ±0.191 interval extends below 0.05. This overstates the human-model alignment evidence and should be corrected, along with the related claim in the Conclusion that Pearson ≈ 0.25 indicates improved alignment.","section":"Section 4.2"},{"comment":"The report-generation evaluation is circular. Section 3.5 extracts diagnostic keywords from the reference reports using Gemini; the prompt in Appendix A.6.3.1 instructs the LLM to use exact phrases from CLINICAL KEYWORDS verbatim and to maximize lexical similarity to the expert reference; Table 3 then scores the generated reports against the same reference reports. Lexical-overlap metrics such as ROUGE, BERTScore, and CheXpert F1 therefore measure prompt compliance with reference-derived vocabulary rather than the model's ability to generate a faithful report from the image. The pipeline should be evaluated with keyword extraction performed on data that is not the evaluation reference, or with a prompt that does not inject reference-derived terms.","section":"Section 3.5, Appendix A.6.3.1, Table 3"},{"comment":"The YOLO-completed bounding boxes for REFLACX are load-bearing for the report-generation stage: they supervise the gaze-attention maps and define the anatomical regions used to ground generated reports. The manuscript acknowledges that these synthetic boxes may bias spatial supervision, but no sensitivity analysis is provided. At minimum, the authors should report the quality of the YOLO completion (e.g., detection accuracy against held-out EyeGaze boxes) and assess whether the report-generation and attention-alignment results change when the completion is degraded or omitted.","section":"Section 4.1, Appendix A.2"}],"minor_comments":[{"comment":"The partition sizes 1,984/493/400 are labeled 80.1%/19.9%/16.2%; these percentages do not sum to 100 and do not match the stated total of 2,877 aligned samples. Please clarify the denominators.","section":"Section 4.1"},{"comment":"The table caption says \"Signal classification performance\"; this appears to be a typo for \"Multimodal classification performance\" or similar.","section":"Table 1"},{"comment":"RadGraph-F1 values around 0.13 are reported as differences between models; without significance tests or confidence intervals on the comparison, the claim that one model \"attains the highest RadGraph-F1\" is not supported.","section":"Table 3"},{"comment":"The appendix reports several different corpus sizes (3,192 patients, 3,689 studies, 2,877 aligned samples, 63 full-multimodal records); a flow diagram reconciling these counts would help reproducibility.","section":"Appendix A.1"},{"comment":"The p-value row is reported as mean ± standard deviation, which is not a meaningful summary for p-values; report the test statistic and confidence interval instead.","section":"Table 1, p-value row"}],"recommendation":"reject","confidential_remarks":"The central classification claim is confounded by the fine-tuning intervention in Table 1, and the report-generation evaluation is circular because the LLM is prompted with reference-derived keywords and then scored against those same references. These are not merely presentation issues; they require new experiments and a reframing of the contribution. The authors may be able to resubmit a substantially revised version that isolates the gaze-loss effect and removes reference-keyword injection from the generation evaluation, but as submitted the evidence does not support the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper as a well-organized study whose headline claim collapses on inspection. The method—a four-term gaze-attention loss combining MSE, KL, correlation, and center-of-mass, with quality weighting—is new, and the application to both EyeGaze and REFLACX in MIMIC-Eye is a genuine addition. The data curation is thorough: fixation normalization, pupil scaling, bounding-box completion, and honest reporting of limitations. The authors clearly know the data and the literature.\n\nBut the abstract says incorporating fixations improves F1 from 0.597 to 0.631 and AUC from 0.821 to 0.849. Table 1 shows those numbers come from the '+ Fine Tune' column, where 493 validation samples are folded into training. The '+ Fixation Enhanced' column, which isolates the proposed loss, sits at F1 0.597 and AUC 0.823—identical to the transcription baseline. So the reported gain is fine-tuning, not gaze supervision. That is a load-bearing error, and it repeats in the introduction and conclusion.\n\nThere is also an internal statistical mistake: Section 4.2 says Pearson r = 0.237 ± 0.191 meets Cohen's moderate threshold of r ≥ 0.30. It doesn't—0.237 is below 0.30, and the interval extends below 0.05. That's not a rounding issue.\n\nThe report-generation stage has a different but related weakness: the five LLMs are compared only within the proposed grounding pipeline, with no baseline that skips keyword extraction or anatomical mapping. Since keywords are extracted from the reference reports and the prompt explicitly asks the model to use them verbatim to maximize lexical overlap, the ROUGE improvements are partly engineered into the setup. The YOLO-completed bounding boxes for REFLACX are a real worry for region grounding, but the classification claim is already unsupported without needing that argument.\n\nWhat's good here: the pipeline is modular, the preprocessing details are useful, and the multi-term gaze loss is a reasonable idea that deserves a clean test. The authors are transparent about many limitations. This is not a sloppy submission in the usual sense—it's a paper where the main experiment's control groups were not aligned.\n\nMy recommendation: send it to peer review, but the reviewers should require a corrected comparison with fine-tuning held constant, a fix to the Pearson interpretation, and an ablation for the report-generation pipeline. The work has enough substance to be salvageable, but it is not publishable as written.","headline":"The paper's central claim—that gaze supervision improves classification—is unsupported because the abstract's numbers come from a column that also fine-tunes on validation data; the isolated gaze loss shows no gain.","tokens_in":23009,"tokens_out":3066,"would_cite":false,"duration_ms":32822,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding radiologist eye-tracking as a training signal improves chest X-ray classification (F1 0.597 to 0.631, AUC 0.821 to 0.849) and produces attention maps aligned with expert gaze.","keywords":["chest X-ray","gaze supervision","eye-tracking","contrastive learning","radiology report generation","multi-label classification","attention alignment","MIMIC-Eye"],"falsifier":"Run the pipeline twice on the REFLACX half of the corpus, once with YOLO-completed bounding boxes and once with radiologist-drawn abnormality ellipses or manually corrected boxes replacing them, then compare macro F1, AUC, and attention-alignment metrics. If the synthetic boxes are noise-free, results should stay roughly equal; a systematic disagreement between the two runs, or a large measured mismatch between YOLO boxes and human annotations, would show the load-bearing assumption does not hold.","tokens_in":21852,"feed_emoji":"🩻","tokens_out":7866,"duration_ms":77142,"temperature":0.7,"pith_summary":"The paper argues that radiologist eye-tracking, used as a training-time spatial signal, makes a chest X-ray classifier both more accurate and more interpretable, and that the same predicted regions can anchor automatic report generation. Using a curated subset of the MIMIC-Eye dataset, it reports that adding gaze supervision raises macro F1 from 0.597 to 0.631 and AUC from 0.821 to 0.849, with attention maps that correlate more closely with radiologist fixations. The report-generation half converts classifier outputs into confidence-weighted keywords, maps them to 17 canonical anatomical regions, and asks an LLM to write region-aware findings and impression sections. If these results hold, gaze data would offer a practical route to explainable and clinically grounded radiology AI without dense pixel-level annotations.","feed_headline":"Gaze supervision lifts chest X-ray diagnosis scores","feed_subtitle":"Radiologists' eye-tracking improved F1 to 0.631 and aligned AI attention maps with expert fixations.","key_machinery":"The load-bearing mechanism is the gaze-attention loss and the contrastive alignment that carries it. The loss combines four terms on predicted versus radiologist attention maps: MSE for pixel-level fidelity, KL divergence for distribution shape, a Pearson correlation term for pattern similarity, and a normalized center-of-mass term for geometric centering, with a per-sample quality weight that scales with fixation count and quality. An InfoNCE objective simultaneously aligns image-text and gaze-text embeddings, and a lightweight decoder predicts attention at three spatial scales. The second-stage machinery is a curated dictionary of 17 thoracic regions with alias lists and a condition-to-region clinical matrix, plus an LLM prompt that turns confidence-weighted keywords and region flags into findings and impression text.","core_discovery":"The paper's central claim is that human gaze can serve as an effective spatial teacher for a multimodal disease classifier, not merely as an extra input feature. It proposes a two-stage framework: a gaze-guided contrastive learner that fuses image, bounding-box mask, text, and fixation embeddings with an InfoNCE objective and a novel multi-term gaze-attention loss, followed by a region-grounded report generator that maps predicted keywords to anatomical regions and prompts an LLM. It claims that supervised attention maps align with radiologist fixations (for example, Jensen-Shannon divergence 0.451 versus an inter-reader upper bound near 0.45) while classification improves, and that the generated reports reach BERTScore 0.732 and CheXpert-F1 0.546, with phrase-level factual grounding still limited at RadGraph-F1 0.129. The authors state that test-time inference uses only the image, so gaze is purely a training signal.","pith_inferences":["Beyond the paper, the gaze-text InfoNCE loss suggests a way to distill eye-tracking into the text encoder, so models trained with gaze could inherit spatial priors even when deployed without an eye-tracker.","Beyond the paper, the claimed attention maps could serve as a filter for LLM output: only report sentences whose named anatomical region appears in the activated region mask would be kept, directly attacking the RadGraph-F1 gap the paper reports.","Beyond the paper, the same four-term gaze loss is modality-agnostic and could be applied to other eye-tracking corpora, such as CT or ultrasound, since none of its terms depends on chest-radiograph structure.","Beyond the paper, the ablation results leave open whether the classification gain comes from the spatial attention loss or from simply having more gaze-derived embeddings; a controlled study separating those two contributions would clarify the mechanism."],"forward_implications":["A chest X-ray classifier can be trained to use radiologist gaze as a weak spatial label and then run image-only at test time, so gaze collection is not needed at deployment.","The same attention maps can be scored against radiologist fixations, giving a quantitative, human-interpretable check on where the model is looking.","A keyword-condition dictionary with 17 anatomical regions plus a structured LLM prompt can generate reports whose semantics match reference reports while still falling short on fine-grained factual relations.","Merging just 493 validation studies into training restores and improves performance, suggesting modest amounts of human annotation can sharpen a gaze-supervised model.","Because all training signals come from the public MIMIC-Eye data and intermediate predictions are retained, the pipeline offers a reproducible testbed for gaze-aware radiology AI."],"supporting_citations":[{"why":"Supplies the InfoNCE contrastive objective used for image-gaze and image-text alignment.","marker":"van den Oord et al., 2019"},{"why":"Provides CheXpert labels, transfer-learning initialization, and the CheXpert-F1 report metric.","marker":"Irvin et al., 2019"},{"why":"The MIMIC-Eye dataset that supplies the aligned radiographs, gaze, bounding boxes, and reports.","marker":"Hsieh et al., 2023"},{"why":"The REFLACX half of the data; its missing region masks motivate the YOLO completion that is load-bearing.","marker":"Lanfredi et al., 2022"},{"why":"The EyeGaze dataset portion with radiologist gaze, transcripts, and bounding-box annotations.","marker":"Karargyris et al., 2021"},{"why":"Provides the Jensen-Shannon divergence and NSS saliency benchmark thresholds used to interpret attention alignment.","marker":"Bylinskii et al., 2019"},{"why":"Bio-ClinicalBERT encodes the report text modality.","marker":"Alsentzer et al., 2019"},{"why":"The ViT-BASE image backbone, transfer-learned on CheXpert.","marker":"Dosovitskiy et al., 2020"}],"fun_headline_variants":["Radiologist gaze lifts AI chest X-ray diagnosis","AI learns from radiologists' eyes to spot chest issues","Gaze-supervised AI improves chest X-ray classification","Eye-tracking boosts chest X-ray AI to 0.631 F1","Seeing like a radiologist: gaze-guided AI for X-rays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically completed location boxes for the REFLACX studies are accurate enough to be treated as expert ground truth, because both the gaze-attention training and the anatomical region mapping for reports rely on those boxes.","fun_headline_variants_meta":{"raw":{"variants":["Radiologist gaze lifts AI chest X-ray diagnosis","AI learns from radiologists' eyes to spot chest issues","Gaze-supervised AI improves chest X-ray classification","Eye-tracking boosts chest X-ray AI to 0.631 F1","Seeing like a radiologist: gaze-guided AI for X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3462,"prompt_tokens":947,"completion_tokens":2515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":563,"tokens_out":2515,"duration_ms":18582,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:15:44.788834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline twice on the REFLACX half of the corpus, once with YOLO-completed bounding boxes and once with radiologist-drawn abnormality ellipses or manually corrected boxes replacing them, then compare macro F1, AUC, and attention-alignment metrics. If the synthetic boxes are noise-free, results should stay roughly equal; a systematic disagreement between the two runs, or a large measured mismatch between YOLO boxes and human annotations, would show the load-bearing assumption does not hold.","supporting_citations":[{"cited_title":"What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 0 (3): 0 740--757, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the Jensen-Shannon divergence and NSS saliency benchmark thresholds used to interpret attention alignment."}],"review_version":2}