{"id":"ba6ec65e-056b-40fa-a429-f2cf6034341a","arxiv_id":"2507.05816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Large language models predict retinopathy of prematurity risk poorly from admission notes alone, over-predict medium and high risk, and positive emotional prompt framing partially corrects this bias.","lead":"Researchers built a Chinese dataset of preterm infant admission notes and tested whether large language models can predict retinopathy of prematurity risk. They found the models are biased toward predicting medium or high risk, and that positive emotional wording in prompts can reduce that bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'positive affect mitigates bias' conclusion is confounded: the positive prompt also changes the stated base rate and the advised decision rule, so the observed shift cannot be attributed to emotional valence alone.","rationale":"The reader's weakest_assumption identifies exactly the concern that matters most: the three affective prompts differ on multiple dimensions beyond valence, so the observed mitigation of bias cannot be cleanly attributed to affect. My reading of the paper does not change the verdict, because the other principal findings are more solid. The bias toward medium- and high-risk predictions is supported by confusion matrices (Figure 3), by the unbalanced-dataset experiment showing much lower accuracy when low-risk samples dominate (Table VII), and by the case study. The dataset contribution (CROP) is real, though its label-construction procedure is described only briefly. The CoT and ICL accuracy gains are internally consistent and reproducible in principle. The affective-framing conclusion, by contrast, is the least supported and the most consequential for the paper's framing as an 'affective bias' study. The proposed test would settle whether the effect is genuine by controlling for content and prior. Since the paper is already CONDITIONAL and this concern is the condition, no verdict change is needed, but the condition should explicitly require the matched-phrasing control before the affective-mitigation claim is accepted.","tokens_in":18571,"tokens_out":1919,"duration_ms":23310,"concrete_test":"For the five models in Table IV, construct matched prompt pairs per valence where the only changed element is a single emotional phrase (e.g., 'You feel confident about outcomes' vs. 'You feel neutral about outcomes' vs. 'You feel pessimistic about outcomes'), keeping all other role definitions, task instructions, output constraints, and any disease-base-rate statements identical in length and content across conditions. Run all three phrasings with temperature 0, report accuracy and the low-risk-misclassification rate (the bias measure used in Section V-D), and compute the mean and range across three paraphrases per valence. If the positive-minus-negative difference shrinks to noise or reverses, the affective-valence claim is a wording/prior artifact; if it persists across multiple phrasings with overlapping ranges, the claim gains real support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's third conclusion (Section V-D, reasserted in the Conclusion) is load-bearing: 'prompts infused with positive affective framing help reduce the bias exhibited by LLMs.' But the three prompts in Section IV-B do not differ only in emotional valence. The positive prompt says, 'You are a pediatric ophthalmologist who is confident in treatment outcomes and believes that most cases can successfully avoid retinopathy,' which explicitly asserts a low disease base rate and directs the model toward low-risk classifications. The negative prompt presumably asserts the opposite. Thus the reported mitigation of the high-risk bias could be a prior shift or a direct instruction-following effect, not an affective one. The prompts also differ in length and in whether they mention specific clinical expectations. Table IV shows no significance tests and no variance estimates; the effect is inconsistent across models (e.g., Qwen-2.5 improves only from 37.76 to 42.29, while o3-mini improves by 6.34 points; o4-mini shows a smaller gain of 3.93 points). Figure 4's headline comparison, 76.79% high-risk predictions under negative framing versus 21.43% under positive framing, is a single illustrative pair, not an aggregate statistic with uncertainty. Because the affective-framing claim rests on one hand-written phrasing per valence, the conclusion is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CROP, a Chinese benchmark dataset of 993 admission records for predicting retinopathy of prematurity (ROP) risk, and Affective-ROPTester, an evaluation framework with Instruction, Chain-of-Thought (CoT), and In-Context Learning (ICL) prompting schemes, with additional affective framings (negative, neutral, positive) embedded in the Instruction prompt. Experiments across a wide range of open-source, closed-source, and medical LLMs show that (i) LLMs alone predict ROP risk poorly, (ii) CoT and ICL improve accuracy, and (iii) LLMs exhibit a bias toward predicting medium and high risk, which the paper claims is mitigated by positive affective framing.","tokens_in":18820,"tokens_out":3362,"duration_ms":38182,"significance":"If the findings hold, CROP will be a useful first Chinese text-only benchmark for ROP risk prediction, and the Affective-ROPTester framework provides a reproducible protocol for evaluating LLM capability and bias in clinical risk stratification. The paper's breadth of model coverage and its confusion-matrix-based evidence for a consistent high/medium-risk bias are valuable. However, the central affective-framing conclusion is not yet empirically established because the prompts differ in more than emotional valence, and the CoT scheme's hand-coded rubric may confound the interpretation of the performance gains. These issues limit the current contribution until addressed.","major_comments":[{"comment":"The claim that positive affective framing mitigates bias is confounded: the three affective prompts do not differ only in emotional valence. The positive prompt ('You are a pediatric ophthalmologist who is confident in treatment outcomes and believes that most cases can successfully avoid retinopathy') explicitly asserts a low base rate and instructs the model toward low-risk classifications, while the negative prompt presumably asserts the opposite. Consequently, the observed reduction in high-risk predictions under positive framing (e.g., Figure 4's 76.79% vs. 21.43%) could be a prior shift or instruction-following effect rather than an emotional effect. The paper should include multiple phrasings per valence, a lexical control that varies wording without changing valence, or an explicit manipulation of the stated base rate to disentangle these factors. In addition, Table IV provides no significance tests or confidence intervals, and the effect is inconsistent across models (e.g., Qwen-2.5 gains 4.53 accuracy points from neutral to positive, while o4-mini loses 1.51 points), so the claim that positive framing helps is not yet supported.","section":"Section IV-B and Section V-D"},{"comment":"The CoT scheme encodes a deterministic clinical rubric: Step 7 maps gestational age <28 weeks or weight <1000 g to high risk, 28-32 weeks and 1000-1500 g to medium risk, and >32 weeks and >1500 g to low risk. The reported performance improvements under CoT therefore largely measure how well this hand-coded rubric matches the dataset labels, rather than the LLM's own reasoning about ROP risk. Since the paper interprets the CoT results as evidence that external knowledge enhances LLM capability, it should explicitly acknowledge this confound, analyze how often the model simply echoes the rubric's thresholds, and ideally compare against a baseline where the same rubric is applied non-LLM. This is load-bearing for the paper's second principal conclusion (Section V-B).","section":"Section IV-C, Scheme 2"},{"comment":"The affective-framing results lack any uncertainty quantification or hypothesis testing. No standard deviations, confidence intervals, or significance tests are reported for Table IV, and the small per-class test sizes (108-112 samples) mean that differences of a few percentage points (e.g., Qwen-2.5: 40.78 neutral vs. 42.29 positive) may be within sampling noise. The paper's conclusion that positive affective framing improves accuracy and mitigates bias should be supported by multiple runs, error bars, and a statistical test across the set of evaluated models. This is essential because the claim is one of the paper's three headline conclusions.","section":"Section V-D and Table IV"}],"minor_comments":[{"comment":"The abstract lists 'two principal observations' but then enumerates three (limited efficacy, affective bias, positive framing mitigation); the Conclusion likewise contains three separate claims.","section":"Abstract and Conclusion"},{"comment":"Equation (1) states that y_j represents the true label of x, but in the argmax expression y_j should denote a candidate label in Y; this is a notation error.","section":"Equation (1)"},{"comment":"The bullet list contains typos: 'instrcution' should be 'instruction' and the sentence 'And also give an instrcution for researchers...' is ungrammatical; this should be corrected.","section":"Section I, Contributions"},{"comment":"The text says the dataset is 'evenly divided' into three categories, but Table I reports 112 low-, 111 medium-, and 108 high-risk test samples; the imbalance is minor but the wording should be adjusted.","section":"Table I and Section III-B"},{"comment":"Figure 4 does not specify which model the confusion matrices are for; this should be stated in the caption or text.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad empirical study with a potentially useful new dataset and a clear documentation of high/medium-risk bias across many LLMs. The main gap is that the affective-framing conclusion, which is central to the paper's stated contributions, is confounded by uncontrolled prompt content and lacks statistical support. I would be willing to reconsider after a revision that adds controls for valence and significance testing, and that clarifies the CoT rubric's role."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is worth a look if you care about clinical LLM evaluation. It introduces CROP, the first Chinese dataset for ROP risk prediction from admission notes alone — 993 balanced records. The evaluation is broad: many LLMs, three prompting schemes, ablations on CoT factors, ICL shot count, model size, and an unbalanced-set robustness check. The bias finding holds up: naive LLMs overpredict medium/high risk, and the confusion matrices plus the unbalanced experiment (accuracy collapses to 19% on a low-skew sample) make that credible. External knowledge in CoT and ICL clearly helps, though the best accuracy is only 61%.\n\nThe soft spot is the third conclusion, that positive affective framing reduces bias. The positive prompt is not just emotionally different; it tells the model 'most cases can successfully avoid retinopathy,' which is a direct prior statement and a decision-rule hint. So the observed shift could be instruction-following or base-rate adjustment, not affect. There are no significance tests, and the effect sizes vary a lot across models. That claim needs multiple phrasings per valence and at least a paired test before I'd trust it.\n\nAlso, the CoT scheme is essentially a hand-coded rubric, so the improvement over the instruction condition partly measures how well the model follows the rubric rather than its own clinical reasoning. That's not a fatal flaw — the paper is honest about it — but it tempers the 'external knowledge helps' framing. And no rule-based baseline (e.g., logistic regression on gestational age and birth weight) means we don't know if the best LLM adds anything over a simple classifier. Data and code are not released, which makes independent verification harder.\n\nOverall, the dataset and evaluation framework are a genuine contribution for the ROP risk prediction niche. The bias finding is solid. The affective framing conclusion is not yet established. I'd send it to review — a serious referee can push on the prompt confound and baselines, and the paper deserves revision rather than rejection.","headline":"Useful new Chinese ROP risk benchmark and a credible bias finding, but the affective-framing mitigation claim is confounded by prompt content beyond valence.","tokens_in":19385,"tokens_out":2313,"would_cite":false,"duration_ms":25968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds a 993-record Chinese benchmark for predicting retinopathy of prematurity (ROP) risk in preterm infants and reports that large language models over-predict medium and high risk, a bias that positive affective prompts…","keywords":["retinopathy of prematurity","large language models","affective bias","prompt engineering","risk prediction","chain-of-thought prompting","in-context learning","Chinese medical benchmark"],"falsifier":"Re-run the CROP evaluations with several paraphrases of each affective prompt, matched for length and lexical content; if accuracy and bias shifts do not track valence consistently across paraphrases, the affective-framing claim is not supported.","tokens_in":18353,"feed_emoji":"🩺","tokens_out":6877,"duration_ms":70921,"temperature":0.7,"pith_summary":"The paper builds what it identifies as the first Chinese benchmark, CROP, of 993 premature-infant admission records labeled low, medium, or high risk for retinopathy of prematurity (ROP), and uses it to ask whether large language models can predict ROP risk from text alone. It finds that with only an instruction prompt the models perform near chance and systematically over-assign medium and high risk, especially to low-risk infants. Feeding the models a hand-built chain of thought based on known risk factors, or a few retrieved examples, raises accuracy substantially; the best model reaches 61.33 percent. The paper's most actionable claim is that positive affective framing in the prompt reduces the over-prediction bias and improves accuracy relative to negative or neutral framing. If true, prompt affect is a real, adjustable lever on diagnostic bias, not just a stylistic footnote.","feed_headline":"Positive prompts curb AI over-prediction of preterm eye risk","feed_subtitle":"New Chinese benchmark tests 20+ LLMs on premature-infant records; best accuracy is 61 percent.","key_machinery":"The load-bearing mechanism is Affective-ROPTester, an evaluation scaffold with three prompting schemes. The Instruction scheme presents the model with a role definition and a restricted three-way label space, probing intrinsic knowledge and bias. The CoT scheme prepends an eight-step reasoning template built from known ROP risk factors, including gestational age, birth weight, oxygen exposure, multiple pregnancy, and maternal complications, thereby supplying external knowledge. The ICL scheme retrieves five nearest training admissions by BM25 and presents them as labeled demonstrations. Superimposed on the Instruction scheme are three hand-written affective framings, negative, neutral, and positive, and this affective component is what carries the paper's bias-mitigation claim.","core_discovery":"The central claim is that LLMs are biased risk predictors for ROP when left to their own knowledge, consistently pushing low- and medium-risk admissions into higher categories, and that this bias can be partially corrected by supplying structured external knowledge or by embedding positive emotion in the prompt. The paper states that prompts infused with positive affective framing help reduce the bias exhibited by LLMs. Accuracy follows the same pattern: negative prompts hurt, positive prompts help, with DeepSeek-R1 rising from 40.18 percent under negative framing to 49.24 percent under positive framing, and with chain-of-thought prompting the best model reaches 61.33 percent accuracy, which the paper still calls insufficient for clinical use.","pith_inferences":["Editorial inference: if positive framing works by shifting the model's prior toward better outcomes, then calibration or class-prior adjustment could achieve the same bias reduction without changing prompt content.","Editorial inference: the affective result may extend to other clinical risk tasks where LLMs systematically over-predict severe outcomes, such as sepsis or ICU mortality screening, and should be tested there.","Editorial inference: because the three prompts differ in wording and content beyond valence, a lexical-control study is needed to separate affect from instruction content; until then the size of the affect effect is uncertain.","Editorial inference: the unbalanced-dataset experiment implies the bias is visible as a drop in accuracy when low-risk cases dominate, so a testable extension is reporting calibration, such as Brier score or expected calibration error, per valence."],"forward_implications":["If correct, LLM-only screening from admission notes is not yet clinically usable for ROP; the ceiling in this benchmark is about 61 percent accuracy with a chain-of-thought prompt.","Structured external knowledge, through risk-factor reasoning or demonstrations, is what makes LLM predictions usable at all, shifting the best open models from roughly 39 percent to 61 percent accuracy.","The documented over-prediction of medium and high risk means naive deployment would generate false alarms disproportionate to low-risk infants.","Positive affective prompts are a cheap, parameter-free intervention that moves accuracy and bias in the desired direction, so prompt wording should be reported in any clinical LLM benchmark.","These effects appear across open, closed, and medical models, indicating the bias is not specific to one model family."],"supporting_citations":[{"why":"Supplies the known ROP risk factors that the chain-of-thought template is built on.","marker":"[15]"},{"why":"Defines the prediction task and the severity grouping used to map diagnoses to low, medium, and high risk labels.","marker":"[17]"},{"why":"Provides the BM25 retrieval algorithm that selects demonstration examples in the in-context learning scheme.","marker":"[41]"},{"why":"Introduces the Qwen-2.5 model family that achieves the best chain-of-thought and in-context learning results and drives the size-scaling experiments.","marker":"[48]"},{"why":"Supplies the DeepSeek-R1 reasoning model used in the affective-framing and stability experiments.","marker":"[2]"},{"why":"Provides the LoRA method used for the fine-tuning comparison that underperforms prompted approaches.","marker":"[58]"},{"why":"Gives prior evidence that prompt design induces bias in pretrained language models, motivating the affective-bias analysis.","marker":"[38]"},{"why":"Provides the clinical classification scheme for ROP severity used to assign risk categories in the dataset.","marker":"[39]"}],"fun_headline_variants":["How emotion in prompts skews AI risk calls for preterm eyes","AI overflags preterm eye risk; positive prompts help","Positive framing trims AI's ROP overprediction bias","LLMs stumble on ROP risk without structured aid","Affective prompts shift LLM bias in preterm eye risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result that positive affect reduces bias rests on the assumption that the three hand-written prompts differ only in emotional valence, but the positive prompt also changes wording, length, expected base rate, and clinical advice, so the measured effect could come from content rather than affect.","fun_headline_variants_meta":{"raw":{"variants":["How emotion in prompts skews AI risk calls for preterm eyes","AI overflags preterm eye risk; positive prompts help","Positive framing trims AI's ROP overprediction bias","LLMs stumble on ROP risk without structured aid","Affective prompts shift LLM bias in preterm eye risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1314,"prompt_tokens":978,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":594,"tokens_out":336,"duration_ms":4106,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:17:42.061724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CROP evaluations with several paraphrases of each affective prompt, matched for length and lexical content; if accuracy and bias shifts do not track valence consistently across paraphrases, the affective-framing claim is not supported.","supporting_citations":[{"cited_title":"Research progress on the risk factors of retinopathy of prematurity,","cited_arxiv_id":null,"evidence_quote":"Supplies the known ROP risk factors that the chain-of-thought template is built on."},{"cited_title":"Development and validation of a deep learning model to predict the occurrence and severity of retinopathy of prematurity","cited_arxiv_id":null,"evidence_quote":"Defines the prediction task and the severity grouping used to map diagnoses to low, medium, and high risk labels."},{"cited_title":"The probabilistic relevance framework: Bm25 and beyond,","cited_arxiv_id":null,"evidence_quote":"Provides the BM25 retrieval algorithm that selects demonstration examples in the in-context learning scheme."},{"cited_title":"Qwen2.5: A party of foundation models,","cited_arxiv_id":null,"evidence_quote":"Introduces the Qwen-2.5 model family that achieves the best chain-of-thought and in-context learning results and drives the size-scaling experiments."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the LoRA method used for the fine-tuning comparison that underperforms prompted approaches."},{"cited_title":"The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,","cited_arxiv_id":null,"evidence_quote":"Gives prior evidence that prompt design induces bias in pretrained language models, motivating the affective-bias analysis."},{"cited_title":"Screening examination of premature infants for retinopathy of prematurity,","cited_arxiv_id":null,"evidence_quote":"Provides the clinical classification scheme for ROP severity used to assign risk categories in the dataset."}],"review_version":1}