REVIEW 3 major objections 5 minor 4 references
How Good is Artificial Intelligence at Automatically Answering Consumer Questions Related to Alzheimer's Disease?
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that an AI language model can automatically generate answers to consumer questions about Alzheimer's disease that two medical annotators rate as mostly relevant, with the original GPT-2 scoring 3.0 out of 4 on relevance.
desk verdict Small, honest feasibility study of GPT-2 for Alzheimer's consumer questions, but the headline comparison rests on noisy ratings with no confidence intervals and the 'slightly better' claim is not supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is GPT-2, a transformer-based language model that generates coherent text by predicting one word at a time, and EduGPT-2, the same model fine-tuned on more than 9,000 patient education documents about diseases, symptoms, treatments, and procedures. The method feeds a consumer question as a prompt, lets the model generate an answer, and then has two annotators score each response on a 4-point relevance scale (4 = most relevant, 1 = completely irrelevant). The comparison of the generic and domain-fine-tuned models is the empirical test of whether transfer learning improves answer quality.
What would settle it
Re-score the same 84 answers with a fresh pair of annotators: if the mean relevance for either model falls below 2.5 on the 1–4 scale, the answer sets are not 'mostly relevant' and the feasibility claim would be contradicted.
Extended reading notes
Core claim
The central discovery is that a pretrained language model without special medical training can generate responses to Alzheimer's-related consumer questions that human raters find largely on-topic. The authors crawled 1,000 question titles from Yahoo! Answers, had annotators classify 277 as well-formed, and selected 84 for evaluation. GPT-2 and EduGPT-2 each produced responses; two medical annotators gave mean relevance scores of 3.0 and 2.8 out of 4, respectively. The authors conclude that automatically answering such consumer questions is feasible, and they note that the original GPT-2 performed slightly better than the transfer-learned EduGPT-2, a result they attribute to possible generalizability issues in the fine-tuned model.
Load-bearing premise
The conclusion rests on the premise that the average of two annotators' relevance scores is a dependable measure of answer quality, even though the annotators disagreed substantially.
Editorial extensions
If this is right
- If replicated, AI-generated answers could give caregivers immediate responses on online health platforms when human answers are slow or missing.
- The finding that the general model beats the fine-tuned one suggests that formal patient education text may not match the informal style of consumer questions.
- The same approach could be tested on other chronic diseases where caregivers face similar information gaps.
- The paper's outcome supports adding medical-accuracy checks before any real-world deployment, since relevance alone does not guarantee correctness.
Reading between the lines
- The reliability of the result is undercut by the reported low inter-annotator agreement (0.43 and 0.22), so the mean scores of 3.0 and 2.8 might not replicate with different raters.
- A natural next experiment would score generated answers for factual correctness in addition to relevance, since a relevant but inaccurate answer could mislead a caregiver.
- Fine-tuning on actual online question-answer pairs, rather than formal education materials, might plausibly reverse the observed ranking between GPT-2 and EduGPT-2.
- Performance may vary by question type (symptoms, treatment, caregiving), and separate sub-analyses could reveal where the models are most and least useful.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a feasibility study of using GPT-2 and a domain-adapted EduGPT-2 to automatically generate answers to consumer questions about Alzheimer's disease (AD). The authors crawled 1,000 question titles from Yahoo! Answers, filtered to 277 'well-formed' questions via manual annotation, randomly selected 84, generated responses with both models, and had two annotators rate relevance on a 1–4 scale. They report mean relevance scores of 3.0 for GPT-2 and 2.8 for EduGPT-2, with inter-annotator agreement scores of 0.43 and 0.22. The conclusion states that the annotations show the feasibility of applying AI models to answer AD-related consumer questions, and notes that GPT-2 performs slightly better than EduGPT-2.
Significance. If the evaluation were reliable, this would be a useful early demonstration that large language models can generate topically relevant answers to consumer health questions in a domain where timely, accurate information is important. The paper is among the first to apply GPT-2 to AD-specific consumer questions and includes a domain-adaptation step. However, the significance is currently limited by the evaluation methodology: the central claim rests on averaged relevance ratings from two annotators whose agreement is low, and no baseline or statistical inference is provided. As it stands, the paper supports only a weak feasibility reading (models produce non-random, often topically relevant text), not a validated claim about answer quality or utility.
major comments (3)
- [Methods (annotation procedure) and Results (reported scores)] The reported inter-annotator agreement is low (0.43 for GPT-2 and 0.22 for EduGPT-2), yet the paper uses the average of the two annotators' scores as the sole outcome measure and does not report per-annotator distributions, the type of agreement coefficient, or confidence intervals. With low agreement, the averaged ratings may not reflect either annotator's judgment, and the mean scores of 3.0 and 2.8 cannot be interpreted as stable. This is load-bearing because the feasibility conclusion in the Conclusion rests entirely on these averages. Please report the agreement metric with a citation, provide the per-annotator score distributions, and compute confidence intervals or a bootstrap estimate for the mean relevance scores; additionally, test whether the means are significantly above the scale midpoint of 2.5.
- [Results (GPT-2 vs. EduGPT-2 comparison)] The claim that GPT-2 performs 'slightly better' than EduGPT-2 is based on a mean difference of 0.2 on a 4-point scale (3.0 vs. 2.8) with no significance test, effect size, or confidence interval. With n=84 paired questions, this difference may easily be within sampling noise. Please provide a paired statistical test (e.g., Wilcoxon signed-rank test) or a bootstrap confidence interval for the mean difference, and avoid comparative claims without such support.
- [Methods (baseline) and Conclusion (feasibility claim)] The evaluation lacks any baseline or comparator condition. A mean relevance score of 3.0 is described as 'most responses are relevant,' but there is no reference point (e.g., human-written answers, a rule-based retrieval system, or random text) to calibrate whether 3.0 is meaningfully good or merely the default response of a fluent language model. Moreover, the outcome is limited to perceived relevance; it does not evaluate factual accuracy, medical safety, or actionability, which are essential for consumer health information. Please add a baseline condition or substantially soften the feasibility claim to 'perceived relevance' and discuss the absence of clinical or factual validation.
minor comments (5)
- [Methods] The term 'inner annotator agreement' should be 'inter-annotator agreement'; also specify the coefficient used (e.g., Cohen's kappa, weighted kappa, or intraclass correlation).
- [Figure 1] Figure 1 is not described in the text; clarify what the bars represent (counts, percentages, or average scores) and whether they show all questions or separated by model.
- [Methods] Provide details of the generation procedure: how each question was formatted as a prompt, the maximum generation length, decoding parameters (temperature, top-k, etc.), and any post-processing or filtering of empty or degenerate outputs.
- [Introduction and Conclusion] The abstract claims 'this is the first study,' while the full text says 'it is rare in the literature'; please use consistent phrasing and support either claim with a brief literature check.
- [Results] The two annotators who classified questions into 'well-formed' and 'poorly-formed' presumably had the same reliability issues as the relevance annotators; please report agreement for the question classification step as well.
Circularity Check
No circularity: the evaluated answers are generated by pretrained models and rated by external annotators, with no fitted parameter or self-citation serving as the load-bearing evidence.
full rationale
The paper's derivation chain is: collect consumer questions from Yahoo! Answers, generate responses with GPT-2 and EduGPT-2, have two annotators rate relevance, and report the mean scores. No model parameter is fitted to the annotation scores; EduGPT-2 is fine-tuned on Mayo Clinic patient education materials, but the evaluation corpus comes from Yahoo! Answers and the human ratings are not fed back into the model. No claimed result is defined in terms of the outcome it purports to establish, and no load-bearing claim rests on a self-citation. The low inter-annotator agreement (0.43 and 0.22) is a measurement-reliability concern, not a circularity concern; the paper's conclusion is weak but not circular. No specific reduction (Eq. X = Eq. Y by construction, or fitted parameter renamed as prediction) can be exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption Relevance of a generated response is a valid proxy for answer quality for consumer health questions.
- domain assumption Yahoo! Answers questions under the 'Alzheimer's disease' topic are representative of caregiver consumer questions.
- domain assumption GPT-2 generated text can be treated as an answer without additional filtering.
- domain assumption Averaging two annotators' relevance scores is valid despite low inter-annotator agreement.
Cite this review
Pith. "Pith review of How Good is Artificial Intelligence at Automatically Answering Consumer Questions Related to Alzheimer's Disease?." pith.science (2026). https://pith.science/paper/3UCBAGIK
@misc{pith2026190810678,
author = {Pith},
title = {Pith review of: How Good is Artificial Intelligence at Automatically Answering Consumer Questions Related to Alzheimer's Disease?},
year = {2026},
howpublished = {\url{https://pith.science/paper/3UCBAGIK}},
note = {Machine review of arXiv:1908.10678}
}
abstract
Alzheimer's Disease (AD) is the most common type of dementia, comprising 60-80% of cases. There were an estimated 5.8 million Americans living with Alzheimer's dementia in 2019, and this number will almost double every 20 years. The total lifetime cost of care for someone with dementia is estimated to be $350,174 in 2018, 70% of which is associated with family-provided care. Most family caregivers face emotional, financial and physical difficulties. As a medium to relieve this burden, online communities in social media websites such as Twitter, Reddit, and Yahoo! Answers provide potential venues for caregivers to search relevant questions and answers, or post questions and seek answers from other members. However, there are often a limited number of relevant questions and responses to search from, and posted questions are rarely answered immediately. Due to recent advancement in Artificial Intelligence (AI), particularly Natural Language Processing (NLP), we propose to utilize AI to automatically generate answers to AD-related consumer questions posted by caregivers and evaluate how good AI is at answering those questions. To the best of our knowledge, this is the first study in the literature applying and evaluating AI models designed to automatically answer consumer questions related to AD.
Reference graph
Works this paper leans on
-
[1]
2019 Alzheimer's disease facts and figures
Gaugler J, James B, Johnson T, Marin A, Weuve J. 2019 Alzheimer's disease facts and figures. Alzheimers & Dementia. 2019 Mar 1;15(3):321-87
work page 2019
-
[2]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL. 2019
work page 2019
-
[3]
Language models are unsupervised multitask learners
Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog. 2019 Feb;1(8)
work page 2019
-
[4]
BERT has a mouth, and it must speak: BERT as a markov random field language model
Wang A, Cho K. BERT has a mouth, and it must speak: BERT as a markov random field language model. arXiv preprint arXiv:1902.04094
arXiv 1902
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.