{"id":"91ea5f3f-ff61-4df5-a04a-3afa998433d7","arxiv_id":"2502.04288","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding geolocation to LLM embeddings is claimed to improve Alzheimer's risk prediction, but the target is a CDC aggregate value and the near-zero error suggests label memorization.","lead":"The paper uses Llama3-70B and GPT-4o embeddings on CDC health statistics to predict Alzheimer's risk scores, reporting that geolocation cuts error by 28-33%. The dataset contains aggregate county-level percentages rather than patient clinical notes, and the near-perfect accuracy indicates the model may be memorizing text labels instead of diagnosing Alzheimer's.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 28.57%/33.47% improvements and R2=0.9999 are artifacts of predicting CDC PLACES aggregate percentages from survey-question text, not patient-level AD risk from clinical notes.","rationale":"I read the paper's aim as using geolocation plus LLM embeddings of clinical text to predict a continuous AD risk score. For the claimed improvement to hold, the target must be a patient-level risk score and the text input must be clinical notes. The methodology section undercuts both assumptions. Table 1 lists CDC survey questions, not clinical text; Section 3 explicitly states that the statistical values are the 'risk scores' and ties them to 'the patients in this area', documenting that the target is a geographic aggregate, not an individual diagnosis. The 'locationdesc', 'Geolocation', 'topic', and year columns encode the place and measure, while 'data value' is one number per question-location-year. A regression from question text to data value can be near-perfect because the question names the statistic being predicted. The paper's own baseline—a Random Forest on categorical and numerical features with no text embeddings—already reaches R2=0.9781, which is implausible for patient-level AD risk prediction and consistent with retrieving area-level percentages from categorical identifiers. The geolocation ablation's relative improvements are computed on MSE values around 1e-9, so the quoted percentage reductions are differences in rounding-scale noise; no statistical tests, confidence intervals, or external baselines are provided. This is not merely disagreement with consensus; the data description is internally inconsistent with the claim that clinical records are being analyzed. The reader's weakest assumption identifies the same load-bearing flaw, and my read does not change the reject verdict.","tokens_in":8333,"tokens_out":3815,"duration_ms":39609,"concrete_test":"Download the CDC PLACES 2024 release and verify the schema: confirm that 'Data_Value' is an aggregate prevalence/percentage at county or census-tract level and that the dataset contains zero patient identifiers or clinical notes. Then reproduce the DMV experiment under an altered input set that replaces the 'question' text with a constant placeholder while retaining geolocation and all other categorical and numerical features. If the R2 falls from 0.9999 to near zero while the original setup reproduces 0.9999, the near-perfect fit and the apparent geolocation improvement are attributable to retrieving the aggregate value from the question text, not to analyzing clinical notes. This single comparison discriminates between a real clinical NLP result and the proposed alternative explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the CDC data to be patient-level clinical notes with a continuous AD risk target. The paper's own Table 1 disproves this: the 'question' column contains PLACES measure names such as 'Percentage of older adults who are experiencing frequent mental distress', and the 'data value' is the corresponding aggregate percentage for a geographic area. Section 3 confirms this by saying 'We will use these statistical values as the risk scores to predict' and 'The higher the score, the more likely the patients in this area are to have Alzheimer's disease.' These rows describe geographic areas, not patients; there are no clinical notes and no individual AD risk scores. The DMV pipeline feeds the question text into Llama3/GPT-4o embeddings and regresses 'data value' from it. Because the target is the answer to the question, an embedding of the question text (plus year and geolocation) can recover the aggregate statistic, yielding R2=0.9999. The paper's own Random Forest baseline reaches R2=0.9781 without any text embeddings, which is implausible for patient-level diagnostic risk prediction and consistent with retrieving area-level percentages from the categorical question/stratification fields. The geolocation ablation is then computed on MSE values of order 1e-9 (0.93e-9 vs 1.59e-9, 1.24e-9 vs 1.87e-9), so the quoted '70.97%', '50.70%', '28.57%', and '33.47%' reductions are relative differences at rounding-error scale, with no confidence intervals or significance tests. The Limitations section mentions only static data and computational cost, and does not disclose the absence of patient-level clinical information.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a 'DMV' framework that concatenates Llama3-70B or GPT-4o text embeddings with geolocation and other tabular features, then trains a Random Forest regressor to predict a continuous Alzheimer's disease risk score. The dataset is described as CDC-derived records with 284K rows, and the authors report near-perfect test performance (R-squared = 0.9999 for both embedding models) as well as geolocation-based error reductions of 28.57% for Llama3-70B and 33.47% for GPT-4o on MAE. The central claim is that geolocation information significantly improves early AD risk prediction from clinical notes.","tokens_in":8651,"tokens_out":3964,"duration_ms":39613,"significance":"If the evidence supported it, combining LLM embeddings with geolocation to improve patient-level Alzheimer's risk prediction would be a practically valuable contribution. The paper is clearly organized, and its experimental protocol (10-fold cross-validation, baseline comparison, and ablation study) is easy to follow. However, no code or data are provided, so reproducibility rests entirely on the authors' description. More importantly, as detailed below, the dataset and target variable do not correspond to patient-level clinical notes or to individual AD risk, so the strong numerical results are not evidence about Alzheimer's diagnosis. The claimed contribution therefore does not currently hold.","major_comments":[{"comment":"The dataset described is not a set of clinical notes. Table 1 shows the input 'question' column containing CDC PLACES measure labels such as 'Percentage of older adults who are experiencing frequent mental distress,' and the text states that these 'statistical values' will be used as the risk scores to predict, with higher scores meaning patients 'in this area' are more likely to have Alzheimer's disease. These are area-level aggregate percentages, not individual patient AD risk scores. The abstract and the Data Description claim that the dataset includes patient records and clinical notes, but the examples shown and the features listed in Tables 2 and 3 provide no evidence that any clinical-note text is used. Because the regression target is the numeric answer corresponding to the question text fed as input, the near-perfect R-squared reflects a lookup of the known aggregate statistic rather than a diagnostic model.","section":"§3, Table 1"},{"comment":"Table 5 reports R-squared = 0.9999 and MSE values of 0.93e-9 and 1.24e-9 for the embedding models. For a patient-level clinical risk score these values would be implausibly high; they are, however, exactly what one expects when the model predicts the published CDC data value from question text and location identifiers. The baseline Random Forest already reaches R-squared = 0.9781 from categorical features such as topic, question, and stratification, which is likewise inconsistent with patient-level diagnosis and consistent with retrieving an area-level percentage from category labels. No confidence intervals or statistical significance tests are reported, so the abstract's claim that geolocation 'significantly decreases' prediction error is unsupported.","section":"§5, Table 5"},{"comment":"The geolocation ablation compares MSE values of order 1e-9 and MAE values of order 1e-7. The reported improvements of 70.97%, 50.70%, 28.57%, and 33.47% are relative differences at a scale where rounding of the underlying metrics could dominate; no confidence intervals are given. These differences are not shown to be statistically or clinically meaningful, and they do not support the conclusion that geolocation data enhance early AD risk assessment.","section":"§5, Tables 6 and 7"},{"comment":"Section 4 states that text from 'patient records' is tokenized and embedded, but the features listed in Tables 2 and 3 include columns such as locationabbr, topic, question, and data value. The embedding input is therefore the question label, not unstructured clinical narrative. Thus the framework does not demonstrate the claimed analysis of patient records, and the conclusions in Section 6 about 'unstructured medical text' and 'rich features extracted from patient records' are not supported by the experiments actually described.","section":"§4, DMV Framework"}],"minor_comments":[{"comment":"The table heading 'Matric Comparison' should read 'Metric Comparison,' and the symbol 'R!' is used inconsistently instead of 'R-squared' or an actual superscript-2 notation.","section":"§5, Table 5"},{"comment":"The model name appears both as 'GPT4-o' and 'GPT-4o'; the paper should adopt one consistent spelling.","section":"Throughout"},{"comment":"The Limitations section mentions static datasets and computational resources but does not acknowledge that the target variable is an aggregate CDC statistic rather than a patient-level risk score; this omission obscures the main limitation of the study.","section":"Limitations"}],"recommendation":"reject","confidential_remarks":"The editor may wish to note that, despite the clinical framing, the experiments amount to retrieving CDC PLACES aggregate percentages from question labels and location fields. The reported metrics are artifacts of this target/input alignment rather than evidence about Alzheimer's diagnosis. A resubmission would require a fundamentally different dataset and evaluation, not just additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before spending referee time on it: the dataset is not clinical records and the target is not an AD risk score. The paper uses CDC PLACES aggregate statistics, where each row is a geographic area and a measure like 'Percentage of older adults who are currently obese.' The target, 'data value,' is literally the answer to the question text used as input. So the near-perfect R² of 0.9999 is expected memorization, not diagnostic insight. The geolocation ablation compares MSE values around 1e-9, so the quoted 28–33% improvements are relative differences at rounding-error scale with no error bars.\n\nWhat is okay: the paper is clearly written, the pipeline is standard, and the authors are transparent about their tables. There is a reasonable intuition that environmental context might matter for dementia risk. But the execution never touches that intuition: no patient-level data, no clinical notes, no actual risk scores. The related work is relevant but the comparisons are superficial.\n\nThe soft spots are not minor. The central claim is unsupported because the data and target are mislabeled. The baseline Random Forest reaches R²=0.9781 without any text embeddings, which is only plausible if the model is retrieving area-level percentages from the categorical fields—further evidence that this is a lookup task, not a predictive model of AD. The limitations section mentions only static data and compute cost, and does not acknowledge the absence of any clinical content. The improvements claimed in the abstract are computed on errors so small that they are meaningless.\n\nWho is this for? Nobody in clinical NLP. It might serve as a cautionary example in a methods class about checking what your target actually is, but not as a scientific contribution. A serious referee would need to send it back for a completely different dataset and framing; the current version is not salvageable by revision.\n\nRecommendation: do not send to peer review. This is a desk reject: the load-bearing assumption is false and the reported effects are artifacts.\n\nThat's my read.","headline":"The paper's central claim—geolocation improves AD risk prediction—is an artifact of predicting CDC aggregate percentages from the question text, not a real clinical result.","tokens_in":9196,"tokens_out":1851,"would_cite":false,"duration_ms":18140,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding geolocation features to Llama3-70B and GPT-4o embeddings cuts Alzheimer's risk-score prediction error by 28.57% and 33.47%.","keywords":["Alzheimer's disease","early detection","geolocation","LLM embeddings","Random Forest regression","risk score prediction","DMV framework"],"falsifier":"Permute the latitude and longitude pairs across rows while keeping the question text intact; if the near-zero mean squared error persists, the model is recovering the stored aggregate value from row-level identifiers rather than from language or location.","tokens_in":8101,"feed_emoji":"🧠","tokens_out":7728,"duration_ms":70483,"temperature":0.7,"pith_summary":"This paper is trying to establish that location information, appended to semantic embeddings of text from a public-health survey, makes an Alzheimer's risk-score regressor materially more accurate. Using the DMV framework of data processing, model training, and validation with Llama3-70B and GPT-4o embeddings feeding a Random Forest regressor, the authors report a 28.57% and 33.47% drop in mean absolute error when geolocation is included. If that result holds, text-plus-place modeling could offer a low-cost way to flag early Alzheimer's risk from records. The reader should note that the score being predicted is an aggregate survey percentage attached to a geographic area, not a patient-level diagnosis.","feed_headline":"Location data cuts AI Alzheimer's risk-score error by up to 33%","feed_subtitle":"Adding GPS coordinates to language-model embeddings shrinks prediction error by roughly a third.","key_machinery":"The DMV framework is a data-processing, model-training, and validation pipeline: missing values are imputed, categorical fields are one-hot encoded, numeric fields are normalized, the text column is embedded by Llama3-70B or GPT-4o, and the embeddings are concatenated with features including latitude and longitude before a Random Forest Regressor predicts the continuous risk score. The geolocation columns are the feature subset whose removal produces the reported error increases, so they carry the paper's central ablation claim.","core_discovery":"On the paper's own terms, the central discovery is that the spatial context of a health-survey question nearly determines the numeric answer once the question text is embedded by a large language model. The task is framed as regressing the 'data value' column from the 'question' text plus latitude, longitude, and other categorical features. With both Llama3-70B and GPT-4o embeddings, the regressor reaches $R^2 = 0.9999$; removing geolocation raises mean squared error by 70.97% and 50.70% respectively and mean absolute error by 28.57% and 33.47% respectively.","pith_inferences":["Because the target variable is an aggregate survey value that varies by question and location, the near-zero error most plausibly reflects the model memorizing the dataset's structured mapping rather than learning a patient-level diagnostic signal; the paper does not test this distinction.","A decisive extension would be to obtain patient-level notes with confirmed Alzheimer's diagnoses and addresses, then test whether location still adds predictive power after controlling for age, education, and comorbidities.","The reported geolocation gains could also be an artifact of the dataset's group structure: each location has many rows, so latitude and longitude may act as row identifiers; a grouped split that holds out all rows for a location together would settle this."],"forward_implications":["If geolocation truly adds this much signal, geographically stratified risk models should outperform text-only models for this kind of survey-derived health outcome.","If the near-perfect fit is genuine, then in this dataset the question text plus location essentially determines the risk score, leaving little room for further clinical variables.","The same embedding-plus-geography recipe could be applied to other spatially patterned health outcomes, such as asthma or diabetes risk from community health surveys.","Clinical deployment would require moving from aggregate survey statistics to individual patient notes, a step the current setup does not test."],"supporting_citations":[{"why":"Introduces Llama3, the embedding model used for the first advanced pipeline.","marker":"Dubey et al. 2024"},{"why":"Introduces GPT-4, whose GPT-4o variant supplies the second set of embeddings.","marker":"Achiam et al. 2023"},{"why":"Establishes the clinical-note NLP approach to Alzheimer's detection that this paper extends.","marker":"Oh et al. 2023"},{"why":"Applies pre-trained large language models to Alzheimer's detection, the baseline this paper builds on.","marker":"Filippo et al. 2024"},{"why":"Supplies the premise that geolocation-based monitoring carries behavioral information relevant to cognitive conditions.","marker":"Peek, Fraccaro, and van der Veer 2023"},{"why":"Reports that geolocation data aids disease prediction in another domain, supporting the rationale for adding spatial features.","marker":"Nguyen and Le 2023"},{"why":"Provides the Random Forest-based configuration used for the regressor.","marker":"Shastry and Sattar 2023"}],"fun_headline_variants":["Location data cuts Alzheimer's risk-score error by 33%","Adding GPS coordinates to clinical notes improves AD prediction by 33%","Geolocation boosts Alzheimer's risk score accuracy by one-third in AI model","DMV framework with location cues cuts AD risk error 33%","Where you live helps AI predict Alzheimer's risk 33% better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that each row of the public-health survey—a location, a question about older adults, and that location's aggregate percentage answer—can stand in for a patient-level clinical note with an Alzheimer's risk score.","fun_headline_variants_meta":{"raw":{"variants":["Location data cuts Alzheimer's risk-score error by 33%","Adding GPS coordinates to clinical notes improves AD prediction by 33%","Geolocation boosts Alzheimer's risk score accuracy by one-third in AI model","DMV framework with location cues cuts AD risk error 33%","Where you live helps AI predict Alzheimer's risk 33% better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2797,"prompt_tokens":886,"completion_tokens":1911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1819}},"tokens_in":502,"tokens_out":1911,"duration_ms":15120,"temperature":1.0,"reasoning_tokens":1819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:52:53.346722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the latitude and longitude pairs across rows while keeping the question text intact; if the near-zero mean squared error persists, the model is recovering the stored aggregate value from row-level identifiers rather than from language or location.","supporting_citations":[],"review_version":1}