{"id":"d3a5975c-99c6-4441-894a-4dfd8834b303","arxiv_id":"2411.16791","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-derived features outperform a mean-value baseline on all 41 city prediction tasks, but the evaluation lacks non-LLM baselines and error bars, so the practical gain is uncertain.","lead":"This study asks large language models to estimate 41 urban metrics, from air pollution to crime, either by direct questioning or by turning the model's internal states into prediction features. The authors find that LLM-based features consistently beat a baseline that simply predicts the average value, suggesting a cheap source of rough estimates for cities that lack official data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is only benchmarked against a mean baseline; simple demographic/geographic features (population, GDP, coordinates) are never tested and could explain the observed LLM-feature gains.","rationale":"The most load-bearing condition for the central claim is that the observed RMSE gains are attributable to city knowledge encoded in LLMs rather than to generic correlates that are trivially available without an LLM. The paper's only experimental control is the No-Feature mean baseline (Table 1). Since the LLM-extracted features include standard indicators such as population, GDP, and built-up area, the improvement over the mean could be reproduced by a simple regressor on those variables. The paper even states in Related Work that it aims to show LLMs achieve comparable accuracy to traditional feature engineering, yet no such traditional baseline is included. This is not a disagreement with consensus about LLM knowledge; it is an internal gap between the claim and the experimental design. The proposed concrete test is feasible: the datasets are public, and the same ML pipeline can be rerun on simple features. If simple features match the LLM features, the paper's headline should be downgraded to a more modest claim about convenience or scalability rather than about unique knowledge. The reader's weakest_assumption identified exactly this missing baseline, so we agree. The model-selection inflation noted in the reader's rationale is a secondary concern but not the single most load-bearing one.","tokens_in":22516,"tokens_out":4373,"duration_ms":199133,"concrete_test":"Reproduce Table 1 with the same 5-fold CV and model-selection protocol, using simple non-LLM baseline features constructed from open data: population, GDP, built-up area, latitude, longitude, and region or country dummies, for all 41 tasks. Compare RMSE of these baselines with Exp-Feature and Imp-Feature. If the simple-feature RMSE is within 5% of the LLM-feature RMSE on most tasks, or beats it, the central claim is unsupported; if the LLM features are clearly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 compares Exp-Feature, Imp-Feature, and Direct-Ask exclusively to No-Feature (the target mean). The abstract and conclusion claim that 'ML models trained on LLM-derived features consistently lead to improved predictive accuracy,' but the only comparator cannot distinguish LLM-specific knowledge from trivial correlates. Many of the paper's own explicit feature sets are standard demographics: Carbon emission and Energy tasks use Population, GDP, Built-up Area (Tables 6, 10-12); Water withdrawal uses Population and Economy (Table 9); Industry uses Population, Economy, Infrastructure, Technology (Tables 22-25). A model trained on just these variables, which are publicly available for most cities and require no LLM, could plausibly match or beat the LLM features, especially on tasks with strong population or economy scaling such as Total energy, Total industry, and Carbon emission. The paper's stated goal in Related Work that LLMs can achieve 'comparable accuracy to traditional feature engineering approaches' is never tested, because no traditional feature-engineering baseline appears in Table 1. Without it, the central causal claim that LLMs contribute knowledge about cities is unsupported; the improvement over the mean could be entirely mediated by variables the LLM is simply regurgitating.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates what large language models (LLMs) know about cities and regions by comparing three ways of using LLMs—directly asking for the target value (Direct-Ask), asking for explicit feature values (Exp-Feature), and using the LLM's last hidden states as implicit features (Imp-Feature)—against a No-Feature baseline that predicts the target mean. Experiments cover 41 tasks in 8 domains at city and region levels, with RMSE as the primary metric. The central claim is that ML models trained on LLM-derived features consistently improve predictive accuracy over the No-Feature baseline, and that LLMs show broad but varying knowledge across global cities, sometimes producing generic or random outputs when they lack knowledge.","tokens_in":22774,"tokens_out":4384,"duration_ms":68574,"significance":"If the central claim holds, the paper offers a scalable, low-cost method for obtaining proxy features for urban variables in data-scarce regions, which would be a useful contribution to computational urban science. The paper has notable strengths: it spans a broad range of tasks (environment, energy, health, transport, crime, industry) and geographical levels, compares explicit and implicit feature extraction, tests both open-source and API-based LLMs, and candidly investigates cases where LLMs produce generic or inconsistent outputs. However, the evaluation design does not currently support the headline claim because it lacks non-LLM feature baselines, uses an optimistic model-selection protocol, and reports no uncertainty measures.","major_comments":[{"comment":"The central claim that LLM-derived features 'consistently lead to improved predictive accuracy' is evaluated only against the No-Feature (mean) baseline. Many explicit feature sets listed in the task descriptions are standard demographics—e.g., Population, GDP, and Built-up Area for Carbon emission and Energy (Tables 6, 10–12), Population and Economy for Water withdrawal (Table 9), and Population, Economy, Infrastructure, and Technology for Industry (Tables 22–25). Without comparing against a model trained on such non-LLM features (population, GDP, coordinates), the observed gains cannot be attributed to LLM knowledge; they could be mediated by variables the LLM is merely regurgitating. The Related Work goal of achieving 'comparable accuracy to traditional feature engineering approaches' is never tested because no traditional feature-engineering baseline appears in Table 1.","section":"Overall Results; Table 1"},{"comment":"The paper reports that 'A set of frequently used ML models are tried including Decision Tree, Random Forest, Gradient Boosting, XGBoost, AdaBoost, and Linear Regressor. The best results are reported' under 5-fold cross-validation. Selecting the best model based on test-fold performance without an outer validation loop or nested cross-validation produces optimistically biased RMSE values and makes the comparison to No-Feature unfair, since No-Feature has no model selection. Please use nested cross-validation, a fixed model, or report the selection procedure and variance across folds.","section":"Experiment Settings"},{"comment":"Several Direct-Ask entries for Llama3.1-8B are listed as 'err' (e.g., Methane, Water withdrawal, Total energy, Traditional energy, Renewable energy, Dengue, Total industry, Mining, Manufacture, Utilities, Patent, Material stocks). The caption says 'err' indicates answers entirely unrelated to the question, but the paper does not state how many queries produced such outputs, whether these entries are excluded from RMSE computation, or how this affects the comparison. Since these are mostly Chinese-city tasks, the claim that LLMs know something about everywhere requires explicit handling of these failures, such as reporting failure rates and analysis of the failures.","section":"Table 1"},{"comment":"The conclusion states that feature-based ML models achieve '18% improvements over the No-Feature method,' but this figure does not appear in Table 1 or elsewhere, and no calculation is provided. If this aggregate improvement is reported, it should be derived from the per-task results (e.g., average relative RMSE reduction with a confidence interval), or the sentence should be revised to state the range of improvements.","section":"Conclusion"},{"comment":"All RMSE values in Table 1 are point estimates without error bars, confidence intervals, or significance tests across cross-validation folds or LLM sampling. Many differences between methods are small (e.g., tasks 29–34, where Exp-Feature and Imp-Feature differ by less than 0.1 on a 0–10 scale), and without measures of uncertainty the claim of 'consistently' improved accuracy is not supported. Paired tests (e.g., Wilcoxon signed-rank over tasks) or per-fold standard errors are needed.","section":"Overall Results"}],"minor_comments":[{"comment":"The paper writes that 'GPT-4o performs better in general,' but Table 1 shows several tasks where Llama3.1-8B Exp-Feature has lower RMSE than GPT-4o Exp-Feature (e.g., tasks 5, 9, 29–34); please qualify this statement.","section":"Overall Results; Table 1"},{"comment":"Two references are placeholders: '(?)' appears for Household CO2 and for Patent. These citations need to be completed.","section":"Task Descriptions"},{"comment":"The 're-scaled deviation' in Figure 7 is not defined; please state how it is computed from the 100 queries.","section":"Q1; Figure 7"},{"comment":"The description says a concatenated mean-max pooling and linear transformation are used, but it does not specify how the linear transformation is trained. If it is learned on the target, this could leak information and must be clarified.","section":"Implicit Feature Extraction"},{"comment":"There are several typos, including 'removed from thes dataset' in the Carbon emission preprocessing, and inconsistent capitalization in feature names (e.g., 'A vg home value' in Table 1).","section":"Data Preprocess; Carbon emission"},{"comment":"Table 4 reports only two tasks for the language comparison; the claim that 'Chinese and English are close in performance for Chinese cities' would be stronger with more tasks and significance testing.","section":"Q4; Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and addresses an interesting question, but the evaluation currently lacks the controls needed to support the central causal claim. The most important fix is to add traditional feature baselines (population, GDP, built-up area, coordinates) to Table 1; without this, the contribution is not established. The model-selection protocol and lack of uncertainty measures are also serious. I would be willing to review a revised version that adds these experiments and revises the claims accordingly. No concerns about citation patterns beyond the placeholder references mentioned in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this is a broad, honestly presented benchmark of LLM knowledge about cities, and the headline result—features extracted from LLMs beat the target mean—is real, but only against the mean. The paper never tests a non-LLM feature set, so the stronger claim that LLMs contribute distinct city knowledge is not actually supported.\n\nThe new stuff: 41 tasks across 8 domains, city-level and region-level, covering environment, energy, transport, crime, and more. That breadth is new. So is the interactive explicit-feature framework where the LLM proposes feature names then scores each city. The paper also documents a useful failure mode: when the LLM doesn't know, it outputs generic round numbers (5 and 50 for 94% of Mining queries). The per-task correlation tables in the appendix are a genuine resource. The held-out fold evaluation for the feature-based methods is non-circular, which is more than you get from a lot of papers in this area.\n\nThe soft spots are real. The only baseline is the target mean. Many of the explicit features are standard demographics—population, GDP, built-up area—which are publicly available for most of these cities. A simple regression on log population and GDP could plausibly match or beat the LLM features on energy, carbon, and industry tasks. The paper's related-work claim about 'comparable accuracy to traditional feature engineering' is exactly what it doesn't test. That is the core weakness and it deserves a fix, not just a caveat. Second, the ML evaluation selects the best of six models per task without nested CV; that inflates accuracy and the paper reports no variance. Third, Llama's Direct-Ask returns 'err' on many Chinese datasets and the paper doesn't explain why. Fourth, no code or data release, which limits reuse even though most input datasets are public.\n\nNone of this makes the paper worthless. The descriptive result—LLM features can be useful signals—survives, and the benchmark could be a starting point for follow-up work. But the central causal claim needs better baselines before I'd trust the comparison.\n\nWho it's for: researchers working on LLM-as-feature-extractor or city-level prediction in data-scarce settings. It deserves a serious referee, but the referee should demand non-LLM baselines (population, GDP, coordinates, maybe a couple of engineered features), nested CV, and error bars.","headline":"A broad, useful benchmark of what LLMs know about cities, but the headline claim is only tested against the target mean, so the case that LLM features add real knowledge is not yet made.","tokens_in":23285,"tokens_out":2919,"would_cite":false,"duration_ms":27855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs carry usable knowledge about cities: ML models trained on LLM-extracted features consistently beat a mean baseline on 41 urban prediction tasks.","keywords":["large language models","city knowledge","feature extraction","urban prediction","global cities","zero-shot estimation","data-scarce regions","LLM uncertainty"],"falsifier":"Run the same 41 tasks with simple engineered features (population, GDP, coordinates, country or region dummies) as the baseline. If those features match or beat the LLM-derived feature models in RMSE, then the paper's conclusion that LLMs provide predictive city knowledge would lose its support, because every gain could be explained by the LLM having recovered only these basic correlates.","tokens_in":22356,"feed_emoji":"🏙️","tokens_out":5508,"duration_ms":48875,"temperature":0.7,"pith_summary":"This paper asks what large language models know about cities and whether that knowledge can be converted into predictive features for urban variables. Across 41 tasks covering environment, energy, health, transport, industry, crime, and city services, the authors show that machine-learning models trained on LLM-extracted features—either explicit scored attributes named by the LLM or compressed hidden-layer representations—consistently beat the mean-value baseline, with an average improvement around 18 percent. The paper argues that LLMs therefore know something about every continent and can serve as a low-cost, scalable source of city knowledge, particularly for places that publish little or no data. It also demonstrates that when an LLM lacks knowledge it tends to emit generic placeholder values or unstable answers, which provides a visible warning flag.","feed_headline":"LLM features beat mean baseline on 41 city tasks","feed_subtitle":"Models trained on LLM-extracted city knowledge cut prediction error, giving a cheap route to data-scarce regions.","key_machinery":"The load-bearing mechanism is the LLM-as-feature-extractor. Explicit feature extraction asks the model to name relevant factors and score each city on a 0–10 scale, while implicit feature extraction takes the model's last hidden layer, pools it with mean-max operations, projects it to 32 dimensions, and feeds the result to a standard regressor. What carries the argument is the observation that the LLM's ordinal information—which cities rank higher on a factor—survives even when its absolute numbers are off, so the ML model can recalibrate the rough scores into accurate predictions.","core_discovery":"The paper's central claim is that LLMs embed usable knowledge about cities worldwide, and that this knowledge can be harvested for prediction. On the forty-one tasks, the best LLM-derived feature method (explicit or implicit) outperforms the No-Feature baseline on nearly every task, while direct queries win on popular datasets whose values likely appear in pretraining. The reason feature-based methods work is that LLM-generated feature values preserve the relative ordering of cities even when the exact numbers are wrong, so a trained regressor can map those rough scores to real scales. The authors further claim that this holds across all continents, and that the failure mode is identifiable: repetitive round numbers or high variance across repeated queries mark tasks the LLM does not actually know.","pith_inferences":["The reported gains might largely come from the LLM recovering well-known correlates such as population, GDP, and climate; a proper test would compare against those variables directly.","For direct-ask wins on widely published datasets, memorization during pretraining is a plausible alternative to reasoning, so the feature-based results on obscure targets are the cleaner evidence.","Output variance and placeholder-value detection could be converted into per-city uncertainty weights, letting a downstream model down-weight exactly the cities the LLM does not know.","The language experiment hints that matching the prompt language to the region (Chinese for Chinese cities) narrows the gap with English; testing this on more languages and regions could give practical prompting guidance."],"forward_implications":["Urban researchers could generate proxy variables for cities that release no data, extending cross-city studies from a few dozen well-documented places to global coverage.","For familiar targets like air quality or home values, querying the LLM directly is competitive; for obscure targets, feature extraction is the more reliable route.","Explicit features work on any LLM and are recommended, since closed models like GPT-4o do not expose hidden layers.","The consistency of LLM answers across repeated queries can be used as a confidence signal, telling analysts when to trust an estimate and when to collect real data.","LLM-derived features can support cheap hypothesis testing before committing resources to official data collection."],"supporting_citations":[{"why":"Supplies the geospatial-knowledge extraction approach that this paper adapts from global regions to cities.","marker":"(Manvi et al. 2023)"},{"why":"Provides the People Near Transit dataset for 85 OECD cities used in the transport tasks and motivating example.","marker":"(Verbavatz and Barthelemy 2020)"},{"why":"Provides the global CO2 emission dataset for 343 cities used in the carbon-emission task.","marker":"(Nangini et al. 2019)"},{"why":"Supplies crime-relevant demographic and POI features and the Chicago nightlife data used for feature comparison.","marker":"(Wang et al. 2016)"},{"why":"Provides the city-level livestock methane emission dataset for China used in the methane task and LLM model comparison.","marker":"(Du et al. 2024)"},{"why":"Provides city-level water withdrawal and industry output datasets for Chinese cities used in multiple tasks.","marker":"(Zhang et al. 2024b)"}],"fun_headline_variants":["LLM features beat baselines on city prediction tasks","LLMs reveal urban knowledge hidden in language","City forecasts improved with LLM-derived features","Harness LLM knowledge for global city predictions","LLM insight: better data for cities you lack info on"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on comparing against a baseline that predicts the mean for every city; if a cheap set of ordinary features such as population, GDP, or geographic coordinates does not appear in the comparison, the improved RMSE cannot by itself prove that LLMs contribute genuine city knowledge.","fun_headline_variants_meta":{"raw":{"variants":["LLM features beat baselines on city prediction tasks","LLMs reveal urban knowledge hidden in language","City forecasts improved with LLM-derived features","Harness LLM knowledge for global city predictions","LLM insight: better data for cities you lack info on"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1251,"prompt_tokens":812,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":428,"tokens_out":439,"duration_ms":5118,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:27.215622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 41 tasks with simple engineered features (population, GDP, coordinates, country or region dummies) as the baseline. If those features match or beat the LLM-derived feature models in RMSE, then the paper's conclusion that LLMs provide predictive city knowledge would lose its support, because every gain could be explained by the LLM having recovered only these basic correlates.","supporting_citations":[{"cited_title":"2019) pro- vides a comprehensive compilation of anthropogenic CO 2 emission data for 343 cities worldwide","cited_arxiv_id":null,"evidence_quote":"Provides the global CO2 emission dataset for 343 cities used in the carbon-emission task."}],"review_version":1}