What can LLM tell us about cities?

Guanjie Zheng; Rui Bao; Yaochen Wang; Yuqi Huang; Zhenhui Jessie Li; Zhixue Song; Zhuoheng Li

REVIEW 5 major objections 6 minor 12 references

What can LLM tell us about cities?

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs carry usable knowledge about cities: ML models trained on LLM-extracted features consistently beat a mean baseline on 41 urban prediction tasks.

desk verdict A broad, useful benchmark of what LLMs know about cities, but the headline claim is only tested against the target mean, so the case that LLM features add real knowledge is not yet made. read the letter →

arxiv 2411.16791 v1 pith:X45SI253 submitted 2024-11-25 cs.CL cs.AIcs.IR

Zhuoheng Li , Yaochen Wang , Zhixue Song , Yuqi Huang , Rui Bao , Guanjie Zheng , Zhenhui Jessie Li This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords largelanguagemodelscityknowledgefeatureextractionurbanpredictionglobalcitieszero-shotestimationdata-scarceregionsLLMuncertainty

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what large language models know about cities and whether that knowledge can be converted into predictive features for urban variables. Across 41 tasks covering environment, energy, health, transport, industry, crime, and city services, the authors show that machine-learning models trained on LLM-extracted features—either explicit scored attributes named by the LLM or compressed hidden-layer representations—consistently beat the mean-value baseline, with an average improvement around 18 percent. The paper argues that LLMs therefore know something about every continent and can serve as a low-cost, scalable source of city knowledge, particularly for places that publish little or no data. It also demonstrates that when an LLM lacks knowledge it tends to emit generic placeholder values or unstable answers, which provides a visible warning flag.

What carries the argument

The load-bearing mechanism is the LLM-as-feature-extractor. Explicit feature extraction asks the model to name relevant factors and score each city on a 0–10 scale, while implicit feature extraction takes the model's last hidden layer, pools it with mean-max operations, projects it to 32 dimensions, and feeds the result to a standard regressor. What carries the argument is the observation that the LLM's ordinal information—which cities rank higher on a factor—survives even when its absolute numbers are off, so the ML model can recalibrate the rough scores into accurate predictions.

What would settle it

Run the same 41 tasks with simple engineered features (population, GDP, coordinates, country or region dummies) as the baseline. If those features match or beat the LLM-derived feature models in RMSE, then the paper's conclusion that LLMs provide predictive city knowledge would lose its support, because every gain could be explained by the LLM having recovered only these basic correlates.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs embed usable knowledge about cities worldwide, and that this knowledge can be harvested for prediction. On the forty-one tasks, the best LLM-derived feature method (explicit or implicit) outperforms the No-Feature baseline on nearly every task, while direct queries win on popular datasets whose values likely appear in pretraining. The reason feature-based methods work is that LLM-generated feature values preserve the relative ordering of cities even when the exact numbers are wrong, so a trained regressor can map those rough scores to real scales. The authors further claim that this holds across all continents, and that the failure mode is identifiable: repetitive round numbers or high variance across repeated queries mark tasks the LLM does not actually know.

Load-bearing premise

The argument rests on comparing against a baseline that predicts the mean for every city; if a cheap set of ordinary features such as population, GDP, or geographic coordinates does not appear in the comparison, the improved RMSE cannot by itself prove that LLMs contribute genuine city knowledge.

Editorial extensions

If this is right

Urban researchers could generate proxy variables for cities that release no data, extending cross-city studies from a few dozen well-documented places to global coverage.
For familiar targets like air quality or home values, querying the LLM directly is competitive; for obscure targets, feature extraction is the more reliable route.
Explicit features work on any LLM and are recommended, since closed models like GPT-4o do not expose hidden layers.
The consistency of LLM answers across repeated queries can be used as a confidence signal, telling analysts when to trust an estimate and when to collect real data.
LLM-derived features can support cheap hypothesis testing before committing resources to official data collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported gains might largely come from the LLM recovering well-known correlates such as population, GDP, and climate; a proper test would compare against those variables directly.
For direct-ask wins on widely published datasets, memorization during pretraining is a plausible alternative to reasoning, so the feature-based results on obscure targets are the cleaner evidence.
Output variance and placeholder-value detection could be converted into per-city uncertainty weights, letting a downstream model down-weight exactly the cities the LLM does not know.
The language experiment hints that matching the prompt language to the region (Chinese for Chinese cities) narrows the gap with English; testing this on more languages and regions could give practical prompting guidance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, what you should know: this is a broad, honestly presented benchmark of LLM knowledge about cities, and the headline result—features extracted from LLMs beat the target mean—is real, but only against the mean. The paper never tests a non-LLM feature set, so the stronger claim that LLMs contribute distinct city knowledge is not actually supported.

The new stuff: 41 tasks across 8 domains, city-level and region-level, covering environment, energy, transport, crime, and more. That breadth is new. So is the interactive explicit-feature framework where the LLM proposes feature names then scores each city. The paper also documents a useful failure mode: when the LLM doesn't know, it outputs generic round numbers (5 and 50 for 94% of Mining queries). The per-task correlation tables in the appendix are a genuine resource. The held-out fold evaluation for the feature-based methods is non-circular, which is more than you get from a lot of papers in this area.

The soft spots are real. The only baseline is the target mean. Many of the explicit features are standard demographics—population, GDP, built-up area—which are publicly available for most of these cities. A simple regression on log population and GDP could plausibly match or beat the LLM features on energy, carbon, and industry tasks. The paper's related-work claim about 'comparable accuracy to traditional feature engineering' is exactly what it doesn't test. That is the core weakness and it deserves a fix, not just a caveat. Second, the ML evaluation selects the best of six models per task without nested CV; that inflates accuracy and the paper reports no variance. Third, Llama's Direct-Ask returns 'err' on many Chinese datasets and the paper doesn't explain why. Fourth, no code or data release, which limits reuse even though most input datasets are public.

None of this makes the paper worthless. The descriptive result—LLM features can be useful signals—survives, and the benchmark could be a starting point for follow-up work. But the central causal claim needs better baselines before I'd trust the comparison.

Who it's for: researchers working on LLM-as-feature-extractor or city-level prediction in data-scarce settings. It deserves a serious referee, but the referee should demand non-LLM baselines (population, GDP, coordinates, maybe a couple of engineered features), nested CV, and error bars.

Referee Report

5 major / 6 minor

Summary. The paper investigates what large language models (LLMs) know about cities and regions by comparing three ways of using LLMs—directly asking for the target value (Direct-Ask), asking for explicit feature values (Exp-Feature), and using the LLM's last hidden states as implicit features (Imp-Feature)—against a No-Feature baseline that predicts the target mean. Experiments cover 41 tasks in 8 domains at city and region levels, with RMSE as the primary metric. The central claim is that ML models trained on LLM-derived features consistently improve predictive accuracy over the No-Feature baseline, and that LLMs show broad but varying knowledge across global cities, sometimes producing generic or random outputs when they lack knowledge.

Significance. If the central claim holds, the paper offers a scalable, low-cost method for obtaining proxy features for urban variables in data-scarce regions, which would be a useful contribution to computational urban science. The paper has notable strengths: it spans a broad range of tasks (environment, energy, health, transport, crime, industry) and geographical levels, compares explicit and implicit feature extraction, tests both open-source and API-based LLMs, and candidly investigates cases where LLMs produce generic or inconsistent outputs. However, the evaluation design does not currently support the headline claim because it lacks non-LLM feature baselines, uses an optimistic model-selection protocol, and reports no uncertainty measures.

major comments (5)

[Overall Results; Table 1] The central claim that LLM-derived features 'consistently lead to improved predictive accuracy' is evaluated only against the No-Feature (mean) baseline. Many explicit feature sets listed in the task descriptions are standard demographics—e.g., Population, GDP, and Built-up Area for Carbon emission and Energy (Tables 6, 10–12), Population and Economy for Water withdrawal (Table 9), and Population, Economy, Infrastructure, and Technology for Industry (Tables 22–25). Without comparing against a model trained on such non-LLM features (population, GDP, coordinates), the observed gains cannot be attributed to LLM knowledge; they could be mediated by variables the LLM is merely regurgitating. The Related Work goal of achieving 'comparable accuracy to traditional feature engineering approaches' is never tested because no traditional feature-engineering baseline appears in Table 1.
[Experiment Settings] The paper reports that 'A set of frequently used ML models are tried including Decision Tree, Random Forest, Gradient Boosting, XGBoost, AdaBoost, and Linear Regressor. The best results are reported' under 5-fold cross-validation. Selecting the best model based on test-fold performance without an outer validation loop or nested cross-validation produces optimistically biased RMSE values and makes the comparison to No-Feature unfair, since No-Feature has no model selection. Please use nested cross-validation, a fixed model, or report the selection procedure and variance across folds.
[Table 1] Several Direct-Ask entries for Llama3.1-8B are listed as 'err' (e.g., Methane, Water withdrawal, Total energy, Traditional energy, Renewable energy, Dengue, Total industry, Mining, Manufacture, Utilities, Patent, Material stocks). The caption says 'err' indicates answers entirely unrelated to the question, but the paper does not state how many queries produced such outputs, whether these entries are excluded from RMSE computation, or how this affects the comparison. Since these are mostly Chinese-city tasks, the claim that LLMs know something about everywhere requires explicit handling of these failures, such as reporting failure rates and analysis of the failures.
[Conclusion] The conclusion states that feature-based ML models achieve '18% improvements over the No-Feature method,' but this figure does not appear in Table 1 or elsewhere, and no calculation is provided. If this aggregate improvement is reported, it should be derived from the per-task results (e.g., average relative RMSE reduction with a confidence interval), or the sentence should be revised to state the range of improvements.
[Overall Results] All RMSE values in Table 1 are point estimates without error bars, confidence intervals, or significance tests across cross-validation folds or LLM sampling. Many differences between methods are small (e.g., tasks 29–34, where Exp-Feature and Imp-Feature differ by less than 0.1 on a 0–10 scale), and without measures of uncertainty the claim of 'consistently' improved accuracy is not supported. Paired tests (e.g., Wilcoxon signed-rank over tasks) or per-fold standard errors are needed.

minor comments (6)

[Overall Results; Table 1] The paper writes that 'GPT-4o performs better in general,' but Table 1 shows several tasks where Llama3.1-8B Exp-Feature has lower RMSE than GPT-4o Exp-Feature (e.g., tasks 5, 9, 29–34); please qualify this statement.
[Task Descriptions] Two references are placeholders: '(?)' appears for Household CO2 and for Patent. These citations need to be completed.
[Q1; Figure 7] The 're-scaled deviation' in Figure 7 is not defined; please state how it is computed from the 100 queries.
[Implicit Feature Extraction] The description says a concatenated mean-max pooling and linear transformation are used, but it does not specify how the linear transformation is trained. If it is learned on the target, this could leak information and must be clarified.
[Data Preprocess; Carbon emission] There are several typos, including 'removed from thes dataset' in the Carbon emission preprocessing, and inconsistent capitalization in feature names (e.g., 'A vg home value' in Table 1).
[Q4; Table 4] Table 4 reports only two tasks for the language comparison; the claim that 'Chinese and English are close in performance for Chinese cities' would be stronger with more tasks and significance testing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LLM-feature predictions are evaluated on held-out folds against external datasets.

full rationale

The paper's predictive claims are generated from LLM features on held-out folds: Exp-Feature and Imp-Feature are fed into ML models evaluated with 5-fold cross-validation, so the target values of test cities are not seen during training. The LLM features are elicited with prompts that include the target variable name but not the target values, and the mapping from features to targets is learned only on training folds. Thus the reported RMSE reductions over the No-Feature mean are not equivalent by construction to the input. Direct-Ask results on familiar datasets are acknowledged memorization ("LLM directly knows the answer"), which is a data-contamination issue, not circularity. The sole use of a mean baseline and the absence of a traditional-feature baseline weaken the external validity of the central claim, but that is a comparison/rigor limitation, not a self-referential derivation. The only same-author citation is the use of crime features from Wang et al. (2016) for the crime tasks; that prior empirical feature set is not a uniqueness theorem and does not force the LLM prediction, so it is not load-bearing circularity. No step in the paper reduces a predicted quantity to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The ledger captures the evaluation choices that support the empirical claim: the model selection rule, the underspecified projection, and the assumptions about memorization and error-bar-free RMSE.

free parameters (2)

ML model selection = best of 6 models
The paper tries Decision Tree, Random Forest, Gradient Boosting, XGBoost, AdaBoost, and Linear Regressor and reports the best RMSE, which is a form of tuning that inflates reported performance.
Implicit feature projection dimension = 32
The hidden states are pooled and linearly projected to 32 dimensions; the paper does not specify whether this projection is trained unsupervised or supervised, so it may introduce an extra fitted component.

assumptions (3)

domain assumption LLM has not memorized the target datasets for most tasks
The paper relies on the idea that the LLM cannot retrieve exact target values for less popular tasks, but it acknowledges direct memorization for popular datasets (PM2.5, COVID-19, home values). No contamination test is performed.
domain assumption Cross-validated selection of the best of six ML models yields unbiased performance
Choosing the best model per task without a nested procedure or a fixed selection rule can lead to optimistic RMSE estimates.
domain assumption RMSE differences are meaningful without significance testing
No error bars, standard deviations, or statistical tests are reported, so it is unclear if differences are reliable.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What can LLM tell us about cities?." pith.science (2026). https://pith.science/paper/X45SI253

@misc{pith2026241116791,
  author       = {Pith},
  title        = {Pith review of: What can LLM tell us about cities?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X45SI253}},
  note         = {Machine review of arXiv:2411.16791}
}

read the original abstract

This study explores the capabilities of large language models (LLMs) in providing knowledge about cities and regions on a global scale. We employ two methods: directly querying the LLM for target variable values and extracting explicit and implicit features from the LLM correlated with the target variable. Our experiments reveal that LLMs embed a broad but varying degree of knowledge across global cities, with ML models trained on LLM-derived features consistently leading to improved predictive accuracy. Additionally, we observe that LLMs demonstrate a certain level of knowledge across global cities on all continents, but it is evident when they lack knowledge, as they tend to generate generic or random outputs for unfamiliar tasks. These findings suggest that LLMs can offer new opportunities for data-driven decision-making in the study of cities.

Figures

Figures reproduced from arXiv: 2411.16791 by the authors.

**Figure 2.** Public transportation data are collected from vari [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** A strong correlation between public transporta [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Features of a specific task are extracted from the [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Extract relevant features impacting city-level en [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 7.** Figure 7: Re-scaled deviation of each sample in 100 queries [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 6.** Figure 6: Query results on Mining for 245 cities. 93.9% [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 8.** Figure 8: Count statistics of the public transportation score [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 9.** Figure 9: Similarity is being observed on Chicago Nightlife [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[5]

2024) presents a comprehensive city-level inventory of livestock methane emission in China from 2010 to 2020, incorporating biolog- ical, management, and environmental variables

Methane Data Description This dataset (Du et al. 2024) presents a comprehensive city-level inventory of livestock methane emission in China from 2010 to 2020, incorporating biolog- ical, management, and environmental variables. The data is measured in tonnes. Data Preprocess The methane emission from livestock, measured in tons per year, are analyzed usin...

work page 2024
[6]

2019) pro- vides a comprehensive compilation of anthropogenic CO 2 emission data for 343 cities worldwide

Carbon emission Data Description This dataset (Nangini et al. 2019) pro- vides a comprehensive compilation of anthropogenic CO 2 emission data for 343 cities worldwide. It integrates data Table 5: Correlation and p-value of Methane features. Features Correlation p-value livestock density 0.2377 0.0000 feeding practices -0.0517 0.3368 manure management -0....

work page 2019
[7]

2021) comprises an emission inventory of urban house- hold CO2-equivalent emissions for 52 major cities in Japan

Household CO 2 Data Description The dataset utilized in this study (Long et al. 2021) comprises an emission inventory of urban house- hold CO2-equivalent emissions for 52 major cities in Japan. It includes fossil fuels related to direct household emis- sions such as gasoline, kerosene, liquefied petroleum gas, and city gas. These emissions are calculated ...

work page 2021
[8]

2020) provides CO2 emissions data for Russia’s 82 constituent entities from 2005 to 2019

Total CO2 Data Description The dataset (Xiao et al. 2020) provides CO2 emissions data for Russia’s 82 constituent entities from 2005 to 2019. For this experiment, the total CO 2 emissions of each entity in 2019 are selected, with emissions measured in million tonnes. Data Preprocess The target variable is calculated by sum- ming the energy-related and pro...

work page 2020
[9]

2024b) is used for this experiment

Water withdrawal Data Description The City-level water withdrawal and scarcity accounts of China dataset (Zhang et al. 2024b) is used for this experiment. The data is measured in 100 mil- lion cubic meters (100 million m3). Data Preprocess The dataset provides information on wa- ter withdrawal, availability, and criticality for different cities in China. ...

work page
[13]

2024), a global collection of publicly available dengue case counts from 1924 to 2023, covering 102 countries

Dengue Data Description This paper introduces the OpenDengue dataset(Clarke et al. 2024), a global collection of publicly available dengue case counts from 1924 to 2023, covering 102 countries. The dataset integrates data from 843 sources, emphasizing its high temporal (weekly/monthly) and spatial resolution. This resource aims to support research on deng...

work page 2024
[14]

The data is measured in counts

COVID-19 Data Description This dataset (CDC 2023) presents a provisional count of deaths involving COVID-19 by county of occurrence in the United States from 2020 to 2023. The data is measured in counts. Data Preprocess Among the six urban-rural codes (Non- core, Medium metro, Small metro, Micropolitan, Large fringe metro, Large central metro), only Large...

work page 2023
[15]

2023) is used to study the impact of environmental factors on health in English cities

Life expectancy Data Description This dataset (Han et al. 2023) is used to study the impact of environmental factors on health in English cities. It includes a variety of variables related to the environment and health, which can be used to analyze the relationship between health conditions in different cities and environmental factors. The data is measur...

work page 2023

Show all 12 references

[16]

Access to Healthy Assets and Hazards

Health access Data Description This dataset (Daras et al. 2019) pro- vides a comprehensive series of national open source low- level geographical measures of accessibility to various health-related features across Great Britain. It includes 14 measures categorized into three d...

2019
[26]

The geocoded data is further allocated to the correspond- ing countries, regions, and cities

Patent Data Description This dataset ( ?) provides geographic coordinates for inventor and applicant locations in 18.8 million patent documents spanning more than 30 years. The geocoded data is further allocated to the correspond- ing countries, regions, and cities. The data i...

2014
[27]

A vg home value Data Description This dataset (Zillow 2024) includes housing data with home value information derived from Table 26: Correlation and p-value of Patent features. Features Correlation p-value population 0.0645 0.1556 education 0.1497 0.0009 economy 0.1814 0.0001 ...

2024
[28]

2020” col- umn in the “Material stock and population 1978-2020

Material stocks Data Description This paper (Li et al. 2023) introduces a dataset that contains the stock of 24 materials contained in 10 types of products, buildings, and infrastructure in all 332 prefecture-level cities in China from 1978 to 2020. We se- lect the per capita ...

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [5]

2024) presents a comprehensive city-level inventory of livestock methane emission in China from 2010 to 2020, incorporating biolog- ical, management, and environmental variables

Methane Data Description This dataset (Du et al. 2024) presents a comprehensive city-level inventory of livestock methane emission in China from 2010 to 2020, incorporating biolog- ical, management, and environmental variables. The data is measured in tonnes. Data Preprocess The methane emission from livestock, measured in tons per year, are analyzed usin...

work page 2024

[2] [6]

2019) pro- vides a comprehensive compilation of anthropogenic CO 2 emission data for 343 cities worldwide

Carbon emission Data Description This dataset (Nangini et al. 2019) pro- vides a comprehensive compilation of anthropogenic CO 2 emission data for 343 cities worldwide. It integrates data Table 5: Correlation and p-value of Methane features. Features Correlation p-value livestock density 0.2377 0.0000 feeding practices -0.0517 0.3368 manure management -0....

work page 2019

[3] [7]

2021) comprises an emission inventory of urban house- hold CO2-equivalent emissions for 52 major cities in Japan

Household CO 2 Data Description The dataset utilized in this study (Long et al. 2021) comprises an emission inventory of urban house- hold CO2-equivalent emissions for 52 major cities in Japan. It includes fossil fuels related to direct household emis- sions such as gasoline, kerosene, liquefied petroleum gas, and city gas. These emissions are calculated ...

work page 2021

[4] [8]

2020) provides CO2 emissions data for Russia’s 82 constituent entities from 2005 to 2019

Total CO2 Data Description The dataset (Xiao et al. 2020) provides CO2 emissions data for Russia’s 82 constituent entities from 2005 to 2019. For this experiment, the total CO 2 emissions of each entity in 2019 are selected, with emissions measured in million tonnes. Data Preprocess The target variable is calculated by sum- ming the energy-related and pro...

work page 2020

[5] [9]

2024b) is used for this experiment

Water withdrawal Data Description The City-level water withdrawal and scarcity accounts of China dataset (Zhang et al. 2024b) is used for this experiment. The data is measured in 100 mil- lion cubic meters (100 million m3). Data Preprocess The dataset provides information on wa- ter withdrawal, availability, and criticality for different cities in China. ...

work page

[6] [13]

2024), a global collection of publicly available dengue case counts from 1924 to 2023, covering 102 countries

Dengue Data Description This paper introduces the OpenDengue dataset(Clarke et al. 2024), a global collection of publicly available dengue case counts from 1924 to 2023, covering 102 countries. The dataset integrates data from 843 sources, emphasizing its high temporal (weekly/monthly) and spatial resolution. This resource aims to support research on deng...

work page 2024

[7] [14]

The data is measured in counts

COVID-19 Data Description This dataset (CDC 2023) presents a provisional count of deaths involving COVID-19 by county of occurrence in the United States from 2020 to 2023. The data is measured in counts. Data Preprocess Among the six urban-rural codes (Non- core, Medium metro, Small metro, Micropolitan, Large fringe metro, Large central metro), only Large...

work page 2023

[8] [15]

2023) is used to study the impact of environmental factors on health in English cities

Life expectancy Data Description This dataset (Han et al. 2023) is used to study the impact of environmental factors on health in English cities. It includes a variety of variables related to the environment and health, which can be used to analyze the relationship between health conditions in different cities and environmental factors. The data is measur...

work page 2023