Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A knowledge-graph benchmark exposes geospatial hallucination in 20 LLMs, and a dynamic fine-tuning method cuts errors by 29.6%.

desk verdict Useful geospatial hallucination benchmark with a solid taxonomy, but the DynamicKTO gain is inflated by same-benchmark beta selection and some tasks look broken. read the letter →

arxiv 2507.19586 v1 pith:XOV4OYFX submitted 2025-07-25 cs.CL cs.AIcs.LG

Shengyuan Wang , Jie Feng , Tianhui Liu , Dan Pei , Yong Li This is my paper
classification cs.CLcs.AIcs.LG
keywords geospatialhallucinationlargelanguagemodelsbenchmarkknowledgegraphKahneman-Tverskyoptimizationdynamicbetafactualityalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Geospatial hallucinations—LLMs asserting places, roads, or spatial relations that do not exist, omitting ones that do, or mixing up attributes like area and category—are common enough to threaten any LLM-based geospatial application, and the paper sets out to show they can be measured and reduced. It constructs a city knowledge graph with points of interest, areas, and roads, uses it to generate multiple-choice questions whose options map to five hallucination types, and reports that 20 leading LLMs score low on the resulting benchmark, often near or below random on individual tasks. It then proposes DynamicKTO, a fine-tuning method that changes the risk-aversion parameter of Kahneman-Tversky Optimization according to whether a training item tests entities, relations, or attributes, and reports a 29.6% improvement over the best fixed-parameter KTO on the abstain-enabled version. The paper claims this brings Llama3.1-8B from 0.3505 to 0.4858 overall accuracy, making a small open model competitive with the best proprietary models it measured.

What carries the argument

The load-bearing object is GEOHALU BENCH, a multiple-choice benchmark generated from SpatialKG, a knowledge graph of points of interest (POIs), areas of interest (AOIs), and roads with relations and attributes. Each question's distractors are tied to hallucination types: fabricated entities are LLM-generated names vetted against the graph, omitted entities are represented by a 'None of the other options' choice, and wrong attribute values are sampled beyond a threshold. The training-side mechanism is DynamicKTO, which keeps KTO's binary desirable/undesirable objective but replaces the fixed $\beta$ with $\beta(x) = \mathrm{Type2Beta}(x)$, set to 0.1 for entity items, 0.3 for relation items, and 0.5 for attribute items. This lets the optimizer tighten or relax alignment per knowledge type rather than applying one risk policy to all data.

What would settle it

Take a random sample of items where a model chose 'None of the other options' and verify against an independent, more complete map source whether the omitted entity truly exists; if a meaningful share of such cases is actually present in the world, then the omission labels—and the accuracy gains built on them—are artifacts of knowledge-graph gaps rather than genuine hallucinations.

Watch

Extended reading notes

Core claim

The paper's central claim is that geospatial hallucination is a distinct, measurable failure mode of LLMs and that a benchmark plus a dynamically parameterized fine-tuning objective can substantially reduce it. On the benchmark, the entity/relation/attribute taxonomy identifies five error types (fabrication and omission for entities and relations, plus attribute confusion), and evaluation across three major cities and three underrepresented cities shows low and uneven accuracy, with omission generally exceeding fabrication. The paper then demonstrates that fixing one risk-aversion value for the entire KTO loss is a bottleneck: because entity, relation, and attribute items differ in difficulty, a per-category beta (0.1, 0.3, 0.5) produces the best result. DynamicKTO lifts Llama3.1-8B's overall score to 0.4858 on the abstain version, a 38.61% relative gain over no fine-tuning and 29.60% over the best fixed-beta KTO, while preserving performance on general benchmarks and on the CityGPT urban-task evaluation.

Load-bearing premise

The load-bearing premise is that GEOHALU BENCH's labels are true: SpatialKG must be complete enough that a 'None of the other options' answer is genuinely an omission, and the distractors must be plausible but non-factual; if the graph misses real places, the benchmark mistakes the model's correct answer for a hallucination.

Editorial extensions

If this is right

  • If the reported gains hold, geospatial factuality can be treated as an alignable skill rather than an emergent property of model size.
  • A small open model can reach accuracy comparable to the best proprietary models on this benchmark, changing the cost equation for trustworthy geospatial applications.
  • The dynamic-beta principle should transfer to any domain where training data splits into difficulty or risk classes, such as medicine, law, or finance.
  • Because omission dominates fabrication, models improved by knowledge injection rather than refusal training are likely to be more useful in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence is that retrieval-augmented generation over the same spatial knowledge graph could match or exceed the training-based gain at inference time, since most errors are omissions of world knowledge rather than confident fabrication.
  • A more stringent test would convert the multiple-choice task to open-form generation and check whether models still avoid fabricated places; the current format may undercount or overcount certain error types depending on option positioning.
  • The taxonomy-level beta tuning suggests a general recipe: compute a difficulty statistic on a sample, then assign beta by cluster; the paper tests this and finds category-level tuning better for geospatial data, but sample-level tuning may win in less structured domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents GEOHALUBENCH, a multiple-choice benchmark for geospatial hallucinations, constructed from a structured geospatial knowledge graph (SpatialKG) with a taxonomy covering entity fabrication/omission, relation fabrication/omission, and attribute confusion. The authors evaluate 20 LLMs on this benchmark, report generally low accuracy and a strong bias against underrepresented regions, and propose DynamicKTO, a variant of KTO in which the risk-aversion parameter beta is assigned per task category (0.1 for Entity, 0.3 for Relation, 0.5 for Attribute). On GEOHALUBENCH-Abstain, Llama3.1-8B fine-tuned with DynamicKTO reaches 0.4858 overall accuracy versus 0.3748 for the best fixed-beta KTO, a relative improvement of 29.6%, with only modest degradation on general benchmarks. The paper also builds Factual-CityGPT by applying DynamicKTO on CityGPT and reports improved hallucination scores with preserved urban task performance.

Significance. If the reported gains are robust, the paper makes a useful contribution: it is one of the first systematic benchmarks for geospatial hallucinations, it evaluates a broad set of 20 LLMs, and it proposes a simple, cost-effective extension of KTO with a task-dependent beta. The authors provide code and data, include a training-cost analysis, and show that the model retains general capabilities on IFEval, BBH, and MMLU. The central claim, however, depends on benchmark validity and on a fair comparison to tuned fixed-beta baselines; both need stronger evidence before the magnitude of the improvement can be accepted. With those issues addressed, the benchmark and method would be valuable to the hallucination mitigation community.

major comments (4)
  1. [§3.2, Table 3; §A.8] The three task-level beta values (0.1, 0.3, 0.5) are selected after inspecting LFactDistance and the model's performance on GEOHALUBENCH itself, with no held-out city or data split used for selection. The headline 29.6% gain is therefore computed relative to a three-point fixed-beta grid, and a tuned fixed-beta KTO or per-category selected betas could close much of the gap. Please add a validation protocol in which betas are chosen on one city (or one fold) and evaluated on a disjoint city (or fold), for example tuning on Beijing and testing on London and New York, to demonstrate that the dynamic assignment, rather than in-sample selection, drives the improvement.
  2. [§2.2.3, Table 13, Figure 5] The construct validity of the benchmark as a measure of geospatial hallucination needs stronger support. In Table 13, several tasks are at or near zero for essentially all models while the random baseline is 0.200 (e.g., AOI-Area: most models score 0.000-0.100, with only Qwen2.5-0.5B reaching 0.320), and Figure 5 shows that adding the 'Cannot Determine' option substantially reorders model rankings. Since 'None of the other options' is labeled as Entity or Relation Omission, a model that prefers a particular option letter or that systematically abstains can be credited or penalized without actually exhibiting the targeted hallucination. I request a per-task option-distribution analysis (proportion of A/B/C/'Cannot Determine' choices per model compared with the random baseline) and a refusal-rate-controlled breakdown of DynamicKTO's gains to rule out format bias and abstention effects.
  3. [§2.3.2, Table 3] The comparison to fixed-beta KTO uses only the grid beta in {0.1, 0.3, 0.5}, which is too coarse to support the claim that the dynamic per-task assignment, rather than the specific beta values, is responsible for the improvement. The dynamic-beta DPO variant (beta-DPO, Wu et al., 2024) is cited in Related Work but is not included as a baseline; it is the direct dynamic-beta analog of DPO and should be evaluated. Please add a denser beta sweep, per-category tuned fixed betas, and beta-DPO to the comparison.
  4. [§3.2, Tables 3 and 5] All fine-tuning results are reported as point estimates without repeated seeds or error bars, even though fine-tuning is stochastic and several paired differences are small (e.g., Relation accuracy: DynamicKTO 0.4256 vs ORPO 0.4320 in Table 3; Qwen2.5-7B improvements of roughly 0.02-0.06 in Table 5). Please report variance over at least three seeds or a paired significance test for the headline DynamicKTO versus fixed-beta KTO comparison; without this, the size of the claimed effect is not statistically grounded.
minor comments (5)
  1. [§A.8] The first bullet under 'Sample-Level' ends with 'based on its' and is an incomplete sentence; please complete the description of how beta is computed per sample.
  2. [Table 13] The row for Llama-3.1-8B-Instruct appears twice with identical values; remove the duplicate row.
  3. [Figure 4] The caption says models are 'listed at name order,' but the vertical order in the figure appears to follow a specific evaluation order; please clarify the ordering criterion in the caption.
  4. [§3.2] The claim that testing three beta values 'confirms that the performance boost is not merely due to trivial hyperparameter optimization' is too strong for a three-point grid; please soften the wording to reflect the limited sweep.
  5. [References and in-text citations] Several in-text citations are formatted inconsistently (e.g., 'Niels et al.(Mündler et al., 2023)'); please standardize the citation style throughout the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DynamicKTO's reported gains are empirical evaluations, not derivations from fitted parameters, and the benchmark's ground truth rests on external geodata.

full rationale

The central derivation is not circular. GEOHALU BENCH ground truth comes from SpatialKG, which is constructed from external OpenStreetMap and Foursquare data, not from the models being evaluated. The hallucination taxonomy is a definitional framework used consistently in benchmark construction and analysis, not a quantity fitted to the outcome. DynamicKTO's beta values (0.1/0.3/0.5 for Entity/Relation/Attribute) are selected by an empirical LFactDistance heuristic described in Appendices A.2 and A.8, and the paper reports measured accuracy on GEOHALU BENCH-Abstain rather than a prediction derived from those values. The 29.6% gain is a post-hoc empirical result; choosing hyperparameters on the evaluation distribution is a generalization and overfitting concern, not a circular reduction, and the comparisons against sequential per-task KTO (Table 11) and against ORPO/SimPO provide non-circular evidence that the dynamic mechanism contributes beyond simple beta selection. The UrbanKG citation is a data-source reference, not a load-bearing self-citation, because the underlying data come from external providers. No uniqueness theorem, self-citation chain, or definitional identity forces the method's success, so no circular step meets the evidence bar.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the correctness of the constructed knowledge graph, the validity of the multiple-choice option mapping to hallucination types, and the hand-selected beta values. No external ground-truth validation or formal proof is provided for these foundations.

free parameters (2)
  • DynamicKTO task-level beta values = 0.1 (Entity), 0.3 (Relation), 0.5 (Attribute)
    Chosen per category based on LFactDistance and benchmark experiments; they are hand-set hyperparameters, not derived from a closed-form rule. Section 2.3.2 and Table 3.
  • KTO lambda coefficients lambda_D and lambda_U = not specified
    Standard KTO hyperparameters not reported in the paper; the reported results depend on them for exact reproduction.
assumptions (3)
  • domain assumption SpatialKG built from OpenStreetMap and Foursquare is a correct and sufficiently complete ground truth for all benchmark labels.
    All correct answers and hallucination labels are derived from SpatialKG in Sections 2.1 and 2.2.3. If the knowledge graph has errors or omissions, the benchmark's factual and hallucinated labels are wrong.
  • domain assumption Selecting a fabricated option or 'None of the other options' in a 3-option MCQ reveals the model's genuine belief or hallucination rather than a format effect or refusal strategy.
    This is the core measurement assumption of GEOHALUBENCH (Section 2.2.3 and Figure 3). The below-random performance of several models and the abstention analysis suggest format effects may be present.
  • ad hoc to paper LFactDistance computed on factual versus hallucinated template sequences is a valid proxy for hallucination tendency and justifies the inverse beta assignment.
    Introduced in Appendix A.2 and used only to motivate the beta choice; it is not independently validated against external hallucination metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning." pith.science (2026). https://pith.science/paper/XOV4OYFX

@misc{pith2026250719586,
  author       = {Pith},
  title        = {Pith review of: Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOV4OYFX}},
  note         = {Machine review of arXiv:2507.19586}
}
read the original abstract

Large language models (LLMs) possess extensive world knowledge, including geospatial knowledge, which has been successfully applied to various geospatial tasks such as mobility prediction and social indicator prediction. However, LLMs often generate inaccurate geospatial knowledge, leading to geospatial hallucinations (incorrect or inconsistent representations of geospatial information) that compromise their reliability. While the phenomenon of general knowledge hallucination in LLMs has been widely studied, the systematic evaluation and mitigation of geospatial hallucinations remain largely unexplored. To address this gap, we propose a comprehensive evaluation framework for geospatial hallucinations, leveraging structured geospatial knowledge graphs for controlled assessment. Through extensive evaluation across 20 advanced LLMs, we uncover the hallucinations in their geospatial knowledge. Building on these insights, we introduce a dynamic factuality aligning method based on Kahneman-Tversky Optimization (KTO) to mitigate geospatial hallucinations in LLMs, leading to a performance improvement of over 29.6% on the proposed benchmark. Extensive experimental results demonstrate the effectiveness of our benchmark and learning algorithm in enhancing the trustworthiness of LLMs in geospatial knowledge and reasoning tasks.

Figures

Figures reproduced from arXiv: 2507.19586 by the authors.

Figure 1
Figure 1. An overview of our work. In order to ad [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the hierarchy and composi [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example case of GEOHALUBENCH, where the multiple-choice options are labeled with distinct colors, and the corresponding hallucination types are highlighted in corresponding colors for clarity. Test Question Here is a multiple-choice question: Which of the following is a point of interest in Bei￾jing? A. Silver Spoon Cafe B. Haidian Library C. None of the other options Please select from A, B, C. Output your answe… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distributions of Hallucination Types and In [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The introduction of the Refuse/Abstain option alters the ranking of LLMs. The line chart illustrates the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: The prompt template of generating POI-related hallucinations. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The prompt template of generating AOI-related hallucinations. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The prompt template of generating Road-related hallucinations. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 19 canonical work pages

  1. [1]

    Pasquale Balsebre, Weiming Huang, and Gao Cong. 2024. Lamp: A language model on the map. arXiv preprint arXiv:2403.09059

    work page Pith review arXiv 2024

  2. [2]

    Forrest Bao, Miaoran Li, Rogger Luo, and Ofer Mendelevitch. 2024. https://doi.org/10.57967/hf/3240 HHEM-2.1-Open

    doi:10.57967/hf/3240 2024

  3. [3]

    Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. 2024. Does your data spark joy? performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476

    arXiv 2024

  4. [4]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. Halc: Object hallucination reduction via adaptive focal-contrast decoding. In Forty-first International Conference on Machine Learning

    work page 2024

  5. [5]

    LMDeploy Contributors. 2023 a . Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy

    work page 2023

  6. [6]

    OpenCompass Contributors. 2023 b . Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    work page 2023

  7. [7]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    arXiv 2024

  8. [8]

    Jie Feng, Yuwei Du, Tianhui Liu, Siqi Guo, Yuming Lin, and Yong Li. 2024 a . Citygpt: Empowering urban spatial cognition of large language models. arXiv preprint arXiv:2406.13948

    arXiv 2024

Show all 59 references
  1. [9]

    Jie Feng, Yuwei Du, Jie Zhao, and Yong Li. 2024 b . Agentmove: Predicting human mobility anywhere using large language model based agentic framework. arXiv preprint arXiv:2408.13986

    2024 arXiv

  2. [10]

    Jie Feng, Jun Zhang, Junbo Yan, Xin Zhang, Tianjian Ouyang, Tianhui Liu, Yuwei Du, Siqi Guo, and Yong Li. 2024 c . Citybench: Evaluating the capabilities of large language model as world model. arXiv preprint arXiv:2406.13945

    2024 arXiv

  3. [11]

    Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 864--870

    2020

  4. [12]

    Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207

    2023 arXiv

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  6. [14]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

    2024

  7. [15]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232

    2023 arXiv

  8. [16]

    Anna A Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H Clark, Carina Kauf, Jennifer Hu, RT Pramod, Gabriel Grand, et al. 2024. Elements of world knowledge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language m...

    2024 arXiv

  9. [17]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  10. [18]

    Yue Jiang, Qin Chao, Yile Chen, Xiucheng Li, Shuai Liu, and Gao Cong. 2024. Urbanllm: Autonomous urban activity planning and management with large language models. arXiv preprint arXiv:2406.12360

    2024 arXiv

  11. [19]

    Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. Wice: Real-world entailment for claims in wikipedia. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7561--7583

    2023

  12. [20]

    Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2023. Realtime qa: What's the answer right now? Advances in neural information processing systems, 36:49025--49043

    2023

  13. [21]

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2023. Certifying llm safety against adversarial prompting. In First Conference on Language Modeling

    2023

  14. [22]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  15. [23]

    Bruce W Lee, Hyunsoo Cho, and Kang Min Yoo. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.82 Instruction tuning with human curriculum . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1281--1309, Mexico City, Mexico. Association for Comp...

    2024 doi

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  17. [25]

    Leybzon and Corentin Kervadec

    Danny D. Leybzon and Corentin Kervadec. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.4 Learning, forgetting, remembering: Insights from tracking LLM memorization during training . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Network...

    2024 doi

  18. [26]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023 a . Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747

    2023 arXiv

  19. [27]

    Zekun Li, Wenxuan Zhou, Yao-Yi Chiang, and Muhao Chen. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.317 G eo LM : Empowering language models for geospatially grounded language understanding . In Proceedings of the 2023 Conference on Empirical Methods in Natural Languag...

    2023 doi

  20. [28]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252

    2022

  21. [29]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

    2023 arXiv

  22. [30]

    Yu Liu, Jingtao Ding, Yanjie Fu, and Yong Li. 2023 b . Urbankg: An urban knowledge graph system. ACM Transactions on Intelligent Systems and Technology, 14(4):1--25

    2023

  23. [31]

    Fiona Macpherson and Dimitris Platchias. 2013. Hallucination: Philosophy and psychology. MIT Press

    2013

  24. [32]

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

    2023 arXiv

  25. [33]

    Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. 2024. Large language models are geographically biased. arXiv preprint arXiv:2402.02680

    2024 arXiv

  26. [34]

    Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, and Stefano Ermon. 2023. Geollm: Extracting geospatial knowledge from large language models. arXiv preprint arXiv:2310.06213

    2023 arXiv

  27. [35]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661

    2020 arXiv

  28. [36]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2025. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198--124235

    2025

  29. [37]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empiric...

    2023

  30. [38]

    Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. 2024. Generating benchmarks for factuality evaluation of language models. In Proceedings of the 18th Conference of the European Chapter...

    2024

  31. [39]

    Niels M \"u ndler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852

    2023 arXiv

  32. [40]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 314--334

    2023

  33. [41]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

    2023

  34. [42]

    Leonardo FR Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022. Factgraph: Evaluating factuality in summarization with semantic graph representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computatio...

    2022

  35. [43]

    Jonathan Roberts, Timo L \"u ddecke, Sowmen Das, Kai Han, and Samuel Albanie. 2023. Gpt4geo: How a language model sees the world's geography. arXiv preprint arXiv:2306.00020

    2023 arXiv

  36. [44]

    Hannah Sansford, Nicholas Richardson, Hermina Petric Maretic, and Juba Nait Saada. 2024. Grapheval: A knowledge-graph based llm hallucination evaluation framework. arXiv preprint arXiv:2407.10793

    2024 arXiv

  37. [45]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  38. [46]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  39. [47]

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. 2023. Where would i go next? large language models as human mobility predictors. arXiv preprint arXiv:2308.15197

    2023 arXiv

  41. [49]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  42. [50]

    Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. 2024. Fundamental limitations of alignment in large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 53079--53112

    2024

  43. [51]

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024. -dpo: Direct preference optimization with dynamic . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  44. [52]

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817

    2024 arXiv

  45. [53]

    Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. 2024. Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. In Proceedings of the ACM on Web Conference 2024...

    2024

  46. [54]

    Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296

    2023 arXiv

  47. [55]

    Yichi Zhang, Zhuo Chen, Yin Fang, Yanxi Lu, Li Fangming, Wen Zhang, and Huajun Chen. 2024. https://doi.org/10.18653/v1/2024.findings-acl.52 Knowledgeable preference alignment for LLM s in domain-specific question answering . In Findings of the Association for Computational Lin...

    2024 doi

  48. [56]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  49. [57]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  50. [58]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.