Pith. sign in

REVIEW 4 major objections 4 minor 29 references

LETToT: Label-Free Evaluation of Large Language Models On Tourism Using Expert Tree-of-Thought

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that domain-expert reasoning trees can replace annotated benchmarks for evaluating LLMs in tourism, and that doing so reverses the ranking of reasoning-enabled small models over larger generic ones.

desk verdict Interesting idea, but the evaluation metric is circular and insensitive to factual errors, so the main empirical claims don't stand. read the letter →

arxiv 2508.11280 v2 pith:OKDF3QSD submitted 2025-08-15 cs.CL cs.AI

Ruiyan Qi , Congding Wen , Weibo Zhou , Jiwei Li , Shangsong Liang , Lingbo Li This is my paper
classification cs.CLcs.AI
keywords label-freeevaluationtree-of-thoughttourismquestionansweringLLMpromptoptimizationexpertknowledgereasoningmodelsAnalyticHierarchyProcess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LETToT is a label-free evaluation framework for tourism question answering. It replaces costly human-labeled benchmarks with a two-layer expert tree-of-thought: a taxonomy of query types and tourism themes plus fine-grained element checklists. The authors argue that this structure both improves LLM answers when used as a prompt (4.99–14.15% relative gains) and provides an objective scoring rubric by counting covered elements and penalizing verbosity. Applying it to five open models from 32B to 671B parameters, they find that scaling laws hold (DeepSeek-V3 leads) but reasoning-fine-tuned smaller models close the gap and, below 72B, significantly beat non-reasoning counterparts on accuracy and conciseness. The point is a scalable, annotation-free alternative for domain-specific LLM evaluation.

What carries the argument

The load-bearing object is the expert ToT element library: three travel-phase query types (planning, pre-trip consultation, on-trip guidance) crossed with eleven tourism themes, each expanded into checklists of verifiable sub-elements (e.g., avalanche warning systems for ice-snow tourism). Evaluation uses the composite score $S_{total} = (\alpha S_{base} + \beta S_{specific}) \cdot F_{eff}$, where $S_{base}$ counts coverage of general planning/consultation/guidance elements, $S_{specific}$ counts theme-specific elements, and $F_{eff}$ is a logistic efficiency factor $N/L$ (elements per character) that rewards concise density. This machinery turns 'quality' into countable coverage plus text e

What would settle it

Take a sample of tourism QA responses, score them with LETToT's $S_{total}$, and collect independent human expert ratings (or tourist outcome judgments) on the same responses. If the LETToT ranking does not correlate with the human ranking—especially over the sub-72B models where it reverses the generic leaderboard—the central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that expert knowledge can be encoded as a hierarchical tree-of-thought and used twice: as an optimizing prompt and as a label-free scoring oracle. The authors claim the optimized expert ToT improves response quality by 4.99–14.15% over baseline prompts across seven quality dimensions. When the same ToT elements are used to grade models, the resulting ranking—DS-V3 first, then DS-70B, DS-32B, Qwen-72B, Qwen-32B—contradicts the HuggingFace Open LLM Leaderboard ordering over the sub-72B models, and the authors read this reversal as evidence that generic benchmarks miss domain-specific competence such as reasoning and conciseness.

Load-bearing premise

The whole framework rests on the assumption that the expert-defined ToT element list is a valid and complete operationalization of tourism answer quality; if those elements do not match what human experts or tourists judge as good answers, the scores and rankings lose their meaning.

Editorial extensions

If this is right

  • Tourism QA can be evaluated without building a labeled benchmark: a validated expert ToT plus the composite score yields a model ranking.
  • Prompt optimization guided by the same ToT improves outputs across all seven quality dimensions, with the largest gains in thematic relevance (+14.15%) and contextual appropriateness (+13.85%).
  • Scaling laws persist in this specialized domain, with DeepSeek-V3 leading, but reasoning-enhanced models at 32B–70B narrow the gap.
  • Below 72B parameters, explicit reasoning architectures significantly outperform non-reasoning models ($p < 0.05$) on accuracy and conciseness, a difference generic leaderboards hide.
  • The modular element structure should transfer LETToT to other expertise-heavy domains by substituting the theme-specific checklists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the efficiency factor makes the metric sensitive to response length and element density, so a model that lists many checklist items tersely could score well without actually being helpful; an independent human correlation study is the natural next test.
  • The reversal against the Open LLM Leaderboard is interpreted by the authors as evidence that their metric captures domain expertise, but an equally plausible reading is that the two instruments measure different constructs; the paper does not test which better predicts tourist satisfaction.
  • One unstated consequence: if label-free expert ToT scores are trustworthy, they could serve as reward signals for preference optimization or as filters for synthetic training data in tourism, removing the need for human preference labels.
  • The framework's taxonomy (3 query types × 11 themes) could transfer to adjacent service domains such as healthcare navigation or legal advice by substituting the element checklists, a direction the conclusion gestures at but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LETToT, a two-stage label-free evaluation framework for tourism QA. In Stage 1, the authors construct a hierarchical expert Tree-of-Thought prompt by iteratively refining tourism-specific elements through LLM-judge feedback and AHP-weighted scoring, and they report 4.99–14.15% relative quality gains over baseline prompts. In Stage 2, the optimized ToT elements are used as a rule-based scoring rubric: Stotal = (α·Sbase + β·Sspecific) · Feff, where coverage of general and theme-specific elements is combined with a logistic length-efficiency factor. Using this metric, five LLMs of 32B–671B parameters are ranked, and the paper claims that reasoning-enhanced sub-72B models significantly outperform non-reasoning counterparts in accuracy and conciseness (p < 0.05), in contrast to generic leaderboards. The paper positions LETToT as a scalable, annotated-benchmark-free alternative for domain-specific evaluation.

Significance. The motivation is timely: tourism QA evaluation is expensive and hallucination-prone, and a reliable label-free protocol would be valuable. The paper contributes a structured expert-ToT taxonomy (Table 6), a query classification, and a reproducible scoring procedure. However, the central metric does not verify factual correctness, and the evaluation rubric overlaps with the prompt being optimized. The headline quality gains and the accuracy/conciseness conclusion are therefore unsupported as stated. The framework's usefulness depends entirely on validating Stotal against external human judgments or a task-specific benchmark; the paper does not provide such validation and instead interprets disagreement with a generic leaderboard as evidence in its favor. Thus, while the scaffolding is interesting, the current evidence does not establish LETToT as a valid evaluation method.

major comments (4)
  1. [Domain-Specific Evaluation, Eq. (3)] Stotal is computed solely from counts of expert-ToT element mentions and text length: Stotal = (α·Sbase + β·Sspecific) · Feff, where Feff = 1/(1+exp(−N/L)). Nowhere does the formula verify that a covered element is true. A response that names a nonexistent attraction, gives incorrect opening hours, or proposes an infeasible itinerary receives full credit for each element it mentions. This is a load-bearing omission because the paper motivates the framework by hallucination ('plausible but incorrect information'). The only accuracy-related component, the Acc dimension in the Stage-1 direct scoring, is not part of the Stage-2 Stotal ranking. Consequently, the RQ2 claim that reasoning models are better in 'accuracy and conciseness' (p < 0.05) is not supported by the metric that produced the p-values.
  2. [Iterative Expert ToT Validation and Refinement; Results RQ1] The prompt-optimization stage and the evaluation stage use the same expert-derived ToT element inventory. Stage 1 refines prompts so that they explicitly instruct the model to cover the elements listed in Table 6, and Stage 2 scores responses by counting coverage of those same elements via Eq. (3). The reported 4.99–14.15% relative gains therefore partly reflect alignment with the rubric by construction. An optimized prompt that explicitly contains the to-be-counted elements is guaranteed to score higher on element coverage, irrespective of whether the end-to-end travel-plan quality improved. To make the effectiveness claim, the optimized and baseline responses should be judged on an external task objective or by human raters who are blind to the optimization criterion.
  3. [Experimental Design; RQ2 comparison to HuggingFace Leaderboard] The paper reports a discrepancy between LETToT's ranking and the HuggingFace Open LLM Leaderboard, and interprets this as evidence of LETToT's domain-specific usefulness. This is not a validation of the metric. Disagreement with a generic benchmark is expected if the metric measures something different; the central question is whether LETToT measures the correct thing. Without human preference judgments on the same responses, or comparison against a dedicated tourism-QA benchmark with factual grounding, the claimed superiority over generic leaderboards is untested. The paper should either provide such external validation or substantially weaken the claim that LETToT is a 'robust alternative.'
  4. [Results RQ2; Figure 7; Answer to RQ2] The statistical claim that reasoning-enhanced sub-72B models 'significantly outperform counterparts in accuracy and conciseness (p < 0.05)' is internally inconsistent with the definition of Stotal. The p-values reported in Figure 7 are computed on Stotal distributions, but Stotal does not measure accuracy. Feff is a density/conciseness proxy, not accuracy. If the authors intend to make an accuracy claim, they need a metric that checks facts (e.g., against a knowledge base or human annotation). As written, the 'accuracy' portion of the claim is unjustified, and the 'conciseness' portion conflates token-efficiency with semantic quality.
minor comments (4)
  1. [Conclusion] The text states '4.99-4.15%' in the conclusion; this appears to be a typo for '4.99–14.15%' used elsewhere. Please correct.
  2. [Figure 6] The y-axis label reads 'range: 0-0'; this is likely a typo and should be the score range actually plotted.
  3. [Throughout] There are several typographical issues (e.g., 'popluar', 'formated', 'DS models exhibits'). A careful proofread is needed.
  4. [References] Several references are cited with incomplete version information and inconsistent formatting (e.g., arXiv preprints without version numbers or access dates). The reference list should be standardized.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ2 'accuracy and conciseness' finding reduces to LETToT's element-coverage metric by construction.

  1. self definitional [Domain-Specific Evaluation with LETToT, Eq. (3); Answer to RQ2]
    "Stotal = (αSbase + βSspecific) · Feff , where Sbase is the base score assessing the coverage of general tourism elements, Sspecific is the specific score evaluating the coverage of elements unique to a given tourism theme, and Feff is the efficiency factor measuring the information density of the text. ... Reasoning-enhanced models (32B–72B) significantly outperform non-reasoning counterparts in accuracy and conciseness (p < 0.05)."

    The RQ2 'accuracy' conclusion is drawn from Stotal, but Stotal is constructed entirely from counts of expert-ToT element mentions (ΣCi, ΣSj) and text length L via Feff; it contains no term that checks whether mentioned elements are true or appropriate. Therefore the statement 'reasoning models outperform in accuracy' is equivalent, under the paper's operationalization, to 'reasoning models produce responses covering more ToT elements in fewer characters.' The outcome variable is the metric's own definition, so the headline finding is a tautology of Eq. (3), not an empirical discovery.

full rationale

LETToT is largely a rubric-construction exercise, and the RQ1 prompt-optimization gains are measured by an LLM-judge on seven generic dimensions, which is not itself circular. The load-bearing reduction occurs in RQ2. The comprehensive score Stotal (Eq. 3) is defined as a weighted sum of element-coverage scores multiplied by an efficiency factor based on element density; there is no factual-verification term. The paper nevertheless reports this score as 'accuracy and conciseness' and concludes reasoning models are more accurate. Since Stotal is the only stage-2 performance measure, the conclusion 'explicit reasoning architectures outperform counterparts in accuracy' is logically equivalent to 'responses from these models contain more ToT elements per character.' The metric's inputs (the expert ToT element list) are also the output claim's criterion, so the headline finding is a restatement of the scoring formula rather than an independent result. No self-citation chain or uniqueness import is present; the circularity is a self-definitional labeling of the metric.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework's central measures rest on manually constructed element lists, expert-judgment-based AHP weights, and unposted tuning of the efficiency factor. These constitute a large set of free parameters that are not fitted to or validated against an independent benchmark.

free parameters (4)
  • α and β (Stotal weights) = α = β = 1 (default)
    Introduced to balance general and specific element coverage; set to 1 without evidence that these weights maximize alignment with human judgments.
  • Logistic efficiency factor tuning = Not specified
    The paper states the logistic function was 'fine-tuned through empirical analysis' but does not report any fitted constants or the selection process.
  • AHP pairwise weights for seven dimensions = Derived from expert pairwise comparisons
    The AHP weights that combine the seven quality dimensions are based on subjective expert judgments and are not validated against outcome data.
  • Expert ToT element inventory (Table 6) = Human-crafted categories and sub-elements
    The entire hierarchy of tourism elements that defines the scoring rubric is manually constructed and is itself a large set of free parameters.
assumptions (5)
  • domain assumption The seven quality dimensions (Thematic Relevance, Contextual Adaptability, Logical Coherence, Creativity, Accuracy, Completeness, Practicality) are the correct axes for tourism answer quality.
    Invoked as the rubric for LLM-judge scoring in Section 'Direct Scoring' and used to validate prompt optimization.
  • ad hoc to paper Counting coverage of expert ToT elements is a valid measure of tourism answer quality.
    The comprehensive scoring formula Stotal is based on element coverage; no external ground truth is used to validate this measure.
  • domain assumption The 3,240 curated QA pairs are representative of tourism questions.
    The dataset is claimed to be systematically curated but is not released, so this cannot be checked.
  • domain assumption The web-sourced texts are treated as high-quality references.
    Used in the 60:40 dataset split; no quality control beyond manual validation with 95% agreement is described.
  • domain assumption The LLM-judge provides unbiased and reliable quality scores.
    The paper mentions potential preference bias and uses two scoring mechanisms to mitigate it, but no debiasing algorithm is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LETToT: Label-Free Evaluation of Large Language Models On Tourism Using Expert Tree-of-Thought." pith.science (2026). https://pith.science/paper/OKDF3QSD

@misc{pith2026250811280,
  author       = {Pith},
  title        = {Pith review of: LETToT: Label-Free Evaluation of Large Language Models On Tourism Using Expert Tree-of-Thought},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKDF3QSD}},
  note         = {Machine review of arXiv:2508.11280}
}
abstract

Evaluating large language models (LLMs) in specific domain like tourism remains challenging due to the prohibitive cost of annotated benchmarks and persistent issues like hallucinations. We propose $\textbf{L}$able-Free $\textbf{E}$valuation of LLM on $\textbf{T}$ourism using Expert $\textbf{T}$ree-$\textbf{o}$f-$\textbf{T}$hought (LETToT), a framework that leverages expert-derived reasoning structures-instead of labeled data-to access LLMs in tourism. First, we iteratively refine and validate hierarchical ToT components through alignment with generic quality dimensions and expert feedback. Results demonstrate the effectiveness of our systematically optimized expert ToT with 4.99-14.15\% relative quality gains over baselines. Second, we apply LETToT's optimized expert ToT to evaluate models of varying scales (32B-671B parameters), revealing: (1) Scaling laws persist in specialized domains (DeepSeek-V3 leads), yet reasoning-enhanced smaller models (e.g., DeepSeek-R1-Distill-Llama-70B) close this gap; (2) For sub-72B models, explicit reasoning architectures outperform counterparts in accuracy and conciseness ($p<0.05$). Our work established a scalable, label-free paradigm for domain-specific LLM evaluation, offering a robust alternative to conventional annotated benchmarks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    Barati, I.; Ghafouri, A.; and Minaei, B. 2025. Bactrainus: Optimizing Large Language Models for Multi-hop Complex Question Answering Tasks. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 1234--1247

    work page 2025

  2. [2]

    Contractor, D.; Shah, K.; Partap, A.; and Singla, P. 2019. Large scale question answering using tourism data. arXiv preprint arXiv:1909.03527

    work page Pith review arXiv 2019

  3. [3]

    Gao, P.; Xie, A.; Mao, S.; Wu, W.; Xia, Y.; Mi, H.; and Wei, F. 2024. Meta Reasoning for Large Language Models (Version 1). arXiv preprint

    work page 2024

  4. [4]

    HuggingFace. 2025. Open LLM Leaderboard. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard. Accessed: 2025-05-11

    work page 2025

  5. [5]

    Hui, W.; and Tu, K. 2024. RoT: Enhancing Large Language Models with Reflection on Search Trees. arXiv preprint. Version 3

    work page 2024

  6. [6]

    W.; and Norman, W

    Kang, S.; Jodice, L. W.; and Norman, W. C. 2020. How do tourists search for tourism information via smartphone before and during their trip? Tourism Recreation Research, 45(1): 57--68

    work page 2020

  7. [7]

    Kumar, P. 2024. Large language models (LLMs): Survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10)

    work page 2024

  8. [8]

    Liang, C.; Feng, Z.; Liu, Z.; Jiang, W.; Xu, J.; Chen, Y.; and Wang, Y. 2024. Textualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM Generation. arXiv preprint. Version 1

    work page 2024

Show all 29 references
  1. [9]

    Liang, P.; Bommasani, R.; Lee, T.; et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  2. [10]

    Lyu, Y.; Li, Z.; Niu, S.; Xiong, F.; Tang, B.; Wang, W.; and Chen, E. 2024. CRUD-RAG: A Comprehensive Benchmark for Retrieval-Augmented Generation of Large Language Models. ACM Transactions on Information Systems, 42(3): 1--35

    2024

  3. [11]

    Maria, G. A. 2016. CLASSIFICATION OF VARIOUS FORMS OF TOURISM. Annals of the University of Oradea, Economic Science Series, 25(2)

    2016

  4. [12]

    A.; and Álvarez Albelo, C

    Martínez-González, J. A.; and Álvarez Albelo, C. D. 2021. Influence of site personalization and first impression on young consumers' loyalty to tourism websites. Sustainability, 13(3): 1425

    2021

  5. [13]

    Mizumoto, A.; and Eguchi, M. 2023. Exploring the Potential of Using an AI Language Model for Automated Essay Scoring. Research Methods in Applied Linguistics, 2(2): 100050

    2023

  6. [14]

    Mo, S.; and Xin, M. 2023. Tree of Uncertain Thoughts Reasoning for Large Language Models (Version 1). arXiv preprint

    2023

  7. [15]

    To use or not to use

    Nautiyal, R.; Polus, R.; Tripathi, A.; and Shaheer, I. 2023. "To use or not to use"-Mobile technology in nature-based tourism experience. Journal of Outdoor Recreation and Tourism, 43: 100667

    2023

  8. [16]

    Ren, R.; Yao, X.; and Cole, S. e. a. 2024. Are Large Language Models Ready for Travel Planning? arXiv preprint

    2024

  9. [17]

    Wan, G.; Wu, Y.; Wang, H.; et al. 2024. Derailer-rerailer: Adaptive verification for efficient and reliable language model reasoning. arXiv preprint arXiv:2408.13940

    2024 arXiv

  10. [18]

    Wang, L.; Liao, L.; Yang, K.; and Tan, H. 2013. A Case Study of Question Answering in Automatic Tourism Service Packaging. Cybernetics and Information Technologies, 13: 143--152

    2013

  11. [19]

    Wang, Z.; Wu, J.; Lai, Y.; Zhang, C.; and Zhou, D. 2024. SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding (Version 2). arXiv preprint

    2024

  12. [20]

    Wei, Q.; Yang, M.; Wang, J.; Mao, W.; Xu, J.; and Ning, H. 2024. TourLLM: Enhancing LLMs with Tourism Knowledge. arXiv preprint. Version 1

    2024

  13. [21]

    Xia, L.; Li, C.; Zhang, C.; et al. 2024. Leveraging error-assisted fine-tuning large language models for manufacturing excellence. Robotics and Computer-Integrated Manufacturing, 88: 102728

    2024

  14. [22]

    Xiang, Z.; and Pan, B. 2011. Travel queries on cities in the United States: Implications for search engine marketing for tourist destinations. Tourism Management, 32(1): 88--97

    2011

  15. [23]

    Xie, J.; Zhang, K.; Chen, J.; et al. 2024. TravelPlanner: A benchmark for real-world planning with language agents. In Proceedings of the 41st International Conference on Machine Learning

    2024

  16. [24]

    Xu, F.; Lin, Q.; and Han, J. e. a. 2025. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. IEEE Transactions on Knowledge and Data Engineering

    2025

  17. [25]

    Yang, L.; Cao, H.; Hao, F.; Zhang, W.; and Ahmad, M. 2020. Research on Tourism Question Answering System Based on Xi’an Tourism Knowledge Graph. Journal of Physics: Conference Series, 1616(1): 012090

    2020

  18. [26]

    L.; Cao, Y.; and Narasimhan, K

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Version 2). arXiv preprint

    2023

  19. [27]

    Yue, M.; Yao, W.; Mi, H.; et al. 2024. DOTS : Learning to reason dynamically in LLMs via optimal reasoning trajectories search. arXiv preprint arXiv:2410.03864

    2024 arXiv

  20. [28]

    Zhang, P.; Wang, J.; and Li, R. 2023. Tourism-type ontology framework for tourism-type classification, naming, and knowledge organization. Heliyon, 9(4)

    2023

  21. [29]

    Zhao, R.; Zhao, F.; Wang, L.; et al. 2024. KG-CoT : Chain-of-thought prompting of large language models over knowledge graphs for knowledge-aware question answering. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 6642--6650

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.