{"id":"07203480-b4f6-46c3-8152-3e503b13272c","arxiv_id":"2508.11280","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LETToT scores LLM tourism answers by counting coverage of expert-designed reasoning elements, and finds reasoning-enhanced small models beat larger non-reasoning models on that rubric.","lead":"The paper presents a system that grades how well language models answer travel questions without needing labeled test answers. It uses expert-designed reasoning steps as the scoring checklist and reports that smaller models with step-by-step reasoning beat larger plain models on these tourism tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LETToT's Stotal counts ToT-element mentions and penalizes length but never verifies factual correctness, so the claimed accuracy/conciseness finding is unsupported.","rationale":"The reader's weakest assumption concerns the validity and completeness of the expert-defined ToT element list and the lack of independent validation against human judgments. I agree that this is a serious problem, but I want to sharpen it into an internal, checkable defect: even granting the expert list as a comprehensive coverage rubric, the formula in Eq. (3) cannot measure factual accuracy. This is not merely an external validity question; it is a mismatch between the paper's stated motivation (hallucination) and the operationalized metric (element counts plus length penalty). The strongest claim about reasoning models winning on 'accuracy and conciseness' is therefore unsupported by the evidence the paper actually computes. I also note that the dataset and code are withheld, which prevents independent replication, but the hallucinated-twin test I propose does not require access to the original dataset—it only needs the scoring formula and a sample of model outputs. If the test shows sensitivity to factual substitutions, then this concern would be substantially weakened; if not, the central evaluation claim collapses. Either way, the reader's REJECT verdict remains appropriate, so I recommend no change to the verdict.","tokens_in":11007,"tokens_out":4372,"duration_ms":52577,"concrete_test":"Take a stratified sample of 100 model-generated tourism responses. For each response, produce a 'hallucinated twin' by replacing concrete factual content (POI names, opening hours, distances, prices, transfer feasibility) with plausible but false substitutes, while keeping the same ToT element coverage, section structure, and character length. Run LETToT on both sets. If the median Stotal difference between correct and hallucinated twins is near zero, or if the model rankings are unchanged, the metric is insensitive to factual accuracy and the RQ2 accuracy/conciseness claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is in Eq. (3). LETToT's comprehensive score is entirely a function of how many expert-ToT elements are mentioned and how short the text is: Stotal = (αSbase + βSspecific) · Feff, with Feff the logistic of (ΣCi + ΣSj)/L. Nowhere does the framework verify that a covered element is true. A response that names a non-existent attraction, gives wrong opening hours, or builds an itinerary with impossible transfer times receives full credit for each element it mentions. For a tourism-QA evaluator whose stated motivation is hallucination ('plausible but incorrect information'), this is not a minor omission: the headline 'accuracy and conciseness (p < 0.05)' finding in RQ2 is not supported by the metric. Stage 1 does include an 'Accuracy' dimension, but that is scored by an LLM-judge and is not part of the stage-2 Stotal ranking; the label-free evaluation drops the only component that could detect falsehood. Moreover, because the prompt-optimization stage maximizes the very same element-coverage rubric, the reported 4.99–14.15% quality gains can be read as alignment with the rubric rather than improvement in end-to-end travel-plan quality. The central claim therefore rests on an unvalidated proxy that is insensitive to the failure mode the paper is most concerned about.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LETToT, a two-stage label-free evaluation framework for tourism QA. In Stage 1, the authors construct a hierarchical expert Tree-of-Thought prompt by iteratively refining tourism-specific elements through LLM-judge feedback and AHP-weighted scoring, and they report 4.99–14.15% relative quality gains over baseline prompts. In Stage 2, the optimized ToT elements are used as a rule-based scoring rubric: Stotal = (α·Sbase + β·Sspecific) · Feff, where coverage of general and theme-specific elements is combined with a logistic length-efficiency factor. Using this metric, five LLMs of 32B–671B parameters are ranked, and the paper claims that reasoning-enhanced sub-72B models significantly outperform non-reasoning counterparts in accuracy and conciseness (p < 0.05), in contrast to generic leaderboards. The paper positions LETToT as a scalable, annotated-benchmark-free alternative for domain-specific evaluation.","tokens_in":11383,"tokens_out":3128,"duration_ms":38327,"significance":"The motivation is timely: tourism QA evaluation is expensive and hallucination-prone, and a reliable label-free protocol would be valuable. The paper contributes a structured expert-ToT taxonomy (Table 6), a query classification, and a reproducible scoring procedure. However, the central metric does not verify factual correctness, and the evaluation rubric overlaps with the prompt being optimized. The headline quality gains and the accuracy/conciseness conclusion are therefore unsupported as stated. The framework's usefulness depends entirely on validating Stotal against external human judgments or a task-specific benchmark; the paper does not provide such validation and instead interprets disagreement with a generic leaderboard as evidence in its favor. Thus, while the scaffolding is interesting, the current evidence does not establish LETToT as a valid evaluation method.","major_comments":[{"comment":"Stotal is computed solely from counts of expert-ToT element mentions and text length: Stotal = (α·Sbase + β·Sspecific) · Feff, where Feff = 1/(1+exp(−N/L)). Nowhere does the formula verify that a covered element is true. A response that names a nonexistent attraction, gives incorrect opening hours, or proposes an infeasible itinerary receives full credit for each element it mentions. This is a load-bearing omission because the paper motivates the framework by hallucination ('plausible but incorrect information'). The only accuracy-related component, the Acc dimension in the Stage-1 direct scoring, is not part of the Stage-2 Stotal ranking. Consequently, the RQ2 claim that reasoning models are better in 'accuracy and conciseness' (p < 0.05) is not supported by the metric that produced the p-values.","section":"Domain-Specific Evaluation, Eq. (3)"},{"comment":"The prompt-optimization stage and the evaluation stage use the same expert-derived ToT element inventory. Stage 1 refines prompts so that they explicitly instruct the model to cover the elements listed in Table 6, and Stage 2 scores responses by counting coverage of those same elements via Eq. (3). The reported 4.99–14.15% relative gains therefore partly reflect alignment with the rubric by construction. An optimized prompt that explicitly contains the to-be-counted elements is guaranteed to score higher on element coverage, irrespective of whether the end-to-end travel-plan quality improved. To make the effectiveness claim, the optimized and baseline responses should be judged on an external task objective or by human raters who are blind to the optimization criterion.","section":"Iterative Expert ToT Validation and Refinement; Results RQ1"},{"comment":"The paper reports a discrepancy between LETToT's ranking and the HuggingFace Open LLM Leaderboard, and interprets this as evidence of LETToT's domain-specific usefulness. This is not a validation of the metric. Disagreement with a generic benchmark is expected if the metric measures something different; the central question is whether LETToT measures the correct thing. Without human preference judgments on the same responses, or comparison against a dedicated tourism-QA benchmark with factual grounding, the claimed superiority over generic leaderboards is untested. The paper should either provide such external validation or substantially weaken the claim that LETToT is a 'robust alternative.'","section":"Experimental Design; RQ2 comparison to HuggingFace Leaderboard"},{"comment":"The statistical claim that reasoning-enhanced sub-72B models 'significantly outperform counterparts in accuracy and conciseness (p < 0.05)' is internally inconsistent with the definition of Stotal. The p-values reported in Figure 7 are computed on Stotal distributions, but Stotal does not measure accuracy. Feff is a density/conciseness proxy, not accuracy. If the authors intend to make an accuracy claim, they need a metric that checks facts (e.g., against a knowledge base or human annotation). As written, the 'accuracy' portion of the claim is unjustified, and the 'conciseness' portion conflates token-efficiency with semantic quality.","section":"Results RQ2; Figure 7; Answer to RQ2"}],"minor_comments":[{"comment":"The text states '4.99-4.15%' in the conclusion; this appears to be a typo for '4.99–14.15%' used elsewhere. Please correct.","section":"Conclusion"},{"comment":"The y-axis label reads 'range: 0-0'; this is likely a typo and should be the score range actually plotted.","section":"Figure 6"},{"comment":"There are several typographical issues (e.g., 'popluar', 'formated', 'DS models exhibits'). A careful proofread is needed.","section":"Throughout"},{"comment":"Several references are cited with incomplete version information and inconsistent formatting (e.g., arXiv preprints without version numbers or access dates). The reference list should be standardized.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript has a clear and useful scaffolding, but the core evaluation metric is insensitive to the very failure mode the paper claims to address, and the reported gains are confounded by rubric alignment. I do not see a local fix: adding a factual verification component to Stotal, collecting human judgments, and rerunning all experiments would change the paper's central results and conclusions. This goes beyond a revision of presentation or a targeted addition; it redefines the framework's main claim. I therefore recommend rejection, though the authors may wish to pursue a revised design with external validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: the paper has a genuinely interesting idea—use expert-derived reasoning structures for label-free evaluation of tourism QA—but the metric it builds is circular and never checks whether the covered elements are factually true. The headline claims about accuracy and conciseness don't follow from the equations.\n\nWhat's actually new: a detailed, tourism-specific ToT element hierarchy (Table 6) that is clearly a lot of work, and a two-stage setup where prompts are optimized against an AHP-weighted rubric, then models are ranked with the same rubric. The empirical comparison of five LLMs from 32B to 671B is also new for this domain. The observation that reasoning-tuned smaller models beat larger non-reasoning ones on this metric is interesting, even if the metric's validity is the problem.\n\nThe soft spot is load-bearing. Eq (3) defines Stotal as a weighted sum of element coverage times an efficiency factor. Nowhere does it verify that a mentioned element is correct. A response that names a non-existent museum or gives wrong opening hours gets full credit for coverage. The paper's stated motivation is hallucination, yet the stage-2 evaluation drops the only accuracy-related component (the LLM-judge dimension in stage 1). So the finding 'reasoning models outperform non-reasoning in accuracy and conciseness (p < 0.05)' is unsupported. The 4.99-14.15% gains from prompt optimization are also partly by construction: the optimized prompt explicitly asks for the elements that the scoring formula counts.\n\nThe 'label-free' claim is also a stretch. Stage 1 uses human annotators and expert feedback to refine the ToT, and the dataset is withheld, so nothing is independently checkable. The comparison to HuggingFace's leaderboard is used to argue for the metric's superiority, but that just assumes the metric is right.\n\nThe paper does acknowledge some limitations in its threat-to-validity section, but it doesn't address the core circularity or the lack of external validation.\n\nWho should read this: anyone working on domain-specific evaluation or prompt optimization will find the taxonomy and the AHP approach worth a look as a case study. But don't cite the empirical results as evidence for model rankings.\n\nRecommendation: it deserves a serious referee because the idea is worth debating, but the current version needs major revision. The authors should validate the scoring formula against human judgments or an established benchmark, and separate prompt optimization from evaluation. If they do that, the paper could become useful. As is, the central claims don't hold.","headline":"Interesting idea, but the evaluation metric is circular and insensitive to factual errors, so the main empirical claims don't stand.","tokens_in":11795,"tokens_out":2460,"would_cite":false,"duration_ms":25412,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that domain-expert reasoning trees can replace annotated benchmarks for evaluating LLMs in tourism, and that doing so reverses the ranking of reasoning-enabled small models over larger generic ones.","keywords":["label-free evaluation","tree-of-thought","tourism question answering","LLM evaluation","prompt optimization","expert knowledge","reasoning models","Analytic Hierarchy Process"],"falsifier":"Take a sample of tourism QA responses, score them with LETToT's $S_{total}$, and collect independent human expert ratings (or tourist outcome judgments) on the same responses. If the LETToT ranking does not correlate with the human ranking—especially over the sub-72B models where it reverses the generic leaderboard—the central claim fails.","tokens_in":10958,"feed_emoji":"🧭","tokens_out":4315,"duration_ms":42214,"temperature":0.7,"pith_summary":"LETToT is a label-free evaluation framework for tourism question answering. It replaces costly human-labeled benchmarks with a two-layer expert tree-of-thought: a taxonomy of query types and tourism themes plus fine-grained element checklists. The authors argue that this structure both improves LLM answers when used as a prompt (4.99–14.15% relative gains) and provides an objective scoring rubric by counting covered elements and penalizing verbosity. Applying it to five open models from 32B to 671B parameters, they find that scaling laws hold (DeepSeek-V3 leads) but reasoning-fine-tuned smaller models close the gap and, below 72B, significantly beat non-reasoning counterparts on accuracy and conciseness. The point is a scalable, annotation-free alternative for domain-specific LLM evaluation.","feed_headline":"Expert reasoning trees replace labels in tourism LLM tests","feed_subtitle":"A label-free score lifts answer quality by up to 14% and flips the ranking of reasoning models vs generic leaderboards.","key_machinery":"The load-bearing object is the expert ToT element library: three travel-phase query types (planning, pre-trip consultation, on-trip guidance) crossed with eleven tourism themes, each expanded into checklists of verifiable sub-elements (e.g., avalanche warning systems for ice-snow tourism). Evaluation uses the composite score $S_{total} = (\\alpha S_{base} + \\beta S_{specific}) \\cdot F_{eff}$, where $S_{base}$ counts coverage of general planning/consultation/guidance elements, $S_{specific}$ counts theme-specific elements, and $F_{eff}$ is a logistic efficiency factor $N/L$ (elements per character) that rewards concise density. This machinery turns 'quality' into countable coverage plus text e","core_discovery":"On its own terms, the paper's central discovery is that expert knowledge can be encoded as a hierarchical tree-of-thought and used twice: as an optimizing prompt and as a label-free scoring oracle. The authors claim the optimized expert ToT improves response quality by 4.99–14.15% over baseline prompts across seven quality dimensions. When the same ToT elements are used to grade models, the resulting ranking—DS-V3 first, then DS-70B, DS-32B, Qwen-72B, Qwen-32B—contradicts the HuggingFace Open LLM Leaderboard ordering over the sub-72B models, and the authors read this reversal as evidence that generic benchmarks miss domain-specific competence such as reasoning and conciseness.","pith_inferences":["Beyond the paper: the efficiency factor makes the metric sensitive to response length and element density, so a model that lists many checklist items tersely could score well without actually being helpful; an independent human correlation study is the natural next test.","The reversal against the Open LLM Leaderboard is interpreted by the authors as evidence that their metric captures domain expertise, but an equally plausible reading is that the two instruments measure different constructs; the paper does not test which better predicts tourist satisfaction.","One unstated consequence: if label-free expert ToT scores are trustworthy, they could serve as reward signals for preference optimization or as filters for synthetic training data in tourism, removing the need for human preference labels.","The framework's taxonomy (3 query types × 11 themes) could transfer to adjacent service domains such as healthcare navigation or legal advice by substituting the element checklists, a direction the conclusion gestures at but does not test."],"forward_implications":["Tourism QA can be evaluated without building a labeled benchmark: a validated expert ToT plus the composite score yields a model ranking.","Prompt optimization guided by the same ToT improves outputs across all seven quality dimensions, with the largest gains in thematic relevance (+14.15%) and contextual appropriateness (+13.85%).","Scaling laws persist in this specialized domain, with DeepSeek-V3 leading, but reasoning-enhanced models at 32B–70B narrow the gap.","Below 72B parameters, explicit reasoning architectures significantly outperform non-reasoning models ($p < 0.05$) on accuracy and conciseness, a difference generic leaderboards hide.","The modular element structure should transfer LETToT to other expertise-heavy domains by substituting the theme-specific checklists."],"supporting_citations":[{"why":"Supplies the tree-of-thought reasoning structure that LETToT adapts into expert hierarchies.","marker":"(Yao et al. 2023)"},{"why":"Provides the generic Open LLM Leaderboard whose sub-72B ranking LETToT's results contradict.","marker":"(HuggingFace 2025)"},{"why":"Supplies the travel-phase query classification (planning, pre-trip, on-trip) used to structure the ToT.","marker":"(Kang, Jodice, and Norman 2020)"},{"why":"Supports behavior-based categorization of tourism queries, used in domain query analysis.","marker":"(Nautiyal et al. 2023)"},{"why":"Provides a tourism-type ontology and classification informing the eleven tourism themes.","marker":"(Zhang, Wang, and Li 2023)"},{"why":"Frames tourism question answering as a distinct domain requiring domain-specific evaluation.","marker":"(Contractor et al. 2019)"},{"why":"Presents TourLLM, a tourism knowledge-graph system whose evaluation needs LETToT addresses.","marker":"(Wei et al. 2024)"}],"fun_headline_variants":["Label-free expert trees score LLMs in tourism","ToT graders beat labels: 14% quality lift in tourism","Expert reasoning trees flip tourism LLM rankings","Tree-of-thought scoring outranks generic LLM benchmarks"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole framework rests on the assumption that the expert-defined ToT element list is a valid and complete operationalization of tourism answer quality; if those elements do not match what human experts or tourists judge as good answers, the scores and rankings lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Label-free expert trees score LLMs in tourism","ToT graders beat labels: 14% quality lift in tourism","Expert reasoning trees flip tourism LLM rankings","Tree-of-thought scoring outranks generic LLM benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1105,"prompt_tokens":799,"completion_tokens":306,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":543,"tokens_out":306,"duration_ms":3985,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:00:54.691235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of tourism QA responses, score them with LETToT's $S_{total}$, and collect independent human expert ratings (or tourist outcome judgments) on the same responses. If the LETToT ranking does not correlate with the human ranking—especially over the sub-72B models where it reverses the generic leaderboard—the central claim fails.","supporting_citations":[{"cited_title":"L.; Cao, Y.; and Narasimhan, K","cited_arxiv_id":null,"evidence_quote":"Supplies the tree-of-thought reasoning structure that LETToT adapts into expert hierarchies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the generic Open LLM Leaderboard whose sub-72B ranking LETToT's results contradict."},{"cited_title":"W.; and Norman, W","cited_arxiv_id":null,"evidence_quote":"Supplies the travel-phase query classification (planning, pre-trip, on-trip) used to structure the ToT."},{"cited_title":"To use or not to use","cited_arxiv_id":null,"evidence_quote":"Supports behavior-based categorization of tourism queries, used in domain query analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a tourism-type ontology and classification informing the eleven tourism themes."},{"cited_title":"Large Scale Question Answering using Tourism Data","cited_arxiv_id":"1909.03527","evidence_quote":"Frames tourism question answering as a distinct domain requiring domain-specific evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents TourLLM, a tourism knowledge-graph system whose evaluation needs LETToT addresses."}],"review_version":1}