Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Large Language Models in the Travel Domain: An Industrial Experience

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A larger open-weights model outperformed a fine-tuned smaller model on generated hotel descriptions and was adopted into production despite about ten times the compute cost.

desk verdict A readable industrial case study with useful cost numbers, but the headline quality comparison rests on unblinded manual scoring of 20 facilities and needs much stronger evaluation before those numbers are taken seriously. read the letter →

arxiv 2507.22910 v1 pith:OWLAP34T submitted 2025-07-18 cs.CL cs.AI

Sergio Di Meglio , Aniello Somma , Luigi Libero Lucio Starace , Fabio Scippacercola , Giancarlo Sperlì , Sergio Di Martino This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelstraveldomainhotelpropertydescriptionshallucinationevaluationQLoRAfine-tuningMixtral8x7BMistral7Bproductiondeploymentcosttrade-offs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports an industrial deployment of CaleidoGen, an LLM-based module that writes consistent, complete accommodation descriptions for the CALEIDOHOTELS booking platform from structured facility features. The authors compare two strategies: QLoRA fine-tuning of Mistral 7B and a refined system prompt on Mixtral 8x7B. On 20 held-out properties, Mixtral 8x7B reached 99.6% feature completeness and 98.8% precision with 1.2% hallucinated details, versus 93%, 96%, and 4% for Mistral 7B-FT, while generating slightly shorter text. The company adopted Mixtral 8x7B as the core model despite needing about 50GB of VRAM and $1.61 per hour, roughly ten times the Mistral configuration's cost. The paper argues this shows LLMs can improve catalog consistency in production, with a clear trade-off between output quality and resource efficiency.

What carries the argument

The operation centers on a structured 'Context' field that turns a facility's catalog entry into categories such as Recreation, Services, Dining, Rooms, Additional Services, and nearby Points of Interest, which both models receive as input. One arm of the comparison adds QLoRA, a quantized low-rank adaptation method, to Mistral 7B; the other arm replaces instruction fields with a refined system prompt on Mixtral 8x7B, requiring customization of the tokenizer template to support the SYSTEM role. Output quality is measured by manually mapping each context feature to the generated text and counting completeness, precision, and hallucinated details, with the process repeated five times to absorb the models' non-deterministic behavior.

What would settle it

Re-run the protocol with independent annotators who do not know which model produced each description, on a larger sample drawn across the platform's data sources; a statistically significant reversal or large disagreement between annotators would refute the claimed ordering.

Watch

Extended reading notes

Core claim

The central claim is that a larger mixture-of-experts model driven by a refined system prompt can outperform a domain-fine-tuned smaller model for structured product descriptions in a real booking system. For the facilities tested, Mixtral 8x7B was more complete (99.6% of context features) and more precise (98.8%), with far fewer hallucinations (1.2%), and its texts were shorter (249 versus 277 words). These gains came at approximately ten times the compute cost: 50GB of VRAM and $1.61 per hour versus 5GB and $0.16 per hour. The authors conclude that Mixtral 8x7B is the best core model for CALEIDOHOTELS, while the fine-tuned Mistral 7B remains a viable option under stricter resource constraints.

Load-bearing premise

The quality comparison rests on the authors' own manual mapping of catalog features to generated text for 20 facilities, repeated five times, with no inter-annotator agreement or independent validation; an unrepresentative or biased mapping would change the reported gaps.

Editorial extensions

If this is right

  • If the reported numbers hold, a booking platform can generate uniform, enriched descriptions for listings whose providers give sparse or inconsistent data.
  • Mixtral 8x7B can be a production-viable core model when factual accuracy matters more than serving cost, as the paper says CALEIDOHOTELS chose.
  • A fine-tuned 7B model can still reach 93-96% quality at one-tenth of the cost, so more constrained deployments have a defensible lower-cost option.
  • Prompt engineering on a larger model can beat fine-tuning on a smaller model for this structured text-generation task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's cost comparison uses current cloud spot prices; if inference costs fall or smaller models improve with better tuning, the adoption decision could reverse.
  • The manual feature-mapping protocol measures text quality, not user behavior; a natural next experiment would link description quality to booking or engagement rates.
  • The same structured-context evaluation could be reused to audit hallucinations in other data-to-text production settings beyond hotel descriptions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes CaleidoGen, an industrial LLM-based system developed for the CALEIDOHOTELS property booking platform to generate consistent accommodation descriptions from heterogeneous catalog data. The authors compare two models: Mistral 7B fine-tuned with QLoRA on a small custom dataset, and Mixtral 8x7B used with a refined system prompt. Based on 20 hand-annotated facilities, they report that Mixtral 8x7B achieves higher completeness (99.6% vs. 93%), higher precision (98.8% vs. 96%), a lower hallucination rate (1.2% vs. 4%), and slightly shorter descriptions (249 vs. 277 words), at roughly ten times the computational cost ($1.61/hour vs. $0.16/hour and 50GB vs. 5GB VRAM). The paper concludes that Mixtral 8x7B was adopted as the core model, and offers practical lessons about prompt design, tokenizer customization, device mapping, and cost trade-offs.

Significance. If the evaluation were robust, this would be a useful industrial data point on the quality/cost trade-off of open-weight LLMs in a concrete travel-domain application. The paper's main strengths are its honest reporting of deployment constraints (VRAM, hourly cost, tokenizer limitations, device-map issues) and the description of an iterative pipeline with three phases. It also makes a specific, falsifiable model comparison. However, the comparative claim rests on a small, author-scored, unblinded sample with no baseline and no inter-annotator reliability measure, so the reported numerical advantages should be treated as preliminary rather than established. As an experience report the paper is valuable; as a controlled evaluation it is currently under-powered.

major comments (4)
  1. [Section IV-A, Table I] The hallucination metric as defined is redundant with precision: at the aggregate level, the reported hallucination rate is exactly 100% minus precision for both models (4% = 100% - 96%, 1.2% = 100% - 98.8%), and both metrics use the same denominator, Total Features Added. The paper should either clarify what additional information the hallucination metric provides or remove it from the headline results. The reported standard deviations are also internally inconsistent with exact complementarity: Mistral 7B-FT shows precision SD of 3.2 but hallucination SD of 3.8, which would be impossible if per-facility values were exact complements.
  2. [Section IV, evaluation protocol] The central model-vs-model claim depends entirely on two authors manually mapping context features to generated text for only 20 facilities, with no described sampling procedure, no blinding to model identity, and no inter-annotator agreement measure. The claim in Section IV that this 'ensuring an unbiased assessment' is stronger than the protocol supports: the annotators know which model produced each description, and a systematic preference for the more fluent Mixtral output could account for the 2.8pp precision gap. Please report inter-annotator reliability on a blind subset, describe how the 20 facilities were selected, and state whether the five repetitions involved re-annotation or only re-generation.
  3. [Section IV-B, Section VI] There is no baseline against the status quo, such as the original provider descriptions or the fallback descriptions previously used by the platform. Thus the conclusion in Section VI that CaleidoGen improves the consistency and reliability of accommodation data is not supported by the presented data; the experimental design only compares two candidate models. Add a baseline comparison or temper the conclusions to a model-vs-model comparison.
  4. [Section IV-B, Table I] No confidence intervals or significance tests are reported for the differences in completeness, precision, or hallucination. Given n=20 and the large per-facility standard deviations (e.g., Mistral completeness SD of 8.8pp), the reader cannot tell whether the completeness gap of 6.6pp is stable or within sampling noise. Report per-facility distributions, bootstrap confidence intervals, or a paired statistical test, and clarify whether the standard deviations are computed across facilities or across the five repetitions.
minor comments (5)
  1. [Abstract] The abstract contains a typo: 'well-know LLMs' should be 'well-known LLMs'.
  2. [Section III-A] The fine-tuning setup is under-specified: the paper mentions a 100-item training dataset but does not report hyperparameters (learning rate, number of epochs, LoRA rank, batch size) or the train/validation split. Providing these would improve reproducibility.
  3. [Section III-B.2.a] The tokenizer modification for Mixtral is an interesting practical detail, but the description is vague ('customize the tokenizer template'). A short example or a reference to the exact change would help readers facing the same issue.
  4. [Section IV] It is unclear whether the five repetitions were run on the same 20 facilities and whether the manual annotation was repeated for each run; please clarify the experimental workflow.
  5. [References] Several references in Sections I and II are to the authors' own previous work on software performance and testing, and their connection to the travel-domain LLM content is not obvious; consider trimming or integrating them more explicitly into the narrative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the model comparison is an external empirical evaluation, not a derivation from its own outputs or a fitted-parameter prediction.

full rationale

The paper's central claims are empirical comparisons between two LLM outputs computed from count-based metrics defined over catalog context features (Completeness, Precision, Hallucinations, Length). The 20 test facilities are stated to be outside the optimization dataset, so the evaluation is external to training. The metrics are not fitted to the headline results and no parameter is tuned to the target comparison. Self-citations appear in the introduction and related work (e.g., refs. [2], [3], [5], [6], [8], [9], [14]), but they support background assertions about performance testing, LLM use, and data preprocessing; none is load-bearing for the Mixtral-versus-Mistral result. The hallucination metric is the complement of precision by definition, so it is redundant evidence rather than an independent measurement; this is a metric-design observation, not a circular derivation. The main threats are methodological (unblinded manual scoring by two authors, no inter-annotator agreement, small sample of 20), which affect confidence but do not make the claim reduce to its own inputs. Therefore no circular step can be exhibited under the required standard, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters were fitted to the reported outcomes; the analysis depends on assumptions about annotation quality, representativeness, and the completeness of the catalog-derived context, rather than on a mathematical derivation. The software system CaleidoGen is an engineering artifact, not a postulated entity.

assumptions (3)
  • domain assumption Manual feature annotation by two authors is an unbiased ground truth for correctness and hallucination.
    Section IV: The evaluation metrics are computed from the authors' mapping of context features to generated text; no inter-annotator agreement or independent validation is reported.
  • domain assumption The 20 test facilities are representative of the catalog and distinct from the training data.
    Section IV: The paper states the facilities were not in the optimization dataset, but gives no sampling description; repeated runs use the same 20 items.
  • domain assumption The structured Context field extracted from the primary catalog is complete and accurate for measuring completeness.
    Section III-A and Section IV: The completeness metric treats context features as ground truth; if the catalog itself is incomplete, the metric is biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models in the Travel Domain: An Industrial Experience." pith.science (2026). https://pith.science/paper/OWLAP34T

@misc{pith2026250722910,
  author       = {Pith},
  title        = {Pith review of: Large Language Models in the Travel Domain: An Industrial Experience},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWLAP34T}},
  note         = {Machine review of arXiv:2507.22910}
}
abstract

Online property booking platforms are widely used and rely heavily on consistent, up-to-date information about accommodation facilities, often sourced from third-party providers. However, these external data sources are frequently affected by incomplete or inconsistent details, which can frustrate users and result in a loss of market. In response to these challenges, we present an industrial case study involving the integration of Large Language Models (LLMs) into CALEIDOHOTELS, a property reservation platform developed by FERVENTO. We evaluate two well-known LLMs in this context: Mistral 7B, fine-tuned with QLoRA, and Mixtral 8x7B, utilized with a refined system prompt. Both models were assessed based on their ability to generate consistent and homogeneous descriptions while minimizing hallucinations. Mixtral 8x7B outperformed Mistral 7B in terms of completeness (99.6% vs. 93%), precision (98.8% vs. 96%), and hallucination rate (1.2% vs. 4%), producing shorter yet more concise content (249 vs. 277 words on average). However, this came at a significantly higher computational cost: 50GB VRAM and $1.61/hour versus 5GB and $0.16/hour for Mistral 7B. Our findings provide practical insights into the trade-offs between model quality and resource efficiency, offering guidance for deploying LLMs in production environments and demonstrating their effectiveness in enhancing the consistency and reliability of accommodation data.

Figures

Figures reproduced from arXiv: 2507.22910 by the authors.

Figure 1
Figure 1. CaleidoGen’s iterative process A. Data Preprocessing One of the output of the catalog analysis phase was the input for Phase 2, the Data Preprocessing phase. The latter serves two primary purposes: (i) to create a training dataset for the model’s fine-tuning, and (ii) to extract the necessary context for creating the test dataset. Developing a dedicated dataset for further training the model is crucial [11], [12], a… view at source ↗
Figure 2
Figure 2. This example presents the formatting of a dataset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example of the finetuning prompt of Mistral 7B [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example of System prompt used with Mixtral 8x7B [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Analysis of the generation • Hallucinations: This evaluation metric focuses on the pres￾ence of hallucinations within the description generated. Hallucinations = Hallucinated Features Total Features Added × 100 • Length of generation: The final metric under considerati…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [1]

    Online search engines and online travel agencies: A comparative approach,

    S. Angeloni and C. Rossi, “Online search engines and online travel agencies: A comparative approach,” Journal of Hospitality & Tourism Research, vol. 45, no. 4, pp. 720–749, 2021. [Online]. Available: https://doi.org/10.1177/1096348020980101

    work page doi:10.1177/1096348020980101 2021

  2. [2]

    Performance testing in open-source web projects: Adoption, maintenance, and a change taxonomy,

    S. Di Meglio, L. L. L. Starace, V . Pontillo, R. Opdebeeck, C. De Roover, and S. Di Martino, “Performance testing in open-source web projects: Adoption, maintenance, and a change taxonomy,” in 41st IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME 2025). IEEE, 2025

    work page 2025

  3. [3]

    Evaluating performance and resource consumption of rest frameworks and execution environments: Insights and guidelines for developers and companies,

    S. Di Meglio and L. L. L. Starace, “Evaluating performance and resource consumption of rest frameworks and execution environments: Insights and guidelines for developers and companies,” IEEE Access, 2024

    work page 2024

  4. [4]

    Tourbert: A pretrained language model for the tourism industry,

    V . Arefieva and R. Egger, “Tourbert: A pretrained language model for the tourism industry,” 2022

    work page 2022

  5. [5]

    Large language models in software engineering: A focus on issue report classification and user acceptance test generation,

    G. De Vito, L. L. L. Starace, S. Di Martino, F. Ferrucci, and F. Palomba, “Large language models in software engineering: A focus on issue report classification and user acceptance test generation,” in Proceedings of the Ital-IA Conference, Naples, Italy , 2024, pp. 48–53

    work page 2024

  6. [6]

    Can large language models automatically generate gis reports?

    L. L. L. Starace and S. Di Martino, “Can large language models automatically generate gis reports?” in International Symposium on Web and Wireless Geographical Information Systems . Springer, 2024, pp. 147–161

    work page 2024

  7. [7]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” 2023. [Online]. Available: https://arxiv.org/abs/2305.14314

    arXiv 2023

  8. [8]

    Starting a new rest api project? a performance benchmark of frameworks and execution environments

    S. Di Meglio, L. L. L. Starace, and S. Di Martino, “Starting a new rest api project? a performance benchmark of frameworks and execution environments.” in IWSM-Mensura, 2023

    work page 2023

Show all 31 references
  1. [9]

    E2e-loader: A tool to generate performance tests from end-to-end gui-level tests,

    S. Di Meglio, L. Libero Lucio Starace, and S. Di Martino, “E2e-loader: A tool to generate performance tests from end-to-end gui-level tests,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), 2025, pp. 747–751

    2025

  2. [10]

    Indicators of website features in the user experience of e-tourism search and metasearch engines,

    T. D. Vila, E. A. Gonz ´alez, N. Ara ´ujo Vila, and J. A. Fraiz Brea, “Indicators of website features in the user experience of e-tourism search and metasearch engines,” Journal of theoretical and applied electronic commerce research, vol. 16, no. 1, pp. 18–36, 2021

    2021

  3. [11]

    An empirical analysis of data preprocessing for machine learning-based software cost estimation,

    J. Huang, Y .-F. Li, and M. Xie, “An empirical analysis of data preprocessing for machine learning-based software cost estimation,” Information and software Technology , vol. 67, pp. 108–127, 2015

    2015

  4. [12]

    What is data preprocessing in ml?

    Y . Gavrilova and O. Bolgurtseva, “What is data preprocessing in ml?”

  5. [13]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” 2023

    2023

  6. [14]

    A visual-based toolkit to support mobility data analytics,

    S. Di Martino, E. Landolfi, N. Mazzocca, F. R. di Torrepadula, and L. L. L. Starace, “A visual-based toolkit to support mobility data analytics,” Expert Systems with Applications , vol. 238, p. 121949, 2024

    2024

  7. [15]

    Fine-tuning large language models for adaptive machine translation,

    Y . Moslem, R. Haque, and A. Way, “Fine-tuning large language models for adaptive machine translation,” arXiv preprint arXiv:2312.12740 , 2023

    2023 arXiv

  8. [16]

    Evaluating large language models: Chatgpt-4, mistral 8x7b, and google gemini benchmarked against mmlu,

    K. Ono and A. Morita, “Evaluating large language models: Chatgpt-4, mistral 8x7b, and google gemini benchmarked against mmlu,” Authorea Preprints, 2024

    2024

  9. [17]

    Open llm leaderboard v2,

    C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf, “Open llm leaderboard v2,” https://huggingface.co/spaces/open-llm-leaderboard/ open llm leaderboard, 2024

    2024

  10. [18]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: ht...

    2023 arXiv

  11. [19]

    A survey of large language models,

    W. X. e. a. Zhao, “A survey of large language models,” 2023. [Online]. Available: http://arxiv.org/abs/2303.18223

    2023 arXiv

  12. [20]

    Prompt engineering for generative ai,

    Google, “Prompt engineering for generative ai,” Get Started with Machine Learning, 2023. [Online]. Available: https://developers.google. com/machine-learning/resources/prompt-eng

    2023

  13. [21]

    Mixtral of experts,

    A. Q. J. et al., “Mixtral of experts,” 2024. [Online]. Available: https://arxiv.org/abs/2401.04088

    2024 arXiv

  14. [22]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    S. Gugger, L. Debut, T. Wolf, P. Schmid, Z. Mueller, S. Mangrulkar, M. Sun, and B. Bossan, “Accelerate: Training and inference at scale made simple, efficient and adaptable.” https://github.com/huggingface/ accelerate, 2022

    2022

  15. [23]

    Tokenizer,

    A. Zucker, “Tokenizer,” 2023, november 6, 2023. [Online]. Available: https://huggingface.co/docs/transformers/main classes/tokenizer

    2023

  16. [24]

    An exploratory study on how non-determinism in large language models affects log parsing,

    M. Astekin, M. Hort, and L. Moonen, “An exploratory study on how non-determinism in large language models affects log parsing,” in Pro- ceedings of the ACM/IEEE 2nd International Workshop on Interpretabil- ity, Robustness, and Benchmarking in Neural Software Engineering , ser....

    2024

  17. [25]

    Finesure: Fine-grained summarization evaluation using llms,

    H. Song, H. Su, I. Shalyminov, J. Cai, and S. Mansour, “Finesure: Fine-grained summarization evaluation using llms,” 2024. [Online]. Available: https://arxiv.org/abs/2407.00908

    2024 arXiv

  18. [26]

    Unveiling llm evaluation focused on metrics: Challenges and solutions,

    T. Hu and X.-H. Zhou, “Unveiling llm evaluation focused on metrics: Challenges and solutions,” 2024. [Online]. Available: https: //arxiv.org/abs/2404.09135

    2024 arXiv

  19. [27]

    Tourllm: Enhancing llms with tourism knowledge,

    Q. Wei, M. Yang, J. Wang, W. Mao, J. Xu, and H. Ning, “Tourllm: Enhancing llms with tourism knowledge,” 2024. [Online]. Available: https://arxiv.org/abs/2407.12791

    2024 arXiv

  20. [28]

    A fine-tuned tourism-specific generative ai concept,

    C. H. Hsu, G. Tan, and B. Stantic, “A fine-tuned tourism-specific generative ai concept,” Annals of Tourism Research, vol. 104, p. 103723, 2024, annals of Tourism Research: 50th Anniversary Issue

    2024

  21. [29]

    Chatgpt and the hospitality and tourism industry: an overview of current trends and future research directions,

    Y . L. Dogan Gursoy and H. Song, “Chatgpt and the hospitality and tourism industry: an overview of current trends and future research directions,” Journal of Hospitality Marketing & Management , vol. 32, no. 5, pp. 579–592, 2023

    2023

  22. [30]

    Ai-powered chatgpt in the hospitality and tourism industry: benefits, challenges, theoretical framework, propositions and future research directions,

    R. A. Rather, “Ai-powered chatgpt in the hospitality and tourism industry: benefits, challenges, theoretical framework, propositions and future research directions,” Tourism Recreation Research, vol. 0, no. 0, pp. 1–11, 2024. [Online]. Available: https://doi.org/10.1080/025082...

    2024

  23. [2020]

    Available: https://serokell.io/blog/data-preprocessing

    [Online]. Available: https://serokell.io/blog/data-preprocessing

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.