Generating Search Explanations using Large Language Models

Arif Laksito; Mark Stevenson

REVIEW 3 major objections 4 minor 15 references

Generating Search Explanations using Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned LLMs generate more accurate and plausible search-result explanations than baseline models.

desk verdict A plausible small win for fine-tuned LLMs on explanation generation, with an abstract that claims more than the metrics support. read the letter →

arxiv 2507.16692 v1 pith:I2GHRPT7 submitted 2025-07-22 cs.IR

Arif Laksito , Mark Stevenson This is my paper · ORCID

classification cs.IR

keywords Aspect-orientedexplanationsExplainableinformationretrievalLargelanguagemodelsFine-tuningQLoRASearchresultMETEORBERTScore

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models, fine-tuned on a suitable corpus, can produce the short aspect-oriented explanations that search engines display alongside results to tell a user why a document matches an underspecified query. The authors build a dataset from Wikipedia in which article titles serve as queries and section headings serve as reference explanations, then fine-tune both encoder-decoder models (BART, T5) and decoder-only models (LLaMA v2 13B, LLaMA v3 70B) and compare them with from-scratch transformer baselines. They report that all fine-tuned models beat the baselines on METEOR, ROUGE-1, and BERTScore, with fine-tuned LLaMA v3 70B attaining the highest scores. A sympathetic reader would care because better search explanations could let users judge result relevance at a glance, and the finding suggests that fine-tuning existing pretrained models is a simpler path to this capability than training specialized architectures from scratch.

What carries the argument

The load-bearing mechanism is the fine-tuning recipe rather than a new architecture. Smaller encoder-decoder models (BART, T5) receive full fine-tuning with a natural-language input format, while the large decoder-only LLaMA models are adapted with QLoRA, which combines 4-bit quantization with low-rank adaptation to fit on one GPU. The evaluation rests on a Wikipedia-derived dataset in which article titles become queries and section headings become reference aspect explanations, with the sections limited to 128–512 tokens, and on the three automatic metrics METEOR, ROUGE-1, and BERTScore that quantify lexical and semantic overlap with those references.

What would settle it

Have human annotators rate the generated explanations for a held-out set of underspecified queries against the actual Wikipedia section headings for accuracy and plausibility, then check whether the fine-tuned LLaMA v3 70B explanations receive higher ratings than the from-scratch transformer baselines. The paper's central claim fails if the higher METEOR, ROUGE-1, and BERTScore received by the fine-tuned models do not correspond to higher human ratings.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-tuning pretrained large language models for aspect-oriented explanation generation yields explanations that are consistently more accurate and plausible than those from baseline systems. In the reported experiments, every fine-tuned model — BART, T5, LLaMA v2 13B, and LLaMA v3 70B — outperforms the from-scratch Transformer, Bert2Bert, Bert2GPT, and zero-shot LLaMA baselines across all three automatic metrics. The strongest result is the fine-tuned LLaMA v3 70B model, which reaches METEOR 0.3222, ROUGE-1 0.4993, and BERTScore 0.5652, while the much smaller fine-tuned T5 remains close on all metrics at a fraction of the training and inference cost. The paper also finds that zero-shot prompting of large decoder-only models alone is not sufficient, so supervised fine-tuning is necessary for the task.

Load-bearing premise

The whole comparison rests on treating Wikipedia section headings as the correct aspects and the three automatic metrics as trustworthy measures of explanation accuracy and plausibility.

Editorial extensions

If this is right

Fine-tuned LLMs can be used directly to generate aspect-oriented explanations for search results, giving a cheaper path than training specialized transformer models from scratch.
Scaling to larger models improves explanation quality on automatic metrics, so organisations with enough compute can push quality higher by fine-tuning 70B-scale models.
Mid-size encoder-decoder models such as T5 and BART offer most of the quality gain at far lower training and inference cost, making them the practical choice when resources are limited.
Zero-shot prompting alone is insufficient for this task, so supervised fine-tuning is required even for very large decoder-only LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension beyond the paper would be to test the same fine-tuning recipe on other explanation sources, such as FAQs, user reviews, or entity descriptions, to see whether the gains hold outside encyclopedic content.
Because the paper evaluates with automatic metrics only, a human-judgment study would clarify whether the higher scores actually translate into explanations users find more plausible and easier to scan.
The low cost of the fine-tuned T5 model suggests a deployable recipe: generate explanations offline for frequent queries with a mid-size model, reserving larger models for hard or rare query aspects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a reasonable small empirical study, but the abstract oversells what the metrics actually show. The real contribution is a clean comparison table showing fine-tuned BART/T5 and QLoRA-tuned LLaMA models beat trained-from-scratch transformer baselines on METEOR/ROUGE/BERTScore for Wikipedia heading generation. That result is plausible and probably reproducible. But the phrase "more accurate and plausible" in the abstract is not supported: those metrics measure lexical and embedding overlap with section headings, not plausibility as judged by a searcher. No human evaluation appears anywhere. If the authors softened the claim to "closer to the reference distribution" or added a small user study, the paper would be honest.

What's new: prior work used modified Transformer architectures trained from scratch; this paper shows pretrained LLMs fine-tuned (full or QLoRA) do much better on this task. That's worth knowing for anyone working on explainable IR. The natural-language input formatting and instruction tuning are minor variations, but the empirical result is new.

Soft spots: (1) The baseline set omits the actual prior task-specific architectures from [9, 14] - they compare against generic Transformer, Bert2Bert, Bert2Gpt trained from scratch, not the query-attention models those papers proposed. So the claim of beating "a range of baseline models" sidesteps the strongest prior systems. (2) LLaMA models got 1 epoch versus 5 for everyone else, so the ranking among fine-tuned models is not apples-to-apples; the broad conclusion likely survives, but the 70B-vs-T5 margin is soft. (3) No significance tests or error bars; the numbers in Table 1 are point estimates. (4) The dataset uses Wikipedia section headings as explanations, which is a convenient proxy but far from what a user would find plausible in a real search UI.

None of these is fatal. The central empirical claim - that fine-tuning LLMs beats training from scratch on this proxy task - is probably true. The fix is mostly reframing: either add human evaluation or scale back the abstract.

Who this is for: IR researchers working on explanation generation or snippet quality; it's a short workshop-style paper, not a definitive study. It deserves a serious referee, but the referee should push for the missing baseline and the human eval.

Referee Report

3 major / 4 minor

Summary. The paper investigates using large language models (LLMs) to generate aspect-oriented explanations for search results. The authors construct a dataset from English Wikipedia in which article titles are queries and section headings are reference explanations. They fine-tune encoder-decoder models (BART, T5) and decoder-only models (LLaMA 13B/70B with QLoRA), and compare against several baselines (a small Transformer, Bert2Bert, Bert2GPT, and zero-shot LLaMA). Using METEOR, ROUGE-1, and BERTScore, they report that all fine-tuned models outperform the baselines, with LLaMA v3 (70B) achieving the highest scores. The paper concludes that larger models yield superior performance while smaller fine-tuned encoder-decoder models remain competitive in efficiency.

Significance. If the empirical comparison is accepted, the paper provides a useful, clearly described benchmark for aspect-based explanation generation using modern LLMs. The dataset construction is described in detail and a repository is referenced, which supports reproducibility. The main finding, that fine-tuning pretrained models substantially improves over from-scratch baselines on automatic metrics, is plausible and consistent with prior work. However, the paper's central claim—that the generated explanations are 'consistently more accurate and plausible'—goes beyond what the automatic overlap-based metrics can establish. No human evaluation or user study is reported, and the metric-to-quality inference is unvalidated for this task. The comparison among models is also confounded by unequal training budgets. These issues are load-bearing for the abstract's claim, so the contribution is currently moderate; with appropriate re-scoping or added evidence, the result would be a solid empirical short paper.

major comments (3)

[Abstract and Section 4 (Results)] The abstract claims that the generated explanations are 'consistently more accurate and plausible' than baselines, but Table 1 only reports METEOR, ROUGE-1, and BERTScore. These metrics measure n-gram or embedding overlap with the Wikipedia section-heading references, not whether a user would judge an explanation as accurate or plausible. The paper contains no human evaluation, expert annotation, or user study. The claim as stated is therefore unsupported; it should either be rephrased to refer to overlap-based similarity or complemented with a human judgment study.
[Section 3 (Evaluation)] The LLaMA models are fine-tuned for only 1 epoch, while all encoder-decoder models are trained for 5 epochs. This introduces a confound: the conclusion in Section 4 that 'larger models yield superior performance' is based on comparing models trained with different training effort. The comparison between FT LLaMA and FT T5/BART, and the comparison among LLaMA variants, would only be meaningful if training budgets are matched or if the effect of epoch count is analyzed.
[Table 1 and Section 4] No error bars, variance estimates, or significance tests are reported for any of the metrics. Since runs appear to be reported for a single seed or configuration, the differences between FT T5 and FT LLaMA v2 (e.g., METEOR 0.2723 vs. 0.2759) and between FT T5 and FT LLaMA v3 (BERTScore 0.5202 vs. 0.5652) may not be statistically reliable. The claim that LLaMA v3 'achieved the highest overall scores' should be qualified or supported with significance testing or repeated runs.

minor comments (4)

[Section 2 (Approach)] The phrase 'we utilize fine-tuning of LLMs' is slightly awkward; 'we fine-tune LLMs' is clearer. Additionally, Figure 1 is referenced but not described in enough detail in the text to understand the input-output structure without looking at the figure.
[Table 1 caption] The caption states 'Training time is reported for one epoch,' but the text in Section 3 says all encoder-decoder models were trained for 5 epochs. Clarify whether the reported training times are per epoch or total, and whether the LLaMA training times are also per epoch.
[Section 3 (Evaluation)] The dataset construction footnote appears after 'randomly assigning groups' but the link is split across lines in the provided text; ensure the repository URL is presented cleanly and that the dataset versioning is described.
[References] Several references omit URLs or DOIs (e.g., [2], [9], [14] are given with arXiv identifiers but some lack the full link). For a camera-ready version, all references should be complete and consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison is a held-out supervised benchmark, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is an empirical comparison: it constructs a dataset from Wikipedia article titles as queries and section headings as reference explanations, splits it into training and test sets, fine-tunes or prompts LLMs, and measures output overlap with held-out reference headings using METEOR, ROUGE-1, and BERTScore. There is no fitted parameter that is later relabeled as a prediction, no self-consistency equation that makes the outcome true by definition, and no load-bearing self-citation, since the defining references [9, 14] are external prior work and the authors of the present paper are Laksito and Stevenson. The possible objection that lexical and embedding overlap metrics do not directly measure human-perceived accuracy or plausibility is a construct-validity limitation, not circularity: it concerns whether the benchmark measures what the abstract claims, not whether the benchmark result was forced by construction. Because the test references are unseen and the models are not designed to reproduce the evaluation metric definitionally, the reported ranking is an ordinary empirical result with independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new theoretical quantities or entities are introduced. The central empirical claim depends on the Wikipedia-as-query/aspect assumption, on automatic metrics standing in for human quality judgments, and on several hand-chosen training hyperparameters. These are all reasonable for a small empirical paper, but they bound the scope of the conclusion.

free parameters (4)

Number of training epochs for encoder-decoder models = 5
Chosen by hand at a fixed value of 5 epochs in Section 3. LLaMA models are trained for only 1 epoch, which confounds model-size comparison with training budget.
Learning rate = 1e-5
Set to 1e-5 for all encoder-decoder models in Section 3. It is not tuned or varied, so results may depend on this choice.
Batch size = 8
Set to 8 for all encoder-decoder models in Section 3. It is not varied or justified.
QLoRA configuration for LLaMA models = not specified
Section 2 mentions 4-bit quantization and low-rank adaptation, but rank, alpha, target modules, and related hyperparameters are not reported, making the exact method hard to reproduce.

assumptions (4)

domain assumption Wikipedia article titles stand in for search queries and section headings stand in for aspect-oriented explanations.
Adopted from prior work [9, 14] and restated in Section 3. If this mapping is unrepresentative of real search behavior, the task definition and downstream evaluation are not valid for actual search systems.
domain assumption METEOR, ROUGE-1, and BERTScore measure explanation accuracy and plausibility.
Section 3 uses these automatic overlap metrics as the only evaluation. The abstract's claim of 'accurate and plausible' rests entirely on this assumption, with no human judgments collected.
domain assumption Training and test data come from the same Wikipedia distribution.
Both fine-tuning and evaluation use sections from the March 2024 English Wikipedia dump, as described in Section 3. Generalization to other query and document distributions is not tested.
domain assumption Pretrained LLM checkpoints are valid starting points for fine-tuning.
The paper fine-tunes BART, T5, and LLaMA without questioning whether pretrained weights transfer to this task. This is standard practice in NLP, but it remains an unstated background assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generating Search Explanations using Large Language Models." pith.science (2026). https://pith.science/paper/I2GHRPT7

@misc{pith2026250716692,
  author       = {Pith},
  title        = {Pith review of: Generating Search Explanations using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2GHRPT7}},
  note         = {Machine review of arXiv:2507.16692}
}

read the original abstract

Aspect-oriented explanations in search results are typically concise text snippets placed alongside retrieved documents to serve as explanations that assist users in efficiently locating relevant information. While Large Language Models (LLMs) have demonstrated exceptional performance for a range of problems, their potential to generate explanations for search results has not been explored. This study addresses that gap by leveraging both encoder-decoder and decoder-only LLMs to generate explanations for search results. The explanations generated are consistently more accurate and plausible explanations than those produced by a range of baseline models.

Figures

Figures reproduced from arXiv: 2507.16692 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computationa...

work page 2005
[2]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36 (Dec. 2023), 10088–10115

work page 2023
[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. doi:10.48550/arXiv.1810.04805 arXiv:1810.04805 [cs]

arXiv doi:10.48550/arxiv.1810.04805 2019
[4]

Mayu Iwata, Tetsuya Sakai, Takehiro Yamamoto, Yu Chen, Yi Liu, Ji-Rong Wen, and Shojiro Nishio. 2012. AspecTiles: Tile-Based Visualization of Diversified Web Search Results. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, Portland Oregon USA, 85–94. doi:10.1145/2348283.2348298

work page arXiv 2012
[5]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81

work page 2004
[6]

Sean MacAvaney, Craig Macdonald, Roderick Murray-Smith, and Iadh Ounis

work page
[7]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, ...

doi:10.3115/1073083 2002
[8]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. (2019)

work page 2019

Show all 15 references

[9]

Razieh Rahimi, Youngwoo Kim, Hamed Zamani, and James Allan. 2021. Explain- ing Documents’ Relevance to Search Queries. (Nov. 2021). arXiv:2111.01314

2021 arXiv
[10]

Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2015. Search Result Diversification. Found. Trends Inf. Retr. 9, 1 (March 2015), 1–90. doi:10.1561/ 1500000040

2015
[11]

Anastasios Tombros and Mark Sanderson. 1998. Advantages of Query Biased Summaries in Information Retrieval. In Proceedings of the 21st Annual Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Melbourne Australia, 2–10. doi:10.1145/...

1998
[12]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy...

2023 doi arXiv
[13]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associa...

2017
[14]

Puxuan Yu, Razieh Rahimi, and James Allan. 2022. Towards Explainable Search Results: A Listwise Explanation Generator. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (2022). doi:10.1145/3477495

2022 doi
[2021]

arXiv:2108.04026 [cs]

IntenT5: Search Result Diversification Using Causal Language Models. arXiv:2108.04026 [cs]

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computationa...

work page 2005

[2] [2]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36 (Dec. 2023), 10088–10115

work page 2023

[3] [3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. doi:10.48550/arXiv.1810.04805 arXiv:1810.04805 [cs]

arXiv doi:10.48550/arxiv.1810.04805 2019

[4] [4]

Mayu Iwata, Tetsuya Sakai, Takehiro Yamamoto, Yu Chen, Yi Liu, Ji-Rong Wen, and Shojiro Nishio. 2012. AspecTiles: Tile-Based Visualization of Diversified Web Search Results. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, Portland Oregon USA, 85–94. doi:10.1145/2348283.2348298

work page arXiv 2012

[5] [5]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81

work page 2004

[6] [6]

Sean MacAvaney, Craig Macdonald, Roderick Murray-Smith, and Iadh Ounis

work page

[7] [7]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, ...

doi:10.3115/1073083 2002

[8] [8]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. (2019)

work page 2019