{"id":"65401a08-59e4-49cf-8b4a-57d070afc0f3","arxiv_id":"2507.16692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuned LLMs generate Wikipedia-section-style search explanations that score higher on automatic overlap metrics than earlier trained-from-scratch transformer baselines.","lead":"This paper applies fine-tuned large language models to write short explanations for search results, using Wikipedia section headings as the training target. The fine-tuned models beat older transformer baselines on automatic text-overlap metrics, with the largest model, LLaMA v3 70B, scoring highest.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that outputs are 'more accurate and plausible' is supported only by automatic overlap metrics against Wikipedia section headings; no human evaluation grounds the accuracy/plausibility wording.","rationale":"I read the paper as a modest empirical study. The training setup is described, the table is internally consistent, and the dataset-construction code is linked, which is helpful. The single load-bearing link in the argument is from automatic metric scores to the abstract's accuracy/plausibility claim. METEOR, ROUGE-1, and BERTScore are all reference-overlap measures; for this dataset the reference is a section heading, not a human judgment of explanation quality. There is no evidence in the paper that higher overlap translates to the perceived qualities the abstract asserts. This is not a disagreement about consensus regarding metrics; it is a correctness risk specific to over-interpreting the numbers. The human evaluation I propose would directly settle it. Since the reader already conditioned the verdict on this missing evidence, my stress test does not move the verdict. I would keep it conditional: the empirical comparison is usable, but the headline claim should not be accepted as stated until human ratings or an equivalent validation are supplied.","tokens_in":4465,"tokens_out":3274,"duration_ms":35065,"concrete_test":"Draw a random sample of 100 test queries from the same split. For each query, collect outputs from FT LLaMA v3 (70B), FT T5, Bert2Gpt, and 0-shot LLaMA, plus the reference section heading. Ask at least three annotators, blinded to model identity, to rate each explanation for (a) accuracy and (b) plausibility on a 1-5 scale. Pre-register a paired comparison of mean ratings; if fine-tuned LLMs do not significantly outperform Bert2Gpt and 0-shot LLaMA, the abstract's central claim is unsupported. As a calibration sanity check, also rate the Wikipedia section headings themselves; if human raters do not consider the reference headings accurate or plausible explanations, the dataset proxy itself is called into question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3 the dataset is built by 'using Wikipedia article titles as queries and their section headings as aspect-based explanations' and the evaluation uses METEOR, ROUGE-1, and BERTScore. These metrics measure n-gram and embedding overlap with the section-heading reference, not whether a user would judge an output as an accurate or plausible explanation for why a document matches a query. Because the reference text is itself a heading, a model can score well by emitting fluent heading-like strings; the numbers in Table 1 therefore support only 'closer to the reference distribution,' not the abstract's 'consistently more accurate and plausible.' No human evaluation, user study, or even expert annotation is reported anywhere in the paper. The absence is load-bearing because the abstract makes a claim about perceived quality, and the automatic metrics do not measure plausibility. Secondary issues (unequal epochs for LLaMA, no prior task-specific architecture in the baseline set) would affect the ranking among fine-tuned models, but the abstract's comparison to 'a range of baseline models' falls mainly on the unvalidated metric-to-quality inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates using large language models (LLMs) to generate aspect-oriented explanations for search results. The authors construct a dataset from English Wikipedia in which article titles are queries and section headings are reference explanations. They fine-tune encoder-decoder models (BART, T5) and decoder-only models (LLaMA 13B/70B with QLoRA), and compare against several baselines (a small Transformer, Bert2Bert, Bert2GPT, and zero-shot LLaMA). Using METEOR, ROUGE-1, and BERTScore, they report that all fine-tuned models outperform the baselines, with LLaMA v3 (70B) achieving the highest scores. The paper concludes that larger models yield superior performance while smaller fine-tuned encoder-decoder models remain competitive in efficiency.","tokens_in":4769,"tokens_out":2127,"duration_ms":24421,"significance":"If the empirical comparison is accepted, the paper provides a useful, clearly described benchmark for aspect-based explanation generation using modern LLMs. The dataset construction is described in detail and a repository is referenced, which supports reproducibility. The main finding, that fine-tuning pretrained models substantially improves over from-scratch baselines on automatic metrics, is plausible and consistent with prior work. However, the paper's central claim—that the generated explanations are 'consistently more accurate and plausible'—goes beyond what the automatic overlap-based metrics can establish. No human evaluation or user study is reported, and the metric-to-quality inference is unvalidated for this task. The comparison among models is also confounded by unequal training budgets. These issues are load-bearing for the abstract's claim, so the contribution is currently moderate; with appropriate re-scoping or added evidence, the result would be a solid empirical short paper.","major_comments":[{"comment":"The abstract claims that the generated explanations are 'consistently more accurate and plausible' than baselines, but Table 1 only reports METEOR, ROUGE-1, and BERTScore. These metrics measure n-gram or embedding overlap with the Wikipedia section-heading references, not whether a user would judge an explanation as accurate or plausible. The paper contains no human evaluation, expert annotation, or user study. The claim as stated is therefore unsupported; it should either be rephrased to refer to overlap-based similarity or complemented with a human judgment study.","section":"Abstract and Section 4 (Results)"},{"comment":"The LLaMA models are fine-tuned for only 1 epoch, while all encoder-decoder models are trained for 5 epochs. This introduces a confound: the conclusion in Section 4 that 'larger models yield superior performance' is based on comparing models trained with different training effort. The comparison between FT LLaMA and FT T5/BART, and the comparison among LLaMA variants, would only be meaningful if training budgets are matched or if the effect of epoch count is analyzed.","section":"Section 3 (Evaluation)"},{"comment":"No error bars, variance estimates, or significance tests are reported for any of the metrics. Since runs appear to be reported for a single seed or configuration, the differences between FT T5 and FT LLaMA v2 (e.g., METEOR 0.2723 vs. 0.2759) and between FT T5 and FT LLaMA v3 (BERTScore 0.5202 vs. 0.5652) may not be statistically reliable. The claim that LLaMA v3 'achieved the highest overall scores' should be qualified or supported with significance testing or repeated runs.","section":"Table 1 and Section 4"}],"minor_comments":[{"comment":"The phrase 'we utilize fine-tuning of LLMs' is slightly awkward; 'we fine-tune LLMs' is clearer. Additionally, Figure 1 is referenced but not described in enough detail in the text to understand the input-output structure without looking at the figure.","section":"Section 2 (Approach)"},{"comment":"The caption states 'Training time is reported for one epoch,' but the text in Section 3 says all encoder-decoder models were trained for 5 epochs. Clarify whether the reported training times are per epoch or total, and whether the LLaMA training times are also per epoch.","section":"Table 1 caption"},{"comment":"The dataset construction footnote appears after 'randomly assigning groups' but the link is split across lines in the provided text; ensure the repository URL is presented cleanly and that the dataset versioning is described.","section":"Section 3 (Evaluation)"},{"comment":"Several references omit URLs or DOIs (e.g., [2], [9], [14] are given with arXiv identifiers but some lack the full link). For a camera-ready version, all references should be complete and consistent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short conference paper (3 pages) with a single clear empirical contribution. The main risk is the oversold 'accurate and plausible' claim relative to the automatic metrics. If the authors cannot add human evaluation, they should rephrase the abstract and conclusions to say 'closer to the reference section headings according to overlap metrics.' The unequal-epoch confound should also be addressed, at least by an ablation or a clear justification in a limitations paragraph. These are fixable within the paper's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable small empirical study, but the abstract oversells what the metrics actually show. The real contribution is a clean comparison table showing fine-tuned BART/T5 and QLoRA-tuned LLaMA models beat trained-from-scratch transformer baselines on METEOR/ROUGE/BERTScore for Wikipedia heading generation. That result is plausible and probably reproducible. But the phrase \"more accurate and plausible\" in the abstract is not supported: those metrics measure lexical and embedding overlap with section headings, not plausibility as judged by a searcher. No human evaluation appears anywhere. If the authors softened the claim to \"closer to the reference distribution\" or added a small user study, the paper would be honest.\n\nWhat's new: prior work used modified Transformer architectures trained from scratch; this paper shows pretrained LLMs fine-tuned (full or QLoRA) do much better on this task. That's worth knowing for anyone working on explainable IR. The natural-language input formatting and instruction tuning are minor variations, but the empirical result is new.\n\nSoft spots: (1) The baseline set omits the actual prior task-specific architectures from [9, 14] - they compare against generic Transformer, Bert2Bert, Bert2Gpt trained from scratch, not the query-attention models those papers proposed. So the claim of beating \"a range of baseline models\" sidesteps the strongest prior systems. (2) LLaMA models got 1 epoch versus 5 for everyone else, so the ranking among fine-tuned models is not apples-to-apples; the broad conclusion likely survives, but the 70B-vs-T5 margin is soft. (3) No significance tests or error bars; the numbers in Table 1 are point estimates. (4) The dataset uses Wikipedia section headings as explanations, which is a convenient proxy but far from what a user would find plausible in a real search UI.\n\nNone of these is fatal. The central empirical claim - that fine-tuning LLMs beats training from scratch on this proxy task - is probably true. The fix is mostly reframing: either add human evaluation or scale back the abstract.\n\nWho this is for: IR researchers working on explanation generation or snippet quality; it's a short workshop-style paper, not a definitive study. It deserves a serious referee, but the referee should push for the missing baseline and the human eval.","headline":"A plausible small win for fine-tuned LLMs on explanation generation, with an abstract that claims more than the metrics support.","tokens_in":5192,"tokens_out":1854,"would_cite":false,"duration_ms":17778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs generate more accurate and plausible search-result explanations than baseline models.","keywords":["Aspect-oriented explanations","Explainable information retrieval","Large language models","Fine-tuning","QLoRA","Search result explanations","METEOR","BERTScore"],"falsifier":"Have human annotators rate the generated explanations for a held-out set of underspecified queries against the actual Wikipedia section headings for accuracy and plausibility, then check whether the fine-tuned LLaMA v3 70B explanations receive higher ratings than the from-scratch transformer baselines. The paper's central claim fails if the higher METEOR, ROUGE-1, and BERTScore received by the fine-tuned models do not correspond to higher human ratings.","tokens_in":4238,"feed_emoji":"🔍","tokens_out":8502,"duration_ms":74792,"temperature":0.7,"pith_summary":"This paper asks whether large language models, fine-tuned on a suitable corpus, can produce the short aspect-oriented explanations that search engines display alongside results to tell a user why a document matches an underspecified query. The authors build a dataset from Wikipedia in which article titles serve as queries and section headings serve as reference explanations, then fine-tune both encoder-decoder models (BART, T5) and decoder-only models (LLaMA v2 13B, LLaMA v3 70B) and compare them with from-scratch transformer baselines. They report that all fine-tuned models beat the baselines on METEOR, ROUGE-1, and BERTScore, with fine-tuned LLaMA v3 70B attaining the highest scores. A sympathetic reader would care because better search explanations could let users judge result relevance at a glance, and the finding suggests that fine-tuning existing pretrained models is a simpler path to this capability than training specialized architectures from scratch.","feed_headline":"Fine-tuned LLaMA 70B beats baselines on search explanations","feed_subtitle":"Fine-tuned models beat baselines on all search-explanation metrics; small T5 and BART stay close.","key_machinery":"The load-bearing mechanism is the fine-tuning recipe rather than a new architecture. Smaller encoder-decoder models (BART, T5) receive full fine-tuning with a natural-language input format, while the large decoder-only LLaMA models are adapted with QLoRA, which combines 4-bit quantization with low-rank adaptation to fit on one GPU. The evaluation rests on a Wikipedia-derived dataset in which article titles become queries and section headings become reference aspect explanations, with the sections limited to 128–512 tokens, and on the three automatic metrics METEOR, ROUGE-1, and BERTScore that quantify lexical and semantic overlap with those references.","core_discovery":"The paper's central claim is that fine-tuning pretrained large language models for aspect-oriented explanation generation yields explanations that are consistently more accurate and plausible than those from baseline systems. In the reported experiments, every fine-tuned model — BART, T5, LLaMA v2 13B, and LLaMA v3 70B — outperforms the from-scratch Transformer, Bert2Bert, Bert2GPT, and zero-shot LLaMA baselines across all three automatic metrics. The strongest result is the fine-tuned LLaMA v3 70B model, which reaches METEOR 0.3222, ROUGE-1 0.4993, and BERTScore 0.5652, while the much smaller fine-tuned T5 remains close on all metrics at a fraction of the training and inference cost. The paper also finds that zero-shot prompting of large decoder-only models alone is not sufficient, so supervised fine-tuning is necessary for the task.","pith_inferences":["A natural extension beyond the paper would be to test the same fine-tuning recipe on other explanation sources, such as FAQs, user reviews, or entity descriptions, to see whether the gains hold outside encyclopedic content.","Because the paper evaluates with automatic metrics only, a human-judgment study would clarify whether the higher scores actually translate into explanations users find more plausible and easier to scan.","The low cost of the fine-tuned T5 model suggests a deployable recipe: generate explanations offline for frequent queries with a mid-size model, reserving larger models for hard or rare query aspects."],"forward_implications":["Fine-tuned LLMs can be used directly to generate aspect-oriented explanations for search results, giving a cheaper path than training specialized transformer models from scratch.","Scaling to larger models improves explanation quality on automatic metrics, so organisations with enough compute can push quality higher by fine-tuning 70B-scale models.","Mid-size encoder-decoder models such as T5 and BART offer most of the quality gain at far lower training and inference cost, making them the practical choice when resources are limited.","Zero-shot prompting alone is insufficient for this task, so supervised fine-tuning is required even for very large decoder-only LLMs."],"supporting_citations":[{"why":"Defines the aspect-based explanation task and contributes the dataset construction method (Wikipedia titles as queries, section headings as references) that this work adopts.","marker":"[9]"},{"why":"Extends explanation generation to a listwise setting and supplies the same data-construction approach used to build the evaluation set.","marker":"[14]"},{"why":"Introduces QLoRA, the parameter-efficient 4-bit quantization and low-rank adaptation method used to fine-tune the LLaMA models.","marker":"[2]"},{"why":"METEOR is one of the three automatic metrics used to score explanation accuracy and plausibility.","marker":"[1]"},{"why":"ROUGE-1 is one of the three automatic metrics used to score explanation accuracy and plausibility.","marker":"[5]"},{"why":"Supplies the standard Transformer architecture that serves as the from-scratch encoder-decoder baseline.","marker":"[13]"},{"why":"Provides the LLaMA family of decoder-only models used for zero-shot baselines and for QLoRA fine-tuning at 13B and 70B scale.","marker":"[12]"},{"why":"BERT serves as the encoder in the Bert2Bert and Bert2GPT from-scratch baselines.","marker":"[3]"},{"why":"GPT-2 serves as the decoder in the Bert2GPT from-scratch baseline.","marker":"[8]"}],"fun_headline_variants":["Fine-tuned LLaMA 70B outshines baselines on search explanations","Small fine-tuned T5 stays close to LLaMA 70B on search snippets","Fine-tuning, not zero-shot, yields better LLM search explanations","LLM fine-tuning improves search explanation accuracy and plausibility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on treating Wikipedia section headings as the correct aspects and the three automatic metrics as trustworthy measures of explanation accuracy and plausibility.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLaMA 70B outshines baselines on search explanations","Small fine-tuned T5 stays close to LLaMA 70B on search snippets","Fine-tuning, not zero-shot, yields better LLM search explanations","LLM fine-tuning improves search explanation accuracy and plausibility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1669,"prompt_tokens":817,"completion_tokens":852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":433,"tokens_out":852,"duration_ms":7864,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:03:27.790290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators rate the generated explanations for a held-out set of underspecified queries against the actual Wikipedia section headings for accuracy and plausibility, then check whether the fine-tuned LLaMA v3 70B explanations receive higher ratings than the from-scratch transformer baselines. The paper's central claim fails if the higher METEOR, ROUGE-1, and BERTScore received by the fine-tuned models do not correspond to higher human ratings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-2 serves as the decoder in the Bert2GPT from-scratch baseline."},{"cited_title":"Explaining Documents' Relevance to Search Queries","cited_arxiv_id":"2111.01314","evidence_quote":"Defines the aspect-based explanation task and contributes the dataset construction method (Wikipedia titles as queries, section headings as references) that this work adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces QLoRA, the parameter-efficient 4-bit quantization and low-rank adaptation method used to fine-tune the LLaMA models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"METEOR is one of the three automatic metrics used to score explanation accuracy and plausibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ROUGE-1 is one of the three automatic metrics used to score explanation accuracy and plausibility."},{"cited_title":"Gomez, Lukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Supplies the standard Transformer architecture that serves as the from-scratch encoder-decoder baseline."}],"review_version":1}