Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ExPerT evaluates personalized long-form text by decomposing it into atomic aspects, matching those aspects against a user-written reference, and scoring content and style alignment per aspect, reporting the highest human-agreement of the…

desk verdict ExPerT is a real and explainable evaluation framework, but the headline 7.2% gain is a post-hoc selection on 100 filtered examples and is not yet established. read the letter →

arxiv 2501.14956 v2 pith:VQU6E7FK submitted 2025-01-24 cs.CL cs.AIcs.IR

Alireza Salemi , Julian Killingback , Hamed Zamani This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords personalizedtextgenerationlong-formLLMevaluationexplainableaspectextractionreference-basedmetricsF-measureprompt-injectionrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that personalized text generation can be evaluated automatically and transparently by an LLM that works at the level of atomic aspects rather than whole documents. Because only the person who wrote the prompt can truly judge a personalized output, ExPerT uses that person's own reference text and asks whether the generated text covers the same aspects and whether the evidence for each matched aspect agrees in content and writing style. The reported result is that this aspect-level score agrees with human judges 74% of the time, beating GEMBA and G-Eval by 7.2% relative, while also producing human-readable rationales rated 4.7 out of 5. If correct, this gives the field an evaluation signal that is more reliable than holistic LLM scoring, harder to cheat with a simple confidence phrase, and explainable enough for a user to see why one output was preferred.

What carries the argument

The central object is the atomic aspect: a single claim or topic unit in a text, together with its evidence, the sentences or phrases that support it. ExPerT uses an off-the-shelf instruction-tuned LLM to extract these aspect-evidence pairs from the reference and generated texts, then runs a one-pass matching step in which each aspect is paired with at most one aspect from the other text or with 'none'. For each matched pair, the LLM makes binary content-alignment and style-alignment decisions on the evidence and writes a rationale; the two decisions are averaged, the default CONTENT/STYLE AVERAGE aggregation, to score evidence similarity. Recall averages the best matched evidence score over reference aspects, precision over generated aspects, and the final score is their harmonic mean, the F-measure. The matching and binary-decision structure is what converts an LLM judgment from an opaque holistic score into a transparent, decomposable calculation.

What would settle it

Design a human-annotated test set in which one reference aspect is deliberately mapped to two distinct generated aspects, run both ExPerT's single-best-match prompt and a many-to-many prompt on it, and check whether the many-to-many variant recovers the human-aligned matches and raises agreement; if it does not, the one-to-one matching assumption is not the limiting factor.

Watch

Extended reading notes

Core claim

The paper's central claim is that personalized long-form generation is best evaluated not by comparing whole texts, but by comparing the atomic aspects each text contains and the evidence attached to those aspects. On the LongLaMP benchmark, ExPerT agrees with human pairwise judgments at 0.74 alignment, a 7.2% relative improvement over the previous best whole-text LLM evaluators GEMBA and G-Eval, both at 0.69. The paper also claims that this decomposition makes the evaluation explainable: human annotators rated its per-step rationales 4.7 out of 5 and could infer which output ExPerT preferred in 94% of cases from the explanations alone. The same experiments show ExPerT resists a simple prompt-injection phrase that inflates GEMBA scores by up to 24.3% relative, and its scores decrease linearly as user profiles are randomly replaced, evidence that it tracks personalization rather than surface text. In short, the paper is trying to establish that a reference-based, aspect-level, content-and-style F-measure is a more effective, more robust, and more transparent way to evaluate personalized text than holistic LLM judging.

Load-bearing premise

The load-bearing premise is that each aspect in one text matches at most one aspect in the other, so the LLM is instructed to pick a single best match or 'none'; if real aspects connect many-to-many, the recall and precision scores undercount aligned content and the reported F-measure no longer measures the alignment the equations describe.

Editorial extensions

If this is right

  • Personalized generation systems can be ranked and iterated on without re-recruiting the original prompt author each time, since the user's reference text supplies the personal ground truth.
  • Evaluation reports become inspectable: recall and precision identify which aspects of the reference were missed, which extra aspects were invented, and where content or style diverged, not just a single score.
  • The prompt-injection and position-bias results suggest that LLM-based evaluation of personalized text need not be abandoned for lack of robustness, provided matching is structured rather than holistic.
  • Model-size experiments indicate that a large open model reaches the same human-agreement level as GPT-4o under ExPerT, so reproducible evaluation does not require a proprietary API.
  • Because the same aspect, content, and style machinery applies to any reference-based long-form generation, the metric should transfer to summarization or translation evaluation tasks, though the paper does not test those settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A many-to-many aspect matching variant, allowing one reference aspect to split across several generated aspects, would likely raise recall and precision on texts where a single idea branches into multiple sub-points; the paper's one-to-one matching may undercount those alignments.
  • If a reference-free model can synthesize a plausible user reference, ExPerT could be applied where no user-written reference exists, turning the metric into a hybrid reference-based and reference-free evaluator.
  • The per-aspect rationales ExPerT produces could be mined as training feedback for personalized generators, giving models a finer-grained error signal than a scalar score.
  • The linear sensitivity to profile substitution suggests ExPerT could serve as a diagnostic for how much of a system's output actually depends on the user profile, not just as a final quality score.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ExPerT proposes an explainable, reference-based evaluation metric for personalized long-form text generation. It decomposes both the reference (expected) and generated outputs into atomic aspects with supporting evidence, matches aspects via an LLM in a recall- and precision-style procedure, and scores matched evidence on content and writing-style alignment using one of five aggregation variants. The final score is the F-measure of the per-aspect precision and recall. Experiments on the LongLaMP benchmark report the highest alignment with human judgments among compared metrics (0.74 versus 0.69 for GEMBA and G-Eval), a 4.7/5 human rating for explanation usefulness, robustness to a simple prompt-injection trick, and sensitivity to profile replacement. The authors release code for the metric.

Significance. If the empirical claims hold, ExPerT is a meaningful contribution to the underexplored problem of evaluating personalized long-form text generation. Its design has several strengths: it is reference-based and thus grounded in the user's own expected output; it provides fine-grained, per-step rationales; it explicitly separates content and style dimensions; and the authors publicly release the code. The robustness analysis against prompt-injection and the linear sensitivity to profile corruption are useful, falsifiable checks. However, the central quantitative claim—that ExPerT outperforms GEMBA and G-Eval by 7.2% relative alignment—is currently under-supported because the winning aggregation variant was selected on the same 100 human-annotated examples used for the final comparison, and because the reported point estimates are not accompanied by confidence intervals or significance tests.

major comments (4)
  1. [Section 3.1 and Figure 3 / Table 1] The headline result of Table 1 (0.74 vs. 0.69) is obtained after selecting the CONTENT/STYLE AVERAGE aggregation from the five variants presented in Figure 3, all evaluated on the same 100 human-annotated examples. Selecting the variant with the maximum observed alignment on the test set inflates the expected alignment relative to a fixed, pre-specified rule, and with n=100 the 0.05 gap over GEMBA/G-Eval is within a plausible sampling error (a 95% CI for 0.74 spans roughly 0.65–0.82). The paper reports no confidence intervals, significance tests, multiple-comparison correction, or a held-out variant selection. To support the claimed superiority, the authors should either pre-register the aggregation rule, use a separate validation set for variant selection, or report corrected (e.g., Bonferroni or bootstrap) intervals and pairwise tests.
  2. [Section 2.2 and Appendix D (Figure 9)] The matching step assumes that each aspect from either text can be matched to at most one aspect from the other set, and the LLM prompt implements this by selecting a single best match or 'none'. However, Appendix D and Figure 9 explicitly state that 'an aspect from the generated output or the expected output can be matched to multiple aspects from the other set' and illustrate this behavior. This is an internal contradiction: if the actual extraction and matching allow many-to-many relations, the linear-time assumption and the single-inference prompt are not consistent with the claims, and the recall/precision scores in Equations (1)–(3) undercount aligned aspects. The authors must either restrict the method (and Figure 9 caption) to one-to-one matching or revise the matching procedure and the definitions of R, P, and F to account for many-to-many alignment.
  3. [Section 3.1, Human Annotation] The 100 samples are not a random sample from the LongLaMP test sets; they are deliberately filtered so that at least one metric selects a different response as better than the others. This means the reported alignment values are conditional estimates on a disagreement-filtered subset, not estimates of alignment on the original test distribution. Consequently, the 0.74 figure cannot be directly interpreted as 'alignment with human judgment on the benchmark' as implied in Table 1 and the abstract. The authors should report the alignment on an unfiltered random sample as well, or clearly frame the numbers as hard-subset discriminative power.
  4. [Section 3.2, first finding] The claim that ExPerT 'achieves the highest alignment with human judgments' is based solely on point estimates with no statistical testing. Given the small sample (100 examples, 3 annotators each, majority vote) and the post-hoc variant selection, the 0.74 vs. 0.69 difference could well be noise. A bootstrap or permutation test over examples, or over annotators, is needed to establish that the difference is not attributable to sampling variation.
minor comments (5)
  1. [Section 2.2] The terminology 'recall- and precision-based manner' is used before the formal definitions; for clarity, the formal definitions of R and P should be moved closer to the matching description or introduced earlier.
  2. [Section 3.2, 'How efficient is ExPerT...'] The cost comparison uses 18.6 LLM calls per instance for ExPerT as measured on the 100 filtered human-annotated examples. Since the number of aspects may differ on a random sample, reporting the range or a formula for the call count in terms of the number of extracted aspects would be more informative.
  3. [Section 3.2, 'How explainable is ExPerT...'] The human explanation study reports that annotators 'correctly identified the output with the higher ExPerT score in 94% of cases.' Because the annotators see the aspects, matching, and rationales but not the final score, this is a reasonable protocol; however, the paper does not report inter-annotator agreement for this task, which would be useful given the 4.7/5 rating.
  4. [Section 3.2, 'How sensitive is ExPerT...'] The profile-replacement curve (Figure 5) would benefit from error bars or a repeat with multiple random profile permutations; the current plot cannot indicate whether the downward trend is statistically reliable.
  5. [Throughout] There are several typographical issues, e.g., 'G-E VAL' in the introduction (should be 'G-Eval'), 'A VERAGE' in Section 2.2, and 'Strick' for 'Stricked' in Figure 6; these should be corrected in a final pass.

Circularity Check

1 steps flagged · score 5.0 of 10

The reported 7.2% improvement is partly a test-set selection artifact: the CONTENT/STYLE AVERAGE variant was chosen as the best of five aggregation rules on the same 100 human-annotated examples used to compute the headline alignment, so the headline number is a selected maximum rather than an independent prediction.

  1. fitted input called prediction [Section 3.2, Figure 3, and Table 1]
    "Among the methods that incorporate both style and content, the CONTENT/STYLE A VERAGE achieves the highest alignment (0.74), followed by CONTENT OR STYLE (0.73). ... Table 1: ExPerT (Content/Style Average) 0.74"

    The paper evaluates five aggregation variants on the same 100 human-annotated examples (Figure 3) and then reports the best-performing variant, CONTENT/STYLE AVERAGE, as ExPerT's headline alignment in Table 1. This is equivalent to selecting a discrete design choice using the target labels and then presenting the selected value as the method's expected alignment. Because no held-out variant selection, confidence interval, or multiple-comparison correction is provided, the 0.74 vs 0.69 gap over GEMBA and G-Eval is at least in part a test-set maximum rather than an out-of-sample prediction. The selection makes the reported improvement statistically forced by construction, although the underlying aspect-matching pipeline is not itself fit to the human labels.

full rationale

The metric's derivation is not circular in the strict equation sense: ExPerT's aspect extraction, matching, and content/style scoring are performed by an off-the-shelf Gemma 2 model, and the F-measure equations in Section 2 do not contain any term estimated from the 100 human-annotated examples. The self-citations to LaMP, LongLaMP, and Personalized RAG are to public, externally available resources and are not used as proof of the metric's validity; they are benchmarks and generation tooling, so they do not constitute load-bearing self-citation. The main circularity-adjacent problem is the aggregation-variant selection: Figure 3 ranks five variants on the exact same 100 examples, and Table 1 then reports the best variant's alignment as the headline result. That is a form of test-set fitting, making the 7.2% relative improvement partly a selected maximum. The conclusion is not entirely forced, since several variants (CONTENT at 0.71, OR at 0.73, AVERAGE at 0.74) beat GEMBA/G-Eval at 0.69, but the precise reported gain is not an independent estimate. Other concerns, such as the at-most-one matching assumption conflicting with the many-to-many example in Appendix D/Figure 9, are internal consistency and validity issues rather than circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The claim relies on the LLM's extraction, matching, and content/style judgments being trustworthy, and on the reference-based framing and content/style dichotomy being the right ones. The only quantity selected from the test data is the aggregation method, listed as a free parameter. No new physical or external entities are introduced; 'aspect' and 'evidence' are internal text decompositions, not independently falsifiable constructs.

free parameters (1)
  • Score aggregation method = CONTENT/STYLE AVERAGE
    Chosen post hoc as the best of five aggregation variants in Figure 3; the reported 0.74 alignment in Table 1 uses this variant, so the headline improvement is partly a test-set selection.
assumptions (5)
  • domain assumption Only the prompt author can reliably assess personalized output; reference-based evaluation is more reliable than reference-free inference of user preferences.
    Section 1 cites Dong et al. (2024) to justify focusing on reference-based evaluation; the entire design depends on this premise.
  • domain assumption Content alignment and writing style alignment are the two key attributes of personalized text generation.
    Section 2 says the metric focuses on these two dimensions and ignores other potential axes such as tone, factual precision, or register.
  • domain assumption The LLM's aspect and evidence extraction is faithful and complete for both the reference and generated texts.
    Section 2.1 relies entirely on the off-the-shelf LLM to emit the JSON aspect lists; no gold-standard extraction quality is reported, so errors propagate into matching and scoring.
  • ad hoc to paper Each aspect can be matched to at most one aspect in the other set, enabling a single-inference best-match prompt.
    Section 2.2 introduces the assumption to reduce matching cost to O(|Ay|+|A_ybar|); Appendix D contradicts it by describing and illustrating many-to-many matches, leaving the operative semantics unresolved.
  • domain assumption The LLM's binary content and style judgments are accurate and largely unbiased.
    The content and style matching prompts in Section 2.2 return binary decisions; the paper demonstrates robustness to one appended phrase and one position-swap test, but not to the general space of evaluator biases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation." pith.science (2026). https://pith.science/paper/VQU6E7FK

@misc{pith2026250114956,
  author       = {Pith},
  title        = {Pith review of: ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQU6E7FK}},
  note         = {Machine review of arXiv:2501.14956}
}
read the original abstract

Evaluating personalized text generated by large language models (LLMs) is challenging, as only the LLM user, i.e., prompt author, can reliably assess the output, but re-engaging the same individuals across studies is infeasible. This paper addresses the challenge of evaluating personalized text generation by introducing ExPerT, an explainable reference-based evaluation framework. ExPerT leverages an LLM to extract atomic aspects and their evidence from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style -- two key attributes in personalized text generation. Additionally, ExPerT generates detailed, fine-grained explanations for every step of the evaluation process, enhancing transparency and interpretability. Our experiments demonstrate that ExPerT achieves a 7.2% relative improvement in alignment with human judgments compared to the state-of-the-art text generation evaluation methods. Furthermore, human evaluators rated the usability of ExPerT's explanations at 4.7 out of 5, highlighting its effectiveness in making evaluation decisions more interpretable.

Figures

Figures reproduced from arXiv: 2501.14956 by the authors.

Figure 1
Figure 1. ExPerT pipeline: Generated and reference outputs are first decomposed into atomic aspects along with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The prompts used for aspect extraction, aspect, content, and writing style matching in ExPerT. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The alignment between ExPerT different methods for content and style score aggregation with human judgment in evaluation. sizes.5 The models are used in ExPerT to score out￾puts, and the alignment of these scores with human judgments is computed. The results are presented in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: The average ExPerT score across varying [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The sorted difference between assigned score by the evaluators to the generated output with trick and the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The evaluation prompts used for LLM-based baselines. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The prompt used for personalizing LLMs by [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: A case study of aspect and evidence extraction, as well as aspect, content, and writing style matching in [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.

Reference graph

Works this paper leans on

61 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

    2005

  2. [2]

    Anja Belz and Ehud Reiter. 2006. https://aclanthology.org/E06-1040 Comparing automatic and human evaluation of NLG systems . In 11th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages 313--320, Trento, Italy. Association for Computational Linguistics

    work page 2006

  3. [3]

    Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2021. https://arxiv.org/abs/2006.14799 Evaluation of text generation: A survey . Preprint, arXiv:2006.14799

    arXiv 2021

  4. [4]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.474 Humans or LLM s as the judge? a study on judgement bias . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301--8327, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.474 2024

  5. [5]

    Hand, and Nishadi Kirielle

    Peter Christen, David J. Hand, and Nishadi Kirielle. 2023. https://doi.org/10.1145/3606367 A review of the f-measure: Its history, properties, criticism, and alternatives . ACM Comput. Surv., 56(3)

    doi:10.1145/3606367 2023

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. https://aclanthology.org/2024.findings-emnlp.592 Can LLM be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126--10141, Miami, Florida, USA. Association for Computational Linguistics

    work page 2024

  8. [8]

    Andrew Fowler, Kurt Partridge, Ciprian Chelba, Xiaojun Bi, Tom Ouyang, and Shumin Zhai. 2015. https://doi.org/10.1145/2702123.2702503 Effects of language modeling and its personalization on touchscreen typing performance . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, page 649–658, New York, NY, USA. Asso...

    arXiv 2015

Show all 61 references
  1. [9]

    Gemini-Team. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 arXiv

  2. [10]

    Gemma-Team. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  3. [11]

    Michael Hanna and Ond r ej Bojar. 2021. https://aclanthology.org/2021.wmt-1.59 A fine-grained analysis of BERTS core . In Proceedings of the Sixth Conference on Machine Translation, pages 507--517, Online. Association for Computational Linguistics

    2021

  4. [12]

    Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. https://doi.org/10.18653/v1/2024.acl-long.745 LLM -rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts . In Proceedings of the 62nd Annual Meeti...

    2024 doi

  5. [13]

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. https://arxiv.org/abs/2310.11564 Personalized soups: Personalized large language model alignment via post-hoc parameter mer...

    2023 arXiv

  6. [14]

    Marvin Kaster, Wei Zhao, and Steffen Eger. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.701 Global explainability of BERT -based evaluation metrics by disentangling along linguistic factors . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P...

    2021 doi

  7. [15]

    Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA

    2015

  8. [16]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Tam...

    2023

  9. [17]

    Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu, and Shirui Pan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.172 How far are we from robust long abstractive summarization? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2682--...

    2022 doi

  10. [18]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.29 Benchmarking cognitive biases in large language models as evaluators . In Findings of the Association for Computational Linguistics: ACL ...

    2024 doi

  11. [19]

    Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. https://doi.org/10.18653/v1/2021.naacl-main.393 Hurdles to progress in long-form question answering . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2021 doi

  12. [20]

    Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, Chien Van Nguyen, Thien Huu Nguyen, and Hamed Zamani

    Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, Chien Van Nguyen, Thien Huu Nguyen, and Hamed Zamani. 2024. https://arxiv.org/abs/2407.11016 Longla...

    2024 arXiv

  13. [21]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  14. [22]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  15. [23]

    Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva, Wei Zhao, Yang Gao, and Steffen Eger. 2022. https://arxiv.org/abs/2203.11131 Towards explainable evaluation metrics for natural language generation . Preprint, arXiv:2203.11131

    2022 arXiv

  16. [24]

    Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. 2024. https://doi.org/10.1145/3589334.3645408 Learning to rewrite prompts for personalized text generation . In Proceedings of the ACM on Web Conference 2024, WWW ’24. ACM

    2024

  17. [25]

    Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. 2023 a . https://arxiv.org/abs/2308.07968 Teach llms to personalize -- an approach inspired by writing education . Preprint, arXiv:2308.07968

    2023 arXiv

  18. [26]

    Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023 b . https://arxiv.org/abs/2310.19740 Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation . Preprint, arXiv:2310.19740

    2023 arXiv

  19. [27]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  20. [28]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  21. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  22. [30]

    Zhuoran Lu, Sheshera Mysore, Tara Safavi, Jennifer Neville, Longqi Yang, and Mengting Wan. 2024. https://arxiv.org/abs/2405.04656 Corporate communication companion (ccc): An llm-empowered writing assistant for workplace social media . Preprint, arXiv:2405.04656

    2024 arXiv

  23. [31]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  24. [32]

    Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. 2023. https://arxiv.org/abs/2311.09180 Pearl: Personalizing large language model writing assistants with generation-calibrated ret...

    2023 arXiv

  25. [33]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Vol...

    2019 arXiv

  26. [34]

    Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. https://arxiv.org/abs/2402.15987 Likelihood-based mitigation of evaluation bias in large language models . Preprint, arXiv:2402.15987

    2024

  27. [35]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  28. [36]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

    2002

  29. [37]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt \"a schel, and Sebastian Riedel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.200 KIL...

    2021 doi

  30. [38]

    Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. 2024. https://arxiv.org/abs/2409.16191 Hellobench: Evaluating long text generation capabil...

    2024 arXiv

  31. [39]

    Alireza Salemi, Juan Altmayer Pizzorno, and Hamed Zamani. 2023 a . https://doi.org/10.1145/3539618.3591629 A symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering . In Proceedings of the 46th International ACM SIGIR Conference on R...

    2023

  32. [41]

    Alireza Salemi, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, Tao Chen, Zhuowan Li, Michael Bendersky, and Hamed Zamani. 2025. https://arxiv.org/abs/2501.04167 Reasoning-enhanced self-training for long-form personalized text generation . Preprint, arXiv:2501.04167

    2025 arXiv

  33. [42]

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.399 L a MP : When large language models meet personalization . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024 doi

  34. [43]

    Alireza Salemi, Mahta Rafiee, and Hamed Zamani. 2023 b . https://doi.org/10.1145/3578337.3605137 Pre-training multi-modal dense retrievers for outside-knowledge visual question answering . In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information R...

    2023

  35. [44]

    Alireza Salemi and Hamed Zamani. 2024 a . https://arxiv.org/abs/2409.09510 Comparing retrieval-augmentation and parameter-efficient fine-tuning for privacy-preserving personalization of large language models . Preprint, arXiv:2409.09510

    2024 arXiv

  36. [45]

    Alireza Salemi and Hamed Zamani. 2024 b . https://doi.org/10.1145/3626772.3657957 Evaluating retrieval quality in retrieval-augmented generation . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pag...

    2024

  37. [46]

    Alireza Salemi and Hamed Zamani. 2024 c . https://arxiv.org/abs/2410.09942 Learning to rank for multiple retrieval-augmented models through iterative utility maximization . Preprint, arXiv:2410.09942

    2024 arXiv

  38. [47]

    Alireza Salemi and Hamed Zamani. 2024 d . https://doi.org/10.1145/3626772.3657733 Towards a search engine for machines: Unified ranking for multiple retrieval-augmented large language models . In Proceedings of the 47th International ACM SIGIR Conference on Research and Develo...

    2024

  39. [48]

    Chris Samarinas, Alexander Krubner, Alireza Salemi, Youngwoo Kim, and Hamed Zamani. 2025. https://arxiv.org/abs/2501.03545 Beyond factual accuracy: Evaluating coverage of diverse factual information in long-form text generation . Preprint, arXiv:2501.03545

    2025 arXiv

  40. [49]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  41. [50]

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. https://arxiv.org/abs/2405.01724 Large language models are inconsistent and biased evaluators . Preprint, arXiv:2405.01724

    2024 arXiv

  42. [51]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, page 3104–3112, Cambridge, MA, USA. MIT Press

    2014

  43. [52]

    Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. 2024. https://arxiv.org/abs/2406.10471 Personalized pieces: Efficient personalized large language models through collaborative efforts . Preprint, arXiv:2406.10471

    2024 arXiv

  44. [53]

    Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024. https://aclanthology.org/2024.emnlp-main.737 Learning personalized alignment for evaluating open-ended text generation . In Proceedings of the 2024 Conference on Empirical Metho...

    2024

  45. [54]

    Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. 2023. https://arxiv.org/abs/2310.11593 Automated evaluation of personalized text generation using large language models . Preprint, arXiv:2310.11593

    2023 arXiv

  46. [55]

    Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.365 INSTRUCTSCORE : Towards explainable text generation evaluation with automatic feedback . In Proceedings of the 2023 Conference ...

    2023 doi

  47. [56]

    Gui-Rong Xue, Jie Han, Yong Yu, and Qiang Yang. 2009. https://doi.org/10.1145/1462198.1462203 User language model for collaborative personalized search . ACM Trans. Inf. Syst., 27(2)

    2009

  48. [57]

    Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. 2024. https://aclanthology.org/2024.naacl-long.132 LLM -based medical assistant personalization with short- and long-term memory coordination . In Proceedings of the 2024 Conference of the North American Chapter of the ...

    2024

  49. [58]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  50. [59]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  51. [60]

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.131 Towards a unified multi-dimensional evaluator for text generation . In Proceedings of the 2022 Conference on Empiric...

    2022 doi

  52. [61]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  53. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.