Improving Fairness of Large Language Models in Multi-document Summarization

Haoyuan Li; Rui Zhang; Snigdha Chaturvedi

REVIEW 4 major objections 4 minor 26 references

Improving Fairness of Large Language Models in Multi-document Summarization

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FairPO, a preference-tuning method built on DPO, improves both summary-level and corpus-level fairness of LLM multi-document summaries across three models and three datasets while preserving relevance and factuality.

desk verdict FairPO is a clearly written, novel application of DPO to summarization fairness, but its perturbation-based pair generation can game the Equal Coverage metric by dropping documents of the overrepresented attribute, so the reported gains may not reflect genuine balanced representation. read the letter →

arxiv 2506.07479 v2 pith:M427FVJB submitted 2025-06-09 cs.CL

Haoyuan Li , Rui Zhang , Snigdha Chaturvedi This is my paper

classification cs.CL

keywords multi-documentsummarizationfairnesspreferencetuningdirectoptimizationequalcoverageparitylargelanguagemodelssocialattributes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-document summarizers that overrepresent one side of a controversy can mislead readers—neglecting negative reviews, for instance, can push customers toward bad products. The paper argues that fairness must be enforced at two scales at once: within each summary (equal coverage of social attribute values) and across a corpus of summaries (no systematic over- or underrepresentation). To achieve this it proposes FairPO, a preference-tuning method that generates training pairs by removing a small slice of documents with a given attribute value, then re-weights the chosen and rejected summaries according to their effect on corpus-level coverage parity. Experiments across Llama3.1, Mistral, and Gemma2 on Amazon, MITweet, and SemEval report that FairPO beats DPO, OPTune, prompting, and policy-gradient baselines on both fairness metrics while preserving summary quality. The central evidence is that a preference signal derived from coverage probabilities can steer an LLM toward fairer summaries without an explicit fairness reward.

What carries the argument

The load-bearing object is the coverage probability $p(d_i, s_j)$, defined as the maximum entailment probability between any chunk $d_{i,l}$ of document $d_i$ and summary sentence $s_j$, averaged over sentences to give $p(d_i, s)$. Averaging these over documents yields the Equal Coverage (EC) and Coverage Parity (CP) scores that FairPO optimizes. With these scores in hand, FairPO generates preference pairs by perturbing the document set: it removes $\alpha\%$ of documents having the most over- or under-represented attribute value, regenerates summaries, and keeps the lowest-EC summary as chosen and the highest-EC as rejected. It then modifies the DPO objective into $\sigma(-m)\beta\big(w_r \log \frac{\pi_\theta(S_r|D)}{\pi_{\mathrm{ref}}(S_r|D)} - w_c \log \frac{\pi_\theta(S_c|D)}{\pi_{\mathrm{ref}}(S_c|D)}\big)$, where the separate weights $w_c, w_r$ are computed from running estimates $O(k), U(k)$ of how often attribute $k$ is over- or under-represented in recent training batches. The weights make chosen summaries that counteract the current imbalance get a larger positive signal, and rejected summaries that worsen it get a larger negative signal.

What would settle it

Swap the entailment model for a different one, such as a larger NLI model or human coverage ratings, and rerun FairPO with everything else fixed; if the Equal Coverage and Coverage Parity gains shrink or disappear, the reported fairness improvement is an artifact of the particular coverage estimator rather than a property of the tuned summaries.

Watch

Extended reading notes

Core claim

FairPO treats fairness in multi-document summarization as a preference-optimization problem. Following the Equal Coverage and Coverage Parity definitions of Li et al. (2024), it first estimates, via an entailment model, the probability that each source document is covered by each generated summary. Using these probabilities it identifies the most over- and under-represented social attribute values in a first summary, then re-summarizes the document set after removing a small fraction of documents with each of those values; the summary with the lowest Equal Coverage among the three becomes the chosen preference and the one with the highest becomes the rejected. For corpus-level fairness, the DPO objective is modified so that chosen summaries that counteract a currently over- or under-represented attribute receive higher weight, and rejected summaries that worsen the imbalance receive higher rejection weight. The paper reports that across Llama3.1, Mistral, and Gemma2, on Amazon reviews, MITweet ideology tweets, and SemEval stance tweets, FairPO improves both Equal Coverage and Coverage Parity over DPO, OPTune, a prompting baseline, and a policy-gradient baseline, with a human study on Amazon finding FairPO summaries fairer than DPO summaries in 18 of 30 pairwise comparisons.

Load-bearing premise

The whole method assumes that the unnamed entailment model used to estimate coverage probabilities faithfully measures how well a summary represents each source document; if that model is biased or noisy, every fairness score and training signal is measuring the wrong thing.

Editorial extensions

If this is right

If FairPO is right, preference tuning with document-set perturbation is a viable training-time route to fairness in multi-document summarization, without requiring users to specify fairness constraints.
The corpus-level weighting means fairness can be improved not just summary-by-summary but at the level of the whole output corpus, countering systematic over- or under-representation across many summaries.
Because FairPO preserves relevance, fluency, and factuality on the tested models and datasets, fairness improvements need not come at the cost of summary quality.
The method transfers across three LLM families and three social-attribute domains (sentiment, ideology, stance), suggesting the mechanism is not tied to one model or one attribute type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper leaves the entailment model in Eq. (1) unnamed, a direct test of robustness is to swap in a different NLI model or human coverage annotations and check whether the EC/CP gains persist; this would tell whether the reported fairness improvement is tied to the specific coverage estimator.
The perturbation step creates counterfactual document sets by deleting documents with a given attribute value; the same mechanism could be repurposed as an auditing tool to measure how much each attribute group shifts a summarizer's output.
The dynamic weighting in Eqs. (7)-(9) is defined for any grouping of documents by a shared attribute, so the same fairness-aware DPO objective could be applied to other grouped criteria, such as topic balance or demographic groups in summarization.
The human evaluation only tested Amazon reviews and only compared FairPO against DPO; extending it to the other datasets and to the OPTune or prompting baselines would test whether the entailment-based preference signal aligns with human fairness judgments beyond that single setting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is applying preference tuning, specifically DPO, to fairness in multi-document summarization. The perturbation-based pair generation and the corpus-level dynamic weighting are new, and the experimental coverage is respectable: three LLMs, three datasets, ablations, a human study, and code made available. The writing is clear and the method is easy to reimplement. That is worth taking seriously.

The soft spot is more serious than the reader's report suggests. The stress-test concern lands. In §3.1, the chosen summary is the one with the lowest Equal Coverage among summaries of the original and perturbed document sets. Since the perturbed sets are created by deleting documents of the overrepresented attribute, a summary that simply omits those documents can score a lower EC on the original document set than a genuinely balanced summary, especially when the original model summary was itself imbalanced. Concretely, with four positive and four negative reviews, a base summary that covers all positives but only some negatives has an EC around 0.25; deleting one positive and then covering the remaining seven documents drops EC to about 0.125. The gain comes from dropping the positive review, not from representing all documents fairly. The stress-test's numeric example says 0.125 is "lower than" the balanced summary's 0.0, which is backwards, but the underlying mechanism is correct with a realistic base summary. The human evaluation only asks annotators whether a summary leans positive, negative, or fair; it never checks whether individual reviews were silently dropped. So the automatic result and the human result can both be consistent with metric gaming.

There are also two smaller issues: the entailment model behind Eq. (1) is never named, which makes the core metric hard to audit, and the main automatic tables report means without standard deviations or significance tests, though the per-split tables do help. The self-citation of Li et al. (2024) for the metric is not itself a flaw, but it does mean the training and evaluation share the same, possibly gameable, objective.

This is not a desk-reject. The idea is useful, the experiments are broad, and the flaw is fixable. A serious referee should ask the authors to report whether FairPO systematically omits documents of the overrepresented attribute, e.g., by measuring document-level coverage rates or adding a human completeness judgment. If the omission effect is real, the method needs a completeness penalty or a different preference-pair construction. I would send it to review, but expect the authors to need to do additional work before the central claim is convincing.

Referee Report

4 major / 4 minor

Summary. The paper proposes FairPO, a preference-tuning method for improving fairness in multi-document summarization. FairPO generates preference pairs by perturbing document sets (removing a small fraction of documents from the over- or under-represented social attribute group) and then applies DPO with dynamically adjusted weights for chosen and rejected summaries to improve both summary-level (Equal Coverage) and corpus-level (Coverage Parity) fairness. The method is evaluated on three LLMs (Llama3.1, Mistral, Gemma2) and three datasets (Amazon, MITweet, SemEval), with automatic fairness metrics, ablations, a human evaluation on Amazon, and an LLM-based quality evaluation. The authors report that FairPO outperforms strong baselines on overall EC and CP while maintaining summary quality.

Significance. If the reported results hold, FairPO would be a useful contribution: it is a clearly specified preference-tuning approach that targets both summary-level and corpus-level fairness in MDS, and it is evaluated across multiple models and datasets with ablations and a small human study. The paper also releases code, which aids reproducibility. However, the central empirical claim is weakened by several load-bearing issues: the entailment model behind the coverage probabilities is unnamed, the main automatic results lack significance testing, the training objective is not fully specified as a loss, and the perturbation-based preference pairs may allow the model to reduce EC by omitting documents from an attribute group rather than by genuinely balancing representation. These concerns need to be addressed before the claims can be fully accepted.

major comments (4)

[§2, Eq. (1)] The entailment model used in Eq. (1) is never named. The coverage probabilities p(di, sj) are the foundation for EC/CP computation, preference-pair selection in §3.1, and the weights in Eqs. (7)–(9); without specifying the model (e.g., a HuggingFace model identifier), its version, and how the entailment probability is extracted, the results are not reproducible and the validity of the automatic evaluation cannot be assessed.
[§3.2, Eq. (5) and App. A.3, Eq. (12)] The training objective is not fully specified. Eq. (5) as written is a weighted log-ratio expression multiplied by a detached σ(−m) factor, not a loss function; the derivative in Eq. (12) is asserted, but no explicit loss whose gradient equals that expression is given. Please state the actual loss (or provide pseudocode) and explain why the stop-gradient on σ(−m) is used; otherwise the method cannot be implemented exactly as intended.
[§4.3, Tables 2–3 and App. A.6] The main automatic results report averages over three dataset splits without standard deviations or significance tests. Several differences are small (e.g., Table 2, Llama3.1 Overall EC: FairPO 4.53 vs. OPTune 4.54; CP 0.50 vs. 0.69), and the per-split tables show noticeable variability. Please add error bars or paired significance tests, and state where FairPO is not significantly better than the best baseline.
[§3.1, §4.5] The perturbation procedure may reduce EC by training the model to omit documents from the over-represented attribute group rather than by balancing coverage. A summary generated from a document set with α% of the over-represented attribute removed can achieve a lower EC on the original set simply by not covering those removed documents; the current human evaluation in §4.5 only asks annotators to judge whether a summary is lean/fair, not whether it omits important content. Please provide per-attribute coverage or omission statistics, or a human completeness evaluation, to rule out that the reported gains are driven by selective omission rather than fair representation.

minor comments (4)

[Tables 3 and 5] The name "FariPO" appears in the Llama3.1 and Mistral rows of Tables 3 and 5; this should be "FairPO".
[Table 2, App. A.5] The Policy Gradient baseline (Policy G.) is only reported for Llama3.1; the main text should state that it was omitted for Mistral and Gemma2 due to training instability, rather than only mentioning this in the appendix.
[§3.2, Eqs. (8)–(9) and App. A.4] The equations use the raw sum Ck(D, S*) as the exponent, but the appendix explains that Ck is actually normalized separately over positive and negative values in the implementation. This normalization should be described in the main text because it changes the meaning of the equations.
[§4.5] The human evaluation covers only the Amazon dataset and only the FairPO vs. DPO comparison; the paper should acknowledge more explicitly that the claimed advantage over other baselines (e.g., OPTune, Prompt) is not human-validated.

Circularity Check

3 steps flagged · score 4.0 of 10

The fairness metric from the authors' prior work doubles as both the training signal and the automatic evaluation; automatic EC/CP gains are substantially self-referential, yet the human evaluation provides partial independent support.

self definitional [Section 3.1 (Perturbation-based Preference Pair Generation), used with Eq. (3); evaluated in Section 4.3 Table 2]
"Among summaries S, S+, S−, FairPO selects the summary with the lowest Equal Coverage value, indicating the best summary-level fairness, as the chosen summary Sc. The summary with the highest Equal Coverage value is selected as the rejected summary Sr."

The chosen summary is defined as the minimum-EC summary and the rejected summary as the maximum-EC summary; DPO is then trained to increase the likelihood of the chosen summary and decrease the rejected one. Consequently, the EC reduction reported in Table 2 is the direct target of the preference signal, not an independent evaluation of fairness. The held-out test split makes the improvement not literally guaranteed, but the automatic result is largely a consequence of optimizing the same quantity that is later reported.
self definitional [Section 3.2 (Fairness-aware Preference Tuning), Eqs. (7)-(9), evaluated in Section 4.3 Table 2]
"To estimate corpus-level fairness, FairPO computes the sum of coverage probability differences for documents with social attribute values of k, Ck(D, S∗) = ... A summary S∗ is considered overrepresenting or underrepresenting the social attribute value k if the sum of coverage probability differences, Ck(D, S∗), is greater or less than zero respectively."

The corpus-level weights wc and wr are computed from Ck(D,S*), which is the per-document-set sum of the same coverage probability differences whose corpus-level average defines Coverage Parity (CP) in Eq. (4). Thus the training weights are explicitly constructed to reduce CP, and the CP gains in Table 2 are the result of that direct optimization signal, not an independent measure of corpus-level fairness.

1 more flagged steps

self citation load bearing [Section 2 (Background)]
"To evaluate fairness in MDS, we use Equal Coverage EC(D, S), a summary-level measure, and Coverage Parity CP(G), a corpus-level measure, proposed by Li et al. (2024). Below, we summarize these concepts as introduced in the original paper."

Li et al. (2024) shares three authors with the present paper (Haoyuan Li, Rui Zhang, Snigdha Chaturvedi), so the only automatic fairness yardstick used in both the training objective and the evaluation tables comes from the authors' own prior work. No external or independently validated fairness metric is reported for the main automatic comparisons. The human evaluation on Amazon and the Prometheus-2 quality evaluation provide some independent grounding, so the self-citation is not total, but the automatic fairness claim rests on a self-defined instrument.

full rationale

FairPO's automatic evaluation is substantially self-referential. The summary-level objective selects preference pairs by Equal Coverage (Eq. 3), and the corpus-level weights in Eqs. (7)-(9) are built from the same coverage-probability differences that define Coverage Parity (Eq. 4). Reporting EC and CP improvements after tuning with those objectives is therefore not an independent test of fairness; the gains are at least partly by construction. Additionally, the EC/CP metrics are cited from Li et al. (2024), a paper with overlapping authorship, so the central automatic fairness claim relies on the authors' own measure. The paper does contain independent evidence: a human evaluation on Amazon in which FairPO-tuned summaries are judged fairer than DPO-tuned summaries in 18 of 30 pairs (p < 0.05), and a Prometheus-2-based quality check. These prevent the claim from being fully circular, but the human evaluation covers only one dataset, only the DPO comparison, and does not validate EC/CP as a faithful measure of fairness. The omission-based failure mode noted in the skeptical reading is a metric-validity risk rather than a formal circularity, but it reinforces that low EC can be achieved by dropping documents of the overrepresented attribute, so the automatic EC/CP numbers should not be read as independent confirmation. Overall, the score is 4: there is meaningful self-citation and the automatic results reduce partly to the training objective, yet the human evaluation gives the central claim some independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

FairPO introduces no new physical or conceptual entities. The load-bearing assumptions are the validity of the entailment-based coverage metric, the informativeness of perturbation-based preference pairs, and the heuristic corpus-level weighting. The main free parameters are alpha, tau, and gamma, all tuned on validation data.

free parameters (4)

alpha (perturbation percentage) = 10%
Fraction of documents removed to create perturbed document sets; tuned on validation (Section 4.2, Appendix A.4).
temperature tau = 1, 2, or 3 depending on model and dataset
Smoothing temperature in the weighting formulas Eqs. (8)-(9); tuned per model and dataset on validation.
discount factor gamma = 0.75 for Llama3.1, 0.5 for other models
Decay factor for recent training steps when estimating overrepresentation and underrepresentation (Appendix A.4).
LoRA rank and scaling factor = 16 and 16
Low-rank adapter configuration chosen for training cost; standard hyperparameter rather than a domain-specific fit.

assumptions (5)

domain assumption The entailment model used in Eq. (1) gives accurate coverage probabilities
All EC/CP scores and training weights depend on these probabilities, yet the model is never named, so this assumption is unverified.
domain assumption EC and CP from Li et al. (2024) are valid measures of fairness
The fairness target is defined by these metrics; the human evaluation covers only one dataset and one comparison, so the metric's validity as a general fairness measure is largely assumed.
domain assumption Preference pairs generated by perturbing document sets and selecting min/max EC summaries are informative for DPO training
The method assumes that removing alpha percent of a social-attribute group and comparing summaries teaches the model to balance coverage; no theoretical guarantee is provided.
ad hoc to paper The corpus-level weighting formula in Eqs. (8)-(9) improves corpus-level fairness
This weighting is a heuristic design with no derivation and no theoretical justification; support comes only from empirical ablations.
domain assumption LoRA-based DPO tuning preserves summary quality
Quality preservation is evaluated only with an LLM judge and only for consistent pairwise judgments, so it is not established independently of the judge model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Fairness of Large Language Models in Multi-document Summarization." pith.science (2026). https://pith.science/paper/M427FVJB

@misc{pith2026250607479,
  author       = {Pith},
  title        = {Pith review of: Improving Fairness of Large Language Models in Multi-document Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M427FVJB}},
  note         = {Machine review of arXiv:2506.07479}
}

read the original abstract

Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact decision-making. For example, a summarization system that tends to overrepresent negative reviews of products can mislead customers into disregarding good products. Previous works measure fairness in MDS at two levels: summary-level and corpus-level. While summary-level fairness focuses on individual summaries, corpus-level fairness focuses on a corpus of summaries. Recent methods primarily focus on summary-level fairness. We propose FairPO, a preference tuning method that focuses on both summary-level and corpus-level fairness in MDS. To improve summary-level fairness, we propose to generate preference pairs by perturbing document sets. To improve corpus-level fairness, we propose fairness-aware preference tuning by dynamically adjusting the weights of preference pairs. Our experiments show that FairPO outperforms strong baselines while maintaining the critical qualities of summaries. The code is available at https://github.com/leehaoyuan/coverage_fairnes.

Figures

Figures reproduced from arXiv: 2506.07479 by the authors.

**Figure 2.** Summarization prompt for the Amazon Dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Sample summaries generated by DPO and FairPO. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 8 canonical work pages

[1]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024
[2]

Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, and Heng Huang. 2024. Optune: Efficient online preference tuning. arXiv preprint arXiv:2406.07657

arXiv 2024
[3]

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

2021
[4]

Kung-Hsiang Huang, Philippe Laban, Alexander R Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint arXiv:2309.09369

arXiv 2023
[5]

Nannan Huang, Haytham Fayek, and Xiuzhen Zhang. 2024. Bias in opinion summarisation from pre-training to adaptation: A case study in political bias. arXiv preprint arXiv:2402.00322

work page Pith review arXiv 2024
[6]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

arXiv 2023
[7]

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language models . Preprint, arXiv:2405.01535

arXiv 2024
[8]

Philipp Koehn. 2004. https://aclanthology.org/W04-3250 Statistical significance tests for machine translation evaluation . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388--395, Barcelona, Spain. Association for Computational Linguistics

work page 2004

Show all 26 references

[9]

Yuanyuan Lei, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Ruihong Huang, and Dong Yu. 2024. Polarity calibration for opinion summarization. arXiv preprint arXiv:2404.01706

2024 arXiv
[10]

Haoyuan Li, Yusen Zhang, Rui Zhang, and Snigdha Chaturvedi. 2024. https://arxiv.org/abs/2412.08795 Coverage-based fairness in multi-document summarization . Preprint, arXiv:2412.08795

2024 arXiv
[11]

Songtao Liu, Ziling Luo, Minghua Xu, LiXiao Wei, Ziyao Wei, Han Yu, Wei Xiang, and Bang Wang. 2023. Ideology takes multiple looks: A high-quality dataset for multifaceted ideology detection. In The 2023 Conference on Empirical Methods in Natural Language Processing

2023
[12]

Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. https://doi.org/10.18653/v1/S16-1003 S em E val-2016 task 6: Detecting stance in tweets . In Proceedings of the 10th International Workshop on Semantic Evaluation ( S em E val-2016) , pa...

2016 doi
[13]

Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. https://doi.org/10.18653/v1/D19-1018 Justifying recommendations using distantly-labeled reviews and fine-grained aspects . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In...

2019 doi
[14]

Olubusayo Olabisi, Aaron Hudson, Antonie Jetter, and Ameeta Agrawal. 2022. https://aclanthology.org/2022.coling-1.542 Analyzing the dialect diversity in multi-document summaries . In Proceedings of the 29th International Conference on Computational Linguistics, pages 6208--622...

2022
[15]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...

2022
[16]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[17]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

2020
[18]

Justus J Randolph. 2005. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss' fixed-marginal multirater kappa. Online submission

2005
[19]

Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Leonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos Garea, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietqu...

2023 doi
[20]

Anurag Shandilya, Kripabandhu Ghosh, and Saptarshi Ghosh. 2018. Fairness of extractive text summarization. In Companion Proceedings of the The Web Conference 2018, pages 97--98

2018
[21]

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

2020
[22]

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

2024 arXiv
[23]

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, et al. 2023. Fair abstractive summarization of diverse perspectives. arXiv preprint arXiv:2311.07884

2023 arXiv
[24]

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

2019 arXiv
[25]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[26]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024

[2] [2]

Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, and Heng Huang. 2024. Optune: Efficient online preference tuning. arXiv preprint arXiv:2406.07657

arXiv 2024

[3] [3]

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

2021

[4] [4]

Kung-Hsiang Huang, Philippe Laban, Alexander R Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint arXiv:2309.09369

arXiv 2023

[5] [5]

Nannan Huang, Haytham Fayek, and Xiuzhen Zhang. 2024. Bias in opinion summarisation from pre-training to adaptation: A case study in political bias. arXiv preprint arXiv:2402.00322

work page Pith review arXiv 2024

[6] [6]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

arXiv 2023

[7] [7]

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language models . Preprint, arXiv:2405.01535

arXiv 2024

[8] [8]

Philipp Koehn. 2004. https://aclanthology.org/W04-3250 Statistical significance tests for machine translation evaluation . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388--395, Barcelona, Spain. Association for Computational Linguistics

work page 2004