Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models

Iacopo Ghinassi; Leonardo Catalano; Tommaso Colella

REVIEW 4 major objections 6 minor 15 references

Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Small language models produce IPCC topic summaries as good as ChatGPT and GPT-4, at far lower estimated energy cost.

desk verdict Useful dataset and an undemonstrated-but-plausible SLM-equivalence claim; the metric ceiling and rough cost estimates mean the conclusions should be softened, but the paper deserves peer review. read the letter →

arxiv 2411.14272 v1 pith:TMZ3YCVU submitted 2024-11-21 cs.CL

Iacopo Ghinassi , Leonardo Catalano , Tommaso Colella This is my paper · ORCID

classification cs.CL

keywords aspect-basedsummarizationsmalllanguagemodelsclimatechangereportszero-shotretrievalaugmentedgenerationenergy-efficientNLPIPCCcarbonfootprint

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that small language models (SLMs) with a few billion parameters can summarize IPCC climate reports by topic just as well as ChatGPT and GPT-4, while consuming much less energy. To test this, the authors release SumIPCC, a dataset of 140 topic-annotated paragraph-summary pairs from the IPCC synthesis reports, and score zero-shot generated summaries with an LLM-as-judge metric. On gold paragraphs, most SLMs scored statistically indistinguishably from the large models, and the authors conclude that the task is 'virtually solved' in this easier setting. When retrieval is added, the task becomes harder: automatic paragraph retrieval degrades all summary qualities, though the smallest competitive model still matches the best large model once energy cost is factored in. The paper's central efficiency conclusion is that large proprietary models are the worst choice when effectiveness is weighted by energy use.

What carries the argument

The central mechanism is a simple fixed prompt template asking the model to summarize a text with respect to a topic, applied zero-shot. Evaluation uses ChatGPT-RTS, an LLM-as-judge metric scoring coherence, consistency, fluency, and relevance, with the relevance prompt re-adapted to the target aspect. The efficiency comparison uses Carburacy, which re-weights a quality score $R$ by the estimated emission cost $C$ as $\gamma = e^{(\log \alpha) R} / (1 + C\beta)$, with $\beta = 100$ for the main experiments and $\beta = 10000$ set empirically for the retrieval experiments. The authors measure $C$ for open models with a fixed prompt and output length, and estimate ChatGPT and GPT-4 costs by scaling the measured cost of Gemma 2B by parameter count, a step they explicitly call a very rough estimate. The dataset SumIPCC provides 140 topic-annotated pairs drawn from the IPCC Summary for Policy-Makers and the corresponding synthesis reports.

What would settle it

Measure the actual CO2 emissions of one ChatGPT and one GPT-4 API call that generates a topic summary on the SumIPCC examples, using direct instrumentation by the provider or an independent metering setup, and recompute the Carburacy rankings with those figures; if the large models' true per-query cost is comparable to or below that of a 2B-parameter small model, the central efficiency conclusion fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that aspect-based summarization of IPCC reports can be performed out of the box by instruction-tuned small models: over the ChatGPT-RTS dimensions of consistency, coherence, fluency, and relevance, models of 1.8B to 8B parameters receive average scores from 4.77 to 4.90, while ChatGPT receives 4.91 and GPT-4 4.89, with no statistically significant difference from the best model for most SLMs. With the Carburacy energy re-weighting, the small models outperform the proprietary ones on the efficiency-effectiveness trade-off. The authors further show that when the input paragraphs are retrieved automatically rather than given as ground truth, model choice matters more, with Llama 3 and Qwen 1.8B leading, but they regard this RAG scenario as needing more research.

Load-bearing premise

The efficiency ranking depends on the estimated energy costs of ChatGPT and GPT-4, which the authors call a very rough estimate made by scaling a measured small-model cost by parameter count; if the large models' true per-query emissions are much lower, the paper's preference for small models would not follow.

Editorial extensions

If this is right

IPCC readers and policy analysts could obtain topic-specific summaries from 2B-8B open models at roughly two orders of magnitude lower estimated energy cost, with ChatGPT-RTS scores close to those of ChatGPT and GPT-4.
Energy-aware evaluation flips the model ranking: in the efficiency-effectiveness trade-off, ChatGPT and GPT-4 rank worst among the tested systems, so future deployments in the climate domain should start from small models.
When paragraphs are retrieved automatically instead of given, all summary qualities drop and the model choice matters more, so a RAG pipeline for climate reports still requires better retrieval or filtering before it can be trusted.
SumIPCC, with its topic-annotated pairs and section and subsection metadata, gives the community a benchmark for zero-shot and fine-tuned aspect-based summarization in expert-curated reports.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the dataset's gold summaries are largely extractive highlights, the high scores may reflect lexical overlap rather than abstractive synthesis; a version of SumIPCC with more abstractive reference summaries would test whether the SLM advantage persists.
Editorial inference: the energy-cost estimates for proprietary models are scaled by parameter count, not measured, so the paper's quantitative efficiency margin is an order-of-magnitude estimate; direct emissions reporting by API providers would sharpen or overturn the ranking.
Editorial inference: the same zero-shot SLM recipe could transfer to other expert-curated reports, such as health or biodiversity assessments, where a small model running on a local GPU would also address the data-access and inclusivity concerns the authors raise.
Editorial inference: the RAG failure suggests that an aspect-aware retriever using section headers could improve retrieval, and the dataset already includes this metadata, enabling that test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real news here is the resource: SumIPCC, 140 topic-annotated summary-paragraph pairs from the IPCC AR5/AR6 synthesis reports, released with code. That is the main contribution, and it looks carefully put together. The second contribution is applying Carburacy re-weighting to zero-shot generative ABS for the first time; the adaptation is straightforward, and it does what it claims.

What the paper does well: it runs nine SLMs plus ChatGPT and GPT-4 under one prompt template, in both gold-paragraph and RAG settings, and reports raw scores plus energy-aware rankings. The RAG results are genuinely informative—scores spread out, smaller models like Qwen 1.8B stay competitive, and the retrieval failure mode is visible. The authors are also honest about the metric's limitations in Section 6.1 and in the Limitations section; they do not hide the self-bias or the rough emissions estimate.

That honesty does not rescue the central claim. The Q2 conclusion that SLMs are as effective as larger LLMs rests on ChatGPT-RTS scores clustered near 4.8–4.9/5. In that saturated region the metric's discrimination is weak, and since the evaluator is ChatGPT itself, the self-bias concern is not hypothetical—it is the known failure mode. Failing to reject the null with a T-test on 140 examples is absence of evidence, not evidence of equivalence. The human check in Appendix A uses only 10 pairs, too small to certify the high-score range. The efficiency comparison depends on proprietary-model costs scaled from Gemma 2B by parameter count, a rough estimate the authors flag, and the RAG Carburacy beta=10000 is empirical with no sensitivity analysis. These are acknowledged, but together they mean the headline 'SLMs are comparable' is plausible, not demonstrated.

None of this kills the paper. The dataset is a contribution on its own, and the RAG comparison is a useful data point for applied climate NLP and sustainable NLP more broadly. The right fix is to downscale the claim: report that SLMs were not significantly worse in this evaluation, note the metric ceiling, and add sensitivity tests on beta and on the cost assumptions. A stratified human evaluation of a larger sample would also help, but the 10-pair check is enough to show the authors are not hiding the problem.

For peer review: yes, send it out. It is a solid applied-resource paper with honest limitations, and a good referee can push the authors to make the equivalence claim match the evidence. I would cite it for SumIPCC and for the energy-aware comparison methodology.

Referee Report

4 major / 6 minor

Summary. The paper introduces SumIPCC, a new dataset of 140 topic-annotated paragraph–summary pairs from IPCC AR5 and AR6 synthesis reports, and uses it to evaluate nine small language models (SLMs), two proprietary large language models (ChatGPT/GPT-4), an extractive baseline, and a retrieval-augmented generation (RAG) setup. The authors adapt the Carburacy carbon-aware re-weighting framework to zero-shot generative summarization, and they report that SLMs are statistically indistinguishable from larger LLMs on the ChatGPT-RTS metric while being much more energy-efficient. The RAG results show larger performance gaps but still identify small models as competitive after re-weighting. The paper releases code and data.

Significance. The dataset and the efficiency-aware evaluation framework are useful contributions for a socially relevant domain. If the central claim holds, the work would support the use of SLMs for aspect-based summarization of climate reports, which is a meaningful practical finding for resource-constrained settings. The paper is commendable for releasing its dataset and code, for adapting Carburacy to generative models, and for explicitly acknowledging limitations of the evaluation metric and the rough nature of the energy estimates. However, the strength of the main conclusion depends on the validity of the ChatGPT-RTS metric in the near-ceiling regime and on the reliability of the estimated energy costs for proprietary models.

major comments (4)

[Section 6.1, Table 4] The claim that SLMs are 'as effective as larger LLMs' (Q2) rests on ChatGPT-RTS scores that are near ceiling (most averages between 4.8 and 4.9 out of 5). The paper itself states that the metric has lower reliability for high-quality summaries and may be biased toward ChatGPT. A two-sample t-test that fails to reject the null does not establish equivalence; with a saturated metric, the test has low power to detect genuine differences. To support the equivalence claim, the authors should report score distributions, effect sizes, and ideally a larger human evaluation that is powered to detect differences in the high-score range. This is load-bearing because the subsequent energy-efficiency trade-off conclusion assumes equal effectiveness.
[Section 5.1, Table 2] The energy costs for ChatGPT and GPT-4 are estimated by scaling the measured cost of Gemma 2B by the assumed parameter ratio, equating both proprietary models to GPT-3's ~175B parameters. The paper calls this 'a very rough estimate' (Section 5.1). The Carburacy re-ranking in Figure 2, which ranks ChatGPT and GPT-4 as the worst efficiency/effectiveness trade-off, depends critically on this estimate. If the true per-query emissions of the proprietary API models are lower (e.g., due to optimized inference hardware or batch serving), the efficiency conclusion would weaken. Please provide a sensitivity analysis over the proprietary cost estimates or justify the scale factor with external evidence.
[Section 3.5, Eq. (11)] The RAG-related Carburacy re-weighting sets beta = 10000 'empirically', while the original work uses beta = 100. No sensitivity analysis or criterion for this choice is given. Because the C values for retriever models span several orders of magnitude (Table 3, e.g., MiniLM at 4.42e-10 vs. other models at ~4e-5), the re-weighted results in Figures 3 and 4 are highly sensitive to beta. The authors should justify the beta value and show whether the reported rankings are stable across a range of beta values. Otherwise, the conclusion that 'comparatively smaller models are the best choice' for retrieval is not robust.
[Appendix A] The human validation of ChatGPT-RTS is based on only 10 summary pairs and two annotators, with agreement around 80% and large standard deviations on several dimensions (Table 6). The paper's Limitations section acknowledges the metric's lower correlation with human judgment for high-quality systems, which is exactly the regime of the main experiment. This evidence is too thin to certify that ChatGPT-RTS discriminates reliably among summaries that all score near the top of the scale. A larger, more systematic human study, or an alternative evaluation protocol, is needed before the 'virtually solved' claim (Section 7) can be supported.

minor comments (6)

[Section 2.2] 'where the firsts have shown improvements' should be 'where the former have shown improvements'.
[Section 3.2, Eq. (6)] The notation 'argmax_{i∈Pind}(cos(q,e_i)), s.t. |Psub|=k' is imprecise: argmax returns a single index, not a set of k indices. Please rephrase as selecting the top-k indices by cosine similarity.
[Section 5.1] 'for the module of the respective model parameters' is likely a typo; 'modulus' or 'ratio' seems intended.
[Table 3] The energy cost for MiniLM (4.42e-10) is several orders of magnitude lower than all other retrievers (4.06e-05 or higher). This is suspicious and should be checked; it may be a unit or measurement error that affects the retrieval re-ranking.
[Section 3.4.2] The MRR@10 cutoff is described only in text; it would be clearer to define it in the metric equation or a footnote for reproducibility.
[Appendix F] The paper says ChatGPT and GPT-4 were accessed via the official API, but no details are given on how their emissions would be attributed; the estimation method in Section 5.1 should be cross-referenced here.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SLM/LLM comparison is an empirical evaluation with an external metric; acknowledged biases and arbitrary hyperparameters are validity concerns, not derivation loops.

full rationale

The paper's derivation chain is not circular. The central empirical claim (Q2: SLMs are comparable to larger LLMs, Section 6.1) is obtained by running a fixed prompt template on 140 ground-truth paragraph/topic pairs and scoring outputs with the external ChatGPT-RTS metric; the claim is not defined in terms of the metric, and no equation maps the metric into the conclusion by construction. The acknowledged fact that ChatGPT-RTS is itself ChatGPT-based and may favor ChatGPT (Section 6.1; Limitations) is a measurement-validity caveat, not a logical reduction. If anything, a ChatGPT bias would inflate ChatGPT's score and make the SLM-equivalence finding more conservative, not less. The near-ceiling scores and small human-validation sample (Appendix A, 10 pairs) are evidence-strength concerns, not circularity. The Carburacy reweighting does contain an arbitrary choice, beta=10000 'empirically set' for retrieval, and the ChatGPT/GPT4 energy costs are rough parameter-count estimates (Section 5.1); these affect the efficiency conclusions, but they are stated assumptions rather than fitted parameters renamed as predictions. The paper contains no load-bearing self-citations: Carburacy and ChatGPT-RTS are external prior works, and the authors do not invoke their own prior results as authority. Issues of metric saturation, low power in the t-test, and absence-of-evidence reasoning are correctness/validity risks for a different review pass; they do not make the derivation equivalent to its inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the validity of the ChatGPT-based evaluation metric, the accuracy of the energy cost estimates (especially for proprietary models), and several hand-set hyperparameters such as beta=10000 and k=2. These are domain assumptions rather than derived results, and they directly influence the paper's main comparative conclusions.

free parameters (5)

beta for RAG Carburacy re-weighting = 10000
Set empirically in Section 3.5 to account for emission scale; no sensitivity analysis, directly affects the re-ranked model order in the RAG experiments.
number of retrieved paragraphs k = 2
Used to build the input text for RAG summarization; no justification or sweep is provided (Sections 3.2 and 6.2).
proprietary cost scale factor = parameter ratio ~87.5 (175B/2B) times Gemma 2B cost
Authors estimate ChatGPT and GPT4 cost by multiplying the measured cost of Gemma 2B by the ratio of model parameters (Section 5.1), described as 'a very rough estimate'.
MRR cutoff = 10
MRR@10 hyperparameter set to 10; retrievers are scored only on the first 10 ranked items (Section 3.4.2).
character threshold for iterative summarization = not reported
Threshold described as 'empirically set' in Appendix C; affects whether input is summarized iteratively, which has a small measured performance cost.

assumptions (5)

domain assumption ChatGPT-RTS metric scores correlate with human judgment for this dataset.
The paper relies on ChatGPT as the evaluator (Section 3.4.1) and validates with only 10 pairs and two annotators (Appendix A), noting the metric is biased toward ChatGPT and less reliable for high-quality summaries.
domain assumption Codecarbon accurately measures per-example CO2 emissions on the T4 GPU.
Energy costs in Tables 2 and 3 are measured with codecarbon (Section 3.5) but no verification or error bars are provided.
domain assumption 4-bit quantization does not materially change model outputs.
All SLMs are quantized to 4 bits (Section 5.1), citing Yao et al. (2024); no per-model check is reported.
domain assumption The manual filtering of 'too broad' summaries does not bias the dataset.
Section 4 reports that overly broad summaries were filtered out, which may remove the hardest cases and inflate apparent model performance.
domain assumption A single fixed prompt template fairly compares all models.
All systems use one prompt template (Section 3.1) with no per-model tuning or validation that the template is equally clear for all models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models." pith.science (2026). https://pith.science/paper/TMZ3YCVU

@misc{pith2026241114272,
  author       = {Pith},
  title        = {Pith review of: Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TMZ3YCVU}},
  note         = {Machine review of arXiv:2411.14272}
}

read the original abstract

The use of Natural Language Processing (NLP) for helping decision-makers with Climate Change action has recently been highlighted as a use case aligning with a broader drive towards NLP technologies for social good. In this context, Aspect-Based Summarization (ABS) systems that extract and summarize relevant information are particularly useful as they provide stakeholders with a convenient way of finding relevant information in expert-curated reports. In this work, we release a new dataset for ABS of Climate Change reports and we employ different Large Language Models (LLMs) and so-called Small Language Models (SLMs) to tackle this problem in an unsupervised way. Considering the problem at hand, we also show how SLMs are not significantly worse for the problem while leading to reduced carbon footprint; we do so by applying for the first time an existing framework considering both energy efficiency and task performance to the evaluation of zero-shot generative models for ABS. Overall, our results show that modern language models, both big and small, can effectively tackle ABS for Climate Change reports but more research is needed when we frame the problem as a Retrieval Augmented Generation (RAG) problem and our work and dataset will help foster efforts in this direction.

Figures

Figures reproduced from arXiv: 2411.14272 by the authors.

**Figure 3.** Retrieval results in terms of MRR@10 metric [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** ChatGPT RTS Average scores for the RAG experiment re-weighted via the Carburacy method. presents the biggest overall drop in performance, as it could have been expected). This evidence suggests that our task in a RAG setting is indeed a more challenging task, which requires further investigation both in terms of the retrieval model being used and in terms of the summarization model. Different LLMs, in fact, appear… view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Average percentage of agreement between human annotators and similarity-based summarization metrics: standard deviation is also included in the form of error bars. B Evaluation Prompts In using the ChatGPT RTS, we have prompted ChatGPT with 4 different prompts per summ…

**Figure 7.** Figure 7: Average comparison of performance across [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 6.** Figure 6: The prompt used for evaluation with ChatGPT [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 8.** Figure 8: An example of three summaries obtained us [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: An example of three summaries obtained using three LLMs: ChatGPT, Llama 3 and Mistral (the weakest LLM among this set of experiments). Prompt indicates the command given to the LLMs, including the text to be summarized and the target aspect. The reference summary is de…

**Figure 10.** Figure 10: Average word count in the reference sum [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Average word count in the target paragraphs [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

**Figure 14.** Figure 14: Rouge-1, rouge-2 and rouge-l scores of the [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

Qwen 0.5B: https://huggingface.co/ Qwen/Qwen1.5-0.5B-Chat

work page
[2]

Qwen 1.8B: https://huggingface.co/ Qwen/Qwen1.5-1.8B-Chat

work page
[3]

Qwen 4B: https://huggingface.co/ Qwen/Qwen1.5-4B-Chat

work page
[4]

Qwen 7B: https://huggingface.co/ Qwen/Qwen1.5-7B-Chat

work page
[5]

In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4381–4401, Toronto, Canada

OASum: Large-scale open domain aspect- based summarization. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4381–4401, Toronto, Canada. Association for Com- putational Linguistics. Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2024. Exploring post-training quantiza- tion in llms from comprehensive study to ...

work page 2023
[6]

Transactions of the Associa- tion for Computational Linguistics, 12:39–57

Benchmarking Large Language Models for News Summarization. Transactions of the Associa- tion for Computational Linguistics, 12:39–57. Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core compe- tency: Survey on evaluation of large language mod- els. In Proceedings o...

work page 2023
[7]

Gemma 7B: https://huggingface.co/ google/gemma-1.1-7b-it

work page
[8]

Phi 3: https://huggingface.co/ microsoft/Phi-3-mini-128k-instruct

work page

Show all 15 references

[9]

All the models run between 2.5 and 10 hours, depending on model size and length of generated summaries: no sampling was applied for replicability

Mistral: https://huggingface.co/ mistralai/Mistral-7B-Instruct-v0.2 The models were all quantized in 4 bit with the bitandbytes python library 4 and run on a single NVIDIA® T4 GPU5 with 16GB of RAM, as pre- viously explained. All the models run between 2.5 and 10 hours, depend...

2023
[11]

Llama 3: https://huggingface.co/ meta-llama/Meta-Llama-3-8B
[12]

Gemma 2B: https://huggingface.co/ google/gemma-1.1-2b-it
[2019]

In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 42–47, Hong Kong, China

Unsupervised aspect-based multi-document abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 42–47, Hong Kong, China. Association for Computational Linguistics. Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, ...

2024 arXiv
[2021]

Transactions of the Associa- tion for Computational Linguistics, 9:211–225

WikiAsp: A Dataset for Multi-domain Aspect- based Summarization. Transactions of the Associa- tion for Computational Linguistics, 9:211–225. IPCC. 2014a. Climate change 2014: Summary for policy-makers. Technical report, IPCC. IPCC. 2014b. Climate change 2014: Synthesis report....

2014 arXiv
[2023]

Proceedings of the AAAI Conference on Artificial Intelligence, 37(12):14417– 14425

Carburacy: Summarization models tuning and comparison in eco-sustainable regimes with a novel carbon-aware accuracy. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12):14417– 14425. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB:...

2023 arXiv
[2024]

Preprint, arXiv:2404.13076

Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In Proceedings of the Third International Conference on Language Resources and Ev...

2002 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Qwen 0.5B: https://huggingface.co/ Qwen/Qwen1.5-0.5B-Chat

work page

[2] [2]

Qwen 1.8B: https://huggingface.co/ Qwen/Qwen1.5-1.8B-Chat

work page

[3] [3]

Qwen 4B: https://huggingface.co/ Qwen/Qwen1.5-4B-Chat

work page

[4] [4]

Qwen 7B: https://huggingface.co/ Qwen/Qwen1.5-7B-Chat

work page

[5] [5]

In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4381–4401, Toronto, Canada

OASum: Large-scale open domain aspect- based summarization. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4381–4401, Toronto, Canada. Association for Com- putational Linguistics. Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2024. Exploring post-training quantiza- tion in llms from comprehensive study to ...

work page 2023

[6] [6]

Transactions of the Associa- tion for Computational Linguistics, 12:39–57

Benchmarking Large Language Models for News Summarization. Transactions of the Associa- tion for Computational Linguistics, 12:39–57. Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core compe- tency: Survey on evaluation of large language mod- els. In Proceedings o...

work page 2023

[7] [7]

Gemma 7B: https://huggingface.co/ google/gemma-1.1-7b-it

work page

[8] [8]

Phi 3: https://huggingface.co/ microsoft/Phi-3-mini-128k-instruct

work page