Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Exploring the Impact of Temperature on Large Language Models:Hot or Cold?

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that temperature effects on LLMs are ability-specific and scale with model size, so a prompt-level selector can beat the default setting.

desk verdict A useful temperature sweep across abilities and model sizes, but the headline Mutation Temperature scaling claim is internally contradicted and not operationally defined. read the letter →

arxiv 2506.07295 v1 pith:JPPK7COZ submitted 2025-06-08 cs.CL

Lujun Li , Lama Sleem , Niccolo' Gentile , Geoffrey Nichil , Radu State This is my paper
classification cs.CL
keywords samplingtemperaturelargelanguagemodelsmodelperformanceevaluationBERT-basedclassifierGPT-basedmutationSuperGLUE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to replace guesswork about sampling temperature in large language models with measurement. The authors evaluate twelve open-weights models in three size classes on six capabilities across temperatures from 0.1 to 1.9, and find that temperature effects are ability-specific and size-dependent. They also define a “mutation temperature,” the point where performance shifts sharply, and report that this threshold rises with model size. On that basis they build a selector that labels a prompt by the ability it requires and chooses the best temperature from the measured curves, which improves SuperGLUE accuracy for small and medium models. The practical payoff is that temperature can be treated as a tunable per-prompt parameter rather than a fixed dial.

What carries the argument

The load-bearing object is the per-ability, per-size temperature–performance curve, measured under a single-prompt protocol with three repetitions, together with the softmax temperature rule $P_i = e^{y_i/T} / \sum_j e^{y_j/T}$ that makes the curve meaningful. The paper uses the curve to define the “mutation temperature,” the value at which performance drops sharply, and to serve as the lookup table for the BERT-based selector, which predicts the dominant ability of a prompt from its text and returns the temperature that maximized the curve for that ability and model size.

What would settle it

Re-score a random sample of the causal reasoning, creativity, and instruction following outputs with human annotators, then recompute the temperature–performance curves; if the curves change sign or lose their measured correlations, the core temperature effects for those abilities do not survive.

Watch

Extended reading notes

Core claim

The central claim is that sampling temperature is not a uniform knob: it changes different abilities in different directions, and those changes shrink as models grow. Statistically, instruction following, machine translation, and summarization show the strongest negative correlations with temperature, with significance values of zero, while causal reasoning and in-context learning are only mildly affected. For small models the relative performance swings reach 186.81% on creativity and 192.32% on machine translation, while large models stay nearly flat. The paper also identifies a “mutation temperature” for each size class, the point where performance drops sharply, and reports that it increases with model size, for example on instruction following it sits between 1.0 and 1.3 for small models, between 1.3 and 1.6 for medium models, and between 1.6 and 1.9 for large models. A BERT-based selector trained to label prompts by the dominant ability picks the temperature that maximized the measured curve for that ability, improving average SuperGLUE accuracy from 0.252 to 0.494 for the smallest model while leaving large models unchanged, and the same optimal temperatures hold under 4-bit and FP16 inference.

Load-bearing premise

The load-bearing premise is that GPT-3.5-turbo's judgments of causal reasoning, creativity, and instruction following match what human raters would say, since those three abilities are scored by the model rather than by exact matches.

Editorial extensions

If this is right

  • Default temperature 1.0 is often suboptimal: for small models, choosing the right temperature roughly doubles SuperGLUE accuracy on some tasks, such as the WIC task.
  • Machine translation and summarization favor low temperatures, while creativity peaks at moderate values around 1.3, so deployment guidelines should be ability-aware.
  • Larger models do not need temperature tuning as urgently; the measured variability and correlations weaken as model size grows.
  • Instruction following degrades abruptly past a model-specific threshold, so staying below temperature 1 is a safe default for strict instruction tasks.
  • Quantization to 4-bit does not change where the optimal temperature sits, so temperature-tuning results transfer across inference precisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mutation-temperature trend holds beyond 80B parameters, a scaling law $T^*\sim c+\lambda\log N$ could let practitioners pick a safe temperature ceiling from model size alone, a testable extension the paper does not run.
  • The selector's reliance on six discrete abilities suggests a natural generalization: train the prompt-to-temperature mapping directly on prompt embeddings, which might handle mixed-ability prompts such as “summarize this in French” better than a single-ability label.
  • Because the judge-based abilities rest on GPT-3.5-turbo, an obvious stress test is to swap the judge for GPT-4o or for human ratings; agreement between judge sources would strengthen the claim that temperature effects on creativity are real.
  • The paper's temperature curves could also be used to calibrate decoding for RAG or agent loops, where repeated sampling at the wrong temperature may compound errors across turns; no such extension is tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper reports a systematic empirical study of sampling temperature (0.1–1.9) on six abilities—causal reasoning, creativity, in-context learning, instruction following, machine translation, and summarization—across 12 open-source LLMs grouped into small, medium, and large size bands, all evaluated with 4-bit AWQ quantization. It reports Pearson and Spearman correlations between temperature and performance, claims stronger negative temperature sensitivity for IF, MT, and SUMM, and infers that larger models are more robust to temperature variation. The paper then proposes a BERT-based and a GPT-based prompt-to-ability selector that chooses the temperature maximizing performance on the main experimental response curves, and demonstrates improvements on three SuperGLUE tasks for small and medium models. Extending temperature to 4.0 and comparing FP16 with 4-bit inference, the paper introduces the notion of a 'Mutation Temperature' and claims it increases with model size.

Significance. If the empirical claims held, the paper would offer practical guidance for temperature selection and a deployable selector for small and medium models. Strengths include the use of publicly specified benchmarks and models, a wide temperature sweep, a concrete selector artifact, and the explicit separation of six abilities rather than a single aggregate accuracy. However, several load-bearing parts are currently unsupported: the unvalidated GPT-3.5 judge for three of the six abilities, the contradictory and undefined 'Mutation Temperature' scaling claim, and statistical inferences drawn from seven temperature points with p-values of 0.00 and no confidence intervals. These issues are fixable with additional experiments and a reanalysis, but they prevent the paper from being accepted in its current form.

major comments (5)
  1. [Abstract; Sections 4.2.2, 5.3.2, 6] The 'Mutation Temperature' is never operationally defined, and the paper contradicts itself. Section 4.2.2 states 'we did not find evidence for a specific “Mutation Temperature” in large models,' while Section 5.3.2 asserts 'each model has its own mutation temperature, with larger models generally exhibiting a higher threshold,' and Section 6 concludes 'large models still have a mutation temperature, but it is higher than 2.0.' Because the abstract's scaling statement ('Mutation Temperature ... increases with model size') depends entirely on this construct, the authors must provide a precise definition (e.g., the smallest temperature at which accuracy drops by a specified fraction below the low-temperature baseline, or a fitted changepoint) and report per-model estimates with uncertainty, or remove the claim.
  2. [Sections 4.1 and 6] For CR, CT, and IF, the evaluation relies on gpt-3.5-turbo-0125 as an LLM judge, but the paper reports no human-alignment study for this specific judge and metric; the authors acknowledge in Section 6 that 'It is difficult to verify the accuracy of the GPT model evaluations without manually checking each case.' The temperature-performance correlations, the optimal-temperature curves, and the selector's T* for these abilities all inherit the judge's noise or bias. This is load-bearing; a subset of items should be human-annotated and agreement (e.g., Cohen's kappa) reported, or the affected claims should be explicitly downgraded.
  3. [Section 5.1 and Table 3] The correlation and range statistics are reported without a clear unit of analysis. With seven temperature points per model, p-values of exactly 0.00 are not meaningful unless the number of models, the aggregation procedure, and confidence intervals are provided. The 'Range Max (%)' metric is never defined in the text (relative to minimum? to a baseline?), and values such as 192.32% for MT-small are misleading without that definition. Please report the per-model temperature-response data, effect sizes with confidence intervals, and a precise definition of every derived statistic.
  4. [Sections 4.2.1 and 5.3.1] The selector's optimal temperature T* = argmax_T D(T, F(xp), M) is read off the same main-experiment performance distributions D that the paper uses to establish its temperature-effect claims; the SuperGLUE experiment is therefore not an independent validation of those response curves. In addition, the paper does not report the BERT classifier's accuracy on ability prediction, nor confidence intervals or significance tests for the ACCB/ACCC improvements (e.g., WIC improving from 0.196 to 0.500 for Llama-3.2-1B). A held-out ability-classification test and comparisons against baselines (e.g., random temperature, or the best single temperature for each ability) are needed.
  5. [Section 5.1] The claim that 'larger models are more robust to temperature-induced variations' is inferred from decreasing Range Max and CV Max across size groups, but no statistical test (e.g., a mixed-effects model with a temperature-by-size interaction, or a bootstrap over models) is provided. Given that the number of models per size band is small (3–5), this inference needs direct support rather than descriptive statistics alone.
minor comments (5)
  1. [Section 5.1 and Table 3] The table is not readable as typeset: rows run together (e.g., '0.41 0.520.820.05') and column headers do not align with the numeric columns; please re-typeset with clear column separation.
  2. [Section 3.1 and Table 1] 'Maltesian' should be 'Maltese', and 'Flore101' should be 'FLORES-101' for consistency with the cited benchmark.
  3. [Table 2] Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct are dated 'Sep 2025', which is inconsistent with the paper's timeline and the known release dates; this appears to be a typo and should be corrected.
  4. [Section 5.1] The text defines CV as 'the ratio of average performance to standard deviation', which is the inverse of the standard coefficient of variation (standard deviation divided by mean); please correct the definition.
  5. [Section 5.2] The statement that all metrics are 'uniformly referred to as accuracy' conflicts with the metric names in Table 1 (spBLEU, Rouge-L, DRFR); please clarify how the values in Figure 2 are normalized and whether the original scales are preserved.

Circularity Check

1 steps flagged · score 2.0 of 10

One tautological renaming (mutation-temperature scaling) but the central empirical derivation and selector transfer are self-contained.

  1. renaming known result [Abstract; §5.3.2; §6]
    "By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature — the point at which significant performance changes occur — increases with model size. (Abstract) ... Large models are more robust to increasing temperatures, while small models experience significant performance loss at lower temperatures, which aligns with our expectations. In particular, each model has its own mutation temperature, with larger models generally exhibiting a higher threshold. (§5.3.2)"

    The 'mutation temperature' is defined as 'the point at which significant performance changes occur.' The claim that this point is higher for larger models is a direct restatement of the immediately preceding observation that 'large models are more robust to increasing temperatures, while small models experience significant performance loss at lower temperatures.' No independent operational definition or measurement of 'mutation temperature' is supplied, and §4.2.2 states 'we did not find evidence for a specific Mutation Temperature in large models.' Thus the abstract's scaling claim does not add information beyond the robustness observation it is derived from; it is a re-labeling of the same curve shape rather than an independent derivation.

full rationale

The temperature selector's T* = argmax_T D(T, F(xp), M) is built from the paper's own measured performance curves, but the SuperGLUE evaluations in Table 4 are out-of-sample prompts not used in D, so the reported accuracy gains (e.g., Llama-3.2-1B WIC 0.196 to 0.500) are not forced by construction. The correlation findings in Table 3 and the FP16-versus-4-bit comparisons are direct empirical summaries of experiments, with no fitted parameter renamed as a prediction. The only near-circular element is the mutation-temperature scaling claim, which restates the observation that larger models are more robust and is internally contradicted by §4.2.2 ('we did not find evidence for a specific Mutation Temperature in large models'). That contradiction and the missing operational definition are correctness/reproducibility concerns, not a case where an equation reduces to its own input. No load-bearing self-citations are present, and the reviewer's concern about the GPT-3.5 judge is a validity risk, not a circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The main empirical claims rest on the validity of GPT-3.5 as a judge, the ability-specificity of the chosen datasets, the sufficiency of three repetitions, and the equivalence of 4-bit and FP16 inference. The selector adds a free parameter in the form of per-ability optimal temperatures derived from the authors' own curves, and the mutation temperature is an under-defined, hand-set breakpoint. No new physical or mathematical entities are introduced beyond the Mutation Temperature label.

free parameters (2)
  • Per-ability optimal temperature schedules used by the selector = Derived from D(T, F(xp), M), e.g., CR/CT optimum near 1.3, MT/SUMM near 0
    T* = argmax_T D is computed from the authors' own measured performance curves, so the 'optimal' values are fitted to the same experimental data that motivates them.
  • Mutation temperature thresholds per ability and model size = IF: 1.0-1.3 (small), 1.3-1.6 (medium), 1.6-1.9 (large); SUMM medium near 1.7; large often none below 4.0
    No formal change-point model is defined; the thresholds are read off the curves in Section 5.2 and Figure 3, so they are effectively hand-chosen breakpoints.
assumptions (5)
  • domain assumption GPT-3.5-turbo judgments are a valid proxy for human evaluation of causal reasoning, creativity, and instruction following.
    Section 3.2 invokes LLM-as-a-Judge alignment with human judgment, but Section 6 admits the judgments were not manually verified. This assumption underpins three of the six ability curves.
  • domain assumption The selected datasets measure the intended six abilities and a single-prompt, single-query protocol isolates each ability.
    Section 3.1 assigns one dataset per ability and argues that a single-prompt format avoids multi-prompt assistance, but there is no independent validation that each dataset isolates the target ability.
  • domain assumption Three repetitions per question provide stable estimates of generation stochasticity.
    Section 4.1 states that each question is tested three times across 12 models, but no variance analysis or confidence intervals are shown for the resulting metrics.
  • domain assumption AWQ 4-bit quantization preserves the temperature effects present in full precision.
    Section 4.1 quantizes all models to 4 bits with AWQ. The FP16 comparison in Section 5.3.2 covers only three models, so the assumption is not fully validated across the model zoo.
  • domain assumption Nucleus sampling with Top-P=0.9, repetition penalty 1.0, and max_new_tokens=1024 is a neutral baseline; varying these parameters does not change the main temperature conclusions.
    Section 4.2.2 and Table 5 show high correlations between temperature curves under most parameter settings, but creativity and instruction following show notable sensitivity to Top-P and repetition penalty, so the baseline is not completely neutral.
invented entities (1)
  • Mutation Temperature
    purpose: A name for the temperature at which a task's performance drops sharply; used to state the main size-scaling finding that the threshold increases with model size.
    The paper provides no operational definition or independent measurement of this threshold. It is inferred post hoc from the authors' own performance curves, and the paper itself is inconsistent about whether large models have one below 4.0.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Impact of Temperature on Large Language Models:Hot or Cold?." pith.science (2026). https://pith.science/paper/JPPK7COZ

@misc{pith2026250607295,
  author       = {Pith},
  title        = {Pith review of: Exploring the Impact of Temperature on Large Language Models:Hot or Cold?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPPK7COZ}},
  note         = {Machine review of arXiv:2506.07295}
}
read the original abstract

The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperature, plays a crucial role in model inference. In this study, we systematically evaluated the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities, conducting statistical analyses on open source models of three different sizes: small (1B--4B), medium (6B--13B), and large (40B--80B). Our findings reveal distinct skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection in practical applications. To address this challenge, we propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt. We demonstrate that this approach can significantly improve the performance of small and medium models in the SuperGLUE datasets. Furthermore, our study extends to FP16 precision inference, revealing that temperature effects are consistent with those observed in 4-bit quantized models. By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature -- the point at which significant performance changes occur -- increases with model size.

Figures

Figures reproduced from arXiv: 2506.07295 by the authors.

Figure 1
Figure 1. Cosine Similarity Using BERT Embedding Model [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Average performance trends for different model sizes, with shaded bands indicating variability. not necessarily reduce performance. In contrast, small models do not demonstrate substantial causal reasoning ability within the scope of this study. Creativity (CT). An optimal temperature of 1.3 is recommended for medium and large models to maximize creativity. Small models show a marked decline in creativity at T = 1.0… view at source ↗
Figure 3
Figure 3. Performance with extended temperature to 4.0 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 2 citations worldwide. Full citation record

  1. How Small Transformation Expose the Weakness of Semantic Similarity Measures

    cs.CL 2025-09 reject novelty 4.0 of 10

    A diagnostic benchmark of text and code transformations finds embedding similarity metrics often conflate opposition with equivalence; LLM judges discriminate better, and Euclidean distance improves code embeddings.

Reference graph

Works this paper leans on

30 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    A learning algorithm for boltzmann machines

    Ackley, D.H., Hinton, G.E., Sejnowski, T.J., 1985. A learning algorithm for boltzmann machines. Cognitive Science 9, 147–

    work page 1985

  2. [2]

    This is not correct! negation-aware evaluation of language generation systems, in: Keet, C.M., Lee, H.Y ., Zarrieß, S

    Anschütz, M., Miguel Lozano, D., Groh, G., 2023. This is not correct! negation-aware evaluation of language generation systems, in: Keet, C.M., Lee, H.Y ., Zarrieß, S. (Eds.), Proceedings of the 16th International Natural Language Generation Conference, Association for Computa- tional Linguistics, Prague, Czechia. pp. 163–175. URL: https://aclanthology.or...

    doi:10.18653/v1/2023 2023

  3. [3]

    LongBench: A bilingual, multitask benchmark for long context understanding, in: Ku, L.W., Martins, A., Srikumar, V

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., Li, J., 2024. LongBench: A bilingual, multitask benchmark for long context understanding, in: Ku, L.W., Martins, A., Srikumar, V . (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Lo...

    2024

  4. [4]

    Deepseek llm: Scaling open-source language models with longtermism

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al., 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954

    arXiv 2024

  5. [5]

    Language models are few-shot learners, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY , USA

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A...

    work page 2020

  6. [6]

    Language gans falling short

    Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., Charlin, L., 2018. Language gans falling short. CoRR abs /1811.02549. URL: http://arxiv.org/abs/1811.02549, arXiv:1811.02549

    arXiv 2018

  7. [7]

    Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., Wu, C.S., 2024. Art or artifice? large language models and the false promise of creativity, in: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY , USA. URL:https://doi.org/10.1145/3613904.3642731, doi:10.1145/3613904.3642731

    arXiv 2024

  8. [8]

    Kl-divergence guided temperature sampling

    Chang, C.C., Reitter, D., Aksitov, R., Sung, Y .H., 2023. Kl-divergence guided temperature sampling. arXiv:2306.01286

    arXiv 2023

Show all 30 references
  1. [9]

    Hierarchical neural story generation, in: Gurevych, I., Miyao, Y

    Fan, A., Lewis, M., Dauphin, Y ., 2018. Hierarchical neural story generation, in: Gurevych, I., Miyao, Y . (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), Association for Computational Linguistics, Melbo...

    2018 doi

  2. [10]

    Frohberg, J., Binder, F., 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models, in: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk,...

    2025

  3. [11]

    The Flores-101 evaluation benchmark for low-resource and multilingual machine translation

    Goyal, N., Gao, C., Chaudhary, V ., Chen, P.J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., Fan, A., 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics 10...

    2022 doi

  4. [12]

    The originality of machines: Ai takes the torrance test

    Guzik, E.E., Byrge, C., Gilde, C., 2023. The originality of machines: Ai takes the torrance test. Journal of Creativity 33, 100065. URL: https://www.sciencedirect.com/science/article/pii/S2713374523000249, doi:https://doi.org/10.1016/j.yjoc.2023. 100065

    2023 doi

  5. [13]

    The curious case of neural text degeneration, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net

    Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y ., 2020. The curious case of neural text degeneration, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net. URL: https://openreview.net/ forum?id=r...

    2020

  6. [14]

    Causal reasoning and large language models: Opening a new frontier for causal- ity

    Kiciman, E., Ness, R.O., Sharma, A., Tan, C., 2024. Causal reasoning and large language models: Opening a new frontier for causal- ity. Transactions on Machine Learning Research (TMLR) URL: https://www.microsoft.com/en-us/research/publication/ causal-reasoning-and-large-langua...

    2024

  7. [15]

    E fficient memory management for large language model serving with pagedattention, in: Proceedings of the 29th Symposium on Operating Systems Principles, pp

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y ., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I., 2023. E fficient memory management for large language model serving with pagedattention, in: Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626

    2023

  8. [16]

    Lin, C.Y ., Och, F.J., 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics, in: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain. pp. 605–612. UR...

    2004

  9. [17]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., Han, S., 2025. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. GetMobile: Mobile Comp. and Comm. 28, 12–17. URL: https://doi.org/10.1145/3714983.3714987, doi:10.1145/ 3714983.3714987

    2025

  10. [18]

    G-eval: NLG evaluation using gpt-4 with better human alignment, in: Bouamor, H., Pino, J., Bali, K

    Liu, Y ., Iter, D., Xu, Y ., Wang, S., Xu, R., Zhu, C., 2023. G-eval: NLG evaluation using gpt-4 with better human alignment, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Com...

    2023

  11. [19]

    Is temperature the creativity parameter of large language mod- els?, in: Grace, K., Llano, M.T., Martins, P., Hedblom, M.M

    Peeperkorn, M., Kouwenhoven, T., Brown, D., Jordanous, A., 2024. Is temperature the creativity parameter of large language mod- els?, in: Grace, K., Llano, M.T., Martins, P., Hedblom, M.M. (Eds.), Proceedings of the 15th International Conference on Computational Creativity, IC...

    2024

  12. [20]

    InFoBench: Evaluating instruction following ability in large language models, in: Ku, L.W., Martins, A., Srikumar, V

    Qin, Y ., Song, K., Hu, Y ., Yao, W., Cho, S., Wang, X., Wu, X., Liu, F., Liu, P., Yu, D., 2024. InFoBench: Evaluating instruction following ability in large language models, in: Ku, L.W., Martins, A., Srikumar, V . (Eds.), Findings of the Association for Computational Linguis...

    2024 doi

  13. [21]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., 2018. Improving language understanding by generative pre-training. URL: https://api.semanticscholar. org/CorpusID:49313245

    2018

  14. [22]

    Renze, M., 2024. The e ffect of sampling temperature on problem solving in large language models, in: Findings of the Association for Compu- tational Linguistics: EMNLP 2024, Association for Computational Linguistics. p. 7346–7356. URL: http://dx.doi.org/10.18653/v1/ 2024.find...

    2024 doi

  15. [23]

    Creativity research: Originality, utility, and integration

    Runco, M., 1988. Creativity research: Originality, utility, and integration. Creativity Research Journal - CREATIVITY RES J 1, 1–7. doi: 10. 1080/10400418809534283

    1988

  16. [24]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  17. [25]

    Superglue: A stickier benchmark for general-purpose language understanding systems, in: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R

    Wang, A., Pruksachatkun, Y ., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S., 2019. Superglue: A stickier benchmark for general-purpose language understanding systems, in: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (...

    2019

  18. [26]

    Benchmarking large language models for news summa- rization

    Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., Hashimoto, T.B., 2024. Benchmarking large language models for news summa- rization. Transactions of the Association for Computational Linguistics 12, 39–57. URL: https://aclanthology.org/2024.tacl-1.3/, doi:10.1162/tac...

    2024 doi

  19. [27]

    A survey of large language models.arXiv:2303.18223

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., Du, Y ., Yang, C., Chen, Y ., Chen, Z., Jiang, J., Ren, R., Li, Y ., Tang, X., Liu, Z., Liu, P., Nie, J.Y ., Wen, J.R., 2023. A survey of large language models.arXiv:2303.18223

    2023 arXiv

  20. [28]

    AGIEval: A human-centric benchmark for evaluating foundation models, in: Duh, K., Gomez, H., Bethard, S

    Zhong, W., Cui, R., Guo, Y ., Liang, Y ., Lu, S., Wang, Y ., Saied, A., Chen, W., Duan, N., 2024. AGIEval: A human-centric benchmark for evaluating foundation models, in: Duh, K., Gomez, H., Bethard, S. (Eds.), Findings of the Association for Computational Linguistics: NAACL 2...

    2024 doi

  21. [29]

    Zhu, Y ., Li, J., Li, G., Zhao, Y ., Li, J., Jin, Z., Mei, H., 2024. Hot or cold? adaptive temperature sampling for code generation with large language models, in: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innova...

    2024 doi

  22. [169]

    URL: https://www.sciencedirect.com/science/article/pii/S0364021385800124, doi: https://doi.org/10.1016/ S0364-0213(85)80012-4

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.