{"id":"325b36a7-9494-42a1-a24e-ee3d14d6886c","arxiv_id":"2501.19317","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Quantizing LLMs from 32 to 8 bits cuts GPU memory by about 76%, shifts affective-generation F1 by up to 10 percentage points in either direction, and lets larger quantized models rival smaller full-precision models in text quality.","lead":"This paper measures how quantizing five open-weight language models to 8, 16, or 32 bits affects GPU memory, inference speed, and the emotional quality of generated text. It reports about 76% memory savings with mixed F1 changes, and suggests that larger 8-bit models can match smaller full-precision models in affective text quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Emotion-classifier F1 values are internally inconsistent (Table 1 Joy F1=0.98 impossible from P=0.91/R=0.92), undermining the F1-based trade-off that is the paper's central quantitative claim.","rationale":"The reader identified the emotion-classifier-as-ground-truth assumption as the weakest point; I agree, and I add a concrete internal inconsistency that makes the concern more acute: Table 1's Joy F1=0.98 is arithmetically impossible given the reported precision and recall. This is a verifiable error in the evaluation metric, and because all F1 scores in Table 2 are produced by the same classifier, the central quantitative claims about quantization-induced F1 changes are not trustworthy without correction or independent validation. The memory-saving figure (76%) is a hardware measurement and is not affected, but the paper's main argument about the 'trade-off' depends on the F1 comparisons. The other issues the reader raised (inference-time inconsistency, missing code/data) are also valid but secondary to the validity of the performance metric. Since the reader already assigned CONDITIONAL, my concern reinforces that judgment rather than moving it; the paper can be accepted only if the classifier evaluation is fixed and the F1 results are confirmed with an independent measure. Therefore the verdict remains CONDITIONAL (no change).","tokens_in":9580,"tokens_out":2940,"duration_ms":26429,"concrete_test":"First, recompute Table 1's Joy F1 from the reported precision (0.91) and recall (0.92). If it is not 0.98, the classifier evaluation pipeline is confirmed buggy. Second, independently re-annotate a random sample of generated sentences (e.g., 100 per configuration) with human raters or with a second, externally validated emotion classifier, and recompute the Table 2 F1 deltas (especially Llama-2-7b 32-bit vs 8-bit and Llama-2-70b 8-bit vs 32-bit). If the direction and magnitude of the 10 pp changes do not replicate, the central trade-off claim collapses to the memory-saving result alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central trade-off claim is quantified almost entirely by F1 scores from a RoBERTa emotion classifier fine-tuned on ISEAR (Section 2.4, Table 2). That classifier's own reported metrics in Table 1 contain an arithmetically impossible value: for Joy, precision=0.91, recall=0.92, but F1=0.98, whereas F1 = 2*P*R/(P+R) = 0.915. This is not rounding noise; it is a 0.065 discrepancy. It indicates either a bug in the evaluation script or an error in reporting. Since the same pipeline computes every F1 in Table 2, all performance comparisons — including the claimed 'decrease of up to 10 pp for larger models' and 'increase of 10 pp for smaller models' — inherit this unreliability. Additionally, Section 2.5 evaluates only generic text quality dimensions (fluency, coherence, etc.) with GPT-4o/GPT-3.5; there is no human or independent validation of whether the generated text actually expresses the target emotion. The classifier's own macro-F1 is only 0.75, with Shame at 0.52, so emotion detection is already weak for some categories. If the classifier has systematic biases — e.g., favoring certain lexical patterns produced by larger or smaller models, or by different quantization levels — the reported F1 differences and the 'larger models at lower quantization outperform smaller, higher-precision models' conclusion could be artifacts. This is the most load-bearing assumption because the memory-saving result (76% reduction) is a straightforward hardware measurement, but the performance trade-off, which determines whether the memory saving is worthwhile, rests entirely on these questionable F1 numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how post-training quantization (8, 16, and 32 bits) affects affective text generation in five open-weight LLMs (Llama-2 7B/13B/70B, Mistral-7B, Mixtral-8x7B). Ten seed prompts are combined with seven emotion categories to generate 210 sentences per model-precision configuration, which are scored by a RoBERTa emotion classifier fine-tuned on ISEAR for whether they express the target emotion (F1), and by GPT-4o and GPT-3.5 judges on five text-quality dimensions. The central claim is that quantizing to 8 bits reduces GPU memory by about 76%, costs up to 10 percentage points of F1 for larger models, gains up to 10 points for smaller models, roughly doubles inference time, and yields text quality in which larger models at lower quantization generally outperform smaller, higher-precision models at similar memory.","tokens_in":9923,"tokens_out":5138,"duration_ms":46202,"significance":"If the findings are correct, the paper offers a practical, actionable trade-off map for deploying LLM-based affective generation on memory-constrained hardware, a question that is relevant and underexplored. The paper has concrete strengths: it systematically covers 15 model-precision configurations, reports hardware measurements (GPU RAM, inference time) that are directly useful, provides a clear and reproducible-looking protocol, and makes code available in principle. The main quantitative claims, however, rest on an emotion classifier whose reported evaluation contains an internal inconsistency, and the F1 comparisons lack any measures of uncertainty. The text-quality conclusions also rely on LLM judges without human validation and are partly contradicted by the paper's own tables. The topic is timely, but the evidence as presented does not yet support the headline trade-off claims.","major_comments":[{"comment":"Table 1 reports for the Joy class a precision of 0.91 and a recall of 0.92, but an F1 score of 0.98. The harmonic mean is 2*0.91*0.92/(0.91+0.92) = 0.915, which is 0.065 away from the reported value and cannot be explained by rounding. Because this classifier computes every F1 score in Table 2, the central quantitative claim about up to ±10 pp changes in F1 inherits this unreliability. The authors must correct the classifier evaluation (or the reporting error) and recompute all downstream F1 values before the trade-off claim can be assessed.","section":"Table 1, §2.4"},{"comment":"The F1 differences that drive the main conclusion — e.g., Llama-2-7b 32-bit at .63 vs 8-bit at .73, or Llama-2-70b 32-bit at .74 vs 8-bit at .64 — are reported as point estimates only. Each cell is based on 210 generated sentences across 7 emotion classes and 10 seed prompts, yet no variance, confidence interval, or significance test is given. With this sample size, differences on the order of 0.10 could easily arise from sampling noise. The paper should provide per-emotion F1 breakdowns, bootstrap intervals, or a significance test (e.g., paired across prompts) to support the claimed 10 pp increases and decreases.","section":"Table 2, §3.1"},{"comment":"The claim that 'generating an instance with 8-bit quantization takes about twice as long as with 32-bit quantization' is not supported by the paper's own data. The 8-bit to 32-bit inference time ratios in Table 2 are 1.32 for Llama-2-7b, 1.39 for Llama-2-13b, 0.67 for Llama-2-70b, 2.11 for Mistral-7B, and 2.33 for Mixtral-8x7B. Similarly, the statement in §3.4 that 'as quantization increases, inference time also increases' is contradicted by Llama-2-70b, whose 8-bit runtime (34.93 s) is lower than its 32-bit runtime (51.93 s). This trade-off component needs to be re-analyzed and the claims revised or qualified.","section":"§3.4, Table 2"},{"comment":"The conclusion that 'larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory' is not uniformly supported by Table 3. Mistral-7B at any quantization level scores higher than Llama-2-70B on all five reported quality dimensions (e.g., fluency 4.6 vs 3.6 at 8 bits), so the 'generally' claim requires a caveat or a more careful ranking analysis. In addition, the emotion-correctness evaluation is entirely delegated to an automatic classifier with macro-F1 0.75 and a Shame F1 of only 0.52; no human evaluation is reported to confirm that the generated text actually expresses the requested emotion. The F1-based trade-off could be an artifact of classifier bias correlated with model size or precision. The authors should either add a human validation sample or explicitly limit the claim to classifier-measured emotion fidelity.","section":"§3.2, Table 3"}],"minor_comments":[{"comment":"The text refers to 'Table ??' when discussing the GPT-3.5 evaluation; this placeholder should be replaced with a reference to Table 4.","section":"Appendix A"},{"comment":"Section 2.5 lists six text-quality dimensions, including Human Likeness, but Tables 3 and 4 report only five columns (Fluency, Native Speaker Likeness, Coherency, Plausibility, Grammar) and omit Human Likeness. Please align the text with the tables or add the missing dimension.","section":"§2.5, Tables 3 and 4"},{"comment":"The sentence 'The inference tasks for the LLMs models are performed and evenly distributed across a GPU cluster equipped with 8 NVIDIA RTX A6000 GPUs, for 10 times per model configuration' is ambiguous: it is unclear whether the entire generation is repeated 10 times per model configuration and, if so, how the 210-sentence counts and the reported F1 values aggregate these repetitions.","section":"§2.3"},{"comment":"The claim that quantization reduces performance 'by an average of 3.6 pp across all configurations' is not derivable from Table 2. For example, the mean F1 across the five models is 0.71 at 8 bits, 0.70 at 16 bits, and 0.69 at 32 bits, which gives an average drop of roughly 1.6 pp from 8 to 32 bits and less from 16 to 32 bits. Please specify the baseline and computation for the 3.6 pp figure.","section":"§3.1"},{"comment":"The macro-averaged precision, recall, and F1 are all reported as .75, which is consistent with rounding, but the per-class F1 values should be recomputed after correcting the Joy F1 and the table regenerated with consistent two-decimal rounding.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical question is legitimate and the hardware measurements are a useful contribution, but the paper currently reads more like an extended abstract than a complete study: the evaluation relies on a single, weakly validated emotion classifier, the reported F1 table contains an arithmetically impossible value, and the inference-time claim is contradicted by the experimental data. I would not invite a resubmission without the authors addressing the classifier inconsistency and providing uncertainty estimates for the F1 comparisons. The self-citations are appropriate given the authors' prior work in affective generation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the memory-saving half of this paper is solid and useful, but the F1-based performance half has an arithmetic error in Table 1 that needs a public fix before the trade-off claim is citable. The 76% GPU reduction is a clean hardware measurement; the F1 comparisons are not yet trustworthy.\n\nWhat's new here: a systematic 8/16/32-bit comparison for affective text generation across Llama-2 7B/13B/70B, Mistral-7B, and Mixtral-8x7B. I don't know of another paper that does exactly this. The qualitative examples in Appendix B support the main intuition: an 8-bit 70B model can produce richer, more coherent emotional text than a 32-bit 7B model while using similar GPU RAM. That's a practically useful observation for anyone deploying these systems.\n\nSoft spots, in rough order of severity:\n\n1. Table 1 reports Joy precision 0.91, recall 0.92, F1 0.98. The harmonic mean is 0.915. That is not rounding noise; it's a 6.5-point discrepancy. Since the same classifier computed every F1 in Table 2, all the ±10pp conclusions inherit this unreliability. It could be a typo in the table, but the paper has to say which and show corrected numbers.\n\n2. Section 3.4 claims 8-bit inference takes about twice as long as 32-bit. Their own Table 2 contradicts this for Llama-2-70B (34.9s at 8-bit vs 51.9s at 32-bit, so 8-bit is faster), and for the 13B model the ratio is 1.4x, not 2x. The claim is an overgeneralization.\n\n3. The \"increase of 10pp for smaller models\" only holds for Llama-2-7B. Mistral-7B is essentially flat (0.73 vs 0.74). With no variance or significance tests, these are single point estimates; differences like 0.73 vs 0.74 are likely noise.\n\n4. No code or data is released for the experiments. Nothing is independently reproducible, and quantization details (the library, the exact method) are not given.\n\nI share the worry about the emotion classifier as ground truth: an ISEAR-fine-tuned RoBERTa with macro-F1 0.75 and Shame F1 0.52 is not a human judge. A small human evaluation on a few configurations would have addressed this directly.\n\nWho is this for: practitioners wanting a deployment rule for emotion-conditioned generation under memory limits. The paper is clearly written, and the limitations section is honest. It deserves a serious referee: the question is real and the measurement, once cleaned up, would be a useful data point. But the current version needs a major revision—fix the arithmetic, reconcile the inference-time claim with Table 2, report variance, and release code and data. If those are fixed, I'd be happy to cite it for the memory-saving result.","headline":"A useful memory-quantization measurement whose F1-based trade-off is compromised by an impossible F1 value and an inference-time claim contradicted by its own table; deserving of a major-revision review.","tokens_in":10486,"tokens_out":4546,"would_cite":false,"duration_ms":40447,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantizing LLMs from 32 to 8 bits cuts GPU memory by about 76% for affective text generation, with larger quantized models often beating smaller full-precision ones at similar memory.","keywords":["quantization","affective text generation","emotion classification","GPU memory","inference time","large language models","Llama-2","Mistral"],"falsifier":"Have human annotators label whether each generated sentence expresses the target emotion, and compare the resulting emotion accuracy to the classifier's F1 across the 15 configurations; if the human-based ranking does not match the classifier's, the reported trade-off is at least partly an artifact of the classifier.","tokens_in":9348,"feed_emoji":"🧠","tokens_out":4892,"duration_ms":41793,"temperature":0.7,"pith_summary":"The paper asks what happens to affective text generation when large language models are quantized to 8, 16, or 32 bits. It finds that dropping from 32 to 8 bits cuts GPU memory usage by about 76% across five open-weight models, at the price of roughly doubling inference time. The emotion-classification F1 of the generated text falls by up to 10 percentage points for the largest model (Llama-2-70B) but can rise by up to 10 points for smaller models (Llama-2-7B). A key result is that larger models at 8 bits generally match or beat smaller models at 16 or 32 bits on both the emotion classifier and on LLM-based text-quality scores, while using about the same memory. This points toward a practical recipe for running capable affective text generation on memory-limited hardware: choose the largest model you can fit at 8 bits rather than a smaller model at full precision.","feed_headline":"8-bit LLMs cut memory 76% with small emotion-text trade-off","feed_subtitle":"Quantized Llama-70B beats full-precision 7B on emotion text at similar memory cost.","key_machinery":"The central object is the precision-bit setting (8, 16, or 32) applied to five open-weight LLMs, where quantization means reducing the number of bits used to store each model weight. The evaluation machinery is an emotion classifier (RoBERTa fine-tuned on the ISEAR dataset) that scores whether generated text expresses the target emotion, plus GPU RAM and inference-time measurements, plus GPT-4o and GPT-3.5 Likert ratings for text quality dimensions like coherence, fluency, and plausibility.","core_discovery":"The paper reports that lowering precision from 32 to 8 bits reduces GPU memory by roughly 76% across all tested configurations, while changing the F1 of an emotion classifier on generated text by up to −10 percentage points for the largest model (Llama-2-70B) and +10 points for smaller models (Llama-2-7B). At the same time, the larger models at 8 bits matched or beat smaller models at 16 or 32 bits in both the classifier's F1 and in GPT-4o text-quality ratings, despite using similar memory. That means an 8-bit Llama-2-70B can be a practical substitute for a 16- or 32-bit 7B or 13B model on memory-limited hardware, with the caveat that inference takes roughly twice as long.","pith_inferences":["If the emotion classifier systematically favors longer or more fluent outputs, the 'larger model at 8 bits beats smaller at full precision' result could be driven by text length or fluency rather than by emotion accuracy; a human study that controls for length would settle this.","The same memory-budget logic likely generalizes to other generation tasks such as summarization or dialogue, but the specific ±10 pp F1 range and the 76% memory figure are tied to this model set, this classifier, and these ten prompts.","A cheap extension would be to test whether the 'bigger and quantized beats smaller and precise' ordering holds when the emotion classifier is replaced by human evaluation on the same generated sentences; the paper's Limitation section effectively invites this check."],"forward_implications":["An 8-bit Llama-2-70B can deliver better emotion-text quality than a 16- or 32-bit 7B or 13B model while using roughly the same GPU memory, making it a practical choice on memory-limited GPUs.","Quantizing from 32 to 8 bits consistently saves about 76% of GPU RAM, but most configurations see inference time roughly double.","The effect of quantization on emotion correctness is not uniform: some small models improve in F1 after quantization, while large models can lose up to 10 percentage points.","Because the evaluation rests on a single emotion classifier and ten seed prompts, the reported F1 numbers are tied to that classifier's notion of 'expressing an emotion' and could shift with a different classifier or prompt set."],"supporting_citations":[{"why":"Supplies the Llama-2 family (7B, 13B, 70B) that serves as the primary testbed for quantization.","marker":"(Touvron et al., 2023)"},{"why":"Supplies Mistral-7B, one of the two model families tested.","marker":"(Jiang et al., 2023)"},{"why":"Supplies Mixtral-8x7B, the second model family's large variant.","marker":"(Jiang et al., 2024)"},{"why":"Provides the ISEAR dataset on which the emotion classifier is fine-tuned.","marker":"(Scherer and Wallbott, 1994)"},{"why":"Provides RoBERTa, the base model for the emotion classifier.","marker":"(Liu et al., 2019)"},{"why":"Justifies using LLMs like GPT-4 and GPT-3.5 for automatic text quality evaluation.","marker":"(Chiang and Lee, 2023)"},{"why":"Presents G-Eval, the basis for the GPT-based quality scoring used in the paper.","marker":"(Liu et al., 2023)"}],"fun_headline_variants":["Quantized 70B LLMs beat full-precision 7B on emotion text","8-bit LLMs: 76% less memory, mixed F1, but big models still win","Memory slashed 76% with 8-bit LLMs, larger models still best","8-bit quantization: 76% memory cut, F1 swings ±10 points","Same memory, better emotion text: 8-bit 70B beats 32-bit 7B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic emotion classifier used to score whether generated text expresses the target emotion is treated as ground truth, and if it is biased toward certain model sizes or precisions, the reported F1 trade-offs could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Quantized 70B LLMs beat full-precision 7B on emotion text","8-bit LLMs: 76% less memory, mixed F1, but big models still win","Memory slashed 76% with 8-bit LLMs, larger models still best","8-bit quantization: 76% memory cut, F1 swings ±10 points","Same memory, better emotion text: 8-bit 70B beats 32-bit 7B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000346,"raw_usage":{"total_tokens":1901,"prompt_tokens":952,"completion_tokens":949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":833}},"tokens_in":568,"tokens_out":949,"duration_ms":7193,"temperature":1.0,"reasoning_tokens":833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:36:51.017636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators label whether each generated sentence expresses the target emotion, and compare the resulting emotion accuracy to the classifier's F1 across the 15 configurations; if the human-based ranking does not match the classifier's, the reported trade-off is at least partly an artifact of the classifier.","supporting_citations":[],"review_version":1}