{"id":"c2d81774-4493-434c-a2a4-af37017236be","arxiv_id":"2507.09687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Generative LSTM classifiers under post-training quantization are far more sensitive than discriminative ones to calibration data class balance and input noise, especially at 3- to 5-bit widths.","lead":"This paper tests how post-training quantization affects two kinds of LSTM text classifiers, generative and discriminative, and finds that generative classifiers lose accuracy more quickly when the data used for calibration is not balanced across classes. The results are a practical warning for edge deployment: the robustness advantage of generative classifiers vanishes under low-bit quantization unless calibration data is chosen carefully.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The class-imbalance mechanism is confounded: on balanced AG News/DBPedia a 25% random calibration sample is nearly class-balanced, so the Section 5.3–5.4 gap cannot be attributed to class imbalance without reported class counts, seeds, and repeated runs.","rationale":"The reader's weakest assumption is also the most load-bearing concern: the headline mechanism depends on class-unconditional calibration being class-imbalanced, but random sampling from class-balanced datasets makes that assumption almost certainly false. This does not by itself disprove the paper's empirical observation that the two calibration procedures differ at low bitwidths, but it does undermine the stated causal explanation and the practical recommendation that class balance is the critical factor. The issue is addressable by reporting calibration set statistics, using multiple seeds, and rerunning with matched sample sizes. Because the paper's central mechanistic claim is currently unsupported rather than proven false, the existing CONDITIONAL verdict is appropriate; no change to the reader's verdict is needed.","tokens_in":18905,"tokens_out":7828,"duration_ms":93865,"concrete_test":"Run a factorial PTQ experiment on AG News and DBPedia with a fixed seed and identical total calibration size across conditions: (A) class-unconditional random 25% sample, reporting its actual class counts; (B) exactly class-balanced 25% sample matched in total size; (C) several additional random 25% draws with different seeds. Repeat each condition 10 times and report mean +/- std test accuracy per bitwidth. If (A) and (B) overlap at 3–6 bits, the class-imbalance mechanism is unsupported; if they differ, the remaining difference is attributable to seed or sample identity rather than class balance. Additionally, inspect the calibration data construction to verify that Sections 5.3 and 5.4 use the same total number of calibration samples; if class-conditional uses more samples, recompute with equal N.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that class-imbalanced calibration data causes insufficient weight adaptation and degraded accuracy for generative LSTM classifiers (Abstract; Section 6.1)—rests primarily on comparing class-unconditional random calibration (Section 5.3) with class-conditional stratified calibration (Section 5.4). AG News and DBPedia training sets are class-balanced, so a 25% random sample of the training set should have class proportions very close to the population. Concretely, for AG News the 25% calibration set has roughly 24k samples, expected 6k per class with binomial standard deviation around 67, i.e., class proportions near 24.7–25.3%; DBPedia is similarly near-balanced. Yet Figures 6 and 7 show a large accuracy gap, especially at 3–5 bits. If the class-unconditional sets are indeed nearly balanced, the observed gap cannot be a consequence of class imbalance. The paper never reports the empirical class distribution of the sampled calibration sets, does not state the random seed, and does not report repeated runs or error bars. The class-conditional protocol is also ambiguous: '25% from each class in a four-class classification task' could mean 25% of each class (same total as class-unconditional) or 25% of the calibration set per class, which would change total calibration size and confound the comparison. The controlled class-coverage experiment in Section 6.1 removes entire classes, which is qualitatively different from the mild imbalance expected from random sampling on balanced datasets; it cannot by itself rescue the mechanism at realistic imbalance levels. Finally, the KS statistics in Figures 12–14 are correlational: they show distributional shifts but do not establish that class imbalance caused those shifts or that insufficient weight adaptation caused the accuracy loss.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical study of post-training quantization (PTQ) for LSTM-based text classifiers, comparing a discriminative classifier (embedding, LSTM, linear layer, softmax) with a generative classifier that scores each class by the autoregressive likelihood of the input under a class-conditioned LSTM. The authors train both models on AG News and DBPedia, then quantize weights and activations to 3-8 bits using Brevitas with class-unconditional (random) or class-conditional (stratified) calibration and GPFQ refinement on the final linear layer. They measure test accuracy across bitwidths, character-level input noise at several rates, and internal weight/activation shifts via KS statistics. The main claims are that discriminative classifiers remain robust under PTQ while generative classifiers are more sensitive to bitwidth, calibration composition, and input noise, and that class-imbalanced calibration data causes 'insufficient weight adaptation' at low bitwidths in generative models.","tokens_in":19223,"tokens_out":6990,"duration_ms":72049,"significance":"If the central claim were established, the paper would offer actionable guidance for deploying generative LSTM classifiers on edge devices: use class-balanced calibration and avoid aggressive low-bit quantization. The study covers two standard datasets, multiple bitwidths, a controlled class-coverage experiment, and a systematic KS-statistic analysis, which are useful contributions. However, the main causal claim currently rests on a comparison that may be confounded by calibration-set size and by the fact that random sampling from balanced datasets produces nearly balanced calibration sets; the paper does not report the empirical class distribution, seeds, or repeated runs. The controlled class-coverage experiment (Section 6.1) supports a weaker claim that extreme class coverage matters, but not the specific mechanism invoked for the Section 5.3-5.4 gap.","major_comments":[{"comment":"The comparison that motivates the paper's central claim is confounded. AG News and DBPedia training sets are class-balanced, so a 25% random calibration sample is expected to be nearly class-balanced (for AG News, per-class proportions around 25% with a standard deviation below 1 percentage point on the calibration sample). The large accuracy gap between Figures 6 and 7 therefore cannot be attributed to class imbalance unless the authors report the actual class counts in the sampled calibration sets, the random seed, and repeated draws. Without this information, the observed gap may be caused by which particular samples were drawn or by other differences between the two calibration protocols. I ask the authors to report these details and to supplement the comparison with a matched experiment in which calibration sets have identical total size and controlled class proportions.","section":"Sections 5.3 and 5.4"},{"comment":"The definition of class-conditional calibration is ambiguous. '25% from each class in a four-class classification task' could mean 25% of each class (preserving the same total calibration size as the class-unconditional setting) or 25% of the calibration set per class (making the calibration set four times larger). These two readings change the calibration-set size, which is known to affect PTQ accuracy, and would confound the comparison in Figure 7. Please specify the sample count per class and total sample size for both calibration protocols.","section":"Section 5.4"},{"comment":"The noise injection procedure is not reproducible as stated. Section 4.3.3 says 'for each character in the input sentence, a substitution is performed with probability epsilon' but then says 'the replacement character is sampled uniformly from the model's vocabulary.' The vocabulary is word-level, so replacing a character with a vocabulary token is not defined; if word-level substitution was intended, the text should say so. In addition, the x-axes of Figures 9 and 10 label epsilon as 'Noise Level (Variance),' but epsilon is defined as a substitution probability. Please clarify the noise model and relabel the axes.","section":"Section 4.3.3 and Figures 9-10"},{"comment":"The KS-statistic argument for 'insufficient weight adaptation' does not establish the claimed mechanism. A smaller KS distance between quantized and full-precision weights at low bitwidths (Figure 12) is interpreted as insufficient adaptation, but it could equally indicate that GPFQ made smaller changes because the calibration data were less representative, or because the optimization objective on a skewed sample did not demand larger changes. The authors should connect the KS magnitudes to accuracy directly, e.g., by showing that the class-unconditional calibration data are actually class-imbalanced and that repeated sampling with matched calibration size reproduces the accuracy gap.","section":"Section 6.1.1"}],"minor_comments":[{"comment":"There are typos in the manuscript, including 'trianing' in Section 5.1, 'calibraiton' in Section 6.1, and 'calibtrated' in Section 6.1.1; these should be corrected.","section":"Section 5.1"},{"comment":"The caption says 'loss difference' but the figure appears to show KDEs of token-level loss; please clarify whether the plotted quantity is the loss itself or a difference relative to full precision.","section":"Figure 15"},{"comment":"The text states GPFQ is applied only to the final linear layer, but the general formulation in Section 4.2.3 describes arbitrary layers; it would help to state explicitly in Section 4.2.3 that the implementation restricts GPFQ to the final linear layer.","section":"Sections 4.2.3 and 4.3.2"},{"comment":"The KS statistics are reported without p-values or confidence intervals; the abstract's phrase 'test statistics derived from nonparametric hypothesis testing' suggests formal tests, but no test outcomes are reported.","section":"Figures 12-14"},{"comment":"The paper does not mention whether code or model configurations are available; given the number of implementation details, a reproducibility appendix would strengthen the paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the class-imbalance attribution. The authors can address it by reporting class counts, seeds, and a matched repeated-sampling experiment. If the gap disappears under matched total size, the paper's central claim would need to be substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper reports a large gap between class-unconditional and class-conditional calibration for generative LSTM classifiers under PTQ, and uses KS statistics to argue that class imbalance causes insufficient weight adaptation. The empirical trend is probably real: generative classifiers do seem more fragile at low bitwidths, and their full-precision noise robustness disappears after quantization. That part is worth taking seriously.\n\nWhat's genuinely new is the object of study—nobody has systematically compared generative vs. discriminative LSTM text classifiers under PTQ with different calibration compositions. The controlled class-coverage experiment in Section 6.1, where entire classes are removed from calibration, is a nice demonstration that class coverage matters for generative models. The noise robustness comparison across bitwidths is also useful. The paper uses public datasets and Brevitas, so the experiments are reproducible in principle.\n\nThe soft spot is load-bearing. AG News and DBPedia are class-balanced, and a 25% random sample of the training set will have class proportions within a fraction of a percent of the population. The paper never reports the actual class counts in the sampled calibration sets, nor seeds, nor repeated runs. So the large accuracy gap in Figures 6 and 7 cannot be attributed to class imbalance. The class-conditional protocol is also ambiguous: \"25% from each class\" could mean 25% of each class (same total size) or 25% of the calibration set per class (different total size), which would confound the comparison. The Section 6.1 experiment removes entire classes, which is a much stronger manipulation than the mild imbalance expected from random sampling on balanced datasets; it doesn't rescue the mechanism at realistic imbalance levels. The KS statistics are correlational—they show shifts but not that class imbalance caused them.\n\nThat said, the paper deserves a serious referee rather than a desk reject. The topic is relevant, the experiments are reproducible with public tools, and the empirical accuracy trends are plausible enough to warrant a careful revision. A revision that adds seeds, error bars, class counts for calibration sets, and a proper imbalance manipulation (e.g., artificially subsampling majority classes to varying degrees) could validate or refute the central claim. For my own work I wouldn't cite the mechanism yet, but I'd watch for a corrected version. Send it to peer review with the expectation of major revision.","headline":"The empirical trend is plausible, but the paper's central mechanism—class imbalance in calibration—is not actually tested, because a 25% random sample of balanced AG News/DBPedia is nearly class-balanced.","tokens_in":19775,"tokens_out":2682,"would_cite":false,"duration_ms":30997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Generative LSTM text classifiers lose far more accuracy than discriminative ones under post-training quantization, and only remain viable below 6 bits when the calibration data is class-balanced.","keywords":["post-training quantization","generative classifiers","discriminative classifiers","LSTM","text classification","calibration data","class imbalance","noise robustness"],"falsifier":"Measure the per-class composition of the 25 percent random calibration sets on AG News and DBPedia; if those sets are nearly class-balanced while the accuracy gap between class-conditional and class-unconditional calibration persists, the proposed imbalance mechanism is not the explanation, and the decisive control would be to draw multiple random calibration sets with matched class counts and show that accuracy tracks the class distribution rather than the seed.","tokens_in":18734,"feed_emoji":"⚖️","tokens_out":7437,"duration_ms":74638,"temperature":0.7,"pith_summary":"The paper tries to establish that generative LSTM text classifiers degrade far more than discriminative ones when post-training quantization (PTQ) is applied, and that the composition of the small calibration set used to set activation scales is a main driver of that degradation. On standard datasets, the generatively trained classifier falls sharply below 6-bit precision when calibration data is sampled without regard to class balance, while the discriminative classifier stays accurate down to 3 bits. The paper argues the mechanism is that class-imbalanced calibration yields skewed activation statistics, and the greedy refinement step then makes insufficient weight corrections in the final linear layer. If this is right, practitioners deploying generative classifiers on edge devices must use class-balanced calibration data, and the apparent robustness of generative classifiers to noise does not survive aggressive quantization. The finding matters because generative classifiers are otherwise attractive for noisy, safety-sensitive edge settings.","feed_headline":"Imbalanced calibration data breaks quantized generative LSTMs","feed_subtitle":"Discriminative LSTMs stay robust down to 3 bits; generative ones need class-balanced calibration to survive.","key_machinery":"The load-bearing mechanism is the calibration step of PTQ followed by Greedy Path-Following Quantization (GPFQ), a data-dependent post-training refinement that adjusts rows of a weight matrix to minimize the forward-pass error induced by quantization on a batch of calibration inputs. In this study GPFQ is applied only to the final linear layer, since the quantization library does not support it for the recurrent LSTM layers. The paper's diagnostic tool is the Kolmogorov–Smirnov (KS) statistic, a nonparametric measure of the largest distance between two distributions; it is used to compare weight distributions before and after calibration and to compare activation distributions produced by class-conditional versus class-unconditional calibration data. The argument proceeds by showing that unbalanced calibration shifts activation statistics relative to balanced calibration, and that those skewed statistics lead GPFQ to make smaller corrective adjustments at low bitwidths, leaving quantized weights misaligned with full-precision behavior.","core_discovery":"The paper's central claim is that discriminative LSTM text classifiers remain robust under post-training quantization while generative LSTM classifiers are markedly more sensitive, with sensitivity appearing as a function of three things: bitwidth, the class composition of the calibration data, and input noise during quantized inference. Using a controlled comparison of class-conditional and class-unconditional calibration, the authors show that the generative classifier's accuracy collapses below 6 bits precisely when calibration is unbalanced, and that balancing the calibration set restores most of the lost accuracy, even reversing the ordering so that the generative model beats the discriminative one at several bitwidths. The authors attribute the failure to insufficient weight adaptation in the final linear layer during greedy path-following quantization, a consequence of activation statistics that do not represent all classes. They further show that the generative classifier's full-precision robustness to character-level noise erodes after quantization, especially at 3 and 4 bits.","pith_inferences":["The paper never reports the empirical class distribution of the 'class-unconditional' 25 percent sample; because AG News and DBPedia training sets are class-balanced, a random sample should be nearly balanced, so the observed gap may partly reflect seed or sample-selection effects rather than class imbalance per se.","Because GPFQ is applied only to the final linear layer, the demonstrated 'insufficient weight adaptation' concerns that layer; whether LSTM gate and cell weights would exhibit the same imbalance sensitivity is untested and is the most direct next experiment.","The results suggest a cheap deployment rule: when labels are unavailable, approximating class balance in calibration data with unsupervised clustering or class-prior estimates may recover much of the generative classifier's low-bitwidth accuracy.","The model comparison is limited to a single 100-hidden-unit LSTM configuration; the same calibration-sensitivity pattern may or may not transfer to transformer text classifiers, which is where edge NLP is moving."],"forward_implications":["Class-balanced calibration data should be treated as a requirement, not an option, for generative LSTM classifiers deployed below 6-bit precision.","Discriminative LSTM classifiers are comparatively indifferent to calibration class composition, so existing PTQ pipelines that ignore class balance are safer for them.","Full-precision robustness to noisy input is not preserved under quantization for generative classifiers; 3-bit and 4-bit generative models lose their noise advantage over discriminative ones.","At several bitwidths with class-conditional calibration, the generative classifier outperforms the discriminative one, so the two model families' rankings can flip depending on calibration quality.","Training-time noise injection can partially restore low-bitwidth robustness, suggesting robustness and quantization decisions interact."],"supporting_citations":[{"why":"supplies the generative and discriminative LSTM architectures and training splits that all experiments build on","marker":"[44]"},{"why":"defines the greedy path-following quantization algorithm whose weight adjustments the paper measures","marker":"[24]"},{"why":"establishes that calibration data composition matters in PTQ and justifies the 25 percent calibration set size","marker":"[21]"},{"why":"the quantization toolchain used for weight and activation quantization, calibration, and GPFQ","marker":"[8]"},{"why":"prior evidence that generative classifiers are robust to noisy labels, the claim the paper tests under quantization","marker":"[19]"},{"why":"supplies the class-conditioned sequence-generation formulation used for generative classification","marker":"[5]"},{"why":"defines the Kolmogorov–Smirnov statistic used to quantify weight and activation shifts","marker":"[16]"},{"why":"extends the KS framework with the goodness-of-fit tables used in the distribution comparisons","marker":"[36]"}],"fun_headline_variants":["Imbalanced calibration breaks quantized generative LSTM classifiers","Balanced calibration data rescues quantized generative LSTMs","Class imbalance in calibration dooms quantized generative LSTMs","Quantized generative LSTMs need balanced calibration data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that class imbalance in calibration causes the generative classifier's collapse assumes that the random 'class-unconditional' calibration set is actually class-imbalanced, but the paper never reports its empirical class distribution, so the effect could be driven by which particular samples were drawn rather than by class balance.","fun_headline_variants_meta":{"raw":{"variants":["Imbalanced calibration breaks quantized generative LSTM classifiers","Balanced calibration data rescues quantized generative LSTMs","Class imbalance in calibration dooms quantized generative LSTMs","Quantized generative LSTMs need balanced calibration data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4439,"prompt_tokens":1024,"completion_tokens":3415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3351}},"tokens_in":640,"tokens_out":3415,"duration_ms":25503,"temperature":1.0,"reasoning_tokens":3351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:49:46.399926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the per-class composition of the 25 percent random calibration sets on AG News and DBPedia; if those sets are nearly class-balanced while the accuracy gap between class-conditional and class-unconditional calibration persists, the proposed imbalance mechanism is not the explanation, and the decisive control would be to draw multiple random calibration sets with matched class counts and show that accuracy tracks the class distribution rather than the seed.","supporting_citations":[{"cited_title":"A Greedy Algorithm for Quantizing Neural Networks","cited_arxiv_id":"2010.15979","evidence_quote":"defines the greedy path-following quantization algorithm whose weight adjustments the paper measures"},{"cited_title":"Robust inference via generative classifiers for handling noisy labels, in: Proceedings of the 36th International Conference on Machine Learning (ICML)","cited_arxiv_id":null,"evidence_quote":"prior evidence that generative classifiers are robust to noisy labels, the claim the paper tests under quantization"},{"cited_title":"Latent-Variable Generative Models for Data-Efficient Text Classification","cited_arxiv_id":"1910.00382","evidence_quote":"supplies the class-conditioned sequence-generation formulation used for generative classification"},{"cited_title":"Sulla determinazione empirica di una legge di distribuzione","cited_arxiv_id":null,"evidence_quote":"defines the Kolmogorov–Smirnov statistic used to quantify weight and activation shifts"},{"cited_title":"Table for estimating the goodness of fit of empirical distributions","cited_arxiv_id":null,"evidence_quote":"extends the KS framework with the goodness-of-fit tables used in the distribution comparisons"}],"review_version":1}