{"id":"ce5504b1-e670-4787-9b8c-b81eed93ee4a","arxiv_id":"2412.18135","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LSAQ assigns higher quantization precision to layers deemed important by the overlap of top-k input and output token sets, and reports small accuracy and perplexity gains over a cosine-similarity baseline.","lead":"The authors propose LSAQ, a system that quantizes different layers of a large language model to different precisions based on a Jaccard-similarity measure of layer importance, adapting to the GPU memory available on an edge device. The paper reports that this layer-specific scheme slightly outperforms a cosine-similarity-based layer-wise baseline on zero-shot tasks and WikiText-2 perplexity for Llama models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Jaccard token-set importance is not validated as a quantizability measure; small end-to-end gains do not establish the central claim.","rationale":"The reader identified the core weak assumption: token-set overlap may not reflect semantic transformation or quantization sensitivity. I agree and sharpen it: the metric's construction is also internally suspect because it uses the input embedding matrix rather than the actual output projection for Llama models, so the 'top-k token sets' are not the model's predicted tokens. The reader's verdict of CONDITIONAL already reflects that the evaluation is narrow and lacks calibration details; my concern is of the same kind but more specific. Since a concrete oracle test can settle whether the metric is actually useful, and the paper as written does not provide that evidence, the conditional verdict remains appropriate. I do not move to REJECT because the proposed test could validate the metric, and the idea is not internally inconsistent beyond the projection issue, which is fixable. The lack of code and error bars further supports keeping the conditional status rather than accepting the paper as-is.","tokens_in":12180,"tokens_out":7983,"duration_ms":77997,"concrete_test":"For Llama-2-7B, quantize each of the 32 layers to INT4 individually (all other layers kept in FP16) and measure the per-layer perplexity increase on WikiText2. Compute Spearman rank correlation between LSAQ's layer-importance scores I_i (Equation 3) and the per-layer PPL degradation, and likewise for cosine-similarity-based importance. If LSAQ's correlation is not positive or is not stronger than cosine's, the claim in Section IV.B.3 fails. As a secondary check, recompute the top-k sets using the model's actual lm_head weight matrix instead of W_E^T; if the layer ordering changes materially, the reported experiments rely on an incorrect vocabulary projection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central assertion (Section IV.B.3) is that Jaccard similarity between top-k token sets (Equation 3) is a more effective layer-importance metric for quantization than cosine similarity. This rests on two unverified conditions. First, the token sets must be the model's actual top-k predictions, but the paper computes them by projecting hidden states through W_E^T (Equations 1-2), the input embedding matrix. For Llama-2 and Llama-3, the output head is not weight-tied with the input embedding (tie_word_embeddings=false), so this projection does not yield the model's vocabulary distribution; the resulting sets are not 'top-k most probable tokens' as claimed. Second, even with a correct projection, Jaccard set overlap discards all probability information inside the overlapping set. A layer can reorder or sharpen probabilities among the same top-k tokens and still be critical for the final prediction; Equation 3 would incorrectly mark it as low-importance. The paper never validates the metric against a per-layer quantization-sensitivity oracle, and the reported improvements over LWQ are small (average accuracy differences of 0.2-1.5 points) with no error bars, repeated runs, or code. The end-to-end results therefore do not confirm that the Jaccard ranking is meaningfully better than cosine or even than a random layer ordering.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LSAQ, a layer-specific adaptive quantization system for LLMs. Layer importance is estimated by projecting each layer's input and output hidden states through the input embedding matrix, taking the top-k tokens, and computing I_i = 1 - Jaccard(C_{i,in}, C_{i,out}); layers with higher importance receive higher precision. A resource-detection module and an allocation algorithm (Algorithm 1) then assign FP16, INT8, or INT4 per layer according to available GPU memory. Experiments on Llama-2-7B/13B and Llama-3-8B compare LSAQ against a cosine-similarity-based LWQ baseline on six zero-shot tasks and WikiText2 perplexity at 7/6/5 average bits, reporting small average accuracy gains and mostly lower perplexity.","tokens_in":12438,"tokens_out":3379,"duration_ms":34197,"significance":"If validated, LSAQ would be a simple and practical heuristic for mixed-precision deployment of LLMs on memory-constrained devices, with the useful property of adapting the quantization scheme to available resources. The paper's intended contribution is clear, and the deployment-oriented framing (offline planning plus online quantization) is sensible. However, the central claim that Jaccard token-set overlap is a more effective layer-importance metric than cosine similarity is not established by the current evidence: the metric is not validated against any per-layer sensitivity oracle, the token-set construction uses an unvalidated projection, and the empirical differences over the single baseline are small and unreplicated. The paper also ships no code or detailed configuration, which limits reproducibility. The idea is worth pursuing, but the current manuscript requires substantial additional validation.","major_comments":[{"comment":"The paper states that projecting hidden states through W_E^T and selecting the top-k indices yields 'the top-k most probable tokens' for the layer input and output. For Llama-2 and Llama-3, the output embedding (language modeling head) is not weight-tied with the input embedding matrix, so X_{i,in} · W_E^T is not the vocabulary logits distribution. The resulting sets are therefore not the model's top-k predicted tokens, and the semantic interpretation attached to the Jaccard metric is not justified. The authors should either use the actual language-model head for this projection, or explicitly reframe the construction as a heuristic similarity measure and validate it as such.","section":"III-A, Equations (1)-(2)"},{"comment":"The load-bearing assumption is that a layer with high Jaccard overlap between its input and output top-k token sets is less important and can tolerate stronger quantization. This assumption is not validated. Jaccard set overlap discards all probability information inside the set: a layer could reorder or sharpen the probabilities among the same top-k tokens and still be critical for the final prediction, while Equation (3) would label it low-importance. The paper needs a direct validation of the importance ranking against a per-layer quantization-sensitivity oracle, for example by measuring the actual perplexity or zero-shot accuracy drop when each individual layer is quantized to INT4 and comparing that oracle ranking with the Jaccard ranking. Without such a test, the end-to-end results cannot confirm that the proposed metric is meaningfully better than cosine similarity or even than a random layer ordering.","section":"Equation (3) and IV.B.3"},{"comment":"The empirical evidence for the central claim is thin. The accuracy differences over the LWQ cosine baseline are typically 0.1–1.5 points on individual tasks, with no error bars, no repeated runs, and no significance tests; one perplexity comparison (Llama-3-8B at 7 bits) actually favors LWQ, which is acknowledged in IV.B.2 but conflicts with the abstract's 'consistently outperforms' wording. Since the only baseline is LWQ with cosine similarity, and both methods use the same quantization engine, the claimed superiority of the Jaccard metric rests on very small margins. The authors should add multiple seeds or calibration sets, report variance, and ideally include additional baselines (e.g., layer-importance by weight-outlier count, random layer ordering, and a sensitivity-oracle upper bound) to demonstrate that the chosen metric, not just the allocation procedure, drives the gains.","section":"IV.B, Tables I and III"},{"comment":"The allocation pseudocode is underspecified at a load-bearing point. Line 8 sets 'Mint8' to the difference between available memory and the memory required for INT4 precision, but the variable name suggests it should be the memory available for INT8 layers; the subsequent formula Nint4 = len - Mint8 / S8to4 should be derived explicitly with units (bytes vs. bits) and the memory overhead for activations or inference buffers should be stated. As written, the formula is not reproducible from the text.","section":"Algorithm 1"}],"minor_comments":[{"comment":"The claim of an advantage in '87.5% of individual tasks' should state the denominator explicitly (e.g., how many model-bit-width-task triples are compared) and clarify whether the 7-bit Llama-2-7B case, where both methods are identical, is included in the count.","section":"IV.B.1"},{"comment":"The abstract says LSAQ 'consistently outperforms' the baselines, but the paper later reports a perplexity exception for Llama-3-8B at 7 bits; the wording should be tempered to match the actual results.","section":"Abstract and IV.B.2"},{"comment":"The sentence 'we transform this value by inversion and addition to obtain the importance metric' is unclear; Equation (3) simply defines I_i as 1 minus the Jaccard similarity, so the description should be rewritten to match the formula.","section":"III.A"},{"comment":"The choice of k is justified only by a citation to [25]; since k directly controls the size of the token sets and hence the granularity of the metric, a sensitivity analysis over k (even a small one) would strengthen the claim that the specific value has minimal impact.","section":"III.A"},{"comment":"The paper would benefit from a discussion of calibration data: the importance detection in Equations (1)-(2) requires input hidden states, but the text does not specify which dataset or how many samples are used to compute the layer importance, nor whether the results are stable across different calibration sets.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the deployment framework is practical, but the core importance metric is not convincingly validated. The non-tied embedding issue is a concrete correctness concern that the authors can address, and the empirical section needs stronger evidence (error bars, additional baselines, a sensitivity oracle). I believe the paper is within scope for a revision rather than a rejection, but the current version does not yet support the strong claims made in the abstract and Section IV.B.3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on LSAQ. The genuinely new bit is using Jaccard similarity between top-k token sets derived from layer input/output hidden states as a layer importance measure for mixed-precision quantization. That is a real departure from cosine similarity and outlier-count metrics in LWQ and ShortGPT, and the idea is simple enough to be practical. The deployment allocation in Algorithm 1 is a standard resource-aware heuristic, not a deep contribution, but it is clearly presented.\n\nWhat the paper does well: it runs experiments on Llama-2-7B/13B and Llama-3-8B across 7/6/5 average bits, and the Jaccard-based allocation beats LWQ on average zero-shot accuracy in most configurations and on WikiText-2 perplexity in 8 of 9 cases. The gains are small, typically 0.1 to 0.7 points accuracy, a bit larger for Llama-3 at 5-bit, but they are consistent. The layer-importance visualization and the direct comparison of layer rankings in Table II are helpful. No code is shipped, but the method is described well enough to reimplement.\n\nWhere I would push back:\n\n1. The \"top-k most probable tokens\" description is wrong for Llama. The paper projects hidden states through the input embedding matrix W_E^T, but Llama's output head is a separate lm_head, not weight-tied. The projection is not the model's vocabulary distribution, so the sets are not probabilities. The method may still work as a heuristic, but the justification in Section III-A overstates what the projection means.\n\n2. The metric discards all probability information within the overlapping set. A layer could sharpen or reorder the same top-k tokens and still be critical; Equation 3 would label it unimportant. The paper does not validate the importance ranking against a per-layer quantization-sensitivity oracle, so the theoretical basis is shaky.\n\n3. The empirical case is thin. One baseline, no error bars, no significance tests, and one perplexity configuration comes out worse. Table IV has an obvious arithmetic inconsistency: for the 6G row, 32 INT8 plus 10 INT4 layers sums to 42 layers in a 32-layer model. Likely a typo, but it signals sloppiness.\n\n4. No code, and calibration details are missing: which samples are used for hidden states, sequence length, the exact k value. The claim that k has little effect is plausible but unsupported.\n\nOverall, the central idea is plausible and the end-to-end results are consistent enough that the paper deserves a serious referee. The flaws are addressable with a corrected formulation, a sensitivity validation, and a stronger baseline comparison. I would send it to review with expectations of major revision, not desk-reject it.\n\nFor your reading group: worth a look if you work on quantization, but it is not a must-read.","headline":"A simple, plausibly useful Jaccard-based layer importance metric for LLM quantization, but the paper's token-set justification is technically off and the evaluation is too thin to fully support the centrality claim.","tokens_in":12961,"tokens_out":3201,"would_cite":true,"duration_ms":30012,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LSAQ claims that Jaccard overlap between a layer's top-k input and output tokens is a better guide to which LLM layers can be safely quantized to INT4 than cosine similarity is.","keywords":["large language models","post-training quantization","layer importance","Jaccard similarity","mixed-precision quantization","edge deployment","Llama","perplexity"],"falsifier":"On Llama-2-7B, compute the Jaccard importance score for all 32 layers, quantize the 10 lowest-scoring layers to INT4 and the remaining 22 to INT8, and compare perplexity against quantizing 10 randomly chosen layers to INT4. The central claim is falsified if the random split matches or beats the Jaccard-based split, or if zeroing or heavily quantizing any high-overlap layer causes a large perplexity jump.","tokens_in":12005,"feed_emoji":"📱","tokens_out":4876,"duration_ms":47577,"temperature":0.7,"pith_summary":"This paper proposes that the layers of a large language model can be ranked by how little they change the model's most likely next-token vocabulary, measured by the Jaccard overlap between the top-k tokens before and after each layer. On that ranking, LSAQ assigns INT8 to high-importance layers and INT4 to low-importance layers, choosing the split from the GPU memory actually available. The paper reports that this allocation beats a cosine-similarity-based layer-wise quantizer at the same average bit-width on zero-shot tasks and WikiText2 perplexity across Llama-2-7B, Llama-2-13B, and Llama-3-8B. The intended payoff is that one model can be deployed at whatever precision fits the device, with the least harmful layers taking the compression.","feed_headline":"Token overlap finds LLM layers safe to compress to 4 bits","feed_subtitle":"A Jaccard-based importance score beats cosine similarity for mixed INT8/INT4 quantization on Llama models.","key_machinery":"The machinery is the top-k token-set Jaccard importance metric (Equation 3), built from hidden states projected to vocabulary space through the embedding matrix. It treats each layer as a semantic transformation: decode input and output hidden states to their most probable tokens, count how much the token sets overlap, and invert that similarity into an importance score. This score feeds an allocation algorithm (Algorithm 1) that, given free GPU memory, maximizes the number of high-precision layers: full FP16 if memory allows, then INT8, then as many INT8 layers as memory allows with the least important layers dropped to INT4. Per-channel INT8/INT4 weight quantization is then applied to the chosen layers.","core_discovery":"The central claim is that semantic layer importance is better captured by token-set overlap than by cosine similarity of hidden states. For each layer, LSAQ projects the last-token hidden state at input and output through the embedding matrix, keeps the k most probable vocabulary tokens, and defines importance as $I_i = 1 - J(C_{i,\\text{in}}, C_{i,\\text{out}})$, where $J$ is the Jaccard similarity of the two top-k token sets. Layers whose input and output token sets overlap heavily are judged redundant and are quantized to INT4 first; layers whose token sets diverge are kept at INT8 or FP16. The paper argues that this ranking is more effective than cosine similarity, and supports it experimentally by showing better zero-shot accuracy and lower perplexity at 7-, 6-, and 5-bit average precision.","pith_inferences":["The Jaccard top-k ranking could be reused beyond quantization, for example to decide which layers to prune or which layers can be skipped in early-exit inference, since it identifies layers that barely change the model's output-token distribution; the paper does not test these uses.","A stronger version of the metric might compare full output distributions rather than top-k token sets, which would avoid the arbitrary choice of k but would be more expensive to compute.","The claim that a high-overlap layer is unimportant would be directly testable by comparing the Jaccard ranking against an oracle sensitivity ranking obtained by perturbing each layer; the paper does not report that comparison."],"forward_implications":["At the same average bit-width, LSAQ reports higher average zero-shot accuracy than the cosine-based LWQ baseline on most tasks and lower WikiText2 perplexity in most of the tested configurations.","The same quantized model can be reconfigured for different devices: for Llama-2-7B, 16GB keeps all 32 layers at FP16, 8GB fits all 32 at INT8, and 6GB fits 22 INT8 layers plus 10 INT4 layers.","Because the importance ranking is computed once offline, online deployment only requires reading available memory and selecting the corresponding INT8/INT4 split, which enables dynamic deployment to edge devices with different resources.","At 5-bit average precision, LSAQ stays within a few points of the 8-bit model on most zero-shot tasks, suggesting that heavily compressed deployments can preserve most of the model's capabilities."],"supporting_citations":[{"why":"Supplies the prior observation that LLM layers are redundant and the cosine-similarity-based notion of layer importance that LSAQ aims to improve.","marker":"[18]"},{"why":"Defines the Layer-Wise Quantization baseline whose cosine-similarity importance metric and same-granularity quantization scheme LSAQ compares against.","marker":"[19]"},{"why":"Provides the Llama-2 models used in the quantization and deployment experiments.","marker":"[7]"},{"why":"Provides the Llama-3 model used in the quantization experiments.","marker":"[27]"},{"why":"Supplies the basis for choosing the value of k when constructing the top-k token sets.","marker":"[25]"},{"why":"Defines the Jaccard similarity coefficient used as the core layer-importance metric in Equation 3.","marker":"[26]"},{"why":"Provides the WikiText2 dataset used for the perplexity evaluation that supports the main comparison.","marker":"[20]"}],"fun_headline_variants":["Token overlap ranks LLM layers for adaptive quantization","LSAQ: Layer importance from token sets guides bit width","Mixed INT4/INT8 via Jaccard similarity on LLM layers","Dynamic LLM quantization adapts to edge device resources"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme assumes that a layer whose top-k most-likely tokens barely change from input to output is safe to compress hardest; if a layer preserves token identities while still computing probability shifts the model relies on, the importance ranking will be misleading.","fun_headline_variants_meta":{"raw":{"variants":["Token overlap ranks LLM layers for adaptive quantization","LSAQ: Layer importance from token sets guides bit width","Mixed INT4/INT8 via Jaccard similarity on LLM layers","Dynamic LLM quantization adapts to edge device resources"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1248,"prompt_tokens":937,"completion_tokens":311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":553,"tokens_out":311,"duration_ms":3753,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:59:49.057229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On Llama-2-7B, compute the Jaccard importance score for all 32 layers, quantize the 10 lowest-scoring layers to INT4 and the remaining 22 to INT8, and compare perplexity against quantizing 10 randomly chosen layers to INT4. The central claim is falsified if the random split matches or beats the Jaccard-based split, or if zeroing or heavily quantizing any high-overlap layer causes a large perplexity jump.","supporting_citations":[],"review_version":1}