{"id":"b27d1529-447a-49ac-834b-1f967b5140f8","arxiv_id":"2411.13814","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"AutoMixQ uses Gaussian-process search to assign 4 or 8 bits per layer of a pruned, LoRA-fine-tuned LLM, reporting memory savings with mixed accuracy results.","lead":"AutoMixQ is a system that picks different numeric precisions for different layers of a pruned large language model during fine-tuning. It reports lower memory use than standard LoRA and LoftQ baselines, but the accuracy gains come from searching directly on each benchmark and are not always consistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AutoMixQ's accuracy advantage is likely a selection artifact: the reported score is the best of ~50 fine-tuned configurations chosen using the same benchmark's metric, with no held-out split, repeated seeds, or error bars.","rationale":"The reader's verdict of REJECT is well-supported. The most load-bearing premise is that P(q) in Eq. 5, measured on the benchmark itself, is a fair and reliable objective, and that the reported AutoMixQ score is an unbiased estimate of real performance. In practice, the paper optimizes directly against the evaluation metric of each benchmark, selecting one configuration from 50 fine-tuned candidates, while the baselines are single runs. This is textbook selection bias: the maximum of many noisy evaluations will exceed a single evaluation even without a true accuracy advantage. The paper's own Table 2 shows AutoMixQ losing to LoRA on several entries (e.g., LLaMA-7B 30% HellaSwag and ARC-e), which is inconsistent with the 'consistently outperforms' wording and further supports that the reported wins are within the expected spread of selection noise. The memory savings are credible, since AutoMixQ uses many 4-bit layers, but the accuracy advantage is the crux of the claimed contribution. The proposed test—using a held-out split, repeated seeds, and a random-search control—directly settles whether the advantage is real or an artifact. Until such evidence is provided, the central performance claim is not established, and REJECT remains the appropriate verdict.","tokens_in":15296,"tokens_out":3238,"duration_ms":27970,"concrete_test":"Run the identical AutoMixQ pipeline but replace P(q) in Eq. 5 with accuracy on a held-out validation split of each benchmark (e.g., 50% of the evaluation set), then report the chosen configuration's accuracy on the untouched half, averaged over at least 5 independent search repetitions with different seeds. Also run a random-search control: sample 50 configurations uniformly at random, fine-tune each, and pick the best on the same split. If AutoMixQ's held-out accuracy does not exceed the random-search control and the LoRA/LoftQ baselines by a margin larger than the seed noise, the claimed consistent accuracy advantage is a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AutoMixQ 'consistently outperforms' LoRA/LoftQ in accuracy rests on comparing a single baseline run against the best of ~50 AutoMixQ configurations, each selected using the same benchmark's evaluation metric as the search objective. Section 4.2 states that 10 initial configs plus 40 GP iterations produce 50 data points, and the Pareto front is built from their measured P(q); the final configuration is then chosen to maximize Eq. 5 on that same benchmark. No held-out split, no repeated seeds, and no error bars are reported. Under zero-shot evaluation on datasets like OBQA (~500 examples) or BoolQ (~3.2k), per-seed accuracy variation is typically several points, so the max-of-50 vs single-run comparison can produce the observed 1-5 point gaps even if the true mean performance of AutoMixQ equals the baselines. The internal inconsistency in Table 2—e.g., LLaMA-7B 30% HellaSwag 61.14 vs LoRA 63.14, ARC-e 58.12 vs 59.22—is exactly what one expects from selection noise, not from a systematic advantage. Therefore the 'superior performance' half of the central claim is not supported by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoMixQ, a framework that combines structured pruning (LLM-Pruner), LoRA fine-tuning, and mixed-precision layer-wise quantization for LLMs. A Gaussian-process surrogate model guides an iterative search over per-layer 4/8-bit quantization configurations, using Pareto optimality to balance memory and downstream accuracy, with the final configuration selected by a weighted objective (Eq. 5). Experiments on LLaMA-7B, LLaMA-13B, and Vicuna-7B across seven commonsense zero-shot benchmarks report lower peak memory than LoRA and LoftQ and, in many cells, higher accuracy.","tokens_in":15485,"tokens_out":6541,"duration_ms":55433,"significance":"The idea of automatically determining per-layer bit-widths for pruned models during LoRA fine-tuning is practically motivated, and the reported memory reductions are credible because AutoMixQ uses 4-bit layers. The paper includes useful implementation details, ablation studies on initialization and data types, and reproducible software versions. If the accuracy claim were properly validated, the GP-guided Pareto search could be a useful tool for resource-constrained fine-tuning. However, the current evaluation does not support the 'superior performance' half of the central claim because the search is optimized directly on the same benchmark used for reporting, and the paper's own tables contain multiple counterexamples to the claim of consistent outperformance.","major_comments":[{"comment":"The reported AutoMixQ accuracy is the result of selecting a configuration from up to 50 actual fine-tuning runs, where the selection objective (Eq. 5) is evaluated on the same downstream benchmark later reported in Table 2. The workflow in Section 3.3 and the implementation in Section 4.2 make clear that the GP is trained on P(q) measured on the target benchmark, and the final configuration is chosen from the Pareto front using that same benchmark's performance. There is no held-out validation split and no repeated seeds. Consequently, the reported AutoMixQ scores are effectively the best-of-50 on the test set, while the LoRA and LoftQ baselines are single runs. This selection protocol inflates AutoMixQ's apparent accuracy and invalidates the claim that AutoMixQ 'consistently outperforms' the baselines.","section":"Section 3.3, Section 4.2, Table 2"},{"comment":"The claim that AutoMixQ 'consistently outperforms both LoRA and LoftQ' in accuracy is contradicted by the paper's own data. For example, LLaMA-7B at 30% pruning shows AutoMixQ worse than LoRA on HellaSwag (61.14 vs 63.14), ARC-e (58.12 vs 59.22), and OBQA (39.00 vs 39.60); at 20% pruning it is worse on HellaSwag and WinoGrande. Even ignoring the selection artifact, the reported numbers do not support a claim of consistent outperformance. The authors should either soften the claim to match the evidence or provide statistically meaningful comparisons (multiple seeds, error bars) that demonstrate an advantage.","section":"Abstract, Section 4.3, Table 2"},{"comment":"The trade-off parameter λ that selects the final configuration from the Pareto front is never specified in the experimental section. Because memory and accuracy of the reported configuration depend directly on λ, the absence of its value makes the main results irreproducible. The paper should state the value(s) of λ used for each benchmark or, better, report the entire Pareto front and how the reported point was chosen.","section":"Section 3.1, Eq. (5), Section 4.2"},{"comment":"No repeated seeds or variance estimates are reported for any method. On small zero-shot benchmarks such as OBQA (~500 examples) and BoolQ (~3.2k instances), per-seed accuracy differences of several percentage points are common. Without error bars, the observed gaps between AutoMixQ and baselines (some as small as 0.1-0.5 points) cannot be distinguished from noise. A valid comparison requires multiple seeds and reporting mean ± standard deviation, ideally on a held-out test split after model selection on a validation split.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The description of the search space as 'a tensor with dimensions L × n^L' is confusing and technically incorrect; the number of possible configurations is n^L, not L × n^L.","section":"Section 3.1"},{"comment":"There is a typo: 'performes' should be 'performs'.","section":"Section 4.1"},{"comment":"The sample generation results in Figures 3a and 3b are anecdotal; the claim that they demonstrate 'significant superiority' is not supported by any quantitative metric.","section":"Appendix E"},{"comment":"Some references are incomplete, e.g., the 'Meng 2024' citation for PiSSA lists only a first name, and the Alpaca citation lacks a proper author list.","section":"References"},{"comment":"The motivating example in Table 1 reports a single random mixed-precision configuration; the paper should clarify whether the performance of that configuration is from one random draw and how representative it is, since the motivating example is used to justify the entire search approach.","section":"Section 2.3, Table 1"}],"recommendation":"reject","confidential_remarks":"The core methodological flaw—selecting the reported configuration on the same test benchmark used for evaluation—is a well-known pitfall that invalidates the performance comparison as presented. The paper also overstates its own results: the table contradicts the 'consistently outperforms' claim. While the memory-savings contribution is plausible, the central accuracy claim is not supported. A major rework with proper held-out validation and repeated seeds would be needed before the paper could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Memory savings are credible, but the accuracy advantage is probably a selection artifact: the reported score is the best of ~50 fine-tuned configurations chosen using the same benchmark's metric, with no held-out split or error bars. The new thing is an end-to-end pipeline: LLM-Pruner structured pruning, LoftQ initialization, per-layer 4/8-bit mixed-precision quantization, and a GP-guided Pareto search over layer-wise bit-width assignments. That combination is not in the cited prior work, and the paper treats it as a system rather than a single trick. The memory numbers are believable because 4-bit layers obviously use less memory than 16-bit LoRA or 8-bit LoftQ. The reported peak-memory reductions (roughly 30-35% vs LoRA, 25% vs LoftQ) are consistent with that. The ablation study is a nice extra; it shows the method isn't tied to one NF4/FP4 choice or one adapter initialization.\n\nThe soft spot is the accuracy claim, and it is load-bearing. The configuration is chosen by running up to 50 actual fine-tunes per benchmark, using the same benchmark's metric as the search objective, and then reporting the best configuration's score on that same benchmark. There is no held-out split, no repeated seeds, no error bars. On small datasets like OBQA (~500 examples) or even BoolQ (~3.2k), per-seed accuracy variation is typically a few points. So comparing a single baseline run against the max-of-50 AutoMixQ runs can easily produce the observed 1-5 point gaps even if the true mean performance is identical. The internal inconsistency in Table 2 makes this concrete: at 30% pruning on LLaMA-7B, AutoMixQ gets 61.14 HellaSwag vs LoRA's 63.14, and 58.12 ARC-e vs LoRA's 59.22. That is exactly what selection noise looks like, not a systematic advantage. The abstract's 'consistently outperforms both LoRA and LoftQ' is simply not supported by the paper's own table.\n\nThe GP and Pareto machinery is standard, but that's fine as an internal search heuristic. The bigger issue is that the evaluation protocol conflates model selection with final evaluation. If the authors fixed that--held-out validation, repeated seeds, reporting mean±std, or at least comparing against baselines with the same search budget--the paper could be a useful practical result. As it stands, the memory contribution is real and the accuracy contribution is unverified.\n\nWho should read this: people working on efficient fine-tuning of pruned LLMs who want a concrete recipe for mixed-precision search. It deserves a serious referee, not a desk reject, because the method is concrete and the memory claim is credible; but the empirical claim needs major rework before publication. I would not cite it in its current form.","headline":"Memory savings are credible, but the accuracy advantage is probably a selection artifact: the reported score is the best of ~50 configs tuned on the same benchmark, with no held-out split or error bars.","tokens_in":816,"tokens_out":1735,"would_cite":false,"duration_ms":28149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoMixQ claims that an automated per-layer bit-width search lets pruned LLMs fine-tune with less memory and better or equal accuracy than LoRA and LoftQ.","keywords":["mixed-precision quantization","parameter-efficient fine-tuning","LoRA","structured pruning","Gaussian process optimization","Pareto optimality","memory-efficient LLM fine-tuning","LLM compression"],"falsifier":"Run the AutoMixQ search with a held-out validation set; if the configuration selected on validation accuracy does not reproduce the reported gains on the test benchmarks, or if uniform 8-bit quantization matches it at equal memory, the central claim of a superior trade-off collapses.","tokens_in":15045,"feed_emoji":"⚙️","tokens_out":6776,"duration_ms":93011,"temperature":0.7,"pith_summary":"This paper argues that combining structured pruning, LoRA fine-tuning, and per-layer mixed-precision quantization works better than treating all layers alike. Pruning leaves layers with different importance and computational loads, so a uniform 4-bit or 8-bit setting is not the best use of a memory budget. AutoMixQ automates the choice of each layer's bit-width by iteratively training a lightweight performance model and selecting configurations on the Pareto frontier of memory versus task accuracy. In experiments on LLaMA-7B, LLaMA-13B, and Vicuna-7B, the framework reports lower peak memory than LoRA and LoftQ while matching or improving accuracy on commonsense benchmarks.","feed_headline":"Mixed-precision search cuts memory a third and beats uniform baselines","feed_subtitle":"AutoMixQ chooses per-layer 4/8-bit settings with a Gaussian-process search, outperforming LoRA and LoftQ on pruned LLMs.","key_machinery":"The load-bearing object is the Gaussian-process performance model trained on actual fine-tuning outcomes, used with Pareto optimality to navigate the exponentially large space of per-layer bit-width assignments. Each candidate configuration $q$ is a vector of 4-bit or 8-bit choices across layers; the GP predicts $P(q)$, memory $M(q)$ is computed from the bit-widths, and configurations that no other configuration dominates in both objectives form the Pareto frontier. The objective $q^{*} = \\arg\\min_{q} (M(q) - \\lambda P(q))$ picks one point on that frontier, balancing memory savings against predicted accuracy.","core_discovery":"The central claim is that AutoMixQ, an end-to-end optimization loop, can find per-layer 4/8-bit quantization configurations for a pruned LLM that dominate uniform-precision baselines on the memory-accuracy trade-off. The loop starts with a few randomly configured fine-tuning runs, fits a Gaussian-process model to predict task performance $P(q)$ from the configuration vector $q$, builds the Pareto frontier over predicted performance and measured memory $M(q)$, and fine-tunes the model under the configuration that best fits the objective $M(q) - \\lambda P(q)$. Repeating this cycle for about 40 iterations yields configurations that, the paper reports, cut peak memory by roughly a third relative to LoRA and a quarter relative to LoftQ while often improving zero-shot accuracy, such as 66.21% versus 62.45% and 58.96% on BoolQ at 30% pruning of LLaMA-7B.","pith_inferences":["Beyond the paper: the same GP-guided Pareto loop could be extended to search over LoRA ranks, activation bit-widths, or pruning ratios jointly, since the objective already treats memory and performance as separable costs.","Beyond the paper: the reported scores come from optimizing on the same benchmark used for the final comparison; a held-out split or repeated seeds would test whether the search is genuinely finding better configurations rather than selecting noise.","Beyond the paper: if benchmark scores are noisy, a GP that models observation noise explicitly, or an acquisition function that penalizes variance, would make the reported trade-off more trustworthy."],"forward_implications":["At equal or better accuracy, AutoMixQ reports peak memory reductions around 30% versus LoRA and roughly 25% versus LoftQ across LLaMA-7B, LLaMA-13B, and Vicuna-7B.","Because the search is self-adjusting, the same pipeline can be applied to other pruning methods, models, and tasks without redesigning the quantization scheme.","Higher pruning rates (up to 50%) still leave AutoMixQ competitive with or ahead of LoRA and LoftQ, so aggressive compression does not erase the benefit of refined quantization.","The framework deliberately restricts choices to 4-bit and 8-bit, since 2-bit quantization does not reduce memory usage under the LoftQ-style initialization used here."],"supporting_citations":[{"why":"Supplies the NF4 4-bit format and the QLoRA quantization-plus-LoRA recipe that AutoMixQ builds on for memory savings.","marker":"Dettmers et al. 2024"},{"why":"LoftQ initialization reconciles quantized weights with low-rank adapters; AutoMixQ uses it as its starting point and as a uniform-precision baseline.","marker":"Li et al. 2023"},{"why":"LLM-Pruner performs the structured pruning that creates the uneven per-layer structure motivating mixed-precision quantization.","marker":"Ma, Fang, and Wang 2023"},{"why":"LoRA provides the low-rank adaptation mechanism whose fine-tuning memory and performance AutoMixQ compares against.","marker":"Hu et al. 2021"},{"why":"Shows layer importance varies, the premise for per-layer precision allocation in pruned models.","marker":"Zhang et al. 2023c"},{"why":"Defines Pareto optimality in multi-objective problems, the criterion AutoMixQ uses to filter candidate configurations.","marker":"Censor 1977"}],"fun_headline_variants":["AutoMixQ: 35% less memory, better zero-shot on BoolQ","Self-adjusting bit-widths beat uniform for pruned fine-tuning","Pareto-optimal quantization: GP search cuts memory a third","Per-layer 4/8-bit configs outdo LoRA and LoftQ in memory-accuracy","Mixed-precision fine-tuning with GP: less memory, more accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the benchmark accuracy used to steer the search is a stable and unbiased measure of real performance, because the same test scores choose the configuration and are then reported as the result.","fun_headline_variants_meta":{"raw":{"variants":["AutoMixQ: 35% less memory, better zero-shot on BoolQ","Self-adjusting bit-widths beat uniform for pruned fine-tuning","Pareto-optimal quantization: GP search cuts memory a third","Per-layer 4/8-bit configs outdo LoRA and LoftQ in memory-accuracy","Mixed-precision fine-tuning with GP: less memory, more accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001042,"raw_usage":{"total_tokens":4400,"prompt_tokens":983,"completion_tokens":3417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":3314}},"tokens_in":599,"tokens_out":3417,"duration_ms":26046,"temperature":1.0,"reasoning_tokens":3314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:50:47.280060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the AutoMixQ search with a held-out validation set; if the configuration selected on validation accuracy does not reproduce the reported gains on the test benchmarks, or if uniform 8-bit quantization matches it at equal memory, the central claim of a superior trade-off collapses.","supporting_citations":[],"review_version":1}