{"id":"edb0b3bb-04c6-46ab-b10d-b4ad7cfdbaff","arxiv_id":"1909.01688","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A gradually shrinking teacher signal during knowledge distillation improves 2-bit quantized ResNet20 accuracy on CIFAR-10 and CIFAR-100 compared with fixed-coefficient distillation.","lead":"Neural networks with very low precision weights often lose accuracy. This paper shows how tuning knowledge-distillation settings recovers much of that loss and proposes gradually reducing the teacher's influence during training, reporting strong 2-bit ResNet20 results on CIFAR-10 and CIFAR-100.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GSLR is the empirical crux of the paper, but its λ schedule is undefined, so the reported 92.7%/67.0% gains cannot be reproduced or attributed to the method.","rationale":"The paper's novel and actionable contribution is GSLR plus the reported accuracy gains. The abstract promises a concrete technique, but Section 3.4 provides only a qualitative description of the schedule. The CIFAR-10 number in the abstract is also not traceable to any table entry labeled GSLR, while the CIFAR-100 number is. This gap is load-bearing because the entire 'robust KD training' claim depends on the schedule; without it, the improvement cannot be reproduced, and the claim that GSLR is 'fail-safe' cannot be evaluated. The reader's weakest assumption identified this same missing schedule; I agree. The paper's softer scientific points, such as the softmax-distribution argument in Table 1 and Figure 1, are plausible and provide some support, but they do not rescue the practical GSLR claim. The appropriate verdict remains conditional, pending schedule disclosure and a rerun under the stated schedule with variance reported. No change to the reader's conditional verdict is needed.","tokens_in":8699,"tokens_out":6376,"duration_ms":63416,"concrete_test":"Ask the authors for the exact λ(t) schedule used in Figure 3 and Table 4, e.g., λ(t) = λ0 * (1 - t/T)^p with λ0=0.5, T=total epochs, and p an exponent. Then rerun the CIFAR-10 and CIFAR-100 2-bit ResNet20 experiments for every teacher size and temperature in Figures 2 and 3, using five seeds each, only that schedule, and no test-set-based selection. Report per-setting means and standard deviations plus a full table. If the 92.7% and 67.0% numbers reproduce and GSLR's worst-case result stays above the hard-label baseline, the robustness claim holds; otherwise GSLR is an artifact of unspecified schedule selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 defines GSLR only as 'use the soft and hard losses equally at the beginning ... gradually reduce the amount of the soft loss as the training proceeds.' No functional form for λ(t), decay rate, epoch boundaries, or endpoint is given in Section 3.4, Algorithm 1, or Figure 3. The CIFAR-100 headline result (67.0%, Table 4) is labeled 'GSLR', but the CIFAR-10 headline result in the abstract (92.7%) does not appear in Table 4; the closest entry is 92.52% for τ=10, λ=0.5 without GSLR. This matters because the paper's practical claim is that GSLR is 'fail-safe' and 'much less sensitive to specific parameter setting' (Section 3.4). Without the schedule, the method cannot be instantiated independently, so replication failures can be blamed on the unspecified details. Figure 3, the only evidence for robustness, reports no numerical values, standard deviations, or seed counts; it is a visual assertion that GSLR lifts results across teacher sizes and temperatures. The central claim therefore rests on an unstated procedure and an unreported summary, making the 'robust' improvement untestable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the use of knowledge distillation (KD) for training quantized deep neural networks (QDNNs), specifically 2-bit (ternary) ResNet20 on CIFAR-10 and CIFAR-100. The authors experimentally compare different teacher models (full-precision, quantized, teacher-assistant) and study the effects of the KD hyperparameters: temperature τ, loss weighting factor λ, and teacher network size. Their main claims are: (i) the softmax distribution of the teacher, rather than the teacher's accuracy, is the key factor for effective QDNN distillation; (ii) the KD hyperparameters are interrelated, so they should be tuned jointly; and (iii) a proposed technique called gradual soft loss reducing (GSLR), which starts with equal soft and hard losses and gradually decreases the soft-loss weight during training, provides robust, 'fail-safe' improvements. The paper reports test accuracies of 92.7% on CIFAR-10 and 67.0% on CIFAR-100 for 2-bit ternary ResNet20, outperforming prior KD-based QDNN methods (QDistill, Apprentice, Guided) while using much smaller student models.","tokens_in":8939,"tokens_out":2952,"duration_ms":25962,"significance":"If the claims hold, the paper would make a useful practical contribution to low-precision network training: a simple, hyperparameter-robust distillation recipe that works with small teacher models, plus a systematic empirical analysis of the interaction between temperature, loss weight, and teacher size. The comparison against prior work is favorable and the student models are substantially smaller, which is practically relevant. However, the central novelty—the GSLR schedule—is left undefined, and the headline CIFAR-10 number in the abstract does not appear in the main results table. These issues currently prevent independent reproduction and make the 'fail-safe' robustness claim untestable. The paper's strengths are its clear experimental scope across teacher types and sizes, and the plausible observation that temperature can compensate for teacher size in QDNN distillation.","major_comments":[{"comment":"The GSLR technique, which is the paper's main contribution, is not fully specified. The text says only that the soft and hard losses are used equally at the beginning and that the soft loss is 'gradually reduced' as training proceeds, but no functional form for λ(t), decay rate, epoch boundaries, or final value is given in Section 3.4, Algorithm 1, or Figure 3. Because the CIFAR-100 result of 67.0% in Table 4 is attributed to GSLR, this omission prevents the method from being instantiated independently and makes replication failures impossible to diagnose. Please provide the exact schedule and, ideally, pseudocode for λ(t) for all reported GSLR experiments.","section":"Section 3.4, Algorithm 1"},{"comment":"The abstract states that the proposed hyperparameters plus GSLR achieve 92.7% test accuracy on CIFAR-10 with 2-bit ternary ResNet20, but Table 4 lists 92.52% as the best CIFAR-10 result (τ=10, λ=0.5, no GSLR) and shows GSLR only for the CIFAR-100 entry. The source of the 92.7% number is unclear. If it comes from a GSLR experiment, the corresponding teacher size, τ, λ, and schedule should be reported; if not, the abstract and the body should be reconciled.","section":"Abstract vs. Table 4"},{"comment":"Figure 3, which is the sole evidence for the 'fail-safe' and 'much less sensitive' claims about GSLR, reports no numerical values, standard deviations, or seed counts. Section 2.2 states that Table 1 results are averages over five runs, but no such statement is made for Figure 3. Without variance information or at least exact plotted values, the reader cannot assess whether the improvements shown in panel (b) over panel (a) are statistically meaningful. Please provide the underlying numbers and the number of runs.","section":"Figure 3"},{"comment":"The hyperparameter selection procedure appears to rely on test-set performance, which is at odds with the paper's stated goal of avoiding 'cherry picking' and being 'fail-proof.' The final reported settings (e.g., τ=10 for CIFAR-10, and the GSLR result for CIFAR-100) are presented after sweeping τ and teacher size in Figures 2 and 3, but no held-out validation or model-selection protocol is described. If the GSLR schedule was chosen after inspecting test results, the robustness claim is weakened. Please clarify how the reported configurations were selected and whether the authors used a validation split.","section":"Section 3.2-3.4"}],"minor_comments":[{"comment":"In Algorithm 1, the parameter is spelled 'Loss wegithed factor'; correct the typo. Also, the algorithm inputs list λ and τ but not the GSLR schedule, which should be included if GSLR is part of the proposed training procedure.","section":"Algorithm 1"},{"comment":"In the CIFAR-10 discussion, the text refers to 'the large value (blue line)' when describing τ=10, but in Figure 2(a) the blue line appears to correspond to τ=1. Please check the color legend and reconcile the description.","section":"Section 3.3, Figure 2(a)"},{"comment":"The table header and text use inconsistent names for the prior method: 'QDistill' appears in the table and 'QDistil' appears in the text. Also, the abstract's 92.7% CIFAR-10 result is absent from the table; the closest entries are 92.52% (CIFAR-10) and 67.0% (CIFAR-100).","section":"Table 4"},{"comment":"Some reference entries have incorrect venue/year pairings; for example, reference [23] lists CVPR 2017 for the original ResNet paper, which was published at CVPR 2016. Please verify all references against the original sources.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a conference-length paper with a potentially useful empirical contribution, but the missing GSLR schedule and the abstract/Table 4 discrepancy are serious reproducibility issues that must be fixed before the paper can be accepted. The authors should also address whether the reported gains are robust to the choice of schedule and whether the schedule was tuned on the test set. No concerns about scientific misconduct, but the 'fail-safe' framing is currently not supported by the evidence presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical story about teacher size and temperature for 2-bit quantized ResNets is worth a look, but the paper's headline method (GSLR) is not specified well enough to reproduce, and the abstract and main table don't agree on the CIFAR-10 number.\n\nWhat's actually here: a clean set of experiments on CIFAR-10/100 showing that for 2-bit ResNet20, the softmax shape from a full-precision teacher matters more than teacher accuracy, and that temperature and teacher width are coupled. The observation that a small teacher (WRN20x1.2) with the right temperature can match a larger one is useful for practitioners. Table 4 also shows a big jump over the QDistill small-network baseline, though part of that gap is likely due to different training recipes rather than KD hyperparameters alone.\n\nThe soft spots are real. Section 3.4 defines GSLR only verbally: equal soft/hard loss at the start, then gradually reduce the soft loss. No functional form, decay schedule, endpoint, or epoch boundaries. Figure 3, the main evidence that GSLR is fail-safe, has no numbers, error bars, or seed counts. The abstract reports 92.7% on CIFAR-10; Table 4's closest entry is 92.52% without GSLR. No code is released. The paper also doesn't compare GSLR to other scheduled-distillation methods, which is a gap because decaying soft-loss weight is a known family of tricks. So the central reproducibility claim fails as written.\n\nThe positive side: the finding about teacher softmax distribution is consistent with the teacher-assistant literature and is demonstrated with at least some repeated runs (\"average of five times\" in Table 1). I believe the authors know what they're doing; the paper is written clearly. The issue is that the one novel component is the least documented.\n\nWho is this for? Someone working on low-precision network training might take away the practical heuristic about tau and teacher width. But as a paper, it needs the schedule spelled out and a proper table with variances before the GSLR claims can be evaluated.\n\nI'd send it to a serious referee rather than desk-reject, because the empirical question is legitimate and the baseline results are useful. But I'd expect major revision: complete GSLR specification, corrected abstract/table consistency, error bars on the key figures, and a comparison with at least one prior scheduled-loss method.","headline":"Useful empirical analysis of teacher size and temperature for 2-bit quantized networks, but the GSLR technique is underspecified and the headline numbers don't line up.","tokens_in":9467,"tokens_out":2136,"would_cite":false,"duration_ms":20614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that when knowledge distillation is used to train quantized neural networks, the decisive factor is the shape of the teacher's softmax distribution, not how accurate the teacher is, and it introduces a simple schedule…","keywords":["knowledge distillation","quantized deep neural networks","2-bit networks","softmax temperature","gradual soft loss reducing","CIFAR-10","CIFAR-100","fixed-point optimization"],"falsifier":"Train a 2-bit ResNet20 on CIFAR-100 with GSLR using a single pre-registered schedule (e.g., linear decay of $\\lambda$ from 0.5 to 0 over the full run) across the full temperature–teacher-size grid; if the 67.0% accuracy is not reproduced and accuracy varies as much as fixed-$\\lambda$ KD, the GSLR robustness claim is falsified.","tokens_in":8488,"feed_emoji":"🧠","tokens_out":5481,"duration_ms":44989,"temperature":0.7,"pith_summary":"This paper claims that when knowledge distillation is used to train quantized deep neural networks, the most important factor is not the teacher network's accuracy but the shape of its softmax distribution, which is controlled by temperature and teacher size. The authors argue that the distillation hyperparameters—temperature, loss coefficient, and teacher size—are interlinked and must be set together. They introduce a technique called gradual soft loss reducing (GSLR), which starts with equal hard and soft losses and gradually reduces the soft loss during training, making low-bit training more predictable. With careful hyperparameter selection plus GSLR, a 2-bit ternary ResNet20 reaches 92.7% on CIFAR-10 and 67.0% on CIFAR-100, surpassing prior distillation-plus-quantization baselines.","feed_headline":"A soft-label schedule pushes 2-bit ResNet20 to 92.7% on CIFAR-10","feed_subtitle":"Gradually reducing the distillation loss makes low-bit networks robust across temperatures and teacher sizes.","key_machinery":"The load-bearing object is the teacher's softmax probability distribution, $p_i = \\exp(z_i/\\tau)/\\sum_j \\exp(z_j/\\tau)$, where $\\tau$ is the temperature: $\\tau$ controls how broad or sharp the soft label is, and the paper shows this shape, not teacher accuracy, determines how much knowledge transfers to the 2-bit student. The second mechanism is GSLR, which sets the distillation loss as a mixture $L = (1-\\lambda)H(y,p_S) + \\lambda H(p_T,p_S)$ with the weight $\\lambda$ starting at 0.5 and gradually decreasing during training. GSLR is what makes the method \"fail-safe\": across a grid of teacher sizes and temperatures, it yields accuracy near or above the hard-loss baseline instead of the wide swings seen with fixed $\\lambda$.","core_discovery":"The central claim is that for quantized networks with very low precision, knowledge distillation succeeds or fails on the softmax distribution of the teacher, not on how accurate the teacher is. A broad soft label transfers useful class-relation information; a sharp one degenerates toward the hard label and gives little beyond conventional training. Because temperature and teacher size shape that distribution, the paper treats them as interacting controls: larger teachers produce sharper distributions, so temperature must rise to compensate, and too high a temperature flattens the label until the teacher's knowledge disappears. On this basis, the paper reports that a small full-precision teacher (WideResNet20x1.5) with temperature 10 and a 0.5 loss weight lifts a 2-bit ResNet20 to 92.52% on CIFAR-10, and that GSLR raises CIFAR-100 accuracy to 67.0%, surpassing prior distillation-plus-quantization baselines that used much larger student models.","pith_inferences":["If the softmax-shape hypothesis holds generally, distillation for other low-capacity students (e.g., pruned or sparse networks) should also benefit from temperature- and size-aware scheduling, not just quantized ones.","The paper omits the functional form of the GSLR schedule (decay rate, epochs, endpoint); a natural test is whether a fixed, pre-registered schedule—say linear decay over the training run—reproduces the reported 67.0%, or whether the gains depend on a schedule selected after seeing test results.","GSLR resembles curriculum and annealing methods in other domains; one could test whether the soft loss acts as a regularizer early in training, so its benefit should hold for other noise-injection or label-smoothing schemes.","A direct extension would apply GSLR to larger datasets and deeper quantized models to see whether the robustness across temperatures persists when the student is not under-parameterized relative to the dataset."],"forward_implications":["A small full-precision teacher can replace a large or quantized teacher, since only the softmax shape matters; this reduces memory and compute for distillation.","Temperature should be scaled with teacher size; the paper's results map which teacher-size and temperature pairs work for CIFAR-10 and CIFAR-100.","GSLR reduces the risk of cherry-picking hyperparameters by stabilizing accuracy across temperatures and teacher sizes.","The same approach yields a 1-bit ResNet20 with 91.3% on CIFAR-10, close to the 2-bit performance of earlier methods.","Setting the loss coefficient during training can be simplified to \"start equal, then decrease\", replacing per-dataset tuning."],"supporting_citations":[{"why":"Supplies the distillation loss and temperature that the paper tunes and extends.","marker":"[6]"},{"why":"Baseline (Apprentice) that first applies KD to low-precision networks; the paper compares against it.","marker":"[16]"},{"why":"Baseline (QDistill) for quantization-aware distillation; the paper's teacher sizes and temperatures are compared with its student models.","marker":"[17]"},{"why":"Provides the teacher-student gap analysis that motivates the paper's claim that large teachers sharpen the softmax and need temperature compensation.","marker":"[18]"},{"why":"Defines the WideResNet20xN teacher family used to sweep teacher size.","marker":"[22]"},{"why":"Baseline (Guided) for CIFAR-100 that the paper outperforms with a much smaller student.","marker":"[25]"}],"fun_headline_variants":["Soft-label decay lifts 2-bit ResNet to 92.7% on CIFAR-10","Distillation hyper-parameters fold into one schedule for quantized nets","Small teacher, hot labels: a recipe for 2-bit accuracy","Gradual label softening beats sharp teachers in low-bit training","One dial links teacher size and temperature for 2-bit distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's fail-safe claim rests on an unspecified GSLR schedule—it says only that soft and hard losses start equal and the soft loss is gradually reduced, giving no decay form or endpoint, so if the improvement depends on a schedule chosen after inspecting test results, the claim that GSLR is robust and reproducible is not supported.","fun_headline_variants_meta":{"raw":{"variants":["Soft-label decay lifts 2-bit ResNet to 92.7% on CIFAR-10","Distillation hyper-parameters fold into one schedule for quantized nets","Small teacher, hot labels: a recipe for 2-bit accuracy","Gradual label softening beats sharp teachers in low-bit training","One dial links teacher size and temperature for 2-bit distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1286,"prompt_tokens":891,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":299}},"tokens_in":507,"tokens_out":395,"duration_ms":4352,"temperature":1.0,"reasoning_tokens":299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:09:35.723131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 2-bit ResNet20 on CIFAR-100 with GSLR using a single pre-registered schedule (e.g., linear decay of $\\lambda$ from 0.5 to 0 over the full run) across the full temperature–teacher-size grid; if the 67.0% accuracy is not reproduced and accuracy varies as much as fixed-$\\lambda$ KD, the GSLR robustness claim is falsified.","supporting_citations":[{"cited_title":"Fixed-point feedfor- ward deep neural network design using weights +1, 0, and -1,","cited_arxiv_id":null,"evidence_quote":"Supplies the distillation loss and temperature that the paper tunes and extends."},{"cited_title":"Knowledge distillation using unlabeled mismatched images","cited_arxiv_id":"1703.07131","evidence_quote":"Baseline (QDistill) for quantization-aware distillation; the paper's teacher sizes and temperatures are compared with its student models."},{"cited_title":"Re- lational knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Provides the teacher-student gap analysis that motivates the paper's claim that large teachers sharpen the softmax and need temperature compensation."}],"review_version":1}