{"id":"b173cd6a-94e8-44bc-93fc-16f130e62224","arxiv_id":"1908.00858","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A teacher-loss confidence weighting for knowledge distillation lets a 93-percent-smaller pose-regression network nearly match the teacher's visual-odometry accuracy.","lead":"The authors train a small visual-odometry network by copying a large teacher network only on the frames where the teacher is accurate, using the teacher's own error as a confidence score. This shrinks the model to 7% of its original parameters while keeping the predicted camera trajectory close to the original.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attentive weighting depends on a fragile max-min normalization (Eqs. 9-11) that can flatten to near-constant weights under one outlier teacher loss or produce negative weights if min(eT)>0; without sensitivity analysis, Table 3's gap over FitNets is not yet tied to the attentiveness mechanism.","rationale":"The reader's weakest_assumption correctly identifies the normalized teacher-loss confidence score as the load-bearing component, and I agree that the method's validity hinges on Phi being a reliable, non-degenerate weighting. My pass sharpens this into a specific, testable concern: the max-min normalization in Eqs. 9-11 can collapse to uniform weights under an outlier and can even produce negative weights if the written formula is used literally. This is not an accusation of error; it is an identification of an unexamined sensitivity in the central mechanism. The internal ablation (Table 1) supports the claim that adding AIL/AHT helps, and Fig. 4 shows AIL beating unweighted baselines at drate=72.78%, which is genuine evidence and should be credited. However, Table 3's much larger gap at drate=92.95% is asserted without error bars, without the Phi distribution, and without a control that randomizes the same weights; therefore the attentive mechanism itself is not yet isolated from schedule and hyperparameter effects. Other issues the reader raised, such as missing code, missing alpha/beta values, and the absent supplementary experiment, are real but secondary: they do not decide whether the mechanism works, they only block exact reproduction. Because the concern is addressable by a focused control experiment rather than a demonstrated contradiction, the correct verdict remains CONDITIONAL, matching the reader, rather than REJECT or UNVERDICTED.","tokens_in":13872,"tokens_out":5844,"duration_ms":64829,"concrete_test":"Rerun the Table 3 experiment at drate=92.95% with three Phi variants while keeping all other training details fixed: (i) the published Eq. 9 formula; (ii) robust percentile normalization Phi = clip(1 - (e_i - q5)/(q95 - q5), 0, 1); and (iii) a random permutation of the same Phi values across training samples. If variant (iii) matches Ours within a few meters ATE, the attentiveness is not load-bearing; if variant (ii) materially changes ATE, the max-min normalization in Eqs. 9-11 is fragile. Also report the histogram of Phi on the KITTI training set and the exact alpha and beta values used for all methods.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the per-sample teacher-loss weights Phi_i = 1 - ||pT-pgt||^2_i / eta (Eq. 9) are what make distillation work, with eta = max(eT)-min(eT) over the entire training set (Eqs. 10-11). This is load-bearing because the paper attributes the 29.0m ATE of Ours in Table 3 to AIL/AHT; if Phi is nearly constant or selects the wrong frames, the method degenerates to unweighted distillation (FitNets/KD) and the reported advantage is unexplained. Three properties of Phi are not established. First, normalizing by a single max-min range makes the weights highly sensitive to one outlier teacher loss: if a single e_j is very large, eta is dominated by it and almost all Phi_i cluster near 1, switching off the attentiveness. Second, the written formula omits subtracting min(eT) in the numerator, so whenever min(eT)>0 the worst samples receive negative weights, producing anti-imitation on exactly the frames the method claims to down-weight; this is either a typo or an unstated design choice. Third, Phi is a static function of the teacher's training errors and never adapts to the student's relative accuracy or to distribution shift; the paper reports only aggregate ATE and does not show the distribution of Phi, the fraction of near-zero or negative weights, or a robustness check against perturbing eta. Without such evidence, the Table 3 gap could instead come from the two-stage schedule, the choice of hint layer, or alpha/beta tuning rather than from the attentive weights themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript addresses knowledge distillation for camera pose regression networks in visual odometry. Because regression outputs lack the 'dark knowledge' available in classification, the authors propose to weight the student's imitation loss by a per-sample confidence based on the teacher's own loss against ground truth. They introduce Attentive Imitation Loss (AIL) for the final pose objective and Attentive Hint Training (AHT) for an intermediate representation, both using the normalized teacher loss Phi_i. Training proceeds in two stages: AHT on the feature/regressor trunk, then AIL on the remaining layers. Experiments on KITTI and Malaga report student networks with up to 92.95% parameter reduction and 2.12x speedup, with ATE close to the teacher and better than standard KD, FitNets, and an upper-bound distillation baseline.","tokens_in":14201,"tokens_out":5894,"duration_ms":54385,"significance":"The central question is whether the teacher-loss-based confidence weighting is a working and necessary recipe for regression distillation. The paper gives a clear formulation and an informative ablation in Table 1: applying attentive weighting in the intermediate and final layers consistently improves ATE over the non-attentive counterparts. The cross-dataset test on Malaga and the comparison of trajectory-level accuracy against the three baselines are valuable and go beyond simple frame-wise RPE. If the normalization issue and robustness concerns are resolved, the method would be a useful contribution to model compression for pose regression. The paper does not provide code, and the quantitative evidence is based on single runs, so independent verification would be needed.","major_comments":[{"comment":"The confidence weight Phi_i as written is not a normalized teacher loss. With eta = max(eT) - min(eT), the expression Phi_i = 1 - ||pT - pgt||^2_i / eta produces values outside [0,1]; in particular, if min(eT) > 0, the worst sample receives Phi < 0, turning imitation into anti-imitation on precisely the frames the method intends to down-weight. If the intended formula is Phi_i = 1 - (||pT - pgt||^2_i - min(eT)) / eta, this needs to be stated explicitly and the discrepancy from the printed equation resolved. Please also report the empirical range of Phi_i on the training and test sets, because a denominator dominated by a single outlier would collapse all weights to ~1 and reduce AIL/AHT to unweighted distillation.","section":"Section 4, Eqs. (9)-(11)"},{"comment":"The load-bearing comparison against FitNets and Chen's OD is not yet attributable to the attentive mechanism. Table 3 compares complete methods that differ in several respects: the two-stage schedule, the choice of hint/guided layers, the attentive weights Phi_i, and the alpha/beta hyperparameters. No sensitivity analysis is given to show that the 29.0m ATE advantage over FitNets (31.96m) is driven by Phi_i rather than by these auxiliary design choices. I recommend ablations that keep all other components identical and vary only the weighting (e.g., replace Phi_i by a constant, by the corrected max-min normalized weights, and by a percentile-based robust normalization), together with the distribution of Phi_i.","section":"Section 7.5, Table 3"},{"comment":"All reported results are based on a single training run per configuration; no standard deviation, seed, or repeated-run statistics are provided. Because Table 3's improvement over FitNets is on the order of 3m in ATE, while Table 1 shows differences of 8-15m between procedures, the reader cannot judge whether the gains are stable. Please report multiple runs (at least 3 seeds) for the main configurations and use error bars or significance tests.","section":"Section 7.3, Tables 1-3"}],"minor_comments":[{"comment":"Table 1 reports 'Rec. Error' of the student's intermediate representation with respect to the teacher's representation; for AHT this is literally the training objective being minimized, so the lower reconstruction error for AHT relative to HT is expected and should not be counted as independent evidence. The ATE column in Table 1 is the informative comparison.","section":"Section 5, Eq. (12)"},{"comment":"The use of ||.||_i and ||.||_j with subscript indices is confusing because the same notation usually denotes a norm or a norm with exponent; the paper defines this convention but should consider a clearer notation such as superscripts or explicit loss variables e_i^{(T)}.","section":"Section 4, notation"},{"comment":"The Malaga evaluation is qualitative because no ground-truth poses exist; the caption in Figure 9 should state this limitation directly in the figure and not rely only on the main text.","section":"Section 7.1, Malaga test"},{"comment":"There are typographical errors such as 'Knowledege Distillation' in the text before Section 3 and inconsistent spacing in equations; the manuscript would benefit from a careful proofread.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a worthwhile problem. The main risks are technical: the normalization typo and the lack of robustness/statistical analysis. I would not treat the 'first work' claim as a blocking issue, but I would ask the authors to verify that no recent regression-distillation works are omitted. If code is not released, the single-run results on KITTI should be interpreted with caution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and, on the evidence here, mostly works: instead of making a student imitate a teacher's pose predictions unconditionally, weight the imitation loss by the teacher's own error against ground truth. That is the attentive imitation loss (AIL), and the same weight is applied to hint training (AHT). The novelty relative to Hinton, FitNets, and Chen's object-detection distillation is real: none of them use a per-sample teacher-loss confidence for regression distillation, and the paper makes a reasonable case that pose regression lacks the \"dark knowledge\" that makes classification KD work.\n\nThe internal ablations are the strongest part. Table 1 shows AIL alone beats student-only and HT-only, and AHT adds a further gain. The comparison in Table 3 places the method ahead of KD, FitNets, and Chen's upper-bound loss at 92.95% parameter reduction. The KITTI/Malaga evaluation is standard, and the observation that a distilled student can beat a subsampled teacher at equal compute is practically useful.\n\nThe soft spots are real but mostly addressable. All results are single runs with no error bars; alpha and beta are never reported; the paper references a supplementary experiment that is missing from this version; and there is no code. More concerning is the stress-test's point about Eq. (9)-(11). Phi_i = 1 - e_i / eta with eta = max(eT)-min(eT). If min(eT) > 0, the worst samples get negative weights, which the text does not discuss. And a single outlier teacher loss inflates eta and flattens all weights toward one, effectively switching off the attention. The paper never reports the distribution of Phi, the fraction of near-zero or negative weights, or any sensitivity check. So the mechanism is not yet fully established; the Table 3 gap could in principle come from the two-stage schedule or the hint-layer choice. That said, Table 1's stepwise ablation—same schedule, differing only in AIL/AHT—does suggest the confidence weighting itself is doing something. I would not call the central claim circular; the weights depend on the teacher, not the student.\n\nThis paper is for people working on model compression for regression, especially VO/SLAM. It deserves serious reviewing: the idea is portable, the experimental comparison is mostly fair, and the weaknesses are fixable with more runs, reported hyperparameters, and a robustness analysis of the normalization. I would send it out.","headline":"Teacher-loss-weighted distillation for pose regression is a genuinely new and plausible idea; the paper needs more runs and a robustness check on the confidence normalization before I'd trust the exact numbers.","tokens_in":14784,"tokens_out":2103,"would_cite":true,"duration_ms":20019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A student network with 92.95% fewer parameters can stay close to a deep pose-regressor teacher when the imitation loss is weighted by the teacher's own training loss as a confidence score.","keywords":["knowledge distillation","pose regression","visual odometry","attentive imitation loss","attentive hint training","teacher confidence weighting","model compression","KITTI"],"falsifier":"Train the same 92.95%-reduced student on KITTI with the per-sample confidence weights $\\Phi_i$ randomly permuted across frames while preserving their distribution; if the permuted student matches the attentive student's 29 m ATE, then the teacher-loss signal itself is not the cause of the improvement.","tokens_in":13663,"feed_emoji":"🧭","tokens_out":8663,"duration_ms":84576,"temperature":0.7,"pith_summary":"Knowledge distillation works well for image classification because softened teacher outputs carry 'dark knowledge' about class similarities; regression has no such signal, so it has been unclear how a small student could learn from a large pose-regression teacher. This paper claims that the gap can be closed by treating the teacher's own error against ground truth as a confidence score: the student imitates the teacher only where the teacher is known to be accurate, and ignores unreliable teacher frames. Concretely, an attentive imitation loss and an attentive hint loss, both weighted by the normalized teacher loss, let a student with 92.95% fewer parameters stay close to the teacher on KITTI (29.0 m absolute trajectory error vs 26.7 m for the teacher) while running 2.12 times faster. The same recipe beats standard distillation, teacher-as-upper-bound, and hint-training baselines, and generalizes from KITTI training to Malaga testing. The underlying claim is that confidence-weighted imitation, not dark knowledge, is what carries knowledge transfer in regression.","feed_headline":"Teacher loss as trust score lets tiny pose network match big one","feed_subtitle":"Weighting imitation by teacher loss keeps a 93%-smaller model near the teacher: 29 m vs 27 m trajectory error.","key_machinery":"The load-bearing object is the normalized teacher loss $\\Phi_i$, a scalar confidence score computed once from the teacher's errors across the whole training set. In Attentive Imitation Loss it multiplies the imitation term $\\lVert p_S - p_T\\rVert^2_i$ so that frames where the teacher errs are down-weighted; in Attentive Hint Training the same $\\Phi_i$ multiplies the L2 distance between teacher and student intermediate representations. The two-stage training procedure first fits the student's intermediate representation with Attentive Hint Training, then trains the remaining layers with Attentive Imitation Loss; the paper reports that this separate schedule gives lower absolute pose error than training both objectives jointly. The normalization $\\eta = \\max(e_T) - \\min(e_T)$ is computed separately for translation and rotation because their teacher-loss distributions differ.","core_discovery":"The central discovery is that the teacher's loss on the training set, normalized to a confidence score by $\\Phi_i = 1 - \\lVert p_T - p_{gt}\\rVert^2_i / \\eta$ with $\\eta = \\max(e_T) - \\min(e_T)$, is an effective per-sample trust weight for distilling a pose regressor. Using $\\Phi_i$ to scale the imitation loss (Attentive Imitation Loss) tells the student to copy teacher predictions mostly where the teacher is close to ground truth, and using the same $\\Phi_i$ to scale the hint loss (Attentive Hint Training) aligns the student's intermediate representation with the teacher's only on trusted frames. With this mechanism, the paper reports that a student retaining only 7.05% of the teacher's parameters keeps absolute trajectory error close to the teacher's (29.03 m vs 26.74 m) and runs 2.12 times faster. The attentive weighting is what separates the method from three baselines in the same setup: unweighted imitation gives 165.2 m ATE, teacher-as-upper-bound gives 46.2 m, and standard hint training gives 32.0 m, against the proposed method's 29.0 m.","pith_inferences":["The authors do not explore it, but the same teacher-loss confidence weighting could be applied to other regression distillation targets with heteroscedastic errors, such as monocular depth or optical flow, where a teacher's per-pixel error varies strongly with scene content.","The min-max normalization is sensitive to outliers: a single catastrophic teacher loss on one training frame would flatten all confidence weights toward 1, silently reducing attentive learning to unweighted imitation, so rank-normalizing or clipping $\\Phi_i$ is a testable robustness improvement.","The results imply that an over-parameterized teacher may be most valuable as a source of reliable training signal rather than as a final predictor, since the distilled student matches or outperforms the supervised teacher on Malaga despite having far fewer parameters."],"forward_implications":["Regression networks can be distilled without dark knowledge: normalized teacher loss is sufficient to transfer pose-estimation ability to a student with 92.95% fewer parameters.","A compressed visual odometry student can run at more than double the frame rate (41 ms vs 87 ms per frame) while staying close to the teacher's accuracy on KITTI, and at some compression ratios it beats the supervised teacher's absolute trajectory error.","The attentive weighting is doing the work: under matched conditions, unweighted imitation gives 165.2 m ATE, teacher-as-upper-bound gives 46.2 m, and standard hint training gives 32.0 m, against 29.0 m for the proposed method.","Training the intermediate representation and the final output in two separate stages, both weighted by teacher loss, yields lower absolute pose error than combining both objectives in one joint loss, and the resulting model generalizes to a different dataset (Malaga) without retraining."],"supporting_citations":[{"why":"Establishes knowledge distillation and dark knowledge; the paper argues dark knowledge is absent in regression and provides the unweighted KD baseline it must outperform.","marker":"[16]"},{"why":"Defines hint training for intermediate representations; the paper's Attentive Hint Training modifies it with teacher-loss weights and uses it as a baseline.","marker":"[25]"},{"why":"Introduces teacher-loss-as-upper-bound distillation for object detection; the paper adapts it for pose regression and compares against it.","marker":"[4]"},{"why":"Provides the teacher network architecture whose layers are removed to build the student.","marker":"[34]"},{"why":"Provides the recurrent pose-regression architecture and LSTM-based 6-DoF pose learning that the teacher and student are built on.","marker":"[33]"},{"why":"Provides the KITTI odometry benchmark used for training and testing the teacher and student models.","marker":"[8]"},{"why":"Provides the Malaga dataset used to test cross-dataset generalization of the distilled model.","marker":"[3]"}],"fun_headline_variants":["Teacher loss weights when to copy for pose distillation","Trust teacher only on confident frames for pose distillation","Weighted imitation keeps tiny pose net near big one","Pose distillation via teacher-loss trust weighting","Teacher's own loss steers which frames to imitate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the teacher's errors measured on the training set tell which teacher predictions are trustworthy, and that those trust weights remain valid on the test sequences the student will later see.","fun_headline_variants_meta":{"raw":{"variants":["Teacher loss weights when to copy for pose distillation","Trust teacher only on confident frames for pose distillation","Weighted imitation keeps tiny pose net near big one","Pose distillation via teacher-loss trust weighting","Teacher's own loss steers which frames to imitate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2956,"prompt_tokens":976,"completion_tokens":1980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1908}},"tokens_in":592,"tokens_out":1980,"duration_ms":14581,"temperature":1.0,"reasoning_tokens":1908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:29:57.030122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 92.95%-reduced student on KITTI with the per-sample confidence weights $\\Phi_i$ randomly permuted across frames while preserving their distribution; if the permuted student matches the attentive student's 29 m ATE, then the teacher-loss signal itself is not the cause of the improvement.","supporting_citations":[{"cited_title":"Hinton, O","cited_arxiv_id":null,"evidence_quote":"Establishes knowledge distillation and dark knowledge; the paper argues dark knowledge is absent in regression and provides the unweighted KD baseline it must outperform."},{"cited_title":"Romero, N","cited_arxiv_id":null,"evidence_quote":"Defines hint training for intermediate representations; the paper's Attentive Hint Training modifies it with teacher-loss weights and uses it as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces teacher-loss-as-upper-bound distillation for object detection; the paper adapts it for pose regression and compares against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the teacher network architecture whose layers are removed to build the student."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the recurrent pose-regression architecture and LSTM-based 6-DoF pose learning that the teacher and student are built on."},{"cited_title":"Geiger, P","cited_arxiv_id":null,"evidence_quote":"Provides the KITTI odometry benchmark used for training and testing the teacher and student models."},{"cited_title":"Blanco-Claraco, F.- ´A","cited_arxiv_id":null,"evidence_quote":"Provides the Malaga dataset used to test cross-dataset generalization of the distilled model."}],"review_version":1}