{"id":"4eca3461-b068-448f-b7d3-52ef393fb577","arxiv_id":"2412.13236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"COSEE trains each early-exit classifier to emphasize samples that would exit at it under simulated thresholds, improving GLUE accuracy at matched speed-up over prior methods.","lead":"This paper proposes COSEE, a training method for early-exiting language models that reweights each sample's loss by where the sample would exit under simulated inference thresholds, and adds a loss that makes exiting signals more discriminative. If the reported GLUE results hold, it is a simple upgrade that improves accuracy at the same inference speed in BERT-style models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 compares methods at different speed-up ratios and with borrowed baselines, so the 'consistently outperforms' claim is not yet supported by the reported evidence.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper should not be accepted without additional experiments. However, the single most load-bearing concern is not primarily the SWM signal-calibration assumption highlighted in the reader's weakest_assumption field; it is the comparison protocol in Table 2. The headline claim is an empirical superiority claim, and the reported table does not support it because speed-up ratios differ across methods, baseline numbers are imported from other papers, and no uncertainty estimates are given. On RTE, COSEE is numerically below ConsistentEE; on MRPC, COSEE operates at a much higher speed-up than ConsistentEE, making direct accuracy comparison invalid. These are concrete, checkable weaknesses that directly undermine the abstract's 'consistently outperforms' statement. The signal-calibration concern is legitimate and worth testing, but it is secondary to the question of whether the experimental evidence actually demonstrates the claimed trade-off improvement. The proposed concrete test — a matched-speed-up re-run of all baselines under identical training conditions, with multiple seeds and accuracy-versus-speed-up curves — would settle the matter. If COSEE still dominates at matched speed-ups, the claim is supported; if not, the paper must be revised to a more modest claim. Since the reader already asked for exactly this kind of evidence and issued a CONDITIONAL verdict, my analysis does not change the recommended verdict.","tokens_in":13216,"tokens_out":3661,"duration_ms":34018,"concrete_test":"Re-run the primary baselines (at least DeeBERT, GPFEE, LeeBERT, BERxiT, DisentangledEE, and ConsistentEE) under the same fine-tuning protocol used for COSEE, with the same backbone, optimizer, sequence length, and training budget. On each GLUE task, tune each method's threshold to hit a common grid of speed-ups (e.g., 1.5x, 2.0x, 2.5x, 3.0x) and plot accuracy versus speed-up with mean +/- std over at least three seeds. Accept the headline claim only if COSEE's curve dominates or is statistically tied at equal speed-ups on the majority of tasks. If COSEE only wins at higher speed-ups but loses at matched speed-ups, the abstract's 'consistently outperforms' claim must be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that COSEE 'consistently outperforms the state-of-the-art methods across all tasks, yielding a better trade-off between performance and efficiency' (Abstract and Section 4.4) — rests on Table 2. In that table, speed-up ratios are not matched. COSEE is reported at 2.70x on MRPC versus 1.59x for ConsistentEE, at 2.56x on QNLI versus 1.72x for ConsistentEE, and at 2.14x on SST-2 versus 1.97x for LeeBERT. On RTE, COSEE is below ConsistentEE (68.7 at 1.96x versus 69.0 at 1.85x). Since higher speed-up normally trades away accuracy, a point better on one axis and worse on the other does not establish a better trade-off. The issue is compounded by Section 4.2: most baseline numbers are taken from their original papers, with different training settings, and no variance estimates or multiple seeds are reported. Thus the headline superiority is not established by the reported evidence, even if the COSEE training recipe itself is plausible. The theoretical concern about SWM relying on signal-difficulty calibration (Sections 3.2 and 3.3) is real but secondary: it can be evaluated once the comparison protocol is fixed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COSEE, a training-time modification of signal-based early exiting for pre-trained language models. At each training step, COSEE randomly samples K exiting thresholds, simulates test-time early exiting to obtain a pseudo exit layer m*_n per sample, and weights the cross-entropy loss of each internal classifier by a softmax over distance to that exit layer (Eq. 3). It adds an online signal calibration (OSC) loss (Eqs. 6-8) that pushes the exiting signals of easy and hard samples apart, and it adopts a normalized energy score (Eq. 11) as the default signal. Experiments on six GLUE tasks with BERT-base and ALBERT-base claim that COSEE consistently outperforms prior early-exit methods at similar speed-ups, supplemented by ablations, convergence curves, and failure-case statistics.","tokens_in":13434,"tokens_out":5588,"duration_ms":49991,"significance":"If the empirical claims held, COSEE would be a valuable and simple contribution: SWM is parameter-free, the method adds negligible parameters, it is evaluated across multiple exiting signals and backbones, and code is released. The paper also contains useful ablations (Figures 4-6) and a failure-case analysis (Appendix D). However, the headline claim of consistent state-of-the-art outperformance is not currently supported by the reported evidence because the comparison protocol in Table 2 does not match speed-up ratios or training setups across methods. With a properly matched evaluation, the method may well be competitive; the contribution's value depends on that empirical confirmation.","major_comments":[{"comment":"The claim that COSEE 'consistently outperforms the state-of-the-art methods across all tasks' is not supported because comparisons are made at different speed-up ratios. For example, on RTE COSEE scores 68.7 at 1.96x while ConsistentEE scores 69.0 at 1.85x; on MNLI both score 83.4 but COSEE is at 1.92x while ConsistentEE is at 1.45x; on QNLI COSEE's 90.2 at 2.56x is compared with ConsistentEE's 89.9 at 1.72x. Since accuracy generally declines as speed-up increases, a point that is better on one axis and worse on the other does not establish a superior trade-off. Please report results at matched speed-ups (e.g., tune every method to approximately 2.00x) or present full Pareto curves with error bars.","section":"Section 4.4 / Table 2"},{"comment":"Most baseline numbers are taken from the original papers (DeeBERT, PABEE, BERxiT, etc.) or from GPFEE and DisentangledEE, so training data, hyperparameters, and inference settings are not matched across methods. No multiple seeds or variance estimates are reported. This makes any 'better trade-off' claim fragile. Please rerun the baselines under the same protocol (same backbone checkpoint, same batch-size-1 inference, same seed) or, at minimum, clearly mark borrowed numbers and report variance for every method.","section":"Section 4.2 / Table 2"},{"comment":"The text states that thresholds are adjusted to achieve 'similar speed-up ratio ... approximately 2.00x (±38%)', but the speed-ups in Table 2 range from 1.45x to 2.86x, and COSEE's own ratios range from 1.92x to 2.70x. A ±38% tolerance does not constitute matched speed-up. Please report the threshold used for each method and task, and either enforce a tighter tolerance or analyze the trade-off with interpolation.","section":"Section 4.3"},{"comment":"Both SWM and OSC rely on the exiting signal's ability to rank sample difficulty. The paper acknowledges that early in training signals may not reflect difficulty, and the easy/hard partition is based on the internal classifier's own correctness, which can be noisy. The warm-up in Eq. (4) and the OSC objective are patches, not guarantees. The current evidence that they work is limited: Figure 5 and Appendix D are shown for SST-2 (and QNLI) only, without confidence intervals or comparison to alternative calibration strategies. Please provide quantitative evidence of signal reliability across all tasks, e.g., DIS curves and premature/delayed exiting rates at matched speed-ups, to show that the weighting direction is not sometimes reversed.","section":"Section 3.3 / Eq. (7)"}],"minor_comments":[{"comment":"The caption says 'Best results are marked in bold', but the bold row appears to be COSEE throughout; on RTE and MNLI, COSEE is not the best. Please correct either the bold marking or the caption.","section":"Table 2 caption"},{"comment":"The phrase 'consistently outperforms the state-of-the-art methods' should be softened to 'is competitive with' or 'often outperforms', given the unmatched comparisons in Table 2.","section":"Abstract and Section 4.4"},{"comment":"The ALBERT comparison has the same unmatched speed-up issue and uses borrowed baselines; please add matched-speed-up comparisons or explicitly state the limitation.","section":"Table 3"},{"comment":"Please clarify the monotonic direction of the normalized energy score relative to sample difficulty; currently a reader must infer it from Figure 3.","section":"Eq. (11)"},{"comment":"The text 'objectiveLossOSC' should be rendered as 'objective Loss_OSC' with proper math formatting.","section":"Figure 2 caption"},{"comment":"Parameter sensitivity is shown only on SST-2 and QNLI; please state whether the chosen defaults generalize to other tasks or provide sensitivity results on at least one additional task.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The contribution is plausible and clearly written, but the empirical protocol needs substantial strengthening. The most important issue is the unmatched speed-up comparison and the reliance on borrowed baseline numbers, not the theoretical framing. I would be willing to review a revised version that provides matched-speed-up comparisons and variance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about COSEE. First, the training recipe is genuinely new and coherent: it softly weights each sample's loss across classifiers based on a simulated exiting layer under multiple thresholds, plus an online signal-calibration loss. This is a sensible extension of ConsistentEE's one-hot router idea, and it targets a real gap—signal-based early exiters have no train/test consistency. Second, the evidence that it 'consistently outperforms' the state of the art is not there yet, because Table 2 compares at different speed-up ratios and mostly against numbers borrowed from other papers.\n\nWhat the paper does well: the ablations show both SWM and OSC help, especially at high speed-up. The generality checks with entropy/softmax signals and ALBERT are useful. The authors acknowledge the early-training calibration problem and patch it with warm-up and OSC. The method is simple, parameter-free for SWM, and the code is released.\n\nSoft spots, in order of importance. The comparison protocol is the big one. On MRPC, COSEE is reported at 2.70x speed-up versus 1.59x for ConsistentEE; on QNLI, 2.56x versus 1.72x; on SST-2, 2.14x versus 1.97x. Higher speed-up usually trades away accuracy, so a point that is better on one axis and worse on the other does not establish a better trade-off. RTE is the clearest counterexample: COSEE gets 68.7 at 1.96x, ConsistentEE gets 69.0 at 1.85x. Second, most baselines are taken from the original papers, under different training settings, with no error bars or multiple seeds. The abstract's 'consistently outperforms' is too strong given Table 2. Third, the circularity in SWM is real but secondary: the weights inherit the signal's mistakes early in training, and warm-up/OSC are patches, not guarantees. For an empirical training method that is acceptable, but it means the story rests on the experiments, and the experiments need to be cleaned up.\n\nWho is this for? Anyone working on early exiting for PLMs. It deserves a serious referee—the method is plausible and the flaws are fixable. The authors should be asked to re-run baselines under matched speed-up, report variance, and rewrite the abstract to match the evidence. If they do, this becomes a useful contribution.","headline":"COSEE's training recipe is worth a look, but its 'consistently outperforms' claim is not supported by the unmatched speed-up comparisons and borrowed baselines in Table 2.","tokens_in":13995,"tokens_out":3320,"would_cite":true,"duration_ms":27057,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COSEE claims that training each early-exit classifier on the samples that will actually exit there, simulated across multiple thresholds, closes the train-test inconsistency that limits signal-based early exiters.","keywords":["early exiting","pre-trained language models","sample weighting","training-testing consistency","energy score","GLUE benchmark","inference efficiency","multi-exit networks"],"falsifier":"Replay a trained COSEE model's training-time simulation: for a fixed threshold, record the $m^*_n$ that SWM would have computed for each development sample from stored per-layer signals, compare it with the layer where that sample actually exits at inference under the same threshold, and compute the disagreement rate; a large disagreement rate would mean the claimed train-test consistency is not achieved.","tokens_in":12997,"feed_emoji":"⚡","tokens_out":6024,"duration_ms":55033,"temperature":0.7,"pith_summary":"COSEE is trying to establish that the main bottleneck in signal-based early exiting for pre-trained language models is a mismatch between training and inference: during training every internal classifier treats every sample equally, but at test time easy samples exit early and hard samples travel deeper. The paper's proposed fix is a calibrated sample weighting mechanism that, at each training step, simulates the test-time exiting process under several randomly chosen thresholds, finds where each sample would exit, and up-weights that sample's loss mainly for classifiers near its exit layer. An online signal calibration objective and a normalized energy score make the exiting signal itself more discriminative, so the simulated exits are trustworthy. If the claim is right, a nearly parameter-free change to the training loss improves accuracy at matched speed-up ratios and keeps the ability to adjust the speed-up ratio after training.","feed_headline":"Sample weighting closes the train-test gap in early exiting","feed_subtitle":"COSEE makes each layer's classifier focus on the samples that exit there, improving GLUE accuracy at about 2x fewer layers.","key_machinery":"The load-bearing mechanism is the sample weighting mechanism built on simulated exit layers. At each training step the model runs each sample through all layers, reads off an exiting signal, draws $K$ thresholds, and computes $m^*_n$ from the current early-exit rule; the loss weight for classifier $m$ is $w_n^{(m)} = e^{-\\beta_t |m - m^*_n|} / \\sum_{m'} e^{-\\beta_t |m' - m^*_n|}$, with $\\beta_t$ warmed up from $\\beta_0$ so unstable early-training exits do not distort the weights. Around it, the online signal calibration objective $\\max(0, S_{\\text{easy}} - S_{\\text{hard}} + \\epsilon)$ explicitly widens the gap between easy and hard samples' exiting signals, and the normalized energy score $E_{\\text{norm}} = (1 + e^{-E})^{-1}$ aligns energy distributions across layers so a single threshold is meaningful everywhere.","core_discovery":"On the paper's own terms, COSEE establishes that consistency between training and testing can be achieved without sacrificing flexible speed-up. The training objective is the mean over $K$ randomly sampled thresholds of a sample-weighted classification loss, where the weight of sample $n$ at classifier $m$ is a soft exponential function of $|m - m^*_n|$ and $m^*_n$ is the layer at which that sample would exit under the current exiting signal at threshold $\\tau$. Because the weights are soft rather than one-hot, the model generalizes to thresholds not seen in training. The paper reports that on the GLUE benchmark with BERT-base this yields a better performance-efficiency trade-off than existing signal-based and router-based early exiters, sometimes matching or beating the full BERT-base at roughly $2\\times$ speed-up, and that the same framework transfers to ALBERT and to entropy and softmax signals.","pith_inferences":["Extension: The weighting scheme is a continuum between uniform training and router-based one-hot training, controlled by $\\beta_t$; a natural extension the paper does not explore is scheduling or adapting $\\beta$ per classifier or per task.","Extension: If the simulated exit layer is a valid difficulty proxy, the same threshold-simulation loss could be applied to other multi-exit architectures beyond text, for example vision or speech models.","Extension: The paper's own premature and delayed exiting rate statistics suggest an immediate diagnostic: those rates could be monitored during training and used to choose $\\beta_0$ or to stop training early, since COSEE claims to reduce both.","Extension: Because COSEE depends on the signal ranking samples correctly, a direct stress test would be to permute the signal values during training; if the gains persist, the weighting mechanism is not actually carrying the result."],"forward_implications":["At matched speed-ups of roughly $2.00\\times$, COSEE reports higher accuracy than DeeBERT, PABEE, BERxiT, LeeBERT, GPFEE, GAML-BERT, PALBERT, DisentangledEE, and ConsistentEE on the GLUE tasks, with especially clear gains at high speed-up.","Because training averages over $K$ thresholds, the trained multi-exit network can later be operated at any threshold: the acceleration ratio is adjusted at inference time, not fixed by the training router.","The OSC objective raises the difficulty-inversion score of the exiting signal, so exit decisions and the weights that depend on them are driven by a more reliable difficulty proxy.","Switching to SWM alone speeds convergence because each classifier concentrates on a difficulty slice of the data, while COSEE adds only classifier parameters, under 0.03% of BERT-base.","COSEE reports lower premature and delayed exiting rates than conventional training, meaning the resulting exit decisions are more reliable under strict speed-up settings."],"supporting_citations":[{"why":"Introduces DeeBERT, the signal-based early-exiting baseline and the saved-layers speed-up metric that COSEE builds on and compares against.","marker":"Xin et al. 2020"},{"why":"Provides the PABEE baseline and represents the weighted-sum training objective that COSEE contrasts with equal treatment of all samples.","marker":"Zhou et al. 2020"},{"why":"Provides the GPFEE baseline and supplies replicated results plus the training and evaluation settings used for the GLUE comparison.","marker":"Liao et al. 2021"},{"why":"Presents ConsistentEE, the router-based early exiter whose fixed speed-up limitation motivates COSEE's threshold-averaged design.","marker":"Zeng et al. 2024"},{"why":"Supplies the energy-based exiting signal and the theoretical grounding that energy separates easy from hard samples, which COSEE adapts into its normalized energy score.","marker":"Akbari, Banitalebi-Dehkordi, and Zhang 2022"},{"why":"Defines the difficulty inversion score used to evaluate whether COSEE's exiting signals are more discriminative.","marker":"Li et al. 2021"},{"why":"Defines the GLUE benchmark datasets used in all experiments.","marker":"Wang et al. 2019"},{"why":"Provides BERT-base, the backbone that all main comparisons are built on.","marker":"Devlin et al. 2019"}],"fun_headline_variants":["Consistent early exit without fixed speed-up ratios","Calibrated weights for early exit that matches training","COSEE: aligning early-exit training with inference","Flexible early exit with training-testing consistency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole weighting scheme rests on the exiting signal being a reliable ranking of sample difficulty from the start of training: the simulated exit layer that sets every sample's weights is read off that signal, and if the signal is miscalibrated early, the weights train classifiers to emphasize the wrong samples.","fun_headline_variants_meta":{"raw":{"variants":["Consistent early exit without fixed speed-up ratios","Calibrated weights for early exit that matches training","COSEE: aligning early-exit training with inference","Flexible early exit with training-testing consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2465,"prompt_tokens":902,"completion_tokens":1563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1502}},"tokens_in":518,"tokens_out":1563,"duration_ms":12145,"temperature":1.0,"reasoning_tokens":1502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:27:24.031502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay a trained COSEE model's training-time simulation: for a fixed threshold, record the $m^*_n$ that SWM would have computed for each development sample from stored per-layer signals, compare it with the layer where that sample actually exits at inference under the same threshold, and compute the disagreement rate; a large disagreement rate would mean the claimed train-test consistency is not achieved.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces DeeBERT, the signal-based early-exiting baseline and the saved-layers speed-up metric that COSEE builds on and compares against."},{"cited_title":"J.; Xu, K.; and Wei, F","cited_arxiv_id":null,"evidence_quote":"Provides the PABEE baseline and represents the weighted-sum training objective that COSEE contrasts with equal treatment of all samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPFEE baseline and supplies replicated results plus the training and evaluation settings used for the GLUE comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents ConsistentEE, the router-based early exiter whose fixed speed-up limitation motivates COSEE's threshold-averaged design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GLUE benchmark datasets used in all experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT-base, the backbone that all main comparisons are built on."}],"review_version":1}