{"id":"28fbad92-5fcd-4489-9443-3118e274371a","arxiv_id":"2509.05385","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SAGE reports 97-99% exact-match accuracy on GSM8K level-1 by triggering LoRA fine-tuning on flagged samples, but the flagging uses ground-truth answers and the final evaluation is not held out.","lead":"SAGE is a framework that tries to let a large language model improve itself while answering questions: it flags suspicious answers, clusters them, and fine-tunes small LoRA adapters on the clusters. The framework's detection step compares the model's answer with the gold answer, so it needs the answer key at test time, and the main result is measured on the same data used for fine-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SAGE's Trigger cannot start without gold answers at inference, so the claimed 'self-adaptation without supervision' is unsupported; the end-to-end evaluation is label-retrospective online fine-tuning, not inference-time self-adaptation.","rationale":"The reader's weakest assumption is exactly the label-availability premise: the Trigger compares predictions with gold answers, so without labels no failures are detected and the adaptation loop never starts. My stress-test confirms this and sharpens it by noting the internal contradiction with the paper's 'without additional supervision' and 'Adaptation autonomy' requirements. The end-to-end evaluation on GSM8K Level-1 uses the gold answer in the Trigger, making the 97.16% ± 4.65% result an oracle-supervised online fine-tuning result, not evidence for self-adaptation during inference. The manuscript contains no code, no released data, and no formal verification, so there is no independent support for the central claim. Because the central claim fails exactly where the reader's verdict says it fails, the REJECT verdict stands without modification. I find no reason to move the verdict in either direction.","tokens_in":12803,"tokens_out":3178,"duration_ms":30787,"concrete_test":"Withhold gold answers from the Trigger's comparison path while keeping the rest of SAGE unchanged, then run the end-to-end GSM8K Level-1 protocol across the three random seeds of Table 4: feed the model only the question and prompt, compute the anomaly score without BLEU/ROUGE/embedding similarities to a reference answer (using only logits margin or a self-consistency proxy), and record the number of triggered samples and final EM/MSE. If triggers are near zero or final EM stays within noise of the Base Model's 2.22%, the label-availability premise is confirmed as load-bearing. A complementary analytical check is to remove s2, s3, and s4 from Eq. (5) and show that the remaining trigger produces no stable clusters on OOD data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is label availability at inference time. In the Trigger module, the paper states: 'given a test set, the LLM first receives a masked version of the gold answer, which is retained as the ground truth. It then generates an answer based on the question ... The Trigger module compares the ground truth with the predicted answer at the token level.' The anomaly score in Eq. (5) integrates BLEU, ROUGE-L, and embedding similarity, all computed against the ground-truth answer. In a genuine deployment stream, gold answers are not available, so the Trigger cannot flag reasoning failures, the Trigger Buffer receives no anomalous samples, and no LoRA adapter can be trained. This contradicts the paper's repeated claims of 'real-time' detection 'without additional supervision' and the stated requirement of 'Adaptation autonomy ... without external supervision.' The end-to-end GSM8K Level-1 evaluation supplies the gold answer to the Trigger, so it demonstrates supervised online fine-tuning with oracle failure detection, not autonomous test-time self-adaptation. The conclusion's acknowledged limitation about 'dataset alignment' does not address this label dependency. This is an internal inconsistency in the central argument, not merely a disagreement with current consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAGE, a three-module framework for test-time self-adaptation of LLMs: a Trigger module that detects reasoning failures from multiple evaluation metrics, a Trigger Buffer that clusters anomalous samples, and a LoRA Store that fine-tunes and retains adapters. The manuscript claims that SAGE enables dynamic knowledge updating during inference, reporting 97.16% ± 4.65% EM accuracy on GSM8K Level-1, and includes module-wise evaluations and ablations.","tokens_in":13211,"tokens_out":2784,"duration_ms":24770,"significance":"If the central claim were valid, SAGE would be a practical advance in continual and test-time adaptation, since a deployed LLM could detect its own failures and improve with lightweight LoRA updates. The paper provides a detailed system description and several ablation comparisons. However, the key claim is not established by the current evidence: the failure signal used by the Trigger is the gold answer, and the end-to-end evaluation is conducted on the same samples used to train the adapters. The reported results therefore demonstrate supervised online fine-tuning with oracle failure detection, not autonomous test-time self-adaptation.","major_comments":[{"comment":"Equation (5) defines the anomaly score AS as a weighted sum of one minus normalized similarity terms, where BLEU, ROUGE-L, and embedding similarity are computed against the ground-truth answer. The text also states that the LLM 'first receives a masked version of the gold answer, which is retained as the ground truth.' Consequently, the Trigger cannot operate in a deployment stream without gold answers, contradicting the abstract's claim of 'real-time' detection and the Preliminaries' requirement that adaptation run 'without external supervision.' The end-to-end experiment therefore demonstrates supervised online fine-tuning with oracle failure detection, not autonomous test-time self-adaptation.","section":"Design of SAGE, Trigger"},{"comment":"The final EM accuracy is reported on the same GSM8K Level-1 samples whose anomalies were clustered and used to train the LoRA adapters (Tables 2 and 3). This is not a held-out evaluation: the adapters are selected by validation accuracy on clusters derived from the test set, and the best of Top-3 adapters is reported for each seed. The headline '97.16% ± 4.65%' is the mean over 9 adapters from three seeds, with the best adapter per seed retained, which inflates the estimate. A held-out split is needed before any claim of generalization can be made.","section":"End-to-End Evaluation of SAGE"},{"comment":"The Wilcoxon signed-rank test (W=0, p=0.0039) compares SAGE against baselines on the same non-held-out data, so it does not establish statistical significance of a generalization improvement. The test also appears to use the post-hoc selected adapter; the selection procedure is not accounted for in the p-value, making the significance claim uninterpretable.","section":"End-to-End Evaluation of SAGE"}],"minor_comments":[{"comment":"The text refers to 'Appendix B.2' for random-seed stability, but the referenced results appear in Appendix Table 4; the in-text citation is ambiguous.","section":"Appendix B.2"},{"comment":"There is an unresolved LaTeX reference in the text: 'Figure reffig:tri-sj' in Appendix B.1.","section":"Appendix B.1"},{"comment":"The symbol '*' in Table 1 is explained only in the caption, not in the main text, and the notation 'LStore500', 'LStore499', and 'SAGE315' is not defined anywhere in the paper.","section":"Table 1"},{"comment":"The paper alternates between 'Lora Store' and 'LoRA Store' (e.g., section headers and the Table 2 caption), and there are typographical errors such as 'unclusterd' and 'Lora Store' in the LoRA Store section.","section":"Throughout"},{"comment":"Many numeric values appear with formatting issues (e.g., '107', '4811.8×10 6', '291851.5×10 10'), which makes the tables difficult to interpret.","section":"Tables 1-3"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The central claim is not supported by the evidence: the Trigger requires gold answers at inference time, and the end-to-end evaluation is not held out. These are load-bearing issues that cannot be fixed by a local revision, as the method as described depends on label availability and the reported accuracy is measured on training data. A major reframing as supervised online fine-tuning with oracle detection would be needed, which would constitute a different contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: the paper builds a modular, trigger-guided LoRA adaptation system and describes the components carefully, but the central claim—that SAGE enables autonomous self-adaptation at inference time—is not supported. The trigger compares predictions against gold answers, so without labels the system never starts. The end-to-end numbers come from training adapters on the same GSM8K level-1 samples used for evaluation, with best-adapter selection among the Top-3 and seed averaging. That makes the reported 97.16% EM an artifact of the setup, not evidence of test-time adaptation.\n\nWhat's genuinely new is the composition: detecting failures, clustering them with HDBSCAN plus stability checks, and maintaining a pool of LoRA adapters with hyperparameter search. The component-level evaluations are not without merit. The trigger separates ID/OOD samples well under the chosen threshold, and the clustering module shows sensible behavior on streaming data. The LoRA store's heatmaps do show that rank and learning rate matter. So the engineering is plausible and the writeup is clear. I'm not saying the authors are sloppy—just that the evaluation doesn't test what the abstract promises.\n\nThe load-bearing problem is in the Trigger module. Eq. (5) is a weighted sum of one minus similarity to the ground truth. Detection is defined as disagreement with the label. The paper claims 'without additional supervision' and 'self-adaptation,' but this is label-retrospective online fine-tuning. The conclusion's note about 'dataset alignment' doesn't address that the trigger requires gold answers at inference. That's an internal inconsistency, not a missing experiment.\n\nThere are also smaller issues: several thresholds (τ, T, η_ARI, η_cos, δ) are not given concrete values, no code or data is released, and some baselines are weak (the LoRA baseline on GSM8K is near zero). The ablation omits the Trigger and LoRA Store modules with a rationale that doesn't actually test their contribution to the end-to-end claim.\n\nWho is this for? If you are working on test-time adaptation and want a cautionary example of how easy it is to accidentally use labels at inference, this paper is useful. As a positive result, it doesn't hold up. For peer review, I'd send it out—the problem is relevant and the framework is clearly described—but I'd signal that a major revision is needed: either reframe as label-retrospective fine-tuning or design a trigger that works without gold answers, and redo the evaluation with a proper held-out split.","headline":"A clearly-described LoRA self-adaptation pipeline whose central test-time claim is undercut by the trigger's need for gold answers and an evaluation that trains and tests on the same data.","tokens_in":13602,"tokens_out":3002,"would_cite":false,"duration_ms":26195,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAGE lets LLMs detect and fix their own reasoning failures at inference time.","keywords":["LLM self-adaptation","test-time training","Low-Rank Adaptation (LoRA)","trigger-guided fine-tuning","streaming clustering","HDBSCAN","atomic reasoning subtasks","anomaly detection"],"falsifier":"Run SAGE on a held-out stream where gold answers are withheld or provided only for a small fraction of samples, and show that the Trigger fires on almost nothing and the reported EM gain disappears; a positive control would use the same stream with gold answers available, confirming that the gain comes from the label-dependent trigger.","tokens_in":12583,"feed_emoji":"⚙️","tokens_out":6946,"duration_ms":47099,"temperature":0.7,"pith_summary":"Large language models typically freeze after training, so they cannot absorb new knowledge while reasoning. This paper argues that complex reasoning can be decomposed into atomic subtasks, and proposes SAGE, a trigger-guided framework in which a model spots its own reasoning failures, clusters the failed samples, and fine-tunes lightweight LoRA adapters at test time. On a selected GSM8K level-1 subtask, SAGE reports exact-match accuracy rising from 2.22% for the base model to a mean of 97.16% ± 4.65% across random seeds, with a Wilcoxon signed-rank test giving p = 0.0039. If this holds, deployed LLMs could self-improve from their own inference mistakes without full retraining.","feed_headline":"SAGE lets LLMs hit 97% exact-match by self-adapting mid-reasoning","feed_subtitle":"SAGE detects a model's reasoning failures, clusters them, and fine-tunes LoRA adapters on the fly without retraining.","key_machinery":"The central object is SAGE's three-module loop: trigger, buffer, store. The Trigger turns each reasoning step into a supervised signal by comparing the model's output with the retained gold answer (masked at input, unmasked for comparison), producing a normalized anomaly score. The Trigger Buffer aggregates flagged samples into stable clusters via HDBSCAN plus stability metrics (Adjusted Rand Index and average cosine similarity) and merges clusters whose centroids are close. The LoRA Store runs Cluster-Aware LoRA Optimization (CLO), which samples LoRA ranks and learning rates, trains candidate adapters, sorts them by accuracy then cross-entropy loss, and retains the Top-3 adapters for future inference.","core_discovery":"On its own terms, SAGE establishes that a static LLM can become a self-adapting system during inference by chaining three components: a Trigger that flags reasoning failures by comparing the model's prediction with a masked gold answer using logits margin, BLEU, ROUGE-L, and embedding similarity; a Trigger Buffer that clusters the flagged samples with streaming HDBSCAN, stability checks, and similarity-based merging; and a LoRA Store that searches over hyperparameters, trains candidate adapters, ranks them by accuracy and loss, and keeps the best three for reuse. The paper's central experimental claim is that on the GSM8K level-1 atomic subtask, SAGE's best adapters reach 99.80% exact match, with a mean of 97.16% ± 4.65% over random seeds and a statistically significant improvement over static baselines.","pith_inferences":["The label-availability premise is the key boundary: the Trigger needs the gold answer at inference time to compute its anomaly score, so in a fully unsupervised stream the pipeline would never fire; a version that replaces this with self-supervised signals (e.g., consistency across decoding attempts, or verification of arithmetic steps) would be a natural testable extension.","If the reported 97% EM holds on broader OOD datasets, the framework suggests a modular alternative to RL-based post-training: instead of one global policy update, a library of small adapters could be grown incrementally, each specialized to a cluster of failures.","The clustering step implicitly assumes that similar failures share a cause; one could test this by measuring whether adapter performance transfers across clusters that are similar by embedding but differ in underlying error type."],"forward_implications":["Deployed LLMs could update themselves during test-time interaction, incorporating new knowledge without full retraining and without modifying the base weights.","Decomposing a task into atomic subtasks makes LoRA fine-tuning effective on sparse, streaming data; the paper reports more than 50% improvement over multitask fine-tuning on atomic tasks.","Failure-triggered, event-driven learning converts static LoRA updates into a continual process that only fires when the model detects an anomaly.","Top-3 adapters retained in the LoRA Store provide reusable, task-specific modules that can be selected for future reasoning calls."],"supporting_citations":[{"why":"Supplies the low-rank adaptation method that all of SAGE's dynamic fine-tuning is built on.","marker":"Hu et al. 2022"},{"why":"Provides the HDBSCAN density-based clustering algorithm used in the Trigger Buffer's streaming clustering.","marker":"Campello, Moulavi, and Sander 2013"},{"why":"Defines BLEU, one of the trigger's surface-level comparison metrics.","marker":"Papineni et al. 2002"},{"why":"Defines ROUGE-L, the structural comparison metric used in the trigger.","marker":"Lin 2004"},{"why":"Provides the sentence-embedding similarity used as the trigger's semantic metric.","marker":"Lin et al. 2017"},{"why":"Defines the logits margin used to measure the model's confidence in next-token prediction.","marker":"Liu et al. 2016"}],"fun_headline_variants":["SAGE self-adapts LLMs mid-reasoning with triggered LoRA updates","Trigger-guided LoRA lets LLMs learn during inference","SAGE: dynamic LoRA tuning during reasoning hits 99.8% exact match","LLMs self-update on the fly via SAGE's trigger-cluster-LoRA pipeline","SAGE: adapt LLMs at inference time without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that the correct answer is available for every test-time sample, because the Trigger detects failures by comparing the generated answer with a masked gold answer; if a real inference stream lacks these labels, no anomalies can be flagged and the self-adaptation loop never starts.","fun_headline_variants_meta":{"raw":{"variants":["SAGE self-adapts LLMs mid-reasoning with triggered LoRA updates","Trigger-guided LoRA lets LLMs learn during inference","SAGE: dynamic LoRA tuning during reasoning hits 99.8% exact match","LLMs self-update on the fly via SAGE's trigger-cluster-LoRA pipeline","SAGE: adapt LLMs at inference time without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2466,"prompt_tokens":876,"completion_tokens":1590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1489}},"tokens_in":492,"tokens_out":1590,"duration_ms":9003,"temperature":1.0,"reasoning_tokens":1489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:27:22.277545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAGE on a held-out stream where gold answers are withheld or provided only for a small fraction of samples, and show that the Trigger fires on almost nothing and the reported EM gain disappears; a positive control would use the same stream with gold answers available, confirming that the gain comes from the label-dependent trigger.","supporting_citations":[{"cited_title":"J.; Moulavi, D.; and Sander, J","cited_arxiv_id":null,"evidence_quote":"Provides the HDBSCAN density-based clustering algorithm used in the Trigger Buffer's streaming clustering."},{"cited_title":"N.; Yu, M.; Xiang, B.; Zhou, B.; and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"Provides the sentence-embedding similarity used as the trigger's semantic metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the logits margin used to measure the model's confidence in next-token prediction."}],"review_version":2}