{"id":"28272fd1-46c8-4a31-85ba-779ce14fa0d3","arxiv_id":"2607.20440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"SGRE extracts a reasoning skeleton from a teacher trace, coarsens its graph, and verbalizes it densely; the final answer is preserved verbatim, and students distilled on the edited traces show large accuracy drops.","lead":"The paper proposes SGRE, a post-hoc editing pipeline that rewrites a teacher LLM's reasoning traces into denser, structurally coarsened forms so that student models trained on them perform much worse, while the final answer is copied exactly to keep accuracy. The approach is motivated by Cognitive Load Theory and could offer an API-deployable defense against knowledge distillation, if the reported effects are not artifacts of the evaluation protocol.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-forcing evaluation may undercount SGRE student accuracy because edited traces already end with a boxed final answer; the paper never states the pre-existing box is stripped.","rationale":"The reader's weakest assumption is exactly the AF protocol confound, and I agree it is the most load-bearing issue. It directly threatens the primary claim of anti-distillation effectiveness. Other weaknesses (no error bars, single LLM judge) weaken confidence but do not invalidate the central result if the AF issue is resolved. Therefore the conditional verdict is appropriate; no change to the reader's verdict is needed. The proposed test would settle whether the concern actually lands.","tokens_in":16985,"tokens_out":7294,"duration_ms":79301,"concrete_test":"Take 100–200 SGRE-edited GSM8K traces whose underlying answer is known correct. Run the exact AF protocol from §4.1: append the prompt, generate 32 tokens, extract the boxed content. Count how many extractions are empty/malformed or differ from the true answer. If the false-failure rate is non-negligible (>5%), the protocol is biased. Then rerun the student distillation evaluation with a corrected protocol (strip any pre-existing \\boxed{...} from the trace before appending AF, or extract only from the appended continuation). If the SGRE student accuracy (Table 1, e.g., Llama-3.2-3B GSM8K 19.02) moves by more than a few points, the headline degradation is inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical claim—SGRE yields the largest student degradation (Table 1)—rests on an evaluation protocol whose interaction with SGRE's output format is unspecified and plausibly biased. Section 4.1 says: 'after generating the reasoning trace, we append \"\\n\\n**Final Answer**\\n[\\boxed{ \" and continue to generate for 32 additional answer tokens. The final prediction is extracted from the content within \"\\boxed{...}\".' But SGRE verbalized traces already end with a boxed final answer, e.g., Appendix E.1: '**Final Answer:** \\[ \\boxed{36} \\]'. The paper does not state whether this pre-existing box is stripped before the AF prompt is appended. If extraction takes the last box, the appended generation after a closed box may produce a second, empty or malformed box for SGRE traces, making correct student answers appear incorrect. Since SGRE traces always end with the preserved final answer, while baseline traces may not (or may be corrupted by ADS), this can differentially deflate SGRE student accuracy and inflate the apparent anti-distillation effect. This is load-bearing because it is the central quantitative claim; the teacher-accuracy preservation is tautological and naturalness is secondary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SGRE, an \"Answer-then-Edit\" anti-distillation framework. The teacher first generates clean reasoning traces; a small LM extracts a reasoning-skeleton graph; graph coarsening merges reasoning steps; and the teacher verbalizes the coarsened skeleton with deliberately elevated textual complexity. The final answer from the original trace is preserved and appended verbatim, which guarantees teacher answer accuracy by construction. Experiments on GSM8K, MATH, and MMLU-Pro, with DeepSeek-R1-Distill-Qwen-7B as teacher and four student models, compare SGRE against ADS and DOGe. The paper reports that SGRE achieves the largest student-performance degradation while retaining teacher accuracy and high trace naturalness.","tokens_in":17295,"tokens_out":5416,"duration_ms":61289,"significance":"The post-hoc editing paradigm is a useful departure from internal-perturbation defenses: it decouples utility preservation from the defense mechanism and makes naturalness more controllable. If the empirical claims are reliable, SGRE offers a practical, efficient anti-distillation defense. The paper is also transparent that final-answer preservation is by design, not an empirical discovery. However, the main evaluation lacks error bars, and the answer-forcing protocol is underspecified in a way that could affect the central student-degradation measurement. The naturalness claim rests on a single LLM judge on a 10% subset. These issues make the significance conditional on additional validation.","major_comments":[{"comment":"The answer-forcing (AF) evaluation protocol is underspecified. The paper states that after generating the reasoning trace the authors append \"\\n\\n**Final Answer**\\n[\\boxed{ \" and extract from the content of \"\\boxed{...}\". SGRE traces already end with a boxed final answer (Appendix E.1 shows \"**Final Answer:** \\[ \\boxed{36} \\]\"). The paper does not state whether this existing box is stripped before appending the AF prompt. If it is not stripped, the final trace contains two boxed answers, and extracting from the last box could yield an empty or malformed box. This concern applies to clean and DOGe traces as well, but it is load-bearing for the central claim in Table 1 that SGRE achieves the largest student degradation. The authors should specify the extraction rule and, if the existing box is not stripped, run a robustness check (e.g., extracting from the first box or stripping the pre-ex","section":"Section 4.1 / Appendix E.1"},{"comment":"The paper reports \"average results over five runs\" but Table 1 contains no standard deviations, confidence intervals, or significance tests. Claims such as \"consistently achieves the largest degradation\" and \"outperforming ADS by a margin of 10.41%\" cannot be assessed without variance information. For example, the student-accuracy differences between SGRE and ADS on MATH (e.g., Llama-3.2-3B: 8.65 vs. 10.14; gemma-3-1b-it: 7.12 vs. 9.55) could be within run-to-run noise. The authors should report per-run numbers or standard deviations across the five runs, and preferably a paired test across datasets or seeds.","section":"Section 4.2 / Table 1"},{"comment":"The naturalness evaluation uses a single LLM judge (ChatGPT-5.5-Instant) on a randomly sampled 10% subset of each dataset, with no inter-annotator agreement, no human baseline, and no variance reported. Since \"superior trace naturalness\" is one of the paper's key advantages over ADS, this evaluation is too thin. At minimum, the authors should report the judge's confidence, run multiple judge models or repeated calls, and provide per-item standard errors or a significance test for the differences between SGRE and the baselines. A small human evaluation would substantially strengthen the claim.","section":"Section 4.4 / Figure 3"},{"comment":"The statement \"SGRE maintains the teacher's answer accuracy without loss\" is not an empirical finding but a design guarantee: the final answer is copied verbatim from the skeleton into the edited trace. The paper acknowledges this in Section 3.3, but Section 4.2 and the abstract present it as an experimentally demonstrated advantage. The presentation should be corrected to describe this as a guarantee, and the evaluation should verify that the implemented pipeline indeed preserves the final answer in all edited traces (e.g., by checking that extraction from edited traces reproduces the original answer).","section":"Section 4.2 / Section 3.3"}],"minor_comments":[{"comment":"The AF prompt string \"\\n\\n**Final Answer**\\n[\\boxed{ \" appears to contain a stray opening square bracket before \"\\boxed{\". This should be checked and fixed if it is a typo, since the extraction rule refers only to \"\\boxed{...}\".","section":"Section 4.1"},{"comment":"The phrase \"approximately 1× additional processing cost\" is ambiguous. If it means the total runtime doubles, it should say \"~2×\" or \"100% overhead\"; as written, \"1×\" could be read as no additional cost.","section":"Section 4.6"},{"comment":"The text says SGRE \"consistently and significantly\" reduces student performance, but the figure does not include error bars or significance tests. Use \"substantially\" or report statistical significance.","section":"Section 4.5 / Figure 4"},{"comment":"Equation (6) uses an undefined operator: the text defines \"L\" as ordered textual concatenation, but the displayed formula shows \"M\" before the set union. Please align the notation.","section":"Section 3.2"},{"comment":"The evaluator \"ChatGPT-5.5-Instant\" [27] should be described more precisely: exact model identifier, access date, and temperature/decoding settings, so the evaluation is reproducible.","section":"Section 4.4"},{"comment":"The number of training runs is five, but it is not stated whether the same five distilled students are used for all methods and datasets, or whether new seeds are drawn per condition. This matters for paired error-bar calculation.","section":"Appendix A.2 / Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is promising but needs a reliability pass before publication. The main issues are the underspecified answer-forcing extraction, missing variance measures, and a thin naturalness evaluation. The AF concern is less clean-cut than the skeptic notes because clean and DOGe traces also end with a boxed answer, but it is still an unsettled methodological point that the authors should address with an explicit protocol and a robustness check. I recommend major revision rather than rejection, since the central idea is defensible and the fixes are within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is real: SGRE moves anti-distillation out of the teacher's internals and into post-hoc trace editing. That is a clean departure from ADS and DOGe, and the three-stage skeleton pipeline (extraction, coarsening, verbalization) is a sensible concrete implementation. The CLT framing is more of a storytelling device than a mechanism, but it motivates the design without overselling it. The paper is also honest that teacher accuracy preservation is by construction: the final answer is copied verbatim, so the teacher-accuracy claim is tautological. Good that they say so.\n\nThe soft spot is exactly what your stress-test note flags, and I think it lands. The AF evaluation protocol appends another \"**Final Answer**\\n[\\boxed{ \" after the generated trace, and the answer is extracted from the last box. SGRE traces already end with a boxed answer, and the paper never says the pre-existing box is stripped. If it is not, forced continuation after a closed box can produce a second, empty or malformed box, and last-box extraction would then undercount SGRE student answers. That would differentially inflate the measured anti-distillation effect because SGRE traces always end with a preserved box, while ADS traces are often corrupted and may not. This is load-bearing: Table 1 is the central empirical claim. The fix is easy—strip the existing box before appending the AF prompt, or extract from the first box, or verify manual extraction on a sample—but it has to be done and reported.\n\nTwo smaller issues are worth naming. No standard deviations are reported despite the claim of \"average results over five runs,\" which makes it hard to judge whether the SGRE-vs-DOGe gaps are meaningful. And the naturalness evaluation is one LLM judge on a 10% subset; that is fine as a sanity check but not as a headline result.\n\nOn balance: the idea is worth taking seriously, the method is new, the paper is coherent, and the evaluation is fixable. I would send it to review, but I would ask for a rerun of the AF evaluation before trusting the numbers. If the effect survives the corrected protocol, this is a solid contribution to the AD line.\n\nBest,\n\n[You]","headline":"A genuinely new post-hoc anti-distillation paradigm with a plausible mechanism, but the headline student-degradation numbers rest on an evaluation protocol that may bias against SGRE and needs a straightforward fix.","tokens_in":17735,"tokens_out":2354,"would_cite":false,"duration_ms":28196,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that post-hoc editing of a teacher LLM's reasoning traces—extracting, coarsening, and rewording their logical skeleton—can sharply reduce the usefulness of those traces for knowledge distillation, while preserving the teac","keywords":["anti-distillation","knowledge distillation","reasoning traces","intellectual property protection","Cognitive Load Theory","skeleton graph","answer-then-edit","trace editing"],"falsifier":"Take an SGRE-edited trace, append the answer-forcing suffix, and decode additional tokens exactly as the paper's protocol does. If the newly generated boxed content is usually empty or malformed while the original boxed answer is correct, then the reported student accuracy is an artifact. Re-running the distillation evaluation with the original boxed answer stripped before appending the suffix, and checking whether student accuracy rises substantially, would settle whether the anti-distillation effect is genuine.","tokens_in":1232,"feed_emoji":"🛡️","tokens_out":1630,"duration_ms":54132,"temperature":0.7,"pith_summary":"To prevent rivals from cheaply copying a proprietary LLM through its public API, this paper proposes editing the model's reasoning traces before they are served. The method first lets the teacher produce a clean answer, then extracts a compressed 'skeleton' of the reasoning, merges and scrambles that skeleton's steps, and rewrites it in dense, indirect language. The resulting traces still lead to the original answer, but students fine-tuned on them pick up far less reasoning ability. Across three benchmarks and several student sizes, the paper reports the largest drops in distilled-student accuracy among compared defenses, with teacher accuracy unchanged and readability close to clean traces.","feed_headline":"Trace editing blocks LLM copying while preserving accuracy","feed_subtitle":"Skeleton-guided rewriting makes students learn far less from served traces, with teacher accuracy and readability intact.","key_machinery":"The reasoning skeleton graph: a DAG extracted from the teacher's trace whose nodes are abstract steps typed support, reasoning, or conclusion, and whose edges are depends_on relations. Skeleton graph coarsening merges nodes sharing predecessors (parallel coarsening) or along single-predecessor chains (serial coarsening) into blocks of at most Bmax steps, perturbing the granularity of the logic. Skeleton verbalization then re-lexicalizes the blocks subject to content fidelity, structural modulation, information control, and language style. The graph is the vehicle for deliberately injecting 'extraneous load' into the trace.","core_discovery":"SGRE shows that an anti-distillation defense does not need to perturb the teacher's internal generation; it can be applied after the fact. The teacher first answers normally, then a pipeline converts the trace into a directed acyclic graph of abstract reasoning steps, coarsens that graph by merging parallel and serial nodes, and verbalizes the coarsened structure using the teacher model under rules that demand dense, non-linear, vague language. Because the final answer is carried verbatim through the pipeline, teacher accuracy is preserved by construction; because the edited trace's logic is fragmented and its wording obfuscated, a student trained with token-level cross-entropy acquires less","pith_inferences":["If served adaptively—clean traces to trusted users, edited traces to suspects—a determined adversary could probe multiple times or apply their own abstraction and denoising to approximate the original skeleton, so the defense is likely best combined with access control and query monitoring.","The Cognitive Load Theory framing suggests the method should hurt students that imitate surface structure more than students already trained to reason from sparse hints; this could be tested by varying student pretraining or using students trained on skeleton-like data.","The paper evaluates distillation as unconditional next-token prediction on full traces; a stronger attack that conditions on the final answer, samples several traces per question, or down-weights late tokens might recover more reasoning ability.","Because the final answer is preserved verbatim, SGRE offers no protection against answer-only distillation that ignores the trace; its value is specifically for defenses against trace-based reasoning distillation."],"forward_implications":["Students trained on SGRE-edited traces show larger accuracy drops than those trained on traces from perturbation-based defenses, across all tested datasets and student model sizes.","The teacher's answer accuracy on the evaluation sets is unchanged by SGRE, because the final answer is copied from the original trace rather than regenerated.","Edited traces score close to clean traces on an LLM-as-a-judge naturalness scale, whereas logit-perturbation defenses degrade naturalness substantially.","SGRE runs faster than token-level penalty sampling, since the extra cost is a single pass of skeleton extraction plus verbalization rather than per-token gradient computation.","The defense's effectiveness persists when different open-weight models are used as the skeleton extractor, indicating the effect comes from the editing paradigm itself."],"fun_headline_variants":["Answer-then-edit: rewrite traces to block distillation","SGRE edits reasoning skeletons after the answer, hurting copycats","Post-hoc trace rewriting keeps teacher accuracy, blocks student theft","Rewrite served reasoning to cut distillation without losing utility","Skeleton-guided editing: answer clean, then obfuscate the logic"],"cache_read_input_tokens":19200,"weakest_assumption_plain":"The comparison rests on the answer-forcing protocol measuring student accuracy fairly even though SGRE traces already end with a boxed final answer; if that pre-existing box is not stripped before the answer-forcing prompt is appended, the protocol may count a second, empty box as the student's answer and overstate the defense's effectiveness.","fun_headline_variants_meta":{"raw":{"variants":["Answer-then-edit: rewrite traces to block distillation","SGRE edits reasoning skeletons after the answer, hurting copycats","Post-hoc trace rewriting keeps teacher accuracy, blocks student theft","Rewrite served reasoning to cut distillation without losing utility","Skeleton-guided editing: answer clean, then obfuscate the logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1250,"prompt_tokens":823,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":567,"tokens_out":427,"duration_ms":5777,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:14:33.239945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an SGRE-edited trace, append the answer-forcing suffix, and decode additional tokens exactly as the paper's protocol does. If the newly generated boxed content is usually empty or malformed while the original boxed answer is correct, then the reported student accuracy is an artifact. Re-running the distillation evaluation with the original boxed answer stripped before appending the suffix, and checking whether student accuracy rises substantially, would settle whether the anti-distillation effect is genuine.","supporting_citations":[],"review_version":1}