Pith. sign in

REVIEW 3 major objections 5 minor 7 references

Improving instruction-following in reasoning traces substantially improves privacy, at some cost to task accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Training large reasoning models to follow instructions inside their reasoning traces, plus switching adapters mid-generation, significantly reduces privacy leaks in their hidden thoughts.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Genuinely useful idea—RT-specific IF training with mid-response LoRA switching—but the privacy metric only catches verbatim leaks, so the central claim is weaker than the numbers suggest. the 3 major comments →

arxiv 2602.24210 v2 pith:FD5OYM27 submitted 2026-02-27 cs.CL cs.AI

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves

Haritz Puerto , Haonan Li , Xudong Han , Timothy Baldwin , Iryna Gurevych This is my paper
classification cs.CL cs.AI
keywords privacyreasoning tracesinstruction followingStaged DecodingLoRA adapterscontextual privacylarge reasoning modelsprompt injection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that privacy leaks in large reasoning models are essentially an instruction-following problem: models violate explicit privacy directives inside their reasoning traces, and those traces can be exposed through prompt injection. The authors show that if you fine-tune models to follow instructions about how to reason, and generate the reasoning and final answer with separate specialized adapters, you can sharply reduce private-data leakage. Across six models, they report gains of up to 20.9 points on instruction-following benchmarks and up to 51.9 percentage points on privacy benchmarks. The central trade-off is that these gains can come at the cost of task utility, especially on math reasoning, because instruction-following and reasoning performance pull against each other. The paper's contribution is a path toward privacy-aware agents built with small adapters rather than full retraining.

Core claim

The authors claim that improving instruction-following within reasoning traces directly translates into better contextual privacy: models trained on a new supervised dataset of reasoning-trace instructions (formatting, style, and reasoning-type constraints) leak far less private information in both their reasoning traces and final answers. They introduce Staged Decoding, which generates the reasoning trace with one LoRA adapter tuned for reasoning-trace instruction-following, then swaps in a second LoRA adapter tuned for final-answer instruction-following before generating the answer. This decoupling consistently maximizes both instruction-following scores and privacy scores, with the larges

What carries the argument

Staged Decoding: a decoding-time strategy that stops at the end-of-thinking token, unloads the low-rank adapter (LoRA) used for the reasoning trace, loads a different LoRA adapter optimized for final-answer instruction-following, and resumes generation. The reasoning-trace adapter is trained on a new SFT dataset of instructions that constrain only the reasoning trace — formatting, style, and reasoning-type constraints — plus progressively broader datasets that also target final-answer and joint constraints. This makes it possible to optimize instruction-following in each output component separately, with negligible computational overhead.

Load-bearing premise

The privacy scores assume that leakage is identifiable with absolute precision via exact string matching; if a model paraphrases, encodes, or indirectly references the confidential token, it is scored as private even though the information still leaks.

What would settle it

Run an evaluation where a human or a semantic-leakage classifier checks whether models trained with this method ever reveal confidential information through paraphrases or indirect references (e.g., describing the password's characters or using synonyms) while the string-matching metric reports no leak; if such cases are common, the reported privacy gains overstate real-world protection.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Privacy-aware reasoning agents can be built without full retraining: small LoRA adapters plus Staged Decoding produce large privacy gains across model families and sizes.
  • Staged Decoding yields the best of both worlds for instruction-following: it preserves the high reasoning-trace IF of the RT-optimized checkpoint while improving final-answer IF, beating baselines in 9 of 12 average-IF comparisons.
  • Improved instruction-following in reasoning traces is a practical lever for contextual privacy, with statistically significant privacy gains on both PasswordEval and PEEP in the paper's evaluation.
  • There is a real but task-dependent trade-off between privacy/instruction-following and utility; privacy benchmarks show a milder trade-off than math reasoning, and the paper suggests scaling training data could narrow the gap.
  • The method's success implies that privacy directives in system prompts are best treated as instructions to follow during reasoning, not just in the final answer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because leakage is measured by exact string matching, the method might be learning to avoid repeating secrets verbatim while still revealing them through paraphrase or indirect reference; a semantic-leakage evaluation would test whether the privacy gains hold up.
  • The two-adapter staged-decoding idea is generalizable beyond privacy—it could control other reasoning-trace properties like language, length, or format by decoupling per-component optimization.
  • The observed utility drop likely stems from the small GSM8K-only training set; training on more diverse, task-relevant data could preserve privacy gains while recovering utility.
  • Combining Staged Decoding with post-hoc anonymization of reasoning traces may approach the privacy ceiling of full anonymization while retaining more task utility, since the method already beats such anonymizers in utility on several models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes that privacy leakage in large reasoning models (LRMs) can be reduced by improving instruction-following (IF) inside the reasoning trace. It introduces an SFT dataset of trace-specific instructions (format, style, reasoning type), trains LoRA adapters for IF in the trace and in the final answer, and proposes Staged Decoding, which switches adapters at the trace/answer boundary. Experiments on six Qwen3/Phi-4 models (1.7B–14B) across IFEval, MathIF, PasswordEval, and PEEP report large IF gains (up to 20.9 points) and large privacy gains (up to 51.9 points), at a cost in task utility on MathIF. The paper concludes that improving IF in LRMs can significantly enhance privacy.

Significance. If the empirical claim holds, the paper provides a practical, adapter-based route to privacy-aware reasoning agents, which is more accessible than full retraining. The evaluation is broad (six models, two families, two privacy benchmarks), the method is simple and reproducible, code is released, and the training/evaluation data are independent. The comparison to the RANA anonymization upper bound is useful. However, the central privacy result rests on an exact-string-match leakage metric, which is explicitly defended in the Limitations section as measuring leakage with 'absolute precision.' This assumption is not safe: paraphrased, segmented, or partial disclosure of a secret is scored as non-leakage, so the reported privacy gains may substantially overstate real-world protection. The claim of consistent gains is also contradicted by at least one clear regression in Table 2.

major comments (3)
  1. [§4.2.2, Limitations] The privacy score is defined as 1 − leak rate, where a leak is scored only when the exact secret string appears verbatim in the trace or final answer. The Limitations section asserts that leakage 'can be identified and measured with absolute precision (i.e., via string matching).' This conflates verbatim reproduction with information disclosure. A model that outputs 'Heritage-Revive', 'H e r i t a g e R e v i v e', or 'the keyword starts with H and ends with e' is counted as private even though the secret is disclosed and usable for exfiltration. Prompt-injection exfiltration does not require verbatim reproduction. The same blind spot affects the RANA comparison in §5.4, since RANA removes strings by exact matching. The reported average privacy gains of 21.65 and 22.69 points may therefore reflect a shift from verbatim to paraphrased or partial disclosure rather than a genuine privacy im
  2. [Table 2, §5.2] The paper claims Staged Decoding yields 'consistent' privacy gains, but Qwen3-1.7B on PasswordEval is a clear counterexample: baseline total privacy is 42.13, while Staged Decoding is 22.60, with RT privacy 22.25 vs. 26.10 and FA privacy 23.30 vs. 74.17. This is a large regression, not a marginal one. Phi-4 3.8B also shows FA privacy dropping from 58.06 to 46.22 on PasswordEval. The reported averages and t-tests across models mask this failure case. The paper should explicitly acknowledge and analyze these regressions, and the claim that improved IF 'significantly enhances privacy' should be qualified to exclude or explain them.
  3. [§5.3, Table 3] The utility cost on MathIF is severe and uniform: all six models drop substantially (e.g., Qwen3-8B from 40.96 to 22.89; Phi4-14B from 40.66 to 25.75). The paper attributes this to an inherent IF–utility trade-off, but the magnitude raises a practical concern for the proposed approach as a privacy intervention. The utility drop may also be partly attributable to the 4-bit quantization and malformed outputs documented in Appendix D; the paper does not disentangle these causes. Since the paper's motivation is building privacy-aware agents, the utility loss should be analyzed more carefully rather than only cited as a known trade-off.
minor comments (5)
  1. [§3.1] The model name 'gptoss-120B' appears to be a typo or placeholder; please clarify the actual model used for rewriting reasoning traces.
  2. [§4.1 / Table 1] The text says Staged Decoding 'maximizes' IF-RT and IF-FA, but in several rows Staged Decoding's IF-FA is below the baseline (e.g., Qwen3-1.7B IFEval: 63.67 vs. 70.26). It maximizes the average, not each component. Rephrase to avoid overclaiming.
  3. [§5.2] The one-tailed t-test is reported with p-values but no test details. With six models and two seeds, state what the units of analysis are and the degrees of freedom; otherwise the significance claim is hard to interpret.
  4. [Appendix F] The PEEP utility judge is validated on only 10 sampled responses, with 4 judged imperfect. This is a small validation; a sentence acknowledging the limitation of the utility metric would be appropriate.
  5. [General] The benchmark name is written inconsistently as 'Math-IF' and 'MathIF'; please standardize. Also, the abstract states 'up to 20.9 IF points' but the gains are sometimes negative; the 'up to' phrasing should be accompanied by a range or caveat.

Circularity Check

0 steps flagged

No significant circularity; the privacy metric is a construct-validity limitation, not a circular reduction.

full rationale

The paper's claimed derivation chain—train on generic RT instruction-following data from GSM8K/DeepSeek-R1 rewrites, improve IF on IFEval and MathIF, then measure privacy on PasswordEval and PEEP—contains no step where an output is defined as an input. The SFT data and privacy benchmarks are disjoint: no privacy benchmark is used for training or checkpoint selection, and the privacy score (1 − leak rate) is computed by exact-string matching of confidential tokens, not derived from IF scores. Staged Decoding simply switches LoRA adapters optimized for IF-RT and IF-FA; its privacy gains are measured, not fitted. The only self-citation with author overlap is Green et al. (2025, Leaky Thoughts), which supplies motivation and the RANA upper-bound baseline; the leakage premise is also shown in the paper's own examples (Tables 12–13), and the cited result is independently falsifiable, so the citation is not load-bearing in a circular sense. The Limitations passage claiming leakage "can be identified and measured with absolute precision (i.e., via string matching)" is a genuine measurement-validity threat—paraphrased or partial disclosure is scored as private—and should lower confidence in real-world privacy gains, but it is a limitation of the metric, not a circular reduction. Verdict: no significant circularity; score reflects only the minor, non-load-bearing self-citation and the metric-validity caveat.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No fitted physical constants or invented entities. The central claim rests on training with LoRA, dev-set checkpoint selection, and the domain assumptions above; the most fragile is exact-match leakage measurement.

free parameters (1)
  • Checkpoint selection on MathIF-GSM8K dev set = best per variant, selected from 36 LoRA checkpoints per model
    All reported IF-RT, IF-FA and privacy numbers are for dev-selected checkpoints, not a fixed training recipe; this is a hand-chosen selection procedure built into the method (§4.1).
axioms (4)
  • domain assumption Privacy directives behave like ordinary instructions, so training on generic RT instructions transfers to privacy constraints.
    Core hypothesis stated in §1 and §6; if false the observed privacy gains are coincidental to the training distribution rather than caused by improved IF.
  • domain assumption Exact string matching of passwords and confidential strings measures leakage with 'absolute precision' (paper's words).
    Used to define privacy scores in §4.2.2 and defended in Limitations; paraphrased leakage would be scored as private.
  • domain assumption The end-of-thinking token cleanly separates RT and FA, and swapping LoRA adapters at that boundary does not corrupt the generation or KV cache.
    Required for Staged Decoding in §3.3; no experiments isolate adapter-switch artifacts.
  • domain assumption Evaluated six models (1.7B–14B) are representative enough to support a general claim about LRMs.
    Used in abstract and conclusion; results vary by model and one model moves opposite on PasswordEval.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves." pith.science (2026). https://pith.science/paper/FD5OYM27

@misc{pith2026260224210,
  author       = {Pith},
  title        = {Pith review of: From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FD5OYM27}},
  note         = {Machine review of arXiv:2602.24210}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large reasoning models (LRMs) produce reasoning traces (RTs) that often contain sensitive information. These leaky thoughts are difficult to control and frequently violate explicit privacy directives. Because RTs can be exposed through prompt injection attacks, this becomes a direct privacy risk to the user. We approach this as a controllability problem: since privacy directives are themselves instructions, improving instruction-following (IF) within the RT provides a direct path to reducing privacy leaks. To this end, we introduce an SFT dataset that teaches models to follow general instructions throughout their reasoning process, and propose Staged Decoding, a simple decoding strategy that decouples RT and answer generation using separate LoRA adapters to maximize IF of each component. We evaluate our approach on six models from two families (1.7B-14B parameters), across two IF benchmarks and two privacy benchmarks. Our method yields substantial improvements, with gains of up to 20.9 points in IF and 51.9 percentage points on privacy benchmarks, though these can come at the cost of task utility due to the trade-off between reasoning performance and IF. Our results show that improving IF in LRMs can significantly enhance privacy, suggesting a promising direction for future privacy-aware LRMs. Our code is available at https://github.com/UKPLab/arxiv2026-controllable-reasoning-models.

Figures

Figures reproduced from arXiv: 2602.24210 by Haonan Li, Haritz Puerto, Iryna Gurevych, Timothy Baldwin, Xudong Han.

Figure 1
Figure 1. Figure 1: Reasoning traces of user agents often include [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Staged Decoding generates the thinking to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of contextual information protected [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 3 linked inside Pith

  1. [3]

    C3oT: generating shorter chain-of-thought without compromising effectiveness. InProceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on In- novative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’25/IAAI’25/EAAI’25. AAAI Press. Woo...

    arXiv 2023

  2. [5]

    Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang

    Large language model agent: A survey on methodology, applications and challenges.Preprint, arXiv:2503.21460. Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. CoT-valve: Length- compressible chain-of-thought tuning. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- per...

    Pith/arXiv arXiv 2025

  3. [6]

    Preprint, arXiv:2507.05391

    Controlling what you share: Assessing language model adherence to privacy preferences. Preprint, arXiv:2507.05391. Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2018. Routing networks: Adaptive selec- tion of non-linear functions for multi-task learning. InInternational Conference on Learning Representa- tions. Dylan Sam, Alexander Robey, Andy Zou, ...

    arXiv 2018

  4. [7]

    TimeTraveller42

    Trade-offs in large reasoning models: An empirical analysis of deliberative and adaptive rea- soning over foundational capabilities.Preprint, arXiv:2503.17979. Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, and Kamalika Chaudhuri. 2025. AgentDAM: Privacy leakage evalu- ation for autonomous web agents. InThe Thirty-ninth...

    arXiv 2025

  5. [2021]

    Michael Han Daniel Han and Unsloth team

    Training verifiers to solve math word prob- lems.Preprint, arXiv:2110.14168. Michael Han Daniel Han and Unsloth team. 2023. Un- sloth.http://github.com/unslothai/unsloth. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetun- ing of quantized LLMs. InThirty-seventh Confer- ence on Neural Information Processing ...

    Pith/arXiv arXiv 2023

  6. [2024]

    In The Twelfth International Conference on Learning Representations

    AgentBench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Jun- wei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, and 7 others

  7. [2025]

    aha moments

    Activated loRA: Fine-tuned LLMs for intrin- sics. InThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and...

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.