Pith. sign in

REVIEW 4 major objections 6 minor 18 references

Rewriting AI-edited prose toward a user's style scores the same from 0.5B to 7B parameters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:59 UTC pith:DRKBD6EP

load-bearing objection A careful, honestly limited empirical pipeline for per-user LoRA rewriting—but the plateau is largely a similarity/copy metric artifact, and the style components show no adaptation signal. the 4 major comments →

arxiv 2607.29238 v1 pith:DRKBD6EP submitted 2026-07-31 cs.CL cs.AIcs.CYcs.ETcs.HC

Small Is Enough: Per-User Style Rewriting of AI-Edited Text via LoRA Adapters

Antorweep Chakravorty This is my paper
classification cs.CL cs.AIcs.CYcs.ETcs.HC
keywords personalized rewritingstyle transferLoRAsmall language modelsparameter-efficient fine-tuningprivacyLLM-as-a-judgeAI-ness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

InMyStyle is a local system that rewrites AI-edited paragraphs to match one user's writing style by training a personal LoRA adapter on paired machine-paraphrased and user-written text. The central claim is that for this measured rewriting task, base-model capacity makes no difference: all four model sizes reach an automatic composite score of 0.69 on 219 held-out pairs under both greedy and sampled decoding. The paper's secondary finding is that LLM judges perceive outputs from larger adapters as less AI-like, but this does not alter the automatic plateau. If correct, the paper shows that per-user style rewriting can be done privately and cheaply on a laptop, with model size shaping compute and perceived AI-ness rather than automatic quality.

Core claim

On the paper's own terms, the discovery is a size-invariant plateau: training Qwen2.5 LoRA adapters of 0.5B, 1.5B, 3B, and 7B parameters on the same paired AI-shadow/user-original data and evaluating with a weighted composite of authorship probability, content similarity, target similarity, AI-tell reduction, and stylometric movement yields a mean composite of 0.69 for every size under both greedy and temperature-0.7 sampling. The larger models achieve lower training and evaluation cross-entropy, but this ordering disappears at the task level. The authors interpret this as evidence that compact models capture recurring user preferences through low-rank updates, making model size a trade-off

What carries the argument

LoRA adapters (low-rank matrix updates to frozen base-model weights) trained with response-only loss, where cross-entropy is computed only over user-written target tokens while the machine-generated shadow input serves as masked conditioning. Paired training data are produced by three local helper LLMs that paraphrase user paragraphs into 'AI-shadow' variants, with paragraph-level split isolation to prevent target leakage. The composite metric Q combines authorship classifier probability, BERTScore content and target similarity, AI-tell reduction, and stylometric distance.

Load-bearing premise

The 0.69 plateau is meaningful only if the automatic composite Q actually measures recovery of the user's style; Q's authorship component is trained on the same user's original paragraphs versus the same AI-shadow inputs used to build the training pairs, so it may reward generic human-vs-machine differences rather than the user's specific voice.

What would settle it

A blinded human study that asks readers to rank rewrites by 'matches this author's writing' would settle it: if humans consistently prefer 7B rewrites over 0.5B rewrites for the same paragraphs, the automatic plateau is a metric artifact. Alternatively, if a second user's corpus shows a different size trend, the result is user-specific.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Users can get personalized, privacy-preserving rewriting on a single laptop with a 0.5B model, no cloud inference or style prompts required.
  • For this metric, compute can be cut roughly sevenfold without measured loss, making the approach feasible on consumer hardware.
  • Per-user adapter storage becomes practical: a separate small PEFT module per user instead of a shared multi-user model.
  • The plateau suggests the bottleneck is the paired-data construction and metric, not the base model's capacity.
  • Larger models still win on perceived AI-ness, so size choice depends on whether priority is automatic quality or human-like feel.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the plateau holds across other base-model families and users, the common assumption that style transfer scales with model capacity is not true for this task; the practical limit may be the quality of the synthetic AI-shadow pairs.
  • Lower perceived AI-ness from the 7B adapter may reflect fewer mechanical artifacts rather than a better match to the user's voice, meaning 'more human-like' and 'more like this author' are separate dimensions.
  • A testable extension: diversifying the corruption operators beyond the fixed AI-tell dictionary could shift the plateau, since the authorship component of Q may be tied to the specific corruption distribution.
  • The plateau could mask a ceiling effect if Q is insensitive above a certain style-recovery threshold; a more discriminative authorship metric might reveal size differences the composite misses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents InMyStyle, a fully local pipeline that creates paired training examples by paraphrasing a target user's paragraphs with three helper LLMs and applying stochastic corruption, then trains LoRA adapters on Qwen2.5 models from 0.5B to 7B with response-only loss and grouped paragraph-level splits. On 219 held-out evaluation pairs, a hand-weighted composite score Q (Eq. 8) is reported as 0.69 for all model sizes under both greedy and sampled decoding; the authors interpret this plateau as evidence that small models are sufficient for the measured rewriting task. A secondary LLM-as-a-judge study finds that InMyStyle outputs are perceived as less AI-like than helper-generated inputs, with perceived AI-ness decreasing as model size increases.

Significance. If the plateau were established under a validated measure of per-user style recovery, the result would be practically valuable: it would support deploying sub-1B local models for personalized rewriting, reducing cost and privacy exposure. The paper has clear strengths: the paragraph-level split prevents target leakage; the response-only loss is well motivated; the model-size sweep is controlled within a single family; and the limitation section is unusually honest. However, the central conclusion relies on a composite metric whose style-sensitive components (A and S) show no measurable adaptation signal, while the similarity components dominate the score. The paper's own LLM-judge data show a size-dependent trend that complicates the plateau interpretation. These issues are load-bearing for the 'small is enough' claim and require additional evidence or careful re-scoping of the conclusion.

major comments (4)
  1. [§4.5, Table 2] The composite Q in Eq. 8 is dominated by content and target similarity. With A≈0.52, C≈0.93, T≈0.89, R≈0.5, and S=0.00, the style-relevant terms A and S contribute only about 0.23 of the 0.69 total, while C and T contribute about 0.37. Stylometric improvement S is exactly 0.00 for all eight conditions, and authorship A hovers near the 0.5 classifier boundary. Thus the plateau is driven by near-saturated similarity and floor-level style signals. To support the 'small is enough' claim, the authors must show that Q is sensitive to style recovery. Please add a no-adaptation baseline (e.g., the AI-shadow input scored directly, or a generic paraphrase without LoRA) and report A and S as primary outcomes, not only Q.
  2. [§4.5, §3.3] The Authorship classifier is trained on the same user's originals versus AI-shadows produced by the same three helper models used to build the training pairs, and the AI-tell dictionary used to corrupt inputs (Eq. 2) is the same dictionary used to compute the R metric (Eq. 6). This creates circularity: A may be measuring 'not a helper-model output' rather than 'this user's style', and R partly measures removal of artificially injected tells. The paper acknowledges this risk in Section 6, but the central conclusion rests on Q. Please provide a cross-user validation of the classifier, a held-out-helper evaluation, or a clear argument for why the evaluation is not in-distribution with respect to the metric.
  3. [§5.2, Table 5] The LLM-judge results show a monotonic decrease in perceived AI-ness with model size (6.24, 5.80, 4.93, 4.45 for 0.5B to 7B), which directly contradicts a uniform plateau across sizes on a human-relevant dimension. The paper labels this descriptive and not overturning the automatic plateau, but it raises the possibility that Q is insensitive to differences that judges perceive. The authors should either treat the size trend as evidence against the unconditional 'small is enough' claim (or qualify it to the automatic metric), or provide a concrete test showing that perceived AI-ness is orthogonal to style recovery. At minimum, the unbalanced sample counts (n from 35 to 40) and lack of variance or statistical testing should be addressed.
  4. [§4.5, Eq. 8] The composite weights 0.40/0.25/0.15/0.15/0.05 are hand-selected with no sensitivity analysis or justification. Because the main conclusion is a plateau in Q, it is essential to show that the plateau is not an artifact of these weights. Please report Q for alternative reasonable weight vectors (e.g., equal weights, style-heavy weights) or provide bootstrap confidence intervals over the 73 source paragraphs. The current rounding of all Q values to 0.69 obscures potentially meaningful variation across models and decoding modes.
minor comments (6)
  1. [Abstract] The system is described as 'privacy first', but the paper does not evaluate privacy guarantees (e.g., data leakage, model inversion, or secure execution). Suggest qualifying to 'local' or 'on-device'.
  2. [§3.3] The corruption probability 0.5 for the AI-tell operator is not justified. If tells are injected in only half the training inputs, the R metric may be inconsistently measured. Consider reporting R separately for corrupted and uncorrupted inputs.
  3. [Table 2] The caption says 'Bold marks the highest value within each decoding condition', but many values tie after rounding. Using exact values or a footnote would avoid ambiguity about which entries are considered best.
  4. [§4.6] The statement 'A new private or temporary conversation was used for every batch' does not specify whether judges were accessed via API or web interface, which affects reproducibility. Please provide the exact retrieval method and date.
  5. [§5.2] Pooled perceived-AI means should be accompanied by per-item variance or inter-judge agreement (e.g., ICC) to assess calibration differences and the reliability of the pooled estimate.
  6. [Table 1 caption] 'T able 1' should be 'Table 1'. Also, 'Separation of Gen' in Section 5.2 is awkward phrasing.

Circularity Check

0 steps flagged

No significant circularity: the plateau follows from a fixed, reported composite on a paragraph-level held-out split; the main caveats are metric-validity limitations, not circular reductions.

full rationale

The paper's central claim — an automatic composite score of 0.69 across model sizes and decoding conditions — comes from a fixed formula (Eq. 8) applied to 219 held-out pairs whose source paragraphs are disjoint from the training paragraphs (Eq. 3). The authorship classifier is trained only on the training partition and applied to evaluation outputs, so it is not fit to the predictions under test. Content and target similarity use BERTScore with fixed roberta-base, and the stylometric and AI-tell components are small contributors (0.15 and 0.05 weights) with S = 0.00 for every condition. The same AI-tell dictionary used to corrupt training inputs is also used in the R metric, but the paper discloses this ('AI-tell reduction uses the same phrase/substitution dictionary as the primary automatic evaluation') and the component is not what drives the plateau. The paper's own Limitations section explicitly states that 'The automatic results are specific to a manually weighted composite. The authorship classifier may capture general human-machine or helper-specific signals in addition to user style, BERTScore can reward conservative copying, and the stylometric profile captures only coarse surface features.' This is an acknowledged validity caveat, not a circular reduction. No parameter is tuned to force equality across model sizes, the components are reported separately, and there are no load-bearing self-citations. Therefore no circular step can be exhibited under the required standard of showing an equation reducing to an input or a fitted parameter renamed as a prediction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central plateau depends on several hand-chosen components, especially the composite weights and the AI-tell dictionary that appears in both training corruption and evaluation. No new entities are introduced; per-user adapters are standard LoRA modules.

free parameters (5)
  • Composite weights (Eq. 8) = w_A=0.40, w_C=0.25, w_T=0.15, w_R=0.15, w_S=0.05
    Hand-chosen weights define the 0.69 plateau; they are not derived from data or theory.
  • AI-tell dictionary and corruption probabilities = AI-tell p=0.5, light-noise p=0.3; dictionary includes 'use'->'utilize' and formulaic phrases
    Manual synthetic corruption; the same dictionary is later used to compute R, so the metric rewards in-distribution behavior.
  • Length-aware generation budget = 1.5w+32 tokens, floor at 30 words, cap at 600 tokens
    Manual heuristic that affects all outputs and therefore every metric, especially on long inputs.
  • Stylometric and AI-tell normalization constants = eR=(R+2)/4, eS=clip((S+0.2)/0.4)
    Hand-selected scaling to bring R and S onto [0,1] before weighting.
  • LoRA/optimization hyperparameters = rank 8, alpha 16, dropout 0.05, epochs 3, lr 2e-4, batch 16, max length 2048
    Fixed across sizes, so they do not explain the plateau, but different settings could change it. Not fitted to the data, but chosen by hand.
axioms (4)
  • domain assumption Helper paraphrases are meaning-preserving and representative of AI-edited prose
    Section 3.3 constructs training pairs from helper outputs; if helpers introduce errors or do not resemble real AI editing, the learned transform is misaligned.
  • domain assumption The user's 36 published papers are a clean sample of their personal writing style
    Section 6 admits temporal, domain, co-author, reviewer, and venue effects contaminate the style signal; the system may learn a mixed distribution.
  • domain assumption The authorship classifier and stylometric features are valid proxies for style
    Section 4.5 defines Authorship as a TF-IDF logistic-regression probability and stylometric improvement via coarse surface features; the paper concedes these may capture generic human-vs-machine signals.
  • domain assumption Response-only loss on AI-shadow/user pairs learns a rewriting function rather than near-identity copying
    Section 3.5 masks input tokens during training, but there is no proof or analysis that the model generalizes beyond reconstructing the target distribution.

pith-pipeline@v1.3.0-daily-deepseek · 11229 in / 13868 out tokens · 125550 ms · 2026-08-03T10:59:33.601719+00:00 · methodology

0 comments
read the original abstract

InMyStyle is a privacy first, single user system that adapts small language models to rewrite AI-edited text towards an individual user's writing style without an instruction prompt at inference. Given a user's documents, it uses multiple local helper LLMs to construct paired training examples and fine tunes LoRA adapters on base models ranging from 0.5B to 7B parameters. Length aware generation budgets and automatic chunking support inputs of different lengths. On 219 evaluation pairs from a scientific-paper corpus, the automatic composite score plateaus at 0.69 [scale 0-1] across all model sizes under both greedy and sampled decoding. This observed plateau suggests that small models are sufficient for the measured rewriting task, with model size determining trade-offs rather than a stable quality ranking. As a secondary evaluation, 400 ratings from five LLM judges give InMyStyle outputs a mean perceived AI-ness score over 20% lower than their helper-AI generated inputs, while mean perceived AI-ness scores decrease with model size within InMyStyle.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, et al.Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Aug. 2024.doi: 10.48550/arXiv.2404.14219. arXiv:2404.14219 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. May 2023.doi:10 . 48550/arXiv.2305.14314. arXiv:2305.14314 [cs.LG]

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.14314 2023

  3. [3]

    Com- ment: ACL 2024

    Liam Dugan, Alyssa Hwang, Filip Trhlik, et al.RAID: A Shared Bench- mark for Robust Evaluation of Machine-Generated Text Detectors. Com- ment: ACL 2024. June 2024.doi:10.48550/arXiv.2405.07940. arXiv: 2405.07940 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.07940 2024

  4. [4]

    Jillian Fisher, Skyler Hallinan, Ximing Lu, Mitchell Gordon, Zaid Har- chaoui, and Yejin Choi.StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements. Aug. 2024.doi:10. 48550/arXiv.2408.15666. arXiv:2408.15666 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.15666 2024

  5. [5]

    2024.doi:10.48550/ARXIV.2407.21783

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al.The Llama 3 Herd of Models. 2024.doi:10.48550/ARXIV.2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  6. [6]

    Published in ICLR 2020

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.The Curious Case of Neural Text Degeneration. Published in ICLR 2020. 2019. doi:10.48550/ARXIV.1904.09751

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.09751 2020

  7. [7]

    Zachary Horvitz, Ajay Patel, Kanishk Singh, Chris Callison-Burch, Kath- leen McKeown, and Zhou Yu.TinyStyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddings. Nov. 2024.doi:10.48550/arXiv. 2406.15586. arXiv:2406.15586 [cs.CL]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.15586 2024

  8. [8]

    Hu, Yelong Shen, Phillip Wallis, et al.LoRA: Low-Rank Adap- tation of Large Language Models

    Edward J. Hu, Yelong Shen, Phillip Wallis, et al.LoRA: Low-Rank Adap- tation of Large Language Models. Comment: Draft V2 includes better base- lines, experiments on GLUE, and more on adapter latency. Oct. 2021.doi: 10.48550/arXiv.2106.09685. arXiv:2106.09685 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.09685 2021

  9. [9]

    Deep Learning for Text Style Transfer: A Survey

    Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. “Deep Learning for Text Style Transfer: A Survey”. In:Computational Linguistics48.1 (2022), pp. 155–205.doi:10.1162/coli_a_00426

    doi:10.1162/coli_a_00426 2022

  10. [10]

    Comment: NeurIPS 2023 camera ready (32 pages)

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer.Paraphrasing Evades Detectors of AI-generated Text, but Re- trieval Is an Effective Defense. Comment: NeurIPS 2023 camera ready (32 pages). Oct. 2023.doi:10.48550/arXiv.2303.13408. arXiv:2303.13408 [cs.CL]. Small Is Enough 19

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.13408 2023

  11. [11]

    2019.doi:10.48550/ARXIV.1907

    Yinhan Liu, Myle Ott, Naman Goyal, et al.RoBERTa: A Robustly Opti- mized BERT Pretraining Approach. 2019.doi:10.48550/ARXIV.1907. 11692

    doi:10.48550/arxiv.1907 2019

  12. [12]

    Armand Nicolicioiu, Eugenia Iofinova, Andrej Jovanovic, et al.Panza: Design and Analysis of a Fully-Local Personalized Text Writing Assis- tant. Feb. 2025.doi:10.48550/arXiv.2407.10994. arXiv:2407.10994 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.10994 2025

  13. [13]

    Text Style Transfer Evaluation Using Large Language Models

    Phil Ostheimer, Mayank Nagda, Marius Kloft, and Sophie Fellenz. “Text Style Transfer Evaluation Using Large Language Models”. In:Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC-COLING 2024). Torino, Italia: ELRA and ICCL, 2024, pp. 15802–15822

    2024

  14. [14]

    Qwen, An Yang, Baosong Yang, et al.Qwen2.5 Technical Report. Jan. 2025.doi:10.48550/arXiv.2412.15115. arXiv:2412.15115 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2025

  15. [15]

    ASurveyofModernAuthorshipAttributionMeth- ods

    EfstathiosStamatatos.“ASurveyofModernAuthorshipAttributionMeth- ods”. In:Journal of the American Society for Information Science and Technology60.3 (Mar. 2009), pp. 538–556.issn: 1532-2882, 1532-2890. doi:10.1002/asi.21001

    doi:10.1002/asi.21001 2009

  16. [16]

    Comment: EMNLP 2024 Main

    Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang.Democratizing LLMs via Personalized Parameter-Efficient Fine-tuning. Comment: EMNLP 2024 Main. Feb. 2025.doi:10.48550/ arXiv.2402.04401. arXiv:2402.04401 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.04401 2024

  17. [17]

    AuthorMix: Modular Authorship Style Transfer via Layer-Wise Adapter Mixing

    Sarubi Thillainathan, Ji-Ung Lee, Michael Sullivan, and Alexander Koller. AuthorMix: Modular Authorship Style Transfer via Layer-Wise Adapter Mixing. Mar. 2026.doi:10 . 48550 / arXiv . 2603 . 23069. arXiv:2603 . 23069 [cs.CL]

    2026

  18. [18]

    Weinberger, and Yoav Artzi.BERTScore: Evaluating Text Generation with BERT

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.BERTScore: Evaluating Text Generation with BERT. 2019.doi: 10.48550/ARXIV.1904.09675

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.09675 2019