Pith. sign in

REVIEW 2 major objections 5 minor 23 references

Playful AI in Professional Email: A Field Experiment on Tone and Recipient Engagement

T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI rewriting of workplace emails changes whether people open and reply only by shifting emotional tone, not by the fact of AI assistance itself.

desk verdict Solid multi-company crossover field experiment: LLM tone shifts positivity with no direct open/reply effects, but the “only through tone” claim rests on exploratory correlational mediation via an automated polarity score. read the letter →

arxiv 2607.11749 v1 pith:OHIKKJJ2 submitted 2026-07-13 cs.AI cs.HC

Ziv Ben-Zion , Teddy Lazebnik This is my paper
classification cs.AIcs.HC
keywords AI-mediatedcommunicationworkplaceemailemotionaltonepositivityfieldexperimentLLMrewritingrecipientengagementmediation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether AI-assisted writing changes how workplace email recipients actually behave, and through what channel. In a three-week randomized crossover field experiment, 121 employees at six companies sent their ordinary work mail under three conditions: unaided writing, GPT rewriting in a playful tone, and GPT rewriting in a professional tone, producing 16,880 emails. Neither AI condition directly changed open rates, reply rates, or response times. Playful rewriting raised emotional positivity and professional rewriting lowered it, and within a given sender the more positive emails were far more likely to be opened and answered. The authors conclude that AI-assisted communication shapes engagement not because a message is AI-edited, but because of the emotional tone of the language recipients read.

What carries the argument

The tone-mediated pathway: experimental condition shifts email positivity (a-path), within-sender-centered positivity predicts opening and replying (b-path), and the product yields a significant Sobel indirect effect of condition on behavior despite a null direct effect of condition.

What would settle it

A follow-up that independently manipulates positivity while holding AI condition and other features fixed, and finds that positivity no longer predicts opens or replies—or that finds a direct condition effect on engagement after positivity is controlled—would falsify the mediated-only claim.

Watch

Extended reading notes

Core claim

Across 16,880 workplace emails, playful GPT rewriting increased message positivity and professional rewriting decreased it, yet neither condition had any direct effect on opening, replying, or response speed. Within-sender positivity strongly predicted both opening (OR about 2.05) and replying (OR about 3.32). Mediation analysis showed significant indirect effects of both AI conditions on behavior through positivity alone, with no remaining direct effect of condition. AI editing mattered only insofar as it changed the emotional character of the language recipients read.

Load-bearing premise

The claim rests on treating an automated positivity score of the final email as the main emotional feature recipients respond to, even though positivity was measured rather than independently manipulated and may covary with other unmeasured message features.

Editorial extensions

If this is right

  • Defaulting workplace AI tools to a formal or professional register can suppress engagement by cooling emotional tone, even when the text looks more polished.
  • Any writing intervention will move recipient behavior only to the extent it changes features recipients actually respond to, such as positivity.
  • Email-by-email warmth within one sender’s ordinary range tracks large differences in open and reply odds, so small tone shifts are behaviorally consequential.
  • Playful tone moves positivity more for external recipients, but positivity more strongly drives opening among internal colleagues who share a baseline expectation of the sender.
  • Prompts and product defaults for writing assistants should be chosen for the direction of tone shift they produce, not only for correctness or polish.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If recipients routinely learn whether an email was AI-edited, a second direct pathway through trust or authenticity judgments may appear alongside the tone pathway documented here.
  • The same content-fixed, tone-randomized design should be testable in chat, support tickets, and performance feedback, where engagement is also a primary outcome.
  • Organizations that standardize on formal-rewriting defaults may systematically trade warmth for perceived competence across many senders at once.
  • Longitudinal adoption studies could reveal whether individual tone shifts aggregate into organization-level changes in how readily people open and answer mail.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reports a three-week randomized crossover field experiment with 121 employees across six companies who sent 16,880 ordinary work emails under three conditions: unaided writing, GPT-5 playful rewriting, and GPT-5 professional rewriting. Playful editing raised automated positivity (B=+0.068) and professional editing lowered it (B=−0.041), yet neither condition produced detectable direct effects on open rates, reply rates, or response times. Within-sender-centered positivity strongly predicted opening (OR=2.05) and replying (OR=3.32), and Sobel product-of-coefficients tests yielded significant indirect effects of condition on behavior through positivity. The authors conclude that AI-assisted writing shapes engagement only via the emotional tone of the language produced, not via AI use per se.

Significance. If the mediated pathway holds, the result is practically and theoretically useful: it supplies field evidence that LLM rewriting is neither a bonus nor a penalty in itself, and that the behavioral value of writing assistants depends on the direction of the tone shift they induce. Strengths include a genuine within-sender crossover in real organizations, large email N, mixed models with sender random intercepts, stratified Cox models, recipient-type splits, a compliance-restricted sensitivity analysis that leaves conclusions intact, and public analysis code. These design features make the null direct effects and the a-path (condition → positivity) credible contributions even if the mediation claim is qualified.

major comments (2)
  1. Results (mediation section) and Methods (Statistical analysis): the central claim that AI editing shapes engagement “only through” tone rests on a non-preregistered, secondary Sobel a×b product whose b-path is correlational. Positivity was measured on the final sent text, never independently manipulated, and Discussion itself notes that unobserved features (length, specificity, topic) that covary with polarity cannot be ruled out. Because the playful and professional prompts alter diction and concreteness alongside valence, residualizing positivity on length/topic/urgency (or reporting partial associations) is needed before the “only through tone” framing is warranted; otherwise the large ORs and Sobel z-values may partly reflect those co-varying features.
  2. Methods (Measures) and Discussion: the mediator is a single continuous spacytextblob polarity score (−1 to +1). The paper treats this score as a sufficient proxy for the emotional features recipients use when deciding to open and reply. Without validation against human ratings of warmth/playfulness/authenticity on a subset of emails, or multi-dimensional affect measures, the claim that the operative channel is “emotional tone” remains under-specified relative to the strength of the abstract and Discussion conclusions.
minor comments (5)
  1. Abstract and Introduction: the primary hypothesis of direct engagement gains from playful editing is stated clearly, but the abstract leads with the mediation result; a brief note that mediation was exploratory would better match the Methods statement that the study was not preregistered.
  2. Table 1 and Results: report email-level N and sender-level clustering more explicitly in the table notes so readers can see the effective sample for the mixed models at a glance.
  3. Methods (Experimental conditions): the playful STYLE variants (pirate, medieval, theatrical) are described only by example; listing the full set of styles used would aid replication.
  4. Figure 1 caption: panel (d) mediation diagram is described but the figure itself is not fully specified in the text; ensure path coefficients and significance markers are legible in the final layout.
  5. References: several 2025–2026 citations (e.g., Microsoft Work Trend Index 2026) should be checked for final publication status and stable DOIs before production.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: experimental a-path, measured mediator, and estimated mediation product are independent of the behavioral outcomes by construction.

full rationale

The paper's load-bearing chain is a randomized crossover field experiment (condition assignment → measured spacytextblob polarity → open/reply behavior) with Sobel product-of-coefficients mediation. Condition is experimentally assigned and is not defined in terms of positivity or engagement. Positivity is an automated polarity score on final sent text (−1 to +1), not a fitted function of open/reply rates. The b-path and a×b indirect effects are estimated from data (within-sender-centered positivity in GLMMs; Sobel z on a×b), not forced by identity or by a parameter fitted to the same outcomes. Null direct c-paths are empirical, not definitional. Self-citations (Ben-Zion et al. on LLM anxiety; Lazebnik & Rosenfeld LLM detector) are background or compliance sensitivity only and do not underwrite the mediation claim. Confounding of polarity with co-varying message features is a validity concern, not circularity by construction. The derivation is self-contained against the study's own experimental benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard experimental and mixed-model machinery plus domain assumptions that automated polarity and open/reply telemetry measure the intended constructs, and that within-sender centering isolates tone. No new physical entities are postulated. Free choices that affect estimates include the LLM style prompts, the detector compliance threshold, and censoring windows for survival models.

free parameters (4)
  • LLM style prompt templates (playful STYLE variants; professional formal rewrite)
    Fixed hand-written prompts define the experimental manipulation intensity and direction; different prompts would change the a-path size and possibly content preservation.
  • LLM-detector compliance threshold (≥0.50 for LLM conditions; <0.50 for Unaided)
    Threshold 0.50 is a chosen cutoff that defines the per-protocol sample (14,931 emails) used in sensitivity analyses.
  • Survival censoring windows (14 days open; 40 days reply)
    Right-censoring times are set to observed extremes and affect Cox estimates of time-to-event outcomes.
  • spacytextblob polarity score as continuous positivity mediator (−1 to +1)
    Choice of this off-the-shelf polarity model (vs. other sentiment systems or human ratings) defines the mediator used in all a/b/indirect paths.
assumptions (5)
  • domain assumption Tracking-pixel first-retrieval events validly indicate recipient opens of workplace email.
    Methods define open events via embedded pixels; privacy clients and image blocking can bias open measurement.
  • domain assumption Automated polarity captures the affective features that drive recipient approach/engagement.
    Load-bearing for interpreting positivity as the operative channel; paper acknowledges valence-only limitation.
  • standard math Within-sender centering of positivity isolates email-level tone from stable sender style.
    Standard mixed-model centering used for the b-path; assumes linear additive sender baselines.
  • domain assumption GPT rewrites preserve factual content and requested actions so condition differences are stylistic, not substantive.
    Prompt instructions and spot fidelity checks support this; residual content drift would confound tone with substance.
  • ad hoc to paper Sobel product-of-coefficients with delta-method SE adequately tests mediation for binary outcomes on the log-odds scale.
    Paper uses Sobel rather than modern bootstrap/CI mediation for GLMMs; this is a modeling choice that affects inference strength for the headline indirect path.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Playful AI in Professional Email: A Field Experiment on Tone and Recipient Engagement." pith.science (2026). https://pith.science/paper/OHIKKJJ2

@misc{pith2026260711749,
  author       = {Pith},
  title        = {Pith review of: Playful AI in Professional Email: A Field Experiment on Tone and Recipient Engagement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHIKKJJ2}},
  note         = {Machine review of arXiv:2607.11749}
}
read the original abstract

Large language models (LLMs) are rapidly reshaping workplace communication, yet whether AI-assisted writing changes how recipients actually behave, and through what channel, remains unknown. Here, in a randomized crossover field experiment, 121 employees across six companies sent work emails under three conditions over three weeks: unaided writing, GPT-5 rewriting in a playful tone, and GPT-5 rewriting in a professional tone. Across 16,880 emails, playful editing increased emotional positivity (B=+0.068, p<0.001), and professional editing decreased it (B=-0.041, p<0.001), yet neither condition directly altered open rates, reply rates, or response times. Instead, within-sender positivity strongly predicted both opening (OR=2.05) and replying (OR=3.32, p<0.001), a significant indirect pathway through which AI editing shaped behavior, in the absence of any direct effect. These findings suggest that AI-assisted communication shapes workplace engagement not through its use, but through the emotional tone of the language it produces.

Figures

Figures reproduced from arXiv: 2607.11749 by the authors.

Figure 1
Figure 1. LLM-assisted email rewriting affects recipient engagement only by shifting emotional tone. (a) Marginal mean positivity scores (±95% CI) by condition. All pairwise contrasts significant after Holm correction (p<0.001). (b) Association between within-sender centered positivity and behavioral outcomes, controlling for condition. Error bars are 95% CIs for OR estimates. (c) Open and reply rates by positivity quartile (… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 1 linked inside Pith

  1. [1]

    Microsoft 2026 Work Trend Index Annual Report https://news.microsoft.com/annual-work-trend-index-2026/

    Microsoft 2026 Work Trend Index Annual Report. Microsoft 2026 Work Trend Index Annual Report https://news.microsoft.com/annual-work-trend-index-2026/

    2026

  2. [2]

    How People Are Really Using AI in 2026

    Zao-Sanders, M. How People Are Really Using AI in 2026. Harvard Business Review (2026)

    2026

  3. [3]

    & Raymond, L

    Brynjolfsson, E., Li, D. & Raymond, L. Generative AI at work. Q. J. Econ. 140, 889–942 (2025)

    2025

  4. [4]

    & Zhang, W

    Noy, S. & Zhang, W. Experimental evidence on the productivity effects of generative artificial intelligence. Science 381, 187–192 (2023)

    2023

  5. [5]

    Elucidating the bonds of workplace humor: A relational process model

    Cooper, C. Elucidating the bonds of workplace humor: A relational process model. Hum. Relat. 61, 1087–1115 (2008)

    2008

  6. [6]

    Mesmer-Magnus, J., Glew, D. J. & Viswesvaran, C. A meta -analysis of positive humor in the workplace. J. Manag. Psychol. 27, 155–190 (2012)

    2012

  7. [7]

    T., Naaman, M

    Hancock, J. T., Naaman, M. & Levy, K. AI -mediated communication: Definition, research agenda, and ethical considerations. J. Comput.-Mediat. Commun. 25, 89–100 (2020)

    2020

  8. [8]

    & Sharot, T

    Glickman, M. & Sharot, T. How human–AI feedback loops alter human perceptual, emotional and social judgements. Nat. Hum. Behav. 9, 345–359 (2025)

    2025

Show all 23 references
  1. [9]

    & West, R

    Salvi, F., Horta Ribeiro, M., Gallotti, R. & West, R. On the conversational persuasiveness of GPT-4. Nat. Hum. Behav. 9, 1645–1653 (2025)

    2025

  2. [10]

    & Lazebnik, T

    Ben-Zion, Z., Elyoseph, Z., Spiller, T. & Lazebnik, T. Inducing state anxiety in LLM agents reproduces human-like biases in consumer decision-making. Npj Artif. Intell. 2, 55 (2026)

    2026

  3. [11]

    Ben-Zion, Z. et al. Assessing and alleviating state anxiety in large language models. NPJ Digit. Med. 8, 132 (2025)

    2025

  4. [12]

    Coda-Forno, J. et al. Inducing anxiety in large language models can induce bias. Preprint at https://doi.org/10.48550/arXiv.2304.11111 (2024)

    2024 doi arXiv

  5. [13]

    Van Kleef, G. A. How Emotions Regulate Social Life: The Emotions as Social Information (EASI) Model. Curr. Dir. Psychol. Sci. 18, 184–188 (2009)

    2009

  6. [14]

    Doshi, A. R. & Hauser, O. P. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Sci. Adv. 10, eadn5290 (2024)

    2024

  7. [15]

    Fredrickson, B. L. The role of positive emotions in positive psychology: The broaden-and- build theory of positive emotions. Am. Psychol. 56, 218 (2001)

    2001

  8. [16]

    T., Cuddy, A

    Fiske, S. T., Cuddy, A. J. & Glick, P. Universal dimensions of social cognition: Warmth and competence. Trends Cogn. Sci. 11, 77–83 (2007)

    2007

  9. [17]

    K., Houser, M

    Stephens, K. K., Houser, M. L. & Cowan, R. L. R U Able to Meat Me: The Impact of Students’ Overly Casual Email Messages to Instructors. Commun. Educ. 58, 303–326 (2009)

    2009

  10. [18]

    Burgoon, J. K. Interpersonal Expectations, Expectancy Violations, and Emotional Communication. J. Lang. Soc. Psychol. 12, 30–48 (1993)

    1993

  11. [19]

    Barsade, S. G. The Ripple Effect: Emotional Contagion and its Influence on Group Behavior. Adm. Sci. Q. 47, 644–675 (2002)

    2002

  12. [20]

    & Boyd, A

    Honnibal, M., Montani, I., Van Landeghem, S. & Boyd, A. spaCy: Industrial-strength natural language processing in Python. (2020)

    2020

  13. [21]

    textblob Documentation

    Loria, S. textblob Documentation. Release 015 2, 269 (2018)

    2018

  14. [22]

    Jakesch, M., Hancock, J. T. & Naaman, M. Human heuristics for AI-generated language are flawed. Proc. Natl. Acad. Sci. 120, e2208839120 (2023)

    2023

  15. [23]

    & Rosenfeld, A

    Lazebnik, T. & Rosenfeld, A. Detecting LLM -assisted writing in scientific communication: Are we there yet? J. Data Inf. Sci. 9, 4–13 (2024). Appendix Supplementary Analysis: Treatment-Compliance Sensitivity Assignment to the Playful and Professional conditions did not guarant...

    2024

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.