Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Evaluating Style-Personalized Text Generation: Challenges and Directions

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that no single metric—BLEU, style embeddings, or an LLM acting as judge—adequately measures style-personalized text generation, and that combining diverse metrics into an ensemble consistently outperforms every individual

desk verdict A genuinely useful benchmark and a transparent evaluation of SPTG metrics, but the headline ensemble claim is undercut by test-set weighting, post-hoc selection, and noise-level gains. read the letter →

arxiv 2508.06374 v3 pith:ZQFZJD7K submitted 2025-08-08 cs.CL

Anubhav Jangra , Bahareh Sarrafzadeh , Silviu Cucerzan , Adrian de Wynter , Sujay Kumar Jauhar This is my paper
classification cs.CL
keywords stylepersonalizationtextgenerationevaluationLLM-as-judgeensemblemetricsdiscriminationbenchmarkBLEUauthorshipattributionlow-resource
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Style-personalized text generation—getting an LLM to write 'like me'—lacks a trustworthy measurement. This paper tries to establish that the field should stop trusting any single evaluator, whether n-gram overlap like BLEU, style-similarity embeddings, or an LLM-as-judge, and instead combine several diverse metrics into an ensemble. To test this, the authors build a binary style-discrimination benchmark across eight writing tasks with three evaluation settings—domain discrimination, authorship attribution, and LLM-personalized-versus-non-personalized—and report that the best ensemble, ρall with performance-weighted voting, reaches 0.821 mean accuracy, beating the best single judge (gpt-4.1 at 0.815) and BLEU (0.733) by a wide margin. The paper also shows that every metric class degrades most sharply in the LLM-personalized setting, where even the best judge reaches only 0.678, and that short, informal domains like Reddit and Enron sit near the random baseline. If correct, evaluations of style-personalized output should report ensemble judgments, and the field should build metrics that capture style in long-form, low-resource, pragmatic contexts.

What carries the argument

The engine is the forced-choice style discrimination task with ensembled voting. Each instance is a triplet ($T_{\mathrm{ref}}$, $T^+$, $T^-$), and the metric must output the index of the stylistically closer candidate, turning evaluation into binary classification. The second mechanism is ensembling: majority voting and performance-weighted voting (PWV), where each metric's vote is weighted by its own overall accuracy. The paper finds that ensembles mixing paradigms—for example, BLEU + ROUGE-L + Wegmann + StyleDistance + gpt-4.1 with PWV—yield the best results, and it credits the diversity among paradigms (shown through pairwise disagreement) for the gain.

What would settle it

Select a random subset of the LLM-setting instances and have several human raters—who have not seen the labels—choose which candidate is stylistically closer to the reference, with an explicit 'no difference' option. The paper's own annotation study found 'Both' to be the most frequent answer for style preference; if raters likewise fail to systematically pick $T^+$ over $T^-$, then the ensemble accuracy in the LLM setting is not a valid measurement of style discrimination.

Watch

Extended reading notes

Core claim

Central claim: on a new low-resource benchmark spanning eight writing domains and three evaluation settings, an ensemble of metrics from three paradigms—n-gram overlap, style embeddings, and LLM-as-judges—with performance-weighted voting reaches 0.821 mean accuracy, beating the best single judge (gpt-4.1 at 0.815) and BLEU (0.733). The paper frames style evaluation as binary discrimination—given a reference text $T_{\mathrm{ref}}$ and two candidates $T^+$, $T^-$, decide which is stylistically closer—so it can test whether metrics discriminate style at all, not just how well they score. It also shows that all metric classes degrade most in the LLM-personalized setting, where even the best jud

Load-bearing premise

The load-bearing premise is that in the LLM-generated setting, the candidate produced with reference style text ($T^+$) really is closer in style to the reference than the candidate produced without it ($T^-$); if the generator fails to echo the style, the labels become arbitrary and the accuracy numbers stop measuring style discrimination.

Editorial extensions

If this is right

  • SPTG papers should report ensemble judgments rather than a single headline metric, since no single evaluator—including gpt-4.1—consistently outperforms the ρall ensemble in this benchmark.
  • The proposed benchmark provides a reusable testbed: eight writing tasks and three settings that let future metrics be checked for genuine discrimination instead of inflated agreement.
  • Evaluation claims about style personalization should be benchmarked in the hardest LLM-personalized setting, where even the best judges hover near 0.678, and on short informal domains where metrics are near random.
  • Performance-weighted voting across paradigms (n-gram, embedding, LLM-judge) is a cheap, model-free way to improve evaluation reliability without training new models.
  • The finding that n-gram metrics are stronger than expected in long-form, low-resource style discrimination suggests that lexical and structural markers carry useful signal at span lengths where prior work found them weak.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ensemble gain probably comes from paradigm diversity rather than from any single strong model: pairwise disagreement is highest across paradigm families (e.g., ROUGE-1 vs StyleDistance disagree on 0.35 of authorship-attribution instances), so deliberately mixing maximally different metric types should yield further gains.
  • If the LLM-setting labels are as unstable as the annotation study hints, the bottleneck in current SPTG may be generation, not evaluation: a testable extension would be to measure whether stronger style-conditioned generators make the $T^+$ vs $T^-$ distinction easier for all metrics, which would show up as rising separability.
  • A natural next step is to move from hard voting to score-averaging or rank-fusion across metrics, which would use information from near-ties—useful because human raters often see 'Both' as the right answer.
  • Extending the ensemble idea to correlate with human preference on existing personalization benchmarks (like LaMP/LongLaMP with human judgments) would directly test whether the ρall ensemble also improves agreement with people, not just agreement with synthetic labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a style-discrimination benchmark for evaluating style-personalized text generation, spanning eight writing domains and three settings: domain discrimination (DD), authorship attribution (AA), and LLM-generated personalized vs. non-personalized discrimination (LLM). It compares n-gram metrics, style embedding metrics, and LLM-as-judge metrics, along with majority-voting and performance-weighted-voting ensembles, and reports that ensembles outperform single evaluators. The paper also includes ablations on LLM-as-judge prompting and a human inter-annotator agreement study.

Significance. If the empirical claims held, the benchmark and the metric comparison would be a useful contribution: the coverage of eight domains, three evaluation settings, multiple metric families, and modern LLM judges is valuable, and the human-IAA analysis speaks to a real difficulty in SPTG evaluation. However, the central claim—that ensembles of diverse metrics consistently outperform single-evaluator methods—is not established by the reported experiments. The benchmark infrastructure and individual metric observations are worth preserving, but the paper's headline conclusion currently rests on an invalid evaluation protocol and an unvalidated ground-truth assumption in the LLM setting.

major comments (3)
  1. [§4.4, §5.2, Table 3] The PWV ensemble weights are estimated from metric accuracies on the same benchmark instances used for evaluation. Section 4.4 defines performance-weighted voting but does not state that weights are computed on a separate split; §5.2 reports the 'best-performing ensemble combination'; and §A says 'We use the same training and test sets across all experiments.' The reported 0.821 mean accuracy for ρall(PWV) is therefore a test-set-selected upper bound, not an unbiased estimate. The varying ensemble compositions across rows also indicate post hoc selection. A held-out procedure (e.g., nested cross-validation or a fixed dev split) is required before any claim of ensemble superiority can be made.
  2. [Tables 2-4] All accuracies are point estimates without confidence intervals, standard errors, or significance tests. With N=636, the standard error of a proportion near 0.8 is about 0.015, so the headline difference between ρall(PWV) at 0.821 and gpt-4.1 at 0.815 is within sampling noise. Moreover, Table 3 contradicts the abstract's 'consistently outperforms': in AA, ρall(PWV) is 0.802 versus gpt-4.1's 0.807, so the ensemble loses on that setting. Even granting clean labels, the central empirical claim is not statistically supported nor consistent across settings.
  3. [§4.2, Appendix E] The LLM setting assumes that T+ = M(q_ref, T'_ref) is stylistically closer to the reference than T- = M(q_ref, ∅), producing a meaningful binary ground-truth label. This assumption is not validated. The human annotation study in Appendix E found that annotators most often chose 'Both' for style preference (Figure 17b), and §6 admits that outputs of the same generator for the same query 'may share overlapping stylistic features.' If human annotators cannot distinguish T+ from T-, the labels are arbitrary, and the LLM-setting accuracies (e.g., 0.693 for ρall in Table 3) do not measure style discrimination. The benchmark's validity for this setting—and any conclusions drawn from it—requires either a validated generation procedure or human-verified labels.
minor comments (5)
  1. [§5.1] 'Close-sourced' should be 'closed-source'.
  2. [Figure 4 caption] The phrase 'T+ or T+' should read 'T+ or T−'.
  3. [Figure 17] The labels contain a typo: 'personalzied' should be 'personalized'.
  4. [Table 3] The composition uses 'gpt4.1' while the text and other tables use 'gpt-4.1'; please make the notation consistent.
  5. [Appendix A] The statement 'We use the same training and test sets across all experiments' is ambiguous given the development set introduced in §5.3 for APO; clarify the exact data splits used for each experiment.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline ensemble claim is fitted, not predicted: PWV weights and the 'best-performing' composition are both derived on the evaluation set itself, so 0.821 vs 0.815 is a selection upper bound, and Table 3's own AA row contradicts 'consistently outperforms.'

  1. fitted input called prediction [§4.4 (PWV definition); §5.2 & Table 3 (best-performing ensemble); Appendix A (same train/test sets)]
    "We explore two ensembling strategies: majority voting (MV) and performance-weighted voting (PWV). ... Table 3 presents the results of the best-performing ensemble combination across different metrics. ... We use the same training and test sets across all experiments."

    PWV weights are the constituent metrics' own accuracies, and no held-out split is described: unlike §5.3 (explicit 64-instance development set), Appendix A says 'We use the same training and test sets across all experiments.' The ρall(PWV) mean (0.821, Table 3) is thus a weighted vote fitted to the same 636 test labels used as the outcome, and §5.2 reports the 'best-performing ensemble combination' selected on that test set (Table 3 even lists different subsets for MV vs PWV, despite §4.4 defining ρall as ensembling over all metrics). 0.821 is a selection upper bound, not a prediction; declaring 'consistently outperforms' against unfitted gpt-4.1 (0.815) compares a fitted maximum to a fixed point, and the table contradicts the claim in AA (0.802 vs 0.807).

full rationale

The benchmark construction and the per-metric evaluation (Tables 1–2) are self-contained, non-circular contributions: each metric's accuracy on the 636-instance test set is computed directly, and the resulting ranking (gpt-4.1 > o4-mini > BLEU > StyleDistance) is an independent empirical result. No self-citation is load-bearing: De Wynter (2025a, 2025b) support contextual claims about prompt sensitivity and APO overfitting, and De Wynter et al. (2024) supplies the meta-prompting technique used to build fquery; none justify the ensemble-superiority claim. The circularity is confined to the evidence for the abstract's central claim. Performance-weighted voting takes the constituents' test accuracies as weights; no development split is described (Appendix A uses the same train/test sets, in contrast to the explicit 64-instance development set in §5.3). §5.2 reports the 'best-performing ensemble combination,' so the composition itself was selected on the reported score, and Table 3's MV and PWV rows list different subsets. The 0.821 mean is therefore a fitted/selected statistic, and comparing it with the unfitted gpt-4.1 single judge (0.815) does not test the claim. The 'consistently outperforms' wording is additionally contradicted by the table's AA row (0.802 vs 0.807), and the 0.006 mean gain is within sampling error at N=636, with no confidence intervals or paired tests reported. Separately, the LLM-setting ground truth (T+ vs T-) is an acknowledged construct-validity risk, not a circularity: the labels are defined by the generation procedure rather than by any metric, and the paper's own §6 and Appendix E document that annotators most often chose 'Both.' That weakens the LLM-setting accuracy numbers as a correctness risk, but it is not a circular derivation. Overall: partial circularity on the central claim through test-set-fitted weights and post-hoc composition selection, while the benchmark and individual-metric results remain independent.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the validity of the benchmark's ground-truth labels and on treating discrimination accuracy as a proxy for evaluation quality. The ensemble advantage also depends on test-set-fitted weights. No new unobserved entities are introduced; the style-discrimination benchmark is a dataset/task construction, not an invented entity.

free parameters (2)
  • PWV ensemble weights = Not reported per component, but derived from accuracy on the evaluation benchmark
    Performance-weighted voting weights are computed from metric accuracy on the same test instances used to report final scores (Section 4.4, Table 3). This is a fitted value that directly influences the reported ensemble advantage.
  • Ensemble composition = e.g., BLEU, ROUGE-L, Wegmann, StyleDistance, gpt-4.1 for ρall(PWV)
    The 'best performing ensemble combination' is selected per setting after inspecting results (Section 5.2, Table 3), which is a post hoc choice over the space of metric subsets.
assumptions (6)
  • domain assumption Texts by the same author are stylistically more similar to each other than to texts by different authors (authorship attribution setting).
    This justifies the ground-truth label in the AA setting, Section 4.2. If author style varies as much within an author as across authors, the labels are meaningless.
  • domain assumption Texts from different domains are stylistically distinguishable (domain discrimination setting).
    DD labels assume domain differences are a valid proxy for style differences, Section 4.2.
  • domain assumption T+ = M(q_ref, T'_ref) is stylistically closer to T_ref than T- = M(q_ref, ∅) in the LLM setting.
    This is the core label-generation assumption in Section 4.2, and the paper's own human study (Appendix E) shows humans often cannot distinguish T+ from T-, so this assumption is fragile.
  • domain assumption Cosine similarity over style embeddings (Wegmann, StyleDistance) is a valid measure of style closeness.
    Used to convert embedding distances into binary labels in Section 4.3; if the embeddings capture topic rather than style, the comparison is invalid.
  • domain assumption Accuracy on the binary style-discrimination task is a valid proxy for the quality of an SPTG evaluation metric.
    Adopted in Section 3 and acknowledged as a simplification in the Limitations section; if discrimination ability does not correlate with real SPTG evaluation validity, the whole benchmark measures something tangential.
  • ad hoc to paper The reconstructed user query fquery(T) approximates a realistic user query in the LLM setting.
    fquery is implemented via meta-prompting (De Wynter et al., 2024) and WildChat filtering (Section 4.2, Appendix A); the validity of the LLM setting depends on these reconstructed queries being representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Style-Personalized Text Generation: Challenges and Directions." pith.science (2026). https://pith.science/paper/ZQFZJD7K

@misc{pith2026250806374,
  author       = {Pith},
  title        = {Pith review of: Evaluating Style-Personalized Text Generation: Challenges and Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQFZJD7K}},
  note         = {Machine review of arXiv:2508.06374}
}
read the original abstract

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.

Figures

Figures reproduced from arXiv: 2508.06374 by the authors.

Figure 1
Figure 1. Pairwise disagreement of evaluation metrics for the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Average pairwise BertScore (Zhang et al., 2020) similarity across Tref , T+, and T−. Semantic overlap between the candidate set T+, T− is signifi￾cantly higher for the LLM compared to other two. factors: (a) both T+ and T− are generated by the same model for the same query (qref ), differing only in the inclusion of reference style text for T+. This is perhaps expected, as generation models ex￾hibit inherent stylist… view at source ↗
Figure 3
Figure 3. Accuracy of ρall(PW V ) across all domains. Enron emails and Reddit microblogs achieve lowest accuracy for AA and LLM evaluation settings, marginally outperforming the Random baseline [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Distribution of LLM-as-judge evaluation metric responses across different evaluation settings. Correct [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Ablation study results for LLM-as-judges prompting for the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Zero-shot evaluation prompt used to evaluate all LLM-as-judge metrics (Section [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Zero-shot evaluation prompt (Pstruct) used to evaluate parseable structured output in Section 5.3. stories* . Each instance was labeled by three anno￾tators. The IAA scores for the content preference was 0.779, for style preference was 0.641, and for copy￾editing prefe…
Figure 8
Figure 8. Figure 8: Zero-shot evaluation prompt (Pbinary) used to evaluate binary output in Section 5.3. <|im_start|>system You are a writing assistant. Your goal is to write a user query that can be used to generate a piece of text. You'll be provided the following inputs in |begin INPUT…
Figure 9
Figure 9. Figure 9: Query reconstruction prompt in chat markup format. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Personalized text generation prompt used to generate [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Non-personalized text generation prompt used to generate [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Classification prompt used to filter out English writing queries from the WildChat dataset ( [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Classification prompt used to obtain the writing task corresponding to the filtered English writing queries [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: WildChat user query classification statistics for writing task identification over randomly selected 50,000 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: WildChat user query classification statistics for categorizing the writing queries obtained from Figure [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Example instance of our evaluation dataset from the [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 18
Figure 18. Figure 18: Agreement statistics for three annotators for [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 17
Figure 17. Figure 17: Distribution of annotation responses for con [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Bashar Alhafni, Vivek Kulkarni, Dhruv Kumar, and Vipul Raheja. 2024. Personalized text generation with fine-grained linguistic control. arXiv preprint arXiv:2402.04914

    arXiv 2024

  5. [5]

    arXiv.org submitters. 2024. https://doi.org/10.34740/KAGGLE/DSV/7548853 arxiv dataset

    arXiv 2024

  6. [6]

    Hayden P Baker, Emma Dwyer, Senthooran Kalidoss, Kelly Hynes, Jennifer Wolf, and Jason A Strelzow. 2024. Chatgpt's ability to assist with clinical documentation: a randomized controlled trial. JAAOS-Journal of the American Academy of Orthopaedic Surgeons, 32(3):123--129

    work page 2024

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  8. [8]

    Archana Reddy Bongurala, Dhaval Save, Ankit Virmani, and Rahul Kashyap. 2024. Transforming health care with artificial intelligence: redefining medical documentation. Mayo Clinic Proceedings: Digital Health, 2(3):342--347

    work page 2024

Show all 48 references
  1. [9]

    James Carney and Cole Robertson. 2019. https://doi.org/10.17633/rd.brunel.7712540.v1 4000 stories with sentiment analysis dataset. brunel university london

    2019 doi

  2. [10]

    Nicholas Diakopoulos. 2019. Automating the news: How algorithms are rewriting the media. Harvard University Press

    2019

  3. [11]

    Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. 2020. A survey on ensemble learning. Frontiers of Computer Science, 14(2):241--258

    2020

  4. [12]

    Edenbd. 2020. https://www.kaggle.com/datasets/edenbd/150k-lyrics-labeled-with-spotify-valence 150k lyrics labeled with spotify valence

    2020

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  7. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [16]

    Vibhu Jain, Yash Goel, M Uma, et al. 2023. Ai powered transformative post generator for linkedin using llm and explicit filter. In 2023 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES), pages 1--7. IEEE

    2023

  9. [17]

    Kowe Kadoma, Dana \"e Metaxa, and Mor Naaman. 2025. Generative ai and perceptual harms: Who's suspected of using llms? In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1--17

    2025

  10. [18]

    Joshua Kaisen, Meng Li, and Shijie Lu. 2024. Ai and productivity: The impact of chatgpt's release on blogging. Available at SSRN 4858507

    2024

  11. [19]

    Mohamed Khalifa and Mona Albadawy. 2024. Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, page 100145

    2024

  12. [20]

    Bryan Klimt and Yiming Yang. 2004. Introducing the enron corpus. In CEAS, volume 4, page 1

    2004

  13. [21]

    Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. 2024 a . Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016

    2024 arXiv

  14. [22]

    V Kumar, Abdul R Ashraf, and Waqar Nadeem. 2024 b . Ai-powered marketing: What, where, and how? International Journal of Information Management, 77:102783

    2024

  15. [23]

    David Lewis. 1987. Reuters-21578 Text Categorization Collection . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C52G6M

    1987 doi

  16. [24]

    Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. 2024 a . Learning to rewrite prompts for personalized text generation. In Proceedings of the ACM on Web Conference 2024, pages 3367--3378

    2024

  17. [25]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024 b . Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579

    2024 arXiv

  18. [26]

    Weijiang Li, Yinmeng Lai, Sandeep Soni, and Koustuv Saha. 2025. Emails by llms: A comparison of language in ai-generated and human-written emails. In Proceedings of the 17th ACM Web Science Conference 2025, pages 391--403

    2025

  19. [27]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  20. [28]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  21. [29]

    Tao Long, Dorothy Zhang, Grace Li, Batool Taraif, Samia Menon, Kynnedy Simone Smith, Sitong Wang, Katy Ilonka Gero, and Lydia B Chilton. 2023. Tweetorial hooks: generative ai tools to motivate science on social media. arXiv preprint arXiv:2305.12265

    2023 arXiv

  22. [30]

    Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D Manning, and Daniel E Ho. 2024. Hallucination-free? assessing the reliability of leading ai legal research tools. Journal of Empirical Legal Studies

    2024

  23. [31]

    Julian John McAuley and Jure Leskovec. 2013. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference on World Wide Web, pages 897--908

    2013

  24. [32]

    Ibomoiye Domor Mienye and Yanxia Sun. 2022. A survey of ensemble learning: Concepts, algorithms, applications, and prospects. Ieee Access, 10:99129--99149

    2022

  25. [33]

    Sheshera Mysore, Debarati Das, Hancheng Cao, and Bahareh Sarrafzadeh. 2025. Prototypical human-ai collaboration behaviors from llm-assisted writing in the wild. arXiv preprint arXiv:2505.16023

    2025 arXiv

  26. [34]

    Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. 2023. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers. arXiv preprint arXiv:2311.09180

    2023 arXiv

  27. [35]

    Andy Nguyen, Yvonne Hong, Belle Dang, and Xiaoshan Huang. 2024. Human-ai collaboration patterns in ai-assisted academic writing. Studies in Higher Education, 49(5):847--864

    2024

  28. [36]

    Richard Yuanzhe Pang and Kevin Gimpel. 2018. Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer. arXiv preprint arXiv:1810.11878

    2018 arXiv

  29. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  30. [38]

    Ajay Patel, Nicholas Andrews, and Chris Callison-Burch. 2022. Low-resource authorship style transfer: Can non-famous authors be imitated? arXiv preprint arXiv:2212.08986

    2022 arXiv

  31. [39]

    Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2024. Styledistance: Stronger content-independent style embeddings with synthetic parallel examples. arXiv preprint arXiv:2410.12757

    2024 arXiv

  32. [40]

    Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, and David West Brown. 2025. Do llms write like humans? variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(8):e2422455122

    2025

  33. [41]

    Timothy A Rengers, Cornelius A Thiels, and Hojjat Salehinejad. 2024. Academic surgery in the era of large language models: a review. JAMA surgery, 159(4):445--450

    2024

  34. [42]

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406

    2023 arXiv

  35. [43]

    Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006. Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing weblogs, volume 6, pages 199--205

    2006

  36. [44]

    Anna Wegmann, Marijn Schraagen, and Dong Nguyen. 2022. Same author or just same topic? towards content-independent style representations. arXiv preprint arXiv:2204.04907

    2022 arXiv

  37. [45]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  38. [46]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  39. [47]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. https://openreview.net/forum?id=Bl8u7ZRlbM Wildchat: 1m chat GPT interaction logs in the wild . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Saurabh Bhausaheb Zinjad, Amrita Bhattacharjee, Amey Bhilegaonkar, and Huan Liu. 2024. Resumeflow: An llm-facilitated pipeline for personalized resume generation and refinement. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Inform...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.