TEASPN: Framework and Protocol for Integrated Writing Assistance Environments

Jun Suzuki; Kentaro Inui; Masato Hagiwara; Takumi Ito; Tatsuki Kuribayashi

REVIEW 4 major objections 5 minor 18 references

TEASPN: Framework and Protocol for Integrated Writing Assistance Environments

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TEASPN claims a single protocol can unify writing-assistance technologies in any editor, and a demo of it won preference from 10 of 12 writers.

desk verdict A useful protocol for writing-assistance integration, undermined by a user study whose fluency metric is too entangled with the demo's own GPT-2 suggestions. read the letter →

arxiv 1909.02621 v1 pith:QC33OUXV submitted 2019-09-05 cs.CL

Masato Hagiwara , Takumi Ito , Tatsuki Kuribayashi , Jun Suzuki , Kentaro Inui This is my paper · ORCID

classification cs.CL

keywords writingassistancetechnologyLanguageServerProtocolclient-servergrammaticalerrorcorrectiontextcompletionparaphraseintegratedenvironmentuserstudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TEASPN is a protocol, modeled on the Language Server Protocol, that standardizes how writing software talks to servers providing writing-assistance technologies—grammar correction, autocomplete, paraphrasing, dictionary hover, example search, and more. The paper's central claim is that this removes the N×M integration burden: every editor that speaks TEASPN can connect to every TEASPN server, so researchers can publish one server and reach many editors. To support the claim, the authors released an open-source SDK and built a demo server with eight assistance features, then tested it with twelve writers in a within-subject experiment. Ten of twelve participants preferred the integrated environment, participants used an average of 3.2 features, and texts written in the integrated condition had lower GPT-2 perplexity, which the authors read as evidence that writers produced more fluent English.

What carries the argument

The load-bearing object is TEASPN itself: a JSON-RPC-based protocol, forked from the Language Server Protocol, in which writing software is a client and assistance technologies are servers. The protocol defines request/response types for eight features, including two (syntax highlighting and search) that go beyond LSP; a server can push suggestions and the client can request completions, corrections, paraphrases, definitions, coreference jumps, and corpus searches. The TEASPN SDK provides a Python handler base class so that a WAT developer only implements the NLP logic—the sample completion handler is a few lines—while the library handles connection, text synchronization, and protocol details. That machinery reduces the integration problem from N×M pairs to N client implementations plus M server implementations.

What would settle it

Take the texts from the integrated and baseline conditions, match them by prompt and by length (e.g., same number of words), and recompute GPT-2 perplexity or have independent raters judge fluency. If the perplexity gap disappears while human ratings do not favor the integrated texts, the paper's fluency conclusion is not supported.

Watch

Extended reading notes

Core claim

The paper proposes that the long-standing 'great divide' between NLP writing-assistance research and real writing software can be closed by a standardized client-server protocol rather than by any single model or product. TEASPN forks the Language Server Protocol, reusing its data models and adding two features—server-side syntax highlighting and external-resource search—that natural-language writing needs. The demo implementation integrates eight assistance technologies under one protocol, and the user study reports that 10 of 12 participants rated the integrated environment better than using the same editor with no assistance plus external tools, 40% judged the integrated-condition text as better, and integrated-condition texts achieved lower perplexity (26.4 vs 37.8). The authors conclude that integrated writing assistance environments built on TEASPN provide a better writing experience and potentially help users write more fluent text.

Load-bearing premise

The claim that TEASPN helps people write more fluently assumes GPT-2 perplexity measures text quality, but the integrated-condition texts were shorter and the two conditions used different prompts assigned at random, so lower perplexity could come from brevity or topic rather than better writing.

Editorial extensions

If this is right

For writing-software developers, supporting TEASPN means writing one client library once; every TEASPN-compatible server (GEC, completion, paraphrase, search) becomes available without per-technology work.
For WAT researchers and developers, the SDK removes the need to write editor-specific integrations; a single server implementation exposes a model to any TEASPN client.
For writers, the integration shifts assistance from separate web apps and checkers into the editor itself, so features like example search, paraphrasing, and dictionary hover are one action away.
If the authors' results hold, integrated writing assistance changes writing behavior: participants voluntarily used an average of 3.2 features, indicating that low integration cost leads to broad feature adoption.
The observed lower perplexity of integrated-condition texts (26.4 vs 37.8) is the paper's evidence that the environment helps writers produce more typical, fluent English, not just that they enjoy the tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The N×M-to-N+M argument means the protocol's value grows quadratically with ecosystem size: once enough editors and WAT servers exist, integration cost is no longer a per-pair problem, which could shift the field's bottleneck to quality of the individual NLP components and latency of live suggestions.
Because the study's participants were mostly non-native English speakers, the fluency gain may be larger for L2 writers; a replication with native speakers would clarify whether integrated assistance primarily closes knowledge gaps (grammar, vocabulary, usage) or improves writing generally.
The same client-server pattern could extend beyond prose writing to other assisted-authoring domains, such as slide decks, scientific papers, or code comments, where NLP models currently need bespoke integrations.
A direct test of the fluency claim would replace GPT-2 perplexity with human-rated fluency or error counts on length-matched and prompt-matched texts; the paper's current evidence leaves room for the perplexity gap to be driven by text length or prompt assignment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

TEASPN is more valuable than it first looks. The protocol and SDK are a real contribution: adapting LSP to natural-language writing, adding server-side syntax highlighting and external search, and releasing an open-source Python SDK. The sample VS Code client under 200 lines is a nice concrete proof of the low integration cost. If you work on writing-assistance tools or on bridging NLP research and end-user editors, this is worth reading and citing as a baseline.

The soft spots are all in the empirical evaluation. The paper claims the integrated environment 'helped them write more fluent text,' but the evidence is thin. Twelve participants, no significance tests, and the 10-of-12 preference could partly come from demand effects, since the authors personally demonstrated every feature. The perplexity result is the bigger problem: the demo's phrase completion uses the same GPT-2 family as the metric. If participants accepted any of those suggestions, the integrated texts are partly written by the model used to score fluency, so lower perplexity is partly self-consistency, not writer fluency. The paper doesn't report how often suggestions were used or accepted, so the confound is unaddressed. The length difference (335 vs 379 chars) and the different prompts only add to the uncertainty. The authors themselves note the length gap but don't engage with the metric issue.

That said, the protocol itself stands. The core claim about integration cost is supported by the architecture and the working SDK; it does not depend on the user study. The evaluation should have been presented as a feasibility demo, not as evidence of improved writing. As an engineering paper, the contribution is solid.

Recommendation: send it to a serious referee. The framework is worth a place in the literature, and the review process can push the authors to either fix the evaluation or soften the claims. I'd cite the protocol and SDK, not the fluency result.

Referee Report

4 major / 5 minor

Summary. The paper proposes TEASPN, a protocol and open-source framework for integrated writing assistance environments, built as a fork of the Language Server Protocol (LSP). It standardizes client-server communication between writing software and writing assistance technologies (WATs), provides an SDK to ease WAT integration, and reports a demo system with features including syntax highlighting, grammatical error detection/correction, completion, rewriting, jump, hover, and search. The authors evaluate the demo with twelve participants who wrote short English texts under an INTEGRATED condition (editor with TEASPN) and a BASELINE condition (same editor without WATs, but with access to external tools). The evaluation reports that most participants preferred the integrated experience, that a minority believed their texts were better in the integrated condition, and that integrated-condition texts had lower GPT-2 perplexity, which the authors interpret as evidence of more fluent writing.

Significance. If the empirical claims held, the paper would make a useful systems contribution: it addresses a real integration bottleneck in writing-assistance technology, and its protocol design builds pragmatically on LSP, the SDK is released as open source, and the sample client reportedly requires under 200 lines of code. The protocol and framework are technically plausible and the demo appears functional. However, the load-bearing empirical claim that the integrated environment helps users write more fluent text is weakly supported by the evidence presented, for reasons detailed in the major comments. The framework contribution itself remains credible and potentially valuable to the NLP and CALL communities, so the paper is worth revising rather than rejecting.

major comments (4)

[§4.2, §5, Table 2] The perplexity comparison is confounded by a circularity: the demo's phrase-completion feature uses a neural language model (Section 4.2, Radford et al. 2019), and the fluency metric in Section 5 is GPT-2 small perplexity. If participants accepted any GPT-2-based completion suggestions, the resulting text is partially generated by the same model family used as the metric, so lower integrated-condition perplexity could partly reflect self-consistency of accepted suggestions rather than improved writer fluency. The paper does not report how often participants accepted phrase completions, or analyze the texts with completion usage controlled. This undermines the specific claim that 'the integrated writing environment helped them write more fluent and/or typical English text.'
[§5, Table 2] The empirical comparison is statistically uninterpretable as reported: with n=12, no significance tests are given for the perplexity difference (37.8 vs. 26.4) or for the questionnaire responses, and the integrated-condition texts were shorter (335 vs. 379 characters on average). Since GPT-2 perplexity is sensitive to text length and topical predictability, and the two prompts ('activity you enjoy' vs. 'hometown') have different lexical predictability, the observed perplexity gap may reflect brevity or prompt assignment rather than fluency. The authors should report per-prompt results, statistical tests (or effect sizes with confidence intervals), and ideally account for text length.
[§5, Figure 5] The subjective-quality result is overstated relative to the abstract and conclusion. Q1 shows 10/12 (83.3%) preferred the integrated experience, which is a valid user-experience claim, but Q2 shows only 40% believed they wrote better texts in the INTEGRATED condition, with 40% reporting 'about the same' and the remainder (20%) favoring BASELINE. With twelve participants, 40% is not distinguishable from chance, and it does not support the conclusion that users 'can write more fluent text.' The paper should either temper the fluency claim to a user-experience/preference claim or provide stronger quality evidence.
[§4.1, §5] The BASELINE condition allowed participants to use any external writing tools (e.g., Grammarly, web dictionaries), but the paper does not report whether or how much these tools were actually used. Without measuring external-tool usage in the BASELINE condition, the comparison conflates the integrated environment with the mere presence of assistance features, weakening the claim that the integrated TEASPN environment specifically provides benefit. A usage log or post-session questionnaire about external tool use is needed.

minor comments (5)

[§5, Figure 5] The reported 40% for Q2 corresponds to about 5 of 12 participants; the paper should state exact counts rather than rounded percentages to avoid ambiguity.
[§1] There is a typographical artifact in the abstract/introduction: 'TEASPN1' should be 'TEASPN' with the footnote marker separated cleanly, and 'W ATs' appears with an inconsistent space throughout.
[§5] The sentence 'the quality of the inidividual W ATs improve' contains a typo ('inidividual' should be 'individual').
[§2] The text says 'RDL/OWL-based format' for the NLP Interchange Format; the standard is RDF/OWL, so this appears to be a typo.
[§4.2] The paper should specify the GPT-2 model size used for phrase completion, since the evaluation uses GPT-2 small for perplexity; if a different size was used for completion, that should be stated explicitly.

Circularity Check

1 steps flagged · score 4.0 of 10

Fluency evidence partially circular: the integrated demo's phrase-completion feature and the fluency metric both use GPT-2, so lower perplexity may reflect accepted model completions rather than independent writer fluency.

other [Section 4.2 (Implementation of the Demo System) and Section 5 (Results and Analysis), Table 2]
"We implemented two types of completion features: one which suggests the likely next phrases given the context using a neural language model (Radford et al., 2019) ... Perplexity was calculated using the pretrained GPT-2 model (small, 117M parameters) (Radford et al., 2019). The texts written in the INTEGRATED condition had lower perplexity, suggesting that the integrated writing environment helped them write more fluent and/or typical English text."

Both the completion feature and the fluency metric invoke the same GPT-2 model (Radford et al., 2019). A phrase completion suggested by the neural LM is by construction a continuation with high probability under that LM, so any text incorporating an accepted completion will have low GPT-2 perplexity regardless of the writer's own skill. Since Q3 reports an average of 3.2 features used and the paper does not report acceptance counts, the 26.4 vs 37.8 perplexity gap conflates model self-consistency with writer fluency. This makes the 'more fluent text' conclusion partially circular, although the protocol and SDK claims are unaffected.

full rationale

The paper's central engineering contributions—the TEASPN protocol, SDK, sample client under 200 lines of TypeScript, and demo server integrating LanguageTool, SpaCy, coreference, WordNet, and Tatoeba/Elasticsearch—are concrete and independent of the fluency metric; they do not reduce to fitted parameters or self-citations. The user-experience preference (10/12) is also not circular. The only questionable link is the Section 5 fluency analysis: the integrated demo's phrase-completion feature and the perplexity metric both use the GPT-2 model (Radford et al., 2019). Accepted completions are likely continuations under GPT-2, so they lower GPT-2 perplexity by construction. The paper neither reports acceptance counts nor controls for this overlap, so the 26.4 vs 37.8 perplexity gap supports the 'more fluent text' claim only partially; it is at least partly a self-consistency artifact, compounded by the reported length difference (335 vs 379 chars) and prompt heterogeneity. This is a partial circularity in one supporting analysis, not a circularity of the protocol design. Hence a moderate score of 4 rather than 0 or 6.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

TEASPN introduces no free parameters or speculative entities; as an engineering contribution, it rests on domain assumptions about protocol standardization and the evaluation's validity.

assumptions (3)

domain assumption The Language Server Protocol is a suitable foundation for a natural-language writing assistance protocol.
TEASPN is a fork of LSP and reuses its data models (Section 3), assuming the IDE-language integration model transfers to writing software and WATs.
domain assumption GPT-2 perplexity is a valid measure of written text fluency or quality.
Section 5 uses perplexity from a pretrained GPT-2 small model to argue integrated texts are more fluent, without validating this metric for short learner texts.
domain assumption Self-reported writing experience is a reliable indicator of writing assistance quality.
Q1 and Q2 are subjective Likert-style responses (Figure 5) and are treated as evidence of effectiveness without objective outcome measures.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TEASPN: Framework and Protocol for Integrated Writing Assistance Environments." pith.science (2026). https://pith.science/paper/QC33OUXV

@misc{pith2026190902621,
  author       = {Pith},
  title        = {Pith review of: TEASPN: Framework and Protocol for Integrated Writing Assistance Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QC33OUXV}},
  note         = {Machine review of arXiv:1909.02621}
}

read the original abstract

Language technologies play a key role in assisting people with their writing. Although there has been steady progress in e.g., grammatical error correction (GEC), human writers are yet to benefit from this progress due to the high development cost of integrating with writing software. We propose TEASPN, a protocol and an open-source framework for achieving integrated writing assistance environments. The protocol standardizes the way writing software communicates with servers that implement such technologies, allowing developers and researchers to integrate the latest developments in natural language processing (NLP) with low cost. As a result, users can enjoy the integrated experience in their favorite writing software. The results from experiments with human participants show that users use a wide range of technologies and rate their writing experience favorably, allowing them to write more fluent text.

Figures

Figures reproduced from arXiv: 1909.02621 by the authors.

**Figure 2.** Architecture of TEASPN. Requests can be sent in both directions, which are often triggered by some events (such as user input) and can be responded with additional data (such as results of GEC). TEASPN clients and servers can be written in any programming language as long as they conform to the protocol. 3.2 Features By integrating a large selection of WATs in a single platform, TEASPN makes them available to write… view at source ↗

**Figure 3.** Code snippet for computing completion periments with end-users (writers) in order to answer the following research question: Does the integrated writing assistance environment developed with TEASPN provide better writing experience and help write better texts? To explore the effectiveness of integrated writing assistance environment, we compared the following two conditions. In the INTEGRATED condition, partici… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Screenshot of our demo system. the authors of this paper showed the participants all the features of the demo system for a demonstration purpose. After the writing sessions, they were asked questions regarding their writing experience. Because the focus of this exper…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 14 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Jim Chang and Jason Chang. 2015. https://doi.org/10.3115/v1/N15-3022 W rite A head2: Mining lexical grammar patterns for assisted writing . In Proceedings of the NAACL 2015 System Demonstrations , pages 106--110

doi:10.3115/v1/n15-3022 2015
[4]

Mei-Hua Chen, Shih-Ting Huang, Hung-Ting Hsieh, Ting-Hui Kao, and Jason S. Chang. 2012. https://www.aclweb.org/anthology/P12-3027 FLOW : A first-language-oriented writing assistant system . In Proceedings of ACL 2012 System Demonstrations , pages 157--162

work page 2012
[5]

Gmail Smart Compose: Real-Time Assisted Writing

Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. https://arxiv.org/pdf/1906.00080.pdf Gmail smart compose: Real-time assisted writing

work page Pith review arXiv 2019
[6]

Anne Vandeventer Faltin. 2003. Natural language processing tools for computer assisted language learning. Linguistik Online, 17(5):137--153

work page 2003
[7]

Maarten van Gompel, Iris Hendrickx, Antal van den Bosch, Els Lefever, and V \'e ronique Hoste. 2014. https://doi.org/10.3115/v1/S14-2005 S em E val 2014 task 5 - L 2 writing assistant . In Proceedings of S em E val 2014 , pages 36--44

work page doi:10.3115/v1/s14-2005 2014
[8]

Sebastian Hellmann, Jens Lehmann, and Soren Auer. 2012. NIF : An ontology-based and linked-data-aware NLP interchange format. Technical report, Working Draft

work page 2012

Show all 18 references

[9]

Toru Ishida. 2006. L anguage G rid: An infrastructure for intercultural collaboration. In IEEE/IPSJ Symposium on Applications and the Internet (SAINT 2006), pages 96--100

2006
[10]

Corrado, L \'a szl \'o Luk \'a cs, Marina Ganea, Peter Young, and Vivek Ramavajjala

Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Gregory S. Corrado, L \'a szl \'o Luk \'a cs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart reply: Automated response suggestion for email. In Proceedings of KDD 2016

2016
[11]

Philippe Langlais, George Foster, and Guy Lapalme. 2000. https://www.aclweb.org/anthology/W00-0507 T rans T ype: a computer-aided translation typing system . In ANLP - NAACL 2000 Workshop: Embedded Machine Translation Systems

2000
[12]

Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. Automated Grammatical Error Detection for Language Learners. Morgan & Claypool

2010
[13]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI

2019
[14]

Hubert Soyer, Goran Topi \'c , Pontus Stenetorp, and Akiko Aizawa. 2015. https://doi.org/10.3115/v1/N15-3019 C ro V e WA : Crosslingual vector-based writing assistance . In Proceedings of NAACL 2015 System Demonstrations , pages 91--95, Denver, Colorado

2015 doi
[15]

Irena Srdanović. 2011. https://www.aclweb.org/anthology/P12-3027 Evaluating e-resources for J apanese language learning . In Proceedings of eLex 2011 , pages 260--267

2011
[16]

Kiri Wagstaff. 2012. Machine learning that matters. In Proceedings of the Twenty-Ninth International Conference on Machine Learning (ICML), pages 529--536

2012
[17]

John Wieting and Kevin Gimpel. 2018. https://www.aclweb.org/anthology/P18-1042 P ara NMT -50 M : Pushing the limits of paraphrastic sentence embeddings with millions of machine translations . In Proceedings of ACL 2018, pages 451--462

2018
[18]

Tzu-Hsi Yen, Jian-Cheng Wu, Jim Chang, Joanne Boisson, and Jason Chang. 2015. https://doi.org/10.3115/v1/P15-4024 W rite A head: Mining grammar patterns in corpora for assisted writing . In Proceedings of ACL - IJCNLP 2015 System Demonstrations , pages 139--144

2015 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Jim Chang and Jason Chang. 2015. https://doi.org/10.3115/v1/N15-3022 W rite A head2: Mining lexical grammar patterns for assisted writing . In Proceedings of the NAACL 2015 System Demonstrations , pages 106--110

doi:10.3115/v1/n15-3022 2015

[4] [4]

Mei-Hua Chen, Shih-Ting Huang, Hung-Ting Hsieh, Ting-Hui Kao, and Jason S. Chang. 2012. https://www.aclweb.org/anthology/P12-3027 FLOW : A first-language-oriented writing assistant system . In Proceedings of ACL 2012 System Demonstrations , pages 157--162

work page 2012

[5] [5]

Gmail Smart Compose: Real-Time Assisted Writing

Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. https://arxiv.org/pdf/1906.00080.pdf Gmail smart compose: Real-time assisted writing

work page Pith review arXiv 2019

[6] [6]

Anne Vandeventer Faltin. 2003. Natural language processing tools for computer assisted language learning. Linguistik Online, 17(5):137--153

work page 2003

[7] [7]

Maarten van Gompel, Iris Hendrickx, Antal van den Bosch, Els Lefever, and V \'e ronique Hoste. 2014. https://doi.org/10.3115/v1/S14-2005 S em E val 2014 task 5 - L 2 writing assistant . In Proceedings of S em E val 2014 , pages 36--44

work page doi:10.3115/v1/s14-2005 2014

[8] [8]

Sebastian Hellmann, Jens Lehmann, and Soren Auer. 2012. NIF : An ontology-based and linked-data-aware NLP interchange format. Technical report, Working Draft

work page 2012