Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Reviewriter: AI-Generated Instructions For Peer Review Writing

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reviewriter embeds a fine-tuned German GPT-2 in a web tool that generates peer-review instructions, and its 14-student evaluation reports positive technology acceptance on every measured construct.

desk verdict Design work and qualitative insights are solid, but the quantitative claim of positive technology acceptance is not supported by the data—worth peer review with careful revisions. read the letter →

arxiv 2506.04423 v1 pith:XPC3AR4L submitted 2025-06-04 cs.HC

Xiaotian Su , Thiemo Wambsganss , Roman Rietsche , Seyed Parsa Neshaei , Tanja Käser This is my paper
classification cs.HC
keywords peerreviewwritingAI-generatedinstructionswriter'sblockGermanGPT-2technologyacceptancelanguagemodelfine-tuningeducationalNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reviewriter is a web tool that gives students AI-generated instructions while they write peer reviews in German, with the goal of breaking writer's block. The paper derives five user requirements from interviews with twelve graduate students, turns them into seven design principles, and implements them around a German GPT-2 model fine-tuned on 11,925 student-written business peer reviews. In an evaluation with fourteen students, all five technology-acceptance measures—perceived ease of use, ease of interaction, excitement, enjoyment, and usefulness—came out above the neutral midpoint of the seven-point scale. Qualitative feedback adds that the instructions helped students find new ideas and continue their reviews, alongside reports of irrelevant content, grammar errors, and long waiting times. The paper therefore aims to establish that a fine-tuned open-source German language model can be embedded in a student-accepted tool for peer review writing, and that the design rationales are reusable by other educational-tool builders.

What carries the argument

The load-bearing mechanism is next-sentence prediction from a German GPT-2 model fine-tuned on 11,925 cleaned student peer reviews. In the Reviewriter interface, the model takes the student's last twenty words after a spacebar press, waits eight seconds, and returns three inline gray suggestions, each capped at about forty-five words, that the student can accept with the Tab key, reject with Esc, or toggle with the arrow keys; accepted text enters the review and feeds the next generation. Around this model the paper places seven functionalities derived from five interview-based user requirements: a clean interface, guidance, keyboard control, a simple text area, inline presentation, complete argument suggestions, and a word-count display. This design keeps the AI output inside a student-controlled writing flow rather than presenting it as a separate chatbot.

What would settle it

A blind evaluation in which more than two raters score a larger sample of generated instructions (for example, 100 per model) on fluency and correctness, or a larger user study using the same technology-acceptance items: if German GPT-2 does not beat BLOOM or if the mean usefulness rating falls at or below the neutral 4, the paper's central claims would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a student-facing tool which generates next-sentence instructions from a German GPT-2 model, fine-tuned on a filtered corpus of 11,925 peer reviews from a business course, is perceived positively by tertiary students in a peer-review writing task. Quantitatively, the fourteen participants rated the tool above the neutral value of 4 on all five technology-acceptance constructs: ease of use (mean 6.07), ease of interaction (5.50), excitement (5.64), enjoyment (5.43), and usefulness (4.64). The authors interpret these scores as favorable acceptance of the generative AI application. Qualitative results support the intended purpose: students reported overcoming writer's block, receiving novel ideas to explore, and writing faster, while a minority found the suggestions irrelevant or ungrammatical and the eight-second waiting time too long. The paper positions this as evidence that embedding a fine-tuned open-source LLM in a deliberately designed writing interface is a viable way to deliver adaptive instructions in real educational settings.

Load-bearing premise

The load-bearing premise is that the 11,925 peer reviews from one business course at a single university are representative of the peer-review writing students will do, and that the two researchers' ratings of just ten generated samples per model are reliable enough to pick German GPT-2 as the right model.

Editorial extensions

If this is right

  • Students can receive three on-demand instruction suggestions without leaving the text editor, using keyboard shortcuts to accept, reject, or toggle between them.
  • A fine-tuned German GPT-2 with 124 million parameters outperformed a 560-million-parameter multilingual model and a 774-million-parameter German model in the authors' fluency and correctness checks, providing an open-source baseline for German educational text generation.
  • The eight-second delay before showing suggestions gives students thinking room while writing; some found it helpful, others found it slowed them down, making latency a tunable design parameter.
  • Peer-review writing tools based on generative AI are perceived as easy to use and enjoyable by students, but perceived usefulness (mean 4.64) is the weakest of the five measured constructs and an obvious target for improvement.
  • The authors' planned next steps include comparing how students perceive reviews written by peers, reviews written with Reviewriter, and fully AI-generated reviews, and integrating the tool into a university peer review system to study long-term effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The model was trained only on business-model peer reviews, so applying it to peer reviews in other disciplines such as lab reports or essays would require new fine-tuning or transfer testing.
  • The acceptance evidence is self-reported and has no control condition; a randomized study comparing review quality, writing fluency, and perceived effort with and without AI instructions would test whether the positive ratings translate into learning gains.
  • The eight-second latency is presented as a way to reduce disruption, but interactive-systems research would predict a tension between helpful reflection and frustrating waiting; measuring when suggestions feel timely is a concrete, testable extension.
  • The paper compares three open-source generative models but not a prompted general-purpose LLM, so whether fine-tuning is necessary or whether the design principles transfer to a zero-shot API model remains open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Reviewriter, a web-based tool that provides AI-generated instructions to support German-speaking students in writing peer reviews. The authors derive design principles from a literature review and interviews with twelve graduate students, implement the tool with a fine-tuned German GPT-2 model, and evaluate it in a mixed-method study with fourteen students. The paper reports positive technology acceptance based on quantitative measures (means on five TAM constructs) and qualitative feedback indicating benefits such as overcoming writer's block, while also noting limitations such as hallucination and response delay.

Significance. If the claims held, the paper would contribute a working prototype, design principles, and a fine-tuning comparison for a specific educational writing task, which is a useful step for the HCI/educational-NLP community. The authors are transparent about their data source, use open-source models, and provide qualitative quotes that give concrete insight into student experiences. However, the central quantitative claim of 'positive technology acceptance' rests on descriptive statistics from a small self-selected sample without inferential tests, and thus the evidence is considerably weaker than the abstract and conclusion suggest.

major comments (3)
  1. [Section 4.2, Table 3] The claim that 'All results are better than the neutral value of four' is not supported by inferential statistics. For perceived usefulness, M=4.64, SD=1.34, N=14, a two-sided one-sample t-test against the neutral midpoint of 4 yields t≈1.79, p≈0.097, which is not significant at the conventional 0.05 level. Since perceived usefulness is the construct most directly tied to adoption, the abstract's 'positive technology acceptance based on quantitative measures' is an overstatement. Please report appropriate inferential tests (e.g., one-sample t-tests or Wilcoxon signed-rank tests with correction for multiple comparisons) and interpret the results accordingly, or explicitly reframe the quantitative results as descriptive and exploratory.
  2. [Section 4.1 and Section 5] The evaluation has no control condition and no baseline measurement of students' prior peer review writing ability or prior familiarity with AI tools. Consequently, the study cannot support causal claims that Reviewriter improves writing or that observed acceptance differs from a non-AI setting. The manuscript should be revised to present the evaluation as a descriptive feasibility/acceptance study, and conclusions in Sections 1 and 5 should be tempered accordingly.
  3. [Section 3.3] The selection of German GPT-2 over BLOOM rests on a human evaluation of only ten generated instructions per model, rated by two researchers, with no reported inter-rater reliability, no explicit rating rubric beyond 'fluency and correctness,' and no example of the evaluation instrument. This is a thin basis for the claim that German GPT-2 is 'the best' model. Please provide the full evaluation details (criteria, scores, agreement), or soften the claim to state that German GPT-2 was chosen based on the authors' informal assessment, not a rigorous human evaluation.
minor comments (5)
  1. [Author affiliation] The affiliation for the second university contains a typo: 'Universtiy' should be 'University'.
  2. [Figure 1 caption] The caption says 'pre-possessed dataset'; this should be 'pre-processed dataset'.
  3. [Appendix A.2, Table 4] The expansion for 'dh' is listed as 'da her'; the correct German term is 'das heißt', typically abbreviated 'd.h.' Please correct this.
  4. [Section 4.2] The sentence 'The results show that the participants rate positively using Reviewriter to receive adaptive instructions' is missing an object; it should read 'rate the use of Reviewriter positively' or similar.
  5. [Appendix B.2] The post-survey includes items on intention to use (e.g., item 1 and item 13), but the paper does not report a separate 'behavioral intention' construct in Table 3. Please clarify whether these items were included in one of the reported constructs or omitted from the analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the acceptance claim is a direct measurement, not a prediction from a fitted model, and the reused corpus is an external prior dataset.

full rationale

The paper does not contain a derivation in which an output is defined in terms of its own inputs or a fitted parameter is later renamed as a prediction. The main claim, 'positive technology acceptance based on quantitative measures' (Abstract; Section 4.2, Table 3), is a direct descriptive summary of the 14-student questionnaire, not a quantity predicted from the fine-tuned model or from the corpus. The fine-tuning corpus is taken from Wambsganss et al. (2022b), a separate, previously published COLING paper with overlapping authors, but the present work uses it as existing training data and independently selects German GPT-2 via training loss plus a human evaluation of fluency and correctness (Section 3.3); the bias/quality properties of the corpus are cited from that earlier external publication, so this is real evidence rather than a self-referential loop. Model choice is an engineering selection, not a 'prediction' of the later acceptance scores. The remaining limitations — small self-selected N, descriptive statistics without inferential tests, and four evaluation participants who had joined the design interviews — are validity and generalizability concerns, not circularity, because no result is equivalent to its inputs by construction. No load-bearing premise is justified solely by the authors' own prior work, so the derivation chain is self-contained for the claims made.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters were fitted to data. The paper relies on domain assumptions about the corpus, the validity of self-report measures, and the adequacy of the small human evaluation. Reviewriter is a software artifact, not a theoretical invented entity.

assumptions (4)
  • domain assumption The 11,925-review corpus is representative of German peer review writing in educational settings.
    The corpus comes from a single business course at one university (Section 3.1). The paper assumes its characteristics generalize to the target task and population.
  • domain assumption Self-reported Likert-scale responses reflect actual user acceptance and system quality.
    The central claim relies on questionnaire responses from 14 students (Section 4.2), with no objective measure of writing quality or behavior.
  • domain assumption The reliability of the two researchers' ratings on ten generated samples is sufficient for model selection.
    Section 3.3 selects German GPT-2 based on this small human evaluation without inter-rater reliability statistics.
  • domain assumption The design decisions (25-word threshold, 8-second delay, three suggestions, 60-token limit) are appropriate and were not tuned to the evaluation data.
    These settings are justified by literature and interviews (Section 3.4), but they are hand-picked and might affect the user experience in ways not fully measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reviewriter: AI-Generated Instructions For Peer Review Writing." pith.science (2026). https://pith.science/paper/XPC3AR4L

@misc{pith2026250604423,
  author       = {Pith},
  title        = {Pith review of: Reviewriter: AI-Generated Instructions For Peer Review Writing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPC3AR4L}},
  note         = {Machine review of arXiv:2506.04423}
}
read the original abstract

Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate \texttt{Reviewriter}, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.

Figures

Figures reproduced from arXiv: 2506.04423 by the authors.

Figure 1
Figure 1. Overview of our methodology: We first gather system needs and requirements from literature and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of Reviewriter to provide AI-generated instructions for students to write peer reviews. First, students enter initial input, which is then used by the German GPT-2 model to generate instructions. The students evaluate the generated content and decide whether to regenerate it. Following this, students are free to edit the instructions. Finally, both the generated text and the student’s text are utilized … view at source ↗
Figure 3
Figure 3. A screenshot of Reviewriter and its main functionalities (F1 - F7) derived from system requirements and design principles. The system provides a clean interface (F1). By clicking the question mark, students get detailed guidance on the peer review writing task and the usage of the tool (F2). A simple text area supports all typical interactions, such as typing, selecting, editing, and deleting text, and caret movemen… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Overview of the study procedure. Students begin with five pre-test questions and two introduction videos. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 35 canonical work pages

  1. [1]

    I like experimenting and trying out new tech- nologies

    work page

  2. [2]

    As a rule, I am hesitant when trying out new technologies

    work page

  3. [3]

    Supporting Cognitive and Emotional Empathic Writing of Students

    Supporting cognitive and emotional empathic writing of students.arXiv preprint arXiv:2105.14815 (2021). Thiemo Wambsganss, Vinitra Swamy, Roman Rietsche, and Tanja Käser. 2022b. Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer- Review Data Modeling. InProceedings of the 29th International Conference on Computational Linguis- tics...

    work page Pith review arXiv 2021

  4. [4]

    When I hear about new technologies I look for a way to experiment with them

    work page

  5. [5]

    I have had experience writing re- views/feedback in the past. B.2 Post-test questions asked during evaluation ofReviewriter • Transition questions: How many times have you accepted Reviewriter’s recommenda- tions? • Technology Acceptance Model

    work page

  6. [6]

    In my circle of friends, I’m usually the first person to try new digital media / new tech- nologies

    work page

  7. [7]

    Compared to other participants, I think I wrote a very convincing re- view/feedback

    work page

  8. [8]

    After using Reviewriter, my ability to write reviews/feedback has improved

    work page

Show all 36 references
  1. [9]

    Assuming the review writing assistance tool is available, the next time I want to write a review/feedback I would use it again

  2. [10]

    With Reviewriter I can write re- views/feedback more effectively

  3. [11]

    Learning to use Reviewriter was easy for me

  4. [12]

    I find using Reviewriter useful for writ- ing reviews/feedbacks

  5. [13]

    I find Reviewriter easy to interact with

  6. [14]

    It would be easy for me to become famil- iar withReviewriter

  7. [17]

    I’m sure I wrote a very insightful re- view/feedback

  8. [18]

    I’m sure I wrote a very convincing re- view/feedback

  9. [19]

    With Reviewriter I can write better re- views/ feedbacks

  10. [20]

    I think I now know more about how to write well-structured, persuasive, and in- sightful reviews/feedbacks

  11. [21]

    Assuming Reviewriter was available, the next time I write a review/feedback I would use it

  12. [22]

    • Evaluate student perceptions on the AI- generated instructions

    After using Reviewriter, my ability to pay attention to the different parts of the review/feedback structure has improved. • Evaluate student perceptions on the AI- generated instructions

  13. [23]

    I expect Reviewriter will help me im- prove my ability to write well-structured reviews/feedbacks

  14. [24]

    I assume Reviewriter would help me improve my ability to write compelling reviews/feedback

  15. [25]

    I assume Reviewriter would help me improve my ability to write insightful reviews/feedback

  16. [26]

    Interacting with the tool was fun and en- joyable for me

  17. [27]

    I expect Reviewriter will help me im- prove my ability to write helpful re- views/feedback

  18. [28]

    • Open-ended questions for qualitative feed- back

    Interacting with the tool was exciting. • Open-ended questions for qualitative feed- back

  19. [29]

    How has Reviewriter impacted your writing process?

  20. [30]

    What did you particularly like about us- ingReviewriter?

  21. [31]

    Do you have any other ideas?

  22. [32]

    What could still be improved?

  23. [33]

    Grammarly)?

    Have you used a writing support program before (e.g. Grammarly)?

  24. [34]

    What is your field of study?

  25. [35]

    Please enter your gender

  26. [36]

    company" with

    Please indicate your mother tongue. B.3 Clustered qualitative student feedback from the evaluation ofReviewriter Source Text Original input First of all, I can say that your first task was solved well. You have structured the environmental and company analysis well, writ- ten ...

  27. [2016]

    Educ.96, C (may 2016), 55–71

    Effects of Different Online Peer-Feedback Approaches on Students’ Performance Skills, Moti- vation and Self-Efficacy in a Dance Course.Comput. Educ.96, C (may 2016), 55–71. https://doi.org/ 10.1016/j.compedu.2016.02.004 María Soledad Ibarra-Sáiz, Gregorio Rodríguez-Gómez, and ...

    2020

  28. [2021]

    Zihan Gao and Jiepu Jiang

    Collaborative peer feedback and learning ana- lytics: Theory-oriented design for supporting class- wide interventions.Assessment & Evaluation in Higher Education46, 2 (2021), 169–190. Zihan Gao and Jiepu Jiang. 2021. Evaluating Human-AI Hybrid Conversational Systems with Chatb...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.