REVIEW 2 major objections 4 minor 37 references
People who frequently use AI for writing tend to misjudge how much of the final text is their own work, while occasional users gauge it more accurately.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 00:24 UTC pith:UP7ZHYVV
load-bearing objection Construct is plausible, but the paper's central accuracy claim is not tested by its own statistics. the 2 major comments →
When AI Blurs the Boundaries of Contribution: An Empirical Study of Authorship Calibration
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that authorship, a construct traditionally assumed to be transparent to the writer, becomes opaque in AI-assisted writing, and the degree of opacity is systematically related to how much AI help a user requests. Concretely, the authors define authorship calibration as the difference between a user's declared authorship (the percentage they say they wrote) and their actual authorship (the percentage of final sentences attributable to their own keystrokes). Across 1,252 sessions they find a near-zero average calibration score but high variability, and when they split sessions at the median number of AI calls, the heavy-AI group shows significantly lower calibration
What carries the argument
The load-bearing instrument is the authorship calibration score, defined as declared authorship minus actual authorship, where actual authorship is computed from text-provenance metadata in the interaction logs. This score operationalizes the otherwise abstract idea of awareness: zero means perfect calibration, positive means overestimation, negative means underestimation. The second piece is the median split on number of AI calls to form low- and high-usage groups, compared with a rank-based test for distributional difference.
Load-bearing premise
The statistical comparison treats each writing session as an independent observation, but the data contain many sessions from the same writer, and sessions from one writer are likely correlated; if that correlation is strong, the reported group difference between light and heavy AI users could be an artifact rather than a real effect.
What would settle it
Reanalyze the session-level data with a multilevel model that includes writer as a random effect and compute the contrast between low- and high-AI-usage sessions within writers; if the contrast is not robustly negative (or the confidence interval includes zero), the frequency conclusion fails. An independent replication with a per-session authorship question asked immediately after writing would also test whether the effect is a survey-recall artifact.
If this is right
- In educational settings, miscalibrated authorship can distort metacognitive monitoring, potentially leading to overconfidence, reduced effort, or unrealistic self-assessment of learning.
- Authorship calibration offers a concrete, computable outcome measure for evaluating human-AI writing tools and interaction designs.
- Interventions that make text provenance visible (e.g., highlighting which sentences came from AI suggestions) could be evaluated by whether they move calibration scores toward zero.
- The near-zero group mean shows most writers are reasonably calibrated, so the problem is not universal but concentrated in high-use sessions, suggesting targeted interventions for that group.
- If heavy AI use degrades calibration, then pedagogical guidance on 'responsible AI use' may need to address self-awareness, not just product quality or plagiarism.
Where Pith is reading between the lines
- The effort-blend explanation suggests a testable sharpening: if asked about the process (edits, prompts, selections) rather than the final product, heavy AI users may claim what they 'worked on' rather than what they 'wrote'; an experiment that varies how the authorship question is framed could show calibration depends on frame.
- The same subtraction-of-declared-and-actual metric could be applied to code generation, image editing, or spreadsheet formulas, where 'authorship' is harder to define but equally relevant to learning analytics.
- Since the dataset pools multiple sessions per writer, a per-writer analysis (e.g., mixed-effects model) would reveal whether the heavy AI effect is driven by a few habitually overestimating users or is robust across individuals.
- The near-zero mean may hide a dissociation: light users are slightly underconfident and heavy users overconfident, meaning that the 'calibration problem' is really two distinct errors with possibly different mechanisms (conservatism vs. effort blend).
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces 'authorship calibration,' defined as the signed difference between a user's declared proportion of authorship and their actual proportion in AI-assisted writing, using the CoAuthor dataset. It categorizes 1,252 sessions into Low- and High-AI usage groups by median number of AI calls, reports descriptive statistics and a Mann–Whitney U test, and concludes that high-AI users tend to misjudge their authorship while low-AI users are more accurately calibrated. The paper frames the contribution as relevant to metacognition and learning analytics.
Significance. If the central claim were empirically established, the concept of authorship calibration could be a useful lens for studying human-AI collaboration and its educational implications. The paper has strengths: it builds on a public, rich dataset; the operationalization of calibration is simple and computable; and the authors provide code for reproducibility. However, the significance hinges on the empirical finding that AI usage frequency affects calibration accuracy, and that finding is not supported by the reported statistical analysis.
major comments (2)
- [Abstract / §5.2] The abstract claims that heavy AI users 'tend to misjudge their authorship' whereas low-AI users 'exhibit more accurate authorship calibration.' Accuracy of calibration is defined in Eq. (1) and the text as the absolute magnitude |Declared − Actual|. However, the only inferential test reported in §5.2 is a Mann–Whitney U test on the signed calibration scores (Declared − Actual). With group means of −0.004 and +0.003, both effectively zero, this test does not show that one group misjudges more. It at most shows a difference in the location/shape of the signed distributions. No test compares absolute miscalibration between groups. The discussion's claim of 'more accurate authorship calibration' for low-AI users is therefore unsupported by the evidence presented.
- [§5.2 / §4.1] The statistical comparison treats each writing session as an independent observation, but the 1,252 filtered sessions come from only 60 authors, with multiple sessions per author (Section 4.1). Sessions from the same author are likely correlated, violating the independence assumption of the Mann–Whitney U test. The reported p < 0.05 is therefore suspect. The analysis should account for clustering, for example by using a mixed-effects model or by aggregating scores per author before comparing groups.
minor comments (4)
- [§5.2] The text refers to 'heatmap confirms this pattern ... (See Figure 3a)' and later 'heatmap distribution ... (See Figure 5b)'. Figure 3 is a calibration curve, not a heatmap; the internal cross-references should be corrected.
- [§5.2] There are typographical and grammatical errors, e.g., 'users from the the Low-AI usage group' and 'our results confirms that the accuracy of authorship calibration vary.' A careful proofread is needed.
- [§4.3] The operationalization of 'declared authorship' relies on a single survey item ('—% of the essay/story is written by me...'). The paper does not discuss the validity or reliability of this item as a measure of perceived authorship, especially when users may interpret 'written by me' differently in AI-assisted contexts.
- [References] Several references are incomplete or informal (e.g., [4] is a Medium post, [24] is titled 'Genai et al. cocreation, authorship, ownership...'). The authors should ensure all citations meet the venue's standards.
Circularity Check
No significant circularity: authorship calibration is computed as a direct difference of two independently sourced quantities, and no load-bearing self-citation or fitted parameter is present.
full rationale
The paper's central construct is operationalized in Equation 1 as Declared Authorship − Actual Authorship, where declared authorship comes from a CoAuthor survey item and actual authorship comes from CoAuthor metadata on the proportion of text written by the user. These are two distinct, externally provided measurements; the paper does not fit a parameter and then rename it as a prediction. The split into Low- and High-AI usage groups is based on the median number of AI calls, following Shibani et al. [29], an external methodological citation, and is not derived from the calibration outcome. There are no self-citations by the present authors, no imported uniqueness theorem, and no ansatz smuggled in by citation. The skeptical concern that the only reported significance test compares signed calibration scores rather than the absolute miscalibration defined in Section 3 is a statistical-support issue about whether the conclusion 'more accurate' is established, not a circularity: the test does not reduce to the definition by construction, and the means and standard deviations are reported observations rather than fitted inputs. The construct-validation worry about using the same survey item to measure perceived authorship is a validity limitation, not a circular derivation. The paper is therefore self-contained with respect to circularity; any concerns about independence of sessions, construct validity, or inferential mismatch belong to a correctness/validity review, not to the circularity pass.
Axiom & Free-Parameter Ledger
free parameters (1)
- Median AI-call threshold for Low/High grouping =
11 AI calls
axioms (4)
- domain assumption The CoAuthor metadata 'actual proportion of text written by user' is an objective measure of actual authorship.
- domain assumption The single survey item '—% of the essay/story is written by me' validly measures perceived authorship.
- domain assumption Writing sessions are statistically independent observations.
- domain assumption Calibration is adequately represented by the arithmetic difference Declared - Actual.
invented entities (1)
-
Authorship calibration
no independent evidence
read the original abstract
The broad adoption of Artificial Intelligence (AI), especially Generative AI, raises pressing questions about how users interact with these systems to produce new content. In this paper, we introduce the concept of authorship calibration, defined as users awareness of their actual authorship when interacting with AI. Using the CoAuthor dataset, we empirically examine how authorship calibration varies across users and how it relates to their frequency of AI use. Our results reveal high variability: users relying heavily on AI tend to misjudge their authorship, whereas those using AI less frequently exhibit more accurate authorship calibration. These findings suggest that AI can obscure users perception of their own authorship. In learning contexts, miscalibration can affect metacognitive monitoring and learning strategies, ultimately impacting learning outcomes. Fostering authorship calibration then appears essential for promoting responsible and educationally meaningful AI integration.
Figures
Reference graph
Works this paper leans on
-
[1]
University students’ insights of generative artificial intelligence (ai) writing tools.Education Sciences, 14(10):1062, 2024
Al-Mothana M Gasaymeh, Mohammad A Beirat, and Asma’a A Abu Qbeita. University students’ insights of generative artificial intelligence (ai) writing tools.Education Sciences, 14(10):1062, 2024
2024
-
[2]
Gwo-Jen Hwang and Nian-Shing Chen. Exploring the potential of generative artificial intelligence in education: applications, challenges, and future research directions.Journal of Educational Technology & Society, 26(2), 2023
2023
-
[3]
The ai ghostwriter effect: When users do not perceive ownership of ai-generated text but self-declare as authors.ACM Transactions on Computer-Human Interaction, 31(2):1–40, 2024
Fiona Draxler, Anna Werner, Florian Lehmann, Matthias Hoppe, Albrecht Schmidt, Daniel Buschek, and Robin Welsch. The ai ghostwriter effect: When users do not perceive ownership of ai-generated text but self-declare as authors.ACM Transactions on Computer-Human Interaction, 31(2):1–40, 2024
2024
-
[4]
AI in writing class: Editor, co-author, ghostwriter, or muse? Medium, 2024
Glenn Kleiman. AI in writing class: Editor, co-author, ghostwriter, or muse? Medium, 2024. Accessed: July 17, 2026
2024
-
[5]
The impact of large language models on higher education: exploring the connection between ai and education 4.0
Iris Cristina Peláez-Sánchez, Davis Velarde-Camaqui, and Leonardo David Glasserman-Morales. The impact of large language models on higher education: exploring the connection between ai and education 4.0. InFrontiers in Education, volume 9, page 1392091. Frontiers Media SA, 2024
2024
-
[6]
Revolutionizing education with ai: Exploring the transformative potential of chatgpt.Contemporary educational technology, 15(3), 2023
Tufan Adiguzel, Mehmet Haldun Kaya, and Fatih Kürsat Cansu. Revolutionizing education with ai: Exploring the transformative potential of chatgpt.Contemporary educational technology, 15(3), 2023
2023
-
[7]
The promise and challenges of generative ai in education
Michail Giannakos, Roger Azevedo, Peter Brusilovsky, Mutlu Cukurova, Yannis Dimitriadis, Davinia Hernandez- Leo, Sanna Järvelä, Manolis Mavrikis, and Bart Rienties. The promise and challenges of generative ai in education. Behaviour & Information Technology, 44(11):2518–2544, 2025
2025
-
[8]
The role of metacognition in enhancing learning.Australian Journal of education, 32(2):127–138, 1988
John Biggs. The role of metacognition in enhancing learning.Australian Journal of education, 32(2):127–138, 1988
1988
-
[9]
Calibration of probabilities: The state of the art
Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D Phillips. Calibration of probabilities: The state of the art. InDecision Making and Change in Human Affairs: Proceedings of the Fifth Research Conference on Subjective Probability, Utility, and Decision Making, Darmstadt, 1–4 September, 1975, pages 275–324. Springer, 1977
1975
-
[10]
Calibration: What is it and why it matters? an introduction to the special issue on calibrating calibration.Learning and Instruction, 24:1–3, 2013
Patricia A Alexander. Calibration: What is it and why it matters? an introduction to the special issue on calibrating calibration.Learning and Instruction, 24:1–3, 2013
2013
-
[11]
Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. InProceedings of the 2022 CHI conference on human factors in computing systems, pages 1–19, 2022
2022
-
[12]
Art and the science of generative ai.Science, 380(6650):1110–1111, 2023
Ziv Epstein, Aaron Hertzmann, Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R Frank, Matthew Groh, Laura Herman, Neil Leach, et al. Art and the science of generative ai.Science, 380(6650):1110–1111, 2023
2023
-
[13]
Reza Hadi Mogavi, Chao Deng, Justin Juho Kim, Pengyuan Zhou, Young D Kwon, Ahmed Hosny Saleh Metwally, Ahmed Tlili, Simone Bassanelli, Antonio Bucchiarone, Sujit Gujar, et al. Exploring user perspectives on chatgpt: Applications, perceptions, and implications for ai-integrated education.arXiv preprint arXiv:2305.13114, 2023
Pith/arXiv arXiv 2023
-
[14]
Zied Bahroun, Chiraz Anane, Vian Ahmed, and Andrew Zacca. Transforming education: A comprehensive review of generative artificial intelligence in educational settings through bibliometric and content analysis.Sustainability, 15(17):12983, 2023
2023
-
[15]
A comprehensive review on generative ai for education.Ieee Access, 12:142733–142759, 2024
Uday Mittal, Siva Sai, Vinay Chamola, and Devika Sangwan. A comprehensive review on generative ai for education.Ieee Access, 12:142733–142759, 2024
2024
-
[16]
Rania Abdelghani, Hélène Sauzéon, and Pierre-Yves Oudeyer. Generative ai in the classroom: Can students remain active learners?arXiv preprint arXiv:2310.03192, 2023
Pith/arXiv arXiv 2023
-
[17]
Catalina Gomez, Sue Min Cho, Shichang Ke, Chien-Ming Huang, and Mathias Unberath. Human-ai collaboration is not very collaborative yet: a taxonomy of interaction patterns in ai-assisted decision making from a systematic review.Frontiers in Computer Science, 6:1521066, 2025
2025
-
[18]
Human-ai interaction: intermittent, continuous, and proactive.Interactions, 28(6):67–71, 2021
Niels Van Berkel, Mikael B Skov, and Jesper Kjeldskov. Human-ai interaction: intermittent, continuous, and proactive.Interactions, 28(6):67–71, 2021
2021
-
[19]
A multidimensional taxonomy for learner-ai interaction.Education and Information Technologies, 29(14):18361–18378, 2024
Bahar Memarian and Tenzin Doleck. A multidimensional taxonomy for learner-ai interaction.Education and Information Technologies, 29(14):18361–18378, 2024
2024
-
[20]
Human-ai collaboration patterns in ai-assisted academic writing.Studies in Higher Education, 49(5):847–864, 2024
Andy Nguyen, Yvonne Hong, Belle Dang, and Xiaoshan Huang. Human-ai collaboration patterns in ai-assisted academic writing.Studies in Higher Education, 49(5):847–864, 2024. 8 When AI Blurs the Boundaries of ContributionA PREPRINT
2024
-
[21]
Modifying ai, enhancing essays: How active engagement with generative ai boosts writing quality
Kaixun Yang, Mladen Rakovi ´c, Zhiping Liang, Lixiang Yan, Zijie Zeng, Yizhou Fan, Dragan Gaševi ´c, and Guanliang Chen. Modifying ai, enhancing essays: How active engagement with generative ai boosts writing quality. InProceedings of the 15th International Learning Analytics and Knowledge Conference, pages 568–578, 2025
2025
-
[22]
Generative models can help writers without writing for them
Kenneth C Arnold, April M V olzer, and Noah G Madrid. Generative models can help writers without writing for them. InIUI Workshops, 2021
2021
-
[23]
Supporting learner agency in collaborative writing with generative ai.British Journal of Educational Technology, 2025
Sujin Kim, Hyo-Jeong So, and Kyudong Park. Supporting learner agency in collaborative writing with generative ai.British Journal of Educational Technology, 2025
2025
-
[24]
Genai et al
Aras Bozkurt. Genai et al. cocreation, authorship, ownership, academic ethics and integrity in a time of generative ai, 2024
2024
-
[25]
Assessment in the age of artificial intelligence.Computers and Education: Artificial Intelligence, 3:100075, 2022
Zachari Swiecki, Hassan Khosravi, Guanliang Chen, Roberto Martinez-Maldonado, Jason M Lodge, Sandra Milligan, Neil Selwyn, and Dragan Gaševi ´c. Assessment in the age of artificial intelligence.Computers and Education: Artificial Intelligence, 3:100075, 2022
2022
-
[26]
Evidence-centered assess- ment for writing with generative ai
Yixin Cheng, Kayley Lyons, Guanliang Chen, Dragan Gaševi´c, and Zachari Swiecki. Evidence-centered assess- ment for writing with generative ai. InProceedings of the 14th learning analytics and knowledge conference, pages 178–188, 2024
2024
-
[27]
Exploring human-ai collaboration in educational contexts: Insights from writing analytics and authorship attribution
Hongchen Pan, Eduardo Araujo Oliveira, and Rafael Ferreira Mello. Exploring human-ai collaboration in educational contexts: Insights from writing analytics and authorship attribution. InProceedings of the 15th International Learning Analytics and Knowledge Conference, pages 903–909, 2025
2025
-
[28]
Automatic authorship analysis in human-ai collaborative writing
Aquia Richburg, Calvin Bao, and Marine Carpuat. Automatic authorship analysis in human-ai collaborative writing. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1845–1855, 2024
2024
-
[29]
Visual repre- sentation of co-authorship with gpt-3: Studying human-machine interaction for effective writing.International Educational Data Mining Society, 2023
Antonette Shibani, Ratnavel Rajalakshmi, Faerie Mattins, Srivarshan Selvaraj, and Simon Knight. Visual repre- sentation of co-authorship with gpt-3: Studying human-machine interaction for effective writing.International Educational Data Mining Society, 2023
2023
-
[30]
Visualizing authorship and contribution of collaborative writing in e-learning environments
Johnny Torres, Sixto García, and Enrique Peláez. Visualizing authorship and contribution of collaborative writing in e-learning environments. InProceedings of the 24th International Conference on Intelligent User Interfaces, pages 324–328, 2019
2019
-
[31]
The hallmark effect: Supporting provenance and transparent use of large language models in writing with interactive visualization
Md Naimul Hoque, Tasfia Mashiat, Bhavya Ghai, Cecilia D Shelton, Fanny Chevalier, Kari Kraus, and Niklas Elmqvist. The hallmark effect: Supporting provenance and transparent use of large language models in writing with interactive visualization. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2024
2024
-
[32]
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
John H Flavell. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10):906, 1979
1979
-
[33]
A conceptual framework for assessing motivation and self-regulated learning in college students
Paul R Pintrich. A conceptual framework for assessing motivation and self-regulated learning in college students. Educational psychology review, 16(4):385–407, 2004
2004
-
[34]
Becoming a self-regulated learner: An overview.Theory into practice, 41(2):64–70, 2002
Barry J Zimmerman. Becoming a self-regulated learner: An overview.Theory into practice, 41(2):64–70, 2002
2002
-
[35]
Using automated time management enablers to improve self-regulated learning.Active Learning in Higher Education, 23(1):3–15, 2022
Henry Khiat. Using automated time management enablers to improve self-regulated learning.Active Learning in Higher Education, 23(1):3–15, 2022
2022
-
[36]
Enhancing self-regulated learning and learning experience in generative ai environments: The critical role of metacognitive support.British Journal of Educational Technology, 2025
Xiaoqing Xu, Lifang Qiao, Nuo Cheng, Hongxia Liu, and Wei Zhao. Enhancing self-regulated learning and learning experience in generative ai environments: The critical role of metacognitive support.British Journal of Educational Technology, 2025
2025
-
[37]
Exploring the relationship between calibration and self-regulated learning.Educational psychology review, 12(4):437–475, 2000
Nancy J Stone. Exploring the relationship between calibration and self-regulated learning.Educational psychology review, 12(4):437–475, 2000. 9
2000
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.