REVIEW 2 major objections 6 minor 57 references
WikiSTAR: A System for Shedding Light on the Hidden History of Scientific Wikipedia Articles
T0 review · 2 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A new system reveals the scientific history hidden in Wikipedia edits.
desk verdict Worth engaging: a solid new task and benchmark for scientific edit classification on Wikipedia, with a real but fixable gap around section linking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the section edit pair: each edited Wikipedia section is matched to its previous version using title similarity (normalized Levenshtein) and content similarity (cosine over embeddings), and the pair is then classified by a large language model against a ten-label taxonomy of scientific edit types. The section level provides the context needed to judge scientific significance while localizing changes, and the taxonomy—spanning additions, removals, clarifications, technical terms, researcher names, narrative shifts, references, wikilinks, and quantitative information—drives both the classifier and the interactive visualizations. The classifier's refined prompt, whi
What would settle it
Take a random sample of the section edit pairs produced by WikiSTAR's linking step and have human annotators judge whether each pair truly represents the same section before and after the edit; if a substantial fraction are misaligned—say, over 10%—the classification labels no longer reliably describe what changed.
Extended reading notes
Core claim
The paper claims that scientifically meaningful changes in Wikipedia articles can be identified at the section level and separated from routine edits using a fixed taxonomy of ten scientific edit types, and that this classification makes the scientific history of an article accessible at a scale previously impossible. The claim is supported by WikiSTAR-BENCH, a human-annotated dataset of 1,387 section edit pairs, on which a carefully prompted LLM classifier reaches macro-F1 0.82, near the human agreement ceiling of 0.91. The paper further claims that the resulting interactive views—trends over time, per-section heatmaps, section evolution, and an edit table—all grounded in classified edits,
Load-bearing premise
The section-linking heuristic that pairs each edited section with its previous version is reported to 'work well in practice' but is never quantitatively evaluated, so misaligned pairs would silently corrupt every classification and view built on them.
Editorial extensions
If this is right
- Researchers can trace when new findings enter a Wikipedia article, how sections are renamed or restructured, and where editorial attention concentrates, without manual reading of thousands of edits.
- Comparisons across articles and fields become feasible, revealing how different scientific domains are represented and updated on Wikipedia.
- The released benchmark gives the community a standard, human-annotated evaluation set for the new task of scientific edit classification.
- Journalists and historians can ground claims about how public knowledge of science changes in concrete, dated edit evidence rather than anecdote.
- The user study suggests the system can prompt new research questions across Wikipedia editing, science journalism, and the history and philosophy of science.
Reading between the lines
- Because the section-linking step is validated only informally, a systematic misalignment of sections across revisions would silently corrupt both the labels and all downstream views; a dedicated validation study of the matching heuristic is a natural next step.
- The taxonomy is content-centric and does not label editorial disputes, reverts, or citation quality changes; extending it to capture controversies or references to retracted work could open new research directions.
- Since the classifier is an LLM prompted in English and evaluated on English Wikipedia, its behavior on other language editions or on less-covered, niche topics is unknown; testing cross-lingual and low-resource transfer is a plausible extension.
- If the approach were applied at the scale of all scientific Wikipedia articles rather than a few, it could produce a large-scale corpus of scientific knowledge evolution, enabling quantitative historiography of science.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. WikiSTAR is an end-to-end system for surfacing scientifically meaningful edits in the revision histories of scientific Wikipedia articles. The pipeline parses revisions into sections, links each edited section to its previous version, classifies each section-edit pair into a ten-label taxonomy of scientific edit types using an LLM, and provides interactive views (history overview, trends over time, per-section heatmap, section evolution, edit table). The paper introduces WIKISTAR-BENCH, a human-annotated benchmark of 1,387 section edits across biology, computer science, and neuroscience, and evaluates seven LLMs as classifiers, reporting best macro-F1 values of 0.82–0.83. A user study with three domain experts reports that the system surfaced new patterns and enabled previously impractical analyses. The paper claims that WikiSTAR makes the scientific edit history of Wikipedia accessible to researchers, journalists, and the public.
Significance. If the results hold, WikiSTAR is a timely and potentially valuable resource for studying how scientific knowledge evolves on Wikipedia. The benchmark is a concrete contribution: it is human-annotated with a high inter-annotator agreement (mean pairwise Cohen's κ = 0.89), per-label and per-domain results are reported, and the code, data, and demo are released. The multi-label evaluation across seven models is methodologically clean, and the prompt-refinement comparison is a useful practical result. The main uncertainty is whether the section-pair construction, which underlies both the benchmark and the deployed system, is accurate enough; this must be addressed before the headline performance numbers can be fully trusted. The user study is encouraging but anecdotal.
major comments (2)
- [Sec. 2.1 ('Matching section edit pairs')] The section-linking heuristic is load-bearing but unvalidated. The paper states that title/content similarity thresholds 'worked well in practice' and defers advanced matching, but gives no accuracy numbers and does not specify the threshold values. Every downstream artifact depends on this step: WIKISTAR-BENCH is built from these pairs, the macro-F1 in Table 1 is computed on them, and the interactive views inherit them. Sections are frequently renamed, split, merged, or moved, so this is not a trivial step. Please report a quantitative matching evaluation (e.g., precision/recall of recovered predecessors on a human-annotated sample stratified by section identity changes), report the thresholds, and analyze how matching errors affect the classification metrics.
- [Secs. 3.1–3.3 (LLM-generated summaries)] The generated History Overview and per-point summaries are a core part of the end-to-end 'accessible history' claim, but their accuracy is not evaluated. The system can return fluent but incorrect narratives; the user study quotes enthusiasm for these features, but no factuality or faithfulness check is reported. Add at least a small human evaluation of summary correctness, or clearly scope the claims to the labeled edit table.
minor comments (6)
- [Sec. 2.3 (Annotation process)] Please specify how many annotators labeled each non-shared example, how labels were aggregated (majority vote, consensus, adjudication), and whether the Non-Scientific Edit label was exclusive or could co-occur with scientific labels. Also clarify how the 104 shared examples were selected.
- [Sec. 2.4 / Table 1] The comparison of GPT-5.4's macro-F1 (computed on n=1,387) to the human agreement ceiling (computed on n=104) is not apples-to-apples. Report the human ceiling on the same benchmark split, or add a caveat that the two numbers are measured on different sets.
- [Table 5 (Appendix C)] The table layout with duplicated Precision/Recall blocks is difficult to read, and identical values across o3-mini, GPT-4o, and GPT-5.1 look suspicious. Please merge the columns and verify the rounding.
- [Sec. 4 (User Study)] The user study is a pilot with three participants, no comparison condition, and self-reported Likert ratings. This is acceptable as qualitative evidence, but please report the full nine-item questionnaire, the participant selection criteria, and any relationship between participants and the authors.
- [Sec. 3.1 and general system description] In the deployed system, generated summaries should be clearly marked as AI-generated in the UI, with links to the underlying source edits. The participants' desire to validate summaries (e.g., P2's comment) supports this design choice.
- [Appendix A, Table 3 note] Minor typo: 'released with our our code repository' should read 'released with our code repository'.
Circularity Check
No circular derivation chain; the benchmark and prompted-LLM evaluation are independent, and the only self-citations are motivational.
full rationale
WikiSTAR's pipeline is: parse revisions into sections, match section edit pairs (§2.1), classify against a 10-label taxonomy with an LLM (§2.2–2.4), and visualize. The performance claim (macro-F1 0.82) is measured against WIKISTAR-BENCH, a human-annotated gold standard (1,387 examples; mean pairwise κ=0.89, Section 2.3). The LLMs are prompted, not trained on the benchmark, so the classifier is not fit to the test set; no parameter is estimated from the labels it is later asked to predict. The taxonomy is author-designed, but the benchmark labels come from independent annotators applying that taxonomy, and the human agreement ceiling (F1 0.91) gives a check that the labeling task is not vacuous. The section-linking heuristic (§2.1) is unvalidated and load-bearing for the pipeline, but it is a data-construction step with external inputs (revision text, similarity measures), not an output derived from the classification; its lack of quantitative evaluation is a correctness/robustness concern, not a circularity. The self-citations to Benjakob et al. (2021, 2023) involve co-author Aviram but are used only to motivate the importance of Wikipedia revision history; they do not supply the classification result or the benchmark. Hence the central claim is self-contained against an external human benchmark, and no step reduces by construction to its own inputs. Score 2 reflects the minor, non-load-bearing self-citation, not substantive circularity.
Assumptions & free parameters
free parameters (1)
- Section matching thresholds
assumptions (3)
- domain assumption Wikipedia revision history retrieved via the MediaWiki API is a complete and faithful record of all edits.
- ad hoc to paper The expert-designed ten-label taxonomy captures what 'scientifically meaningful change' means, with Non-Scientific Edit covering everything else.
- domain assumption LLM classifiers, evaluated on Biology, Computer Science, and Neuroscience, generalize to other scientific articles beyond the benchmark domains.
Cite this review
Pith. "Pith review of WikiSTAR: A System for Shedding Light on the Hidden History of Scientific Wikipedia Articles." pith.science (2026). https://pith.science/paper/32WX33EC
@misc{pith2026260712441,
author = {Pith},
title = {Pith review of: WikiSTAR: A System for Shedding Light on the Hidden History of Scientific Wikipedia Articles},
year = {2026},
howpublished = {\url{https://pith.science/paper/32WX33EC}},
note = {Machine review of arXiv:2607.12441}
}
read the original abstract
Wikipedia plays a key role in shaping public understanding of science, and its openly accessible revision history is a unique record of how scientific knowledge evolves over time. Yet scientifically meaningful revisions are obscured by the sheer volume of routine edits, leaving each article's scientific history hidden. We present WikiSTAR (Scientific Tracking of Article Revisions), an interactive system for exploring scientifically meaningful changes across an article's revision history. Using an LLM classifier with an expert-designed multi-label taxonomy, WikiSTAR first tags edit types such as the addition of technical terms, new research findings, and changes in scientific narrative. Then, through interactive views, an article's full revision history can be traced at any granularity - from aggregate trends that reveal when and in which sections scientific content was added or refined, down to individual edits - showing how scientific knowledge develops at a scale previously impossible. In a user study, experts from three domains found that WikiSTAR surfaced new patterns and research questions and enabled previously impractical analyses. We release our system, code and a human-annotated benchmark.
Figures
Reference graph
Works this paper leans on
-
[1]
Omer Benjakob, Rona Aviram, and Jonathan Sobel. 2021. Citation needed? wikipedia and the covid-19 pandemic. BioRxiv
2021
-
[2]
Omer Benjakob, Olha Guley, Jean-Marc Sevin, Leo Blondel, Ariane Augustoni, Matthieu Collet, Louise Jouveshomme, Roy Amit, Ariel Linder, and Rona Aviram. 2023. Wikipedia as a tool for contemporary history of science: A case study on crispr. PLoS One, 18(9):e0290827
2023
-
[3]
Jacob Cohen. 1960. https://api.semanticscholar.org/CorpusID:15926286 A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20:37 -- 46
1960
-
[4]
Johannes Daxenberger and Iryna Gurevych. 2012. https://api.semanticscholar.org/CorpusID:11053125 A corpus-based study of edit categories in featured and non-featured wikipedia articles . In International Conference on Computational Linguistics
2012
-
[5]
Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. https://api.semanticscholar.org/CorpusID:271571434 ...
2024
-
[6]
Romy Fieblinger, Md Tanvirul Alam, and Nidhi Rastogi. 2024. Actionable cyber threat intelligence using knowledge graphs and large language models. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 100--111. IEEE
2024
-
[7]
Fabian Fl \"o ck and Maribel Acosta. 2014. https://doi.org/10.1145/2566486.2568026 WikiWho : Precise and efficient attribution of authorship of revisioned content . In Proceedings of the 23rd International Conference on World Wide Web (WWW), pages 843--854. ACM
arXiv 2014
-
[8]
Yuhan Guo, Qin Han, Yuke Lou, Yiming Wang, Can Liu, and Xiaoru Yuan. 2023. https://doi.org/10.1109/PacificVis56936.2023.00025 Edit-history vis: An interactive visual exploration and analysis on wikipedia edit history . In 2023 IEEE 16th Pacific Visualization Symposium (PacificVis), pages 157--166. IEEE
arXiv 2023
Show all 57 references
-
[9]
OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex P...
2024
-
[10]
Jun Liu and Sudha Ram. 2011. https://api.semanticscholar.org/CorpusID:15299619 Who does what: Collaboration patterns in the wikipedia and their impact on article quality . ACM Trans. Manag. Inf. Syst., 2:11:1--11:23
2011
-
[11]
OpenAI . 2025 a . https://openai.com/index/introducing-gpt-5/ Introducing gpt-5 . Accessed: 2026-06-02
2025
-
[12]
OpenAI . 2025 b . https://cdn.openai.com/o3-mini-system-cardfeb10.pdf Openai o3-mini system card . Accessed: 2026-02-18
2025
-
[13]
Dheeraj Rajagopal, Xuchao Zhang, Michael Gamon, Sujay Kumar Jauhar, Diyi Yang, and Eduard Hovy. 2022. One document, many revisions: a dataset for classification and description of edit intents. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, page...
2022
-
[15]
heroes of crispr
Arno Simons, Wolfgang Kircheis, Marion Schmidt, Martin Potthast, and Benno Stein. 2024. https://api.semanticscholar.org/CorpusID:268060504 Who are the “heroes of crispr”? public science communication on wikipedia and the challenge of micro-notability . Public Understanding of ...
2024
-
[16]
Misha Teplitskiy, Grace Lu, and Eamon Duede. 2015. https://api.semanticscholar.org/CorpusID:10220883 Amplifying the impact of open access: Wikipedia and the diffusion of science . Journal of the Association for Information Science and Technology, 68
2015
-
[17]
Vi \'e gas, Martin Wattenberg, and Kushal Dave
Fernanda B. Vi \'e gas, Martin Wattenberg, and Kushal Dave. 2004. https://doi.org/10.1145/985692.985765 Studying cooperation and conflict between authors with history flow visualizations . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pag...
2004
-
[18]
Wikimedia Foundation . 2025. MediaWiki Action API . https://www.mediawiki.org/wiki/API:Main_page. Accessed: 2026-07-08
2025
-
[19]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://api.semanticscholar.org/Cor...
2025
-
[20]
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in wikipedia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000--2010
2017
-
[21]
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages=
Identifying semantic edit intentions from revisions in wikipedia , author=. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages=
2017
-
[22]
arXiv preprint arXiv:2306.12587 , year=
Aries: A corpus of scientific paper edits made in response to peer reviews , author=. arXiv preprint arXiv:2306.12587 , year=
-
[23]
arXiv preprint arXiv:2410.02028 , year=
Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions , author=. arXiv preprint arXiv:2410.02028 , year=
-
[24]
2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) , year=
Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability , author=. 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) , year=
2023
-
[25]
psychometrika , volume=
Coefficient alpha and the internal structure of tests , author=. psychometrika , volume=. 1951 , publisher=
1951
-
[26]
PLoS One , volume=
Wikipedia as a tool for contemporary history of science: A case study on CRISPR , author=. PLoS One , volume=. 2023 , publisher=
2023
-
[27]
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages=
Language of vandalism: Improving wikipedia vandalism detection via stylometric analysis , author=. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages=
-
[28]
Proceedings of the 51st annual meeting of the Association for Computational Linguistics (volume 1: long papers) , pages=
Linguistic models for analyzing and detecting biased language , author=. Proceedings of the 51st annual meeting of the Association for Computational Linguistics (volume 1: long papers) , pages=
-
[29]
Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages=
One document, many revisions: a dataset for classification and description of edit intents , author=. Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages=
-
[30]
arXiv preprint arXiv:2404.03428 , year=
Edisum: Summarizing and Explaining Wikipedia Edits at Scale , author=. arXiv preprint arXiv:2404.03428 , year=
-
[31]
arXiv preprint arXiv:2203.03802 , year=
Understanding iterative revision from human-written text , author=. arXiv preprint arXiv:2203.03802 , year=
-
[32]
Journal of biological rhythms , volume=
A Clockwork Wikipedia: from a broad perspective to a case study , author=. Journal of biological rhythms , volume=. 2018 , publisher=
2018
-
[33]
bioRxiv , pages=
High Impact: Wikipedia sources and edit history document two decades of the climate change field , author=. bioRxiv , pages=. 2023 , publisher=
2023
-
[34]
BioRxiv , year=
Citation needed? Wikipedia and the COVID-19 pandemic , author=. BioRxiv , year=
-
[35]
Information & Culture , volume=
Writing the Revolution: Wikipedia and the Survival of Facts in the Digital Age by Heather Ford , author=. Information & Culture , volume=. 2024 , publisher=
2024
-
[36]
Quantitative science studies , volume=
COVID-19 research in Wikipedia , author=. Quantitative science studies , volume=. 2020 , publisher=
2020
-
[37]
arXiv preprint arXiv:2409.09010 , year=
Contri (e) ve: Context+ Retrieve for Scholarly Question Answering , author=. arXiv preprint arXiv:2409.09010 , year=
-
[38]
2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) , pages=
Actionable cyber threat intelligence using knowledge graphs and large language models , author=. 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) , pages=. 2024 , organization=
2024
-
[39]
ArXiv , year=
How Does Knowledge Come By? , author=. ArXiv , year=
-
[40]
International Conference on Computational Linguistics , year=
A Corpus-Based Study of Edit Categories in Featured and Non-Featured Wikipedia Articles , author=. International Conference on Computational Linguistics , year=
-
[41]
ACM Trans
Who does what: Collaboration patterns in the wikipedia and their impact on article quality , author=. ACM Trans. Manag. Inf. Syst. , year=
-
[42]
Conference on Empirical Methods in Natural Language Processing , year=
Automatically Classifying Edit Categories in Wikipedia Revisions , author=. Conference on Empirical Methods in Natural Language Processing , year=
-
[43]
ArXiv , year=
Understanding Iterative Revision from Human-Written Text , author=. ArXiv , year=
-
[44]
Heroes of CRISPR
Who are the “Heroes of CRISPR”? Public science communication on Wikipedia and the challenge of micro-notability , author=. Public Understanding of Science (Bristol, England) , year=
-
[45]
The World Wide Web Conference , year=
Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability , author=. The World Wide Web Conference , year=
-
[46]
Journal of the Association for Information Science and Technology , year=
Amplifying the impact of open access: Wikipedia and the diffusion of science , author=. Journal of the Association for Information Science and Technology , year=
-
[47]
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI) , pages =
Studying Cooperation and Conflict between Authors with History Flow Visualizations , author =. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI) , pages =. 2004 , publisher =
2004
-
[48]
Proceedings of the 23rd International Conference on World Wide Web (WWW) , pages =
Fl. Proceedings of the 23rd International Conference on World Wide Web (WWW) , pages =. 2014 , publisher =
2014
-
[49]
2023 IEEE 16th Pacific Visualization Symposium (PacificVis) , pages =
Edit-History Vis: An Interactive Visual Exploration and Analysis on Wikipedia Edit History , author =. 2023 IEEE 16th Pacific Visualization Symposium (PacificVis) , pages =. 2023 , publisher =
2023
-
[50]
Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) , volume =
Towards Better Visual Tools for Exploring Wikipedia Article Development---The Use Case of ``Gamergate Controversy'' , author =. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) , volume =
-
[51]
2023 IEEE 16th Pacific Visualization Symposium (PacificVis) , year=
Edit-History Vis: An Interactive Visual Exploration and Analysis on Wikipedia Edit History , author=. 2023 IEEE 16th Pacific Visualization Symposium (PacificVis) , year=
2023
-
[52]
Educational and Psychological Measurement , year=
A Coefficient of Agreement for Nominal Scales , author=. Educational and Psychological Measurement , year=
-
[53]
2025 , url=
Qwen3 Technical Report , author=. 2025 , url=
2025
-
[54]
2024 , url=
The Llama 3 Herd of Models , author=. 2024 , url=
2024
-
[55]
2024 , url=
GPT-4o System Card , author=. 2024 , url=
2024
-
[56]
OpenAI o3-mini System Card , year =
-
[57]
Introducing GPT-5 , year =
-
[58]
Scientometrics , year=
A diachronic perspective on citation latency in Wikipedia articles on CRISPR/Cas-9: an exploratory case study , author=. Scientometrics , year=
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.