REVIEW 3 major objections 6 minor 35 references
SciCom Wiki: Fact-Checking and FAIR Knowledge Distribution for Scientific Videos and Podcasts
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A Wikibase-based platform for science videos and podcasts can make non-textual media FAIR and computationally fact-checkable, and its prototype was judged suitable by stakeholders and users.
desk verdict A genuinely useful FAIR catalog for science media backed by a well-run requirements study, but the central fact-checking claim is an unvalidated pipeline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a three-stage pipeline: a linked-data digital library whose data model covers more than 200 media qualities, with a full-text wiki storing transcripts and a search page as the main interface; an LLM-based statement extractor that turns transcripts, PDFs, or web text into aligned triples; and a veracity checker that searches the ground-truth knowledge graph for exact matches and otherwise applies the shortest-path graph-distance method from the computational fact-checking literature to estimate semantic proximity. This graph-distance score is the load-bearing proxy for factuality, and the accuracy score $s_{\mathrm{acc}} = \sum_i s_i w_i$ is currently reduced to veracity because the additional metrics, such as context, clearness, authenticity, subjectivity, and confidence, lack computational quantifications.
What would settle it
Take a set of climate statements with known expert verdicts, including hedged, sarcastic, and true-but-misleading formulations, run them through the pipeline, and compare the veracity scores with the expert ratings; if the scores do not separate true from false statements substantially better than chance, the graph-distance proxy is not a valid accuracy measure.
Extended reading notes
Core claim
The central claim is that a digital library built on linked-data wiki technology, extended with a full-text wiki and search and import services, provides a feasible foundation for the Science Communication Knowledge Infrastructure: it makes videos and podcasts findable, accessible, interoperable, and reusable, and it enables a neurosymbolic computational fact-checking workflow. In that workflow, heterogeneous media are first converted into text, then into subject-predicate-object triples by a large language model, aligned by normalizing synonyms and base forms, and compared against a trusted ground-truth knowledge graph assembled from the headline statements of the authoritative climate-science synthesis report. Statements that are not exact matches receive a veracity indication by a shortest-path graph-distance check, and an overall accuracy score is computed as $s_{\mathrm{acc}} = \sum_i s_i w_i$; in the current implementation only veracity contributes ($w_{\mathrm{ver}}=1$). The paper reports that expert interviews and a public survey found the tool necessary and usable, and concludes that the SciCom Wiki is suitable to address the requirements raised.
Load-bearing premise
The claim collapses if the graph-distance score between LLM-extracted triples and trusted-report triples does not actually track whether a statement is true; the authors themselves call the path check an indication, not a sufficient check.
Editorial extensions
If this is right
- A FAIR wiki node can serve as a central access point for scientific videos and podcasts, aggregating metadata and transcripts that are today scattered across isolated repositories.
- Computational fact-checking of audio-visual media becomes feasible at scale: LLM extraction plus knowledge-graph alignment produces a per-statement veracity indication without requiring every video to be watched and checked by hand.
- The same pipeline transfers to other domains once a trusted, machine-readable ground-truth knowledge graph exists, since the extraction and alignment steps are presented as domain-independent.
- Because the path check is explicitly not a sufficient verification, the practical system still depends on human-in-the-loop supervision to catch hallucinated or misaligned triples.
- User demand is not hypothetical: the surveys and interviews show that stakeholders want accuracy filtering and fact-check support more than reach metrics such as views or subscriber counts.
Reading between the lines
- Editorial inference: the validity of the veracity score has not yet been demonstrated against an external benchmark; comparing it with human expert verdicts on a sample of climate claims would be the natural next test.
- Editorial inference: the architecture could be reused outside climate communication, for instance in health or election discourse, wherever a consensus synthesis exists to build the ground-truth graph.
- Editorial inference: the authors' own discussion implies that the accuracy score should be treated as a triage signal that prioritizes claims for professional fact-checkers, not as a final verdict, since context, sarcasm, and subjectivity are explicitly out of scope.
- Editorial inference: a successful deployment probably requires community and institutional scaffolding like the one that sustains Wikipedia rather than the software alone, which the paper also acknowledges.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SciCom Wiki, a Wikibase-based FAIR digital library for scientific videos and podcasts, together with a neurosymbolic computational fact-checking pipeline that converts media text into knowledge graph triples and scores statements against a ground-truth knowledge graph built from IPCC AR6 headline statements. Requirements were elicited from 53 survey participants and refined in 11 interviews; the library was evaluated with 14 participants (including UEQ and ASQ instruments), and the fact-checking tool was evaluated through 10 expert interviews and a 43-participant user survey. The authors claim that the system as a whole is a suitable foundation for a FAIR Science Communication Knowledge Infrastructure and that it addresses the most requested feature, fact-checking.
Significance. If the fact-checking claim were validated, the paper would make a useful contribution to an important and under-served area: FAIR, collaborative infrastructure for non-textual science communication media. The strengths of the paper are its open-source artifacts, reproducible survey instruments, explicit requirement elicitation, and a transparently reported usability evaluation with appropriate statistical tests (Wilcoxon/t-tests, Bonferroni-Holm correction). The weakness is that the central fact-checking claim is not validated: no precision/recall, no comparison with human fact-checkers, no error analysis of the LLM triple extraction, and no verification of the ground-truth knowledge graph are reported. The accuracy score reduces by construction to a veracity heuristic that the authors themselves describe as only an indication, not a sufficient check. The paper therefore demonstrates a well-received prototype library and a fact-checking pipeline architecture, but not a validated fact-checking capability.
major comments (3)
- [Section 4.1, 'Veracity Checking' and 'Quantify additional metrics'] The scientific accuracy score s_acc is defined as a weighted sum, but the paper states that no computational quantification was found for the additional metrics and that w_ver=1; hence s_acc reduces to the veracity score. The veracity check itself uses exact match or a Ciampaglia-style path distance that the paper explicitly says 'should only be interpreted as an indication, not a sufficient check.' No evaluation of the correctness of this score is reported: there is no precision/recall, no comparison against human fact-checkers, no gold-standard set, and no error analysis of the LLM triple extraction. The user survey in Section 4.2 measures perceived usefulness and necessity, not whether the scores are accurate. Consequently, the paper's central claim that the system addresses the most requested feature, fact-checking, is not established by the presented evidence.
- [Section 4.1, 'Extension' and 'Alignment'] The ground-truth knowledge graph is constructed from IPCC AR6 headline statements using the same LLM-based triple extraction pipeline that the expert interviews in Section 4.2 characterize as 'unreliable for semantic parsing' and prone to hallucinated or non-reproducible triples. The paper reports no verification of the ground-truth graph, such as human validation of the extracted triples or comparison with an independently constructed climate knowledge graph. If the ground truth contains extraction errors, the veracity scores computed against it inherit those errors, so the fact-checking output is not trustworthy even under the paper's own definitions.
- [Abstract and Section 3.3, 'Results'] The abstract concludes that the SciCom Wiki 'was found suitable to address the raised requirements,' but the reported evaluation does not fully support this: the hypothesis H_A,criteria was not met (only three of the seven top-priority criteria were rated 'good' or better), and Task 4 (finding a computer science video longer than 60 minutes from 2013-2014) did not reject the null hypothesis in Table 1. The paper acknowledges these limitations in the text, yet the abstract and conclusion state the suitability claim without qualification. The claim should be tempered to reflect that the system met the majority but not all of the evaluated requirements.
minor comments (6)
- [Section 1, Introduction] Typo: 'We present a our work' should read 'We present our work.'
- [Section 5, Discussion] Typo: 'to modularly adress all' should read 'to modularly address all.'
- [Section 2, Background] Typo: 'even their ressources and scalability are limited' should read 'even their resources and scalability are limited.'
- [Section 3.3, 'Results'] The phrase 'confirming showing positive objective efficiency' is grammatically awkward; consider 'confirming positive objective efficiency.'
- [Figure 5 caption] The caption contains an apparent rendering artifact: a long string of hexadecimal-like codes appears before the word 'against the benchmark.' This should be cleaned up.
- [Section 4.1, 'Veracity Checking'] The reference to Ciampaglia et al. [7] is appropriate, but the paper does not explain how the path distance is computed (e.g., path length, node degrees) despite the score being central to the fact-checking claim; a brief formal definition would aid reproducibility.
Circularity Check
No significant circularity: the fact-checking score is an explicitly qualified graph-distance heuristic against an external IPCC ground truth, and the platform evaluations use independent stakeholder and user groups.
full rationale
The paper's central chain is requirements elicitation, implementation of a Wikibase-based digital library, evaluation by surveys and interviews, and a neurosymbolic fact-checking pipeline whose ground truth is the IPCC AR6 synthesis, an external peer-reviewed source. None of these steps derives its conclusion from its own inputs. The veracity score is an operationalization, not a definition of truth: the authors state that the path check 'should only be interpreted as an indication, not a sufficient check' (Section 4.1, Veracity Checking), and that 'our accuracy scoring is currently limited to veracity (wver = 1)' (Section 4.1, Quantify additional metrics). This is an acknowledged validation gap rather than a circular reduction, because the knowledge graph being checked against is not produced by the tool itself but by IPCC headline statements. The only self-citations are the two co-author master theses [28, 31] used to point to implementation details; they do not carry the central argument, impose no forbidden alternatives, and are not invoked as uniqueness theorems. The requirements and usability evaluations involve independent participant groups (53 surveyed and 11 interviewed for requirements, 14 for prototype evaluation; 10 experts and 43 survey participants for the fact-checking tool), measuring perceived usefulness, task performance, and user experience rather than fitted predictions. No equation, parameter, or success measure is shown to be equivalent to its input by construction. Therefore the paper exhibits no significant circularity.
Assumptions & free parameters
free parameters (1)
- Accuracy score weights w_i in s_acc = sum(s_i * w_i) =
w_ver = 1, all other w_i = 0
assumptions (3)
- domain assumption IPCC AR6 headline statements constitute a correct and sufficient ground truth for climate claims.
- domain assumption LLM-based triple extraction produces triples that faithfully represent the meaning of media statements.
- domain assumption Graph proximity between aligned triples and ground-truth triples approximates factual veracity.
invented entities (1)
-
Scientific accuracy score s_acc
Cite this review
Pith. "Pith review of SciCom Wiki: Fact-Checking and FAIR Knowledge Distribution for Scientific Videos and Podcasts." pith.science (2026). https://pith.science/paper/XIL42GAR
@misc{pith2026250507912,
author = {Pith},
title = {Pith review of: SciCom Wiki: Fact-Checking and FAIR Knowledge Distribution for Scientific Videos and Podcasts},
year = {2026},
howpublished = {\url{https://pith.science/paper/XIL42GAR}},
note = {Machine review of arXiv:2505.07912}
}
read the original abstract
Democratic societies need accessible, reliable information. Videos and Podcasts have established themselves as the medium of choice for civic dissemination, but also as carriers of misinformation. The emerging Science Communication Knowledge Infrastructure (SciCom KI) curating non-textual media is still fragmented and not adequately equipped to scale against the content flood. Our work sets out to support the SciCom KI with a central, collaborative platform, the SciCom Wiki, to facilitate FAIR (findable, accessible, interoperable, reusable) media representation and the fact-checking of their content, particularly for videos and podcasts. Building an open-source service system centered around Wikibase, we survey requirements from 53 stakeholders, refine these in 11 interviews, and evaluate our prototype based on these requirements with another 14 participants. To address the most requested feature, fact-checking, we developed a neurosymbolic computational fact-checking approach, converting heterogenous media into knowledge graphs. This increases machine-readability and allows comparing statements against equally represented ground-truth. Our computational fact-checking tool was iteratively evaluated through 10 expert interviews, a public user survey with 43 participants verified the necessity and usability of our tool. Overall, our findings identified several needs to systematically support the SciCom KI. The SciCom Wiki, as a FAIR digital library complementing our neurosymbolic computational fact-checking framework, was found suitable to address the raised requirements. Further, we identified that the SciCom KI is severely underdeveloped regarding FAIR knowledge and related systems facilitating its collaborative creation and curation. Our system can provide a central knowledge node, yet a collaborative effort is required to scale against the imminent (mis-)information flood.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Domo Resource - Data Never Sleeps 11.0 (2023),https://www.domo.com/learn/ infographic/data-never-sleeps-11
work page 2023
-
[2]
Auer, S., Oelen, A., Haris, M., Stocker, M., D’Souza, J., et al.: Improving Access to Scientific Literature with Knowledge Graphs. Bibliothek Forschung und Praxis 44(3), 516–529 (Dec 2020).https://doi.org/10.1515/bfp-2020-2042, publisher: De Gruyter
-
[3]
In: Proceedings of the 7th Linguistic Annotation Workshop and Interop- erability with Discourse
Banarescu, L., Bonial, C., Cai, S., other: Abstract meaning representation for sem- banking. In: Proceedings of the 7th Linguistic Annotation Workshop and Interop- erability with Discourse. pp. 178–186. Association for Computational Linguistics (2013),https://aclanthology.org/W13-2322
work page 2013
-
[4]
Brennen, J.S., Simon, F.M., Howard, P.N., Nielsen, R.K.: Types, sources, and claims of COVID-19 misinformation (2020),https://ora.ox.ac.uk/objects/ uuid:178db677-fa8b-491d-beda-4bacdc9d7069, publisher: Reuters Institute for the Study of Journalism
work page 2020
-
[5]
Burns, T.W., O’Connor, D.J., Stocklmayer, S.M.: Science Communication: A Contemporary Definition. Public Understanding of Science12(2), 183–202 (Apr 2003).https://doi.org/10.1177/09636625030122004, publisher: SAGE Publica- tions Ltd
-
[6]
Calvin, K., Dasgupta, D., Krinner, G., et al.: IPCC, 2023: Full report [core writ- ing team, h. lee and j. romero (eds.)]. IPCC, geneva, switzerland. (2023).https: //doi.org/10.59327/IPCC/AR6-9789291691647,https://www.ipcc.ch/report/ ar6/syr/, edition: First
-
[7]
Ciampaglia, G.L., Shiralkar, P., Rocha, L.M., other: Computational fact check- ing from knowledge networks10(6), e0128193 (2015).https://doi.org/10.1371/ journal.pone.0128193, publisher: Public Library of Science
work page 2015
-
[8]
Environmental Research Letters8(2), 024024 (Jun 2013).https://doi.org/10
Cook, J., Nuccitelli, D., Green, S.A., Richardson, M., Winkler, B., et al.: Quan- tifying the consensus on anthropogenic global warming in the scientific literature. Environmental Research Letters8(2), 024024 (Jun 2013).https://doi.org/10. 1088/1748-9326/8/2/024024
work page 2013
Show all 35 references
-
[9]
In: Latest Advances and New Visions of Ontology in Information Science
Davarpanah,A., Babaie,H.A., Huang,G.:Climate systemontology: Aformalspec- ification of the complex climate system. In: Latest Advances and New Visions of Ontology in Information Science. IntechOpen (2023).https://doi.org/10.5772/ intechopen.110809
2023
-
[10]
Dessí, D., Osborne, F., Reforgiato Recupero, D., Buscaldi, D., Motta, E.: SCI- CERO: A deep learning and NLP approach for generating scientific knowledge graphs in the computer science domain258, 109945 (2022).https://doi.org/ 10.1016/j.knosys.2022.109945
2022
-
[11]
The MIT Press (2010)
Edwards, P.N.: A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming. The MIT Press (2010)
2010
-
[12]
org/resources/emissions-gap-report-2023, section: publications
Environment, U.N.: Emissions Gap Report 2023 (Aug 2023),http://www.unep. org/resources/emissions-gap-report-2023, section: publications
2023
-
[13]
Public Understanding of Science32(5), 605– 621 (Jul 2023).https://doi.org/10.1177/09636625221148054, publisher: SAGE Publications Ltd
Fähnrich, B., Weitkamp, E., Kupper, J.F.: Exploring ‘quality’ in science communi- cation online: Expert thoughts on how to assess and promote science communica- tion quality in digital media contexts. Public Understanding of Science32(5), 605– 621 (Jul 2023).https://doi.org/10...
2023 doi
-
[14]
Nature438(7070), 900–901 (Dec 2005).https://doi.org/10.1038/438900a, publisher: Nature Publishing Group SciCom Wiki 17
Giles, J.: Internet encyclopaedias go head to head. Nature438(7070), 900–901 (Dec 2005).https://doi.org/10.1038/438900a, publisher: Nature Publishing Group SciCom Wiki 17
2005 doi
-
[15]
Universitätsver- lag Göttingen (2007).https://doi.org/10.17875/gup2007-208, accepted: 2020- 04-15T02:34:09Z
Hagenhoff, S., Seidenfaden, L., Ortelbach, B., Schumann, M.: Neue Formen der Wissenschaftskommunikation: eine Fallstudienuntersuchung. Universitätsver- lag Göttingen (2007).https://doi.org/10.17875/gup2007-208, accepted: 2020- 04-15T02:34:09Z
2007 doi
-
[16]
In: Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers)
He, L., Lee, K., Lewis, M., Zettlemoyer, L.: Deep semantic role labeling: What works and what’s next. In: Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). pp. 473–483. Association for Computational Linguistics (...
2017
-
[17]
Islam, S., Proma, A., Zhou, Y., Akter, S.N., Wohn, C., Hoque, E.: KnowUREn- vironment: An automated knowledge graph for climate change and environmental issues (Nov 2022),https://www.climatechange.ai/papers/aaaifss2022/3
2022
-
[18]
Frontiers in Communication4(Oct 2019).https://doi.org/10.3389/ fcomm.2019.00055, publisher: Frontiers
Kappel, K., Holmen, S.J.: Why Science Communication, and Does It Work? A Taxonomy of Science Communication Aims and a Survey of the Empirical Ev- idence. Frontiers in Communication4(Oct 2019).https://doi.org/10.3389/ fcomm.2019.00055, publisher: Frontiers
2019
-
[19]
Computer Supported Cooperative Work (CSCW) 19(3), 377–415 (Aug 2010).https://doi.org/10.1007/s10606-010-9113-z
Karasti, H., Baker, K.S., Millerand, F.: Infrastructure Time: Long-term Matters in Collaborative Development. Computer Supported Cooperative Work (CSCW) 19(3), 377–415 (Aug 2010).https://doi.org/10.1007/s10606-010-9113-z
2010 doi
-
[20]
Science & Technology Studies29(1), 2–12 (Feb 2016).https://doi.org/ 10.23987/sts.55406, number: 1
Karasti, H., Millerand, F., Hine, C.M., Bowker, G.C.: Knowledge infrastructures: Part I. Science & Technology Studies29(1), 2–12 (Feb 2016).https://doi.org/ 10.23987/sts.55406, number: 1
2016 doi
-
[21]
Teorie vědy / Theory of Science35(1), 3–28 (May 2013).https://doi.org/ 10.46938/tv.2013.172
Kulczycki, E.: Transformation of Science Communication in the Age of Social Me- dia. Teorie vědy / Theory of Science35(1), 3–28 (May 2013).https://doi.org/ 10.46938/tv.2013.172
2013 doi
-
[22]
SIGCHI Bull.23(1), 78–81 (Jan 1991).https: //doi.org/10.1145/122672.122692
Lewis, J.R.: Psychometric evaluation of an after-scenario questionnaire for com- puter usability studies: the ASQ. SIGCHI Bull.23(1), 78–81 (Jan 1991).https: //doi.org/10.1145/122672.122692
1991
-
[23]
Royal Society Open Science (Jan 2019).https://doi.org/10.1098/ rsos.180932, publisher: The Royal Society Publishing
MacKenzie, L.E.: Science podcasts: analysis of global production and output from 2004 to 2018. Royal Society Open Science (Jan 2019).https://doi.org/10.1098/ rsos.180932, publisher: The Royal Society Publishing
2004
-
[24]
In: Proceedings of the 24th International Conference on World Wide Web
Marín Arraiza, P., Strobel, S.: The TIB|AV Portal as a Future Linked Media Ecosystem. In: Proceedings of the 24th International Conference on World Wide Web. pp. 733–734. WWW ’15 Companion, Association for Computing Machinery, New York, NY, USA (May 2015).https://doi.org/10.11...
2015
- [25]
-
[26]
International Journal of Interactive Multimedia and Artificial Intelligence4(4), 40–44 (2017).https://doi.org/10
Schrepp, M., Hinderks, A., Thomaschewski, J.: Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence4(4), 40–44 (2017).https://doi.org/10. 25968/opus-3397, publisher: Universidad Inte...
2017
-
[27]
International Journal of Humanities and Arts Computing 16(1),78–93(Mar2022).https://doi.org/10.3366/ijhac.2022.0278,publisher: Edinburgh University Press
Schöch, C., Hinzmann, M., Röttgermann, J., Dietz, K., Klee, A.: Smart Modelling for Literary History. International Journal of Humanities and Arts Computing 16(1),78–93(Mar2022).https://doi.org/10.3366/ijhac.2022.0278,publisher: Edinburgh University Press
2022
-
[28]
Stehr, N.: Eine digitale Wissensinfrastruktur zur Bereitstellung von Informatio- nen über wissenschaftliche Videos und Podcasts (Apr 2025).https://doi.org/ 10.15488/18996, publisher: Hannover : Gottfried Wilhelm Leibniz Universität 18 Wittenborg et al
2025 doi
-
[29]
Tenzer, F.: Daten - Volumen der weltweit generierten Daten bis 2027 (May 2023),https://de.statista.com/statistik/daten/studie/267974/umfrage/ prognose-zum-weltweit-generierten-datenvolumen/, data from IDC Global DataSphere
2027
-
[30]
In: Walker, M., Ji, H., Stent, A
Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a Large- scale Dataset for Fact Extraction and VERification. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lingu...
2018 doi
-
[31]
Master’s thesis, Hannover : Gottfried Wilhelm Leibniz Universität (2024).https://doi.org/10.15488/17173
Tremel, C.S.: Scientific Knowledge fit for society - Scoring scientific accuracy in climate change related news articles. Master’s thesis, Hannover : Gottfried Wilhelm Leibniz Universität (2024).https://doi.org/10.15488/17173
2024 doi
-
[32]
Varvantakis, C.: The Wikibase Software: Data and Collections in the Linked Open Data Web (Jan 2025).https://doi.org/10.5281/zenodo.14655779
2025 doi
-
[33]
Com- mun
Vrandečić, D., Krötzsch, M.: Wikidata: a free collaborative knowledgebase. Com- mun. ACM57(10), 78–85 (Sep 2014).https://doi.org/10.1145/2629489
2014 doi
-
[34]
Sci- entific Data3(1), 160018 (Mar 2016).https://doi.org/10.1038/sdata.2016.18, publisher: Nature Publishing Group
Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., Bouw- man, J., Brookes, A.J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C.T., Finkers, R., Gonzalez-Bel...
2016
-
[35]
Wittenborg,T.:SciComWiki[EN]-FAIRknowledgeinfrastructureforeducational content | SMWCon Fall 2024 (2024),https://doi.org/10.5446/69939
2024 doi
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.