{"id":"85c8144b-3170-4bb3-a7db-d9aa675bc856","arxiv_id":"2505.07912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors built a FAIR digital library and a neurosymbolic fact-checking prototype for scientific videos and podcasts, and validated usability but not fact-checking accuracy.","lead":"This paper presents a collaborative platform, SciCom Wiki, for cataloging scientific videos and podcasts as linked open data, and a neurosymbolic fact-checking tool that converts media into knowledge graphs to compare them against a climate-science ground truth. It reports that the platform was found usable in small usability tests and that a fact-checking prototype was perceived as helpful, but it does not measure fact-checking accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy score is an unvalidated graph-distance heuristic; without a gold-standard evaluation the fact-checking claim is unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the knowledge-graph distance metric is treated as a veracity measure without validation. My stress-test sharpens this: since the other accuracy metrics are unimplemented, the reported scientific accuracy score reduces entirely to the path-check heuristic, which the authors themselves call insufficient. The paper deserves credit for transparently flagging this limitation and for the solid requirements and usability evaluation of the SciCom Wiki platform; those parts of the contribution are supported by survey and interview data. However, the fact-checking component is the most requested feature and is presented as a central contribution, so the missing accuracy validation is not a peripheral caveat. Because the reader already issued a CONDITIONAL verdict on exactly this basis, no verdict adjustment is needed: the condition should be a quantitative evaluation of the fact-checking output against human judgments or an established benchmark.","tokens_in":15328,"tokens_out":3145,"duration_ms":32886,"concrete_test":"Build a gold-standard test set of 50–100 statements from climate-related videos, podcasts, and news articles, each independently labeled as supported, refuted, or not-checkable by two professional fact-checkers against IPCC AR6. Run the full pipeline (Whisper transcription, LLM triple extraction, ontology alignment, GraphDB exact and path checks) and compute precision/recall of extracted triples against manually extracted triples, plus the correlation or AUC between the system's s_acc scores and the human labels. If the path-based score does not outperform an exact-match-only or random baseline, the fact-checking claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's fact-checking contribution rests on the claim that media statements can be scored for scientific accuracy by (a) LLM-extracting triples and (b) checking them against an IPCC-derived knowledge graph via exact match or Ciampaglia path distance (§4.1, Veracity Checking). The authors explicitly state that the path check 'should only be interpreted as an indication, not a sufficient check,' and because no computational quantification was found for the additional metrics, the reported accuracy score is currently limited to veracity alone (w_ver = 1). Thus the entire accuracy score inherits the validity of an explicitly insufficient heuristic. No benchmark, FEVER-style evaluation, human-fact-checker agreement, or error analysis is reported for either the LLM triple extraction, which the expert interviews call unreliable, or the veracity score. The user survey measures perceived usefulness, not correctness. Consequently, the 'computational fact-checking' claim, the most requested feature and a central stated contribution, is not established; the system currently demonstrates a pipeline and UI mock-up rather than a validated fact-checking capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SciCom Wiki, a Wikibase-based FAIR digital library for scientific videos and podcasts, together with a neurosymbolic computational fact-checking pipeline that converts media text into knowledge graph triples and scores statements against a ground-truth knowledge graph built from IPCC AR6 headline statements. Requirements were elicited from 53 survey participants and refined in 11 interviews; the library was evaluated with 14 participants (including UEQ and ASQ instruments), and the fact-checking tool was evaluated through 10 expert interviews and a 43-participant user survey. The authors claim that the system as a whole is a suitable foundation for a FAIR Science Communication Knowledge Infrastructure and that it addresses the most requested feature, fact-checking.","tokens_in":15475,"tokens_out":3889,"duration_ms":36077,"significance":"If the fact-checking claim were validated, the paper would make a useful contribution to an important and under-served area: FAIR, collaborative infrastructure for non-textual science communication media. The strengths of the paper are its open-source artifacts, reproducible survey instruments, explicit requirement elicitation, and a transparently reported usability evaluation with appropriate statistical tests (Wilcoxon/t-tests, Bonferroni-Holm correction). The weakness is that the central fact-checking claim is not validated: no precision/recall, no comparison with human fact-checkers, no error analysis of the LLM triple extraction, and no verification of the ground-truth knowledge graph are reported. The accuracy score reduces by construction to a veracity heuristic that the authors themselves describe as only an indication, not a sufficient check. The paper therefore demonstrates a well-received prototype library and a fact-checking pipeline architecture, but not a validated fact-checking capability.","major_comments":[{"comment":"The scientific accuracy score s_acc is defined as a weighted sum, but the paper states that no computational quantification was found for the additional metrics and that w_ver=1; hence s_acc reduces to the veracity score. The veracity check itself uses exact match or a Ciampaglia-style path distance that the paper explicitly says 'should only be interpreted as an indication, not a sufficient check.' No evaluation of the correctness of this score is reported: there is no precision/recall, no comparison against human fact-checkers, no gold-standard set, and no error analysis of the LLM triple extraction. The user survey in Section 4.2 measures perceived usefulness and necessity, not whether the scores are accurate. Consequently, the paper's central claim that the system addresses the most requested feature, fact-checking, is not established by the presented evidence.","section":"Section 4.1, 'Veracity Checking' and 'Quantify additional metrics'"},{"comment":"The ground-truth knowledge graph is constructed from IPCC AR6 headline statements using the same LLM-based triple extraction pipeline that the expert interviews in Section 4.2 characterize as 'unreliable for semantic parsing' and prone to hallucinated or non-reproducible triples. The paper reports no verification of the ground-truth graph, such as human validation of the extracted triples or comparison with an independently constructed climate knowledge graph. If the ground truth contains extraction errors, the veracity scores computed against it inherit those errors, so the fact-checking output is not trustworthy even under the paper's own definitions.","section":"Section 4.1, 'Extension' and 'Alignment'"},{"comment":"The abstract concludes that the SciCom Wiki 'was found suitable to address the raised requirements,' but the reported evaluation does not fully support this: the hypothesis H_A,criteria was not met (only three of the seven top-priority criteria were rated 'good' or better), and Task 4 (finding a computer science video longer than 60 minutes from 2013-2014) did not reject the null hypothesis in Table 1. The paper acknowledges these limitations in the text, yet the abstract and conclusion state the suitability claim without qualification. The claim should be tempered to reflect that the system met the majority but not all of the evaluated requirements.","section":"Abstract and Section 3.3, 'Results'"}],"minor_comments":[{"comment":"Typo: 'We present a our work' should read 'We present our work.'","section":"Section 1, Introduction"},{"comment":"Typo: 'to modularly adress all' should read 'to modularly address all.'","section":"Section 5, Discussion"},{"comment":"Typo: 'even their ressources and scalability are limited' should read 'even their resources and scalability are limited.'","section":"Section 2, Background"},{"comment":"The phrase 'confirming showing positive objective efficiency' is grammatically awkward; consider 'confirming positive objective efficiency.'","section":"Section 3.3, 'Results'"},{"comment":"The caption contains an apparent rendering artifact: a long string of hexadecimal-like codes appears before the word 'against the benchmark.' This should be cleaned up.","section":"Figure 5 caption"},{"comment":"The reference to Ciampaglia et al. [7] is appropriate, but the paper does not explain how the path distance is computed (e.g., path length, node degrees) despite the score being central to the fact-checking claim; a brief formal definition would aid reproducibility.","section":"Section 4.1, 'Veracity Checking'"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a better fit for a systems/UX venue if the fact-checking validation is added or the claims are appropriately scoped. As it stands, the 'computational fact-checking' contribution is a pipeline prototype without correctness evaluation. The authors should be encouraged to either provide a small-scale benchmark against human fact-checkers or clearly reframe the contribution as an infrastructure for future fact-checking, not a validated fact-checker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The quick take: this is a solid requirements-driven systems paper with an honest usability evaluation, but the headline \"fact-checking\" contribution is a pipeline demo, not a validated capability. The stress-test worry is on target: the accuracy score reduces to an unvalidated graph-distance heuristic, and the authors themselves admit the path check is \"not a sufficient check.\"\n\nWhat's genuinely new is the SciCom Wiki itself: a Wikibase-based FAIR catalog for scientific videos and podcasts with faceted search, transcript storage, and import from the World Lecture Project. The requirements elicitation (53 survey respondents, 11 interviews) is well done, and the 14-participant usability evaluation uses standard instruments (UEQ, ASQ) with appropriate statistics. That part supports the claims made. The survey finding that fact-checking is the most requested feature is a useful data point for the community. Credit also goes to shipping code and survey data, which makes the usability results reproducible.\n\nThe soft spot is the computational fact-checking in Section 4. The pipeline (Whisper transcription, LLM triple extraction, alignment, exact match or Ciampaglia path distance on an IPCC-derived knowledge graph) is assembled from known components, and there is no evaluation of whether the final score correlates with veracity. No precision/recall, no FEVER-style benchmark, no human fact-checker agreement. The expert interviews and 43-participant survey gauge perceived need and usability, not correctness. The authors are transparent about this: they state the scoring is \"currently limited to veracity (w_ver = 1)\" and that the path check is \"only ... an indication.\" That transparency keeps the paper honest, but the abstract's implication that the system \"facilitate[s] ... the fact-checking of their content\" overstates what has been shown. The load-bearing claim should be softened to \"a prototype pipeline for accuracy scoring that awaits validation.\"\n\nMinor issues: some rough writing, a garbled figure, and heavy reliance on the two master's theses for implementation details. That's acceptable for a conference paper but means a reader cannot fully evaluate the system without chasing the theses.\n\nWho benefits: people building FAIR infrastructure for non-textual science media, and researchers working on requirements for science communication tools. The requirements data and the Wikibase architecture are citation-worthy. The fact-checking section, as it stands, is a starting point rather than a result. I'd send this to peer review with a clear request to either benchmark the accuracy score against a gold standard or explicitly reframe the contribution as infrastructure-plus-pipeline. A serious referee will find enough substance to work with.\n\nRecommendation: accept conditionally, with major revision requiring validation of the veracity score or an honest demotion of the fact-checking claim.","headline":"A genuinely useful FAIR catalog for science media backed by a well-run requirements study, but the central fact-checking claim is an unvalidated pipeline.","tokens_in":16059,"tokens_out":1907,"would_cite":false,"duration_ms":18613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Wikibase-based platform for science videos and podcasts can make non-textual media FAIR and computationally fact-checkable, and its prototype was judged suitable by stakeholders and users.","keywords":["science communication","knowledge infrastructure","FAIR data","computational fact-checking","knowledge graphs","LLM triple extraction","videos and podcasts","Wikibase"],"falsifier":"Take a set of climate statements with known expert verdicts, including hedged, sarcastic, and true-but-misleading formulations, run them through the pipeline, and compare the veracity scores with the expert ratings; if the scores do not separate true from false statements substantially better than chance, the graph-distance proxy is not a valid accuracy measure.","tokens_in":15095,"feed_emoji":"🎙️","tokens_out":8535,"duration_ms":75282,"temperature":0.7,"pith_summary":"The paper tries to establish that a central, collaborative knowledge platform for scientific videos and podcasts, built on linked-data wiki technology, can represent non-textual media according to FAIR principles and support computational fact-checking of their claims. This matters because video and podcast content increasingly carries civic information and misinformation, while existing knowledge infrastructures curate text far better than audio-visual media. Based on requirement surveys with 53 stakeholders, 11 interviews, a 14-participant prototype evaluation, 10 expert interviews, and a 43-person public survey, the authors conclude that the SciCom Wiki is suitable to address the raised requirements. The load-bearing demonstration is a climate-communication pipeline that turns media statements and a trusted climate-science synthesis report into a shared knowledge graph and scores statements by graph proximity. The paper stops short of claiming that the scoring is a complete check, calling it an indication rather than a sufficient verification.","feed_headline":"A wiki makes science videos and podcasts checkable","feed_subtitle":"A prototype scored video claims against climate-science ground truth, and evaluators called the platform usable.","key_machinery":"The central mechanism is a three-stage pipeline: a linked-data digital library whose data model covers more than 200 media qualities, with a full-text wiki storing transcripts and a search page as the main interface; an LLM-based statement extractor that turns transcripts, PDFs, or web text into aligned triples; and a veracity checker that searches the ground-truth knowledge graph for exact matches and otherwise applies the shortest-path graph-distance method from the computational fact-checking literature to estimate semantic proximity. This graph-distance score is the load-bearing proxy for factuality, and the accuracy score $s_{\\mathrm{acc}} = \\sum_i s_i w_i$ is currently reduced to veracity because the additional metrics, such as context, clearness, authenticity, subjectivity, and confidence, lack computational quantifications.","core_discovery":"The central claim is that a digital library built on linked-data wiki technology, extended with a full-text wiki and search and import services, provides a feasible foundation for the Science Communication Knowledge Infrastructure: it makes videos and podcasts findable, accessible, interoperable, and reusable, and it enables a neurosymbolic computational fact-checking workflow. In that workflow, heterogeneous media are first converted into text, then into subject-predicate-object triples by a large language model, aligned by normalizing synonyms and base forms, and compared against a trusted ground-truth knowledge graph assembled from the headline statements of the authoritative climate-science synthesis report. Statements that are not exact matches receive a veracity indication by a shortest-path graph-distance check, and an overall accuracy score is computed as $s_{\\mathrm{acc}} = \\sum_i s_i w_i$; in the current implementation only veracity contributes ($w_{\\mathrm{ver}}=1$). The paper reports that expert interviews and a public survey found the tool necessary and usable, and concludes that the SciCom Wiki is suitable to address the requirements raised.","pith_inferences":["Editorial inference: the validity of the veracity score has not yet been demonstrated against an external benchmark; comparing it with human expert verdicts on a sample of climate claims would be the natural next test.","Editorial inference: the architecture could be reused outside climate communication, for instance in health or election discourse, wherever a consensus synthesis exists to build the ground-truth graph.","Editorial inference: the authors' own discussion implies that the accuracy score should be treated as a triage signal that prioritizes claims for professional fact-checkers, not as a final verdict, since context, sarcasm, and subjectivity are explicitly out of scope.","Editorial inference: a successful deployment probably requires community and institutional scaffolding like the one that sustains Wikipedia rather than the software alone, which the paper also acknowledges."],"forward_implications":["A FAIR wiki node can serve as a central access point for scientific videos and podcasts, aggregating metadata and transcripts that are today scattered across isolated repositories.","Computational fact-checking of audio-visual media becomes feasible at scale: LLM extraction plus knowledge-graph alignment produces a per-statement veracity indication without requiring every video to be watched and checked by hand.","The same pipeline transfers to other domains once a trusted, machine-readable ground-truth knowledge graph exists, since the extraction and alignment steps are presented as domain-independent.","Because the path check is explicitly not a sufficient verification, the practical system still depends on human-in-the-loop supervision to catch hallucinated or misaligned triples.","User demand is not hypothetical: the surveys and interviews show that stakeholders want accuracy filtering and fact-check support more than reach metrics such as views or subscriber counts."],"supporting_citations":[{"why":"Supplies the shortest-path semantic proximity method used to approximate statement veracity.","marker":"[7]"},{"why":"The authoritative climate-science synthesis report whose headline statements form the ground-truth knowledge graph.","marker":"[6]"},{"why":"The FAIR data principles that define the interoperability and reusability standards the platform is designed to meet.","marker":"[34]"},{"why":"The open collaborative knowledge base whose linked-data model the platform reuses.","marker":"[33]"},{"why":"A domain-specific linked-data knowledge base that motivated the architecture of building dedicated wiki instances.","marker":"[27]"},{"why":"An automated climate-change knowledge graph whose construction and trusted-triple approach inform the ground-truth pipeline.","marker":"[17]"},{"why":"The underlying thesis that details the digital library's implementation and evaluation.","marker":"[28]"},{"why":"The underlying thesis that details the fact-checking tool's implementation and evaluation.","marker":"[31]"},{"why":"The user-experience benchmark used to interpret the usability evaluation results.","marker":"[26]"}],"fun_headline_variants":["FAIR wiki fact-checks science videos and podcasts","Neurosymbolic fact-checking for science media","SciCom Wiki: linked-data fact-checking for science videos","Wiki-driven fact-checking for science podcasts and videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if the graph-distance score between LLM-extracted triples and trusted-report triples does not actually track whether a statement is true; the authors themselves call the path check an indication, not a sufficient check.","fun_headline_variants_meta":{"raw":{"variants":["FAIR wiki fact-checks science videos and podcasts","Neurosymbolic fact-checking for science media","SciCom Wiki: linked-data fact-checking for science videos","Wiki-driven fact-checking for science podcasts and videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1550,"prompt_tokens":1070,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":686,"tokens_out":480,"duration_ms":4262,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:13:28.543757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of climate statements with known expert verdicts, including hedged, sarcastic, and true-but-misleading formulations, run them through the pipeline, and compare the veracity scores with the expert ratings; if the scores do not separate true from false statements substantially better than chance, the graph-distance proxy is not a valid accuracy measure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the shortest-path semantic proximity method used to approximate statement veracity."},{"cited_title":"International Journal of Humanities and Arts Computing 16(1),78–93(Mar2022).https://doi.org/10.3366/ijhac.2022.0278,publisher: Edinburgh University Press","cited_arxiv_id":null,"evidence_quote":"A domain-specific linked-data knowledge base that motivated the architecture of building dedicated wiki instances."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An automated climate-change knowledge graph whose construction and trusted-triple approach inform the ground-truth pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The underlying thesis that details the digital library's implementation and evaluation."},{"cited_title":"Master’s thesis, Hannover : Gottfried Wilhelm Leibniz Universität (2024).https://doi.org/10.15488/17173","cited_arxiv_id":null,"evidence_quote":"The underlying thesis that details the fact-checking tool's implementation and evaluation."},{"cited_title":"International Journal of Interactive Multimedia and Artificial Intelligence4(4), 40–44 (2017).https://doi.org/10","cited_arxiv_id":null,"evidence_quote":"The user-experience benchmark used to interpret the usability evaluation results."}],"review_version":1}