{"id":"806792db-3d4b-4ff8-8deb-12e4e5a151da","arxiv_id":"2506.15794","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Veracity is an open-source LLM-plus-web-search fact-checking app with a 0 to 100 reliability score and explanations, but no evaluation of its accuracy is included.","lead":"This demo paper describes Veracity, an open-source app that uses an LLM and web search to score the truth of user-submitted claims. It is aimed at the public, journalists, and researchers, but reports no accuracy measurements or user studies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 60% share-recommendation threshold is an unvalidated, user-facing binary verdict; without calibration data the central fact-checking value is not established.","rationale":"The reader identified the same weakest assumption: the reliability score and 60% threshold lack calibration and evaluation. I agree that this is the load-bearing concern because the paper's stated value proposition is accurate, grounded fact-checking and a share recommendation directly tied to the score. The paper gives no accuracy, calibration, or baseline measurements, and the threshold is presented as a fixed decision rule without justification. The architecture description, screenshots, and citations to Tian et al. provide some support that the system exists and follows a reasonable retrieval-augmented LLM design, but they do not validate the score's truthfulness. I also note the open-source repositories are only referenced as 'links' with no commit hashes, which weakens reproducibility claims. My recommendation matches the reader's: CONDITIONAL, requiring either narrowing claims to a system description or adding an evaluation of score accuracy and calibration. I do not see internal inconsistency in the code or a more fundamental flaw; the central risk is empirical validation, not logic. Hence agreement with the reader and no verdict change.","tokens_in":5118,"tokens_out":1371,"duration_ms":14073,"concrete_test":"Collect a benchmark set of at least 200 claims spanning true, false, and mixed claims (e.g., from Snopes or the FEVER dataset), run the deployed system end-to-end, and compute calibration of the reliability score against ground truth plus accuracy/precision/recall at the 60% threshold. Report the calibration curve and the confusion matrix; if the threshold yields a false-positive rate above a pre-specified acceptable level (e.g., >10%) or the score is miscalibrated by more than 0.1 Brier score, then the share recommendation should be revised or explicitly labeled as unvalidated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that Veracity provides 'grounded veracity assessments' that are reliable enough for public fact-checking. The weakest load-bearing assumption is that the LLM's reliability score—and especially the 60% threshold that converts it into a user-facing share recommendation—is accurate enough to be useful. Section 3 introduces 'Score-based analysis' and Section 2.1 (item 5) specifies the threshold, but the paper presents no evaluation, benchmark, calibration study, or comparison against existing misinformation-detection systems. The threshold is presented as though it were a decision boundary separating true from false, but its choice is unexplained and untested. Because the share recommendation is the most consequential user-facing output (it directly tells users whether to spread a claim), a wrong score distribution or a poorly chosen threshold could cause the system to actively amplify misinformation while appearing trustworthy. The software may run correctly, but the paper's claim of being a production-ready factuality assessment application depends on this unvalidated mapping; the claim of 'grounded' assessments likewise depends on whether retrieved sources actually support the score, which is also unevaluated. This is not a disagreement with consensus but a missing internal validation of a decision rule the paper itself treats as core.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Veracity, an open-source AI fact-checking system that combines large language models (LLMs) with web retrieval agents to analyze user-submitted claims and provide grounded veracity assessments. The system displays retrieved sources, a numerical reliability score, an explanation of the model's reasoning, and a share recommendation based on a 60% threshold. The paper describes the architecture (frontend/backend, Model-View-Controller), the AI techniques (LLM plus web search, following Tian et al. 2024), and interactive elements including user feedback and an expert dashboard. The central claim is that Veracity is a production-ready, transparent, and accessible fact-checking tool, with open-source repositories available.","tokens_in":5442,"tokens_out":2920,"duration_ms":29013,"significance":"If the system works as described, Veracity could be a valuable open-source contribution to the fact-checking ecosystem, offering a test-bed for research and a practical tool for the public. The paper's strengths are its clear architecture description, its grounding in prior work on retrieval-augmented misinformation detection, and its emphasis on transparency and user-facing explainability. However, the paper provides no quantitative evaluation, calibration, or comparison against existing systems, so the central claims of effectiveness, groundedness, and production readiness are currently unverified. The open-source availability and detailed system description are useful, but the lack of empirical evidence is a significant gap.","major_comments":[{"comment":"The reliability score and the 60% share-recommendation threshold are presented as core innovations, but the paper provides no evidence that the LLM-generated score is calibrated or that the threshold produces correct recommendations. This is load-bearing because the share recommendation is the most consequential user-facing output. Without a calibration study on a standard fact-checking benchmark, the claim of 'grounded veracity assessments' and the production-readiness claim in Section 5 are unsupported.","section":"Section 3, 'Score-based analysis'"},{"comment":"The choice of 60% as the threshold for a positive share recommendation is unexplained and untested. If the score distribution is miscalibrated, the system may recommend sharing false claims. The authors should either justify the threshold empirically or clearly present it as a configurable placeholder with a planned calibration procedure.","section":"Section 2.1, item 5"},{"comment":"The paper describes Veracity as 'production ready,' but no evaluation of accuracy, latency, robustness, or user experience is provided. For a demo paper, a description of the system is valuable, but the production-readiness claim requires at least some quantitative evidence or a clear statement that this is a prototype.","section":"Section 5, Conclusion"},{"comment":"The paper states that the system is an implementation of Tian et al. (2024), but it does not compare Veracity's performance against that baseline or any other misinformation-detection system. Without such a comparison, it is unclear what the system adds beyond the existing method, especially regarding the claimed innovation of a reliability score.","section":"Section 2.3"}],"minor_comments":[{"comment":"The frontend and backend wikis are cited as '[link]' with no actual URLs; please provide the links or remove the references.","section":"Section 2.2"},{"comment":"The claim 'This is the first tool of its kind to present a reliability score' is strong and unsubstantiated; consider softening it or providing a survey of existing tools to support it.","section":"Section 3"},{"comment":"Several references are incomplete or inconsistently formatted, e.g., the Zhou et al. (2024) entry ends with 'arxiv [preprint](2024)' without a full citation.","section":"References"},{"comment":"The numerical annotations in Figure 1 are referenced in the text, but the figure itself is not included in the text review; ensure the figure is legible and the annotations match the described items.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a demo/system description without evaluation. The overlap with Tian et al. (2024) is significant and the authors should clearly delineate the novel contributions and provide independent evidence of the system's value. If the authors add a modest evaluation and temper the claims, the paper could be acceptable as a demo."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Veracity. The paper is an honest demo paper. The core architecture is directly inherited from Tian et al. 2024 (shared authors), and the paper says so up front. What's actually new is the packaging: the source display panel, the LLM-justified numerical score, and the chat-style interface. That is real, concrete, and useful as a community resource. The system may well work as described; the architecture and screenshots are internally consistent, and the authors should get credit for open-sourcing it and framing it as a test-bed.\n\nThe soft spot is exactly what the stress-test highlights. The 60% threshold converts the LLM reliability score into a user-facing share recommendation, and there is no calibration, accuracy measurement, or comparison behind it. The paper repeatedly calls assessments \"grounded\" and the application \"production ready,\" but offers no evaluation at all. For a demo paper that is tolerable, but those claims need to be softened or backed by at least a sanity-check experiment. Also, the \"first tool of its kind\" novelty claim is unsupported, and the GitHub links appear only as footnote placeholders, not as actual URLs with commit hashes, which hurts reproducibility.\n\nI don't think this is a load-bearing flaw in the sense that the paper's main deliverable is a working system, not a measured result. The stress-test concern is fair, though: if the share recommendation is meant to be taken seriously, an arbitrary 60% cutoff is not obviously safer than any other threshold. That said, the paper itself does not present the threshold as a validated scientific result; it describes it as a design choice. The real problem is the gap between this modest system description and the confidence in the abstract and conclusion.\n\nWho is this for? People building open-source fact-checking tools, and demo-track reviewers who care about artifact claims. It deserves a serious referee because the artifact is real and the gap between claims and evidence is precisely the kind of thing reviewers should probe. I wouldn't cite it for accuracy results, but I might cite it as an example of an open-source deployment.\n\nRecommendation: send it to peer review, ideally a demo track, with the request that the authors either add a small calibration evaluation or explicitly label the 60% threshold as an arbitrary product choice, not a validated decision rule.","headline":"A clear demo paper describing a real open-source fact-checking tool, but with unsupported reliability claims and an uncalibrated 60% share threshold.","tokens_in":5883,"tokens_out":1992,"would_cite":false,"duration_ms":23622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This demo paper describes Veracity, an open-source fact-checking system in which an LLM teams with web retrieval agents to return a 0-100 reliability score, a plain-language explanation, and the sources behind the verdict.","keywords":["fact-checking","misinformation","large language models","web retrieval agents","open-source software","reliability scoring","transparent AI","media literacy"],"falsifier":"Run Veracity on a set of claims whose truth has already been established by professional fact-checkers and compare the system's score to the known labels; if the share recommendation (score above 60 percent) does not distinguish true from false claims better than chance, the central claim that the tool provides reliable fact-checking would be refuted.","tokens_in":4932,"feed_emoji":"✅","tokens_out":6443,"duration_ms":68032,"temperature":0.7,"pith_summary":"The paper presents Veracity, an open-source fact-checking application that combines a large language model with web retrieval agents to assess user-submitted claims. The authors' central claim is that this design produces transparent, grounded veracity assessments: users see a 0-100 reliability score, an explanation of the reasoning, and the sources used. The system is aimed at the general public and at expert users such as journalists, who get an aggregate dashboard. If the system works as described, it would give individuals an accessible alternative to platform moderation, and give researchers a test bed for fact-checking strategies.","feed_headline":"Open-source app pairs LLMs with web search to fact-check claims","feed_subtitle":"Anyone can paste a claim and see a 0-100 reliability score with named sources, all runnable locally.","key_machinery":"The central mechanism is the LLM/web-search-agent loop. Given a claim, the LLM decides whether retrieval is needed and issues queries through a web agent; retrieved sources are passed to the LLM as evidence, and the user sees the selected sources with credibility ratings. The score-based analysis is the second piece: the LLM must assign a 0-100 reliability score and justify it in text, and the interface maps the score to an actionable message, including a share recommendation when the score exceeds 60 percent.","core_discovery":"The central claim is that an open-source, production-ready factuality assessment application can be built by teaming an LLM with a web retrieval agent: the LLM decides whether to search, reads documents returned by a web search, and produces a reliability score plus a natural-language explanation, while the user is shown the sources and their average credibility ranking. The authors present this as the first tool of its kind to offer a numerical reliability score with an LLM justification, along with multilingual support, a messaging-app-style interface, and an expert dashboard. As a demo paper, its claim is about the existence and design of the system rather than a measured accuracy result; the force of the claim is that such a tool can be open, transparent, and immediately usable.","pith_inferences":["The paper leaves open whether the LLM's 0-100 score is calibrated; a natural extension is to run the open-source system on a labeled benchmark to test whether the 60 percent share threshold separates true from false claims.","Because the interface turns the score into a share recommendation, an uncalibrated score could amplify rather than reduce misinformation at the boundary; a calibration study is the logical next step.","The same architecture could plausibly be extended to image, video, or audio claims by swapping the retrieval agent and using a multimodal LLM, though the paper does not discuss this.","The built-in user feedback mechanism (star ratings plus tags) could be mined as a human calibration signal, turning the app into a data-collection instrument for fact-checking research."],"forward_implications":["A user can paste a claim and receive a scored, explained verdict without needing to judge the sources on their own.","The open-source code can be downloaded and run locally, so fact-checking does not depend on a single proprietary service.","Experts such as journalists can see aggregate claim trends through the dashboard, which could help them spot emerging misinformation narratives.","Researchers can treat Veracity as a test bed for new retrieval, scoring, and explanation strategies.","Because multilingual support is built in, non-English claims can be checked through the same interface."],"supporting_citations":[{"why":"Supplies the core method: teaming an LLM with web retrieval agents for evidence-based misinformation detection, which Veracity implements.","marker":"[Tian et al., 2024]"},{"why":"Provides the news-domain quality ratings used to display each source's documented credibility.","marker":"[Lin et al., 2023]"},{"why":"Supports the design choice that retrieving online evidence improves fact verification.","marker":"[Bekoulis et al., 2021]"},{"why":"Grounds the premise that LLMs can detect misinformation, a precondition for the score-based analysis.","marker":"[Pelrine et al., 2023a]"},{"why":"Documents opportunities and challenges of using LLMs to combat misinformation, motivating the retrieval-plus-LLM combination.","marker":"[Chen and Shu, 2024]"},{"why":"Establishes the gap the paper addresses: high-accuracy fact-checking systems are not yet helpful and trustworthy, motivating transparent explanations and source display.","marker":"[Augenstein et al., 2024]"}],"fun_headline_variants":["Open-source fact-checker uses LLMs and web search for verdicts","Veracity: free tool scores claims 0-100 with cited sources","Chat-style fact-check app: LLM + web retrieval gives reliability score","Open-source AI fact-checking: paste a claim, get a grounded score","Veracity: transparent AI fact-checking with source-backed scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's value rests on the assumption that the LLM's reliability score—and the 60 percent cutoff for recommending a share—correctly tracks whether a claim is actually true, since no calibration, benchmark, or evaluation is reported to support it.","fun_headline_variants_meta":{"raw":{"variants":["Open-source fact-checker uses LLMs and web search for verdicts","Veracity: free tool scores claims 0-100 with cited sources","Chat-style fact-check app: LLM + web retrieval gives reliability score","Open-source AI fact-checking: paste a claim, get a grounded score","Veracity: transparent AI fact-checking with source-backed scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1302,"prompt_tokens":797,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":410}},"tokens_in":413,"tokens_out":505,"duration_ms":5223,"temperature":1.0,"reasoning_tokens":410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:50:45.577978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Veracity on a set of claims whose truth has already been established by professional fact-checkers and compare the system's score to the known labels; if the share recommendation (score above 60 percent) does not distinguish true from false claims better than chance, the central claim that the tool provides reliable fact-checking would be refuted.","supporting_citations":[{"cited_title":"High level of correspon- dence across different news domain quality rating sets","cited_arxiv_id":null,"evidence_quote":"Provides the news-domain quality ratings used to display each source's documented credibility."},{"cited_title":"A review on fact extraction and verification.ACM Computing Surveys (CSUR), 55(1):1–35,","cited_arxiv_id":null,"evidence_quote":"Supports the design choice that retrieving online evidence improves fact verification."},{"cited_title":"Combat- ing misinformation in the age of llms: Opportunities and challenges.AI Magazine, 45(3):354–368,","cited_arxiv_id":null,"evidence_quote":"Documents opportunities and challenges of using LLMs to combat misinformation, motivating the retrieval-plus-LLM combination."},{"cited_title":"Factuality challenges in the era of large language models and opportunities for fact-checking.Nature Machine Intelligence, 6(8):852–863, Aug","cited_arxiv_id":null,"evidence_quote":"Establishes the gap the paper addresses: high-accuracy fact-checking systems are not yet helpful and trustworthy, motivating transparent explanations and source display."}],"review_version":1}