{"id":"9318b635-0421-4283-935f-649041ebef92","arxiv_id":"2507.05965","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using Olmo to extract atomic facts and Gemma to verify them against Wikipedia, OpenFActScore reproduces the original FActScore ranking of 10 LLMs with a Pearson correlation above 0.99.","lead":"OpenFActScore reimplements the FActScore factuality metric so it can run on open Hugging Face models instead of closed-source models like InstructGPT and ChatGPT. The authors report that open models rank 10 language models in nearly the same order as the original metric, with a Pearson correlation above 0.99.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"0.99 correlation rests on 10 widely-spread models with no uncertainty estimate; the test may be too easy to establish ranking preservation.","rationale":"The reader's weakest assumption concerned the representativeness of the benchmark data and human labels; my concern is related but distinct: the headline correlation's statistical robustness and the discriminative difficulty of the 10-model testbed. The paper's central claim is that OpenFActScore preserves the order of models produced by the original FActScore. With only 10 points spanning a wide range, and no uncertainty quantification, the evidence for ranking preservation is weaker than the abstract suggests. A lack of CI or per-instance data prevents verification. However, this does not invalidate the qualitative conclusion; it strengthens the need for the CONDITIONAL verdict already given. I agree with the reader that the paper is promising but needs revision, so no verdict change is required.","tokens_in":9610,"tokens_out":9563,"duration_ms":100855,"concrete_test":"Rerun the full OpenFActScore pipeline on a larger, more challenging set of model outputs (at least 20 models, including several with similar factuality, e.g., different seeds or adjacent checkpoints) and report Pearson r, Spearman rho, and Kendall tau with bootstrap 95% CIs. If Kendall tau against the original FS(A) drops below 0.9 on the near-tie subset, the 'ranking preserved' claim is not robust. As a minimal alternative, bootstrap the existing 10 pairs (10,000 resamples) and report the CI for r; if the lower bound is <0.95, the abstract's 'over 0.99' is misleading.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the Pearson r>0.99 between OpenFActScore and the original FActScore (Section 4.3, Table 3). This number is computed from exactly 10 model-level scores. No confidence interval, p-value, or per-sample breakdown is reported, and the 10 models span an extremely wide quality range (FS(A) from 17.3 to 73.1). With n=10, even a strong r has a wide CI; more importantly, a testbed with such a large spread makes ranking preservation nearly trivial. The paper's conclusion also says '11 different language models' while Table 3 lists 10, indicating reporting imprecision. The claim that 'the ranking of the models stays the same' is therefore not well-supported for models that are close in factuality (e.g., Alpaca 13B vs Vicuna 7B). Without release of per-entity atomic facts and validation labels, the 0.99 cannot be independently verified. This is the load-bearing weak point because the entire contribution—open models substituting closed ones—hinges on the claim that the open pipeline reproduces the original ordering.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OpenFActScore, an open-source reimplementation of the FActScore factuality evaluation framework. It replaces the original closed-source components (InstructGPT for atomic fact generation, ChatGPT/InstLlama for validation) with Hugging Face-compatible models, and describes the implementation, prompts, and design choices. The authors evaluate four open models: Llama-3.1, Gemma, Qwen, and Olmo, using BERTScore-F1 against human-corrected atomic facts for AFG and error rates against human FActScores for AFV. Based on these results, they select Olmo for AFG and Gemma for AFV, apply this pipeline to 10 model outputs from the original FActScore repository, and report that OpenFActScore scores correlate with the original FActScore settings at Pearson r > 0.99 with the same model ranking.","tokens_in":9815,"tokens_out":7759,"duration_ms":86258,"significance":"If the statistical claims are properly supported, this is a useful and reproducible engineering contribution: it shows that a fully open pipeline can approximate the ordinal behavior of a closed-source factuality metric on a standard benchmark. The released codebase, the use of the original human annotations, and the explicit choice of a fully open model (Olmo) are genuine strengths. The scientific novelty is modest, however, since the paper is primarily a reimplementation and evaluation study rather than a new metric. The main limitation is that the headline 0.99 correlation rests on only 10 aggregate model-level points with no uncertainty quantification, so the strength of the conclusion is currently overstated.","major_comments":[{"comment":"The claim that 'the Pearson correlation between FActScore and OpenFActScore in both setting (A) and (B) is over 0.99' is computed from exactly 10 model-level scores with no confidence interval, p-value, or per-entity breakdown. The table does show that the 10 aggregate scores are identically ordered across the three columns, so the ranking claim is supported for this specific sample. However, a high Pearson r is expected when the 10 points span a very wide range (FS(A) from 17.3 to 73.1), and this does not establish that the ranking is reliable for models that are close in factuality (e.g., Alpaca 13B vs Vicuna 7B differ by only about 1 point in FS(OFS)). Please report Fisher-z or bootstrap confidence intervals, Spearman/Kendall rank correlations, and per-entity agreement, and release the per-entity scores so the 0.99 can be independently verified.","section":"Section 4.3, Table 3"},{"comment":"The 'Cumulative ER' column is the sum of the absolute error rates across the three subject models (e.g., Llama-3.1: 2.8 + 1.6 + 9.0 = 13.4), but it is labeled and discussed as a cumulative error rate. Because signed errors cancel, this metric can conceal large but opposing errors and is not an appropriate loss for model selection. Please report signed error rates and mean absolute error separately, and confirm that the choice of Gemma for AFV and the overall conclusion remain justified under the corrected metric.","section":"Section 4.2, Table 2"},{"comment":"The paper does not provide the OpenFActScore atomic facts, validation labels, or per-entity scores used to compute the FS(OFS) column in Table 3, so the central quantitative claim cannot be recomputed from the manuscript alone. Given that the stated contribution is reproducibility, please release these artifacts alongside the code, including exact model versions, decoding parameters, and the retrieval setting (e.g., Wikipedia dump and GTR model version) used for the final evaluation.","section":"Section 4.3 and Data Availability"}],"minor_comments":[{"comment":"The conclusion states that 'we evaluated productions from 11 different language models,' but Table 3 lists 10 models; please clarify whether the count includes the three subject models in Tables 1-2 or whether a row is missing from Table 3.","section":"Conclusion and Table 3"},{"comment":"The abstract contains a typo ('setupt' for 'setup') and should be corrected.","section":"Abstract"},{"comment":"There are several spelling errors, including 'oringial', 'implementatoin', 'differense', 'Althought', and 'architechture'; please proofread the manuscript.","section":"Throughout"},{"comment":"There is an inconsistent capitalization 'OpenFactScore' in the surrounding text, which should be 'OpenFActScore'.","section":"Section 4.3"},{"comment":"The AFG evaluation uses BERTScore-F1 with a best-match assignment per sentence, but the paper does not report standard deviations across entities or across the three subject models; adding such variation would help interpret the 0.35% difference that motivates choosing Olmo over Gemma.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The main risks are statistical and reproducibility-related: the 0.99 correlation is computed from 10 points without uncertainty intervals, and the underlying per-entity scores are not released. There is also a mild in-sample selection issue: the model choices (Olmo and Gemma) are made on the same human-annotated benchmark family that is reused for the final 10-model comparison, although the subject models in Table 3 are different from those in Tables 1-2. If the authors provide confidence intervals, per-entity results, and the missing artifacts, the paper could be suitable for publication as an open-source reproducibility study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does what it claims—gives you a working open-source FActScore with any HuggingFace model—and the empirical story is mostly honest, but the 0.99 Pearson correlation is a lot weaker than it looks. I'd still send it out, with a request for better stats.\n\nWhat's genuinely new: the code, the modular breakdown, and the per-module comparison of Gemma/Qwen/Llama/Olmo on AFG and AFV against human annotations. That's useful for anyone who wants to avoid closed APIs. The finding that Olmo generates good atoms but validates poorly, while Gemma validates well, is a nice result. The decision to keep Olmo for generation because it is fully open is principled.\n\nThe soft spot is the headline correlation. Ten models, FActScore range 17–73, no error bars, no permutation test. With that spread, a high r is almost guaranteed, and it says little about whether ordering is preserved for models that are close (Alpaca 13B vs Vicuna 7B). The conclusion says '11 different language models' while Table 3 lists 10. Also, the model choice for the pipeline (Olmo+Gemma) is made on the same human-annotated data used in the final 10-model comparison, so the 0.99 is partly in-sample selection. The cumulative error rate in Table 2 is computed as a sum of absolute deviations, though it's described as a summation of error rates; that's a minor but fixable inconsistency.\n\nNone of this is fatal. The paper is transparent about design choices, ships code, and provides a concrete path for others to verify. It's a resource paper, not a conceptual advance. The right audience is NLP evaluation people who want a cheap, reproducible factuality metric. I'd send it to peer review, with the clear expectation that the authors tighten the statistics, report per-entity scores, and fix the counting/definition slips. If they do that, I'd cite it.","headline":"Useful open-source FActScore with a plausible but over-sold 0.99 correlation; worth reviewing with statistical fixes.","tokens_in":10311,"tokens_out":3289,"would_cite":true,"duration_ms":34650,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenFActScore is an open-source reimplementation of FActScore that claims fully open language models can reproduce the original metric's model ranking, with a Pearson correlation over 0.99 across 10 language models.","keywords":["factuality evaluation","atomic fact generation","atomic fact validation","open-source language models","FActScore","hallucination detection","long-form text generation","model-based evaluation"],"falsifier":"Have human annotators score the same 10 model outputs used in Table 3 and compare their ranking with OpenFActScore's ranking; if the human-model rank correlation is much lower than 0.99, then the high correlation shows agreement with the closed-source estimator rather than with human factuality. Alternatively, compute OpenFActScore on a fresh set of model outputs from a different domain and check whether the model ranking still matches human judgment.","tokens_in":9419,"feed_emoji":"📊","tokens_out":6450,"duration_ms":62973,"temperature":0.7,"pith_summary":"OpenFActScore is an open-source reimplementation of the FActScore pipeline for measuring whether long-form text generated by language models is factual. The paper claims that fully open models can stand in for the proprietary InstructGPT and ChatGPT components of the original, and that the resulting estimates are not only close but rank models identically: across 10 language models, OpenFActScore's scores correlate with the original FActScore at $r > 0.99$. The authors select Olmo for atomic fact generation and Gemma for atomic fact validation, documenting design changes such as chat templates and system prompts. A sympathetic reader would care because factuality evaluation is otherwise tied to closed, paid APIs, which limits reproducibility and cost-effective auditing.","feed_headline":"Open models reproduce FActScore rankings at 0.99","feed_subtitle":"A fully open model pair scores 10 language models in the same order as the closed-source original.","key_machinery":"The load-bearing mechanism is the two-stage FActScore pipeline itself, made model-agnostic. Atomic Fact Generation (AFG) splits each sentence into atomic facts, short statements each carrying one piece of information, using a prompt with a BM25-selected demo; Atomic Fact Validation (AFV) retrieves the top five Wikipedia passages with a GTR retriever, prepends them to a 'True or False?' prompt, and parses the validator's first token. OpenFActScore's engineering contribution is an HFModel class that loads any openly available causal language model with its chat template and system prompt, letting the user swap AFG and AFV models freely. The chosen pair, Olmo for AFG and Gemma for AFV, is what produces the $r > 0.99$ correlation with the original scores.","core_discovery":"The central discovery is that a fully open model pair can approximate the original closed-source FActScore estimator. Using Olmo to decompose model outputs into atomic facts and Gemma to verify each fact against retrieved Wikipedia passages yields FActScore estimates that correlate with the original scores from both FActScore settings at Pearson $r > 0.99$ across 10 language models, preserving the model ranking even though the absolute OpenFActScore values are lower. This is presented as an extension of FActScore rather than a new metric, with the open pipeline intended to make factuality evaluation reproducible and independent of commercial APIs.","pith_inferences":["If the correlation generalizes beyond the biography benchmark, OpenFActScore could serve as a free, reproducible audit tool for hallucination rates in both open and commercial models.","The lower absolute scores suggest OpenFActScore is calibrated differently from the original; users who care about absolute factuality thresholds rather than rankings should recalibrate before comparing scores across papers.","A natural test is to run the same two-stage pipeline on other domains with factual ground truth, such as news summarization or non-English biography writing, to see whether the $r > 0.99$ transfers.","Because the paper chooses Olmo over Gemma for AFG based on a roughly 0.35 percent score difference plus openness, swapping in a slightly better open generator could change the correlation and should be checked before relying on the specific model pair."],"forward_implications":["Factuality scores for a new language model can be computed with no closed-source API calls, as long as a Wikipedia dump and an open validator are available.","Model rankings from the original FActScore benchmark are preserved by the open pipeline, so comparative conclusions from that benchmark carry over to OpenFActScore.","Gemma and Llama 3.1 are more reliable validators than Olmo and Qwen, indicating that strong atomic fact generation does not automatically imply strong atomic fact validation.","The published system prompts and chat-template handling make the exact scoring procedure reproducible by other researchers."],"supporting_citations":[{"why":"Supplies the original FActScore method, the human-annotated benchmark, the 10 model outputs, and the reference scores in settings (A) and (B) that OpenFActScore is compared against.","marker":"Min et al. (2023a)"},{"why":"Provides Olmo, the model chosen for atomic fact generation because it is fully open and nearly tied with Gemma on AFG BERTScore-F1.","marker":"OLMo et al. (2025)"},{"why":"Provides Gemma, the model chosen for atomic fact validation after showing the lowest cumulative error rate relative to human annotations.","marker":"Team et al. (2025)"},{"why":"Supplies BERTScore-F1, the measure used to estimate semantic overlap between generated atomic facts and human-annotated atomic facts.","marker":"Zhang* et al. (2020)"},{"why":"Supplies the model-loading library used by the HFModel class to load and prompt open models, enabling the model-agnostic design.","marker":"Wolf et al. (2019)"},{"why":"Supplies the GTR retriever used to fetch the top Wikipedia passages for each atomic fact during validation.","marker":"Ni et al. (2022)"},{"why":"Documents InstructGPT, the closed model originally used for atomic fact generation that OpenFActScore replaces with open models.","marker":"Ouyang et al. (2022)"}],"fun_headline_variants":["Open FActScore matches closed-source rankings at r=0.99","Open-source factuality eval hits 0.99 correlation with original","Olmo + Gemma reproduce FActScore rankings at r=0.99","Open pipeline achieves 0.99 Pearson correlation with FActScore","OpenFActScore: open models match closed-source factuality scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the original FActScore benchmark data, specifically the human-annotated atomic facts and the 10 model outputs used in Table 3, is an accurate and representative testbed for factuality evaluation; if these outputs are easy to rank or the human labels are noisy, a 0.99 correlation may only show agreement with the original automated pipeline, not with genuine human factuality judgments.","fun_headline_variants_meta":{"raw":{"variants":["Open FActScore matches closed-source rankings at r=0.99","Open-source factuality eval hits 0.99 correlation with original","Olmo + Gemma reproduce FActScore rankings at r=0.99","Open pipeline achieves 0.99 Pearson correlation with FActScore","OpenFActScore: open models match closed-source factuality scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001172,"raw_usage":{"total_tokens":4821,"prompt_tokens":893,"completion_tokens":3928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":3833}},"tokens_in":509,"tokens_out":3928,"duration_ms":30027,"temperature":1.0,"reasoning_tokens":3833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:13:50.876985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators score the same 10 model outputs used in Table 3 and compare their ranking with OpenFActScore's ranking; if the human-model rank correlation is much lower than 0.99, then the high correlation shows agreement with the closed-source estimator rather than with human factuality. Alternatively, compute OpenFActScore on a fresh set of model outputs from a different domain and check whether the model ranking still matches human judgment.","supporting_citations":[],"review_version":1}