Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification

Jaeyoon Jung; Kunwoo Park; Seunghyun Yoon; Yejun Yoon

REVIEW 4 major objections 5 minor 11 references

Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HerO 2, an open-source fact-checking pipeline, ranked second in the AVeriTeC 2025 shared task with an AVeriTeC score of 0.271 while averaging the shortest runtime, 29.19 seconds per claim, among the top three systems.

desk verdict A competent shared-task system description with an externally verified second-place leaderboard finish, but the internal ablations don't cleanly support the headline component claims. read the letter →

arxiv 2507.11004 v1 pith:FVP4QUOX submitted 2025-07-15 cs.CL

Yejun Yoon , Jaeyoon Jung , Seunghyun Yoon , Kunwoo Park This is my paper

classification cs.CL

keywords factverificationAVeriTeCsharedtaskevidenceretrievalanswerreformulationAWQquantizationQwen3efficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents HerO 2, an open-source fact-checking system entered in the AVeriTeC 2025 shared task. It claims that four changes to the previous HerO system — summarizing retrieved documents into paragraph-level evidence, rewriting that evidence into answer-form text, quantizing the veracity-prediction model to 4-bit with AWQ, and switching to newer Qwen3 and Llama3.1 backbones — improve both evidence quality and speed. On the test set, HerO 2 ranked second with an AVeriTeC score of 0.271 and averaged 29.19 seconds per claim, the shortest runtime among the top three systems. The paper argues this shows efficient, reproducible fact verification is achievable with open models under tight constraints.

What carries the argument

The pipeline's load-bearing mechanism is the evidence-quality loop: HyDE-FC query expansion retrieves top-10 whole documents from a dense index (gte-base-en-v1.5); Qwen3 8B summarizes each document into one paragraph; Qwen3 8B conditions on the claim and a generated question to reformulate that summary into answer-form; and a fine-tuned Qwen3 32B, quantized to 4-bit with AWQ, reads the top-10 question-answer pairs plus the claim to output a verdict. The answer reformulation step is what makes the evidence align with the Ev2R model-based metric, and the quantization is what makes the stronger veracity model fit the hardware limit.

What would settle it

Run the HerO 2 pipeline on the development set, then measure factual consistency between each original retrieved document and its summarized or reformulated evidence, for example by checking atomic-fact overlap with a strong LLM judge. If summarized evidence introduces substantial unsupported facts or drops facts needed to answer the generated question, or if replacing summarized evidence with the original documents raises the Ev2R Q+A score, the reported contribution of the summarization and reformulation steps would be undermined.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that document-level retrieval followed by LLM summarization and answer reformulation produces evidence that scores substantially higher under the Ev2R recall metric than sentence- or chunk-level retrieval (0.522 vs 0.374 top-10 Q+A), and that AWQ 4-bit quantization of a fine-tuned Qwen3 32B veracity model improves accuracy by 0.018 over Qwen3 8B while fitting in 23GB VRAM. Together these components yield second place with the shortest runtime.

Load-bearing premise

The system assumes that the Qwen3 8B document summarization and answer reformulation steps preserve the factual content of the retrieved web documents, since no faithfulness evaluation is reported and any hallucinated or omitted fact would flow into the evidence score and the final verdict.

Editorial extensions

If this is right

Document-level retrieval with summarization beats finer-grained retrieval targets under Ev2R, so future systems can simplify retrieval granularity.
Answer reformulation consistently improves Q+A over raw summaries, suggesting evidence should be shaped to the question-answer format the scorer expects.
AWQ quantization gives accuracy gains on top of a larger backbone instead of costing accuracy, making 32B veracity models practical on 23GB GPUs.
The 29.19-second per-claim runtime shows the one-minute constraint can be met with open models, supporting deployment in realistic fact-checking pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported gains depend on the summarization LLM being faithful; a direct faithfulness check (e.g., comparing atomic facts before and after summarization) would tell whether the evidence scores measure retrieval or compression artifacts.
Because the paper reports single-run dev results without variance except on the test set, the 0.018 AWQ accuracy difference may be within run-to-run noise; a paired multi-seed comparison would sharpen the claim.
The Ev2R threshold of 0.5 means the AVeriTeC score depends on both evidence quality and veracity accuracy, so improving evidence quality alone might push more claims over the threshold, an interaction the paper does not isolate.
The approach is transferable to other evidence-based verification tasks with model-based recall metrics, where answer-formatted evidence could improve scores without retraining the verifier.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid shared-task write-up. HerO 2 is HerO plus document summarization, answer reformulation, AWQ-quantized Qwen3 32B for veracity, and updated backbones. It took second on the AVeriTeC 2025 leaderboard with the shortest runtime among the top three (29.19s vs 53.67 and 31.71). That result is externally anchored, so the central claim is trustworthy at the system level.

What's genuinely new is the combination, not any single module. Answer reformulation—turning retrieved summaries into answer-form text conditioned on the claim and question—is a reasonable and somewhat novel trick for Ev2R, which is a recall metric that rewards semantic coverage of atomic facts. The AWQ 4-bit quantization of a 32B veracity model under the task's one-minute and VRAM constraints is also a practical data point. Credit where due: code is public, the leaderboard position is official, and the paper is honest about the development/testing split.

The soft spots are real but not fatal. Development-set support for the two headline components is weaker than the prose suggests. In Table 3, at the operating point used by the final system (top-10), adding document summarization to document retrieval actually drops Ev2R recall from 0.522 to 0.487; only after answer reformulation does it climb to 0.556. So the paper does not show that summarization alone improves evidence at top-10. More importantly, neither the summaries nor the reformulated answers are checked for faithfulness. Since Ev2R is computed by an LLM, hallucinated but fluent evidence could inflate the score. That matters because the final pipeline feeds these generated texts into veracity prediction. A small faithfulness study, even human-rated on 100 examples, would close the gap.

The other thing worth flagging: Table 2 contains an outlier—Chunk(4 sentences) at top-10 gives 0.115, far below its top-5 value of 0.41. That suggests a bug or a one-off run, and it undermines confidence in the development-set numbers, which appear to be single runs without error bars or significance tests. Also, the task description says models over 10B parameters are prohibited, but the final veracity model is Qwen3 32B AWQ. If that's a literal parameter cap, there's a rule-compliance question the paper should address explicitly; if the cap is actually about memory, say so.

Bottom line: the leaderboard result is solid and the system is useful for applied fact-checking. The component-level attribution is not yet established. A serious referee should ask for corrected, repeated ablations and a faithfulness check. I'd send it out; it's a legitimate workshop paper with more value than the average system description.

Referee Report

4 major / 5 minor

Summary. The paper describes HerO 2, Team HUMANE's system for the AVeriTeC 2025 shared task. HerO 2 extends the authors' previous HerO system with dense document retrieval using gte-base-en-v1.5, document summarization and answer reformulation performed by Qwen3 8B, a fine-tuned Qwen3 32B veracity predictor quantized to 4-bit with AWQ, and updated backbone language models throughout the pipeline. On the official test set, HerO 2 ranks second with an AVeriTeC score of 0.271, behind CTU AIC at 0.332, and it achieves the shortest average runtime among the top three systems at 29.19 seconds per claim. The paper also reports development-set ablations for retrieval target type, the effect of summarization and answer reformulation, and the choice of veracity prediction model.

Significance. The official leaderboard result is externally verified and gives the paper a solid headline claim: HerO 2 is the second-best system in a competitive shared task and the fastest among the top three systems, using only open models and fitting within the task's hardware constraints. The code release supports reproducibility, and the runtime advantage is directly relevant to the shared task's efficiency goal. However, the paper's internal, development-set evidence for two headline components—document summarization and answer reformulation—is not conclusive as reported, and the lack of any faithfulness evaluation is a real gap given that Ev2R is an LLM-based recall metric. The paper's contribution as a system description is still useful, but the causal attribution of the second-place result to those components needs stronger support.

major comments (4)

[§4.2, Table 3] At the operating point used by the final system (top-10 evidence), adding document summarization to document-based retrieval decreases Ev2R recall from 0.522 to 0.487; the increase to 0.556 appears only after answer reformulation is also applied. The paper therefore does not demonstrate that summarization alone improves evidence quality, and the abstract and conclusion's attribution of the result to 'document summarization and answer reformulation' as a package is not supported by the isolated ablation. The comparison also lacks error bars or significance tests, so it is unclear whether the reported differences are stable.
[§3.2–3.3, Table 3] No faithfulness or factual-overlap evaluation is reported for the Qwen3 8B summaries and reformulated answers. Because Ev2R recall is computed by an LLM decomposing ground-truth evidence into atomic facts, fluent but hallucinated summaries or answers could inflate the recall score without being faithful to the retrieved documents. The authors should either provide a faithfulness check (e.g., manual or automatic overlap between the generated evidence and source documents) or explicitly qualify the claim that summarization and reformulation improve evidence quality.
[§4.2, Table 2] The reported value for Chunk (4 sentences) at Top-10, 0.115, is far below both the Top-5 value for the same target (0.41) and the Top-3 value for Sentence retrieval (0.289). This anomaly is not discussed and suggests either an evaluation bug or high variance in the development-set pipeline. Since all development-set numbers come from single runs, this value undermines confidence in the internal ranking of retrieval strategies and in the comparison between Tables 2 and 3. The authors should investigate and correct this value, or report multiple runs with variance.
[§4.2, Table 4] The claim that AWQ post-training quantization of Qwen3 32B 'yields a +0.018 improvement in accuracy' over Qwen3 8B is based on a single run with no significance test or stated variance. A 1.8-point accuracy difference on a development set is plausibly within noise, especially given the F1 scores are essentially identical (0.382 vs. 0.385). The text should either present repeated runs or temper the conclusion that quantization 'enables higher accuracy'.

minor comments (5)

[Throughout] The system name is spelled inconsistently as 'Hero 2' in several places (e.g., Section 1, Figure 1) and 'HerO 2' elsewhere; please use a single spelling consistently.
[Tables 2–3] The captions state that 'the number of retrieved evidence documents' or 'candidates' is varied, but the rows in Table 3 appear to be cumulative pipeline additions. Please clarify whether '+ document summarization' and '+ answer reformulation' are cumulative, and define precisely what each row includes.
[§4.1] The text says 'In other cases, we employed mxbai-embed-large-v1' but does not specify which experiments used this embedding model; please state where gte-base-en-v1.5 is used and where mxbai-embed-large-v1 is used.
[References] Several references contain formatting errors, including 'InProceedings' with a missing space, 'and 1 others' in the author lists, and inconsistent spacing in 'A VeriTeC' (e.g., in the Schlichtkrull et al. entry). Please correct these.
[§4.1] Footnote 1 gives a Hugging Face URL for Qwen hyperparameters without an accessed date; please add an access date or a version identifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the system is evaluated on an external shared-task leaderboard and the ablations are empirical.

full rationale

The paper contains no derivation chain in which an output is defined in terms of an input or in which a fitted quantity is renamed as a prediction. Its central result, the second-place AVeriTeC 2025 score of 0.271, is computed by the task organizers using the externally defined Ev2R metric (Akhtar et al., 2024), not by the authors' own formulas. The development-set comparisons in Tables 2-4 are straightforward empirical ablations that guided module choices, and the test-set numbers are independent of those choices except in the normal sense of system selection. The self-citations to HerO and HyDE-FC (Yoon et al., 2024) supply a baseline and a query-expansion component, but the paper does not invoke those citations to justify the effectiveness of the new components; that effectiveness is supported by locally run experiments scored by an external LLM. The absence of faithfulness checks for summarization and reformulation is a validity or robustness concern, not a circularity, since no claim reduces by construction to the metric that measures it. The anomalous value in Table 2 (chunk-of-4 top-10 recall 0.115) and the inconsistent top-10 effect of summarization in Table 3 are evidence-quality issues, not circular reasoning. Therefore no circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

This is an empirical systems paper, so the ledger contains hand-chosen system configuration numbers rather than fitted physical constants. The main choices are the retrieval count, the number of Q+A pairs, the quantization bit-width, and decoding parameters. The paper relies on the Ev2R metric and on LLM faithfulness as unvalidated domain assumptions.

free parameters (4)

Number of retrieved documents = 10
Selected via development set comparison in Table 2; top-10 document-level retrieval gave the highest Q+A (Ev2R recall) of 0.522.
Number of question-answer pairs = 10
Selected via development set comparison in Table 3; using the top-10 reformulated Q+A pairs gave the highest Q+A score of 0.556.
AWQ bit-width = 4
Chosen to fit the 23GB VRAM of an A10G GPU; the paper does not compare 4-bit AWQ against unquantized Qwen3 32B, so any accuracy loss from quantization is untested.
Veracity decoding hyperparameters = temperature 0.9, top-p 0.7, top-k 1, fallback top-2
Hand-selected values listed in Section 4.1; no sensitivity analysis is reported.

assumptions (3)

domain assumption Ev2R is a valid measure of evidence quality
The system is optimized and evaluated against Ev2R recall computed by Llama 3.3 70B with a threshold of 0.5; the validity of the entire ranking depends on this metric.
domain assumption Instruction-tuned LLMs produce faithful summaries and answers
Qwen3 8B performs document summarization and answer reformulation; no faithfulness checks are reported, so any hallucination or omission is implicitly assumed not to occur.
domain assumption The official shared task evaluation is accurate
The central claim (second place, fastest runtime) relies on the leaderboard and runtime measurements provided by the AVeriTeC 2025 organizers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification." pith.science (2026). https://pith.science/paper/FVP4QUOX

@misc{pith2026250711004,
  author       = {Pith},
  title        = {Pith review of: Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVP4QUOX}},
  note         = {Machine review of arXiv:2507.11004}
}

read the original abstract

This paper presents HerO 2, Team HUMANE's system for the AVeriTeC shared task at the FEVER-25 workshop. HerO 2 is an enhanced version of HerO, the best-performing open-source model from the previous year's challenge. It improves evidence quality through document summarization and answer reformulation, optimizes veracity prediction via post-training quantization under computational constraints, and enhances overall system performance by integrating updated language model (LM) backbones. HerO 2 ranked second on the leaderboard while achieving the shortest runtime among the top three systems, demonstrating both high efficiency and strong potential for real-world fact verification. The code is available at https://github.com/ssu-humane/HerO2.

Figures

Figures reproduced from arXiv: 2507.11004 by the authors.

**Figure 1.** Pipeline of our system System Query Expansion Evidence Retrieval Evidence Summarization Question Generation Answer Reformulation Veracity Prediction Baseline HyDE-FC (Llama3.1 8B) Hybrid (BM25/SFR-embedding-2) NA Llama3 8B NA Llama3.1 8B HerO 2 Dense (gte-base-en-v1.5) Qwen3 8B Qwen3 8B Qwen3 8B Qwen3 32B AWQ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** An example of the instruction prompt used [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 1 canonical work pages

[1]

Mubashara Akhtar, Michael Schlichtkrull, and Andreas Vlachos. 2024. Ev2r: Evaluating evidence retrieval in automated fact-checking. arXiv preprint arXiv:2411.05375

arXiv 2024
[2]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024
[3]

Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. https://www.mixedbread.ai/blog/mxbai-embed-large-v1 Open source strikes bread - new fluffy embeddings model

2024
[4]

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281

arXiv 2023
[5]

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

2024
[6]

Mark Rothermel, Tobias Braun, Marcus Rohrbach, and Anna Rohrbach. 2024. https://doi.org/10.18653/v1/2024.fever-1.12 I n F act: A strong baseline for automated fact-checking . In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 108--112, Miami, Florida, USA. Association for Computational Linguistics

doi:10.18653/v1/2024.fever-1.12 2024
[7]

Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos. 2024. https://aclanthology.org/2024.fever-1.1/ The automated verification of textual claims ( AV eri T e C ) shared task . In Proceedings of the Seventh...

work page 2024
[8]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

arXiv 2025

Show all 11 references

[9]

Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park. 2024. https://doi.org/10.18653/v1/2024.fever-1.15 H er O at AV eri T e C : The herd of open large language models for verifying real-world claims . In Proceedings of the Seventh Fact Extraction and VERification Worksho...

2024 doi
[10]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Mubashara Akhtar, Michael Schlichtkrull, and Andreas Vlachos. 2024. Ev2r: Evaluating evidence retrieval in automated fact-checking. arXiv preprint arXiv:2411.05375

arXiv 2024

[2] [2]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

[3] [3]

Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. https://www.mixedbread.ai/blog/mxbai-embed-large-v1 Open source strikes bread - new fluffy embeddings model

2024

[4] [4]

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281

arXiv 2023

[5] [5]

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

2024

[6] [6]

Mark Rothermel, Tobias Braun, Marcus Rohrbach, and Anna Rohrbach. 2024. https://doi.org/10.18653/v1/2024.fever-1.12 I n F act: A strong baseline for automated fact-checking . In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 108--112, Miami, Florida, USA. Association for Computational Linguistics

doi:10.18653/v1/2024.fever-1.12 2024

[7] [7]

Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos. 2024. https://aclanthology.org/2024.fever-1.1/ The automated verification of textual claims ( AV eri T e C ) shared task . In Proceedings of the Seventh...

work page 2024

[8] [8]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

arXiv 2025