Pith. sign in

REVIEW 3 major objections 5 minor 32 references

STEC: Evidence Compression for Deep Search in Open-domain Multi-Hop QA

T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read After multi-trajectory deep search, multi-hop QA picks better finals by comparing compressed, candidate-specific evidence than by ranking raw trajectories or bare answer strings.

desk verdict Clean modular fix for multi-trajectory answer selection; real but small gains, and the Compress step is still a black box. read the letter →

arxiv 2607.10795 v1 pith:6NL6QYHM submitted 2026-07-12 cs.AI cs.CL

Xinkang Li , Rong Jiang , Xin Song , Ye Wang , Yue Han , Changjian Li This is my paper
classification cs.AIcs.CL
keywords DeepSearchMulti-HopQuestionAnsweringTrajectoryEvidenceCompressionAnswerVerificationRetrieval-AugmentedGenerationTest-TimeScaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM search agents for open-domain multi-hop questions often produce many trajectories that support different candidates and mix redundant, incomplete, or conflicting evidence. Ranking raw trajectories dumps noise on the verifier; comparing answer strings alone throws away the support behind each candidate. STEC reframes the last step as candidate-centered evidence comparison: it groups trajectories by normalized answer identity, compresses each group into a structured evidence card, and has a verifier choose among those cards without inventing answers outside the search outputs. On four multi-hop benchmarks it reports the strongest overall exact-match among the methods compared, and an ablation that removes the compression step lowers average performance. A reader who already believes multi-trajectory search is useful should care because the coverage of many traces only helps if the final selection can actually use the evidence they gathered.

What carries the argument

Answer-Level Evidence Compression: for each normalized candidate a it constructs Ca = (Va, E+a, E−a, Pa, Γa)—raw answer variants, supporting evidence units, conflicting evidence units, a concise reasoning path from support to a, and group statistics—so Evidence-Guided Answer Verification compares aligned candidate cards rather than heterogeneous raw trajectories.

What would settle it

On fixed multi-trajectory outputs, replace Compress with a deliberately lossy or randomly labeled compressor (or with human/oracle cards) and re-run verification: if STEC’s gain over Verifier Only vanishes under lossy labels and does not improve under oracle cards, answer-level evidence compression is not what is driving better final selection.

Watch

Extended reading notes

Core claim

The paper argues that final answer selection after multi-trajectory search is a candidate-centered evidence comparison problem, not a raw-trajectory or string-ranking problem. STEC groups trajectories by normalized answer identity, builds for each candidate a compact representation of answer variants, supporting evidence, conflicting evidence, a concise reasoning path, and group statistics, then runs an evidence-guided verifier constrained to that candidate set. Empirically STEC achieves the best macro-average EM (0.347) among reported baselines on HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle, with ablation support that answer-level evidence compression contributes to the gain.

Load-bearing premise

The compression step can extract, de-duplicate, and label supporting versus conflicting evidence and a faithful reasoning path from each answer group without losing key facts or inventing new ones.

Editorial extensions

If this is right

  • Final selection after multi-trajectory search can be treated as constrained comparison among existing candidates rather than open-ended generation or raw-trace ranking.
  • Multi-trajectory search pays off more when evidence is reorganized at answer granularity before verification.
  • Larger trajectory budgets improve average exact match, and compression remains helpful as N grows.
  • The same compression-plus-verification pattern improves over verifier-only across 3B/7B base and instruct settings, with clearer relative gains when the model is smaller.
  • Answer-string majority or direct trajectory comparison can be replaced by structured candidate-level evidence comparison without leaving the search-produced candidate set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Candidate-level evidence cards could transfer to other multi-sample agent settings (tool use, code synthesis, multi-agent debate) where several traces propose different finals.
  • If compression fidelity is the real bottleneck, measuring E+/E− label accuracy and path faithfulness against the original trajectories is the next decisive experiment.
  • Verifier prompts or training may gain more from structured evidence fields than from dumping full chain-of-thought traces.
  • Systems that already sample many search trajectories may improve final accuracy more by reorganizing existing outputs than by increasing N alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes STEC, a post-search final-answer selection framework for open-domain multi-hop QA. After multi-trajectory search produces heterogeneous trajectories with possibly conflicting candidates, STEC groups trajectories by normalized answer identity, compresses each group into a candidate-specific evidence card Ca=(Va,E+a,E−a,Pa,Γa), and runs an evidence-guided verifier constrained to the existing candidate set. On HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle with Search-R1 trajectories and a Qwen2.5-7B backbone, STEC reports the best macro-average EM (0.347) among the listed baselines, with ablations and N/model analyses offered as support that answer-level compression contributes.

Significance. Final selection after multi-trajectory deep search is a real and under-addressed bottleneck: raw-trajectory comparison is noisy, and answer-string comparison discards evidence. Framing selection as candidate-centered evidence comparison is a useful design shift and is compatible with existing search generators. The main-table gains over Search-R1 (+0.046 avg EM) and consistent wins across four datasets are practically meaningful if the mechanism holds. Strengths include a clear problem statement, constrained selection (no free-form answer invention), multi-dataset evaluation, and sensitivity checks over N and model size/tuning. The contribution is primarily empirical and systems-level rather than theoretical; its lasting value depends on whether compression fidelity and the ablation isolation of that step can be made rigorous.

major comments (3)
  1. §3.3 (Eqs. 5–8): The central mechanism is Compress(q,a,Ga)→Ca=(Va,E+a,E−a,Pa,Γa), including extraction, de-duplication, support/conflict labeling, and construction of a faithful reasoning path Pa. The section is purely procedural: no algorithm, prompt, model choice, or fidelity metric is given for E+/E− labels or Pa. Without an independent check that compression preserves evidence quality (or at least does not systematically invent/mislabel units), the claim that structured cards are a valid proxy for evidence comparison remains an untested assumption rather than a demonstrated design contribution.
  2. Table 4 and §4.3: The ablation gain of Answer-Level Evidence Compression is only +0.008 avg EM (0.339→0.347), with a slight drop on MuSiQue (0.194→0.193) and no error bars, seeds, or significance tests. More critically, “Verifier Only” is defined only as removing compression “while keeping the other settings unchanged.” It is unclear whether the control receives raw multi-trajectory dumps, grouped but uncompressed trajectories, or answer strings alone. If the control already benefits from grouping or a stronger verifier prompt, the +0.008 does not isolate compression fidelity as the operative mechanism.
  3. §4.1 Implementation and Table 3: Main results fix the trajectory source to Search-R1 (N=8) and the backbone to Qwen2.5-7B. STEC is presented as a general evidence-compression framework, but all primary gains are measured as a post-processor on one generator. Without at least one alternative trajectory source (e.g., Search-o1 or IRCoT rollouts) under the same selection module, it is hard to separate STEC’s contribution from Search-R1’s coverage and answer distribution. This is load-bearing for the generality claim in the abstract and conclusion.
minor comments (5)
  1. Fig. 1 and Fig. 2 use heavily garbled OCR-style labels (e.g., “7[KYZOUT 'TY]KX”), which makes the overview figures hard to read; please regenerate with clean text.
  2. Table 1 notation is helpful, but Norm(·) (§3.1) is described only as “conservative” surface normalization; a precise rule list or pseudocode would aid reproducibility.
  3. §4.5 uses 1,000-example subsets for HotpotQA/2Wiki/MuSiQue without stating sampling procedure or whether the same subset is used across model configurations.
  4. Related work (§2.2) covers verifier-based test-time scaling well; a short explicit contrast with majority vote / self-consistency over answer strings would sharpen the positioning of evidence-guided selection.
  5. Bamboogle has only 125 test examples; report absolute counts or confidence intervals when claiming best EM (0.456) on that set.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: STEC is an empirical selection framework whose EM gains are measured on external multi-hop benchmarks, not forced by definition or self-citation.

full rationale

STEC’s load-bearing claim is empirical: after multi-trajectory search, grouping trajectories by normalized answer identity, compressing each group into Ca=(Va,E+a,E−a,Pa,Γa), and verifying over those cards improves final Exact Match relative to baselines and to a Verifier-Only ablation (Tables 3–4; avg EM 0.347 vs 0.339). That claim is not obtained by construction from the method’s inputs. Equations (1)–(9) only define the pipeline (trajectory set T(q), Norm, Unique, Ga, Compress, Verifier constrained to A(q)); they do not equate the reported EM to a fitted identity or to a quantity already used as input. Search-R1 is reused as an upstream trajectory generator and as a baseline, which is ordinary system dependence, not a self-citation uniqueness theorem that forces the result. Prior work by overlapping authors (e.g., Search-o1) appears as related work/baseline, not as a load-bearing uniqueness or ansatz that the present derivation reduces to. There is no fitted-parameter-as-prediction step, no renaming of a known closed-form result as a first-principles derivation, and no self-definitional loop (X defined via Y then “predicted” as Y). Underspecification of Compress and a small ablation delta are correctness/measurement concerns, not circularity. Against external HotpotQA/2Wiki/MuSiQue/Bamboogle EM, the paper is self-contained; circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on standard multi-hop QA evaluation practice, on the usefulness of multi-trajectory search outputs as a candidate pool, and on an ad-hoc but explicit evidence-card schema. Free choices include trajectory budget N and conservative answer normalization. No physical entities are postulated; the main invented construct is the candidate-specific evidence representation used as the verification interface.

free parameters (3)
  • N (number of search trajectories)
    Default N=8 is a hand-chosen compute budget that materially affects candidate coverage and reported EM (Table 5); not derived from theory.
  • Answer normalization rules (Norm)
    Case/punctuation/article/whitespace normalization is a design choice that defines answer identity groups Ga; different rules would change candidate sets and all downstream comparisons.
  • Evidence-card field schema and compression policy
    Which units enter E+/E−, how Pa is summarized, and how Γa is counted are implementation choices that define the verification input C(q).
assumptions (4)
  • domain assumption Exact Match after answer normalization is an adequate primary metric for multi-hop open-domain QA performance.
    All main claims in §4 use EM only; no human preference or partial-credit evidence is provided.
  • domain assumption Multi-trajectory search already places useful candidates and evidence into T(q) often enough that selection, not generation of new answers, is the right final stage.
    STEC is constrained to choose â ∈ A(q) from search outputs (§3.1); gains presuppose coverage by the upstream searcher.
  • ad hoc to paper A five-field candidate-specific representation Ca=(Va,E+a,E−a,Pa,Γa) is a sufficient and comparable interface for evidence-guided verification.
    Introduced in §3.3 as the compression target without independent validation that these fields capture all decision-relevant evidence.
  • domain assumption Conservative surface-form normalization merges only equivalent answers and does not incorrectly collapse distinct meanings.
    Stated in §3.1 as the basis for A(q) and Ga; correctness of grouping depends on this.
invented entities (1)
  • Candidate-specific evidence representation Ca
    purpose: Provide a uniform, compressed evidence interface per normalized answer so verification compares candidates rather than raw trajectories.
    Defined in §3.3 as Ca=(Va,E+a,E−a,Pa,Γa); it is a paper-specific data structure whose usefulness is argued mainly by downstream EM gains, not by external independent measurement of representation quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STEC: Evidence Compression for Deep Search in Open-domain Multi-Hop QA." pith.science (2026). https://pith.science/paper/6NL6QYHM

@misc{pith2026260710795,
  author       = {Pith},
  title        = {Pith review of: STEC: Evidence Compression for Deep Search in Open-domain Multi-Hop QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NL6QYHM}},
  note         = {Machine review of arXiv:2607.10795}
}
read the original abstract

In open-domain multi-hop question answering (QA), LLM-based search agents offer a promising approach to knowledge-intensive QA by combining retrieval with reasoning. Existing methods mainly improve open-domain multi-hop QA through reasoning paradigms, retrieval interaction, and search strategy optimization. However, using multiple search trajectories introduces a challenging final answer selection problem. Different trajectories may support different candidates, and the retrieved information can be heterogeneous, redundant, incomplete, or conflicting. Directly comparing raw trajectories exposes the verifier to noisy and unaligned content, while comparing answer strings ignores the evidence supporting each candidate, making reliable final selection difficult. To address this challenge, we propose STEC, an evidence compression framework for final answer selection in multi-hop QA. STEC selects the final answer from the existing candidate set through two mechanisms: (1) Answer-Level Evidence Compression, which groups trajectories by normalized answer identity and converts each answer group into a candidate-specific evidence representation; and (2) Evidence-Guided Answer Verification, which compares these representations and selects the final answer from the candidate set. The design shifts final selection from raw trajectory comparison to candidate-level evidence comparison. We evaluate STEC on four open-domain multi-hop QA benchmarks against representative baselines. Experimental results show that STEC performs best overall among the compared methods, and ablation results provide evidence that answer-level evidence compression contributes to final answer selection.

Figures

Figures reproduced from arXiv: 2607.10795 by the authors.

Figure 1
Figure 1. Illustration of open-domain multi-hop question answering. A system retrieves relevant information from a large-scale open corpus and reasons across multiple hops to derive the final answer. multi-step reasoning. Unlike single-hop question answering, a multi-hop question usually cannot be answered with a single retrieved passage. The model must combine information from multiple sources through reliable multi-step rea… view at source ↗
Figure 2
Figure 2. Overview of STEC. Given a question, multi-trajectory search first produces search trajectories with candidate answers. STEC then groups trajectories by normal￾ized answer identity, compresses each answer group into a candidate-specific evidence representation, and performs evidence-guided verification to select the final answer from the existing candidate set. duces multiple trajectories for a question; (2) Answer-L… view at source ↗
Figure 3
Figure 3. Effect of the number of search trajectories. Left: EM scores of Verifier Only under different values of N. Right: EM scores of STEC under different values of N. Comparing the two methods, STEC matches Verifier Only at N = 2 and achieves higher average EM at N = 4 and N = 8. This comparison suggests that Answer￾Level Evidence Compression remains effective under larger trajectory budgets. The per-dataset results furth… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 4 canonical work pages

  1. [1]

    uttler, H., Lewis, M., Yih, W.-T., Rockt

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K"uttler, H., Lewis, M., Yih, W.-T., Rockt"aschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In: Advances in Neural Informa- tion Processing Systems 33, pp. 9459–9474 (2020)

    2020

  2. [2]

    In: The Eleventh International Conference on Learning Representations (ICLR 2023) (2023)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Syn- ergizing Reasoning and Acting in Language Models. In: The Eleventh International Conference on Learning Representations (ICLR 2023) (2023)

    2023

  3. [3]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Trivedi, H., Balasubramanian, N., Khot, T., Sabharwal, A.: Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014–10037. Association for Computa- tional Linguistics, Toronto (2023). ht...

    doi:10.18653/v1/2023.acl-long.557 2023

  4. [4]

    In: The Twelfth International Con- ference on Learning Representations (ICLR 2024) (2024)

    Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H.: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In: The Twelfth International Con- ference on Learning Representations (ICLR 2024) (2024)

    2024

  5. [5]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, pp

    Li, X., Dong, G., Jin, J., Zhang, Y., Zhou, Y., Zhu, Y., Zhang, P., Dou, Z.: Search-o1: Agentic Search-Enhanced Large Reasoning Models. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, pp. 5420–5438. Association for Computational Linguistics, Suzhou (2025). https://doi.org/10.18653/v1/2025.emnlp-main.276

    doi:10.18653/v1/2025.emnlp-main.276 2025

  6. [6]

    Jiang, J., Chen, J., Li, J., Ren, R., Wang, S., Zhao, W.X., Song, Y., Zhang, T.: RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Ver- ification and Refinement. In: Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Pap...

    work page doi:10.18653/v1/2025.naacl-long.361 2025

  7. [7]

    In: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) (2025)

    Li, X., Jin, J., Dong, G., Qian, H., Wu, Y., Wen, J.-R., Zhu, Y., Dou, Z.: Web- Thinker: Empowering Large Reasoning Models with Deep Research Capability. In: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) (2025)

    2025

  8. [8]

    In: Pro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, pp

    Zheng, Y., Fu, D., Hu, X., Cai, X., Ye, L., Lu, P., Liu, P.: DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. In: Pro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 414–431. Association for Computational Linguistics, Suzhou (2025). https://doi.org/10.18653/v1/2025.emn...

    doi:10.18653/v1/2025.emnlp-main.22 2025

Show all 32 references
  1. [9]

    Proceedings of the AAAI Conference on Artificial Intelligence 40(23), 19380–19388 (2026)

    Tan, J., Dou, Z., Yu, Y., Cheng, J., Liu, L., Xie, J., Wen, J.-R.: HierSearch: A Hi- erarchical Enterprise Deep Search Framework Integrating Local and Web Searches. Proceedings of the AAAI Conference on Artificial Intelligence 40(23), 19380–19388 (2026). https://doi.org/10.1609...

    2026 doi

  2. [10]

    arXiv preprint arXiv:2503.09516 (2025) STEC: Evidence Compression for Open-domain Multi-Hop QA 15

    Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., Han, J.: Search- R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint arXiv:2503.09516 (2025) STEC: Evidence Compression for Open-domain Multi-Hop QA 15

    2025 arXiv

  3. [11]

    In: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) (2025)

    Chen, M., Sun, L., Li, T., Sun, H., Zhou, Y., Zhu, C., Wang, H., Pan, J.Z., Zhang, W., Chen, H., Yang, F., Zhou, Z., Chen, W.: ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. In: Advances in Neural Information Processing Systems 38 (NeurIPS 2025) (2025)

    2025

  4. [12]

    arXiv preprint arXiv:2503.05592 (2025)

    Song, H., Jiang, J., Min, Y., Chen, J., Chen, Z., Zhao, W.X., Fang, L., Wen, J.- R.: R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2503.05592 (2025)

    2025 arXiv

  5. [13]

    arXiv preprint arXiv:2505.04588 (2025)

    Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y., Jiang, Y., Xie, P., Zhang, Y., Huang, F., Zhou, J.: ZeroSearch: Incentivize the Search Capability of LLMs with- out Searching. arXiv preprint arXiv:2505.04588 (2025)

    2025 arXiv

  6. [14]

    In: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), Melbourne/Naarm, Australia, to appear (2026)

    Wen, T., Dong, G., Dou, Z.: SmartSearch: Process Reward-Guided Query Refine- ment for Search Agents. In: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), Melbourne/Naarm, Australia, to appear (2026)

    2026

  7. [15]

    Proceedings of the AAAI Conference on Artificial Intelligence 40(40), 34142–34150 (2026)

    Xu, J., Du, X., Ao, Y., Zhao, P., Li, Y., Zhong, L., Yuan, L., Bo, Z., Wang, X., Sun, M., Gui, Z., Zhang, D., Wang, Z., Wang, Q., Hou, Y., Yin, Z., Wang, H., Chen, H., Liang, L., Zhou, J.: Thinker: Training LLMs in Hier- archical Thinking for Deep Search via Multi-Turn Interac...

    2026 doi

  8. [16]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp

    Zhang, D.-C., Zhao, Y., Wu, J., Zhang, L., Li, B., Yin, W., Jiang, Y., Li, Y.-F., Tu, K., Xie, P., Huang, F.: EvolveSearch: An Iterative Self-Evolving Search Agent. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 13123–13136. Ass...

    2025 doi

  9. [17]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp

    Thatikonda, R.K., Buntine, W., Shareghi, E.: Logical Reasoning with Outcome Reward Models for Test-Time Scaling. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 26102–26112. Association for Computational Linguistics, Suzhou (2025...

    2025 doi

  10. [18]

    arXiv preprint arXiv:2505.14999 (2025)

    Jiang, E.H., Luo, H., Pang, S., Li, X., Qi, Z., Li, H., Yang, C.-F., Lin, Z., Li, X., Xu, H., Chang, K.-W., Wu, Y.N.: Learning to Rank Chain-of-Thought: Using a Small Model. arXiv preprint arXiv:2505.14999 (2025)

    2025

  11. [19]

    In: The Thirteenth International Conference on Learning Representations (ICLR 2025) (2025)

    Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., Kumar, A.: Rewarding Progress: Scaling Automated Process Veri- fiers for LLM Reasoning. In: The Thirteenth International Conference on Learning Representations (ICLR 2025) (2025)

    2025

  12. [20]

    Proceedings of the AAAI Conference on Artificial In- telligence 40(41), 34932–34940 (2026)

    Zhao, J., Liu, R., Zhang, K., Zhou, Z., Gao, J., Li, D., Lyu, J., Qian, Z., Qi, B., Li, X., Zhou, B.: GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning. Proceedings of the AAAI Conference on Artificial In- telligence 40(41), 34932–34940 (2026)....

    2026 doi

  13. [21]

    In: Proceedings of the 42nd In- ternational Conference on Machine Learning, PMLR 267, pp

    Zhao, E., Awasthi, P., Gollapudi, S.: Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification. In: Proceedings of the 42nd In- ternational Conference on Machine Learning, PMLR 267, pp. 77272–77309 (2025)

    2025

  14. [22]

    In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answer- ing. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380. Assoc...

    2018 doi

  15. [23]

    In: Proceedings of the 28th International Conference on Computational Linguistics, pp

    Ho, X., Duong Nguyen, A.-K., Sugawara, S., Aizawa, A.: Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625. International Committee on Computationa...

    2020 doi

  16. [24]

    Transactions of the Association for Computational Linguistics 10, 539–554 (2022)

    Trivedi, H., Balasubramanian, N., Khot, T., Sabharwal, A.: MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics 10, 539–554 (2022). https://doi.org/10.1162/tacl_a_00475

    2022 doi

  17. [25]

    In: Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp

    Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N.A., Lewis, M.: Measuring and Narrowing the Compositionality Gap in Language Models. In: Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 5687–5711. Association for Computational Linguistics (2023)...

    2023 doi

  18. [26]

    In: Advances in Neural Information Processing Systems 35, pp

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems 35, pp. 24824– 24837 (2022)

    2022

  19. [27]

    Journal of Machine Learning Research 25(70), 1–53 (2024)

    Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V.Y., Hu...

    2024

  20. [28]

    arXiv preprint arXiv:2501.12948 (2025)

    DeepSeek-AI, Guo, D., Yang, D., et al.: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  21. [29]

    arXiv preprint arXiv:2402.00157 (2024)

    Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., Yin, W.: Large Language Models for Mathematical Reasoning: Progresses and Challenges. arXiv preprint arXiv:2402.00157 (2024)

    2024 arXiv

  22. [30]

    Trans- actions on Machine Learning Research (2025)

    Chen, J., Ren, J., Chen, X., Yang, C., Sun, R., Yoon, J., Arik, S.O.: SETS: Lever- aging Self-Verification and Self-Correction for Improved Test-Time Scaling. Trans- actions on Machine Learning Research (2025)

    2025

  23. [31]

    In: Conference on Language Modeling (COLM

    Lifshitz, S., McIlraith, S.A., Du, Y.: Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers. In: Conference on Language Modeling (COLM

  24. [32]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp

    Chang, K., Shi, Y., Wang, C., Zhou, H., Hu, C., Liu, X., Luo, Y., Ge, Y., Xiao, T., Zhu, J.: Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 18462–1847...

    2025 doi

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.