Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction

Eneko Agirre; Mikel Zubillaga; Oier Lopez de Lacalle; Oscar Sainz

REVIEW 3 major objections 5 minor 36 references

Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For document-level information extraction, sampling and selection consistently outperform greedy decoding and set a new state of the art.

desk verdict A genuine first TTS-for-DocIE adaptation with a clean unsupervised selector, but every sampled result rests on one seed and the 'consistent' claim is contradicted by the paper's own non-reasoning tables. read the letter →

arxiv 2601.18395 v2 pith:NNJ2GTTU submitted 2026-01-26 cs.CL

Mikel Zubillaga , Oscar Sainz , Oier Lopez de Lacalle , Eneko Agirre This is my paper

classification cs.CL

keywords document-levelinformationextractiontest-timescalingsamplingandselectionrewardmodelF1votingreasoningmodelsrejectiontemplate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that greedy decoding, the standard way to evaluate LLMs on document-level IE, systematically undersells their real capability. It proposes ThinkTwice: generate N candidate templates, then pick one via either unsupervised F1 Voting (average pairwise F1 agreement) or a supervised reward model. With reasoning-oriented LLMs, the gap between the best of N samples and greedy grows, and both selectors recover a large part of it, beating previous systems on three benchmarks. The supervised variant adds a rejection-sampling method to create silver reasoning traces, addressing the lack of human gold traces for this task. If correct, evaluation practice for document-level IE should shift from single greedy outputs to sampling plus selection.

What carries the argument

The central mechanism is 'sample-then-select': the generator produces N structured template candidates under constrained decoding (with reasoning traces for reasoning models), and a selector chooses one. Two selectors are introduced: F1 Voting, which scores each candidate by its average pairwise F1 against the other N-1 candidates, exploiting agreement as a quality signal; and a supervised reward model trained with a Bradley-Terry margin loss on silver preference pairs (chosen vs rejected templates ranked by F1 against gold). A rejection-sampling loop supplies the silver training data: generate N candidate reasoning-template pairs, keep the top K by F1 against gold, fine-tune the model, and

What would settle it

Run the same sampling-and-selection pipeline on a held-out document-level IE dataset with non-overlapping template structures, and measure (1) the oracle gap between greedy and best-of-64, and (2) whether F1 Voting or the reward selector recovers any of it. If the oracle gap is near zero, sampling has nothing to exploit; if the selectors pick no better than a random candidate, the selection mechanism is not finding better templates.

Watch

Extended reading notes

Core claim

The central claim is that the stochastic variability of LLM outputs is an asset, not a nuisance: for a fixed document, sampling 64 candidate templates and selecting one via F1 Voting or a reward model consistently beats the single greedy output, often by several F1 points, and more so for reasoning models. The paper further claims that a reward model trained on automatically generated silver preferences—built from the same rejection-sampling loop—is the strongest selector, surpassing the unsupervised methods and establishing state-of-the-art results in zero-shot, supervised, and cross-lingual settings. The key empirical observation is the widening oracle gap: the best of 64 samples scores fa

Load-bearing premise

The supervised gains rest on the assumption that the silver reasoning traces—generated by the model itself and kept only when the paired template scores well against the gold template—teach genuine document reasoning rather than template formatting or dataset artifacts; the paper reports that only 68-71% of training instances passed a 'gold-quality' filter, with no human evaluation of the traces.

Editorial extensions

If this is right

Greedy decoding under-reports LLM capability on document-level IE; sampling and selection should become the default evaluation mode.
Unsupervised F1 Voting improves over greedy in zero-shot settings and beats majority voting, especially for complex, long-tail templates.
The supervised reward selector is the strongest variant, beating all unsupervised selectors and previously published systems on the English benchmark.
Training the reasoning LLM on silver traces produced by rejection sampling transfers across languages, outperforming reference systems trained on target-language data in several languages.
Reasoning-oriented models benefit more from sampling than non-reasoning models, making them the preferred backbone for document-level IE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The wide gap between the selected output and the oracle best-of-N suggests the bottleneck has shifted from generation to selection; better selectors may yield further gains without improving the generator at all.
F1 Voting's success implies that for structured outputs, agreement under a task-specific metric is a usable quality signal; the same idea could transfer to other structured prediction tasks where exact-match majority voting fails.
If the silver-traces training works as claimed, it offers a general recipe for fine-tuning reasoning models on tasks without gold chain-of-thought: generate, filter by task score, and iterate. The reported trace-quality ceiling (roughly two-thirds of traces judged 'gold-quality') is a clear target for better filters or reinforcement learning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a real, well-scoped first adaptation of test-time scaling to document-level IE, and F1 Voting is a sensible structured-output analogue of majority voting. But the empirical claim is weaker than the abstract suggests. Every sampled result comes from one fixed seed (Appendix D says seed 42), and the paper's own Table 5 shows F1 Voting below greedy for both non-reasoning models—so "consistently outperforming" is doing more work than the data supports.

What's new: the combination of sampling multiple candidate templates, selecting via pairwise F1 agreement, and the rejection-sampling loop that produces silver reasoning traces for fine-tuning. Individually those components exist, but the paper is the first to put them together for DocIE, and the zero-shot gains on BETTER with reasoning models are non-trivial (17.10 vs 14.78 for Llama R1; 20.02 vs 16.12 for Qwen think). The cross-lingual transfer, especially the reward selector, is a nice addition. Code and hyperparameters are provided, and the paper is honest about the oracle gap and about the missing gold reasoning traces.

Soft spots, in rough proportion. The single-seed evaluation is the main one. Without variance estimates or significance tests, a 0.2 F1 difference on Korean or Russian could easily be sampling noise. The deltas that matter—Reward on MUC 42.13 vs 35.97, F1 Voting on BETTER—are large enough that they might survive a seed change, but "consistently" across every language and model is not established. Second, the abstract overclaims: Table 5 contradicts the unqualified "consistently outperforming greedy" for non-reasoning models (Llama 3.3 F1 1.48 vs greedy 3.20). Third, the silver traces: only 67–71% of training instances got "gold-quality" traces, with no human evaluation, so the supervised fine-tuning gains are not clearly attributable to reasoning quality. One minor labeling issue: Table 6 calls Reward "zero-shot" but the selector is trained on English data; it is a supervised selector applied in zero-shot LLM inference.

The circularity worry does not land—reward is trained against gold F1, which is external supervision. The method is coherent and the paper is clearly written.

Who it is for: people working on LLM-based IE and test-time scaling for structured outputs. It deserves a serious referee. If I were the editor, I would send it out, with the request that the authors report multiple seeds (or at least give variance) and fix the abstract.

Referee Report

3 major / 5 minor

Summary. The paper proposes THINKTWICE, an inference-time sampling-and-selection framework for document-level information extraction (DocIE). For each input document, the LLM generates N=64 candidate templates under constrained decoding, and a selector chooses the final template. Two selectors are introduced: an unsupervised F1 Voting method that scores each candidate by its average F1 similarity to the other candidates, and a supervised reward model trained on silver preference data. The supervised pipeline includes a rejection-sampling procedure that generates silver reasoning traces, filters them by F1 against gold templates, and fine-tunes both the reasoning LLM and the reward model. Experiments on MUC-4, MultiMUC, and BETTER Granular report that THINKTWICE consistently outperforms greedy decoding, especially for reasoning models, and achieves state-of-the-art results in zero-shot, supervised, and cross-lingual settings. The paper also makes its code publicly available.

Significance. If the empirical claims hold, the paper makes a useful contribution to LLM-based DocIE: it challenges the default of greedy decoding and adapts test-time scaling ideas to structured extraction. The framework is simple, well-motivated, and the open-source code is a strength. The main evidence, however, is currently a set of single-seed point estimates with no uncertainty quantification; several improvements are modest, and the quality of the silver reasoning traces is not independently verified. These gaps prevent the paper from fully supporting the 'consistently outperforms' claim and the specific attribution of gains to reasoning quality.

major comments (3)

[Section 5.2, Table 1, Appendix D] All reported results are single-seed point estimates. Appendix D states that whenever sampling is used, 'we fix the random seed to 42.' Tables 1, 5, 6, and 7 report no standard deviations, confidence intervals, or significance tests. The observed gains are often modest — e.g., MUC Llama R1 F1 Voting 21.23 vs greedy 18.68; MultiMUC average 13.22 vs 11.46 — so the 'consistently outperforms' claim in the Abstract and Section 5.2 is not yet established. Please run the full pipeline over multiple seeds (e.g., 5) and report means ± std, and perform a paired significance test (e.g., bootstrap or Wilcoxon over documents) for the greedy-vs-selected comparison.
[Section 5.3] The supervised fine-tuning pipeline relies on model-generated silver reasoning traces filtered by F1 against gold templates. The paper reports that only 67.77% (Llama R1) and 71.46% (Qwen 3) of training instances received traces judged 'gold-quality', but no human evaluation or direct analysis of these traces is provided. If the accepted traces predominantly teach the model to follow the JSON template or exploit dataset surface patterns rather than to reason over document evidence, the fine-tuning gains attributed to ThinkTwice may not reflect improved reasoning. Please provide either a human evaluation of a sample of accepted and rejected traces, or an ablation that trains on the same silver data without the reasoning-trace component, to isolate the contribution of the traces.
[Section 3.4 and Section 4.3] The F1 Voting selector uses a 'simplified F1 scorer' for the BETTER dataset, while the final evaluation uses the official BETTER scorer. The manuscript does not quantify how the simplified scorer differs from the official scorer, nor whether the simplified scorer is a good proxy for the official metric. Since selection and evaluation use different scoring functions, the candidate selected by the simplified F1 may not be optimal under the official metric, which could affect the zero-shot BETTER results in Table 1 and Table 5. Please either use the official scorer in selection or report the correlation between simplified F1 and official scores on a development set.

minor comments (5)

[Section 5.2] Typo: 'we sue' should be 'we use'.
[Section 5.3] Typo: 'calib' should be 'calibrate'.
[Table 3] The table uses commas as decimal separators (0,7) while the text uses periods; please make the style consistent.
[Figure 2 caption] The symbol '1⃝' is unclear; use a standard circled number or plain text.
[Table 1 header] The column header 'BETTERA VG' appears to be a formatting artifact; should read 'BETTER Avg'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the selectors are either unsupervised or trained on external gold-derived labels, and the only self-citation is not load-bearing.

full rationale

This is an empirical systems paper rather than a derivation, so the main circularity patterns do not apply. The unsupervised F1 Voting selector (Eq. 1) scores candidates by their average similarity to other sampled candidates only; it never uses gold annotations, so selecting with it and then scoring with the evaluation metric is not a reduction to the input. The supervised reward model (Eqs. 2-4) is trained on pairwise preferences induced by F1 against gold templates; this is ordinary supervised learning with external labels, not a fitted parameter being renamed as a prediction. The rejection-sampling silver-data loop (Section 3.3) filters the model's own outputs by F1 against gold on the training set and then fine-tunes on the survivors; this is expert iteration / self-training with gold supervision, and evaluation is on held-out test annotations, so the gains are not forced by construction. The paper reports oracle upper bounds explicitly and acknowledges that only 67.77% and 71.46% of training instances received gold-quality traces, which is a stated limitation rather than a hidden circular step. The one overlapping-author citation, [Sainz et al., 2024] for guideline-aware prompting, is a component choice shared by all compared configurations and is not load-bearing for the central sampling-vs-greedy claim. The single-seed evaluation concern raised by the skeptic is a statistical-evidence / correctness risk, not a circularity. No quoted equation reduces to its own input, so the honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest mostly on dataset/metric choices and on the quality of automatically generated silver traces; no new physical or mathematical entities are introduced. The main free parameters affect sampling diversity and silver-data filtering, and they are reasonable for an empirical NLP method.

free parameters (6)

sample_count_N = 64
Number of candidate templates generated per document during inference and rejection sampling (Section 4.2). Controls cost and the ceiling shown by the oracle; chosen during development.
top_K_selected = 8 per iteration; final 4 (Qwen3) / 1 (Llama R1)
Number of silver reasoning-template pairs kept per instance for fine-tuning (Sections 3.3, 4.2).
reward_margin_scaling_lambda = 3
Scaling factor for the margin in the Bradley-Terry reward loss (Eq. 3), taken from Touvron et al. (2023) rather than tuned for DocIE.
inference_sampling_hyperparameters = temperature 0.6-0.7, top_p 0.8-1.0, top_k -1/20 per model
Diversity of the 64 samples depends on these; selected from each model's recommended settings (Table 3).
rejection_sampling_iterations = 8 (Llama R1), 14 (Qwen3)
Iterative fine-tuning run until the selection score no longer improved (Section 4.2).
fine_tuning_hyperparameters = LR 5e-5 (LLM) / 2e-5 (reward); epochs 5/4; batch 128/512
Chosen by best checkpoint on the development split (Appendix D).

assumptions (5)

domain assumption CEAF-RME/F1 is a valid proxy for template extraction quality and is used to rank silver candidates against gold annotations.
The whole silver-data and reward-preference pipeline depends on this metric (Sections 3.3, 3.4, 4.3).
domain assumption The simplified F1 scorer used by F1 Voting approximates the official BETTER/MUC scorers.
Used to compute pairwise similarities in Eq. 1; authors state they implemented a simplified F1 scorer for BETTER (Section 4.3).
ad hoc to paper Model-generated silver reasoning traces filtered by gold F1 are adequate training targets for a reasoning LLM.
Load-bearing for supervised ThinkTwice; only 67.77%/71.46% of traces reached 'gold-quality' and no human evaluation is reported (Section 5.3).
standard math Bradley-Terry preference model with margin loss is a valid way to train the reward selector.
Equation 4; standard pairwise ranking objective.
domain assumption Constrained JSON-schema decoding reliably produces parseable templates from all sampled candidates.
If many samples are malformed, selection quality degrades; paper notes constrained decoding is insufficient for non-reasoning models on BETTER (Section 5.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction." pith.science (2026). https://pith.science/paper/NNJ2GTTU

@misc{pith2026260118395,
  author       = {Pith},
  title        = {Pith review of: Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNJ2GTTU}},
  note         = {Machine review of arXiv:2601.18395}
}

read the original abstract

Document-level Information Extraction (DocIE) aims to produce an output template with the entities, relations, and events of interest occurring in the given document. Standard practices include prompting decoder-only LLMs using greedy decoding to avoid output variability. Rather than treating this variability as a limitation, we show that sampling can produce substantially better solutions than greedy decoding, especially when using reasoning models. We thus propose ThinkTwice, a sampling and selection framework in which the LLM generates multiple candidate templates for a given document, and a selection module chooses the most suitable one. We introduce both an unsupervised method that exploits agreement across generated outputs, and a supervised selection method using reward models trained on labeled DocIE data. To address the scarcity of golden reasoning trajectories for DocIE, we propose a rejection-sampling-based method to generate silver training data that pairs output templates with reasoning traces. Our experiments show the validity of unsupervised and supervised ThinkTwice, consistently outperforming greedy baselines and the supervised state-of-the-art.

Figures

Figures reproduced from arXiv: 2601.18395 by the authors.

**Figure 2.** THINKTWICE architecture, with the inference process at the bottom. The supervised option includes two steps: ⃝1 The iterative procedure to generate the silver dataset with trajectories and to fine-tune the reasoning model; ⃝2 Training the selector with silver preference data that include trajectories. In the following sections, we describe the main components of THINKTWICE, our sampling and selection method. We sta… view at source ↗

**Figure 3.** Zero-shot results for the greedy reasoning and non [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Cross-lingual transfer results for THINKTWICE trained on English (MUC-4) and evaluated on MultiMUC. We also report a state-of-the-art system trained on the target language as a reference. older and less capable LLMs. In contrast, performance improves significantly whe…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 1 linked inside Pith

[1]

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324--345, 1952

1952
[2]

Iterative document-level information extraction via imitation learning

Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron White, and Benjamin Van Durme. Iterative document-level information extraction via imitation learning. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 1858--1874, Dubrovnik, Croa...

2023
[3]

Are more LLM calls all you need? towards the scaling properties of compound AI systems

Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AI systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

2024
[4]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

2025
[5]

Rethinking negative instances for generative named entity recognition

Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, Yan Bowen, and Min Zhang. Rethinking negative instances for generative named entity recognition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024 , pages 3461--3475, Bangkok, Thailand, August 2024. Association for Computationa...

2024
[6]

Template filling with generative transformers

Xinya Du, Alexander Rush, and Claire Cardie. Template filling with generative transformers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computation...

2021
[7]

M ulti MUC : Multilingual template filling on MUC -4

William Gantt, Shabnam Behzad, Hannah An, Yunmo Chen, Aaron White, Benjamin Van Durme, and Mahsa Yarmohammadi. M ulti MUC : Multilingual template filling on MUC -4. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 349--3...

2024
[8]

Joseph Gatto, Omar Sharif, Parker Seegmiller, and Sarah M. Preum. Document-level event-argument data augmentation for challenging role types. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 25109-...

2025

Show all 36 references

[9]

The llama 3 herd of models, 2024

Aaron Grattafiori, , and Zhiyu Ma. The llama 3 herd of models, 2024

2024
[10]

M essage U nderstanding C onference- 6: A brief history

Ralph Grishman and Beth Sundheim. M essage U nderstanding C onference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics , 1996

1996
[11]

Twenty-five years of information extraction

Ralph Grishman. Twenty-five years of information extraction. Natural Language Engineering , 25(6):677–692, 2019

2019
[12]

Document-level entity-based extraction as template generation

Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. Document-level entity-based extraction as template generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Proce...

2021
[13]

Openai o1 system card, 2024

OpenAI: Aaron Jaech, , and Zhuohan Li. Openai o1 system card, 2024

2024
[14]

S ci REX : A challenge dataset for document-level information extraction

Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. S ci REX : A challenge dataset for document-level information extraction. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Associat...

2020
[15]

Overview of the tac 2010 knowledge base population track

Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. Overview of the tac 2010 knowledge base population track. In Third text analysis conference (TAC 2010) , volume 3, pages 3--3, 2010

2010
[16]

Document-level event argument extraction by conditional generation

Sha Li, Heng Ji, and Jiawei Han. Document-level event argument extraction by conditional generation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceeding...

2021
[17]

Revisiting large language models as zero-shot relation extractors

Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation extractors. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 6877--6892, Singapore, December 2023. ...

2023
[18]

Let's verify step by step, 2023

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023

2023
[19]

The IARPA BETTER program abstract task four new semantically annotated corpora from IARPA ' s BETTER program

Timothy Mckinnon and Carl Rubino. The IARPA BETTER program abstract task four new semantically annotated corpora from IARPA ' s BETTER program. In Nicoletta Calzolari, Fr \'e d \'e ric B \'e chet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi...

2022
[20]

F ourth M essage U nderstanding C onference ( MUC -4): Proceedings of a Conference Held in M c L ean, V irginia, J une 16-18, 1992 , 1992

1992
[21]

Zero: Memory optimizations toward training trillion parameter models, 2020

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020

2020
[22]

Gollie: Annotation guidelines improve zero-shot information-extraction

Oscar Sainz, Iker Garc\' a-Ferrero, Rodrigo Agerri, Oier Lacalle, German Rigau, and Eneko Agirre. Gollie: Annotation guidelines improve zero-shot information-extraction. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on ...

2024
[23]

Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning

Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations , 2025

2025
[24]

Llama 2: Open foundation and fine-tuned chat models, 2023

Hugo Touvron, , and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023

2023
[25]

Revisiting relation extraction in the era of large language models

Somin Wadhwa, Silvio Amir, and Byron Wallace. Revisiting relation extraction in the era of large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

2023
[26]

Ace 2005 multilingual training corpus

Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia , 57:45, 2006

2005
[27]

GPT - RE : In-context learning for relation extraction using large language models

Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. GPT - RE : In-context learning for relation extraction using large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empiric...

2023
[28]

Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023

2023
[29]

Inference scaling laws: An empirical analysis of compute-optimal inference for llm problem-solving

Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for llm problem-solving. In The Thirteenth International Conference on Learning Representations , 2025

2025
[30]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

2025
[31]

Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017) , pages 35--45, 2017

2017
[32]

In-context learning for few-shot nested named entity recognition, 2024

Meishan Zhang, Bin Wang, Hao Fei, and Min Zhang. In-context learning for few-shot nested named entity recognition, 2024

2024
[33]

A survey on test-time scaling in large language models: What, how, where, and how well?, 2025

Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025

2025
[34]

Qwen3 embedding: Advancing text embedding and reranking through foundation models

Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 , 2025

2025 arXiv
[35]

A survey of generative information extraction

Zikang Zhang, Wangjie You, Tianci Wu, Xinrui Wang, Juntao Li, and Min Zhang. A survey of generative information extraction. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st Internation...

2025
[36]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324--345, 1952

1952

[2] [2]

Iterative document-level information extraction via imitation learning

Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron White, and Benjamin Van Durme. Iterative document-level information extraction via imitation learning. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 1858--1874, Dubrovnik, Croa...

2023

[3] [3]

Are more LLM calls all you need? towards the scaling properties of compound AI systems

Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AI systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

2024

[4] [4]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

2025

[5] [5]

Rethinking negative instances for generative named entity recognition

Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, Yan Bowen, and Min Zhang. Rethinking negative instances for generative named entity recognition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024 , pages 3461--3475, Bangkok, Thailand, August 2024. Association for Computationa...

2024

[6] [6]

Template filling with generative transformers

Xinya Du, Alexander Rush, and Claire Cardie. Template filling with generative transformers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computation...

2021

[7] [7]

M ulti MUC : Multilingual template filling on MUC -4

William Gantt, Shabnam Behzad, Hannah An, Yunmo Chen, Aaron White, Benjamin Van Durme, and Mahsa Yarmohammadi. M ulti MUC : Multilingual template filling on MUC -4. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 349--3...

2024

[8] [8]

Joseph Gatto, Omar Sharif, Parker Seegmiller, and Sarah M. Preum. Document-level event-argument data augmentation for challenging role types. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 25109-...

2025