{"id":"6c02e5d3-7997-4083-ac4e-09917977139e","arxiv_id":"2601.18395","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Sampling many candidate extractions and selecting by agreement or a learned reward model outperforms greedy decoding for document-level information extraction.","lead":"This paper shows that asking an LLM many times to fill an information-extraction template and then picking the best candidate beats the usual single 'greedy' answer, especially for reasoning models. The authors' ThinkTwice framework gets state-of-the-art scores on document-level extraction benchmarks and across languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-seed evaluation: all 'consistent' gains over greedy rest on one seed (42); no variance/error bars, so the central empirical claim is not yet established.","rationale":"The reader's verdict is CONDITIONAL, and my analysis does not move it: the concern I identify—absence of multi-seed variance estimates—reinforces the need for conditions before full acceptance. I partially agree with the reader's weakest_assumption: silver trace quality is a legitimate secondary risk for the supervised pipeline, but it is not the most load-bearing issue for the paper's central claim. The zero-shot results in Table 1 already support the sampling-and-selection thesis without any fine-tuning, so the trace-quality concern cannot invalidate the core 'don't be greedy' message on its own. By contrast, every comparison in every table depends on a single seed. The paper fixed seed 42 for reproducibility, which is good practice, but reproducibility of one seed is not evidence of robustness across seeds. Since the claimed improvements are small (2–4 F1 points) and the selectors are far from the oracle upper bound, sampling noise could plausibly account for the observed gaps. The concrete test I propose—multi-seed replication with confidence intervals—would settle whether the improvements are consistent rather than seed-specific. If the deltas vanish under other seeds, the headline claim would need substantial qualification; if they persist, the paper's empirical foundation would be much stronger. The reader's rationale already noted 'absence of variance estimates,' so my concern is not new, but I elevate it to the primary load-bearing issue rather than the trace-quality concern. The paper provides code, hyperparameters, and compute details, which is credit, but it does not mitigate the single-seed gap. Therefore I recommend keeping the CONDITIONAL verdict, with an added explicit condition that multi-seed robustness be demonstrated.","tokens_in":15502,"tokens_out":8043,"duration_ms":100081,"concrete_test":"Re-run the zero-shot THINKTWICE (F1-Voting) and greedy baselines on MUC-4, MultiMUC, and BETTER with at least 5 independent seeds (e.g., 42, 123, 2024, 7, 99), keeping all other hyperparameters fixed. Report the mean and 95% bootstrap confidence interval for the per-dataset delta over greedy. If any dataset's CI includes 0, the 'consistently outperforms' claim is unsupported; if all CIs exclude 0, the concern is resolved. For the supervised reward-model results, repeat with at least 2 seeds or report a bootstrap over the 64 sampled candidates per instance to bound selection variability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ThinkTwice sampling+selection consistently outperforms greedy decoding—is an empirical claim, but every reported number comes from a single fixed random seed. Appendix D states: 'When sampling is required, each model is sampled 64 times per prompt. In those cases, to ensure reproducibility, we fix the random seed to 42.' Tables 1, 2, 5, 6, and 7 report only point estimates, with no standard deviations, confidence intervals, or significance tests. The observed deltas are modest: e.g., Table 1 gives MUC Llama-R1 F1-Voting 21.23 vs. greedy 18.68, MultiMUC average 13.22 vs. 11.46, BETTER 17.10 vs. 14.78; Qwen shows similar 2–4 point gains. With one seed, these differences could reflect the particular sample drawn rather than a robust property of the selector. This matters because Figure 1 shows the oracle (best-of-N) rises steeply with N, but the proposed selectors remain far below oracle (e.g., Llama-R1 MUC oracle 42.32 vs. F1-Voting 21.23), so selecting a good candidate is nontrivial and seed sensitivity is plausible. The paper's use of 'consistently' (Abstract, Section 5.2, Section 6) is therefore under-supported. This is not an accusation of cherry-picking; it is a missing-evidence gap in a load-bearing empirical assertion. The reader's identified concern about silver reasoning-trace quality is also valid, but it affects only the supervised fine-tuning component; the zero-shot sampling-and-selection claim is independent of trace quality, whereas the single-seed issue affects all reported comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes THINKTWICE, an inference-time sampling-and-selection framework for document-level information extraction (DocIE). For each input document, the LLM generates N=64 candidate templates under constrained decoding, and a selector chooses the final template. Two selectors are introduced: an unsupervised F1 Voting method that scores each candidate by its average F1 similarity to the other candidates, and a supervised reward model trained on silver preference data. The supervised pipeline includes a rejection-sampling procedure that generates silver reasoning traces, filters them by F1 against gold templates, and fine-tunes both the reasoning LLM and the reward model. Experiments on MUC-4, MultiMUC, and BETTER Granular report that THINKTWICE consistently outperforms greedy decoding, especially for reasoning models, and achieves state-of-the-art results in zero-shot, supervised, and cross-lingual settings. The paper also makes its code publicly available.","tokens_in":15966,"tokens_out":4823,"duration_ms":52863,"significance":"If the empirical claims hold, the paper makes a useful contribution to LLM-based DocIE: it challenges the default of greedy decoding and adapts test-time scaling ideas to structured extraction. The framework is simple, well-motivated, and the open-source code is a strength. The main evidence, however, is currently a set of single-seed point estimates with no uncertainty quantification; several improvements are modest, and the quality of the silver reasoning traces is not independently verified. These gaps prevent the paper from fully supporting the 'consistently outperforms' claim and the specific attribution of gains to reasoning quality.","major_comments":[{"comment":"All reported results are single-seed point estimates. Appendix D states that whenever sampling is used, 'we fix the random seed to 42.' Tables 1, 5, 6, and 7 report no standard deviations, confidence intervals, or significance tests. The observed gains are often modest — e.g., MUC Llama R1 F1 Voting 21.23 vs greedy 18.68; MultiMUC average 13.22 vs 11.46 — so the 'consistently outperforms' claim in the Abstract and Section 5.2 is not yet established. Please run the full pipeline over multiple seeds (e.g., 5) and report means ± std, and perform a paired significance test (e.g., bootstrap or Wilcoxon over documents) for the greedy-vs-selected comparison.","section":"Section 5.2, Table 1, Appendix D"},{"comment":"The supervised fine-tuning pipeline relies on model-generated silver reasoning traces filtered by F1 against gold templates. The paper reports that only 67.77% (Llama R1) and 71.46% (Qwen 3) of training instances received traces judged 'gold-quality', but no human evaluation or direct analysis of these traces is provided. If the accepted traces predominantly teach the model to follow the JSON template or exploit dataset surface patterns rather than to reason over document evidence, the fine-tuning gains attributed to ThinkTwice may not reflect improved reasoning. Please provide either a human evaluation of a sample of accepted and rejected traces, or an ablation that trains on the same silver data without the reasoning-trace component, to isolate the contribution of the traces.","section":"Section 5.3"},{"comment":"The F1 Voting selector uses a 'simplified F1 scorer' for the BETTER dataset, while the final evaluation uses the official BETTER scorer. The manuscript does not quantify how the simplified scorer differs from the official scorer, nor whether the simplified scorer is a good proxy for the official metric. Since selection and evaluation use different scoring functions, the candidate selected by the simplified F1 may not be optimal under the official metric, which could affect the zero-shot BETTER results in Table 1 and Table 5. Please either use the official scorer in selection or report the correlation between simplified F1 and official scores on a development set.","section":"Section 3.4 and Section 4.3"}],"minor_comments":[{"comment":"Typo: 'we sue' should be 'we use'.","section":"Section 5.2"},{"comment":"Typo: 'calib' should be 'calibrate'.","section":"Section 5.3"},{"comment":"The table uses commas as decimal separators (0,7) while the text uses periods; please make the style consistent.","section":"Table 3"},{"comment":"The symbol '1⃝' is unclear; use a standard circled number or plain text.","section":"Figure 2 caption"},{"comment":"The column header 'BETTERA VG' appears to be a formatting artifact; should read 'BETTER Avg'.","section":"Table 1 header"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the core idea is promising. The main barrier is statistical: every number comes from a single seed, which is insufficient for the 'consistently outperforms' claim. The silver-trace quality issue also needs a direct assessment. If the authors supply multi-seed results and significance tests, and provide evidence about trace quality, I would be supportive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real, well-scoped first adaptation of test-time scaling to document-level IE, and F1 Voting is a sensible structured-output analogue of majority voting. But the empirical claim is weaker than the abstract suggests. Every sampled result comes from one fixed seed (Appendix D says seed 42), and the paper's own Table 5 shows F1 Voting below greedy for both non-reasoning models—so \"consistently outperforming\" is doing more work than the data supports.\n\nWhat's new: the combination of sampling multiple candidate templates, selecting via pairwise F1 agreement, and the rejection-sampling loop that produces silver reasoning traces for fine-tuning. Individually those components exist, but the paper is the first to put them together for DocIE, and the zero-shot gains on BETTER with reasoning models are non-trivial (17.10 vs 14.78 for Llama R1; 20.02 vs 16.12 for Qwen think). The cross-lingual transfer, especially the reward selector, is a nice addition. Code and hyperparameters are provided, and the paper is honest about the oracle gap and about the missing gold reasoning traces.\n\nSoft spots, in rough proportion. The single-seed evaluation is the main one. Without variance estimates or significance tests, a 0.2 F1 difference on Korean or Russian could easily be sampling noise. The deltas that matter—Reward on MUC 42.13 vs 35.97, F1 Voting on BETTER—are large enough that they might survive a seed change, but \"consistently\" across every language and model is not established. Second, the abstract overclaims: Table 5 contradicts the unqualified \"consistently outperforming greedy\" for non-reasoning models (Llama 3.3 F1 1.48 vs greedy 3.20). Third, the silver traces: only 67–71% of training instances got \"gold-quality\" traces, with no human evaluation, so the supervised fine-tuning gains are not clearly attributable to reasoning quality. One minor labeling issue: Table 6 calls Reward \"zero-shot\" but the selector is trained on English data; it is a supervised selector applied in zero-shot LLM inference.\n\nThe circularity worry does not land—reward is trained against gold F1, which is external supervision. The method is coherent and the paper is clearly written.\n\nWho it is for: people working on LLM-based IE and test-time scaling for structured outputs. It deserves a serious referee. If I were the editor, I would send it out, with the request that the authors report multiple seeds (or at least give variance) and fix the abstract.","headline":"A genuine first TTS-for-DocIE adaptation with a clean unsupervised selector, but every sampled result rests on one seed and the 'consistent' claim is contradicted by the paper's own non-reasoning tables.","tokens_in":16410,"tokens_out":2983,"would_cite":true,"duration_ms":31393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For document-level information extraction, sampling and selection consistently outperform greedy decoding and set a new state of the art.","keywords":["document-level information extraction","test-time scaling","sampling and selection","reward model","F1 voting","reasoning models","rejection sampling","template extraction"],"falsifier":"Run the same sampling-and-selection pipeline on a held-out document-level IE dataset with non-overlapping template structures, and measure (1) the oracle gap between greedy and best-of-64, and (2) whether F1 Voting or the reward selector recovers any of it. If the oracle gap is near zero, sampling has nothing to exploit; if the selectors pick no better than a random candidate, the selection mechanism is not finding better templates.","tokens_in":15417,"feed_emoji":"🎲","tokens_out":4492,"duration_ms":47454,"temperature":0.7,"pith_summary":"The paper argues that greedy decoding, the standard way to evaluate LLMs on document-level IE, systematically undersells their real capability. It proposes ThinkTwice: generate N candidate templates, then pick one via either unsupervised F1 Voting (average pairwise F1 agreement) or a supervised reward model. With reasoning-oriented LLMs, the gap between the best of N samples and greedy grows, and both selectors recover a large part of it, beating previous systems on three benchmarks. The supervised variant adds a rejection-sampling method to create silver reasoning traces, addressing the lack of human gold traces for this task. If correct, evaluation practice for document-level IE should shift from single greedy outputs to sampling plus selection.","feed_headline":"Sampling beats greedy decoding for document-level extraction","feed_subtitle":"Greedy decoding hides what models can do; sampling and selection recovers it.","key_machinery":"The central mechanism is 'sample-then-select': the generator produces N structured template candidates under constrained decoding (with reasoning traces for reasoning models), and a selector chooses one. Two selectors are introduced: F1 Voting, which scores each candidate by its average pairwise F1 against the other N-1 candidates, exploiting agreement as a quality signal; and a supervised reward model trained with a Bradley-Terry margin loss on silver preference pairs (chosen vs rejected templates ranked by F1 against gold). A rejection-sampling loop supplies the silver training data: generate N candidate reasoning-template pairs, keep the top K by F1 against gold, fine-tune the model, and","core_discovery":"The central claim is that the stochastic variability of LLM outputs is an asset, not a nuisance: for a fixed document, sampling 64 candidate templates and selecting one via F1 Voting or a reward model consistently beats the single greedy output, often by several F1 points, and more so for reasoning models. The paper further claims that a reward model trained on automatically generated silver preferences—built from the same rejection-sampling loop—is the strongest selector, surpassing the unsupervised methods and establishing state-of-the-art results in zero-shot, supervised, and cross-lingual settings. The key empirical observation is the widening oracle gap: the best of 64 samples scores fa","pith_inferences":["The wide gap between the selected output and the oracle best-of-N suggests the bottleneck has shifted from generation to selection; better selectors may yield further gains without improving the generator at all.","F1 Voting's success implies that for structured outputs, agreement under a task-specific metric is a usable quality signal; the same idea could transfer to other structured prediction tasks where exact-match majority voting fails.","If the silver-traces training works as claimed, it offers a general recipe for fine-tuning reasoning models on tasks without gold chain-of-thought: generate, filter by task score, and iterate. The reported trace-quality ceiling (roughly two-thirds of traces judged 'gold-quality') is a clear target for better filters or reinforcement learning."],"forward_implications":["Greedy decoding under-reports LLM capability on document-level IE; sampling and selection should become the default evaluation mode.","Unsupervised F1 Voting improves over greedy in zero-shot settings and beats majority voting, especially for complex, long-tail templates.","The supervised reward selector is the strongest variant, beating all unsupervised selectors and previously published systems on the English benchmark.","Training the reasoning LLM on silver traces produced by rejection sampling transfers across languages, outperforming reference systems trained on target-language data in several languages.","Reasoning-oriented models benefit more from sampling than non-reasoning models, making them the preferred backbone for document-level IE."],"fun_headline_variants":["ThinkTwice: Sampling and selection outperform greedy decoding","Variability is valuable: Sampling improves document extraction","Sample 64 candidates, pick the best: Better than greedy for DocIE","Oracle gap: Sampling 64 times beats greedy for DocIE","Why greedy decoding underperforms sampling in DocIE"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The supervised gains rest on the assumption that the silver reasoning traces—generated by the model itself and kept only when the paired template scores well against the gold template—teach genuine document reasoning rather than template formatting or dataset artifacts; the paper reports that only 68-71% of training instances passed a 'gold-quality' filter, with no human evaluation of the traces.","fun_headline_variants_meta":{"raw":{"variants":["ThinkTwice: Sampling and selection outperform greedy decoding","Variability is valuable: Sampling improves document extraction","Sample 64 candidates, pick the best: Better than greedy for DocIE","Oracle gap: Sampling 64 times beats greedy for DocIE","Why greedy decoding underperforms sampling in DocIE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001147,"raw_usage":{"total_tokens":4565,"prompt_tokens":685,"completion_tokens":3880,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":3798}},"tokens_in":429,"tokens_out":3880,"duration_ms":27438,"temperature":1.0,"reasoning_tokens":3798,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:58:01.399806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same sampling-and-selection pipeline on a held-out document-level IE dataset with non-overlapping template structures, and measure (1) the oracle gap between greedy and best-of-64, and (2) whether F1 Voting or the reward selector recovers any of it. If the oracle gap is near zero, sampling has nothing to exploit; if the selectors pick no better than a random candidate, the selection mechanism is not finding better templates.","supporting_citations":[],"review_version":1}