{"id":"26d36f00-aad1-418b-8425-6016ef68bf12","arxiv_id":"2608.00538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-play dialogue framework with a self-trained questioner improves zero-shot named entity recognition over basic prompting, but not consistently over the strongest existing methods.","lead":"This paper presents DE-NER, a multi-turn dialogue framework in which one language model asks clarifying questions, another answers, and a third produces the final named-entity predictions, all without labeled data. On three benchmarks the trained questioner improves over plain prompting by about 3.75 F1 points, but the gains are smaller against stronger published baselines and the method fails on a smaller model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'outperform competitive baselines' claim contradicts Appendix B: DE-NER loses to ReverseNER, CMAS, and Self-Improving on several benchmarks; +3.75% is only vs Prompt.","rationale":"The reader's stated 'weakest_assumption' concerns the self-referential training signal in Section 3.2 and the Mistral-7B collapse in Table 2. That is a legitimate concern, but it is not the most load-bearing issue for the paper's central claim. The most direct problem is that the abstract claims superiority over 'competitive baselines,' while Appendix B shows DE-NER losing to ReverseNER, CMAS, and Self-Improving on multiple datasets. The reader's rationale does mention the Appendix B losses, but the formal weakest-assumption analysis does not center on this empirical contradiction. Therefore my agreement is partial. The stress-test concern is concrete and internally verifiable: the paper's own numbers contradict the headline. This does not necessarily invalidate the method's mechanism; it makes the paper's central empirical claim unsupported as stated. The appropriate verdict remains CONDITIONAL: the authors can address this by adding a fair same-model comparison with ReverseNER/CMAS in the main table, reporting the +3.75% only against the actual baselines used, and softening the abstract. If the comparison reveals DE-NER is not competitive, the verdict should move to REJECT; if the comparison shows DE-NER is competitive under fair conditions, the claim holds. Since we cannot adjudicate without that rerun, CONDITIONAL is the right call.","tokens_in":10031,"tokens_out":5628,"duration_ms":54244,"concrete_test":"Run a controlled comparison using the same base LLM for all methods. Specifically, use the released code to evaluate DE-NER with GPT-4o-mini as all three roles on CoNLL03 and WikiGold, and run ReverseNER's released code under the same setting and evaluation script. If DE-NER's F1 remains below ReverseNER's reported 77.78 and 78.45, then the abstract's claim of outperforming competitive baselines is empirically false. Additionally, recompute the average improvement against the mean of all baselines listed in Table 7, not just Prompt-based; if DE-NER is not above that mean, the '+3.75%' framing must be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in the abstract and conclusion is that DE-NER 'outperform[s] the competitive baselines in zero-shot settings across multiple benchmarks, with an average improvement of 3.75% F1 points.' Appendix B (Table 7) reports the numbers behind this claim, and they do not support it. On CoNLL03, DE-NER scores 74.35, below ReverseNER's 77.78 and Self-Improving's 74.51. On WikiGold, DE-NER scores 74.97, below ReverseNER's 78.45 and CMAS's 76.23. On GENIA, DE-NER scores 51.70, below Self-Improving's 52.06. The only baseline DE-NER consistently outperforms is the in-house 'Prompt-based' single-turn prompting baseline: 74.35/74.97/51.70 vs. 70.58/71.55/47.65. The claimed '+3.75%' average is exactly the difference between DE-NER's average 67.01 and Prompt-based's 63.26. Thus the abstract's 'competitive baselines' phrasing is misleading: DE-NER is not superior to the published strong baselines, and the main-text Table 1 omits these comparisons. This is not a question of tuning or statistical noise; the reported numbers directly contradict the headline empirical claim. The contribution may still be a modest improvement over single-turn prompting and an untrained dialogue ablation, but that is a much weaker claim than what the paper asserts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DE-NER, a zero-shot named entity recognition framework built around a multi-turn dialogue among three LLM roles: a trainable Questioner, a frozen Roleplayer, and a frozen Responser. The Questioner is trained by self-play on an unlabeled corpus: K stochastic dialogues are run, an entity-level majority-vote pseudo-label is derived, the dialogue trajectory with highest overlap with that pseudo-label is selected, and the Questioner is optimized to reproduce the clarification questions in that trajectory. Experiments on CoNLL03, WikiGold, and GENIA report that DE-NER improves over a single-turn Prompt baseline and over an untrained dialogue Base, with especially large gains on MISC and some biomedical types. The abstract and conclusion further claim that DE-NER outperforms the competitive baselines with an average gain of 3.75% F1 points.","tokens_in":1648,"tokens_out":1949,"duration_ms":79863,"significance":"If the headline claim held, the paper would show that a self-trained questioner can elicit useful clarification from an LLM and improve zero-shot NER without external knowledge or manual demonstrations. The paper has strengths: it releases code, it proposes a concrete dialogue-elicitation mechanism with a trajectory-optimization objective, and it conducts multi-model and multi-dataset experiments. However, the evidence as presented does not support the headline. Appendix B, which the main text does not cite, reports that DE-NER is below ReverseNER on CoNLL03 and WikiGold, below Self-Improving on CoNLL03 and GENIA, and below CMAS on WikiGold. Table 2 further shows a catastrophic drop with a smaller backbone (Mistral-7B). The contribution may reduce to a modest, sometimes noisy, improvement over the in-house Prompt and Base, which is a substantially weaker claim than the one made in the abstract and conclusion.","major_comments":[{"comment":"The central empirical claim is not supported by the paper's own numbers. DE-NER scores 74.35/74.97/51.70 on CoNLL03/WikiGold/GENIA, while Appendix B lists ReverseNER at 77.78/78.45, Self-Improving at 74.51/73.98/52.06, and CMAS at 76.23 on WikiGold. The claimed +3.75% is exactly DE-NER's average (67.01) minus the Prompt baseline's average (63.26); it is not an improvement over the published competitive baselines. Furthermore, the comparison in Table 7 is confounded: ReverseNER uses GPT-4o-mini while the main DE-NER results use GPT-3.5-Turbo, so the appendix table is not an apples-to-apples comparison. The abstract and conclusion must be corrected, and the main text should include same-backbone comparisons with ReverseNER, Self-Improving, and CMAS rather than omitting them from Table 1.","section":"Abstract, §5, and Appendix B (Table 7)"},{"comment":"The scaling-law analysis directly undermines the generality claim. When Mistral-7B-instruct-v0.2 is used as the Roleplayer/Responser, DE-NER collapses to 25.56 F1, far below both Prompt (36.85) and Base (48.59). The Limitations section states that effectiveness on smaller models remains an open question, but Table 2 already provides a measurement and it is negative. The paper needs to explain this failure or explicitly restrict the method's scope to large backbones; otherwise the claimed robustness across model scales is unsupported.","section":"§4.3, Table 2, and Limitations"},{"comment":"The training signal is self-referential and underspecified. Pseudo-labels are obtained by majority voting over the same model's outputs, and the optimal trajectory is selected by maximizing overlap with those pseudo-labels: D*_i = argmax Overlap(y_hat, y_tilde). The unlabeled corpus used for self-training is never identified, and there is no ablation against random trajectory selection, selection by agreement with a different model, or an oracle. Given that the improvements of DE-NER over Base are modest (e.g., 51.70 vs 51.48 on GENIA, 74.97 vs 73.53 on WikiGold), the paper should demonstrate that the trajectory-selection criterion is what drives the gain, rather than the self-agreement itself. At minimum, specify the unlabeled corpus and report the selected-trajectory statistics.","section":"§3.2, Equation for D*_i"}],"minor_comments":[{"comment":"The symbol N is used both for the number of dialogue turns and for one of the GENIA entity types in Table 1, which is confusing. Please disambiguate, e.g., use n_turns for the former.","section":"§3.2 / Table 1 notation"},{"comment":"Typo: 'Tabel 6' should be 'Table 6'. Also, the Roleplayer prompt in Table 5 is extremely short; one sentence describing the role is likely not the exact prompt used, so please include the full prompt.","section":"Appendix A"},{"comment":"The label 'k-shot' is misleading. These are in-context demonstrations sampled from the selected trajectories, not a few-shot training regime. Rename to 'number of demonstrations' and clarify how the demonstrations are formatted.","section":"Table 3"},{"comment":"MIT-Restaurant results are reported but never discussed in the main text, and no conclusion is drawn from them. Either integrate this analysis or remove the table.","section":"Appendix C, Table 8"},{"comment":"The sentence 'LLMs know what LLMs need to accomplish tasks, which could be lagged by manual interferences' is unclear and grammatically awkward. Please rewrite it to state the intended conclusion about static demonstrations.","section":"§4.3"},{"comment":"Standard deviations are reported but no significance tests are given. For the small differences on GENIA and WikiGold (e.g., 51.70 vs 51.48 and 74.97 vs 73.53), a paired bootstrap or similar test would help establish whether the improvement is meaningful.","section":"Several tables"}],"recommendation":"major_revision","confidential_remarks":"The overclaim in the abstract is substantial and is directly contradicted by the paper's own Appendix B. The mixed-backbone comparison in Table 7 makes the situation worse. I would urge the editor to require the authors to provide same-backbone comparisons with ReverseNER, Self-Improving, and CMAS, and to rewrite the abstract and conclusion to match the actual empirical scope. If they cannot produce such comparisons, the paper should be rejected rather than conditionally accepted on a weaker claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you. DE-NER is a dialogue-elicitation framework for zero-shot NER: a trainable Questioner asks clarifying questions of a frozen Roleplayer and a frozen Responser combines the dialogue. The new bit is the training strategy — the Questioner is optimized on dialogue trajectories selected by overlap with the model's own majority-vote pseudo-labels. That is not in the cited CMAS, CrossAgentIE, or Self-Improving papers, so there is real novelty here. The reported gains are modest but real over the in-house baselines: on CoNLL03, DE-NER gets 74.35 vs 72.28 for the untrained Base, and on WikiGold 74.97 vs 73.53. The GENIA improvement is smaller, under 0.3 F1. The paper also includes a reasonable error analysis and released code, which helps reproducibility.\n\nNow the soft spots. The headline claim 'outperform the competitive baselines' is not supported by the paper's own Appendix B. On CoNLL03, DE-NER (74.35) trails ReverseNER (77.78) and Self-Improving (74.51); on WikiGold it trails ReverseNER (78.45) and CMAS (76.23); on GENIA it trails Self-Improving (52.06 vs 51.70). The 3.75% average improvement is computed against the single-turn Prompt, not against competitive baselines. That is an overclaim, and the main table omits these comparisons, which is misleading. Also, the self-training signal is self-referential: trajectories are selected for agreeing with the model's own majority-vote pseudo-labels. The evaluation is held-out, so it is not circular in the fatal sense, but the premise that this reliably identifies effective questions is only as strong as the pseudo-labels, and Table 2 shows it can collapse: Mistral-7B drops from 48.59 (Base) to 25.56 (DE-NER). That is a serious robustness issue that the paper sweeps under 'scaling law' without explanation. And the unlabeled corpus used for self-training is never specified, so the distributional setup is not reproducible.\n\nNone of these are fatal to the underlying idea. The mechanism is interesting, and the paper is honest enough to include the failure case and the comparison table, even if the abstract ignores them. What it needs is a rewrite of the claims, the strong baselines moved into the main table, significance tests or at least error bars on the key comparisons, a specification of the unlabeled data, and some analysis of when the self-training helps versus hurts.\n\nWho should read it: people working on multi-agent or self-training methods for NER and IE. It deserves a serious referee, but with the expectation of major revision.","headline":"New dialogue-elicitation recipe, modest gains, but the abstract overclaims the baselines and the small-model collapse is unexplained.","tokens_in":10903,"tokens_out":1809,"would_cite":false,"duration_ms":17710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DE-NER claims zero-shot NER improves when a learnable questioner asks clarifying questions in self-play dialogue, trained via majority-vote pseudo-label trajectory selection, gaining an average of 3.75 F1 points.","keywords":["zero-shot named entity recognition","large language models","dialogue elicitation","self-play","questioner training","majority-vote pseudo-labels","entity boundary disambiguation","zero-shot information extraction"],"falsifier":"Swap the roleplayer and responser to a mid-sized open model such as Mistral-7B on CoNLL03: DE-NER's F1 falls to 25.56, below the 48.59 untrained-dialogue baseline, which contradicts the unconditional claim that dialogue elicitation improves zero-shot NER.","tokens_in":9921,"feed_emoji":"💬","tokens_out":5503,"duration_ms":50288,"temperature":0.7,"pith_summary":"The paper claims that a zero-shot named entity recognizer can improve itself by asking clarifying questions in a dialogue, with no human annotations, demonstrations, or external knowledge. The setup has three roles: a questioner, a roleplayer, and a responser. The questioner is trained on its own successful dialogue trajectories, selected by overlap with majority-vote pseudo-labels. Across CoNLL03, WikiGold, and GENIA, the trained dialogue model beats a plain prompt and an untrained dialogue baseline by an average of 3.75 F1 points. The gains are largest for ambiguous entity types such as MISC and for biomedical entities.","feed_headline":"Self-questioning lifts zero-shot NER by 3.75 F1 points","feed_subtitle":"A trained questioner draws out clarifying answers, lifting zero-shot NER by about 3.75 F1 points on three benchmarks.","key_machinery":"Dialogue trajectory optimization with entity-level majority voting: run K high-temperature dialogues per sentence, vote entity spans and types across runs to form pseudo-labels, pick the trajectory whose final prediction overlaps most with the pseudo-label, and train the Questioner to imitate that trajectory's questions. This turns unlabeled text into a training signal for asking questions, without external supervision. The three-role self-play — Questioner, Roleplayer, Responser — is the interaction mechanism that carries the argument.","core_discovery":"The central claim is that LLMs' latent NER knowledge is better extracted by multi-turn self-generated clarification than by single-turn prompting or fixed demonstrations. DE-NER is a three-role self-play pipeline: a learnable Questioner examines an initial prediction and asks one clarifying question per turn aimed at type ambiguity, boundary conflict, missed entities, or type errors; a frozen Roleplayer answers only from the input sentence; and a frozen Responser integrates the dialogue into a final prediction. The Questioner is trained without labels by running K stochastic dialogues on an unlabeled corpus, taking entity-level majority voting as pseudo-labels, selecting the trajectory whose","pith_inferences":["The same pseudo-label trajectory selection could apply to other structured extraction tasks, such as relation extraction or event argument extraction, because the selection rule is not NER-specific.","Since gains widen with larger frozen models, the questioner's benefit may track the amount of latent knowledge in the roleplayer and responser; testing questioner size separately would isolate this.","Because pseudo-labels come from the model's own votes, systematic model errors could be reinforced; a non-majority selection rule or an external validator might avoid the collapse seen with Mistral-7B.","The unlabeled self-training corpus is unspecified, so its domain closeness to the target benchmarks is a hidden variable; varying that corpus would reveal how much of the gain is corpus-dependent."],"forward_implications":["Zero-shot NER can be improved by self-questioning rather than prompt engineering or demonstrations, reducing human intervention.","Gains concentrate on ambiguous types such as MISC, DNA, and protein, where boundary and type uncertainty matter most.","The method transfers to held-out domains such as biomedical text and restaurant queries without retraining.","Dialogue elicitation is not model-size agnostic: with Mistral-7B as roleplayer and responser, DE-NER's F1 on CoNLL03 drops to 25.56 versus 48.59 for the untrained dialogue baseline.","Static in-context demonstrations do not help the trained questioner; instance-specific reasoning degrades when fixed examples are added."],"supporting_citations":[{"why":"Provides the chat-based zero-shot information extraction baseline that DE-NER extends and compares against.","marker":"Wei et al., 2023"},{"why":"Shows that teaching models to ask clarifying questions improves reasoning, motivating the trainable questioner.","marker":"Andukuri et al., 2024"},{"why":"Establishes a self-improving zero-shot NER baseline using self-annotated demonstrations and self-consistency, a comparison point and precursor to pseudo-label voting.","marker":"Xie et al., 2024"},{"why":"ReverseNER is a self-generated example-driven zero-shot NER baseline used in comparison.","marker":"Wang et al., 2024"},{"why":"CMAS is a cooperative multi-agent zero-shot NER baseline used in comparison.","marker":"Wang et al., 2025"},{"why":"Supplies the CoNLL03 benchmark used for main evaluation.","marker":"Sang and De Meulder, 2003"},{"why":"Supplies the WikiGold benchmark used for main evaluation.","marker":"Balasuriya et al., 2009"},{"why":"Supplies the GENIA biomedical benchmark used for main evaluation.","marker":"Ohta et al., 2002"}],"fun_headline_variants":["Self-play Q&A sharpens zero-shot NER by 3.75 F1","Multi-turn questioning lifts zero-shot NER 3.75 F1","Dialogue elicitation boosts NER: +3.75 F1 in zero-shot","LLM chat unlocks better zero-shot NER, +3.75 F1","Question-and-answer loop upgrades zero-shot NER by 3.75 F1"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the model's own majority-vote pseudo-labels mark the dialogue trajectories that genuinely improve entity extraction, so the trained questioner transfers to new text; the Mistral-7B collapse shows this premise can fail.","fun_headline_variants_meta":{"raw":{"variants":["Self-play Q&A sharpens zero-shot NER by 3.75 F1","Multi-turn questioning lifts zero-shot NER 3.75 F1","Dialogue elicitation boosts NER: +3.75 F1 in zero-shot","LLM chat unlocks better zero-shot NER, +3.75 F1","Question-and-answer loop upgrades zero-shot NER by 3.75 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1013,"prompt_tokens":649,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":393,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":393,"tokens_out":364,"duration_ms":5108,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:41:16.986793+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the roleplayer and responser to a mid-sized open model such as Mistral-7B on CoNLL03: DE-NER's F1 falls to 25.56, below the 48.59 untrained-dialogue baseline, which contradicts the unconditional claim that dialogue elicitation improves zero-shot NER.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CoNLL03 benchmark used for main evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WikiGold benchmark used for main evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GENIA biomedical benchmark used for main evaluation."}],"review_version":1}