{"id":"2852b20e-ee20-4f56-8052-287842392816","arxiv_id":"2506.08120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs often choose NO_RELATION in closed-option relation extraction even when their reasoning names a better relation, causing silent information loss.","lead":"This paper introduces a named behavior, conservative bias, where LLMs answering multiple-choice relation questions default to no_relation even when their reasoning identifies a better option. The study matters for relation extraction systems because a safe default can silently discard real relationships.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'twice as often as hallucination' headline is not supported because CBR (Eq. 1) is conditional on Hobson's choice while HR (Eq. 2) is a per-task rate; Table 1 compares incompatible denominators.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that posture. My primary concern differs slightly from the reader's weakest assumption: rather than focusing on chain-of-thought faithfulness, I focus on the denominator mismatch between CBR and HR. This is a direct, internal metric-consistency problem that can be checked from the paper's own equations and table, and it specifically targets the headline 'twice as often' claim. The paper does real work — multiple models, two datasets, three prompt tiers, and a semantic-similarity validation — which means the underlying phenomenon is plausible and may be recoverable. However, the headline comparison cannot be verified from Table 1 as reported, and a reconstruction of the intended unconditional rate yields dataset-dependent results that often contradict the abstract. This is not an external-validity dispute or a disagreement with consensus; it is a formal issue in the definition and interpretation of the paper's own metrics. The concrete test above would settle whether the 2x claim survives a common-denominator recomputation. Because the issue is fixable by reanalysis and the core qualitative finding may survive, the reader's CONDITIONAL verdict remains appropriate; I would not accept the paper as-is, but I would not reject the research direction.","tokens_in":9479,"tokens_out":6397,"duration_ms":76818,"concrete_test":"Recompute every row of Table 1 using raw counts: set the unconditional conservative-bias rate to CBR × HCR (equivalently N_CB / N_total) and compare it to HR from the same row, using the paper's stated aggregation rule if one is provided (the paper does not state one). If the 'twice as often' claim fails to hold under a common denominator for a majority of model/dataset/temperature configurations, the abstract's magnitude claim is unsupported and should be revised to a conditional-rate statement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing quantitative claim is that 'conservative bias occurs twice as often as hallucination.' This comparison is not supported by the paper's own metric definitions. Eq. (1) defines CBR = N_CB / N_HC, i.e., a rate conditional on the model having already defaulted to a Hobson's choice (NO_RELATION or a suboptimal option). Eq. (2) defines HR = N_H / N_total, an unconditional per-task rate. Table 1 reports both as percentages and Section 4.1.1 reads them as directly comparable, but they are not on the same denominator. The unconditional CB rate would be CBR × HCR. Using the table's own values, GPT-4 on TACRED at temperature 0.2 has CBR 7.99% and HCR 1.23%, so an unconditional CB rate near 0.10%, versus an HR of 15.47% — the opposite of 'twice as often.' GPT-4 on REFinD at 0.2 gives CBR 1.14% × HCR 57.72% ≈ 0.66%, versus HR 0.04%, a ratio of about 16x, not 2x. Thus the abstract's single quantitative headline is not derivable from the measurements, and the direction of the claimed effect reverses across datasets. The existence of a conservative-bias behavior may still be recoverable, but its stated magnitude is currently an artifact of mixing conditional and unconditional rates.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines Conservative Bias (CB) in LLM relation extraction as the tendency to default to NO_RELATION or a suboptimal option while the chain of thought suggests a more appropriate relation. It introduces the notion of Hobson's choice and four metrics (HCR, CBR, HR, NRR), and evaluates GPT-4, Llama3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 on REFinD and TACRED under constrained, semi-constrained, and open-ended prompts. The paper claims that conservative bias occurs twice as often as hallucination and that CB-flagged suggestions can be used to expand relation labels. The main evidence is Table 1 and the semantic similarity results in Table 2 and Appendix A.1.","tokens_in":9751,"tokens_out":9774,"duration_ms":107071,"significance":"The paper identifies a plausible and under-explored failure mode: closed-option relation extraction may discard relations that the model can articulate in its reasoning. If the central claims were established, the results would matter for how relation extraction is prompted and evaluated, and the proposed HCR/CBR/HR/NRR framework would be a useful starting point. The experimental design is systematic in covering three prompt types, two datasets, three models, and two temperature settings, and Appendix A.3 provides concrete prompt templates. However, the headline magnitude of the effect is not supported by the metrics as defined, and the detection of CB relies on the model's own reasoning and same-family outputs rather than on ground truth. The contribution is therefore promising but not yet demonstrated.","major_comments":[{"comment":"The headline 'conservative bias occurs twice as often as hallucination' is not supported by the reported metrics because CBR and HR are not on a common denominator. CBR = N_CB/N_HC is conditional on the model having already made a Hobson's choice, whereas HR = N_H/N_total is an unconditional per-task rate. Using Table 1's own values, GPT-4 on TACRED at temperature 0.2 has CBR 7.99% and HCR 1.23%, so the unconditional CB rate is roughly 7.99% × 1.23% ≈ 0.10%, against an HR of 15.47%: the opposite of 'twice as often.' On REFinD at temperature 0.2 the unconditional rate is 1.14% × 57.72% ≈ 0.66% against HR 0.04%, a ratio of about 16, not 2. The correct unconditional CB rate, CBR × HCR, should be reported, or the comparison should be reframed as a conditional one; the abstract and §4.1.1 should be revised accordingly.","section":"§3.2, Eqs. (1)-(2); §4.1.1, Table 1"},{"comment":"The evaluation is restricted to the subset of instances whose gold label is NO_RELATION/OTHER (45% of REFinD, 79.5% of TACRED). On that subset the gold label asserts that no relation exists, so the model's chain-of-thought claim that 'a valid relation exists' cannot be checked against ground truth; under the dataset's own labeling, the suggested relation is unsupported. CB detection therefore rests entirely on the model's self-report, and the reported N_CB may count reasoning-time hallucinations as conservative bias. To support the claim of information loss, the experiment should include instances with positive gold relations that are absent from the option set, or validate a sample of CB-flagged suggestions against human judgments.","section":"§4 (Data); §3 (CB conditions)"},{"comment":"The semantic validation compares CB-flagged constrained outputs only with the same model family's semi-constrained and open-ended outputs, scored by SBERT or a GPT-4 semantic-similarity prompt. This establishes self-consistency across prompt variants, not correctness: all three prompts can be wrong in the same way. The paper's Limitations section explicitly acknowledges the absence of human evaluation. Please add human evaluation of a sample of CB-flagged instances (both the suggested relation and the final label), or compare the CB suggestions against an external relation-extraction gold standard, and report the precision of the CB detector.","section":"§4.1.2, Table A.3; §8"},{"comment":"The definition of N_HC is internally inconsistent. The prose says HCR captures defaults to 'NO_RELATION (or suboptimal option)' and condition (iii) of CB includes selecting 'the least incorrect (suboptimal) option,' but Eq. (1) defines N_HC as 'Number of times the model selects NO_RELATION as the least incorrect option.' If suboptimal non-NO_RELATION choices are counted in N_HC, then CBR's denominator is not simply the NO_RELATION count and the metric's interpretation changes; if they are not, then the prose overstates HCR. Please define N_HC unambiguously and align the prose, the condition list, and the equation.","section":"§3.2, Eq. (1)"}],"minor_comments":[{"comment":"The 'quadrupling (4x)' statement for semi-constrained REFinD does not hold for temperature 0.5: 40.68/7.27 ≈ 5.6.","section":"§4.1.1"},{"comment":"The similarity threshold of 0.7 is said to align with Okazaki and Tsujii (2010), but that paper addresses approximate dictionary matching rather than semantic similarity; please replace the citation or justify the threshold empirically.","section":"§4.1.2"},{"comment":"Labeling run-to-run Cohen's kappa as 'inter-annotator agreement' is misleading; these are repeated model runs, not independent annotators.","section":"Table A.4"},{"comment":"The 'µ' column heading is undefined in the caption; it appears to be the mean similarity but should be stated explicitly.","section":"Table 2"},{"comment":"The paper says multiple iterations were run per temperature, but Table 1 reports only a single percentage per condition; please state the number of iterations and report the variance or confidence intervals for the headline rates.","section":"§4.1.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"No conflict of interest. The paper's broad novelty claim that conservative bias 'has not been explored' is worth a careful prior-art check against work on abstention, selective prediction, and model refusal behavior in relation extraction; this is not the basis for my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper if you work on relation extraction with LLMs; the core observation is worth taking seriously. What is actually new here is the operationalization: the paper names conservative bias, defines HCR, CBR, HR, and NRR, and compares constrained, semi-constrained, and open-ended prompts. That three-tier design is a genuinely useful way to expose the default-to-NO_RELATION tendency. The REFinD results give a concrete instance: GPT-4 shows low hallucination but high conservatism, and the paper is honest enough to call its findings initial benchmarks rather than definitive measurements.\n\nThe soft spot is load-bearing. The abstract's 'twice as often' claim does not survive contact with the paper's own equations. CBR in Eq. 1 is conditional on N_HC, while HR in Eq. 2 is a per-task rate. Table 1 reads them as comparable, but they are not. Recomputing: GPT-4 on TACRED at 0.2 has CBR 7.99% and HCR 1.23%, so the unconditional CB rate is about 0.10%, against an HR of 15.47%. On REFinD at 0.2 the ratio runs the other way, about 16x. The direction of the effect flips depending on what you condition on. The existence of the behavior is not in dispute, but the stated magnitude is an artifact.\n\nThe validation loop is also self-referential. CB detection depends on the model's chain-of-thought being faithful, and the semantic similarity check compares the model's own constrained outputs against its semi-constrained or open-ended outputs. That is consistency, not correctness. Human labels or an external benchmark would break the loop. The 0.7 threshold is fine, but the paper leans on it more than it should. The limitations section mentions the need for human evaluation, which is good, but that need is not a minor caveat; it is central to whether the quantitative claim holds.\n\nFor all that, I would send this to review. The phenomenon is plausible, the prompt design is reusable, and the flaws are fixable. A revision that recomputes unconditional rates and adds a small human-annotated set could turn this into a solid empirical paper. As it stands, treat the headline with suspicion but the direction as worth pursuing. I'd bring it to a reading group to talk about denominators.","headline":"The phenomenon is real and the three-tier prompt design is a useful contribution, but the headline 'twice as often as hallucination' doesn't survive contact with the paper's own metrics.","tokens_in":10317,"tokens_out":2839,"would_cite":false,"duration_ms":31620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In relation extraction, LLMs default to 'no relation' about twice as often as they hallucinate.","keywords":["conservative bias","relation extraction","LLM hallucination","Hobson's choice","NO_RELATION default","semantic similarity","chain-of-thought reasoning","prompt design"],"falsifier":"A direct check: take a labeled set and remove the true relation from the option list, then see whether the model's reasoning names the true relation before the model outputs NO_RELATION. The conservative-bias claim predicts this happens far above chance; if the named relation matches the gold label about as often as a random relation from the options would, the measured information loss is an artifact of self-consistent but incorrect reasoning.","tokens_in":9248,"feed_emoji":"🤐","tokens_out":6666,"duration_ms":78383,"temperature":0.7,"pith_summary":"The paper argues that a large language model's main failure mode in relation extraction is not making things up. When the true relation is missing from the fixed option list, the model usually plays safe: it outputs NO_RELATION or the least bad option, even though its own step-by-step reasoning names the relation it would prefer. The authors call this conservative bias and model it as a Hobson's choice, where every available option is wrong but the model must choose one. Their headline finding is that conservative bias occurs about twice as often as hallucination, with the ratio varying by model, dataset, and temperature. If correct, this matters because closed-option extraction silently discards information the model can express, and some of that lost signal may be recoverable from reasoning.","feed_headline":"LLMs pick 'no relation' twice as often as they hallucinate","feed_subtitle":"Closed-option extraction can silently lose relations the model itself names in its chain of thought.","key_machinery":"The load-bearing object is the pair of counting definitions built around the Hobson's choice: HCR measures how often the model defaults to NO_RELATION or a suboptimal option, and CBR measures how often, among those defaults, the model's reasoning nevertheless names a more suitable relation. Detection requires three prompt tiers: constrained prompts give a fixed option list, semi-constrained prompts allow proposing a novel relation, and open-ended prompts impose no list at all. The step-by-step instruction exposes the reasoning that conservative bias is defined on, and the semantic-similarity comparison with semi-constrained and open-ended outputs is what validates that the reasoning-suggested relation is a real lost alternative rather than noise.","core_discovery":"The paper's central claim is that LLMs show a systematic, measurable conservative bias in relation extraction: when the correct relation is not among the provided options, the model recognizes a valid relation in its reasoning, is aware that the option is unavailable, and nevertheless concludes with NO_RELATION (or the least incorrect option) rather than risk an unsupported label. The authors formalize this with the Conservative Bias Rate, $CBR = N_{CB}/N_{HC}$, the share of Hobson's-choice defaults in which the reasoning names a better relation, and the Hallucination Rate, $HR = N_H/N_{total}$. They observe an inverse relationship between CBR and HR: GPT-4 shows very low hallucination on constrained prompts (0.02--0.04% on one financial dataset) alongside elevated conservative bias, while a smaller open model shows the opposite pattern. To show the lost labels are real, they compare reasoning-suggested labels against the same model's semi-constrained and open-ended outputs using SBERT and a GPT-4 semantic-similarity prompt, finding 54--62% of conservative-bias labels on the financial dataset exceed the 0.7 similarity threshold. The conclusion is that conservative bias is a distinct failure mode, more frequent than hallucination in several settings, and potentially recoverable.","pith_inferences":["A testable extension the paper does not run: the same conservative bias may appear in any closed-option classification with an incomplete option set, so the mechanism is probably not specific to relation extraction.","If the reasoning is faithful, adding an explicit 'none of these, but here is my best guess' option to constrained prompts should recover much of the lost signal; this is a directly testable prompt variant the authors only gesture at.","The paper attributes the behavior to alignment, but that attribution is speculative; comparing instruction-tuned and base checkpoints of the same model on identical prompts would separate alignment-induced caution from a more general uncertainty behavior."],"forward_implications":["Relation extraction pipelines that score only constrained final labels will under-report relations the model can articulate, so recall estimates will be too pessimistic.","Suppressing hallucination in an extraction system will, under this account, push answers toward NO_RELATION; the two goals trade off rather than align.","Reasoning text becomes a recoverable signal: labels suggested in chain-of-thought and validated against semi-constrained or open outputs can expand or refine relation schemas.","Evaluation reports for LLM relation extraction should include a conservative-bias rate alongside hallucination rate, since both are failure modes with opposite costs."],"supporting_citations":[{"why":"Supplies the REFinD financial dataset, its 22 relation types, and the relation extraction instances the constrained, semi-constrained, and open-ended prompt comparisons run on.","marker":"Kaur et al., 2023"},{"why":"Supplies the TACRED dataset, the large newswire-based relation extraction benchmark used as the second testbed for measuring conservative bias.","marker":"Zhang et al., 2017"},{"why":"Provides the prior work on mitigating entity-level hallucination in relation extraction that motivates the paper's inverse trade-off between hallucination and conservative bias.","marker":"Su et al., 2024"},{"why":"Supplies the step-by-step verification style of prompting that the paper uses to expose the reasoning in which conservative bias is detected.","marker":"Lightman et al., 2023"},{"why":"Supplies the 0.7 semantic similarity threshold used to decide when a reasoning-suggested relation matches the relation produced by semi-constrained or open-ended prompts.","marker":"Okazaki and Tsujii, 2010"},{"why":"Supplies the Cohen's Kappa interpretation used to report inter-annotator agreement across multiple model runs at fixed temperature.","marker":"McHugh, 2012"}],"fun_headline_variants":["Conservative bias: LLMs default to 'no relation' at double hallucination rate","LLMs choose 'no relation' over hallucination twice as often","Hobson's choice: LLMs take safe 'none' over a risky relation","LLMs' conservative bias: real relations lost to avoid hallucination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured conservative-bias rate rests on trusting that the chain-of-thought reasoning the model prints truly reflects what relation it recognizes, so if the reasoning is plausible but unfaithful, a 'recognized valid relation' may not actually exist and the claimed information loss is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Conservative bias: LLMs default to 'no relation' at double hallucination rate","LLMs choose 'no relation' over hallucination twice as often","Hobson's choice: LLMs take safe 'none' over a risky relation","LLMs' conservative bias: real relations lost to avoid hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1636,"prompt_tokens":931,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":547,"tokens_out":705,"duration_ms":8134,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:19:25.235813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check: take a labeled set and remove the true relation from the option list, then see whether the model's reasoning names the true relation before the model outputs NO_RELATION. The conservative-bias claim predicts this happens far above chance; if the named relation matches the gold label about as often as a random relation from the options would, the measured information loss is an artifact of self-consistent but incorrect reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TACRED dataset, the large newswire-based relation extraction benchmark used as the second testbed for measuring conservative bias."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 0.7 semantic similarity threshold used to decide when a reasoning-suggested relation matches the relation produced by semi-constrained or open-ended prompts."}],"review_version":1}