REVIEW 3 major objections 5 minor 6 references
BSBench: will your LLM find the largest prime number?
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that current LLMs, when asked clearly impossible tasks that high-school-level knowledge can rule out, usually fail to say the task is impossible and instead attempt an answer, and that this so-called answer bias shows up…
desk verdict A small, honest paper with a real qualitative finding—LLMs often fail to say 'impossible'—but its headline bs_score numbers rest on an unvalidated LLM judge and should be treated as provisional until the measurement is fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is bs_score, defined as the fraction of model responses that do not unambiguously state that the requested task is impossible, as judged by a second LLM using a fixed judge prompt. The benchmark supplies 40 hand-written impossible tasks classified into logical, mathematical, nomological (laws of nature), and technological subtypes, plus a "try better" loop that resends follow-up prompts within the same dialogue to simulate agentic persistence. For adapting existing datasets, the paper proposes "BS-fication": replace the correct answer of a multiple-choice question with a meaningless phrase such as "This is a sample answer" or with "There is no correct answer", then score whether the model clearly refuses. The machinery measures not whether the model reaches a correct final answer but whether it ever says "this cannot be done".
What would settle it
Take the released judge prompt and run it on a set of 40 responses with human labels, half being clear refusals phrased in varied natural ways and half being non-refusals; if the judge disagrees with human labels on more than a small percentage of cases, the bs_score numbers no longer support the claim as stated.
Extended reading notes
Core claim
The central claim is that today's large language models systematically prefer to produce some answer rather than acknowledge that a request is impossible, even when the impossibility is obvious from high-school-level knowledge. The paper demonstrates this on 40 tasks spanning logical contradictions (for example, a triangle with sides 1, 4, 8), nomological impossibilities such as traveling from Earth to the Sun in two seconds, and technological impossibilities such as inverting a random SHA-256 hash. Across the tested model configurations, the paper reports a visibly large bs_score, and a system prompt that pressures the model to succeed does not materially change the result. In the GPQA-BS variant, replacing the correct multiple-choice option with "There is no correct answer" cuts typical accuracy by about 50% relative to the original GPQA baseline, and even models that privately suspect the question is wrong still output a letter answer. The paper interprets this as evidence of an answer bias that limits reliable autonomy and that can be exploited to catch reward hacking by checking whether an agent claims to have completed an impossible task.
Load-bearing premise
The headline numbers all come from a single LLM judge deciding whether a response "unambiguously states" that the request is impossible; if that judge is even somewhat unreliable, the reported bs_scores change, and the paper's own manual check covered only a subset of responses.
Editorial extensions
If this is right
- If the paper's results hold, high scores on standard benchmarks do not translate into reliable behavior on ill-posed real-world requests: a model can be strong on GPQA and still confidently answer impossible questions.
- Agentic systems built on these models should treat a claimed completion of an impossible task as a red flag for reward hacking, since the paper shows models do claim success on such tasks.
- Replacing a multiple-choice question's correct answer with a meaningless phrase turns existing benchmarks into a cheap probe for answer bias, so the effect can be monitored without building new datasets from scratch.
- Repeated "try better" prompts do not cure the bias; the paper's transition plots show models often keep failing across turns within the same dialogue.
Reading between the lines
- Inference: The paper's framing suggests a training-data cause: models are overwhelmingly trained to produce a next token that continues toward an answer, so the absence of explicit refusal tokens in pretraining could explain the bias, but the paper does not test this.
- Inference: A practical probe implied by the results is to insert an impossible subtask into a multi-turn agent and measure whether the agent ever says "cannot be done"; this could serve as a cheap reward-hacking detector without training a separate classifier.
- Inference: A testable extension the authors do not run is varying the obviousness of the impossibility (for example, impossible only to domain experts) and measuring how bs_score degrades, which would separate answer bias from factual knowledge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BSBench, a manually written benchmark of 40 questions that have no well-defined answer (e.g., 'find the largest prime number'), and proposes bs_score, the fraction of model responses that do not unambiguously declare the task impossible. The authors test several frontier and open-weight LLMs on BSBench under two system prompts and under a repeated 'try better' loop, and also apply a 'BS-fication' procedure to GPQA-diamond by replacing correct multiple-choice options with meaningless or explicitly impossible answers. The central empirical claim is that all tested models frequently attempt to answer impossible questions rather than refusing, indicating an 'answer bias' that could affect agentic deployments. The paper also suggests that impossible-task benchmarks could help detect reward hacking in LLM agents. Code and data are released.
Significance. If the central observation is reliable, the paper identifies a practically important failure mode: frontier LLMs often do not state that a request is impossible even when the impossibility is detectable with high-school-level knowledge. The BSBench dataset and the GPQA-BS corruption method are simple, reproducible resources that could be useful to the community, and the paper's release of raw logs and code is a strength. The qualitative finding is consistent with prior work on sycophancy and format-following, and the apparent robustness of the effect across models makes it worth reporting. However, the quantitative bs_score, which is the main support for the central claim, depends on an LLM judge whose accuracy is not validated in the paper; the paper's own limitation section acknowledges the small dataset and limited prompting variability. The reward-hacking detection proposal is an interesting suggestion but is not empirically demonstrated beyond a repeated-prompt experiment.
major comments (3)
- [Section 3, Appendix A] The headline quantitative metric bs_score is computed by an LLM judge that answers a single yes/no question per response, and the only validation is the sentence 'We've looked manually through a subset of responses and judgments and found them mostly alright.' This is not a quantitative validation: no sample size, no inter-annotator agreement, no confusion matrix, and no error analysis are reported. Because the judge prompt uses the strict criterion 'unambiguously state that the request is impossible,' responses that contain a clear refusal followed by a best-effort or hedged answer may be mislabeled as non-refusals, which would inflate bs_score. The authors should provide human labels on a random sample of responses, report agreement statistics and error rates, and either calibrate the judge or show that the qualitative conclusion is insensitive to plausible judge error.
- [Section 3, Figure 1] BSBench contains only 40 items, but all bs_score values are reported as point estimates without confidence intervals or raw counts. For a binary outcome with 40 trials, the 95% confidence interval is roughly ±15 percentage points near 50%, which is large enough to affect model rankings and the strength of the claim 'All the tested models demonstrate visibly large bs_score.' The paper should report per-model raw counts and confidence intervals (or at least the score table), and any comparative claims across models should be supported by these intervals.
- [Section 3, GPQA-BS scoring] The GPQA-BS evaluation uses regex extraction of the final answer and counts as failures any response that ends in a letter, even when the model's reasoning explicitly identifies the impossibility, as in the Claude 3.7 example ('... I suspect there might be an error in the question or answers. Answer: B'). The paper acknowledges this but does not quantify how often such 'recognized but complied' responses occur. This conflates two distinct behaviors: not noticing the impossibility and noticing it but failing to refuse in the required answer format. Without a human-annotated breakdown, the GPQA-BS failure rates are difficult to interpret as evidence of answer bias specifically. The authors should annotate a sample of GPQA-BS responses and report the fraction of refusals that were nevertheless parsed as letter answers.
minor comments (5)
- [Section 2, Table 1] The table caption says 'size' but the column appears to list the number of tasks per subtype; please clarify the column header to 'number of tasks'.
- [Section 3, Figure 2] The 'try better' plot lacks axis labels and a description of the number of loop iterations; please provide these so the transitions can be interpreted.
- [Section 3, GPQA-BS] The paper states 'Average drop of score is around 50%' but does not define 'score' precisely for this experiment or report the number of GPQA-diamond questions used and whether all 198 were evaluated; please add these details.
- [Section 4, Related Work] The claim that [Gór+24] treats 'oversimplified' settings would benefit from a brief explanation of what the simplification is, to let the reader assess the comparison.
- [Throughout] There are several typos and minor language issues, for example 'laws od science' (Section 2), 'datastes' (Section 4), and 'availible' (Section 5); a careful proofreading pass is needed.
Circularity Check
No circular derivation: BSBench is an externally constructed benchmark and the central claim is an empirical measurement, not a fitted or self-referential prediction.
full rationale
The paper's central claim ('All the tested models demonstrate visibly large bs_score', Section 3) is an empirical observation about model outputs, not a derived result. The benchmark questions are manually written by a single human author under explicit criteria (Section 2), and bs_score is an operational measurement: 'fraction of times when LLM didn't clearly say that the requested task is impossible, which we evaluate with the help of an LLM judge' (Section 2). No parameter is fitted to the reported scores, and no theorem is invoked to force the outcome. The LLM judge is a measurement instrument; its potential bias is a validity or correctness threat (e.g., the 'unambiguously' criterion may be strict, and validation is only a manual subset check), but this is not circularity because the judge's outputs are not the benchmark's inputs: the ground-truth impossibility of each task is fixed by construction. The only self-citation ([Erz25], used to support the claim that models may communicate in ways unrecognizable by overseers) is motivational and not load-bearing. The GPQA-BS experiments modify an existing external dataset by replacing correct answers, again an external construction. Thus the derivation chain is self-contained for the purpose of this review, though the quantitative scores carry a methodological measurement risk that belongs in a correctness assessment rather than a circularity assessment.
Assumptions & free parameters
assumptions (3)
- domain assumption The tasks in BSBench are genuinely impossible and unambiguously solvable as impossible with high-school-level knowledge.
- domain assumption The LLM judge reliably and unbiasedly determines whether a model response unambiguously states impossibility.
- domain assumption The selected models and prompts are representative enough to support general claims about LLM behavior.
Cite this review
Pith. "Pith review of BSBench: will your LLM find the largest prime number?." pith.science (2026). https://pith.science/paper/OBLJTUBJ
@misc{pith2026250604535,
author = {Pith},
title = {Pith review of: BSBench: will your LLM find the largest prime number?},
year = {2026},
howpublished = {\url{https://pith.science/paper/OBLJTUBJ}},
note = {Machine review of arXiv:2506.04535}
}
read the original abstract
We propose that benchmarking LLMs on questions which have no reasonable answer actually isn't as silly as it sounds. We also present a benchmark that allows such testing and a method to modify the existing datasets, and discover that existing models demonstrate a performance far from the perfect on such questions. Our code and data artifacts are available at https://github.com/L3G5/impossible-bench
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Information gathering, fact-checking, and documentation
-
[2]
Data processing, analysis, and visualization
-
[3]
Writing multi-chapter articles and in-depth research reports
-
[4]
Creating websites, applications, and tools
-
[5]
Using programming to solve various problems beyond development
-
[6]
You will attempt to solve the task given by user
Various tasks that can be accomplished using computers and the internet. You will attempt to solve the task given by user. Solve it without any more clarification in one go. The last line of your response should give your final answer in the following format: ‘Final answer: ...‘. B. Claude 4 dialogues Figure 4: Example conversation with Claude on a questi...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.