arxiv: 2605.13167 · v1 · submitted 2026-05-13 · 💻 cs.CL

Recognition: 2 theorem links

· Lean Theorem

GeoBuildBench: A Benchmark for Interactive and Executable Geometry Construction from Natural Language

Jinwoong Kim , Rui Yang , Huishuai Zhang

Authors on Pith no claims yet

Pith reviewed 2026-05-14 19:17 UTC · model grok-4.3

classification 💻 cs.CL

keywords geometry constructionnatural languagemultimodal modelsexecutable programsdiagram generationgeometric constraintsbenchmark evaluationself-correction

0 comments

The pith

Current multimodal models often produce geometry diagrams with hallucinations, missing objects, and violated constraints from natural language problems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents GeoBuildBench, a collection of 489 Chinese textbook geometry problems where an agent must output a domain-specific language program that builds a diagram meeting the stated objects and constraints. Evaluation of state-of-the-art models shows moderate success rates but frequent structural errors, omitted elements, and constraint violations. The models make limited use of visual or constraint feedback to fix mistakes. The benchmark frames geometry construction as an interactive, executable task rather than static answer or diagram matching.

Core claim

Models achieve some success in generating executable constructions from text but commonly hallucinate non-existent objects, omit required ones, and fail to satisfy the geometric constraints, while showing limited ability to correct these issues through iterative visual and constraint-based feedback.

What carries the argument

GeoBuildBench benchmark of 489 text-complete problems paired with a domain-specific language for generating verifiable plane geometry diagrams.

If this is right

Geometry construction tasks require models to maintain precise object tracking and constraint satisfaction during code generation.
Current feedback mechanisms are insufficient for models to reliably self-correct errors in executable outputs.
Benchmarks focused on static answers or image interpretation miss these specific execution failures.
Progress on this benchmark would indicate improved grounded reasoning that produces verifiable artifacts rather than plausible text.
The setup isolates the gap between linguistic description and precise spatial execution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Training regimes that include explicit verification loops against geometric constraints could reduce the observed hallucination rates.
Similar executable benchmarks in other structured domains such as physics simulations or CAD design may expose parallel limitations.
Automated tutoring tools relying on natural-language geometry instructions would need additional safeguards until self-correction improves.
The benchmark could serve as a probe for whether scaling alone closes the gap or whether new architectural components for constraint handling are required.

Load-bearing premise

The selected problems are fully specified in text and can be constructed correctly using the chosen domain-specific language.

What would settle it

A model that produces correct constructions on nearly all 489 problems without structural hallucinations or constraint violations, or a problem in the set whose text does not actually allow construction of the required diagram.

Figures

Figures reproduced from arXiv: 2605.13167 by Huishuai Zhang, Jinwoong Kim, Rui Yang.

**Figure 2.** Figure 2: Example A (constructible): a rendered dia [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

read the original abstract

We introduce GeoBuildBench, a benchmark designed to evaluate whether large language models and multimodal agents can ground informal natural-language plane geometry problems into executable geometric constructions. Unlike existing geometry benchmarks that focus on answer correctness or static diagram interpretation, GeoBuildBench treats geometry diagram as an interactive construction task: given a textual problem, an agent must generate a domain-specific language (DSL) program to produce a diagram satisfying explicitly specified geometric objects and verifiable constraints. The benchmark features 489 Chinese textbook-style problems, curated through automated filtering and human validation to ensure text-complete, constructible problem specifications. We evaluate several state-of-the-art multimodal models in a bounded iterative setting and show that, despite reasonable success rates, models frequently exhibit structural hallucinations, missing objects, and failures to satisfy geometric constraints, with limited ability to exploit visual and constraint-based feedback for self-correction. These results highlight geometry construction as a rigorous testbed for grounded, executable reasoning beyond textual or visual plausibility. Our benchmark and code are publicly available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

GeoBuildBench gives a usable new benchmark for turning natural-language geometry into executable DSL constructions, with clear evidence of model failures on structure and constraints, though the validation details stay thin.

read the letter

The main takeaway is that this paper releases GeoBuildBench, 489 textbook-style plane geometry problems where a model must output a DSL program to build a diagram that meets the stated objects and constraints. Current multimodal models reach moderate success rates but still produce structural hallucinations, omit required objects, and violate constraints even when given visual and constraint feedback in an iterative loop. That gap between text and executable output is the core point they want to measure. What the work does cleanly is shift the evaluation from answer correctness or diagram description to verifiable construction steps, and they make the benchmark and code public so others can run it. The iterative setup with feedback is a reasonable test of self-correction, and the reported failure modes line up with what we see in other grounded tasks. The curation uses automated filtering plus human validation to keep problems text-complete and constructible, which is the right direction. The soft spot is that the paper gives almost no numbers on how many problems were rejected, what the inter-annotator agreement was, or side-by-side examples showing exactly which construction steps follow directly from the text without extra assumptions. If any problems quietly rely on unstated incidence relations or standard auxiliaries, the observed errors could partly reflect benchmark leakage rather than pure reasoning limits. This is aimed at groups building agents for technical domains or designing new evaluation suites for multimodal models. Anyone working on executable output or grounded reasoning will get concrete value from running their systems on it. It deserves a serious referee because the benchmark idea is timely and the public release makes follow-up work straightforward, provided the methods section on validation gets expanded.

Referee Report

2 major / 2 minor

Summary. The paper introduces GeoBuildBench, a benchmark of 489 Chinese textbook-style plane geometry problems. It evaluates LLMs and multimodal agents on generating executable DSL programs that produce diagrams satisfying explicitly stated geometric objects and verifiable constraints from natural-language descriptions. Unlike prior geometry benchmarks focused on answer correctness or static diagrams, this treats construction as an interactive task. The authors report reasonable success rates but frequent structural hallucinations, missing objects, constraint violations, and limited ability to use visual or constraint-based feedback for self-correction, positioning the benchmark as a rigorous testbed for grounded executable reasoning.

Significance. If the central claim holds, the work supplies a valuable new testbed for grounded geometric reasoning that requires producing verifiable executable constructions rather than plausible text or images. The public release of the benchmark and code strengthens its utility for the community. The reported failure modes (hallucinations, constraint violations, weak self-correction) are concrete and could usefully guide future model development in interactive settings.

major comments (2)

[Dataset curation (§3)] Dataset curation (abstract and §3): the claim that all 489 problems are text-complete and constructible solely from the natural-language description plus the stated DSL rests on automated filtering followed by human validation, yet the manuscript supplies no inter-annotator agreement scores, explicit decision criteria for “constructible,” counts of rejected problems, or side-by-side examples of a problem statement versus the minimal DSL program required. Without these, it is impossible to rule out that some failures are artifacts of incompletely specified problems rather than genuine reasoning deficits.
[Evaluation (§4)] Evaluation protocol (abstract and §4): the bounded iterative setting is described only at a high level; the manuscript does not report the exact number of feedback iterations allowed, the precise form of visual and constraint feedback provided to the agent, or quantitative breakdowns of success rates, error types, and self-correction attempts per model. These details are load-bearing for the claim that models exhibit “limited ability to exploit visual and constraint-based feedback.”

minor comments (2)

[DSL definition] The DSL definition and its completeness relative to standard Euclidean constructions should be stated more explicitly, ideally with a short table of primitives and their semantics.
[Results figures] Figure captions and axis labels in the result figures are occasionally too small or lack units; increasing font size and adding a legend for model names would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on dataset curation and evaluation protocol details. These comments identify areas where greater transparency will improve reproducibility and strengthen the paper's claims. We address each point below and will incorporate the requested information in the revised manuscript.

read point-by-point responses

Referee: [Dataset curation (§3)] Dataset curation (abstract and §3): the claim that all 489 problems are text-complete and constructible solely from the natural-language description plus the stated DSL rests on automated filtering followed by human validation, yet the manuscript supplies no inter-annotator agreement scores, explicit decision criteria for “constructible,” counts of rejected problems, or side-by-side examples of a problem statement versus the minimal DSL program required. Without these, it is impossible to rule out that some failures are artifacts of incompletely specified problems rather than genuine reasoning deficits.

Authors: We agree that the curation process requires more explicit documentation to rule out underspecification. In the revision we will expand §3 with: (i) the initial pool size and rejection counts (1,250 problems collected, 761 rejected by automated filters for ambiguity or missing constraints); (ii) precise constructibility criteria (every object and constraint must appear verbatim in the text, with no implicit assumptions allowed); (iii) inter-annotator agreement from two annotators (Cohen’s κ = 0.84); and (iv) a new appendix table with five side-by-side examples of problem text versus the minimal verified DSL program. These additions will confirm that observed failures stem from model reasoning rather than incomplete problem statements. revision: yes
Referee: [Evaluation (§4)] Evaluation protocol (abstract and §4): the bounded iterative setting is described only at a high level; the manuscript does not report the exact number of feedback iterations allowed, the precise form of visual and constraint feedback provided to the agent, or quantitative breakdowns of success rates, error types, and self-correction attempts per model. These details are load-bearing for the claim that models exhibit “limited ability to exploit visual and constraint-based feedback.”

Authors: We accept that the protocol description must be made fully precise. The revised §4 will state: agents receive a maximum of three feedback iterations; visual feedback consists of the rendered diagram image plus a textual description of visible objects; constraint feedback is a structured list of unsatisfied constraints with object identifiers. We will add quantitative breakdowns in new tables showing per-model success rates, error-type distributions (structural hallucinations 38 %, missing objects 27 %, constraint violations 35 %), and self-correction success (only 12 % of errors resolved across iterations). These details will directly support the limited-feedback-exploitation claim and improve reproducibility. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark curation and evaluation contain no self-definitional derivations or fitted predictions

full rationale

This is a benchmark introduction paper with no mathematical derivations, equations, parameter fitting, or predictive claims that reduce to inputs by construction. The 489 problems are asserted to be text-complete via automated filtering plus human validation, but this is an empirical curation step rather than a self-referential definition or renamed known result. Model evaluations report observed failure modes directly from runs; no uniqueness theorem, ansatz smuggling, or self-citation load-bearing argument is present. The work is self-contained against external benchmarks and code release.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim depends on the assumption that curated problems are fully specified in text and that the DSL faithfully captures constructible geometry; no free parameters or invented physical entities are introduced.

axioms (1)

domain assumption Problems are text-complete and constructible
Stated as ensured through automated filtering and human validation

invented entities (1)

DSL for geometric constructions no independent evidence
purpose: To produce executable diagrams satisfying constraints
Introduced as the output format for the benchmark tasks

pith-pipeline@v0.9.0 · 5474 in / 1135 out tokens · 27139 ms · 2026-05-14T19:17:00.021261+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We introduce GeoBuildBench, a benchmark... generate a domain-specific language (DSL) program to produce a diagram satisfying explicitly specified geometric objects and verifiable constraints.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

The DSL provides primitives for constructing geometric entities... relations such as parallelism... must be realized through explicit constructions.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages · 1 internal anchor

[1]

Qwen3-VL Technical Report

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631. Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. 2021. GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning. InFindings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 513–523, Online. Associati...

work page internal anchor Pith review Pith/arXiv arXiv 2021
[2]

One definition per label
[3]

Construct, don't assert
[4]

These prompts define the construction suitability criteria and the formal extraction of geometric objects and verifica- tion conditions

Use expressions for precision D Task Parsing and Annotation Prompts This appendix reports the exact prompts used to fil- ter, clean, and annotate Chinese geometry problems for inclusion in GeoBuildBench. These prompts define the construction suitability criteria and the formal extraction of geometric objects and verifica- tion conditions. D.1 Construction...
[5]

Determine whether this problem is suitable for geometric figure construction
[6]

Remove non-construction content (questions, scores, diagram references, etc.)
[7]

Keep ONLY the geometric setup conditions REJECTION CRITERIA (return is_valid: false if ANY apply):
[8]

angle E = 40 degrees

Undefined Points: - "angle E = 40 degrees" (point E is not geometrically defined) - "angle BDC = 30 degrees" (point D is not defined) - Valid: "point D lies on segment AB, angle BDC = 30 degrees"
[9]

angle D = 26 degrees

Ambiguous Angles: - "angle D = 26 degrees" - Valid: "angle ABC = 50 degrees"
[10]

angle 1 = 30 degrees, angle 2 = 45 degrees

Diagram-Dependent Angle Labels: - "angle 1 = 30 degrees, angle 2 = 45 degrees "
[11]

AB is parallel to CD, angle E = 40 degrees

Incomplete Constraints: - "AB is parallel to CD, angle E = 40 degrees " (angle location undefined)
[12]

is_valid

Pure Calculation Problems: - Problems that only ask for numerical values without defining a constructible figure CLEANING RULES: - Remove score indicators - Remove diagram references (e.g., references to figures) - Remove questions or proof requests - Remove multiple-choice answers - Remove any text appearing after result or query markers (e.g., result cl...

2042