{"id":"5ccade15-2e97-43c3-af49-96f3b551fa55","arxiv_id":"2505.14340","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of plane geometry problem solving that classifies methods into an encoder-decoder framework and analyzes hallucination and data leakage in current benchmarks.","lead":"This paper organizes recent research on teaching AI systems to solve plane geometry problems from diagrams and text into one map. It also identifies where those systems fail, especially by inventing diagram details and by using benchmarks that leak answers.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central taxonomy depends on unverified E/I/D/O assignments; Table A1 appears to classify Visual Sketchpad, an inference-time prompting method, as a learned seq-to-seq decoder, so the survey's map may misrepresent the literature.","rationale":"Good faith reading: the survey is a broad, useful synthesis with a coherent high-level framework, and it makes honest statements about its two-dimensional scope. The reader's ACCEPT is defensible. However, my role is to stress-test the central claim. The claim 'we fill the missing PGPS survey gap' is operationalized by the Table A1 taxonomy, not by any new experiments. That taxonomy's accuracy is therefore load-bearing. I found a concrete reason to doubt it: Visual Sketchpad, an inference-time prompting framework, is assigned the same trained encoder-decoder tuple as supervised fine-tuned VLMs. This is not a matter of taste or disagreement with community consensus; it is an empirical question about what the cited papers contain. The proposed audit would settle it. My verdict is CONDITIONAL rather than REJECT because the taxonomy is plausibly fixable with a verification pass, and the rest of the survey (task definitions, benchmark discussion, limitations) remains useful regardless. I also note the duplicate Table A1 row for E2 I1 D1 O1 as a minor quality signal, not as the core concern.","tokens_in":20231,"tokens_out":4846,"duration_ms":47897,"concrete_test":"Audit Table A1 against the original papers: for each row, check whether the cited method actually contains the claimed encoder, intermediate representation, decoder, and output format, using the paper's architecture diagram and released code when available. Focus first on Visual Sketchpad (Hu et al., 2024b), GOLD (Zhang and Moshfeghi, 2024), FGeo-SSS (Zhang et al., 2024a), GeoDRL (Peng et al., 2023), and DFE-GPS (Zhang et al., 2025e). Define a misassignment as a row whose source describes no component matching the assigned tuple (e.g., a prompting-only method assigned D3, or a natural-language-input method assigned I1). If at least two of the audited rows are misassigned, the taxonomy needs correction before the survey can serve as a reliable map of the field.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central contribution is the encoder-decoder taxonomy in Section 3 and Table A1, which assigns each method one tuple (encoder, intermediate representation, decoder, output format). The taxonomy is not validated against source code or even against the original method descriptions, and at least one assignment looks actively wrong: Visual Sketchpad (Hu et al., 2024b) is listed as E2 I2 D3 O3, i.e., a neural encoder producing an embedding and a sequence-to-sequence decoder. In the source paper it is an inference-time prompting framework that interleaves VLM calls with image-sketch editing; it has no trained decoder explicitly generating output from an intermediate embedding. Forcing it into the same slot as Math-LLaVA and MAVIS glosses over a qualitatively different mechanism. The same worry applies more broadly: end-to-end VLMs do not expose a separable decoder, so labeling their hidden states as I2 is an interpretive choice, not an architectural fact. If several Table A1 rows misclassify methods, the structured overview promised in the abstract loses evidential value, and the central claim that the paper fills the missing-survey gap is weakened. The limitations paragraph is honest, but it does not repair the taxonomy's factual basis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys the emerging literature on plane geometry problem solving (PGPS), a multimodal reasoning task in which models must combine diagram and text inputs to produce answers or proofs. The authors organize the field around an encoder–decoder framework: methods are decomposed into an encoder that produces either a formal-language description or an embedding vector, and a decoder that emits a theorem sequence, a logic program, or a natural-language description. On this basis they review rule-based and neural encoders, formal-language and embedding-based decoders, and the associated benchmarks and datasets. The final sections discuss two cross-cutting challenges: hallucination in diagram perception, illustrated with a small GPT-4.1 probe using the authors' GeoDANO engine, and benchmark quality issues such as data leakage and weak diagram–text interdependence. The paper positions itself as the first dedicated, up-to-date survey centered specifically on PGPS.","tokens_in":20489,"tokens_out":2433,"duration_ms":25439,"significance":"If its taxonomy is reliable, the paper provides a genuinely useful organizing device for a fragmented and fast-moving literature. The encoder–decoder language gives practitioners a compact notation for comparing methods, and the appendix table listing each method's E/I/D/O combination is a practical reference. The discussion of hallucination and data leakage is timely and concrete, including an original small error analysis (GPT-4.1 at 59% on GeoDANO-sampled same-side questions) and a useful comparison of benchmark comprehensiveness in Table A3. The paper also makes an honest scope statement in its Limitations section, restricting itself to two-dimensional geometry. The main weakness is that the central taxonomy is asserted rather than validated: several Table A1 assignments are at least debatable, and at least one appears to be factually wrong, which reduces confidence in the survey's core contribution as a map of the literature.","major_comments":[{"comment":"The classification of Visual Sketchpad (Hu et al., 2024b) as E2 I2 D3 O3 is not supported by the cited paper. In its source, Visual Sketchpad is an inference-time prompting framework that interleaves calls to a frozen multimodal language model with editing of a visual sketch; it does not train or contain a sequence-to-sequence decoder that maps an intermediate embedding to a natural-language output. Forcing it into the same E/I/D/O slot as Math-LLaVA and MAVIS misrepresents a qualitatively different mechanism. Because Table A1 is the paper's central organizing contribution, this misclassification is load-bearing: it undermines the claim that the taxonomy faithfully summarizes the surveyed methods. The authors should either reclassify Visual Sketchpad (for example, as a prompting method outside the learned-encoder/decoder scheme, with an explicit caveat in Table A1) or, more generally, validate each Table A1 assignment against the original method description and state the criteria for assigning I2 to end-to-end models.","section":"Table A1, Section 3"},{"comment":"The sentence 'UniGeo (Chen et al., 2022) is currently the only benchmark designed explicitly to systematically measure reasoning capabilities' is an absolute claim that is not justified by any systematic search or comparison. The paper itself later discusses benchmarks that evaluate reasoning steps or process-level correctness (e.g., Jaiswal et al., 2024; Zhang et al., 2025a; Sun et al., 2024 appear in the references), and other geometry reasoning benchmarks exist in the broader literature. As written, this claim overstates the uniqueness of UniGeo and weakens the completeness of the task overview. The authors should either qualify the claim to refer to a specific definition of 'reasoning benchmark' or provide a systematic comparison of benchmarks that include reasoning-step evaluation.","section":"Section 2.2.2 (Reasoning tasks)"},{"comment":"The taxonomy assigns every method a single (encoder, intermediate representation, decoder, output) tuple, but the paper does not establish that this decomposition is jointly exhaustive or that the four components are separable for all surveyed methods. In particular, end-to-end vision-language models such as GeoDANO and MAVIS do not expose a distinct encoder and decoder with a well-defined intermediate embedding; labeling their hidden representations as I2 is an interpretive choice rather than an architectural fact. The authors should state explicitly whether the taxonomy is intended as a functional decomposition or an architectural description, and add a caveat that some assignments, especially for end-to-end models, are approximate. Without that caveat, the table gives a false impression of precision and the survey's central claim loses support.","section":"Section 3.1, Table A1"}],"minor_comments":[{"comment":"There is a missing period between 'utilized across different methods' and 'Through the analysis'; please correct this punctuation error.","section":"Section 7"},{"comment":"The table uses symbols E1, E2, I1, I2, D1, D2, D3, O1, O2, O3 without a legend in the caption; readers must infer their meaning from Figure 3. Please add a one-line explanation in the Table A1 caption or refer explicitly to Figure 3.","section":"Table A1"},{"comment":"The table uses '⃝' and '×' symbols without a legend; please define them (e.g., '⃝ = satisfies the criterion, × = does not satisfy the criterion') in the caption or text.","section":"Table A3"},{"comment":"The phrase 'L(V)LM-assisted annotation' is used with varying capitalization (L(V)LM, LLMs, VLMs); please standardize the terminology throughout the appendix.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a competent survey with a useful framework, and the deficiencies identified in my major comments are fixable: reclassifying or explicitly qualifying the Visual Sketchpad row, qualifying the UniGeo uniqueness claim, and adding a caveat about the interpretive nature of the E/I/D/O decomposition for end-to-end models. The use of the authors' own GeoDANO engine in the GPT-4.1 probe is a mild self-citation but not a correctness issue; it is disclosed transparently. The paper fits the scope of the journal as a survey paper. I would be comfortable with acceptance after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The survey does something real: it gives the PGPS community a shared vocabulary. The encoder-decoder framing with intermediate and output representations is a reasonable organizing device, and Table A1 plus Table A3 are genuinely useful reference points. The limitations section is honest about the 2D scope, and the paper doesn't oversell itself. If you work in or near this area, this saves you the literature sweep.\n\nThe main soft spot is exactly what the stress-test note flags. Table A1 lists Visual Sketchpad as E2 I2 D3 O3, but that method is an inference-time prompting framework that interleaves VLM calls with sketch editing — it has no trained decoder generating output from an intermediate embedding. Forcing it into the same slot as MAVIS and Math-LLaVA glosses over a qualitatively different mechanism. The same worry generalizes: end-to-end VLMs don't expose a separable decoder, so labeling their hidden states as I2 is an interpretive choice, not an architectural fact. That doesn't sink the survey, but a paper whose central contribution is the taxonomy needs to either validate the assignments against the source papers or add a caveat that the E/I/D/O slots are approximate. The UniGeo claim in Section 2.2.2 — that it is currently the only dedicated reasoning benchmark — also reads as asserted without a systematic search; it may be true, but it needs a softer qualifier or a citation to a broader benchmark review.\n\nThe GPT-4.1 hallucination probe is small (100 synthetic problems, one model) and the paper wisely presents it as illustrative rather than as a measurement. It's fine as motivation, not a contribution.\n\nOverall: the framework holds up as a descriptive map, and the tables are the kind of thing people will cite. The flaws are fixable. I'd send it to peer review with a request that the authors audit Table A1 row by row and either correct the Visual Sketchpad entry, add a column for mechanism (trained vs. prompting), or add an explicit disclaimer that the decomposition is architectural rather than mechanistic. That's a revision, not a rejection.","headline":"A useful survey map of a niche field, with a few taxonomy entries that need checking before it goes out.","tokens_in":572,"tokens_out":1358,"would_cite":true,"duration_ms":30544,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that plane geometry problem solving can be organized into a single encoder-decoder taxonomy, and that its two central obstacles are diagram hallucination and benchmark data leakage.","keywords":["plane geometry problem solving","multi-modal reasoning","vision-language models","encoder-decoder taxonomy","diagram understanding","benchmark analysis","hallucination","data leakage"],"falsifier":"Find one published PGPS system that genuinely straddles the grid — for example, a system that produces both a formal-language description and an embedding vector for the same problem instance, or that switches between logic-program and natural-language outputs depending on input difficulty — and the survey's claim that each method occupies a single E/I/D/O combination is shown to be incomplete. A concrete check is to take the 29 methods listed in Table A1, reimplement or inspect the released code of a random sample, and see whether any method's actual inference-time pipeline behaves differently from its assigned tuple.","tokens_in":20058,"feed_emoji":"📐","tokens_out":7619,"duration_ms":63136,"temperature":0.7,"pith_summary":"This survey argues that automated plane geometry problem solving (PGPS) is now a distinct research area large enough to deserve its own review, and that the right way to see the field is as an encoder-decoder pipeline. The encoder turns the diagram-plus-text input into either a formal-language description or an embedding vector; the decoder converts that representation into a theorem sequence, a logic program, or a natural-language answer. The paper assigns every surveyed method a single combination of these four choices, and uses that grid to show where the field is stuck: models misread diagrams (hallucination during encoding) and many benchmarks do not actually force multi-modal reasoning (data leakage and text-only-solvable problems). A reader accepts the survey if the taxonomy is faithful enough to organize the literature without distorting individual methods.","feed_headline":"One taxonomy maps every plane-geometry solver","feed_subtitle":"Two failure modes emerge: hallucination in diagram reading and answer leakage in benchmarks.","key_machinery":"The central object is the E/I/D/O tuple, the four-coordinate classification illustrated in Figure 3, with E the encoder, I the intermediate representation, D the decoder, and O the output representation. The intermediate representation carries most of the explanatory weight: formal-language descriptions make the system's reading of the diagram explicit and open to symbolic theorem proving, while embedding vectors make the perception step opaque but allow end-to-end training. The taxonomy's power is comparative — it pins each method to exactly one E/I/D/O combination in Table A1, which is what turns a list of systems into a map of design trade-offs and failure locations.","core_discovery":"The paper's central claim is that the entire PGPS literature fits a modular encoder-decoder architecture, and that the decisive design choice is the intermediate representation between encoder and decoder. In this view, a PGPS system is fully described by four coordinates: encoder type (rule-based or neural), intermediate representation (formal-language description composed of an entity set and a predicate set, versus an embedding vector), decoder type (rule-based axiomatic search, graph-based, or sequence-to-sequence), and output format (theorem sequence, logic program, or natural-language description). The survey maintains that this four-coordinate grid is complete for the methods it reviews, that it lets the field compare systems on equal terms, and that it localizes the open problems: perception errors happen in the encoding stage, while evaluation errors come from benchmarks that are text-solvable, leak data by reusing diagrams, or rely on synthetic diagrams that miss real-world clutter.","pith_inferences":["Editorial inference: if diagram hallucination is the dominant failure mode, then spending capacity on perception — e.g., fine-tuning encoders on synthetic geometry diagrams with domain adaptation — should yield larger accuracy gains than scaling the reasoning model, a hypothesis that could be tested by ablating the encoder while holding the decoder fixed.","Editorial inference: the E/I/D/O grid suggests a diagnostic protocol: for any PGPS system, deliberately corrupt the diagram (or the text) and measure where the error enters; the survey's framework predicts that embedding-based systems will fail silently at perception while formal-language systems will fail by producing malformed or inconsistent predicates.","Editorial inference: the leakage critique extends to a practical recipe for benchmark construction — generate synthetic diagrams that mimic real-world styles (markers, clutter, labeled points), ensure each problem is unsolvable from text alone by construction, and withhold diagram templates across train/test splits; this would satisfy all three criteria the survey lists as simultaneously unmet."],"forward_implications":["If the taxonomy is faithful, new PGPS systems can be described and compared by their E/I/D/O tuple, making architecture choices explicit and reproducible.","The paper's error analysis implies that improving diagram perception — through visual prompting, segmentation masks, or auxiliary-line drawing — should reduce the dominant error type in current vision-language PGPS systems.","The benchmark analysis implies that no current benchmark simultaneously has realistic diagrams, no data leakage, and genuine diagram-text interdependence, so reported accuracy on existing benchmarks likely overstates true multi-modal reasoning.","Because L(V)LM-annotated datasets contain hallucinated geometry relations, models trained on those datasets inherit the errors; auditing and cleaning such datasets is a prerequisite for reliable progress.","The survey's synthesis implies that the field's immediate bottlenecks are perceptual (reading diagrams accurately) and evaluative (building leakage-free, interdependence-aware benchmarks), not the reasoning decoder itself."],"supporting_citations":[{"why":"Defines the direct-answer task and the first rule-based diagram-plus-text parser (GEOS) that later systems build on.","marker":"(Seo et al., 2015)"},{"why":"Introduces G-Aligner, the early diagram-to-text alignment method that anchors the rule-based encoder lineage.","marker":"(Seo et al., 2014)"},{"why":"InterGPS supplies the formal-language plus symbolic-search paradigm and the widely reused text parser that the survey treats as the reference formal pipeline.","marker":"(Lu et al., 2021)"},{"why":"GeoQA provides the dominant direct-answer benchmark and the logic-program output format, defining the numerical-reasoning setting.","marker":"(Chen et al., 2021)"},{"why":"UniGeo is the only benchmark explicitly built for reasoning-step construction, anchoring the theorem-sequence task.","marker":"(Chen et al., 2022)"},{"why":"MathVerse supplies the diagram-text interdependence evaluation and the error analysis connecting incorrect answers to perception failures.","marker":"(Zhang et al., 2025a)"},{"why":"MAVIS exemplifies the synthetic-data-engine plus L(V)LM annotation pipeline whose hallucinations motivate the survey's data-quality warnings.","marker":"(Zhang et al., 2025b)"},{"why":"GeomVerse is the synthetic benchmark without data leakage that the survey uses as a contrast in its comprehensivity analysis.","marker":"(Kazemi et al., 2024)"}],"fun_headline_variants":["Survey puts every plane geometry solver on one grid","Plane geometry AI: one taxonomy, two failure modes","New survey: plane geometry solvers fit one architecture","Plane geometry: one framework for all, with two pitfalls","Survey: plane geometry reasoning fits encoder-decoder grid"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's organizing power rests on the assumption that every reviewed method can be assigned exactly one encoder type, one intermediate representation, one decoder type, and one output format without distortion, even though the assignments were made by reading papers rather than running the code.","fun_headline_variants_meta":{"raw":{"variants":["Survey puts every plane geometry solver on one grid","Plane geometry AI: one taxonomy, two failure modes","New survey: plane geometry solvers fit one architecture","Plane geometry: one framework for all, with two pitfalls","Survey: plane geometry reasoning fits encoder-decoder grid"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1187,"prompt_tokens":851,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":258}},"tokens_in":467,"tokens_out":336,"duration_ms":3619,"temperature":1.0,"reasoning_tokens":258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:39.151431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find one published PGPS system that genuinely straddles the grid — for example, a system that produces both a formal-language description and an embedding vector for the same problem instance, or that switches between logic-program and natural-language outputs depending on input difficulty — and the survey's claim that each method occupies a single E/I/D/O combination is shown to be incomplete. A concrete check is to take the 29 methods listed in Table A1, reimplement or inspect the released code of a random sample, and see whether any method's actual inference-time pipeline behaves differently from its assigned tuple.","supporting_citations":[],"review_version":1}