{"id":"b3c99135-5699-4c1a-9d5b-3b9eeeed462f","arxiv_id":"2412.10455","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Geo-LLaVA combines retrieval-augmented fine-tuning with in-context learning, reporting 65.25% and 42.36% on selected subsets of GeoQA and the new GeoMath dataset.","lead":"Geo-LLaVA is a multimodal model that solves plane and solid geometry problems by retrieving similar solved questions and showing them as in-context examples to a fine-tuned LLaVA. The authors release GeoMath, a new dataset of about 10,000 Chinese high-school solid geometry problems with reasoning steps, and report accuracy gains over general multimodal baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is contradicted by the paper's own Table 2: G-llava-13B scores 67.00% on GeoQA+ vs Geo-LLaVA's 65.25%, and 'selected questions' are never defined, so the central claim is either false or unsubstantiated.","rationale":"The paper's stated contribution is a state-of-the-art geometry-solving LMM. The most load-bearing condition for that claim is that the reported 65.25% GeoQA+ and 42.36% GeoMath numbers are directly comparable with prior work on the same test questions. The paper itself violates this condition: Table 2 reports G-llava-13B at 67.00% on GeoQA+, above Geo-LLaVA's 65.25%, while Section 4.1 only gives aggregate test-set sizes and the abstract refers vaguely to 'selected questions.' No test-selection procedure, exact split, or code is provided. Hence either the comparison uses different splits, in which case the SOTA claim is unsubstantiated, or the splits match, in which case the SOTA claim is false. The ICL-leakage concern raised by the reader is real and worth testing, but it is not the primary blocker; a correct controlled comparison is a prerequisite that is missing. This reinforces the REJECT verdict: the empirical central claim cannot be certified from the paper as written.","tokens_in":8601,"tokens_out":3964,"duration_ms":36402,"concrete_test":"Recompute the GeoQA+ row by running G-llava-13B and Geo-LLaVA-13B on the exact same 1,509 test questions used for Table 2. If G-llava reproduces 67.00% while Geo-LLaVA gives 65.25%, the SOTA claim fails; if Geo-LLaVA surpasses 67.00%, release the test split and inference logs so the comparison can be audited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—'state-of-the-art performance of 65.25% and 42.36%'—requires that Geo-LLaVA be compared against prior LMMs on the same test questions. The paper does not satisfy this. Table 2 lists G-llava-13B at 67.00% on GeoQA+, which is higher than Geo-LLaVA's 65.25%; if the comparison is on the same split, the SOTA claim is false, and if the test sets differ, the claim is unsubstantiated because the 'selected questions' are never described and baselines are not re-evaluated on the same subset. No code, model, or test-selection script is released, so the reader cannot determine which case holds. This internal inconsistency is load-bearing independently of the ICL-leakage risk flagged by the reader: even setting leakage aside, the headline result is not supported by the paper's own evidence. The minimum requirement is a controlled comparison on an identical, fully specified test split, with baselines re-run under the same protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Geo-LLaVA, a roughly 13B-parameter large multimodal model for geometry problem solving, built on LLaVA-1.5 with LoRA fine-tuning and a CLIP/BERT dual-tower retrieval network. The authors introduce a new solid-geometry QA dataset called GeoMath, sourced from Chinese high school education websites and machine-translated into English, and combine retrieval-augmented supervised fine-tuning with inference-time in-context learning. The abstract and conclusion claim state-of-the-art results of 65.25% on GeoQA+ and 42.36% on GeoMath on 'selected questions,' with an ablation showing incremental gains from image-context data, QA data, meta-training, and inference-time ICL.","tokens_in":8875,"tokens_out":5639,"duration_ms":58862,"significance":"If the reported numbers were obtained under a controlled evaluation protocol, the paper would make a useful contribution: GeoMath is a new solid-geometry QA resource, and the combination of retrieval and in-context learning for geometry LMMs is timely. The paper also provides an ablation that attempts to isolate the effect of each component, and it compares against several public and proprietary baselines. However, the experimental protocol as reported does not support the headline claims: the state-of-the-art claim is contradicted by the paper's own Table 2, the test-subset selection is never specified, and the inference-time retrieval procedure risks answer leakage from the training set. No code, model, or test-selection script is released, so the contributions cannot be independently validated in their current form.","major_comments":[{"comment":"The abstract claims state-of-the-art performance of 65.25% on GeoQA+, but Table 2 reports G-llava-13B at 67.00% on the same dataset, which is higher than Geo-LLaVA's 65.25%. If the two numbers were obtained on the same test split, the SOTA claim is false; if they were obtained on different splits, the claim is unsubstantiated because the paper never defines what 'selected questions' means, never states the selection criteria, and never re-evaluates G-llava or other baselines on the same subset. The paper must specify the exact test split, re-run all baselines on that split, and release the selection script; without this, the headline comparison is not decidable from the manuscript.","section":"Abstract; §4.3, Table 2"},{"comment":"The retrieval protocol is both internally ambiguous and a source of potential answer leakage. Section 3.4 says K is set to 1 for retrieving the most similar training sample at inference, while Section 4.2.1 says the retrieval model selects the top 5 samples to construct the GeoMath-Meta data; the relationship between these two settings is not explained. More importantly, because the retrieval pool is the same training set used for fine-tuning and the test split is drawn from the same distribution, the nearest training neighbor may be a near-duplicate whose answer the model can copy rather than reason through. The paper does not quantify retrieval similarity, does not hold out the retrieval pool, and does not report a random-retrieval or no-retrieval control that would separate genuine in-context reasoning from memorization. A controlled comparison with the retrieval pool held out, with near-duplicates removed, and with random and oracle retrieval baselines is needed to support the claimed ICL gains.","section":"§3.4; §4.2.1"},{"comment":"The text states that the model was fine-tuned five times to compute the mean and standard deviation of the evaluation metric, but Table 2 reports only point estimates with no standard deviations, confidence intervals, or significance tests. This matters because the central SOTA difference (Geo-LLaVA 65.25 vs. G-llava 67.00) is 1.75 percentage points, and the final ICL ablation gain is only 0.88 percentage points; without variance information, neither the SOTA claim nor the ICL benefit can be distinguished from noise. The authors should report the standard deviations and specify the number of seeds used for each configuration.","section":"§4.3"}],"minor_comments":[{"comment":"The dataset name is inconsistently spelled as 'PSDK9K' and 'PSDK-9K'; it should be PGPS9K to match the reference [33].","section":"§4.1, Table 1"},{"comment":"GeoQA+ is cited as [11] in the introduction and as [9] in the experiments section; the reference numbering should be harmonized.","section":"Introduction; References"},{"comment":"The term 'meta-training' is used loosely: the method concatenates retrieved examples during supervised fine-tuning rather than performing task-level meta-learning across tasks. The authors should either align the terminology with the MetaICL literature and discuss the relationship, or rename the component to avoid overclaiming novelty.","section":"§3.4, §4.2.1"},{"comment":"The left panel contains repeated handwritten-style text 'K Retrieval Questions' and the pipeline labels are difficult to read; a vectorized figure with clearly separated training and inference stages would improve reproducibility of the described protocol.","section":"Figure 2"},{"comment":"The LoRA configuration is not specified: the paper states that LoRA is used but does not report rank, alpha, or which modules are adapted. These details are needed for replication.","section":"§4.2.2"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the unverifiable evaluation protocol: the paper's own Table 2 lists a baseline above the claimed SOTA, and the test subset is not described. The leakage risk in Section 3.4 compounds the problem, since the inference-time retrieval pool is the same training set. A revision would need to redo the evaluation on a fully specified split with baselines re-run under an identical protocol and a leakage-controlled retrieval setup, which is beyond a local fix. I see no path to acceptance without those changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. GeoMath, roughly 10K solid-geometry QA pairs with reasoning steps drawn from Chinese high-school materials, fills a concrete gap in a field that has mostly been plane-geometry-only. The framework is also sensible as an engineering exercise: a CLIP/BERT dual-tower retriever, LoRA fine-tuning of LLaVA-1.5, and retrieval-augmented in-context learning at inference. The ablations show consistent gains from adding IC, QA, and meta-training, which is worth respecting.\n\nBut the central empirical claim does not survive contact with the paper's own Table 2. The abstract says state-of-the-art 65.25% on GeoQA+, yet Table 2 lists G-llava-13B at 67.00% and DPE-NGS at 66.09%. Either the comparison is on different test splits or the claim is simply false; either way it is unsupported as written. The phrase \"selected questions\" is never defined, and baselines are not rerun on those same questions, so the 65.25% is not a controlled comparison.\n\nThe retrieval setup has a more subtle but serious problem. At inference, the model retrieves the single most similar sample from the same training set used for fine-tuning (K=1, Section 3.4). If that nearest neighbor is a near-duplicate of the test question, the model is effectively copying a solution it has already seen. The paper never quantifies the similarity between test questions and their retrieved neighbors and never holds out the retrieval pool. That risks inflating the ICL numbers. Also, the paper says it finetuned five times to get mean and standard deviation, but Table 2 shows single numbers with no variance.\n\nThe citation pattern is fine, and the related work is honest about prior LMMs, though the text's claim that models like G-llava \"lag behind GPT-4V\" is inconsistent with Table 2, where G-llava beats GPT-4V.\n\nBottom line: this is a workshop paper with a useful dataset and a reasonable framework, but the published numbers are not trustworthy in their current form. The authors need to release the data and code, define the test selection, rerun all baselines on the identical split, and either remove the near-duplicate leakage or quantify it. I would not accept the empirical claims as-is, but the dataset contribution justifies sending it to referees under the condition that those issues are fixed. As a reader, I would find the paper valuable mainly as a cautionary example of how retrieval-augmented evaluation can leak training answers.","headline":"The GeoMath dataset is a real gap-filler, but the SOTA claim is contradicted by the paper's own Table 2 and the ICL evaluation leaks from the training set.","tokens_in":9390,"tokens_out":2768,"would_cite":false,"duration_ms":28967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multimodal model given one retrieved solved geometry example before answering attains 65.25 percent on GeoQA+ and 42.36 percent on a new solid-geometry benchmark.","keywords":["geometry problem solving","large multimodal model","in-context learning","retrieval augmentation","solid geometry","GeoMath dataset","meta-training","LoRA fine-tuning"],"falsifier":"Compare each test question with its retrieved neighbor by image overlap and answer text; if a large share of neighbors share the answer or are near-duplicates, the in-context-learning gain is likely leakage. A stronger test would rebuild the retrieval pool excluding every training item similar to any test question above a threshold and rerun the evaluation; if accuracy falls sharply, the benchmark numbers measure memorization rather than problem solving.","tokens_in":8374,"feed_emoji":"📐","tokens_out":6963,"duration_ms":67667,"temperature":0.7,"pith_summary":"This paper proposes that a vision-language model can solve geometry exam problems, both plane and solid, if before answering it is given a retrieved worked example of a similar problem as in-context context. To make that work, the authors build a new dataset, GeoMath, of about ten thousand solid-geometry questions with image descriptions and step-by-step reasoning, and a model, Geo-LLaVA, that is fine-tuned with such retrieved examples during training and given one again at inference. The model reports state-of-the-art accuracy of 65.25% on selected GeoQA+ questions and 42.36% on selected GeoMath questions, compared against general vision-language models and prior geometry-specialised models. The significance, if the numbers hold, is that solid-geometry reasoning becomes tractable for multimodal language models and that retrieval of similar solved problems is a practical way to inject geometric knowledge without a symbolic solver.","feed_headline":"Retrieved solved examples push geometry AI to 65.25 percent","feed_subtitle":"A retrieval step that supplies one similar worked example lifts accuracy on plane and solid geometry benchmarks.","key_machinery":"The mechanism is a two-part retrieval-and-generate loop. A dual-tower retrieval network, trained with a contrastive loss, maps the question text and image into a shared space and returns the single most similar training sample (K=1). That sample's text and image are merged, with images vertically stacked into one image because the backbone accepts a single image, and wrapped in a meta-prompt together with the current question. The backbone is a 13-billion-parameter vision-language model fine-tuned with LoRA on the meta-prompted data; at inference the same retrieval supplies an in-context example before the model generates a solution and answer. The 'meta' component is that retrieval is used both in the fine-tuning stage and at test time, so the model learns from the format of solved examples.","core_discovery":"The central claim is that meta in-context learning, training the model on pairs of a question and a retrieved similar question with its solution, then supplying the same kind of retrieved example at inference, makes a 13-billion-parameter multimodal model competitive or better on geometry QA than much larger general models. Concretely, Geo-LLaVA records 65.25% accuracy on the selected GeoQA+ test questions and 42.36% on the selected GeoMath test questions, with the ablation showing each added component (image-context data, QA data, meta-training, and inference-time ICL) increasing accuracy. The paper further claims this is the first model to handle solid geometry problems, producing picture descriptions and reasoning steps for them.","pith_inferences":["The reported gains could be partly an artifact of retrieval leakage: the top retrieved neighbor is drawn from the same dataset used for fine-tuning, and the test split comes from that dataset's distribution, so a neighbor may be a near-duplicate whose answer is memorized rather than reasoned. The paper does not measure question-to-neighbor similarity or hold the retrieval pool out from the test se","A straightforward test would replace the retrieved neighbor with a random or deliberately dissimilar example; if accuracy stays high, the gain is from training rather than from the inference-time retrieval.","The same retrieval-plus-fine-tuning loop could transfer to other diagram-heavy mathematics, such as trigonometry or vector problems, provided the retrieval pool is cleanly separated from evaluation.","The vertical merging of retrieved and target images into one input may blur spatial relationships; models that accept multiple images separately might show different gains, a comparison the paper does not run."],"forward_implications":["If the reported accuracies hold, geometry problem solving no longer requires a separate symbolic engine; a retrieval-augmented multimodal model can produce answers and readable reasoning steps directly.","Solid geometry, previously almost absent from geometry-QA benchmarks, becomes an addressable task because the new GeoMath dataset provides images, questions, and reasoning steps.","The ablation's stepwise gains imply that image-description data, QA fine-tuning, and meta-training each contribute independently, suggesting a recipe other visual reasoning tasks could copy.","Because the method retrieves from a training pool at inference, it can be updated with new worked examples without retraining the generator, as long as the retrieval index is refreshed."],"supporting_citations":[{"why":"Supplies the multimodal backbone that Geo-LLaVA fine-tunes with LoRA.","marker":"[22]"},{"why":"Provides the meta-training and in-context learning formulation that the paper adapts to geometry.","marker":"[24]"},{"why":"Provides the GeoQA+ plane-geometry benchmark and the DPE-NGS baseline used in the comparison.","marker":"[9]"},{"why":"Provides a geometry-specialised baseline model that Geo-LLaVA is designed to outperform.","marker":"[16]"},{"why":"Provides a large general vision-language baseline whose geometry performance the paper argues Geo-LLaVA surpasses.","marker":"[1]"},{"why":"Supplies the contrastive visual-language pretraining approach used for the dual-tower retrieval network.","marker":"[25]"},{"why":"Supplies the text encoder used in the retrieval network.","marker":"[13]"},{"why":"Supplies the image encoder used in the retrieval network and the multimodal backbone.","marker":"[14]"},{"why":"Provides low-rank adaptation (LoRA), the efficient fine-tuning method applied to the backbone.","marker":"[19]"}],"fun_headline_variants":["Retrieved solved examples push geometry AI to 65.25%","Geometry AI improves to 65.25% via meta in-context learning","First AI to solve solid geometry problems","Retrieval-augmented training boosts geometry AI to 65.25%","Geo-LLaVA: geometry AI with meta-ICL hits 65.25%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy rests on treating the single retrieved training question as a helpful worked example rather than a near-copy of the test question, since the retrieval pool is the same dataset used for fine-tuning and the test split is drawn from that dataset.","fun_headline_variants_meta":{"raw":{"variants":["Retrieved solved examples push geometry AI to 65.25%","Geometry AI improves to 65.25% via meta in-context learning","First AI to solve solid geometry problems","Retrieval-augmented training boosts geometry AI to 65.25%","Geo-LLaVA: geometry AI with meta-ICL hits 65.25%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000955,"raw_usage":{"total_tokens":4065,"prompt_tokens":930,"completion_tokens":3135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3039}},"tokens_in":546,"tokens_out":3135,"duration_ms":23747,"temperature":1.0,"reasoning_tokens":3039,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:56.735970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare each test question with its retrieved neighbor by image overlap and answer text; if a large share of neighbors share the answer or are near-duplicates, the in-context-learning gain is likely leakage. A stronger test would rebuild the retrieval pool excluding every training item similar to any test question above a threshold and rerun the evaluation; if accuracy falls sharply, the benchmark numbers measure memorization rather than problem solving.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multimodal backbone that Geo-LLaVA fine-tunes with LoRA."},{"cited_title":"An augmented benchmark dataset for geometric ques- tion answering through dual parallel text encoding","cited_arxiv_id":null,"evidence_quote":"Provides the GeoQA+ plane-geometry benchmark and the DPE-NGS baseline used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a large general vision-language baseline whose geometry performance the paper argues Geo-LLaVA surpasses."}],"review_version":1}