{"id":"189171af-7933-48dc-aead-a74d5c15b840","arxiv_id":"2607.18767","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Cropping scanned exam pages to a question's bounding box improved handwritten-answer grading accuracy and cut token/compute cost across eight vision-language models.","lead":"This paper tests whether cropping a scanned handwritten exam to the question's bounding box improves small vision-language model grading accuracy and lowers compute. On 126 Australian Physics Olympiad responses, cropping helped or tied 15 of 16 model-condition comparisons and cut token counts by roughly half.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 accuracy averages cannot be reproduced from Table 2, undermining the reported significance of the central claim.","rationale":"The reader's weakest assumption about bounding-box placement is plausible, but it is not the most decisive issue for the central claim. The larger problem is that the paper's headline aggregate statistics — the numbers used for its significance claim — are internally inconsistent with the per-model results in Table 2. This is a correctness risk that directly affects whether the main claim is supported. My own computation from Table 2 gives different row means than Table 3, and the reported SEMs are too small for the stated across-model aggregation. A paired comparison of the Table 2 values may still show a significant benefit, so I would not reject the paper outright; the appropriate verdict is CONDITIONAL, requiring the authors to supply corrected aggregate statistics or raw data and rerun the significance test. The reader's concern about fixed crops affecting transferability remains secondary and does not change this conclusion.","tokens_in":5986,"tokens_out":13613,"duration_ms":269723,"concrete_test":"Recompute each row of Table 3 directly from the per-model accuracies in Table 2 (equal weight per model) and recompute the paired difference between bounding-box and no-bounding-box conditions per model. If the corrected means and SEMs match Table 3, the inconsistency is typographical and the claim can be re-evaluated. If they do not match, the authors must provide raw per-response accuracies or corrected tables, and the 99.8% significance statement must be re-derived before the central claim is accepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3's supporting table is internally inconsistent. Averaging the eight model accuracies in Table 2 for each condition gives: CoT+BB 0.796, CoT+noBB 0.709, no-CoT+BB 0.818, no-CoT+noBB 0.708 — while Table 3 reports 0.817, 0.734, 0.840, 0.728. These differences (~0.02) exceed the reported standard errors. Moreover, the SEMs in Table 3 (~0.019) are far smaller than the between-model variability visible in Table 2 (model accuracies range roughly 0.49–0.94; SD ≈ 0.14, SEM ≈ 0.05), so the stated uncertainty does not match the stated aggregation. The paper's significance statement — the '3.08 times the quadrature' comparison — is computed from Table 3. If Table 3 is not derivable from Table 2, the central quantitative support for 'bounding boxes significantly improve grading accuracy' is unreliable. This is more load-bearing than the bounding-box coverage assumption: even if every crop contained the answer, the reported significance could be an artifact of incorrect aggregation or transcription.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of whether cropping scanned handwritten exam pages to a question-specific bounding box improves the accuracy and computational efficiency of small vision-language models on a binary short-answer grading task. Using 126 responses to Question 1a of the 2025 Australian Physics Olympiad, the author evaluates eight open-weight VLMs (4B–72B active parameters) under four conditions defined by the combination of chain-of-thought prompting (CoT vs. no-CoT) and image input (full A3 double-page spread vs. bounding-box crop). Accuracy is measured against human marker agreement, and FLOPs are estimated using the standard 2nN approximation on measured token counts. The paper reports that bounding boxes improve or tie accuracy in 15 of 16 model-condition comparisons and reduce token counts and FLOPs by roughly half, and it claims 'significantly improves grading accuracy' on the basis of a statistical comparison involving Table 3.","tokens_in":6279,"tokens_out":9310,"duration_ms":84644,"significance":"If the central claim holds, the paper offers a simple, actionable preprocessing step for VLM-based grading of handwritten exams: crop the page to the region containing the target response, thereby reducing visual clutter and input tokens. This could make local, privacy-preserving SLM deployment more feasible in educational settings. The study has several strengths: it uses real anonymized student exam data, external human labels with no discrepancies, multiple open-weight models, and measured input/output token counts with a transparent FLOPs approximation. The computational-cost reduction is robust and not dependent on the disputed statistical test. However, the quantitative evidence for the 'significantly improves accuracy' claim is undermined by an internal inconsistency between Table 2 and Table 3, and the significance test is based on a post-hoc extremal contrast. These issues are fixable but must be addressed before the central claim can be considered established.","major_comments":[{"comment":"Table 3 cannot be derived from Table 2. Averaging the eight model accuracies in Table 2 gives CoT+BB=0.796, CoT+no-BB=0.709, no-CoT+BB=0.818, no-CoT+no-BB=0.708, whereas Table 3 reports 0.817, 0.734, 0.840, 0.728. The discrepancies are about 0.02–0.025, which exceed the standard errors reported in Table 3 (0.019–0.024). Moreover, the SEMs in Table 3 (~0.02) are far smaller than the between-model SEM computed from Table 2 (e.g., CoT+BB values range 0.55–0.94, SD≈0.14, SEM≈0.05). Because the paper's significance statement (the '3.08 times the quadrature' comparison) is computed from Table 3, the quantitative support for the accuracy claim is currently unreliable. The authors must report exactly how Table 3 was aggregated, correct it using the per-model data, and base the significance test on the actual trial-level data or on an appropriate model-based analysis.","section":"§3, Tables 2 and 3"},{"comment":"The significance test uses a post-hoc extremal contrast: it compares the lowest bounding-box mean (0.817) with the highest no-bounding-box mean (0.734) from Table 3, i.e., it selects the most favorable contrast after inspecting the data. Even setting aside the Table 3 inconsistency, this ignores the pairing of conditions within each model and the large between-model variability visible in Table 2. A proper test should compare the marginal BB vs. no-BB means, or fit a mixed-effects model with model as a random effect and CoT/BB as fixed effects, so that the repeated measures on the same eight models are accounted for. The current statistic treats the two group means as independent and provides no justification for the 99.8% confidence claim.","section":"§3, significance statement"},{"comment":"The bounding box is a single fixed crop chosen once by the researchers from visual inspection of the scanned exams, and no per-page verification is reported that the crop contains the student's answer to Q1a in all 126 papers. If any response fell outside the crop, the model would see only a region without the answer and would likely output 0, biasing the BB accuracy downward. While this bias is conservative for the BB-vs-no-BB comparison, the absence of verification leaves an unresolved threat to internal validity, and the transferability of a single hand-picked box to other exam layouts is unexamined. Please report any failed crops or a verification protocol, and discuss how the fixed-box approach would generalize.","section":"§2.1, Bounding Boxes"},{"comment":"The text says '126 of the 588 student exam papers were chosen randomly as the dataset for each of our four experimental conditions.' It is unclear whether the same 126 papers were used in all four conditions or whether each condition used an independently sampled 126. If the latter, student-level difficulty is not controlled across conditions, and model-level comparisons between conditions are not matched. This ambiguity also affects which statistical test is appropriate. Please clarify the sampling design.","section":"§2.1, dataset description"}],"minor_comments":[{"comment":"The CoT and no-CoT prompts describe the page as containing 'a question about snails,' whereas the target question is about human walking (footnote 2 explains this). This wording is likely to confuse models, especially in the cropped condition where the snail diagram is absent; consider using a prompt that matches the actual question content.","section":"§2.1, prompts"},{"comment":"The figure caption describes error bars as 'maximum and minimum values of the Wilson Score interval at the Z=3 (p=99.7%) significance level.' A Wilson interval is at a confidence level, not a significance level; state the confidence level explicitly (e.g., 99.7%) and clarify what 'maximum and minimum values' refers to.","section":"Figure 2"},{"comment":"The text says all evaluated models are 'small enough to run on upper-end consumer-grade hardware,' but this is questionable for Qwen2.5-VL 72B and Llama 4 Scout (109B total parameters). Consider revising to avoid overstating hardware accessibility.","section":"§2.2"},{"comment":"The phrase '3.08 times the squared quadrature of the standard errors' is ambiguous and likely meant '3.08 times the quadrature (root-sum-square) of the standard errors.' Reword for clarity.","section":"§3"},{"comment":"There are minor typographical issues, e.g., 'using of CoT prompting' in Section 2, and the run-together 'PromptingTo investigate' in Section 2.1. A careful proofread is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central idea is simple, clearly presented, and the compute-cost reduction is convincingly demonstrated. The main obstacle is the irreproducible Table 3 and the ad-hoc significance test; these are fixable within the manuscript's scope by recomputing the aggregates and using a proper statistical analysis (e.g., mixed-effects modeling or a paired test based on the eight models). I would be willing to review a revised version. The paper would also be strengthened by releasing the per-model, per-condition response data or a small aggregation script to make Table 3 verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: useful empirical note with a plausible practical result, but its central significance claim rests on Table 3, and Table 3 does not match Table 2. That has to be fixed before I'd trust the headline.\n\nWhat's actually new: eight open-weight VLMs (4B–72B) on 126 real handwritten physics responses, four conditions crossing chain-of-thought with bounding-box cropping. The cropping direction wins almost everywhere: 15 of 16 model-condition pairs improved or tied, and average token counts fell by more than half, so the FLOPs saving is solid. No fitted parameters enter the comparison; the FLOPs formula is standard; the labels are human marks. For practitioners deploying SLMs locally, this is exactly the kind of cheap preprocessing evidence they need. It is not a mechanism paper, and the paper doesn't claim it is.\n\nSoft spots, in order. First, Table 3 is not derivable from Table 2. Averaging the eight model accuracies in Table 2 gives 0.796 (CoT+BB), 0.709 (CoT+noBB), 0.818 (noCoT+BB), 0.708 (noCoT+noBB). Table 3 reports 0.817, 0.734, 0.840, 0.728. The SEMs in Table 3 are also far smaller than the between-model spread visible in Table 2. The paper's \"3.08 times the quadrature\" significance statement is computed from Table 3, so the main quantitative support for \"significantly improves accuracy\" is unreliable until this is reconciled. This is likely a transcription or aggregation error, but it is load-bearing.\n\nSecond, only one question is used, with a self-selected consent sample, and the bounding boxes were chosen once by inspection without per-page verification that every answer fell inside. The crops were generous, so this is plausible; it's still a claim about generalizing to other layouts that isn't tested.\n\nThird, no code or data is released, which is unfortunate for a dataset that appears to be consent-based and could be shared.\n\nWho this is for: AIED practitioners working with local SLMs and scanned handwritten work. A workshop or short-paper venue is the right level. It deserves a serious referee, but the referee should ask for the Table 2/3 reconciliation and a re-run of the significance test before publication. As it stands, the direction of the effect is probably right, but the stated confidence level is not supported.","headline":"Useful empirical preprocessing note, but the significance claim rests on a table I can't reproduce from the paper's own model-level data; fix that before relying on it.","tokens_in":6694,"tokens_out":3671,"would_cite":false,"duration_ms":29984,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cropping handwritten exam answers to a bounding box improves grading accuracy and cuts compute for small vision-language models.","keywords":["small language models","vision-language models","automated grading","bounding boxes","image cropping","handwritten exams","chain-of-thought prompting","computational efficiency"],"falsifier":"Re-run the eight models on the same 126 scans with bounding boxes generated independently per page (for example, by locating the handwriting), and check whether the fixed boxes ever exclude a written answer. If a nontrivial number of answers fall outside the fixed crops, or if accuracy with per-page crops does not beat full-page accuracy, the paper's central claim fails.","tokens_in":5892,"feed_emoji":"✂️","tokens_out":4583,"duration_ms":42456,"temperature":0.7,"pith_summary":"This paper asks whether a simple preprocessing step — cropping a scanned handwritten exam page to just the box containing the target answer — makes small vision-language models better graders. Across 126 student responses to one physics short-answer question and eight open-weights models from 4B to 72B parameters, every model graded more accurately when shown the cropped region instead of the full double-page spread. The crop also roughly halved the input tokens and cut estimated inference FLOPs from about 41–45 trillion to 17–18 trillion. The authors take this as evidence that removing visual clutter before the model sees the image is a crucial step for making locally deployable small models practical for large-scale educational grading.","feed_headline":"Cropping exam scans lifts accuracy and halves compute","feed_subtitle":"In a 126-exam test, cropping to the answer region beat full-page input for every model and cut FLOPs by more than half.","key_machinery":"The load-bearing mechanism is the bounding-box crop: a rectangle, chosen once by visual inspection of the exam layout, that isolates the pixels containing the target question's answer. It operates on two fronts: it shrinks the image so the vision encoder produces fewer patches and tokens, and it removes unrelated visual content that competes for the model's attention. The paper also tracks chain-of-thought prompting as a second experimental lever, but the evidence suggests the crop, not the prompt, is what moves accuracy. The cost model used to quantify the savings is the standard approximation C ≈ 2nN, where n is total tokens and N active parameters, which translates token reduction directl","core_discovery":"The paper's central claim is that presenting a vision-language model with a fixed bounding-box crop of the answer region, rather than the entire scanned page, improves grading accuracy and lowers computational cost across all tested models. In the cleanest comparison (no chain-of-thought prompting), average accuracy rose from 0.728 on full pages to 0.840 on crops, while average tokens fell from 1463 to 679 and estimated inference FLOPs fell from 41 to 17 trillion. The authors attribute the gain to two mechanisms: the crop removes visual distractions (other questions, diagrams, scribbles) that confuse small models, and fewer image patches means fewer visual tokens, which directly cuts compute","pith_inferences":["Not tested in the paper: whether the fixed crop would hold up on exams with varied layouts or multi-part questions whose answers move; an adaptive region proposal (for example, a layout model or handwriting locator) would be the natural next test.","The token savings imply that, for a fixed compute budget, cropping effectively lets a school run a larger local model at the same cost — the two interventions trade off directly.","Since the crop was chosen by human inspection of the same dataset, part of the gain may be layout-specific; applying the same method to a different exam would show how much of the benefit survives.","The CoT findings suggest that for low-reasoning binary grading, prompting for step-by-step reasoning mostly wastes tokens; in constructed-response grading that requires actual reasoning, CoT might interact differently with cropping."],"forward_implications":["For short-answer grading of fixed-layout paper exams, cropping to the answer region is a reliable accuracy boost and a token/compute saver, so exam systems should treat layout-aware cropping as a default preprocessing step.","Because the benefit held across all eight models from 4B to 72B parameters, the result transfers across architectures and sizes rather than being a quirk of one model.","The larger models reached roughly the same accuracy with crops as without, meaning cropping lets smaller, cheaper local models approach the accuracy of much larger cloud-hosted models.","Chain-of-thought prompting did not consistently help and sometimes hurt; for simple binary grading tasks, the added tokens buy no accuracy, so simpler prompting is preferable.","Reduced FLOPs per graded response lower energy use and cost, strengthening the case for on-premises grading of sensitive student data."],"fun_headline_variants":["Bounding boxes boost small-model exam grading accuracy","Cropping scans cuts compute, lifts grading accuracy","Tiny models grade better with cropped exam images","Answer crops: key to cheap, accurate exam grading","Cropped pages make small LMs grade exams smarter"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the fixed bounding box, placed once by visual inspection, actually contains every student's answer on all 126 pages; the paper does not verify per page that the crop captured the response, so a misaligned answer could distort the apparent benefit of cropping.","fun_headline_variants_meta":{"raw":{"variants":["Bounding boxes boost small-model exam grading accuracy","Cropping scans cuts compute, lifts grading accuracy","Tiny models grade better with cropped exam images","Answer crops: key to cheap, accurate exam grading","Cropped pages make small LMs grade exams smarter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3225,"prompt_tokens":700,"completion_tokens":2525,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":2451}},"tokens_in":444,"tokens_out":2525,"duration_ms":15048,"temperature":1.0,"reasoning_tokens":2451,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:21:55.676047+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the eight models on the same 126 scans with bounding boxes generated independently per page (for example, by locating the handwriting), and check whether the fixed boxes ever exclude a written answer. If a nontrivial number of answers fall outside the fixed crops, or if accuracy with per-page crops does not beat full-page accuracy, the paper's central claim fails.","supporting_citations":[],"review_version":1}