{"id":"b729e825-aaac-41da-96ea-6a2031b1ddfc","arxiv_id":"2505.00746","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using a sliding window over per-token Shannon entropy from GPT-4o log-probabilities, the paper flags high-uncertainty spans that, in a 36-image proof-of-concept, tend to contain most math OCR errors.","lead":"Sliding-window Shannon entropy computed from GPT-4o per-token confidence scores can highlight the spans of an OCR transcript most likely to contain recognition errors. A small qualitative proof-of-concept suggests these hotspots catch most human-flagged mistakes, offering a lightweight way to focus proofreading of machine-transcribed math.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Error labels are collected with hotspots visible to annotators, so the reported overlap may be an artifact of the evaluation protocol rather than evidence for the method.","rationale":"The reader identified the permissive union labeling as the weakest assumption. My analysis agrees that the error labels are the sole empirical support for the central claim, but the more precise and more damaging problem is that the annotation protocol is not independent of the hotspot display: annotators saw the highlighted windows while marking errors, and the any-flag-counts rule plus lack of inter-annotator agreement inflates in-hotspot errors while potentially missing out-of-hotspot errors. This is a measurement artifact, not merely label permissiveness. The paper itself concedes in Section 6.3 that it provides no rigorous quantitative benchmarks, and Section 5 contains no numerical overlap statistics. The proposed concrete test—blinded re-annotation with a random-window baseline and agreement metrics—directly settles whether the central claim survives unbiased measurement. The verdict remains CONDITIONAL because the method is simple, the mathematical derivation in Section 3.6 is sound, and the claim could still hold under proper evaluation; the condition is that the authors must provide blinded, quantitative evidence.","tokens_in":13067,"tokens_out":3041,"duration_ms":38175,"concrete_test":"Re-run the annotation on the same 36 transcripts in a fully blinded protocol: annotators mark all errors in the complete LaTeX output without seeing any entropy heat-mapping or highlighting, and without knowing which windows are hotspots. Then compute the recall of the top-3 entropy windows against this unbiased error set, compare it to the recall of three randomly selected 10-token windows, and report inter-annotator agreement (e.g., pairwise F1 or Krippendorff's alpha). If blinded recall is substantially above the random-window baseline and the original union labels are close to the blinded labels, the concern is resolved; otherwise the abstract's 'vast majority' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the vast majority of true GPT-4o transcription errors fall inside the top-3 entropy windows—rests entirely on the human error labels collected in Sections 4.2–4.3. That protocol is not a valid unbiased test of the claim. Annotators were shown the GPT output with entropy hotspots visually highlighted in color (Section 4.3, Side-by-Side Interface) and then asked to flag errors; any token flagged by any of three annotators was counted as an error, with no inter-annotator agreement requirement (Section 4.2). This makes the error set systematically biased toward hotspot tokens: annotators are more likely to scrutinize and flag highlighted regions, while errors outside hotspots can be missed because low-entropy text is visually de-emphasized. The union rule compounds this by counting any single annotator's in-hotspot flag while not recovering outside errors missed by all. Section 5 reports only qualitative impressions ('nearly all misread symbols... fell squarely under these high-entropy windows') and Section 6.3 concedes the results do not yield rigorous quantitative benchmarks. Without a blinded labeling protocol, a random or even inverse-entropy selection of windows could appear to contain 'most errors' under the same measurement procedure. The load-bearing assumption is therefore not merely that labels are permissive, but that the labeling process itself is independent of the hotspots being evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc error-localization method for GPT-based mathematical OCR. Per-token log-probabilities returned by the GPT-4o API are converted into truncated Shannon entropies, which are then smoothed with a fixed-length sliding window; the highest-average-entropy windows are presented as hotspots. The authors contribute a short proof that the top-k-plus-tail entropy is a lower bound on the true token entropy, a linear-time algorithm for window aggregation and hotspot selection, a proof-of-concept study on 12 scanned arXiv pages at three resolutions (36 image instances), and a public GitHub repository with code and annotation guidelines. The central empirical claim, stated in the abstract and Section 5, is that the vast majority of human-verified GPT-4o transcription errors fall inside the top-3 entropy windows. The paper explicitly frames itself as a proof-of-concept and concedes in Section 6.3 that it does not provide rigorous quantitative benchmarks.","tokens_in":13246,"tokens_out":2298,"duration_ms":24797,"significance":"If the central claim were established with a valid evaluation protocol, the method would be a useful lightweight triage tool: it runs in O(n) time once log-probabilities are available, requires no model retraining, and is directly applicable to widely available API outputs. The theoretical lower-bound proof in Section 3.6 is correct, and the release of code and annotation guidelines supports replication. However, the empirical evidence for the headline claim is currently anecdotal and is confounded by the evaluation protocol: annotators see the entropy-based hotspots before flagging errors, and no precision, recall, or baseline comparison is reported. The contribution is therefore better assessed as a promising proof-of-concept than as a validated error-localization method.","major_comments":[{"comment":"The annotation protocol is not a valid unbiased test of the central claim. Annotators compare the original image against a transcript in which the entropy hotspots are visually highlighted in color, and are then asked to flag errors. This makes it substantially more likely that tokens inside highlighted windows are scrutinized and flagged, while errors outside the highlighted regions are more easily missed because low-entropy text is visually de-emphasized. The reported overlap between hotspots and errors may therefore be an artifact of the labeling procedure rather than evidence for the method. The authors should re-run annotation with the entropy highlights hidden during error marking (or use a within-subject design with blinded passes) and only then compute the overlap.","section":"Section 4.3 (Side-by-Side Interface)"},{"comment":"The union rule for error labels—any token flagged by any of the three annotators counts as an error—amplifies the bias described above. With no requirement of inter-annotator agreement, an in-hotspot token flagged by a single annotator is counted as an error, while an out-of-hotspot token missed by all three annotators is not counted at all. The paper should report per-annotator error sets, pairwise agreement, and majority-vote results; the central claim should be evaluated under the most conservative reasonable labeling rule, not the most permissive one.","section":"Section 4.2 (Annotation Protocol)"},{"comment":"The abstract claims that the 'vast majority of true errors' fall inside the high-entropy regions, but Section 5 reports no quantitative measure of this overlap. Section 6.3 explicitly concedes that the results 'do not yield rigorous quantitative benchmarks.' To support the headline claim, the authors should report token-level recall and precision (or window-level equivalents) for the top-M windows, along with error counts per transcript and a confidence interval across the 36 instances. A necessary control is a random-window baseline with the same number of windows: if the method's recall is not clearly above that baseline, the central claim is unsupported.","section":"Section 5 and Section 6.3"},{"comment":"The experimental units are not independent: the 36 image instances come from only 12 source pages, each rendered at three resolutions, and the same model outputs are used across resolutions. This dependency should be accounted for when reporting any aggregate statistics, and the paper should state whether hotspot recall is computed per page or per resolution instance. Without this distinction, the effective sample size for the headline claim is 12 pages, not 36.","section":"Section 5.4 and Section 4.1"}],"minor_comments":[{"comment":"The proof of the lower bound labels the entropy chain-rule equation as '(1)', but Equation (1) in Section 2.1 is already the definition of Shannon entropy; renumbering the equations would avoid ambiguity.","section":"Section 3.6.3"},{"comment":"The claim that annotators focused on '10–15% of the tokens' does not match the stated parameters for all transcripts: three windows of W=10 in a 600-token document cover 5%, while in a 200-token document they cover 15%. The paper should report the actual covered fraction per transcript.","section":"Section 5.3"},{"comment":"The text contains typos and ligature artifacts (e.g., 'transcrib e' in the abstract, various 'confidence' forms) that should be cleaned up in a revision.","section":"Abstract and Section 3.4"},{"comment":"The rank-based hotspot selection does not specify whether overlapping windows are merged or de-duplicated; with W=10 and a dense entropy peak, the top-3 windows may overlap heavily, effectively covering fewer than 30 distinct tokens. This should be clarified and, if necessary, post-processed to non-overlapping spans.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the lower-bound derivation and the algorithmic pipeline are sound, and the paper is honest about its limitations. The central issue is that the abstract's 'vast majority' claim is supported only by a biased annotation protocol and qualitative impressions. This is fixable with a blinded annotation study and basic token/window-level metrics, so I see major revision rather than rejection, provided the authors are willing to substantially strengthen the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes a straightforward idea: compute per-token Shannon entropy from GPT-4o's top-k log-probabilities, smooth it with a sliding window, and flag the highest-entropy windows as likely OCR error locations. The ingredients are all known, but the specific application to localizing errors in math transcription is new. The math in Section 3.6 is correct, if standard: merging the tail into one bucket cannot increase entropy. The pipeline is simple, O(n), and the code and annotation guidelines are released. The paper is also refreshingly honest about its limitations. That honesty makes the soft spot harder to ignore. The central claim that the vast majority of true errors fall inside the top three entropy windows is supported only by an evaluation that is not a valid test of that claim. Annotators were shown the GPT output with the entropy hotspots highlighted in color, then asked to flag errors. Any token flagged by any of three annotators counted as an error, with no inter-annotator agreement required. This protocol biases the error set toward hotspot tokens and makes the reported overlap partly an artifact of attention. The paper concedes in Section 6.3 that it does not yield rigorous quantitative benchmarks, but the concession undersells the problem: it is not just that metrics are missing, it is that the measurement procedure cannot separate the method's signal from the highlighting itself. Still, the core idea is plausible. Entropy spikes should correlate with visual ambiguity, and visual ambiguity is where OCR errors happen. The paper would be worth engaging with if the authors redid the evaluation with a blinded protocol, plus inter-annotator agreement, precision/recall, and a baseline comparison against random or inverse-entropy windows. The current version is a proof-of-concept with a confounded evaluation, not a validated result. For a reader working on post-editing LLM-based OCR, this is a useful starting point. It deserves peer review, but the referee should ask for major revision: blinded annotation and real metrics. Without that, the main claim remains untested.","headline":"A simple, plausible entropy-based method for localizing GPT OCR errors, but the central empirical claim is compromised by an annotation protocol that shows annotators the hotspots before they label errors.","tokens_in":699,"tokens_out":866,"would_cite":false,"duration_ms":29476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sliding-window average of per-token Shannon entropy places the vast majority of human-verified GPT-4o transcription errors inside a handful of 10-token hotspots.","keywords":["sliding-window entropy","Shannon entropy","mathematical OCR","GPT-4o transcription","token-level uncertainty","error localization","vision-language models"],"falsifier":"Re-annotate the same 36 transcripts counting as errors only tokens flagged by at least two of the three annotators; if the fraction of these consensus errors inside the top-three 10-token windows is close to the fraction of the transcript those windows cover (roughly 10–15%), the claimed concentration is an artifact of permissive labeling.","tokens_in":12781,"feed_emoji":"🔍","tokens_out":11082,"duration_ms":109173,"temperature":0.7,"pith_summary":"The paper proposes a lightweight error-localization tool for GPT-4o transcription of scanned mathematics: take the token-level log-probabilities the model already exposes, convert them to per-token truncated Shannon entropies, and average them over sliding windows to form an \"uncertainty landscape\" of the output. The central empirical claim is that on 36 image instances (12 scanned research pages rendered at 72, 150, and 300 dpi), the three 10-token windows with the highest average entropy contain the vast majority of human-verified transcription errors. If true, human proofreaders or automatic re-prompting can focus on a small, highlighted fraction of the transcript instead of rereading the whole page. The paper also proves a short information-theoretic fact: the top-k-plus-tail entropy computed from API log-probabilities is a lower bound on the true Shannon entropy, because coarse-graining the tail can only decrease entropy. The evaluation is deliberately qualitative, using a small corpus and permissive annotation, and is offered as a proof of concept rather than a benchmark.","feed_headline":"Three 10-token entropy windows catch most GPT-4o OCR errors","feed_subtitle":"Proofreaders can skip most of a scanned math page and still see the bulk of transcription mistakes.","key_machinery":"The load-bearing object is the sliding-window truncated Shannon entropy signal. At each token position i, the API supplies probabilities $p_j$ for the k most likely tokens; the remaining mass is placed in a tail bucket $p_{\\text{tail}} = 1 - \\sum_{j=1}^{k} p_j$, and the paper defines $\\hat H(i) = -\\sum_{j=1}^{k} p_j \\log_2 p_j - p_{\\text{tail}} \\log_2 p_{\\text{tail}}$. Averaging $\\hat H(i)$ over windows of length W, $A_i = \\frac{1}{W}\\sum_{r=i}^{i+W-1}\\hat H(r)$, produces the uncertainty landscape; the top-M windows ($M=3$, $W=10$ in the experiments) are the hotspots. This signal converts black-box API log-probabilities into a visually highlightable heat-map and is the entire mechanism by which the paper localizes likely errors. The theoretical anchor is the coarse-graining inequality $H(f(X)) \\le H(X)$ for a deterministic merging map f, which justifies treating the truncated value as a conservative, non-overshooting estimate of the model's uncertainty.","core_discovery":"The paper's central claim is that the places where a vision-language model's own token distribution is most spread out coincide with the places where its OCR output is most likely to be wrong. For each decoded token, it computes a truncated Shannon entropy from the probabilities of the k most likely tokens plus one tail bucket, then averages these values over sliding windows of length W=10 and keeps the top three windows as hotspots. In its proof-of-concept corpus, the paper finds that the vast majority of human-flagged errors—missing symbols, mismatched braces, confused Greek letters, dropped subscripts—fall inside those hotspots, while lower-entropy spans can be skimmed. The accompanying theoretical result is that this truncated entropy is a strict lower bound on the true Shannon entropy; by the coarse-graining property of entropy, merging all non-top-k tokens into a single tail event cannot increase the entropy. The paper presents the finding as first evidence for a triage tool, not as a quantitative benchmark.","pith_inferences":["A stricter annotation rule (at least two of the three annotators must flag a token) would likely shrink the error set, and the measured concentration of errors inside high-entropy windows could drop; re-annotating the released transcripts under that rule would show how much of the result depends on the permissive union labeling.","The same sliding-window entropy signal could be applied to speech recognition or handwritten-text recognition transcripts, where local token competition also marks confusion; the paper lists this as future work but does not test it.","If entropy peaks track visual ambiguity rather than actual mistakes, the false-alarm rate may be irreducible at low resolutions; combining the heat-map with a character-aware language-model score could better separate 'uncertain' from 'wrong.'"],"forward_implications":["Post-editors of GPT-based mathematical OCR can inspect roughly 30 tokens per page (three 10-token windows) and still see most of the real transcription errors, cutting proofreading time.","The hotspots can be fed back to the model as localized re-prompting queries, so corrections are attempted only on suspicious spans rather than on the entire transcript.","Because the computation is linear-time once log-probabilities are retrieved and requires no retraining, the method runs on any API that exposes top-k token probabilities.","Image resolution directly shapes the signal: 72 dpi transcriptions produce many high-entropy windows, while 300 dpi reduces the number of hotspots by roughly 80%, making the heat-map a diagnostic for scan quality.","Entropy hotspots still include false alarms—windows where the model is uncertain but transcribed correctly—so the method is a triage aid for human review, not an autonomous error detector."],"supporting_citations":[{"why":"Supplies the Shannon entropy definitions and the coarse-graining inequality used in the paper's lower-bound proof.","marker":"[1]"},{"why":"Describes the GPT-4o system whose API exposes top-k token log-probabilities, the raw signal the pipeline consumes.","marker":"[4]"},{"why":"Gives empirical evidence that token-level entropy peaks reflect a language model's confusion, the premise that high entropy marks likely error zones.","marker":"[10]"},{"why":"Motivates the segment-wise post-OCR correction setting that the sliding-window approach is designed to feed.","marker":"[25]"}],"fun_headline_variants":["Sliding entropy windows pinpoint GPT-4o OCR slips","High-entropy token spans expose most OCR mistakes","Entropy hotspots flag GPT-4o transcription errors","Top three entropy windows snag most OCR errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the union of whatever the three annotators flagged is an accurate list of the transcript's true errors, even though the paper did not require annotators to agree with one another.","fun_headline_variants_meta":{"raw":{"variants":["Sliding entropy windows pinpoint GPT-4o OCR slips","High-entropy token spans expose most OCR mistakes","Entropy hotspots flag GPT-4o transcription errors","Top three entropy windows snag most OCR errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":2998,"prompt_tokens":911,"completion_tokens":2087,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":527,"tokens_out":2087,"duration_ms":14371,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:02:34.326755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 36 transcripts counting as errors only tokens flagged by at least two of the three annotators; if the fraction of these consensus errors inside the top-three 10-token windows is close to the fraction of the transcript those windows cover (roughly 10–15%), the claimed concentration is an artifact of permissive labeling.","supporting_citations":[{"cited_title":"Cover and Joy A","cited_arxiv_id":null,"evidence_quote":"Supplies the Shannon entropy definitions and the coarse-graining inequality used in the paper's lower-bound proof."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"Describes the GPT-4o system whose API exposes top-k token log-probabilities, the raw signal the pipeline consumes."},{"cited_title":"Can llms express their uncertainty? an empirica l evaluation of confidence elicitation in llms","cited_arxiv_id":null,"evidence_quote":"Gives empirical evidence that token-level entropy peaks reflect a language model's confusion, the premise that high entropy marks likely error zones."},{"cited_title":"Icdar 2019 competition on post-ocr text correction","cited_arxiv_id":null,"evidence_quote":"Motivates the segment-wise post-OCR correction setting that the sliding-window approach is designed to feed."}],"review_version":1}