{"id":"2061da8b-ec55-457c-a07a-8c3621e46563","arxiv_id":"2412.01221","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper reports that GPT-4o's token-level uncertainty, computed as the negative log-likelihood of its output, rises monotonically as image resolution falls from 300 to 72 dpi on a single test page.","lead":"This paper measures how confident the GPT-4o model is when converting math equations from images into LaTeX code, using token probabilities at four image resolutions. It reports that lower-resolution images make the model less certain and more error-prone, suggesting image quality matters for AI-based math OCR.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 'entropy' is computed as negative log-likelihood of the greedy tokens (Eq 21), not the per-token Shannon entropy (Eq 13), so the experiments do not measure the conditional entropy H(Y|X) that the theory claims to explain.","rationale":"The reader's weakest_assumption correctly identifies the mapping from Eq (21) to H(Y|X) as the load-bearing step, and my analysis confirms this is the most serious flaw. The manuscript itself is internally inconsistent: Section 4.5 computes entropy over the distribution, while Section 5.1.3 computes negative log-likelihood of the sampled token. The example calculation also contains an arithmetic error, but the deeper issue is conceptual. Even if the experimental trend is plausible, it is not a measurement of conditional entropy, so the theoretical sections do not explain the data. The use of Fano's inequality is additionally inverted, compounding the problem. Since the central claim is precisely that entropy decreases with resolution, and the measured quantity is not entropy, the verdict REJECT stands. No further adjustment is needed beyond what the reader already concluded.","tokens_in":13190,"tokens_out":4816,"duration_ms":45196,"concrete_test":"Run the provided GitHub code for the 300 dpi image and, for each token position i, compute both -log2 Pi(yi) (Eq 21) and Hi = -sum_y Pi(y|x,y1:i-1) log2 Pi(y|x,y1:i-1) using the returned logprobs for all available tokens. Sum the per-position entropies and compare to the reported 7.77 bits. If the two sums differ materially (e.g., by more than 10%), the paper's 'entropy' is not Shannon entropy. Repeat for the 72 dpi image to check whether the discrepancy changes with resolution, which would affect even the qualitative trend.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on identifying the measured quantity with conditional entropy. Section 5.1.3 defines Hi = -log2 Pi(yi | x, y1:i-1), the self-information of the greedily chosen token. Section 4.1 defines Hi as the entropy of the next-token distribution, summing over the whole vocabulary. These differ: for a token with probability 0.6, Eq (21) gives 0.737 bits, whereas Eq (13) gives about 1.22 bits (and the paper's own example in Eq (18) uses the latter). The experimental values (7.77, 12.28, 18.27, 38.91 bits) are therefore sums of negative log-probabilities along one deterministic path, not estimates of H(Y|X), which is an expectation over all possible outputs. Section 4.6 asserts a 'direct relation' without derivation, but the theoretical results of Section 3 apply to H(Y|X), not to the self-information of a single sample. Consequently, the experimental monotonic trend does not validate the entropy-based framework; at most it shows that the model's reported token likelihoods decrease with downsampling. This is load-bearing because without the entropy identification, the paper's explanatory story collapses: Fano's inequality is also misused (a lower H(Y|X) does not imply a lower error probability; it only gives a lower bound on Pe), and the theoretical discussion cannot be tested by the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to quantify the uncertainty of GPT models in converting images of mathematical equations into LaTeX code by using entropy measures. The theoretical part (Sections 2--3) models OCR as a noisy channel and argues that higher image resolution increases mutual information I(X;Y) and decreases conditional entropy H(Y|X), relying on the Data Processing Inequality and Fano's Inequality. The experimental part (Section 5) uses GPT-4o on one arXiv page rendered at 72, 96, 150, and 300 dpi, computes a quantity called total entropy as the sum of negative log-probabilities of the greedily generated tokens (Eq. 21), and reports that this quantity increases monotonically as resolution decreases. The paper concludes that entropy analysis empirically validates the theoretical framework and that entropy is a practical uncertainty measure for GPT-based mathematical OCR.","tokens_in":13491,"tokens_out":5105,"duration_ms":43164,"significance":"If the central claim were sound, the paper would provide a simple information-theoretic explanation and diagnostic for resolution-dependent failures of LLM-based math OCR, which is a practically relevant task. The paper ships code and uses a publicly available source document, which is a positive reproducibility feature. However, the load-bearing identification of the measured quantity with Shannon entropy is incorrect, and the theoretical derivation contains unsupported and misapplied inequalities. As presented, the paper does not establish either that entropy (in the Shannon sense) decreases with resolution or that the reported monotonic trend is an entropy phenomenon. The core contribution is therefore not supported, and the framing overstates what the experiments show.","major_comments":[{"comment":"The experimental quantity Hi = -log2 Pi(yi | x, y1:i-1) is the self-information of the greedily decoded token, not the Shannon entropy of the next-token distribution defined in Eq. (13). For a token with probability 0.6, Eq. (21) gives 0.737 bits, whereas Eq. (13) gives about 1.22 bits, as the paper's own example in Eq. (18) illustrates. The reported totals of 7.77, 12.28, 18.27, and 38.91 bits are sums of negative log-likelihoods along a single deterministic output path, not estimates of H(Y|X), which is an expectation over all possible outputs. Section 4.6 asserts a 'direct relation' without derivation, but the theoretical results of Section 3 apply to H(Y|X), not to the self-information of a single sample. Consequently, the experimental monotonic trend does not validate the entropy-based framework.","section":"Section 5.1.3, Eq. (21) and Section 4.1, Eq. (13)"},{"comment":"The central inequality I_high-res(X;Y) > I_low-res(X;Y) is simply asserted without proof, and all subsequent claims about H(Y|X) in Eqs. (7) and (9) follow only from this assertion. The Data Processing Inequality is also misapplied: for the actual processing chain X -> f(X) -> Y, in which the OCR system observes only the downsampled image, DPI gives I(X; f(X)) >= I(X;Y), not I(f(X);Y) <= I(X;Y). In fact, if Y is generated solely from f(X), then I(X;Y) = I(f(X);Y), so the inequality in Eq. (11) does not establish that downsampling reduces the mutual information between the original image and the recognized text.","section":"Section 3.2.1, Eq. (6) and Section 3.2.3, Eq. (11)"},{"comment":"The paper states that 'the lower H(Y|X) implies the lower error rates Pe.' This is not a consequence of Fano's inequality, which gives the lower bound H(Y|X) >= H(Pe) + Pe log(|Y|-1). A smaller H(Y|X) does not imply a smaller Pe; the inequality only constrains achievable error probabilities from below. The argument that low entropy predicts high accuracy is therefore not supported by the cited inequality.","section":"Section 3.2.4, Eq. (12) and surrounding text"},{"comment":"Section 4.4 explains that exact entropy requires the full next-token distribution over the vocabulary and that an approximation may be computed by summing over a subset of tokens as in Eq. (17). Section 5.1.3 instead uses only the negative log-probability of the single sampled token. This is not the subset approximation described in Eq. (17), because no sum over a subset of the vocabulary is performed. The manuscript thus defines two different quantities under the same symbol Hi without reconciling them.","section":"Section 4.4, Eq. (17) vs Section 5.1.3, Eq. (21)"}],"minor_comments":[{"comment":"The experimental basis is a single page from one arXiv paper, one image per resolution, and a single generation with fixed seed and temperature 0. The strong claims in the abstract about a 'clear relationship' and 'reduced uncertainty' are not supported by any statistical repetition, error bars, or significance testing.","section":"Section 5, Table 1"},{"comment":"The notation H(y|x) is overloaded: it is used both for the total entropy defined in Eq. (14) and for the normalized entropy defined in Eq. (16). Using different symbols, such as H_norm or h_total, would avoid confusion.","section":"Section 4.2, Eq. (16)"},{"comment":"The reported normalized entropy is labeled in 'bits' throughout Table 1, but a dimensionless quantity divided by n log2|V| cannot have units of bits; this labeling should be corrected.","section":"Section 5.1.3"},{"comment":"The stated limitation that entropy estimates 'may be approximations' if the model does not output the full distribution does not describe the actual computation in Eq. (21), which is not an approximation of Eq. (13) but a different quantity. This discrepancy should be acknowledged explicitly.","section":"Section 6, first bullet"}],"recommendation":"reject","confidential_remarks":"The paper's central contribution depends on equating the sum of negative log-probabilities of generated tokens with the conditional entropy H(Y|X). This identification is wrong, and the theoretical derivation in Section 3 relies on an asserted inequality and misapplied information-theoretic results. These are not local presentation issues but fundamental problems with the paper's core claim. The empirical observation that token log-likelihoods decrease with downsampling might be salvageable as a separate, more modest empirical finding, but that would require substantial rewriting and would not match the title and abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2412.01221. The empirical observation is real: GPT-4o's summed negative log-likelihood over its greedy output for a fixed math PDF page increases as resolution drops (7.77 to 38.91 bits). That's a plausible, reproducible practical finding, and the GitHub repo and clear write-up are in its favor.\n\nWhat's not real is the paper's framing. Section 3 develops theory for the conditional entropy H(Y|X). Section 4.6 asserts a 'direct relation' between that and the 'entropy' H(y|x) computed in the experiments. But the experiments use Eq (21): Hi = -log2 Pi(yi | ...), the self-information of the greedily chosen token, not the Shannon entropy of the next-token distribution from Eq (13). These are different: the latter is an expectation over the vocabulary. So the measured numbers are negative log-likelihood sums, not estimates of H(Y|X). The monotonic trend does not validate the entropy-based theory. This is the load-bearing flaw.\n\nThe theoretical derivation also has a soft spot: Eq (6), the claim that high-res increases mutual information, is asserted, not proved. DPI actually supports the direction of the inequality, but the paper's application is loose because the model's output sequence changes with resolution, so the comparison isn't a clean DPI setup. Fano's inequality is used in the direction low H implies low Pe, which is fine as a contrapositive, so I'd not count that as an additional error. But it doesn't rescue the entropy identification.\n\nOther weaknesses: one page, four resolutions, no repeated trials or error bars, accuracy judged visually, no comparison across models or PDFs. The claimed 'gap' in the literature is asserted, not demonstrated; semantic entropy and related work already occupy that space.\n\nIf the authors reframed the paper as 'negative log-likelihood increases with downsampling for GPT-4o on one page,' it would be a modest empirical note. As written, the central claim about entropy is unsupported, and the theory section doesn't connect to the measurements. I'd desk reject it; it needs a complete rework to be a credible contribution. Not worth a serious referee slot in its current form.","headline":"The paper's empirical trend is real, but it's negative log-likelihood, not entropy, so the central claim is unsupported.","tokens_in":13966,"tokens_out":4436,"would_cite":false,"duration_ms":39836,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that GPT uncertainty in math OCR scales with image resolution, measurable by token entropy.","keywords":["GPT","entropy analysis","mathematical OCR","LaTeX recognition","image resolution","uncertainty quantification","conditional entropy","mutual information"],"falsifier":"Compute the true next-token entropy at each position from the full vocabulary distribution (not just the sampled token's probability) for the same four images, and check whether total entropy still rises monotonically from 300 to 72 dpi; if it does not, the paper's trend is an artifact of the negative-log-likelihood approximation rather than a property of the model's uncertainty.","tokens_in":12978,"feed_emoji":"🧮","tokens_out":5218,"duration_ms":43582,"temperature":0.7,"pith_summary":"This paper claims that the uncertainty of a GPT model extracting LaTeX from images of math equations can be read directly from the entropy of its generated tokens: higher-resolution images produce lower entropy, and lower-resolution images produce higher entropy and more errors. The authors model the OCR task as a noisy information channel, identify the sum of negative log probabilities of generated tokens with the conditional entropy H(Y|X) of the recognized text given the image, and support the resolution trend with the Data Processing Inequality and Fano's Inequality. In experiments with GPT-4o on a page converted at 300, 150, 96, and 72 dpi, total entropy rises from 7.77 bits to 38.91 bits while accuracy falls from perfect to severely degraded. The claim matters because it offers a token-level, ground-truth-free uncertainty signal for a practical OCR use case.","feed_headline":"Lower image resolution raises GPT math-OCR uncertainty","feed_subtitle":"Entropy of generated LaTeX tokens climbs from 7.77 to 38.91 bits as dpi drops from 300 to 72.","key_machinery":"The central object is the conditional entropy of the generated token sequence, H(y|x) = -Σ log2 P_i(y_i | x, y_{1:i-1}), summed over output positions, together with its normalized version dividing by n log2|V|. The paper identifies this empirically computed quantity with the theoretical conditional entropy H(Y|X) of the OCR channel, and uses three information-theoretic tools to explain the experiments: the identity I(X;Y) = H(Y) - H(Y|X) with H(Y) assumed constant for fixed text, the Data Processing Inequality applied to resolution reduction as a deterministic function, and Fano's Inequality H(Y|X) ≥ H(P_e) + P_e log(|Y|-1) linking conditional entropy to error probability. This machinery turns token-level log probabilities into a claimed measure of recognition uncertainty and error likelihood.","core_discovery":"The paper's central claim is that, for GPT-based mathematical OCR, the conditional Shannon entropy of the output token sequence is a faithful proxy for the model's uncertainty, and that this entropy is governed by input image resolution through the information-theoretic relation I(X;Y) = H(Y) - H(Y|X) with H(Y) approximately constant for a fixed text. Empirically, using the GPT-4o model with temperature zero and logprobs enabled, the paper reports a monotone increase in total entropy as resolution drops: 7.77 bits at 300 dpi, 12.28 at 150 dpi, 18.27 at 96 dpi, and 38.91 bits at 72 dpi, with normalized per-token entropy rising from 0.0171 to 0.082 bits. The authors take this as validating that lower resolution reduces mutual information between image and text, raising conditional entropy, and that higher conditional entropy implies higher error probability by Fano's Inequality. The contribution is the direct application of conditional entropy computed from token log-probabilities as a quantitative uncertainty measure in math-OCR.","pith_inferences":["The paper does not establish whether the entropy-accuracy link is monotone at finer resolution gradations or across varied document layouts; a natural extension is testing multiple pages and fonts to see if a single normalized-entropy threshold predicts errors.","The reported total entropy mixes sequence length with per-token confidence, so a stricter test would recompute entropy from the full vocabulary distribution at each position; if the trend vanishes, the result is an artifact of negative log-likelihood rather than true entropy.","One implication the authors leave implicit is that entropy could drive an adaptive OCR pipeline, re-rendering an image at higher resolution whenever normalized entropy exceeds a threshold.","The Data Processing Inequality argument holds for deterministic downsampling; stochastic degradation such as blur or noise adds randomness that may break the simple monotone claim, a testable edge case."],"forward_implications":["In deployed math-OCR systems, normalized token entropy can flag low-confidence extractions for human review without needing a reference transcription.","Image resolution becomes a tunable parameter with a measurable uncertainty cost: below roughly 100 dpi, the reported entropy rises sharply, so a quality floor can be set from entropy data.","Because the entropy is computed from a single generation, it offers a cheap, per-output uncertainty estimate for closed models that expose logprobs, no ensembles or internal states required.","The monotone relation between resolution and entropy gives a concrete validation target for other GPT-based OCR models: the same experimental protocol can compare uncertainty calibration across models."],"supporting_citations":[{"why":"Defines the GPT-4o model and API used for all experiments, whose logprob output is the measured quantity.","marker":"[25]"},{"why":"Technical report for GPT-4o, the model the paper configures with temperature 0 and logprobs.","marker":"[26]"},{"why":"The arXiv paper whose page is converted to JPEG at 72, 96, 150, and 300 dpi; it is the sole input source for the four experiments.","marker":"[27]"}],"fun_headline_variants":["Low-res images inflate GPT math-OCR entropy","Entropy tracks GPT uncertainty in math OCR","Resolution governs GPT math-OCR uncertainty","Higher DPI slashes GPT math-OCR entropy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument collapses if the sum of negative log probabilities of the sampled tokens is not a good stand-in for the model's true conditional entropy: for example, if the model's reported per-token probabilities are miscalibrated or if the missing probability mass over the rest of the vocabulary is large.","fun_headline_variants_meta":{"raw":{"variants":["Low-res images inflate GPT math-OCR entropy","Entropy tracks GPT uncertainty in math OCR","Resolution governs GPT math-OCR uncertainty","Higher DPI slashes GPT math-OCR entropy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1338,"prompt_tokens":948,"completion_tokens":390,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":564,"tokens_out":390,"duration_ms":3800,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:38:12.372799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the true next-token entropy at each position from the full vocabulary distribution (not just the sampled token's probability) for the same four images, and check whether total entropy still rises monotonically from 300 to 72 dpi; if it does not, the paper's trend is an artifact of the negative-log-likelihood approximation rather than a property of the model's uncertainty.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GPT-4o model and API used for all experiments, whose logprob output is the measured quantity."},{"cited_title":"Quantum Data Compression and Quantum Cross Entropy","cited_arxiv_id":"2106.13823","evidence_quote":"The arXiv paper whose page is converted to JPEG at 72, 96, 150, and 300 dpi; it is the sole input source for the four experiments."}],"review_version":1}