{"id":"3a085347-6fb7-4cd7-bdab-3c1c683b708d","arxiv_id":"2608.00536","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A power-law annealing of edit-distance-based rewards improves GRPO-style RL for document parsing across text, tables, and formulas.","lead":"This paper proposes a reward-sharpening mechanism called Step-Aware Annealing that makes reinforcement learning for document parsing more effective. It reports state-of-the-art text, table, and formula extraction on standard benchmarks using an unchanged 3B vision-language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not protected against benchmark overfitting: B.7 selects Δγ and (α,β) using the exact OmniDocBench and DocElemHard scores reported in Table 3, so the 'consistent SAA improvement' may reflect selection on the test benchmarks rather than a fixed recipe.","rationale":"The paper's contribution is an empirical recipe: SAA plus element-specific reference rewards yields consistent gains and SOTA. The strongest evidence is Table 3 and the supplemental drill-downs. However, the supplementary material itself (B.7) reveals that the two headline hyperparameters of the recipe were selected by maximizing the same aggregate scores that later appear as the paper's main result. This makes the central claim vulnerable in a way that no internal consistency of the ablations can repair: the comparison is not between a fixed method and baselines, but between a method and baselines after choosing the method's parameters on the evaluation sets. The B.9 statement about checkpoint selection does not cover this. The formula reward is not literally the formula evaluation metric (CDM), so the reader's 'rewards identical to evaluation metrics' is only partially accurate; the more precise issue is selection on the test benchmarks. The proposed replication check would settle it. Since this strengthens the existing CONDITIONAL rather than overturning the paper, UNCHANGED is the right verdict, with the condition made explicit: report validation-based hyperparameter selection or demonstrate transfer to a fresh benchmark.","tokens_in":18459,"tokens_out":5289,"duration_ms":66423,"concrete_test":"Perform a pre-registered replication: fix Δγ=8, γ_init=1, α=0.8, β=0.2, τ, and all RL settings before any benchmark evaluation; carve a validation split out of the 612k RL pool (or use a separate internal benchmark) for all sensitivity analyses; then evaluate on OmniDocBench and DocElemHard exactly once. Report DocPO versus the non-annealed baseline and versus Δγ=4 on this untouched test evaluation. If the margin over the nearest comparison shrinks to within run-to-run noise or reverses, the claim that SAA robustly produces the reported consistent gains is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Supplementary B.7 (Tables 14–15) evaluates the SAA curvature range Δγ on OmniDocBench and DocElemHard, and the formula weights (α,β) on OmniDocBench; the values adopted in the final recipe (Δγ=8, α=0.8, β=0.2) are the ones that score highest on the same final benchmarks used for the headline SOTA comparison in Table 3. This is a test-set selection procedure. The statement in B.9 that no final benchmark test set is used for early stopping or model selection does not rule out choosing hyperparameters on the test benchmarks; in fact Table 14 explicitly reports test-set performance for Δγ=0, 4, and 8 and labels Δγ=8 as DocPO. Therefore the central empirical claim that SAA consistently improves GRPO-style RL and reaches SOTA is at least partly a claim about a configuration selected on the evaluation distribution. The Limitations section also concedes that proxy rewards can be gamed. The concern is not that the method is fabricated: the ablations are internally consistent and the external WikiTableSet and UniMERNet results provide some transfer evidence. The problem is that a single, unseeded run plus hyperparameters chosen on the test set cannot support the strong 'consistent' and 'SOTA' wording without an independent validation split.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Step-Aware Annealing (SAA), a power-law reward sharpening mechanism for GRPO-style reinforcement learning in document parsing. Base rewards are element-specific: NED for text, weighted TEDS for tables, and a syntax-gated Rubric+edit reward for formulas. SAA raises the exponent gamma over training according to an adaptive schedule controlled by a Dynamic Dispersion Controller. The authors instantiate this in the DocPO framework on a Qwen2.5-VL-3B backbone and report improved text, table, and formula metrics on OmniDocBench and their self-constructed DocElemHard benchmark, including state-of-the-art scores without post-processing or vision-encoder modification. The supplementary material contains ablation studies, hyperparameter sensitivity analyses, external benchmark results on WikiTableSet and UniMERNet, and formal propositions on the reward-sharpening transform, together with an explicit caveat that these propositions do not guarantee monotonic growth of the GRPO advantage.","tokens_in":18771,"tokens_out":3866,"duration_ms":47808,"significance":"If the empirical claims hold under a properly controlled protocol, the contribution is practically valuable: SAA is a simple, reward-agnostic technique that could improve RL-based document parsing without architectural changes, and the paper documents its recipe, prompts, and dataset construction in unusual detail. The theoretical propositions in the supplement are correct but modest, and the authors honestly state that they describe raw reward spacing, not advantage dynamics. The paper's strengths include a public code link, detailed ablations isolating SAA, external transfer benchmarks, and explicit limitations acknowledging that proxy rewards can be gamed. The main weakness is that the headline numbers are currently tied to hyperparameters selected on the same benchmarks used for final reporting, and all results appear to come from single-seed runs; the significance of the central claim is therefore conditional on additional validation.","major_comments":[{"comment":"The final hyperparameter recipe is selected on the evaluation benchmarks. Table 14 reports OmniDocBench and DocElemHard overall scores for Δγ = 0, 4, and 8 and labels Δγ = 8 as DocPO; Table 15 selects α = 0.8, β = 0.2 on OmniDocBench CDM. The statement in B.9 that no final benchmark test set is used for early stopping or model selection does not exclude using those same benchmarks to choose hyperparameters. Since Table 3 then reports state-of-the-art results on exactly these benchmarks, the central claim that SAA 'consistently improves' GRPO-style RL is partly a claim about a configuration selected on the evaluation distribution. Please add an independent validation split for hyperparameter selection, re-report final numbers under that protocol, or explicitly restrict the SOTA claim to the selected configuration with the selection disclosed as part of the result.","section":"§3.4, §B.7, §B.9"},{"comment":"All RL runs appear to be single-seed. No random seed is reported, and no error bars, confidence intervals, or significance tests are given for the central comparisons in Tables 2, 3, and 4, or for the supplementary tables. Headline gains such as text NED 0.0238→0.0125 on OmniDocBench and formula CDM 87.69→92.88 on DocElemHard could be within run-to-run variance, especially given the small formula subset in DocElemHard. Please report results over at least three seeds with mean and standard deviation, or provide a clear justification for why seed variance is negligible.","section":"§4.1, Table 3"},{"comment":"The evaluation metrics are the same quantities, or the same metric family, as the RL rewards: NED for text, TEDS for tables, and edit-distance/rubric signals for formulas. The Limitations section correctly concedes that proxy rewards can be gamed or miss semantic nuances. The external benchmarks in B.5 provide some transfer evidence, but they are reported for a single configuration and do not rule out overfitting to the metric family. Please add a reward-hacking analysis or an independent assessment of parsing quality, such as human spot-checks or a held-out metric not used as a reward, to support the claim that the gains reflect genuine parsing quality rather than metric exploitation.","section":"§3.1–3.3, §4.1, Limitations"}],"minor_comments":[{"comment":"The window size s_window is described as 'set to 3' without specifying the unit; please clarify whether it is measured in training steps, gradient updates, or mini-batches, and how this interacts with the global batch size of 128.","section":"§3.4, Eq. (6)"},{"comment":"The training-progress plot appears to show a single run for each method; if multiple seeds are added, this figure should include error bands or representative individual curves so the '~1.8x faster' and '+1.5 points' claims can be assessed for stability.","section":"§4.2, Figure 7"},{"comment":"In the matched comparison with peer RL parsers, the text says all models 'receive the same cropped patches'; please state explicitly whether the peer models were only evaluated on these patches or were also fine-tuned/RL-trained on the same training patches, since this affects the interpretation of the comparison.","section":"§B.3"},{"comment":"The table drill-down in Table 7 includes the caution that vertical and structured-text subsets are small (n=6 and n=15), but the formula and text drill-downs in Table 12 do not report subset sample sizes; please add them or add a comparable caution.","section":"§B.6, Table 12"},{"comment":"The notation for the base time constant is inconsistent: Eq. (6) uses τ, while B.7 refers to τ_base; please unify the notation to avoid confusion.","section":"§3.4 and §B.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for the venue and the method is plausible, but the empirical protocol needs strengthening before publication. The hyperparameter selection on the final benchmarks and the absence of multi-seed results are the main barriers. I do not see evidence of fabrication; the issue is statistical and methodological control. If the authors add an independent validation split and seed variance estimates, the paper could become a solid practical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before you read this. First, the paper's core idea is genuinely simple and plausibly useful: instead of a fixed power-law reward sharpening, anneal the exponent over training, and the ablations show it consistently beats non-annealed rewards across text, tables, and formulas. Second, the headline SOTA numbers are softer than they look, because the authors selected their two key hyperparameters (Δγ and the formula weights) on the exact benchmarks they then report as final results. The stress-test note is right about this.\n\nWhat is actually new: Step-Aware Annealing (a power-law exponent that grows with training, plus a dispersion-based adaptive time constant), element-specific rewards (NED, weighted TEDS, and a syntax-gated rubric+edit for formulas), and the DocElemHard benchmark. The paper does a lot well. The ablations isolate SAA cleanly. The matched element-level comparison against INFINITY Parser and olmOCR 2 is fair. The external WikiTableSet and UniMERNet results show transfer. The supplementary propositions are correct, and the authors honestly note they do not guarantee monotonic advantage growth. The stability diagnostics — rollout diversity and actor entropy — are a nice touch.\n\nSoft spots, in proportion. The main one is the hyperparameter selection in B.7. Table 14 chooses Δγ=8 on OmniDocBench and DocElemHard, Table 15 chooses α=0.8, β=0.2 on OmniDocBench, and those constants then go into the final recipe reported on those same benchmarks. The B.9 note about not using the final benchmark for early stopping is true for checkpoints, but it doesn't cover this. That is partial test-set circularity, and with single-seed runs and no released code, the words 'consistent' and 'state-of-the-art' outrun the evidence. The rewards also equal the evaluation metrics, which invites reward hacking; the authors acknowledge this in limitations. Two smaller items: the formula rubric uses a 7B LLM to judge a 3B policy, and DocElemHard is constructed partly from accumulated failure cases of earlier model versions, so its composition is not independent. None of these sink the paper, but they move it from established fact to promising result.\n\nWho is this for? People working on RL for structured generation or document AI. It deserves a serious referee. I'd ask for multiple seeds, a real validation split for hyperparameters, and code release before trusting the SOTA claim. Verdict: send to peer review, conditional accept after major revision.","headline":"Simple reward-sharpening annealing that consistently helps in ablations, but the SOTA claim is compromised by test-benchmark hyperparameter tuning and single-seed runs; worth peer review with a required major revision.","tokens_in":19291,"tokens_out":2844,"would_cite":true,"duration_ms":31427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes Step-Aware Annealing, a reward sharpening mechanism that makes GRPO-style document parsing reinforcement learning effective in the high-accuracy regime, and shows that a standard 3B vision-language backbone with…","keywords":["document parsing","reinforcement learning","reward shaping","step-aware annealing","GRPO","tree edit distance","LaTeX formula recognition","table recognition"],"falsifier":"A concrete test: evaluate the DocPO-trained policy on a new document corpus whose ground truth is annotated by humans and scored with a metric different from the RL rewards—for example, rendered-image similarity for tables instead of HTML-tree TEDS, or human-judged semantic equivalence for formulas. If the SAA-trained model's margin over the non-annealed baseline shrinks or vanishes under that scoring, the claim that SAA improves genuine parsing quality rather than merely the trained metrics would be refuted; if the margin survives, the claim is supported.","tokens_in":18272,"feed_emoji":"📄","tokens_out":8622,"duration_ms":90757,"temperature":0.7,"pith_summary":"This paper tries to solve a practical problem: reinforcement learning for document parsing stalls when outputs become nearly correct, because near-miss transcriptions score almost as highly as perfect ones. Its answer is Step-Aware Annealing (SAA), a schedule that gradually sharpens any normalized reference-based reward by raising it to a growing power, so that small quality gaps among high-scoring candidates become learnable signals. The authors build DocPO around SAA, pairing it with element-specific rewards: normalized edit distance for text, weighted tree edit distance for tables, and a syntax-gated rubric-plus-edit reward for formulas. With an unmodified 3B vision-language backbone and no post-processing, the method reports strong results on both OmniDocBench and a newly constructed hard-element benchmark. A sympathetic reader would take the claim to be that reward-level optimization, not architecture, is the main lever still available for high-precision document understanding.","feed_headline":"Reward sharpening lifts 3B document parser to 95.49","feed_subtitle":"A growing power-law on rewards lets a stock 3B model beat custom-encoder parsers with no post-processing.","key_machinery":"The load-bearing object is Step-Aware Annealing (SAA), a power-law reward shaper $f_\\gamma(M)=M^\\gamma$ applied to any reference reward normalized to $[0,1]$, with the curvature exponent $\\gamma$ annealed from 1 toward 9 during training. The exponential schedule's time constant is adjusted per task by the Dynamic Dispersion Controller, a rolling coefficient of variation of recent rewards that accelerates sharpening when outputs are dispersed and keeps the schedule conservative when rewards are already concentrated. SAA preserves the base reward's ranking—what counts as correct never changes—while making near-ties among top candidates easier for GRPO's within-group advantage to distinguish; the relative gap between nearby scores follows $\\rho_\\gamma=(1+\\delta/m)^\\gamma-1$, strictly increasing in $\\gamma$. Around SAA, DocPO wraps element-specific base rewards: normalized string edit distance for text, weighted tree edit distance (via the APTED algorithm) for tables, and for formulas a hard LaTeX-syntax gate multiplied by a convex combination of a zero-shot binary semantic rubric and edit-distance similarity.","core_discovery":"On its own terms, the central discovery is that reward discriminability—not model capacity, vision-encoder design, or post-processing—is the limiting factor for RL-based document parsing. Using a standard GRPO setup, the paper shows that reference-based rewards from the edit-distance family stagnate in the high-accuracy regime because near-correct outputs receive nearly identical scores. Step-Aware Annealing replaces the base reward $M$ with $M^\\gamma$, annealing $\\gamma$ from 1 toward 9 on an exponential schedule modulated by a task-wise dispersion controller; the monotonic transform preserves reward ordering while amplifying the relative gap between nearby high-scoring candidates, and the paper proves this relative gap grows monotonically with $\\gamma$. With only this reward-level change on an unmodified 3B backbone and no post-processing, the unified model attains an overall metric of 95.49 on OmniDocBench and 93.76 on DocElemHard, surpassing specialized parsers with custom visual front-ends. The authors take this as evidence that reward-level optimization is a complementary and often cheaper route to high-precision document understanding.","pith_inferences":["If SAA is truly reward-agnostic, a natural extension is to apply it to other dense-similarity-reward tasks with saturation, such as code repair, machine translation, or structured prediction; the paper does not test this, but the mechanism's only requirement is a normalized reference reward.","The formula reward leans on a 7B zero-shot rubric at training time; given the reported 97.5% rubric accuracy on 200 pairs, a smaller distilled verifier or a learned proxy could plausibly replace it and cut RL cost, an extension the paper leaves implicit.","Because SAA preserves reward ordering and does not alter exploration, the gains likely come from sharper credit assignment rather than broader search; a head-to-head comparison against exploration-focused RL methods would clarify the division of labor.","The new DocElemHard benchmark and the 612k-patch RL pool are themselves part of the recipe; an ablation training SAA on a smaller or unfiltered pool would separate the contribution of reward annealing from that of data curation."],"forward_implications":["A standard, unmodified 3B backbone with reward-level optimization can beat specialized document parsers that rely on custom vision encoders, extra pretraining, or post-processing on the benchmarks tested.","The same annealing schedule improves text, table, and formula rewards, making SAA a general plug-in for any normalized reference-based reward rather than a per-task reward design.","The annealing schedule itself is load-bearing: fixed static power-law exponents underperform the dynamic schedule, and removing the dispersion controller costs roughly 0.28 TEDS points on OmniDocBench tables.","SAA accelerates and lifts optimization in the high-accuracy regime: the table variant crosses a 91.0 TEDS threshold about 1.8 times earlier than the non-annealed baseline and converges roughly 1.5 points higher.","The gains concentrate on hard, visually confusable layouts—vertical tables, spanning-cell tables, fuzzy scans, watermarks, and mixed-language blocks—where near-correct outputs are hardest to distinguish."],"supporting_citations":[{"why":"Defines the Qwen2.5-VL backbone that DocPO leaves unchanged at 3B and 7B scale.","marker":"[1]"},{"why":"OmniDocBench is the main public benchmark whose NED, TEDS, and CDM metrics double as the RL rewards.","marker":"[21]"},{"why":"Supplies the APTED algorithm used to compute the weighted tree edit distance table reward.","marker":"[22]"},{"why":"Provides the memory-efficient tree edit distance variant used for the table reward.","marker":"[23]"},{"why":"Serves as a binary unit-test reward RL baseline compared under the same element-level protocol.","marker":"[25]"},{"why":"Serves as an edit-distance RL parser baseline whose string-level reward motivates element-specific rewards.","marker":"[27]"},{"why":"Specialized VLM with a custom encoder, used as a comparison in the main table and in drill-down analyses.","marker":"[30]"},{"why":"Specialized compact VLM with dynamic-resolution encoder and post-processing, the strongest listed competitor on overall scores.","marker":"[3]"},{"why":"Learned render-and-compare reward model for tables, contrasted with the reference-based reward approach.","marker":"[34]"},{"why":"Precedent for adaptively emphasizing hard samples in detection training, related to the dispersion-based scheduling in DDC.","marker":"[31]"}],"fun_headline_variants":["Reward sharpening lifts 3B parser to 95.49","Annealed rewards beat custom encoders in document parsing","Reward curvature annealing unlocks 3B parsing accuracy","Step-aware rewards sharpen document parsing without extra supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated scores used as rewards actually measure parsing quality, so making the model chase tiny differences in those scores makes it genuinely better rather than just better at the test; if that alignment fails, the reported gains could be reward hacking rather than real understanding.","fun_headline_variants_meta":{"raw":{"variants":["Reward sharpening lifts 3B parser to 95.49","Annealed rewards beat custom encoders in document parsing","Reward curvature annealing unlocks 3B parsing accuracy","Step-aware rewards sharpen document parsing without extra supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00067,"raw_usage":{"total_tokens":3060,"prompt_tokens":956,"completion_tokens":2104,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2038}},"tokens_in":572,"tokens_out":2104,"duration_ms":29873,"temperature":1.0,"reasoning_tokens":2038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:54:08.479614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: evaluate the DocPO-trained policy on a new document corpus whose ground truth is annotated by humans and scored with a metric different from the RL rewards—for example, rendered-image similarity for tables instead of HTML-tree TEDS, or human-judged semantic equivalence for formulas. If the SAA-trained model's margin over the non-annealed baseline shrinks or vanishes under that scoring, the claim that SAA improves genuine parsing quality rather than merely the trained metrics would be refuted; if the margin survives, the claim is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OmniDocBench is the main public benchmark whose NED, TEDS, and CDM metrics double as the RL rewards."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the APTED algorithm used to compute the weighted tree edit distance table reward."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the memory-efficient tree edit distance variant used for the table reward."},{"cited_title":"Progressive Hard-case Mining across Pyramid Levels for Object Detection","cited_arxiv_id":"2109.07217","evidence_quote":"Precedent for adaptively emphasizing hard samples in detection training, related to the dispersion-based scheduling in DDC."}],"review_version":2}