REVIEW 3 major objections 5 minor
DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper proposes Step-Aware Annealing, a reward sharpening mechanism that makes GRPO-style document parsing reinforcement learning effective in the high-accuracy regime, and shows that a standard 3B vision-language backbone with…
desk verdict Simple reward-sharpening annealing that consistently helps in ablations, but the SOTA claim is compromised by test-benchmark hyperparameter tuning and single-seed runs; worth peer review with a required major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Step-Aware Annealing (SAA), a power-law reward shaper $f_\gamma(M)=M^\gamma$ applied to any reference reward normalized to $[0,1]$, with the curvature exponent $\gamma$ annealed from 1 toward 9 during training. The exponential schedule's time constant is adjusted per task by the Dynamic Dispersion Controller, a rolling coefficient of variation of recent rewards that accelerates sharpening when outputs are dispersed and keeps the schedule conservative when rewards are already concentrated. SAA preserves the base reward's ranking—what counts as correct never changes—while making near-ties among top candidates easier for GRPO's within-group advantage to distinguish; the relative gap between nearby scores follows $\rho_\gamma=(1+\delta/m)^\gamma-1$, strictly increasing in $\gamma$. Around SAA, DocPO wraps element-specific base rewards: normalized string edit distance for text, weighted tree edit distance (via the APTED algorithm) for tables, and for formulas a hard LaTeX-syntax gate multiplied by a convex combination of a zero-shot binary semantic rubric and edit-distance similarity.
What would settle it
A concrete test: evaluate the DocPO-trained policy on a new document corpus whose ground truth is annotated by humans and scored with a metric different from the RL rewards—for example, rendered-image similarity for tables instead of HTML-tree TEDS, or human-judged semantic equivalence for formulas. If the SAA-trained model's margin over the non-annealed baseline shrinks or vanishes under that scoring, the claim that SAA improves genuine parsing quality rather than merely the trained metrics would be refuted; if the margin survives, the claim is supported.
Extended reading notes
Core claim
On its own terms, the central discovery is that reward discriminability—not model capacity, vision-encoder design, or post-processing—is the limiting factor for RL-based document parsing. Using a standard GRPO setup, the paper shows that reference-based rewards from the edit-distance family stagnate in the high-accuracy regime because near-correct outputs receive nearly identical scores. Step-Aware Annealing replaces the base reward $M$ with $M^\gamma$, annealing $\gamma$ from 1 toward 9 on an exponential schedule modulated by a task-wise dispersion controller; the monotonic transform preserves reward ordering while amplifying the relative gap between nearby high-scoring candidates, and the paper proves this relative gap grows monotonically with $\gamma$. With only this reward-level change on an unmodified 3B backbone and no post-processing, the unified model attains an overall metric of 95.49 on OmniDocBench and 93.76 on DocElemHard, surpassing specialized parsers with custom visual front-ends. The authors take this as evidence that reward-level optimization is a complementary and often cheaper route to high-precision document understanding.
Load-bearing premise
The load-bearing premise is that the automated scores used as rewards actually measure parsing quality, so making the model chase tiny differences in those scores makes it genuinely better rather than just better at the test; if that alignment fails, the reported gains could be reward hacking rather than real understanding.
Editorial extensions
If this is right
- A standard, unmodified 3B backbone with reward-level optimization can beat specialized document parsers that rely on custom vision encoders, extra pretraining, or post-processing on the benchmarks tested.
- The same annealing schedule improves text, table, and formula rewards, making SAA a general plug-in for any normalized reference-based reward rather than a per-task reward design.
- The annealing schedule itself is load-bearing: fixed static power-law exponents underperform the dynamic schedule, and removing the dispersion controller costs roughly 0.28 TEDS points on OmniDocBench tables.
- SAA accelerates and lifts optimization in the high-accuracy regime: the table variant crosses a 91.0 TEDS threshold about 1.8 times earlier than the non-annealed baseline and converges roughly 1.5 points higher.
- The gains concentrate on hard, visually confusable layouts—vertical tables, spanning-cell tables, fuzzy scans, watermarks, and mixed-language blocks—where near-correct outputs are hardest to distinguish.
Reading between the lines
- If SAA is truly reward-agnostic, a natural extension is to apply it to other dense-similarity-reward tasks with saturation, such as code repair, machine translation, or structured prediction; the paper does not test this, but the mechanism's only requirement is a normalized reference reward.
- The formula reward leans on a 7B zero-shot rubric at training time; given the reported 97.5% rubric accuracy on 200 pairs, a smaller distilled verifier or a learned proxy could plausibly replace it and cut RL cost, an extension the paper leaves implicit.
- Because SAA preserves reward ordering and does not alter exploration, the gains likely come from sharper credit assignment rather than broader search; a head-to-head comparison against exploration-focused RL methods would clarify the division of labor.
- The new DocElemHard benchmark and the 612k-patch RL pool are themselves part of the recipe; an ablation training SAA on a smaller or unfiltered pool would separate the contribution of reward annealing from that of data curation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Step-Aware Annealing (SAA), a power-law reward sharpening mechanism for GRPO-style reinforcement learning in document parsing. Base rewards are element-specific: NED for text, weighted TEDS for tables, and a syntax-gated Rubric+edit reward for formulas. SAA raises the exponent gamma over training according to an adaptive schedule controlled by a Dynamic Dispersion Controller. The authors instantiate this in the DocPO framework on a Qwen2.5-VL-3B backbone and report improved text, table, and formula metrics on OmniDocBench and their self-constructed DocElemHard benchmark, including state-of-the-art scores without post-processing or vision-encoder modification. The supplementary material contains ablation studies, hyperparameter sensitivity analyses, external benchmark results on WikiTableSet and UniMERNet, and formal propositions on the reward-sharpening transform, together with an explicit caveat that these propositions do not guarantee monotonic growth of the GRPO advantage.
Significance. If the empirical claims hold under a properly controlled protocol, the contribution is practically valuable: SAA is a simple, reward-agnostic technique that could improve RL-based document parsing without architectural changes, and the paper documents its recipe, prompts, and dataset construction in unusual detail. The theoretical propositions in the supplement are correct but modest, and the authors honestly state that they describe raw reward spacing, not advantage dynamics. The paper's strengths include a public code link, detailed ablations isolating SAA, external transfer benchmarks, and explicit limitations acknowledging that proxy rewards can be gamed. The main weakness is that the headline numbers are currently tied to hyperparameters selected on the same benchmarks used for final reporting, and all results appear to come from single-seed runs; the significance of the central claim is therefore conditional on additional validation.
major comments (3)
- [§3.4, §B.7, §B.9] The final hyperparameter recipe is selected on the evaluation benchmarks. Table 14 reports OmniDocBench and DocElemHard overall scores for Δγ = 0, 4, and 8 and labels Δγ = 8 as DocPO; Table 15 selects α = 0.8, β = 0.2 on OmniDocBench CDM. The statement in B.9 that no final benchmark test set is used for early stopping or model selection does not exclude using those same benchmarks to choose hyperparameters. Since Table 3 then reports state-of-the-art results on exactly these benchmarks, the central claim that SAA 'consistently improves' GRPO-style RL is partly a claim about a configuration selected on the evaluation distribution. Please add an independent validation split for hyperparameter selection, re-report final numbers under that protocol, or explicitly restrict the SOTA claim to the selected configuration with the selection disclosed as part of the result.
- [§4.1, Table 3] All RL runs appear to be single-seed. No random seed is reported, and no error bars, confidence intervals, or significance tests are given for the central comparisons in Tables 2, 3, and 4, or for the supplementary tables. Headline gains such as text NED 0.0238→0.0125 on OmniDocBench and formula CDM 87.69→92.88 on DocElemHard could be within run-to-run variance, especially given the small formula subset in DocElemHard. Please report results over at least three seeds with mean and standard deviation, or provide a clear justification for why seed variance is negligible.
- [§3.1–3.3, §4.1, Limitations] The evaluation metrics are the same quantities, or the same metric family, as the RL rewards: NED for text, TEDS for tables, and edit-distance/rubric signals for formulas. The Limitations section correctly concedes that proxy rewards can be gamed or miss semantic nuances. The external benchmarks in B.5 provide some transfer evidence, but they are reported for a single configuration and do not rule out overfitting to the metric family. Please add a reward-hacking analysis or an independent assessment of parsing quality, such as human spot-checks or a held-out metric not used as a reward, to support the claim that the gains reflect genuine parsing quality rather than metric exploitation.
minor comments (5)
- [§3.4, Eq. (6)] The window size s_window is described as 'set to 3' without specifying the unit; please clarify whether it is measured in training steps, gradient updates, or mini-batches, and how this interacts with the global batch size of 128.
- [§4.2, Figure 7] The training-progress plot appears to show a single run for each method; if multiple seeds are added, this figure should include error bands or representative individual curves so the '~1.8x faster' and '+1.5 points' claims can be assessed for stability.
- [§B.3] In the matched comparison with peer RL parsers, the text says all models 'receive the same cropped patches'; please state explicitly whether the peer models were only evaluated on these patches or were also fine-tuned/RL-trained on the same training patches, since this affects the interpretation of the comparison.
- [§B.6, Table 12] The table drill-down in Table 7 includes the caution that vertical and structured-text subsets are small (n=6 and n=15), but the formula and text drill-downs in Table 12 do not report subset sample sizes; please add them or add a comparable caution.
- [§3.4 and §B.7] The notation for the base time constant is inconsistent: Eq. (6) uses τ, while B.7 refers to τ_base; please unify the notation to avoid confusion.
Circularity Check
The headline OmniDocBench/DocElemHard results are partly fitted: B.7 selects Δγ=8 and α=0.8/β=0.2 on these final benchmarks and then reports the same benchmarks as SOTA in Table 3. The SAA reward-shaping math itself is self-contained.
-
fitted input called prediction
[Supplementary B.7, Tables 14–15; main Section 3.4 and Table 3]
"Table 14: Sensitivity to Δγ on element-level parsing (Overall ↑). Δγ OmniDocBench DocElemHard 0 (w/o SAA) 94.42 90.53 4 94.87 91.65 8 (DocPO) 95.49 93.76 ... The chosen setting, α=0.8 and β=0.2, achieves the highest score."
The final DocPO configuration (Δγ=8, α=0.8, β=0.2) is selected by taking the maximum observed score on OmniDocBench, and for Δγ also on DocElemHard, in Tables 14–15. The same two benchmarks then appear as the headline SOTA evaluation in main Table 3. Therefore the reported 'consistent improvement' and the 95.49/93.76 numbers are, at least in part, the argmax of the tested hyperparameter grid on the evaluation distribution, not an independent prediction from a fixed recipe. This is a fitted-input-called-prediction pattern: the reported result is the selection criterion itself. The SAA math in Section 3.4 and B.10 is not circular; the circularity is confined to the empirical claim that this exact configuration beats all baselines on these benchmarks.
-
other
[Supplementary B.9]
"No final benchmark test set is used for early stopping or model selection. The stabilization criterion in the main paper refers to a held-out validation split, composed of hard cases and separated from the training pool before RL. It covers mixed-script or noisy text, span-heavy tables, and long or multi-line formulas. OmniDocBench and DocElemHard are used only for final reporting."
B.7 reports test-set sensitivity for Δγ and (α,β) on OmniDocBench/DocElemHard and adopts the best-scoring values as DocPO, so the statement that these benchmarks are used only for final reporting is false as written. The final configuration was explicitly model-selected on the same benchmarks that produce the headline numbers, which converts the supposedly independent final evaluation into a selected-best evaluation. This does not change the mathematics of SAA, but it undermines the strength of the empirical claim derived from those numbers.
full rationale
The core Step-Aware Annealing derivation is self-contained and not circular: Proposition B.1 and Proposition B.2 follow directly from the definition f_γ(M)=M^γ, and the analytic claim that increasing γ amplifies relative reward gaps is a straightforward consequence of the power-law transform. Using NED/TEDS/CDM-style metrics as RL rewards is an intended optimization setup rather than circularity, though it carries reward-hacking risk as the Limitations section concedes. The main circularity is empirical and concrete: Supplementary B.7 selects the final hyperparameters (Δγ=8, α=0.8, β=0.2) by taking the best scores on OmniDocBench and DocElemHard, and main Table 3 then reports those same benchmarks as evidence of state-of-the-art performance. Supplementary B.9 explicitly claims no final benchmark is used for model selection, which is inconsistent with B.7. The external WikiTableSet and UniMERNet results in B.5 provide some independent transfer evidence and show the method is not fabricated, but they do not cure the test-set selection issue for the headline OmniDocBench/DocElemHard numbers. Overall score 6 reflects that one central reported result is partly fitted by construction on the evaluation benchmarks, while the proposed mechanism itself retains independent analytical content.
Assumptions & free parameters
free parameters (6)
- gamma_init (SAA starting exponent) =
1
- Delta_gamma (SAA maximum curvature range) =
8
- tau_base (exponential time constant) =
roughly half of total training steps
- DDC window size s_window =
3
- Formula reward weights alpha and beta =
0.8 and 0.2
- APTED structural and content costs =
2 for structural ops, 1 for content mismatches
assumptions (5)
- standard math A monotone power-law transform of a normalized reward preserves the argmax ranking within a rollout group
- standard math GRPO advantage normalization with no KL constraint is a valid optimization target
- domain assumption Qwen2.5-7B-Instruct zero-shot rubric reliably judges semantic equivalence of LaTeX formulas
- domain assumption NED, TEDS, and CDM are faithful proxies for document parsing quality
- domain assumption Pre-filtering samples with all-perfect or all-zero rollouts does not bias RL
Cite this review
Pith. "Pith review of DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards." pith.science (2026). https://pith.science/paper/GG7EXLYF
@misc{pith2026260800536,
author = {Pith},
title = {Pith review of: DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards},
year = {2026},
howpublished = {\url{https://pith.science/paper/GG7EXLYF}},
note = {Machine review of arXiv:2608.00536}
}
read the original abstract
Reinforcement learning (RL) for document parsing often relies on reference-based rewards rooted in edit distance (e.g., tree edit distance), yet it remains hard to optimize in the high-accuracy regime because such rewards become weakly discriminative: near-correct outputs receive very similar scores, providing limited learning signal for hard cases. We propose Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning. Built on SAA, we introduce DocPO, a document policy optimization framework with element-specific, reference-based rewards anchored by edit-distance signals: normalized string edit distance (NED) for text, tree edit distance similarity (TEDS) for tables, and a hybrid Rubric+edit reward for formulas. Experiments on OmniDocBench and DocElemHard show that SAA consistently improves GRPO-style RL across document elements over non-annealed rewards, without requiring additional human supervision for reward construction.
Figures
Figures from the paper (5 more)
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.