{"id":"85660e2d-1d64-4edd-9842-f4528d34f7a4","arxiv_id":"2608.06607","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-inference routing saves cost only when many documents are hard for the cheap model and those hard cases are visible in document features; a pilot test identifies such genres, with 31-77% savings where both hold.","lead":"This paper tests whether simple document features can predict which documents are hard to extract, so a cheap model handles easy cases and a costly model is reserved for hard ones. It offers a two-part checklist to decide before deployment whether routing will save money, with reported savings of 31-33% on receipts and 77% on degraded forms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-draw F1 labels sit on a noise floor equal to the 0.02 tolerance, so the quality constraint and headline savings need a de-noised re-run before the central claim is established.","rationale":"The paper's central quantitative claim is that routing reduces cost by 31-33% on receipts and 77% on DeepForm while keeping quality within 0.02 F1 of always-large. This claim depends on labels and quality values that are each derived from a single stochastic extraction. The paper itself quantifies the per-document F1 standard deviation as about 0.02, the same size as the tolerance and the label threshold. That makes the quality constraint in Eq. (4) satisfiable by noise, especially because the threshold is selected on the same noisy labels used to evaluate the constraint. The reported bootstrap intervals for receipt savings include values below the 30% success criterion, and DeepForm savings have no interval at all. The stability of the threshold-free AUC across tau does not address this, because AUC measures ranking quality while the savings claim depends on the exact boundary and the tolerance constraint. A de-noised replication with multiple draws per document, or a noise-resampling simulation based on the existing pilot, would settle whether the headline savings and quality guarantee survive. This is the same concern the reader identified as the weakest assumption, so I agree. The reader's CONDITIONAL verdict already captures this; no change to the verdict is needed. I considered the small-sample pilot prediction (n=2) as an alternative, but it is explicitly anecdotal and less central than the quantitative cost-quality claim. The paper's negative results on invoices and nutrition labels, and its conservative canonical scorer, are genuine supporting evidence that routing is not universally beneficial; the remaining risk is specifically the measurement noise floor at the tolerance, not the direction of the findings.","tokens_in":13131,"tokens_out":5593,"duration_ms":55535,"concrete_test":"Re-run the full router pipeline (feature extraction, oracle labeling, RF training, no-peek threshold selection, and test evaluation) on the CORD, SROIE, and DeepForm test sets using k >= 5 repeated extractions per (tier, document), averaging F1 per document to form de-noised gap labels and de-noised quality estimates. Recompute the headline savings subject to the de-noised constraint Q(t) >= Q_large - delta. If the de-noised saving falls below 30% or the de-noised quality deficit exceeds 0.02, the headline claim is not robust to decoding noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that every F1 gap, oracle label, and the quality constraint in Eq. (4) are computed from one stochastic extraction per (tier, document) at the provider's default sampling (Sec. IV-D, Sec. VI). The paper's own re-extraction pilot measures per-document F1 std approx 0.02, exactly equal to the tolerance delta=0.02 and the label threshold tau=0.02. Near-threshold labels are therefore noisy: about 20% of documents within 0.10 F1 of tau flip under averaging. The no-peek threshold in Sec. V-F is selected on training labels to satisfy Q(t) >= Q_large - delta on these same single-draw F1s. Because the same single draws define both the labels and the evaluation quality, the measured 'within 0.02 F1' can be achieved by decoding noise rather than by genuine expected quality: the threshold optimizer can pick a point that meets the constraint on the realized draw even if the true expected Q(t) is lower. The reported savings (33% CORD, 31% SROIE, 77% DeepForm) come without a check of whether the constraint holds under averaged F1; the bootstrap CIs on receipts savings ([26,42]% and [27,35]%) include values below the 30% success criterion, and DeepForm has no CI. The threshold-free AUC stability across tau (Table X) does not settle this, because the cost claim depends on the boundary and the tolerance constraint, not only on ranking quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies pre-inference routing for structured document field extraction: choose a cheap or expensive LLM per document based on 13 intrinsic features (OCR quality, image quality, layout, content/structure) before any extraction call. Documents are labeled large-required when the per-document F1 gap exceeds tau=0.02; a calibrated random forest estimates P(large-required), and the operating threshold is selected to minimize cost subject to Q(t) >= Q_large - delta with delta=0.02. Experiments on CORD/SROIE receipts, DeepForm ad-buy forms, DocILE invoices, POIE nutrition labels, and VRDU report that routing saves 31-33% on receipts and 77% on DeepForm, does not help on invoices/nutrition/VRDU, does not transfer across genres, and is robust to model pair (5x and 3x cost ratios), to canonical vs strict scoring, and to a tau sweep. The paper also compares pre-inference routing with confidence cascades, reports a no-peek threshold-transfer protocol, and releases code.","tokens_in":13444,"tokens_out":10394,"duration_ms":94491,"significance":"If the savings figures survive scrutiny, the paper makes a practical contribution: a two-condition diagnostic (routing headroom and feature-detectable difficulty) with conservative field-type canonical scoring, a threshold-free AUC as a selection-free metric, and explicit negative results on genres where routing fails. Strengths include held-out bootstrap CIs on the receipt savings, threshold-transfer checks, replication on a second model pair, and the observation that a bag-of-words router matches engineered features, which supports the claim that the genre, not the router design, is the binding constraint. The main gap is statistical: every F1 label and the quality constraint in Eq. (4) are computed from a single stochastic extraction per (tier, document), and the tolerance delta equals the measured per-document F1 noise, so the central cost-savings claim is not yet fully established. The DeepForm 77% headline also lacks uncertainty quantification, and the relationship between the no-peek and frontier SROIE savings is inconsistent across tables.","major_comments":[{"comment":"All oracle labels, training targets, and the quality constraint in Eq. (4) are computed from one stochastic extraction per (tier, document) at the provider's default sampling. The paper's own re-extraction pilot reports per-document F1 standard deviation of about 0.02 and about 20% label flips within 0.10 F1 of tau under averaging; since tau=delta=0.02, the label boundary sits on this noise floor. The no-peek threshold in Sec. V-F is selected on training predictions to satisfy Q(t) >= Q_large - delta using the same single draws that define the labels, so the constraint can be satisfied on the realized draw even if the expected Q(t) is below the tolerance. AUC stability across tau (Table X) does not resolve this, because the cost claim depends on the operating point and the boundary. Please re-run with averaged F1 over several draws for both labels and evaluation, or provide an explicit bound showing the worst-case expected Q(t) under the measured noise, before the headline savings are reported.","section":"Sec. IV-D and Sec. VI"},{"comment":"Both CORD and SROIE are described as clearing the predefined success criterion (cutting cost by at least 30% within 2 F1 points), but the reported bootstrap 95% CIs are [26,42]% for CORD and [27,35]% for SROIE; both include values below 30%. The point estimates clear the criterion, but the uncertainty does not rule out failing it. Please report the fraction of bootstrap resamples meeting the >=30% criterion, or present the success claim together with the CI explicitly stated.","section":"Sec. V-C, Table VII"},{"comment":"At tau=0.02, Table VII reports a SROIE saving of 31%, while Table X reports 39%, Table IX lists the RF route as 39%, and Sec. V-F calls 39% the achievable frontier and 31% the no-peek threshold result. If Table VII is meant to report the no-peek threshold-transfer saving, the caption must state this; as written, a reader sees a direct contradiction among the headline tables. Please harmonize the labels so that each table states explicitly whether the saving is the evaluation-set frontier value or the blind-threshold value.","section":"Table VII vs Table X and Sec. V-F"},{"comment":"The DeepForm headline of 77% cost saving is reported on a 100-document held-out test without a confidence interval, and the observed quality gap is exactly 0.020, equal to delta; with the noise floor described in Sec. VI, this point estimate is fragile. Please provide a bootstrap CI for the saving and a CI for the quality-constraint margin, or present the 77% result as preliminary rather than as a headline number.","section":"Table VIII"}],"minor_comments":[{"comment":"The claim that a small labeled pilot can predict whether routing will work is based on only two positive pilot runs; please present this as an anecdotal illustration, or add at least one pilot on a genre where the diagnostic predicts routing should fail.","section":"Abstract and Introduction"},{"comment":"The statement that setting tau=delta is for simplicity should be clarified: delta is a user-chosen deployment tolerance, while tau is a modeling choice, and a mismatch between them would change the training labels. The paper should state explicitly that the reported numbers hold for delta=0.02 and that delta was not swept.","section":"Sec. IV-C"},{"comment":"The DeepForm feature-signal-gated pilot is mentioned as confirming both conditions, but no pilot numbers (headroom, pilot AUC) are reported; please include the pilot results so the reader can assess the two-case pilot claim.","section":"Sec. V-D"},{"comment":"The DeepForm CI for the RF-minus-text difference is reported in the text but not in the table; moving it to the table would avoid confusion.","section":"Table XI"},{"comment":"The term half-tolerance generalization buffer is used for the DeepForm threshold selection but is never defined; please define it or replace it with the explicit threshold-selection rule.","section":"Sec. V-D"},{"comment":"Several references contain stray spaces in author initials (for example, refs. 5 and 12 print V . with a space); please clean up the bibliography formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest, well-scoped, and already includes several conservative design choices, which is to its credit. The single-draw F1 noise issue is the main barrier and, as the skeptic notes, it is load-bearing for the cost-savings claim. I do not see grounds for rejection: the threshold-free AUC, threshold-transfer checks, and two-model-pair replication suggest the qualitative finding is likely correct, but the noise floor equals the tolerance, so the quantitative headline needs either averaged re-extraction or a rigorous noise bound. The discrepancy between Table VII and Table X is likely a reporting confusion rather than a deep flaw, but it must be fixed. Given the scope of the required statistical re-run, major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the two-condition diagnostic: routing pays only when the cheap model fails often enough (headroom) and those failures are predictable from intrinsic page features. That framing, plus the negative results, is the real contribution. The paper tests five genres and is admirably candid: no transfer across genres (CORD to SROIE at 0.55 AUC), bag-of-words routing matching engineered features, and a canonical scorer that lowers the headline AUC and savings. The field-type canonical scoring, leakage-resistant feature set, no-peek threshold transfer, and replication across two model pairs at 5x and 3x cost ratios are real strengths. The negative results are not buried; they are the argument that the genre, not the router, is the binding constraint.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing for the cost claim. Every F1 gap, oracle label, and the quality constraint in Eq. (4) come from a single stochastic extraction per (tier, document). The paper's own re-extraction pilot finds per-document F1 std ~0.02, exactly equal to the tolerance delta and the label threshold tau. So the threshold optimizer can satisfy the quality constraint on the realized draw even when the expected quality is lower. The receipt savings CIs ([26,42]% and [27,35]%) bleed below the 30% success criterion, DeepForm's 77% has no CI on 100 documents, and the AUC stability across tau does not settle the boundary-dependent cost claim. This does not sink the diagnostic, but the headline \"within 0.02 F1\" needs a de-noised re-run (averaged F1 over several draws) before the savings are established.\n\nTwo smaller items: the pilot-prediction claim rests on two successful cases, and the GitHub release should pin a commit hash so the reproduction is verifiable. Both are minor.\n\nWho this is for: practitioners running LLM-based document extraction who want a cheap pre-deployment test, and routing researchers who will find the clean negative transfer result useful. It deserves a serious referee; the central conceptual claim is supported, and the noise issue is fixable with re-runs and proper CIs. I would accept it for review.","headline":"A genuinely honest and useful paper on when pre-inference routing pays for document extraction; the headline cost numbers sit on a single-draw noise floor that needs de-noising before the savings claim is final.","tokens_in":13950,"tokens_out":1721,"would_cite":true,"duration_ms":18146,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pre-inference routing of each document to a cheap or expensive extractor pays off exactly when the cheap model fails often enough and those failures are predictable from visible page features — and that a small…","keywords":["document understanding","model routing","cost-efficient inference","information extraction","large language models","pre-inference features","extraction F1","difficulty prediction"],"falsifier":"Take a corpus the diagnostic flags as routable on a 100–200 document pilot (headroom near or above 40% and routing AUC near or above 0.7), deploy the router on a held-out set with the threshold fixed on the pilot, and measure the realized saving at the 0.02 F1 tolerance; a realized saving far below the reported scale, or a quality loss beyond tolerance, would falsify the two-condition claim. A more direct check of the measurement floor: re-extract the pilot set several times per (tier, document), average the F1, and recompute the headroom fraction — if the averaged headroom moves by more than the reported savings margins, the headline numbers sit on single-draw noise rather than on stable document difficulty.","tokens_in":12908,"feed_emoji":"🧾","tokens_out":13327,"duration_ms":97478,"temperature":0.7,"pith_summary":"Most document-extraction systems run the same model on every document. This paper asks whether a cheap model can be used for the easy ones and a strong, expensive model reserved for the few that need it, with the choice made before extraction from signals like OCR confidence, blur, and layout density. It argues that this pre-inference routing helps only when two conditions both hold: enough documents genuinely need the strong model, and that need is visible in the document itself. When both hold — photographed receipts and degraded faxed ad-buy forms — the calibrated router cuts cost by 31–33% and 77% respectively while staying within 0.02 F1 of always choosing the large model; when either condition is missing, as with digital invoices or near-ceiling nutrition labels, routing does not help. The practical payoff is a before-you-deploy test: a small labeled pilot that measures the two conditions, which predicted the outcome correctly in both cases where it was tried first.","feed_headline":"Routing cuts document-extraction cost by up to 77%","feed_subtitle":"A 13-signal pre-read of each page decides when the cheap model suffices, saving money at near-identical quality.","key_machinery":"The load-bearing object is the two-condition diagnostic, operationalized as the headroom fraction and the routing AUC. Headroom is the share of documents labeled large-required by the gap rule $y(d)=\\mathbf{1}[g(d)>\\tau]$ with $g(d)=q_\\ell(d)-q_s(d)$; feature-detectability is the cross-validated AUC of a calibrated random forest that maps 13 pre-inference features — OCR confidence and its variance, token fragmentation, blur, contrast, word-height variation, line crowding and density, aspect ratio, item density, and token count — to that label. The decision side sweeps the router's probability threshold across a cost–quality Pareto frontier, choosing the cheapest point within a tolerance $\\delta=0.02$ F1 of always using the large tier, under a token-based cost model in which the tier pair is priced at a clean 5× ratio (and a second pair at 3×). A field-type canonical scorer, which compares money as numbers, dates by components, and text by normalized exact match, keeps harmless reformatting from being counted as extraction failure.","core_discovery":"The paper's central claim is that whether document routing works for a corpus is itself predictable, and it reduces to two measurable conditions. The first is headroom: the fraction of documents whose quality gap $g(d)=q_\\ell(d)-q_s(d)$ between the expensive and cheap extraction tier exceeds a tolerance $\\tau=0.02$ F1; the second is feature-detectability: the AUC with which a router can predict that gap from 13 pre-inference features computed from the page image, OCR text, and box geometry. On six datasets spanning five genres, routing succeeds where both conditions are favorable — pooled receipts (held-out AUC 0.707; 31–33% savings) and the degraded ad-buy forms (held-out AUC 0.916; 77% savings at within-0.02-F1 quality) — and fails where either is absent. The paper also establishes two negative results that delimit the method: a bag-of-words text router matches the engineered features, so the bottleneck is the genre rather than the router, and the router does not transfer across datasets, even within a genre, so it must be refit per deployment.","pith_inferences":["The two-condition diagnostic may be the more durable result than the router: it can serve as a cheap go/no-go gate before committing to routing, cascading, or fine-tuning on any new corpus.","Because a bag-of-words text router matches the engineered image and layout features, detectable difficulty seems to live in the document's surface lexicon and structure rather than in fine-grained image quality; a testable extension is whether an OCR-free vision-language extractor, which sees the page directly, changes the headroom and predictability on genres the diagnostic currently calls unrout","The paper's own measurement floor (per-document F1 standard deviation about 0.02, with roughly 20% of near-threshold documents flipping label under averaging) implies that a few re-extractions per document on the pilot would sharpen the headroom estimate at trivial cost, and headroom should ideally be reported with its own confidence interval."],"forward_implications":["On a new corpus, a small labeled pilot that measures headroom and feature-detectability tells you in advance whether pre-inference routing will save money; both pilot predictions the paper ran were correct.","Where the diagnostic says route, the savings are substantial at the stated quality tolerance: 31–33% on receipts and 77% on degraded ad-buy forms, with the no-peek threshold check confirming the saving is not a test-set artifact.","Pre-inference routing is strictly cheaper than a confidence cascade under full-document re-extraction, because a cascade pays the cheap extraction on every document before deciding; where escalation is frequent, routing beats even a perfect-trigger oracle cascade.","The router must be refit per dataset: transfer fails across genres and even across two receipt datasets, so a deployed router needs its own labeled sample on the target corpus.","The effect persists across two model pairs with 5× and 3× cost ratios, so the savings are not an artifact of one particular price gap."],"supporting_citations":[{"why":"One of the two receipt datasets; pooled with SROIE to produce the held-out AUC 0.707 router and the 33% CORD saving.","marker":"[17]"},{"why":"Second receipt dataset; supplies the 31% no-peek saving and the bimodal gap distribution routing exploits.","marker":"[18]"},{"why":"The degraded faxed ad-buy form corpus, the main non-receipt positive case where routing saves 77% at the stated quality tolerance.","marker":"[21]"},{"why":"The low-headroom registration-form corpus whose near-zero mean gap shows a domain that rarely needs the large tier.","marker":"[20]"},{"why":"The invoice corpus supplying the negative case: ample headroom but no feature-predictable signal (within-genre AUC 0.52).","marker":"[23]"},{"why":"The nutrition-label corpus supplying the other negative case: predictable features but almost no headroom (16%).","marker":"[24]"},{"why":"The cascade-style baseline the paper claims to be strictly cheaper than under full-document re-extraction.","marker":"[10]"},{"why":"The learned weak-to-strong binary router that decides after generation, defining the post-inference approach the paper contrasts with pre-inference routing.","marker":"[11]"}],"fun_headline_variants":["Predict-and-route: 77% cheaper extraction when conditions align","Routing success hinges on two measurable conditions","Pre-flight check: when the cheap extractor suffices","77% savings in document extraction with smart routing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each document's difficulty label is the F1 gap between two single stochastic extraction runs, and the 0.02 tolerance sits inside the measured decoding noise (per-document F1 standard deviation about 0.02), so near-threshold labels, and the savings measured against them, rest on a noisy measurement floor.","fun_headline_variants_meta":{"raw":{"variants":["Predict-and-route: 77% cheaper extraction when conditions align","Routing success hinges on two measurable conditions","Pre-flight check: when the cheap extractor suffices","77% savings in document extraction with smart routing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1375,"prompt_tokens":1049,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":665,"tokens_out":326,"duration_ms":3984,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:16:35.330105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus the diagnostic flags as routable on a 100–200 document pilot (headroom near or above 40% and routing AUC near or above 0.7), deploy the router on a held-out set with the threshold fixed on the pilot, and measure the realized saving at the 0.02 F1 tolerance; a realized saving far below the reported scale, or a quality loss beyond tolerance, would falsify the two-condition claim. A more direct check of the measurement floor: re-extract the pilot set several times per (tier, document), average the F1, and recompute the headroom fraction — if the averaged headroom moves by more than the reported savings margins, the headline numbers sit on single-draw noise rather than on stable document difficulty.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two receipt datasets; pooled with SROIE to produce the held-out AUC 0.707 router and the 33% CORD saving."},{"cited_title":"Huang, K","cited_arxiv_id":null,"evidence_quote":"Second receipt dataset; supplies the 31% no-peek saving and the bimodal gap distribution routing exploits."},{"cited_title":"Svetlichnaya","cited_arxiv_id":null,"evidence_quote":"The degraded faxed ad-buy form corpus, the main non-receipt positive case where routing saves 77% at the stated quality tolerance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The low-headroom registration-form corpus whose near-zero mean gap shows a domain that rarely needs the large tier."},{"cited_title":"ˇSimsa, M","cited_arxiv_id":null,"evidence_quote":"The invoice corpus supplying the negative case: ample headroom but no feature-predictable signal (within-genre AUC 0.52)."},{"cited_title":"Kuang, W","cited_arxiv_id":null,"evidence_quote":"The nutrition-label corpus supplying the other negative case: predictable features but almost no headroom (16%)."}],"review_version":1}