{"id":"91858b7f-feb3-47a5-b8c5-f3fb2d652193","arxiv_id":"2607.21780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new Bangla–English benchmark shows vision-language models can cluster pages of shuffled government-form packets but cannot reliably reconstruct their original page order.","lead":"The authors built a new benchmark of 1,950 document packets from Bangladeshi government forms, mixing Bangla and English pages, and asked five leading vision-language models to split each packet back into its original documents. The models grouped pages into the right documents fairly reliably, but failed to reconstruct the original page order whenever pages were shuffled — a gap that better instructions and language only partially close.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Khondo's ordering bottleneck may be partly task underdetermination: ground-truth order is taken from unverified source PDFs and ~2.2% of pages lack image-only order cues, so a human upper-bound check is needed.","rationale":"The paper's central empirical claim—that page-order reconstruction, not clustering, is the primary bottleneck in vision-based packet splitting—depends on the assumption that Khondo's ground-truth order is well-defined and recoverable from page images. The reader's weakest_assumption identifies exactly this: source PDF order is taken as ground truth without verification, and a nontrivial fraction of pages have no image-only order cues. I agree that this is the most load-bearing concern. If the task is partly underdetermined, then the measured ordering deficit conflates model failure with benchmark ambiguity, and the claim that the difficulty is 'genuine' rather than an artifact of prompt wording or data construction is not fully supported. The paper's own A.4 acknowledgment of ~2.2% Uncertain pages, combined with the lack of any human upper-bound measurement, makes this a real risk rather than a hypothetical. A human-ordering experiment would settle it directly: high human Sord would validate the benchmark and strengthen the central claim; low human Sord would require reinterpreting the ordering results. Since the reader already reached CONDITIONAL on related framing concerns and my concern reinforces rather than overturns that verdict, I recommend UNCHANGED. I do not see a need to move to REJECT, because the dataset construction, leakage prevention, and clustering results are independently valuable, and the ordering-bottleneck conclusion may well survive the proposed test.","tokens_in":26111,"tokens_out":6160,"duration_ms":66098,"concrete_test":"Select a stratified random sample of 100 shuffled test packets (e.g., 25 each from MonoRand, PolyInt, PolyRand, and a mixed set) and have 2–3 independent human annotators reconstruct each document's page order using only the page images and the same cues listed in the OAw prompt (printed page numbers, field flow, section continuity, signatures), with no access to the source PDF order. Compute Sord against Khondo ground truth and inter-annotator agreement. If human Sord is ≈0.95 or higher with high agreement, the ordering bottleneck is a genuine model limitation. If human Sord is materially lower (e.g., <0.85) or agreement is low on a substantial subset, Khondo's ordering ground truth is partly underdetermined, and the paper should report an ambiguity-adjusted ceiling or exclude such pages from the ordering metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MLLMs genuinely struggle with page-order reconstruction and that this is a model limitation rather than a prompting artifact—requires Khondo's ground-truth page order to be recoverable from page images alone. §4.1 defines ground truth as the order of pages in the source PDF from forms.portal.gov.bd, recorded at assembly time, but the paper does not verify that these PDFs are internally correctly ordered, nor that each page image carries legible order cues (printed page numbers, continuation of fields/sections, signatures). A.4 concedes that ~2.2% of pages are labeled Uncertain—legacy non-Unicode fonts (Bijoy/SutonnyMJ), maps/diagrams, blank/near-empty scans—and that these pages are retained in packets. With mean packet length 14.6 pages, roughly a quarter of packets contain at least one such page; for a document whose pages include a map, a blank continuation sheet, or a legacy-font page with unreadable glyphs, no image-only cue fixes its position. If a nontrivial share of documents have underdetermined order, the measured Sord deficit on MonoRand/PolyInt/PolyRand overstates model failure: part of what is labeled 'ordering bottleneck' is irreducible ambiguity in the benchmark itself. The paper asserts in A.4 that these pages 'do not materially affect' the ordering ground truth, but no human-ceiling measurement supports that assertion, and without it the headline conclusion remains conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Khondo, a benchmark for document packet splitting on Bangladeshi government forms, containing 1,950 packets over five concatenation schemes (MonoSeq, MonoRand, PolySeq, PolyInt, PolyRand), with ground-truth document boundaries, domain labels, and page order recorded at assembly time. The authors evaluate five zero-shot MLLMs, fine-tune three/four small open-weight MLLMs, and run two controlled analyses: a prompt-sensitivity study (order-aware vs. order-agnostic prompts) and a cross-lingual study (Bangla-only vs. English-only packets). Their main finding is that MLLMs cluster pages into source documents reasonably well but fail to reconstruct original page order once pages are shuffled; the ordering gap is reduced, but not eliminated, by explicit ordering instructions, and is larger for Bangla than for English packets.","tokens_in":26373,"tokens_out":4754,"duration_ms":51389,"significance":"If the results hold, Khondo is a valuable first benchmark for vision-native packet splitting in a low-resource language. The construction has clear strengths: form-level disjoint train/validation/test splits prevent content leakage, domain balance is within ~1.6 pp of uniform, ground truth is recorded at assembly time, language labels use blind dual-human review with Cohen's κ=0.90, and the dataset/code are public. The paper also ships a scaling study for fine-tuned Qwen3-VL. However, the two headline controlled analyses contain load-bearing framing flaws — the 'order-agnostic' prompt actually contains an order instruction, and the cross-lingual comparison does not hold language fixed — so the central interpretive claims require additional work before the benchmark's conclusions can be accepted as stated.","major_comments":[{"comment":"The OAg prompt is not order-agnostic. Figure 5 explicitly states: 'page_ordinals within each subdocument must be in the original page order.' This is an instruction about page order. The paper's claim that the OAg prompt 'contains no instruction regarding page order' is therefore false, and the paired comparison OAw vs. OAg does not isolate the effect of order instructions. Additionally, the OAg prompt tells the model to 'Group consecutive pages from the same sub-document together,' while the OAw prompt explicitly permits non-adjacent pages; this confounds grouping constraints with ordering instructions. The conclusion that 'order-specific instructions are necessary, but not sufficient' is not supported by this experiment. The authors should re-run the prompt-sensitivity study with an OAg prompt that truly removes all references to original page order and that keeps the grouping constrai","section":"§5.4 and Fig. 5"},{"comment":"The ground-truth page order is taken as the order of pages in the source PDFs from forms.portal.gov.bd, but the paper does not verify that these PDFs are internally correctly ordered, nor that the order is recoverable from page images alone. Appendix A.4 concedes that ~2.2% of pages are labeled Uncertain (legacy non-Unicode fonts, maps/diagrams, blank/near-empty scans) and that these pages are retained in packets. With mean packet length 14.6 pages, roughly a quarter of packets contain at least one such page. For such packets, no image-only cue fixes the position of those pages, so part of the measured Sord deficit on MonoRand/PolyInt/PolyRand may reflect task underdetermination rather than model failure. The assertion in A.4 that these pages 'do not materially affect' the ordering ground truth is unsupported. The authors should provide a human-ceiling measurement of Sord on shuffled var","section":"§4.1 and A.4"},{"comment":"The definition of Sord is incomplete for impure predicted clusters. The paper says Sord compares the predicted page order with the ground-truth order over the m pages of each predicted multi-page sub-document, but if a predicted cluster merges pages from two or more ground-truth documents, there is no single ground-truth page order for that set. The manuscript does not describe how such predicted clusters are matched to ground-truth documents or how τa is computed in this case. Also, defining Sord as 1 when a packet contains no predicted multi-page sub-document can inflate the overall score for degenerate outputs (e.g., all pages as singletons). These choices affect every ordering result in Tables 2, 3, 5, and 6 and need to be specified and justified.","section":"§5.1, Eq. (1)-(2)"},{"comment":"The cross-lingual experiment is described as producing conditions that 'differ only in language,' but the Bangla-only and English-only packets are assembled from different source forms. Language is therefore confounded with form identity, layout, content, and possibly page complexity. Figure 15 shows one matched example, but the controlled comparison is not based on translated versions of the same forms. The claim that English packets are ordered 0.13–0.36 better than Bangla packets because of language requires matched or counterbalanced forms. At minimum, the authors should control for form identity (e.g., by using translated pairs) or provide evidence that the form pools are equivalent in difficulty.","section":"§5.5 and Table 6"}],"minor_comments":[{"comment":"The axis labels and some text in these figures appear as unicode-escaped tokens (e.g., '/uni00000030/...'), making them unreadable in the submitted PDF. Please replace with rendered text.","section":"Figures 2, 3, 7, 8"},{"comment":"The proprietary model names (Gemini-3.5-Flash, GPT-5.4, Qwen 3.6 Plus, etc.) and open-weight models should be accompanied by exact version identifiers and access dates for reproducibility, as API versions can change behavior.","section":"§5.2"},{"comment":"A paired permutation test is mentioned but no p-values or effect sizes are reported. Please include them in Table 5 or the caption.","section":"§A.3.1"},{"comment":"Minor typo: 'vision-native; where models operate directly on page images' should read 'vision-native, where models operate directly on page images.'","section":"Abstract"},{"comment":"The formula for Kendall's τa is written as (nc − nd)/C(m,2); this is correct only when there are no ties, but the notation should be defined explicitly and the handling of ties stated.","section":"Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The OAg prompt issue is the most serious problem: the controlled analysis at the center of the paper's prompt-sensitivity claim compares two prompts that both instruct 'original page order,' so the headline number '0.43–0.67 recovery' is not measuring what the paper says. This is fixable by re-running the API experiments with a truly order-agnostic prompt, but it is a genuine re-analysis, not a text edit. The ground-truth order verification and the Sord metric definition also need to be addressed before the benchmark's claims are fully load-bearing. The dataset construction itself is careful and the resource is likely to be useful even if the controlled analyses are revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Khondo is a real contribution — the first packet-splitting benchmark for a low-resource language, vision-native by design — and its headline finding holds up: across five flagship MLLMs and three shuffled variants, clustering stays around 0.55–0.85 while ordering falls to 0.05–0.82, versus near 0.95 on sequential packets. The paper deserves a serious referee. But the two controlled analyses that turn this into a claim about \"genuine difficulty\" have framing flaws a revision should fix.\n\nWhat's new is the dataset, and it's well built: 1,950 packets, 28,513 pages, five concatenation schemes, 14 domains, form-level disjoint splits, domain balance within 1.6 points of uniform, dual-human-reviewed language labels (κ = 0.90), ground truth recorded at assembly. The fine-tuning results — ordering improves but stays far behind clustering on shuffled variants — make the task's learnability concrete.\n\nSoft spots, in proportion. The \"order-agnostic\" prompt is mislabeled: its constraints still say \"page_ordinals within each subdocument must be in the original page order,\" contradicting the caption's claim that it contains no ordering instruction. The comparison still shows explicit re-sequencing helps, but re-label and re-analyze. The cross-lingual comparison is not clean: Bangla-only and English-only packets come from different source forms, so language is confounded with document identity. The 0.13–0.36 gap is plausible, but it's not a controlled experiment. And everything is a single run at temperature 1.0 with no error bars.\n\nThe underdetermination point is fair but not fatal. Ground truth comes from source PDFs never verified as internally ordered, and 2.2% of pages (legacy fonts, maps, near-blanks) carry no image-only order cues. The assertion in A.4 that these \"do not materially affect\" ordering comes without a human-ceiling measurement. A quick human upper-bound check would settle it. I doubt it reverses the finding — the gap is far too large for 2.2% ambiguous pages — but that claim should be evidence, not prose.\n\nMinor: the Table 3 caption calls PolyInt \"independently sampled using full-page shuffling,\" contradicting the round-robin definition in §3. And the DocSplit overlap — a Khondo co-author is first author of DocSplit — should be declared.\n\nThis is for anyone working on multimodal document understanding, page-image reasoning, or low-resource document evaluation. The benchmark will be used regardless of how the explanatory claims shake out. Send it to referees; expect revisions.","headline":"A genuinely new, carefully built benchmark whose central finding — clustering survives shuffling while page-order reconstruction collapses — is solid, but the two controlled analyses are not as clean as the prose claims, so the paper needs a revision pass, not a desk rejection.","tokens_in":26978,"tokens_out":6835,"would_cite":true,"duration_ms":65294,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new vision-native benchmark for splitting packets of Bangladeshi government forms shows that multimodal AI models can group pages into the right documents but cannot reliably reconstruct the original page order once pages are shuffled—and","keywords":["document packet splitting","page-order reconstruction","multimodal LLM evaluation","Bangla document understanding","low-resource document benchmark","vision-language models","Bangladeshi government forms","page stream segmentation"],"falsifier":"Give careful human annotators the same shuffled packet pages (images only, no text search) and measure their Sord; if humans also plateau near 0.7–0.8, the claim that models face an intrinsic ordering difficulty weakens in favor of a task that is partly impossible. A sharper test: re-score the frontier models on a Khondo subset from which all Uncertain and legacy-font pages are removed; if the English-over-Bangla ordering gap shrinks substantially, the gap is driven by unreadable script cues rather than by the ordering operation itself.","tokens_in":25891,"feed_emoji":"🗂️","tokens_out":8008,"duration_ms":70842,"temperature":0.7,"pith_summary":"The paper claims that the bottleneck in splitting scanned government-form packets is not deciding which pages belong to which document—it is reconstructing the original page order after pages have been shuffled. To make this case, it introduces Khondo, the first vision-native packet-splitting benchmark for a low-resource language: 1,950 packets assembled from 423 real Bangladeshi government forms across 14 domains, in five concatenation schemes from sequential to fully shuffled. On Khondo, flagship multimodal models cluster pages reasonably well (clustering scores of roughly 0.55–0.85 on shuffled variants) but order them poorly: the best model drops from about 0.95 to 0.68–0.82 when pages are shuffled, and small open models reach only about 0.32 even after fine-tuning. Two controlled experiments attribute part of the gap to prompt instructions (order-aware prompts recover 0.43–0.67 of the lost ordering accuracy but no more) and part to language (identical English packets are ordered 0.13–0.36 better than Bangla ones). If the finding is right, progress in low-resource document automation hinges on teaching models to read visual order cues—printed page numbers, field flow, section continuity—in Bangla, not on better page grouping.","feed_headline":"Reordering shuffled pages stumps AI form splitters","feed_subtitle":"New Bangla–English benchmark shows the real bottleneck is restoring shuffled page order, worst for Bangla.","key_machinery":"The instrument is the benchmark itself, designed to control the sources of difficulty. Khondo assembles packets from real Bangladeshi government forms under five concatenation variants: MonoSeq and PolySeq keep pages in original order (ordering is trivial), MonoRand shuffles within each document's block, PolyInt interleaves documents round-robin, and PolyRand globally shuffles. The variants isolate stressors—interleaving and shared domains stress clustering; shuffling stresses ordering. Two further controls, the paired order-aware versus order-agnostic prompts and the size-matched Bangla-only versus English-only packets, separate prompting effects from language effects. The scoring is equall","core_discovery":"The central claim, stated the way the authors would state it: on Khondo, a bilingual (Bangla–English), vision-native benchmark built from authentic Bangladeshi government forms, zero-shot multimodal language models can group pages into their source documents fairly well, but page-order reconstruction is the primary bottleneck. Clustering stays in the 0.55–0.85 range even when documents are interleaved or globally shuffled, whereas ordering collapses from Sord ≈ 0.95 on sequential packets to 0.68–0.82 for the best flagship model, and to roughly 0.05–0.15 for weaker models that effectively copy the presented order. Two paired manipulations show the deficit is not merely prompting: explicit reo","pith_inferences":["Beyond the paper: roughly 2% of Khondo pages (legacy non-Unicode Bangla fonts, maps, near-blank sheets) carry little or no image-only order cue, so part of the measured ordering gap is likely task underdetermination; a human-annotation ceiling on shuffled packets would partition the deficit into recoverable-versus-unknowable order.","Beyond the paper: the English-over-Bangla advantage might stem from models reading Bangla text as low-level texture rather than structured cues—a testable hypothesis: translate field labels or provide in-prompt OCR for Bangla pages and see whether the ordering gap closes.","Beyond the paper: the same five-variant design could be applied to other low-resource scripts to test whether the ordering bottleneck tracks script complexity or generalizes across languages.","Beyond the paper: a page's position in a form often follows deterministic cues (printed page numbers, tables continuing across pages); an order-from-layout-only evaluation could isolate whether the deficit is linguistic or purely visual."],"forward_implications":["If the ordering bottleneck is intrinsic, document-packet-splitting systems for low-resource administrative corpora should be judged primarily on their page-order reconstruction ability, not clustering accuracy.","Order-aware prompting is a necessary component of any practical deployment—without it models simply 'copy the input order'—but deployment will also need trainable order-reconstruction capacity.","Because English packets are ordered 0.13–0.36 better than Bangla ones, English-only evaluation overstates readiness for the real Bangla-dominant distribution; benchmarks must preserve the natural language skew.","Fine-tuning small open MLLMs lifts shuffled-variant ordering from near zero to roughly 0.32, and scaling to 32B pushes packet scores toward but not past the zero-shot flagship level, so the task is learnable but not yet solved.","Khondo provides a concrete target for future work: closing the shuffled-variant Sord gap toward the roughly 0.95 achieved when pages arrive in sequence."],"fun_headline_variants":["AI groups Bangla form pages but fails to reorder them","Page-order puzzle stumps vision models on Bangla forms","Shuffled Bangla forms: AI can split, but not sort","Bangla page order is the real hurdle for multimodal model","Why shuffled Bangla forms trip up AI: ordering, not clustering"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a packet's ground-truth page order—taken from the source PDF's order at assembly time—is genuinely recoverable from the page images alone; if a sizable share of pages (legacy-font glyphs, maps, blank continuation sheets, roughly 2% labeled Uncertain) provide no visual order cue, then part of the measured ordering bottleneck is an underdetermined task, not a model failure.","fun_headline_variants_meta":{"raw":{"variants":["AI groups Bangla form pages but fails to reorder them","Page-order puzzle stumps vision models on Bangla forms","Shuffled Bangla forms: AI can split, but not sort","Bangla page order is the real hurdle for multimodal model","Why shuffled Bangla forms trip up AI: ordering, not clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000346,"raw_usage":{"total_tokens":1757,"prompt_tokens":789,"completion_tokens":968,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":881}},"tokens_in":533,"tokens_out":968,"duration_ms":7534,"temperature":1.0,"reasoning_tokens":881,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:44:02.404865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give careful human annotators the same shuffled packet pages (images only, no text search) and measure their Sord; if humans also plateau near 0.7–0.8, the claim that models face an intrinsic ordering difficulty weakens in favor of a task that is partly impossible. A sharper test: re-score the frontier models on a Khondo subset from which all Uncertain and legacy-font pages are removed; if the English-over-Bangla ordering gap shrinks substantially, the gap is driven by unreadable script cues rather than by the ordering operation itself.","supporting_citations":[],"review_version":1}