{"id":"09dfa190-5158-4ba9-8b04-29c29285d6b0","arxiv_id":"2608.03884","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BANGLAWILD is the first in-the-wild Bengali scene text benchmark with dual verbatim/standard labels, and its evaluation shows visual mis-recognition dominates errors while conjunct-related errors are nearly closed.","lead":"Introduces BANGLAWILD, a benchmark of 2,535 images of Bengali text photographed on real-world surfaces like walls, vehicles, and packaging, each with exact transcriptions and diagnostic labels. Evaluations of 15 vision-language models and 3 OCR systems find that most errors come from visual misreading, not from complex Bengali orthography, and that larger models do not always outperform smaller ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual-dominance claim rests on a string-only classifier; Table 2's ~60% visual share is not yet validated against image-grounded human cause labels.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: Appendix H assigns causal categories from (GT, prediction) pairs only, and the headline 'visual mis-recognition dominates' is inferred from those string-edit labels. This is not a minor methodological quibble; the entire inversion of Bengali-OCR priorities rests on the ratio of visual to orthographic errors. The classifier's deterministic, image-free rules guarantee reproducibility but not construct validity. A single substitution at edit distance 2 is labeled 'visual grapheme misread' even if the model knew the glyph and substituted a confusable lexeme from its language prior; a word with <40% character overlap is labeled 'lexical word substitution' and then counted as visual, even if the output is a fluent hallucination. The paper's own examples illustrate this ambiguity. The proposed human-judgment study directly tests whether the taxonomy tracks perceptual causes; it is feasible and would settle the central claim. If validated, the paper's contribution is strong; if not, the finding should be reframed as 'string-edit-dominant errors' rather than 'visual mis-recognition.' The dataset and leaderboard remain valuable either way, so conditional acceptance is appropriate; code and data release is a separate but also necessary condition.","tokens_in":34349,"tokens_out":5183,"duration_ms":49870,"concrete_test":"Sample ~300 error events from Table 2 for Gemini-2.5-Flash, Qwen3.5-9B, and EasyOCR, stratified by Table 2 categories. For each event, show the original image, ground truth, and prediction to 2–3 native Bengali annotators; ask them to classify the dominant cause as (a) visual/glyph misread, (b) orthographic (matra/homophone/conjunct/nasal), (c) language-prior/lexical substitution, (d) segmentation/Unicode, or (e) other. Compare these human labels to Appendix H classifier labels via Cohen's kappa, and compare the human-labeled visual share to the claimed 56–60% (and EasyOCR's 3.7× ratio). If the human visual share is within ~10 percentage points and kappa ≥0.6, the central claim is validated; if a large fraction of 'lexical word substitutions' is instead human-labeled as language-prior, the conclusion should be reframed as 'visual plus lexical/language-prior errors dominate' rather than '","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that Bengali scene-text failure is dominated by visual misrecognition rather than orthographic confusion—depends entirely on the Appendix H error classifier. That classifier receives only the (ground truth, prediction) string pair and explicitly has no access to the source image, prompt, or model identity. Categories carrying causal labels are therefore assigned from edit statistics, not perceptual evidence. 'Visual grapheme misread' is defined as a single substitution within edit distance 2 that is not in a homophone lookup; 'lexical word substitution' is defined by character overlap <40% or edit distance ≥60% of word length. Neither rule can distinguish a vision-encoder miss from a language-prior replacement, hallucination, or decoding artifact. The headline 56–60% 'visual' share thus partly reflects the classifier's decision to count all low-overlap word replacements as visual. The conjunct/hasanta estimate is more credible because it is keyed to a specific Unicode diacritic, but the comparative visual-to-orthographic ratio is computed on the unvalidated taxonomy. Determinism and cross-model consistency do not establish construct validity: the same string-edit rule can mislabel the same cause for all models. The paper's own qualitative examples include substitutions such as কপাট→েলাহ that look like language-prior or hallucination errors, not obviously visual misreads, which underscores the risk. The benchmark, annotation, and leaderboard remain solid; only the causal interpretation of the taxonomy lacks support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BANGLAWILD, a benchmark of 2,535 in-the-wild Bengali scene text images with verbatim gold transcriptions, categorical and diagnostic attribute labels, and dual verbatim/standard-form annotation. It evaluates 15 vision-language models, 3 conventional OCR systems, and 6 LoRA fine-tuned variants under three prompting strategies, and complements edit-distance metrics with an LLM-as-a-judge protocol. The central finding is that visual mis-recognition, not orthographic confusion, dominates errors (~56–60% for the strongest systems), with conjunct/hasanta errors contributing under 2%, challenging a long-standing assumption in Bengali OCR research.","tokens_in":34679,"tokens_out":5672,"duration_ms":55949,"significance":"The dataset construction is careful and well-documented: three independent native annotators, expert adjudication, strong inter-annotator agreement (Cohen's κ 0.81–0.93, mean pairwise CER 3.2%), bootstrapped confidence intervals, and transparent handling of the oracle P3 condition and of in-family judge bias. The dual verbatim/standard annotation is a valuable contribution that enables measuring silent over-correction and hallucination, which conventional OCR cannot exhibit. If the error-taxonomy claim were validated, the paper would provide an important, field-shaping result for Bengali OCR and VLM evaluation. However, the central claim currently rests on a string-only classifier whose 'visual' categories are not confirmed by any image-grounded evidence, so the headline result is not yet established. The benchmark and leaderboard are likely to be useful regardless.","major_comments":[{"comment":"The error classifier receives only the (ground truth, prediction) string pair and explicitly has no access to the source image. The categories 'visual grapheme misread', 'lexical word substitution', and 'multi-character word corruption' are assigned from edit-distance thresholds and character-overlap rules, not from any perceptual evidence. Therefore the claim that ~56–60% of errors are 'visual mis-recognition' is not supported: the same string-edit pattern can arise from a vision-encoder miss, a language-prior replacement, a hallucination, or a decoding artifact. For instance, কপাট→েলাহ is counted as 'lexical word substitution' and described as a visual misreading, but the output shares no visual overlap with the source and is more plausibly a language-prior or hallucination error. The taxonomy needs validation against human cause-labeling on a sample of error pairs (with images), or th","section":"§6 and Appendix H (Table 2)"},{"comment":"The headline visual-to-orthographic ratio (4.4× for Gemini Flash, 5.8× for Qwen3.5-9B) is computed from the unvalidated taxonomy. 'Lexical word substitution' is defined as any whole-word replacement with character overlap <40% or edit distance reaching 60% of word length; this captures any low-overlap error, regardless of cause. Because the classifier cannot distinguish visual errors from hallucination or language-prior errors, the ratio is not a meaningful estimate of the perceptual failure distribution. The subsequent conclusion that 'the bottleneck lies in the in-the-wild imagery rather than any specific model class' is a causal claim that the string-only classifier cannot support. A sensitivity analysis using stricter, visually grounded definitions, or a human-annotated cause sample, is required.","section":"§6, Table 2: visual:orthographic ratio"},{"comment":"The claim that the 'classical conjunct bottleneck has largely closed' (conjunct/hasanta errors under 2% for VLMs) is also dependent on the decision hierarchy in Appendix H. Any error that rewrites a word into a completely different form (e.g., a lost conjunct embedded in a hallucinated word) is routed to 'lexical word substitution' or 'semantic hallucination', never to 'conjunct/hasanta error'. The conjunct rate may therefore be artificially deflated by the classifier's priority order. At minimum, the paper should report how many errors with a hasanta/conjunct in the ground truth were reassigned to non-orthographic categories, or provide a per-glyph analysis of conjunct-containing ground-truth words.","section":"§6 and §7: conjunct/hasanta conclusion"}],"minor_comments":[{"comment":"The taxonomy table reports only percentages of residual errors. Because the models have very different error rates (exact-match accuracy ranges from 0.16% to 35.6%), absolute counts or per-image rates would help assess the practical impact of each category.","section":"Table 2"},{"comment":"The thresholds defining the categories (edit distance 2, character overlap <40%, edit distance ≥60% of length) are presented as fixed rules without justification or sensitivity analysis. A brief rationale or a test of neighboring thresholds would strengthen reproducibility.","section":"Appendix H"},{"comment":"The claim of a 'persistent gap' in which larger models do not outperform smaller ones is based on only two Gemini models and three Qwen variants, which are also architecturally distinct. The paper appropriately cautions about this, but the wording in the abstract and conclusion ('persistent gap') is stronger than the evidence warrants.","section":"§5"},{"comment":"The qualitative examples used to support the visual-bottleneck claim (e.g., ফ্লয্াটিট→ফয্াক্টির, কপাট→েলাহ) are presented as self-evidently visual misreadings. Since the paper has the actual images, showing the source images (or a crowd-sourced human judgment of those specific cases) would be more convincing than relying on the reader's intuition.","section":"§6"},{"comment":"The 'word-boundary segmentation' category includes whitespace-only normalization and a token split that leaves the character sequence intact. These events inflate WER but are not recognition failures; counting them separately from the visual/orthographic ratio would be cleaner.","section":"Appendix H, word-boundary segmentation"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and evaluation infrastructure are solid and likely citable. The main risk is that the paper's most interesting claim—that Bengali scene-text failure is visual rather than orthographic—is not yet supported by the evidence. I would advise the editor to ask for either (a) a validation study with human cause labels on a stratified sample of error pairs, or (b) a substantial reframing of the error taxonomy as descriptive string-edit categories. The conjunct/hasanta claim needs similar scrutiny. If the authors can provide that validation, the paper would be a strong contribution; in the current form, the headline is an overreach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe benchmark is the contribution; the headline finding is the risk. BanglaWild is the first in-the-wild Bengali scene text benchmark with dual verbatim/standard annotation, and it evaluates both conventional OCR and generative VLMs on the same images. The annotation is careful—three annotators, expert adjudication, kappa 0.81–0.93, pairwise CER 3.2%—and the evaluation is thorough: bootstrapped CIs, an out-of-family judge for the Gemini models, and an oracle condition properly flagged. That is real, useful work.\n\nThe central claim—that visual misrecognition dominates failure at 56–60%, with conjunct errors under 2%—doesn't yet have support. The Appendix H classifier sees only the (ground truth, prediction) string pair. It has no image. The category 'visual grapheme misread' is any single substitution within edit distance two that isn't in a homophone list; 'lexical substitution' is any low-overlap word replacement. Those are string-edit rules, not perceptual causes. A language-prior hallucination that outputs a plausible different word lands in the same bucket as a vision encoder missing a character. Determinism across models doesn't fix that; it just makes the same mislabel consistent. The conjunct/hasanta estimate is safer, since it's keyed to a specific diacritic, but the visual-to-orthographic ratio is computed on the unvalidated taxonomy. The paper's own qualitative examples (কপাট→েলাহ) look visual, but the counts are not demonstrated to be visual. The paper never checks whether the taxonomy matches human judgments of why errors occurred.\n\nMinor points: data and code are not yet released, so the benchmark isn't independently reproducible. The non-monotonic scaling result is carefully hedged, which is good. Some context×content cells have few images; the paper flags that.\n\nBottom line: the benchmark deserves peer review and likely publication after revision. The error taxonomy needs validation against human cause labels, or the causal claims need to be softened to 'string-edit profiles' rather than 'visual misrecognition.' A serious referee can push exactly there. If the authors validate the taxonomy or release the data and code, this becomes a solid resource.","headline":"Solid benchmark, but the visual-dominance claim rests on a string-only classifier that has not been validated; the dataset is the real contribution.","tokens_in":35164,"tokens_out":2508,"would_cite":true,"duration_ms":24350,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bengali scene text recognition fails primarily through visual mis-recognition, not orthographic confusion, according to a new 2,535-image in-the-wild benchmark.","keywords":["Bengali scene text recognition","in-the-wild benchmark","vision-language models","error taxonomy","visual mis-recognition","orthographic confusion","conjunct/hasanta errors","LoRA fine-tuning"],"falsifier":"Take a sample of mis-transcriptions and have native Bengali readers label the cause (visual confusion vs spelling choice) from the image; compare with the taxonomy's visual/orthographic assignment. If human labels disagree with the classifier's visual majority, the paper's central finding is an artifact of edit-distance heuristics.","tokens_in":34261,"feed_emoji":"🪧","tokens_out":6022,"duration_ms":50148,"temperature":0.7,"pith_summary":"BanglaWild is a first attempt to measure how well modern systems actually read Bengali text in the wild: 2,535 mobile-phone photographs of walls, signboards, vehicles, packaging, and banners, each with a verbatim gold transcription and an orthographically standard form where the in-image spelling deviates. The paper's central claim is that on this benchmark, visual mis-recognition—not orthographic confusion—dominates failure, with visual categories making up roughly 56–60% of errors for the strongest systems and conjunct/hasanta errors below 2%, a profile that also holds for the one conventional OCR baseline that reads Bengali reliably. It also finds that larger models do not consistently beat smaller ones within the same family, that prompt language mainly gates cross-script drift, and that LoRA fine-tuning rescues catastrophic failures in weak models without lifting the ceiling on already competent ones. If these findings hold, they redirect Bengali OCR research from conjunct decoding toward visual robustness, and they argue for dual verbatim/standard evaluation whenever generative models transcribe text with non-standard spelling.","feed_headline":"Seeing, not spelling, trips Bengali OCR models","feed_subtitle":"A 2,535-image in-the-wild benchmark finds ~60% of errors are visual misreads; conjunct errors fall under 2%.","key_machinery":"The load-bearing object is the BANGLAWILD benchmark itself, with its dual verbatim/standard gold transcription: the verbatim form makes recognition errors measurable while the standard form exposes silent over-correction. The argument is carried by a fifteen-class deterministic error taxonomy (Appendix H) that classifies every (ground truth, prediction) pair by a rule-based cascade and edit-distance thresholds, producing the visual-versus-orthographic split; the classifier sees only the string pair, never the image. Supporting mechanisms are the LLM-as-a-Judge protocol (validated against native annotators, ρ=0.89) that supplies a meaning-aware score alongside CER/WER, and LoRA fine-tuning us","core_discovery":"The paper introduces BANGLAWILD, a benchmark of 2,535 in-the-wild Bengali scene text images paired with verbatim gold transcriptions and, for the 3.43% of images with non-standard spelling, an orthographically standard form. Using a fifteen-class deterministic error taxonomy applied to four VLM families and three conventional OCR baselines, it reports that lexical substitution, single-character visual misreads, and multi-character word corruption together account for ~56–60% of Gemini-2.5-Flash and Qwen3.5-9B errors, while conjunct/hasanta errors contribute 1.0–1.6%; EasyOCR shows the same visual-dominant profile at a 3.7× visual-to-orthographic ratio. The paper interprets this as inverting","pith_inferences":["If visual mis-recognition is the true bottleneck, then accuracy should improve more from better detection and image preprocessing (e.g., curved-text rectification, clutter reduction) than from stronger language priors; this is testable by re-running the benchmark with rectified images.","The non-monotonic scaling across model sizes suggests training-data composition or benchmark overlap effects; a controlled study holding architecture fixed and varying only Bengali scene-text pretraining data could settle it.","The dual verbatim/standard annotation could be adopted for other low-resource scripts with non-standard spelling, turning silent over-correction into a measurable quantity rather than an anecdote.","Because prompt language only gates cross-script contamination, prompting in Bengali is a cheap but limited fix; the paper's profile predicts that visual robustness gains will require model-level changes, not instruction changes."],"forward_implications":["Bengali OCR research should de-emphasize conjunct decoding: conjunct/hasanta errors are 1.0–1.6% of VLM errors and 4.6% for EasyOCR, far below the ~56–60% share of visual mis-recognition categories.","Benchmarks need dual verbatim/standard annotations: over-correction and fluent hallucination are structurally invisible to conventional OCR and to edit-distance metrics, yet they dominate deployment risk for addresses, receipts, and medical packaging.","Prompt language has a narrow, script-gating effect: Bengali prompts reduce cross-script contamination (e.g., 6.8%→5.5% on Gemini) but leave visual error categories unchanged within ±1.5 pp.","LoRA fine-tuning on 1,268 images converts catastrophic failures (refusals, off-target captioning) into ordinary errors but slightly regresses the strongest open model (Qwen3.5-9B: 20.38→21.53 CER), so small-scale adaptation does not lift the ceiling.","Rankings depend on the metric: CER and an LLM judge diverge at the top, so deployment decisions should look at both, and tokenization/Unicode artifacts inflate WER by 22–25% of errors without changing readability."],"supporting_citations":[{"why":"Supplies the ICDAR RRC-MLT multi-script benchmark lineage and the prior Indic error typologies the taxonomy is cross-checked against.","marker":"(Nayef et al., 2019)"},{"why":"Provides cross-script transfer baselines and Indic error typology used to retain orthographic categories in the taxonomy.","marker":"(Gunna et al., 2022)"},{"why":"Establishes the LLM-as-a-Judge protocol and validation approach used for the semantic scoring.","marker":"(Zheng et al., 2023)"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method used to test whether adaptation improves zero-shot recognition.","marker":"(Hu et al., 2022)"},{"why":"Documents in-family judge bias, motivating the out-of-family re-scoring of the two Gemini systems.","marker":"(Panickssery et al., 2024)"},{"why":"Text-centric VLM benchmark (OCRBench v2) whose English/Chinese focus defines the measurement gap this paper fills.","marker":"(Fu et al., 2025)"},{"why":"MTVQA benchmark; its translated-text constraint motivates the paper's natively photographed imagery.","marker":"(Tang et al., 2024)"},{"why":"BN-HTRd, the canonical Bengali handwritten corpus, exemplifying the controlled-condition resources this benchmark departs from.","marker":"(Rahman et al., 2022)"},{"why":"Datasheet template followed for documenting dataset composition, collection, and intended use.","marker":"(Gebru et al., 2021)"},{"why":"Prior work on VLM over-correction in handwritten OCR that motivates the dual verbatim/standard annotation.","marker":"(Seong et al., 2026)"}],"fun_headline_variants":["Visual errors dominate Bengali scene-text OCR","Bengali OCR: visual misreads ~60%, conjuncts under 2%","Larger models don't beat smaller ones on Bengali OCR","New benchmark shows Bengali OCR stumbles on sight, not spelling"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim that ~60% of errors are visual rests on a rule-based classifier that judges errors from the text pair alone, without seeing the image; if its edit-distance categories do not track the true perceptual cause, the visual-dominance finding would be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Visual errors dominate Bengali scene-text OCR","Bengali OCR: visual misreads ~60%, conjuncts under 2%","Larger models don't beat smaller ones on Bengali OCR","New benchmark shows Bengali OCR stumbles on sight, not spelling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1678,"prompt_tokens":801,"completion_tokens":877,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":808}},"tokens_in":545,"tokens_out":877,"duration_ms":7658,"temperature":1.0,"reasoning_tokens":808,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:26:29.598853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of mis-transcriptions and have native Bengali readers label the cause (visual confusion vs spelling choice) from the image; compare with the taxonomy's visual/orthographic assignment. If human labels disagree with the classifier's visual majority, the paper's central finding is an artifact of edit-distance heuristics.","supporting_citations":[{"cited_title":"2019 , pages =","cited_arxiv_id":null,"evidence_quote":"Supplies the ICDAR RRC-MLT multi-script benchmark lineage and the prior Indic error typologies the taxonomy is cross-checked against."}],"review_version":1}