{"id":"6e0d0ec1-5b05-47e0-b5cd-23069ef8a423","arxiv_id":"1909.00823","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A YOLOv3-based system detects Bengali digits and operators and evaluates handwritten arithmetic expressions, reporting 98.6% mAP on a new self-made dataset.","lead":"HishabNet detects and localizes handwritten Bengali digits and operators in an image using the YOLOv3 object detector, then evaluates the arithmetic expressions it finds. It introduces a new synthetic dataset of 40,000 annotated images, but shares neither code nor data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 for grouping detections into expressions is internally inconsistent as written, so the claimed end-to-end calculation is not reproducible and the 132/140 result cannot be verified from the paper.","rationale":"The strongest claim requires two things: a working detector and a working expression-grouping/evaluation procedure. The detector part is standard YOLOv3 on a custom dataset; its validation mAP could be inflated by augmentation leakage, but the end-to-end result (132/140 on a separate test set) does not depend on the mAP. The grouping procedure, Algorithm 1, is the novel element that converts independent detections into expressions and numbers. If that procedure is not correctly defined, the paper does not demonstrate the claimed functionality. Reading the pseudocode in Section III-D carefully, the indexing is inconsistent: the sort uses a column that is later used for the vertical span, and the membership test compares height rather than vertical center. No consistent interpretation yields the described row-grouping behavior. This is a concrete technical flaw, not a stylistic issue. Even if we apply the most natural correction, the algorithm is a fragile heuristic that is not validated on non-linear layouts. The reader's weakest assumption pointed to the heuristic's limitations; this attack is more fundamental: the algorithm as printed cannot be executed. However, because the authors could fix this by releasing code and a corrected algorithm, a conditional acceptance with that requirement remains appropriate. Thus we recommend no change to the reader's verdict.","tokens_in":8829,"tokens_out":8829,"duration_ms":82572,"concrete_test":"Re-implement Algorithm 1 from the printed pseudocode, using the variables exactly as defined (I[i][j] with j=1..4 for xcenter,ycenter,w,h), and run it on the 20 'multiple expressions' test images' detections (or, if the dataset is unavailable, on a synthetic set of three horizontal expression images). If the printed algorithm cannot be parsed into a deterministic procedure, or if it fails to separate the expressions correctly, the concern lands. Additionally, request the authors' exact implementation and confirm it matches the pseudocode; if it differs, the paper does not describe the evaluated system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section III-D, Algorithm 1 is the sole mechanism described for partitioning detections into separate mathematical expressions and ordering them into numbers. The pseudocode is not well-formed: the indexing 'I i;j=2' is ambiguous, and the subsequent operations use columns j=2 and j=4 inconsistently. If j=2 is the x-center (as implied by 'Sort I according to xcenter'), then the vertical span Ymin/Ymax is computed from x-coordinates; if j=2 is the y-center, the preceding sort is not by x. Moreover, the membership test 'Ymin <= I i=k;j=4 <= Ymax' compares the height of the candidate box (column 4) against the vertical span, rather than its vertical center. Under no consistent reading does the printed algorithm group boxes by vertical position, which is the behavior claimed in the text. The paper does not supply corrected pseudocode, a link to code, or an error analysis of the 8 failures in Table III (the 20-image 'multiple expressions' category is the worst at 16/20). Even under the most natural intended fix, the heuristic uses a hand-chosen 0.5*h margin and assumes horizontally arranged, single-line expressions; it does not address fractions, superscripts, or vertically staggered terms, which are common in handwritten math and are not present in the test set. Therefore the central claim of a functional end-to-end expression evaluator rests on a procedure that is underspecified and internally inconsistent as published.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HishabNet, an end-to-end system that uses YOLOv3 to detect and localize handwritten Bengali digits, operators, brackets, decimal points, and equals signs, and then forms and evaluates arithmetic expressions by grouping detections according to their positions. The model is trained on a newly created \"Hishab\" detection dataset of 40,000 images, reports a validation mean average precision of 98.6%, and achieves 132/140 correct calculations on a custom test set of expression images; the backbone is also evaluated on the NumtaDB and CMATERdb classification benchmarks.","tokens_in":9077,"tokens_out":6621,"duration_ms":69829,"significance":"The core idea is sensible and the paper makes a concrete, falsifiable claim: a YOLOv3 detector plus simple positional grouping can evaluate handwritten Bengali arithmetic. If the result holds, it is a useful engineering demonstration, and the Hishab dataset with bounding-box annotations could be a resource for the community. Credit is due for reporting quantitative results on a new dataset and for testing the backbone on public benchmarks; nevertheless, the paper's own evidence is currently too thin: the grouping algorithm is not reproducible as written, the expression-evaluation protocol is undefined, and no baseline comparisons or external validation are provided.","major_comments":[{"comment":"The pseudocode is internally inconsistent, so the claimed end-to-end calculation cannot be reproduced from the text. If column 2 is the x-center, as implied by the instruction \"Sort I according to xcenter, I_{i;j=2}\", then Ymin/Ymax are computed from x-centers and the membership test compares only the height (column 4) against that span, never checking vertical position; if column 2 is instead the y-center, the preceding sort is not by x. The paper provides no code, no corrected pseudocode, and no runnable specification, and this algorithm is the sole mechanism for converting detections into expressions. Please supply a corrected, unambiguous algorithm and either a reference implementation or a worked example showing how the test-set images are grouped.","section":"Section III-D, Algorithm 1"},{"comment":"The \"correct prediction\" criterion is undefined, and the failure analysis is missing. It is not stated whether a correct image requires the final numeric answer only, or also perfect detection, grouping, and parse order; the eight failures, including four in the hardest \"multiple expressions\" category, are not analyzed, so the reader cannot tell whether errors come from the detector, the grouping heuristic, the calculator, or the test-set construction. Specify the evaluation metric and report per-stage error counts and examples of failure, especially for the 16/20 multiple-expression category.","section":"Section IV, Table III"},{"comment":"The algorithm assumes expressions are horizontally arranged and uses the vertical span of the leftmost box plus a hand-chosen 0.5-times-height margin. This is stated as sufficient for identifying and evaluating multiple mathematical expressions, but fractions, superscripts, subscripts, and vertically staggered terms are not represented in Table III, even though they are common in handwritten mathematics. The claim should be restricted to single-line horizontal expressions until these cases are shown to work, or the test set and algorithm should be extended to cover them.","section":"Section III-D, grouping heuristic"},{"comment":"The validation is not yet sufficient to support the performance claims. The 98.6% mAP is computed on the authors' own validation split, with no stated IoU threshold, no class-wise AP, and no comparison to any other detector; Table II is a reproduction of published results for other algorithms on COCO and is not an evaluation of this model. Likewise, the abstract's \"outperforms previously suggested architectures\" is not supported by any head-to-head comparison in the results. I recommend adding a baseline comparison on Hishab, reporting class-wise AP and the IoU criterion, and, if the claim is to be made, benchmarking the backbone against published methods on the same public splits.","section":"Sections I and IV, validation"}],"minor_comments":[{"comment":"The arithmetic is slightly inconsistent: 18 classes with nearly 2,180 images per class plus 800 multi-class images gives about 40,040 images, not 40,000, and the construction of the 800 multi-class images and the annotation format should be described precisely.","section":"Section III-B, dataset description"},{"comment":"The classification of a wrong-class box with IoU above the threshold as FN is nonstandard; most detection evaluations would count a correctly localized but misclassified box as a false positive, so please justify or correct the convention.","section":"Section II-E, Table I"},{"comment":"Equation (2) has a typographical issue with the leading label and the superscript/subscript placement, and the notation \"IOU truth pred\" is hard to read; please re-typeset and renumber the equations consistently.","section":"Section III-A, Equation (2)"},{"comment":"The paper never states the IoU threshold t used for the mAP calculation, nor whether mAP is averaged over the 18 classes equally; without this information the 98.6% figure is not reproducible.","section":"Section IV, mAP reporting"},{"comment":"The 140-image expression test set is described only as containing expressions of \"various complexities\"; please state how it was constructed, whether it is a subset of the Hishab validation split, and whether it is disjoint from the training data.","section":"Section IV, test set"},{"comment":"The Hishab dataset, trained weights, and evaluation scripts are not made available, and no repository URL is given; a public release would be necessary for the results to be independently verified.","section":"General, reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a conference-length report, and its main risk is independent verification: the headline mAP is on a self-built dataset, the expression metric is undefined, and the grouping algorithm is not reproducible as printed. I see no deliberate circularity, but the reliance on a self-built dataset makes external or at least baseline comparison essential. If the authors provide corrected pseudocode, code, and stronger evaluation, the core idea could become publishable; as it stands I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is YOLOv3 applied to a new Bengali handwritten digit/operator detection dataset, plus a rule-based parser that groups detections into expressions and evaluates them. The detection half is standard but sensible, and the backbone's accuracy on NumtaDB and CMATERdb is genuinely good. The problem is that the grouping step, Algorithm 1, is not merely underspecified but internally inconsistent as printed, so the headline result of 132/140 correct expressions cannot be reproduced from the paper.\n\nWhat's new: Hishab is, as far as I know, the first Bengali handwritten digit dataset with bounding-box annotations, and treating expression recognition as detection-plus-grouping is novel for Bengali even though each component is off-the-shelf. The 98.6% mAP is plausible on a controlled self-made dataset, and the public-dataset accuracies suggest the backbone is solid.\n\nWhere it falls short: Algorithm 1's column indexing is contradictory—the text says sort by xcenter but the pseudocode uses j=2 for both xcenter and ycenter, and then compares a box's height (column 4) against vertical margins. No consistent reading yields the vertical grouping the text describes. Second, the expression evaluation uses only 140 hand-picked images, with no error analysis beyond Table III; the 8 failures are not examined. Third, the claim that the backbone 'outperforms' previous classifiers is not backed by any baseline comparison in this paper. Fourth, no code or dataset link is provided, so the results cannot be checked. The grouping heuristic additionally assumes horizontally arranged, single-line expressions, which excludes fractions, superscripts, and vertically staggered terms—common in handwritten math—and the test set includes none of those cases.\n\nThe detection approach itself likely works; the weakness is the parser and the thin evaluation. A resubmission with corrected pseudocode, released data and code, a proper benchmark, and failure analysis could turn this into a useful contribution. As it stands, it reads like a promising technical report whose central claim is not yet supported.\n\nI would send it to peer review because a competent referee can give the authors a clear path toward a solid paper, but I would not accept it in anything close to current form. It is worth a reading-group slot as a concrete case of detection alone not solving expression parsing.","headline":"A YOLOv3-based Bengali digit detector with a new bounding-box dataset, undermined by an internally inconsistent expression-grouping algorithm and evaluation that is too thin to support the strong claims.","tokens_in":9634,"tokens_out":3206,"would_cite":false,"duration_ms":33484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A detector that reads handwritten Bengali math expressions end-to-end, reporting 98.6% mAP and solving 132 of 140 test expressions.","keywords":["Bengali handwritten digits","object detection","YOLOv3","mathematical expression recognition","Hishab dataset","CNN","handwritten math","localization"],"falsifier":"Feed the system a handwritten expression containing a fraction such as 3/4 or a superscript like $2^{3}$. If the grouping algorithm cannot reconstruct the expression (e.g., it merges numerator and denominator or misorders the superscript), then the horizontal-sorting assumption is invalid and the claimed end-to-end capability does not extend to these common notational forms.","tokens_in":8581,"feed_emoji":"🧮","tokens_out":1697,"duration_ms":18751,"temperature":0.7,"pith_summary":"This paper argues that handwritten Bengali mathematical expressions can be recognized and evaluated by treating digits and operators as objects to detect, rather than as a single classification problem. The authors build a YOLOv3-based detector, train it on a new annotated dataset they call Hishab, and show that the detected boxes can be grouped into expressions and computed. They report a validation mAP of 98.6% and that the full system correctly solves 132 of 140 test images of varying complexity. If true, this would be a working pipeline for an under-served script and a step beyond single-digit classification.","feed_headline":"How a detector reads Bengali math by hand","feed_subtitle":"YOLOv3-based system spots digits and operators, then reconstructs and solves expressions — 132 of 140 test cases correct.","key_machinery":"The central mechanism is YOLOv3, a one-stage object detector that predicts bounding boxes and class probabilities from a single convolutional pass. Detected objects are then assembled into expressions by Algorithm 1, which sorts detections by x-center and groups them by the vertical extent of the leftmost box, using a 0.5×height margin. This spatial grouping is what lets the system turn isolated digit detections into multi-digit numbers and expressions.","core_discovery":"The paper's central claim is that a single-stage object detection network, YOLOv3 with a Darknet-53 backbone, can simultaneously localize and classify Bengali numerals and arithmetic operators in an image, and that those detections contain enough spatial information to reconstruct numbers and perform arithmetic. The authors introduce the Hishab dataset of 40,000 annotated images spanning 18 classes, report an overall validation mean average precision of 98.6%, and demonstrate correct evaluation of handwritten expressions in categories ranging from single-digit arithmetic to multi-digit decimal expressions with brackets and multiple expressions per image. They also report that the same backbone, when trained as a classifier, achieves 99.6252% accuracy on NumtaDB and 99.0833% on CMATERdb.","pith_inferences":["The paper leaves implicit that the expression-assembly step is the actual bottleneck: the detector itself is evaluated only on detection mAP, while the end-to-end accuracy (132/140) is a joint measure of detection and grouping, so failures could stem from either stage. A natural extension is to replace the hand-crafted grouping with a learned parser or a graph neural network over detected boxes.","The 0.5×height margin in Algorithm 1 is a free parameter that is not cross-validated. Its sensitivity could be tested by perturbing it on a held-out set, and such a test would quantify how robust the grouping is to variations in handwriting size and spacing.","The same architecture could be evaluated on handwritten math in other scripts, or on printed math, to see whether the detection-driven approach transfers without re-engineering the grouping logic.","Because the dataset is synthetic in part (augmentations applied to a primary dataset), the reported mAP may be optimistic relative to fully natural handwritten images; a test on a separate real-world dataset would clarify this."],"forward_implications":["If the reported performance holds, the same detection-plus-grouping approach could be applied to other under-resourced scripts, since the method does not depend on Bengali-specific features beyond the training data.","The Hishab dataset could support further research on Bengali handwritten object detection, as it is the first dataset of its kind with bounding-box annotations for digits and operators.","The system's ability to handle multiple expressions in one image suggests that document-level math recognition, not just isolated equations, is feasible with this architecture.","The reported classifier accuracies on NumtaDB and CMATERdb indicate that the backbone CNN is a competitive feature extractor for Bengali digit recognition.","The grouping heuristic, if extended to handle fractions and superscripts, could generalize the system to a wider class of mathematical notation."],"supporting_citations":[{"why":"Provides the YOLOv3 architecture that the detection model is built on.","marker":"[1]"},{"why":"Introduces the YOLO algorithm, framing detection as regression, which the method adopts.","marker":"[2]"},{"why":"Darknet is the open-source framework used to implement and train the model.","marker":"[20]"},{"why":"NumtaDB is one of the public datasets used to evaluate the backbone CNN's classification accuracy.","marker":"[27]"},{"why":"CMATERdb is the other public dataset used to evaluate the backbone CNN's classification accuracy.","marker":"[28]"}],"fun_headline_variants":["YOLOv3 detects and solves handwritten Bengali expressions","Detector achieves 98.6% mAP on Bengali math images","Hishab dataset enables automated Bengali math solving"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a mathematical expression is always arranged horizontally, so that sorting detected objects by their horizontal centers and grouping by the leftmost object's vertical span is enough to reconstruct the expression; this breaks for fractions, superscripts, and vertically staggered layouts.","fun_headline_variants_meta":{"raw":{"variants":["YOLOv3 detects and solves handwritten Bengali expressions","Detector achieves 98.6% mAP on Bengali math images","Hishab dataset enables automated Bengali math solving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000913,"raw_usage":{"total_tokens":3898,"prompt_tokens":898,"completion_tokens":3000,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2947}},"tokens_in":514,"tokens_out":3000,"duration_ms":21265,"temperature":1.0,"reasoning_tokens":2947,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:34:07.304075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the system a handwritten expression containing a fraction such as 3/4 or a superscript like $2^{3}$. If the grouping algorithm cannot reconstruct the expression (e.g., it merges numerator and denominator or misorders the superscript), then the horizontal-sorting assumption is invalid and the claimed end-to-end capability does not extend to these common notational forms.","supporting_citations":[{"cited_title":"You only look once: Unified, real-time object detection,","cited_arxiv_id":null,"evidence_quote":"Introduces the YOLO algorithm, framing detection as regression, which the method adopts."},{"cited_title":"Darknet: Open source neural networks in c,","cited_arxiv_id":null,"evidence_quote":"Darknet is the open-source framework used to implement and train the model."},{"cited_title":"NumtaDB - Assembled Bengali Handwritten Digits","cited_arxiv_id":"1806.02452","evidence_quote":"NumtaDB is one of the public datasets used to evaluate the backbone CNN's classification accuracy."},{"cited_title":"A genetic algorithm based region sampling for selection of local features in handwritten digit recognition application,","cited_arxiv_id":null,"evidence_quote":"CMATERdb is the other public dataset used to evaluate the backbone CNN's classification accuracy."}],"review_version":1}