{"id":"1cd15a0d-e8b3-4899-865e-22d3057ef445","arxiv_id":"2501.02811","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A team reports winning a street-view shop sign recognition competition with a multi-stage OCR pipeline built from known components, but provides no code, data, or rigorous ablations.","lead":"This paper describes the first-place competition solution for recognizing storefront signboards in street-view images. It combines existing OCR techniques, including Mask R-CNN detection, a graph neural network for text grouping, and self-supervised text recognition pre-training.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The per-method gains in Table 1 are the load-bearing evidence for the paper's scientific claims, but they lack a baseline, protocol, and error bars; the causal attribution of every component is unverifiable.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the per-method gains in Table 1 are asserted without ablation protocol, error bars, or any indication of the evaluation split. I agree with that assessment. The first-place competition score itself is an external fact if the competition records are accurate, but the paper's scientific content is the decomposition of that score into method contributions. The missing baseline makes Table 1 not merely imprecise but uninterpretable: the reader cannot tell whether the gains are measured against a valid baseline, whether they are additive, or whether they were selected on the test set. This is a claim-without-derivation problem, not a disagreement with any external consensus. The rejection is justified by the absence of evidence, and the paper could in principle be rehabilitated by releasing code and a proper ablation study, but as submitted the verdict remains unchanged.","tokens_in":12278,"tokens_out":4557,"duration_ms":46446,"concrete_test":"Ask the authors for the evaluation code, the exact base configuration, and the per-stage validation logs, then rerun the seven Table 1 ablations on a fixed validation split with at least three random seeds, holding all other components constant. If any reported gain is not reproduced within approximately ±0.5 F-score points (or if the reproduced gains do not sum to the difference between the reported baseline and the final 0.6672), the causal attribution in Table 1 fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5 (Table 1) is the only quantitative support for the abstract's claim that 'comprehensive experiments have validated the effectiveness of these methods.' The table lists seven 'Gain (%)' values — DCN 0.6, BoxDQN 1.6, ViT backbone 2.5, self-supervised pre-training 3.2, multimodal modelling & GNN 2.9, perspective rectification 2.6, Center Loss 0.8 — summing to 14.2. If these are F-score percentage-point gains on the competition metric, the implied baseline is about 0.525, but no baseline score is reported. The paper does not state what configuration is ablated, whether each component is toggled individually or cumulatively, whether the measurement is on the ~500-image test set (which may have been used for model selection) or on a held-out validation split, or whether any run is repeated. Because the components interact (e.g., BoxDQN adjusts text boxes for the recognizer; self-supervised pretraining initializes the ViT backbone), the individual gains are not necessarily additive, and the implicit additive baseline is not justified. Without this protocol, the central scientific contribution — that these techniques cause the reported improvement — is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a first-place solution for a streetscape shop sign recognition competition. The method is a multi-stage pipeline: (1) signboard detection using Mask R-CNN enhanced with deformable convolutions, keypoint regression on quadrilateral vertices, and data augmentation; (2) joint text detection and key information extraction via a two-stage detector, graph neural network, and a reinforcement-learning-based BoxDQN component; (3) text recognition using a SAR-based model upgraded with a ViT backbone, bidirectional LSTMs, self-supervised pre-training, and Center Loss; and (4) heuristic reading-sequence prediction. The authors report an F-score of 0.6672 and first place among 14 teams, and Table 1 lists per-method gains for seven components. The central claim is that comprehensive experiments validate the effectiveness of these methods.","tokens_in":12531,"tokens_out":3107,"duration_ms":32942,"significance":"If the competition result is accurate, the paper documents a competitive end-to-end system for a practical OCR task, and the external first-place score is a meaningful existence proof. However, the scientific value beyond that score is currently weak: the ablation table (Table 1) lacks a baseline, protocol, error bars, and dataset split information, so the per-method attributions are not verifiable. No code or data is provided. The paper's contribution as a research article therefore rests almost entirely on an unreproducible competition score and a list of known techniques applied in a pipeline.","major_comments":[{"comment":"Table 1 reports seven 'Gain (%' values (0.6, 1.6, 2.5, 3.2, 2.9, 2.6, 0.8) with no stated baseline, no definition of what configuration or metric the gains are measured on, no indication of whether each component is toggled individually or added cumulatively, and no error bars or number of runs. If these are F-score percentage-point increments on the competition metric and are additive, the implied baseline is about 0.525, but no baseline score appears anywhere in the paper. Without this protocol information, the abstract's claim that 'comprehensive experiments have validated the effectiveness of these methods' is unsupported.","section":"Section 2.5, Table 1"},{"comment":"The evaluation set contains only about 500 images, and the paper does not state whether the gains in Table 1 were computed on that test set or on a held-out validation split. Since the described pipeline was developed for the competition, it is likely that the test set was used for model selection, making the per-method gains post-hoc attributions rather than blind predictions. The authors should clarify the evaluation protocol, and if the test set was used, they should provide a controlled validation protocol or cross-validation to support causal claims.","section":"Section 3.2 and Section 1.3.1"},{"comment":"The text recognition baseline is referenced as 'SAR [ ? ]' with a missing citation, and the self-supervised pre-training methods are described only at a high level (Sequential Contrast Learning and stroke/semantic masked image modeling) without specifying the pre-training data, training epochs, or how these methods are adapted to the Chinese shop-sign setting. The BoxDQN integration is also described in one sentence. These omissions make the method impossible to reproduce or compare against prior work.","section":"Section 2.3"},{"comment":"Several components are underspecified: the graph neural network construction and multimodal feature grouping in Section 2.2, the 'copy-and-paste' and random perspective augmentation details in Section 2.1, and the reading-sequence heuristics in Section 2.4. For a methods-focused paper, this level of detail is insufficient for a reader to reimplement the pipeline or to attribute performance to specific design choices.","section":"Sections 2.1-2.4"}],"minor_comments":[{"comment":"There is a typo: 'Mutli-modal' should be 'Multi-modal'.","section":"Table 1"},{"comment":"The phrase 'the mural information and textual information' appears to be a typo; 'mural' is likely intended to be 'visual'.","section":"Section 2.3"},{"comment":"The reference list contains many entries that are never cited in the body (e.g., [3], [4], [11], [13], [27], [29], [30], [48]-[53], and others), and reference [1] is empty. The missing citation for SAR also needs to be supplied.","section":"References"},{"comment":"The evaluation metric description does not specify the matching criterion for a correct detection (e.g., IoU threshold) or the exact text matching rule (e.g., exact string match, handling of Chinese characters). This makes the reported F-score difficult to interpret.","section":"Section 3.1"},{"comment":"The paper does not name the competition (e.g., year or organizing body) in the title or abstract, which makes the context of the claimed first-place result less clear to readers.","section":"Title and Abstract"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be a competition solution report that has been submitted as a research paper. The external first-place score is notable, but the paper's scientific contribution is not currently substantiated: Table 1 lacks even a baseline value, and no code or data is provided. The reference list also contains a large number of apparently uncited or irrelevant entries, which suggests the manuscript has not been carefully edited. If the authors were to resubmit, they would need to provide a rigorous ablation protocol, clarify the evaluation split, and make code or detailed experimental settings available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. This is a competition write-up, not a research paper. The first-place score (F=0.6672, combined 97.11) is externally grounded and the pipeline description is clear, but the paper gives you no way to verify where the gains came from.\n\nWhat is actually new: nothing scientifically. Every component is from prior work, as the authors themselves cite. The contribution is an engineering integration that won a 14-team competition. That is worth respecting as an achievement, and the paper describes the data and metric well enough that someone entering a similar challenge could use it as a starting point.\n\nThe soft spot is Table 1. Seven gains are listed with no baseline, no ablation protocol, no error bars, and no statement about whether the ~500-image test set was used for model selection. The gains sum to 14.2 percentage points, which implies a baseline around 0.525, but that number never appears. The components interact, so the individual attributions are not self-evidently additive. As written, 'comprehensive experiments have validated the effectiveness' is unsupported. This is a missing-evidence problem, not a fraud.\n\nThe citation list is a separate red flag: a missing placeholder on SAR, and a long tail of references about pavement texture, idling vehicles, sarcasm detection, and financial sentiment. That suggests the bibliography was not actually checked. It makes me doubt the carefulness of the analysis, though it does not by itself invalidate the competition result.\n\nWho is this for? Practitioners entering a street-view OCR competition, or teachers who want a concrete example of what a multi-stage pipeline looks like. As a peer-reviewed article, it fails the bar: the central claims are not independently checkable and no code or data are released. I would not send it to a serious venue. A workshop or a technical blog would be the right home. If the authors release code and a controlled ablation, it could become a useful technical report.\n\nRecommendation: desk reject if it comes to your journal. Not worth referee time in its current form.","headline":"A first-place competition score that's externally grounded but an ablation table with no baseline, protocol, or error bars; treat as a technical report, not a research paper.","tokens_in":13006,"tokens_out":3387,"would_cite":false,"duration_ms":30653,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-stage pipeline combining signboard detection, graph-based text grouping, and self-supervised pretrained recognition won first place in the shop-sign recognition competition with an F-score of 0.6672.","keywords":["scene text recognition","store signboard detection","street view OCR","key information extraction","graph neural network","self-supervised pre-training","perspective text rectification","competition solution"],"falsifier":"Re-run the pipeline on the same competition test set, removing one technique at a time with a fixed random seed and reporting standard deviations; if removing self-supervised pre-training does not lower the F-score by roughly 3.2 points, or if any of the seven listed gains reverse sign, the causal attribution in the paper's results table is refuted.","tokens_in":12080,"feed_emoji":"🏪","tokens_out":9372,"duration_ms":80928,"temperature":0.7,"pith_summary":"This paper reports a first-place solution to a street-view shop sign recognition competition, where the task is to locate store signboards in photos and read the store names written on them. The authors claim that a four-stage pipeline — signboard detection, joint text detection with key information extraction, separate horizontal and vertical text recognition, and rule-based reading-sequence prediction — beat 13 other teams with an F-score of 0.6672, an oral defense score of 92.86, and a combined score of 97.11. The contribution is presented as the combination of a Mask-RCNN-based signboard detector with deformable convolutions and keypoint regression, a graph neural network that assigns detected text to signboards, a ViT-backed recognizer trained with two self-supervised pre-training schemes, and perspective rectification of signboard quadrilaterals. A sympathetic reader would care because the paper claims this modular recipe is enough to win a real-world OCR challenge with only about 5,000 training images.","feed_headline":"Four-stage pipeline wins shop-sign OCR contest with F-score 0.6672","feed_subtitle":"A graph network that links text to signboards carries the winning entry.","key_machinery":"The key machinery is the joint text detection and key information extraction network, which fuses positional and image embeddings of detected text boxes and passes them through a graph neural network to decide which text belongs to a store signboard. This stage is trained end-to-end from a two-stage text detector, following the feature-sampling-and-grouping paradigm the paper cites as [73]. Supporting machinery includes the signboard detector (Mask-RCNN with deformable convolutions, four-vertex keypoint regression branches, and copy-and-paste plus perspective augmentation), perspective rectification that transforms the minimum enclosing quadrilateral into a corrected rectangular region, and a text recognizer pre-trained with sequential contrastive learning and with stroke-and-semantic masked image modeling based on MAE.","core_discovery":"The paper's central claim is that a multi-stage system, rather than a single end-to-end model, is the winning design for street-view shop sign recognition. The authors report that their system won first place among 14 teams on the competition's evaluation set, with a store-sign recognition F-score of 0.6672. They attribute the margin to seven quantified gains: deformable convolutions (+0.6%), BoxDQN reinforcement-learning box adjustment (+1.6%), a ViT backbone (+2.5%), self-supervised pre-training (+3.2%), multimodal modeling with a graph neural network (+2.9%), perspective text rectification (+2.6%), and Center Loss for text recognition (+0.8%). The pipeline proceeds by detecting signboard quadrilaterals with instance segmentation, detecting text and simultaneously deciding which text belongs to a store signboard, recognizing the text with horizontal and vertical models trained under two self-supervised objectives, and merging text blocks into store names using reading-order heuristics without a learned model.","pith_inferences":["The paper does not provide code, checkpoints, or an ablation protocol, so the only way to confirm that the Table 1 gains are causal is to re-run the pipeline on the same data with controlled removals.","Because no separate validation split is described, the evaluation set may have been used for model selection; if so, the reported gains could be optimistically biased relative to a fresh test set.","The same four-stage recipe could transfer to other text-in-the-wild tasks such as license plate recognition or document photo OCR, but transfer performance is not reported.","A learned sequence model or vision-language model could plausibly improve on the rule-based reading-order merging stage for complex multi-block signboards, a direction the paper leaves implicit."],"forward_implications":["A modular four-stage pipeline can outperform end-to-end single-model approaches on a real street-view OCR benchmark, at least on the competition's evaluation set.","Self-supervised pre-training is the largest single contributor in this system, worth a claimed +3.2% F-score gain, so investing in unlabeled text-image pre-training may pay off more than architecture changes.","Perspective rectification of the signboard quadrilateral before recognition contributes +2.6%, meaning geometric correction is a meaningful part of the accuracy budget.","The graph neural network for assigning text to signboards contributes +2.9%, supporting the paper's claim that joint text detection and key information extraction share useful features.","The authors state that the framework can be quickly implemented and applied in industry for tasks such as rebuilding realistic maps."],"supporting_citations":[{"why":"Defines the street-view shop sign recognition task, dataset, and evaluation that the reported solution was built to win.","marker":"[74]"},{"why":"Provides the Mask R-CNN base architecture that the signboard detection stage modifies.","marker":"[17]"},{"why":"Supplies deformable convolutional networks, credited with a +0.6% gain in signboard detection.","marker":"[10]"},{"why":"Introduces BoxDQN, the reinforcement-learning method that adjusts text box shapes for better recognition, credited with +1.6%.","marker":"[71]"},{"why":"Supplies the ViT backbone used in the text recognizer, credited with +2.5%.","marker":"[14]"},{"why":"Provides sequence-to-sequence contrastive learning, one of the two self-supervised pre-training objectives for the recognizer.","marker":"[2]"},{"why":"Supplies the masked autoencoder scheme that the stroke-and-semantic masked image modeling pre-training is based on.","marker":"[18]"},{"why":"Provides the feature sampling and grouping paradigm that the joint text detection and key information extraction network follows.","marker":"[73]"}],"fun_headline_variants":["Four-stage pipeline wins shop-sign OCR with F-score 0.6672","Reinforcement learning and graph nets win streetscape sign recognition","Self-supervised pretraining propels sign OCR team to first place","Winning sign OCR: multimodal fusion, BoxDQN, and text rectification","Graph network links text to signboards in first-place OCR system"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported improvements from each added technique are real and caused by that technique, since the paper gives no controlled comparisons or uncertainty estimates.","fun_headline_variants_meta":{"raw":{"variants":["Four-stage pipeline wins shop-sign OCR with F-score 0.6672","Reinforcement learning and graph nets win streetscape sign recognition","Self-supervised pretraining propels sign OCR team to first place","Winning sign OCR: multimodal fusion, BoxDQN, and text rectification","Graph network links text to signboards in first-place OCR system"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1386,"prompt_tokens":880,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":496,"tokens_out":506,"duration_ms":5911,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:03:00.393763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the same competition test set, removing one technique at a time with a fixed random seed and reporting standard deviations; if removing self-supervised pre-training does not lower the F-score by roughly 3.2 points, or if any of the seven listed gains reverse sign, the causal attribution in the paper's results table is refuted.","supporting_citations":[{"cited_title":"Character recognition competition for street view shop signs","cited_arxiv_id":null,"evidence_quote":"Defines the street-view shop sign recognition task, dataset, and evaluation that the reported solution was built to win."},{"cited_title":"Optimal boxes: boosting end-to-end scene text recognition by adjusting annotated bounding boxes via reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Introduces BoxDQN, the reinforcement-learning method that adjusts text box shapes for better recognition, credited with +1.6%."},{"cited_title":"Few could be better than all: Feature sampling and grouping for scene text detection","cited_arxiv_id":null,"evidence_quote":"Provides the feature sampling and grouping paradigm that the joint text detection and key information extraction network follows."}],"review_version":1}