{"id":"08b206b2-b954-44a9-b83b-e053d077d767","arxiv_id":"2507.20025","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RICE improves vision encoders by applying cluster discrimination at the region level and unifying object and OCR classification targets in one pretraining framework.","lead":"RICE is a new method for pretraining image encoders that learns from object regions and text regions, not only whole images. It reports large gains over CLIP and SigLIP on OCR, detection, segmentation, and multimodal benchmarks, though part of the object signal is inherited from CLIP's own features.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claims RICE 'consistently outperforms' prior methods, but Table 1 shows RICE-560px loses to Qwen2.5-ViT and SigLIPv2 on a majority of benchmarks; the central claim is overstated.","rationale":"I chose the overclaim over the reader's pseudo-label concern because it is a direct, internally verifiable contradiction of the central claim. The paper's own Table 1 shows that at 560px RICE loses on a substantial fraction of benchmarks, so the abstract's 'consistently outperforms' is not merely imprecise but false as written. The pseudo-label concern is a mechanism-level hypothesis about whether gains come from CLIP distillation versus genuinely better object semantics; even if that hypothesis is correct, the empirical results still support a useful contribution (region-level cluster discrimination plus OCR supervision). The overclaim, by contrast, requires a straightforward revision to the paper's headline claims and does not depend on external assumptions. The reader's verdict of CONDITIONAL remains appropriate, but the condition should include toning down 'consistently' to match the actual results; I therefore recommend no change to the verdict itself. My concrete test is a simple tally from the existing table, so it can be settled immediately without new experiments.","tokens_in":17042,"tokens_out":10600,"duration_ms":125057,"concrete_test":"Compute a win/loss tally across every benchmark in Table 1 for each RICE configuration against its closest baseline (RICE-336px vs AIMv2-336px; RICE-378px vs SigLIPv2-384px; RICE-560px vs SigLIPv2-560px and Qwen2.5-ViT-560px). If the number of losses is non-negligible (e.g., at least three in any comparison), revise the abstract, introduction, and conclusion from 'consistently outperforms' to 'outperforms on most OCR and perception benchmarks', and add a limitation section acknowledging resolution-dependent tradeoffs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RICE 'consistently outperforms previous methods' (Abstract; Sec. 5). This is contradicted by the paper's own Table 1. At 560px, RICE-ViT-L-14-560px loses to Qwen2.5-ViT-ViT-H-14-560px on 9/15 benchmarks (InfoVQA -2.76, ChartQA -0.76, TextVQA -4.74, OCRBench -55, OCRBenchV2 -0.70, AI2D -1.88, MMEPer -31, RealworldQA -0.92, MMStar -4.51). It also loses to SigLIPv2-560px on 8/15 benchmarks (TextVQA -1.27, OCRBench -20, OCRBenchV2 -0.33, AI2D -0.10, MMEPer -12, POPE -0.32, RealworldQA -3.14, MMStar -2.59), including the 'Other Avg' aggregate (-0.14). Thus the phrase 'consistently outperforms' is not supported by the reported results. The reader's weakest assumption—that CLIP-derived pseudo-labels limit object semantics—is plausible but does not invalidate the empirical gains; the overclaim is a direct internal inconsistency that misstates the findings.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RICE, a region-aware cluster discrimination method for visual representation learning. It builds a billion-scale region dataset from SAM masks and PaddleOCR text, assigns object pseudo-labels by k-means on frozen CLIP features (Eq. 1), and introduces a Region Transformer layer with a mask-guided attention mechanism. The training loss combines a single-label object cluster discrimination loss and a multi-label OCR loss. Experiments evaluate RICE as a vision encoder in LLaVA-NeXT and LLaVA-OneVision frameworks, on referring segmentation, detection, tracking, and show hyperparameter ablations. The paper claims RICE consistently outperforms prior methods across segmentation, dense detection, and visual perception for MLLMs.","tokens_in":17356,"tokens_out":4154,"duration_ms":48440,"significance":"If the results hold, RICE provides a single vision encoder that improves region-level object semantics and OCR in one forward pass, with strong results on OCR-centric benchmarks and referring segmentation. The controlled LLaVA-NeXT setup with Qwen2.5-7B reduces some comparison bias, and the authors release the pretrained models, which supports reproducibility. The scale of pretraining (13B samples) is substantial. However, the object supervision is inherited from CLIP features, so the claimed object-semantic improvement is partly a distillation of CLIP geometry rather than an independent signal. The empirical gains still appear meaningful, especially at 336/378px, but the headline claim of 'consistently outperforms' is not supported by the paper's own Table 1 at 560px.","major_comments":[{"comment":"The claim that RICE 'consistently outperforms previous methods' is contradicted by Table 1. At 560px, RICE ViT-L-14-560px loses to Qwen2.5-ViT-ViT-H-14-560px on 9 of 15 benchmarks (e.g., TextVQA -4.74, OCRBench -55, MMStar -4.51) and loses to SigLIPv2-560px on 8 of 15 benchmarks, including the 'Other Avg' aggregate (-0.14). The text in Sec. 4.2 selectively highlights wins on InfoVQA and DocVQA while omitting the negative differences. Please rephrase the claim to specify the regimes where RICE is superior (e.g., OCR-centric tasks and lower resolutions) and report an aggregate or significance test that supports any global statement.","section":"Abstract, Sec. 5, Table 1"},{"comment":"The loss formulations are incomplete and thus not reproducible. Eq. (3) as written is 'Lobject = log(1 + exp(-sim(...)) + log(1 + sum ...' with mismatched parentheses; the second log term has no closing parenthesis and the negative-class sum is not clearly inside the log. Eq. (4) has the same problem. Please give precise, well-parenthesized definitions, and also specify the sampling procedure for negative centers in the object loss (the reference to [2] is insufficient without the exact uniform-sampling subset defined in the text).","section":"Sec. 3.3, Eqs. (3) and (4)"},{"comment":"The ablation study covers hyperparameters (N, K, rho, M) but does not isolate the main components of the method: the Region Transformer layer, the object cluster discrimination loss, and the OCR loss. Without removing each component in turn, the gains cannot be attributed to the proposed mechanisms. Add component ablations on at least one detection and one OCR benchmark, e.g., training with only the OCR loss, only the object loss, and with global attention instead of region attention.","section":"Sec. 4.5"},{"comment":"The object pseudo-labels are produced by k-means on frozen CLIP features, so the object supervision is a distillation of CLIP's feature geometry. The paper should state this explicitly and analyze its implication: the improvement over CLIP on object-centric tasks may come from region-level training and OCR labels rather than from genuinely better object semantics. Provide an experiment that separates these factors, such as training RICE with random or uniform region labels (or with the object branch removed) to measure the marginal contribution of the cluster-based object loss.","section":"Sec. 3.1, Eq. (1)"},{"comment":"All results are reported from single runs with no variance estimates or statistical significance. Many differences in Table 1 are within a few tenths of a point (e.g., RICE-560 vs SigLIPv2-560 on POPE is -0.32). For a claim of 'consistently outperforms', please report standard deviations over at least 2-3 seeds for the principal comparisons (Table 1 and Table 3), or use paired tests where feasible.","section":"Tables 1-4"}],"minor_comments":[{"comment":"Column headers in Table 5a are '1 5 10 20' while the row label says 'Testset'; these are values of the sampled-region count N, not test sets. Rename the column header to 'N' for clarity.","section":"Table 5a"},{"comment":"The text says features are extracted 'using the CLIP model' without specifying which variant (e.g., ViT-L/14, OpenCLIP, or Data Filtering Networks). Specify the exact CLIP checkpoint used for pseudo-label generation.","section":"Eq. (1)"},{"comment":"The phrase 'processing a total of 13 billion samples during the initial pretraining stage' is ambiguous: it is unclear whether this is the number of unique samples, the number of samples seen per epoch, or the cumulative number over all epochs. Clarify the relation to dataset sizes.","section":"Sec. 4.1"},{"comment":"The term 'previous methods' is used broadly but the comparison set includes Qwen2.5-ViT, which is described as a specialized backbone. Specify in the abstract or introduction whether the claim covers only methods available at the time of submission or also contemporaneous specialized models.","section":"Abstract and Sec. 5"},{"comment":"Define the similarity function sim(·) explicitly (e.g., cosine similarity after L2 normalization) and note the margin m and scale s from Sec. 4.1, which do not appear in the loss equations.","section":"Eq. (3)-(4)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a large-scale pretraining effort and controlled MLLM evaluation, but the internal inconsistency between the abstract claim and Table 1, the incomplete loss equations, and the missing component ablations prevent acceptance in current form. The CLIP-distillation concern is a genuine interpretational issue that the authors should address rather than ignore. The scope of the requested changes is limited to experiments and text revision, so major_revision is the appropriate verdict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: RICE is a solid engineering contribution that does something genuinely new at region level—cluster discrimination on object regions plus multi-label OCR tokens in one encoder, with a mask-guided Region Transformer to batch variable-size regions. The evaluation is extensive and the controlled LLaVA-NeXT + Qwen2.5 setup is a real plus. But the headline claim \"consistently outperforms\" is not supported by their own Table 1, and the object supervision is partly CLIP distilled. That doesn't kill the paper, but it needs fixing.\n\nWhat's new: prior cluster discrimination (UNICOM, MLCD) works at image level; RICE moves to region-level, combining SAM regions with CLIP k-means labels and PaddleOCR text tokens, and unifies both in a single classification loss. The Region Transformer layer with a visibility mask is a neat solution to the variable-size region problem. The gains on detection, segmentation, tracking, and DocVQA are consistent across many benchmarks, and ablations on key hyperparameters are included. Releasing models is good practice.\n\nSoft spots, in order of severity:\n\n1. The \"consistently outperforms\" language in Abstract/Conclusions is contradicted by Table 1 at 560px: RICE-560px loses to Qwen2.5-ViT on 9/15 benchmarks and to SigLIPv2 on 8/15, including the Other Avg aggregate. The text in Sec 4.2 says it \"maintains its lead,\" which is not what the table shows. This needs to be fixed with more careful phrasing and a balanced discussion of where RICE wins and loses.\n\n2. The object labels come from k-means on frozen CLIP features (Eq. 1), so object semantics are partly distilled from CLIP. This is a real limitation, but it's shared with UNICOM and many region-level methods; it doesn't invalidate the gains, and the OCR branch uses independent PaddleOCR text. Still, the paper should be explicit that the object side is CLIP-dependent.\n\n3. All tables report single runs with no variance. For a paper with a claimed +50 OCRBench gain over CLIP, that's acceptable as a first report, but it weakens the precision of the claims. Ablations are on 10% of data only, so hyperparameter conclusions may not transfer to the full-scale model.\n\nThe math and architecture are coherent. The citation pattern is fine; self-citations to UNICOM/MLCD are appropriate since those are the direct baselines.\n\nBottom line: this is a useful, well-executed paper that overstates its own results in the abstract. A serious editor should send it to review; the authors should be asked to tone down the consistency claim, report variance or at least multiple seeds, and discuss the CLIP-distillation dependence. I'd bring it to reading group and would cite it for the region-level cluster discrimination idea if I were working on vision encoders.","headline":"Solid region-level cluster discrimination paper with a real overclaim in the abstract; worth reviewing but needs a tone-down and variance reporting.","tokens_in":17879,"tokens_out":2444,"would_cite":true,"duration_ms":26261,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RICE, a region-aware cluster discrimination pretraining method, produces a vision encoder that outperforms previous encoders on segmentation, dense detection, and OCR-heavy multimodal perception, including surpassing a specialized OCR…","keywords":["visual representation learning","region-based pretraining","cluster discrimination","OCR","dense prediction","segmentation","multimodal LLM","vision transformer"],"falsifier":"Train a variant of RICE where the object region labels are replaced by random assignments to the one million centers, keeping the region masking and OCR losses identical. If the variant still matches RICE's gains on COCO detection and DocVQA, then the specific object cluster targets are not what drives the improvement.","tokens_in":16832,"feed_emoji":"🖼️","tokens_out":9871,"duration_ms":99957,"temperature":0.7,"pith_summary":"This paper argues that a vision encoder pretrained with region-aware cluster discrimination—supervising region-level object recognition and OCR through a single classification loss—can outperform global-contrastive encoders such as CLIP and SigLIP on dense prediction, OCR, segmentation, and multimodal perception tasks. The authors build RICE, which extracts region embeddings through a Region Transformer layer that masks attention to each region, and trains them against one million semantic cluster centers for objects and token embeddings for OCR text. On benchmarks such as DocVQA, COCO detection, referring segmentation, and video tracking, RICE consistently beats the baselines, and at 560px resolution it surpasses the specialized backbone of a leading multimodal model on document question answering. If the results hold, a single pretrained encoder could provide object-level semantics and OCR reading in a single forward pass, reducing the need for separate specialized vision backbones.","feed_headline":"One pretraining method lifts OCR, detection, segmentation at once.","feed_subtitle":"Region-aware cluster discrimination gives one encoder top scores on OCR, detection, and segmentation.","key_machinery":"The central mechanism is the unified region cluster discrimination loss, which turns every image region into a classification problem: object regions are classified against one million k-means cluster centers derived from frozen CLIP features (single-label), and OCR regions are classified against token embeddings of the detected text (multi-label). The Region Transformer layer carries out this supervision: it applies a region-specific visibility mask (0 inside a region, −∞ outside) to a standard multi-head attention operation, so each region's class token only sees tokens within that region, and produces fixed-length embeddings for all regions in one forward pass. Training uses a margin-based softmax with L2-normalized features and centers, and a random negative-sampling strategy that selects 10% of class centers as negatives to avoid conflicting gradients.","core_discovery":"RICE is a fully region-supervised visual representation learning method. The paper's central claim is that treating image regions as classification targets—single-label object clusters plus multi-label OCR token embeddings—under one unified cluster discrimination loss produces a vision transformer whose local features transfer to a broad set of dense and OCR-heavy tasks better than global vision-language contrastive models and previous cluster discrimination models. The method constructs a dataset of 400 million images with 2 billion object regions (from segmentation masks) and 400 million OCR regions (from detected text), assigns each object region a pseudo-label by nearest-neighbor matching of its CLIP feature to one of one million k-means centers, and then trains a ViT with additional Region Transformer layers that restrict self-attention to region tokens. The object and OCR losses share the same softmax-margin formulation, enabling distributed training at a global batch size of 32K on 64 GPUs. The paper reports that this encoder outperforms CLIP, SigLIP, MLCD, and others on COCO/LVIS detection and segmentation, referring segmentation, DocVQA/OCRBench, and MLLM visual perception, reaching 87.38% DocVQA at 560px in a standard multimodal LLM setup and beating a specialized OCR-optimized vision backbone.","pith_inferences":["If the gains persist when the frozen CLIP feature extractor is replaced by a different (e.g., SigLIP-derived) feature backbone for clustering, region cluster discrimination could become a general recipe layered on any strong global encoder.","The paper's cluster pseudo-labels are essentially a quantization of CLIP's region feature manifold; one could test a testable extension where the cluster centers are learned end-to-end instead of frozen, or re-clustered periodically, to see whether object semantics improve beyond the CLIP ceiling.","The random negative sampling rate ρ=0.1 and the use of only 10 sampled regions per image suggest a high degree of redundancy in both region data and negative classes; ablating region sample diversity (e.g., by sampling the largest or most salient regions) might reveal whether small, hard-detectable regions benefit more from the region supervision.","The unified loss treats OCR tokens as positive classes; an implicit corollary is that the same framework could be extended to other dense annotation types (e.g., depth or keypoint tokens) as positive classes, but the paper does not test this."],"forward_implications":["A single vision encoder from this method can serve as the vision tower for multimodal LLMs, improving OCR and dense understanding without a separate OCR-optimized backbone.","Region-level supervision transfers to object tracking: RICE improves video tracking success rates on LaSOT, TrackingNet, GOT-10k, and TNL2K over prior encoders.","The unified object/OCR classification loss scales to billion-scale region data, suggesting that region-level cluster discrimination is a practical path to further scaling visual representation learning.","Because the region attention layer produces fixed-length region embeddings in one forward pass, downstream tasks like referring segmentation can directly consume region features without region-specific fine-tuning of the encoder.","The ablation results indicate that 10 sampled regions per image, 1–2 million cluster centers, and a 0.1 negative sampling ratio are near-optimal; these are actionable defaults for follow-up work."],"supporting_citations":[{"why":"Supplies frozen CLIP features for clustering region pseudo-labels and serves as the main baseline.","marker":"[50]"},{"why":"Generates the fine-grained mask regions that form the object region dataset.","marker":"[30]"},{"why":"Extracts OCR text regions and their labels from the image datasets.","marker":"[18]"},{"why":"Provides the cluster discrimination loss and the one-million-center labeling recipe adopted for object regions.","marker":"[3]"},{"why":"Supplies the multi-label cluster discrimination formulation that the OCR loss extends.","marker":"[4]"},{"why":"Gives the margin-based softmax with L2 normalization used in both region losses.","marker":"[16]"},{"why":"Contributes the random negative-class sampling strategy that keeps the loss efficient.","marker":"[2]"},{"why":"Implements hierarchical k-means on GPU, making the 2-billion-region clustering tractable.","marker":"[17]"},{"why":"Defines the referring segmentation setup in which RICE's region features are evaluated.","marker":"[31]"}],"fun_headline_variants":["Region cluster pretraining boosts OCR, detection, and segmentation all at once","A single regional clustering loss teaches dense tasks and OCR together","Region-aware clustering pretraining outperforms CLIP on dense tasks and OCR","Unified region loss pushes vision transformer beyond global contrastive models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The object region pseudo-labels come from k-means on frozen CLIP features, so RICE's object supervision is only as good as CLIP's ability to organize region features into semantically meaningful clusters; if those clusters are noisy or biased, the model inherits that noise.","fun_headline_variants_meta":{"raw":{"variants":["Region cluster pretraining boosts OCR, detection, and segmentation all at once","A single regional clustering loss teaches dense tasks and OCR together","Region-aware clustering pretraining outperforms CLIP on dense tasks and OCR","Unified region loss pushes vision transformer beyond global contrastive models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000418,"raw_usage":{"total_tokens":2163,"prompt_tokens":967,"completion_tokens":1196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1122}},"tokens_in":583,"tokens_out":1196,"duration_ms":10005,"temperature":1.0,"reasoning_tokens":1122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:50:44.422879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a variant of RICE where the object region labels are replaced by random assignments to the one million centers, keeping the region masking and OCR losses identical. If the variant still matches RICE's gains on COCO detection and DocVQA, then the specific object cluster targets are not what drives the improvement.","supporting_citations":[{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Generates the fine-grained mask regions that form the object region dataset."},{"cited_title":"Arcface: Additive angular margin loss for deep face recognition","cited_arxiv_id":null,"evidence_quote":"Gives the margin-based softmax with L2 normalization used in both region losses."},{"cited_title":"Lisa: Reasoning segmenta- tion via large language model","cited_arxiv_id":null,"evidence_quote":"Defines the referring segmentation setup in which RICE's region features are evaluated."}],"review_version":1}