{"id":"0286b906-5c74-4884-a6da-e2b9db3892da","arxiv_id":"2412.13161","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BanglishRev provides 1.74 million Daraz product reviews in Bangla, English, Banglish and code-mixed text, and a BanglishBERT model trained on rating-derived labels reports 94% accuracy on a manual sentiment benchmark.","lead":"This paper releases BanglishRev, a 1.74-million-review dataset of Bengali, English, Banglish and code-mixed product reviews from Daraz Bangladesh, with ratings and metadata. A sentiment model trained on rating-derived labels reached 94% accuracy on an existing manually labeled benchmark, but the evaluation has caveats that the authors do not address.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern is train/test overlap between BanglishRev and the Rashid et al. test set; dataset value stands, but the 94% accuracy needs an overlap check.","rationale":"The reader's weakest assumption—possible train/test leakage because both corpora were scraped from Daraz, combined with rating-threshold labels—is the most load-bearing concern I could identify. It is concrete, testable, and directly affects the headline accuracy and the claim of beating the baseline, yet it does not undermine the dataset's existence or its primary value as a resource. I agree with the reader that the verdict should be CONDITIONAL: the resource should be released and used, but the evaluation claims need an overlap check, multiple seeds, and confidence intervals before the accuracy claim can be taken at face value. No additional concern rises to the level of changing the verdict. The paper contains no formal proofs to check, but it does ship a public dataset link and precise scraping/normalization details, which are forms of independent support for the resource claim. The weaknesses the paper itself flags—class imbalance (78% 5-star), potential spam, seller-deleted negative reviews, and the computational cost—are acknowledged limitations rather than hidden assumptions, and they are consistent with making the benchmark claim conditional rather than unconditional. One internal inconsistency worth noting but not load-bearing: the dataset analysis text lists 37.7% Banglish, then 25.04% Bangla, 31.27% English, and 5.92% code-mixed, and the prose says '25.04% in English' in one sentence while the later breakdown gives 31.27% English with 25.04% Bangla; the abstract and figures do not depend on this, and the later detailed breakdown is self-consistent. This is a writing-quality issue, not a correctness risk for the central claim. Overall, the reader's concern is the right one, and the conditional verdict is appropriate.","tokens_in":9300,"tokens_out":2702,"duration_ms":19398,"concrete_test":"Compute exact-match overlap (after the same normalization pipeline used in Section 4.1.1) between the BanglishRev review texts and the Rashid et al. (2024) test reviews, and also measure near-duplicate similarity (e.g., token-set Jaccard >= 0.9) to catch lightly edited duplicates. If overlap is above a small threshold (say > 0.5% of the test set), re-run Experiment 2 with those reviews removed and report the new accuracy/F1. Additionally, re-run Experiment 2 with three random seeds and report mean and 95% confidence intervals to check whether the 94% figure is stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that BanglishRev is the largest Bengali e-commerce review dataset and that rating-derived labels train a useful sentiment model—rests on two empirical pillars: (1) the dataset is indeed 1.74M written reviews with 3.2M ratings across 128k products, and (2) the trained BanglishBERT model reaches 94% accuracy/0.94 F1 on the Rashid et al. (2024) test set. The paper reports the collection pipeline in enough detail to make the resource credible, and it self-reports limitations about class imbalance and training cost. The weakest point is not fabrication or internal inconsistency but evaluation leakage: both BanglishRev and Rashid et al. (2024) are scraped from Daraz, and the paper reports no overlap check between the 1.74M training reviews and the ~78k annotated test reviews. Since Daraz reviews are public and the collection windows may overlap, identical reviews could appear on both sides, inflating the 94% figure. A second, smaller issue is that the rating threshold (rating > 3 vs <= 3) is itself a proxy label, and the baseline in Table 2 uses manually annotated labels, so the comparison is not apples-to-apples; however, the paper explicitly frames the experiment as 'effectiveness of rating as sentiment classification label', so this is a framing caution rather than a fatal flaw. The dataset contribution is independently valuable and the limitation section is candid, so the concern is real but conditional, not a rejection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BanglishRev, a large-scale dataset of product reviews from Daraz Bangladesh, containing 1.74 million written reviews drawn from 3.2 million ratings across 128k products, with rich metadata such as review text, rating, dates, likes/dislikes, seller responses, and image links. The authors describe their scraping pipeline, present descriptive statistics and language-distribution analyses, and report a sentiment-classification experiment in which a BanglishBERT model is trained on BanglishRev with ratings converted to binary labels (rating >= 4 positive, <= 3 negative) and evaluated on the externally annotated Rashid et al. (2024) test set, achieving a claimed accuracy of 94% and F1 of 0.94. The paper also discusses limitations, ethical considerations, and several future research directions.","tokens_in":9552,"tokens_out":4629,"duration_ms":43095,"significance":"If the dataset is as described, it is a substantial contribution: it is an order of magnitude larger than existing Bangla e-commerce review datasets, includes metadata that other datasets lack, and is released on HuggingFace for public use. The paper is transparent about its scraping procedure and limitations, and it explicitly anonymizes user and product IDs. The sentiment evaluation, however, is the main supporting evidence for the dataset's utility, and that evaluation currently has unresolved threats to validity (possible train/test leakage, insufficient statistical rigor, and confounded comparisons). The dataset itself is valuable, but the magnitude of the claimed experimental result needs to be verified with additional analysis.","major_comments":[{"comment":"The claimed superiority of the BanglishRev-trained model over the Rashid et al. baseline is not substantiated because no overlap check is reported between the BanglishRev training reviews and the Rashid et al. test reviews. Both datasets are scraped from Daraz Bangladesh, and the BanglishRev collection period (April–May 2024, §4.2) could overlap with the period from which the Rashid et al. reviews were collected. Since the paper does not report any deduplication or overlap analysis, identical reviews may appear in both the training set and the test set, inflating the reported 94% accuracy and 0.94 F1. Please conduct and report an exact-match and near-duplicate overlap analysis (e.g., MinHash or normalized text hashing) between the BanglishRev training data and the Rashid et al. test set, and rerun the evaluation on the non-overlapping subset or after removing overlapping instances from training. Without this, the central experimental claim is unsupported.","section":"§4.1.2, Tables 3 and 5"},{"comment":"The experiments compare models that differ in both training data size and label source, so the observed accuracy difference cannot be attributed to 'rating as sentiment label' (the section's stated focus). The baseline is trained on 80% of the 78k manually labeled Rashid et al. reviews, whereas the BanglishRev models are trained on 1.74M rating-derived labels. To support the claim that rating information is an effective substitute for manual sentiment labels, add a control experiment that trains on a random subsample of BanglishRev of comparable size to the Rashid et al. training split (or trains on the Rashid et al. reviews with rating-derived labels), so that data volume and label source are not confounded.","section":"§4.1, Tables 2–5"},{"comment":"Results are reported from a single run with no variance estimates, and the numbers are not stable: after one epoch Experiment 2 gives 95% accuracy, while after three epochs it gives 94%. The 1-point gap relative to the baseline's 93% is within likely stochastic noise. Please report the mean and standard deviation over multiple random seeds, and if feasible a paired significance test over the common test set. Additionally, the abstract's 'F1 score of 0.94' is the weighted average, which is dominated by the majority positive class (>78% five-star reviews, §4.2); the negative-class recall of 0.72 in Table 3 is the actual weakness and should be reported in the abstract or at least in the main text near the accuracy claim.","section":"§4.1.2, Tables 3 and 5"}],"minor_comments":[{"comment":"The language distribution sentence is internally inconsistent: it states '37.7% of the total reviews are written in Banglish, 21.2% in English, 25.04% in English and only 5.9% in code-mixed language,' but the counts in the next paragraph give 25.04% Bangla, 31.27% English, 5.92% code-mixed, and 37.77% Banglish. Please correct the text and ensure Figure 3 matches.","section":"§3.2"},{"comment":"The number of root categories is inconsistent: §3.1 says 'eight root categories' while the caption of Figure 4 says '12 root categories' and §4.3 also says '8 root categories.' Please clarify the correct number and use it consistently.","section":"§3.1 and §4.3"},{"comment":"The sentence 'Experiment 1 and 2 are conducted with the BanglishRev dataset being used for training' is confusing because Experiment 1 is the baseline trained and tested entirely on Rashid et al. Please rephrase to explicitly state the training and test set for each of the three experiments.","section":"§4.1.2"},{"comment":"There are several typographical errors: 'languague' (Introduction), 'scarped' (§3.1), 'Implemetation' (§4.1.1), 'Google Colaboratory' (§4.1.1), and 'Heath and Beauty' (§3.2). A careful proofread is needed.","section":"Throughout"},{"comment":"The dataset is a central contribution, so its long-term availability matters. Please verify that the HuggingFace URL is stable and consider providing a versioned DOI or commit hash, along with a data card that describes the anonymization procedure and the intended use conditions.","section":"Abstract and dataset link"}],"recommendation":"major_revision","confidential_remarks":"The dataset resource is the primary contribution and appears credible; the experimental evaluation is secondary but is used to justify the dataset's utility. The missing overlap check is the key concern because both datasets come from the same platform and the collection windows may overlap. If the authors can provide a clean overlap analysis and rerun the evaluation, the paper could be acceptable. I would not reject outright, as the issue is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the resource is real and worth having; the headline accuracy number is not yet trustworthy. The dataset itself is the contribution. At 1.74M written reviews with ratings, dates, likes, seller replies, and image links, it is an order of magnitude larger than the 78k Rashid et al. benchmark, and the collection pipeline (category tree, product URLs, review pages) is described concretely enough that the corpus is credible. The paper is honest about the 78% five-star imbalance and about not balancing it. That candor counts.\n\nThe main soft spot is the evaluation. Both BanglishRev and the Rashid et al. test set are scraped from Daraz, review text is public, and the collection windows can overlap (BanglishRev was collected April–May 2024; Rashid et al. 2024 is earlier but Daraz pages persist). The paper reports no de-duplication or overlap check between the 1.74M training reviews and the test set. If any test reviews appear in training, the 94% accuracy and the claim of beating the baseline are inflated. This is a real leak risk, not a hypothetical. The fix is straightforward: check exact and near-duplicate review IDs/text and report the overlap; also add multiple seeds, confidence intervals, and a class-balanced evaluation. Without that, the benchmark claim should be read as potentially optimistic.\n\nA second, smaller issue: the comparison in Table 3 vs Table 2 is not apples-to-apples. Experiment 1 trains on manually annotated labels; Experiments 2/3 train on rating-derived labels. The paper frames this as measuring rating-as-label effectiveness, so it is a framing caution rather than a flaw, but the better-than-baseline phrasing overstates it.\n\nThere is also a concrete internal inconsistency in Section 3.2. The first language distribution says 37.7% Banglish, 21.2% English, 25.04% English, 5.9% code-mixed; the second enumeration says 25.04% Bangla, 31.27% English, 5.92% Bangla-English, 37.77% Banglish. The second set of numbers is the coherent one; the first must contain a typo. An editor should ask them to fix it.\n\nOverall: the dataset is a genuine resource for Bengali e-commerce NLP and low-resource work, and the paper is honest enough about limitations that it deserves a serious referee. I would accept it for review but ask for an overlap check, error bars, and a corrected language distribution before publication.","headline":"The dataset is real, large, and useful; the 94% accuracy needs an overlap check before it is trusted.","tokens_in":10116,"tokens_out":1938,"would_cite":true,"duration_ms":16485,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces BanglishRev, claimed to be the largest e-commerce review dataset for Bengali customers, with 1.74 million written reviews from 3.2 million ratings across about 128,000 products, and reports that a BanglishBERT model…","keywords":["BanglishRev","Bengali e-commerce reviews","code-mixed text","Banglish transliteration","sentiment analysis","BanglishBERT","rating-derived labels","Daraz Bangladesh"],"falsifier":"Compute the overlap between BanglishRev's training reviews and the manually annotated test set after text normalization; if a material share of test reviews appear in training, the reported accuracy is inflated. A complementary check is to annotate a random sample of BanglishRev reviews and compare the rating-derived labels with human sentiment judgments to see whether the proxy holds outside the test benchmark.","tokens_in":9091,"feed_emoji":"🛒","tokens_out":7738,"duration_ms":62611,"temperature":0.7,"pith_summary":"BanglishRev is presented as the largest e-commerce product-review dataset for the Bengali-speaking market: 1.74 million written reviews drawn from 3.2 million ratings across roughly 128,000 products on the Daraz Bangladesh platform. Each review carries rich metadata—the rating, dates of review and purchase, likes and dislikes, seller replies, and image links—so the corpus is positioned to support not only sentiment analysis but also spam detection, market analysis, and consumer-behavior studies. The paper's main experiment tests whether star ratings can substitute for human sentiment labels: a BanglishBERT model trained on BanglishRev with ratings above 3 marked positive and 3 or below marked negative is evaluated on a previously published manually annotated 78,000-review benchmark. The model reaches 94% accuracy and 0.94 F1, which the authors take as evidence that the dataset's scale compensates for the noise in rating-derived labels.","feed_headline":"Bengali review dataset hits 94% sentiment accuracy at 1.74M reviews","feed_subtitle":"Using star ratings as labels, the model beats a manually annotated baseline—a route to low-cost Bangla sentiment analysis.","key_machinery":"The load-bearing artifact is the dataset itself, produced by scraping Daraz's three-tier category tree, collecting product URLs, and then extracting review details page by page. The experimental machinery is a normalization pipeline that detects whether a review is Bangla, English, Banglish, or code-mixed, transliterates Banglish words to Bangla phonetics with the avro library, and feeds the text into BanglishBERT, a BERT-based model pre-trained with an ELECTRA-style replaced-token-detection objective. The rating-to-label mapping—positive for ratings greater than 3, negative for ratings 3 or below—is what converts the unlabeled corpus into supervised training data, and the previously published manually annotated benchmark supplies the held-out test set.","core_discovery":"The central claim is that BanglishRev is the largest and most metadata-rich public review dataset for Bengali e-commerce, and that it can serve as effective training data for binary sentiment analysis even though it carries no manual sentiment labels. Using rating-derived labels, a BanglishBERT model trained on BanglishRev reaches 94% accuracy and 0.94 F1 on a manually annotated 78,000-review test set, outperforming a baseline model trained and tested on that same manually annotated set (93% accuracy with a weaker negative-class F1 of 0.78). The paper further documents the corpus's language mix—roughly 37.8% Banglish, 31.3% English, 25.0% Bangla, and 5.9% code-mixed—and notes that more than 78% of ratings are five stars, an imbalance it deliberately preserves for downstream tasks.","pith_inferences":["A natural but untested consequence is that the rating-proxy model's reported F1, measured on a test set balanced differently than real Daraz traffic, overstates performance on the platform's actual review stream, where positive reviews dominate.","The class-wise results after three epochs (positive precision 0.96 and recall 0.98; negative precision 0.84 and recall 0.72) imply asymmetric errors: a deployed system would miss a substantial share of negative reviews unless thresholds are re-tuned.","Because the corpus comes from a single platform, the paper's conclusions about Bengali e-commerce language patterns may not transfer to other marketplaces; collecting comparable reviews from booksellers, food delivery, and other verticals would test the generality.","The image links, if maintained, could support multimodal analyses—for example, whether photo similarity to the product listing predicts satisfaction—but the dataset's reliance on external URLs makes this capability fragile over time."],"forward_implications":["If BanglishRev is as large as reported, it becomes the default scale reference for Bengali e-commerce NLP, replacing prior datasets that top out at 78k reviews.","The metadata—review and purchase dates, likes, dislikes, seller replies, and image links—makes rating prediction, spam review detection, and consumer behavior analysis feasible on the same corpus, not just sentiment.","The rating-labeling scheme, if transferable, offers a low-cost path to sentiment classifiers for other low-resource languages where manual annotation is scarce.","The observed dense clusters of near-identical five-star reviews provide a concrete starting point for spam-review detection research in Bengali.","The language-mix statistics (37.8% Banglish, 31.3% English, 25.0% Bangla, 5.9% code-mixed) give a realistic picture of how Bengali customers actually write, which can inform future tokenizers and transliteration tools."],"supporting_citations":[{"why":"Supplies the manually annotated 78,130-review benchmark used as the held-out test set and as the baseline comparator for the 94% accuracy claim.","marker":"Rashid et al. (2024)"},{"why":"Provides BanglishBERT, the pretrained Bangla-English language model fine-tuned on BanglishRev for sentiment classification.","marker":"Bhattacharjee et al. (2021)"},{"why":"Introduces the BERT architecture that BanglishBERT builds on, grounding the model choice and tokenization settings.","marker":"Devlin et al. (2018)"},{"why":"Describes the ELECTRA replaced-token-detection objective used to pre-train BanglishBERT, which the paper credits with its cross-lingual transfer.","marker":"Clark et al. (2020)"},{"why":"Supplies prior evidence that star ratings can serve as noisy sentiment labels, the premise behind the rating-threshold labeling in the main experiment.","marker":"Nguyen et al. (2014)"}],"fun_headline_variants":["1.74M Bangla-English reviews fuel 94% sentiment accuracy","Code-mixed review dataset powers 94% F1 sentiment model","Largest Bengali e-commerce review dataset hits 0.94 F1","Rating-labeled Banglish reviews achieve 94% sentiment accuracy","BanglishRev: 1.74M reviews, 94% accuracy, no manual labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that none of the reviews in the manually annotated test set also appear in BanglishRev's training data, even though both corpora come from the same e-commerce platforms, and the paper reports no de-duplication or overlap check; any leakage would inflate the 94% accuracy.","fun_headline_variants_meta":{"raw":{"variants":["1.74M Bangla-English reviews fuel 94% sentiment accuracy","Code-mixed review dataset powers 94% F1 sentiment model","Largest Bengali e-commerce review dataset hits 0.94 F1","Rating-labeled Banglish reviews achieve 94% sentiment accuracy","BanglishRev: 1.74M reviews, 94% accuracy, no manual labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3667,"prompt_tokens":1062,"completion_tokens":2605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":2508}},"tokens_in":678,"tokens_out":2605,"duration_ms":17704,"temperature":1.0,"reasoning_tokens":2508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:21:30.581004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between BanglishRev's training reviews and the manually annotated test set after text normalization; if a material share of test reviews appear in training, the reported accuracy is inflated. A complementary check is to annotate a random sample of BanglishRev reviews and compare the rating-derived labels with human sentiment judgments to see whether the proxy holds outside the test benchmark.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the manually annotated 78,130-review benchmark used as the held-out test set and as the baseline comparator for the 94% accuracy claim."},{"cited_title":"Q., Vu, T., Pham, S","cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence that star ratings can serve as noisy sentiment labels, the premise behind the rating-threshold labeling in the main experiment."}],"review_version":1}