{"id":"9580048b-edff-4492-8611-f443fb348b91","arxiv_id":"2502.07058","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs predict sentiment worse on Taiwan Mandarin than Mainland Mandarin reviews, using a new contextually paired dataset from Booking.com.","lead":"This paper builds a paired dataset of Taiwan Mandarin and Mainland Mandarin hotel reviews from Booking.com, matching reviews for the same hotel with similar ratings and lengths, and tests six LLMs on sentiment prediction. The authors find that most models perform worse on Taiwan Mandarin reviews, and they argue this low-cost approach can benchmark language-variety disparities at scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human study shows TW reviews are harder for humans too, so the LLM gap may reflect task difficulty rather than LLM-specific bias.","rationale":"The reader's weakest assumption is exactly the point I identify: the pairing criteria do not rule out content differences that make TW reviews harder, and the human study confirms that humans also find TW harder. This is the most load-bearing issue because the paper's headline claim—LLMs specifically underperform in Taiwan Mandarin—loses its LLM-specific meaning if the gap is simply a task difficulty effect. The paper itself acknowledges this in Section 5 ('Plausible') and in Limitations, but the abstract and conclusion still present the finding as an LLM disparity. The MT experiment provides some evidence for a variety-level effect, but it is not decisive because translation may alter content. A direct human-LLM comparison on the same pairs would settle whether LLMs exhibit an additional bias beyond human difficulty. I agree with the reader's CONDITIONAL verdict: the method is novel and useful, but the central claim needs qualification and additional evidence. Since my concern aligns with the reader's existing analysis, no verdict adjustment is needed.","tokens_in":14538,"tokens_out":5145,"duration_ms":45718,"concrete_test":"Use the exact 50 TW-CN pairs from the human study (Appendix B.2) and run the six LLMs under the same structured and plain prompts. Compute accuracy and MSE gaps (CN − TW) for humans and for each LLM. Use a paired bootstrap or mixed-effects model to test whether the mean LLM gap is significantly larger than the human gap on the same pairs. If the LLM gap is not significantly larger, the central claim must be reframed as a task-difficulty difference rather than an LLM-specific disparity. Also report how many of the 6 models show a gap in the same direction as humans.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'LLMs consistently underperform in Taiwan Mandarin' is not established as an LLM-specific bias because the paper's own human study (Section 5, Appendix B.2) shows human raters also perform significantly worse on TW reviews (accuracy 28.75% vs 37.00%, p=.016; MSE 3.510 vs 2.795, p=.036). This indicates the TW reviews in these pairs are intrinsically harder to predict, independent of model. Pairing on same hotel, same coarse rating, and length bin (Section 3.1) does not control for content properties like directness, lexical choice, or framing that affect predictability. The MT experiment (Table 3) shows translating TW to CN improves GPT-4o, which suggests a variety effect, but it does not isolate LLM bias from human difficulty. Moreover, Table 1 shows that several models (e.g., Gemma2 9B/27B, Llama3-8B) perform better on TW in structured or short settings, contradicting 'consistently.' Without a direct comparison showing LLMs underperform more than humans on identical pairs, the observed gap is confounded by task difficulty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes using contextually aligned online hotel reviews to benchmark LLM performance across language varieties. The authors collect 22,918 review pairs from Booking.com, each containing one Taiwan Mandarin (TW) and one Mainland Mandarin (CN) review of the same hotel with ratings in the same coarse 3-class bin and similar text length. Six LLMs (GPT-4o, three Llama3 sizes, two Gemma2 sizes) predict the 1-10 rating under structured, plain, and shuffled input formats. The authors report that in most model/setting/length combinations, accuracy is lower and MSE higher for TW reviews, that disrupting review structure widens the gap, that shorter reviews show larger MSE gaps, that machine translation from TW to CN improves GPT-4o performance while the reverse direction is mixed, and that human raters also predict CN reviews more accurately than TW reviews. Confound analyses address writing quality, code-mixing, and score distributions. The central claim stated in the abstract is that 'LLMs consistently underperform in Taiwan Mandarin.'","tokens_in":14726,"tokens_out":11128,"duration_ms":90823,"significance":"The dataset construction method is the paper's core contribution, and it is a real one: pairing reviews by the same hotel, coarse rating class, and length bin yields 22,918 TW-CN pairs at a fraction of the cost of native-speaker instance-level translation, and the code and data are released. The paper is notably transparent about confounds: it reports human validation of writing quality and content-rating agreement, code-mixing rates, score distributions, an MT directionality experiment, and a human rating-prediction study. The aggregate empirical pattern - 36 of 38 significant accuracy gaps and 45 of 47 significant MSE gaps favoring Mainland Mandarin across six models and three input formats - is a substantial and falsifiable finding, and if it survives the confound-control analyses suggested below it would justify the abstract's claim.","major_comments":[{"comment":"The authors' own human study shows that native speakers also predict ratings significantly better for Mainland Mandarin reviews than for Taiwan Mandarin reviews (accuracy 37.00% vs. 28.75%, p=.016; MSE 2.795 vs. 3.510, p=.036), and the paper labels this confound 'Plausible.' Yet the abstract and Section 7 nonetheless assert that 'LLMs consistently underperform in Taiwan Mandarin.' As it stands, the evidence establishes a performance disparity on a contextually aligned rating-prediction task, not an LLM-specific bias: the TW reviews in the paired sample appear intrinsically harder to predict, for humans and models alike. Because this attribution is the paper's headline claim, the point is load-bearing. The paper should either (i) reframe the claim as a task-level disparity, or (ii) provide a direct comparison of model gaps with human gaps on identical pairs, or an analysis stratifying pairs by human-rated predictability, to support the bias reading. The post-hoc exclusion of the two lowest-performing participants (both Taiwan-Mandarin speakers, Appendix B.2) further complicates the human baseline and should be justified or analyzed with and without exclusion.","section":"Section 5; Appendix B.2"},{"comment":"The word 'consistently' is contradicted by the paper's own data. In the structured-short condition, Gemma2 9B and Gemma2 27B show significantly better accuracy on TW than on CN (Delta Acc = -1.24** and -1.07**), and in 8 of the 54 model/setting/length cells in Table 1 accuracy is numerically higher for TW (e.g., Llama3-8B structured overall, Delta Acc = -1.01). Of the 38 statistically significant accuracy gaps, 2 favor TW. The direction of the disparity therefore depends on model, input format, and metric; 'consistently' overstates the pattern. The finding should be reported as a majority pattern with the counts (e.g., 36 of 38 significant accuracy comparisons and 45 of 47 significant MSE comparisons favor CN), and the abstract's wording should be aligned with that.","section":"Table 1, rows for Gemma2"},{"comment":"Invalid predictions are excluded rather than modeled, and the invalid counts reported in Table 6 are substantial and model-dependent: Llama3-8B has 168 invalid predictions in the plain condition and 139 in the structured condition, and Llama3-405B has 130 invalid in the structured condition. Because the analysis keeps only pairs in which both members produced a valid score, any systematic relationship between invalidity and language variety would bias the estimated gap; for example, if TW reviews more often trigger off-range or non-numeric outputs, dropping those pairs would differentially remove hard TW items. The paper should report invalid counts separately by variety and setting, and include a sensitivity analysis (e.g., treating invalid outputs as a separate outcome class or imputing them) to demonstrate that the headline gaps are robust to the exclusion mechanism.","section":"Appendix D; Section 4"},{"comment":"The 54 experiments are reported with per-cell significance stars and no correction for multiple comparisons; at alpha=.05 one expects roughly 2.7 spurious significant cells by chance. The aggregate directional counts (36/38 and 45/47) are clearly robust to this concern, but load-bearing secondary claims are not. In particular, the claim that 'shorter reviews tend to produce larger MSE gaps' is inferred from comparing the pattern of stars across rows of Table 2 rather than from a test of the length by variety interaction; the paper should confirm this with a model that includes the interaction (with model as a random effect, or per-model interaction tests), and should also state which statistical test produced the reported p-values (paired t-test, McNemar, or bootstrap).","section":"Section 4; Tables 1-2"}],"minor_comments":[{"comment":"The caption says the significance notation and color coding 'follow the same conventions as in Table 2'; this should refer to Table 1.","section":"Table 2 caption"},{"comment":"The opening sentence, 'Of the world's 7,000 languages, sixty (60) million people speak British English, 23 million speak Taiwan Mandarin, and 10 million speak European Portuguese, compared to...' is ungrammatical and appears to be a damaged list item; please rewrite it.","section":"Section 1"},{"comment":"It is not stated whether the Mainland Mandarin reviews were validated by native Mainland Mandarin speakers or by the same five Taiwan Mandarin raters; this matters for interpreting the small quality differences, so rater backgrounds should be reported for both groups.","section":"Section 3.2"},{"comment":"Participants were shown 'both the English text and its translation'; since the study evaluates Chinese reviews, please clarify why the English text was displayed and whether it could influence the quality and agreement judgments.","section":"Appendix B.1"},{"comment":"The text refers to 'presented in 9' where Table 9 is meant.","section":"Appendix G"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is honest and well-executed on the empirical side; the dispute is interpretive, not a matter of fabrication or circularity. The risk is that the abstract overclaims relative to the authors' own Section 5 caution. I have asked in the major comments for the claim to be reframed or supported by a direct human-vs-model comparison on identical pairs, and for a sensitivity analysis of the invalid-prediction exclusion. The dataset and pairing methodology are genuinely useful to the community, and the paper is appropriate for the journal's scope. No concerns about citation practice or novelty disclosure; the positioning against Ziems et al.'s VALUE/MultiVALUE work and DialectBench is fair."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the method is genuinely new and worth building on; the headline empirical claim is real but weaker than the abstract suggests, because the authors' own human study shows TW reviews are harder for humans too. I'd send it out, but I'd ask for a softened abstract and a release of the paired data.\n\nThe core idea is to use naturally occurring online reviews as contextually aligned pairs—same hotel, same coarse rating bin, same length bin—so that two Mandarin varieties are compared on the same real-world scenario without paying for expert translation. That is a genuine contribution. The authors construct 22,918 TW-CN pairs from Booking.com, run six LLMs across three input formats, and find a majority of experiments show lower accuracy and higher MSE on Taiwan Mandarin. They also do a careful job examining confounds: writing quality, code-mixing, rating distributions, and a human-score-prediction study. The MT asymmetry result is a nice secondary finding.\n\nThe soft spots are in proportion. The human study is the big one: native speakers also predict TW ratings significantly worse (acc 28.75 vs 37.00, p=.016; MSE 3.510 vs 2.795, p=.036). That means the TW reviews in these pairs are intrinsically harder to predict, independent of any LLM bias. The pairing controls for hotel, coarse rating, and length, but not for content properties—directness, lexical choice, framing—that affect predictability. So the headline \"LLMs consistently underperform in Taiwan Mandarin\" is overstated; it should say \"on this contextually aligned sentiment-prediction task, most models show a gap, and part of that gap appears to reflect task difficulty rather than model-specific bias.\" Table 1 also shows some models do better on TW in some settings (e.g., Gemma2 9b/27b structured), so \"consistently\" is not literally accurate. The multiple comparisons across 54 experiments are not corrected; with ~50 tests, a handful of false positives is expected. Excluding invalid predictions rather than modeling them is a minor issue, though the appendix does report the counts. And the dataset is not released—only the processing code—which limits reproducibility.\n\nNone of this kills the paper. The method is the contribution, and it is a good one. The authors are transparent about the confounds and explicitly caveat the human-study interpretation. The citation pattern is normal and covers prior dialect-benchmarking work. I'd want the abstract revised and the data released, but this deserves serious refereeing. My bottom line: conditionally accept after those revisions.","headline":"A genuinely new method for cross-variety benchmarking, but the headline claim of consistent TW underperformance is weakened by the authors' own human study showing TW reviews are harder for humans too.","tokens_in":15274,"tokens_out":2146,"would_cite":true,"duration_ms":18637,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs consistently underperform on Taiwan Mandarin reviews even when hotel, rating, and length are matched.","keywords":["language varieties","Taiwan Mandarin","Mainland Mandarin","sentiment analysis","large language models","dialect bias","contextual alignment","hotel reviews"],"falsifier":"A decisive check would be a re-run in which every Taiwan Mandarin review is paired with a Mainland Mandarin review produced by professional human translation of the same content, holding hotel, rating class, length, and format constant; if the performance gap disappears on such semantically identical pairs, the reported disparity is attributable to content rather than to language variety, and the paper's central claim fails.","tokens_in":14342,"feed_emoji":"📉","tokens_out":6854,"duration_ms":59130,"temperature":0.7,"pith_summary":"This paper argues that international hotel-review platforms like Booking.com can be mined to compare how well large language models handle different varieties of the same language, without the usual cost of hiring native speakers to construct parallel examples. It pairs reviews written in Taiwan Mandarin with reviews written in Mainland Mandarin for the same hotel, the same coarse rating class, and a similar text length, then asks six LLMs to predict each review's 1-10 rating. Across 54 model-and-format experiments, models were significantly worse on Taiwan Mandarin reviews in the large majority of cases; the gap grew when the review's section structure was removed and was largest on short texts. The paper's larger claim is that this low-cost, contextually aligned design can be reused across many language varieties to expose where LLMs still fail.","feed_headline":"Six LLMs underperform on Taiwan Mandarin hotel reviews","feed_subtitle":"Matched Booking.com reviews show accuracy gaps that widen when section structure is stripped away","key_machinery":"The load-bearing mechanism is the contextually aligned review pair: each pair contains one Taiwan Mandarin and one Mainland Mandarin review from the same hotel, falling in the same coarse rating class (1-3, 4-7, 8-10) and in the same 10-token text-length bin. This pairing is meant to give instance-level comparability between language varieties without the expensive native-speaker translation or annotation used in earlier efforts. The experimental apparatus then varies input format - structured (title/positive/negative), plain (concatenated), or shuffled (random order) - to see how much of the variety gap depends on format cues, and compares GPT-4o, Llama3 (8B, 70B, 405B), and Gemma2 (9B, 27B).","core_discovery":"The central discovery is that LLMs consistently underperform on Taiwan Mandarin compared with Mainland Mandarin on a rating-prediction sentiment task built from naturally occurring hotel reviews. In the paper's results, 38 of 54 experiments showed significant accuracy differences between the varieties and 47 of 54 showed significant differences in mean squared error; among those significant differences, 36 of 38 accuracy gaps and 45 of 47 MSE gaps indicated worse performance on Taiwan Mandarin. The authors also report that the gap widens when inputs lose their section structure (plain or shuffled versus structured) and that shorter reviews show larger MSE gaps, while machine translation from Taiwan to Mainland Mandarin improves predictions but the reverse direction does not. A human-participant study found that native speakers also predicted ratings less accurately for Taiwan Mandarin reviews, which the paper takes as a caution that part of the gap may reflect task difficulty or content rather than a language-variety-specific deficit.","pith_inferences":["The same 'same hotel, same rating class, same length' pairing could be exported to other varieties with large review corpora, such as Brazilian versus European Portuguese or British versus American English, creating near-free dialect audits for future models.","Since native-speaker raters also scored Taiwan Mandarin lower, the study implies that any benchmark that reports an LLM gap should include a human-difficulty control; otherwise the gap cannot be attributed to the model.","A testable follow-up is to calibrate each model's outputs on a per-variety rating distribution: if the Taiwan gap shrinks after calibration, part of the disparity is a scoring-scale artifact rather than a language-understanding failure."],"forward_implications":["Six leading LLMs are currently less reliable at predicting Taiwan Mandarin reviews than Mainland Mandarin ones on matched hotel-review pairs.","The reliability gap grows when the model sees a plain paragraph or shuffled sections rather than the original title/positive/negative structure.","Short Taiwan Mandarin reviews are where the model's mean squared error is largest relative to Mainland Mandarin, so low-context user text is the riskiest input.","Machine translation between the two varieties is not a neutral pairing tool: TW-to-CN translation improves predictions, while CN-to-TW does not."],"supporting_citations":[{"why":"Supplies evidence that LLMs already show sentiment-prediction disparity across English varieties and motivates measuring the same across Mandarin varieties.","marker":"(Ziems et al., 2022)"},{"why":"Establishes instance-level alignment as the most rigorous comparison method that the paper extends with low-cost pairing.","marker":"(Ziems et al., 2023)"},{"why":"Quantifies dialect gaps across languages and supports the paper's machine-translation asymmetry interpretation.","marker":"(Kantharuban et al., 2023)"},{"why":"Recent Taiwan Mandarin understanding benchmark that motivates targeted cross-variety evaluation.","marker":"(Chen et al., 2024)"},{"why":"Case study showing text-variety differences can drive NLP performance disparities outside English.","marker":"(Blodgett and O'Connor, 2017)"},{"why":"Provides the code-mixing and cultural-context evidence used to test code-mixing as a confound.","marker":"(Ochieng et al., 2024)"}],"fun_headline_variants":["Taiwan Mandarin hotel reviews trip up six LLMs","LLMs consistently worse on Taiwan Mandarin reviews","Language variety gap: Six LLMs fail Taiwan Mandarin","Hotel reviews reveal LLM disparity for Taiwan Mandarin","Six LLMs biased against Taiwan Mandarin on hotel reviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument collapses if matched reviews still differ in something other than language variety, such as how explicitly the writer states their opinion, what they complain about, or how they use star ratings, because then the performance gap could come from task difficulty rather than from the model's handling of the variety itself.","fun_headline_variants_meta":{"raw":{"variants":["Taiwan Mandarin hotel reviews trip up six LLMs","LLMs consistently worse on Taiwan Mandarin reviews","Language variety gap: Six LLMs fail Taiwan Mandarin","Hotel reviews reveal LLM disparity for Taiwan Mandarin","Six LLMs biased against Taiwan Mandarin on hotel reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3704,"prompt_tokens":877,"completion_tokens":2827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2754}},"tokens_in":493,"tokens_out":2827,"duration_ms":17324,"temperature":1.0,"reasoning_tokens":2754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:54:39.429797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be a re-run in which every Taiwan Mandarin review is paired with a Mainland Mandarin review produced by professional human translation of the same content, holding hotel, rating class, length, and format constant; if the performance gap disappears on such semantically identical pairs, the reported disparity is attributable to content rather than to language variety, and the paper's central claim fails.","supporting_citations":[{"cited_title":"VALUE: Understanding Dialect Disparity in NLU","cited_arxiv_id":"2204.03031","evidence_quote":"Supplies evidence that LLMs already show sentiment-prediction disparity across English varieties and motivates measuring the same across Mandarin varieties."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes instance-level alignment as the most rigorous comparison method that the paper extends with low-cost pairing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Quantifies dialect gaps across languages and supports the paper's machine-translation asymmetry interpretation."}],"review_version":1}