{"id":"66906669-6ea4-4c6e-ba1f-e6556ba350f3","arxiv_id":"2605.27296","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs differ from humans in recognizing cultural stylistic features in translated texts, rely mainly on surface linguistic cues rather than stylistic structure in the Hong Kong case, and show inconsistent alignment between recognition and generation.","lead":"The paper builds a benchmark of stylized movie titles and ads from Hong Kong and mainland China to test whether LLMs can recognize and produce culturally specific language styles. A general reader might care because the results point to a gap between surface word patterns and deeper cultural feel that could affect real-world AI use in global media and marketing.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Logistic regression probes may fail to isolate stylistic structure if surface features (e.g., lexical or n-gram cues) are not explicitly controlled or orthogonalized","rationale":"The reader's weakest_assumption directly identifies the load-bearing step in the argument. The abstract-only limitation noted by the reader is the reason for UNVERDICTED; the full-text methods would need to demonstrate clean separation in the probes and human validation of C4STYLI distinctions for the claim to be secure. No other internal inconsistency appears from the provided abstract and claim description.","tokens_in":1674,"tokens_out":325,"duration_ms":21968,"concrete_test":"Re-run the logistic regression probes from the ablation section using only surface-feature controls (e.g., bag-of-words or character n-grams) versus the reported structural features; if probe accuracy on Hong Kong texts remains high (>70%) under surface-only controls, the claim that recognition relies primarily on surface information weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Hong Kong stylistic recognition in LLMs relies primarily on surface-level information—depends on the structural ablation succeeding at removing stylistic structure while preserving surface signals. The C4STYLI curation assumes the texts encode distinct Hong Kong vs. mainland stylistic structures distinguishable by humans, but if the probe features (whatever representation or input they use) entangle surface and structure, or if translation artifacts in movie titles/slogans introduce non-stylistic confounds, the ablation result does not support the interpretation of limited sensitivity to Hong Kong-specific structure.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript curates C4STYLI, a benchmark of highly stylized translated movie titles and advertising slogans from Hong Kong and the Chinese Mainland, to evaluate LLMs on behavioral recognition and productive competence in aesthetic stylistics. Extensive evaluations show LLMs differ from humans in stylistic recognition (varying by domain), that recognition and generation performance are not consistently aligned, and that logistic regression probes indicate stylistic recognition in the Hong Kong setting relies primarily on surface-level linguistic information rather than stylistic structure, suggesting limited sensitivity to Hong Kong-specific stylistic structure.","tokens_in":1799,"tokens_out":416,"duration_ms":25354,"significance":"If the ablation results hold after proper controls, the work offers a concrete empirical benchmark for cultural stylistic awareness in LLMs, grounded in direct model outputs and human comparisons rather than circular definitions. It highlights a potential gap between surface cues and deeper structural sensitivity in cross-cultural settings, which could guide future probing methods and training objectives for cultural competence.","major_comments":[{"comment":"Abstract and structural ablation description: the central claim that Hong Kong stylistic recognition relies primarily on surface-level information (rather than structure) is load-bearing on the logistic regression probes successfully isolating stylistic structure from surface features. No details are given on probe input representations, feature sets, how lexical/n-gram cues are orthogonalized or controlled, dataset size, or statistical controls, so the ablation result does not yet support the interpretation of limited sensitivity to Hong Kong-specific structure.","section":"Abstract / structural ablation"},{"comment":"C4STYLI curation and human evaluation: the assumption that the texts encode distinct Hong Kong versus mainland stylistic structures that humans reliably distinguish is central to interpreting the LLM results, yet no information is provided on curation criteria, translation artifact controls (e.g., movie titles/slogans), dataset size, or inter-annotator agreement.","section":"Abstract / C4STYLI benchmark"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive comments. We address each major point below and will revise the manuscript to supply the requested methodological details.","responses":[{"response":"We agree that the current description of the logistic regression probes is insufficient to support the interpretation. In the revised manuscript we will add a dedicated subsection (or appendix) specifying the probe input representations, the exact feature sets, the procedures used to orthogonalize or control for lexical and n-gram cues, the dataset sizes employed, and the statistical controls applied. These additions will allow readers to evaluate whether the probes successfully isolate stylistic structure.","revision_made":"yes","referee_comment":"[Abstract / structural ablation] Abstract and structural ablation description: the central claim that Hong Kong stylistic recognition relies primarily on surface-level information (rather than structure) is load-bearing on the logistic regression probes successfully isolating stylistic structure from surface features. No details are given on probe input representations, feature sets, how lexical/n-gram cues are orthogonalized or controlled, dataset size, or statistical controls, so the ablation result does not yet support the interpretation of limited sensitivity to Hong Kong-specific structure."},{"response":"We acknowledge that the manuscript currently provides limited information on benchmark construction. The revised version will include an expanded methods section (and appendix) that details the curation criteria for C4STYLI, the specific controls applied to mitigate translation artifacts in movie titles and slogans, the precise dataset sizes, and the inter-annotator agreement statistics from the human evaluation. These additions will strengthen the grounding for the assumption that the texts encode distinct stylistic structures.","revision_made":"yes","referee_comment":"[Abstract / C4STYLI benchmark] C4STYLI curation and human evaluation: the assumption that the texts encode distinct Hong Kong versus mainland stylistic structures that humans reliably distinguish is central to interpreting the LLM results, yet no information is provided on curation criteria, translation artifact controls (e.g., movie titles/slogans), dataset size, or inter-annotator agreement."}],"tokens_in":1355,"tokens_out":442,"duration_ms":24142,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The central result is that in the Hong Kong setting, LLM stylistic recognition tracks surface linguistic signals more than the actual stylistic structure, per their logistic regression probes.\n\nThey built C4STYLI from translated movie titles and advertising slogans that aim to capture Hong Kong versus mainland aesthetic differences. The evaluations test both recognition and generation, show LLMs diverge from human judgments, and note that recognition and generation do not track each other consistently across domains. The structural ablation is the piece that tries to explain why recognition works the way it does.\n\nThe new element is the benchmark itself plus the specific patterns on surface dependence and the recognition-generation split. Those empirical observations are not in the prior work the abstract cites, and the domain variation adds a useful data point for people tracking cultural handling in models.\n\nThe soft spot is the ablation. The abstract gives no numbers on dataset size, no description of how surface versus structure was operationalized in the probes, and no controls or agreement stats. If the logistic features still mix surface cues with structure, or if the curated texts carry translation artifacts, the claim that models show limited sensitivity to Hong Kong-specific structure does not follow. The stress-test concern about entanglement lands here because the abstract supplies no evidence that the separation succeeded.\n\nThis is for researchers running targeted evaluations of LLM cultural or stylistic competence. A reader who wants concrete numbers on where current models fall short on one language pair will find usable material, but only if the methods section fills in the gaps.\n\nIt deserves peer review so the ablation can be checked directly.","headline":"The paper's main observation is that LLMs lean on surface features for Hong Kong stylistic recognition in their new benchmark, but the ablation details are too thin to support that cleanly.","tokens_in":2330,"tokens_out":396,"would_cite":false,"duration_ms":27102,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLMs recognize Hong Kong stylistic features mainly through surface-level words rather than deeper stylistic structure.","keywords":["LLMs","stylistic recognition","cultural awareness","Hong Kong","aesthetic stylistics","C4STYLI","benchmark","ablation"],"falsifier":"An experiment in which humans fail to distinguish the Hong Kong and mainland texts in C4STYLI at above-chance levels, or in which LLMs maintain high recognition accuracy after surface features are removed, would undermine the central claim.","tokens_in":2568,"feed_emoji":"","tokens_out":629,"duration_ms":23133,"temperature":0.7,"pith_summary":"The paper introduces C4STYLI, a benchmark of translated movie titles and advertising slogans drawn from Hong Kong and mainland Chinese sources, to test how well LLMs handle aesthetic stylistics. Evaluations show LLMs differ from humans in style recognition, with performance varying by domain, and recognition ability does not reliably match generation ability. Structural ablation using logistic regression probes reveals that in the Hong Kong setting, LLMs depend primarily on surface linguistic cues instead of the underlying stylistic patterns. This points to limited sensitivity to Hong Kong-specific stylistic structure.","feed_headline":"LLMs spot Hong Kong style via surface words, not structure","feed_subtitle":"A benchmark of movie titles and slogans shows recognition depends on linguistic cues rather than stylistic patterns in the Hong Kong case.","key_machinery":"The C4STYLI benchmark of highly stylized translated movie titles and slogans, paired with structural ablation via logistic regression probes that isolate stylistic structure from surface features.","core_discovery":"In the Hong Kong setting, stylistic recognition in LLMs relies primarily on surface-level linguistic information rather than stylistic structure. This suggests limited sensitivity to Hong Kong-specific stylistic structure. The C4STYLI benchmark shows LLMs differ from humans in recognition, with domain variation and inconsistent alignment between recognition and generation tasks.","pith_inferences":["Similar surface-versus-structure probes could be run on other regional Chinese varieties or non-Chinese cultures to check whether the surface reliance pattern is widespread.","If surface cues dominate, targeted training on examples that emphasize structural differences might shift model behavior toward deeper stylistic capture.","The C4STYLI texts could serve as a fixed test set for measuring whether future models close the gap with human stylistic discrimination."],"forward_implications":["Stylistic recognition and generation performance in LLMs are not consistently aligned.","LLMs differ from humans in stylistic recognition ability.","Recognition performance varies across text domains such as movie titles and slogans.","LLMs show limited sensitivity to Hong Kong-specific stylistic structure in the tested setting."],"fun_headline_variants":["Surface linguistics drive LLM Hong Kong style recognition","LLMs depend on surface cues not stylistic structure","Limited LLM sensitivity to Hong Kong stylistic structure","Stylistic recognition in LLMs skips Hong Kong structure"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The logistic regression probes cleanly separate stylistic structure from surface linguistic features, and the C4STYLI texts encode distinct Hong Kong versus mainland stylistic structures that humans can reliably distinguish.","fun_headline_variants_meta":{"raw":{"variants":["Surface linguistics drive LLM Hong Kong style recognition","LLMs depend on surface cues not stylistic structure","Limited LLM sensitivity to Hong Kong stylistic structure","Stylistic recognition in LLMs skips Hong Kong structure"]},"model":"grok-4.3","cost_usd":0.006965,"raw_usage":{"total_tokens":3113,"prompt_tokens":600,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":69653000,"prompt_tokens_details":{"text_tokens":600,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2456,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":600,"tokens_out":57,"duration_ms":24953,"temperature":1.0,"reasoning_tokens":2456,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T17:50:25.484531+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment in which humans fail to distinguish the Hong Kong and mainland texts in C4STYLI at above-chance levels, or in which LLMs maintain high recognition accuracy after surface features are removed, would undermine the central claim.","supporting_citations":[],"review_version":1}