{"id":"7df16f12-1243-44a9-af76-e86cf7ca9868","arxiv_id":"2412.20440","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"CASAT adds session segmentation, retrieval-augmented plot summaries, and style statistics to LLM prompts for context-aware entertainment translation into Indian languages.","lead":"This paper proposes CASAT, a pipeline that feeds movie-scene context and style descriptors into a large language model to translate entertainment dialogues into Hindi, Bengali, and Telugu. It reports modest COMET gains and win-ratio improvements over prompt-only baselines, but the appendix is unfinished and no code or data are released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline win-ratio and COMET gains are not established: the GPT-4o judge lacks human validation and the COMET deltas lack significance testing, with the paper's own appendix conceding a metric mismatch for creative translations.","rationale":"The paper's central claim is a comparative empirical statement about translation quality. The only support is two automatic metrics whose validity for this task is not shown. The win-ratio is a GPT-4o preference judgment with no disclosed prompt, no human calibration, and no handling of ties, so it could reflect prompt-induced stylistic biases (e.g., length or explicitness) rather than viewer-relevant quality. COMET is a reference-free quality-estimation model; it is not designed to reward culturally creative divergences, and the appendix itself admits the one-to-many nature of the task and the resulting BLEU mismatch. The presence of a concrete cell (Mistral-7B En-Tel) where COMET decreases while the win-ratio is 0.8 strengthens the worry that at least one metric is measuring something orthogonal to the intended construct. Even taking the metrics at face value, the absence of confidence intervals or significance tests means the word 'significant' in the abstract is unsubstantiated; several deltas are 0.01–0.03. A human-evaluation correlation study plus bootstrap CIs would directly settle whether the measured gains reflect actual entertainment-translation quality. Because the reader's REJECT verdict already rests on this evaluation-validity weakness, my read does not change the verdict.","tokens_in":16769,"tokens_out":12790,"duration_ms":111643,"concrete_test":"Run a blind human preference study on a stratified sample of 200 sentence-pairs per language direction from the Table 1 test set (including the Mistral-7B En-Tel and LLaMa3-8B En-Ben cells), presenting source, CASAT output, and baseline output in randomized order to at least three professional subtitle translators or dubbing editors per pair, and ask them to choose the better translation and rate adequacy, fluency, and engagement. Compute Cohen's kappa between the GPT-4o win-ratio judgments and the human majority preference. Additionally, bootstrap 95% confidence intervals for each Table 1 COMET delta on the 800-sample cells. If human–GPT-4o kappa is below 0.4, or if the COMET confidence intervals straddle zero in more than half of the rows, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: CASAT 'consistently outperforms baseline LLMs in terms of win-ratio' and delivers 'significant improvement in the COMET scores' (Abstract). The only evidence is Table 1/3 COMET (wmt22-cometkiwi-da) plus a GPT-4o win-ratio defined in the Evaluation Metrics paragraph. For the claim to hold, these metrics must track viewer/dubbing preference and the reported deltas must be outside noise; neither is established. The win-ratio uses GPT-4o as judge with no judge prompt, order randomization, tie option, or human correlation; the cited GPTScore work does not validate dubbing-preference judgment. COMET is reference-free quality estimation trained on adequacy/fluency judgments, not calibrated for the one-to-many creative translations that define this task. The paper's own appendix ('Possible Reasons for Low Automatic metric scores') concedes such translations 'are rarely literal or word-for-word' and 'often result in lower BLEU scores,' yet BLEU and COMET are still used as headline evidence. Table 1 shows a direct conflict: for Mistral 7B En-Tel, CASAT's COMET falls from 0.42 to 0.41 and BLEU from 0.1 to 0.07 while the GPT-4o win-ratio is 0.8, indicating at least one metric is not measuring the intended construct. No confidence intervals, bootstrap, or significance tests are reported; many COMET gains are 0.01–0.05, plausibly within metric noise on 800 samples. The central empirical claim is therefore unsafe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CASAT (Context And Style Aware Translation), a pipeline for translating entertainment dialogue without timestamps or speaker IDs. CASAT segments a movie or series into 'sessions' by tonal genre, uses an LLM to write plot summaries for nearby sessions, retrieves relevant plot chunks via a reranking RAG module, and extracts style descriptors (POS-based content/function words, syllabic word frequencies, modal words/idioms, intent, emotion) through a Domain Adaptation Module. These context and style signals are concatenated into a prompt for an LLM translator. The method is evaluated on web-scraped parallel subtitle data for English-to-Hindi, English-to-Bengali, and English-to-Telugu, across seven LLM baselines, with BLEU, reference-free COMET (wmt22-cometkiwi-da), and a GPT-4o-based win-ratio. The authors report COMET gains of roughly 0.01--0.05 and win-ratios mostly in the 0.6--0.85 range, and conclude that context and style awareness consistently improve LLM-based entertainment translation.","tokens_in":17216,"tokens_out":4255,"duration_ms":41294,"significance":"If the empirical claims were reliable, CASAT would be a practical, language-agnostic recipe for improving LLM translation in the under-served entertainment domain, with the useful property of not depending on timestamps or speaker identification. Strengths of the submission include a clear problem formulation, explicit pseudocode for the algorithm, a hyperparameter table (Table 6), and ablation experiments (Table 2) that attempt to attribute gains to context versus style. However, the central claim is empirical, and the evidence as presented is not yet convincing: the automatic metrics are used in a regime the paper itself acknowledges is problematic for creative translation, the win-ratio judge is not validated, and no uncertainty quantification is provided. The significance would be much higher with a human evaluation or a validated creative-translation metric.","major_comments":[{"comment":"The abstract's claim of 'significant improvement' and 'consistently outperforms' is not supported by any significance testing, confidence intervals, or error bars. The COMET deltas in Table 1 are mostly 0.01 to 0.05, which is plausibly within metric noise for 800 samples per language direction. The win-ratio uses GPT-4o as a judge, but the paper does not specify the judge prompt, tie handling, order randomization, or any human correlation; the cited GPTScore work does not establish that GPT-4o is a valid judge of dubbing-style preference. Without these controls, the central empirical claim is not established.","section":"Experiments, Evaluation Metrics; Table 1"},{"comment":"The manuscript itself concedes that entertainment translations in this dataset 'are rarely literal or word-for-word' and 'often result in lower BLEU scores,' yet BLEU and COMET are used as headline evidence in Tables 1 and 2. This is a metric-validity problem: the one-to-many mapping of creative translation means a low BLEU or moderate COMET may not indicate poor quality. Table 1 provides a direct internal contradiction: for Mistral 7B En-Tel, CASAT decreases BLEU (0.10 to 0.07) and COMET (0.42 to 0.41) while the GPT-4o win-ratio is 0.8, so at least one of the metrics is not tracking the intended construct. A human evaluation, or a validated metric for creative subtitles, is needed to support the headline claims.","section":"Appendix, 'Possible Reasons for Low Automatic metric scores'; Table 1"},{"comment":"Several parts of the appendix referenced in the main text are missing or incomplete. The main text refers to 'the exact prompt used for baseline LLMs' in the Appendix, but no such prompt appears. The subsection 'Choice of hyper-parameters' is empty. Table 7 has a placeholder caption for a 13-column table with no contents. These omissions prevent the reader from verifying the experimental setup and reproducing the comparisons, which is load-bearing for the central empirical claim.","section":"Appendix"}],"minor_comments":[{"comment":"The abstract claims the method is 'language and LLM-agnostic,' but the experiments cover only English-to-Indian-language directions (En-Hi, En-Ben, En-Tel). This overstates the empirical scope; the claim should be softened or supported by additional language pairs.","section":"Abstract and Experiments"},{"comment":"There is a duplicated phrase: 'classifies each session into one of the three three primary tonal categories.' Please remove the extra 'three'.","section":"Methodology, 'Adaptive Session Classification and Segmentation'"},{"comment":"The header of Table 3 reads 'Wini-ratio' instead of 'Win-ratio.'","section":"Table 3"},{"comment":"The sentence 'As detailed in Section ,' contains an empty cross-reference; it should point to the specific section or be reworded.","section":"Appendix, 'Additional Ablation Experiments'"},{"comment":"The example text inside Figure 4 appears to repeat the same clause ('The conversation's have many long sentences, rich connotations, specific narratives, detailed reasoning, and full of emotions') twice in different boxes; verify that this is not a copy-paste artifact.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an incomplete draft: the placeholder Table 7 and the empty 'Choice of hyper-parameters' subsection suggest the appendix was not finalized. More importantly, the evaluation section needs a human-validated quality signal before the central claim can be trusted; a straightforward revision with significance testing, a full appendix, and either human evaluation or a validated creative-translation metric would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of the CASAT paper. The core idea is sensible: segment movie dialogues into sessions, retrieve plot context via RAG, extract style descriptors via a domain adaptation module, and feed all that into an LLM prompt for translation. For Indian-language content without timestamps or speaker IDs, that's a reasonable engineering recipe, and the paper is straightforward about what it does. The ablation showing context alone helps more than style alone, and that CASAT combines both, is a nice sanity check.\n\nBut the empirical claims don't hold up. The headline 'consistent outperformance' rests on two fragile metrics. Reference-free COMET deltas are 0.01–0.05 on 800 samples with no significance tests, and the paper's own appendix admits creative translations 'are rarely literal' and 'often result in lower BLEU scores' — yet BLEU and COMET are the headline evidence. The GPT-4o win-ratio has no judge prompt, no order randomization, no tie option, and no human correlation. There's also a direct red flag in Table 1: for Mistral 7B En-Tel, CASAT drops COMET from 0.42 to 0.41 and BLEU from 0.1 to 0.07, yet the win-ratio is 0.8. At least one of those metrics is not measuring what viewers care about.\n\nThe manuscript is also unfinished: Table 7 is a placeholder caption, baseline prompts are missing from the appendix, and the hyper-parameter section is nearly empty. That alone would justify a reject in most venues.\n\nWhat's genuinely useful: the problem is real, the session-based segmentation with RAG is a plausible way to inject long-range context without timestamps, and the style descriptor idea (dialogue-level POS/function words, modal particles, session intent) is concretely specified. If a human evaluation showed these descriptors actually change viewer preference, this would be a solid applied contribution.\n\nMy take: reject as-is, but not because the idea is bad. It needs a completed appendix, a small human or dubbing-expert evaluation on a sample, and at minimum bootstrapped confidence intervals on COMET. If the authors fix that, it becomes a reasonable conference paper. I would not send this version to referees — the placeholder table and missing prompts make it impossible to check. I'd tell the authors to resubmit with the evaluation fixed.","headline":"A plausible prompt-engineering recipe for Indian-language entertainment translation, but the evaluation is too fragile and the manuscript too unfinished to accept.","tokens_in":17726,"tokens_out":3059,"would_cite":false,"duration_ms":28310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that feeding a large language model a per-scene estimate of plot context and style, rather than a fixed creative prompt, yields consistently better entertainment translations across eight LLMs, as measured by…","keywords":["entertainment translation","neural machine translation","large language models","context-aware prompting","style adaptation","retrieval-augmented generation","Indian languages","subtitle translation"],"falsifier":"A blind human preference study on the same test sets would settle the claim: if bilingual viewers or professional subtitle translators do not choose CASAT translations over baseline LLM outputs at a rate consistent with the reported win-ratio, the measured gains would not reflect audience-relevant quality.","tokens_in":16578,"feed_emoji":"🎬","tokens_out":8187,"duration_ms":71402,"temperature":0.7,"pith_summary":"This paper tries to show that movie and series translation into Indian languages improves substantially when the prompt given to a large language model is rebuilt for each scene from two estimates: the current plot context and the scene's style. It introduces CASAT, a pipeline that splits a script into tonal sessions, retrieves plot summaries relevant to the line being translated from a vector store, and extracts style markers such as word-class frequencies, modal words, idioms, intent, and emotion. On web-scraped subtitles for English-to-Hindi, English-to-Bengali, and English-to-Telugu, CASAT-augmented LLMs outperform their own baseline prompts and traditional MT systems on reference-free COMET and on a GPT-4o win-ratio, across eight LLMs from 7B to 70B parameters. The significance is that the method requires no timestamps, speaker IDs, or retraining, so it could be a drop-in wrapper for existing localization pipelines.","feed_headline":"LLM translation wins more when prompts adapt to plot and style","feed_subtitle":"Retrieving plot context and scene style for each line beats static prompts across eight LLMs and three languages.","key_machinery":"The central object is the time-varying prompt $p_t = h(\\tilde{c}_t, \\tilde{s}_x)$ that adapts to the current session's estimated context $\\tilde{c}_t$ and style $\\tilde{s}_x$. The context estimate comes from a retrieval-augmented generation module: an offline LLM creates plot summaries of K-session windows, chunks them into a vector database, and a retriever-reranker selects the N most relevant chunks for the source sentence. The style estimate comes from a Domain Adaptation Module that pools dialogue-level statistics (content/function word distributions via an Indic POS tagger, frequent syllabic words, modal words, idioms) with session-level signals (punctuation-derived intent and LLM-extracted emotion). The session segmentation algorithm (Alg 1) is what makes these estimates time-dependent, by grouping sentences into nearly constant-tone sessions; without it the prompt collapses to a static style hint.","core_discovery":"The central claim is that context and style are not static properties of entertainment content but vary from scene to scene, so a translation prompt conditioned on static instructions misses what a line needs. CASAT operationalizes this by generating context and style estimates per session. A session classifier groups sentences into serious, casual, or neutral segments using cosine similarity over sentence embeddings, with windowing to keep sessions coherent. An offline LLM writes plot summaries for K consecutive sessions, which are chunked and indexed in a vector database; at translation time a retriever and reranker select the N most relevant chunks as context. In parallel, a Domain Adaptation Module computes style descriptors: part-of-speech distributions for content versus function words, frequent monosyllabic and polysyllabic words, modal particles and idioms, punctuation-derived intent, and an LLM-generated emotion label. These estimates are concatenated into the prompt that guides the LLM's translation. The experiments report that this adaptive prompt beats a fixed creative prompt, that plot context alone improves relevance while style alone improves literal fidelity, and that combining the two gives the best COMET and win-ratio results across all tested LLMs.","pith_inferences":["If the win-ratio reflects a genuine audience preference, the same session-aware prompt construction could be applied to other creative-generation tasks, such as dubbing scripts segmentation or dialogue rewriting, where style descriptors are already estimated.","The paper's style descriptors were designed for Indian languages; a testable extension would be running CASAT on other language families to see whether the same descriptors (syllable patterns, modal words) transfer or need to be redesigned.","The authors note that the offline plot-design component is a target for removal; if the whole pipeline could run online, live streaming localization would become the natural deployment scenario."],"forward_implications":["Entertainment translation into Indian languages can be improved without retraining models: wrapping off-the-shelf LLMs with session-adaptive prompts raises COMET scores and win-ratios over both static creative prompts and traditional MT systems like IndicTrans2 and NLLB.","The method generalizes across model families and sizes because it only changes the prompt; the paper reports gains across eight LLMs from 7B to 70B and three language pairs (English to Hindi, Bengali, and Telugu).","Context and style contribute complementary gains: plot context improves relevance as measured by COMET and win-ratio, style improves literal fidelity as measured by BLEU, and the full CASAT prompt yields the best combined scores.","Retrieving plot chunks from a vector store outperforms passing the surrounding sentences as context, indicating that non-adjacent scenes often carry the decisive information for a line, and that the retrieval window should not be limited to immediate neighbors.","Larger LLMs exploit the adaptive prompt better, so the quality gap between base and CASAT-augmented translation grows with model capacity."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation formulation (Maximum Inner-Product Search) that the Context Retrieval module extends for plot-context retrieval.","marker":"(Lewis et al. 2021)"},{"why":"Supplies the reranker (Re2G retrieve-rerank-generate) used to select the N most relevant plot chunks.","marker":"(Glass et al. 2022)"},{"why":"Provides the Domain Adaptation Module concept for style definition that the paper adapts for dialogue- and session-level style extraction.","marker":"(Tao et al. 2024)"},{"why":"Supplies wmt22-cometkiwi-da, the reference-free COMET metric used for the main quality comparisons.","marker":"(Rei et al. 2022)"},{"why":"Backs the use of GPT-4o as a quality judge that replicates human-level judgment for the win-ratio metric.","marker":"(Fu et al. 2023)"},{"why":"Provides IndicTrans2, one of the traditional MT systems that CASAT-augmented LLMs are compared against.","marker":"(Gala et al. 2023)"},{"why":"Motivates the need for cultural awareness and creativity in LLM translation, which the paper frames as the problem CASAT addresses.","marker":"(Yao et al. 2024)"}],"fun_headline_variants":["Adaptive prompts boost LLM entertainment translation","Per-scene context and style improve LLM translations","CASAT: adaptive context-style prompting for better dubbing","LLMs translate entertainment better with dynamic prompts","Session-aware prompts beat static ones for subtitle translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that reference-free COMET and a GPT-4o preference judge measure what audiences actually value in entertainment translation, since no human evaluation is included in the study.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive prompts boost LLM entertainment translation","Per-scene context and style improve LLM translations","CASAT: adaptive context-style prompting for better dubbing","LLMs translate entertainment better with dynamic prompts","Session-aware prompts beat static ones for subtitle translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2485,"prompt_tokens":997,"completion_tokens":1488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1416}},"tokens_in":613,"tokens_out":1488,"duration_ms":10263,"temperature":1.0,"reasoning_tokens":1416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:21:32.381358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blind human preference study on the same test sets would settle the claim: if bilingual viewers or professional subtitle translators do not choose CASAT translations over baseline LLM outputs at a rate consistent with the reported win-ratio, the measured gains would not reflect audience-relevant quality.","supporting_citations":[],"review_version":1}