{"id":"53ecb527-899e-4082-ae0e-d4f25c18a622","arxiv_id":"2501.01123","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"TED adds dialogue-aware attention weighting (turn priority, speaker/listener factors) to a RoBERTa-based turn-averaging model and reports the best IEMOCAP score, though the gain is minimal.","lead":"This paper introduces TED, a method that adjusts Transformer attention weights during emotion classification to emphasize the current turn and its speaker. On four conversation datasets it reports top scores, but the gains over simpler baselines are tiny and may be within run-to-run noise.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The IEMOCAP SOTA is a post hoc best-test-grid selection without variance, so the 0.06-point lead over EmoBERTa and the dialogue-feature gain over TBM are not established.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the reported SOTA is a post-selection maximum over a test-performance grid, with no evidence of dev-set-based model selection. The paper's own Section 5.2 concedes that no significant differences were found across dialogue-feature combinations, and Tables 3 and 5 show that feature and speaker variants often hurt. This makes the central contribution of TED, the dialogue feature attention, unmeasurable from the reported numbers. The TBE and TBM components do have independent support in Tables 4 and 6, where they clearly improve over the current-turn baseline, and the methodology section is mostly reproducible in principle. However, the headline claim 'state-of-the-art on IEMOCAP' depends on a 0.06-point margin over EmoBERTa with no variance and a configuration chosen from the test grid. That is not sufficient evidence, so the reader's REJECT verdict is appropriate and should be preserved. If the concrete dev-set-selection check were to pass and seed variance turned out to be small, the verdict could be revisited as CONDITIONAL, but based on the text as written the concern lands.","tokens_in":11839,"tokens_out":3463,"duration_ms":35262,"concrete_test":"Re-run the protocol with configuration selection on the development set only: for each dataset, choose the context window, dialogue-feature combination, and gamma that maximize W-Avg F1 on the development split, then evaluate only that single configuration on the test split across the five fixed seeds. Report per-seed values, mean, and standard deviation. If the selected IEMOCAP test W-Avg F1 does not exceed 68.57 (EmoBERTa) and does not exceed TBM by more than the seed-level noise, the SOTA and dialogue-feature-improvement claims collapse. Additionally, run a paired significance test (e.g., per-dialogue bootstrap or McNemar) comparing the selected TED configuration against TBM and against EmoBERTa.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that TED achieves state-of-the-art performance on IEMOCAP (68.63 W-Avg F1 vs. 68.57 for EmoBERTa) and that the dialogue feature attention improves over the turn-based MHSA baseline. In Table 3, the IEMOCAP row reported in Table 2 (past context, Listener, Normdist) is one of twelve combinations of context window, dialogue-feature target, and priority decay. The value 68.63 is the maximum of the past-context column, exceeding the TBM baseline (68.50) by only 0.13 F1. The authors state in Section 5.2 that 'Varying the combinations of features yielded no significant differences,' and no variance or confidence intervals are reported despite five trials. The paper never states that the context window, dialogue-feature type, and gamma were locked on the development set before test evaluation. If these choices were instead selected after inspecting test labels, the headline number is an optimistic maximum over the grid, not the performance of a pre-specified model. The same issue affects the claim that dialogue features help: some feature combinations, such as past/Listener/Constant on IEMOCAP (68.12), are worse than TBM (68.50), and the claimed SOTA margin is smaller than typical seed noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TED (Turn Emphasis with Dialogue), a method for emotion recognition in conversation that builds on a pretrained RoBERTa encoder, averages token embeddings into turn-based vectors (TBE), applies multi-head self-attention over turn vectors (TBM), and then reweights attention scores with dialogue features: a turn-priority factor (constant or a normal distribution centered on the current turn) and a speaker/listener target mask. The method is evaluated on IEMOCAP, MELD, EmoryNLP, and DailyDialog. The central empirical claims are that TED reaches state-of-the-art performance on IEMOCAP (68.63 W-Avg F1 vs. 68.57 for EmoBERTa) and that the dialogue-feature attention provides an improvement over the turn-based MHSA baseline.","tokens_in":12154,"tokens_out":5212,"duration_ms":48830,"significance":"If the empirical claims were supported, the paper would offer a simple and inexpensive way to inject turn-level dialogue structure into pretrained-model ERC, with the useful negative result that adding positional encoding, feed-forward layers, or larger attention stacks to TBM does not help. The TBE/TBM decomposition is a clear and reusable ablation framework. However, the load-bearing empirical evidence is not established: the headline IEMOCAP number is a selected maximum over a test-result grid, no variance or significance is reported despite five trials, and the paper itself states that feature combinations yielded no significant differences. The contribution is therefore currently more suggestive than demonstrated.","major_comments":[{"comment":"The headline IEMOCAP result is a selected maximum over a test-result grid rather than the performance of a pre-specified model. Section 5.1 says the IEMOCAP column in Table 2 uses 'past, Listener, and Normdist,' and Table 3 shows that this is one of twelve combinations of context window, dialogue-feature type, and decay; 68.63 is the maximum of the past-context column. The paper never states that these configuration choices were locked on the development set before evaluating the test set. Given that the margin over EmoBERTa is 0.06 W-Avg F1 and the IEMOCAP grid spread is 0.51 (68.12 to 68.63), the state-of-the-art claim is not supported. The authors should select configurations on development data alone and report the corresponding test result, or use nested cross-validation.","section":"Section 5.1, Tables 2 and 3"},{"comment":"No variance, confidence interval, or significance test is reported for any of the headline comparisons. Section 4.2 says each experiment has five trials and the average is reported, but the per-seed scores are not given. Section 5.2 then states that 'Varying the combinations of features yielded no significant differences in the results.' This is internally inconsistent with presenting 68.63 as a meaningful improvement over TBM's 68.50 and over EmoBERTa's 68.57. The dialogue-feature gain is within the range that the paper itself calls nonsignificant. Report per-seed values or standard deviations and a paired test for the IEMOCAP comparison.","section":"Section 4.2 and Section 5.2"},{"comment":"The choice of network size and turn-priority coefficient is also not tied to a development-set protocol. Table 8 lists # attention layers in {2,4,6}, # heads in {4,8,16}, and gamma in {1.5,2,3,5}; Table 6 shows that TBM's IEMOCAP score varies from 67.63 to 68.50 with layer/head count. No statement says these hyperparameters were selected on the development set. Combined with the Grid selection in Table 3, the reported numbers are best-of-grid results, so the expected performance of TED under a fixed configuration is unknown.","section":"Section 4.2, Tables 6 and 8"}],"minor_comments":[{"comment":"The phrase 'archives state-of-the-art performance' should be 'achieves state-of-the-art performance'.","section":"Abstract"},{"comment":"The past+future TBM MELD value is reported as 65.69 in Table 5, but the corresponding value in Table 3 is 65.60; please reconcile the discrepancy.","section":"Table 5"},{"comment":"The notation eα^t_j is used both for a scalar attention probability and for a vector over turns; clarify the indexing to avoid ambiguity.","section":"Equation (5) and Equation (7)"},{"comment":"The statement that future turns 'do not contribute to a performance improvement' should be qualified: on IEMOCAP the difference between past-only (66.70) and past+future (66.62) TBE is 0.08, which is likely within noise given that no variance is reported.","section":"Section 5.3"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim hinges on a post hoc best-grid selection and a 0.06-point difference with no significance evidence. This is a load-bearing methodological issue that cannot be fixed by local revision; the evaluation would need to be redone under a locked development-set protocol with variance reporting. The idea itself is reasonable, and a resubmission with a properly controlled evaluation could be considered, but the current manuscript does not substantiate its headline claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know the main thing about this paper: the core claim that the dialogue-feature attention gets SOTA on IEMOCAP does not survive contact with the paper's own tables. The 68.63 W-Avg F1 is the best of a twelve-cell grid in Table 3, only 0.13 above the TBM baseline, and some cells are below TBM. The authors themselves write that varying the combinations 'yielded no significant differences.' No variance or significance is reported despite five trials. That is not a SOTA story.\n\nWhat the paper does well: the TBE (averaging token vectors per turn) and TBM (turn-level self-attention) building blocks are a simple, cheap inductive bias for multi-turn ERC, and the ablations show they matter. On IEMOCAP, TBE w/ past jumps to 66.20 from a 55.76 baseline, and TBM adds another 1.8 points. That is a real, plausibly reproducible gain. The writing is clear, the experimental setup is described in enough detail to re-run, and the ablations of speaker tokens vs. speaker attention vs. listener attention are useful negative results.\n\nThe soft spots are, in order of severity. First, the headline result is a post hoc best-test-grid selection. The paper never states that the context window, dialogue-feature type, and gamma were locked on the development set before looking at test labels, and several cells in the same column are lower than TBM. That is selection bias, and it is load-bearing for the SOTA claim. Second, the dialogue-feature contribution itself is tiny and the paper concedes it is not significant. Third, the 'numerous turns' narrative is speculative: the turn-priority gain is inconsistent across datasets.\n\nThis paper is for ERC practitioners who want a simple turn-level baseline or a quick ablation template. It deserves a serious referee, but only with the requirement that the authors either lock hyperparameters on a dev split or report distribution statistics across the grid. Without that, the paper should be rejected as-is.\n\nMy recommendation: send it to peer review, but expect heavy revision. The TBE/TBM result is worth preserving.","headline":"A clearly-written but overclaimed ERC paper: the TBE/TBM components seem useful, but the headline SOTA and the dialogue-feature benefit are not supported by the paper's own numbers.","tokens_in":12631,"tokens_out":1733,"would_cite":false,"duration_ms":19105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that explicitly emphasizing the current turn inside the attention mechanism improves emotion recognition in conversation, reporting a state-of-the-art weighted F1 of 68.63 on the IEMOCAP benchmark.","keywords":["emotion recognition in conversation","turn emphasis","attention mechanism","dialogue features","turn-based encoding","speaker information","IEMOCAP","multi-turn context"],"falsifier":"Re-run TED on IEMOCAP with the configuration reported in Table 2 (past-only context, listener weighting, normal-distribution priority, $\\gamma$ fixed) locked before touching the test labels, and check whether the weighted F1 still tops 68.57; if the margin reverses or vanishes, the state-of-the-art claim is an artifact of configuration selection rather than of the dialogue layer.","tokens_in":11626,"feed_emoji":"🎭","tokens_out":10551,"duration_ms":84224,"temperature":0.7,"pith_summary":"The paper claims that emotion recognition in conversation improves when the model explicitly emphasizes the current turn inside the attention mechanism, rather than letting special tokens alone separate turns. It proposes TED, which averages each turn's token vectors, runs multi-head self-attention over those turn vectors, and rescales the attention scores with a priority factor derived from turn position and speaker identity. TED reports a state-of-the-art weighted F1 of 68.63 on IEMOCAP, edging the 68.57 of the previous best system, with competitive numbers on MELD, EmoryNLP, and DailyDialog. If the claim holds, it shows that a cheap, explicit prior on which turn matters can outperform richer architectural machinery for multi-turn emotion modeling.","feed_headline":"Turn-emphasis attention edges out prior IEMOCAP emotion record","feed_subtitle":"Explicit turn priority in attention scores 68.63 weighted F1 on IEMOCAP, beating the prior 68.57.","key_machinery":"The load-bearing object is the reweighted attention score $\\tilde{\\alpha}^t_j = \\beta_t \\exp(\\alpha^t_j)\\,/\\,\\sum_k \\beta_k \\exp(\\alpha^k_j)$ applied in the last self-attention layer over turn vectors. The factor $\\beta_t$ is the dialogue feature: either a constant $\\gamma$ or a normal-distribution peak $\\beta_t = 1 + \\gamma \\exp\\left(-(t-t_c)^2/2\\sigma^2\\right)$ centered on the current turn, optionally restricted so the boost applies only to same-speaker turns or to listener turns. It turns turn position and speaker identity into an explicit prior on attention weights, which is what \"emphasizing the current turn\" means in this paper. Two earlier components feed it: TBE (Turn-Based Encoding), which concatenates past, current, and future turns with [TURN] and [SEP] tokens and averages each turn's token vectors from a pretrained encoder, and TBM (Turn-Based Multi-Head Self-Attention), which runs self-attention between those turn vectors. The paper's finding is that the attention rescaling, not added transformer layers or speaker tokens in the input, carries the improvement.","core_discovery":"TED's central claim is that multi-turn emotion recognition improves when the model explicitly emphasizes the current turn inside attention rather than relying on special tokens. The method first encodes the conversation by concatenating past, current, and optionally future turns with [TURN] and [SEP] separators through a pretrained RoBERTa-large, then averages each turn's token vectors into one vector per turn. Multi-head self-attention runs over these turn vectors, and a final dialogue layer rescales the softmax attention scores with a factor $\\beta_t$ built from turn position (a normal-distribution peak at the current turn, or a constant) and optionally gated by whether a turn belongs to the same speaker or the listener. The paper reports 68.63 weighted F1 on IEMOCAP, the best number in its comparison table and above EmoBERTa's 68.57, with strong results on MELD, EmoryNLP, and DailyDialog. Its ablations trace most of the improvement to the turn-based encoding and turn-level attention, with the dialogue features contributing a small additional gain.","pith_inferences":["The per-dataset configurations in Table 2 were read off the same test-score grid shown in Table 3; if they were not locked before test evaluation, an independent assessment with pre-registered settings would show whether the 0.06-point IEMOCAP margin over EmoBERTa survives.","The normal-distribution priority is a hand-shaped prior; replacing it with a learned per-turn weight would remove the $\\gamma$ and $\\sigma$ hyperparameters and make the method a drop-in component for any turn-level Transformer.","Because the speaker and listener gates barely change the scores relative to all-turn attention, the decisive component may be TBE plus TBM alone; reporting both numbers side by side on new datasets would show when the dialogue layer earns its extra hyperparameters.","The IEMOCAP advantage is attributed to long dialogues with many turns; stratifying results by dialogue length would test whether turn emphasis helps precisely where turn counts are high."],"forward_implications":["TED reports a weighted F1 of 68.63 on IEMOCAP, above the 68.57 of EmoBERTa and the 68.03 of DAG-ERC.","Turn-based encoding and turn-level attention carry most of the gain: on IEMOCAP the TBE model alone reaches 66.70 against a 55.76 current-turn-only baseline, and TBM lifts this to 68.50.","The dialogue layer improves on plain TBM by a small margin on each dataset, with the best configuration depending on the dataset (past-only context and normal-distribution priority for IEMOCAP, past-plus-future and constant priority for DailyDialog).","Adding transformer extras to TBM does not help: positional encoding and feed-forward layers both leave IEMOCAP and MELD scores at or below the MHSA-only version, and smaller heads and layers beat larger ones.","Putting speaker identity back into the input as special tokens hurts performance (67.44 vs 68.50 for TBM on IEMOCAP), which the paper reads as evidence that global attention plus explicit turn emphasis, rather than speaker tokens, drives the gain."],"supporting_citations":[{"why":"EmoBERTa: the prior best system on IEMOCAP (68.57 weighted F1) that TED's headline number must beat, and the special-token multi-turn input approach that TED extends.","marker":"(Kim and V ossen, 2021)"},{"why":"DAG-ERC: the previous IEMOCAP state of the art (68.03) in the comparison table.","marker":"(Shen et al., 2021)"},{"why":"Supplies the multi-head self-attention formula that TBM reuses and that the dialogue layer rescales.","marker":"(Vaswani et al., 2017)"},{"why":"RoBERTa-large: the pretrained encoder whose token vectors are averaged into turn vectors.","marker":"(Liu et al., 2019)"},{"why":"HiTrans: the closest Transformer baseline with turn-separating CLS tokens that TED compares against and improves on.","marker":"(Li et al., 2020a)"},{"why":"IEMOCAP: the benchmark dataset where the state-of-the-art claim is made.","marker":"(Busso et al., 2008)"},{"why":"COSMIC: supplies the train/development/test split convention the experiments follow, and a comparison baseline.","marker":"(Ghosal et al., 2020)"}],"fun_headline_variants":["Turn-emphasis attention nudges IEMOCAP emotion record to 68.63","Explicit turn priority in attention boosts ERC performance","TED: attention with turn emphasis tops IEMOCAP benchmark","Dialogue-aware attention for emotion recognition improves IEMOCAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline IEMOCAP result assumes the per-dataset choice of context window, dialogue feature type, and priority coefficient was fixed before the test scores were seen, rather than selected from the grid of test results in Table 3.","fun_headline_variants_meta":{"raw":{"variants":["Turn-emphasis attention nudges IEMOCAP emotion record to 68.63","Explicit turn priority in attention boosts ERC performance","TED: attention with turn emphasis tops IEMOCAP benchmark","Dialogue-aware attention for emotion recognition improves IEMOCAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1228,"prompt_tokens":907,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":523,"tokens_out":321,"duration_ms":3544,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:14.633004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run TED on IEMOCAP with the configuration reported in Table 2 (past-only context, listener weighting, normal-distribution priority, $\\gamma$ fixed) locked before touching the test labels, and check whether the weighted F1 still tops 68.57; if the margin reverses or vanishes, the state-of-the-art claim is an artifact of configuration selection rather than of the dialogue layer.","supporting_citations":[],"review_version":1}