{"id":"51b753e4-68cf-440b-8b85-787f26ada670","arxiv_id":"2507.04708","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"This paper introduces an e-commerce review benchmark for jointly detecting eight Plutchik emotions and the exact text spans that explain them, and shows that a structured self-reflection prompt helps some LLMs but not all.","lead":"The authors create a labeled dataset of 2,400 e-commerce reviews that marks emotions and the exact text spans that trigger them, and compare many language models on this new task. Their structured prompting trick helps some models but not others, and the dataset is not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EOT-DETECT's reported superiority may be an artifact of the union-plus-longest gold standard and lenient partial-match/Rouge metrics; its own prompt instructs exactly the verbose, multi-span outputs those metrics reward.","rationale":"The paper has genuine assets: a clearly defined joint task, a carefully sampled multi-domain dataset, expert annotation with reported agreement, and a broad LLM benchmark. Those contributions could be valuable even if the prompt-comparison claim fails. But the central claim is comparative, and for that comparison to be trustworthy the evaluation target must not systematically reward the proposed method's output style. Section 7 constructs the gold standard as the union of all unique trigger spans with the longest span kept on overlap; combined with Partial Match and Rouge metrics, this rewards models that emit many long overlapping spans. EOT-DETECT's I3/I5 instructions explicitly demand comprehensive emotion coverage, balance, and a final self-check, so it is the condition most likely to benefit from that leniency. Whether this bias actually explains the reported gains is testable: recompute with a majority-vote trigger gold and exact-match-only metric. If the gains persist, the framework's advantage is real; if they vanish, the headline overstates the evidence. The paper's own tables add a second, independent problem: for several models, including GPT-4o and Llama-3.1-8B, EOT-DETECT is worse than zero-shot, so 'consistently outperforms' is not supported even under the paper's preferred scoring. I agree with the reader's weakest assumption that the aggregated gold standard and the lack of a demonstrated held-out development split are the least secure conditions; resolving them would settle whether the central claim is robust or an artifact.","tokens_in":19060,"tokens_out":12481,"duration_ms":133088,"concrete_test":"Recompute all model scores on the same 2,400 reviews using a stricter trigger gold standard: include a span only if at least two of the three annotators marked it (majority-vote intersection), and evaluate triggers with exact-match F1 only, dropping Partial Match and Rouge. Rerun ZS, ZS-CoT, and EOT-DETECT for all 23 models and compare average and paired per-model differences. If EOT-DETECT's advantage over ZS/CoT shrinks below a paired-bootstrap significance threshold or reverses, the reported 'surpasses' claim is an artifact of the union-plus-longest aggregation and lenient metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EOT-DETECT surpasses zero-shot and chain-of-thought across e-commerce domains rests on the evaluation protocol in Section 7. The aggregated gold standard keeps all unique trigger spans from three annotators and, on overlap, retains the longest span. Trigger metrics include Partial Match and Rouge-1/Rouge-L, which give partial credit for overlapping or long predicted spans. EOT-DETECT's instructions (Section 4.3: I3 'Maintain Balance', I5 'Final Self-Check' for Emotion/Trigger Coverage) explicitly push the model to emit more emotions and more/longer triggers than ZS or ZS-CoT. The scoring setup therefore systematically favors EOT-DETECT's output distribution, independent of whether the extracted spans are the ones a single careful reader would endorse. The paper offers no majority-vote or intersection trigger gold, no exact-match-only analysis, and no significance tests or per-domain tables. Appendix B.1 says the decoding configuration was 'empirically determined' and 'validated through extensive testing across our evaluation suite' without naming a held-out development split, so tuning on the test set cannot be excluded. The tables also show the 'consistently outperforms' wording is too strong: GPT-4o emotion F1 drops from 0.75 (ZS) to 0.53 (EOT), R1 from 0.73 to 0.57; Llama-3.1-8B drops from 0.51 to 0.17; several Gemma/Qwen rows are flat or worse. But those inconsistencies are additional evidence; the load-bearing issue is that even the average advantage is not yet shown to be real rather than a measurement artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a joint task, Emotion detection and Opinion Trigger extraction (EOT), for e-commerce reviews, and introduces EOT-X, a human-annotated dataset of 2,400 reviews from Amazon, Yelp, and TripAdvisor with Plutchik's eight primary emotions and extractive opinion triggers. The authors evaluate 23–26 LLMs with zero-shot, zero-shot Chain-of-Thought, and their structured prompting framework EOT-DETECT, which contains five instruction steps including a final self-check. The paper's central claim is that EOT-DETECT outperforms zero-shot and CoT across e-commerce domains, and it also contributes an edge-deployable fine-tuned model (EOT-Llama). The dataset and annotation effort are substantial, but the reported results and evaluation protocol raise serious concerns about the validity of the headline comparison.","tokens_in":19307,"tokens_out":5357,"duration_ms":53894,"significance":"If the main comparison claim were sustained, the paper would provide a reusable benchmark for emotion-opinion trigger analysis and a prompt framework that reliably improves LLM extraction. The strengths are real: the dataset is multi-domain, the annotations involve three expert raters with high inter-annotator agreement (average κ=0.89 for emotions, 0.84 for triggers), the evaluation spans many open and closed models, and the authors commit to releasing data and models. However, the paper's own tables do not support the central claim: EOT-DETECT degrades performance on several strong models, no significance tests are provided, and the evaluation protocol may systematically favor the framework's verbose output style. The significance can only be assessed after these issues are resolved.","major_comments":[{"comment":"The central claim that EOT-DETECT 'consistently outperforms' zero-shot and chain-of-thought is contradicted by the paper's own numbers. For GPT-4o, emotion F1 drops from 0.75 (ZS) to 0.53 (EOT); for Llama-3.1-8B-Instruct, it drops from 0.51 to 0.17; Gemma-2-27B drops from 0.43 to 0.31; and Qwen2.5-7B is flat (0.56 vs. 0.55). Several trigger scores also decline. No significance tests, confidence intervals, or per-domain breakdowns are reported. The authors need to either revise the claim to specify the models and domains where EOT-DETECT helps, or report a proper statistical comparison (e.g., paired bootstrap or McNemar's test) over reviews.","section":"Section 7, Tables 2 and 3"},{"comment":"The aggregated trigger gold standard retains all unique trigger spans from the three annotators and, on overlap, keeps the longest span; the trigger metrics include Partial Match and Rouge-1/Rouge-L, which give partial credit for long or overlapping spans. EOT-DETECT's instructions (I3 'Maintain Balance' and I5 'Final Self-Check') explicitly direct the model to include more emotions and more triggers. This scoring setup systematically favors verbose, multi-span outputs, independent of whether the extracted spans are the ones a single careful reader would endorse. The paper should provide a majority-vote or intersection-based trigger gold standard, an exact-match-only trigger F1 analysis, and a comparison of output lengths across methods to rule out this artifact.","section":"Section 7 (Evaluation on Aggregated Gold Standard)"},{"comment":"The inference configuration (temperature=0.2, top_p=0.95, top_k=25, max_tokens=2500) is stated to have been 'empirically determined' and 'validated through extensive testing across our evaluation suite,' but no held-out development split is described. If prompts or decoding hyperparameters were tuned on the test set, the reported superiority of EOT-DETECT would be inflated. The authors should specify a fixed development split, describe any tuning on it, and report results on the test set only after that tuning is frozen.","section":"Appendix B.1"},{"comment":"Several rows in Table 3 contain identical metric values for different models or prompts, which raises data-integrity concerns. For example, Phi-3.5-mini-instruct's three rows (ZS, ZS-CoT, EOT) exactly match o1-mini's three rows in Table 2, and Qwen2.5-0.5B-Instruct_zs matches Mistral-7B-Instruct-v0.3_zs Cot. These exact duplicates across distinct models are implausible and need to be checked and corrected; if they are formatting errors, the corrected table must be provided before the experimental claims can be evaluated.","section":"Table 3 (open-source models)"}],"minor_comments":[{"comment":"The abstract says 23 LLMs were evaluated, while Section 6 says three PLMs, three proprietary models, and 20 open-source models (26 total); Appendix A lists a different set of models. These numbers should be reconciled and the model list made complete.","section":"Abstract and Section 6"},{"comment":"The text reports an average emotion agreement of 0.89, but Table 1's Overall Average row shows 0.88; also, the text references 'Table X' rather than an actual table number. Please correct these inconsistencies and resolve the placeholder references (including 'Appendix X' in Section 1).","section":"Section 5.2, Table 1"},{"comment":"The trigger-level metric 'Partial Match (PM)' is never formally defined. Please state the exact matching criterion (e.g., token-level overlap threshold) and consider reporting trigger-level F1 alongside R1/RL.","section":"Section 7, Tables 2 and 3"},{"comment":"The model name 'claude sonet 3.5' is a typo for 'Claude Sonnet 3.5'; please also use consistent lowercase/uppercase naming across tables.","section":"Table 2"},{"comment":"The claim that EOT-X is 'the first human-annotated benchmark dataset' for emotion-opinion trigger extraction in e-commerce should be made more precise, since EMOTRIGGER (Singh et al., 2024) and other emotion-cause datasets exist in adjacent domains; the novelty should be positioned against those works explicitly.","section":"Section 2 (Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The dataset and the annotation effort are the strongest parts of this submission, but the experimental section needs substantial rework before the central claims can be accepted. Please verify the duplicate rows in Table 3 and require a properly held-out development split and significance testing. If the authors cannot support the claimed superiority of EOT-DETECT after reanalysis, the contribution should be reframed around the dataset and the per-model findings rather than the prompting framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. EOT-X, with 2,400 expert-annotated e-commerce reviews spanning six domains and Plutchik's eight emotions with span-level opinion triggers, is a useful resource. The inter-annotator agreement numbers (emotion κ≈0.89, trigger κ≈0.84) are credible, and the multi-trigger, multi-emotion annotation scheme is a reasonable extension of prior ECPE and EMOTRIGGER work. If the data is released, it will be a legitimate benchmark for affective computing in e-commerce.\n\nThe problem is the paper's central claim. The abstract and Section 7 say EOT-DETECT \"consistently outperforms\" zero-shot and chain-of-thought, but the paper's own Table 2 shows that GPT-4o emotion F1 drops from 0.75 (zero-shot) to 0.53 (EOT-DETECT), and Llama-3.1-8B drops from 0.51 to 0.17 in Table 3. Many other models are flat or worse. That is not consistency; it is model-dependent behavior. The paper also omits significance tests entirely, and the pre-LLM baselines are dismissed without numbers.\n\nThe deeper worry is that the evaluation protocol might manufacture the average gain. The aggregated gold standard preserves all unique triggers from three annotators and, on overlap, keeps the longest span. Combined with partial-match and Rouge metrics, this rewards exactly the verbose, multi-span outputs that EOT-DETECT's prompt instructs models to produce (I3 \"Maintain Balance\" and I5 \"Final Self-Check\" push for comprehensive coverage). The paper reports no exact-match-only analysis and no intersection-based gold standard, so we cannot tell whether EOT-DETECT finds better spans or simply more and longer ones. That is a load-bearing flaw, and it is not addressed.\n\nA separate reproducibility issue: Appendix B.1 says the decoding configuration was \"empirically determined\" and \"validated through extensive testing across our evaluation suite,\" with no mention of a held-out development split. Combined with the unreleased dataset, I cannot rule out tuning on the test set.\n\nWho is this for? Researchers who want an e-commerce emotion/trigger benchmark and who are willing to treat the paper as a dataset contribution rather than a prompt-engineering result. The dataset deserves peer review; the prompting claim needs major revision. I would like to see the data released, a clear dev/test split, significance tests, exact-match metrics, and a softened claim that EOT-DETECT helps some models and hurts others. Without those changes, the headline result should not be trusted.","headline":"The EOT-X dataset is a real contribution; the EOT-DETECT superiority claim is not supported by the paper's own tables and may be an artifact of the evaluation protocol.","tokens_in":19995,"tokens_out":1768,"would_cite":false,"duration_ms":19971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a joint task, EOT, in which a model must both identify which of eight primary emotions a customer review expresses and extract the exact verbatim spans that caused each emotion.","keywords":["emotion detection","opinion trigger extraction","e-commerce reviews","large language models","structured prompting","self-reflection","Plutchik emotion taxonomy","benchmark dataset"],"falsifier":"Re-run the 23-model comparison with strict exact-span matching on a majority-vote gold standard, and choose temperature and top-p on a separate development split; if EOT-DETECT no longer beats zero-shot and chain-of-thought, the claimed advantage is an evaluation or tuning artifact.","tokens_in":18757,"feed_emoji":"😊","tokens_out":6232,"duration_ms":63342,"temperature":0.7,"pith_summary":"The paper introduces a joint task, EOT, in which a model must both say which of eight primary emotions a customer review expresses and extract the verbatim text spans, called opinion triggers, that explain each emotion. To support the task, it presents EOT-X, a set of 2,400 human-annotated reviews from Amazon, TripAdvisor, and Yelp, along with annotator agreement scores it reports as almost perfect. It then evaluates 23 large language models and claims that EOT-DETECT, a structured prompting framework whose instructions end in a self-check step, outperforms plain zero-shot and chain-of-thought prompting across domains. The paper also shows that a fine-tuned 1-billion-parameter model, EOT-Llama, can handle the task on consumer-grade hardware. If these claims hold, the community gains a reusable benchmark and a prompt recipe for explainable emotion analysis of customer feedback.","feed_headline":"A five-step self-check prompt finds what triggers review emotions","feed_subtitle":"A new 2,400-review benchmark and a self-check prompt tell sellers which exact phrases caused joy, anger, or trust","key_machinery":"The mechanism is the structured prompt plus a final verification loop: the instructions direct the model to focus on key emotions, link each emotion to one or more contiguous substrings, preserve minority emotions, enforce extractive output, and then run a self-check confirming emotion coverage, trigger coverage, emotion faithfulness, and trigger verifiability. This five-step sequence is the component the paper credits for the gains, and it works by turning the LLM's generation into a constrained extractive pass rather than an open-ended labeling exercise.","core_discovery":"EOT-DETECT is defined as a prompt tuple $P=\\langle S, T, I, R\\rangle$: a system message that sets an expert persona, a task description that restricts emotions to the eight Plutchik categories plus Neutral and insists triggers be exact substrings, five instructions that move from emotion identification through trigger linking, balance, clarity, and a final self-check, and the input review. The self-check explicitly asks the model to verify emotion coverage, trigger coverage, emotion faithfulness, and trigger verifiability before answering. On EOT-X, with a gold standard aggregated from three expert raters by majority vote for emotions and union-with-longest-overlap for triggers, the paper reports that EOT-DETECT beats zero-shot and chain-of-thought baselines for several strong models, including Mistral-7B-Instruct-v0.2 at emotion F1 0.86 and Claude Sonnet 3.5 at 0.81, and that the fine-tuned EOT-Llama outperforms models up to seven times larger. The central discovery is that a structured, self-verifying prompt makes modern LLMs more faithful at explaining why a review expresses an emotion.","pith_inferences":["Editorial inference: the reported trigger metrics (partial match and ROUGE) plus the gold-standard rule that keeps every unique trigger span and the longest overlap may reward models that emit long or numerous spans, so an exact-span-only evaluation could reorder the leaderboard.","Editorial inference: the decoding configuration was empirically tuned and validated without a clearly specified development split, so part of the EOT-DETECT advantage could come from matching the test distribution; a validation-split replication is a natural check.","Editorial inference: the same joint formulation could transfer to app-store reviews, support tickets, or survey comments, where the verbatim span that caused an emotion is the actionable unit.","Editorial inference: the self-check instruction is a soft form of constrained decoding; a testable extension is whether forcing spans through constrained generation gives the same gain without prompt-based self-verification."],"forward_implications":["The EOT-X dataset gives future work a fixed, human-verified target for comparing emotion-and-trigger systems across Amazon, TripAdvisor, and Yelp.","A 1-billion-parameter fine-tuned model can serve as an on-device analyzer, reducing the need for API calls in e-commerce feedback processing.","Adding the self-check step is a model-agnostic intervention that lifts several open- and closed-source models above their zero-shot and chain-of-thought baselines.","Because triggers are verbatim spans, model outputs can be audited against the original review, making the analysis interpretable for product and customer-experience teams.","Closed-source models lead on absolute scores, but the best open-source models with EOT-DETECT are competitive, narrowing the practical gap."],"supporting_citations":[{"why":"Supplies the eight-primary-emotion taxonomy that fixes the label set for EOT-X and all prompts.","marker":"(Plutchik, 1980)"},{"why":"Defines the zero-shot chain-of-thought baseline that EOT-DETECT is compared against.","marker":"(Kojima et al., 2022)"},{"why":"Provides the zero-shot prompting baseline and the general LLM paradigm the framework builds on.","marker":"(Brown et al., 2020)"},{"why":"Motivates using expert raters instead of crowdsourcing for emotion annotation quality.","marker":"(Mohammad and Turney, 2013)"},{"why":"Supplies the inter-annotator agreement statistic used to validate EOT-X labels.","marker":"(Fleiss, 1971)"},{"why":"Documents existing LLM limitations on emotion-trigger identification in social media, motivating the e-commerce joint task.","marker":"(Singh et al., 2024)"},{"why":"Provides the Amazon Reviews source from which the e-commerce portion of EOT-X is sampled.","marker":"(Hou et al., 2024)"},{"why":"Provides the TripAdvisor source used for the travel-domain portion of EOT-X.","marker":"(Li et al., 2014)"},{"why":"Provides the Yelp open dataset used for the restaurant-domain portion of EOT-X.","marker":"(Yelp, 2025)"}],"fun_headline_variants":["Self-check prompt links review emotions to opinion triggers","New benchmark and prompt trace emotions to exact triggers","Structured self-check prompt reveals why reviews feel emotional","LLMs with self-check pinpoint emotion triggers in reviews","Prompt with self-reflection explains what drives review emotions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the aggregated gold standard being a fair target and on the prompts and decoding settings not being tuned on the test reviews.","fun_headline_variants_meta":{"raw":{"variants":["Self-check prompt links review emotions to opinion triggers","New benchmark and prompt trace emotions to exact triggers","Structured self-check prompt reveals why reviews feel emotional","LLMs with self-check pinpoint emotion triggers in reviews","Prompt with self-reflection explains what drives review emotions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1584,"prompt_tokens":951,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":560}},"tokens_in":567,"tokens_out":633,"duration_ms":6983,"temperature":1.0,"reasoning_tokens":560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:42:09.123931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 23-model comparison with strict exact-span matching on a majority-vote gold standard, and choose temperature and top-p on a separate development split; if EOT-DETECT no longer beats zero-shot and chain-of-thought, the claimed advantage is an evaluation or tuning artifact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the eight-primary-emotion taxonomy that fixes the label set for EOT-X and all prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents existing LLM limitations on emotion-trigger identification in social media, motivating the e-commerce joint task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Yelp open dataset used for the restaurant-domain portion of EOT-X."}],"review_version":1}