{"id":"cf5817f1-ee3c-4032-9e51-71d76c74d6ff","arxiv_id":"2506.19483","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-3.5 and GPT-4 can generate and evaluate turn-level commonsense dialogue expansions guided by ATOMIC relation definitions, with GPT-4 achieving the highest reranking accuracy.","lead":"The paper tests whether GPT-3.5 and GPT-4 can generate dialogue responses that follow specific commonsense relation definitions from the ATOMIC knowledge base, and then whether the same models can identify which relation was used for each generated turn. It reports that GPT-4 is the best evaluator, with top-1 accuracy around 0.24, and that the approach is viable for turn-level commonsense data augmentation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is a self-consistency cycle: the same prompt definitions are used for generation and ranking, and no manual annotation or external ground truth validates the 0.24 top-1 result as commonsense quality.","rationale":"The reader's weakest assumption identifies relation-definition ambiguity as the key risk, citing Section 4's overlap note. I agree that this is part of the problem, but I see it as nested inside a more fundamental issue: the evaluation is circular. The ground-truth label for each generated response is the very relation definition that was used to prompt the generator, and the evaluator is asked to rank those same definitions. Thus the reported accuracy primarily measures whether the evaluator can detect the generation prompt, not whether the response contains valid commonsense. This circularity remains even if the relation definitions were perfectly unambiguous. The paper's own Future Work section explicitly says manual annotations are needed, which is an in-scope admission that no external validation exists. This makes the central claim unsupported by the evidence presented, independently of the overlap issue. The reader's verdict of REJECT remains appropriate: the contribution is preliminary and the prompt templates may be reusable, but the headline claim is not established. I would not change the verdict, though I would strengthen the rationale by pointing to the self-consistency design of the evaluation rather than only to relation ambiguity.","tokens_in":9455,"tokens_out":3419,"duration_ms":36098,"concrete_test":"Select a random sample of 100-200 augmented turns stratified by relation and source dataset. Have at least three annotators perform two tasks: (a) choose the best-matching ATOMIC relation from the same 12 definitions used in the paper, and (b) rate each generated turn for naturalness and contextual appropriateness on a Likert scale. Compute human top-1 accuracy and inter-annotator agreement, and compare them with GPT-4's 0.24 top-1 / 0.415 MRR. If human accuracy is also near 0.24 or annotators disagree strongly, the task is inherently ambiguous and the reported numbers are an artifact of the relation definitions. If human accuracy is substantially higher, then the GPT-4 reranker is genuinely weak and the generated turns still lack validation of commonsense quality. Either outcome directly tests whether the evaluation framework measures commonsense or merely prompt recovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('effectively harnesses LLMs capabilities for commonsense reasoning and evaluation') rests on Table 2 and the confusion matrices in Figure 2. The evaluation task is set up in Section 3.3 as a ranking problem: each generated turn is labeled with the ATOMIC relation whose definition was inserted into the generation prompt, and the evaluator is then asked to rank the same 12 definitions from Figure 1 (right). This measures whether an LLM can recover its own generation condition, i.e., a self-consistency check, not whether the generated utterance expresses correct, contextually appropriate commonsense. The paper itself acknowledges the lack of external validation: Section 5 states 'we plan to increase the number of dialogues and perform manual annotations to better detect which commonsense attributes are better reflected in the expanded turns,' which confirms that no human annotation was performed. Section 4 also reports 'an overlapping issue concerning the definitions of the commonsense relations,' further undermining the reliability of the labels. With top-1 accuracy at 0.24 versus a chance baseline of 0.083, the numbers are uninterpretable as evidence of commonsense quality. The ACCENT comparison is not apples-to-apples either, since top-k accuracy is applied to a continuous alignment score that was not designed for forced-choice ranking. Therefore the reported results do not support the abstract's claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a prompt-based method for turn-level dialogue data augmentation conditioned on 12 ATOMIC commonsense relations, and an automatic evaluation framework that asks GPT-3.5 and GPT-4 to rank these relation definitions according to their fit with a generated turn. Using 200 dialogues from five datasets (1,721 turns, each augmented 12 times), the authors report top-k accuracy and MRR for the reranking task, compare with ACCENT, and present confusion matrices. They conclude that the approach effectively harnesses LLMs for commonsense reasoning and evaluation, with GPT-4 the best evaluator and GPT-3.5 a cost-effective generator.","tokens_in":9848,"tokens_out":4303,"duration_ms":41385,"significance":"If the evaluation were valid, the paper would offer a simple and transparent prompt-based pipeline for commonsense-conditioned dialogue augmentation and automatic quality assessment. The strengths are concrete: full prompt templates are provided in Figure 1, the model outputs are cross-evaluated (GPT-3.5 generations scored by GPT-4 and vice versa), and the data span five dialogue datasets. However, the current evaluation is essentially a self-consistency check: the same relation definitions used in generation are used as the ranking options during evaluation, so the reported numbers measure whether an LLM can recover its own generation condition, not whether the generated turns express correct or contextually appropriate commonsense. The paper acknowledges the lack of external validation in Section 5. Consequently, the central claim is not supported by the evidence presented.","major_comments":[{"comment":"The evaluation is circular with respect to the central claim. The generation prompt includes the same 12 relation definitions from Table 1, and the evaluation prompt asks the model to rank exactly those definitions; the ground-truth label is the relation whose definition was inserted during generation. Thus the evaluation measures whether the evaluator can recover the generation condition, not whether the generated turn reflects real commonsense. The paper itself states in Section 5 that manual annotations are future work, confirming that no external ground truth or human validation was used. Because the abstract's claim is about effective commonsense reasoning and evaluation, this circularity is load-bearing.","section":"Section 3.3, Figure 1"},{"comment":"The comparison with ACCENT is not apples-to-apples. ACCENT is designed to output a continuous event-commonsense alignment score for a response, not a ranking over 12 relational categories. The paper does not explain how ACCENT's output was converted into top-k accuracy and MRR. Without specifying this adaptation, the ACCENT baseline rows (e.g., top-1 0.08) are uninterpretable, and the claimed superiority of the GPT-based rerankers is not established.","section":"Section 4, Table 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported. The datasets are small (40 dialogues per dataset; only 20 for GPT-4 expansions), and the best top-1 accuracy of 0.24, while above the chance level of 1/12 ≈ 0.083, is not shown to be statistically different from other configurations (e.g., 0.245 vs. 0.21). The absence of variance estimates makes the ranking of methods unreliable.","section":"Section 3.2, Table 2"},{"comment":"The paper acknowledges 'an overlapping issue concerning the definitions of the commonsense relations.' This is a problem for the ground-truth labels themselves: if the Table 1 paraphrases are not mutually exclusive, then a 'correct' label is ambiguous, and both the confusion matrices and the top-k numbers are difficult to interpret. The authors should quantify label reliability (e.g., with human agreement on a sample) or restructure the relation set to reduce ambiguity.","section":"Section 4, Figure 2"}],"minor_comments":[{"comment":"The conclusion says the approach is 'powered by open-source large language models,' but GPT-3.5 and GPT-4 are proprietary. Please correct this wording, e.g., 'instruction-following large language models.'","section":"Section 5"},{"comment":"There is a typo: 'Aproaches' should be 'Approaches.'","section":"Section 1"},{"comment":"The sentence 'Various works has already addressed the issue' should use the plural verb 'have.'","section":"Section 2"},{"comment":"The table layout is hard to parse because the 'Top-k' and 'MRR' headers repeat and the row for 'One-Shot GPT-3.5 w/Comet' contains only dashes for the GPT-4 column. Please clarify that these evaluations were not run and separate the column groups more clearly.","section":"Table 2"},{"comment":"The evaluation prompt contains typos: 'reponse' should be 'response,' and 'for each target reponse' appears twice.","section":"Figure 1"},{"comment":"The phrase '1,721 turns were augmented a total of 12 times each turn' is awkward; suggest 'each of the 1,721 turns was augmented 12 times.'","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The paper is a short preliminary report whose evaluation design is self-referential: the same ATOMIC relation definitions are used for generation and evaluation, and no human annotation or external ground truth validates the reported numbers. The cross-evaluation between GPT-3.5 and GPT-4 is an interesting idea, and the prompt templates are transparent, but the central claim is unsupported as written. A substantially revised version with a validated evaluation (e.g., human judgment on a sample, a proper adaptation of ACCENT, and significance testing) could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a modest but real step in turn-level commonsense generation and evaluation. It extends SODA and ACCENT by generating one alternative response per ATOMIC relation and using a listwise ranking prompt (from RankGPT) to identify which relation a generated turn reflects. The prompt templates are clearly presented, the cross-evaluation of GPT-3.5 and GPT-4 is a nice touch, and Table 3 gives concrete examples. That is the useful part.\n\nThe soft spot is the evaluation design. The generation prompt inserts a relation definition; the evaluation prompt asks the model to rank the same definitions against the generated turn. So the 0.24 top-1 accuracy (vs. 0.083 chance) mostly measures whether the model can recover its own generation condition, not whether the generated utterances are commonsense-correct or contextually appropriate. The paper itself acknowledges relation overlap and says manual annotations are future work, so there is no external ground truth on generation quality. The ACCENT comparison is not apples-to-apples because ACCENT is a continuous alignment score, not a forced-choice rank over 12 categories. And with 40 dialogues per dataset (20 for GPT-4), no error bars, and no significance tests, the numbers are indicative at best.\n\nThe central claim in the abstract, that the approach 'effectively harnesses' LLMs for commonsense reasoning and evaluation, is overstated. I would frame it as: can we prompt-invert the generation condition? That is still a legitimate question, and the paper shows a feasible method, but not a proven measure of commonsense quality.\n\nWho is this for: people working on dialogue data augmentation, LLM evaluation, or ATOMIC-style resources. They will find the prompts and the cross-eval idea reusable. It is not ready for a main track, but it deserves a serious referee for a workshop or an arXiv iteration. If the dataset is released, that data alone would be a useful community asset.\n\nMy recommendation: send it to review, but tell the authors that the evaluation needs either human annotation or a reframed claim. With that revision, it could be a solid workshop paper.","headline":"A useful turn-level extension of SODA/ACCENT with clean prompt templates, but the evaluation measures self-consistency rather than commonsense quality, so the headline claim is overstated.","tokens_in":10284,"tokens_out":3196,"would_cite":false,"duration_ms":33234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that instruction-tuned large language models can generate commonsense-aware dialogue turns and rank which commonsense relation each turn was built on, with GPT-4 as evaluator outperforming both ACCENT and GPT-3.5.","keywords":["commonsense knowledge","event commonsense reasoning","natural language generation","prompt-based generation and evaluation","open-domain dialogues","large language models","ATOMIC commonsense relations","listwise reranking"],"falsifier":"Manually annotate the intended ATOMIC relation for each of the 1,721 augmented turns using the same twelve definitions and measure inter-annotator agreement (e.g., Cohen's kappa). If human annotators agree with each other no more often than GPT-4 agrees with the ground-truth labels (24% of turns), the evaluation task is underdetermined and the reported ranking gaps do not measure commonsense generation quality.","tokens_in":9229,"feed_emoji":"💬","tokens_out":8684,"duration_ms":80358,"temperature":0.7,"pith_summary":"The paper tries to show that large language models can both generate commonsense-infused dialogue turns and grade whether a generated turn actually reflects a given commonsense relation. If this works, dialogue systems could augment their training data on demand and automatically check the augmented data without hand annotation. Concretely, the paper expands 1,721 dialogue turns from five open-domain datasets along 12 ATOMIC commonsense relations, then asks GPT-3.5 and GPT-4 to rank which relation each generated turn was meant to express. It reports that GPT-4 ranks the correct relation first in 24% of cases (MRR 0.415), beating ACCENT and GPT-3.5, and that GPT-3.5 and GPT-4 generations are judged similar in quality. The authors frame the results as preliminary.","feed_headline":"GPT-4 picks the right commonsense label in 24% of dialogue turns","feed_subtitle":"Compared with ACCENT and GPT-3.5, the LLM reranker shows promise—and exposes how fuzzy the twelve commonsense categories are.","key_machinery":"The mechanism is a pair of prompt templates. The expansion prompt takes a dialogue context plus twelve human-readable definitions of ATOMIC relations and asks the model to output one next utterance per relation. The evaluation prompt, adapted from a listwise ranking prompt template (reference [31]), presents the same twelve definitions plus the generated turns and asks the model to rank the most fitting definition for each turn; ranking quality is then measured with top-k accuracy and MRR. A secondary mechanism is the cross-confusion matrix, which compares which relations each model assigns to turns generated by itself or by the other model, serving as a proxy for generation quality. The twelve ATOMIC relation definitions are the vocabulary that anchors both prompts and defines the task.","core_discovery":"The central claim is that, with carefully phrased instruction prompts, an off-the-shelf LLM can perform both halves of commonsense data augmentation: generating a plausible next turn conditioned on a specified ATOMIC relation (such as xIntent, oEffect, or IsAfter), and later identifying which relation a generated turn instantiates. The evidence is a ranking experiment in which twelve relation definitions are listed, the model orders them by fit to each expanded turn, and the correct definition's rank is scored with top-k accuracy and MRR. GPT-4 as evaluator reaches top-1 accuracy 0.24 and MRR 0.415 on GPT-4-generated turns, outperforming ACCENT (top-1 0.08) and GPT-3.5. Cross-confusion matrices indicate that GPT-3.5 and GPT-4 generations are roughly interchangeable in judged quality, while the evaluation diagonal is weaker than one would hope because several relation definitions overlap, a point the paper itself raises.","pith_inferences":["If the method generalizes, any taxonomy with mutually exclusive utterance classes—dialogue acts, intents, politeness strategies—could be plugged into the same two-prompt pipeline, making the contribution a general mechanism for instruction-based data augmentation rather than a claim about ATOMIC in particular.","The 0.24 top-1 ceiling suggests the bottleneck may be the relation definitions rather than the models; a human-annotation study on the same 1,721 turns would separate definitional ambiguity from model error and could provide a gold standard for calibrating the metric.","Because GPT-4 expansions were generated for only half the dialogues, cost per dialogue drives sample size; the cross-confusion matrices hint that a cheaper generation model could be substituted, and running the full comparison at scale would test whether that cost-saving trade-off holds."],"forward_implications":["Turn-level augmentation can multiply a small dialogue corpus by 12 dimensions of commonsense, producing responses that are on average 35% longer than the original turns, with no task-specific training.","GPT-4 can serve as an automatic evaluator for commonsense attribute fidelity, outperforming ACCENT on top-1 accuracy and MRR, potentially reducing the need for hand annotation in early pipeline checks.","GPT-3.5 is a viable cheaper generator, since cross-confusion matrices show its outputs are judged similarly to GPT-4's, so cost-conscious setups could substitute it for generation.","Because the same LLM can both generate and rank, the pipeline can be closed-loop: generate with one model, evaluate with the same or another, without training a separate classifier.","As the paper itself notes, overlapping relation definitions cap how much ranking accuracy can tell us about commonsense quality; improving the metric will require cleaner relation definitions."],"supporting_citations":[{"why":"Defines the 12 ATOMIC relation types whose paraphrased definitions drive both generation and evaluation prompts.","marker":"[10]"},{"why":"Serves as the baseline automatic commonsense evaluation metric that the proposed LLM reranking is compared against in Table 2.","marker":"[26]"},{"why":"Supplies the listwise ranking prompt template adapted for the commonsense-relation ranking task.","marker":"[31]"},{"why":"Demonstrates prior use of GPT-3.5 to validate commonsense types in distilled dialogues, motivating the authors' evaluation design.","marker":"[12]"},{"why":"Provides the COMET examples inserted into the one-shot expansion prompt variant.","marker":"[2]"},{"why":"Supports the zero-shot listwise ranking approach adopted for evaluation.","marker":"[28]"},{"why":"Supports the effectiveness of listwise and pairwise ranking prompting for text classification.","marker":"[23]"},{"why":"Provides the GPT-4 model used as generator and evaluator.","marker":"[21]"},{"why":"Provides the ChatGPT/GPT-3.5-class model used as the cheaper generation and evaluation option.","marker":"[20]"}],"fun_headline_variants":["GPT-4 ranks commonsense labels 24% top-1 in dialogue turns","LLM evaluator beats ACCENT on commonsense attribution","GPT-4 evaluator: 24% top-1 on 12 commonsense relations","Fuzzy commonsense relations trip GPT-4 evaluator at 24%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking experiment assumes the twelve ATOMIC relation definitions are unambiguous and mutually exclusive, so that picking the right one measures how well a generated turn instantiates commonsense; the paper's own confusion matrices indicate the definitions overlap, which would cap the achievable accuracy regardless of model skill.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 ranks commonsense labels 24% top-1 in dialogue turns","LLM evaluator beats ACCENT on commonsense attribution","GPT-4 evaluator: 24% top-1 on 12 commonsense relations","Fuzzy commonsense relations trip GPT-4 evaluator at 24%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2726,"prompt_tokens":1040,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":1603}},"tokens_in":656,"tokens_out":1686,"duration_ms":11740,"temperature":1.0,"reasoning_tokens":1603,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:30:25.059039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate the intended ATOMIC relation for each of the 1,721 augmented turns using the same twelve definitions and measure inter-annotator agreement (e.g., Cohen's kappa). If human annotators agree with each other no more often than GPT-4 agrees with the ground-truth labels (24% of turns), the evaluation task is underdetermined and the reported ranking gaps do not measure commonsense generation quality.","supporting_citations":[{"cited_title":"Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi","cited_arxiv_id":null,"evidence_quote":"Defines the 12 ATOMIC relation types whose paraphrased definitions drive both generation and evaluation prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the baseline automatic commonsense evaluation metric that the proposed LLM reranking is compared against in Table 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the listwise ranking prompt template adapted for the commonsense-relation ranking task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChatGPT/GPT-3.5-class model used as the cheaper generation and evaluation option."}],"review_version":1}