{"id":"28a99c41-bb26-4e83-9aa5-d56e4b4f77d9","arxiv_id":"2505.23304","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PaMA uses LLM-generated event patterns to align clusters with classes, improving H-score by up to 12.58% on event-centric GCD benchmarks while staying competitive on standard GCD datasets.","lead":"This paper introduces Event-Centric Generalized Category Discovery, a harder version of a text classification problem where models must find both known and brand-new scam categories in long, imbalanced reports. It also presents PaMA, a framework that uses a large language model to mine and refine event patterns, reporting large gains on two fraud-related benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 12.58% Scam Report gain is obtained with ρ=25 tuned on the test set; with the default ρ=1, H-score drops to 43.59, so the headline margin is inflated by test-set selection.","rationale":"The reader's stated weakest assumption concerns unvalidated LLM pattern quality (Section 3.2); that is a genuine risk, but the paper's own sensitivity analysis makes a more directly observable problem visible. Section 4.3.4 and the Implementation Details jointly show that the main Scam Report result uses ρ=25 selected to maximize performance on the evaluation set itself, while the default ρ=1 (used in the ablations) yields H-score 43.59 rather than 50.88. Thus the central 12.58% claim is not a fair estimate of expected performance under a proper model-selection protocol. This does not refute the method: the ranking-filtering-mining idea is plausible, and the Telecom Fraud Case result with ρ=1 still shows a 10.40% gain. However, the headline quantitative claim needs qualification or re-estimation with validation-based hyperparameter selection. I therefore keep the reader's CONDITIONAL verdict: the paper should be accepted only if the authors release code/data and re-run with ρ chosen a priori, plus error bars. My agreement is partial because the reader mentioned the ρ issue in the rationale but did not make it the weakest assumption, instead focusing on LLM pattern alignment. Both concerns are real; the ρ selection is the most load-bearing because it directly affects the reported magnitude of the central result.","tokens_in":19058,"tokens_out":2959,"duration_ms":30947,"concrete_test":"Re-run the Scam Report experiment with ρ selected on a held-out validation split (or fixed at ρ=1) and report per-seed ACCK, ACCN, and H-score with standard deviations over at least five seeds. If the validation-selected H-score falls below 46, or if the ρ=1 result of 43.59 is confirmed, the headline improvement over GLEAN drops from 12.58 points to at most about 9 points and must be re-reported without test-set tuning. As a secondary check, tune a comparable re-weighting hyperparameter for each baseline on the same validation split to see whether the relative gap closes further.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states: \"In our main experiments, we use the optimal sample weight of ρ = 25 for the scam report, while the ablations use the initial value of ρ = 1.\" Figure 4 then shows H-score peaking at ρ=25 and declining at larger values, meaning this hyperparameter was selected using the same test set on which the headline 12.58% gain is reported. At the default ρ=1, the full method in Table 2 gives H-score 43.59, which is only 8.82 points above the best baseline GLEAN (34.77), not 12.58. Because roughly eight ρ values were scanned (0, 1, 5, 10, 20, 25, 30, 50), the maximum is expected to be optimistic even under noise. The paper also reports no error bars despite claiming three runs, and baselines are not given the same test-set tuning opportunity. The load-bearing condition is that the EC-GCD advantage survives an honest model-selection protocol; the current evidence is consistent with at least part of the margin coming from test-set overfitting rather than from the method's alignment mechanisms.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Event-Centric Generalized Category Discovery (EC-GCD), a textual GCD setting with long, complex, imbalanced, and subjectively labeled event narratives, and contributes a new Chinese Scam Report benchmark. The proposed method, PaMA, clusters unlabeled BERT embeddings, ranks clusters by compactness and size, uses Qwen2.5-72B to generate and refine event patterns, reassigns low-confidence pseudo-labels on the basis of those patterns, and trains with instance-level and prototype-level contrastive losses. Experiments compare PaMA against six GCD baselines on two EC-GCD datasets and three standard GCD benchmarks, reporting large H-score gains on EC-GCD and competitive results on base GCD.","tokens_in":19331,"tokens_out":5495,"duration_ms":58905,"significance":"Assuming the results survive an honest evaluation protocol, the paper makes a useful contribution: it defines a realistic EC-GCD problem, introduces a new benchmark with reported 91.8% annotator agreement, and presents a modular, clearly ablated LLM-guided alignment method. The design choices—cluster ranking, pattern refinement, pseudo-label reassignment, and the hybrid prototype objective—are each individually tested, which is a strength. The main weakness is methodological: the headline gain on Scam Report is obtained with a test-set-tuned hyperparameter, and the absence of variance estimates leaves several smaller margins indistinguishable from noise. The core idea is plausible and worth publishing after a corrected evaluation.","major_comments":[{"comment":"The Scam Report main result in Table 1 is reported with ρ=25, yet Figure 4 shows H-score peaking at ρ=25 on exactly the benchmark reported in Table 1, meaning the headline 12.58-point gain is achieved under a hyperparameter selected on the test set. Under the default ρ=1 used in the ablations, Table 2 reports H-score 43.59, which is 8.82 points above the best baseline GLEAN (34.77), not 12.58. Because the ablations use a different ρ from the main table, Table 2 is not directly comparable to Table 1. Please select hyperparameters on a held-out validation split, or make the default-ρ results primary and treat the ρ sweep as sensitivity analysis.","section":"§4.1 and §4.3.4 (Figure 4)"},{"comment":"§4.1 states that baselines were implemented based on their original designs with the LLM replaced for consistency, while the Table 1 caption says some results are cited from An et al. [6, 7]. These statements are mutually inconsistent. If the LOOP and GLEAN numbers are quoted from earlier papers, they may have been obtained with a different LLM, backbone, or data split, which would make the comparison to PaMA unfair. Please list exactly which cells are reproduced in this paper and which are quoted, and confirm that quoted numbers use the same evaluation protocol.","section":"§4.1 vs. Table 1"},{"comment":"The paper reports results averaged over three runs but gives no standard deviations or confidence intervals. Several claims rest on small margins, such as Telecom Fraud ACCN (+1.10) and CLINC H-score (−0.19), so without variance estimates those differences are not distinguishable from run-to-run noise. Please report per-run values or error bars for at least the main tables.","section":"§4.1 and Table 1"},{"comment":"The central mechanism—LLM-extracted patterns aligning with human annotation criteria—is never quantitatively validated. There is no measure of pattern accuracy, no human agreement study on generated patterns, and no analysis of how pattern errors propagate through pseudo-label reassignment. Such validation is needed to support the paper's attribution of performance gains to the alignment mechanism rather than to other components or to hyperparameter selection.","section":"§3.2 and §4"}],"minor_comments":[{"comment":"The claim that the average document length in the new dataset is '14 times that of previous GCD datasets' is not substantiated anywhere; please add a table with average lengths for all datasets.","section":"Introduction, Challenge ❶"},{"comment":"The terms 'true positives and false positives' are used without defining the reference set; please specify whether these are relative to cluster assignments, current pseudo-labels, or matched known classes.","section":"§3.2.3"},{"comment":"The subsection 'The Number of Low-Confidence Samples' refers to varying 'top-entropy samples,' while the heading and Figure 7(b) use 'low-confidence'; unifying the terminology would avoid confusion.","section":"Appendix D"},{"comment":"The legend entries in Figure 2 are densely packed; enlarging and separating the legend would improve readability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection of ρ is the central issue; if the authors rerun the main experiments with a validation split and report error bars, the paper could become acceptable. I did not find derivation-level circularity, but the empirical claims are fragile as presented. Also, the Scam Report dataset availability is conditional on internal review; please require a clear release or accessibility statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on GCD or LLM-in-the-loop clustering, but the headline number is inflated. The paper introduces EC-GCD, a realistic variant with long imbalanced event narratives, and a new Scam Report benchmark. PaMA's flow — rank clusters, extract patterns with an LLM, refine them against labeled data, reassign low-confidence samples — is sensible, and the ablation table shows each piece earns its keep. On Telecom Fraud Case, where rho is the default 1, PaMA beats GLEAN by about 15.6 H-score points, so the method has real signal.\n\nThe problem is the Scam Report headline. Section 4.1 says the main experiments use rho=25, selected from Figure 4, and the ablations use rho=1. That means the 12.58% gain is the result of test-set hyperparameter tuning. With rho=1, Table 2 reports H-score 43.59, which is 8.82 above GLEAN, not 12.58. The paper gives no error bars despite saying three runs, and the baselines were not given the same tuning freedom. The rho sweep looks like a classic optimism curve, so part of the margin is suspect. The Telecom result saves the paper's main thesis, but the Scam Report comparison needs to be rerun with rho fixed before reporting.\n\nOther soft spots are minor: the '14 times longer' claim is not tied to a specific baseline; pattern quality from Qwen2.5-72B is not validated against the 91.8% human consistency; and Scam Report plus code is promised but not released. The base GCD results are mixed but acceptable — BANKING H-score is solid, StackOverflow drops 6 points, CLINC ties.\n\nBottom line: the idea and the benchmark are worth engaging. I'd take the paper for review, but I'd insist on an honest model-selection protocol, error bars, and a release plan for the data.","headline":"A useful EC-GCD benchmark and a sensible LLM-pattern pipeline, but the Scam Report headline is inflated by test-set selection of rho.","tokens_in":19866,"tokens_out":2124,"would_cite":false,"duration_ms":19726,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PaMA claims that LLM-extracted, label-refined event patterns realign clusters with human classes in long, imbalanced event narratives, yielding up to 12.58% H-score gains over prior GCD methods","keywords":["Generalized Category Discovery","Event-Centric GCD","LLM pattern mining","class imbalance","pseudo-label reassignment","prototype learning","scam detection","cluster-class alignment"],"falsifier":"Replace the refined LLM patterns with surface-keyword or random patterns while keeping the rest of PaMA fixed and re-measure H-score on Scam Report and Telecom Fraud Case: if the reported gains persist, the pattern-alignment mechanism is not the cause. A more direct check is to compare each refined pattern against held-out human annotation rationales and see whether pattern-class agreement predicts per-class accuracy gains.","tokens_in":18874,"feed_emoji":"🔍","tokens_out":6446,"duration_ms":65006,"temperature":0.7,"pith_summary":"The paper argues that Generalized Category Discovery fails on event-centric texts such as police fraud transcripts and scam complaints, because representation-based clustering groups by surface wording while humans classify by latent event structure, and because rare classes are swallowed by dominant ones. It introduces PaMA, which uses a large language model to write a textual pattern for each cluster, then refines those patterns against labeled samples so they encode human annotation criteria; the patterns reassign low-confidence samples and shape prototypes. On two event-centric datasets, including a newly built Scam Report benchmark, PaMA reports up to 12.58% higher H-score than prior GCD methods, with balanced known/novel accuracy and competitive results on standard benchmarks. If correct, this means LLM-written pattern descriptions are a workable bridge between unsupervised clusters and human category definitions in high-stakes, long-document settings.","feed_headline":"LLM patterns lift scam-category discovery by 12.58%","feed_subtitle":"A pattern-mining pipeline aligns clusters with human labels on long, imbalanced event narratives.","key_machinery":"The load-bearing object is the event pattern: a short LLM-generated natural-language description of the archetypal scam or event flow behind a cluster. PaMA's ranking-filtering-mining pipeline computes a ranking score $\\mathrm{Score}(C_k)=\\sigma\\cdot\\mathrm{Comp}_k+(1-\\sigma)\\cdot\\mathrm{Size}_k$, where compactness favors low intra-cluster variance and size favors dominant classes, so patterns for well-formed dominant clusters are generated first and minority clusters are not crowded out. Filtering removes samples already explained by existing patterns; consensus-driven extraction asks the LLM to induce a dominant pattern from remaining samples. Pattern refinement then rewrites each pattern using labeled positives and negatives, and prototypes combine statistical and semantic cues as $P_i=\\beta\\mu_i+(1-\\beta)f_\\theta(p_i)$ with EMA updates; these patterns carry the argument because they are the interface between feature-space clusters and human classification criteria.","core_discovery":"PaMA's central claim is that in event-centric GCD the cluster-class alignment problem is best solved at the level of latent event patterns rather than at the level of feature vectors. PaMA clusters unlabeled embeddings, ranks clusters by compactness and size, asks an LLM to induce a representative pattern for each cluster while filtering out samples that already match existing patterns, and then refines each pattern using labeled true positives and false positives so that patterns respect annotator boundaries. Low-confidence and unstable samples are reassigned to patterns, instance-level InfoNCE and hybrid prototype contrastive losses train the encoder, and prototypes combine class centers with pattern embeddings under exponential moving average. The paper reports H-score 50.88% on Scam Report versus 38.03% for the best baseline, 74.05% on Telecom Fraud Case, and the highest H-score on BANKING with competitive results on the other base benchmarks.","pith_inferences":["The pattern-as-prototype interface suggests a paper-external test: measure pattern agreement with annotator labels and use it as a predictor of per-class GCD accuracy; the paper reports no such validation.","Because the LLM pattern prompts are domain-agnostic, the pipeline should transfer to other event-centric corpora such as medical case notes and legal judgments, but the paper only evaluates on two fraud-domain datasets, so that transfer remains conjecture.","A deployment risk the paper acknowledges indirectly: detailed scam patterns readable by fraudsters could be used to evade detection, making dataset desensitization essential before release.","The average annotation consistency of 91.8% on Scam Report suggests label noise itself may limit the ceiling for alignment; modeling annotator disagreement explicitly could be a natural follow-up."],"forward_implications":["If the gains hold, LLM-written pattern descriptions can serve as an interpretable, label-efficient bridge between unsupervised clusters and human category definitions in long-document domains such as fraud, legal, and clinical reporting.","The reported 12.58% H-score gain over the strongest baseline on Scam Report implies that previous GCD methods leave most of the difficulty in aligning clusters to subjective classes, not in representation quality alone.","The ablation showing that reassignment weight $\\rho=25$ is optimal implies that reassigned pseudo-labels carry useful signal but must be down-weighted relative to confirmed samples to avoid injecting noise.","Because refinement only uses known-class labeled data, the paper's own results imply that aligning genuinely novel categories to human criteria remains unsolved and is the next bottleneck."],"supporting_citations":[{"why":"Defines the GCD problem formulation and the known-total-category assumption that PaMA inherits, and supplies the K-estimation strategy for open-world settings.","marker":"[1]"},{"why":"Provides the multi-task pretraining and contrastive learning procedure that PaMA uses for encoder initialization.","marker":"[4]"},{"why":"Supplies the decoupled prototypical network and the proximity-based cluster-to-class matching that PaMA adapts, along with separate known/novel prototype objectives.","marker":"[5]"},{"why":"Is the LOOP baseline and the source of the evaluation protocol, including ACCK/ACCN/H-score with Hungarian alignment, and several reported baseline numbers.","marker":"[6]"},{"why":"Is the TAN baseline that aligns labeled and unlabeled prototypes, and is the strongest competitor on Scam Report.","marker":"[7]"},{"why":"Provides the predictive-uncertainty sample selection method that PaMA uses in its high-confidence sample selection.","marker":"[8]"},{"why":"Is the GLEAN baseline that generates cluster-level class descriptions with LLM feedback; PaMA extends this idea with refinement and cluster ranking.","marker":"[9]"},{"why":"Provides the Hungarian algorithm used for cluster-class matching and for computing clustering accuracy in the metrics.","marker":"[29]"},{"why":"Supplies the Telecom Fraud Case dataset, one of the two EC-GCD evaluation benchmarks.","marker":"[33]"},{"why":"Is the Qwen2.5-72B model used as PaMA's LLM for pattern generation and refinement.","marker":"[36]"}],"fun_headline_variants":["LLM pattern mining lifts event-category discovery 12.58%","12.58% H-score gain via LLM event-pattern mining","LLM patterns align clusters in imbalanced event narratives","Event GCD: PaMA mines LLM patterns for imbalanced classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the LLM can extract and refine event patterns that match human annotation criteria, and the paper does not quantitatively validate pattern quality; if the LLM's patterns diverge from annotator judgments, the pseudo-label reassignment and prototype updates would propagate those errors.","fun_headline_variants_meta":{"raw":{"variants":["LLM pattern mining lifts event-category discovery 12.58%","12.58% H-score gain via LLM event-pattern mining","LLM patterns align clusters in imbalanced event narratives","Event GCD: PaMA mines LLM patterns for imbalanced classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3524,"prompt_tokens":886,"completion_tokens":2638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2564}},"tokens_in":502,"tokens_out":2638,"duration_ms":19389,"temperature":1.0,"reasoning_tokens":2564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:39.573675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the refined LLM patterns with surface-keyword or random patterns while keeping the rest of PaMA fixed and re-measure H-score on Scam Report and Telecom Fraud Case: if the reported gains persist, the pattern-alignment mechanism is not the cause. A more direct check is to compare each refined pattern against held-out human annotation rationales and see whether pattern-class agreement predicts per-class accuracy gains.","supporting_citations":[{"cited_title":"Generalized category discovery","cited_arxiv_id":null,"evidence_quote":"Defines the GCD problem formulation and the known-total-category assumption that PaMA inherits, and supplies the K-estimation strategy for open-world settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-task pretraining and contrastive learning procedure that PaMA uses for encoder initialization."},{"cited_title":"Gen- eralized category discovery with decoupled prototypical network","cited_arxiv_id":null,"evidence_quote":"Supplies the decoupled prototypical network and the proximity-based cluster-to-class matching that PaMA adapts, along with separate known/novel prototype objectives."},{"cited_title":"Generalized category discovery with large language models in the loop","cited_arxiv_id":null,"evidence_quote":"Is the LOOP baseline and the source of the evaluation protocol, including ACCK/ACCN/H-score with Hungarian alignment, and several reported baseline numbers."},{"cited_title":"Transfer and alignment network for generalized category discovery","cited_arxiv_id":null,"evidence_quote":"Is the TAN baseline that aligns labeled and unlabeled prototypes, and is the strongest competitor on Scam Report."},{"cited_title":"Actively learn from llms with uncertainty propagation for generalized category discovery","cited_arxiv_id":null,"evidence_quote":"Provides the predictive-uncertainty sample selection method that PaMA uses in its high-confidence sample selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Hungarian algorithm used for cluster-class matching and for computing clustering accuracy in the metrics."},{"cited_title":"Overview of CCL23-eval task 6: Telecom network fraud case classification","cited_arxiv_id":null,"evidence_quote":"Supplies the Telecom Fraud Case dataset, one of the two EC-GCD evaluation benchmarks."}],"review_version":1}