{"id":"f66da0dd-8c3e-453e-b114-5abd1a8a86b8","arxiv_id":"2508.08272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A real-time story identification system combining BGE-M3 embeddings, NER, and online clustering attains AMI about 0.57 on Slovene news, well below offline clustering's 0.84.","lead":"This paper builds a real-time system that groups news articles into stories by combining text embeddings, named entity recognition, and online clustering. A news monitoring company could use it to track events as they happen, but the evaluation shows it still falls well short of offline clustering quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's human-evaluation claim is unsupported: Section 5 reports only automatic AMI/Silhouette scores, and the reference labels come from the system being replaced, so the real-time method's 'sensible results' lack independent evidence.","rationale":"The paper is an honest engineering comparison and explicitly acknowledges that online clustering underperforms offline clustering. The load-bearing issue is not the magnitude of the numbers but the absence of evidence for the abstract's human-evaluation claim. The only human evaluation mentioned is the reworking of reference labels in Section 3; no results of human judgment about the proposed system's clusters appear in Sections 5 or 6. This is a correctness/evidence risk, not a style issue. The reader's weakest assumption about label validity overlaps with this concern, hence 'partial': a fresh independent annotation would simultaneously test label validity and supply the missing human evaluation. I do not recommend REJECT because the automatic comparisons are useful and the claim could be repaired by adding the missing evaluation or by narrowing the abstract. Keeping the CONDITIONAL verdict is therefore appropriate.","tokens_in":11212,"tokens_out":3295,"duration_ms":46137,"concrete_test":"Ask the authors to supply the human-evaluation protocol and results promised in the abstract, or run one: take a random sample of about 300 articles from the 10-day window, have two annotators not involved in label construction assign story labels from scratch, compute the AMI of the best real-time configuration against these fresh labels, and report inter-annotator agreement (e.g., Cohen's kappa). If the fresh-label AMI is substantially below 0.569, or if inter-annotator agreement is low, the labels carry the result rather than the method; if the fresh-label AMI is comparable and agreement is high, the human-evaluation claim receives the independent support it currently lacks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the real-time approach 'produces sensible results as judged by human evaluators.' The body contains no such evaluation. Section 5 reports AMI, Silhouette, and outlier classification accuracy against the Section 3 labeled dataset; human input appears only in the construction of those labels, when the company's graph-based clustering pipeline was 'manually re-evaluated, restructured, and re-ordered' (Section 3). Thus the best real-time result (DBSTREAM with named entities and topic-term merging, AMI 0.569 in Table 5) measures agreement with labels derived from the same offline clustering paradigm the paper aims to replace, and no human judgment of the proposed system's output is reported. If the labels were not independently validated, the central claim reduces to 'our online method partially reproduces a proprietary offline clustering,' which is much weaker than what the abstract asserts. The large gap between the online 0.569 and offline BGE-M3 0.838 further means the real-time system would need strong separate evidence to justify the phrase 'sensible results.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a real-time news story identification pipeline that combines text embeddings (BGE-M3, TF-IDF, DistilBERT, etc.), named entity recognition, optional summarization, online clustering algorithms (DBSTREAM, TextClust) and BERTopic-based topic modeling, plus post-clustering merging and outlier detection. The method is evaluated on a Slovene news dataset of 6,400 articles labeled with 4,028 stories over a ten-day window in March 2023. The authors report automatic metrics (AMI, Silhouette, outlier classification accuracy) for offline and online configurations, with the best online configuration (DBSTREAM with named entities and topic-term merging) reaching AMI 0.569, compared to offline BGE-M3 k-means AMI 0.838. The abstract claims that the approach 'produces sensible results as judged by human evaluators,' but no direct human evaluation of the proposed system's output is reported in the body.","tokens_in":11421,"tokens_out":3509,"duration_ms":44457,"significance":"If the central claims were fully supported, the paper would offer a practically useful contribution to media monitoring: a real-time story identification system with a systematic comparison of text representations, online clustering methods, and post-clustering heuristics. The ablation structure is a strength, and the focus on named entities and time constraints is well motivated by the story identification task. However, the current evidence is not sufficient to support the headline claim of human-judged quality, because the evaluation labels are derived from the same offline clustering paradigm the authors aim to replace, and the reported metrics are purely automatic.","major_comments":[{"comment":"The abstract states that the real-time approach 'produces sensible results as judged by human evaluators,' but Section 5 reports only automatic AMI, Silhouette, and outlier classification accuracy against the Section 3 labeled dataset. There is no reported evaluation in which human evaluators judge the stories produced by the proposed system. The human involvement described in Section 3 is limited to the construction of the labeled dataset, not to assessing the output of the proposed pipeline. This is load-bearing: the headline claim as written is unsupported by the evidence in the paper. Either add a genuine human evaluation of the proposed system's output or revise the abstract and conclusion to state that the system is evaluated only against a reference labeling.","section":"Abstract and Section 5"},{"comment":"The ground-truth labels used throughout Section 5 are generated by 'replicat[ing] the current system's dynamic clustering method' (embeddings, pairwise cosine similarity threshold tau, Louvain community detection with gamma = 0.1), followed by manual re-evaluation, restructuring, and re-ordering by the media monitoring company. The proposed method is therefore evaluated against labels derived from the same graph-based, offline clustering paradigm that the paper aims to replace. High AMI values may partly reflect agreement with the old system's clustering behavior rather than with independently established story identity. This circularity weakens every quantitative comparison in Tables 1-5. The authors should provide independent human story annotations (e.g., with inter-annotator agreement) or otherwise validate that the reference labels represent true story identity rather than the output of one particular clustering paradigm.","section":"Section 3 and Section 5"},{"comment":"Several load-bearing hyperparameters are introduced without sensitivity analysis or a clear validation protocol: the outlier distance threshold (Section 5.1), the fading factor lambda tied to the assumed 10-day story window (Section 4.2.2), the post-clustering merge threshold d (Section 5.3), and the ground-truth generation thresholds tau and gamma (Section 3). The paper states that the outlier threshold is set on a random 10% development sample, but it does not report how the other thresholds were selected or whether they were tuned on the same test set. Since the claimed practical advantage is real-time deployability, the robustness of results to these choices should be demonstrated, for example by reporting parameter sweeps or a train/validation/test split for all tuned parameters.","section":"Section 4.2.2 and Section 5.3"},{"comment":"The online clustering evaluation in Table 4 and the post-clustering results in Table 5 report single AMI values per configuration with no indication of variance, number of runs, or statistical significance. Online clustering algorithms and the batch simulation (initial 10% set, then batches of 10 articles) may be sensitive to initialization and ordering; a single run cannot establish that one configuration reliably outperforms another. The authors should report multiple runs or at least a sensitivity analysis over random seeds and batch compositions.","section":"Section 5.2"}],"minor_comments":[{"comment":"The wording 'we first manually detect outliers using a distance-based threshold' and 'We manually cluster similar micro-clusters' is confusing: the surrounding text indicates that these are automated, threshold-based procedures, not manual inspection. Please replace 'manually' with 'algorithmically' or specify where human intervention actually occurs.","section":"Section 4.2.3"},{"comment":"The outlier evaluation uses a balanced 50/50 split of outliers and non-outliers, which changes the base rate relative to the natural dataset (where most stories are singletons). The reported outlier classification accuracy is therefore not directly interpretable as performance on the real distribution; reporting precision/recall or an unbalanced evaluation would be more informative.","section":"Section 5.1"},{"comment":"The text contains typographical and formatting errors, e.g., 'Theever-increasingamountofglobalnewspresentsanoverwhelmingchal-lenge' in the Introduction and the missing space in 'BERTopic Grootendorst [12]' in Section 4.2.1. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"Table 1 reports BGE-M3 as 0.8378 while the text says 0.838; please make the rounding consistent. Additionally, the claim in Section 5.1 that 'using only named entities outperforms using only the text' refers specifically to outlier accuracy, not to overall clustering; clarifying this would avoid misreading.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful applied contribution, but the abstract's human-evaluation claim is not supported by the reported experiments, and the ground-truth construction creates a circularity that affects all comparisons. These issues can be addressed within the manuscript's scope by adding an independent human evaluation and/or clearly repositioning the claims, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of Skvorc et al. The paper does something concrete: it builds a real-time story identification pipeline for Slovene news, combining BGE-M3 embeddings, SloNER named entities, DBSTREAM/TextClust online clustering, BERTopic, and a post-hoc merging step. That specific combination is new, and the ablations are genuinely useful. The finding that named entities help outlier detection but hurt general clustering, and that topic-term merging on named entities gives the best online AMI (0.569), is a practical insight. The paper is also honest: it reports that online clustering lags offline k-means (0.838) and doesn't hide it.\n\nThe main problem is the abstract's claim of 'sensible results as judged by human evaluators.' The body contains no human evaluation. All reported numbers are automatic AMI/Silhouette/accuracy against the Section 3 labels. Those labels themselves come from the company's graph-based clustering pipeline—embedding, pairwise threshold, Louvain—then 'manually re-evaluated, restructured, and re-ordered.' So the ground truth is partly generated by the same paradigm the paper aims to replace. That's circular in a way that weakens the central claim. The stress-test note is right on this.\n\nThat said, the circularity is not total. The manual restructuring step does inject human judgment, and the paper describes it clearly. But without a description of how much restructuring, or any independent human annotation of the proposed system's output, the claim as stated is unsupported. The gap between 0.569 and 0.838 might still be acceptable if the real-time constraint is essential, but 'sensible' needs a human baseline.\n\nOther soft spots: no code or data released, and several key hyperparameters (outlier distance threshold, fading factor, merge threshold d) are described but not reported numerically. These are fixable in revision.\n\nOverall: this is a legitimate engineering contribution, not a field-changer. It deserves peer review because the core idea is practical and the evaluation, once the human-evaluation claim is either added or removed, can be made sound. I'd send it to a workshop or a systems-oriented NLP venue. I would not cite it in my own work without code/data.\n\nRecommendation: send to peer review with a request to fix the abstract, describe the label generation's human component, and release thresholds/code.","headline":"Solid engineering combination for real-time news story identification, but the abstract's human-evaluation claim is unsupported and the ground-truth labels come from the paradigm being replaced.","tokens_in":12001,"tokens_out":2316,"would_cite":false,"duration_ms":27135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A real-time pipeline assigns news articles to ongoing stories as they appear online, using online stream clustering with named-entity-based post-merging; on a 6,400-article Slovene dataset it reaches AMI 0.569 against 0.838 for offline…","keywords":["story identification","real-time news clustering","online topic modeling","named entity recognition","news monitoring","BGE-M3","DBSTREAM","TextClust"],"falsifier":"Take a sample of the same 6,400 Slovene articles and have independent annotators, blind to the system output and to the company's clustering, assign them to stories; if the real-time pipeline's agreement with those independent labels is far below the 0.569 AMI reported against the company-restructured labels, the paper's evidence for sensible real-time story identification would collapse.","tokens_in":11021,"feed_emoji":"📰","tokens_out":7129,"duration_ms":71064,"temperature":0.7,"pith_summary":"This paper claims that news articles can be assigned to evolving stories as they appear online, without rerunning a full clustering algorithm over the entire archive each time. The proposed system embeds articles, detects named entities, feeds the vectors through an online stream clusterer such as DBSTREAM or TextClust, optionally guided by BERTopic, and then merges micro-clusters whose topic terms or named entities overlap within a 10-day window. On a labeled set of 6,400 Slovene articles covering 4,028 stories, the best real-time configuration reaches an adjusted mutual information (AMI) score of 0.569, compared with 0.838 for offline $k$-means with the same BGE-M3 embeddings. The paper's central claim is that this real-time result is sensible for production news monitoring, even though it does not match offline quality.","feed_headline":"Live news grouping scores 0.569 AMI on Slovene stories","feed_subtitle":"Real-time pipeline with named entities and topic merging beats raw stream clustering and nears offline quality.","key_machinery":"The load-bearing mechanism is the micro-cluster-plus-post-merge design. Online stream clusterers create micro-clusters with a fading factor $\\lambda$ that gradually reduces the weight of older articles and effectively expires stories after 10 days; a separate batch step then merges micro-clusters when their topic-modeling keywords or named entities are close enough. Named entity recognition supplies the entities, BGE-M3 supplies dense text embeddings, and BERTopic supplies the topic terms. The 10-day window is the time-based constraint that defines a story's lifespan and sets the fading schedule.","core_discovery":"The central discovery is that online topic-modeling and stream-clustering methods can be adapted to story identification, and that the main gains come from a post-clustering merge step driven by named entities. Plain DBSTREAM with BERTopic reaches an AMI of 0.398; adding a merging step that compares topic terms computed on named entities lifts it to 0.569. Named entities alone are better than full text for detecting single-article stories (outlier classification accuracy 0.762 versus 0.720), but they slightly hurt general clustering (AMI 0.678 versus 0.838), so the paper's recommended design uses full-text embeddings for clustering and named entities for outlier detection and merging. The paper presents the pipeline as a viable real-time alternative to the offline graph-based clustering that the evaluation labels were derived from.","pith_inferences":["The 10-day fading window caps story length by construction; long-running investigations that span more than ten days would be split into separate stories, so a multi-scale time window is a natural testable extension.","Because the evaluation labels were produced by manually restructuring the output of the company's existing embedding-and-Louvain pipeline, the reported AMI may partly measure agreement with that pipeline's notion of a story rather than an independent ground truth.","The batching of post-clustering merges after each day's articles means the system is real-time only up to a one-day lag; applications that need instant alerts would require a finer merge schedule.","The threshold for outlier detection was tuned on a random 10% development slice of the same dataset, so its transfer to other time periods, languages, or news domains is untested."],"forward_implications":["Production monitoring systems can assign articles to stories continuously as they arrive, without waiting for a full re-clustering of the archive.","Named entities are a cheap, high-value signal for outlier detection and for merging clusters that describe the same people, places, or events.","The gap between the best online result (AMI 0.569) and the offline result (AMI 0.838) quantifies the price of real-time processing on this dataset.","Summarizing articles before embedding does not help and hurts clustering performance, so the pipeline can skip summarization and save runtime.","The approach is built from language-dependent components such as embeddings, named entity recognition, and topic modeling, so transferring it to another language requires replacing those components; the paper evaluates only Slovene data."],"supporting_citations":[{"why":"Supplies the online shared-density micro-cluster algorithm that the real-time pipeline wraps and fine-tunes.","marker":"[13]"},{"why":"Provides the one-pass textual stream clustering baseline that the paper adapts and compares against DBSTREAM.","marker":"[3]"},{"why":"Supplies BERTopic, used to generate topic-modeling terms that drive the post-clustering merging step.","marker":"[12]"},{"why":"Supplies the Slovene named entity recognition model used for outlier detection and for computing named-entity topic terms.","marker":"[19]"},{"why":"Provides the BGE-M3 embedding model that achieved the best offline clustering AMI and is used in the online runs.","marker":"[7]"},{"why":"Provides the Louvain community detection used in the replicated graph-based pipeline that produced the evaluation labels.","marker":"[6]"},{"why":"Supplies the Slovene news summarizer whose integration was tested and found to reduce clustering performance.","marker":"[28]"}],"fun_headline_variants":["Named entities boost real-time story merging to 0.569 AMI","Real-time story ID hits 0.569 AMI with NE merge","Story clustering online: AMI 0.569 via entity merge","Adapting stream clustering for news stories: 0.569 AMI","Entity-aware merge lifts live story grouping to 0.569"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation labels correctly represent what a news story is; they were created by hand from the output of the company's existing embedding-and-Louvain clustering pipeline, and every automatic score in the paper compares against those labels.","fun_headline_variants_meta":{"raw":{"variants":["Named entities boost real-time story merging to 0.569 AMI","Real-time story ID hits 0.569 AMI with NE merge","Story clustering online: AMI 0.569 via entity merge","Adapting stream clustering for news stories: 0.569 AMI","Entity-aware merge lifts live story grouping to 0.569"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000894,"raw_usage":{"total_tokens":3844,"prompt_tokens":924,"completion_tokens":2920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2827}},"tokens_in":540,"tokens_out":2920,"duration_ms":22750,"temperature":1.0,"reasoning_tokens":2827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:14:51.976322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of the same 6,400 Slovene articles and have independent annotators, blind to the system output and to the company's clustering, assign them to stories; if the real-time pipeline's agreement with those independent labels is far below the 0.569 AMI reported against the company-restructured labels, the paper's evidence for sensible real-time story identification would collapse.","supporting_citations":[{"cited_title":"Clustering data streams based on shared density between micro-clusters.IEEE transactions on knowl- edge and data engineering, 28(6):1449–1461, 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the online shared-density micro-cluster algorithm that the real-time pipeline wraps and fine-tunes."},{"cited_title":"Textual one-pass stream clustering with automated distance threshold adaption","cited_arxiv_id":null,"evidence_quote":"Provides the one-pass textual stream clustering baseline that the paper adapts and compares against DBSTREAM."},{"cited_title":"Pytorch model for Slovenian named entity recognition SloNER 1.0","cited_arxiv_id":null,"evidence_quote":"Supplies the Slovene named entity recognition model used for outlier detection and for computing named-entity topic terms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Louvain community detection used in the replicated graph-based pipeline that produced the evaluation labels."},{"cited_title":"One model to rule them all: Ranking Slovene summarizers","cited_arxiv_id":null,"evidence_quote":"Supplies the Slovene news summarizer whose integration was tested and found to reduce clustering performance."}],"review_version":1}