{"id":"d06e84c1-7d79-46fc-82ce-f40e4f529cec","arxiv_id":"2507.23669","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Sentence-embedding retrieval ranks the correct AI Incident in the top three for about 98% of test reports when titles and descriptions are combined, but possible train/test leakage likely inflates that number.","lead":"This paper tests whether standard text-retrieval tools can automatically match new AI incident reports to existing incidents in the AI Incident Database. It finds that sentence-embedding models rank the correct incident at or near the top for most reports, but the evaluation may overstate accuracy because reports describing the same incident were split across training and test sets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline Accuracy@3 of 0.982 rests on a random report-level split in Section 4.1 that mixes reports of the same incident across train and test; it may measure near-duplicate retrieval rather than generalization to unseen reports.","rationale":"I read the paper as a practical information-retrieval benchmark for the AIID curation workflow. The most load-bearing gap is evaluation realism. Section 4.1 describes a random split of reports without grouping by incident, and Section 3.1 confirms that multiple reports can belong to one incident. This creates a near-duplicate training/test overlap that can inflate all of the central numbers, especially the RQ1 comparison between title-only and title-plus-description, because long descriptions are likely to contain overlapping language from other reports of the same event. The RQ3 analysis has an additional, independent problem: Table 5 shows Fold 3 performance dropping below Fold 2, so the statement that performance 'consistently improves' is contradicted by the paper's own numbers, and the fixed test set is not temporally held out. The paper is clearly motivated and the limitations section is honest about ignoring report-group structure in modeling, but it does not flag this evaluation confound. The reader's CONDITIONAL verdict is appropriate, and my stress-test does not change that verdict; it sharpens the reason: the evaluation split, not the model family, is the critical assumption that needs to be fixed before the headline accuracy can be trusted.","tokens_in":12839,"tokens_out":12027,"duration_ms":135117,"concrete_test":"Re-run the full fine-tuning and evaluation for multi-qa-MiniLM-L6-cos-v1 using an incident-disjoint split: assign all reports belonging to the same AI Incident to the same 75/12.5/12.5 fold, then report both per-report and per-incident Accuracy@K, MRR@K, and NDCG@K for K = 3, 5, 10, for both title-only and title-plus-description inputs. If per-incident Accuracy@3 remains close to 0.982, the concern is resolved; if it drops materially (e.g., by more than 5 points), the headline numbers are inflated by same-incident near-duplicate leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a report-to-incident matcher can operate with high precision. That claim is supported only by a random 75/12.5/12.5 split of reports in Section 4.1. Section 3.1 explicitly states that a single AI Incident can have multiple reports, and Table 1 illustrates this. With 3,805 reports over 815 incidents (about 4.7 reports per incident), most test reports will share their target incident with one or more training reports. The model is fine-tuned on report-incident pairs and then evaluated on a different report of the same incident, so the test query is a near-duplicate of content seen during training. The headline Accuracy@3 of 0.982, and the RQ1 improvement from 0.772 (title only) to 0.982 (title plus description), may therefore largely reflect matching the same event's text rather than generalizing to genuinely new incidents. The same problem affects RQ3: the fixed validation/test sets in Figure 2 are not temporal holdouts, and Table 5 already violates the claimed monotonic trend at Fold 3 (Accuracy@3 = 0.883 versus 0.940 in Fold 2). Without an incident-disjoint or chronological split, the quantitative claims in the abstract are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-based framework to automate the association of new reports in the AI Incident Database (AIID) with existing AI Incidents. The task is formalized as ranking candidate incidents by semantic similarity between report text (title and description) and incident text. The authors benchmark BM25, BM25+T5, cross-encoders, and sentence transformer models on a snapshot of 3,805 reports and 815 incidents, and report that sentence transformers, especially multi-qa-MiniLM-L6-cos-v1, outperform all baselines, with Accuracy@3 reaching 0.982 when titles and descriptions are combined. They also analyze robustness to description length and the effect of training-set size, concluding that retrieval performance improves consistently as training data grows.","tokens_in":13125,"tokens_out":3428,"duration_ms":35221,"significance":"If the reported results are valid, the paper offers a practical tool for AIID curators and a useful benchmark for report-to-incident retrieval. The task is well motivated, the dataset is public and clearly described, and the evaluation uses standard retrieval metrics with error bars over multiple runs. The manuscript also commits to releasing code upon acceptance. However, the central quantitative claims rest entirely on the soundness of the evaluation split, and the current split does not support the stated generalization claims; the practical significance is therefore conditional on a corrected experimental protocol.","major_comments":[{"comment":"The random 75/12.5/12.5 split is performed at the report level, but Section 3.1 explicitly states that a single AI Incident may be associated with multiple reports and the dataset contains 3,805 reports for 815 incidents. Because the retrieval target is the incident, test reports from an incident whose other reports appear in the training set are near-duplicates of training content. Consequently, the reported Accuracy@3 of 0.982 likely measures near-duplicate retrieval rather than generalization to previously unseen incidents. The evaluation should be rerun with an incident-disjoint split, or with a temporal split that ensures no incident spans both training and test sets.","section":"Section 4.1, with Section 3.1 and Table 1"},{"comment":"The claim of consistent improvement with training data is contradicted by Table 5: Fold 3 is worse than Fold 2 for every metric (e.g., Accuracy@3 drops from 0.940 to 0.883, MRR@3 from 0.878 to 0.808, and NDCG@3 from 0.894 to 0.827), and Accuracy@10 for Fold 5 equals Fold 1. In addition, Figure 2 shows that only the training folds are chronologically ordered; the validation and test sets are fixed but inherit the same random report-level split, so the same-incident leakage remains and RQ3 does not isolate generalization to genuinely new incidents.","section":"Section 4.5, Table 5"},{"comment":"The headline conclusions—that sentence transformers consistently outperform all other approaches and that combining title and description yields large gains—are based entirely on the leaked split. Since every test report may share its target incident with training reports, the quantitative claims in the abstract and in RQ1 cannot be interpreted as estimates of deployment performance for linking a new report to a previously unobserved incident. The authors should re-evaluate all three research questions under an incident-disjoint protocol and report whether the relative ordering of models and the reported gains persist.","section":"Abstract and Sections 4.2-4.4"}],"minor_comments":[{"comment":"The phrase 'mitigate these risk' should read 'mitigate these risks'; a similar grammar issue appears in the Introduction ('systematically study and mitigate these risk').","section":"Abstract and Introduction"},{"comment":"References [28] and [29] are duplicate entries for the same paper (Nogueira et al., 'Document ranking with a pretrained sequence-to-sequence model'); one should be removed and the citations consolidated.","section":"References"},{"comment":"The sentence 'the test set contains 475 variants' should say '475 reports' (12.5% of 3,805), since 'variant' is a distinct AIID concept introduced in Section 2.3.","section":"Section 4.1"},{"comment":"The text should clarify whether the validation and test sets in the progressive protocol are fixed from the original random split or are temporal holdouts; as written, the terms 'temporal' and 'chronologically ordered' apply only to the training folds.","section":"Figure 2 and Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The central issue is fixable in principle: rerunning the experiments with an incident-disjoint split would directly address the leakage concern. However, if the corrected results drop substantially, the contribution may reduce to a simple baseline comparison on a small dataset. It would also be helpful for the authors to report how many test reports have an incident that also appears in the training set, as this number quantifies the severity of the leakage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a sensible, clearly written benchmark for a real curation task, but the main result — Accuracy@3 of 0.982 — does not yet mean what the abstract says. The split leaks information across reports of the same incident, so the model may be doing near-duplicate matching rather than generalizing to genuinely new incidents.\n\nWhat the paper actually does well: it formalizes report-to-incident linking as a retrieval task over the AI Incident Database, benchmarks several standard models, and finds sentence transformers win. The task is genuinely useful for AIID maintenance, and the authors describe the setup in enough detail that someone could reproduce it. They also write an honest limitations section, acknowledging the reliance on text alone and the absence of metadata. New benchmark numbers on a public database are a contribution, even if no new architecture is proposed.\n\nThe soft spots are structural, not cosmetic. Section 3.1 states that one incident has multiple reports, and Table 1 shows near-duplicate wording across those reports. Section 4.1's random 75/12.5/12.5 split does not group by incident. With 3,805 reports over 815 incidents, most test reports share their target incident with at least one training report. Fine-tuning on report-incident pairs and then testing on another report of the same incident is closer to paraphrase detection than to linking new events. The headline accuracy needs to be recomputed with an incident-disjoint split (e.g., group reports by incident before splitting, or use a temporal split aligned with report dates). The paper's RQ3 claim also has a visible problem: Table 5 shows Fold 3 dropping below Fold 2 on every metric (Accuracy@3 goes from 0.940 to 0.883), yet the text calls the trend consistent. And the code is not available until acceptance, so the numbers cannot be checked independently right now.\n\nWho is this paper for? People maintaining AIID or related incident databases, and researchers who want a baseline for report-to-incident retrieval. It is not a methods advance, but it is a useful practical evaluation. With the split fixed and the RQ3 text corrected, it would be a solid, citable benchmark. As it stands, the central quantitative claims are not established.\n\nI would send this to peer review rather than desk reject it, but with a clear request for major revision: recompute with an incident-disjoint split, add error bars or confidence intervals on the RQ3 curves, and soften the consistency claim. The task is worth doing and the authors seem capable of doing it properly.","headline":"Useful task and honest limitations, but the headline numbers rely on a random report-level split that likely measures near-duplicate retrieval, so the central claims need an incident-disjoint split before they hold.","tokens_in":13606,"tokens_out":1636,"would_cite":false,"duration_ms":18368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that associating a new AI failure report with the correct previously documented incident can be automated by ranking incidents with sentence-embedding cosine similarity, and that combining the report's title and…","keywords":["AI Incident Database","semantic retrieval","sentence embeddings","incident report matching","ranking","AI safety","transformer models","database curation"],"falsifier":"Run the same experiments with an incident-aware split that keeps all reports of an incident in one partition, or a purely temporal split by incident date, and compare Accuracy@3; a large drop from 0.982 would show the reported number depends on near-duplicate overlap between training and test.","tokens_in":12659,"feed_emoji":"🤖","tokens_out":5732,"duration_ms":50666,"temperature":0.7,"pith_summary":"The AI Incident Database (AIID) holds thousands of reports of real-world AI failures, but matching each new report to the incident it describes is done by human editors, which is slow and hard to scale. The paper tries to automate that match by treating it as a ranking problem: embed a new report and every known incident with a sentence transformer, score their cosine similarity, and return the top-ranked incidents. On a random 75/12.5/12.5 split of 3,805 reports across 815 incidents, the best model (multi-qa-MiniLM-L6-cos-v1) reaches Accuracy@3 of 0.982 when the report title and description are combined, up from 0.772 with title alone. The authors read this as evidence that manual AIID curation can be replaced by a scalable semantic-retrieval assistant.","feed_headline":"Sentence embeddings match AI incident reports with 98% accuracy","feed_subtitle":"Transformer retrieval on title plus description lifts top-three accuracy from 0.772 to 0.982.","key_machinery":"The load-bearing object is the sentence-transformer dual encoder: report texts and incident texts are mapped into the same dense vector space, and cosine similarity between a report vector and each incident vector produces the ranking. The pipeline concatenates title and description, cleans and stopword-filters the text, embeds both sides, then returns a ranked incident list. The authors fine-tune three sentence transformers and select multi-qa-MiniLM-L6-cos-v1 as the best configuration for answering their three research questions.","core_discovery":"The central claim is that report-to-incident association in the AIID is a semantic similarity ranking task and that fine-tuned sentence-embedding models solve it almost perfectly. On the test split, multi-qa-MiniLM-L6-cos-v1 achieves Accuracy@3 of 0.982 ± 0.006, MRR@3 of 0.963 ± 0.010, and NDCG@3 of 0.968 ± 0.008 using title and description together, and it beats BM25, BM25+T5, and cross-encoders at every cutoff. The paper further claims performance is stable across description lengths and improves monotonically as the training set grows, which would make the retriever a practical tool for database maintenance.","pith_inferences":["If reports of one incident bleed across the random train/test split, the 0.982 Accuracy@3 likely measures near-duplicate detection; an incident-grouped split is the natural stress test.","A production tool would need a rejection threshold below which no incident is suggested, and the paper does not derive such a threshold from the reported scores.","The same ranking formulation transfers to other multi-report incident repositories, such as aviation safety or vulnerability databases, if their ground-truth incident groupings are exposed.","Ignored metadata like harm type or system function could be used as a hard pre-filter, which would likely raise precision further at no embedding cost."],"forward_implications":["Curators of the AIID could be given a short ranked candidate list per new report, shrinking a manual database-wide scan to a few confirmations.","Because title plus description beats title alone by 15–25 points, any deployed matcher should ingest full report text rather than headlines.","The monotone improvement with training-fold size means the retriever can be re-fine-tuned as the database grows without redesigning the pipeline.","The method is lightweight enough to run on a single workstation with one GPU, making it feasible as a real-time assistant during report submission."],"supporting_citations":[{"why":"Supplies the AI Incident Database snapshot and the definition of incidents as the dataset source for all experiments.","marker":"[23]"},{"why":"Defines the AI Incident / AI Issue distinction that justifies filtering out speculative entries.","marker":"[24]"},{"why":"Documents the manual curation workflow that the proposed retrieval system is designed to assist.","marker":"[30]"},{"why":"Provides BM25, the lexical baseline all neural models are compared against.","marker":"[35]"},{"why":"Provides the BM25+T5 query-expansion baseline used as a hybrid retriever.","marker":"[29]"},{"why":"Introduces Sentence-BERT, the architecture underlying the sentence-embedding models.","marker":"[32]"},{"why":"Presents the MultiQA sentence-embedding family from which the best model is drawn.","marker":"[33]"},{"why":"Introduces MiniLM, the base model of the best-performing multi-qa-MiniLM-L6-cos-v1.","marker":"[42]"},{"why":"Introduces MPNet, the architecture of the all-mpnet-base-v2 comparison model.","marker":"[40]"}],"fun_headline_variants":["AI incident reports auto-matched via sentence embeddings","Semantic search ranks AI incident reports with 98% accuracy","Automated AI failure tracking via semantic similarity","Sentence embeddings match AI incident reports at 98% top-3","Retrieval model automates AI incident reporting with 98% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy depends on a random split that does not keep all reports of the same incident together; if the same event's text appears in both training and testing, the model may be rewarded for recognizing near-duplicates rather than generalizing to genuinely new incidents.","fun_headline_variants_meta":{"raw":{"variants":["AI incident reports auto-matched via sentence embeddings","Semantic search ranks AI incident reports with 98% accuracy","Automated AI failure tracking via semantic similarity","Sentence embeddings match AI incident reports at 98% top-3","Retrieval model automates AI incident reporting with 98% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2819,"prompt_tokens":943,"completion_tokens":1876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1795}},"tokens_in":559,"tokens_out":1876,"duration_ms":12557,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:27:50.869607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiments with an incident-aware split that keeps all reports of an incident in one partition, or a purely temporal split by incident date, and compare Accuracy@3; a large drop from 0.982 would show the reported number depends on near-duplicate overlap between training and test.","supporting_citations":[{"cited_title":"Indexing AI Risks with Incidents, Issues, and Variants","cited_arxiv_id":"2211.10384","evidence_quote":"Defines the AI Incident / AI Issue distinction that justifies filtering out speculative entries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces MiniLM, the base model of the best-performing multi-qa-MiniLM-L6-cos-v1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces MPNet, the architecture of the all-mpnet-base-v2 comparison model."}],"review_version":1}