{"id":"e4d347b8-90d7-4aa5-a6e7-74fdcfa09c6d","arxiv_id":"2412.14802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A retrieve-and-rerank model with BPE tokenization outperforms prior stack trace deduplication methods on four open-source datasets and a new industrial dataset, while running faster than most baselines.","lead":"The authors build a two-stage system for deduplicating crash stack traces: a fast embedding model retrieves candidate traces, then a cross-encoder reranks them using repeated-frame signals. They release SlowOps, a JetBrains industrial dataset with roughly ten times more reports per category than existing open-source benchmarks, and report higher accuracy and lower latency than most baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed all-dataset accuracy advantage may rest on an unspecified BPE tokenizer split; if the tokenizer is fit on the full corpus including test reports, small 0.01–0.04 Acc@1 margins over text-embedding-3-small could be inflated by test-set leakage.","rationale":"I read the paper as making an empirical claim: a two-stage BPE/biLSTM embedding model plus a cross-encoder reranker is simultaneously more accurate (Acc@1) than baselines on all five datasets and fast enough in practice. The condition that has to hold for this claim is that the comparison is free of test-set information in the model's preprocessing. The BPE tokenizer statement is the most direct place where that condition could fail: 'same dataset as the model' is ambiguous, and the evaluation section is silent on tokenizer split. BPE is unsupervised, so the effect is probably not massive, but the reported margins are small enough that even modest leakage matters. The reader's weakest_assumption identifies exactly this; I agree. I considered no-error-bars as an alternative, but the all-five-datasets pattern provides some protection, and the BPE concern is more specific and more directly testable from the released code. I recommend keeping the CONDITIONAL verdict: the concern does not by itself refute the paper, but the central accuracy claim should not be accepted as final until the tokenizer split is resolved by code inspection and a training-only retraining run.","tokens_in":15734,"tokens_out":6673,"duration_ms":61733,"concrete_test":"Inspect the released replication package to determine whether the BPE tokenizer is fitted inside the training split or before the temporal split. Concretely, check the data-loading script for the call that trains the BPE tokenizer and whether it receives train reports only or the full dataset. Then rerun the Table II evaluation with a tokenizer fitted on the training split only; if any Acc@1 entry for 'Ours (Embedding model + Reranker)' drops below text-embedding-3-small on Ubuntu or Eclipse, the all-datasets claim fails. As a secondary check, compute the fraction of test-report tokens that are absent from a training-only BPE vocab; a large fraction would demonstrate that the full-corpus tokenizer is encoding test-specific information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the proposed model 'outperforms other models on all datasets' (Section IV-E) depends on a fair comparison, and the fairness condition is least secure at the preprocessing step. Section III-A1 states: 'The BPE tokenizer is trained using the same dataset as the model,' but the paper never specifies that this means only the training split. Section IV-B describes a chronological 70/10/20 split into train/validation/test but does not state where tokenizer fitting happens relative to that split. If the BPE vocab is learned from all reports, including test reports, then the tokenizer can encode test-specific subword patterns, and the model's embeddings for test frames can be artificially well-formed. The reported Acc@1 margins over text-embedding-3-small are 0.03 (Ubuntu), 0.02 (Eclipse), 0.06 (NetBeans), 0.05 (Gnome), and 0.05 (SlowOps); a leakage of even a few percent on the smaller-margin datasets could flip the comparison. Because the same code is released, this is checkable; until it is checked, the headline accuracy claim is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage stack trace deduplication system: a biLSTM-based embedding model trained with InfoNCE on BPE-tokenized stack frames, followed by a cross-encoder reranker that adds a learned significance vector to frames appearing in both traces. The authors introduce SlowOps, a new industrial dataset from JetBrains with an order of magnitude more reports per category than existing open-source datasets. They evaluate the approach on Ubuntu, Eclipse, NetBeans, Gnome, and SlowOps using Acc@1 for attachment accuracy, ROC-AUC for new-category creation, and wall-clock time per report. The main claim is that the proposed approach outperforms existing methods (including text-embedding-3-small) on all datasets in accuracy while being faster than most baselines. Code and data are released.","tokens_in":15977,"tokens_out":5134,"duration_ms":34430,"significance":"If the accuracy and speed claims hold, the paper makes a practical contribution to crash report deduplication: the two-stage design (fast ANN retrieval plus a more accurate reranker) is a sensible architecture that other practitioners could adopt, and the SlowOps dataset fills a clear gap by providing a large, industrial, manually curated benchmark with many reports per category. The paper also improves on prior evaluations by measuring ROC-AUC for new-category decisions and by timing the retrieval pipeline with precomputed embeddings. The explicit release of code, data, and a Zenodo dataset is a strength that should be credited. The main risk is that several small accuracy margins over the strongest baseline rest on a preprocessing choice (BPE tokenizer training set) that is not fully specified, and on single-run measurements without variance estimates.","major_comments":[{"comment":"The paper states in Section III-A1 that \"The BPE tokenizer is trained using the same dataset as the model,\" but it does not specify that this means only the training split of the chronological 70/10/20 split described in Section IV-B. If the BPE vocabulary and token frequencies are learned from the full dataset including test reports, then the subword representation of test frames could be artificially well-formed, giving the proposed model an advantage over baselines that use pre-defined tokenizers (e.g., text-embedding-3-small). The reported Acc@1 margins in Table II over text-embedding-3-small are only 0.02 on Eclipse, 0.03 on Ubuntu, 0.05 on Gnome, 0.06 on NetBeans, and 0.05 on SlowOps, so a small leakage effect could change the outcome on some datasets. Please clarify that the tokenizer is fit only on the training split, or retrain the tokenizer accordingly and rerun the experiments if this was not already the case.","section":"Section III-A1 and Section IV-B"},{"comment":"All Acc@1 and ROC-AUC values are reported as single point estimates with no confidence intervals, standard deviations, or significance tests. The margins over the strongest baseline (text-embedding-3-small) are small on several datasets (for example, 0.02 on Eclipse and 0.03 on Ubuntu in Table II). Without repeated runs or bootstrap intervals, it is not possible to assess whether these differences are reliable or within noise. Please provide results over multiple random seeds, or at least bootstrap confidence intervals for the test-set metrics, and, where appropriate, a statistical significance test for the pairwise comparison against the strongest baseline.","section":"Tables II and III"},{"comment":"The threshold T used for the new-category creation decision is described as \"trained for each particular dataset\" by selecting the value that gives the best F1 score, but the paper does not state whether this selection is made on the validation split or on the test split. If T is chosen using test labels, the reported ROC-AUC numbers in Table III are not directly affected because ROC-AUC is threshold-free, but any claim about the operational behavior of the new-category decision in a deployed system would be optimistic. Please explicitly state which split is used for threshold selection and, ideally, report precision/recall (or F1) on the test split for the chosen T.","section":"Section IV-B"}],"minor_comments":[{"comment":"The statement that \"for creating new categories, our approach ... ties with an LLM text-embedding-3-small on SlowOps\" is only true for the embedding-model-only variant (ROC-AUC 0.99 vs. 0.99); with the reranker, Table III shows 0.96 vs. 0.99, which is not a tie. Please clarify which variant this claim refers to.","section":"Introduction and Section IV-E2"},{"comment":"The speed comparison reports average time per report on Ubuntu but does not state how many repetitions were performed or give any measure of variance. Please report the number of runs and, if possible, a standard deviation or confidence interval.","section":"Table IV"},{"comment":"Reference [24] is cited as \"text-embedding-ada-003\" but the baseline is called text-embedding-3-small elsewhere; please correct the reference title/identifier to match the actual model used.","section":"References"},{"comment":"The name \"Rodriguez et al.\" appears in the text while the reference list uses \"Rodrigues et al.\"; please unify the spelling.","section":"Section IV-D2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong practical contribution and the release of code and the SlowOps dataset is a clear asset. The main concern is the fairness of the comparison to baselines if the BPE tokenizer is trained on the test split; because the code is released, this is checkable, and the authors should be asked to either confirm that only the training split is used or rerun the experiments otherwise. The lack of variance estimates is a second concern, especially for the small margins over text-embedding-3-small. These issues are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a genuinely useful, practice-oriented paper on stack trace dedup, and the SlowOps dataset plus the multi-faceted evaluation are the real contributions. The architecture—BPE tokenization, a biLSTM embedding model with FAISS, then a reranker that adds a learned significance vector to frames appearing in both traces—is a modest but sensible twist on retrieve-and-rerank, and it works. The head-to-head with text-embedding-3-small is fair and informative. I'd send this to review, with one condition: the BPE tokenizer split needs to be clarified.\n\nWhat the paper does well: it evaluates on five datasets, including a new industrial one with about 50 unique reports per category versus around 1-3 for the open-source ones. It measures accuracy, new-category creation (ROC-AUC), and time assuming pre-computed embeddings, which is the right setting for large systems. It releases code and data. On Ubuntu, Eclipse, NetBeans, Gnome, and SlowOps, the full pipeline beats the strongest baseline (text-embedding-3-small) by 0.02 to 0.06 in Acc@1, and does it at 144.5 ms per report versus 1021.2 ms. That is a legitimate result.\n\nThe soft spots are real but not fatal. The paper says the BPE tokenizer is 'trained using the same dataset as the model' without stating that only the training split is used. If the tokenizer sees test reports, its vocab encodes test-specific subwords, and with margins this small, that could flip the comparison. The released code makes this checkable, but until it is checked, the 'outperforms on all datasets' claim is conditional. Second, there are no error bars or repeated runs; the margins over text-embedding-3-small are 0.01-0.06, so single seeds are thin. Third, the threshold T is fit per dataset and several hyperparameters (K, vocab size, max pairs, temperature) are chosen in preliminary experiments; the paper admits this in the threats section, but it still means the headline numbers are somewhat post-hoc. Fourth, the speed comparison is only on Ubuntu, which is fine as a starting point, not as a general claim. Minor: on SlowOps, the reranker's ROC-AUC (0.96) is below the embedding-only model (0.99) and ties text-embedding-3-small only if you use the embedding-only variant; the text glosses over this.\n\nBottom line: the dataset and evaluation framework deserve a serious referee. I'd accept for review and ask for the tokenizer split, error bars, and a clearer statement of which variant is being compared on ROC-AUC. The central claim is plausible; it just needs tightening.","headline":"A practical stack-trace dedup system with a valuable new dataset and a realistic evaluation; the accuracy claims are plausible but rest on a possible tokenizer leakage path and single runs, so condition acceptance on those being fixed.","tokens_in":16527,"tokens_out":2746,"would_cite":true,"duration_ms":18590,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a two-stage embedding-plus-reranker model outperforms all compared stack-trace deduplication methods on every tested dataset while remaining fast enough for real-time use.","keywords":["stack trace deduplication","crash report grouping","byte pair encoding","cross-encoder reranker","approximate nearest neighbor","ROC-AUC","industrial dataset","biLSTM"],"falsifier":"Retrain the BPE tokenizer on only the training split, rerun the Acc@1 and ROC-AUC evaluations, and compare; if the scores drop materially, test-set leakage through tokenization was inflating the reported gains.","tokens_in":15498,"feed_emoji":"🐞","tokens_out":7088,"duration_ms":53300,"temperature":0.7,"pith_summary":"This paper claims that automatic deduplication of error reports can be made both more accurate and faster than current alternatives by splitting the task into two stages: a lightweight embedding model retrieves the most plausible matching stack traces, and a reranker re-scores those candidates using shared frames. To test this in realistic conditions, the authors introduce SlowOps, an industrial dataset with an order of magnitude more stack traces per category than existing open-source datasets, and measure not only category-assignment accuracy but also the ability to create new categories and per-report latency. Across Ubuntu, Eclipse, NetBeans, Gnome, and SlowOps, the two-stage model reports the highest Acc@1, ahead of the strongest baseline text-embedding-3-small, and reaches this accuracy in 144.5 ms per report (8.7 ms without reranking). The paper's central claim is that a practical stack-trace deduplication system can have both leading accuracy and real-time speed.","feed_headline":"Two-stage model beats stack-trace dedup baselines on every dataset","feed_subtitle":"A retriever plus a reranker that marks shared frames lifts accuracy at 144.5 ms per report.","key_machinery":"The load-bearing machinery is the two-stage retrieve-then-rerank pipeline. The first stage uses BPE tokenization (a fixed 10,000-token subword vocabulary) plus a biLSTM encoder and FAISS-style approximate nearest neighbor search so that unseen package names still tokenize and candidate retrieval is fast. The second stage is a cross-encoder reranker: it encodes each frame of a query-candidate pair, and whenever a frame occurs in both traces it adds a learned significance vector V to that frame's embedding on both sides, so the two traces interact before the pair is scored by an MLP. That shared-frame signal is what lets the reranker correct mistakes made by the embedding-only stage; the paper reports that even this limited interaction improves accuracy.","core_discovery":"The central discovery is that a two-stage retrieval architecture outperforms every compared method on all five datasets. The embedding stage tokenizes stack frames with byte-pair encoding (BPE), encodes each trace with a biLSTM, and uses approximate nearest neighbor search to return the top K candidates; the reranker stage is a biLSTM cross-encoder that adds a learned significance vector to the embeddings of frames that appear in both the query and the candidate, then scores the pair with an MLP. With the reranker, Acc@1 is 0.65 on Ubuntu, 0.75 on Eclipse, 0.52 on NetBeans, 0.45 on Gnome, and 0.98 on SlowOps, beating text-embedding-3-small on every benchmark. The same model also earns the best or tied-best ROC-AUC for creating new categories on all datasets except SlowOps, where the embedding-only version leads, and it runs in 144.5 ms with reranking or 8.7 ms without, compared with 1021.2 ms for the LLM baseline.","pith_inferences":["An implication the authors leave implicit: the reranker's accuracy gain is probably driven by the learned significance vector for shared frames, so ablating that vector would isolate how much of the improvement comes from cross-trace interaction rather than from a larger model.","Because SlowOps contains only Slow Operation Assertion reports, where the informative frames sit in the middle of the trace, methods that weight top frames heavily may be unfairly disadvantaged, and rankings could differ on other error types.","The speed comparison assumes embeddings of known reports are pre-computed; in a live stream, new reports must periodically be re-embedded, so an end-to-end ingestion test would clarify real throughput beyond per-report scoring time."],"forward_implications":["Deployments can precompute embeddings of known reports and only rerank the top K candidates, so the per-report latency of a high-accuracy system is tens of milliseconds rather than seconds.","BPE tokenization removes the need for fixed frame and package vocabularies, so new packages added after training still receive meaningful token vectors instead of random initializations.","The high accuracy on SlowOps suggests that datasets with many reports per category are substantially easier, and that benchmark results on small open-source datasets may understate real-world performance.","The threshold T that balances attaching to an existing category versus creating a new one must be tuned per dataset, since category structures differ by orders of magnitude."],"supporting_citations":[{"why":"Provides the S3M baseline and the biLSTM-based encoding scheme that the embedding model extends.","marker":"[13]"},{"why":"Provides the DeepCrash baseline and the prior time-measurement setup the paper argues is unrealistic.","marker":"[15]"},{"why":"Provides the FaST baseline and the ROC-AUC evaluation of new-category creation that the paper extends to deep models.","marker":"[16]"},{"why":"Supplies byte-pair encoding, the subword tokenization used to handle unseen stack frames.","marker":"[21]"},{"why":"Supplies the approximate nearest neighbor search used for fast candidate retrieval.","marker":"[22]"},{"why":"Is the strongest LLM embedding baseline and the main accuracy competitor on all datasets.","marker":"[24]"},{"why":"Provides the open-source Ubuntu, Eclipse, NetBeans, and Gnome benchmarks used in evaluation.","marker":"[19]"},{"why":"Provides the TraceSim datasets and alignment method used as additional public evaluation data.","marker":"[20]"}],"fun_headline_variants":["Two-stage model wins stack-trace dedup on all 5 datasets","Retriever plus reranker beats stack-trace baselines and runs faster","Stack trace dedup: two-stage model outperforms and is faster","New dedup model tops accuracy and speed on real-world traces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The byte-pair tokenizer is trained on the same dataset as the model, and the paper does not state that only the training split is used, so tokenization could encode information from test reports and inflate the reported accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage model wins stack-trace dedup on all 5 datasets","Retriever plus reranker beats stack-trace baselines and runs faster","Stack trace dedup: two-stage model outperforms and is faster","New dedup model tops accuracy and speed on real-world traces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2316,"prompt_tokens":1100,"completion_tokens":1216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":1139}},"tokens_in":716,"tokens_out":1216,"duration_ms":7349,"temperature":1.0,"reasoning_tokens":1139,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:53:55.537054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the BPE tokenizer on only the training split, rerun the Acc@1 and ROC-AUC evaluations, and compare; if the scores drop materially, test-set leakage through tokenization was inflating the reported gains.","supporting_citations":[{"cited_title":"S3M: Siamese stack (trace) similarity measure,","cited_arxiv_id":null,"evidence_quote":"Provides the S3M baseline and the biLSTM-based encoding scheme that the embedding model extends."},{"cited_title":"Deep- Crash: deep metric learning for crash bucketing based on stack trace,","cited_arxiv_id":null,"evidence_quote":"Provides the DeepCrash baseline and the prior time-measurement setup the paper argues is unrealistic."},{"cited_title":"FaST: A linear time stack trace alignment heuristic for crash report deduplication,","cited_arxiv_id":null,"evidence_quote":"Provides the FaST baseline and the ROC-AUC evaluation of new-category creation that the paper extends to deep models."},{"cited_title":"Billion-scale similarity search with GPUs,","cited_arxiv_id":null,"evidence_quote":"Supplies the approximate nearest neighbor search used for fast candidate retrieval."},{"cited_title":"(2022) text-embedding-ada-003","cited_arxiv_id":null,"evidence_quote":"Is the strongest LLM embedding baseline and the main accuracy competitor on all datasets."},{"cited_title":"The unreasonable effectiveness of traditional information retrieval in crash report dedupli- cation,","cited_arxiv_id":null,"evidence_quote":"Provides the open-source Ubuntu, Eclipse, NetBeans, and Gnome benchmarks used in evaluation."},{"cited_title":"TraceSim: An alignment method for computing stack trace similarity,","cited_arxiv_id":null,"evidence_quote":"Provides the TraceSim datasets and alignment method used as additional public evaluation data."}],"review_version":1}