{"id":"1514d133-f2d8-474f-a578-f808d68fe901","arxiv_id":"2504.16934","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An IDF-based tool that highlights the three rarest stack-trace frames was integrated into JetBrains' bug processing and received a usefulness score of 3.6/5 from 18 developers.","lead":"JetBrains built a feature that pre-highlights the three rarest frames in a stack trace to help developers find bugs faster. Eighteen developers rated it 3.6 out of 5 for usefulness, suggesting the idea has promise but needs better explanations and learned models.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The usefulness claim rests on an untested premise that rare frames are important; survey scores do not validate whether the IDF selection actually helps developers.","rationale":"The reader's weakest-assumption analysis correctly identifies the unvalidated link between frame rarity and importance. This is the single most load-bearing concern because the entire value of the feature depends on it: if the highlighted frames are wrong, the survey's usefulness score reflects the general concept of highlighting rather than the specific IDF selection, and the feature could even mislead developers. The concern is not an external disagreement with community consensus; it is an internal, falsifiable gap in the evidence chain. The paper itself provides the path to validation by mentioning that manual selections are a core saved artifact of the existing system, yet it never connects this data to the evaluation. I do not think this warrants rejection: the report is honest, the feature is deployed at scale, and the survey provides some evidence of perceived value, which is appropriate for an industrial tool note. However, the conditional verdict should remain, because the evaluation does not currently establish that the IDF heuristic identifies what developers consider important. Requiring a retrospective comparison against manual selections is a modest, concrete condition that would materially strengthen the central claim.","tokens_in":2433,"tokens_out":2152,"duration_ms":22874,"concrete_test":"Mine the internal bug-processing tool's history for stack traces that already have developer manually selected frames recorded. For each such trace, compute the IDF top-3 suggestions from the corpus as it existed at that time, then measure overlap (e.g., precision, recall, or whether at least one suggestion matches a manual selection). Compare this overlap against two baselines: top-3 by document frequency (most common frames) and three randomly chosen frames. If IDF top-3 does not significantly outperform random selection, the heuristic is not validated; if it clearly outperforms both baselines, the core assumption is supported. This is a purely retrospective, low-cost check using data the tool already stores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the feature pre-highlights potentially important frames, and that it is useful (mean 3.6/5). This claim depends on the IDF heuristic: the three rarest frames in a stack trace contain the most specific information about the bug. That premise is never tested. The survey asks developers to rate usefulness and visualization, but it does not ask whether the highlighted frames are actually relevant, whether a different selection would be better, or whether the highlights changed their behavior. Crucially, the tool already records the frames that developers manually mark as important (Section I, Figure 1b), and these manual selections are saved and shared. A direct comparison between IDF top-3 suggestions and these manual selections is both feasible and necessary. Without such a comparison, a plausible alternative is that the highlights are often noise — rare frames may come from user-specific configuration, third-party libraries, or stack frames that are unique but unrelated to the root cause. The survey comment that developers wanted the feature to 'motivate its selection' also suggests the IDF rationale was not self-evidently correct. The paper is transparent about this being an initial version, but it still asserts the feature 'potentially important', which is precisely what remains unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the development, integration, and evaluation of a feature in JetBrains' internal bug-processing tool that automatically pre-highlights three stack frames with the highest inverse document frequency (IDF) in a corpus of stack traces. The paper describes the UI, the IDF-based selection heuristic, and a survey of 18 JetBrains developers who rated the feature's usefulness at a mean of 3.6/5 and its visualization at 4.0/5, along with qualitative feedback. The authors state that the feature was bundled into production and is now displayed for all developers at the latest version, and they position the work as an initial step toward more automatic stack-trace analysis.","tokens_in":2786,"tokens_out":4601,"duration_ms":49144,"significance":"If the IDF heuristic actually identifies frames that developers find important, this is a low-cost, scalable addition to crash-triage tooling, and the paper benefits from a real deployment at a large software company with tens of millions of stack traces. The paper is transparent about the feature being an initial version and about the survey being simple. Its concrete strengths are the production integration and the honest reporting of modest, mixed feedback. However, the central premise that rare frames are the important frames is never validated, and the survey does not measure whether the highlighted frames are actually relevant. As it stands, the paper supports a descriptive claim about a deployed feature and its perceived usefulness, not a claim about the correctness or effectiveness of the IDF selection logic. The contribution would be significantly strengthened by comparing the IDF suggestions with the manual frame selections that the tool already records, which is a feasible and natural experiment. The significance of the current version is therefore modest, though the deployment experience may be valuable to practitioners.","major_comments":[{"comment":"The IDF selection logic is not validated against any ground truth. The tool already records the frames that developers manually select as important (Section I, Fig. 1b), and the paper does not compare the IDF top-3 suggestions with these human selections. Without such a comparison, the premise that 'the rarest frames might contain the most specific information about a particular stack trace' remains unsupported, and the feature could be highlighting rare but irrelevant frames. A direct precision-at-k or rank-correlation comparison on a sample of real stack traces with saved manual selections is both feasible and necessary to support the claim that the highlighted frames are potentially important.","section":"Section II (Approach)"},{"comment":"The evaluation evidence for usefulness is a self-report Likert rating from 18 developers who were aware that they were rating a new feature, with no control condition, no baseline, and no statistical characterization beyond a mean. The survey asks about usefulness and visualization but does not ask whether the highlighted frames were actually judged relevant, whether they helped locate the bug, or whether they changed the developer's behavior. A mean of 3.6/5 documents a mildly positive opinion, but it does not test whether the highlighted frames are, in fact, important. I recommend reporting score distributions, adding questions about the correctness and helpfulness of the specific highlighted frames, and ideally conducting a within-subject comparison of the same stack traces with and without highlighting. The qualitative comment that developers wanted the feature to 'motivate its selection' further indicates that the IDF rationale was not self-evidently persuasive.","section":"Section II (Survey methodology and Results)"},{"comment":"The title and abstract claim that the feature 'find[s] important stack frames,' but the paper provides no evidence that the highlighted frames coincide with frames that developers consider important. The only quantitative result is a usefulness rating, which is consistent with a feature that is visually pleasant but not actually correct. I ask the authors to either soften the claim to 'suggest potentially relevant frames' or provide the validation described in the first major comment. This is a load-bearing issue because the central contribution is precisely the identification of important frames, not merely the addition of a UI affordance.","section":"Title and Abstract"}],"minor_comments":[{"comment":"The paper does not define the exact frame identity used to compute IDF (e.g., full method signature vs. normalized method name, or whether user-specific frames are filtered). A sentence specifying the tokenization and normalization would clarify the heuristic.","section":"Section II (Approach)"},{"comment":"The survey methodology does not state the exact Likert question wording, the response rate, or whether all 18 respondents were among the 25 contacted. The sentence '18 of whom confirmed that they recently used the tool and saw this feature, so we used their responses' could be read as either 18 respondents or 18 confirmed users, and this should be clarified.","section":"Section II (Survey methodology)"},{"comment":"The paper reports only mean scores and no distributions, standard deviations, or individual-rating counts. Reporting these would help the reader assess the strength and consistency of the ratings, especially with n=18.","section":"Section II (Results and future work)"},{"comment":"The reference list has minor formatting issues, including extra spaces in author names (e.g., 'Y . V asiliev') and inconsistent capitalization in the title of reference [10]. These are simple copyediting fixes.","section":"Section I"},{"comment":"The choice k=3 for the number of highlighted frames is presented without any sensitivity analysis or explicit rationale beyond avoiding too many or too few suggestions. A brief justification or a small sensitivity check would strengthen the design.","section":"Section II (Approach)"}],"recommendation":"major_revision","confidential_remarks":"This is a short industry experience paper. The deployment is real and the writing is clear, but the research contribution is currently thin because the core heuristic is unvalidated. The authors have access to an obvious ground truth—the manual frame selections already recorded by their tool—so the validation I request is well within the scope of the manuscript. I would not recommend rejection because the feature is deployed and the missing evidence is obtainable, but the paper should not be accepted in its current form. Depending on the venue, the authors may also wish to position the paper explicitly as an experience report rather than an empirical evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a modest, transparent industrial experience report, and it is exactly as limited as it looks. The new thing is the application of IDF to stack traces to pre-highlight three rare frames, plus a small deployment survey. The writing is honest and the deployment scale (tens of millions of stack traces) is real, but the evaluation only tells you that 18 developers found the UI moderately useful, not that the IDF selection is actually pointing at important frames.\n\nWhat it does well: the feature is simple, unobtrusive, and integrated into a real production tool. The authors are appropriately careful about the claim ('potentially important'), they report the survey results plainly, and they list sensible future directions (learning, motivating selection). The IDF baseline is a reasonable first cut, and the idea of using the existing manual selections as a ground truth is implicit in their description of the system — which makes its absence from the evaluation more noticeable.\n\nWhere it is soft: the central premise — that rare frames are the important ones — is never validated. The tool already lets developers manually mark frames as important, and those marks are saved and shared. The paper could have compared the IDF top-3 against these manual selections on a sample of traces. That comparison is feasible, and without it the 3.6/5 usefulness score is hard to interpret: it could reflect that the highlights are often relevant, or that any unobtrusive nudge feels mildly useful, or simply the novelty of seeing something pre-highlighted. The survey doesn't ask whether the highlighted frames were actually relevant, and one developer's request for the feature to 'motivate its selection' suggests the rationale wasn't self-evident. The absence of a baseline is a real gap for any causal claim, but the paper doesn't make a strong causal claim; it's more a first-step report. Still, as published, a reader could easily over-read the survey as evidence that IDF-based highlighting works.\n\nBottom line: a short, useful artifact for someone building similar tooling, but not a result that establishes the value of the heuristic. It deserves a fast, serious review — it should be published somewhere as an industrial experience report, with the missing comparison called out as future work or addressed. I'd send it to review rather than desk reject, and I'd tell the authors to add the manual-selection comparison or tighten the language to 'we deployed this and developers gave it a 3.6; we haven't yet shown it helps.'","headline":"A short, honest industrial note that describes a plausible feature and a weak evaluation; the heuristic linking rarity to importance is never actually tested.","tokens_in":3148,"tokens_out":1988,"would_cite":false,"duration_ms":19798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rarest stack frames are a useful first guess at what matters in a crash trace.","keywords":["stack trace analysis","inverse document frequency","frame importance","crash report triage","bug processing","developer survey","production feature"],"falsifier":"Collect a sample of stack traces from the tool, record both the IDF-highlighted frames and the frames developers manually save as important, and measure their agreement; if the overlap is no better than chance or than the three most frequent frames, the rarity heuristic is falsified.","tokens_in":77,"feed_emoji":"🐞","tokens_out":5907,"duration_ms":115916,"temperature":0.7,"pith_summary":"The paper reports on a production feature that automatically guesses which frames in a stack trace deserve a developer's attention. The guess is simple: within each trace, highlight the three frames that are rarest across the company's entire corpus of received stack traces, using inverse document frequency. The hypothesis is that rare frames carry the most issue-specific information, so they make a reasonable starting point before a developer manually selects the frames they consider important. Eighteen developers who had used the feature in the internal bug-processing tool rated its usefulness at a mean of 3.6 out of 5 and the visualization at 4.0. The paper treats this as evidence that the idea is viable and that the same interface can later host smarter selectors.","feed_headline":"The three rarest frames in a crash trace get highlighted first","feed_subtitle":"IDF-based highlighting ships in the internal tool; 18 developers scored it 3.6 out of 5 for usefulness.","key_machinery":"The mechanism is IDF (inverse document frequency), a corpus-relative measure of how rare a frame is: a frame that appears in few of the received stack traces gets a high score. For each incoming trace, the tool ranks its frames by IDF and flags the top three with bold text and an exclamation icon. This rarity ranking is the load-bearing object; the surrounding UI is deliberately designed so that the pre-highlighting recedes once a developer makes manual selections.","core_discovery":"The central claim is that an information-retrieval staple, inverse document frequency computed over a large corpus of stack traces, can serve as a training-free proxy for frame importance. In its deployed form, the tool pre-highlights the three highest-IDF frames in every stack trace, on the reasoning that frames appearing rarely across the corpus are the ones most specific to the particular failure. Because the suggestions are visually subordinate to manual selections, they help orient the developer without overriding the shared, saved manual highlighting that the workflow already depends on. The paper's claim is not that IDF is optimal, but that this lightweight heuristic is useful enough to ship, with the survey results offered as initial validation.","pith_inferences":["A direct test the paper does not run: compare the IDF-highlighted frames against the frames developers manually mark as important; agreement would validate the rarity premise, while its absence would show the tool highlights noise, and the saved manual selections make this comparison inexpensive.","Because IDF is computed over the whole corpus, its suggestions depend on corpus composition: in a system with many duplicate common crashes, a misspelled or one-off frame can look rare for the wrong reason, so per-product or per-component IDF might behave differently.","The survey measures what developers think of the feature, not whether it accelerates triage; a click-through rate on highlighted frames or a time-to-first-analysis measure would test the productivity benefit the feature is meant to provide.","The same rarity idea could be validated externally on public crash-report databases, where manual frame importance could be inferred from fix commits or from clusters of similar reports."],"forward_implications":["If the heuristic holds, every newly opened stack trace starts with a visible candidate set, shortening the time before a developer engages with the error.","Manual selections remain the source of truth, so the suggestions can be ignored or corrected without changing the existing triage workflow.","The 3.6/5 usefulness score, with 4.0 for visualization, implies the concept is accepted but that developers want explanation, such as tooltips and a stated reason for each suggestion.","The same UI can host more powerful selectors, including learned models, without redesigning the workflow.","Because IDF requires no labeled data, the highlighting can be deployed in any system that already collects a large stream of stack traces."],"supporting_citations":[{"why":"Supplies the inverse document frequency method used to rank frames by rarity.","marker":"[10]"},{"why":"Establishes the motivating problem of processing field crash reports efficiently.","marker":"[1]"},{"why":"Shows automated duplicate detection in bug tracking, a related part of the triage pipeline.","marker":"[2]"},{"why":"One of the prior stack-trace processing techniques that define the tool's corpus and deduplication context.","marker":"[5]"},{"why":"Describes the newer stack-trace deduplication used in the same internal system so the corpus reflects distinct issues.","marker":"[6]"},{"why":"Represents the existing automated task in the same tool that the highlighting feature augments.","marker":"[9]"}],"fun_headline_variants":["IDF picks rare frames to spotlight in crash traces","Rare frames flagged first in JetBrains' bug tool","Training-free frame importance via IDF in stack traces","Stack trace triage: IDF highlights key frames"],"cache_read_input_tokens":5376,"weakest_assumption_plain":"Section II's premise that the rarest frames contain the most issue-specific information is load-bearing, and the paper does not validate it against the frames developers actually select, so if rarity does not track importance, the pre-highlighting points at noise.","fun_headline_variants_meta":{"raw":{"variants":["IDF picks rare frames to spotlight in crash traces","Rare frames flagged first in JetBrains' bug tool","Training-free frame importance via IDF in stack traces","Stack trace triage: IDF highlights key frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3074,"prompt_tokens":713,"completion_tokens":2361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":329,"completion_tokens_details":{"reasoning_tokens":2298}},"tokens_in":329,"tokens_out":2361,"duration_ms":17568,"temperature":1.0,"reasoning_tokens":2298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:57.674817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a sample of stack traces from the tool, record both the IDF-highlighted frames and the frames developers manually save as important, and measure their agreement; if the overlap is no better than chance or than the three most frequent frames, the rarity heuristic is falsified.","supporting_citations":[{"cited_title":"Sushentsev, A","cited_arxiv_id":null,"evidence_quote":"Supplies the inverse document frequency method used to rank frames by rarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the motivating problem of processing field crash reports efficiently."},{"cited_title":"Dhaliwal, F","cited_arxiv_id":null,"evidence_quote":"Shows automated duplicate detection in bug tracking, a related part of the triage pipeline."},{"cited_title":"Bartz, J","cited_arxiv_id":null,"evidence_quote":"One of the prior stack-trace processing techniques that define the tool's corpus and deduplication context."},{"cited_title":"Aggregation of Stack Trace Similarities for Crash Report Deduplication","cited_arxiv_id":"2205.00212","evidence_quote":"Describes the newer stack-trace deduplication used in the same internal system so the corpus reflects distinct issues."},{"cited_title":"(Accessed: 01.12.2024) PyCharm","cited_arxiv_id":null,"evidence_quote":"Represents the existing automated task in the same tool that the highlighting feature augments."}],"review_version":1}