Finding Important Stack Frames in Large Systems

Aleksandr Khvorov; Denis Sushentsev; Yaroslav Golubev

REVIEW 3 major objections 5 minor 12 references

Finding Important Stack Frames in Large Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rarest stack frames are a useful first guess at what matters in a crash trace.

desk verdict A short, honest industrial note that describes a plausible feature and a weak evaluation; the heuristic linking rarity to importance is never actually tested. read the letter →

arxiv 2504.16934 v1 pith:NOJEHTGK submitted 2025-01-12 cs.SE

Aleksandr Khvorov , Yaroslav Golubev , Denis Sushentsev This is my paper · ORCID

classification cs.SE

keywords stacktraceanalysisinversedocumentfrequencyframeimportancecrashreporttriagebugprocessingdevelopersurveyproductionfeature

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports on a production feature that automatically guesses which frames in a stack trace deserve a developer's attention. The guess is simple: within each trace, highlight the three frames that are rarest across the company's entire corpus of received stack traces, using inverse document frequency. The hypothesis is that rare frames carry the most issue-specific information, so they make a reasonable starting point before a developer manually selects the frames they consider important. Eighteen developers who had used the feature in the internal bug-processing tool rated its usefulness at a mean of 3.6 out of 5 and the visualization at 4.0. The paper treats this as evidence that the idea is viable and that the same interface can later host smarter selectors.

What carries the argument

The mechanism is IDF (inverse document frequency), a corpus-relative measure of how rare a frame is: a frame that appears in few of the received stack traces gets a high score. For each incoming trace, the tool ranks its frames by IDF and flags the top three with bold text and an exclamation icon. This rarity ranking is the load-bearing object; the surrounding UI is deliberately designed so that the pre-highlighting recedes once a developer makes manual selections.

What would settle it

Collect a sample of stack traces from the tool, record both the IDF-highlighted frames and the frames developers manually save as important, and measure their agreement; if the overlap is no better than chance or than the three most frequent frames, the rarity heuristic is falsified.

Watch

Extended reading notes

Core claim

The central claim is that an information-retrieval staple, inverse document frequency computed over a large corpus of stack traces, can serve as a training-free proxy for frame importance. In its deployed form, the tool pre-highlights the three highest-IDF frames in every stack trace, on the reasoning that frames appearing rarely across the corpus are the ones most specific to the particular failure. Because the suggestions are visually subordinate to manual selections, they help orient the developer without overriding the shared, saved manual highlighting that the workflow already depends on. The paper's claim is not that IDF is optimal, but that this lightweight heuristic is useful enough to ship, with the survey results offered as initial validation.

Load-bearing premise

Section II's premise that the rarest frames contain the most issue-specific information is load-bearing, and the paper does not validate it against the frames developers actually select, so if rarity does not track importance, the pre-highlighting points at noise.

Editorial extensions

If this is right

If the heuristic holds, every newly opened stack trace starts with a visible candidate set, shortening the time before a developer engages with the error.
Manual selections remain the source of truth, so the suggestions can be ignored or corrected without changing the existing triage workflow.
The 3.6/5 usefulness score, with 4.0 for visualization, implies the concept is accepted but that developers want explanation, such as tooltips and a stated reason for each suggestion.
The same UI can host more powerful selectors, including learned models, without redesigning the workflow.
Because IDF requires no labeled data, the highlighting can be deployed in any system that already collects a large stream of stack traces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test the paper does not run: compare the IDF-highlighted frames against the frames developers manually mark as important; agreement would validate the rarity premise, while its absence would show the tool highlights noise, and the saved manual selections make this comparison inexpensive.
Because IDF is computed over the whole corpus, its suggestions depend on corpus composition: in a system with many duplicate common crashes, a misspelled or one-off frame can look rare for the wrong reason, so per-product or per-component IDF might behave differently.
The survey measures what developers think of the feature, not whether it accelerates triage; a click-through rate on highlighted frames or a time-to-first-analysis measure would test the productivity benefit the feature is meant to provide.
The same rarity idea could be validated externally on public crash-report databases, where manual frame importance could be inferred from fix commits or from clusters of similar reports.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a modest, transparent industrial experience report, and it is exactly as limited as it looks. The new thing is the application of IDF to stack traces to pre-highlight three rare frames, plus a small deployment survey. The writing is honest and the deployment scale (tens of millions of stack traces) is real, but the evaluation only tells you that 18 developers found the UI moderately useful, not that the IDF selection is actually pointing at important frames.

What it does well: the feature is simple, unobtrusive, and integrated into a real production tool. The authors are appropriately careful about the claim ('potentially important'), they report the survey results plainly, and they list sensible future directions (learning, motivating selection). The IDF baseline is a reasonable first cut, and the idea of using the existing manual selections as a ground truth is implicit in their description of the system — which makes its absence from the evaluation more noticeable.

Where it is soft: the central premise — that rare frames are the important ones — is never validated. The tool already lets developers manually mark frames as important, and those marks are saved and shared. The paper could have compared the IDF top-3 against these manual selections on a sample of traces. That comparison is feasible, and without it the 3.6/5 usefulness score is hard to interpret: it could reflect that the highlights are often relevant, or that any unobtrusive nudge feels mildly useful, or simply the novelty of seeing something pre-highlighted. The survey doesn't ask whether the highlighted frames were actually relevant, and one developer's request for the feature to 'motivate its selection' suggests the rationale wasn't self-evident. The absence of a baseline is a real gap for any causal claim, but the paper doesn't make a strong causal claim; it's more a first-step report. Still, as published, a reader could easily over-read the survey as evidence that IDF-based highlighting works.

Bottom line: a short, useful artifact for someone building similar tooling, but not a result that establishes the value of the heuristic. It deserves a fast, serious review — it should be published somewhere as an industrial experience report, with the missing comparison called out as future work or addressed. I'd send it to review rather than desk reject, and I'd tell the authors to add the manual-selection comparison or tighten the language to 'we deployed this and developers gave it a 3.6; we haven't yet shown it helps.'

Referee Report

3 major / 5 minor

Summary. This paper reports the development, integration, and evaluation of a feature in JetBrains' internal bug-processing tool that automatically pre-highlights three stack frames with the highest inverse document frequency (IDF) in a corpus of stack traces. The paper describes the UI, the IDF-based selection heuristic, and a survey of 18 JetBrains developers who rated the feature's usefulness at a mean of 3.6/5 and its visualization at 4.0/5, along with qualitative feedback. The authors state that the feature was bundled into production and is now displayed for all developers at the latest version, and they position the work as an initial step toward more automatic stack-trace analysis.

Significance. If the IDF heuristic actually identifies frames that developers find important, this is a low-cost, scalable addition to crash-triage tooling, and the paper benefits from a real deployment at a large software company with tens of millions of stack traces. The paper is transparent about the feature being an initial version and about the survey being simple. Its concrete strengths are the production integration and the honest reporting of modest, mixed feedback. However, the central premise that rare frames are the important frames is never validated, and the survey does not measure whether the highlighted frames are actually relevant. As it stands, the paper supports a descriptive claim about a deployed feature and its perceived usefulness, not a claim about the correctness or effectiveness of the IDF selection logic. The contribution would be significantly strengthened by comparing the IDF suggestions with the manual frame selections that the tool already records, which is a feasible and natural experiment. The significance of the current version is therefore modest, though the deployment experience may be valuable to practitioners.

major comments (3)

[Section II (Approach)] The IDF selection logic is not validated against any ground truth. The tool already records the frames that developers manually select as important (Section I, Fig. 1b), and the paper does not compare the IDF top-3 suggestions with these human selections. Without such a comparison, the premise that 'the rarest frames might contain the most specific information about a particular stack trace' remains unsupported, and the feature could be highlighting rare but irrelevant frames. A direct precision-at-k or rank-correlation comparison on a sample of real stack traces with saved manual selections is both feasible and necessary to support the claim that the highlighted frames are potentially important.
[Section II (Survey methodology and Results)] The evaluation evidence for usefulness is a self-report Likert rating from 18 developers who were aware that they were rating a new feature, with no control condition, no baseline, and no statistical characterization beyond a mean. The survey asks about usefulness and visualization but does not ask whether the highlighted frames were actually judged relevant, whether they helped locate the bug, or whether they changed the developer's behavior. A mean of 3.6/5 documents a mildly positive opinion, but it does not test whether the highlighted frames are, in fact, important. I recommend reporting score distributions, adding questions about the correctness and helpfulness of the specific highlighted frames, and ideally conducting a within-subject comparison of the same stack traces with and without highlighting. The qualitative comment that developers wanted the feature to 'motivate its selection' further indicates that the IDF rationale was not self-evidently persuasive.
[Title and Abstract] The title and abstract claim that the feature 'find[s] important stack frames,' but the paper provides no evidence that the highlighted frames coincide with frames that developers consider important. The only quantitative result is a usefulness rating, which is consistent with a feature that is visually pleasant but not actually correct. I ask the authors to either soften the claim to 'suggest potentially relevant frames' or provide the validation described in the first major comment. This is a load-bearing issue because the central contribution is precisely the identification of important frames, not merely the addition of a UI affordance.

minor comments (5)

[Section II (Approach)] The paper does not define the exact frame identity used to compute IDF (e.g., full method signature vs. normalized method name, or whether user-specific frames are filtered). A sentence specifying the tokenization and normalization would clarify the heuristic.
[Section II (Survey methodology)] The survey methodology does not state the exact Likert question wording, the response rate, or whether all 18 respondents were among the 25 contacted. The sentence '18 of whom confirmed that they recently used the tool and saw this feature, so we used their responses' could be read as either 18 respondents or 18 confirmed users, and this should be clarified.
[Section II (Results and future work)] The paper reports only mean scores and no distributions, standard deviations, or individual-rating counts. Reporting these would help the reader assess the strength and consistency of the ratings, especially with n=18.
[Section I] The reference list has minor formatting issues, including extra spaces in author names (e.g., 'Y . V asiliev') and inconsistent capitalization in the title of reference [10]. These are simple copyediting fixes.
[Section II (Approach)] The choice k=3 for the number of highlighted frames is presented without any sensitivity analysis or explicit rationale beyond avoiding too many or too few suggestions. A brief justification or a small sensitivity check would strengthen the design.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the IDF highlighting is an unsupervised design choice, not a fitted prediction or a self-citation-derived result.

full rationale

The paper's derivation chain is short and does not contain a circular step. The feature selects the three frames with the highest inverse document frequency in each stack trace, and the authors explicitly present this as a heuristic: 'the rarest frames might contain the most specific information about a particular stack trace.' IDF is defined directly from the corpus of stack traces with no fitted parameters, no outcome labels, and no claimed equivalence between 'important' and 'rarest' as a definition. The top-3 cutoff is a deliberate UI choice, not a fitted value. The survey measures developers' perceived usefulness and visualization convenience, which is an evaluation of an initial product idea, not a prediction derived from the same data used to fit it. The self-citations in the references (e.g., prior work on stack trace deduplication and assignee prediction) are contextual and not load-bearing for the IDF selection rule. Because the paper does not claim to derive importance from first principles or validate the heuristic against manual selections, there is no reduction of a result to its own inputs. The main weakness, that the rarity heuristic itself is untested, is a validity concern, not circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing assumptions are the rare-equals-important heuristic and the validity of self-report ratings as a proxy for usefulness.

free parameters (1)

k (number of highlighted frames) = 3
Chosen by hand to avoid confusing developers with too few or too many suggestions. Not fitted to data, but the usefulness rating is not shown to be robust to this choice.

assumptions (3)

domain assumption Rare frames are more likely to be important for diagnosing a stack trace.
The entire feature is based on this heuristic; Section II states 'the rarest frames might contain the most specific information.' No validation against manual developer selections is provided.
domain assumption Manual frame selection by developers represents ground-truth importance.
The paper treats developers' manual choices in the pre-existing tool as the reference behavior, but it does not measure agreement between IDF suggestions and these manual choices.
domain assumption Self-reported Likert ratings measure actual usefulness.
The evaluation relies on subjective ratings of 18 developers rather than on objective measures such as triage time or fix accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Finding Important Stack Frames in Large Systems." pith.science (2026). https://pith.science/paper/NOJEHTGK

@misc{pith2026250416934,
  author       = {Pith},
  title        = {Pith review of: Finding Important Stack Frames in Large Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOJEHTGK}},
  note         = {Machine review of arXiv:2504.16934}
}

read the original abstract

In this work, we developed, integrated, and tested a feature that automatically highlights potentially important frames in stack traces. The feature was implemented in the internal bug-processing tool at JetBrains that processes tens of millions of stack traces. We surveyed 18 developers at JetBrains who provided valuable feedback on the idea and the implementation.

Figures

Figures reproduced from arXiv: 2504.16934 by the authors.

**Figure 1.** The course of work (a-b) in the pre-existing system without the feature, (c-d) with the proposed feature. The top part shows the initial UI of viewing a stack trace, the bottom — after manually selecting important frames. As for the way to suggest the frames, our initial version uses inverse document frequency (IDF) [10] over the full corpus of received stack traces. In the given stack trace, three frames with the h… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[2]

Dhaliwal, F

T. Dhaliwal, F. Khomh, and Y. Zou, ``Classifying field crash reports for fixing bugs: A case study of Mozilla Firefox ,'' in 2011 27th IEEE International Conference on Software Maintenance (ICSM), 2011, pp. 333--342

work page 2011
[3]

Jalbert and W

N. Jalbert and W. Weimer, ``Automated duplicate detection for bug tracking systems,'' in 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN), 2008, pp. 52--61

work page 2008
[4]

Modani, R

N. Modani, R. Gupta, G. Lohman, T. Syeda-Mahmood, and L. Mignet, ``Automatically identifying known software problems,'' in 2007 IEEE 23rd International Conference on Data Engineering Workshop, 2007, pp. 433--441

work page 2007
[5]

Bartz, J

K. Bartz, J. W. Stokes, J. Platt, R. Kivett, D. Grant, S. Calinoiu, and G. Loihile, ``Finding similar failures using callstack similarity,'' in SysML08: Third Workshop on Tackling Computer Systems Problems with Machine Learning Techniques, 2008

work page 2008
[6]

Aggregation of Stack Trace Similarities for Crash Report Deduplication

N. Karasov, A. Khvorov, R. Vasiliev, Y. Golubev, and T. Bryksin, ``Aggregation of stack trace similarities for crash report deduplication,'' arXiv preprint arXiv:2205.00212, 2022

work page Pith review arXiv 2022
[7]

Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios

E. Shibaev, D. Sushentsev, Y. Golubev, and A. Khvorov, ``Stack trace deduplication: Faster, more accurately, and in more realistic scenarios,'' arXiv preprint arXiv:2412.14802, 2024

work page Pith review arXiv 2024
[8]

(Accessed: 01.12.2024) IntelliJ IDEA

JetBrains. (Accessed: 01.12.2024) IntelliJ IDEA . [Online]. Available: https://www.jetbrains.com/idea/

work page 2024

Show all 12 references

[9]

(Accessed: 01.12.2024) PyCharm

------. (Accessed: 01.12.2024) PyCharm . [Online]. Available: https://www.jetbrains.com/pycharm/

2024
[10]

Sushentsev, A

D. Sushentsev, A. Khvorov, R. Vasiliev, Y. Golubev, and T. Bryksin, `` DapSte p: Deep assignee prediction for stack trace error representation,'' in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 1em plus 0.5em minus 0.4em IEEE, 2...

2022
[11]

Lerch and M

J. Lerch and M. Mezini, ``Finding duplicates of your yet unwritten bug report,'' in 2013 17th European conference on software maintenance and reengineering, 2013, pp. 69--78

2013
[12]

X. Du, M. Liu, J. Li, H. Wang, X. Peng, and Y. Lou, ``Resolving crash bugs via large language models: An empirical study,'' arXiv preprint arXiv:2312.10448, 2023

2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[2] [2]

Dhaliwal, F

T. Dhaliwal, F. Khomh, and Y. Zou, ``Classifying field crash reports for fixing bugs: A case study of Mozilla Firefox ,'' in 2011 27th IEEE International Conference on Software Maintenance (ICSM), 2011, pp. 333--342

work page 2011

[3] [3]

Jalbert and W

N. Jalbert and W. Weimer, ``Automated duplicate detection for bug tracking systems,'' in 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN), 2008, pp. 52--61

work page 2008

[4] [4]

Modani, R

N. Modani, R. Gupta, G. Lohman, T. Syeda-Mahmood, and L. Mignet, ``Automatically identifying known software problems,'' in 2007 IEEE 23rd International Conference on Data Engineering Workshop, 2007, pp. 433--441

work page 2007

[5] [5]

Bartz, J

K. Bartz, J. W. Stokes, J. Platt, R. Kivett, D. Grant, S. Calinoiu, and G. Loihile, ``Finding similar failures using callstack similarity,'' in SysML08: Third Workshop on Tackling Computer Systems Problems with Machine Learning Techniques, 2008

work page 2008

[6] [6]

Aggregation of Stack Trace Similarities for Crash Report Deduplication

N. Karasov, A. Khvorov, R. Vasiliev, Y. Golubev, and T. Bryksin, ``Aggregation of stack trace similarities for crash report deduplication,'' arXiv preprint arXiv:2205.00212, 2022

work page Pith review arXiv 2022

[7] [7]

Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios

E. Shibaev, D. Sushentsev, Y. Golubev, and A. Khvorov, ``Stack trace deduplication: Faster, more accurately, and in more realistic scenarios,'' arXiv preprint arXiv:2412.14802, 2024

work page Pith review arXiv 2024

[8] [8]

(Accessed: 01.12.2024) IntelliJ IDEA

JetBrains. (Accessed: 01.12.2024) IntelliJ IDEA . [Online]. Available: https://www.jetbrains.com/idea/

work page 2024