{"id":"5b5fa2b6-e4f7-473b-8ce0-86d159582077","arxiv_id":"2411.17548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Statistical pruning of trace functions reduces tracing overhead by up to roughly 80% while holding performance model R2 above 0.93 on five benchmark programs.","lead":"The paper shows that tracing only a small set of performance-sensitive functions, found with statistical criteria, can cut tracing overhead by over 80% while keeping machine-learning performance models accurate (R2 up to 0.999) and able to detect injected regressions. This matters because production tracing is usually too expensive to run continuously; a low-overhead method could enable per-commit performance monitoring.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimized trace set is chosen from one hand-tuned input generator and tested on another sample from the same generator; production workloads with a different input distribution can activate pruned functions, so the R2 and regression-detection claims are not yet validated outside the…","rationale":"The reader identified the same load-bearing concern: transferability of performance-sensitivity labels from generated inputs to unseen inputs and production workloads. I agree with that assessment. The paper's central claim, that a small set of traced functions suffices for accurate performance modeling and regression detection in production, requires that functions pruned under the training input distribution remain non-critical under the deployment distribution. The evaluation does not test this because both the 2,500-input analysis set and the 333-input evaluation set are generated by the same hand-tuned generator. The paper deserves credit for acknowledging this in Section 5.3 and for providing a replication package, but the acknowledgement does not resolve the concern. In my view this does not require rejection: the empirical results are credible within the studied distribution, and a conditional acceptance with a required cross-distribution validation is appropriate. Therefore I would not change the reader's CONDITIONAL verdict.","tokens_in":21592,"tokens_out":5288,"duration_ms":63649,"concrete_test":"For one studied program, build the performance-sensitive function set from the 2,500 generated inputs using the criterion that previously gave the best R2. Then evaluate the optimized model on inputs drawn from a genuinely different distribution, e.g., the benchmark's own validation inputs, fuzzing with a different grammar/seed, or real workload traces from a production-like run. Report R2/MAE and the regression-detection rate for regressions injected into functions that were pruned but become active under the new inputs. If R2 drops substantially below the reported 0.931-0.999 range, or if detection of such regressions fails, the transferability claim does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that performance-sensitivity labels learned from 2,500 generated inputs transfer to new inputs and to production workloads. Section 3.1.2 describes a manual, program-specific input generator; Section 3.3.3 then evaluates the optimized models on 333 new inputs sampled from that same generator. Because both sets come from the same distribution, the held-out evaluation cannot detect distribution shift. Section 5.3 explicitly concedes that 'certain functions might exhibit uncovered behavior with unseen inputs,' which can affect which functions are deemed performance-sensitive. Concretely, a function pruned because it has low entropy or low CoV under generated inputs could become performance-critical under a production workload through a different call path, a data-dependent branch, or an input regime the generator never explores. Since optimized tracing discards that function entirely, the resulting model has no feature for its contribution; if that function's behavior correlates with total execution time in the new regime, the model's R2 can degrade and regressions localized there can be missed. This is not an internal formal inconsistency: entropy/CoV are computed on aggregate self-time, so some call-frequency variation is captured. The unresolved part is coverage of the input space, and the paper's production-readiness claim rests on an untested stationarity assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a set of statistical pruning criteria (Shannon entropy, coefficient of variation, correlation-based redundancy removal, feature significance, a union criterion, and a static score) to identify performance-insensitive functions and exclude them from application-level tracing. Performance models are then built from call frequencies of the remaining functions to predict execution time. The evaluation on five C/C++ benchmark programs reports that only 5.64%–21.93% of functions need tracing, that tracing overhead is reduced by more than 80% in most cases, that optimized models reach R2 values up to 0.999 and sometimes outperform full-tracing models, and that model-based regression detection with an injected delay works better than directly comparing execution times.","tokens_in":21940,"tokens_out":8348,"duration_ms":68220,"significance":"If the empirical claims hold, the work is a practical contribution to performance modeling: it attacks a real cost (tracing overhead) and evaluates the payoff in both model accuracy and regression detection. The paper has concrete strengths: a replication package is announced, the study uses diverse benchmark programs of different sizes, optimized models are compared against a full-tracing baseline, and regression detection is compared against a direct execution-time baseline. It is also fair to say that the central modeling claim is not circular by construction: optimized models are tested on held-out inputs, not on the training inputs. However, the strength of the contribution is limited by the absence of uncertainty quantification, by model/criterion selection on the test data, and by the fact that the optimized tracing set is selected and evaluated on inputs from the same hand-built generator, leaving the production-readiness claim undemonstrated.","major_comments":[{"comment":"The central quantitative claims are reported as single point estimates with no repeated runs, confidence intervals, or standard deviations. Execution-time overhead in Table 3 varies enormously (e.g., 77.1% to 2243.9% for full tracing), and the claim that tracing overhead is reduced by more than 80% in most cases is based on average values whose variability is unknown. I ask for repeated measurements or, at minimum, confidence intervals and statistical tests before the overhead-reduction claim can be assessed.","section":"§4.2–§4.3, Tables 3–4"},{"comment":"The performance-sensitive functions are identified from 2,500 inputs generated by a manual, program-specific generator, and the optimized models are evaluated on 333 new inputs sampled from the same generator. Section 5.3 explicitly concedes that 'certain functions might exhibit uncovered behavior with unseen inputs.' This is precisely the load-bearing limitation: a production workload with a different input distribution can activate a pruned function through a different call path, and the optimized model would then have no feature for that function. The abstract’s production-readiness claim is therefore not supported by the current evidence. A concrete remedy is to evaluate on held-out input regimes, fuzzed inputs, or production traces, and to temper the production claim accordingly.","section":"§3.1.2, §3.3.3, §5.3"},{"comment":"For each program, the 'best' pruning criterion and the 'best' machine-learning model are selected from a large pool (10 criteria and multiple model types) using the same held-out test set. Reporting the best R2 after such selection, without nested cross-validation or a separate validation set, overstates the expected performance and can explain the cases where optimized models 'sometimes outperform' full tracing. Please specify the model-selection protocol more precisely and evaluate the final selected configuration on an independent test set.","section":"§3.3.1, §4.3, Table 4"},{"comment":"The pipeline depends on several ad hoc thresholds: the mean ± 3*std outlier rule, the top-99th-percentile unique-call-frequency filter, the p<0.05 feature-significance cutoff, the Spearman ρ≥0.7 correlation-removal threshold, and the loess-derivative/Ckmeans change-point split. No sensitivity analysis is reported for these choices. Since the method is described as fully automated and the results are used to justify pruning, I ask for a sensitivity analysis over reasonable ranges of these parameters, or at least a discussion of which thresholds are load-bearing.","section":"§3.2.1–§3.2.3"}],"minor_comments":[{"comment":"Several cross-references appear to be wrong: 'see Section 3.4.3' for input generation should refer to Section 3.1.2, and the replication-package references to Section 3.4.3 should be corrected.","section":"§4.1"},{"comment":"The column layout of Table 2 is unclear in the submitted text, and some reported counts are inconsistent with the stated semantics: for example, the 'All' column for SU2 reports 9 functions while its constituent criteria (Entropy: 42, Performance Correlations: 48) are much larger. Please clarify the column definitions and verify the values.","section":"Table 2"},{"comment":"The optimized-model row for freqmine reports counts out of 14 versions while all other rows use 15; please explain the missing version or correct the denominator.","section":"Table 5"},{"comment":"The statement that overhead reduction 'remains consistent across all tools' is not supported by any experiment with alternative tracers; it should be removed or qualified.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"I see the contribution as potentially publishable after revision, but the lack of uncertainty quantification and the same-distribution evaluation of the pruning criteria are real gaps. The Table 2 inconsistency should also be checked before camera-ready; if the 'All' column is meant to be a union, the numbers as printed are internally contradictory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is empirical: it systematically compares ten criteria (nine dynamic, one static) for deciding which functions to keep under tracing, on five C/C++ benchmarks, and shows that a small subset can yield models with R2 above 0.93 and often close to 0.99. The overhead reductions are concrete and measured, including storage and execution time. The replication package is a genuine plus. This is an extension of Mußler and Lehr's instrumentation refinement, and the paper says so honestly; the novelty is in the breadth of criteria, the automated thresholding via loess derivatives and one-dimensional clustering, and the regression-detection evaluation.\n\nThe central claim holds up for the narrow setting tested: when training and test inputs come from the same generator, optimized tracing matches or beats full tracing. The held-out 333 inputs are drawn from the same hand-tuned generator as the 2,500 used to rank functions, so the evaluation cannot detect distribution shift. Section 5.3 candidly admits that unseen inputs can change which functions are performance-sensitive, and that concession undercuts the abstract's \"ready to be used in production\" claim. That is the biggest soft spot. A function pruned because it is stable under generated inputs could become critical under a production workload via a different call path, and the optimized model would have no feature for it.\n\nThere are smaller issues. The best pruning criterion and the best ML model are selected per program using the same test data, without multiple-comparison correction, so the reported R2 values are optimistic. There are no error bars or repeated runs. Several preprocessing thresholds (mean±3std, top-99th-percentile unique call counts, correlation ρ≥0.7, p<0.05) are ad hoc. Some criteria produce empty function sets for some programs (StaPerfSens for SU2, Feature Significance for 657.xz), which the paper reports but does not really discuss. Regression detection is tested with only one injected regression type (a time delay), selected by call-frequency clusters, so it is a proof of concept rather than a broad validation.\n\nNone of these are fatal; they are the usual gap between a controlled benchmark study and a production tool. The paper deserves a serious referee and likely conditional acceptance after sensitivity analyses: error bars or repeated runs, a held-out input distribution that differs from training (e.g., fuzzing or a different workload), correction for selection, and a toned-down production claim.\n\nFor a reading group, this is a useful example of how to evaluate tracing-overhead reduction empirically, with a clear methodology and honest limitations. I would cite it if working on tracing or performance regression detection, though I would not rely on the production-readiness conclusion.","headline":"Solid empirical extension of prior instrumentation-refinement work, with a useful systematic comparison of pruning criteria, but the production-readiness claim goes beyond what the same-generator evaluation can support.","tokens_in":118,"tokens_out":1647,"would_cite":true,"duration_ms":36265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that tracing only the few performance-sensitive functions is enough to build accurate performance models and detect regressions, cutting overhead by more than 80% in most cases.","keywords":["tracing optimization","performance modeling","performance regression detection","function pruning","Shannon entropy","coefficient of variation","black-box models","tracing overhead"],"falsifier":"Run the optimized and full-tracing models on a workload from a different distribution than the training generator—e.g., production traffic logs—and check whether a pruned function dominates runtime while the optimized model's R² drops sharply relative to full tracing.","tokens_in":21435,"feed_emoji":"📉","tokens_out":6072,"duration_ms":63223,"temperature":0.7,"pith_summary":"The paper claims that most traced functions carry little performance information, so they can be omitted from tracing without hurting—and sometimes improving—the resulting performance models. Across five C/C++ benchmark programs, only 5.64% to 21.93% of functions were identified as performance-sensitive, and optimized models reached R² scores from 0.931 to 0.999, often matching or beating full-tracing baselines. Tracing overhead fell by more than 80% in most cases. The same optimized models detected injected performance regressions more reliably than directly comparing execution times, supporting use in production settings where tracing costs matter.","feed_headline":"Only 5–22% of functions need tracing for accurate models","feed_subtitle":"Optimized tracing keeps performance models accurate while cutting overhead by more than 80%.","key_machinery":"The central mechanism is the classification of functions into performance-sensitive and performance-insensitive sets using dynamic statistical criteria (Shannon entropy, coefficient of variation, Spearman correlation removal, and linear-regression feature significance) and a static score (StaPerfSens). These criteria rank functions, and an automatic threshold from one-dimensional clustering (Ckmeans.1d.dp) plus derivative analysis splits them. The selected functions' call frequencies then feed into AutoML-tuned regressors (e.g., CatBoost, BayesianRidge), and model-prediction differences between program versions serve as regression detectors.","core_discovery":"The paper establishes that a small subset of functions—those whose execution time varies with input and contributes to the model—sufficiently explains program execution time. Using call frequencies of only these functions as features, black-box regressors predict execution time with high accuracy, and the pruned models sometimes outperform full-tracing models because removing correlated functions reduces noise. The paper further shows that comparing predictions from old and new version models, via the Mann–Whitney U test and Cliff's delta, can flag regressions even when direct execution-time comparison fails.","pith_inferences":["The sensitivity labels depend on the input distribution used during training; production workloads that exercise pruned functions through unseen call paths could invalidate the model, so periodic re-identification or coverage-guided input generation would be a natural safeguard.","The same pruning logic could extend beyond function calls to kernel events or I/O operations, potentially yielding similar overhead reductions in other tracing contexts.","Comparing model predictions between versions could be adapted into a continuous online anomaly detector using sliding windows rather than release-level snapshots."],"forward_implications":["Optimized tracing can run at commit level or in production, where full tracing is too expensive.","Model accuracy does not require more trace data; pruning can remove correlated noise and improve predictions.","The best pruning criterion is program-specific, so the framework must select the criterion and model type per program.","Regression detection works by statistically comparing the old and new versions' model predictions, catching regressions that raw execution time comparisons miss."],"supporting_citations":[{"why":"Mußler et al. 2011 – static exclusion of less important functions from instrumentation, the basis for the static StaPerfSens approach.","marker":"[45]"},{"why":"Lehr et al. 2018 – instrumentation refinement via static/dynamic analyses that the dynamic pruning criteria extend.","marker":"[42]"},{"why":"Liao et al. 2020 – black-box performance models and the Mann–Whitney U / Cliff's delta methodology used for regression detection.","marker":"[43]"},{"why":"Ardelean et al. 2018 – tracing-based cloud performance analysis showing that not all code regions matter for performance.","marker":"[6]"},{"why":"uftrace tracer used to collect function-level trace data in all experiments.","marker":"[39]"},{"why":"Ckmeans.1d.dp one-dimensional clustering used to automate the threshold between sensitive and insensitive functions.","marker":"[57]"},{"why":"PyCaret AutoML library used to train, tune, and select the machine learning performance models.","marker":"[5]"}],"fun_headline_variants":["Trace only 5-22% of functions, cut overhead by 80%","Pruned tracing matches full-tracing model accuracy","Automated function selection reduces tracing cost","With 5-22% of traces, models hit 99% R²","Slash tracing overhead 80% while keeping regressions visible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance-sensitivity labels learned from 2,500 generated inputs carry over to new inputs and production workloads; if a pruned function becomes performance-critical under an unseen input or call path, the optimized model will miss the regression.","fun_headline_variants_meta":{"raw":{"variants":["Trace only 5-22% of functions, cut overhead by 80%","Pruned tracing matches full-tracing model accuracy","Automated function selection reduces tracing cost","With 5-22% of traces, models hit 99% R²","Slash tracing overhead 80% while keeping regressions visible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1231,"prompt_tokens":928,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":544,"tokens_out":303,"duration_ms":21395,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:01:25.031218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the optimized and full-tracing models on a workload from a different distribution than the training generator—e.g., production traffic logs—and check whether a pruned function dominates runtime while the optimized model's R² drops sharply relative to full tracing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mußler et al. 2011 – static exclusion of less important functions from instrumentation, the basis for the static StaPerfSens approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lehr et al. 2018 – instrumentation refinement via static/dynamic analyses that the dynamic pruning criteria extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Liao et al. 2020 – black-box performance models and the Mann–Whitney U / Cliff's delta methodology used for regression detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ardelean et al. 2018 – tracing-based cloud performance analysis showing that not all code regions matter for performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"uftrace tracer used to collect function-level trace data in all experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PyCaret AutoML library used to train, tune, and select the machine learning performance models."}],"review_version":1}