{"id":"1edfdec2-6240-475b-a9ff-f0c34bd572b4","arxiv_id":"2506.15523","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Atys combines language-aware sampling, thread-level pruning, and adaptive sampling frequency to profile hotspot functions in large-scale cloud microservices at a fraction of the cost.","lead":"Atys is a distributed profiling framework that finds the busiest functions in cloud microservices by sampling call stacks with language-specific tools, then pruning inactive threads and adjusting sampling rates. It claims large cost savings in profiling overhead while keeping accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.3's fitted FSP models contradict the reported results: at P99 pruning they give ~43% MAPE and ~50% time reduction, not 0.58% and 6.8%.","rationale":"The central quantitative claim is FSP's 6.8% time reduction at 0.58% MAPE under P99 pruning. The paper's own Section 5.3 supplies a model that should predict this operating point. It does not: direct substitution gives roughly 50% time reduction and 43% MAPE at the P99 threshold. This is not a statistical concern about generality; it is an internal arithmetic contradiction in the headline number. The reader correctly flagged in-sample fitting, missing figures, and lack of error bars, but the most load-bearing issue is narrower and testable: the fitted model and the reported result cannot both be correct. I also note that the FDA claim has metric inconsistencies (abstract says MSE, Section 6.4 evaluates MAE; Section 1 reports an average sampling rate of 3,578 Hz while Section 6.4 reports 357.8 Hz), but the FSP contradiction is more direct because it undermines the first headline claim without relying on external assumptions. I maintain the conditional verdict: the framework is plausible, but the authors must fix the FSP model or clarify the definition of p before the 6.8%/0.58% claim can be accepted.","tokens_in":15935,"tokens_out":9725,"duration_ms":90798,"concrete_test":"Recover the SPECjbb2015 per-thread sample data and raw stack traces used for Figure 4 and Figure 6 from the Atys repository (https://github.com/ottoSJTU/Atys). Recompute the actual aggregation time and the top-50 MAPE at the P99 pruning threshold p = (1840 − 838)/1840 = 0.545 and at p = 0; then evaluate the paper's fitted models T(p) = −1.0614p + 114.44 and MAPE(p) = 984.368·ln(1.099 − 0.001p) at p = 54.5. If the measured reduction and MAPE are 6.8% and 0.58%, the fitted equations in Section 5.3 must be corrected; if the equations reproduce the reported numbers, report the correct definition of p. This single recomputation decides whether the current FSP headline is supportable.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.3 fits T(p) and MAPE(p) on the SPECjbb2015 run and uses them in the optimization (Eq. 1) to choose the pruning level. The reported FSP claim is the P99 configuration: retaining the top 838 of 1,840 threads, i.e. pruning p = (1,840 − 838)/1,840 = 54.5% of threads, yields a 6.8% aggregation-time reduction and 0.58% MAPE over the top-50 functions. Substituting p = 54.5 into the paper's own fits gives T(54.5) = −1.0614·54.5 + 114.44 ≈ 56.6, which is a 50.5% reduction from the p = 0 baseline of 114.44, not 6.8%; and MAPE(54.5) = 984.368·ln(1.099 − 0.001·54.5) ≈ 42.9%, not 0.58%. Even if the logarithm is base 10, the model gives about 18.6%, still far from 0.58%. Solving the stated constraint MAPE(p) ≤ 0.58 with this model gives p ≥ 98.4%, i.e. pruning at least 98.4% of threads would be needed to obtain the claimed error, incompatible with the P99 threshold. Thus the headline quantitative claim is not reproducible from the model presented in the paper. The discrepancy could stem from a typo in the fitted coefficients, a different definition of p (e.g., percent of samples retained), or a misreported P99 error, but any of these means the FSP evaluation is internally inconsistent as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Atys, a distributed profiling framework for large-scale cloud microservices. Atys combines a language-agnostic adaptation mechanism (async-profiler for Java, py-spy for Python, Perf for compiled languages), a two-level flamegraph aggregation method, a function selective pruning (FSP) strategy that retains only high-sample-count threads to reduce aggregation cost, and a frequency dynamic adjustment (FDA) scheme that adapts the sampling rate based on JS divergence between consecutive hotspot-function distributions. The evaluation on a 13-node cluster with SPECjbb2015 and VGG16 reports that FSP reduces aggregation time by 6.8% with 0.58% MAPE for the top 50 hotspot functions, and that FDA achieves MSE comparable to 10 kHz sampling at 12.4% of the cost (an 87.6% reduction). Scalability up to 1,000 monitored instances is claimed with about 175 MB memory and 6% CPU on the Prometheus server.","tokens_in":16358,"tokens_out":6833,"duration_ms":62583,"significance":"If substantiated, Atys would be a useful practical contribution: the architecture is clean, the workloads are realistic, and the open-source prototype is a reproducible artifact. The two key ideas—pruning uninformative daemon threads and adapting sampling frequency to workload periodicity—are sensible and could reduce profiling overhead in production. However, as written the paper's central quantitative claims are not internally consistent or fully evidenced: the FSP model in Section 5.3 contradicts the reported P99 results, the evaluation is in-sample and single-run, FSP is evaluated on only one benchmark despite the abstract's claim of two, the FDA sampling-rate figures disagree by an order of magnitude, and several supporting figures are missing placeholders. These issues must be resolved before the contributions can be accepted.","major_comments":[{"comment":"The fitted FSP models contradict the reported headline result. Section 5.3 defines p as the percentage of pruned threads and fits T(p) = -1.0614p + 114.44 and MAPE(p) = 984.368 * log(-0.001p + 1.099). The P99 configuration used in Section 6.3 prunes 1,002 of 1,840 threads, i.e. p = 54.5. Substituting p = 54.5 into the paper's own model gives T = 114.44 - 1.0614*54.5 ≈ 56.6, a 50.5% reduction from the p=0 baseline, not the claimed 6.8%; and MAPE ≈ 42.9% (natural log) or 18.6% (log base 10), not 0.58%. Solving the stated constraint MAPE(p) ≤ 0.58 yields p ≥ 98.4%, implying that 98.4% of threads would need to be pruned to achieve the claimed error, which is incompatible with the P99 threshold. The statement that T(p) and MAPE(p) are 'both monotonically increasing' is also inconsistent with the defined T(p), which decreases with p. The FSP claim is therefore not reproducible from the models presented in the paper, and a corrected model, a corrected definition of p, or a corrected error figure is required.","section":"Section 5.3, Eq. (1); Section 6.3, Figure 4"},{"comment":"The evaluation is in-sample for both FSP and FDA. The T(p) and MAPE(p) models in Section 5.3 are fitted to the same SPECjbb2015 run that is then used in Section 6.3 to report the 0.58% MAPE and 6.8% time reduction. Similarly, the FDA thresholds θ_specjbb = 0.5, θ_vgg = 0.05, and λ = 0.8 are selected after pre-profiling the same two workloads, and the accuracy/cost numbers in Section 6.4 are reported on those same workloads. There is no held-out workload, no cross-validation, and no discussion of how the reported error would change for a different service. The paper claims general applicability for large-scale microservices, but the evidence is consistent only with per-deployment tuning and in-sample estimation, which generally yields optimistic accuracy. Please provide out-of-sample results or clearly reframe the claims as per-deployment tuning demonstrations.","section":"Section 5.3, 6.3, 6.4"},{"comment":"The abstract states that 'Cluster-scale experiments on two benchmarks show that the FSP strategy achieves a 6.8% reduction in time with a mere 0.58% MAPE', but the FSP evaluation in Section 6.3 is performed only on SPECjbb2015 ('we employ the function selective pruning strategy ... on the stack traces gathered from a specific execution of Specjbb2015'). No FSP results are reported for VGG16 or any other workload. The cross-benchmark claim in the abstract is therefore unsupported; please either report FSP results for the second benchmark or revise the abstract and introduction to attribute the FSP result to SPECjbb2015 alone.","section":"Abstract; Section 6.3"},{"comment":"The FDA sampling-rate figures are inconsistent by an order of magnitude. The introduction states that 'the FDA scheme, averaging a sampling rate of 3,578 Hz, yields results with a mean squared error (MSE) comparable to a 10,000 Hz sampling rate, at only 12.4% of the cost.' Section 6.4, however, reports that 'Atys has an average sampling frequency of 357.8Hz' for the Java benchmark, while also mentioning a comparison at '5,00Hz' (presumably 500 Hz). Since the 12.4% cost figure and the comparison to a 10 kHz baseline depend on which sampling rate is correct, the FDA cost/accuracy tradeoff cannot be evaluated as stated. Please correct the number and ensure the abstract, introduction, and evaluation are mutually consistent.","section":"Introduction vs. Section 6.4"},{"comment":"The manuscript contains unresolved figure placeholders for load-bearing results. Section 4.2 references 'Figure ?? and Figure ??' for the RandomGen flamegraph aggregation example and 'Figure ??' for the aggregated flamegraph, and Section 6.5 references 'Figure ??' for memory usage and 'Figure ??' for CPU usage; Figures 3(a)–3(c) and 8(a)–8(b) are shown in the text but the in-text citations are missing. The scalability claims of 175 MB memory and 6% CPU at 1,000 instances are presented without an associated readable figure or table, and the two-level aggregation example cannot be checked. Please fix all figure references and, ideally, provide numerical data for the scalability experiment.","section":"Section 4.2, Section 6.5, Figures 3 and 8"}],"minor_comments":[{"comment":"The text 'CPU usage comparable to async-profiler at 5,00Hz' appears to be a typo for '500 Hz'; please correct it.","section":"Section 6.4"},{"comment":"The phrases 'fitting MAPE=4.75' and 'fitting MAPE=15.73' are unclear; please define what these numbers represent (e.g., residual fit error, model MAPE, or something else).","section":"Section 5.3"},{"comment":"The abstract refers to 'MAPE in stack traces aggregation' while Section 6.3 specifies MAPE for the top 5, 10, 20, and 50 hotspot functions; the abstract should match the actual metric definition.","section":"Abstract and Section 6.3"},{"comment":"The sentence 'Since T(p) and MAPE(p) are both monotonically increasing' is inconsistent with the fitted form of T(p), which is decreasing; please correct the monotonicity statement or redefine T(p) to represent time saved rather than aggregation time.","section":"Section 5.3"},{"comment":"The quantitative claims are based on what appears to be a single run per configuration; reporting multiple runs with variance (e.g., mean and standard deviation) would substantially strengthen the credibility of the reported improvements.","section":"Throughout evaluation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of cs.PF and has a useful prototype, but the internal inconsistency between the fitted FSP model in Section 5.3 and the reported results in Section 6.3 is a load-bearing correctness issue that cannot be fixed by a simple edit; it requires either re-fitting the model, redefining p, or re-reporting the measured values. The in-sample tuning and the missing figure references also weaken the evaluation substantially. I recommend major revision, not rejection, because the core ideas and the prototype are plausible and the technical issues appear fixable within the manuscript's scope. The authors should also be asked to clarify whether the P99 pruning rule is intended as a universal default or as an example of per-service tuning, since Section 5.1 and Section 5.3 give conflicting indications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kai,\n\nQuick take on arXiv:2506.15523. The system Atys has a sensible combination of known ideas: language-agnostic kernel selection, two-level flamegraph aggregation, thread-level pruning, and JS-divergence-based adaptive sampling. That particular package is new as far as I know, and the authors have shipped code. The engineering is plausible and the paper is clearly written enough to follow.\n\nThe problem is the evaluation, and it is not cosmetic. The stress-test arithmetic checks out: for the SPECjbb run, P99 means pruning about 54.5% of threads (1002 of 1840). Plug that into their own fitted MAPE(p) in Section 5.3, and you get roughly 43% (natural log) or 19% (base-10 log), not the 0.58% claimed in Section 6.3. Their T(p) also predicts roughly 50% aggregation-time reduction, not 6.8%. The authors even note the model gets worse near P99, but the magnitude is off by two orders. Either the coefficients are wrong, the definition of p is different, or the P99 numbers are misreported. As written, the headline FSP claim is not reproducible from the paper's own equations.\n\nThe other soft spots are more ordinary: Figure 3 and Figure 8 are placeholders, there are no error bars or repeated runs, the abstract says 3,578 Hz average sampling while the body says 357.8 Hz, and the abstract talks about MSE while the body uses MAE. Parameters like theta and lambda are tuned on the same two workloads used for the final evaluation, so the accuracy numbers are in-sample. None of these are fatal by themselves, but taken with the FSP contradiction they mean the central quantitative claims are not yet supported.\n\nWhat I do not want to lose: the design is reasonable, and the idea of pruning at thread level rather than stack level is worth thinking about. The JS-divergence scheme is not revolutionary but it is a sensible adaptive rule. The paper would be a good candidate for a workshop or a major-revision round if the authors can clear up the model contradiction, rerun with held-out workloads, and provide the missing figures. Right now I would be reluctant to rely on any of the specific performance numbers.\n\nI would send it to review, because the ideas deserve careful referee attention and the flaws look fixable. But I would not cite it yet.\n\nBest,","headline":"A plausible microservice profiling system whose headline numbers are undermined by an internal inconsistency in the FSP model.","tokens_in":16866,"tokens_out":5276,"would_cite":false,"duration_ms":45078,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Atys shows that hotspot profiling of cloud microservices can be made roughly 87.6% cheaper by pruning uninformative threads and adapting sampling frequency, while keeping hotspot-function error near 0.58%.","keywords":["performance profiling","hotspot functions","cloud microservices","sampling frequency adaptation","thread-level pruning","flamegraph aggregation","distributed systems","profiling cost reduction"],"falsifier":"Run Atys on a long-lived production microservice whose request handling spawns short-lived threads, collect the full trace at a high rate, then re-rank the top 50 functions after P99 pruning; if the MAPE on function CPU time consistently exceeds the paper's 0.58%, the concentration assumption fails. A more direct check is to measure the cumulative sample share of the bottom pruned threads during a burst of rare, expensive requests and see whether it exceeds 1% then.","tokens_in":15755,"feed_emoji":"⚡","tokens_out":7670,"duration_ms":70633,"temperature":0.7,"pith_summary":"This paper sets out to make hotspot-function profiling practical for large-scale cloud microservices, where thousands of instances make per-instance high-rate sampling too expensive. It proposes Atys, a distributed profiling framework with four components: a language-agnostic kernel selector, two-level flamegraph aggregation, a function selective pruning (FSP) strategy, and a frequency dynamic adjustment (FDA) scheme. The core quantitative claim is that the information needed to rank hotspot functions is concentrated in a few threads and a few time windows: keeping only P99 threads cuts aggregation time by 6.8% with a 0.58% mean absolute percentage error for the top 50 functions, and adapting sampling frequency cuts cost by 87.6% while keeping mean squared error on par with high-rate sampling. If these numbers generalize beyond the two evaluated workloads, continuous profiling of production fleets becomes affordable, and profiling overhead no longer scales linearly with instance count.","feed_headline":"Atys cuts profiling cost by 87.6% with tiny accuracy loss","feed_subtitle":"P99 thread pruning and adaptive sampling keep top-50 hotspot error at 0.58% MAPE.","key_machinery":"The load-bearing object is the P99 thread-pruning rule together with a Jensen–Shannon divergence gated frequency controller. P99 pruning treats a thread's sample count as a proxy for its informative worth, discarding all threads outside the smallest set that accounts for 99% of samples; because the number of threads (about 1,840 in the motivating run) is far smaller than the number of stack traces, sorting and cutting threads is cheap, and the paper fits simple models T(p) and MAPE(p) to choose the pruning percentage under a user-entered error bound. The FDA controller represents the top 10 hotspot functions' CPU-time shares as a probability distribution, compares consecutive 10-second windows with JS divergence, multiplies frequency by λ when the divergence is below the threshold θ for several windows, and divides by λ when it is above; this turns high-rate reference samples into anchors that substitute for low-rate samples during stable phases.","core_discovery":"The paper's central claim is that the call-stack data needed for hotspot-function identification in large distributed services can be drastically reduced without moving the answer. Atys prunes stack traces by thread: threads are ranked by sample count and all but the P99 threads—the smallest set covering 99% of samples—are dropped before aggregation; the remaining traces are aggregated at two levels so users see service-wide flamegraphs instead of per-instance snapshots. It then makes sampling frequency self-adjusting: the CPU-time distribution of the top ten hotspot functions is monitored across consecutive time windows, and the Jensen–Shannon divergence between consecutive distributions triggers exponential frequency increases or decreases. On the paper's cluster experiments, these two mechanisms yield a 6.8% aggregation-time reduction at 0.58% MAPE for the top 50 functions, and an 87.6% sampling-cost reduction at accuracy comparable to a 10,000 Hz sampling rate; the central collector stays near 175 MB memory and below 6% CPU when monitoring 1,000 instances.","pith_inferences":["The paper demonstrates the P99 concentration on two long-running workloads; a natural extension is to test whether the same concentration holds for short-lived serverless or request-per-thread microservices, where threads are born and die too quickly for the ranking to stabilize.","The FDA controller's reliance on JS divergence between consecutive windows assumes periodicity; for non-periodic bursty workloads the same anchor-and-substitute idea could be made reactive to request-rate signals rather than to the sampling history itself.","The paper leaves the threshold θ, decay λ, window length, and the number of top functions as user parameters; a natural follow-up is to auto-tune them from the fitted T(p)/MAPE(p) models, making the whole system parameter-free.","If the pruning is performed at each aggregation level rather than only at local profilers, the aggregation time savings would compound with the level of the hierarchy; the paper's two-level design suggests but does not measure this compounding."],"forward_implications":["With FDA's cost reduction, continuous profiling can run by default on long-lived services instead of being started only for targeted investigations, since the average sampling rate and overhead drop by roughly an order of magnitude.","FSP's thread-level pruning is a pre-aggregation step, so it can be applied to existing sampling profilers and monitoring pipelines without changing their kernels; any profiler that exposes per-thread stack traces could use it.","Two-level aggregation makes service-level flamegraphs the primary view, removing per-instance variance and cutting the data moved to the central collector by about 99%; multi-level grouping extends this to fleets beyond 1,000 instances.","The reported memory (around 175 MB for 1,000 monitored instances) and CPU (below 6%) footprints imply one central node can support a large production fleet, so profiling capacity no longer dominates deployment cost."],"supporting_citations":[{"why":"Supplies the Java benchmark whose per-thread sample distribution motivates the P99 pruning rule and the fitted cost/error models.","marker":"[13]"},{"why":"Supplies the Python image-classification workload; its periodic hotspot pattern drives the frequency adjustment experiments.","marker":"[14]"},{"why":"Provides the Java profiling kernel Atys selects for interpreted-language services; FDA accuracy is evaluated against it.","marker":"[23]"},{"why":"Provides the Python profiling kernel used by Atys and compared with baselines on error and cost.","marker":"[47]"},{"why":"Serves as the baseline multi-machine profiler whose unoptimized kernel deployment is the comparison point.","marker":"[11]"},{"why":"Provides the monitoring and time-series storage layer, so the scalability measurements are made on its server.","marker":"[31]"},{"why":"Supports the observation that microservice workloads show periodic variation, which the FDA scheme exploits.","marker":"[49]"},{"why":"Defines the divergence measure used to compare hotspot-function distributions across consecutive time windows.","marker":"[52]"},{"why":"Supplies the flamegraph representation that the two-level aggregation turns into a service-wide view.","marker":"[12]"},{"why":"Documents the production scale (hundreds of thousands of instances per service) that sets Atys's target workload.","marker":"[3]"}],"fun_headline_variants":["Atys cuts profiling cost 87.6% with only 0.58% error","P99 thread pruning and adaptive sampling cut profiling cost 87.6%","Find hotspot functions with 87.6% less sampling and 0.58% error","Atys: 87.6% cheaper profiling via P99 pruning and adaptive sampling","Adaptive frequency and P99 pruning reduce profiling cost 87.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that in any large cloud service a tiny fraction of threads accounts for almost all profiling samples, so the P99 rule can discard the rest without moving the hotspot ranking; if a real service spreads samples evenly across many short-lived threads, pruning would lose informative frames and the reported 0.58% error would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Atys cuts profiling cost 87.6% with only 0.58% error","P99 thread pruning and adaptive sampling cut profiling cost 87.6%","Find hotspot functions with 87.6% less sampling and 0.58% error","Atys: 87.6% cheaper profiling via P99 pruning and adaptive sampling","Adaptive frequency and P99 pruning reduce profiling cost 87.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1843,"prompt_tokens":989,"completion_tokens":854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":745}},"tokens_in":605,"tokens_out":854,"duration_ms":7497,"temperature":1.0,"reasoning_tokens":745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:32:30.002202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Atys on a long-lived production microservice whose request handling spawns short-lived threads, collect the full trace at a high rate, then re-rank the top 50 functions after P99 pruning; if the MAPE on function CPU time consistently exceeds the paper's 0.58%, the concentration assumption fails. A more direct check is to measure the cumulative sample share of the bottom pruned threads during a burst of rare, expensive requests and see whether it exceeds 1% then.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Java benchmark whose per-thread sample distribution motivates the P99 pruning rule and the fitted cost/error models."},{"cited_title":"al., async-profiler, https://github.com/async-profiler/async-profiler (2017)","cited_arxiv_id":null,"evidence_quote":"Provides the Java profiling kernel Atys selects for interpreted-language services; FDA accuracy is evaluated against it."},{"cited_title":"al., py-spy, https://github.com/benfred/py-spy(2023)","cited_arxiv_id":null,"evidence_quote":"Provides the Python profiling kernel used by Atys and compared with baselines on error and cost."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the baseline multi-machine profiler whose unoptimized kernel deployment is the comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the monitoring and time-series storage layer, so the scalability measurements are made on its server."},{"cited_title":"Gregg, Flame graphs, https://www.brendangregg.com/flamegraphs.html (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the flamegraph representation that the two-level aggregation turns into a service-wide view."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the production scale (hundreds of thousands of instances per service) that sets Atys's target workload."}],"review_version":2}