{"id":"345e9618-aae3-408a-bf10-064c5be87e3b","arxiv_id":"1908.09048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Griffin uses an interpretable random forest to attribute a slow job's runtime deviation to individual metrics and rank the likely causes without labeled data.","lead":"Microsoft's Griffin system automatically ranks likely causes of slowdowns in analytics jobs by training an interpretable random forest to predict job runtime without labeled incidents. The paper reports a production deployment and says suggested causes mostly match domain-expert-validated causes, but the evidence is based on about seven incidents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Griffin's core claim that TreeInterpreter delta contributions rank actual slowdown causes is unvalidated: with correlated features the attribution is non-unique and no baseline comparison is reported.","rationale":"I read the paper in good faith. The regression framing is sensible, the random forest is a reasonable choice for runtime prediction, and the TreeInterpreter identity in Eq. 3 is mathematically correct for the trained model. However, the system's advertised value is not runtime prediction but root-cause reasoning. The leap from 'the model's prediction difference decomposes into feature contributions' to 'these contributions rank the causes of the slowdown' is exactly the weakest link. The paper itself acknowledges high feature correlations (up to 0.95, Section 5.2) and claims tree-based models are robust to them, but robustness for prediction does not imply uniqueness or causal validity for attribution. The validation in Section 7.1 is anecdotal: seven to eight incidents, no quantitative agreement score, no comparison against even simple baselines, and the paper reports no confidence thresholds t1/t2/p or feature list. The Auto dataset example in Section 7.5 is illustrative but not evidence for causal correctness. These are not fatal flaws; the deployment and early indications are real evidence of usefulness. But the central claim, as stated in the Abstract, is stronger than the support. My concern matches the reader's identified weakest assumption. The reader's CONDITIONAL verdict already captures this: the approach is plausible and internally consistent, but the empirical support for causal ranking is thin. I therefore see no reason to change that verdict. The proposed test—comparing against SHAP or a linear surrogate—is a tractable way to check whether the specific attribution method matters, which would settle whether the concern actually lands. If Griffin's ranking is not robust to the attribution choice, the paper's claim to discover actual causes would need to be softened to 'suggest plausible features consistent with the model's predictive logic.'","tokens_in":14266,"tokens_out":2940,"duration_ms":31340,"concrete_test":"Re-analyze the incidents in Table 2 (or a larger held-out set) by computing feature attribution with an alternative method, e.g., SHAP (TreeSHAP) on the same random forest and a linear regression on standardized features, and compare top-1 and top-5 agreement with the engineer-validated reasons. If the alternative attributions achieve equal or better agreement than Griffin's TreeInterpreter ranking, then the specific decomposition is not what makes Griffin succeed, and the causal ranking claim is unsupported. If Griffin's ranking is stable across all attribution methods on these incidents, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The system's central claim (Abstract, Section 7.1) is that the relative feature contributions in an interpretable regression model identify and rank the causes of job slowdowns. This requires that the delta feature contributions Δf_ck in Eq. 3 correspond to the causal influence of each feature on the runtime deviation. The paper establishes only an algebraic identity: for the trained random forest, the difference between predictions decomposes exactly into per-feature TreeInterpreter contributions (Eq. 9). But TreeInterpreter attributes predictive signal along the decision paths; it is not a causal estimator. When features are correlated—and Section 5.2 states that correlations up to 0.95 are deliberately retained—the split-based contributions are non-unique: a different but equally accurate forest, or an equally valid attribution method such as SHAP, can distribute the same prediction difference differently across the same features. The paper never tests whether the ranking is stable under such alternative attributions, nor does it compare against simple baselines like a linear model's standardized coefficients or direct feature deltas relative to baseline. The empirical validation in Section 7.1 consists of only seven to eight incidents with no quantitative agreement metric, no control, and one acknowledged mismatch. Thus the observed agreement with experts could be driven by the model's sensitivity to the dominant feature rather than by a valid causal ranking. This is a load-bearing gap because the entire product claim, and the 'no misleading predictions' assertion, rest on this attribution being meaningful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Griffin, a system deployed in Microsoft's production analytics clusters that aims to automatically rank the potential causes of recurring job slowdowns without using labeled incident data. The system trains a random forest to predict job runtime from job-, machine-, and cluster-level features, then uses TreeInterpreter to decompose each prediction into per-feature contributions relative to a baseline computed from similar historical runs. The contributions are ranked and presented to users with a high/medium/low confidence level. The evaluation in Section 7 compares random forest against linear, gradient-boosted, and deep models using MARE, studies scalability of a global versus per-template model, reports agreement with engineer-validated causes on eight historical incidents, and gives an out-of-domain illustration using the Auto gas-mileage dataset.","tokens_in":14354,"tokens_out":3878,"duration_ms":41205,"significance":"If validated, Griffin would be a practically valuable contribution: it is a production anomaly-reasoning system that avoids labeled data, takes a job-centric rather than machine-centric view, and is built on a formal interpretability decomposition (Eqs. 1-3). Strengths of the paper include the explicit problem formalization, the MARE evaluation of model families (Table 3), the demonstration that a global model generalizes to unseen job templates (Section 7.3, jobs 6-8 in Table 2), and the concrete deployment description (Section 6). The paper is also honest about acknowledged limitations, notably the job-3 mismatch in Table 2. However, the central claim that the emitted feature ranking identifies actual slowdown causes is not yet supported by the evidence: the causal interpretation of the TreeInterpreter decomposition is an assumption, the validation set is tiny and lacks a quantitative agreement metric, and no baseline attribution method is compared. The contribution is therefore promising but currently conditional.","major_comments":[{"comment":"The central claim that Griffin's reasons are \"highly correlated\" with expert-validated reasons rests on only eight incidents, of which one is an acknowledged miss (job 3), two are top-5 rather than top-1 agreements (jobs 2 and 4), and none are accompanied by a quantitative agreement metric such as top-1/top-k hit rate, rank correlation, or an inter-rater reliability measure. The text summarizes this small sample as \"highly correlated,\" which overstates the evidence. The authors should report a precise agreement metric with uncertainty bounds, compare against a simple baseline such as ranking raw feature deviations from the baseline (without any trained model), and ideally validate on a larger incident corpus or with multiple expert raters.","section":"Section 7.1, Table 2"},{"comment":"The decomposition in Eq. (3) is an algebraic identity for the trained random forest, not a causal attribution: TreeInterpreter distributes predictive signal along decision paths, and when features are correlated (Section 5.2 states that correlations up to 0.95 are deliberately retained), the per-feature contributions are non-unique across equally accurate forests or across alternative attribution methods. The paper never tests the stability of the resulting ranking under alternative attributions (e.g., SHAP, permutation-based importance, or leave-one-covariate analysis) nor compares against simpler feature-delta baselines. Because the system's output is a ranked list of causes, this gap is load-bearing; the observed expert agreement could be driven by the model's sensitivity to a dominant feature rather than by a valid causal ranking.","section":"Section 4.2-4.3, Eq. (3) and (9)"},{"comment":"The confidence thresholds t1, t2, and p are tuned on validation data and are then used post hoc to interpret the validation outcome: job 3's mismatch is attributed to \"low confidence,\" while high- and medium-confidence cases are counted as successes. This creates a risk of circularity: if confidence is correlated with prediction error or with the chance of expert agreement, the reported agreement over the remaining cases may be optimistic. The paper should report agreement rates stratified by confidence level, and should explicitly state how many incidents would be excluded by a low-confidence flag before the \"highly correlated\" claim is made.","section":"Section 4.4 and Table 2"},{"comment":"The statement that \"Griffin's tree-based models have an innate feature of being robust to correlated features\" is not substantiated for the ranking task. Sections 7.2-7.4 evaluate prediction accuracy (MARE), but the ranking task is exactly where non-identifiability under correlation matters. A simple experiment with synthetic or shuffled correlated features, or retraining the forest with different random seeds to measure rank stability, would clarify whether the emitted ranking is robust or whether it reflects split-based contributions that could redistribute arbitrarily among correlated features.","section":"Section 5.2"}],"minor_comments":[{"comment":"The caption reads \"Two occurrences of the same job, broken down .\" with a dangling phrase; it should be completed, e.g., \"broken down by stage.\"","section":"Figure 2"},{"comment":"The table uses placeholders R1 through R6 without defining them in the caption or text; the sentence \"we use Rx variables for the rest\" does not tell the reader what those reasons are. A legend or explicit list of the actual features is needed.","section":"Table 2"},{"comment":"The notation y_β (baseline runtime) and y'_β (model prediction on baseline features) is confusing; the text says \"the model prediction is very accurate\" for baseline jobs and then uses y_β in Eq. (3) as if it were the actual baseline runtime, but Eq. (3) actually compares to y'_β. Please unify the notation and clarify which quantity Eq. (3) approximates.","section":"Section 4.2"},{"comment":"The Auto dataset illustration is anecdotal and has no ground-truth ranking against which to validate the delta contributions; it should be explicitly labeled as a demonstration of the mechanism rather than as additional validation of the cause-ranking claim.","section":"Section 7.5"},{"comment":"The system name appears as both \"Griffin\" (in the arXiv metadata) and \"Griffin\" (in the text); please unify the spelling, including the ligature.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry systems paper whose core value is the integration of a known interpretability technique (TreeInterpreter) into a production anomaly-reasoning pipeline. The main novelty is the application and deployment, not the attribution method itself. The small validation set and the lack of any attribution baseline are the principal obstacles; these are fixable within the manuscript's scope by adding experiments and substantially tempering the causal language. I see no citation-ethics or scope concerns beyond the usual expectation of reporting negative or null results for the validation set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the one thing to know: Griffin is a real, deployed system that does something genuinely useful—automatically ranking possible causes of job slowdowns without labeled incidents. The application of TreeInterpreter-style contribution decomposition to job-centric anomaly reasoning at this scale is new for this problem setting, and the paper deserves credit for building it and for being upfront about low-confidence cases.\n\nThe other thing to know: the main evidence for the central claim is much thinner than the abstract suggests. Section 7.1 covers seven or eight incidents. One is a miss (job 3, low confidence), two are top-5 hits, and there is no quantitative agreement metric, no inter-rater reliability, and no comparison against a simpler attribution baseline. The \"highly correlated\" wording in the abstract is doing more work than the data supports.\n\nThe deeper soft spot is the attribution step. Eq. 3 is an algebraic identity for the random forest—the decomposition sums exactly to the prediction difference. That is fine as an accounting identity. But the paper implicitly treats each feature's delta contribution as the cause of the slowdown. TreeInterpreter attributes predictive signal along decision paths; it is not a causal estimator. With correlations up to 0.95 deliberately retained, a different but equally accurate forest could shift contributions across features. The paper never tests whether the ranking is stable under alternative attributions (e.g., SHAP or a linear model's standardized coefficients), and it does not compare against a trivial baseline like per-feature deviations from the job template's baseline. Without that, the agreement with experts could be driven by the model picking up on the dominant feature, not by a valid causal ranking.\n\nThere are minor issues too: the confidence thresholds t1/t2/p and the final feature list are not reported, and no data or code is available. These are typical for an industry paper, but they limit reproducibility.\n\nNone of this is a fatal flaw. The system is plausible, the math is standard, and the paper is honest about some limitations—it explicitly says low confidence means the reasons may fall outside the features used. The problem is real and the approach is a reasonable first cut. The evidence is just not yet strong enough to call it a validated root-cause ranking system.\n\nWho is this for? Engineers and researchers working on failure diagnosis in large cloud platforms, and anyone interested in unsupervised explanation of prediction deviations. It is worth a serious peer review—a strong referee could push the authors to add a larger validation set, baseline comparisons, and a stability analysis of the decompositions. I would like to see that version. My own verdict would be conditional, not accept, but this is the kind of paper that deserves referee time.","headline":"Griffin is a plausible, well-engineered system for ranking slowdown causes without labels, but the central causal claim rests on a thin validation set and an unexamined attribution assumption.","tokens_in":15127,"tokens_out":2365,"would_cite":true,"duration_ms":24969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an interpretable random forest trained to predict job runtime can rank the causes of job slowdowns without labeled incident data, and that its rankings match expert-validated diagnoses in a production analytics…","keywords":["job slowdown","root cause analysis","anomaly reasoning","interpretable random forest","unlabeled data","cloud analytics platform","runtime prediction","feature attribution"],"falsifier":"A controlled experiment on historical incidents would settle it: take a slowdown whose cause engineers confirmed, hold the accused feature at its baseline value while leaving other features unchanged, and see whether Griffin's ranking still names it; alternatively, add a synthetic feature highly correlated with the true cause and check whether it displaces the true cause. If the top-ranked reason follows the model's feature usage rather than the known cause, the attribution is not causal.","tokens_in":13900,"feed_emoji":"🔍","tokens_out":9985,"duration_ms":89834,"temperature":0.7,"pith_summary":"Griffin claims that the cause of a job slowdown can be discovered without any human-labeled incident data by explaining, rather than merely predicting, job runtime. It trains a random forest to predict runtime from job-, machine-, and cluster-level telemetry, then decomposes the gap between a slowed job's runtime and its template baseline into per-feature contributions and ranks those contributions as the reasons for the slowdown. In a production analytics platform, the paper reports, the ranked reasons agree with causes manually validated by domain-expert engineers, and no high-confidence prediction in the evaluated incidents pointed to a wrong cause. If the claims hold, slowdown triage that currently takes engineers hours of log and trace analysis can become an automated, ranked explanation served from a job ID.","feed_headline":"No labels needed: a random forest ranks why cloud jobs slow down","feed_subtitle":"The decomposition turns a job's runtime gap into ranked causes that match expert-validated reasons.","key_machinery":"The load-bearing mechanism is the delta-feature-contribution decomposition of a random-forest prediction. Along one tree, each split on feature $x_k$ at node $m$ contributes $(\\bar{y}_{m,j} - \\bar{y}_{m-1,j})$ to the prediction, and summing these along the root-to-leaf path makes the tree's prediction a constant plus the sum of per-feature contributions; averaging over all $J$ trees preserves the additive form $y = c + \\sum_k f_c^k$. Griffin applies the same decomposition to a baseline job's features and subtracts, so the gap between a slowed job's predicted runtime and the baseline predicted runtime equals the sum of per-feature deltas. Since the model predicts baseline runtimes with a mean absolute ratio error of about 2.2%, that gap is treated as the slowdown itself, and the ranking of positive deltas is the system's output.","core_discovery":"Griffin's central claim is that the feature attributions of an accurate, interpretable runtime model are a usable ranking of why a specific job slowed down. Given a slow job with features $x$ and a template baseline $x_\\beta$, the paper uses the tree-interpreter decomposition of a random forest, which writes any prediction as $y = c + \\sum_k f_c^k$, and subtracts the baseline decomposition to obtain $y - y'_\\beta = \\sum_k \\Delta f_c^k$, where $\\Delta f_c^k = f_c^k - f_c^{\\beta,k}$. The features with the largest positive deltas are presented as the ranked reasons for the slowdown, with a confidence level that combines the model's relative prediction error with the spread of individual tree predictions. The paper argues that, on the evaluated historical incidents, these rankings correlate with causes manually validated by domain-expert engineers, including for job templates the model never trained on, and that no high-confidence prediction in this set was misleading.","pith_inferences":["The recipe should transfer to any recurring workload with a stable baseline and rich telemetry—database queries, serverless invocations, or ML training runs—where a job template and an expected runtime can be defined.","The paper does not compare its attribution against simpler baselines such as raw feature deviations from baseline or linear-model coefficients; such a comparison would show whether the random-forest decomposition adds diagnostic power beyond reporting which features changed.","If the rankings are reliable, they could be fed directly to an autotuner that adjusts parallelism, data layout, or queueing parameters when a cause is identified, moving from diagnosis to automatic remediation—a direction the conclusion gestures toward but does not implement.","The confidence levels are calibrated on validation data, so a mature deployment should track whether high-confidence predictions continue to match engineer-confirmed causes over time as jobs and clusters evolve."],"forward_implications":["A user who sees a slowdown can get a ranked list of likely causes immediately from a job ID, so user-induced slowdowns (for example, increased data written) can be closed without system-administrator investigation.","A global model trained over many job templates can reason about jobs from templates it has never seen, as demonstrated by the out-of-template jobs in the validation set.","A low-confidence prediction still narrows the investigation: it rules out the monitored features and tells the engineer where not to look.","Because no labels are required, years of historical telemetry become usable for training, removing the labeling bottleneck that limits earlier anomaly-reasoning approaches.","The same additive-decomposition recipe transfers to other regression problems with a natural baseline, as shown on an automobile gas-mileage dataset where it explains why one car gets better mileage than its peers."],"supporting_citations":[{"why":"Supplies the tree-interpreter decomposition that rewrites each random-forest prediction as a constant plus a sum of per-feature contributions, which Griffin subtracts between a slow job and its baseline to rank causes.","marker":"[26]"},{"why":"Provides the per-tree prediction distribution used to estimate the confidence interval around the forest prediction, which feeds Griffin's confidence levels.","marker":"[19]"},{"why":"Documents the scale and recurrent nature of jobs in the analytics platform, motivating and enabling the template-baseline definition Griffin depends on.","marker":"[9]"},{"why":"Establishes that runtime predictability and SLOs matter for recurring production jobs, supplying the business rationale for automated slowdown diagnosis.","marker":"[16]"},{"why":"Defines the DAG-of-stages/tasks job model and the SQL-like processing engine from which Griffin's job-level features and execution telemetry are drawn.","marker":"[33]"}],"fun_headline_variants":["Griffon ranks cloud job slowdown causes without labeled data","No labels needed: random forest attributes pinpoint job slowdowns","From runtime gap to ranked causes: Griffon's production trick","Unlabeled jobs get expert-level slowdown diagnosis via tree splits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking is assumed to reflect the true cause of the slowdown: Griffin treats the random forest's per-feature split contributions as causal deltas, so the features whose predicted contribution to runtime changed most are declared the reasons; with correlated features, the model can assign shared signal to the wrong one.","fun_headline_variants_meta":{"raw":{"variants":["Griffon ranks cloud job slowdown causes without labeled data","No labels needed: random forest attributes pinpoint job slowdowns","From runtime gap to ranked causes: Griffon's production trick","Unlabeled jobs get expert-level slowdown diagnosis via tree splits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000926,"raw_usage":{"total_tokens":3992,"prompt_tokens":996,"completion_tokens":2996,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2928}},"tokens_in":612,"tokens_out":2996,"duration_ms":21226,"temperature":1.0,"reasoning_tokens":2928,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:23:32.463874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment on historical incidents would settle it: take a slowdown whose cause engineers confirmed, hold the accused feature at its baseline value while leaving other features unchanged, and see whether Griffin's ranking still names it; alternatively, add a synthetic feature highly correlated with the true cause and check whether it displaces the true cause. If the top-ranked reason follows the model's feature usage rather than the known cause, the attribution is not causal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the tree-interpreter decomposition that rewrites each random-forest prediction as a constant plus a sum of per-feature contributions, which Griffin subtracts between a slow job and its baseline to rank causes."},{"cited_title":"Meinshausen","cited_arxiv_id":null,"evidence_quote":"Provides the per-tree prediction distribution used to estimate the confidence interval around the forest prediction, which feeds Griffin's confidence levels."},{"cited_title":"Curino, S","cited_arxiv_id":null,"evidence_quote":"Documents the scale and recurrent nature of jobs in the analytics platform, motivating and enabling the template-baseline definition Griffin depends on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that runtime predictability and SLOs matter for recurring production jobs, supplying the business rationale for automated slowdown diagnosis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DAG-of-stages/tasks job model and the SQL-like processing engine from which Griffin's job-level features and execution telemetry are drawn."}],"review_version":1}