{"id":"c82dd790-9c1c-4f46-8c32-3fe479c5379a","arxiv_id":"2608.01041","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ML predictors rank processor configurations well in aggregate but fall below chance on counter-intuitive reversals and barely beat a no-feature baseline for closely matched policies.","lead":"Machine-learned performance predictors are tested on whether they can rank processor designs window by window. They handle broad structural changes but miss the local reversals and close policy races where cycle simulation still matters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The capacity-independence claim rests on an unmeasured Bayes ceiling; observed failures could be underfitting rather than partial observability.","rationale":"I focused on the capacity/underfitting concern rather than the Concorde-representativeness concern because the empirical finding that four specific predictor instantiations fail on hard ranking windows is well supported by the protocol; the leap to a fundamental trace-based limit is what changes the practical conclusion from 'these four models underperform on difficult ranking windows' to 'no instruction-stream model can do better.' The paper's Section VI-C proves only a conditional ceiling for a fixed observable window X, not that the tested models attain that ceiling, and Section V-D's high regression accuracy does not rule out differential underfitting. A capacity-scaling experiment or a state-aware upper-bound model would settle whether the failure is due to missing hidden state or to insufficient model/training capacity. Since the reader's verdict is already CONDITIONAL and explicitly flags the overstatement of the capacity claim, my concern does not change the recommended verdict; it sharpens the condition that should be attached to the paper's central limitation claim.","tokens_in":18259,"tokens_out":7595,"duration_ms":73303,"concrete_test":"Run a capacity/learning-curve ablation on the two best families: train OneDSE and Concorde at 1x, 4x, and 16x hidden width, with 1 versus 20 seeds, on the same 80/10/10 splits, and record macro strict-pair match minus training-majority and the CIW match. If the curves plateau at or below the reported +2.09 pp and roughly 39.9% CIW values, the capacity explanation is falsified and the paper's interpretation survives. If either metric climbs materially with capacity or seed count, the four-family failure is at least partly underfitting, and the 'not a matter of model capacity' conclusion must be retracted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VI-C derives A*_ab = E_X max P(S|X) and invokes the data-processing inequality and Fano's inequality, but it never estimates this Bayes ceiling or shows that the four trained families are near it. The observed failures are therefore consistent with two very different explanations: partial observability (the paper's claim) or inadequate capacity and training signal (e.g., NeuroScalar and SimNet use one completed seed, and no learning-curve or capacity-scaling experiment is reported). The supporting evidence that 'this failure is not a matter of model capacity' is indirect: high within-one-cycle regression accuracy (95.8%) for the LSTM, and a t-SNE plot that the paper itself disclaims as not proof of non-learnability. Differential ranking error can remain high even when absolute cycle regression is accurate, so the regression accuracy does not establish that the ranking failures are irreducible. The information-theoretic bound is logically valid for a fixed feature window X, but its force depends on showing that H is genuinely non-recoverable from any feasible trace representation and that the Bayes accuracy is near chance; neither is measured. Thus the paper's strongest negative conclusion, that the limitation is not about model capacity, exceeds the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether machine-learned performance predictors can support design-space-exploration ranking at instruction-window granularity. It instantiates four predictor families (NeuroScalar-style LSTM, SimNet-style CNN/MLP, Concorde-style summary model, OneDSE-style Transformer) on a common trace-based protocol with 29 SPEC CPU2006 traces, two design regimes (Structural Parameters and Behavioral Policies), chronological splits, tie-aware metrics, and a feature-free majority baseline. In the SP regime, aggregate ordering is strong (77–89% all-window agreement) but the counter-intuitive-window (CIW) match is only 23.3–39.9%, below the 50% random strict-ordering reference for all four families. In the BP regime, 37.8% of pair-windows are ties, most strict margins are a few cycles, and only the selected OneDSE head beats the majority baseline, by 2.1 percentage points. The paper argues via an information-theoretic argument (§VI-C) that this failure is not a matter of model capacity but of partial observability: ranking outcomes depend on hidden microarchitectural state absent from the instruction stream, and no trace-based predictor can exceed the Bayes accuracy determined by observable inputs alone.","tokens_in":18410,"tokens_out":2329,"duration_ms":22658,"significance":"The paper addresses a genuinely important question for ML-based design-space exploration: regression accuracy does not imply ranking fidelity, especially for closely matched policies and for local reversals of a structural prior. The experimental protocol is a real strength: common input alignment across configurations, chronological splits with guard bands, paired trace bootstrapping, tie-aware evaluation, and a feature-free majority baseline are all appropriate and carefully described. The CIW diagnostic is a useful new evaluation lens, and the margin analysis correctly explains why ranking and regression accuracy can diverge. If the empirical claims survive scrutiny, the paper would be a valuable caveat to the ML-for-architecture literature. The main weakness is that the strongest theoretical conclusion—that the failure is not a matter of capacity—is not directly supported: the Bayes ceiling A* is never estimated, no capacity-scaling experiments are reported, and the model families are simplified instantiations of the systems whose names they carry.","major_comments":[{"comment":"The central claim that the ranking failure is 'not a matter of model capacity' is load-bearing but is not established. Eq. (11) defines the Bayes accuracy A*_ab = E_X max_s P(S_ab=s|X), but the paper never estimates this quantity and never shows that the four trained families operate near it. The observed failures are equally consistent with underfitting or under-training; indeed, §IV(e) states that NeuroScalar-style and SimNet-style results use only one completed seed, and no learning-curve or capacity-scaling experiment is reported. The indirect evidence cited in §V-D (95.8% within-one-cycle regression accuracy) does not resolve this, because, as the paper itself notes, differential ranking error can remain high even when absolute regression is accurate. Please either estimate A* (for example, with a calibrated probabilistic model, a nearest-neighbor lower bound, or an oracle-feature experiment) or add capacity/training-signal scaling studies; alternatively, soften the capacity-independence claim to 'consistent with partial observability' rather than asserting it as established.","section":"§VI-C, Eq. (11)"},{"comment":"The external validity of the negative result is limited by the way the predictor families are instantiated. For Concorde, the full system combines an analytical front-end with lightweight cache and branch simulation, but Table I states that this work instantiates only the ML backbone over window statistics. If the real Concorde, or the real NeuroScalar/SimNet/OneDSE systems with dynamic-state inputs, would rank these windows correctly, the conclusion that 'no trace-based predictor can exceed the Bayes accuracy' would not apply to those systems. The paper explicitly disclaims exact reimplementation, but the abstract and conclusion state the limitation in generality. Please add a state-aware variant (for example, supplementing features with cache-hit levels, prefetch confidence, or replacement metadata) to test whether the failure is specifically due to the six-field trace, or restrict the capacity-independence claim to the exact feature set used and state that the result does not cover predictors with dynamic-state inputs.","section":"§III, Table I"},{"comment":"The headline CIW claim—that all four families fall below the 50% random strict-ordering reference—is reported as point estimates without uncertainty quantification. Table V reports trace-bootstrap confidence intervals for the BP results, but Table X reports only single numbers per pair and model. Given that CIW fractions range from 13.2% to 43.5% and that the number of CIW windows per pair may be small, some of these estimates could be statistically indistinguishable from 50% or from each other. Please provide paired trace-bootstrap intervals for the CIW match rates, or at least per-pair sample sizes, so the claim is not overstated.","section":"§V-G, Table X"}],"minor_comments":[{"comment":"The heading contains a typo: 'Realiability' should be 'Reliability'.","section":"§V-C"},{"comment":"The notation in Eq. (2) is unclear: the concatenation bracket notation and the variables s and R would benefit from a sentence defining the split positions explicitly, especially since the text already explains the window structure in words.","section":"§III-B, Eq. (2)"},{"comment":"The symmetric-log scale in Figure 1 is unusual for a box plot and the caption does not state what the box bounds or whiskers represent; please add a clear legend and a short description of the box elements.","section":"Figure 1"},{"comment":"Figure 6 is explicitly disclaimed as not proof of non-learnability, which is appropriate; the accompanying claim that the raw feature space 'does not contain an obvious low-dimensional partition' is qualitative and should be labeled as such in the main text, not only in the caption.","section":"§VI-D, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's experimental protocol is careful and the empirical ranking results are worth publishing if the CIW uncertainty is characterized and the capacity-independence claim is either supported or appropriately qualified. The main risk is that the abstract and conclusion overstate the theoretical result relative to the evidence. I suggest inviting a revision that adds either A* estimates or capacity scaling experiments, and that clarifies the scope of the claim for systems with dynamic-state inputs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the empirical core is worth engaging, but the paper oversells the capacity-independence claim and contradicts itself on whether OneDSE beats the majority baseline.\n\nWhat's new: the window-level ranking evaluation with CIW diagnostics, tie-aware metrics, margin decomposition, and the feature-free majority baseline. That is a real contribution to how ML predictors should be validated. The two-regime split (structural vs behavioral) is well motivated, and the protocol is careful: chronological splits, paired trace bootstrap, guard bands, head selection on validation only. Give credit where due.\n\nSoft spots. First, the abstract and conclusion say \"no model family reliably beats a feature-free majority baseline,\" but Table V shows OneDSE at +2.09pp with 95% CI [0.52, 3.68]. That is a statistical beat, small but reliable. The paper later acknowledges it, so the global statement needs fixing.\n\nSecond, the Section VI-C argument that failure is \"not a matter of model capacity\" is not supported. Equation 11 defines the Bayes accuracy, but they never estimate it or show the trained families are near it. The observed failures are equally consistent with underfitting, especially since LSTM and SimNet use one completed seed and no learning-curve experiment is reported. The high regression accuracy (95.8%) does not save the argument; the paper itself shows regression error and differential ranking error can diverge. The honest version is: within these instantiations, ranking fails on the hard cases, and partial observability is a plausible explanation, not a proven ceiling.\n\nThird, the Concorde-style instantiation drops the analytical front-end, so it is not the actual Concorde system. The paper admits this, but then the conclusion about \"trace-based predictors\" should be scoped to the ML-backbone-only version.\n\nOn the CIW result: point estimates all below 50% are suggestive, but no confidence intervals are reported there, and CIWs are only 22.4% of non-tied windows, so per-pair samples may be thin. I would not call that a decisive failure for each family.\n\nWho this is for: researchers building ML performance predictors and architects doing DSE. The diagnostics are worth adopting regardless of the theoretical claim. With the capacity overclaim trimmed and the OneDSE inconsistency fixed, this is a strong paper. I would send it to peer review, not desk reject, asking for major revision focused on estimating (or honestly reframing) the Bayes ceiling and reporting seed-level variance.","headline":"Solid empirical study of window-level ranking for ML performance predictors, but the capacity-independence claim is overreached and the abstract contradicts its own Table V on OneDSE.","tokens_in":18989,"tokens_out":2060,"would_cite":true,"duration_ms":17691,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that learned performance predictors should be evaluated by how well they rank designs per instruction window, not by regression error alone, and that four predictor families fail exactly where rankings reverse or margins…","keywords":["machine learning for architecture","design-space exploration","pairwise ranking","counter-intuitive windows","partial observability","instruction-window prediction","prefetching and replacement policies","cycle-level simulation"],"falsifier":"Train the same four predictor families on the same aligned traces and windows but add the hidden dynamic state as features (cache-hit levels, prefetch-queue occupancy, replacement metadata, outstanding-miss counts); if BP strict-pair match then decisively exceeds the feature-free majority baseline and CIW match crosses the 50% random-ordering reference on the five clear-prior pairs, the paper's Bayes-ceiling claim for trace-only inputs is falsified.","tokens_in":18033,"feed_emoji":"🖥️","tokens_out":8194,"duration_ms":68976,"temperature":0.7,"pith_summary":"The paper tries to establish that standard validation metrics for learned performance predictors—regression error or aggregate accuracy—miss what design-space exploration actually needs: reliable ordering of candidate designs at instruction-window granularity. It evaluates four predictor families, an LSTM, a CNN/MLP, a window-summary model, and a Transformer, under a common protocol with the same trace-derived inputs and targets. For structural changes like issue width or cache size, aggregate ordering is strong, but on counter-intuitive windows—where the expected-slower configuration is actually faster—all four families fall below the 50% random strict-ordering reference. For behavioral policies like prefetching and replacement, ground-truth ties cover 37.8% of pair-windows and no family reliably beats a feature-free majority baseline, with the best selected head improving by only 2.1 percentage points. The paper argues this is not a model-capacity problem: because the decisive microarchitectural state is absent from the instruction trace, a Bayes ceiling limits any trace-based predictor.","feed_headline":"Machine-learned chip predictors miss the reversals that matter","feed_subtitle":"Four predictor families stay near a majority baseline on close policy pairs—hidden hardware state is the likely wall.","key_machinery":"The load-bearing objects are counter-intuitive windows, the pair-margin decomposition, and a partial-observability Bayes limit. A counter-intuitive window (CIW) is a non-tied instruction window in which the configuration expected to be slower is faster; conditioning the ranking metric on these windows isolates reversal detection from recovery of the dominant hardware trend. The margin decomposition, expressed as $P(\\text{misrank}) \\le P(|\\Delta|\\le \\gamma) + P(|D|\\ge \\gamma)$, shows that narrow true margins amplify prediction error. The Bayes limit, $A^*_{ab} = \\mathbb{E}_X[\\max_s P(S_{ab}=s|X)]$, together with the data-processing inequality $I(S;Z) \\le I(S;X)$, carries the argument that hidden policy state, not model expressiveness, bounds trace-based ranking.","core_discovery":"The central claim is that ML-based microarchitectural performance prediction should be evaluated as a ranking problem, not only a regression problem, and that under that test the predictors fail in a specific, structured way. Four predictor families share a protocol: they see the same six-field instruction features and predict cycle counts for the middle 500 instructions of 1500-instruction windows. In the structural-parameter regime, all-window pairwise agreement reaches 77–89%, but on the five pairs with a clear architectural prior, counter-intuitive windows make up 22.4% of non-tied windows and CIW match is only 23.3–39.9%, below the 50% random strict-ordering reference. In the behavioral-policy regime, 37.8% of pair-windows are ground-truth ties, most strict pairs differ by only a few cycles, and no model family reliably beats a train-pair-majority baseline; the best selected Transformer head improves by 2.1 percentage points, with gains concentrated at wide margins. The paper concludes that this pattern is not a capacity failure: when the true ranking label depends on hidden microarchitectural state that is not among the observed instruction features, no trace-based predictor can exceed the Bayes accuracy set by observable inputs alone.","pith_inferences":["If the Bayes-limit argument is right, adding cheap dynamic-state summaries—sampled cache-hit levels, prefetch-queue occupancy, or replacement metadata at checkpoints—should move BP ranking above the trace-only ceiling; the paper discusses this tradeoff but does not test it.","The same partial-observability ceiling should apply to other learned policy-ranking tasks, such as branch-predictor choices, DRAM scheduling, or cache-replacement decisions, where simulator-internal state is invisible to the input features.","A practical hybrid emerges implicitly: use trace-based ML to prune the structural space, then reserve cycle-level simulation for the surviving behavioral-policy candidates; the paper sketches this as a middle ground but does not evaluate it end-to-end.","The learnable minority of benchmarks with repeated memory phases suggests that observable phase structure is what makes ranking learnable; a targeted feature-selection study of those phase patterns could identify when trace-based ranking is viable before deployment."],"forward_implications":["Architects should not treat regression accuracy or aggregate ranking as evidence that a learned predictor can order closely matched designs at the per-window level.","ML surrogates remain useful for coarse structural pruning, where margins are large; the paper's SP results put that regime at 77–89% agreement and describe it as a positive control.","Future evaluations of learned predictors should report tie-aware, margin-stratified, pair-level ranking against a feature-free majority baseline, because those metrics expose the failures documented here.","For behavioral-policy selection at instruction-window granularity, cycle-level simulation remains necessary, since the best measured gain over the majority baseline is only 2.1 percentage points and that gain is concentrated on high-margin pairs that are already easy to rank."],"supporting_citations":[{"why":"Supplies the LSTM sequence-predictor family and the six-field microarchitecture-independent trace-input assumption.","marker":"[27]"},{"why":"Supplies the CNN/MLP latency-predictor family and motivates the regression-style success metric the paper argues is insufficient.","marker":"[16]"},{"why":"Supplies the window-summary predictor family and the aggregate-level DSE deployment that contrasts with per-window ranking.","marker":"[19]"},{"why":"Supplies the Transformer predictor family and the regression/classification head-selection protocol used for the selected result.","marker":"[23]"},{"why":"Supplies the instrumented cycle-level simulator used to generate instruction features and ground-truth cycle-count targets.","marker":"[4]"},{"why":"Defines one of the data prefetchers in the behavioral-policy design space.","marker":"[20]"},{"why":"Defines the other data prefetcher in the behavioral-policy design space.","marker":"[2]"},{"why":"Defines one of the replacement policies in the behavioral-policy design space.","marker":"[25]"},{"why":"Defines the other replacement policy in the behavioral-policy design space.","marker":"[18]"}],"fun_headline_variants":["ML chip predictors miss the close calls that matter","Hidden hardware state blocks ML ranking on tight pairs","Ranking test reveals ML predictors fail on key reversals","ML predictors strong on average, weak on critical windows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes its four style instantiations faithfully represent the actual NeuroScalar, SimNet, Concorde, and OneDSE systems; in particular, Concorde's analytical front-end with dynamic cache and branch state is removed in the common protocol, so the conclusion that trace-based predictors cannot rank these policies better would not extend to the full system if that front-end supplies the missing hidden state.","fun_headline_variants_meta":{"raw":{"variants":["ML chip predictors miss the close calls that matter","Hidden hardware state blocks ML ranking on tight pairs","Ranking test reveals ML predictors fail on key reversals","ML predictors strong on average, weak on critical windows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2569,"prompt_tokens":1148,"completion_tokens":1421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":764,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":764,"tokens_out":1421,"duration_ms":8946,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:13:05.254748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same four predictor families on the same aligned traces and windows but add the hidden dynamic state as features (cache-hit levels, prefetch-queue occupancy, replacement metadata, outstanding-miss counts); if BP strict-pair match then decisively exceeds the feature-free majority baseline and CIW match crosses the 50% random-ordering reference on the five clear-prior pairs, the paper's Bayes-ceiling claim for trace-only inputs is falsified.","supporting_citations":[{"cited_title":"Neuroscalar: A deep learning framework for fast, accurate, and in-the-wild cycle-level performance prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM sequence-predictor family and the six-field microarchitecture-independent trace-input assumption."},{"cited_title":"Simnet: Accurate and high-performance computer architecture simulation using deep learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the CNN/MLP latency-predictor family and motivates the regression-style success metric the paper argues is insufficient."},{"cited_title":"Onedse: A unified microprocessor metric prediction and design space exploration framework,","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer predictor family and the regression/classification head-selection protocol used for the selected result."}],"review_version":1}