{"id":"9dcaf685-7cf0-4acf-91aa-1830191f0aa4","arxiv_id":"2605.06968","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"New hardware-usage-based similarity metrics can identify matching computational kernels between proxy applications and performance suites on both CPU and GPU systems.","lead":"The paper proposes performance similarity metrics based on how computational kernels use hardware resources like compute units and memory. These could help check whether proxy applications accurately represent the behavior of full HPC simulation codes on CPUs and GPUs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Validation of 'correct' kernel matching lacks independent ground truth or runtime equivalence check","rationale":"The reader's weakest assumption directly identifies the missing external validation step. Because the full text was stated to be available yet the abstract-level claim remains unsupported by any reported independent check, the concern is load-bearing for the headline result. No other internal inconsistency or circularity is visible from the given material.","tokens_in":1693,"tokens_out":335,"duration_ms":17456,"concrete_test":"Extract the specific Kripke and RAJA kernels declared as matched in the paper; instrument both with the same hardware counters plus wall-clock time and strong-scaling curves on the CPU and GPU platforms described; if the matched pair differs by >15% in any normalized performance metric not already used in the similarity vector, the validation claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that the proposed hardware-usage similarity metrics 'correctly match' a Kripke kernel to a RAJA kernel. For this to hold, there must exist an external criterion of correctness (e.g., expert labeling, measured runtime/scaling parity, or application-level outcome equivalence) that is not derived from the same usage-pattern vectors. The abstract states the validation occurred but supplies no description of that criterion, no quantitative similarity scores, and no side-by-side performance data confirming the matched kernels behave equivalently under load. If the 'correctness' label is assigned by inspecting the same metrics used to compute similarity, the validation is circular and does not demonstrate that the metrics identify performance-equivalent kernels.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes hardware-usage-based performance similarity metrics to assess how well proxy applications represent the characteristics of HPC simulation codes. It defines and characterizes two broad categories of kernels exhibiting similar performance behavior, evaluates pairwise similarity on kernels from the Kripke proxy application and the RAJA Performance Suite on both CPU-only and GPU-accelerated systems, and claims validation by correctly matching one Kripke kernel to a RAJA kernel.","tokens_in":1803,"tokens_out":471,"duration_ms":26270,"significance":"If the metrics can be shown to identify performance-equivalent kernels via an independent ground-truth criterion (rather than circular use of the same usage vectors), the approach would provide a scalable alternative to manual proxy validation in HPC, which is a recognized bottleneck. The dual-platform evaluation (CPU and GPU) and use of established suites (Kripke, RAJA) are positive elements that would strengthen applicability if quantitative results were supplied.","major_comments":[{"comment":"Abstract: The central validation claim ('We validate that our similarity metrics correctly match a kernel in the Kripke proxy app to a kernel in the RAJA Performance Suite') supplies no quantitative similarity scores, no definition or formula for the metrics, no error analysis, and no description of the independent correctness criterion (e.g., expert labeling or measured runtime/scaling parity). This renders the primary result unverifiable from the text.","section":"Abstract"},{"comment":"Evaluation (presumed §4–5): No side-by-side runtime, scaling, or application-level outcome data are presented to confirm that the matched kernels behave equivalently under load. Without such external evidence, the claim that hardware-usage patterns alone suffice to establish performance equivalence rests on an untested assumption and cannot support the paper's conclusions.","section":"Evaluation section"}],"minor_comments":[{"comment":"The abstract and introduction would benefit from explicit definitions or equations for the two proposed similarity metrics and the two kernel categories before the evaluation is described.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early draft; the absence of any numerical results or metric definitions suggests the validation section may be incomplete rather than intentionally omitted."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments on our manuscript. We address each major comment below and describe the revisions we will incorporate.","responses":[{"response":"We agree the abstract is high-level and omits specifics. The similarity metrics are defined via hardware-usage vector comparisons (detailed with formulas in Section 3), and the validation consists of the metrics producing the highest pairwise score for one specific Kripke-RAJA kernel pair on both CPU and GPU platforms. To improve verifiability, we will revise the abstract to include representative quantitative similarity scores, a brief statement of the metric formulation, and clarification that correctness follows from consistent high-similarity matches across platforms.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central validation claim ('We validate that our similarity metrics correctly match a kernel in the Kripke proxy app to a kernel in the RAJA Performance Suite') supplies no quantitative similarity scores, no definition or formula for the metrics, no error analysis, and no description of the independent correctness criterion (e.g., expert labeling or measured runtime/scaling parity). This renders the primary result unverifiable from the text."},{"response":"The evaluation section reports the pairwise similarity scores derived from hardware-usage vectors and identifies the matching kernel pair on the basis of those scores. We acknowledge that direct runtime and scaling comparisons would constitute stronger external corroboration. We will add such measurements for the matched kernels (and a few non-matched controls) on both the CPU-only and GPU-accelerated systems in the revised evaluation section.","revision_made":"yes","referee_comment":"[Evaluation section] Evaluation (presumed §4–5): No side-by-side runtime, scaling, or application-level outcome data are presented to confirm that the matched kernels behave equivalently under load. Without such external evidence, the claim that hardware-usage patterns alone suffice to establish performance equivalence rests on an untested assumption and cannot support the paper's conclusions."}],"tokens_in":1357,"tokens_out":430,"duration_ms":49578,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a practical method for checking how similar computational kernels are across codes and proxies, using metrics based on hardware usage patterns rather than full profiling. They introduce two categories of kernels that share performance characteristics and then define pairwise similarity metrics. The evaluation uses kernels from the Kripke proxy app and the RAJA Performance Suite, run on both CPU-only and GPU systems. They report that the metrics correctly identify a match between one Kripke kernel and one in RAJA. This is new in the way it ties the metrics directly to hardware usage for this matching task, and the validation on these specific applications adds some concrete evidence. It does a good job highlighting the problem of manual benchmark selection in HPC and offering an automated path forward. The focus on both CPU and GPU is also a plus, given how architectures are evolving. The soft spot is in the validation step. The claim that the metrics correctly match the kernels requires some external way to know the match is right, like measured performance parity or independent labeling. The abstract does not provide quantitative similarity scores, details on the metric formulas, or confirmation that the equivalence was checked separately from the usage patterns themselves. If the full paper has those elements with actual numbers and comparisons, it strengthens the work; otherwise the central result stays hard to verify. The assumption that hardware usage alone suffices for performance equivalence also needs scrutiny, as context like data movement or algorithm specifics might matter. This paper is for HPC performance engineers and developers who select or design proxy applications for testing new hardware. A reader in that space can take the metrics and try them on their own codes, which gives it some immediate utility. It deserves a serious referee. The idea is grounded in a real pain point, the evaluation is on relevant software, and the methods appear defined independently. Reviewers can focus on clarifying the validation process and adding the missing quantitative support. I recommend sending it to peer review.","headline":"The paper defines hardware-usage similarity metrics and two kernel categories then applies them to match a Kripke kernel with one from RAJA, but the validation step needs clearer independent evidence.","tokens_in":2293,"tokens_out":466,"would_cite":false,"duration_ms":31741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"HPC kernel clustering via hardware counters; no RS overlap","alignment":"orthogonal","rationale":"Paper defines Euclidean similarity on top-down/roofline metrics and applies agglomerative/k-means clustering to match kernels (e.g., LTIMES) across Kripke and RAJAPerf. Central machinery is empirical performance analysis with no connection to J-cost, φ-ladder, 8-tick periodicity, or any RS forcing theorem.","tokens_in":50659,"confidence":"high","tokens_out":112,"duration_ms":7186,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Performance similarity metrics based on hardware usage patterns correctly match a kernel from the Kripke proxy application to one in the RAJA Performance Suite.","keywords":["performance similarity metrics","computational kernels","proxy applications","HPC benchmarks","hardware usage patterns","Kripke","RAJA Performance Suite"],"falsifier":"A case where two kernels receive a high similarity score from the metrics yet display clearly different run times, scaling behavior, or resource bottlenecks when executed on identical hardware.","tokens_in":2572,"feed_emoji":"📈","tokens_out":604,"duration_ms":32192,"temperature":0.7,"pith_summary":"The paper introduces metrics that compare computational kernels by how they use hardware resources such as compute units and memory bandwidth. These metrics are tested on kernels from the Kripke proxy and the RAJA Performance Suite across both CPU-only and GPU systems. The evaluation confirms that the metrics identify matching kernels without needing manual inspection of full code behavior. A reader would care because current ways to check if benchmarks represent real simulation codes are slow and do not scale as HPC hardware adds more parallelism and accelerators. If the metrics hold, developers could automatically judge how well proxy apps stand in for production codes when assessing new machines.","feed_headline":"Hardware usage metrics match Kripke kernel to RAJA proxy","feed_subtitle":"New scores based on resource patterns automate checks on whether benchmarks represent real HPC codes on CPU and GPU systems.","key_machinery":"Pairwise performance similarity metrics derived from hardware usage patterns, which categorize kernels into two groups exhibiting comparable performance behavior.","core_discovery":"By defining two broad categories of kernels that share performance traits and computing pairwise similarity scores from hardware usage data, the authors show that one kernel in Kripke aligns with a kernel in RAJA on both CPU and GPU platforms, validating the metrics for assessing how well benchmarks represent full codes.","pith_inferences":["The method could be extended to build libraries of equivalent kernels across many more proxy applications.","If validated further, it would allow quicker screening of new architectures by running only matched proxy kernels instead of full codes.","Adding memory hierarchy or communication pattern data to the metrics might strengthen matches for communication-heavy kernels."],"forward_implications":["Comparison of codes and their proxy representations no longer requires labor-intensive manual review.","Benchmark suites can be checked for coverage of real application performance traits on emerging hardware.","Hardware designers and code developers gain a scalable way to evaluate how well proxies capture production code behavior.","The approach works for both CPU-only and GPU-accelerated systems using the same metrics."],"fun_headline_variants":["Hardware metrics match Kripke kernel to RAJA on CPU and GPU","Kernel similarity scores tie Kripke to RAJA via hardware usage data","Pairwise metrics align Kripke kernel with RAJA on CPU and GPU","Hardware usage similarity matches Kripke to RAJA benchmarks"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Hardware usage patterns alone, without full runtime profiling or application-specific details, are enough to decide whether two kernels will show equivalent performance.","fun_headline_variants_meta":{"raw":{"variants":["Hardware metrics match Kripke kernel to RAJA on CPU and GPU","Kernel similarity scores tie Kripke to RAJA via hardware usage data","Pairwise metrics align Kripke kernel with RAJA on CPU and GPU","Hardware usage similarity matches Kripke to RAJA benchmarks"]},"model":"grok-4.3","cost_usd":0.007876,"raw_usage":{"total_tokens":3572,"prompt_tokens":628,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":78762000,"prompt_tokens_details":{"text_tokens":628,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2869,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":628,"tokens_out":75,"duration_ms":29265,"temperature":1.0,"reasoning_tokens":2869,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-11T01:05:11.101968+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A case where two kernels receive a high similarity score from the metrics yet display clearly different run times, scaling behavior, or resource bottlenecks when executed on identical hardware.","supporting_citations":[],"review_version":1}