{"id":"3371ca9b-e8f8-420a-b4ae-90fe1d09f2e8","arxiv_id":"2606.13255","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"An embedding-based method learns solver-problem relationships from performance data to predict optimal configurations for linear solvers on unseen problems, achieving 17% better accuracy than feature-based models on SuiteSparse matrices.","lead":"The paper introduces an embedding-based framework for predicting optimal linear solver configurations from performance data on sparse matrices. A smart generalist might read it to see how machine learning can reduce trial-and-error in choosing solvers for scientific simulations.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Projection of unseen matrices into performance embedding space via inexpensive features lacks isolated validation","rationale":"The reader's weakest assumption directly identifies the generalization step that the headline metrics depend on. Full-text access does not remove the need for an explicit test isolating the inexpensive-feature projection; the current aggregate results leave that step untested.","tokens_in":1776,"tokens_out":322,"duration_ms":11825,"concrete_test":"Hold out 20% of matrices (stratified by nonzero pattern and condition number); train embeddings only on the remaining 80%; project the held-out set using solely the inexpensive feature subset; recompute MAPE and 1-error on this strict unseen split and compare to a non-embedding baseline using the same inexpensive features. If the reported 24% MAPE reduction disappears, the projection assumption does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on inexpensive numerical features (e.g., basic matrix statistics) being sufficient to embed new problems into a space learned from solver-performance pairs, enabling multilabel prediction. Section 3 describes the framework as decoupling embedding learning from feature projection, yet the experiments (621 SuiteSparse matrices, 101 PETSc configs) report aggregate gains (17% top-accuracy, 37% MAPE drop) without an ablation that holds the embedding fixed and varies only the projection features, or that measures embedding-space distortion for out-of-distribution matrices. If inexpensive features fail to preserve performance-relevant distances, the reported competitiveness with reduced features collapses to a classical feature-based model.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes a modular embedding-based framework for linear solver performance prediction that decouples learning of solver-problem relationships from inexpensive numerical feature projection into the embedding space. It targets multilabel prediction using user-centric metrics (MAPE, nDCG, 1-error) and reports results on 621 SuiteSparse matrices across 101 PETSc configurations, claiming a 17% gain in top-prediction accuracy over classical feature-based models (with further gains when expensive features are added), plus 37% MAPE reduction and 46% 1-error reduction; the embedding approach remains competitive under reduced features.","tokens_in":1905,"tokens_out":473,"duration_ms":13260,"significance":"If the central claims hold after addressing validation gaps, the work could advance practical solver selection in scientific computing by offering a flexible, low-cost alternative to expensive-feature or problem-restricted prior methods. The data-driven embedding from observed performance data and focus on multilabel/user-centric metrics are strengths; the modular decoupling, if isolated, would be a notable contribution over monolithic feature-based baselines.","major_comments":[{"comment":"Section 3: The framework overview claims decoupling of embedding learning from feature projection, yet no ablation is reported that holds the learned embedding fixed while varying only the inexpensive projection features (or measures embedding-space distortion for out-of-distribution matrices). Without this, the reported competitiveness under reduced features cannot be attributed specifically to the embedding approach rather than classical feature effects.","section":"Section 3"},{"comment":"Section 4 (experiments): Aggregate gains (17% top-accuracy, 37% MAPE drop) are presented across 621 matrices and 101 configurations, but the support for the projection assumption lacks isolated validation (e.g., no hold-out embedding test or distortion metric). This is load-bearing for the central claim that inexpensive features suffice to project unseen problems accurately.","section":"Section 4"}],"minor_comments":[{"comment":"Abstract and Section 4: Clarify whether the 17% accuracy gain is measured with or without expensive features, and provide per-configuration breakdowns or variance to support the broad-range claim.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments. We address each major comment below and will revise the manuscript to include the requested isolated validations of the embedding decoupling and projection assumption.","responses":[{"response":"We agree that the current manuscript lacks an explicit ablation that holds the learned embedding fixed while varying only the inexpensive projection features, and does not report embedding-space distortion metrics for out-of-distribution matrices. This limits the strength of the attribution to the embedding approach. In the revision we will add such an ablation study together with distortion analysis.","revision_made":"yes","referee_comment":"[Section 3] Section 3: The framework overview claims decoupling of embedding learning from feature projection, yet no ablation is reported that holds the learned embedding fixed while varying only the inexpensive projection features (or measures embedding-space distortion for out-of-distribution matrices). Without this, the reported competitiveness under reduced features cannot be attributed specifically to the embedding approach rather than classical feature effects."},{"response":"We concur that the experiments section does not provide isolated validation such as a hold-out embedding test or distortion metric to directly support the projection assumption. This is a valid concern for the central claim. We will add these analyses in the revised manuscript.","revision_made":"yes","referee_comment":"[Section 4] Section 4 (experiments): Aggregate gains (17% top-accuracy, 37% MAPE drop) are presented across 621 matrices and 101 configurations, but the support for the projection assumption lacks isolated validation (e.g., no hold-out embedding test or distortion metric). This is load-bearing for the central claim that inexpensive features suffice to project unseen problems accurately."}],"tokens_in":1442,"tokens_out":369,"duration_ms":13530,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one thing to know is that this paper presents a modular embedding-based method for predicting which linear solvers will perform well on a given sparse matrix, using inexpensive features to place new problems into a space trained on observed performance.\n\nIt does a solid job showing improvements over classical feature-based models on a decent-sized public dataset from SuiteSparse with PETSc solvers. The 17% gain in top prediction accuracy and the drops in MAPE and error when using more features are concrete, and keeping the edge with reduced features is useful. Focusing on multilabel prediction and metrics that reflect relative performance rather than just picking the best one makes sense for real use.\n\nThe main soft spot is around the projection mechanism. The framework decouples learning the embedding from using cheap features to project new matrices, but the experiments give overall numbers without a specific test that holds the embedding constant and checks how well the cheap features preserve the relevant distances in that space. If that projection doesn't hold up for out-of-distribution cases, the advantage over simpler models shrinks. The stress-test note flags this correctly based on what's described.\n\nThis paper is for people in scientific computing who want practical tools for solver selection. Someone looking for ways to reduce the cost of trying many configurations would find the results relevant.\n\nIt should go to peer review. The idea is implementable and the data is public, so referees can check the details and ask for the missing ablations.","headline":"Embedding framework for solver performance prediction improves on baselines but lacks targeted validation for its core projection step.","tokens_in":2373,"tokens_out":354,"would_cite":false,"duration_ms":16533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"An embedding-based framework learns solver-problem relationships from performance data to predict optimal linear solvers more accurately than classical feature-based models.","keywords":["embedding-based solver selection","linear solver performance prediction","multilabel prediction","sparse matrix features","SuiteSparse collection","PETSc configurations","MAPE and nDCG metrics"],"falsifier":"Running the trained model on a fresh collection of matrices never seen during embedding construction or feature projection and finding that top-prediction accuracy and MAPE no longer improve over a classical feature-based baseline.","tokens_in":2671,"feed_emoji":"📊","tokens_out":671,"duration_ms":14154,"temperature":0.7,"pith_summary":"The paper establishes a modular framework that separates the learning of solver performance from the choice of matrix features, allowing inexpensive numerical features to map new problems into an embedding space built from observed solver runs. This separation matters because libraries offer over a hundred solver and preconditioner combinations whose relative speed varies sharply across sparse linear systems, yet picking the right one often requires expensive matrix analysis or exhaustive testing. Experiments across 621 SuiteSparse matrices and 101 PETSc configurations show the embedding approach raises top-prediction accuracy by 17 percent and cuts mean average percentage error by 37 percent when expensive features are available, while staying competitive and still 24 percent better on error metrics when restricted to cheap features only.","feed_headline":"Embedding method raises solver prediction accuracy 17%","feed_subtitle":"Projects cheap matrix features into a performance-learned space, cutting MAPE 37% and 1-error 46% across 621 matrices and 101 PETSc setups.","key_machinery":"The modular embedding space that decouples performance modeling from feature representation and downstream prediction.","core_discovery":"The framework learns solver-problem relationships directly from observed performance data inside a shared embedding space; inexpensive numerical features then project unseen matrices into that same space so that a downstream multilabel predictor can rank the 101 solver configurations by expected performance, measured with user-centric metrics such as MAPE and nDCG rather than classification accuracy alone.","pith_inferences":["The learned embedding space may expose natural clusters of matrices that share similar solver preferences, enabling transfer to problems outside the original training distribution.","The decoupling of modeling and features could be applied to other configuration-selection tasks such as choosing preconditioners for iterative methods or selecting time-stepping schemes.","If the embedding is updated online with new performance measurements, the predictor could adapt to hardware changes or evolving problem distributions without retraining from scratch."],"forward_implications":["Solver selection can be performed without recomputing expensive matrix properties for every new problem.","The same embedding can support multilabel ranking that reflects relative runtime rather than binary success or failure.","Performance remains competitive when the feature budget is deliberately reduced.","The approach scales to 101 distinct PETSc configurations across hundreds of matrices from a standard public collection."],"fun_headline_variants":["Embedding projects matrix features into performance space","17% accuracy increase for solver selection with embeddings","Embedding reduces MAPE 37% for 101 solver configurations","Performance data learns solver embeddings for prediction"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Inexpensive numerical features suffice to place new matrices accurately inside an embedding space that was built only from performance observations on a training set of matrices.","fun_headline_variants_meta":{"raw":{"variants":["Embedding projects matrix features into performance space","17% accuracy increase for solver selection with embeddings","Embedding reduces MAPE 37% for 101 solver configurations","Performance data learns solver embeddings for prediction"]},"model":"grok-4.3","cost_usd":0.008855,"raw_usage":{"total_tokens":3998,"prompt_tokens":697,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":88549500,"prompt_tokens_details":{"text_tokens":697,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3245,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":697,"tokens_out":56,"duration_ms":23036,"temperature":1.0,"reasoning_tokens":3245,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T05:11:52.485594+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the trained model on a fresh collection of matrices never seen during embedding construction or feature projection and finding that top-prediction accuracy and MAPE no longer improve over a classical feature-based baseline.","supporting_citations":[],"review_version":1}