{"id":"b38e6026-24f5-44b2-ad34-803cd7081853","arxiv_id":"2412.18972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces hardware-aware modifications to Model Spider for recommending pre-trained models on IoT devices, but provides no implementation or evaluation.","lead":"This paper proposes two ways to adapt the Model Spider recommender so it considers IoT hardware restrictions like CPU, memory, and power. The authors also lay out a research agenda for collecting benchmark data to rank pre-trained models on constrained devices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is a proposal only: no data, implementation, or evaluation support Fusion/Shadow, and Eq. (1)'s weighting semantics are internally inconsistent.","rationale":"We read the paper as a position/research-agenda contribution, not an empirical study. The prose is honest about the missing ground truth, and the proposed directions are coherent. However, the title and contribution 2 claim to 'introduce methods', and the abstract says 'introduce a novel, hardware-aware method for PTM selection.' For that claim to hold, the methods must at least be well-specified and trainable. They are not: no data, no implementation, and Eq. (1) is internally inconsistent. The reader's CONDITIONAL verdict is appropriate; we would not move to REJECT because the proposal is coherent and the agenda is valuable, and a corrected ranking procedure plus a small proof-of-concept could make it ACCEPT. Thus no change to the verdict is needed.","tokens_in":7885,"tokens_out":5799,"duration_ms":57885,"concrete_test":"Run a small-scale proof-of-concept: select 10–20 ImageNet PTMs and 3 IoT boards (e.g., Raspberry Pi 4, Jetson Nano, Odroid), measure per-(model,device) execution time, energy, and accuracy, and build ground-truth rankings using the stated weighted-Copeland procedure with w_i as additive weights. Then train Model Spider Fusion with and without hardware-spec tokens and compare ranking quality (Kendall tau) on held-out devices. If hardware-augmented ranking does not beat task-only ranking, the core premise fails; independently, recompute Eq. (1) with additive weights and confirm that exponent-based rankings differ, demonstrating the current equation is not a valid weighting scheme.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Model Spider Fusion and Shadow enable hardware-aware PTM recommendation—rests on two unsecured conditions. First, no dataset, implementation, or experiment is reported; §III-B admits the required ground-truth rankings do not exist, and Algorithm 1 is a proposed procedure, not executed data. Second, the one concrete formal component, Eq. (1) in §IV-B, is internally inconsistent: the text says each metric is assigned a weight w_i in [0,1] with Σw_i=1 and that metrics are 'scaled by adjustable w_i', but Eq. (1) uses w_i as an exponent on normalized metrics rather than as an additive weight. For normalized values below 1, larger 'weights' decrease the term; for values above 1, they increase it. This does not implement the stated weighted-Copeland aggregation and makes the proposed ground-truth rankings ill-defined. Because both proposed methods depend on such ground truth for training (Fusion) or for a hardware ranking (Shadow), the central hardware-aware recommendation claim is currently unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing pre-trained model (PTM) recommendation systems ignore hardware constraints, which is a key limitation for IoT deployment. It identifies two gaps: lack of IoT-specific inputs and lack of ground-truth rankings for IoT devices. To address the first gap, it proposes two modifications to the Model Spider framework: Model Spider Fusion, which appends hardware specification tokens to task tokens, and Model Spider Shadow, which combines task-based and hardware-based rankings using Copeland's method. It also proposes a dataset-creation methodology and a weighted objective function for building ground-truth rankings, and it lays out a research agenda with four research questions. The paper contains no implementation, dataset, or experimental evaluation; its contribution is a proposal plus an agenda.","tokens_in":8052,"tokens_out":2914,"duration_ms":30854,"significance":"If the proposed approaches were validated, they could address a real and recognized need: selecting PTMs for resource-constrained IoT devices without exhaustive on-device evaluation. The paper's strengths are its clear identification of a gap in current recommendation systems, its explicit discussion of the missing ground-truth data, and its concrete architectural suggestions that build on an existing, published framework. The use of established aggregation methods (Copeland's method) and existing sustainability metrics also provides a useful starting point. However, the central claim is currently a proposal only: no evidence is presented that hardware tokens improve ranking accuracy, and no ground-truth rankings are produced. The paper is honest about this limitation, but the title and abstract present the method as an accomplished contribution rather than a research agenda.","major_comments":[{"comment":"The text states that each metric is assigned a weight w_i in [0,1] with Σw_i=1 and that metrics are 'scaled by adjustable w_i', but Eq. (1) places w_i as an exponent: (HW_i(α)/T_i)^{w_i}. For normalized values below 1, a larger 'weight' decreases the term, which is the opposite of the intended weighting; for values above 1, it amplifies the term. The equation therefore does not implement the described weighted-Copeland aggregation, and the proposed ground-truth rankings built from this objective are ill-defined. Please revise the formulation so that the free parameters have the stated additive-weight semantics, or explicitly redefine what w_i means.","section":"§IV-B, Eq. (1)"},{"comment":"The central claim that Model Spider Fusion and Model Spider Shadow enable hardware-aware PTM recommendation is unsupported because no dataset, implementation, or experiment is reported. Section III-B explicitly states that ground-truth rankings for IoT devices do not exist and that Copeland aggregation over heuristic rankings is currently infeasible, and Algorithm 1 is a proposed procedure rather than an executed one. Both proposed methods depend on such ground truth: Fusion needs it for training the similarity mechanism, and Shadow needs it for forming a hardware ranking. Please either provide empirical validation (even on a small benchmark) or substantially reframe the contribution as a research agenda with clearly labeled hypotheses rather than as a working method.","section":"§III-B and §IV-A"},{"comment":"Algorithm 1's measurement procedure is underspecified. Lines 10-20 loop batch_size from 1 to 100, but the results of this sweep are neither aggregated nor compared, and the subsequent loop fixes batch_size to 32 with no explanation of how the two phases relate. There is also no repetition of measurements to account for variance in execution time, power, or CPU temperature, which makes it difficult to see how the metrics in Table II can be reported reliably. Please clarify the intended protocol, including the number of runs, statistical aggregation, and the role of the batch-size sweep.","section":"§IV-B, Algorithm 1"}],"minor_comments":[{"comment":"Both figures appear to be exported directly from a design tool and contain artifacts such as timestamps, URLs, and 'whimsical.com' watermarks; please replace them with clean publication-quality figures.","section":"Figures 1 and 2"},{"comment":"The 'Hardware Aware' column uses '×' for no and 'Yes' for yes, while the proposed methods are labeled 'MS Fusion' and 'MS Shadow' before these abbreviations are defined; please use consistent symbols and the full method names from §IV-A.","section":"Table I"},{"comment":"The notation 'P wi = 1' should be written as 'Σ w_i = 1'; as printed, the summation symbol is missing.","section":"§IV-B, Eq. (1) description"},{"comment":"Research questions RQ1-RQ4 are introduced but never revisited; since the paper is a research agenda, please add a paragraph explaining how each research question will be answered by the proposed data-collection and ranking methodology.","section":"§III-A and §III-B"},{"comment":"Several references are incomplete, lacking venues, volume/page numbers, or full author lists, for example [1], [2], [7], [14], [21], and [22]; please provide complete bibliographic entries.","section":"References"},{"comment":"The abstract claims to 'introduce a novel, hardware-aware method for PTM selection', but the body presents the method as a proposal without validation; please align the wording with the paper's actual contribution, e.g., 'propose'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closer to a vision or position paper than to a completed systems contribution. The authors honestly acknowledge the missing ground-truth data in Section III-B, and the formal issue in Eq. (1) is fixable. If the journal is open to research-agenda papers, a careful revision that reframes the claims and corrects the objective function could be viable; otherwise, the lack of any empirical component is a substantial scope concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a position paper, not a results paper. It identifies a real gap—PTM recommenders ignore device hardware—and sketches two plausible extensions to Model Spider. But it ships no implementation or evaluation, and the single formal piece, Eq. (1), is inconsistent with its own description. I'd treat it as a workshop-level research agenda, not as a validated method.\n\nWhat's actually new and useful: framing the problem as two gaps (hardware-blind inputs and missing ground-truth rankings) is fair and actionable. The two modifications—Fusion, which appends hardware tokens to task tokens, and Shadow, which combines task and hardware rankings with Copeland's method—are a reasonable design space. The paper is honest that ground-truth rankings don't exist yet, and the related-work table is a handy summary. The research questions (RQ1-RQ4) are concrete and worth pursuing.\n\nSoft spots, in order of seriousness. First, the abstract claims a 'novel hardware-aware method,' but there is no empirical validation: no dataset, no implementation, no comparison to existing recommenders. The load-bearing assumption—that static specs like CPU model and RAM, encoded as tokens, predict on-device performance without measurement—is plausible but untested. Second, Eq. (1) is internally inconsistent. The text says each metric is assigned a weight w_i in [0,1] with sum 1 and that metrics are 'scaled by adjustable w_i,' but the equation uses w_i as an exponent on (HW_i/T_i). For normalized values below 1, a larger exponent decreases the term; for values above 1, it increases it. That is not a weighted sum, and it makes the proposed ground-truth ranking ill-defined. Since both Fusion and Shadow depend on that ground truth, this is a load-bearing flaw, not a typo. Third, Algorithm 1's batch-size sweep from 1 to 100 before fixing 32 is unexplained and would need justification.\n\nThe reference list is relevant, though it leans on the authors' own group; that is not a problem given the topic.\n\nWho's this for? A reader starting in hardware-aware model selection might get a useful map of the space and a list of open problems. It does not deserve full peer review as a results paper, but it could be a solid workshop contribution or a 'call to action' if the authors are open to feedback. My recommendation: don't desk-reject with prejudice—send it to a workshop venue, or ask the authors to add a small proof-of-concept and fix Eq. (1), then review it as a regular paper.","headline":"A coherent position paper with a real gap and two sensible proposals, but no implementation or evaluation, and Eq. (1) is internally inconsistent; workshop-level, not yet a results paper.","tokens_in":8641,"tokens_out":4442,"would_cite":false,"duration_ms":46467,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes two hardware-aware extensions to the Model Spider recommender so that pre-trained models can be ranked for IoT devices from specification sheets instead of on-device trial runs.","keywords":["pre-trained model selection","IoT hardware","hardware-aware recommendation","Model Spider","transfer learning","model ranking","edge deployment","Copeland's method"],"falsifier":"Benchmark a set of pre-trained models fine-tuned on the same task across several IoT devices with different CPUs and memory sizes, recording actual execution time, power, and accuracy. Build Model Spider Fusion and Shadow rankings using only each device's spec sheet; if the predicted rankings correlate no better with measured performance than a task-only ranking does, the central claim is refuted.","tokens_in":7657,"feed_emoji":"⚙️","tokens_out":10303,"duration_ms":72659,"temperature":0.7,"pith_summary":"Pre-trained models make it cheap to build IoT applications, but choosing which model will actually run well on a small device usually requires slow, expensive on-device tests. This paper argues that existing model-recommendation systems ignore hardware constraints and proposes two ways to make the Model Spider ranking framework hardware-aware: Fusion, which appends device specifications to the task description, and Shadow, which creates separate task and hardware rankings and merges them with Copeland's method. The paper also specifies the missing ingredient—a ground-truth benchmark of models measured across IoT devices—and gives a data-collection procedure and a tunable ranking objective. If the proposals hold, an engineer could get a ranked shortlist of candidate models for a target device straight from its spec sheet, without running each model on the device first.","feed_headline":"Two tweaks make pre-trained-model ranking device-aware","feed_subtitle":"Fusion and Shadow fold CPU, RAM, and memory specs into ranking so engineers can skip on-device trials.","key_machinery":"The carrying object is Model Spider's token-and-attention similarity mechanism, in which each pre-trained model is encoded as a token, each task as a token, and a multi-head attention block scores how well a model fits a task. The paper's new components are a hardware extractor $\\Psi_h$ that turns device specifications into a hardware token, and Copeland's method for merging independent rankings. Model Spider Fusion appends the hardware token to the task token so that the similarity score sees both the task and the device; Model Spider Shadow computes separate task and hardware rankings and combines them by pairwise majority. The proposed ground-truth objective is $f(\\alpha) \\cdot \\sum_i (HW_i(\\alpha)/T_i)^{w_i}$, a weighted combination of model performance and normalized hardware metrics.","core_discovery":"The central claim is that hardware-aware model selection can be achieved by injecting device specifications (CPU model, RAM, memory size, I/O speed) into an attention-based model recommender: either by enriching the task token with a hardware token, or by treating hardware as a second ranking criterion and aggregating the two rank lists with Copeland's method. The authors define the missing ground truth as a benchmark of fine-tuned models measured on devices with metrics such as execution time, memory utilization, power consumption, CPU temperature, accuracy, and carbon footprint, all aggregated by a weighted objective. They state that no such ranked comparison data currently exists and that this absence is the main barrier to hardware-aware recommendation.","pith_inferences":["If spec-sheet tokens carry real predictive signal, the attention mechanism may learn latent clusters of similar devices (for example, devices with comparable memory behave alike), allowing rankings to transfer to unseen hardware.","A low-cost extension would be to append the same hardware tokens to simpler heuristic transferability scorers, giving a cheaper baseline against the learned recommenders.","An ablation study that masks one spec field at a time (CPU versus RAM versus I/O) would reveal which hardware attributes actually drive ranking changes—something the paper does not specify.","The proposed weighted objective is single-criterion after weighting; a Pareto-front treatment of execution time, power, and accuracy would let engineers see trade-offs rather than commit to fixed weights."],"forward_implications":["Engineers could obtain a ranked shortlist of candidate pre-trained models for a specific IoT device without running on-device forward passes.","The ranking objective can be tuned by adjusting the weights $w_i$ to prioritize speed, energy efficiency, or accuracy, making recommendations customizable per deployment.","A public ground-truth benchmark of fine-tuned models measured across IoT devices would enable studies of correlations between hardware specifications and model characteristics.","The dual-ranking design can be extended beyond hardware to include energy or cost selectors, so recommendations can reflect sustainability goals.","Once ground-truth rankings exist, the effect of quantization and distillation on hardware-aware rankings becomes directly testable."],"supporting_citations":[{"why":"Supplies the token-and-attention Model Spider framework that Fusion and Shadow modify.","marker":"[15]"},{"why":"Defines the power, time, and carbon-footprint metrics used as ranking criteria.","marker":"[20]"},{"why":"Introduces Copeland's method, used to merge the task and hardware rankings in Shadow.","marker":"[26]"},{"why":"Provides the formal treatment of Copeland's method that justifies the aggregation step.","marker":"[27]"},{"why":"Supplies a low-level hardware benchmarking methodology for single-board computers that shapes the data-collection procedure.","marker":"[19]"},{"why":"Documents performance bugs in model reuse that motivate the need for hardware-aware ranking.","marker":"[23]"},{"why":"Surveys hardware-aware neural architecture search and informs the weighted hardware terms in the ranking objective.","marker":"[28]"},{"why":"Demonstrates platform-aware optimization with a weighted objective, the template for the proposed ground-truth ranking formula.","marker":"[29]"}],"fun_headline_variants":["Hardware-aware model ranking skips device trials","Inject device specs into attention to rank PTMs","Copeland aggregation makes model picks device-aware","No labels? Hardware-aware PTM ranking from specs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a static specification of a device—CPU model, RAM, memory size, and I/O speed—is enough to predict how well a model will actually run on that device, so that rankings produced from spec sheets will match rankings measured on real hardware.","fun_headline_variants_meta":{"raw":{"variants":["Hardware-aware model ranking skips device trials","Inject device specs into attention to rank PTMs","Copeland aggregation makes model picks device-aware","No labels? Hardware-aware PTM ranking from specs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1279,"prompt_tokens":809,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":425,"tokens_out":470,"duration_ms":97010,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:20.592645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Benchmark a set of pre-trained models fine-tuned on the same task across several IoT devices with different CPUs and memory sizes, recording actual execution time, power, and accuracy. Build Model Spider Fusion and Shadow rankings using only each device's spec sheet; if the predicted rankings correlate no better with measured performance than a task-only ranking does, the central claim is refuted.","supporting_citations":[{"cited_title":"Model spider: Learning to rank pre-trained models efficiently,","cited_arxiv_id":null,"evidence_quote":"Supplies the token-and-attention Model Spider framework that Fusion and Shadow modify."},{"cited_title":"A comparative analysis for optimizing machine learning model deployment in IoT devices,","cited_arxiv_id":null,"evidence_quote":"Defines the power, time, and carbon-footprint metrics used as ranking criteria."},{"cited_title":"A reasonable social welfare function,","cited_arxiv_id":null,"evidence_quote":"Introduces Copeland's method, used to merge the task and hardware rankings in Shadow."},{"cited_title":"The copeland method: I.: Relationships and the dictionary,","cited_arxiv_id":null,"evidence_quote":"Provides the formal treatment of Copeland's method that justifies the aggregation step."},{"cited_title":"LwHBench: A low- level hardware component benchmark and dataset for single board computers,","cited_arxiv_id":null,"evidence_quote":"Supplies a low-level hardware benchmarking methodology for single-board computers that shapes the data-collection procedure."},{"cited_title":"Challenges and practices of deep learning model reengineering: A case study on computer vision","cited_arxiv_id":null,"evidence_quote":"Documents performance bugs in model reuse that motivate the need for hardware-aware ranking."}],"review_version":1}