{"id":"92547ba0-74ae-4884-a2ec-9f096e0caaa2","arxiv_id":"2412.02116","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ILASH-NAS automatically chooses where to branch a shared multi-task network, reducing search and inference energy on edge devices compared to AutoKeras.","lead":"This paper introduces ILASH, a way to build one neural network that handles several tasks at once by sharing early layers, and ILASH-NAS, a search method that picks how to share those layers. The authors report up to 16x lower search energy and 3x lower inference energy versus AutoKeras on facial and 2D semantic datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ILASH-Pred's cross-dataset transfer is unvalidated; the GN feature encoding cannot identify the target dataset, so the 10–16x efficiency claim rests on an assumption the paper never tests.","rationale":"The paper's central claim is that ILASH-Pred provides an energy-efficient NAS for multi-task edge applications by using a learned surrogate to avoid exhaustive search. The architecture itself is plausible: layer sharing is a reasonable way to reduce inference cost, and the ILASH-Heu search, while expensive, can generate a dataset of branch-quality labels. The reader correctly identified the load-bearing assumption: Auto-ILASH must transfer across datasets/task splits. My analysis agrees and strengthens the concern with a structural argument: the encoding in Algorithm 2 contains only local convolution hyperparameters, and the training data from MTFL/CelebA contains no feature that identifies UTKFace's accuracy distribution. Thus the surrogate's ability to predict GN on an unseen dataset is not just empirically unverified; it is not supported by the input representation. The only evidence offered (Table III) is in-distribution performance on a merged random split, which does not test the leave-one-out regime actually used in Sec. IV-A. If transfer fails, ILASH-Pred's speedup over ILASH-Heu disappears, and the comparison to AutoKeras becomes an unfair one that ignores the cost of generating surrogate training data. This is not an internal inconsistency in the proposed method; it is a missing validation step with a clear experimental protocol. I therefore keep the reader's CONDITIONAL verdict: the idea is worth pursuing, but acceptance should require the leave-one-out transfer test described above, plus an amortized-cost accounting that includes ILASH-Heu data generation.","tokens_in":14170,"tokens_out":4423,"duration_ms":49605,"concrete_test":"Reproduce the UTKFace leave-one-out fold: train Auto-ILASH (Decision Tree with the paper's hyperparameters) on the MTFL+CelebA ILASH Datasets only. For each task added to UTKFace, enumerate all branchable layers, compute Auto-ILASH's predicted GN for each, and run ILASH-Heu's temp-model training/evaluation on the same layers to get ground-truth GN. Report Spearman rank correlation between predicted and ground-truth GN, and compare ILASH-Pred's selected branch against (a) the trivial 'deepest allowed layer' heuristic and (b) the true best branch. If rank correlation is not significantly positive, or if ILASH-Pred's final accuracy is not better than the deepest-layer heuristic, the transfer premise fails and the efficiency claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ILASH-Pred's advertised 10–16x search-energy advantage over AutoKeras (Fig. 3, Sec. V.B) rests entirely on Auto-ILASH ranking branch points on a dataset it was never trained on: UTKFace uses MTFL+CelebA, and each Taskonomy split uses the other two (Sec. IV-A). Table III only reports 7:3 within-distribution regression metrics on the merged ILASH Dataset; it never reports held-out leave-one-target prediction error or rank correlation. The structural encoding (Algorithm 2) contains only kernel size, padding, and stride of the branch layer and its immediate neighbors, while the predicted GN (Eq. 3) includes the target task's validation accuracy. No dataset or task-identity feature is present, so the surrogate cannot know the accuracy scale of an unseen dataset. If the surrogate simply learns to rank by lrindex/lrtotal, ILASH-Pred degenerates to a near-trivial 'branch as deep as allowed' policy, and the claimed efficiency gain is not evidence of a transferable predictor. The amortized cost of the ILASH-Heu runs that generate training data is also excluded from the comparison, so even with transfer the headline efficiency claim requires an amortization argument that the paper does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ILASH, a layer-sharing architecture for multi-task neural networks, and ILASH-NAS, a neural architecture search framework with two modes: ILASH-Heu, which exhaustively evaluates candidate branch points using a hand-defined Goodness metric, and ILASH-Pred, which replaces those evaluations with a machine-learned predictor (Auto-ILASH) trained on data collected by ILASH-Heu. The authors evaluate on UTKFace, MTFL, CelebA, and Taskonomy, compare search efficiency with AutoKeras, and claim up to 16x reductions in energy, time, and CO2 emission during search while keeping final model accuracy stable.","tokens_in":14492,"tokens_out":5413,"duration_ms":54882,"significance":"If the claims hold, the paper offers a practical, energy-conscious NAS approach for multi-task edge applications, with a useful separation between heuristic data collection and predictor-based search. The work deserves credit for measuring inference energy on four real edge devices, for attempting a leave-one-out protocol for the predictor, and for comparing against AutoKeras, which is reasonably identified as the only publicly available multi-task NAS framework. The central contribution, however, rests on two premises that are not yet established: the cross-dataset transferability of Auto-ILASH and a fair amortized accounting of the cost of generating its training data. One results table also appears to duplicate earlier rows, which raises a data-integrity concern that must be resolved before the results can be accepted.","major_comments":[{"comment":"Table III reports MAE, MSE, RMSE, and R2 on the merged ILASH Dataset using a 7:3 split. This is an in-distribution evaluation. Algorithm 3, however, applies Auto-ILASH to a dataset it has never seen: UTKFace is predicted after training on MTFL and CelebA, and each Taskonomy split is predicted after training on the other two splits. The paper never reports held-out prediction error or rank correlation on these leave-one-target-out cases. Because the entire speedup of ILASH-Pred over ILASH-Heu rests on the predictor being useful on unseen datasets, please add leave-one-target-out regression and ranking metrics and compare them against a trivial baseline such as predicting the mean GN or using lrindex/lrtotal alone. Without this, the transfer premise is unsupported.","section":"Section IV.A and Table III"},{"comment":"The reported search efficiency compares ILASH-Pred runtime and energy (e.g., 1.58 hours and 0.287 kWh for Taskonomy split 1 in Table VI) against ILASH-Heu (60.2 hours and 6.34 kWh). But ILASH-Heu is not merely a baseline to be beaten; it is the data-generation step that produces the ILASH Dataset on which Auto-ILASH is trained. The 38x speedup claimed in Section V.C is therefore the cost of using the predictor after the training data already exist. Please provide an amortized comparison that includes the cost of the ILASH-Heu runs used to create the training data, or state how many target searches are needed for ILASH-Pred to break even. This is load-bearing for the headline 10-16x and 38x efficiency claims.","section":"Section V.B, Section V.C, and Table VI"},{"comment":"Table V appears to be a duplicate of Table IV: the Taskonomy split 1 block is identical to the UTKFace block (1.0, 0.957, 351.69, etc.), the split 2 block is identical to the MTFL block, and the split 3 block is identical to the CelebA block. If these are not actual Taskonomy measurements, the Taskonomy inference conclusions in Section V.C are unsupported. Please correct the table or provide the real measured values; if the values are correct, please explain why they coincide exactly.","section":"Table V"},{"comment":"The claim that final accuracy remains mostly stable and that ILASH-Pred is 10-16x more efficient is made from a single set of runs. No standard deviations, seed variation, or repeated-task-order statistics are reported, even though the paper itself states that task ordering affects results. Table IV reports only energy and FPS, not accuracy; accuracy appears only in Figure 3, which is not reproduced in the text. Please report means and variances over at least three seeds, and include the per-task accuracy values in a table so the accuracy-stability claim can be checked.","section":"Figure 3 and Tables IV and VI"}],"minor_comments":[{"comment":"The text contains typos such as 'verage power' instead of 'average power'; please proofread the manuscript.","section":"Section II"},{"comment":"The abbreviation 'ILSH Pred' appears several times instead of 'ILASH-Pred'; please make the notation consistent.","section":"Section IV.D"},{"comment":"The sentence before Equation (5) says 'We calculate the MAE using the following equation,' but the equation defines the MSE; please correct the wording.","section":"Equation (5)"},{"comment":"The encoding loop accesses M odel.layers[idx + i] for i = -1 and i = 1, which is out of range when idx is the first or last layer; please specify boundary handling such as clamping or skipping.","section":"Algorithm 2"},{"comment":"In the ILASH-Heu row for CelebA, the reported KWh-PUE value is 8.31 x 10^-8 while the CO2 emission is 0.793 x 10^-8 lbs, which is inconsistent by a factor of roughly ten; please verify the decimal point.","section":"Table IV, CelebA row"},{"comment":"The Auto-ILASH hyperparameter list for Random Forest mentions 'max features' twice with different values; please clarify the intended setting.","section":"Section IV.A"},{"comment":"The value of the Goodness weight Gth is never reported, and no sensitivity analysis is provided; since Gth controls the trade-off between accuracy and layer position, please state the value used in all experiments.","section":"Equation (3)"}],"recommendation":"major_revision","confidential_remarks":"The duplication of Table V with Table IV is a serious data-integrity issue and should be checked before further review; it may indicate a pipeline error. I would also ask the authors to supply the raw per-task accuracy values, the value of Gth, and the held-out leave-one-target prediction errors. Finally, the comparison with AutoKeras should state clearly that the search spaces differ, since ILASH-Pred searches branch points in a fixed MobileNet-based architecture while AutoKeras searches a broader space; this is not necessarily fatal, but it affects how the efficiency ratio should be interpreted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does one thing well: it builds and tests a layer-sharing multi-task architecture (ILASH) that genuinely reduces inference energy on edge devices, and the comparison with AutoKeras on UTKFace/MTFL/CelebA shows consistent efficiency gains. That part is real, and the edge-device energy tables (Tables IV and V) are the strongest contribution.\n\nThe novelty is modest. Branching multi-task networks are standard, and using an ML surrogate to pick branch points is a known NAS idea. What is new is the specific combination plus the green/energy framing, but that does not make the search framework itself a breakthrough.\n\nThe soft spot is the predictive component. The surrogate (Auto-ILASH) is trained on heuristic runs from two datasets and applied to a third (leave-one-out), but the paper never reports how well the surrogate predicts on the held-out dataset. Table III reports only within-distribution regression error on the merged ILASH Dataset. Meanwhile the feature encoding (Algorithm 2) contains only kernel/pad/stride of the branch layer and its neighbors—no dataset or task identity. The Goodness metric includes validation accuracy, which the encoding cannot infer for an unseen dataset. So it is entirely possible the surrogate is just learning to branch as deep as allowed, and the 'intelligent' search is a depth heuristic in disguise. The paper does not rule this out.\n\nA second issue is the headline 16x search-energy claim. It compares ILASH-Pred to AutoKeras without counting the energy spent by ILASH-Heu to generate the surrogate training data. Table VI shows ILASH-Heu takes tens of hours and several kWh; an amortization calculation would reduce the reported advantage substantially, though the advantage may not vanish.\n\nMinor issues: no error bars, no code release, and the results are sensitive to task order. These are fixable.\n\nOverall, the architecture and inference results deserve attention, but the search-efficiency claim needs validation of transfer and amortization. I'd send it to review with a request for those experiments, rather than reject it outright.","headline":"Useful energy-efficiency results for layer-shared multi-task models, but the predictive NAS speedup rests on an unvalidated cross-dataset transfer assumption.","tokens_in":14939,"tokens_out":4135,"would_cite":true,"duration_ms":42617,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that layer-sharing plus a learned predictor can make multi-task neural architecture search up to 16 times more energy-efficient.","keywords":["neural architecture search","multi-task learning","layer sharing","edge computing","energy efficiency","CO2 emission","predictive NAS"],"falsifier":"Run ILASH-Pred on UTKFace using an Auto-ILASH model trained only on MTFL and CelebA, and compare the chosen branch points, final task accuracies, and total search energy against ILASH-Heu on UTKFace; if the predictor does not beat the heuristic in both accuracy and energy, the transfer premise is unsupported. A simpler check is to measure the Decision Tree's MAE on held-out UTKFace predictions when the model was trained without UTKFace.","tokens_in":13996,"feed_emoji":"⚡","tokens_out":4153,"duration_ms":41636,"temperature":0.7,"pith_summary":"The paper tries to establish that a new class of layer-shared neural networks (ILASH) plus a predictive architecture search (ILASH-NAS) can make multi-task edge AI much cheaper. The key idea is to share a trunk of layers among tasks and branch off only near the output, so one model handles several tasks at once. The search is sped up by a machine-learning model that predicts which layer is the best branching point, avoiding the repeated candidate training of classic NAS. If the claim holds, engineers can build multi-task models for phones and sensors with far less energy, time, and carbon cost, and the deployed models also run at higher frame rates.","feed_headline":"Layer-shared NAS cuts search energy up to 16x","feed_subtitle":"A predictive search framework picks branch points without retraining, making multi-task edge AI far cheaper.","key_machinery":"The search is driven by a Goodness metric, $GN = acc \\times (1 - G_{th}) + \\frac{lr_{index}}{lr_{total}} \\times G_{th}$, which balances validation accuracy against the relative depth of the branching layer using a user-set green threshold. ILASH-Pred uses a Decision Tree regressor (Auto-ILASH) trained on logs of heuristic searches; each candidate branching layer is encoded by the kernel size, padding, and stride of that layer and its two neighbors. The regressor predicts GN for every legal branch, the best branch is chosen without training the candidate, and the final branched model is trained once on all tasks together.","core_discovery":"The central claim is that a hybrid layer-sharing architecture, built by a branching heuristic and then searched by a learned goodness predictor, can match the accuracy of a general multi-task NAS like AutoKeras while using 10–16 times less energy, CO2 emission, and search time, and can run at several times the frame rate on edge devices. On the Taskonomy 2D-semantic tasks, the predictive search is about 38 times faster than the paper's own heuristic search with similar final model error. The paper also claims that the final ILASH models cut inference energy by about three times compared with AutoKeras-generated models.","pith_inferences":["If the transfer assumption holds, the predictor could be reused across similar datasets without re-running the costly heuristic search, but the paper does not report held-out prediction error; reporting that error would show how far the approach generalizes.","The layer encoding could be enriched with channel counts, receptive field, or FLOPs to make the predictor more portable, which is a natural testable extension.","The Goodness metric collapses accuracy and layer position into one scalar; a multi-objective view might reveal that accuracy and efficiency trade off differently than GN suggests, especially when different tasks have very different difficulties."],"forward_implications":["Multi-task models for edge devices could be built with a single shared trunk plus task-specific branches, shrinking model size and raising frames per second.","Architecture search becomes cheap enough to run on modest hardware, lowering the carbon footprint per model.","The same branching-plus-prediction scheme could be applied to other domains such as speech, video, or NLP once a heuristic search has generated a training set for the predictor.","For 2D semantic tasks, the paper claims to be the first multi-task NAS that supports them, with roughly 38 times faster search than its own heuristic baseline."],"supporting_citations":[{"why":"AutoKeras is the comparison baseline; the paper measures ILASH's search efficiency and model quality against its greedy, random, and Bayesian tuners.","marker":"[31]"},{"why":"Supplies the AutoKeras library details used in the experimental comparison and cited alongside [31].","marker":"[32]"},{"why":"Provides the energy and CO2 emission equations (1) and (2) used to compute kWh-PUE and CO2e for both search and inference.","marker":"[37]"},{"why":"MobileNet is the base model for the classification/regression tasks, and its architecture shapes the layer-sharing search space.","marker":"[42]"},{"why":"Taskonomy is the dataset used to evaluate 2D semantic tasks, the setting where ILASH claims to be the only multi-task NAS framework.","marker":"[17]"}],"fun_headline_variants":["Predictive NAS cuts multi-task search energy 16x","Layer-sharing NAS: 16x less energy for multi-task AI","AI search framework cuts energy 16x for edge tasks","Multi-task edge AI: predictive NAS cuts energy 16x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speedup rests on the assumption that a goodness predictor trained on heuristic search logs from some datasets will reliably rank branching choices on a different dataset; the paper never reports the predictor's error on the held-out dataset.","fun_headline_variants_meta":{"raw":{"variants":["Predictive NAS cuts multi-task search energy 16x","Layer-sharing NAS: 16x less energy for multi-task AI","AI search framework cuts energy 16x for edge tasks","Multi-task edge AI: predictive NAS cuts energy 16x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2720,"prompt_tokens":908,"completion_tokens":1812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1742}},"tokens_in":524,"tokens_out":1812,"duration_ms":14430,"temperature":1.0,"reasoning_tokens":1742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:48:35.446651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ILASH-Pred on UTKFace using an Auto-ILASH model trained only on MTFL and CelebA, and compare the chosen branch points, final task accuracies, and total search energy against ILASH-Heu on UTKFace; if the predictor does not beat the heuristic in both accuracy and energy, the transfer premise is unsupported. A simpler check is to measure the Decision Tree's MAE on held-out UTKFace predictions when the model was trained without UTKFace.","supporting_citations":[{"cited_title":"Auto-keras: An efficient neural architecture search system,","cited_arxiv_id":null,"evidence_quote":"AutoKeras is the comparison baseline; the paper measures ILASH's search efficiency and model quality against its greedy, random, and Bayesian tuners."},{"cited_title":"Autokeras: An automl library for deep learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the AutoKeras library details used in the experimental comparison and cited alongside [31]."},{"cited_title":"Energy and policy consider- ations for modern deep learning research,","cited_arxiv_id":null,"evidence_quote":"Provides the energy and CO2 emission equations (1) and (2) used to compute kWh-PUE and CO2e for both search and inference."},{"cited_title":"Taskonomy: Disentangling task transfer learning,","cited_arxiv_id":null,"evidence_quote":"Taskonomy is the dataset used to evaluate 2D semantic tasks, the setting where ILASH claims to be the only multi-task NAS framework."}],"review_version":1}