{"id":"3683296d-c263-4f9c-8c6e-a1d59c23ab74","arxiv_id":"2607.00154","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"EVOTS evolves modular Transformer-like architectures for multivariate time-series forecasting and reports competitive or improved MSE versus a fixed Transformer baseline on ETT datasets across multiple settings and horizons.","lead":"The paper introduces EVOTS, an evolutionary search framework that automatically designs Transformer-like models for time series forecasting tasks. A smart generalist might read it to see how automated architecture search can adapt models to different forecasting problems without manual design rules.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Performance improvements rest on single-run point estimates from a stochastic search process without reported variance or significance tests.","rationale":"The reader's weakest assumption targets the genome representation's ability to explore without hand-crafted rules, which is a prerequisite but not the direct support for the performance claim. The load-bearing uncertainty for the headline result is the reliability of the MSE numbers themselves given the stochastic optimizer. This is an internal experimental-design issue rather than an external-consensus disagreement, and it can be settled by the proposed multi-run check without requiring new datasets.","tokens_in":1725,"tokens_out":339,"duration_ms":13811,"concrete_test":"Re-execute the full EVOTS evolutionary search (same population size, generations, and mutation rates) five times on ETTh1 multivariate-to-multivariate with horizon 96 using distinct random seeds; compute mean and std of final best-model MSE for each run. If the baseline MSE lies inside the 95% CI of the evolved-model MSEs in at least three of the five runs, the improvement claim lacks robustness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim states that evolved architectures achieve competitive and, in several cases, improved MSE versus a strong Transformer baseline in the multivariate-to-multivariate setting. Evolutionary search is inherently stochastic (population initialization, mutation, crossover, and selection all depend on random seeds). The abstract provides no indication of multiple independent evolutionary runs, standard deviations, or statistical tests comparing the evolved models to the baseline. If the reported improvements are from single executions, they may reflect lucky search trajectories rather than systematic superiority enabled by the modular genome and repair mechanism.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces EVOTS, an evolutionary neural architecture search framework for discovering task-adaptive Transformer-like models for multivariate time-series forecasting. Architectures are encoded via a modular genome representation with a repair mechanism to enforce validity; the method is evaluated on the four ETT benchmark datasets under univariate-to-univariate, multivariate-to-univariate, and multivariate-to-multivariate settings with horizons 96/192/336/720. The central empirical claim is that, in the multivariate-to-multivariate setting, the evolved models achieve competitive and in several cases improved MSE relative to a strong Transformer baseline, while additional analyses address performance differences across settings and wall-clock training time.","tokens_in":1819,"tokens_out":557,"duration_ms":11855,"significance":"If the empirical claims hold under proper statistical controls, the work would provide evidence that evolutionary search with a modular genome can discover flexible, high-performing Transformer variants for time-series forecasting without hand-crafted rules, addressing an underexplored area. The practical runtime results and multi-setting evaluation would further strengthen the contribution, though the current presentation leaves the robustness of the performance gains open to question.","major_comments":[{"comment":"Abstract and experimental results section: the reported MSE improvements in the multivariate-to-multivariate setting are presented as single point estimates from the evolutionary search process. Because initialization, mutation, crossover, and selection are stochastic, the absence of multiple independent runs, standard deviations, or statistical significance tests (e.g., paired t-tests or Wilcoxon tests against the baseline) makes it impossible to determine whether observed gains reflect systematic superiority or lucky trajectories; this directly undermines the central claim.","section":"Abstract / Experimental results"},{"comment":"Experimental setup (baseline implementation and data handling): the abstract states competitive MSE results but provides no details on how the Transformer baseline was implemented, whether hyperparameter search was performed for it, the precise train/validation/test splits, or any statistical testing protocol. These omissions are load-bearing for the comparison claim and must be supplied with sufficient specificity to allow reproduction.","section":"Experimental results"}],"minor_comments":[{"comment":"The description of the repair mechanism and genome representation would benefit from a small illustrative example (e.g., a before/after repair diagram) to clarify how structural validity is maintained without hand-crafted rules.","section":"Method"},{"comment":"Wall-clock training time is reported as a coarse cost indicator; it would be clearer to also report the number of evaluated architectures or total search budget in FLOPs or GPU-hours for context.","section":"Experimental results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which highlight important aspects of statistical robustness and reproducibility. We address each major comment below and will revise the manuscript to incorporate the suggested improvements.","responses":[{"response":"We agree that single-run point estimates limit the strength of the claims given the stochastic nature of the evolutionary process. In the revised manuscript, we will report results from multiple independent evolutionary runs (at least five per setting), include means and standard deviations, and add statistical significance tests (Wilcoxon signed-rank tests) comparing evolved models to the baseline. These additions will be placed in the experimental results section and reflected in an updated abstract.","revision_made":"yes","referee_comment":"[Abstract / Experimental results] Abstract and experimental results section: the reported MSE improvements in the multivariate-to-multivariate setting are presented as single point estimates from the evolutionary search process. Because initialization, mutation, crossover, and selection are stochastic, the absence of multiple independent runs, standard deviations, or statistical significance tests (e.g., paired t-tests or Wilcoxon tests against the baseline) makes it impossible to determine whether observed gains reflect systematic superiority or lucky trajectories; this directly undermines the central claim."},{"response":"We acknowledge the need for greater specificity. The revised version will expand the experimental setup subsection to include: (i) the exact Transformer baseline architecture and hyperparameters, (ii) whether and how hyperparameter tuning was performed for the baseline, (iii) the precise train/validation/test split ratios and any preprocessing steps for the ETT datasets, and (iv) the full statistical testing protocol. This information will enable full reproduction of the comparisons.","revision_made":"yes","referee_comment":"[Experimental results] Experimental setup (baseline implementation and data handling): the abstract states competitive MSE results but provides no details on how the Transformer baseline was implemented, whether hyperparameter search was performed for it, the precise train/validation/test splits, or any statistical testing protocol. These omissions are load-bearing for the comparison claim and must be supplied with sufficient specificity to allow reproduction."}],"tokens_in":1455,"tokens_out":447,"duration_ms":14151,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that EVOTS uses an evolutionary search with a modular genome representation and repair mechanism to discover Transformer-like architectures for time series forecasting. It gets some improved MSE numbers over a baseline in the multivariate-to-multivariate case on ETT data.\n\nThe new part is applying this kind of search to the time series domain with that particular genome setup, which avoids needing manual design rules. It does well by testing across univariate-to-univariate, multivariate-to-univariate, and multivariate-to-multivariate, plus different horizons up to 720 steps.\n\nThe soft spot is the reporting of results. Evolutionary search is stochastic, so single point estimates without variance or multiple runs make it hard to trust that the gains come from the approach rather than lucky searches. No details on baseline implementation or stats tests either.\n\nThis is for researchers focused on architecture search in forecasting. It might give ideas on how to encode the search space, but it's not a big shift for the field.\n\nI would send it to peer review so the full paper can be checked for those experimental details.","headline":"EVOTS applies evolutionary NAS to time-series Transformers via a modular genome and repair step, but the MSE gains are single-run point estimates without variance or tests.","tokens_in":2307,"tokens_out":297,"would_cite":false,"duration_ms":20060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Evolutionary search discovers competitive Transformer architectures for time series forecasting without hand-crafted rules.","keywords":["evolutionary neural architecture search","transformer models","time series forecasting","multivariate forecasting","neural architecture search","ETT benchmarks"],"falsifier":"If repeated evolutionary runs on the ETTm1 or ETTm2 datasets in the multivariate-to-multivariate setting produce architectures whose MSE is consistently higher than the fixed Transformer baseline across the tested horizons, the performance claim would be falsified.","tokens_in":2616,"feed_emoji":"🧬","tokens_out":637,"duration_ms":15880,"temperature":0.7,"pith_summary":"The paper introduces an evolutionary neural architecture search framework that encodes Transformer-like models via a modular genome for multivariate time series forecasting tasks. It shows that the resulting architectures achieve mean squared error competitive with or better than a fixed Transformer baseline specifically in the multivariate-to-multivariate setting across ETT family datasets and multiple forecast horizons. This matters because it indicates that search can adapt model structure to forecasting needs rather than relying on fixed designs. The framework includes a repair mechanism to keep generated architectures valid during evolution and reports training times as a measure of practicality.","feed_headline":"Evolved Transformers match or beat baseline on time series tasks","feed_subtitle":"Modular genome and repair mechanism let search discover task-adaptive models competitive on ETT multivariate benchmarks.","key_machinery":"modular genome representation with repair mechanism that enforces structural validity to enable exploration of diverse architectures","core_discovery":"EVOTS encodes architectures as modular genomes allowing flexible composition of attention, feed-forward, and projection components, then applies evolutionary search with a repair mechanism that enforces validity. In experiments on ETTh1, ETTh2, ETTm1, and ETTm2 under univariate-to-univariate, multivariate-to-univariate, and multivariate-to-multivariate regimes with horizons 96 to 720, the evolved models reach competitive or improved MSE relative to a strong Transformer baseline in the multivariate-to-multivariate case.","pith_inferences":["The same genome-plus-repair approach could be tested on other sequence tasks such as natural language or audio modeling.","If the evolved models generalize to unseen datasets outside the ETT family, the method would reduce dependence on task-specific manual tuning.","Extending the genome to include additional operator types or different evolutionary selection pressures would be a direct next experiment."],"forward_implications":["Evolved architectures remain competitive across univariate-to-univariate and multivariate-to-univariate settings as well.","Performance holds for forecast horizons of 96, 192, 336, and 720 steps.","Wall-clock training times stay within ranges that indicate practical computational cost.","The search process operates without hand-crafted design rules yet produces valid high-performing models."],"fun_headline_variants":["Evolutionary search finds Transformers for ETT time series forecasting","Modular genomes support evolutionary search of time series models","EVOTS achieves competitive results on ETT multivariate forecasting","Repair mechanism ensures valid evolutionary Transformer architectures"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The modular genome representation together with the repair mechanism enables effective exploration of a diverse architecture space without relying on hand-crafted design rules.","fun_headline_variants_meta":{"raw":{"variants":["Evolutionary search finds Transformers for ETT time series forecasting","Modular genomes support evolutionary search of time series models","EVOTS achieves competitive results on ETT multivariate forecasting","Repair mechanism ensures valid evolutionary Transformer architectures"]},"model":"grok-4.3","cost_usd":0.005172,"raw_usage":{"total_tokens":2531,"prompt_tokens":709,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":51724500,"prompt_tokens_details":{"text_tokens":709,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1763,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":709,"tokens_out":59,"duration_ms":12353,"temperature":1.0,"reasoning_tokens":1763,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T19:48:45.211552+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If repeated evolutionary runs on the ETTm1 or ETTm2 datasets in the multivariate-to-multivariate setting produce architectures whose MSE is consistently higher than the fixed Transformer baseline across the tested horizons, the performance claim would be falsified.","supporting_citations":[],"review_version":1}