{"id":"c05a2d23-c086-4f90-8cff-93051ea48ac1","arxiv_id":"1908.02880","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A surrogate-assisted genetic algorithm that excludes already-seen items outperforms a plain genetic algorithm and random search on two of three 2D benchmark functions, but not on the third.","lead":"This paper tests a recommender system that uses a genetic algorithm guided by a learned approximation of user taste, instead of asking the user to rate many items. On two of three toy benchmarks it beats a standard genetic algorithm and random search with few real evaluations, but on the third benchmark it loses to random search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's unqualified performance claim is contradicted by the paper's own Schwefel results, and the comparison uses parameters tuned in-sample on the same benchmarks, leaving the headline advantage undemonstrated.","rationale":"The reader correctly identified both the Schwefel contradiction and the limited external validity. My stress-test lands on a related but more internal weakness: even for the benchmarks where SAGRS appears to win, the comparison does not establish an algorithm-level advantage because every SAGRS configuration was selected using the same functions on which it is then evaluated. This is a correctness risk for the central claim, not merely a question of how far the results transfer to real recommender systems. I credit the paper for explicitly listing its limitations in Section 5, including the unavailability of an interactive-system claim and the untested generalization beyond the three benchmarks. The framework is plausible and the experiments are transparent enough to be re-run, but the abstract overstates what the evidence supports. Since the conditional verdict already requires qualifying the central claim and improving the evaluation, my concern does not move the verdict; it reinforces it. The only change I would emphasize is that the qualification must cover the in-sample parameter selection, not just the benchmark-to-human transfer.","tokens_in":10009,"tokens_out":3132,"duration_ms":37235,"concrete_test":"Re-run the Section 4.3 comparison with parameters selected out-of-sample: split the benchmark functions into tuning versus test sets (for example, tune on Bohachevsky and Ackley, then test on Schwefel plus at least one unseen function such as Rastrigin or Griewank), or use nested resampling within each function. Use at least 50 independent seeds and report the mean and 95% bootstrap confidence interval of best true-objective fitness at matched evaluation budgets. If, on held-out functions, SAGRS is not better than the random recommender with the same surrogate, the abstract's outperformance claim must be restricted to the tuned benchmarks and to LSM on easier functions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's central comparison supports the abstract only if SAGRS consistently beats the genetic algorithm and the random recommender. On Schwefel (Figure 8) the authors state the opposite: 'the Random Recommender shows a comparably better performance than the SAGRS' and SAGRS only 'slightly outperform[s]' GA. Thus the headline claim 'given a very limited amount of allowed evaluations on the true objective, our approach outperforms these baseline methods' is false as stated. The comparison is also in-sample: all SAGRS meta-parameters (evaluation rate, population handling, suggestions per cycle, number of cycles) are selected in Section 4.2 using the same three benchmarks that are then used for the comparison in Section 4.3, as indicated by the legends citing the previously evaluated optimal settings. The GA and random recommender configurations are likewise chosen for the comparison rather than independently tuned or held out. With only 10 repetitions and box plots, no significance or confidence statements support the ordering. The acknowledged limitations in Section 5 (only R^2, no human evaluation, need to show generalization beyond benchmarks) further narrow the claim. A load-bearing requirement for the central claim is that the advantage is a property of the algorithm, not of per-benchmark parameter selection; this is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a surrogate-assisted genetic recommender system (SAGRS) in which a pool of user-evaluated items is used to train a surrogate model (polynomial regression or radial basis function network), the surrogate serves as fitness function for a genetic algorithm that proposes new items, and newly evaluated items are removed from the search space before the surrogate is updated. Human evaluation is replaced by three continuous benchmark functions (Bohachevsky, Ackley, Schwefel) on R^2, and the system is compared with a conventional genetic algorithm and a 'Random Recommender'. The authors tune system parameters in Section 4.2 and report in Section 4.3 that SAGRS outperforms the baselines on Bohachevsky and Ackley, while on Schwefel the Random Recommender performs comparably or better. Section 5 explicitly acknowledges that human evaluation was not tested, that only two-dimensional benchmarks were used, and that generalization beyond the chosen functions remains to be shown.","tokens_in":10307,"tokens_out":5081,"duration_ms":56632,"significance":"If the claimed advantage were established, the paper would be a useful step toward interactive recommender systems that combine content-based recommendation with surrogate-assisted evolutionary search, especially because it explicitly removes already evaluated items from the search space and updates the surrogate after each suggestion. The manuscript is honest in reporting the Schwefel result where the baseline is competitive, and it clearly lists limitations in Section 5. However, as presented, the central empirical claim is not fully supported: the headline statement is contradicted by one of the paper's own benchmark comparisons, and the parameter tuning is performed in-sample on the same benchmarks used for the final comparison. The architectural idea is promising, but the current evidence is insufficient to support the unqualified claim of superiority.","major_comments":[{"comment":"The abstract's claim that 'given a very limited amount of allowed evaluations on the true objective, our approach outperforms these baseline methods' is contradicted by the paper's own Schwefel results: Section 4.3 states that on Schwefel the SAGRS only 'slightly outperform[s]' the GA and that 'the Random Recommender shows a comparably better performance than the SAGRS.' The headline claim should be restricted to the benchmarks and configurations where it is actually demonstrated, or replaced by an aggregate comparison with appropriate confidence statements.","section":"Abstract; §4.3, Fig. 8"},{"comment":"The SAGRS meta-parameters (evaluation rate, population-reset choice, number of suggestions per cycle, number of recommendation cycles) are selected in Section 4.2 using the same three benchmark functions that are then used for the comparison in Section 4.3, and the figure legends explicitly cite these previously evaluated settings. The GA and Random Recommender baselines are configured for the comparison rather than independently tuned or validated on held-out functions. The reported advantage is therefore in-sample and may reflect per-benchmark tuning rather than an intrinsic property of the algorithm. Please add a held-out validation, cross-validation over parameter choices, or a sensitivity analysis showing that the conclusions are robust across a parameter grid.","section":"§4.2 and §4.3"},{"comment":"The formalization in Section 3 defines the goal as arg max of the utility function and states that individuals with the highest predicted fitness are recommended, while Section 4.1 states that all benchmark functions 'are to be minimized with a best fitness value of 0' and all comparison plots display lower 'Best Fitness' as better. No sign transformation or maximization/minimization conversion is specified. This ambiguity is load-bearing for every figure and should be resolved, for example by defining the utility as the negative of the benchmark objective or by explicitly converting minimization to maximization.","section":"§3 vs §4.1"},{"comment":"All quantitative comparisons are based on only 10 repetitions per configuration, and no significance tests or confidence intervals are reported. Given the visible overlap between many box plots, especially in Figure 8, claims such as one model 'clearly yields profit' or that SAGRS outperforms the Random Recommender need at least paired non-parametric tests across the 10 runs (e.g., a Wilcoxon signed-rank test) to be supported.","section":"§4.2 and §4.3"}],"minor_comments":[{"comment":"There is a typo: 'aurrogate-assisted genetic algorithm' should read 'surrogate-assisted genetic algorithm.'","section":"§3, first paragraph"},{"comment":"The 'Random Recommender' is not random search in the usual sense, because it recommends the initially best-estimated individuals according to a surrogate model; the name and the exact baseline behavior should be explained more explicitly to avoid confusion.","section":"§4.3"},{"comment":"The definition of 'accepted suggestion' as 'evaluated better than the worst evaluated item at that time' is given only in prose; a formal definition would improve reproducibility.","section":"§4.2"},{"comment":"The figure legends carry all the essential parameter settings and are very long; a table summarizing the exact configuration for each model, benchmark, and baseline would make the experimental setup much easier to check.","section":"Figures 3–8"},{"comment":"The evaluation-budget accounting is inconsistent: Section 4.2.3 mentions about 1000 real fitness evaluations at maximum, while with an initial pool of 100 items and 100 cycles of 4 suggestions the budget appears to be 500; Section 5 also refers to 'the optimized amount of 1000 evaluations.' Please reconcile these numbers.","section":"§4.2.3 and §5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest GECCO Companion paper with a legitimate new combination—surrogate-assisted genetic search for recommendations where already-evaluated items are removed from the search space—but the abstract's performance claim is contradicted by the paper's own Schwefel results, and the comparison is in-sample. Worth a skim for the idea, not for the evidence.\n\nWhat's actually new: the specific twist of treating recommendation as optimization on S \\ S' with a fixed evaluation budget and updating the surrogate as items are recommended is, as far as the citations show, not present in prior surrogate-assisted interactive GA work. The paper also does something honest: it explicitly separates the algorithm's capability from human interaction, and in Section 5 it says real-world applicability cannot be stated. It compares two surrogate families (polynomial least squares, RBF) and systematically sweeps evaluation rate, population handling, suggestions per cycle, and number of cycles. The authors deserve credit for that.\n\nWhere it gets soft. First, the abstract says the approach outperforms the baselines 'given a very limited amount of allowed evaluations.' On Schwefel, Section 4.3 says the opposite: the Random Recommender performs better than the SAGRS. So the headline claim only holds for the easier benchmarks, and even there it is conditional on tuned settings. Second, the parameterization is chosen in Section 4.2 using the same three benchmark functions that are then used in Section 4.3; the legends even cite the previously evaluated 'optimal' settings. That makes the comparison in-sample, and with only 10 repetitions per setup there are no significance or confidence statements to lean on. Third, there is a sign inconsistency: Section 3 formalizes the recommendation objective as arg max of the utility function, while Section 4.1 says the benchmark functions are to be minimized. That is a real formalization slip, even if the empirical plots make the intended direction clear. Finally, as the authors themselves note, everything is done on smooth 2D continuous benchmarks; no human evaluations, no high-dimensional or discrete item spaces, no noise or non-stationarity. The contribution is a proof-of-concept, not a tested recommender.\n\nWho this is for: someone working on interactive evolutionary computation or surrogate-assisted search who wants one concrete example of an exclusion-based recommendation loop. It does not move the needle for recommender systems broadly.\n\nRecommendation: it deserves peer review—a desk rejection would be too harsh—but it needs a revised abstract, a fix for the sign issue, and ideally a held-out evaluation or at least significance testing. I would send it to review, not publish as-is.","headline":"Modest workshop paper with a legitimate new search-space twist; the headline performance claim does not survive the authors' own Schwefel results and the comparison is in-sample.","tokens_in":10795,"tokens_out":2816,"would_cite":false,"duration_ms":28367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68W50","68T20"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a surrogate-assisted genetic recommender system can outperform conventional genetic algorithms and random search under a very limited budget of true user evaluations, by fitting a model to evaluated items and using…","keywords":["surrogate model","genetic algorithm","recommender system","interactive evolutionary computation","fitness approximation","polynomial regression","radial basis function","benchmark functions"],"falsifier":"Run the same comparison on a real or high-dimensional item space and check whether the surrogate-assisted recommender still beats random search; the paper already contains one disconfirming case, since on the Schwefel benchmark the random recommender matches or beats the surrogate-assisted system.","tokens_in":9822,"feed_emoji":"🧬","tokens_out":5573,"duration_ms":56131,"temperature":0.7,"pith_summary":"This paper claims that a recommender system built from a surrogate-assisted interactive genetic algorithm can find items a user will like after very few evaluations of the user's true preferences. The authors simulate the user by three continuous benchmark functions on $\\mathbb{R}^2$ and fit a surrogate, either least-squares polynomial regression or a radial-basis-function network, to an initial pool of 100 evaluated items. A genetic algorithm then optimizes new suggestions against that surrogate; each suggested item is evaluated by the \"user,\" removed from the search space, and added to the training pool to update the model. On the Bohachevsky and Ackley objectives the system outperforms both a conventional genetic algorithm and a random recommender under a tight evaluation budget. The authors state plainly that human evaluation was not tested, so the approach's applicability as an interactive system cannot be asserted.","feed_headline":"Few ratings can beat genetic and random search in recommendations","feed_subtitle":"Surrogate model trained on 100 rated items guides a genetic algorithm to better suggestions on two of three test landscapes.","key_machinery":"The load-bearing component is the evolving surrogate utility function $\\hat{u}_c(s, S')$: a model of user preference fitted to the pool $S'$ of already-evaluated items. Two variants are tested: a second-order polynomial fitted by least squares (LSM), $\\hat{y} = \\beta_0 + \\sum_i \\beta_i x_i + \\sum_j \\beta_{n+j} x_j^2$, and a radial-basis-function network (RBF) with Gaussian activation $\\phi(x) = 1 - e^{-x^2/(2\\sigma^2)}$. The surrogate serves as the fitness function for a genetic algorithm whose job is to maximize predicted utility over the not-yet-evaluated subspace; the best individuals are returned as recommendations, evaluated on the true objective, and then removed from the search space while the surrogate is retrained. The paper also treats three algorithmic controls as part of the machinery: evaluation rate (genetic-algorithm generations before recommending), whether the population is reset after each recommendation, and the number of suggestions per cycle, all of which modulate convergence toward false optima.","core_discovery":"On its own terms, this paper proposes that a content-based recommender can be built by treating the user's preference as an unknown fitness function and using a surrogate-assisted interactive genetic algorithm to explore the item space. An initial pool of 100 evaluated items trains a meta-model; the meta-model then acts as the fitness function for a genetic algorithm whose best-scoring individuals are recommended to the user, evaluated against the true objective, removed from the search space, and folded back into the training pool. The authors claim that with a very limited number of true evaluations this loop outperforms both a conventional genetic algorithm and a random-search recommender on the tested objectives. The detailed results support the claim on the Bohachevsky and Ackley benchmarks but not on Schwefel, where the surrogate-assisted system only matches the random recommender and is sometimes worse, so the abstract's unqualified outperformance claim is broader than the data.","pith_inferences":["If real preferences are high-dimensional, discrete, or noisy, the smooth $\\mathbb{R}^2$ benchmark assumption likely breaks down; a testable extension is to run the same recommendation loop on a discrete item space with a known utility function and compare against random search.","The paper's limitation discussion suggests that a most-uncertain or novelty-based selection criterion for recommendations could improve exploration; this is an editorial inference, not a claim the paper establishes.","A changing fitness landscape, such as the Moving Peaks benchmark the authors themselves propose, would directly test the claimed adaptability to changing user taste.","The LSM-over-RBF advantage hints that global approximation may matter more than interpolation fidelity when the evaluation budget is small, a hypothesis the paper's data support but do not prove."],"forward_implications":["With very limited true evaluations (around 1000 or fewer), a content-based recommender can improve suggestions without access to other users' data.","The approach can adapt to shifting preferences in principle, since the surrogate is retrained after every evaluated suggestion.","A surrogate with a global perspective (LSM) tends to beat a local interpolator (RBF), suggesting model choice matters more than evolutionary parameter tuning.","On hard multimodal objectives such as Schwefel, the benefit over random search disappears, so gains are not guaranteed across item spaces.","Excluding already-evaluated items from the search space may support exploration and model updating but also limits exploitation of known-good regions."],"supporting_citations":[{"why":"Supplies the three benchmark functions used as the simulated user's utility and the implementation used for testing.","marker":"[1]"},{"why":"Supplies the diversity-based method for setting the width $\\sigma$ of the RBF activation functions.","marker":"[4]"},{"why":"Supplies the second-order polynomial surrogate form and the survey of fitness-approximation methods that frames the meta-model choices.","marker":"[8]"},{"why":"Supplies the evolution-control taxonomy used to position the system and the suggestion to use Gaussian processes as certainty-aware surrogates.","marker":"[9]"},{"why":"Supplies an earlier interactive evolutionary-computation approach with a meta-model that motivates combining surrogates with human evaluation.","marker":"[13]"},{"why":"Supplies a prior surrogate-assisted interactive genetic algorithm with co-trained radial-basis-function networks, the closest algorithmic baseline for the proposed system.","marker":"[21]"}],"fun_headline_variants":["Surrogate genetic recommender beats baselines on two of three functions","Limited ratings: surrogate genetic recommender wins on two test functions","Surrogate-assisted genetic recommender outperforms baselines on two of three benchmarks","With few ratings, surrogate genetic search tops random and GA on two benchmarks","Recommender uses surrogate model to evolve suggestions from few user ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a user's liking can be faithfully represented by the smooth two-dimensional benchmark functions used as stand-ins, so that measured gains would transfer to real recommendation; the paper itself states that human evaluation was not tested, so interactive applicability cannot be asserted.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate genetic recommender beats baselines on two of three functions","Limited ratings: surrogate genetic recommender wins on two test functions","Surrogate-assisted genetic recommender outperforms baselines on two of three benchmarks","With few ratings, surrogate genetic search tops random and GA on two benchmarks","Recommender uses surrogate model to evolve suggestions from few user ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4780,"prompt_tokens":866,"completion_tokens":3914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":3822}},"tokens_in":482,"tokens_out":3914,"duration_ms":30545,"temperature":1.0,"reasoning_tokens":3822,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:31:11.728260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparison on a real or high-dimensional item space and check whether the surrogate-assisted recommender still beats random search; the paper already contains one disconfirming case, since on the Schwefel benchmark the random recommender matches or beats the surrogate-assisted system.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the three benchmark functions used as the simulated user's utility and the implementation used for testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the diversity-based method for setting the width $\\sigma$ of the RBF activation functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second-order polynomial surrogate form and the survey of fitness-approximation methods that frames the meta-model choices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evolution-control taxonomy used to position the system and the suggestion to use Gaussian processes as certainty-aware surrogates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies an earlier interactive evolutionary-computation approach with a meta-model that motivates combining surrogates with human evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a prior surrogate-assisted interactive genetic algorithm with co-trained radial-basis-function networks, the closest algorithmic baseline for the proposed system."}],"review_version":1}