Benchmarking Surrogate-Assisted Genetic Recommender Systems

Philipp Altmann; Thomas Gabor

REVIEW 4 major objections 5 minor 21 references

Benchmarking Surrogate-Assisted Genetic Recommender Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a surrogate-assisted genetic recommender system can outperform conventional genetic algorithms and random search under a very limited budget of true user evaluations, by fitting a model to evaluated items and using…

desk verdict Modest workshop paper with a legitimate new search-space twist; the headline performance claim does not survive the authors' own Schwefel results and the comparison is in-sample. read the letter →

arxiv 1908.02880 v1 pith:QSBIZ43W submitted 2019-08-08 cs.NE

Thomas Gabor , Philipp Altmann This is my paper · ORCID

classification cs.NE MSC 68W5068T20

keywords surrogatemodelgeneticalgorithmrecommendersysteminteractiveevolutionarycomputationfitnessapproximationpolynomialregressionradialbasisfunctionbenchmarkfunctions

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a recommender system built from a surrogate-assisted interactive genetic algorithm can find items a user will like after very few evaluations of the user's true preferences. The authors simulate the user by three continuous benchmark functions on $\mathbb{R}^2$ and fit a surrogate, either least-squares polynomial regression or a radial-basis-function network, to an initial pool of 100 evaluated items. A genetic algorithm then optimizes new suggestions against that surrogate; each suggested item is evaluated by the "user," removed from the search space, and added to the training pool to update the model. On the Bohachevsky and Ackley objectives the system outperforms both a conventional genetic algorithm and a random recommender under a tight evaluation budget. The authors state plainly that human evaluation was not tested, so the approach's applicability as an interactive system cannot be asserted.

What carries the argument

The load-bearing component is the evolving surrogate utility function $\hat{u}_c(s, S')$: a model of user preference fitted to the pool $S'$ of already-evaluated items. Two variants are tested: a second-order polynomial fitted by least squares (LSM), $\hat{y} = \beta_0 + \sum_i \beta_i x_i + \sum_j \beta_{n+j} x_j^2$, and a radial-basis-function network (RBF) with Gaussian activation $\phi(x) = 1 - e^{-x^2/(2\sigma^2)}$. The surrogate serves as the fitness function for a genetic algorithm whose job is to maximize predicted utility over the not-yet-evaluated subspace; the best individuals are returned as recommendations, evaluated on the true objective, and then removed from the search space while the surrogate is retrained. The paper also treats three algorithmic controls as part of the machinery: evaluation rate (genetic-algorithm generations before recommending), whether the population is reset after each recommendation, and the number of suggestions per cycle, all of which modulate convergence toward false optima.

What would settle it

Run the same comparison on a real or high-dimensional item space and check whether the surrogate-assisted recommender still beats random search; the paper already contains one disconfirming case, since on the Schwefel benchmark the random recommender matches or beats the surrogate-assisted system.

Watch

Extended reading notes

Core claim

On its own terms, this paper proposes that a content-based recommender can be built by treating the user's preference as an unknown fitness function and using a surrogate-assisted interactive genetic algorithm to explore the item space. An initial pool of 100 evaluated items trains a meta-model; the meta-model then acts as the fitness function for a genetic algorithm whose best-scoring individuals are recommended to the user, evaluated against the true objective, removed from the search space, and folded back into the training pool. The authors claim that with a very limited number of true evaluations this loop outperforms both a conventional genetic algorithm and a random-search recommender on the tested objectives. The detailed results support the claim on the Bohachevsky and Ackley benchmarks but not on Schwefel, where the surrogate-assisted system only matches the random recommender and is sometimes worse, so the abstract's unqualified outperformance claim is broader than the data.

Load-bearing premise

The load-bearing premise is that a user's liking can be faithfully represented by the smooth two-dimensional benchmark functions used as stand-ins, so that measured gains would transfer to real recommendation; the paper itself states that human evaluation was not tested, so interactive applicability cannot be asserted.

Editorial extensions

If this is right

With very limited true evaluations (around 1000 or fewer), a content-based recommender can improve suggestions without access to other users' data.
The approach can adapt to shifting preferences in principle, since the surrogate is retrained after every evaluated suggestion.
A surrogate with a global perspective (LSM) tends to beat a local interpolator (RBF), suggesting model choice matters more than evolutionary parameter tuning.
On hard multimodal objectives such as Schwefel, the benefit over random search disappears, so gains are not guaranteed across item spaces.
Excluding already-evaluated items from the search space may support exploration and model updating but also limits exploitation of known-good regions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If real preferences are high-dimensional, discrete, or noisy, the smooth $\mathbb{R}^2$ benchmark assumption likely breaks down; a testable extension is to run the same recommendation loop on a discrete item space with a known utility function and compare against random search.
The paper's limitation discussion suggests that a most-uncertain or novelty-based selection criterion for recommendations could improve exploration; this is an editorial inference, not a claim the paper establishes.
A changing fitness landscape, such as the Moving Peaks benchmark the authors themselves propose, would directly test the claimed adaptability to changing user taste.
The LSM-over-RBF advantage hints that global approximation may matter more than interpolation fidelity when the evaluation budget is small, a hypothesis the paper's data support but do not prove.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a modest GECCO Companion paper with a legitimate new combination—surrogate-assisted genetic search for recommendations where already-evaluated items are removed from the search space—but the abstract's performance claim is contradicted by the paper's own Schwefel results, and the comparison is in-sample. Worth a skim for the idea, not for the evidence.

What's actually new: the specific twist of treating recommendation as optimization on S \ S' with a fixed evaluation budget and updating the surrogate as items are recommended is, as far as the citations show, not present in prior surrogate-assisted interactive GA work. The paper also does something honest: it explicitly separates the algorithm's capability from human interaction, and in Section 5 it says real-world applicability cannot be stated. It compares two surrogate families (polynomial least squares, RBF) and systematically sweeps evaluation rate, population handling, suggestions per cycle, and number of cycles. The authors deserve credit for that.

Where it gets soft. First, the abstract says the approach outperforms the baselines 'given a very limited amount of allowed evaluations.' On Schwefel, Section 4.3 says the opposite: the Random Recommender performs better than the SAGRS. So the headline claim only holds for the easier benchmarks, and even there it is conditional on tuned settings. Second, the parameterization is chosen in Section 4.2 using the same three benchmark functions that are then used in Section 4.3; the legends even cite the previously evaluated 'optimal' settings. That makes the comparison in-sample, and with only 10 repetitions per setup there are no significance or confidence statements to lean on. Third, there is a sign inconsistency: Section 3 formalizes the recommendation objective as arg max of the utility function, while Section 4.1 says the benchmark functions are to be minimized. That is a real formalization slip, even if the empirical plots make the intended direction clear. Finally, as the authors themselves note, everything is done on smooth 2D continuous benchmarks; no human evaluations, no high-dimensional or discrete item spaces, no noise or non-stationarity. The contribution is a proof-of-concept, not a tested recommender.

Who this is for: someone working on interactive evolutionary computation or surrogate-assisted search who wants one concrete example of an exclusion-based recommendation loop. It does not move the needle for recommender systems broadly.

Recommendation: it deserves peer review—a desk rejection would be too harsh—but it needs a revised abstract, a fix for the sign issue, and ideally a held-out evaluation or at least significance testing. I would send it to review, not publish as-is.

Referee Report

4 major / 5 minor

Summary. The paper proposes a surrogate-assisted genetic recommender system (SAGRS) in which a pool of user-evaluated items is used to train a surrogate model (polynomial regression or radial basis function network), the surrogate serves as fitness function for a genetic algorithm that proposes new items, and newly evaluated items are removed from the search space before the surrogate is updated. Human evaluation is replaced by three continuous benchmark functions (Bohachevsky, Ackley, Schwefel) on R^2, and the system is compared with a conventional genetic algorithm and a 'Random Recommender'. The authors tune system parameters in Section 4.2 and report in Section 4.3 that SAGRS outperforms the baselines on Bohachevsky and Ackley, while on Schwefel the Random Recommender performs comparably or better. Section 5 explicitly acknowledges that human evaluation was not tested, that only two-dimensional benchmarks were used, and that generalization beyond the chosen functions remains to be shown.

Significance. If the claimed advantage were established, the paper would be a useful step toward interactive recommender systems that combine content-based recommendation with surrogate-assisted evolutionary search, especially because it explicitly removes already evaluated items from the search space and updates the surrogate after each suggestion. The manuscript is honest in reporting the Schwefel result where the baseline is competitive, and it clearly lists limitations in Section 5. However, as presented, the central empirical claim is not fully supported: the headline statement is contradicted by one of the paper's own benchmark comparisons, and the parameter tuning is performed in-sample on the same benchmarks used for the final comparison. The architectural idea is promising, but the current evidence is insufficient to support the unqualified claim of superiority.

major comments (4)

[Abstract; §4.3, Fig. 8] The abstract's claim that 'given a very limited amount of allowed evaluations on the true objective, our approach outperforms these baseline methods' is contradicted by the paper's own Schwefel results: Section 4.3 states that on Schwefel the SAGRS only 'slightly outperform[s]' the GA and that 'the Random Recommender shows a comparably better performance than the SAGRS.' The headline claim should be restricted to the benchmarks and configurations where it is actually demonstrated, or replaced by an aggregate comparison with appropriate confidence statements.
[§4.2 and §4.3] The SAGRS meta-parameters (evaluation rate, population-reset choice, number of suggestions per cycle, number of recommendation cycles) are selected in Section 4.2 using the same three benchmark functions that are then used for the comparison in Section 4.3, and the figure legends explicitly cite these previously evaluated settings. The GA and Random Recommender baselines are configured for the comparison rather than independently tuned or validated on held-out functions. The reported advantage is therefore in-sample and may reflect per-benchmark tuning rather than an intrinsic property of the algorithm. Please add a held-out validation, cross-validation over parameter choices, or a sensitivity analysis showing that the conclusions are robust across a parameter grid.
[§3 vs §4.1] The formalization in Section 3 defines the goal as arg max of the utility function and states that individuals with the highest predicted fitness are recommended, while Section 4.1 states that all benchmark functions 'are to be minimized with a best fitness value of 0' and all comparison plots display lower 'Best Fitness' as better. No sign transformation or maximization/minimization conversion is specified. This ambiguity is load-bearing for every figure and should be resolved, for example by defining the utility as the negative of the benchmark objective or by explicitly converting minimization to maximization.
[§4.2 and §4.3] All quantitative comparisons are based on only 10 repetitions per configuration, and no significance tests or confidence intervals are reported. Given the visible overlap between many box plots, especially in Figure 8, claims such as one model 'clearly yields profit' or that SAGRS outperforms the Random Recommender need at least paired non-parametric tests across the 10 runs (e.g., a Wilcoxon signed-rank test) to be supported.

minor comments (5)

[§3, first paragraph] There is a typo: 'aurrogate-assisted genetic algorithm' should read 'surrogate-assisted genetic algorithm.'
[§4.3] The 'Random Recommender' is not random search in the usual sense, because it recommends the initially best-estimated individuals according to a surrogate model; the name and the exact baseline behavior should be explained more explicitly to avoid confusion.
[§4.2] The definition of 'accepted suggestion' as 'evaluated better than the worst evaluated item at that time' is given only in prose; a formal definition would improve reproducibility.
[Figures 3–8] The figure legends carry all the essential parameter settings and are very long; a table summarizing the exact configuration for each model, benchmark, and baseline would make the experimental setup much easier to check.
[§4.2.3 and §5] The evaluation-budget accounting is inconsistent: Section 4.2.3 mentions about 1000 real fitness evaluations at maximum, while with an initial pool of 100 items and 100 cycles of 4 suggestions the budget appears to be 500; Section 5 also refers to 'the optimized amount of 1000 evaluations.' Please reconcile these numbers.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; only in-sample parameter tuning on the same external benchmarks weakens the headline comparison.

fitted input called prediction [Section 4.3 (Comparison of System Performance), using 'optimal settings' from Section 4.2 (Parameter Optimization); legends of Figures 6-8.]
"Concrete settings for all of those variable parameters of each system compared are annotated in the legends of the plots. ... As parameters for this random-search-adopted recommender, an evaluation rate of 0 and a reset population-handling technique as well as the optimal settings for the number of suggestions and the amount of recommendation cycles (as evaluated in the previous sub-section) are used."

The SAGRS configurations in the central comparison are not independent of the test bed: Section 4.2 selects the evaluation rate, population-handling technique, number of suggestions, and number of cycles by running the same three benchmark objectives (Bohachevsky, Ackley, Schwefel) that are then used in Section 4.3. The comparison reuses 'the optimal settings' from that tuning on those same objectives. The reported advantage is therefore partly an in-sample selection artifact rather than an out-of-sample prediction: the compared configurations were chosen to look good on exactly the functions used to substantiate the abstract's claim. The benchmark functions themselves are external and the algorithm is not a mathematical derivation, so this is a weak form of the pattern.

full rationale

The central claim is an empirical benchmark result, not a derivation. The benchmark functions are external (DEAP, [1]), the surrogate models are standard (least-squares polynomial regression and RBF interpolation), and there are no load-bearing self-citations or uniqueness theorems. The one in-sample element is the parameter optimization in Section 4.2, which tunes system settings on the same three functions later used for the headline comparison in Section 4.3; the paper's own Schwefel results (Figure 8) also contradict the unqualified abstract claim. These issues are methodological rather than definitional, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities; its contribution is algorithmic. The free parameters are mostly standard model choices and GA settings, but the per-benchmark tuning of evaluation rate, suggestion count, and population handling in Section 4.2 is a genuine fitting step that affects the reported comparison.

free parameters (5)

RBF width sigma = Average pairwise distance between currently evaluated items
Set by a diversity-based heuristic from Bishop 1995, not optimized for prediction accuracy; controls the locality of the RBF interpolation.
Polynomial degree and structure for LSM = Second order polynomial without cross terms
Model choice taken from Jin 2005; determines the approximation capacity of the surrogate.
GA parameters (selection factor, mutation probability, recombination probability) = 0.9, 0.1, 0.05
Fixed by hand and used for all systems; no sensitivity analysis is provided.
System configuration (evaluation rate, suggestions per cycle, population reset, cycle count) = Per-benchmark optimal settings from Section 4.2
Tuned on the same benchmark functions used for the performance comparison in Section 4.3, making the reported advantage in-sample.
Initial pool size = 100 evaluated items
Chosen by hand; the surrogate is fit to these 100 items before any recommendations are made.

assumptions (4)

domain assumption User utility is a real-valued function u_c: S -> R with item space S = R^2.
Stated in Section 3; central to mapping recommendations to benchmark optimization.
domain assumption The three benchmark functions (Bohachevsky, Ackley, Schwefel) are representative proxies for human preference.
Stated in Section 3 and explicitly flagged as a limitation in Section 5; the entire evaluation depends on this.
standard math The least squares normal equations are solvable, i.e., X^T X is invertible.
Section 3 uses Theta = (X^T X)^-1 X^T Y; assumes full column rank of the design matrix.
domain assumption The surrogate model's predicted fitness landscape is smooth enough for the GA to optimize meaningfully.
The GA optimizes the surrogate; no analysis of surrogate error or local optima is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking Surrogate-Assisted Genetic Recommender Systems." pith.science (2026). https://pith.science/paper/QSBIZ43W

@misc{pith2026190802880,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Surrogate-Assisted Genetic Recommender Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QSBIZ43W}},
  note         = {Machine review of arXiv:1908.02880}
}

read the original abstract

We propose a new approach for building recommender systems by adapting surrogate-assisted interactive genetic algorithms. A pool of user-evaluated items is used to construct an approximative model which serves as a surrogate fitness function in a genetic algorithm for optimizing new suggestions. The surrogate is used to recommend new items to the user, which are then evaluated according to the user's liking and subsequently removed from the search space. By updating the surrogate model after new recommendations have been evaluated by the user, we enable the model itself to evolve towards the user's preferences. In order to precisely evaluate the performance of that approach, the human's subjective evaluation is replaced by common continuous objective benchmark functions for evolutionary algorithms. The system's performance is compared to a conventional genetic algorithm and random search. We show that given a very limited amount of allowed evaluations on the true objective, our approach outperforms these baseline methods.

Figures

Figures reproduced from arXiv: 1908.02880 by the authors.

**Figure 2.** Function plots for the (a) Bohachevsky, (b) Ackley, and (c) Schwefel benchmark functions for two-dimensional input. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Benchmarking different Evaluation Rates (x-axis) on the Bohachevsky (red/orange), Ackley (blue) and Schwefel [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Testing the impact of the number of suggestions (x-axis) per recommendation cycle on the Bohachevsky (red/orange), [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Evaluating the optimal number of recommendation cycles (x-axis) needed for both LSM and RBF models on the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Performance comparison of the SAGRS using an [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Performance comparison of the SAGRS using an [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

[1]

[n. d.]. Benchmarks – DEAP 1.2.2 documentation. http://deap.readthedocs.io/en/ master/api/benchmarks.html. ([n. d.]). Accessed: 2019-04-02

work page 2019
[2]

G Adomavicius and A Tuzhilin. 2005. Toward the next generation of recom- mender systems: a survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6 (2005), 734–749

work page 2005
[3]

Marko Balabanović and Yoav Shoham. 1997. Fab: content-based, collaborative recommendation. Commun. ACM 40, 3 (March 1997), 66–72

work page 1997
[4]

Christopher M Bishop. 1995. Neural Networks for Pattern Recognition . Oxford University Press

work page 1995
[5]

John S Breese, David Heckerman, and Carl Myers Kadie. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. UAI (1998)

work page 1998
[6]

Agoston E Eiben, James E Smith, et al. 2003. Introduction to evolutionary computing. Vol. 53. Springer

work page 2003
[7]

William C Hill, Larry Stead, Mark Rosenstein, and George W Furnas. 1995. Rec- ommending and Evaluating Choices in a Virtual Community of Use. CHI (1995), 194–201

work page 1995
[8]

Yaochu Jin. 2005. A comprehensive survey of fitness approximation in evolution- ary computation. Soft Computing 9, 1 (Jan. 2005), 3–12

work page 2005

Show all 21 references

[9]

Yaochu Jin. 2011. Surrogate-assisted evolutionary computation - Recent advances and future challenges. Swarm and Evolutionary Computation 1, 2 (2011), 61–70

2011
[10]

Yaochu Jin and Jürgen Branke. 2005. Evolutionary optimization in uncertain environments-a survey. IEEE Trans. Evolutionary Computation 9, 3 (2005), 303– 317

2005
[11]

Yaochu Jin, M Olhofer, and B Sendhoff. 2002. A framework for evolutionary opti- mization with approximate fitness functions. IEEE Transactions on Evolutionary Computation 6, 5 (Oct. 2002), 481–494

2002
[12]

Yaochu Jin and Bernhard Sendhoff. 2004. Reducing Fitness Evaluations Using Clustering Techniques and Neural Network Ensembles. In Genetic and Evolution- ary Computation – GECCO 2004 . Springer, Berlin, Heidelberg, Berlin, Heidelberg, 688–699

2004
[13]

Raffi R Kamalian, Alice M Agogino, and Hideyuki Takagi. 2007. Use of interactive evolutionary computation with simplified modeling for computationally expen- sive layout design optimization. IEEE Congress on Evolutionary Computation (2007)

2007
[14]

Ken Lang. 1995. NewsWeeder: Learning to Filter Netnews. In Machine Learning Proceedings 1995. Elsevier, 331–339

1995
[15]

Han-Saem Park, Ji-Oh Yoo, and Sung-Bae Cho. 2006. A context-aware music recommendation system using fuzzy bayesian networks with utility theory. In International conference on fuzzy systems and knowledge discovery . Springer, 970– 979

2006
[16]

Michael Pazzani and Daniel Billsus. 1997. Learning and Revising User Profiles: The Identification of Interesting Web Sites. Machine Learning 27, 3 (1997), 313– 331

1997
[17]

Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens. In the 1994 ACM conference. ACM Press, New York, New York, USA, 175–186

1994
[18]

Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Introduction to Recom- mender Systems Handbook. Recommender Systems Handbook Chapter 1 (2011), 1–35

2011
[19]

Mourad Sefrioui and Jacques Périaux. 2000. A Hierarchical Genetic Algorithm Using Multiple Models for Optimization. PPSN 1917, Chapter 86 (2000), 879–888

2000
[20]

B Sheth and P Maes. 1993. Evolving agents for personalized information filtering. In 9th IEEE Conference on Artificial Intelligence for Applications . IEEE Comput. Soc. Press, 345–352

1993
[21]

Xiaoyan Sun, Dunwei Gong, Yaochu Jin, and Shanshan Chen. 2013. A New Surrogate-Assisted Interactive Genetic Algorithm With Weighted Semisupervised Learning. IEEE Transactions on Cybernetics 43, 2 (2013), 685–698

2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

[n. d.]. Benchmarks – DEAP 1.2.2 documentation. http://deap.readthedocs.io/en/ master/api/benchmarks.html. ([n. d.]). Accessed: 2019-04-02

work page 2019

[2] [2]

G Adomavicius and A Tuzhilin. 2005. Toward the next generation of recom- mender systems: a survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6 (2005), 734–749

work page 2005

[3] [3]

Marko Balabanović and Yoav Shoham. 1997. Fab: content-based, collaborative recommendation. Commun. ACM 40, 3 (March 1997), 66–72

work page 1997

[4] [4]

Christopher M Bishop. 1995. Neural Networks for Pattern Recognition . Oxford University Press

work page 1995

[5] [5]

John S Breese, David Heckerman, and Carl Myers Kadie. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. UAI (1998)

work page 1998

[6] [6]

Agoston E Eiben, James E Smith, et al. 2003. Introduction to evolutionary computing. Vol. 53. Springer

work page 2003

[7] [7]

William C Hill, Larry Stead, Mark Rosenstein, and George W Furnas. 1995. Rec- ommending and Evaluating Choices in a Virtual Community of Use. CHI (1995), 194–201

work page 1995

[8] [8]

Yaochu Jin. 2005. A comprehensive survey of fitness approximation in evolution- ary computation. Soft Computing 9, 1 (Jan. 2005), 3–12

work page 2005