{"id":"a5cbbd65-de02-4fa7-8f7a-dd36a95985f2","arxiv_id":"2411.16206","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Random axis-aligned subspaces let a simple expected-improvement rule generate large parallel batches without extra hyperparameters, and the method wins on most CEC2017 benchmarks.","lead":"This paper proposes a batch Bayesian optimization method that picks one query point from each of several random axis-aligned subspaces of the search space, so many points can be evaluated in parallel. It is designed for large batches, where many existing batch methods lose efficiency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 'significant speedup' over sequential EI is demonstrated on an iteration-count axis, where q=128 necessarily spends 512 evaluations in only 4 iterations; per-evaluation Table 1 shows only comparable results, so the speedup claim conflates parallelism with optimization efficiency.","rationale":"In good faith, the paper's central empirical claim has real support: ESSI with small and medium batch sizes (q=4–32) beats sequential EI on most of the 58 benchmark problems at equal evaluation counts, and it beats the seven batch EI baselines at q=4, 16, and 128 on the majority of problems. The method is simple, parameter-light, and the comparison against batch EI methods at a fixed total evaluation budget is fair in terms of optimization efficiency. The main weakness I see is not the absence of a theoretical guarantee for the subspace heuristic in Equation (9), since the CEC2017 benchmark already consists of rotated functions and the method performs well there. Instead, the most load-bearing concern is the evidence used for the 'speedup' claim in Section 4.2: plotting regret against iterations, when each iteration consumes q evaluations, makes larger batch sizes look faster even if their per-evaluation efficiency is no better or worse. Table 1 shows this: at q=128, ESSI is not better than EI on most problems, yet the text and Figure 4 convey a strong speedup. The reader's rationale already noted this issue, though their stated weakest assumption was the Eq. (9) heuristic; I therefore mark partial agreement. The proposed concrete test would settle the concern without discarding the paper's legitimate contributions: if ESSI(q=128) still reaches target regret in fewer wall-clock seconds under parallel evaluation, the practical scalability claim survives even if the per-evaluation efficiency claim does not. So the reader's CONDITIONAL verdict remains appropriate, with the condition being that the speedup claims be restated on the correct axis.","tokens_in":33760,"tokens_out":8156,"duration_ms":231642,"concrete_test":"Recompute the Section 4.2 comparison with the horizontal axis changed from 'number of iterations' to 'total number of objective function evaluations' (and separately to wall-clock time including GP model training, acquisition optimization, and parallel batch evaluation) using the same 30 runs. Report the number of evaluations needed by ESSI(q) and EI(q=1) to reach a common target regret, for example the final median regret achieved by EI after 512 evaluations, and the area under the regret-versus-evaluations curve. If ESSI(q=128) requires no fewer evaluations than EI to reach that target, the 'speedups convergence' claim in Sections 4.2 and 5 should be revised to a parallel-throughput claim rather than an optimization-efficiency claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison in Section 4.2 (Figure 4) plots simple regret against 'number of iterations.' Because each ESSI iteration evaluates q points, the q=128 curve reaches 512 total evaluations after only 4 iterations, while sequential EI needs 512 iterations. A lower curve at x=4 is therefore not evidence of better optimization efficiency; it reflects that 128 evaluations were spent per iteration. Table 1, which holds total evaluations fixed at 512, shows the same comparison on the correct axis: at q=128, ESSI is significantly better than EI on only 24/58 problems, worse on 16/58, and similar on 18/58. For q=4–32, ESSI is genuinely better on most problems, but the general statement in Section 5 that the method 'speedups the convergence significantly' and the discussion around Figure 4 overstate what the data show for large q. The wall-clock reduction claim is valid only under the assumptions that objective evaluations dominate the cost and that q parallel workers are available; it is not a statement about optimization efficiency per evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ESSI (Expected SubSpace Improvement), a batch Bayesian optimization method. At each iteration, q axis-aligned subspaces are selected randomly; in each subspace, a candidate is found by maximizing an EI-like acquisition function in which all coordinates outside the subspace are pinned to the current best solution x_min. The resulting q points are evaluated in parallel. The method is tested on 58 CEC2017 benchmark problems (d=10 and d=30) with 30 runs, compared against sequential EI with q from 2 to 128 and against seven batch EI baselines at q=4, 16, and 128, with Wilcoxon signed-rank tests. Results are reported in Tables 1-4 and convergence curves in Figures 4, 6, 7, and 8.","tokens_in":34027,"tokens_out":6486,"duration_ms":57110,"significance":"If the claims were fully supported, the method would be a valuable contribution to batch BO: it is simple, has essentially no additional hyperparameters beyond the batch size, scales to q=128, and beats the included batch EI baselines on a majority of benchmark problems. The empirical program is extensive (58 problems, 30 runs, three batch sizes, statistical tests), and the paper provides a public Matlab implementation. However, the central claim of 'significant speedup' over sequential EI is overstated: the main evidence for it is plotted on an iteration-count axis, which conflates parallelism with per-evaluation sample efficiency. On the per-evaluation axis (Table 1), ESSI at q=128 is comparable to sequential EI rather than better. The subspace heuristic itself is reasonable but has no theoretical grounding, and the paper does not discuss when it could fail. The method's genuine strength is its competitive and often superior performance against other batch EI methods at the same batch size.","major_comments":[{"comment":"The claim of 'significant speedup' over sequential EI is established on an iteration-count axis that conflates parallelism with per-evaluation efficiency. In Figure 4, the horizontal axis is 'number of iterations'; ESSI with q=128 performs 512 evaluations in only 4 iterations, whereas the q=1 EI baseline performs 4 evaluations in 4 iterations. A lower regret at iteration 4 therefore reflects the 128 evaluations already consumed, not improved sample efficiency. Table 1, which fixes the total number of evaluations at 512, is the correct per-evaluation comparison: at q=128, ESSI is significantly better than EI on 24/58 problems, significantly worse on 16/58, and similar on 18/58. The abstract's 'speedups the convergence significantly' and the Section 4.2 statement that 'the convergence speed of the proposed ESSI approach can be improved as we increase the batch size from 2 to 128' are not supported for large q on a per-evaluation basis. Please re-plot the sequential-EI comparison on an evaluation-count axis (or wall-clock time with q parallel workers) and revise the claims to separate parallel wall-clock speedup from sample efficiency.","section":"Section 4.2, Figure 4, Table 1"},{"comment":"The ESSI acquisition function fixes all non-subspace coordinates at the current best x_min. Consequently, maximizing ESSI in a subspace is a heuristic that is guaranteed to improve only the lower-dimensional subspace, not necessarily the full-dimensional objective. The paper does not discuss conditions under which this heuristic could fail, such as strongly non-separable or rotated objectives where no axis-aligned subspace contains an improving direction from x_min. Since the CEC2017 test suite is composed of rotated and composition functions, the empirical success is nontrivial, but the absence of any analysis of this assumption is a gap. Please add a paragraph explaining the heuristic, its limitations, and the intuition (e.g., coordinate-descent-like behavior, random subspace coverage over iterations) for why it works in practice.","section":"Section 3.1, Eq. (9)"},{"comment":"The q-EI baseline is computed with only 1000 Monte Carlo samples for batch sizes up to q=128. In dimension d=30 with q=128, a 128-dimensional integral is approximated by 1000 samples; this estimator has high variance and is known to be biased for large q, which can disadvantage q-EI in the comparison. Please justify the MC sample count by reporting estimator variance (e.g., across different MC seeds) or by using a more accurate q-EI approximation (e.g., the analytical formula for small q and the Fq-EI decomposition for large q), and show that the ranking of methods is not sensitive to this choice.","section":"Section 4.3, q-EI implementation"},{"comment":"The acquisition optimization time comparison in Figure 5 measures the wall-clock time for ESSI with parallel parfor across q acquisition tasks versus sequential EI with one task per iteration. The text phrase 'the speedup of our proposed ESSI' attributes the reduction to the method itself, but a large part of the reduction comes from parallelizing q independent acquisition optimizations; any batch method with independent acquisition subproblems could obtain a similar speedup on the same hardware. The claim 'our proposed ESSI is able to reduce the acquisition optimization time significantly' should be rephrased to describe the parallel implementation, and the text should clarify that the linear speedup for expensive objective evaluation is a property of batch evaluation in general rather than specific to ESSI.","section":"Section 4.2, Figure 5"}],"minor_comments":[{"comment":"The text says 'We also use Wilson signed rank test' where it should be 'Wilcoxon signed rank test'; the same typo appears in the captions of Tables 2-4. Section 4.2 uses the correct spelling.","section":"Section 4.3"},{"comment":"Section 3.3 suggests eliminating duplicate subspaces by checking after each selection, but Algorithm 2 (Step 4-8) does not include such a check. Please either implement the rejection step in the algorithm or explicitly state that duplicate subspaces are allowed and quantify how often they occur.","section":"Section 3.3 / Algorithm 2"},{"comment":"Minor wording: 'the number of acquisition samples is set 512' should be 'set to 512', and 'the maximal number of generation' should be 'generations'.","section":"Section 4.1"},{"comment":"The abstract uses 'speedups the convergence'; this should be 'speeds up the convergence'. Also, Section 5's claim of 'no additional parameter' should be qualified, since the batch size q is itself a parameter, as are the GA population size and generation limit used for acquisition optimization.","section":"Abstract and Section 5"},{"comment":"In Figure 5, the numeric values on the bars are not labeled with units in the bar chart itself; the y-axis is already labeled 'time in seconds', but the reader must infer that the numeric labels are seconds. Please make the unit explicit in the figure or caption.","section":"Section 4.2, Figure 5"},{"comment":"The definition of z in Eq. (9) is informal; writing it as a vector with the subspace coordinates inserted at their original positions would remove ambiguity about the ordering of coordinates in the Gaussian process input.","section":"Section 3.1, Eq. (9)"},{"comment":"The convergence curves in Figures 4, 6, 7, and 8 show medians and quartiles, but the corresponding simple regret tables report only the average of 30 runs. Reporting the median (or both) in the tables would make the tables and figures consistent.","section":"Section 4, general"},{"comment":"Reference [33] is to the authors' own ECI work, which is disclosed in the text as the s=1 special case of ESSI; this is appropriate, but it should be mentioned explicitly that ECI is used as a self-comparison baseline in Section 3.1.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core subspace idea is publishable. The main concern is the overstated speedup claim over sequential EI; with the axes corrected and the conclusions revised, the contribution would be the empirical performance against other batch methods at equal batch size. The comparison against q-EI with a low Monte Carlo sample count should be checked for fairness. Please also consider whether the journal expects some theoretical analysis of the subspace heuristic; at minimum, the limitations section should be expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, ESSI is a clean, easy-to-implement batch BO idea: draw q random axis-aligned subspaces, pin the coordinates outside each subspace to the current best xmin, and maximize EI inside the subspace. When s=d it is EI, when s=1 it is the same author's ECI, and the paper says so. That honesty matters. Second, the empirical program is genuinely extensive: 58 CEC2017 problems, 30 runs, batch sizes 2–128, and Wilcoxon tests against both sequential EI and seven EI-family batch methods. On the per-evaluation axis (their Table 1), ESSI beats sequential EI on 46–49 of 58 problems for q=4–32. That is a real result.\n\nThe soft spots are real but mostly addressable. The convergence plots (Figs. 4 and 8) use number of iterations as the x-axis. With q=128, ESSI consumes 512 evaluations in 4 iterations while sequential EI has only evaluated 4 points. A lower curve at that axis is not evidence of better optimization efficiency; it is evidence of parallelism. The conclusion's \"speedups the convergence significantly\" is too strong for large q. Table 1, holding evaluations fixed, tells the honest story: at q=128 ESSI is better on 24, worse on 16, and similar on 18 against EI. That is \"comparable,\" not \"significant speedup.\" The speedup claim is valid as wall-clock time under the assumption that objective evaluations dominate and q workers are available, but that is a different claim.\n\nTwo smaller issues. Algorithm 2 does not include the duplicate-subspace rejection that Section 3.3 mentions, so it is unclear whether the experiments used it. And the abstract's code link differs from the one in the full text. Both are easy fixes.\n\nThe baselines are all EI-family, so the \"state-of-the-art\" claim is limited to that family. Against that standard the method holds up: it beats q-EI, Fq-EI, MACE, and the sequential-batch methods on a majority of problems even at q=128. The lack of a comparison to non-EI scalable batch methods (e.g., Thompson sampling or TuRBO) is a scope limitation, not a flaw in what they claim.\n\nWho is this for? Anyone working on parallel BO with large batch sizes who wants a parameter-light heuristic that does not require retraining a fantasy model. The paper deserves peer review, and revision should focus on fixing the iteration-axis plots and qualifying the speedup claim. I would cite the subspace idea.","headline":"A simple subspace-sampling batch BO heuristic that genuinely helps at moderate batch sizes, but the large-q 'speedup' claim leans on an iteration axis that conflates parallelism with efficiency.","tokens_in":34545,"tokens_out":3076,"would_cite":true,"duration_ms":41174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that batch Bayesian optimization scales to large batches by maximizing expected improvement inside random axis-aligned subspaces, and that the resulting points stay competitive with sequential EI and seven earlier batch…","keywords":["Bayesian optimization","batch evaluation","expected improvement","subspace acquisition","Gaussian process","parallel computing","expensive black-box optimization","CEC2017"],"falsifier":"Run ESSI and sequential EI on rotated versions of separable benchmark problems, such as rotated Rosenbrock or rotated Ackley in 10 and 30 dimensions, using the same 512 additional evaluations and 30 repeats as the paper, and compare simple regret; if ESSI does not match or beat sequential EI on the majority of these rotated problems, or if its advantage disappears specifically when the optimum requires simultaneous movement of many coordinates, the central claim fails. A simpler check: on a problem whose minimum lies far from $x_{\\min}$ along a direction not aligned with any single coordinate, ESSI's batch points all share $x_{\\min}$ in most coordinates, so they should cluster near $x_{\\min}$ and stall unless the subspaces eventually cover the needed direction.","tokens_in":33561,"feed_emoji":"🎯","tokens_out":9770,"duration_ms":79766,"temperature":0.7,"pith_summary":"Standard Bayesian optimization proposes one query at a time, and most batch extensions lose quality as the batch grows. This paper claims a simple fix: draw a batch of random axis-aligned subspaces of the design space, maximize the expected-improvement acquisition function inside each subspace with all other coordinates pinned to the current best point, and evaluate the resulting points in parallel. The resulting method, called Expected SubSpace Improvement (ESSI), has no extra hyperparameters and scales to batch sizes as large as 128. On the 58 rotated and shifted CEC2017 problems, the authors report that ESSI beats or matches sequential EI on most problems and outperforms seven earlier batch expected-improvement methods on a majority of problems at batch sizes 4, 16, and 128. The paper also notes that extending the idea to expensive constraints is left for future work.","feed_headline":"Subspace sampling scales Bayesian optimization to 128-point batches","feed_subtitle":"Random coordinate slices keep acquisition cheap and results competitive with sequential EI and seven batch methods.","key_machinery":"The load-bearing object is the Expected SubSpace Improvement (ESSI) function,\n$$\\mathrm{ESSI}(y) = (f_{\\min} - \\mu(z))\\Phi\\!\\left(\\frac{f_{\\min} - \\mu(z)}{\\$\\sigma$(z)}\\right) + \\$\\sigma$(z)\\varphi\\!\\left(\\frac{f_{\\min} - \\mu(z)}{\\$\\sigma$(z)}\\right),$$\nwhere $z$ is obtained from the current best point $x_{\\min}$ by overwriting the coordinates of the chosen axis-aligned subspace with $y$, and $\\Phi$ and $\\varphi$ are the standard normal cumulative distribution and density functions. This is exactly the expected-improvement formula, but confined to a slice of the design space through the current best point. The machinery works by solving $q$ such slice problems independently, one per randomly selected subspace, so a batch arises from diversity of subspaces rather than from penalizing or hallucinating points; the paper notes that $s=d$ recovers standard EI and $s=1$ recovers the expected coordinate improvement criterion. The random subspace selection, with dimension $s$ drawn uniformly from $1$ to $d$, is what keeps the method parameter-free and gives each acquisition subproblem an average dimension of $(d+1)/2$.","core_discovery":"The central claim is that the bottleneck of batch Bayesian optimization is the acquisition side, not the search geometry: rather than optimizing one joint acquisition function over a $d\\times q$-dimensional space, one can give each batch point its own low-dimensional improvement problem. For an axis-aligned subspace $Y$ of dimension $s$, the ESSI function is the usual expected-improvement expression evaluated at the point $z$ obtained by taking the current best solution $x_{\\min}$ and replacing its $s$ subspace coordinates by the candidate coordinates $y$. Maximizing ESSI for $q$ randomly chosen subspaces yields $q$ different query points, each differing from $x_{\\min}$ in only a few coordinates, and these points can be evaluated in parallel. The authors argue this is why the method scales: the average subspace dimension is $(d+1)/2$, so each inner acquisition problem is cheaper than the original $d$-dimensional EI problem, and the $q$ inner problems are independent and can be solved concurrently.","pith_inferences":["A testable implication the paper does not address: because each batch point differs from $x_{\\min}$ only in the coordinates of one random subspace, ESSI behaves like a randomized coordinate-descent explorer, so on landscapes whose optima require simultaneous coordinated changes of many variables, such as strongly rotated valleys, its advantage over full-dimensional EI should shrink or reverse.","The subspace mechanism is acquisition-agnostic: the same draw-random-axis-aligned-subspaces-and-pin-the-rest construction could be applied to probability of improvement, lower confidence bound, or knowledge-gradient acquisition, producing batch variants by the same argument.","One could isolate the source of the gains by comparing ESSI's random subspaces with fixed, length-scale-informed, or greedily selected subspaces; if random selection is already near-optimal, the decomposition itself is what matters, whereas sensitivity to subspace choice would point to tuning opportunities.","Because the paper reports under-linear parallel speedups in acquisition optimization (about 1.6-2.7x on 30-dimensional problems at small to medium batch sizes), implementation overhead can dominate when objective evaluations are cheap; on many-core machines this suggests task scheduling or granularity changes are a natural next step."],"forward_implications":["If the claim holds, batch Bayesian optimization no longer degrades as the batch reaches 128: ESSI reports better simple regret than seven batch EI baselines on 46-56 of the 58 CEC2017 problems at $q=128$, while beating sequential EI on 24 and matching it on 18.","Wall-clock speedup becomes nearly linear when expensive evaluations dominate, because $q$ evaluations run in parallel and acquisition optimization time itself drops; the paper reports 32-73x speedups in acquisition optimization time on 10-dimensional problems at $q=128$.","High-dimensional problems benefit more from the subspace decomposition, since each ESSI acquisition problem has dimension averaging $(d+1)/2$ rather than $d$; the authors observe larger advantages over competitors on 30-dimensional than on 10-dimensional problems.","The method does not need artificial penalization, fake objective values, multi-objective solvers, or Monte Carlo batch integration, so it can be dropped into an existing EI-based Bayesian optimization loop with essentially no tuning beyond the batch size itself."],"supporting_citations":[{"why":"Supplies the expected-improvement criterion and the Kriging/EGO formulation that the ESSI formula in Eq. (9) directly extends.","marker":"[2]"},{"why":"Introduces the Kriging Believer baseline and the fake-value sequential-batch idea that ESSI is contrasted with in the experiments.","marker":"[11]"},{"why":"The multi-point expected improvement method whose high cost and batch-size limitations motivate the subspace approach; used as a comparison algorithm.","marker":"[12]"},{"why":"Fast multi-point expected improvement baseline; ESSI is compared against it under batch sizes 4, 16, and 128.","marker":"[15]"},{"why":"Expected Improvement and Mutual Information baseline, a sequential-batch approach ESSI is compared with.","marker":"[17]"},{"why":"Pseudo expected improvement baseline, a sequential-batch penalty-style method compared with ESSI.","marker":"[18]"},{"why":"Multi-objective acquisition ensemble (MACE) baseline; its Pareto-front batch selection is contrasted with ESSI's subspace batch selection.","marker":"[22]"},{"why":"Defines expected coordinate improvement, which ESSI recovers when the subspace dimension is one, establishing the generalization relationship.","marker":"[33]"},{"why":"Defines the CEC2017 benchmark suite, the rotated and shifted test problems on which all numerical comparisons are made.","marker":"[34]"},{"why":"Multi-scale search baseline (MSMR) that ESSI is compared against.","marker":"[35]"}],"fun_headline_variants":["Subspace acquisition functions scale batch Bayesian optimization","Random subspaces make batch BO cheap and parallel","Batch BO: give each point its own subspace, scale up","Subspace-decomposed acquisition for large batch BO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that improving expected improvement inside a randomly chosen slice of coordinates through the current best point yields points that improve the objective over the whole space, with no theoretical guarantee and the most risk on rotated or tightly coupled functions.","fun_headline_variants_meta":{"raw":{"variants":["Subspace acquisition functions scale batch Bayesian optimization","Random subspaces make batch BO cheap and parallel","Batch BO: give each point its own subspace, scale up","Subspace-decomposed acquisition for large batch BO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1833,"prompt_tokens":885,"completion_tokens":948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":886}},"tokens_in":501,"tokens_out":948,"duration_ms":9192,"temperature":1.0,"reasoning_tokens":886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:45.268625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ESSI and sequential EI on rotated versions of separable benchmark problems, such as rotated Rosenbrock or rotated Ackley in 10 and 30 dimensions, using the same 512 additional evaluations and 30 repeats as the paper, and compare simple regret; if ESSI does not match or beat sequential EI on the majority of these rotated problems, or if its advantage disappears specifically when the optimum requires simultaneous movement of many coordinates, the central claim fails. A simpler check: on a problem whose minimum lies far from $x_{\\min}$ along a direction not aligned with any single coordinate, ESSI's batch points all share $x_{\\min}$ in most coordinates, so they should cluster near $x_{\\min}$ and stall unless the subspaces eventually cover the needed direction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the expected-improvement criterion and the Kriging/EGO formulation that the ESSI formula in Eq. (9) directly extends."},{"cited_title":"Chevalier, D","cited_arxiv_id":null,"evidence_quote":"The multi-point expected improvement method whose high cost and batch-size limitations motivate the subspace approach; used as a comparison algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Expected Improvement and Mutual Information baseline, a sequential-batch approach ESSI is compared with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pseudo expected improvement baseline, a sequential-batch penalty-style method compared with ESSI."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multi-objective acquisition ensemble (MACE) baseline; its Pareto-front batch selection is contrasted with ESSI's subspace batch selection."},{"cited_title":"Zhan, Expected coordinate improvement for high-dimensional bayesian opti- mization, Swarm and Evolutionary Computation 91 (2024) 101745","cited_arxiv_id":null,"evidence_quote":"Defines expected coordinate improvement, which ESSI recovers when the subspace dimension is one, establishing the generalization relationship."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CEC2017 benchmark suite, the rotated and shifted test problems on which all numerical comparisons are made."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multi-scale search baseline (MSMR) that ESSI is compared against."}],"review_version":1}