{"id":"317472d8-bde4-435a-bf5a-8c9803382f25","arxiv_id":"2412.07779","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"EoT applies multi-objective evolutionary search with crossover, mutation, and clustering to MLLM reasoning and reports improved Pass@K accuracy on MathVista, Math-Vision, and GSM8K.","lead":"Evolution of Thought (EoT) uses a genetic algorithm, NSGA-II, to evolve diverse reasoning paths for multimodal language models, scoring each answer for quality and novelty. The paper reports higher accuracy than several baselines on math benchmarks, but its efficiency claim is contradicted by its own table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated reference-based quality score MQ (Sec. 3.1) is the load-bearing link: if MQ does not track ground-truth correctness, EoT's selection, crossover, mutation, and CA all optimize self-consistency instead of accuracy, leaving the central performance claim unsupported.","rationale":"The paper's central claim is superior reasoning performance and efficiency. The efficiency half is already internally questionable (Table 3: EoT uses 2K+1 steps and 15.23 s/answer, worse than ToT's 2K and 14.62 s), but the performance half rests entirely on the quality objective MQ. The authors correctly identify that direct LLM scoring is noisy and attempt to stabilize it with a reference answer; however, using the same MLLM to generate the reference and to score does not establish validity. Without evidence that MQ is monotone with correctness, all selection pressure in the algorithm is self-referential. The novelty objective is also defined on the population, not on correctness. So the only link from the MOO search to higher accuracy is MQ. This is the single most load-bearing assumption. A quick correlational check could settle it. I agree with the reader that this is the weakest assumption. Other issues (small margins, no error bars, hyperparameter selection) compound the problem, but the MQ validity is the structural crux. If MQ were shown to correlate with correctness, the framework would still need error bars and proper efficiency accounting; but as written, without that validation, the claimed improvements are not explained by the proposed mechanism.","tokens_in":16201,"tokens_out":5903,"duration_ms":51794,"concrete_test":"Take a random sample of 100–200 questions per benchmark from the test sets. Run EoT to produce all candidate answers (initial, crossover, mutation, and the reference answer). Label each answer correct/incorrect against the ground-truth boxed answer. Compute Spearman rank correlation and AUROC between MQ scores and the correctness labels. Then rerun EoT with MQ replaced by scores from a random permutation (preserving novelty objective only) and compare Pass@1. If the MQ–correctness correlation is not significantly positive (e.g., rho < 0.2) or if random-MQ EoT matches the reported Pass@1, the quality objective is not demonstrably driving accuracy, and the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EoT improves reasoning performance. The only correctness signal inside the search loop is the quality score MQ(A) = f(pq, ps, Aref, A) (Section 3.1), where Aref is a reference answer generated by the same MLLM in one pass. Nothing in the paper establishes that this self-score correlates with ground-truth correctness. If MQ is biased or noisy, then non-dominated sorting, parent selection, crossover/mutation, and the CA cluster-drop step all select for what the model considers 'good' rather than what is actually correct. The final evaluation uses exact-match against labels, so the optimization objective and the evaluation metric are not shown to be aligned. The paper does not report any validation of MQ (e.g., correlation with correctness) and does not ablate MQ against random scoring. Consequently, the reported Pass@1 and Pass@k gains could arise from sampling more candidates and aggregating, not from the multi-objective quality signal. This is the weakest condition for the central claim; if it fails, the framework's mechanism is not connected to accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Evolution of Thought (EoT), a multi-objective reasoning framework for multimodal large language models. EoT represents candidate reasoning paths as a population and applies NSGA-II-style non-dominated sorting over two objectives: a reference-based quality score MQ(A)=f(pq,ps,Aref,A) and a novelty score MN(A) based on edit and semantic distances. Crossover and mutation operations generate offspring, and a Condensation-Aggregation (CA) mechanism clusters candidates, drops low-quality clusters, and prompts the model to aggregate the remaining candidates into a final answer. The authors report experiments on MathVista, Math-Vision, and GSM8K with Qwen2-VL, LLaVA-NeXT, and Phi-3.5-vision, comparing against IO, CoT, ToT, MCTSr, and other baselines, and claim superior performance and efficiency.","tokens_in":16460,"tokens_out":7851,"duration_ms":64976,"significance":"If the central claims could be supported, the paper would offer a useful new application of multi-objective evolutionary search to LLM and MLLM reasoning, and the CA mechanism is a sensible answer to the redundancy problem in population-based search. The manuscript is clearly organized and the ablation study covers the main hyperparameters. However, the evidence as presented is not yet sufficient: the quality objective is an unvalidated self-score, the efficiency claim is contradicted by the paper's own Table 3, and no statistical uncertainty is reported. The core idea is promising and the shortcomings are addressable, so the result is of potential interest to the evolutionary computation and LLM reasoning communities.","major_comments":[{"comment":"The quality objective MQ(A)=f(pq,ps,Aref,A) is computed by the same MLLM that generates Aref, and no evidence is given that this score tracks ground-truth correctness. Since non-dominated sorting, parent selection, and the CA cluster-dropping step all rely on MQ, the reported gains in Tables 1 and 2 could be produced by optimizing the model's self-consistency rather than reasoning accuracy. Please validate MQ (e.g., correlation or calibration against correctness) and include an ablation that replaces MQ with a trivial or random score to show that the quality signal, not merely population expansion, drives the improvements.","section":"Sec. 3.1, Eq. (1)"},{"comment":"The efficiency claim in Section 4.3 is not supported by Table 3: EoT uses 2K+1 inference steps versus 2K for ToT, and EoT's time per answer is 15.23s versus 14.62s for ToT. The text states that EoT requires fewer reasoning steps and less time per step, but the table shows the opposite for ToT on both values. Please correct the claim, clarify whether 'time per step' is the intended metric, and provide token or cost measurements.","section":"Sec. 4.3, Table 3"},{"comment":"The described ranking procedure ('rank them according to the number of candidates they dominate... those with the same dominance count are placed in the same level') is not NSGA-II's non-dominated sorting, which partitions the population into Pareto fronts and uses crowding distance within fronts. As written, the algorithm is a dominance-count heuristic, and the claimed connection to NSGA-II and its diversity guarantees is inaccurate. Please clarify the exact ranking and whether crowding distance is used.","section":"Sec. 3.2, Step 2"},{"comment":"No error bars, multiple runs, or significance tests are reported. Many margins are small, for example Qwen2-VL Pass@8 on MathVista (EoT 67.50 vs MCTSr 66.75) and LLaVA-NeXT Pass@1 on MathVista (EoT 21.50 vs MAD 21.00), so single-run differences may be noise. Report repeated runs with means and confidence intervals, or paired significance tests, before claiming superiority.","section":"Tables 1 and 2"},{"comment":"Hyperparameters N, T, r, K, κ, and m are tuned on the same benchmarks used for the final comparison, with r=1:5 and K=3 reported as best on the challenging Math-Vision dataset. Without a separate validation split or a fixed default configuration, the reported results may be inflated by test-set selection. Please specify the selection protocol or validate on held-out tasks.","section":"Sec. 4.4, Fig. 3"},{"comment":"The evaluation protocol for Table 1 is under-specified. Section 4.2 calls it Pass@1, but Table 1 includes methods that output a single answer, and for EoT it is not stated whether the reported value is the top-1 candidate after non-dominated sorting or the output of the CA aggregation. Please state explicitly how each number is obtained.","section":"Sec. 4.2 and Table 1"}],"minor_comments":[{"comment":"The notation A/A to denote the set excluding A is easily confused with quotient notation; please replace it with A\\A or an explicit subscript.","section":"Sec. 3.1, Eq. (2)"},{"comment":"Equation (3) includes a factor of 2 in the denominator, but the text says DSE is the semantic distance; the relation between this normalized cosine distance and the edit-distance term should be spelled out.","section":"Sec. 3.1, Eq. (3)"},{"comment":"In Table 1, the Phi-3.5-vision CoT GSM8K improvement is 42.50, not 42.20; the Figure 3 caption reads 'Impact of and m' and is missing κ.","section":"Table 1 and Fig. 3 caption"},{"comment":"The line 'Expand the candidate set size: N × = 2' should be written as N ← 2N or similar; the printed notation is ambiguous.","section":"Algorithm 1, line 22"},{"comment":"The manuscript does not mention code or data release; please include a reproducibility statement with exact prompts, model versions, and hyperparameter settings.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is the unvalidated self-score MQ; if that is fixed with a correlation analysis and an ablation, and the efficiency claim is corrected, the paper could become suitable for publication. I would also ask the editor to check whether the single-run results and test-set hyperparameter selection meet the journal's standards for empirical papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central idea is worth a serious look, but the current evidence does not carry the claims in the abstract. EoT applies NSGA-II to search over MLLM reasoning paths, balancing a quality score against a novelty score, then condenses and aggregates the survivors. That specific combination is not in the cited literature, and it is a legitimate extension rather than a conceptual breakthrough. The framework is clearly described, the prompts are included, and the ablation covers the main free parameters. Credit where due: the multi-objective formulation is sensible, the novelty metric is well-defined, and the condensation-aggregation step is a practical answer to the problem of summarizing a large candidate set.\n\nThe soft spots are substantial. First, the efficiency claim in the abstract is contradicted by Table 3: EoT uses 2K+1 inference steps and 15.23 seconds per answer, while ToT uses 2K steps and 14.62 seconds. EoT is only more efficient than MCTSr, not than ToT, so 'superior efficiency' is overbroad. Second, the load-bearing quality score MQ is a model self-score against a reference answer generated by the same model. The paper never validates that this score correlates with ground-truth correctness, and it never ablates it against random scoring. If MQ is unreliable, then the Pareto sorting, crossover, mutation, and cluster-dropping all optimize for self-consistency rather than accuracy, and the reported gains could come mostly from sampling more candidates plus aggregation. This is the weakest link in the chain, and the paper does not address it. Third, the accuracy gains over strong baselines are often small (e.g., 45.42 vs 45.00 for MAD on Qwen2-VL MathVista) with no error bars or significance tests, and hyperparameter choices like N, T, r, K, κ, and m appear to be selected on the same benchmarks that are later reported as results.\n\nThe citation pattern is reasonable and the writing is honest about limitations in some places, but the central claims in the abstract go beyond what the experiments support. This paper deserves a serious referee, not a desk rejection: the idea is timely and the negative results on the quality scorer would be informative. But as written I would expect a reject or major-revision recommendation. I would not cite it yet, and I would not build on its efficiency numbers. I would bring it to a reading group mostly to discuss whether the self-score problem is fixable, not as a model of how to report search-method evaluations.","headline":"The core idea is real but the evidence is thinner than the abstract admits; the efficiency claim is contradicted by the paper's own Table 3.","tokens_in":16976,"tokens_out":2070,"would_cite":false,"duration_ms":21676,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that searching for reasoning paths in multimodal language models should optimize answer quality and diversity jointly, and shows that a genetic multi-objective algorithm (NSGA-II) with crossover, mutation, and…","keywords":["Evolution of Thought","multi-objective optimization","NSGA-II","reasoning path search","multimodal LLM","answer diversity","mathematical reasoning","Pareto front"],"falsifier":"Take a benchmark, compute the correlation between the model's $M_Q$ scores and ground-truth correctness; if it is near zero, EoT's selection is not optimizing for correctness. Alternatively, run EoT with reference answers that are deliberately wrong and see whether Pass@1 drops; if it does not, the quality score is not load-bearing.","tokens_in":16031,"feed_emoji":"🧬","tokens_out":5770,"duration_ms":46469,"temperature":0.7,"pith_summary":"The paper argues that reasoning in multimodal large language models can be improved by treating the search for good reasoning paths not as a single-objective optimization over answer quality but as a multi-objective problem that jointly maximizes quality and diversity. It introduces Evolution of Thought (EoT), which uses the NSGA-II genetic algorithm—crossover, mutation, and non-dominated sorting—to evolve a population of candidate answers along a Pareto front, and a Condensation-Aggregation mechanism that clusters and prunes redundant paths before summarizing a final answer. Experimental results on MathVista, Math-Vision, and GSM8K with three open vision-language models show gains in Pass@1 and Pass@K over Tree-of-Thoughts, Monte-Carlo Tree Self-Refine, and other baselines, at comparable or lower inference cost. If the claim holds, it suggests that explicit diversity pressure and information sharing between candidate solutions are practical levers for improving both accuracy and efficiency of model reasoning.","feed_headline":"Reasoning as multi-objective evolution beats tree search","feed_subtitle":"Evolving an answer population along a Pareto front cuts local optima and inference cost.","key_machinery":"The NSGA-II evolutionary loop is the engine: each candidate answer is scored by a quality metric $M_Q$ that compares against a reference answer generated by the same model, and a novelty metric $M_N$ combining normalized edit distance with semantic distance from Sentence BERT embeddings. Non-dominated sorting ranks solutions into Pareto layers and selects parents, then prompt-based crossover merges two parent answers and prompt-based mutation rewrites one parent to maintain diversity. The Condensation-Aggregation mechanism uses K-Medoids clustering on a joint distance matrix to discard the lowest-quality clusters and pass only cluster medoids to a final aggregation prompt, keeping the summarization context short.","core_discovery":"The central claim is that the local optima and inefficiency of existing reasoning-path search methods stem from optimizing quality alone, and that balancing quality with a novelty objective, computed from edit distance and semantic similarity, steers the search onto a Pareto front that preserves diverse, high-quality candidates. This is operationalized through NSGA-II's non-dominated sorting, followed by crossover and mutation prompts that recombine and perturb parent answers, and a final condensation-aggregation stage that clusters candidates by distance, drops low-quality clusters, and asks the model to synthesize a refined answer. The paper reports that EoT outperforms single-objective tree-search baselines on three mathematical reasoning benchmarks across three base models, including on Pass@k diversity-based evaluation, and that the performance gain is not sensitive to population size or number of generations.","pith_inferences":["The framework could generalize to open-ended generation tasks where multiple valid answers exist, since it explicitly maintains a Pareto front of alternatives.","The novelty metric could be replaced or augmented with task-specific diversity measures, such as structural diversity in code generation or plan diversity in embodied tasks.","A testable extension is to feed an ensemble of reference answers instead of a single one, which may further stabilize the quality score and reduce reference bias.","If the correlation between $M_Q$ and true correctness is weak, the observed gains may instead come from the extra computation and aggregation, which a careful ablation could isolate."],"forward_implications":["Reasoning search methods for multimodal models can treat diversity as a first-class objective rather than a byproduct of sampling.","Population-based search with crossover and mutation is fully parallelizable, avoiding the sequential cost of tree search.","Reference-based scoring reduces the instability of direct self-scoring, enabling more consistent quality evaluation.","The clustering-based condensation step lets the model summarize a large candidate set without exceeding context length limits."],"supporting_citations":[{"why":"Supplies the NSGA-II multi-objective genetic algorithm that EoT uses as its evolutionary search engine.","marker":"[5]"},{"why":"Provides the non-dominated sorting and elitist selection procedures EoT relies on for Pareto ranking.","marker":"[6]"},{"why":"Introduces the graph-based aggregation baseline (GoT) that EoT's Condensation-Aggregation mechanism extends and contrasts with.","marker":"[2]"},{"why":"Presents the Tree-of-Thoughts baseline whose single-objective search EoT compares against on diversity and quality.","marker":"[36]"},{"why":"Defines the Monte Carlo Tree Self-Refine baseline used for efficiency and quality comparisons.","marker":"[38]"},{"why":"Provides the Sentence-BERT semantic encoder used in the novelty score $M_N$.","marker":"[22]"},{"why":"Introduces Chain-of-Thought prompting, the basic reasoning baseline all search methods build on.","marker":"[31]"},{"why":"Supplies the MathVista benchmark used for multimodal mathematical reasoning evaluation.","marker":"[17]"},{"why":"Supplies the Math-Vision benchmark used for multimodal mathematical reasoning evaluation.","marker":"[28]"},{"why":"Supplies the GSM8K benchmark used for text-only mathematical reasoning evaluation.","marker":"[4]"}],"fun_headline_variants":["Evolving reasoning with Pareto fronts beats tree search","Multi-objective evolution boosts reasoning diversity and quality","Genetic algorithm for reasoning paths outperforms tree search","Pareto-optimal answers: diverse and high-quality reasoning","Balancing quality and diversity in reasoning via NSGA-II"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality score the model gives each answer, compared with a reference answer it wrote itself, is a trustworthy measure of whether the answer is actually correct.","fun_headline_variants_meta":{"raw":{"variants":["Evolving reasoning with Pareto fronts beats tree search","Multi-objective evolution boosts reasoning diversity and quality","Genetic algorithm for reasoning paths outperforms tree search","Pareto-optimal answers: diverse and high-quality reasoning","Balancing quality and diversity in reasoning via NSGA-II"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2890,"prompt_tokens":884,"completion_tokens":2006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1930}},"tokens_in":500,"tokens_out":2006,"duration_ms":13990,"temperature":1.0,"reasoning_tokens":1930,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:48:20.206977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark, compute the correlation between the model's $M_Q$ scores and ground-truth correctness; if it is near zero, EoT's selection is not optimizing for correctness. Alternatively, run EoT with reference answers that are deliberately wrong and see whether Pass@1 drops; if it does not, the quality score is not load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NSGA-II multi-objective genetic algorithm that EoT uses as its evolutionary search engine."},{"cited_title":"A fast and elitist multiobjective genetic algo- rithm: Nsga-ii","cited_arxiv_id":null,"evidence_quote":"Provides the non-dominated sorting and elitist selection procedures EoT relies on for Pareto ranking."},{"cited_title":"Graph of thoughts: Solving elaborate prob- lems with large language models","cited_arxiv_id":null,"evidence_quote":"Introduces the graph-based aggregation baseline (GoT) that EoT's Condensation-Aggregation mechanism extends and contrasts with."},{"cited_title":"Tree of thoughts: Deliberate problem solving with large language models","cited_arxiv_id":null,"evidence_quote":"Presents the Tree-of-Thoughts baseline whose single-objective search EoT compares against on diversity and quality."},{"cited_title":"Sentence-bert: Sen- tence embeddings using siamese bert-networks","cited_arxiv_id":null,"evidence_quote":"Provides the Sentence-BERT semantic encoder used in the novelty score $M_N$."},{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"Introduces Chain-of-Thought prompting, the basic reasoning baseline all search methods build on."},{"cited_title":"Measuring multimodal mathemat- ical reasoning with math-vision dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Math-Vision benchmark used for multimodal mathematical reasoning evaluation."}],"review_version":1}