{"id":"36a092b9-427b-4dd9-a2b7-7d731900789d","arxiv_id":"1908.06183","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Two new Mastermind heuristics, an SA variant (mean 5.79) and MERC (mean 4.71 queries on MM(6,4)), are both measurably worse than the 1977 Knuth minimax baseline (mean 4.47).","lead":"This paper tests two search algorithms for the classic code-breaking game Mastermind: a simulated-annealing variant and a heuristic that picks guesses to shrink the set of still-possible secret codes as fast as possible on average. Both are run on 5,000 simulated games and compared with random guessing and the classic 1977 Knuth algorithm.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 'Knuth' baseline cannot be Knuth: a max of 7 on MM(6,4) contradicts the five-guess guarantee stated in Section III, and Algorithm 1 is not minimax; Table III's comparisons to SA and MERC hang on an unverified stand-in.","rationale":"The reader's weakest assumption points to the same issue: the 'Knuth' baseline is the linchpin of the comparison table, and the paper contains direct evidence that the implemented baseline is not Knuth's minimax. A correct Knuth baseline has a known guarantee of at most five guesses on MM(6,4), so a reported maximum of seven is not a sampling artifact; it signals a different algorithm. The pseudocode in Algorithm 1 reinforces this: the stated selection rule is not the minimax rule described in the prose, so even if the maximum were a typographical error, the baseline is not reproducible from the paper. Without a verified baseline, the central quantitative contribution of Table III cannot be interpreted, because all claims about SA and MERC are relative to that row. The paper also contains a separate issue in Algorithm 3 (the inner consistency count uses q(m',c') where a correct expected-reduction computation would use q(c,c')), and the aggregation is underspecified; this is an additional reason the work cannot be accepted as presented, but it is secondary to the baseline problem for the paper's comparison claim. The correct disposition remains conditional: the authors should provide a precise algorithm, code, and data, and the table must be recomputed against a real Knuth baseline.","tokens_in":5916,"tokens_out":9148,"duration_ms":96054,"concrete_test":"Enumerate all 1296 master codes for MM(6,4) and implement Knuth's minimax exactly: first guess 1122; at each step, for every code in the full 1296-code space (not only S), compute the 14 feedback-class partition sizes induced on S, score each candidate by the maximum partition size, and choose the code with the smallest maximum. Report the maximum and mean game lengths over all 1296 secrets. If the exact implementation gives max 5 (as Knuth's guarantee requires), then Table III's 'Knuth' max of 7 cannot come from that algorithm, and the baseline in the paper is a different, unreported heuristic; if it gives max greater than 5, the reference implementation differs from the published algorithm. Either way, Table III's comparative claims need to be recomputed against a verified baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the empirical comparison is that the row labeled 'Knuth' in Table III implements the five-guess minimax algorithm described in Section III. The paper's own text undermines this: Section III states Knuth's method guarantees a solution in at most five guesses, while Table III reports a maximum of 7 queries for MM(6,4) and 6 for MM(5,4) for that same baseline. Moreover, Algorithm 1 is not Knuth's minimax: it says 'qi <- Min(E_{c in S}[Max(|Q|)])', which is not a well-formed argmin and, as written, evaluates an expectation over the consistency set rather than selecting the candidate query whose largest partition is smallest; Knuth also chooses among all candidate codes, not only the current consistency set S. Because every comparative statement about SA and MERC, including the headline means (SA 5.7916 and MERC 4.714 vs 'Knuth' 4.468 on MM(6,4)), is measured against this row, the central quantitative claim of the paper is unsupported unless the baseline is independently verified. This is not a disagreement with consensus; it is an internal inconsistency between the stated guarantee and the reported data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents two search algorithms for Mastermind: a simulated-annealing variant (SA) and a heuristic called maximum expected reduction in consistency (MERC), and compares them with random search and with Knuth's minimax five-guess algorithm. Experiments on MM(6,4), MM(5,4), and MM(4,4) with n=5000 games per configuration report mean query counts, with the main numerical results in Table III (e.g., for MM(6,4): MERC mean 4.714, SA mean 5.7916, Knuth mean 4.468, random mean 639.9). The abstract claims novelty for the two approaches but does not claim superiority over prior art; the paper explicitly takes Knuth's 4.467 expected queries as the reference point.","tokens_in":6018,"tokens_out":5324,"duration_ms":50765,"significance":"If the empirical results are reproducible, the paper provides a modest but useful data point: MERC is a simple consistency-based heuristic that vastly outperforms random search and operates in the same range as, though slightly worse than, the classic Knuth baseline, while the SA variant is weaker in these experiments. The paper does not claim state-of-the-art performance, and the MERC idea is a natural, clearly motivated objective. The explicit pseudocode for all algorithms and the large sample size are strengths. However, the central comparative claim depends on the correctness of the implemented Knuth baseline, and the manuscript itself contains an internal inconsistency about that baseline; because of this, the main quantitative conclusion is not yet supported.","major_comments":[{"comment":"","section":"Section III and Table III"},{"comment":"","section":"Section V and Algorithm 3"},{"comment":"","section":"Section VI, Table III"}],"minor_comments":[{"comment":"","section":"Abstract"},{"comment":"","section":"Section II"},{"comment":"","section":"Section IV, Eq. (4)"},{"comment":"","section":"Section II"},{"comment":"","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early draft rather than a finished journal submission, with numerous typos and missing references. The central issue is the internally inconsistent Knuth baseline; this is fixable but requires rerunning the experiments with a verified implementation. The paper's scope and contribution are modest but suitable for a workshop or short-paper venue once the empirical comparison is made trustworthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the headline comparison in Table III is unsupported because the 'Knuth' baseline cannot be Knuth as reported. Section III states Knuth's method guarantees a solution in at most five guesses for MM(6,4), yet the table reports a maximum of 7 for that same algorithm on that same game. One of those two statements is wrong. Algorithm 1's pseudocode doesn't rescue it: 'qi <- Min(E_{c in S}[Max(|Q|)])' is not a valid minimax rule and doesn't match the prose about choosing the code with the smallest maximum partition. So the means tracking SA and MERC against 'Knuth' (5.79, 4.71 vs 4.47) are comparisons against an unverified stand-in.\n\nThe paper does have real content. Section II's math is correct: the query-partition count Qp = (p+3)p/2 yields 14 for MM(6,4), and the code-enumeration formula checks out at 1296. The lower-bound theorem for m(c,2) is a genuine little result. The experimental design (5000 games, mean/median/max/std) is the right shape for this kind of heuristic study.\n\nWhere it gets soft: MERC's 'novelty' is overstated. Choosing the code that minimizes the expected residual consistency set is the same as minimizing the sum of squared partition sizes, a standard greedy criterion found in the older literature (Koyama and Lai, among others). The SA variant is a routine stochastic hill-climber with a Hamming-distance-1 neighborhood and a fixed acceptance constant; the 'temperature' argument is weak because the denominator just grows as more queries accumulate. The absence of code or data is more than a nicety here – the central empirical claim hinges on an implementation detail we can't inspect.\n\nSo who is the paper for? A reader who wants a compact derivation of the partition count and a concrete example of why baseline verification matters in empirical AI papers. As a research contribution, it doesn't hold in current form. I'd not send it to a serious journal until the baseline is reimplemented and the pseudocode fixed, but it might be useful as a workshop note after that.","headline":"A Mastermind heuristic paper with correct background math but a headline comparison built on a 'Knuth' baseline that cannot be Knuth as reported.","tokens_in":6717,"tokens_out":4794,"would_cite":false,"duration_ms":46554,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Consistency heuristic cracks Mastermind in 4.7 guesses on average","keywords":["Mastermind","simulated annealing","maximum expected reduction in consistency","consistency set","minimax baseline","random search","combinatorial search"],"falsifier":"Re-run the paper's algorithm for the minimax baseline, MERC, and simulated annealing on all 1,296 codes of MM(6,4), using the same initial code and feedback rules, and record query counts. If the baseline's maximum is five, the reported maximum of seven in Table III is an implementation or transcription error and the comparisons need re-checking; if the baseline's maximum really is seven, then it is not the five-guess minimax algorithm and the comparative claims stand only against an unverified stand-in.","tokens_in":5547,"feed_emoji":"🧩","tokens_out":8870,"duration_ms":78410,"temperature":0.7,"pith_summary":"This paper proposes two search algorithms for Mastermind and compares them against random guessing and the classic five-guess minimax baseline. The first, a simulated-annealing variant, scores candidate codes by how far their feedback departs from observed feedback and accepts inconsistent candidates with an annealed probability. The second, maximum expected reduction in consistency (MERC), always asks the code from the consistency set that promises the smallest expected remaining set of possible master codes. Across 5,000 simulated games on the standard four-peg, six-color game, MERC averages 4.714 queries and the annealing variant averages 5.7916, versus 639.9 for random search and 4.468 for the minimax baseline. The practical point is that a cheap heuristic can shrink hundreds of queries to under five, while the paper's own numbers show neither new method overtakes the minimax reference.","feed_headline":"Consistency heuristic cracks Mastermind in 4.7 guesses on average","feed_subtitle":"Beats random search by two orders of magnitude, yet remains shy of the classic five-guess minimax baseline.","key_machinery":"The load-bearing object is the consistency set $S_i$, the set of all codes that produce the same feedback history as the hidden master code. MERC scores each candidate $c \\in S_i$ by the expected size of the consistency set after asking $c$, computed by simulating every possible master code and counting survivors, and then asks the $c$ with the smallest expected size. The simulated-annealing variant instead forms an augmented set from $S_i$ plus Hamming-distance-one neighbors of the current query, scores neighbors by the sum of absolute differences in black and white peg counts, and accepts inconsistent neighbors with probability $2/(\\text{score}+1)$.","core_discovery":"The paper's central claim is that a consistency-set reduction rule, MERC, and a simulated-annealing search each solve Mastermind far faster than random guessing, with MERC nearly matching the established minimax baseline. On the standard four-peg, six-color game (MM(6,4)), MERC's mean of 4.714 queries and maximum of 7 compare with a 4.468 mean and a reported maximum of 7 for the minimax baseline and 639.9 for random search; the annealing method trails at 5.7916 mean and 13 maximum. The paper also reports the same ranking for smaller variants: MERC averages 4.206 on MM(5,4) and 3.751 on MM(4,4), always far below random search but never below the minimax baseline.","pith_inferences":["A corrected baseline could change the paper's headline comparison: if the Table III minimax entry actually implements a different selection rule, the 4.468 mean may not reflect the true five-guess algorithm, and MERC's gap to the genuine benchmark could be larger or smaller than reported.","Because MERC restricts queries to the consistency set, it may throw away information that inconsistent but informative queries can provide, as the annealing variant's neighborhood does; a hybrid that occasionally tries such codes could lower MERC's worst-case maximum.","The expected-size criterion is essentially a one-step entropy minimization, so a direct comparison with an information-theoretic strategy on the same code variants would quantify how much of the gap to the minimax baseline comes from greedy lookahead and how much from the consistency restriction.","The reported standard deviations (around 0.9 guesses) make the 0.25-guess difference between MERC and the minimax baseline small; paired trials over the same random seeds would clarify whether that gap is statistically reliable."],"forward_implications":["On the paper's data, MERC reduces the average guess count on MM(6,4) from 639.9 for random search to 4.714, suggesting that expected consistency-set reduction is a strong practical substitute when exhaustive minimax search is unavailable.","The simulated-annealing variant is the weakest informed method tested, averaging 5.7916 guesses, which indicates that adding stochastic neighborhood search to a consistency set does not by itself produce a competitive solver.","The ranking of methods is stable across smaller variants: MERC beats random search by roughly two orders of magnitude on MM(5,4) and MM(4,4) as well, while the minimax baseline remains best.","If the baseline is a faithful implementation, the paper's main comparative point is that neither novel method surpasses the 4.468-mean benchmark, so future work should measure against that benchmark rather than against random search."],"supporting_citations":[{"why":"Supplies the five-guess minimax strategy whose 4.467 expected queries serve as the reference baseline for the experiments.","marker":"[3]"},{"why":"Presents a version of the same minimax baseline from the AI literature, cited alongside [3] as the basis of the comparison.","marker":"[4]"},{"why":"Provides the combinatorial formulas for code counts and the lower-bound theorem the paper uses to frame the search problem.","marker":"[2]"}],"fun_headline_variants":["Consistency rule nears optimal Mastermind solving","MERC beats random Mastermind search by two orders","MERC solves Mastermind in 4.7 guesses on average","Near-minimax Mastermind via consistency-set search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the assumption that the minimax baseline in the experiment is the genuine five-guess minimax algorithm, even though the paper's own Table III reports a maximum of seven queries for that baseline while also stating the algorithm guarantees at most five.","fun_headline_variants_meta":{"raw":{"variants":["Consistency rule nears optimal Mastermind solving","MERC beats random Mastermind search by two orders","MERC solves Mastermind in 4.7 guesses on average","Near-minimax Mastermind via consistency-set search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2058,"prompt_tokens":759,"completion_tokens":1299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":375,"completion_tokens_details":{"reasoning_tokens":1234}},"tokens_in":375,"tokens_out":1299,"duration_ms":12994,"temperature":1.0,"reasoning_tokens":1234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:55:49.466867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's algorithm for the minimax baseline, MERC, and simulated annealing on all 1,296 codes of MM(6,4), using the same initial code and feedback rules, and record query counts. If the baseline's maximum is five, the reported maximum of seven in Table III is an implementation or transcription error and the comparisons need re-checking; if the baseline's maximum really is seven, then it is not the five-guess minimax algorithm and the comparative claims stand only against an unverified stand-in.","supporting_citations":[{"cited_title":"The Computer as Mastermind","cited_arxiv_id":null,"evidence_quote":"Supplies the five-guess minimax strategy whose 4.467 expected queries serve as the reference baseline for the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents a version of the same minimax baseline from the AI literature, cited alongside [3] as the basis of the comparison."},{"cited_title":"An Optimal Mastermind (4,7) Strategy and More Results in the Expected Case, 2013","cited_arxiv_id":null,"evidence_quote":"Provides the combinatorial formulas for code counts and the lower-bound theorem the paper uses to frame the search problem."}],"review_version":1}