{"id":"db5d52f0-4445-4628-a151-9c26793eb91c","arxiv_id":"1908.05472","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A multi-expert FreeCiv agent that used reinforcement learning to pick among conflicting human rules beat every single-expert rule set and won in 287 average turns versus 291 for the best expert.","lead":"The authors built a FreeCiv-playing system that stores strategies from 11 human experts as rules and uses reinforcement learning to choose between those rules when they conflict. The trained system beat each individual expert rule set and won in slightly fewer turns, suggesting RL can act as an automatic conflict resolver inside knowledge-based systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The strongest claim lacks an ablation: no multi-expert KB without RL is run, so the trained agent's wins and 287-turn average may just reflect having all expert rules; RL's specific contribution is not isolated.","rationale":"I read the paper as claiming not just a working hybrid but a specific mechanism: RL conflict resolution improves over static expert selection. The reader's conditional verdict focuses on fixed-map generalization; I agree that is a concern, but the more immediate load-bearing issue is that no control isolates RL. Section 4.5 shows expert KBs are strong; Section 4.6 defines RL action space over all KIs; Section 5 compares only to single experts. An ablation with a random or fixed resolver over the same union would separate 'more knowledge' from 'learning to choose'. Because this is an empirical omission, not evidence of fraud, the appropriate verdict remains conditional, with the added condition that such an ablation (or release of the simulation harness) be provided. I therefore disagree with the reader's choice of weakest assumption; the fixed-map issue is secondary and can be tested later, but the missing baseline challenges the central claim even on the reported maps.","tokens_in":17758,"tokens_out":7296,"duration_ms":76430,"concrete_test":"Re-run the Section 5 contest (same five maps, same starting positions, same 110-game schedule) using the same multi-expert KB but replacing the learned conflict selector with (a) uniform-random selection among conflicting KIs and (b) a fixed first-match priority. If either baseline wins all games or matches the 287-turn average, the RL policy is not the cause of the reported results; if the baselines lose games or need materially more turns, the RL attribution is supported. Also plot mean episode return over the 600 training games to evidence the claimed improvement with experience.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that RL, used for conflict resolution, lets a multi-expert KB win FreeCiv and improve turns-to-win. Section 5 supports this by comparing the trained agent to 11 single-expert KBs: the trained agent averaged 287 turns and won every contest game. But Section 4.6 builds the multi-expert KB by simply unioning all expert KI sets, and Section 5 never runs that same union with a non-learning conflict-resolution rule (random choice, first-match priority, or hand-coded priority). Therefore the reported superiority may be due to the multi-expert KB containing every expert's rules, not to the Monte-Carlo policy selecting among conflicts. The comparison is asymmetric: the trained agent has access to all 11 experts' strategies, while each opponent has only its own. A 4-turn average gap (287 vs 291) with no variance or error bars could also be noise. This is a missing control for the paper's novel component, not just a generalization issue.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Knowledge-Based Reinforcement Learning (KB-RL), a hybrid method that uses a rule-based knowledge system for acting and Monte Carlo reinforcement learning for resolving conflicts among rules originating from multiple human experts. The method is demonstrated on the full FreeCiv strategy game. Eleven expert knowledge bases (KI sets) are built from human players and made to compete in a 550-game tournament. A multi-expert knowledge base is then formed by combining all expert rules, and an RL agent learns to select among conflicting rules with a reward of -1 per turn. Section 5 reports that the trained agent won every contest game against the individual expert knowledge bases and won in 287 turns on average versus 291 turns for the best expert base (Tatamo). The paper interprets these results as evidence that RL can serve as an automated conflict-resolution mechanism that combines multiple expert strategies into a stronger player.","tokens_in":17985,"tokens_out":5579,"duration_ms":56300,"significance":"If the central claim is supported, the paper would offer a practically relevant demonstration that reinforcement learning can be used to integrate conflicting knowledge from multiple experts, turning rule conflicts into a learned policy rather than requiring manual knowledge engineering. The scale of the experiment is a genuine strength: 550 tournament games, 1,100 recorded game histories, a publicly released game-history dataset (DOI in reference [56]), and full evaluation against 11 expert rule sets. The contribution is, however, not yet convincingly isolated: the evaluation lacks a baseline of the multi-expert knowledge base without RL, uses the same five fixed maps for training and testing, and reports aggregate averages without error bars. The turn-count improvement is also directly encoded in the reward function, so it cannot by itself validate the learned policy. The win-rate result is promising but needs an appropriate control to be attributed to RL.","major_comments":[{"comment":"The central claim that RL improves gameplay is not supported by an adequate control. The trained agent uses the multi-expert knowledge base formed by taking the union of all 11 expert KI sets, while each opponent in Section 5 uses only its own single-expert KI set. To attribute the observed wins and the 287-turn average to the learned conflict-resolution policy, the authors must also run the same multi-expert knowledge base with a non-learning conflict-resolution rule such as random selection, first-match priority, or a fixed priority order. As the experiment stands, the superiority of the trained agent may be due entirely to the fact that its rule base contains all experts' rules rather than to what RL contributes.","section":"Section 4.6 / Section 5"},{"comment":"The evaluation is performed on the same five fixed maps and starting positions that were used during training. Section 4.3 states that maps and starting positions were deliberately kept fixed, and Section 4.6 trains and Section 5 tests on those same setups. The abstract's phrase 'various game settings' is therefore an overstatement, and the all-wins result may reflect memorization of these specific maps. Moreover, Section 5 reports only average turns (287 vs. 291) with no standard deviations, per-agent distributions, or significance tests; with ten games per pairing, the four-turn gap is within plausible noise. The paper should report the full distribution of results and, ideally, evaluate on held-out maps or random starts.","section":"Section 4.3 / Section 4.6 / Section 5"},{"comment":"The reward function returns -1 for each turn played, so reducing the number of turns to win is exactly the objective being optimized. The observed shortening of the game length is therefore an expected consequence of the training signal and cannot by itself serve as independent evidence that KB-RL 'improves the gameplay.' The independent evidence is the win rate against expert rule sets, but that evidence is obtained on the same maps used in training and lacks the multi-expert baseline described above. The paper should separate these two claims and treat the win-rate result, not the turn count, as the primary validation of the RL component.","section":"Section 4.6"},{"comment":"The state representation itself is fitted to the tournament data: the 33 features are selected by their correlation with won/lost outcomes, the feature weights are derived from that analysis, and the number of clusters k=185 is chosen by experimentation on the same data. Because the final evaluation uses the same maps and the same game conditions, there is a risk of overfitting the state space to the tournament environment. A validation procedure that separates state-representation selection from evaluation would strengthen the claim that the learned policy generalizes.","section":"Section 4.6"}],"minor_comments":[{"comment":"The abstract uses 'multiple exerts' where 'multiple experts' is meant; the same typo appears in Section 1.","section":"Abstract / Section 1"},{"comment":"The caption labels subfigure b) as Chaos and c) as Medium Islands, but Section 4.3 refers to 'Medium Islands' as Figure 4 b) and 'Chaos' as Figure 4 c); the labels should be made consistent.","section":"Figure 4"},{"comment":"In Equation (1), the symbol G_s is used for both the state return and a component of the variance, and the expected-return term is not explicitly defined; please introduce distinct notation for the mean and variance.","section":"Eq. (1)"},{"comment":"Expert names are used inconsistently: 'Alex' in Table 1 appears as 'Alex DrKaffee' in Figure 7, and several other names are shortened in captions. Please unify the naming.","section":"Table 1 / Figure 7"},{"comment":"The RDF references contain typographical errors: '20014' should be '2014' and '2919' should be '2019'.","section":"References"},{"comment":"The description of the policy is not fully precise: the paper states that a Normal distribution is used to sample actions, but it is not clear whether the learned policy remains stochastic during the post-training contest or whether epsilon-exploration is switched off at evaluation time; please specify the exact evaluation policy.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"The core request is the missing multi-expert baseline. If the authors can run the union of all expert KI sets with a simple non-learning conflict-resolution method and show that the RL policy still wins or matches it, the paper would be substantially strengthened. The turn-count metric should be repositioned as a training objective rather than independent validation. The paper's engineering contribution is real, but as submitted the central attribution claim is not fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely new: instead of forcing a group of human experts to agree on one FreeCiv strategy, the authors keep all their possibly conflicting rule sets in one knowledge base and let Monte Carlo reinforcement learning pick among the conflicting rules at runtime. That is a clean and practical answer to a known pain in knowledge engineering, and I have not seen it done this way in the KBS or FreeCiv literature. The engineering is real: eleven expert rule sets, a 550-game round-robin tournament, a clustering-based state space, and a trained agent that then won every one of its 110 games against the single-expert agents. The game histories are deposited (Zenodo), which is a point in their favor even without a code release.\n\nWhat the paper does well is show that a multi-expert KB can be built and played at all, and that the RL-trained agent, with access to all experts' rules, beats each expert individually. That is a meaningful existence proof. The Section 5 numbers — 287 average turns for the trained agent versus 291 for the best expert — are less convincing than the all-wins result, because shortening turns is exactly the reward the RL optimizes. The all-wins result is not encoded in the reward, so that part is independently informative.\n\nThe soft spots are real and mostly concentrate in one place. The stress-test note is right: there is no ablation where the union of all expert rule sets is run with a non-learning conflict resolver (random choice, first-match, priority order). Without that control, the trained agent's dominance could simply come from having all eleven experts' rules available, not from the RL policy's arbitration. That is a missing control for the paper's own novel component, not a minor generalization issue. The fixed maps and starting positions further limit how far the results generalize, and there are no error bars on the 287-versus-291 gap. These are fixable in revision: run the union with a few cheap conflict-resolution baselines, add variance reporting, and test on at least a couple of unseen maps.\n\nThe citation pattern looks reasonable for the claims made; the paper does not lean on self-citation. The writing is straightforward and the authors are honest about what they did, including the long 8–12 hour games and the practical constraints. I would not say the central argument is circular — the all-wins against experts is a separate, credible outcome — but the stronger claim about RL specifically improving play is not yet isolated.\n\nWho should read this: anyone working on knowledge-based systems, multi-expert knowledge acquisition, conflict resolution, or hybrid symbolic/RL game agents. It deserves a serious referee. If I were the editor, I would send it out with a clear request that the authors add the missing ablation and at least basic uncertainty measures before it is citable as evidence that RL beats hand-coded conflict resolution.","headline":"A genuinely original hybrid idea—RL as the conflict-resolution layer for a multi-expert FreeCiv knowledge base—but the evaluation lacks the one control that would isolate RL's contribution, so the central claim is real and under-supported at the same time.","tokens_in":18503,"tokens_out":1418,"would_cite":true,"duration_ms":17132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning can settle conflicts between expert rules and turn 11 FreeCiv strategies into one stronger player.","keywords":["Knowledge-Based Reinforcement Learning","conflict resolution","multi-expert knowledge base","FreeCiv","strategy games","Monte Carlo reinforcement learning","state-space clustering","rule-based systems"],"falsifier":"Take the trained agent, without any further training, and run it on FreeCiv maps and starting positions randomly generated outside the five used in the paper; if it no longer wins the majority of games against the same expert knowledge bases, or its average turns to win rise above 291, the reported superiority is an artifact of the fixed evaluation setups.","tokens_in":17582,"feed_emoji":"🎮","tokens_out":7432,"duration_ms":66596,"temperature":0.7,"pith_summary":"The paper sets out to show that a classic rule-based AI can be improved by using reinforcement learning as the arbiter of conflicting expert knowledge. Instead of asking knowledge engineers to hand-resolve disagreements among 11 human FreeCiv players, the authors merge all their rule sets into one multi-expert knowledge base and train a policy that picks which rule to fire when several match. The reported result is that the trained KB-RL agent completes full FreeCiv games, wins every contest game against each of the 11 expert knowledge bases, and wins in fewer turns on average (287) than the best single expert base (291). If this holds, multi-expert knowledge acquisition no longer has to discard or flatten diverse strategies; a learning system can preserve them and recombine them into a stronger player. The method matters beyond games because many real-world automation problems face the same issue of multiple experts giving different but individually reasonable procedures.","feed_headline":"Trained agent beats all 11 expert FreeCiv strategies","feed_subtitle":"Reinforcement learning settles conflicting expert rules and wins in 287 turns, fewer than the best single expert's 291.","key_machinery":"The load-bearing object is the multi-expert knowledge base combined with the RL conflict resolver. A Knowledge Item is a single rule with three blocks (on, when, do) that encodes one atomic expert action; conflicts arise when more than one Knowledge Item matches the current situation. The resolver compresses FreeCiv's enormous state space with $k$-means clustering of 33 game features into 185 clusters, then maintains state-action values for each Knowledge Item per cluster using Monte-Carlo returns, with an $\\epsilon$-greedy policy built from a normal distribution over the values. This machinery converts the knowledge-engineering problem of choosing one consistent rule set into an RL problem of learning which expert move is best in each cluster, and it is what lets the system combine strategies rather than average them.","core_discovery":"KB-RL is a rule-based knowledge system in which every expert strategy is encoded as Knowledge Items (rules with on/when/do blocks), and the inference engine applies abductive reasoning to fire matching rules. When several Knowledge Items conflict, a Monte-Carlo reinforcement learning policy decides among them: the game state is compressed by $k$-means clustering of 33 weighted features into 185 clusters, each Knowledge Item is treated as an action, and the episode return is the negative number of turns to finish, with losses penalised more heavily for being destroyed than for losing the space race. After 600 training episodes, the trained agent won all 110 games it played against expert knowledge bases in the five setups, averaged 287 turns to win versus 291 for the best expert base, and visibly composed a new strategy from fragments of different experts, such as limiting early city counts, expanding mid-game, and settling freely late.","pith_inferences":["A natural stress test is to evaluate the trained agent on maps and starting positions it never saw; if its win rate and turn advantage shrink, the reported dominance is tied to the five fixed setups rather than general FreeCiv skill.","The method should transfer to non-game rule-based domains with episodic success signals, but only where each episode can be replayed cheaply enough to collect hundreds of samples; without a simulator the 600-episode training cost becomes a practical barrier.","The average-turn difference of four turns is small, so the more convincing evidence of superiority is the perfect win record; re-running the contest with more than 10 games per matchup would test whether that record is robust.","Because the reward is purely turn-minimisation, the trained agent might sacrifice robustness or risk aversion; a reward that also penalises near-losses could produce a different and possibly safer strategy profile."],"forward_implications":["A multi-expert knowledge base no longer needs to be made internally consistent by hand; conflicting rules can be kept and ranked by learned value.","The KB-RL agent can play and complete the full FreeCiv game, not just isolated sub-tasks such as city building or defense.","Merging 11 expert rule sets produced an agent that won every game against each individual expert base, so combining strategies can beat the best single strategy.","The learned policy shortened average winning time to 287 turns versus the best expert's 291, and the paper reports improvement with more training games.","The approach gives a template for applying RL as a conflict-resolution layer in other rule-based systems where multiple experts provide alternative procedures."],"supporting_citations":[{"why":"Defines the reinforcement learning and Monte-Carlo methods that the conflict-resolution policy is built on.","marker":"[53]"},{"why":"Supplies the state-space dimensionality reduction rationale for clustering the game state.","marker":"[54]"},{"why":"Provides the Protocol Analysis technique used to elicit and encode expert game strategies into rules.","marker":"[48]"},{"why":"Releases the 1100 tournament game histories that form the clustering dataset and the evaluation record.","marker":"[56]"},{"why":"Earlier FreeCiv learning-to-win work using Monte-Carlo reinforcement learning, the main comparison for full-game play.","marker":"[11]"},{"why":"Case-based approach to learning winning tactics in a real-time strategy game, a related strategy-game-learning baseline.","marker":"[6]"},{"why":"Extends case-based tactic learning to defeating novel opponents, another strategy-game-learning comparison.","marker":"[41]"},{"why":"Defines ontology as a formal shared conceptualisation, which the paper uses to design the game's semantic network schema.","marker":"[22]"}],"fun_headline_variants":["KB-RL beats every expert FreeCiv strategy","Reinforcement learning settles expert conflicts to win FreeCiv","Knowledge-based RL wins FreeCiv, beats all experts","Hybrid AI fuses expert rules, learns to beat them all","Monte-Carlo RL arbitrates expert rules, wins FreeCiv"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the five fixed maps and fixed starting positions used in training and in the final contest are representative enough of FreeCiv's 'various game settings' that the trained agent's clean sweep over expert rule bases reflects general skill rather than memorisation of those specific setups.","fun_headline_variants_meta":{"raw":{"variants":["KB-RL beats every expert FreeCiv strategy","Reinforcement learning settles expert conflicts to win FreeCiv","Knowledge-based RL wins FreeCiv, beats all experts","Hybrid AI fuses expert rules, learns to beat them all","Monte-Carlo RL arbitrates expert rules, wins FreeCiv"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000407,"raw_usage":{"total_tokens":2101,"prompt_tokens":919,"completion_tokens":1182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1096}},"tokens_in":535,"tokens_out":1182,"duration_ms":8202,"temperature":1.0,"reasoning_tokens":1096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:13:11.051442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained agent, without any further training, and run it on FreeCiv maps and starting positions randomly generated outside the five used in the paper; if it no longer wins the majority of games against the same expert knowledge bases, or its average turns to win rise above 291, the reported superiority is an artifact of the fixed evaluation setups.","supporting_citations":[{"cited_title":"Sutton and Andrew G","cited_arxiv_id":null,"evidence_quote":"Defines the reinforcement learning and Monte-Carlo methods that the conflict-resolution policy is built on."},{"cited_title":"State-space dimensionality reduction in markov decision processes","cited_arxiv_id":null,"evidence_quote":"Supplies the state-space dimensionality reduction rationale for clustering the game state."},{"cited_title":"Eliciting expertise","cited_arxiv_id":null,"evidence_quote":"Provides the Protocol Analysis technique used to elicit and encode expert game strategies into rules."},{"cited_title":"FreeCiv games played by Knowledge-based Reinforcement Learning, 2019","cited_arxiv_id":null,"evidence_quote":"Releases the 1100 tournament game histories that form the clustering dataset and the evaluation record."},{"cited_title":"Learning to Win by Reading Manuals in a Monte-Carlo Framework","cited_arxiv_id":"1401.5390","evidence_quote":"Earlier FreeCiv learning-to-win work using Monte-Carlo reinforcement learning, the main comparison for full-game play."},{"cited_title":"Aha, Matthew Molineaux, and Marc Ponsen","cited_arxiv_id":null,"evidence_quote":"Case-based approach to learning winning tactics in a real-time strategy game, a related strategy-game-learning baseline."},{"cited_title":"Aha, and Marc J","cited_arxiv_id":null,"evidence_quote":"Extends case-based tactic learning to defeating novel opponents, another strategy-game-learning comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ontology as a formal shared conceptualisation, which the paper uses to design the game's semantic network schema."}],"review_version":1}