{"id":"dc3f497a-64e4-4862-81f7-f1575b926015","arxiv_id":"2508.17959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An iterative metacognitive feedback loop lets a fast LLM match or outperform a standalone reasoning model on graph coloring and code debugging, with selective fallback to the reasoning model only when needed.","lead":"SOFAI-LM pairs a fast, cheap language model with a slower reasoning model, using a metacognitive module that checks the fast model's answers and sends it feedback to retry. On graph coloring and code debugging, the hybrid system matches or beats the reasoning model alone while often using less compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph coloring 'solve rate' is not a decision-problem metric: Eq. 2 cannot score 'NOT SOLVABLE', Algorithm 1 never handles it, so the RQ4 42%-vs-2% result is ambiguous unless unsolvable instances are excluded and the claim narrowed.","rationale":"The reader's weakest_assumption identified the unsolvable-instance gap, and my independent reading of the algorithm confirms it is even more concrete: Eq. 2 cannot score the required 'NOT SOLVABLE' output, and Algorithm 1 has no code path for it. I agree this is the most load-bearing concern because the graph coloring domain supplies the largest claimed advantage (42% vs 2%) and is explicitly framed as a decision problem. The code debugging leg uses a well-defined external verifier (LeetCode hidden tests), so that half of the empirical case is not affected by this concern. I am not arguing the architecture is wrong or that the authors are being deceptive; the issue is an incomplete specification of the evaluation metric. For this reason I do not move the verdict to REJECT. The appropriate disposition remains CONDITIONAL: the authors should define the scoring of 'NOT SOLVABLE', state whether unsolvable instances were included in each reported rate, and ideally report solvable and unsolvable subsets separately. The paper would also benefit from releasing the generated graph instances and the evaluation script so the 42%-vs-2% result can be reproduced under the corrected metric.","tokens_in":22433,"tokens_out":8762,"duration_ms":85849,"concrete_test":"Re-run the graph coloring evaluation on the full generated set (solvable plus unsolvable, 100 instances per size) with an explicit scoring rule: solved iff (instance is solvable and C(y)=1.0) or (instance is unsolvable and output is exactly 'NOT SOLVABLE'). Report SOFAI-LM and standalone LRM success rates separately for solvable and unsolvable subsets and for the combined set. If the 42%-vs-2% figures change materially, or if Algorithm 1 cannot produce a correct 'NOT SOLVABLE' result because it lacks that branch, the RQ4 decision-problem claim is unsupported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ4 graph-coloring comparison rests on an undefined evaluation for unsolvable instances. The paper defines the task as the graph coloring decision problem, states that both solvable and unsolvable instances are generated, and instructs solvers to respond with exactly 'NOT SOLVABLE' when no k-coloring exists. However, the only correctness function introduced, C(y) in Eq. 2, is the fraction of properly colored edges; it cannot assign a score to a 'NOT SOLVABLE' response. Algorithm 1 in the supplementary material has no branch that recognizes this token: after the LLM loop it always calls ParseColoring on the LRM response and returns a coloring. Thus an unsolvable instance can never be solved by the pipeline as specified. This makes the RQ4 headline numbers (SOFAI-LM 42% vs LRM 2% for graph size 25) uninterpretable as decision-problem solve rates: if unsolvable instances are in the denominator, the reported rates are not measuring the task defined in the paper; if only solvable instances are counted, the claim about solving the decision problem is narrower than stated and the paper should say so explicitly. Since the graph coloring domain carries the largest reported advantage, this ambiguity is load-bearing for the general claim that SOFAI-LM outperforms standalone LRMs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SOFAI-LM, a training-free metacognitive architecture that pairs a fast LLM (System 1) with a slower LRM (System 2) under a metacognitive governance module. The LLM iteratively refines solutions using correctness feedback from a domain-specific evaluator, optionally with episodic memory; if the LLM does not converge within a fixed number of iterations, the LRM is invoked with one of three prompting strategies (problem-only, best attempt, or full history). The authors evaluate the architecture on the graph coloring decision problem and on DebugBench (Python and C++ subsets), across multiple LLM/LRM combinations. They report that the feedback-driven LLM alone can match or exceed standalone LRMs (RQ1), that multi-line feedback with minimal memory is best in graph coloring while single-line feedback is used for code debugging (RQ2), that the value of passing LLM context to the LRM differs by domain (RQ3), and that the full SOFAI-LM pipeline outperforms standalone LRMs in both accuracy and inference time (RQ4).","tokens_in":22726,"tokens_out":4290,"duration_ms":43020,"significance":"If the empirical claims hold, the paper makes a practical contribution: a training-free, model-agnostic way to raise the accuracy of a fast LLM on constraint-heavy reasoning tasks while using the expensive LRM only selectively. The strengths include the use of exact verifiers for graph coloring, execution-based evaluation via the LeetCode API for code debugging, comparisons against external benchmarks (DebugBench), and ablations across multiple LLM and LRM combinations. The central claim is plausible but the supporting evidence is weakened by an undefined evaluation protocol for unsolvable graph coloring instances, post-hoc selection of feedback, memory, and prompting variants on the test sets, and inconsistent timing statements. These issues directly affect the headline RQ4 numbers, which carry the paper's main argument.","major_comments":[{"comment":"Eq. (2) defines C(y) as the fraction of properly colored edges, so a response of 'NOT SOLVABLE' cannot be assigned a score by this function. Algorithm 1 (lines 3-30) contains no branch that recognizes the 'NOT SOLVABLE' token; it always calls ParseColoring on the final response and returns a coloring. Since the Experimental Setting states that both solvable and unsolvable graph instances are generated, the RQ4 solve rates (e.g., 42% vs 2% for size 25) are ambiguous as decision-problem rates: if unsolvable instances are in the denominator, the pipeline as specified cannot solve them; if they are excluded, the claim about solving the decision problem is narrower than stated. Please specify how unsolvable instances were handled and report the success rates separately for solvable and unsolvable instances.","section":"Problem domains; Supplementary Algorithm 1"},{"comment":"The headline RQ4 comparison uses per-domain variants chosen from the same test sets: MLF+MEM for graph coloring (Fig. 3) and SLF+MEM for code debugging (adopted 'because of this analysis'), and LRM prompting with PO for graph coloring versus FH for code debugging (Fig. 4). No held-out validation split is described, and no significance tests or error bars are reported; with 100 graph instances per size, the RQ4 advantage could be substantially inflated by selection. Please provide a validation protocol or report how sensitive the RQ4 conclusions are to the choice of feedback, memory, and prompting variants.","section":"Results RQ2/RQ3/RQ4"},{"comment":"The timing statements are inconsistent. The RQ1 text says 'the LRM, though faster on average, performs substantially worse on larger graph sizes,' while footnote 2 says 'iterating the use of the LLM dominates the LRM, since it can solve many more problem instances while using much less time.' These two statements cannot both be true as written. Please clarify which configuration is faster on average, and define the sense in which one configuration dominates another (e.g., Pareto dominance on success rate and time), so that the efficiency claim is unambiguous.","section":"Results RQ1 and footnote 2"},{"comment":"Algorithm 1 invokes all three LRM prompting variants (PO, BA, FH) on every fallback and only afterward selects a final response ('Choose final r* based on selected variant'). This contradicts the described 'selective fallback' mechanism and, if all three calls are executed, would triple the LRM inference cost. Please specify which variant is actually executed in each reported experiment and how the reported time accounts for any unused LRM calls.","section":"Supplementary Algorithm 1, lines 18-30"}],"minor_comments":[{"comment":"There are typographical errors: 'stduy' should be 'study' and 'employes' should be 'employed' (or 'employees' depending on intent).","section":"Introduction; Experimental Setting"},{"comment":"The caption says the comparison is between 'SOFAI-LM and LLM approaches,' but the comparison is actually against the LRM; the caption should refer to the LRM.","section":"Figure 5 caption"},{"comment":"The sample graph coloring prompt says 'Graph Coloring Decision Problem (< 5 colors)' and 'Use < 5 distinct colors' while the instance has k=4; the formal definition is 'at most k colors.' The phrasing should be rendered as at most 4, or equivalently less than 5, to avoid confusion.","section":"Supplementary Figure 6 sample prompt"},{"comment":"The numerator of Eq. (2) appears to contain a sum that is typeset as a bare 'P'; please typeset it as a summation symbol.","section":"Equation (2)"},{"comment":"Several related-work references use a truncated format (e.g., 'Wei, J. e. a. 2022'), which is inconsistent with the full author lists in the main reference list; the bibliography should be unified.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant topic, and the architecture is simple and reproducible in principle. The main risks are empirical: the graph-coloring decision-problem evaluation is undefined for unsolvable instances, and the central RQ4 comparison uses configuration choices made on the same test data. If the authors can clarify the unsolvable-instance handling and add a validation or sensitivity analysis, the claims may be substantially strengthened. I also note that the related-work section is heavily self-referential to the SOFAI line, though this is prior work rather than circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper has one result worth taking seriously and one load-bearing ambiguity. The worth-taking result is RQ3: sharing the LLM's failed attempts with the LRM helps in code debugging but hurts in graph coloring. That domain-contrast finding is concrete and suggests a design principle: failed attempts are useful when fixes are local, noise when a global re-solve is needed.\n\nWhat's new is the integration of a SOFAI-style metacognitive loop with iterative feedback and selective LRM fallback, training-free and tested across several 8B model combinations and two domains. The RQ1 curves consistently show improvement from iteration, and the multiple model combos in the supplement give the architecture some generality credit. On that level, it is a solid extension of existing self-refine/reflexion ideas rather than a breakthrough.\n\nThe soft spots are real. The graph coloring evaluation never defines how a 'NOT SOLVABLE' response is scored. Eq. 2 only measures edge coloring, and Algorithm 1 has no branch that returns 'NOT SOLVABLE' — it always calls the LRM and parses a coloring. The paper generates unsolvable instances, but the reported solve rates appear to count only solvable ones, without saying so. That makes the RQ4 42%-vs-2% number ambiguous, and since that is the largest claimed advantage, it is not a minor detail. The authors need to state that the decision problem is only partially evaluated, or extend the correctness function.\n\nThe experimental protocol also has issues: feedback, memory, and LRM prompting variants were selected after looking at the test-set results, with no error bars or significance tests. The choice of SLF for code debugging is asserted rather than tested. Time statements are inconsistent: the LRM is faster on small graphs but the text later calls the iterative LLM as dominating in time. And the worked kth-factor example labels a clearly correct implementation as failing, which undercuts the walk-through.\n\nNone of this sinks the architecture. But as written, the paper overclaims. It deserves a serious referee, not a desk reject, and a revision that closes the unsolvable-instance gap, adds variance estimates, releases code, and fixes the example. If you're in the LLM-efficiency space, it is worth bringing to reading group and citing the RQ3 finding with caution.","headline":"Plausible hybrid architecture with a genuinely useful domain-contrast finding, but the headline accuracy claim is undercut by an undefined treatment of unsolvable graph instances and post-hoc configuration picking.","tokens_in":23293,"tokens_out":6060,"would_cite":true,"duration_ms":55929,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free metacognitive feedback loop lets a fast LLM match or beat a dedicated reasoning model on graph coloring and code debugging while using less time.","keywords":["metacognition","feedback loop","fast and slow reasoning","graph coloring","code debugging","large reasoning models","training-free adaptation","selective fallback"],"falsifier":"Run SOFAI-LM and the standalone LRM on the same set of known-unsolvable graph instances with 5-25 vertices and count how often each emits exactly 'NOT SOLVABLE' rather than an invalid coloring; if the pipeline is not reliably correct on those instances, the claimed advantage over the LRM applies only to the solvable subset and the decision-problem claim fails.","tokens_in":22245,"feed_emoji":"🧠","tokens_out":6227,"duration_ms":60864,"temperature":0.7,"pith_summary":"This paper tries to establish that an ordinary, fast language model, when run in a loop with a module that checks its answers and feeds targeted corrections back to it, can solve hard reasoning problems as well as or better than a larger, slower reasoning model. The authors generalize the SOFAI fast-and-slow cognitive architecture into SOFAI-LM, replacing fine-tuning with runtime feedback and using the slow model only as a selective fallback. In their experiments on graph coloring and on Python and C++ debugging, the feedback-driven LLM closes or reverses the accuracy gap and does so with lower average inference time. If right, this matters because it points to a cheaper way to get reliable reasoning from language models without retraining.","feed_headline":"Metacognitive loop lets fast LLMs beat reasoning models","feed_subtitle":"On graph coloring and code debugging, a feedback-driven LLM beats a slower reasoning model on both accuracy and time.","key_machinery":"The load-bearing mechanism is the metacognitive governance module (MC) interposed between the System-1 LLM and the System-2 LRM. MC computes a correctness score—fraction of properly colored edges or test pass ratio—turns the specific failures into feedback, and re-prompts the LLM for up to T iterations, tracking stagnation and storing the episode in memory. When the LLM loop fails, MC invokes the LRM with one of three prompt levels: problem-only, best LLM attempt, or full feedback history. The same module, without model fine-tuning, applies to both domains; only the evaluation and feedback-generation pieces are domain-specific.","core_discovery":"SOFAI-LM's central claim is that a metacognitive controller can make a fast LLM surpass a standalone reasoning model on two contrasting reasoning tasks. The controller evaluates each candidate answer with a domain-specific correctness function, generates iterative feedback (structured multi-line feedback or compressed single-line feedback, plus adaptive subproblems), stores a limited episodic memory, and after a fixed number of iterations decides whether to invoke the slower LRM. In the paper's head-to-head comparison, the complete pipeline solves 70% of Python and 73% of C++ debugging problems while taking less time, versus 37% and 40% for the standalone LRM; on size-25 graph coloring it solves 42% versus 2%. The authors also report that feedback format changes outcomes: multi-line feedback beats single-line feedback and minimal memory beats extended history, and that passing failed-attempt history to the LRM helps in code debugging but hurts in graph coloring.","pith_inferences":["An implication not developed in the paper: the same architecture could be applied to any domain with an external verifier that can name concrete failures—formal proof checking, constraint satisfaction, database query repair—so the feedback loop would replace task-specific fine-tuning.","A testable extension suggested by the global/local asymmetry: measure the locality of corrections automatically and use it to choose PO versus BA/FH prompting for the fallback, potentially eliminating the need to hand-tune the strategy per domain.","If unsolvable graph instances were scored explicitly, the graph-coloring advantage might narrow, because the correctness function currently gives no separate credit for correctly answering 'NOT SOLVABLE'; this is the main unresolved boundary of the decision-problem claim.","The authors leave implicit that iteration count T is a cost lever: at small T the loop is cheap but may under-perform, at large T it approaches LRM-level compute; a policy that stops early on detected stagnation may be the real driver of the efficiency gain."],"forward_implications":["Increasing the number of feedback iterations (5, 10, 15) monotonically improves LLM success rate in both domains, with the largest jump at LLM@5.","The architecture is model-agnostic: the same feedback loop raises success rates for different base LLMs, and the selective fallback outperforms multiple standalone LRMs.","Feedback design is consequential: multi-line feedback with minimal episodic memory is the best configuration tested on graph coloring, while single-line feedback is used for debugging.","For local-repair tasks like debugging, giving the LRM the LLM's best attempt or full history raises success; for global-consistency tasks like graph coloring, it lowers success.","Because the slow LRM is invoked only after the LLM loop fails, average inference cost per solved instance can fall even while accuracy rises."],"supporting_citations":[{"why":"Defines the original SOFAI fast-and-slow architecture that SOFAI-LM generalizes.","marker":"(Fabiano et al. 2025)"},{"why":"Introduces the SOFAI architecture and its metacognitive solver selection, the conceptual base for the paper.","marker":"(Booch et al. 2021)"},{"why":"Supplies the DebugBench benchmark and the Python/C++ debugging instances used for evaluation.","marker":"(Tian et al. 2024)"},{"why":"Provides the DIMACS format used to represent graph coloring instances.","marker":"(Johnson and Trick 1996)"},{"why":"Describes DeepSeek-R1, one of the LRM fallback models compared against.","marker":"(Guo et al. 2025)"},{"why":"Provides the Granite 3.3 8B model used as both fast LLM and, in thinking mode, LRM.","marker":"(IBM Research 2025)"},{"why":"Provides Llama 3.1, the second fast LLM tested.","marker":"(Dubey et al. 2024)"},{"why":"Describes Qwen3, another LRM fallback tested.","marker":"(Yang et al. 2025)"}],"fun_headline_variants":["Metacognition boosts fast LLM to beat reasoning models","Fast LLM with metacognitive feedback outperforms slow reasoners","Metacognitive loop gives fast LLM win over reasoning models","LLM plus metacognition: faster and better than reasoning models","Feedback-driven LLM beats reasoning models with less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the graph-coloring correctness score—the fraction of properly colored edges—adequately captures success on the decision problem, even though the paper never specifies how a 'NOT SOLVABLE' answer is scored and reports results only for solvable instances.","fun_headline_variants_meta":{"raw":{"variants":["Metacognition boosts fast LLM to beat reasoning models","Fast LLM with metacognitive feedback outperforms slow reasoners","Metacognitive loop gives fast LLM win over reasoning models","LLM plus metacognition: faster and better than reasoning models","Feedback-driven LLM beats reasoning models with less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1808,"prompt_tokens":999,"completion_tokens":809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":726}},"tokens_in":615,"tokens_out":809,"duration_ms":8328,"temperature":1.0,"reasoning_tokens":726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:58:57.578867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SOFAI-LM and the standalone LRM on the same set of known-unsolvable graph instances with 5-25 vertices and count how often each emits exactly 'NOT SOLVABLE' rather than an invalid coloring; if the pipeline is not reliably correct on those instances, the claimed advantage over the LRM applies only to the solvable subset and the decision-problem claim fails.","supporting_citations":[{"cited_title":"B.; Loreggia, A.; Mattei, N.; Murugesan, K.; Pallagani, V.; Rossi, F.; Srivastava, B.; and Venable, K","cited_arxiv_id":null,"evidence_quote":"Defines the original SOFAI fast-and-slow architecture that SOFAI-LM generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the SOFAI architecture and its metacognitive solver selection, the conceptual base for the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DebugBench benchmark and the Python/C++ debugging instances used for evaluation."},{"cited_title":"S.; and Trick, M","cited_arxiv_id":null,"evidence_quote":"Provides the DIMACS format used to represent graph coloring instances."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Granite 3.3 8B model used as both fast LLM and, in thinking mode, LRM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Llama 3.1, the second fast LLM tested."}],"review_version":2}