{"id":"6cc141db-54ce-4b95-a408-4bb10fe9c01f","arxiv_id":"2502.01344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-role (id, superego, ego) multi-agent prompting framework improves LLM reasoning accuracy by 1-4 points on multi-hop QA and math benchmarks.","lead":"This paper presents PSSD, a prompting framework that gives a language model three roles based on Freudian psychology (id, superego, ego) to generate initial answers, criticize them with rules, and execute a correction script. The authors report small accuracy gains on four reasoning benchmarks over methods that use more compute or external tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No cross-dataset evaluation of the distilled rule set U; PSSD's claimed generalizability is untested and the rule component's contribution is small, so the generalizability claim rests on a potentially benchmark-specific crutch.","rationale":"The reader's weakest assumption is the same one I would stress: U is the only component of PSSD that is built from the benchmark training data and injected at inference time, so its transferability is the condition that must hold for the 'generalizability' half of the central claim. I agree with the reader's assessment. I do not think this concern by itself forces rejection: the three-role framework is coherent, PSSD-SFT is a reasonable integration, and the issue is directly testable with one additional experiment. The post-hoc selection of m and n (footnote 2) and the absence of significance testing are additional causes for caution, but the load-bearing gap is the missing cross-dataset validation of U. The existing CONDITIONAL verdict therefore remains appropriate; no adjustment is needed.","tokens_in":16732,"tokens_out":9470,"duration_ms":84417,"concrete_test":"Rebuild U using only the GSM8K and MATH training splits, then run PSSD on AdvHotpotQA and 2WikiMultiHopQA with that cross-domain U (and the reverse pairing), comparing EM and PM against the same-distribution U results in Table 1. If the cross-domain U loses more than about 1.0 EM point relative to the same-domain U, U is benchmark-specific and the generalizability claim is unsupported; if the gap is within noise, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PSSD's design is generalizable rests on the rule set U (Eq. 3), distilled by GPT-4 from the training splits of the same four benchmarks on which the method is evaluated, with m=3 and n=10 chosen post hoc (footnote 2). At test time U is inserted into the superego prompt and drives key-point generation, so the observed gains could reflect matching the benchmark-specific question types in U rather than a transferable self-denial mechanism. No experiment builds U on one distribution and evaluates on another, and no out-of-distribution reasoning benchmark is used. This matters even though the ablation in Table 3 shows the rules alone contribute only +0.64 EM on AdvHotpotQA and +0.60 on GSM8K: the paper explicitly credits U with supplying the 'persuasive references' needed for confident denial, and the overall 'superiority and generalizability' statement is part of the central claim. The concern is sharpened by internal inconsistencies in Table 1 (e.g., PSSD is not the best EM method on 2WikiMultiHopQA), which make the empirical support less robust than the prose suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PSSD, a multi-agent prompting framework that operationalizes Freudian id/superego/ego roles to improve LLM reasoning. The id role generates multiple initial reasoning paths, the superego role uses a pre-distilled rule set U to produce key-point guidance, and the ego role builds and executes a script before giving a final answer. The paper also introduces PSSD-SFT, a LoRA fine-tuned variant that merges the three roles into a single model. Experiments are reported on AdvHotpotQA, 2WikiMultiHopQA, GSM8K, and MATH, comparing against fine-tuning, tool-augmented, and multi-agent debate baselines. The central claims are that PSSD achieves state-of-the-art results on nearly all tasks and metrics, that it is generalizable, and that its role design reduces the number of LLM calls relative to comparable debate methods.","tokens_in":16944,"tokens_out":6886,"duration_ms":59298,"significance":"The three-role framework is a plausible and potentially useful way to structure LLM self-correction, and the paper contains several strengths: the ablation in Table 3 demonstrates that each role contributes incrementally, the PSSD-SFT variant addresses resource efficiency by merging roles into a single LoRA model, and Table 5 quantifies API-call reductions relative to Self-Contrast and other debate methods. If the empirical claims were fully supported, the framework would be a practical contribution to tool-free reasoning enhancement. However, the current evidence does not support the strength of the advertised claims: the rule set U is distilled from the same distributions on which the method is evaluated, no statistical uncertainty is reported, and the main results table contradicts parts of the prose. These issues are load-bearing for the paper's central generalization and state-of-the-art claims.","major_comments":[{"comment":"The statement that PSSD 'consistently achieves state-of-the-art results across nearly all tasks in all metrics' is contradicted by Table 1. On 2WikiMultiHopQA, ReAct achieves EM 45.24 and VE achieves EM 42.26, both above PSSD's 41.96; on GSM8K, Self-Contrast achieves PM 86.71 versus PSSD's 84.75; and on 2WikiMultiHopQA, ReAct's RM 14.32 is higher than PSSD's 9.48. The prose should be revised to state the exact per-metric rankings and to acknowledge these exceptions explicitly.","section":"Section 4.2, Table 1"},{"comment":"The generalizability claim is not supported by the experimental design because the rule set U is distilled from the training splits of the same four datasets used for evaluation, and the hyperparameters m=3 and n=10 are selected based on experimental results (footnote 2). No experiment builds U on one distribution and evaluates on another, and no out-of-distribution question-type benchmark is used. Since U is inserted into the superego prompt at test time and is credited with providing persuasive references for confident denial, the observed gains could in part reflect matching benchmark-specific question types in U rather than a transferable self-denial mechanism. At minimum, the authors should add a cross-dataset transfer experiment for U and report whether the rule set transfers to held-out question types; the marginal contribution of U in Table 3 (+0.64 EM on AdvHotpotQA, +0.60 on GSM8K) does not remove this concern because the paper explicitly credits U with supplying the references needed for the superego's guidance.","section":"Section 3.3.1, Eq. (3), Section 4.2"},{"comment":"No confidence intervals, error bars, or significance tests are reported, and the test sets are small (308 examples for AdvHotpotQA and 336 for 2WikiMultiHopQA, per Table 6). Several reported advantages are within a few percentage points, such as PSSD's +2.28 EM over ReAct on AdvHotpotQA and PSSD-SFT's +0.33 EM over AugGPT on AdvHotpotQA in Table 2, so the strength of the 'superior performance' claims exceeds what the experiments can support. The authors should provide repeated-run statistics, bootstrap confidence intervals, or significance tests, and temper the prose accordingly.","section":"Section 4.2, Table 6"},{"comment":"Equations (2) and (3) are not well-formed. In Eq. (2), P_b is written as an arg max over b while the maximization should be over the pattern set T, and the probability P(T|q_t^b, K_h^b, K_s^b) is never defined. In Eq. (3), U is written as an arg max over n, which is not a meaningful optimization; the intended operation is presumably a clustering or selection over question types. These equations are the formal statement of the rule-construction procedure, so they should be corrected or replaced with a precise algorithmic description.","section":"Section 3.3.1, Eqs. (2)-(3)"}],"minor_comments":[{"comment":"The manuscript contains numerous grammatical errors and typos, including 'The enhance of accuracy', 'errorneous', 'goolge engine', and 'confidential determination' (likely 'confident determination'). A thorough language edit is needed.","section":"Abstract and throughout"},{"comment":"The consistency analysis is qualitative and based only on kernel density plots. If this analysis is meant to support the claim that PSSD instills greater confidence, the authors should report a quantitative measure of distribution difference or at least describe the density estimation procedure and sample sizes.","section":"Section 4.6, Figure 2"},{"comment":"The table formatting makes the training and test sizes ambiguous; for each dataset, the Size, Training, and Test values should be clearly separated in the table cells.","section":"Table 6"},{"comment":"The 200-sample comparison between PSSD and Self-Contrast is reported without any measure of uncertainty. If this comparison is retained, it should be accompanied by a binomial test or a bootstrap confidence interval.","section":"Appendix A.2"},{"comment":"The notation d_<i^b is not defined precisely; please clarify that it denotes the response tokens of the structured record before position i, and state the LoRA rank and other hyperparameters used in fine-tuning.","section":"Section 3.5.2, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the rule set U is valid and is the main blocker for acceptance. The paper's central idea is defensible, but the evidence needs to be strengthened with cross-dataset rule transfer, proper uncertainty quantification, and a rewrite of the state-of-the-art claim to match the actual rankings in Table 1. If the authors can provide these, the paper would be much closer to the claims it makes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: PSSD is a multi-agent prompting recipe that organizes LLM self-correction into id (multiple initial attempts), superego (rule-guided key-point generation), and ego (scripted execution). The three-role integration is genuinely new, and it shows consistent gains on four reasoning benchmarks, especially GSM8K and MATH, over CoT-SC and Self-Contrast. The PSSD-SFT variant, which folds the roles into one LoRA-tuned model, is a sensible practical move and the compatibility study with ReAct/VE is a nice extra. The paper is clearly written and the empirical story is directionally convincing.\n\nThe soft spots are real but not fatal. The rule set U is distilled from the training splits of the same four benchmarks, and m=3, n=10 were tuned on those benchmarks. The stress-test note is right that there is no cross-dataset test of whether U transfers. However, the ablation shows U contributes only about +0.6 EM on AdvHotpotQA and GSM8K, so it is a small crutch, not the load-bearing pillar. The bigger issue is the prose: the claim of \"state-of-the-art results across nearly all tasks\" overstates Table 1, since ReAct beats PSSD on 2WikiMultiHopQA by a clear margin. There are also no error bars or significance tests, and the consistency analysis is informal kernel-density eyeballing rather than a quantitative test. The GitHub release is only prompts, so full reproducibility is limited.\n\nNone of this sinks the paper. The id and ego roles do the heavy lifting, and the gains are consistent across four datasets. The rule component is a modest bonus, and the lack of transfer testing is a fixable omission rather than a fundamental flaw. The concern about circularity is overstated because the rules are fitted to benchmark distributions but their contribution is small; the method would likely still work without them.\n\nThis paper is for people working on LLM reasoning and multi-agent prompting. It is a useful baseline and the SFT variant is worth citing. It deserves a serious referee; my recommendation is to send it out, but ask the authors for significance testing, a cross-dataset transfer experiment for U, and a toned-down abstract. I'd bring it to a reading group as a representative of the prompting-methods genre, but not as a paradigm shifter.","headline":"A credible three-role prompting method with consistent but modest gains; the rule component is overhyped and under-tested, but the core idea is sound enough for serious review.","tokens_in":17534,"tokens_out":1553,"would_cite":true,"duration_ms":15793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLMs improve reasoning accuracy by running an internal self-denial loop—an id role that generates attempts, a superego role that critiques them with pre-summarized rules, and an ego role that executes a corrective…","keywords":["large language models","self-denial","multi-agent debate","reasoning correction","psyche structure","key points","script generation","LoRA fine-tuning"],"falsifier":"Take a held-out reasoning set whose question types are absent from the four training sets, and compare PSSD against the paper's own no-rule ablation, i.e. superego without $U$. If the full method is not more accurate than the no-rule version, the rule-based guidance is not what generates the gain.","tokens_in":16446,"feed_emoji":"🧠","tokens_out":8963,"duration_ms":74555,"temperature":0.7,"pith_summary":"The paper claims that the element missing from current LLM correction methods is self-denial: the model should first decide, with confidence, that an earlier output might be wrong, and then repair it along targeted steps, rather than just regenerate, debate, or call a tool. To operationalize this, the paper builds PSSD, a three-role prompting pipeline modelled on the psychoanalytic division of the mind into id, superego, and ego. An id role writes several intuitive reasoning attempts; a superego role, guided by a pre-summarized rule set, returns key points that criticize those attempts; an ego role converts the key points into an executable script and runs it to give a final answer. The authors report consistent accuracy gains across AdvHotpotQA, 2WikiMultiHopQA, GSM8K, and MATH, with fewer LLM calls than the multi-agent debate baseline, and they show a fine-tuned single-model variant, PSSD-SFT, also beats the fine-tuning competitors. If this is right, internal, rule-guided self-correction is a cheap and transferable way to strengthen LLM reasoning.","feed_headline":"Three-role 'psyche' prompting lifts LLM reasoning accuracy","feed_subtitle":"A rule-guided superego and script-writing ego beat debate and tool baselines on QA and math, with fewer calls.","key_machinery":"The object that carries the argument is the three-role psyche pipeline, together with the pre-summarized rule set that feeds it. The id role is a simple prompted LLM that emits several reasoning attempts; the superego role is the same LLM prompted to judge those attempts using rules $U=\\{u_1,\\dots,u_{10}\\}$, distilling the judgment into key points; the ego role is the same LLM prompted to write a concrete script $S$, execute it as $S'$, and then answer from $S'$ plus all prior material. The rule set is constructed in a contrastive offline stage: for each training question, GPT-4's high-quality key points are compared with a smaller model's suboptimal key points, the resulting patterns are collected, and GPT-4 summarizes them into ten rules for ten question types. In the single-model variant PSSD-SFT, two LoRA adapters trained on the id task and on the superego/ego task are merged into the base weights, so one forward pass sequence runs the whole self-denial loop.","core_discovery":"The central claim, stated on the paper's own terms, is that multi-agent reasoning methods work only when each agent is able to deny its own earlier answer in a targeted way. PSSD implements that ability by assigning the three psychoanalytic roles to LLMs in sequence: the intuition-based id role produces $l=5$ initial reasoning paths for a question; the rule-driven superego role uses a pre-extracted rule set $U$ to generate key points that single out flaws in those paths; and the script-centric ego role turns the question, attempts, and key points into a step-by-step script, executes the script, and then synthesizes all of it into the final answer $R$. The rule set is built offline: GPT-4 contrasts high-quality key points with suboptimal ones from a smaller model, extracts patterns from training questions, and summarizes ten rule types that the superego then reuses at inference time. The paper reports that on the four benchmarks PSSD beats CoT with self-consistency, ReAct, VE, and Self-Contrast on almost all metrics, with e.g. EM scores of 47.08 on AdvHotpotQA, 41.96 on 2WikiMultiHopQA, 96.80 on GSM8K, and 77.20 on MATH, and that PSSD-SFT, obtained by merging two LoRA adapters into one open-source model, outperforms the fine-tuning baselines on most of the same tasks.","pith_inferences":["One immediate test the paper leaves open is cross-dataset transfer: running PSSD on a fifth benchmark with the same rule set would show whether the ten rules generalize beyond the four training distributions.","The reported PM and RM numbers suggest the bottleneck shifts from generation to selection: the correct answer is often already present among the id's attempts, so the crucial skill is picking and repairing, not creating new candidates.","Because PSSD-SFT collapses three roles into one model, an extension would be to train the entire loop on a larger base model or to make the number of correction rounds adaptive, using the ego's confidence to stop early.","The confidence analysis shows PSSD can be overconfident on wrong samples; pairing the loop with a cheap external verifier, as the compatibility study starts to do, is a natural place to look for the next gain."],"forward_implications":["A single rule-guided correction loop can replace resource-heavy debate: PSSD uses three agents and five LLM calls per question, versus 7.8 for Self-Contrast, and PSSD-SFT uses one model and two calls.","The superego's rule set is a reusable artifact: once the ten rules are summarized from training data, they are applied to every new question, so the per-question inference cost stays low.","Tool-based retrieval is not excluded by the method: adding ReAct or VE on top of PSSD raises exact match further, for example PSSD+ReAct reaches 48.05 on AdvHotpotQA, showing self-denial and external verification are complements.","Ablation results imply each role earns its place: removing the superego drops EM by 1.92 on AdvHotpotQA and 2.40 on GSM8K, while removing the ego drops it by 1.63 and 1.20."],"supporting_citations":[{"why":"Supplies the GSM8K grade-school math benchmark used for mathematical reasoning evaluation.","marker":"[3]"},{"why":"Supplies the id/superego/ego psychoanalytic structure that the three-role design is modelled on.","marker":"[7]"},{"why":"Supplies the MATH benchmark used to evaluate advanced mathematical reasoning.","marker":"[8]"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method PSSD-SFT uses to merge the three roles into one model.","marker":"[10]"},{"why":"GPT-4-turbo technical report; GPT-4 is the default model for PSSD and for the rule extraction process.","marker":"[14]"},{"why":"Provides the CoT-SC baseline and the self-consistency idea underlying the id role's multiple attempts and the confidence metric.","marker":"[21]"},{"why":"ReAct is a tool-using baseline and also the retrieval component combined with PSSD in the compatibility study.","marker":"[25]"},{"why":"Self-Contrast is the multi-agent debate baseline PSSD compares against for accuracy, stability, and number of LLM calls.","marker":"[29]"},{"why":"Verify-and-Edit is a second tool-using baseline and compatibility target that shows PSSD can be stacked with external verification.","marker":"[30]"}],"fun_headline_variants":["Psyche roles let LLMs deny errors and fix answers","Id, ego, superego prompts improve LLM reasoning","PSSD: human psyche structure for self-correcting LLMs","Multi-agent psyche prompting boosts reasoning accuracy","Three-role psyche method helps LLMs self-deny mistakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ten-type rule set extracted from the four benchmark training sets keeps paying off on new questions; if those rules are tuned to the distribution of these particular benchmarks, the superego's key-point guidance would add nothing on unseen problem types.","fun_headline_variants_meta":{"raw":{"variants":["Psyche roles let LLMs deny errors and fix answers","Id, ego, superego prompts improve LLM reasoning","PSSD: human psyche structure for self-correcting LLMs","Multi-agent psyche prompting boosts reasoning accuracy","Three-role psyche method helps LLMs self-deny mistakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1665,"prompt_tokens":1110,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":726,"tokens_out":555,"duration_ms":5352,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:36:25.363811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out reasoning set whose question types are absent from the four training sets, and compare PSSD against the paper's own no-rule ablation, i.e. superego without $U$. If the full method is not more accurate than the no-rule version, the rule-based guidance is not what generates the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the id/superego/ego psychoanalytic structure that the three-role design is modelled on."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Provides LoRA, the parameter-efficient fine-tuning method PSSD-SFT uses to merge the three roles into one model."},{"cited_title":"Le, Ed H","cited_arxiv_id":null,"evidence_quote":"Provides the CoT-SC baseline and the self-consistency idea underlying the id role's multiple attempts and the confidence metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Self-Contrast is the multi-agent debate baseline PSSD compares against for accuracy, stability, and number of LLM calls."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Verify-and-Edit is a second tool-using baseline and compatibility target that shows PSSD can be stacked with external verification."}],"review_version":1}