{"id":"ec072159-8182-4d0c-b087-4ce3d8a9b96e","arxiv_id":"2501.15826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The MADP framework uses Explorer, Empathizer, and Interpreter agents to plan LLM mental health responses, reporting roughly 4-5% improvements that may be fragile due to weak evaluation.","lead":"This paper proposes a three-agent AI framework based on cognitive behavioral therapy to write more empathetic and structured mental health support replies. If the gains hold, smaller open-source chatbots could give higher-quality support at lower cost, which matters for scalable mental health assistance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline improvements rest on scores from Claude 3.5 Sonnet with no judge validation; the near-uniform poor scores for human responses suggest a style bias that could inflate MADP's gains.","rationale":"The reader's weakest assumption—that Claude 3.5 Sonnet's automatic ratings are a valid and unbiased measure—is indeed the single most load-bearing concern. Every headline improvement, including the MADP-LLM claims, is a difference between scores produced by this one judge. My reading adds two concrete aggravating observations. First, Table 2's near-uniformly low scores for human responses are a red flag that the judge has a systematic stylistic preference; no human judge would plausibly rate all human-provided support at 4–6 while every LLM output scores above 7. This suggests the judge is not measuring the intended psychological-support constructs. Second, the human validation in Section 5.8 uses only 18 posts with no inter-annotator agreement or significance testing, so it cannot rescue the automatic evaluation. The paper does have real strengths: clear baselines, multiple backbone LLMs, two datasets, and a reasonable theoretical grounding in the ABC model. The framework itself is plausible, and the automatic gains are consistent across models and languages, which is a positive sign. However, consistency across a biased judge does not establish validity; a single systematic bias would propagate across all rows. The reader's CONDITIONAL verdict is appropriate: the empirical claims are plausible but not established. I do not recommend changing the verdict because the concern is about missing validation rather than demonstrated fraud, and the required validation is a straightforward and attainable addendum. I considered the potential train/test overlap between the original test posts and the MADP fine-tuning dataset as a secondary concern, but the paper is ambiguous about which test set Table 3 uses, and the automatic-judge issue affects even the prompt-only comparisons in Table 1, so the judge bias remains the priority.","tokens_in":13268,"tokens_out":6065,"duration_ms":56560,"concrete_test":"Run a judge-validation study on a stratified sample of responses from Table 1 conditions (e.g., 15 each from GPT4o, GPT4o-MADP, LLaMA3-8b, LLaMA3-8b-MADP, GLM4-9b, GLM4-9b-MADP) plus 15 original human responses. Have at least three psychology-trained raters score them blindly on the same 1–10 Analytical, Empathy, Guidance, and Comprehensive scales; compute Krippendorff's alpha and the correlation between mean human scores and Claude's scores. Recompute the Table 1 improvement percentages using human scores; if the MADP advantage shrinks to near zero or the correlation is low (e.g., <0.3), the central claim is not supported. As an auxiliary check, compare average response length across conditions and run a regression of human scores on condition with length as a covariate to see whether the MADP effect survives controlling for verbosity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of significant MADP improvements is measured entirely by Claude 3.5 Sonnet's automatic ratings (Section 5.2). The paper never validates this judge against human ratings: no human-rated judge-correlation study, no inter-annotator agreement, and the only human check is on 18 posts (Section 5.8) without reliability statistics. A concrete red flag is in Table 2: 'Human Response' baselines score 4.52–6.79, far below every LLM-generated response, which is implausible if the judge measures genuine support quality. This pattern suggests the judge rewards LLM-typical stylistic features—length, formatting, empathetic phrasing—rather than the four intended constructs. Since MADP systematically produces longer, more structured, and more emotionally expressive responses, every improvement percentage in Tables 1–3 could be inflated or spurious. The same unvalidated judge is used for both the framework comparison and the fine-tuned MADP-LLM evaluation, so the entire empirical edifice depends on this single measurement premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MADP, a multi-agent deductive planning framework for mental health question answering. The framework operationalizes the CBT ABC model through three agents (Explorer, Empathizer, Interpreter) that produce a multi-agent dialogue, a support plan, and a final support response. The authors also construct a MADP dataset using GPT4o and fine-tune LLaMA3-8b and GLM4-9b into MADP-LLM. Experiments on EMH and PsyQA report automatic evaluations by Claude 3.5 Sonnet and a small human evaluation, claiming average improvements of roughly 4-6% across Analytical, Empathy, Guidance, and Comprehensive scores, with fine-tuned models approaching or exceeding GPT4o on several metrics.","tokens_in":13495,"tokens_out":3002,"duration_ms":30035,"significance":"If the reported gains are reliable, the work is useful: it is one of the few MHQA approaches that explicitly models interactions among multiple CBT elements rather than a single cognitive prompt, and the idea of distilling a planning-based multi-agent pipeline into a small open-source model for local deployment is practically valuable. The paper also evaluates across English and Chinese datasets and includes an ethical statement cautioning against unsupervised real-world use. However, the central empirical claim rests entirely on an unvalidated LLM judge and a very small human evaluation, so the quantitative contribution is not yet established.","major_comments":[{"comment":"All headline improvement figures are produced by Claude 3.5 Sonnet as an automatic judge, but the judge is never validated against human ratings. No judge-human correlation, no agreement statistics, and no calibration analysis are reported. The human evaluation in §5.8 covers only 18 posts and reports no inter-annotator agreement. Because every percentage in Tables 1-3 depends on this single measurement premise, the paper should report a human-judge correlation study, per-item agreement, and confidence intervals or significance tests before claiming that MADP 'significantly enhances' performance.","section":"§5.2, Tables 1-3"},{"comment":"The 'Human Response' baselines score 4.52 and 6.79 on average, far below every LLM-generated response (all above 7.09). This pattern is implausible if the judge measures genuine support quality and suggests that the judge may reward LLM-typical stylistic features such as length, formatting, and empathetic phrasing rather than the four intended constructs. Since MADP systematically produces longer and more structured responses, the improvement percentages could be inflated by this style bias. The authors should investigate this with human-rated examples and, if the bias is confirmed, re-evaluate or re-calibrate the automatic judge.","section":"Table 2"},{"comment":"The fine-tuned MADP-LLM models are evaluated with the same unvalidated Claude judge, and their training labels are generated by GPT4o through the MADP pipeline. This self-referential setup means the reported 'match or exceed GPT4o' results could reflect the judge's preference for a particular output style rather than genuine support quality. The paper provides no human evaluation of MADP-LLM and no comparison of fine-tuned models against human responses beyond the suspiciously low -H scores in Table 3. Independent human evaluation of MADP-LLM, or at least judge validation on these models' outputs, is needed.","section":"§5.7, Table 3"},{"comment":"The statistical basis for the claims is absent. Scores are reported to one decimal place with percentage changes to two decimals, but no error bars, standard deviations, confidence intervals, or paired significance tests are provided. The test sets contain only 90 posts per dataset, so the differences in Tables 1-3 may be within sampling noise. The word 'significantly' in the abstract and §5.5 is not supported by any statistical test.","section":"§5.5 and §5.6"}],"minor_comments":[{"comment":"There is a typo: 'approachs' should be 'approaches'.","section":"§1"},{"comment":"The table formatting has readability issues, for example 'LLaMA3-8bft-MApr7.60' and 'LLaMA3-8bft-MApr7.81' run the model name and score together; please insert spacing or use a clearer separator.","section":"Table 3"},{"comment":"The dataset description says test pairs are 'randomly selected' but does not report the random seed or selection procedure. Also, for EMH it is not stated how the 'ideal answer' was chosen when multiple responses exist; this should be specified for reproducibility.","section":"§5.1"},{"comment":"The term MADP-LLM is used both as a singular specialized model in the abstract and as a family of fine-tuned models in Table 3. Please clarify whether MADP-LLM is one model or a collection of models.","section":"§4.1 and §4.2"},{"comment":"No data or code availability statement is provided. Since the paper introduces a new dataset and fine-tuned models, releasing the dataset, prompts, and training scripts would substantially support reproducibility.","section":"General"},{"comment":"The figure contains stray text such as 'oo' and the prompt text is very small; a cleaner, larger rendering would help readers follow the three stages.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the paper is not circular in its zero-shot comparisons, but the evaluation methodology is the main obstacle. The lack of judge validation and statistical testing affects every quantitative claim. I would not reject on novelty grounds; the proposed framework is a legitimate contribution if the empirical evidence is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen and Liu propose MADP, a three-agent framework inspired by the reverse ABC model, plus a planning stage, and they fine-tune small models on GPT4o-generated plans and responses. The design is genuinely new as a combination: prior work used single-agent CBT prompts or generic multi-agent debates, not the reverse A->C->B decomposition with a separate planning step. The paper is clearly written, the prompts are concrete, and the distillation of planning data into 8-9B models for local deployment is a practical contribution.\n\nThe evaluation is the weak load-bearing wall. All headline numbers come from Claude 3.5 Sonnet ratings, and the paper gives no evidence that those ratings correlate with human judgments. The only human check is 18 posts with three evaluators and no inter-annotator agreement reported. There are no error bars or significance tests anywhere, so \"significantly enhances\" is doing a lot of work. The stress-test concern about Table 2 is on point: human responses average 4.5-6.8 while every LLM scores above 7, which is implausible unless the judge is rewarding LLM-style length and formatting rather than actual support quality. That bias would inflate all the MADP gains, since MADP responses are longer and more structured.\n\nThe fine-tuning section has an additional self-referential element: GPT4o generates both the support plans and the target responses, and then the same unvalidated judge evaluates the fine-tuned models against baselines. The core zero-shot comparison is not circular, but the magnitude of the improvement is unproven.\n\nWhat holds up: the framework is grounded in a recognizable psychological model and the ablation separating response-only from plan-plus-response training is a sensible experiment design. The authors also flag the ethical limits of deployment, which is honest.\n\nWho should read it: people building applied MHQA systems or distilling structured reasoning into small models. For that audience the design ideas are useful even if the numbers are soft.\n\nRecommendation: send it to peer review, but require judge validation against human ratings, significance testing, a larger human study with agreement stats, and release of the dataset and prompts. The claims need tempering until then.","headline":"A plausible CBT-inspired multi-agent framework whose empirical claims rest on an unvalidated LLM judge; the idea is worth a round of revisions.","tokens_in":13975,"tokens_out":2642,"would_cite":false,"duration_ms":26343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-agent deduction built on the CBT ABC model improves LLM mental-health support scores by about five percent, and fine-tuning small models on the distilled data lets them match the larger model.","keywords":["multi-agent reasoning","mental health question answering","cognitive behavioral therapy","ABC model","support planning","LLM fine-tuning","empathetic response generation","cross-lingual transfer"],"falsifier":"Run a blinded preference study in which clinically trained raters score the same 180 test responses on the four dimensions with a published inter-rater agreement metric; if the human scores show no reliable difference between MADP and single-agent baselines while the automatic evaluator reports a 4–6 percent gap, the central improvement claim is falsified.","tokens_in":13098,"feed_emoji":"🧠","tokens_out":10374,"duration_ms":81473,"temperature":0.7,"pith_summary":"What the paper is trying to establish: a one-turn mental-health question answer (MHQA) system can do substantially better if, before writing a reply, it runs a three-agent deduction over the seeker's situation using the ABC model of cognitive behavioral therapy. The proposed framework, MADP, replaces single-agent CBT prompting with an Explorer, an Empathizer, and an Interpreter that converse in the reverse ABC order (activating event → emotional consequence → belief) and then produce a structured support plan. Across GPT4o, LLaMA3-8b, and GLM4-9b, the paper reports average gains of 4–6 percent on Analytical, Empathy, Guidance, and Comprehensive scores over the same models without the framework, on both an English and a Chinese dataset. It also reports that small open-source models fine-tuned on MADP-generated post–plan–response triples match or exceed GPT4o on several metrics, which matters because it points to a low-cost way to deploy structured, empathetic mental-health support locally.","feed_headline":"Three-agent CBT reasoning lifts LLM mental-health scores by ~5%","feed_subtitle":"Fine-tuning small models on plan-plus-response data lets them match larger closed-source systems.","key_machinery":"The load-bearing object is the ABC model of CBT — activating event (A), beliefs/cognitions (B), and consequences (C) — transposed into three named agents: Explorer ($A_{EX}$), Empathizer ($A_{EM}$), and Interpreter ($A_{IN}$). The framework's move is to run these agents in the reverse direction, $A \\to C \\to B \\to A$, so that emotion is understood and voiced before cognition is interpreted, and then to separate 'support planning' from response writing: the multi-agent dialogue $d_i$ is condensed into a support plan $k_i$ of concrete points and a theme, and the final response $r_i$ is generated only from post plus plan. This deduce-then-plan-then-write chain is the component that gets distilled into the fine-tuned MADP-LLM.","core_discovery":"The paper's central claim, stated on its own terms, is that mental-health support quality improves when reasoning about a help-seeker is decomposed along the CBT ABC model and run in reverse: first the Explorer identifies the activating events (A), then the Empathizer names the emotional consequences (C) and connects with them, and finally the Interpreter interprets the beliefs (B) behind those emotions and offers reframing. The authors attribute earlier methods' weak empathy scores to their focus on the cognitive element alone, and they trace MADP's improvement to the Empathizer's role in establishing emotional connection before cognitive interpretation. They further claim that capturing this reasoning as a support plan and using it as training data can distill the benefit into smaller models: MADP-LLM, fine-tuned with LoRA on plan-and-response data, matches or exceeds GPT4o's scores on several metrics despite a much smaller parameter count.","pith_inferences":["The reverse ABC decomposition may generalize beyond one-turn support to longer counseling dialogues or to non-CBT frameworks, since the ordering principle — emotion before interpretation — is a general counseling heuristic; a testable extension is to apply the same three-agent order to crisis-hotline transcripts.","The dataset's post–plan–response triples enable an intervention study the paper does not run: corrupt or remove the plan at inference time and measure the drop in rated quality, which would isolate how much of the gain comes from planning versus the multi-agent dialogue.","Because the automatic judge is the only large-scale outcome measure, correlating its scores with blinded clinician ratings on the same 180 responses would test whether the 4–6 percent improvement reflects support quality or output style.","The paper's conclusion that human responses underperform LLM responses is drawn from dataset supporter posts rather than professional counselors; comparing against trained counselors' replies would test whether the shortfall reflects the support quality or the dataset."],"forward_implications":["If the reported gains hold, closed-source frontier models are not necessary for high-scoring mental-health support: MADP-LLM, a LoRA fine-tune of LLaMA3-8b or GLM4-9b, reaches or passes GPT4o on several metrics on both English and Chinese test sets.","Because the same MADP prompts raise scores across three different backbones, the multi-agent deduction transfers across model families, sizes, and languages rather than being a quirk of one model.","The support plan acts as an explicit intermediate representation, so the framework produces a checkable record of what the model understood before it wrote the reply, which matters for accountability in a sensitive domain.","On the paper's evidence, cross-lingual fine-tuning helps: each small model improves more on the dataset written in its non-native language after MADP training, suggesting the plan-conditioned data teaches transferable support structure."],"supporting_citations":[{"why":"Supplies the EMH dataset (55 mental-health subreddits) and the three communication mechanisms — Explorations, Emotional Reactions, Interpretations — that define the agents' behaviors.","marker":"[Sharma et al., 2020]"},{"why":"Supplies the PsyQA dataset and the one-turn counseling question-answer task the framework targets.","marker":"[Sun et al., 2021]"},{"why":"The single-agent CBT prompt baseline that the paper argues misses interactions among CBT elements and whose empathy score motivates the Empathizer agent.","marker":"[Na, 2024]"},{"why":"The Cue-CoT baseline that first infers the seeker's state and then responds; MADP is compared against it in Table 2.","marker":"[Wang et al., 2023a]"},{"why":"LoRA, the parameter-efficient fine-tuning method used to train MADP-LLM from small open-source backbones.","marker":"[Hu et al., 2022]"},{"why":"The cognitive-reconstruction theory the Interpreter agent uses to reframe the seeker's beliefs.","marker":"[Carli, 1999]"},{"why":"HealMe, a recent CBT-based single-agent model, illustrating the prior work that MADP extends.","marker":"[Xiao et al., 2024]"}],"fun_headline_variants":["Reverse CBT reasoning helps LLMs answer mental-health posts better","Multi-agent plan fine-tunes small LLMs to match GPT-4o on MHQA","Explorer-Empathizer-Interpreter: new CBT multi-agent plan for mental-health QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 4–6 percent improvements rest on the assumption that the automatic evaluator's scores measure real support quality and that the 18-post human evaluation confirms them; if the judge merely prefers the more structured style of MADP answers, the effect sizes in the tables would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Reverse CBT reasoning helps LLMs answer mental-health posts better","Multi-agent plan fine-tunes small LLMs to match GPT-4o on MHQA","Explorer-Empathizer-Interpreter: new CBT multi-agent plan for mental-health QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000853,"raw_usage":{"total_tokens":3707,"prompt_tokens":946,"completion_tokens":2761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2701}},"tokens_in":562,"tokens_out":2761,"duration_ms":15588,"temperature":1.0,"reasoning_tokens":2701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:55:12.586379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a blinded preference study in which clinically trained raters score the same 180 test responses on the four dimensions with a published inter-rater agreement metric; if the human scores show no reliable difference between MADP and single-agent baselines while the automatic evaluator reports a 4–6 percent gap, the central improvement claim is falsified.","supporting_citations":[{"cited_title":"Miner, David C","cited_arxiv_id":null,"evidence_quote":"Supplies the EMH dataset (55 mental-health subreddits) and the three communication mechanisms — Explorations, Emotional Reactions, Interpretations — that define the agents' behaviors."},{"cited_title":"PsyQA : A Chinese dataset for generating long counseling text for mental health support","cited_arxiv_id":null,"evidence_quote":"Supplies the PsyQA dataset and the one-turn counseling question-answer task the framework targets."},{"cited_title":"Cognitive reconstruction, hindsight, and reactions to victims and perpetrators","cited_arxiv_id":null,"evidence_quote":"The cognitive-reconstruction theory the Interpreter agent uses to reframe the seeker's beliefs."},{"cited_title":"HealMe : Harnessing cognitive reframing in large language models for psychotherapy","cited_arxiv_id":null,"evidence_quote":"HealMe, a recent CBT-based single-agent model, illustrating the prior work that MADP extends."}],"review_version":1}