{"id":"68bfa421-45ef-443f-9041-fd8c2c01cb2a","arxiv_id":"2601.13481","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A multi-agent prompt-rewriting loop is claimed to improve LLM emotion diagnosis accuracy, but its evaluation appears to optimize on the test set and lacks replication details.","lead":"This paper describes APOLO, a five-agent system that automatically rewrites instructions to a language model so it can detect emotions like depression, anxiety, and loneliness in text. The authors report strong gains on six benchmarks, but their optimization loop appears to score and select prompts on the test set, so the headline results are not trustworthy as presented.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"APOLO's reported gains are not independent estimates: Eq. (1), Algorithm 1 line 15, and §3.4/§4.1 compute the optimization reward on the test set, so Table 1 scores are test-set selected/stopped, not held-out evaluations.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the optimization procedure uses D_test reward in Eq. (1), Algorithm 1 line 15, and Sections 3.4/4.1, so the final reported scores are not independent held-out estimates. My reading of the full text confirms this and adds the internal inconsistency: §4.1 mentions a validation set for early stopping while Algorithm 1 and §3.4 use D_test, so the paper cannot simultaneously have both a clean validation-based selection and a test-set-based stopping rule. Because every headline number in Table 1 depends on this protocol, the central claim is unsupported by the evidence as presented. I considered whether this could be treated as a minor wording issue, but the formal objective in Eq. (1) explicitly maximizes over D_test, and Eq. (9) defines the reward as a sum over D_test, making the contamination structural, not cosmetic. The proposed re-evaluation with a genuine train/validation/test split would settle whether APOLO actually improves over OPRO on unseen data. Until that is done, the empirical contribution cannot be accepted.","tokens_in":23077,"tokens_out":2806,"duration_ms":33389,"concrete_test":"Re-run APOLO on all six datasets with a strict three-way split: use a held-out validation split for R(t) in Algorithm 1 line 15 and for the early-stopping rule, select the final prompt based only on validation reward, and evaluate that prompt exactly once on an untouched test split. Compare the resulting Macro F1 and Micro F1 (per dataset and averaged) against Table 1's reported averages of 49.25% and 64.72% and against OPRO. If the advantage over OPRO shrinks substantially, reverses, or falls within run-to-run noise, the original Table 1 numbers are test-set artifacts rather than evidence of generalization.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that APOLO 'consistently improves diagnostic accuracy and robustness' over baselines (Table 1). For that claim to hold, the reported numbers must be independent evaluations on data not used to guide prompt selection or stopping. The paper contradicts this in multiple places. Eq. (1) defines the objective as arg max over D_test of f(pi_tar(x|p), Y). Algorithm 1 line 15 sets R(t) = sum over (x,Y) in D_test f(...), and line 16 stops when the test-set reward gain falls below δ. Section 3.4 says the Target agent evaluates on 'the held-out test set' and uses that to close the optimization loop. Section 4.1 says the Target agent evaluates 'on the entire test set of the respective dataset to track the optimization trajectory comprehensively.' The only contrary sentence is the early-stopping description in §4.1, which mentions a 'validation set' — an internal inconsistency, not a fix. Under either reading, the paper has not provided an independent evaluation: if D_test is the final test split, every headline score is selected/stopped using test labels; if D_test is actually a validation split, then the paper mislabels its evaluation set and the claimed test-set generalization is unverified. This invalidates the main comparison in Table 1 and propagates to the ablations in Table 2, the convergence analysis in Figure 5, the initial-prompt robustness in Table 3, and the efficiency analysis in Figure 4, all of which rely on the same protocol. This is not a disagreement with consensus; it is a correctness risk in the core empirical argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APOLO, a multi-agent prompt-optimization framework for linguistic emotion diagnosis. APOLO models prompt refinement as a POMDP and uses Planner, Teacher, Critic, Student, and Target agents to iteratively generate and refine prompts. Experiments across six emotion-diagnosis datasets and three LLM backbones report consistent gains over baselines (e.g., +2.96 Macro F1 and +3.40 Micro F1 over OPRO on average), together with ablations, convergence, and sample-efficiency analyses. The central claim is that APOLO improves diagnostic accuracy and robustness for LLM-based emotion diagnosis.","tokens_in":23626,"tokens_out":7372,"duration_ms":77244,"significance":"The topic is timely, and the multi-agent POMDP formulation is a plausible design for automated prompt optimization in a high-stakes domain. The paper is broad in coverage: six datasets, three backbones, ablations, and efficiency analyses. If the reported numbers were independent held-out estimates, the framework would be of practical interest for mental-health text analysis. However, the evaluation as specified uses test-set labels inside the optimization loop, so the headline results are not independent estimates. The theoretical bound is also non-substantive. The framework's empirical contribution is therefore not established by this manuscript.","major_comments":[{"comment":"The optimization loop uses D_test as the reward signal. Eq. (1) defines the objective as argmax over D_test; Algorithm 1 line 15 sets R(t) by summing f over D_test, and line 16 stops when the D_test reward gain falls below δ. Section 3.4 calls D_test the 'held-out test set,' and Section 4.1 says the Target agent evaluates 'on the entire test set... to track the optimization trajectory comprehensively.' The only mention of a validation set is the early-stopping sentence in §4.1, which contradicts the algorithm. Thus every score in Table 1 is selected or early-stopped using test labels, not an independent evaluation. This invalidates the central comparison and propagates to Tables 2–4 and Figures 4–5, which use the same protocol. The authors must define a genuine validation split, use it for all prompt selection and early stopping, evaluate the final prompt once on an untouched test split,","section":"§3.4, Eq. (9); Algorithm 1 lines 15–16; §4.1"},{"comment":"The 'Unified improvement bound' is not a substantive theoretical guarantee. The quantities A_i, B_r, and B_c are introduced by assumption, and the proof sketch states that standard KL-regularized policy improvement gives ΔR_i ≥ A_i − σ²/(2λ), which is essentially the conclusion being proved. No measurable or verifiable conditions are supplied, so the bound cannot be checked or falsified. The claim of monotonic improvement is therefore unsupported. Either derive a bound from observable quantities or clearly present this as intuition rather than a theorem.","section":"§3.5, Eq. (11)"},{"comment":"The evaluation protocol is not specified in enough detail to assess fairness. It is unclear what data the baselines (APE, ProTeGi, OPRO, PE2) use for prompt selection and early stopping. Table 4 compares APOLO using 0/1/3 samples against baselines using 20–100 samples, but because APOLO's numbers come from the test-contaminated loop, this does not establish sample efficiency. In addition, no standard deviations, confidence intervals, or significance tests are reported for any table or figure; several differences are small, and statistical testing is needed to support the claims.","section":"§4.1, Table 4"}],"minor_comments":[{"comment":"The text says 'two distinct large language models' but immediately lists three (GPT-5-mini, DeepSeek-V3, Qwen-32B).","section":"§4.1"},{"comment":"The section heading contains a typo: 'Converagence' should be 'Convergence.'","section":"§5.2"},{"comment":"The distribution D in the joint policy optimization objective is not defined. It should be specified as the validation distribution (or training/validation split) that is not used for final test evaluation.","section":"Eq. (8)"},{"comment":"The final Target prompt for the DepressionEmo task is internally contradictory: it instructs the model to identify 'all relevant psychological emotion labels' and gives a multi-label example, but then ends with 'Your output MUST be a single word representing the emotion.' This needs to be reconciled for a multi-label task.","section":"Table 5"},{"comment":"The Planner is described as producing a 'six-step plan,' but the enumeration lists two 'Step 2' entries, so the plan structure is unclear.","section":"§6.3"},{"comment":"The conclusion states results on 'multilingual' benchmarks, but all six datasets are English. This overclaim should be removed or supported.","section":"Conclusion"},{"comment":"The efficiency analysis does not explain how token counts and scaling curves are computed, nor whether the Target agent's test-set scoring calls are included in the token totals. This should be clarified.","section":"§4.3, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The test-set contamination is severe and undermines every empirical headline. I view this as a borderline reject/major-revision case. If the authors cannot supply a clean re-analysis with a true held-out test set and unchanged conclusions, the paper should not be accepted; a major revision is warranted only because the framework itself is potentially sound and the evaluation flaw is in principle correctable by re-running the experiments with a proper validation split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the APOLO results in Table 1 are not independent estimates. The objective in Eq. (1), the reward in Algorithm 1 line 15, and the Target agent's evaluation in §3.4 all use the test set to select and stop the prompt search. §4.1 mentions a validation set for early stopping but then says the Target evaluates on the entire test set to track the optimization trajectory. Under either reading, the reported numbers are test-set-selected, so the claimed gains over OPRO and the other baselines do not establish generalization.\n\nWhat is genuinely useful: the paper is open about being an extension of MARS, the six-dataset/three-LLM evaluation is extensive, and the ablation study is internally consistent. The case study in §6.3 gives a concrete sense of how the Teacher-Critic-Student loop refines prompts. If a proper validation split were used and the final test evaluation were truly held out, the framework would probably still show some benefit, though the gains over OPRO are modest.\n\nThe soft spots, in order:\n1. The test-set leakage invalidates the central claim. This is not a minor quibble; every headline number in Table 1, plus the ablations and convergence plots, inherits it.\n2. The POMDP and risk/cost machinery is decorative. Eqs. (3)-(5) and the 'partially observable decision system' are not implemented as a real POMDP; the actual mechanism is the Socratic dialogue from MARS. The free parameters gamma_r and gamma_c never appear in the experiments, and the theoretical bound in Eq. (11) is constructed from definitions, not proved.\n3. No code or error bars. For a prompt-optimization claim, readers need seeds, variance, and the final prompts used; the paper gives one example prompt.\n\nWho is this for? Someone working on LLM prompt optimization for applied affective NLP might read it as a data point, but only after the evaluation is fixed. As is, I would not cite it.\n\nRecommendation: send to peer review only if the editor and reviewers are willing to demand a re-run with a clean validation/tuning split. The flaw is fixable in principle, and the experimental infrastructure is substantial. But if the journal expects the current numbers to stand, it should be desk rejected.","headline":"The reported gains are test-set-selected rather than independently evaluated; the framework is an honest but incremental extension of MARS wrapped in decorative POMDP language.","tokens_in":24067,"tokens_out":2124,"would_cite":false,"duration_ms":20882,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-agent prompt-optimization loop improves LLM emotion diagnosis on six benchmarks, with the largest gains on multi-label depression data.","keywords":["automated prompt optimization","multi-agent LLM","emotion diagnosis","mental health","emotional comorbidity","POMDP","multi-label classification","affective computing"],"falsifier":"Run APOLO with a true held-out test set that is never touched during optimization (use only a development split for the Target agent's reward), then compare Macro F1 to the baselines under identical budgets; if the advantage over the strongest baseline disappears or reverses, the central claim that the multi-agent loop improves diagnostic accuracy is not supported.","tokens_in":23010,"feed_emoji":"🧠","tokens_out":3569,"duration_ms":37736,"temperature":0.7,"pith_summary":"The paper tries to establish that automatic prompt optimization can make large language models more reliable at diagnosing emotions in mental-health text. It proposes APOLO, a closed loop of five LLM agents — planner, teacher, critic, student, target — that treats instruction refinement as a partially observable Markov decision process. The claim is that this loop finds prompts that catch co-occurring emotions better than hand-written or single-agent optimized prompts, improving Macro F1 by about three points on average across six datasets and three backbones. If true, it would give clinicians and online mental-health platforms a way to improve LLM diagnostic accuracy without retraining or hand-engineering prompts.","feed_headline":"Automated five-agent loop lifts emotion diagnosis across six datasets","feed_subtitle":"Prompt refinement catches co-occurring depression and anxiety cues, with strongest gains on multi-label mental-health text.","key_machinery":"The carrying mechanism is the closed-loop multi-agent refinement cycle. A Planner turns the diagnosis task into a sequence of sub-goals under risk and cost penalties; a Teacher–Critic–Student triad runs Socratic dialogue to rewrite the prompt step by step; a Target agent scores the resulting prompt on a held-out test set and decides whether to continue. The framework models this as a Partially Observable Markov Decision Process whose latent state is the diagnostic hypothesis and whose observation is the current instruction, so prompt edits are treated as uncertainty-reducing actions rather than raw text mutations.","core_discovery":"The central claim is that structuring prompt optimization as a multi-agent decision process, with risk- and cost-aware trajectory planning plus Socratic teacher-critic-student refinement, yields prompts that diagnose co-occurring emotional states more accurately than baseline prompting and optimization methods. On six emotion benchmarks and three LLM backbones, the optimized prompts are reported to outperform the strongest baseline by roughly 3% Macro F1, with especially large gains on the multi-label depression dataset where exact-match ratio rises to about 29%. The paper attributes the gain to the planner decomposing the task into sub-goals, the teacher-critic-student triad iteratively ref","pith_inferences":["If the loop is re-run with a separate validation set — as one passage in the paper claims, though its Algorithm 1 uses the test set to compute the reward and stop — the reported three-point gains may shrink; the paper's own algorithm makes the headline numbers maxima selected on the test data, so an independent held-out evaluation is the natural next check.","The same Planner–Teacher–Critic–Student loop could be pointed at other high-stakes text tasks — suicide-risk triage, adverse-event detection in clinical notes, or crisis-counselor response scoring — where co-occurring categories and safety constraints matter.","Emotional comorbidity is handled implicitly through prompt phrasing; a more explicit approach would condition the prompt on predicted label subsets, which the current framework does not attempt.","The reported efficiency advantage suggests prompt optimization can be made cheap enough for per-task deployment, but the comparison depends on how baselines' token budgets are tallied; a like-for-like cost accounting would strengthen that claim."],"forward_implications":["Finding one good prompt can be automated end-to-end, so clinicians could obtain better instructions for LLM-based emotion screening without prompt-engineering expertise.","Gains concentrate on rare and co-occurring emotions, which matters for detecting suicide intent, hopelessness, and worthlessness that flat single-label prompts miss.","The method transfers across three very different LLM backbones, suggesting the optimization mechanism rather than a specific model drives the improvement.","One in-context example suffices to beat baselines that consume up to 100 samples, cutting the annotation and API cost of prompt search.","The optimized prompt for depression diagnosis evolves into an explicit multi-step chain-of-thought instruction, giving a legible audit trail for why certain emotions are selected."],"fun_headline_variants":["Multi-agent prompt optimizer sharpens mental-health emotion reads","Five-agent prompt loop boosts depression and anxiety diagnosis","Multi-agent refinement yields more robust emotion diagnosis","LLM prompt optimizer improves mental-health emotion accuracy","Multi-agent prompt tuning catches co-occurring mental-health cues"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's headline numbers rest on the assumption that optimizing the prompt using feedback computed on the test set still yields a prompt that generalizes to unseen data — if the test set was used to select the prompt, the reported gains are maxima chosen on that test data, not independent estimates.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent prompt optimizer sharpens mental-health emotion reads","Five-agent prompt loop boosts depression and anxiety diagnosis","Multi-agent refinement yields more robust emotion diagnosis","LLM prompt optimizer improves mental-health emotion accuracy","Multi-agent prompt tuning catches co-occurring mental-health cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2674,"prompt_tokens":763,"completion_tokens":1911,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1838}},"tokens_in":507,"tokens_out":1911,"duration_ms":14824,"temperature":1.0,"reasoning_tokens":1838,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:31:19.609675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run APOLO with a true held-out test set that is never touched during optimization (use only a development split for the Target agent's reward), then compare Macro F1 to the baselines under identical budgets; if the advantage over the strongest baseline disappears or reverses, the central claim that the multi-agent loop improves diagnostic accuracy is not supported.","supporting_citations":[],"review_version":1}