{"id":"49b94707-c986-496e-b6ad-017fbc7526dc","arxiv_id":"2507.08325","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent LLM pipeline for rewriting e-commerce CRM messages reports large quality gains, but the gains are judged by the same model that produces the rewrites, so they are not independently validated.","lead":"CRMAgent is a multi-agent LLM system that rewrites weak e-commerce CRM messages using a merchant's own top performers or similar campaigns from other merchants. The paper reports large gains in audience fit and marketing effectiveness, but those gains are scored by the same AI model that writes the rewrites, so external validation is still missing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-referential GPT-4o evaluation is the sole measure of quality; no human or behavioral validation supports the central outperformance claim.","rationale":"The reader's weakest-assumption identification is correct and is the most load-bearing concern in the paper. For the abstract's claim to hold, EvaluateAgent judgments must proxy customer preferences or engagement. This condition is least secure because the only validation of model outputs uses the same model family that generated them, with no external anchor. The paper has real strengths: a clear multi-agent architecture, detailed prompts, use of real engagement data to construct exemplars, and ablations across model configurations. But those strengths do not address external validity. The appendix's quality analyses using o3 are also automated and do not provide human validation. Therefore rejection is justified, but as an 'insufficient evidence' rejection, not a claim that the system cannot work. If a human evaluation or behavioral A/B test were supplied, the verdict could change substantially.","tokens_in":11964,"tokens_out":3275,"duration_ms":39229,"concrete_test":"Using the existing Appendix A engagement logs, compute the rank correlation (e.g., Spearman) between EvaluateAgent's audience-match and marketing-effectiveness scores for original merchant templates and their observed 7-day engagement scores, within each audience segment. If the correlate is near zero or negative, the judge does not measure real message quality and the central claim fails. Complement this with a randomized A/B test on a subset of generated versus original templates measuring click-through and conversion over 7 days; if generated templates do not beat originals on behavioral metrics, the claimed gains are artifacts of LLM self-preference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that CRMAgent delivers significant gains in audience-match and marketing-effectiveness—is measured exclusively by EvaluateAgent, which is GPT-4o (Section 4.1). The same model family generates the rewrites (TemplateAgent) and judges them, and the scoring prompts (Figures 5–6) reward exactly the stylistic properties TemplateAgent is instructed to add (urgency, clarity, incentive framing; Figures 3–4). No human raters, click-through rates, conversions, or live A/B results appear anywhere; the only data with real engagement signals (Appendix A) is used to pick exemplars, not to validate outputs. Ablation Table 2 shows the conclusion is evaluator-dependent: switching EvaluateAgent to DeepSeek-R1 changes the preference rate from 78.44% to 55.40%, and the paper itself attributes the discrepancy to a 'mismatch between its scoring criteria and human-aligned quality judgments' without supplying any human baseline. Position is also not counterbalanced: Appendix B.2 states A always corresponds to the original and B to the generated message. Thus the 9.09% audience and 38.44% market gains are unvalidated preferences of the generator's own model, not evidence about customer behavior. Additionally, Table 1's 'consistently outperforms' is internally questionable: audience score decreases in 5 of 11 segments, with the overall positive delta driven by two early-stage segments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CRMAgent, a multi-agent LLM system that rewrites underperforming e-commerce CRM message templates. It combines a ContentAgent for diagnosing template weaknesses, a RetrievalAgent for finding similar high-performing exemplars, a TemplateAgent for rewriting via group-based learning, retrieval-and-adaptation, or rule-based fallback, and an EvaluateAgent for scoring and pairwise preference judgments. The authors report experiments on a dataset derived from 3 million CRM message logs across 11 audience segments, claiming that generated templates consistently outperform merchant originals, with a 9.09% relative increase in audience-match score, a 38.44% relative increase in marketing-effectiveness score, and a 78.44% preference rate from the evaluator.","tokens_in":12226,"tokens_out":3244,"duration_ms":36035,"significance":"If the central claim were supported by valid evidence, CRMAgent would be a practically useful system for e-commerce CRM content generation, and the paper's modular multi-agent design and the use of segment-specific exemplars are reasonable ideas. The paper is transparent about its prompts and architecture, provides an ablation study across different backbone models, and reports segment-level results that allow readers to see variance. However, the empirical foundation is not valid for the claims made: the only quality signal comes from an LLM judge (GPT-4o) from the same model family that generates the rewrites, and no human ratings, click-through, conversion, or live A/B metrics are provided. The reported 'significant gains' are therefore unvalidated preferences of the generator's own model, not evidence about real message quality or customer behavior. The paper's own ablation and appendix statements further undermine the robustness of the evaluation. As it stands, the paper does not deliver on its advertised contribution of demonstrating practical impact.","major_comments":[{"comment":"The central claim that CRMAgent 'consistently outperforms' merchant originals rests entirely on EvaluateAgent, which is implemented with GPT-4o—the same model family used by TemplateAgent to generate the rewrites. The scoring and comparison prompts (Figures 5 and 6) explicitly reward urgency, clarity, incentive framing, and strong calls to action, which are exactly the features TemplateAgent is prompted to add (Figures 3 and 4). Thus the 78.44% preference rate and the score gains in Table 1 measure how well the generated text matches the evaluator's stylistic preferences, not whether real customers find the messages more relevant or persuasive. No human ratings, click-through rates, conversions, or A/B results are reported anywhere. This is a circular evaluation and does not support the abstract's claim of 'significant gains in both audience-match and marketing-effectiveness metrics.'","section":"§4.1, Figures 5–6"},{"comment":"The claim of 'consistently outperforms' is contradicted by the segment-level audience scores: audience score declines in 5 of 11 segments (Unpaid Order Buyers −1.21%, Post-Purchase Group −3.46%, Price-Drop Group −4.01%, Active Old Followers −4.19%, Repeat Buyers −4.08%) and is unchanged for Frequent Buyers. The overall +9.09% is driven mainly by two early-stage segments (Potential New Customers +41.87% and New Buyers +20.66%). An aggregate relative change over a non-uniform distribution of segment sizes does not justify the word 'consistently,' and the paper does not provide significance tests or confidence intervals for these differences.","section":"Table 1"},{"comment":"The ablation study shows that the main result is evaluator-dependent: when EvaluateAgent is switched to DeepSeek-R1, the preference rate drops from 78.44% to 55.40% while the score improvements are larger. The paper attributes this to a 'potential mismatch between its scoring criteria and human-aligned quality judgments,' but no human baseline is supplied. This admission that the GPT-4o evaluator's criteria may not be human-aligned undermines the validity of the primary evaluation, since the entire outperformance claim depends on that specific evaluator.","section":"§4.2, Table 2"},{"comment":"The comparison evaluation is not position-balanced. Appendix B.2 states that 'A and B correspond to the original and generated versions respectively,' meaning Message A is always the original and Message B is always the generated message. The prompt in Figure 6 claims the evaluator does not know which is which, but because the assignment is fixed, any positional bias in the LLM judge is fully confounded with the generation effect. The paper should counterbalance the presentation order across pairs, or at minimum report a position-controlled analysis.","section":"Appendix B.2"}],"minor_comments":[{"comment":"The scoring scale is described as three levels (1, 3, 5), which precludes intermediate ratings; this coarse scale may inflate observed score differences and should be justified or replaced with a finer scale.","section":"§3.4, Figure 5"},{"comment":"The paper does not report any measure of inter-evaluator agreement or reliability for EvaluateAgent, and it does not report the variance of the scores; without this, the aggregate Δ (%) values are hard to interpret.","section":"§4.1, Table 1"},{"comment":"The error-type analysis in Appendix C is performed by o3, another LLM, and the 'custom evaluation sheet' is provided in Appendix D, but there is no validation of this automated error labeling against human labels; this is an additional unvalidated LLM-as-judge component.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is unsupported by the evidence as presented. The only quality metric is an LLM judge from the same model family as the generator, and the paper itself provides evidence that the judge's preferences are not stable across models. The paper would need a fundamentally new evaluation (human ratings, behavioral outcomes, or at least a validated, position-balanced non-generator judge) to support its claims, which is beyond a routine revision. I see no grounds for acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a sensible, well-described industrial system for rewriting underperforming e-commerce CRM templates, but the paper's headline claim rests entirely on an evaluation in which the same model family writes the copy and grades it. That makes the 78.44% preference rate and the 38.44% 'market score' gain evidence about GPT-4o's stylistic preferences, not about customer behavior.\n\nWhat's actually new: the three-mode routing is a reasonable contribution. When a merchant has good templates in the same audience segment, the system learns from those; otherwise it retrieves cross-merchant examples by embedding product category and voucher metadata; if nothing matches, it falls back to a rule-based rewrite. That is a practical design for a real problem. The prompts in Figures 2-6 are concrete and the case studies give a clear sense of the output. The ablation is also honest enough to reveal that the conclusion depends on the evaluator: switching EvaluateAgent to DeepSeek-R1 drops the preference rate to 55.40%, which the paper attributes to a mismatch between its scoring criteria and human-aligned quality.\n\nThe soft spots are in the evaluation. There are no human ratings, no click-through or conversion metrics, no live A/B test. The scoring prompts reward urgency, clarity, and incentive framing, which is exactly what TemplateAgent is told to add. Position is not counterbalanced: the paper states in Appendix B.2 that A is always the original and B is always the generated message. And Table 1 undercuts 'consistently outperforms': audience score falls in 5 of 11 segments, and the overall positive delta is driven mainly by two early-stage segments. The dataset and code are not available, so the reader has to take the pipeline and the numbers on faith.\n\nWho should read this: practitioners building LLM-based copy systems will find the architecture useful, and the self-referential evaluation is a cautionary tale for anyone tempted to use LLM judges without validation. It does not change theory or open a new research direction.\n\nFor peer review: I'd send it out if the venue can demand a human or behavioral evaluation as a condition of revision. The system description deserves referee time, but the empirical claim as written is not supported. That's a revise-major, not a straight accept.","headline":"Useful industrial system, but the only evidence for its headline claim is the same LLM that writes the copy judging the copy.","tokens_in":12756,"tokens_out":2612,"would_cite":false,"duration_ms":28752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRMAgent is a four-agent LLM system that rewrites weak e-commerce CRM messages, and the paper reports it beats merchants' originals on LLM-judged audience fit and marketing effectiveness.","keywords":["CRM message templates","multi-agent LLM system","e-commerce private-domain messaging","retrieval-augmented generation","template rewriting","audience segmentation","LLM evaluation","marketing copy generation"],"falsifier":"Run a randomized A/B test on the same 11 audience segments, sending the original template to one set of customers and the generated template to another, and compare seven-day click-through and conversion rates; if the generated templates do not produce higher engagement, or if human raters prefer the originals, the claimed marketing-effectiveness gains are not demonstrated.","tokens_in":11759,"feed_emoji":"📨","tokens_out":8972,"duration_ms":84097,"temperature":0.7,"pith_summary":"CRMAgent is a multi-agent LLM system for rewriting underperforming e-commerce CRM messages, such as push notifications and in-app email, into more persuasive templates. The paper's central claim is that, across 3,957 message pairs in 11 audience segments, the generated templates beat the merchants' original templates: the LLM judge scores audience fit 9.09% higher, marketing effectiveness 38.44% higher, and prefers the generated version in 78.44% of blinded comparisons. The system is designed so that merchants without copywriting expertise could automate the rewrite using their own successful messages, similar cross-merchant templates, or a fallback rewrite when no reference exists. A sympathetic reader would care because the method is concrete and the reported gains are sizable, though the evaluation rests entirely on LLM judgments with no human or behavioral data reported.","feed_headline":"LLM agents rewrite CRM messages, lifting market scores 38%","feed_subtitle":"Four-agent system beats merchant originals on audience fit and persuasion in 78% of blinded LLM comparisons.","key_machinery":"The load-bearing mechanism is the four-agent decomposition with a three-path generation router inside TemplateAgent. In path one, group-based learning uses a merchant's own top-quartile messages from the same audience segment as exemplars; in path two, retrieval-and-adaptation fetches cross-merchant templates matched on audience segment, product category, and voucher type using dense embeddings from a distilled sentence-embedding model indexed for fast similarity search; in path three, a rule-based zero-shot rewrite handles cases with no usable references. EvaluateAgent is the outcome instrument: it produces the three-level audience-match and marketing-effectiveness scores and the blinded pairwise preference that constitute the paper's evidence. In the reported configuration, ContentAgent, TemplateAgent, and EvaluateAgent all run on GPT-4o, so the same model family both writes the copy and grades it.","core_discovery":"The paper claims that a task-specialized multi-agent LLM pipeline can turn weak e-commerce CRM templates into stronger ones. CRMAgent decomposes the work into four roles: ContentAgent compares low- and high-performing templates within an audience segment to explain why some messages engage better; RetrievalAgent searches for high-performing cross-merchant templates with matching audience label, product category, and voucher type; TemplateAgent rewrites the original message using either in-group exemplars, retrieved exemplars, or a rule-based prompt; EvaluateAgent scores both versions on audience match and marketing effectiveness or selects the preferred version without knowing which is original. The empirical payload is the head-to-head result: generated messages score 4.56 versus 4.18 on audience fit (a 9.09% relative gain) and 4.61 versus 3.33 on marketing effectiveness (a 38.44% relative gain), and EvaluateAgent prefers the generated version in 78.44% of blind comparisons. The paper adds BERTScore-F1 of 0.87 and chrF of 24.22 to show that rewrites keep the original intent while changing the wording. The authors present this as evidence that multi-agent LLM systems can serve as scalable CRM content support.","pith_inferences":["The decisive test the paper does not run is behavioral: a live A/B test measuring click-through, conversion, or unsubscribe rates on original versus generated templates would show whether the LLM judge's 78.44% preference transfers to real customers; until then, the practical claim should be read as 'preferred by an LLM evaluator' rather than 'preferred by customers'.","A straightforward extension would be a human evaluation panel drawn from the target audience segments scoring the same anonymized pairs; if human preference lands well below 78.44%, that would indicate the judge is rewarding the stylistic features its own prompt asks for, such as urgency and concrete discounts, rather than generally persuasive copy.","Because ContentAgent, TemplateAgent, and EvaluateAgent share the same GPT-4o family, the diagnosis and the grade are not independent; rerunning evaluation with a different model or human raters would make the reported improvements more credible.","The audience-score dips for price-drop seekers, repeat buyers, and new followers suggest the rewrite policy trades off targeting precision for urgency; an extension would be per-segment rewrite policies that preserve price-sensitivity cues where they are load-bearing."],"forward_implications":["If CRMAgent works as reported, a merchant without a copywriting team can upload underperforming templates and audience labels and receive a rewritten template plus an explanation of what was wrong with the original and why the new version fits the segment better.","The retrieval path makes the system improve as the shared template pool grows: more high-performing cross-merchant examples means more campaigns get exemplar-guided rewrites instead of the rule-based fallback.","Because the largest market-score gains appear where baseline scores are lowest, such as potential new customers with a 52.23% relative gain, the biggest practical payoff would be in acquisition-oriented segments, while mature segments like frequent buyers would gain less.","The reported BERTScore-F1 of 0.87 and chrF of 24.22 suggest the rewrite can be paired with an automatic similarity guardrail to detect when a generated template drifts from the original offer or intent."],"supporting_citations":[{"why":"Supplies the GPT-4o backbone used by ContentAgent, TemplateAgent, and EvaluateAgent in the main experiments.","marker":"[15]"},{"why":"Provides the sentence-embedding method that represents CRM template metadata for retrieval.","marker":"[22]"},{"why":"Supplies the distilled MiniLM model used to compute dense embeddings for retrieval.","marker":"[25]"},{"why":"Provides the fast similarity index used to retrieve top-k cross-merchant templates.","marker":"[3]"},{"why":"Supplies BERTScore-F1, the semantic-alignment metric used to verify rewrites preserve the original intent.","marker":"[29]"},{"why":"Supplies chrF, the character n-gram metric used to measure surface-level overlap between original and generated messages.","marker":"[18]"}],"fun_headline_variants":["Multi-agent LLM rewrites CRM messages with 38% lift","Four-agent LLM pipeline improves e-commerce CRM templates","CRMAgent: AI agents outdo merchants at writing CRM copy","LLM agents raise CRM message effectiveness by 38%","Blind comparison: LLM-generated CRM copy preferred 78%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the same GPT-4o model family that rewrites the messages also judges them reliably, because no human raters, click-through rates, conversions, or live A/B results are reported anywhere in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM rewrites CRM messages with 38% lift","Four-agent LLM pipeline improves e-commerce CRM templates","CRMAgent: AI agents outdo merchants at writing CRM copy","LLM agents raise CRM message effectiveness by 38%","Blind comparison: LLM-generated CRM copy preferred 78%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1632,"prompt_tokens":987,"completion_tokens":645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":603,"tokens_out":645,"duration_ms":6895,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:22:09.312116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a randomized A/B test on the same 11 audience segments, sending the original template to one set of customers and the generated template to another, and compare seven-day click-through and conversion rates; if the generated templates do not produce higher engagement, or if human raters prefer the originals, the claimed marketing-effectiveness gains are not demonstrated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies chrF, the character n-gram metric used to measure surface-level overlap between original and generated messages."}],"review_version":1}