{"id":"8d980473-f75b-4493-be83-9cefbaf17ad5","arxiv_id":"2507.18993","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Agent0 coordinates an architect LLM, a sentinel LLM, and an AutoML oracle to discover multi-value categorical features from text, reporting a 0.005+ RIG lift in one production case study.","lead":"Agent0 uses a team of LLM agents to automatically extract categorical features from raw text and refine the extraction prompts based on AutoML feedback. A single production case study reports a small click-through-rate lift, but no data, code, or comparison baselines are provided.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection bias in the headline RIG: the +0.005 lift in §6 is the maximum over candidate prompts evaluated on the same last-20% split used for selection, so it overstates the expected production lift.","rationale":"The reader's weakest assumption is the same as my primary concern: using RIG on the last 20% both to select and to report the best prompt. I agree with the conditional verdict. I do not see a more fundamental internal inconsistency; the architecture is clearly described and the limitation section is candid. The recommendation remains UNCHANGED because the conditional verdict already encodes the need for a holdout evaluation. If the holdout test fails, the verdict should move toward REJECT for the reported effectiveness claim; if it passes, the conditional concern is resolved.","tokens_in":8172,"tokens_out":2707,"duration_ms":28440,"concrete_test":"Run Agent0's full prompt search once on the training/eval split, then evaluate only the single selected best prompt on a never-used holdout partition (e.g., a separate time slice or random 10% withheld from the final 20% split), recording RIG for that one prompt. Repeat the entire search with 3–5 different random seeds to obtain a distribution of holdout RIG values; if the mean holdout lift is not significantly above 0 with a confidence interval excluding 0, the headline claim of a 0.005+ lift is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of §8—that Agent0 'unequivocally demonstrated' the capacity to augment embedding pipelines—rests on the RIG lift reported in §6. §5 defines RIG on the last 20% of one production dataset, and §6 reports the 'final discovered prompt that yielded the highest lift over the baseline model (0.005+ rig)'. The same evaluation split is used to select the best prompt and to produce the reported score, and Figure 8 shows a distribution of prompt scores rather than a single confirmatory measurement. With an unreported number of candidate prompts, the maximum of the distribution is an upward-biased estimate of the value of the selected prompt; if the true effect is zero, many candidates will still yield a positive best RIG by chance. The paper does not provide a holdout evaluation, error bars, or the number of prompts tried, so the 0.005+ rig cannot be distinguished from selection noise. A second unresolved layer is sentinel stability: the feature extraction is performed once by Gemini-Flash, and no repeated-extraction variance is reported, so the RIG could reflect one run's extraction noise. These weaknesses are acknowledged in spirit in §7 (oracle bias toward DCNv2), but the selection-on-eval issue is not addressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Agent0, an agentic LLM system for automated multi-value feature extraction from text in recommender systems. The system combines an Architect LLM that rewrites extraction prompts, a fast Sentinel LLM that applies prompts to raw text at scale, and an AutoML Oracle that scores candidate features via relative information gain (RIG) on a production DCNv2 model. The Architect and Sentinel run as multiple asynchronous agents sharing a network-hosted memory of prompt-score tuples, with the top and bottom prompts fed back as few-shot examples. The authors report a production case study in which the best discovered prompt yields a '+0.005+ RIG' lift over a baseline without the new feature, and they conclude that the closed-loop methodology is practical and effective and that Agent0 has 'unequivocally demonstrated' its capacity to augment embedding-based pipelines.","tokens_in":8546,"tokens_out":4717,"duration_ms":44339,"significance":"If the central empirical claim survives proper validation, Agent0 is a useful industrial contribution: it automates a labor-intensive part of feature engineering, produces interpretable multi-value categorical features, and its multi-agent shared-memory design is a concrete and sensible engineering pattern. The RIG metric is well-defined, the prompt examples are concrete, and the system description is clear. However, the paper currently provides only one case study with a metric that is used both for prompt selection and for reporting the final score; there are no error bars, no separate holdout, no repeated runs, and no comparisons against hand-written prompts, embedding features, or random prompts. The significance is therefore potential rather than established, and the strong wording of the conclusions in Section 8 is not supported by the evidence presented.","major_comments":[{"comment":"The headline result, the '+0.005+ RIG' lift reported in Section 6, is the maximum over candidate prompts assessed on the same last-20% split that is used for prompt selection. With no separate holdout, no statement of the number of candidate prompts, and no repeated runs, this number is an in-sample maximum and is upward-biased as an estimate of the selected prompt's true value. The Section 8 conclusion that Agent0 has 'unequivocally demonstrated' practical effectiveness therefore does not follow from the presented evidence.","section":"§5–§6, Fig. 8"},{"comment":"No comparison baselines are reported. To support the central claim that the discovered feature augments embedding-based pipelines, the authors must compare against at least a hand-written prompt, a raw text-embedding feature, and a simple content-based baseline under identical evaluation. Without such comparisons, the observed RIG difference cannot be attributed to Agent0's closed-loop discovery process.","section":"§6, §8"},{"comment":"Sentinel stability is not assessed. The prompt is applied once by Gemini-Flash to produce feature values, and the paper does not report repeated extractions, temperature variation, or any measure of extraction variance. Since the RIG is computed on a single derived feature, extraction noise alone could produce a spurious positive lift; the experiment should include multiple extractions or a stability analysis.","section":"§3, §6"},{"comment":"The experimental protocol is under-specified: the dataset size and domain, the number of agents, the number of candidate prompts evaluated, the number of AutoML runs per prompt, and the computational budget are not reported. The claim in Section 6 of '2-3x faster per research-cycle' is also made without supporting measurements. These details are necessary to assess whether the effect is robust and reproducible.","section":"§5–§6"}],"minor_comments":[{"comment":"There is a typo: 'Agen0's internals' should read 'Agent0's internals'.","section":"§6"},{"comment":"The phrase 'Consider an example in 1' should refer explicitly to 'Figure 1' or 'Example 1'.","section":"§6"},{"comment":"Figure 5 has no caption, even though the surrounding text refers to a three-step procedure; a caption and labeled steps would improve readability.","section":"Fig. 5"},{"comment":"References [3] and [7] are informal web/tech-report URLs without version or access dates; they should be formatted consistently with the other references.","section":"References"},{"comment":"The sentence claiming that compact features 'dramatically reduce computational overhead and accelerate training latency' is not supported by any measurement in the paper; it should be tempered or supported.","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"The paper is closer to an industry experience report than a full evaluation study. I would be willing to accept a revision that adds a proper holdout evaluation, baseline comparisons, and variance estimates. If the authors cannot provide those, the strong conclusions in Section 8 should be substantially weakened. The manuscript also needs a clearer statement of what is new relative to existing LLM-agent recommendation surveys and AlphaEvolve."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know about this paper? It is an industry system description of Agent0, a three-part loop: an architect LLM rewrites extraction prompts, a sentinel LLM extracts multi-value tags from text, and an AutoML oracle scores the resulting features by relative information gain (RIG) on the last 20% of one production dataset. That composition appears to be new, and the paper is candid about several real limitations (oracle bias, latency, text-only scope). The writing is clear enough that a practitioner could rebuild the loop from the description.\n\nThe problem is the empirical claim. Section 6 reports a best-discovered prompt with \"0.005+ rig\" over baseline, and Section 8 says Agent0 \"unequivocally demonstrated\" its capacity to augment embedding pipelines. That lift is the maximum over an unreported number of candidate prompts, all scored on the same last-20% split used to select the best prompt. There is no holdout, no error bars, no repeated extractions, no comparison against a hand-written prompt or embedding features. With many prompts tried, the maximum RIG is expected to be positive by chance. The paper's own Figure 8 shows a distribution of scores, and it never tells us how many prompts were evaluated or how many were positive. So the +0.005 cannot be distinguished from selection noise.\n\nThere is also a secondary but real issue: the conclusions overstate what the evidence supports. The stress-test note is right that the selection-on-eval issue is not addressed, even though Section 7 does acknowledge oracle architecture bias.\n\nThat said, this is not a waste of time. The architecture is plausible, the problem (automating interpretable multi-value feature construction) is genuinely relevant, and the paper reports operational lessons (caching, memory consolidation, beam-search-like agent behavior) that would help anyone building a similar system. The citations check out for a paper of this type, and the self-citation is not a problem.\n\nWho is this for? A researcher or practitioner working on LLM agents for recommender features will get value from the system design, even though the headline result is not supported. A serious referee should engage with it, but the paper needs a dedicated holdout, error bars, and a comparison to at least one hand-written prompt before the effectiveness claim can stand.\n\nRecommendation: send it to peer review with a request for major revision on the evaluation. I would not desk-reject it.","headline":"Plausible system description, but the headline RIG lift is a best-of-search value on the evaluation split, so the effectiveness claim is not yet supported.","tokens_in":8962,"tokens_out":1999,"would_cite":false,"duration_ms":19930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent0 shows that a self-improving trio of LLM agents and an AutoML oracle can discover compact text features that lift click-through-rate models.","keywords":["language model agents","automated feature engineering","recommender systems","multi-value features","prompt optimization","AutoML evaluation","click-through rate","text extraction"],"falsifier":"Re-run the exact best prompt found by the loop on a fresh holdout slice of the same production stream, or on a subsequent time window, with the same base model; if the relative information gain over the no-feature baseline drops to zero or negative, the central claim is falsified. An online A/B test that adds the feature and shows no click-through-rate improvement would be the production-grade version of the same test.","tokens_in":7983,"feed_emoji":"🤖","tokens_out":7031,"duration_ms":68461,"temperature":0.7,"pith_summary":"Agent0 claims that a closed loop of two language-model agents plus an AutoML scorer can take raw, noisy document text and automatically discover compact multi-value features—comma-separated tags such as topics, entities, or intents—that make click-through-rate models perform better. The point is that the hardest, most manual step in recommender development, deciding which aspects of text deserve extraction, can be recast as prompt engineering and automated. In a production case study the loop found a prompt whose extracted feature improved the base model by more than 0.005 in relative information gain over the no-feature baseline. If the result holds, text feature engineering becomes something an agent loop can iterate on, and the discovered features remain human-readable instead of being opaque embeddings.","feed_headline":"Agentic LLM loop finds text features that lift click-through rate","feed_subtitle":"It automates the manual feature-engineering step, and the winning prompt stays human-readable.","key_machinery":"The load-bearing object is the closed feedback loop among three components. The Architect is a strong reasoning LLM that rewrites a seed prompt; the Sentinel is a fast LLM that applies the prompt to batches of raw text and produces the comma-separated tag values that become a categorical feature; the Oracle is an AutoML evaluator that adds the feature as an embedding field, retrains the base model, and scores the result by relative information gain versus a no-feature baseline. Because only the prompt changes between iterations, prompt space is the search space, and the shared memory of scored prompt-feature pairs lets each new proposal be guided by what previously worked and what failed.","core_discovery":"Agent0's central claim is that its Architect-Sentinel-Oracle model automates the discovery of high-signal features from text. An architect LLM rewrites extraction prompts; a fast sentinel LLM applies the current prompt to each document and emits the multi-value feature values; an AutoML oracle retrains the downstream model with and without that feature and returns a relative-information-gain score. The score is fed back to the architect together with the best and worst five prompts seen so far, and the cycle repeats. The paper reports that this closed loop, run with several parallel agents sharing only a memory store, converged on an interpretable prompt whose feature produced a lift of more than 0.005 RIG over a production deep-cross baseline, and argues that such compact features complement embeddings while reducing training overhead.","pith_inferences":["Because the reported 0.005+ lift is the maximum over many prompt trials, the cleanest confirmation would be to evaluate that single selected prompt on a fresh holdout slice or a later time window; the paper does not describe such a second-stage test.","The Architect-Sentinel-Oracle pattern is scoring-function-agnostic: swapping the oracle for a different objective, such as conversion lift, calibration, or diversity, would extend the same loop to other targets without redesign.","If the oracle's score is noisy or architecture-specific, the loop could overfit to quirks of one base model; evaluating the selected prompt against several model architectures would reveal whether the feature is genuinely high-signal or merely a lucky match."],"forward_implications":["Feature engineering for text can be treated as an optimization problem over extraction prompts, so improvements in LLM reasoning or AutoML evaluation translate directly into better discovered features.","Discovered features remain compact and interpretable: a data scientist can read and approve the extraction prompt before deployment, unlike an opaque embedding vector.","The loop can be re-run as document streams change, re-scoring and refining prompts without requiring a full model rebuild.","Parallel agents sharing a memory store behave like a beam search over prompt space, and the paper's observations suggest that heterogeneous agents would explore that space more widely."],"supporting_citations":[{"why":"Supplies the agent-based autonomous-loop precedent that Agent0 adapts to data-centric feature discovery.","marker":"[7]"},{"why":"Defines the deep-cross network used as the base model in the oracle, fixing the evaluation setup that any discovered feature must improve.","marker":"[11]"},{"why":"Identifies the fast LLM used for the sentinel network that converts raw text into feature values at scale.","marker":"[3]"}],"fun_headline_variants":["LLM agent loop auto-discovers text features that lift CTR","Closed-loop LLM agents mine text for recommendation feature lift","Auto feature engineering: LLM agents find text signals that lift CTR","Interpretable prompts from LLM agents lift recommendation CTR","Agent0: LLM loop auto-discovers high-value text features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the oracle's relative-information-gain score, computed on the last 20% of one production dataset with a single base model, is a stable and transferable measure of how much the discovered feature will help; if that score does not transfer to a fresh split or to live traffic, the claimed practical lift is not established.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent loop auto-discovers text features that lift CTR","Closed-loop LLM agents mine text for recommendation feature lift","Auto feature engineering: LLM agents find text signals that lift CTR","Interpretable prompts from LLM agents lift recommendation CTR","Agent0: LLM loop auto-discovers high-value text features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2961,"prompt_tokens":869,"completion_tokens":2092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2007}},"tokens_in":485,"tokens_out":2092,"duration_ms":13235,"temperature":1.0,"reasoning_tokens":2007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:03:00.793363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact best prompt found by the loop on a fresh holdout slice of the same production stream, or on a subsequent time window, with the same base model; if the relative information gain over the no-feature baseline drops to zero or negative, the central claim is falsified. An online A/B test that adds the feature and shows no click-through-rate improvement would be the production-grade version of the same test.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the agent-based autonomous-loop precedent that Agent0 adapts to data-centric feature discovery."},{"cited_title":"Accessed: 2025-05-23","cited_arxiv_id":null,"evidence_quote":"Identifies the fast LLM used for the sentinel network that converts raw text into feature values at scale."}],"review_version":2}