Artificial Intelligence · arXiv:2609.04871 v1 · submitted 4 Sep 2026 · 40 references
AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems
The paper says
“Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment.”
more of the abstract
In \textbfDASHEN, NetEase's gaming-community app, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review—the internal gate for full-traffic rollout. Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. We present \textbfAutoLR, initially built as \textbfAuto Launch Review and later extended upstream into an autonomous research-to-launch harness. AutoLR combines three system mechanisms: a \textbfmulti-expert council that debates and adversarially reviews proposals; a \textbfdeterministic evidence-weighted exploration–exploitation selector that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge—such as game communities, player characteristics, and content-interaction patterns—with posterior evidence from configurations, patches, logs, failures, and offline outcomes. LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 8 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A transparent industrial case study whose real contributions are the KEEP ratchet and a scale audit; treat the empirical claims as provenance-limited rather than proven. This line is wrong
6 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 40 references say.
The paper says, at Section 4.1
The review did not quote this passage.
Pith says: Objection 1 of 6 · major
The 1,586-evaluation corpus that carries RQ1 and all Section 5.3 statistics is produced by a path-aware deduplication keyed on (scenario, path session, iteration), where the session is inferred from the enclosing archive path, and no validation against an independent identifier (training-platform job ID, experiment-management session ID, or git trunk hash at evaluation start) is reported.
more of objection 1
If archive paths merge or split distinct experiment sessions, the 3,289 canonical records and the 1,586 completed evaluations change, and with them every corpus-level rate in Table 2. Please add a validation of the dedup key against an independent identifier on at least a sample of sessions, or explicitly state that all scale and distributional claims are contingent on an unvalidated reconstruction.
The paper says, at Sections 3.6 and 5.2
The review did not quote this passage.
Pith says: Objection 2 of 6 · major
Section 3.6 concedes that the audited archive does not retain complete immutable linkage between offline candidates and online packages, yet the introduction to Table 1 in Section 5.2 says the nine records provide "evidence that AutoLR-generated candidates progressed beyond offline evaluation into online A/B testing and production launch decisions."
more of objection 2
As the archive stands, the records document positive reported movements for engineer-selected packages; they cannot be shown to originate from specific AutoLR packages. Please reword the RQ2 claim so that the lineage gap is reflected where the claim is made, not only in the limitations section, state the number of packaged candidates admitted to A/B testing if that count is recoverable, and apply the same separation to the search and user-growth transfers, which are asserted on the basis of a single Launch Review each with no reported magnitudes.
The paper says, at Section 5.2, Table 1
The review did not quote this passage.
Pith says: Objection 3 of 6 · major
The headline sums (+5.75%, +10.83%, +5.55%) are computed under the convention that metrics not separately disclosed in a review are Flat, and the table caption defines Flat as "no reported overall-traffic movement." A metric that was measured and moved negatively and a metric that was never measured are therefore recorded identically, and the sums are an artifact of what each review chose to disclose.
more of objection 3
Please report per record which Flat cells are measured-flat versus not disclosed, and provide a worst-case recomputation of the sums under the assumption that undisclosed cells were negative rather than zero.
The paper says, at Section 5.1
The review did not quote this passage.
Pith says: Objection 4 of 6 · major
The claim that "In the two studied scenarios, AutoLR has become the primary workflow for routine model iteration" is stronger than the evidence reported: the audit establishes that AutoLR completed 1,586 evaluations, but no contemporaneous denominator of all routine model iterations in these scenarios is provided, so the relative share ("primary") cannot be verified and no comparison against a parallel human-driven pipeline is offered.
more of objection 4
Please either supply the scenario-level denominator or downgrade the sentence to "a routine workflow," which is what the corpus actually supports.
The paper says, at Sections 4.1 and 5.3
The review did not quote this passage.
Pith says: Objection 5 of 6 · major
Section 4.1 states that uncertainty intervals are computed by resampling whole sessions because evaluations within a session are correlated (shared data windows, active trunks, and experiment history), but no such intervals appear in Section 5.3 for the KEEP/PACK fractions, reference-floor crossing rates, or IQR comparisons. The text promises a statistical treatment that the results never deliver. Please report the session-resampled intervals for the key rates, or state explicitly why they were not informative and remove the promise.
The paper says, at Section 5.3, Table 2
The review did not quote this passage.
Pith says: Objection 6 of 6 · major
The promotion-resolution replay applies the common reference floor tau_ref = 10^-3 to every evaluation even though 480 of the 810 single-/two-column-feed rows have recorded threshold provenance, and the operative values of tau_keep and tau_pack are never disclosed. A provenance-restricted analysis, together with a sensitivity sweep over plausible tau_keep values, would show whether the central RQ3 observation (reference floor comparable to the IQR of realized deltas) survives under the thresholds actually used, and would materially sharpen the KEEP-ratchet risk statement.
5 minor objections
The paper says, at Section 3.3
Pith says: minor objection 1 of 5
The memory gate's composite score is never given a functional form: beyond p_hat_succ, the seven listed features (expected improvement, novelty, feasibility, stability, confidence, risk) are named but not defined or combined, which prevents any independent simulation of the routing policy; since the paper disclaims selector superiority, a brief specification or a pointer to released code would be sufficient.
The paper says, at Figure 8A
Pith says: minor objection 2 of 5
The caption of Figure 8A contains a text-reflow artifact ("Horizontal segments show the empirical [ , ] intervals.Q Q0.25 0.75") that must be corrected.
show 3 more
The paper says, at Section 5.3
Pith says: minor objection 3 of 5
The discrepancy between the reference-floor crossing rates (4.7% and 8.5%) and the observed KEEP fractions (3.82% and 5.03%) is never discussed; clarifying how many crossers fell below tau_pack or were ineligible or payload-incomplete would help the reader interpret Table 2.
The paper says, at Equations (6) and (8)
Pith says: minor objection 4 of 5
The relationship between tau_ref, tau_keep, and tau_pack is never stated explicitly; please clarify whether tau_ref is a proxy for tau_keep, and state the ordering tau_pack < tau_keep and the relationship between tau_ref and tau_keep directly.
The paper says, at Section 2.2
Pith says: minor objection 5 of 5
References [17] and [27] are an engineering blog post and an internal technical report, respectively; for archival purposes, please provide stable version or venue information where available, since the related-work comparison leans heavily on these sources.
What would settle it
Obtain the raw 4,250 ledger rows and, for each row, an independent identifier: training-platform job ID, experiment-management run/session ID, and the git trunk hash at evaluation start. Rebuild the 3,289 canonical keys and the 1,586 completed evaluations using those identifiers instead of path-session by iteration, and recompute Table 2's fractions from the rebuilt corpus. Then, for each of the nine Launch Review records, require a unique offline package ID and commit hash linking the packaged candidate to the reviewed launch. If the independently keyed corpus differs by more than a pre-specified tolerance, for example 5% of completed evaluations, or if any launch record lacks package lineage, the headline scale and launch evidence must be revised; otherwise the audit concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 8 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 6 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Read it. The conditional verdict is the right one. The genuinely new content is the KEEP ratchet and the Launch-Review-centered production audit; the harness architecture itself overlaps with Google's Self-Evolving Recommendation System, Kuaishou's AgentX, RecHarness, NOVA, and PILOT, and the authors say so plainly. The formal lifecycle is coherent: Equations (4)-(7) cleanly capture scalar scoring, protected-metric guardrails, terminal actions, and the fact that a KEEP rewrites the trunk. The Beta(1,1) smoothing is correctly labeled as a scoring feature rather than a posterior, and the paper repeatedly declines to claim component-level superiority.
Credit where it is earned: the limitations section is unusually honest. It admits disabled noise calibration, no reusable calibration profile, incomplete threshold and guardrail provenance, no matched-budget ablations, no immutable offline-to-online lineage, and no confidence intervals online. It also refuses to treat missing structured payload as compliance. That transparency makes the design discussion worth reading even where the empirical evidence is incomplete.
The soft spots are real, and the stress-test concern lands. The 1,586-evaluation corpus rests on an unvalidated path-session dedup key; without a check against an independent identifier such as a training-platform job ID or scheduler run ID, the canonical counts, IQRs, and KEEP/PACK fractions are only as good as that key's grouping. The nine Launch Review records are also weaker than the phrase "positive online lifts" suggests: metrics not disclosed are treated as flat, denominators and confidence intervals are absent, and the archive does not retain immutable linkage tying an online package to its AutoLR candidate. The paper admits the lineage gap, so this is a provenance limit rather than evidence of selective reporting. Still, RQ2 supports "engineer-selected candidates reached review with positive reports," not "AutoLR-generated models caused these lifts."
The KEEP ratchet is the strongest single idea and follows from Equation (7) regardless of the audit. It deserves citing as a design guardrail for any self-evolving system that promotes candidates to a persistent baseline.
Who the paper is for: engineers and researchers building governed LLM harnesses for recommenders. It deserves a serious referee. The right ask is validation of the dedup key against an independent identifier, or a reframing of RQ1 and RQ2 as a case study rather than an effectiveness evaluation. I would accept it for peer review.
The strongest objection
The most load-bearing link is the unvalidated path-aware dedup key. Section 4.1 reports the drops 4,247 parseable writes to 3,289 canonical keys to 1,586 completed evaluations using the key (scenario, path session, iteration), where a session is identified from the enclosing archive path; Section 5.3 computes KEEP/PACK fractions, IQRs, and crossing rates from this same corpus, and RQ1's 'routine production iteration' claim is carried by the 1,586 count. No check against an independent identifier, such as a training-platform job ID, scheduler run ID, experiment-management session ID, or git trunk hash at evaluation start, is reported. The companion gap is admitted in Section 3.6: the audited archive does not retain complete immutable linkage between offline candidates and online packages, so Table 1's nine Launch Review records cannot be shown to originate from specific AutoLR packages rather than from engineer-modified or separately developed candidates. If path sessions merge or split distinct experiments, the canonical corpus and the session-level resampling basis change; if the nine launch records lack package lineage, the positive online lifts described in RQ2 are not evidence about AutoLR. This is a provenance concern, not an accusation of selective reporting: the paper's own limitation statement concedes the missing lineage.
The reader's conditional verdict is the right one. The central empirical claim depends on the audit's reconstruction of completed evaluations and on the Launch Review records. The paper is unusually candid: it states that no code or data is public, that there are no confidence intervals or candidate denominators, that immutable offline-to-online lineage is not retained, and that no matched-budget ablations exist. That candor is creditworthy, but it does not make the load-bearing count reliable. The most consequential technical issue is that the entire reconstruction is keyed by archive-path heuristics with no reported validation set, no manual sample check, and no independent identifier cross-check. A wrong session key would directly change the 1,586 count, the KEEP/PACK fractions, and the session-resampling basis. Separately, the nine Launch Review records cannot be tied to AutoLR-generated packages under the paper's own admission that complete immutable lineage is absent, so the launch evidence is weaker than the summary language in Section 5.2 suggests. I would not move to REJECT because the paper avoids overclaiming: the sums are explicitly descriptive, effects are not pooled, causal attribution is disclaimed, and the KEEP ratchet is identified rather than hidden. The appropriate posture remains CONDITIONAL, with the concrete requirement that the raw-ledger replay and package lineage check be supplied. If the replay disagrees, the empirical evidence should be downgraded to anecdotal; if it agrees, the operational-viability claim would be substantially supported.
The referee's report
The conceptual contribution is real and internally disciplined. Equations (2)-(8) form a consistent decision contract, and the KEEP ratchet follows directly from the paper's own state-transition rule (Eq. 7), so the central reliability finding is a derivation from the system definition rather than a post hoc observation. The paper also deserves explicit credit for its anti-overclaiming discipline: a one-run offline crossing is labeled E1 rather than a confirmed effect, missing historical payload is not treated as compliance, the Beta(1,1) quantity is declared a scoring feature rather than a posterior, and Section 6.2 lists exactly the analyses that were not performed. If the reconstructed corpus is accepted, the paper provides a rare longitudinal production data point on governed LLM-agent iteration. The significance of the empirical findings, however, is bounded by the same concessions: the scale and deployment claims rest on an unvalidated path-session dedup key, the online records lack package lineage, and the Flat convention is an ad-hoc axiom that shapes the headline sums.
Pith's reading of the paper (a machine's summary, not the authors')
AutoLR is a system for automating the whole research-to-launch-review path of an industrial recommender, not just isolated stages. The paper's central claim is that such a harness can operate as the routine model-iteration workflow in production: in a gaming-community app's two main feeds, the system completed 1,586 offline evaluations, and nine engineer-selected candidate packages produced positive reported online movements. The design principle is asymmetric authority: LLM agents propose, debate, and write code, while deterministic controllers run experiments, extract metrics, enforce guardrails, and change persistent state. Alongside that record, the paper names the main reliability risk—the KEEP ratchet—where a single noisy offline promotion becomes the new baseline and creates progress that may not reproduce. If true, the result matters because it shows a governed autonomous discovery loop is deployable and because it identifies where self-evolving recommenders must fail closed.
What carries the argument. The load-bearing mechanism is asymmetric authority wrapped in a governed lifecycle. A multi-expert council has role-specialized agents propose independently from a frozen task contract, then uses roundtable synthesis and adversarial review to converge on structured candidates. A deterministic evidence-weighted selector builds a bounded, diversity-aware candidate window, takes council reranking as advisory only, and applies a memory gate before GPU training, using $\hat{p}_{\mathrm{succ}}=(n_{\mathrm{success}}+1)/(n_{\mathrm{attempt}}+2)$ as a smoothed scoring feature rather than posterior sampling. A layered knowledge system separates external research, production-system facts, and domain knowledge from an evolving experiment memory of configurations, patches, logs, failures, and outcomes. The mechanism behind the paper's caution is stateful promotion: among the terminal actions only KEEP writes the active trunk, so each accepted one-run result becomes the baseline for subsequent comparisons—the KEEP ratchet.
Core claim. The paper's discovery is operational rather than component-level: AutoLR has moved from an offline experimentation prototype into routine production iteration in two core recommendation scenarios. A path-aware audit reduced 4,250 raw ledger records to 1,586 unique completed evaluations; of the 1,583 with numeric deltas, reference-floor crossing was selective (4.7% and 8.5%) and observed KEEP fractions were 3.82% and 5.03% in the two feeds. Nine Launch Review records report positive relative movements in content-consumption penetration, total consumption time, and valid content views, summing descriptively to +5.75%, +10.83%, and +5.55%, with the paper's explicit caveat that these sums are not pooled treatment effects. The reliability finding is the KEEP ratchet: KEEP is the only terminal action that rewrites the active trunk, so a one-run threshold crossing changes the baseline for every later experiment; because noise calibration was disabled in the audited configuration, the loop could not separate true improvement from accumulated selection noise. The author's claim is therefore that the integrated governed loop is production-viable and that durable autonomy requires conservative, evidence-calibrated commitment, not that any single mechanism—council, selector, or knowledge memory—has been proven causally superior.
The premise it rests on. The load-bearing premise is that the audit's reconstruction of the experiment ledger and the nine Launch Review tables is a faithful, unbiased record of what actually ran and what actually launched, including the path-aware deduplication of 4,250 ledger records into 1,586 completed evaluations and the convention that metrics not separately disclosed in a review are treated as flat.
If the paper is right (Pith's inferences, not the authors')
Under the paper's evidence, routine upstream model iteration in an industrial recommender can be delegated to LLM agents as long as deterministic controllers own execution, metric extraction, guardrails, and persistent state transitions.
Human authority can sit at the deployment boundary: the audited configuration automates research through candidate packaging while engineers keep online A/B admission and Launch Review, and the nine positive review records pass through that gate.
Autonomous promotion should fail closed: without a compatible noise-calibration profile, a fixed KEEP floor does not protect the trunk from one-run selection noise, so protocol-matched repeat confirmation is needed before rewriting the baseline.
Routine autonomy does not require frontier models: after harness and loop optimization, iterations ran on a mixed lower-cost model stack at an observed RMB 3–4 per iteration in LLM API spend, though the migration was observational and not causally attributable to any single factor.
The audit convention that metrics not separately disclosed in a review are Flat means the true review table could hide negative or missing movements; re-extracting the raw A/B values for undisclosed cells would settle whether the reported positive sums are overstated.
The KEEP ratchet should generalize beyond this deployment: any iterative loop that promotes accepted candidates to the incumbent baseline inherits the same stateful amplification, so the paper's fail-closed calibration recommendation is a design constraint for self-evolving recommenders in general.
A direct matched experiment would isolate the ratchet: run the same loop twice from a shared trunk, one with KEEP enabled and one with promotion replaced by PACK-only packaging, and compare final trunk quality under equal trial budgets; the audited archive cannot do this because it lacks matched-budget ablations.
The cost observation suggests cheaper models may suffice for routine governed iteration, but model family, harness version, and task mix changed together in the migration, so a controlled model swap on a fixed workload would be needed to confirm that cost reduction alone preserves output quality.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment. In \textbfDASHEN, NetEase's gaming-community app, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review—the internal gate for full-traffic rollout. Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. We present \textbfAutoLR, initially built as \textbfAuto Launch Review and later extended upstream into an autonomous research-to-launch harness. AutoLR combines three system mechanisms: a \textbfmulti-expert council that debates and adversarially reviews proposals; a \textbfdeterministic evidence-weighted exploration–exploitation selector that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge—such as game communities, player characteristics, and content-interaction patterns—with posterior evidence from configurations, patches, logs, failures, and offline outcomes. LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.67
“We present \textbf{AutoLR}, initially built as \textbf{Auto Launch Review} and later extended upstream into an autonomous research-to-launch harness.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.29 · human participants 0.25 · controlled comparison 0.18 · randomized assignment 0.13 · states quantitative result 0.04 · compares with prior results 0.08 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment. | background 1.00 | 0.10 | 0.07 | asserted 1.00 |
| In \textbf{DASHEN, NetEase's gaming-community app}, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. | background 1.00 | 0.14 | 0.03 | asserted 1.00 |
| Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review---the internal gate for full-traffic rollout. | background 0.99 | 0.13 | 0.03 | asserted 1.00 |
| Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. | gap 0.95 | 0.14 | 0.10 | asserted 0.82 |
| We present \textbf{AutoLR}, initially built as \textbf{Auto Launch Review} and later extended upstream into an autonomous research-to-launch harness. | contribution 1.00 | 0.88 | 0.17 | asserted 1.00 |
| AutoLR combines three system mechanisms: a \textbf{multi-expert council} that debates and adversarially reviews proposals; a \textbf{deterministic evidence-weighted exploration--exploitation selector} that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge---such as game communities, player characteristics, and content-interaction patterns---with posterior evidence from configurations, patches, logs, failures, and offline outcomes. | method 0.91 | 0.81 | 0.19 | asserted 1.00 |
| LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions. | method 0.98 | 0.67 | 0.13 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.49
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,863 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (40)
- [1] Avrim Blum and Moritz Hardt. 2015. The Ladder: A Reliable Leaderboard for Ma- chine Learning Competitions. InProceedings of the 32nd International Conference on Machine Learning. 1006–1014 on Pith
- [2] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, et al. 2021. Accounting for Variance in Machine Learning Benchmarks. InProceedings of MLSys on Pith
- [3] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al on Pith
- [4] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Rec- ommendation Approaches. InProceedings of the 13th ACM Conference on Recom- mender Systems on Pith
- [5] Alex Deng and Xiaolin Shi. 2016. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining on Pith
- [6] Pavel Dmitriev, Somit Gupta, Dong Woo Kim, and Garnet Vaz. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining on Pith
- [7] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mor- datch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. (2023). arXiv:2305.14325 [cs.CL] on PithPith's reviewarXiv
- [8] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2015. The Reusable Holdout: Preserving Validity in Adaptive Data Analysis.Science349, 6248 (2015), 636–638 on Pith
- [9] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural Architecture Search: A Survey.Journal of Machine Learning Research20, 55 (2019), 1–21 on Pith
- [10] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. InAdvances in Neural Information Processing Systems on Pith
- [11] Dietmar Jannach. 2022. Multi-Objective Recommender Systems: Survey and Challenges. InProceedings of the Multi-Objective Recommender Systems Workshop at ACM RecSys on Pith
- [12] Adel Javanmard and Andrea Montanari. 2015. On Online Control of False Discovery Rate. (2015). arXiv:1502.06197 [math.ST] on PithPith's reviewarXiv
- [13] Ziyu Jiang, Daniel Schmidt, Divya Srikanth, Diyi Xu, Ian Kaplan, David Jacenko, and Yuhuai Wu. 2025. AIDE: AI-Driven Exploration in the Space of Code. (2025). arXiv:2502.13138 [cs.AI] on PithPith's reviewarXiv
- [14] Manas Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K. Adams, Pranav Khaitan, Jiahui Liu, and Quoc V. Le. 2020. Neural Input Search for Large Scale Recommendation Models. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data... on Pith
- [15] Ramesh Johari, Leo Pekelis, and David J. Walsh. 2015. Always Valid Inference: Bringing Sequential Analysis to A/B Testing. (2015). arXiv:1512.04922 [stat.ME] on PithPith's reviewarXiv
- [16] Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled Experiments on the Web: Survey and Practical Guide.Data Mining and Knowledge Discovery18, 1 (2009), 140–181 on Pith
- [17] Ashwin Kumar, Erwin Gao, Matan Levi, Sheela Yadawad, Sherman Wong, Sneha Iyer, and Vinodh Kumar Sunkara. 2026. Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation. Engineering at Meta. Published March 17, 2026 on Pith
- [18] Changxin Lao, Fei Pan, Guozhuang Ma, Han Li, Huihuang Lin, Jijun Shi, Kangzhi Zhao, Kun Gai, Mo Zhou, Qinqin Zhou, et al . 2026. AgentX: To- wards Agent-Driven Self-Iteration of Industrial Recommender Systems. (2026). arXiv:2606.26859 [cs.AI] on PithPith's reviewarXiv
- [19] Minyong R. Lee and Milan Shen. 2018. Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments. InProceedings of the 24th Zhang et al. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 491–499 on Pith
- [20] Jiuning Lin, Ruiquan Lan, Xiaodong Zhu, Bin Zhang, Chengyu Lai, Chuxin Chen, Dimin Wang, Han Zhu, Hongtao Cheng, Jialin Zhu, Lingqing Zhang, Shuai Zhong, Tao Wang, Weipeng Huang, Yinjiang Cai, Yinnan Song, Yuan Liu, Zhibo Xiao, Zhixin Ma, and Zihong Huang. 2026. PILOT Technica... on PithPith's reviewarXivdoi:10.48550/arxiv.2608.18637
show 20 more
- [21] Haoran Ling, Yuecheng Li, Zeyu Song, Jing Yao, Shuwen Kang, Chi Lu, Wenjin Wu, and Peng Jiang. 2026. RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems. (2026). arXiv:2607.29241 [cs.IR] on PithPith's reviewarXiv
- [22] Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xi- uqiang He, Zhenguo Li, and Yong Yu. 2020. AutoFIS: Automatic Feature In- teraction Selection in Factorization Models for Click-Through Rate Prediction. InProceedings of the 26th ACM SIGKDD Internationa... on Pith
- [23] S. Liu, L. Fang, Y. Sun, S. Huang, Q. Luo, X. Chen, D. Liu, C. Ma, Z. Chai, H. Wang, et al. 2026. NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems. (2026). arXiv:2606.27243 [cs.IR] on PithPith's reviewarXiv
- [24] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. (2024). arXiv:2408.06292 [cs.AI] on PithPith's reviewarXiv
- [25] Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan O. Arik, and Tomas Pfister. 2025. MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement. InAdvances in Neural Information Processing Systems on Pith
- [26] Alexander Novikov, Ngan Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, et al . 2025. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. (2025). arXiv:250... on PithPith's reviewarXiv
- [27] Prithvi Rajasekaran. 2026. Harness Design for Long-Running Application Devel- opment. Anthropic Engineering on Pith
- [28] Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-Networks for Sequence Tagging. In Proceedings of EMNLP on Pith
- [29] Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. InProceedings of the 14th ACM Conference on Recommender Systems on Pith
- [30] Daniel Russo and James Zou. 2016. Controlling Bias in Adaptive Data Analysis Using Information Theory. InProceedings of AISTATS. 1232–1240 on Pith
- [31] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools.Advances in Neural Information Processing Systems(2023) on Pith
- [32] Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D. Barrett, and Arnu Pretorius. 2024. Should We Be Going MAD? A Look at Multi-Agent Debate Strategies for LLMs. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learnin... on Pith
- [33] James E. Smith and Robert L. Winkler. 2006. The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis.Management Science52, 3 (2006), 311–322 on Pith
- [34] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. InAdvances in Neural Information Processing Systems on Pith
- [35] Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2013. Auto-WEKA: Combined Selection and Hyperparameter Optimization of Clas- sification Algorithms. InProceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 847–855 on Pith
- [36] Haochen Wang, Yi Wu, Daryl Chang, Li Wei, and Lukasz Heldt. 2026. Self- Evolving Recommendation System: End-To-End Autonomous Model Optimiza- tion With LLM Agents. (2026). arXiv:2602.10226 [cs.LG] on PithPith's reviewarXiv
- [37] X. Wu, Y. Zhuan, R. Wei, H. Chen, D. Bai, J. Liu, X. Wang, X. Wang, L. Wang, and X. Cheng. 2026. AgenticRecTune: Multi-Agent with Self-Evolving Skillhub for Recommendation System Optimization. (2026). arXiv:2604.26969 [cs.IR] on PithPith's reviewarXiv
- [38] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations on Pith
- [39] Marc-Andre Zoller and Marco F. Huber. 2021. Benchmark and Survey of Auto- mated Machine Learning Frameworks.Journal of Artificial Intelligence Research 70 (2021), 409–472 on Pith
- [2025] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. InInternational Conference on Learning Representations on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment