Pith. sign in

REVIEW 5 major objections 5 minor 39 references

GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-based agents can automatically optimize off-policy evaluation code, and a minimal two-agent architecture proved the most reliable in a 504-run benchmark.

desk verdict The abstract oversells the headline numbers and the Scope-RL metric may only show that agents can inflate an estimator's own output, but the benchmark, failure analysis, and artifacts make this worth a serious review. read the letter →

arxiv 2511.00802 v2 pith:GFNX57DY submitted 2025-11-02 cs.SE cs.CLcs.LG

Jie JW Wu , Ayanda Patrick Herlihy , Ahmad Saleem Mirza , Ali Afoud , Fatemeh Fard This is my paper
classification cs.SEcs.CLcs.LG
keywords off-policyevaluationLLM-basedagentscodeoptimizationofflineA/BtestingbenchmarkOpenBanditPipelineScope-RLtwo-agentframework
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language model agents can act as automated 'growth hackers' for off-policy evaluation (OPE): given an existing OPE implementation, an agent iteratively rewrites the code, runs the evaluation, and keeps the best-iteration result. To test this, the authors build a benchmark on two public OPE libraries and compare standalone LLMs with three agent architectures. The central result is that a minimal two-agent framework—one agent analyzes code and results and writes instructions, the second implements them—executes successfully in every run and produces the largest improvements among positive outcomes. If correct, the claim means OPE tuning can move from manual hyperparameter search to automated code-space optimization, reducing the cost of offline A/B testing in production.

What carries the argument

The load-bearing mechanism is a two-agent loop: a Prompter/Analyzer agent reads the original OPE code and its results, identifies strengths and weaknesses, and writes a modification instruction file; a Coder agent implements the instruction into new code; the new code is executed and scored. Iterations are independent (each starts from the original code), avoiding context degradation, and a final lightweight LLM selects the best-performing iteration from the aggregated logs. The paper also contributes GrowthHacker, a benchmark harness that turns Jupyter notebooks into runnable Python, executes modified code against logged data, and records a success/failure and performance trajectory.

What would settle it

Re-run the two-agent pipeline on OBP synthetic and Scope-RL notebooks, then compare every iteration's estimator output against known ground-truth policy values: if the selected 'best' iterations have larger relative_policy_value but worse mean squared error against ground truth, the claimed feasibility of OPE optimization collapses. A simpler check: re-evaluate the selected iteration on a held-out portion of logged data; if gains evaporate, the optimization was overfitting.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based agents can autonomously and iteratively optimize off-policy evaluation implementations by modifying source code, and that a deliberately simple two-agent architecture outperforms richer multi-agent frameworks. In 504 experiments across the Open Bandit Pipeline and Scope-RL libraries, the two-agent framework achieved 100% execution success, a 45% positive-outcome rate, and a 106.7% average improvement (5.5% median) among successful optimizations, while CrewAI and AutoGen showed lower or more erratic gains. The authors interpret this as establishing feasibility of automated 'growth hackers' for continuously improving OPE systems, with implications fo

Load-bearing premise

The load-bearing premise is that the proxy metrics the agents optimize are faithful indicators of OPE quality on the logged data—assuming, for Scope-RL, that larger relative_policy_value means better estimation, and that choosing the best iteration on the same logged data does not merely fit noise.

Editorial extensions

If this is right

  • OPE systems in production can be improved without manual hyperparameter expertise, since agent-modifiable code space subsumes hyperparameter tuning.
  • Agent-framework choice and code-modification strategy interact: the paper finds agent-applied patches most reliable overall, whole-code replacement best for its two-agent design, and no single pairing dominates.
  • For Scope-RL's continuous action spaces, safe optimization requires explicit bounds (bandwidth >= 1.0, learning rate <= 3e-4) to avoid parameter explosions.
  • The same iterative code-refinement loop could apply to other data-driven evaluation pipelines, not just OPE.
  • Reliability differences (0% vs 7-10% failure rates) show that simple, focused agent designs can beat general-purpose multi-agent frameworks on specialized coding tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Scope-RL's objective is the estimator's own relative_policy_value, an agent that inflates this score may not actually reduce error against the true policy value; checking against ground truth (available in synthetic OBP or on-policy Scope-RL runs) would separate genuine improvement from objective-gaming.
  • Selecting the best iteration on the same logged data used for optimization risks overfitting; a held-out evaluation split would test whether the gains persist on fresh data.
  • The framework could be extended to optimize mean squared error directly when ground truth exists, which would align agent incentives with OPE's actual goal.
  • The 9,999% extreme-value threshold may obscure large real gains or failures; variance-aware or log-scale metrics would give a cleaner risk picture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces GrowthHacker, a benchmark for using LLMs and LLM-based agents to iteratively modify the code of off-policy evaluation (OPE) implementations, with the goal of improving OPE performance on the Open Bandit Pipeline (OBP) and Scope-RL libraries. Four approaches are compared: a standalone default LLM, AutoGen, CrewAI, and a custom two_agent framework. The authors report success/failure rates, positive-outcome rates, average/median improvement among positive outcomes, and a comparison of code modification modes. The main claimed result is that the two_agent framework is the most reliable and produces the strongest improvements, establishing the feasibility of LLM agents as automated 'growth hackers' for OPE.

Significance. If the empirical claims were supported, this would be a useful contribution: the paper addresses a genuine gap in automating OPE optimization in code space rather than hyperparameter space, and it releases a replication package. The OBP results use ground-truth-based relative estimation error and therefore have meaningful content. The two_agent architecture is simple and reported to be reliable. However, the current evidence does not establish the central feasibility claim. The Scope-RL evaluation defines improvement as an increase in the estimator's own output metric without any ground-truth or held-out validation, and the best-iteration selection is performed on the same logged data used for optimization. These issues, together with substantial discrepancies between the abstract and body numbers, mean the headline claims are not yet credible. The contribution is potentially salvageable with additional validation and corrected reporting, but the current version is not acceptable.

major comments (5)
  1. [Abstract; §5.3.2; Table 9; Figure 4] The headline numbers differ materially between the abstract and the body. The abstract reports a 78% positive-outcome rate for two_agent, a 4.4% median improvement, and a 37.9% average improvement for CrewAI; the body reports 45%, 5.5%, and 31.7%, respectively. These are not cosmetic differences: they change which framework appears strongest on key metrics. The authors must reconcile these numbers and specify exactly which subsets and aggregation rules produce them.
  2. [§2.2; §4.2.4; §5.3.2] The problem formulation in §2.2 defines the optimization objective as minimizing the relative estimation error of the OPE estimator, but the Scope-RL evaluation treats increases in relative_policy_value as improvements. The paper does not show that a larger relative_policy_value implies a more accurate estimator. Since the agent modifies estimator code, an increase could simply result from inflating the estimator's own output (e.g., by changing importance weights or bandwidth in a way that is not validated against any ground truth). The Scope-RL-based feasibility claim therefore requires an independent check against ground-truth policy values or a held-out dataset; the current evaluation does not provide one.
  3. [§3.4; §5.2.1] The framework selects the 'best' iteration post hoc using the same logged data on which the optimization ran. This selection procedure can reward iterations that overfit to noise or that artificially inflate the metric, so the reported positive-outcome rates and improvement magnitudes may reflect selection bias rather than genuine improvement. Additionally, §5.2.1 states that OBP performance changes are computed from baseline to the final iteration, which is inconsistent with the best-iteration selection described in §3.4. The authors need to specify which comparison is used everywhere and justify why the selection procedure does not invalidate the reported improvements.
  4. [§4.2.1; Tables 3–5; Table 9] The two_agent framework was evaluated on only 18 runs total, and the OBP tables show n=1 per estimator–policy cell with zero standard deviation. Claims such as 'two_agent consistently achieved substantial improvement' are not supported by a single observation per cell. More runs are needed to estimate variance and to support comparative statements against frameworks with n=6–9 per cell. The aggregate improvement metrics in Table 9 also lack sample sizes and confidence intervals; given the right-skewed distributions, the reported averages should be accompanied by uncertainty estimates.
  5. [§4.2.4; §5.1] There is no non-LLM baseline, such as random search, grid search, or a simple hyperparameter optimizer, on the same tasks. Without such a control, the paper cannot attribute the observed improvements to the LLM agent's code-modification ability rather than to the iterative search process itself. Adding at least one classical baseline would substantially strengthen the claim that LLM agents are specifically useful for this task.
minor comments (5)
  1. [§1] Typo: 'GrowthHacke' should be 'GrowthHacker'.
  2. [§6.2] Typo: 'attarcted' should be 'attracted'.
  3. [References] The reference list contains duplicates and inconsistencies: RAPGen appears as both [22] and [34], and the SBLLM citation appears as [14] and [17] with different titles. These need to be reconciled.
  4. [Table 9 and Figure 4] The aggregate improvement metrics and outcome proportions do not report the number of runs they are based on. Please add n for each entry so readers can assess the stability of the estimates.
  5. [§5.2.2] The text says 'policy-averaged aggregation' but then reports 'Total=816' and similar numbers; it is unclear whether these totals are runs, estimator-policy combinations, or pooled observations. Clarify the aggregation level.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical benchmark and its claims rest on experimental runs, not on definitional or self-cited reductions.

full rationale

GrowthHacker is an empirical benchmark paper rather than a formal derivation, so the circularity burden is minimal. Its central claim—that LLM-based agents can iteratively modify OPE code and improve measured OPE metrics—is supported by experimental runs on OBP and Scope-RL, not by a chain of equations that reduce to their inputs. The Scope-RL evaluation uses relative_policy_value as both the optimization objective and the success metric, and the framework selects the best iteration from the same logged data used for optimization; while this is a legitimate validity concern about metric overfitting or selection bias, it is not circularity in the required sense: the agent's code modifications are not defined in terms of the reported outcome, and no fitted parameter is renamed as a prediction. The OBP experiments use ground-truth-based relative estimation error, providing an external benchmark. The paper's self-citations (e.g., [35], [36], [37]) appear in background and related-work contexts and are not load-bearing for the empirical claims. The Limitations section acknowledges instability and parameter explosions but does not assert any circular step. Therefore, no specific circular reduction can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark is empirical and rests on standard OPE estimator assumptions, the representativeness of selected notebooks, the validity of its metrics (especially Scope-RL's relative_policy_value), and the correctness of the bypass logic. The most fragile assumption is that maximizing relative_policy_value without ground truth improves OPE.

free parameters (4)
  • extreme_value_threshold = 9,999%
    Hand-chosen threshold for classifying 'EXTREME' outcomes (§4.2.4); directly affects reported extreme rates and stability comparisons.
  • iteration_count = 7
    Number of optimization iterations used for agents (§3.3); affects both success rates and measured improvements.
  • base_llm_model = Gemini-2.5-Flash
    Underlying LLM chosen for cost/performance (§4.2.1); results may not generalize to other models.
  • best_iteration_selector = lightweight LLM
    A lightweight LLM selects the best iteration from heterogeneous outputs (§3.3); this selection step is not a fixed deterministic rule and could bias reported improvements.
assumptions (5)
  • domain assumption Standard OPE estimators (DM, IPW, DR) and MSE/relative estimation error are valid measures of OPE performance.
    The evaluation is built on these metrics (§2.1, §4.2.4); if they are not valid for the test notebooks, the optimization targets are meaningless.
  • domain assumption Positive change in Scope-RL's relative_policy_value represents improvement in OPE quality.
    Stated in §4.2.4; this is the weakest assumption because no ground-truth policy value is used to verify that higher estimates mean better estimation.
  • domain assumption The selected OBP and Scope-RL notebooks are representative of OPE practice.
    The benchmark's generalizability claim relies on this selection (§4.2.3); only 3 OBP and 15 Scope-RL notebooks are used.
  • domain assumption The bypass logic that loads pre-trained artifacts instead of executing training calls preserves the optimization task.
    Introduced in §4.2.3 to avoid 30-60 minute retraining; if bypassing changes what the agent optimizes, the Scope-RL results may not reflect real OPE pipelines.
  • domain assumption Modifications that improve metrics on logged data indicate improved OPE performance without held-out validation.
    Best-iteration selection and all improvement metrics use the same logged data (§3.4, §4.2.4); no train/test split or cross-validation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents." pith.science (2026). https://pith.science/paper/GFNX57DY

@misc{pith2026251100802,
  author       = {Pith},
  title        = {Pith review of: GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFNX57DY}},
  note         = {Machine review of arXiv:2511.00802}
}
read the original abstract

With data-driven development now widely adopted, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands resources for design, implementation, and deployment, and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring weeks of data collection. To address this, the growing research area of off-policy evaluation (OPE), or offline A/B testing, assesses new technologies offline using previously collected logged data. OPE is also a fundamental problem in reinforcement learning and is important where online testing is expensive or risky, such as healthcare, recommender systems, education, and robotics. Despite advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can automatically optimize OPE implementations. We propose GrowthHacker, a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the metrics to guide subsequent optimization. We evaluate methods on Open Bandit Pipeline (OBP) and Scope-RL, and develop a two_agent framework that addresses limitations of existing frameworks while reducing complexity. Across both libraries, two_agent shows the highest reliability (98.1%-100% success rate) and positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated "growth hackers" to continuously improve OPE systems, with implications for scaling data-driven decision-making where manual optimization is expensive.

Figures

Figures reproduced from arXiv: 2511.00802 by the authors.

Figure 1
Figure 1. Flow Diagram for GrowthHacker, a benchmark system for optimizing Off-Policy Evaluation through code modification, using [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Visual Illustration for Two-Agent Architecture [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Visual Illustration for Performance of Estimator on RTB Basics (discrete). [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Stacked bar chart showing normalized outcome proportions per framework. Each bar sums to 100%. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 9 linked inside Pith

  1. [1]

    Florian Auer, Rasmus Ros, Lukas Kaltenbrunner, Per Runeson, and Michael Felderer. 2021. Controlled experimentation in continuous experimentation: Knowledge and challenges.Information and Software Technology134 (2021), 106551

    2021

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

  3. [3]

    Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. 2020. Never Give Up: Learning Directed Exploration Strategies.ArXivabs/2002.06038 (2020). https://api.semanticscholar.org/CorpusID:211126477

    arXiv 2020

  4. [4]

    Rafael Barbarroxa, Luis Gomes, and Zita Vale. 2025. Benchmarking large language models for multi-agent systems: A comparative analysis of AutoGen, CrewAI, and TaskWeaver. InAdvances in Practical Applications of Agents, Multi-Agent Systems, and Digital Twins: The PAAMS Collection. Springer, 39–48

    2025

  5. [5]

    Alina Beygelzimer and John Langford. 2009. The offset tree for learning with partial labels. InProceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 129–138

    2009

  6. [6]

    Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising.Journal of Machine Learning Research 14, 11 (2013)

    2013

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    arXiv 2021

  8. [8]

    Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao. 2023. On the opportunities and challenges of offline reinforcement learning for recommender systems.ACM Transactions on Information Systems(2023)

    2023

Show all 39 references
  1. [9]

    Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly robust policy evaluation and optimization. InStatistical Science, Vol. 29. 485–511

    2014

  2. [10]

    Aleksander Fabijan, Pavel Dmitriev, Helena Holmstrom Olsson, and Jan Bosch. 2018. The online controlled experiment lifecycle.IEEE Software37, 2 (2018), 60–67. 1https://zenodo.org/records/17496870 Manuscript submitted to ACM GrowthHacker: Automated Off-Policy Evaluation Optimiz...

    2018

  3. [11]

    Fabian Fagerholm, Alejandro Sanchez Guinea, Hanna Mäenpää, and Jürgen Münch. 2017. The RIGHT model for continuous experimentation. Journal of Systems and Software123 (2017), 292–305

    2017

  4. [12]

    Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at facebook.IEEE Internet Computing17, 4 (2013), 8–17

    2013

  5. [13]

    Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda.Journal of Systems and Software123 (2017), 176–189

    2017

  6. [14]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-based llms for code optimization. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 254–266

    2024

  7. [15]

    Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline a/b testing for recommender systems. InProceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 198–206

    2018

  8. [16]

    Alois Gruson, Praveen Chandar, Christophe Charbuillet, James McInerney, Samantha Hansen, Damien Tardieu, and Ben Carterette. 2019. Offline evaluation to make decisions about playlistrecommendation algorithms. InProceedings of the Twelfth ACM International Conference on Web Sea...

    2019

  9. [17]

    Danning Jiang, Zheng Wu, Chin-Yi Hsieh, Tse-Hsun Chen, et al. 2023. Impact of large language models on generating software specifications.arXiv preprint arXiv:2306.03324(2023)

    2023 arXiv

  10. [18]

    Thorsten Joachims and Adith Swaminathan. 2016. Counterfactual evaluation and learning for search, recommendation and ad placement. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 1199–1201

    2016

  11. [19]

    Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nataka, and Yuta Saito. 2023. SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation.arXiv preprint arXiv:2311.18206(2023)

    2023 arXiv

  12. [20]

    2020.Trustworthy online controlled experiments: A practical guide to a/b testing

    Ron Kohavi, Diane Tang, and Ya Xu. 2020.Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press

    2020

  13. [21]

    Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. 2018. Breaking the curse of horizon: Infinite-horizon off-policy estimation.Advances in Neural Information Processing Systems31 (2018)

    2018

  14. [22]

    Yihan Mao, Minghui Pan, Wing Lam Lin, Wei Chen, and Tse-Hsun Chen. 2023. RAPGen: An approach for fixing code inefficiencies in zero-shot. arXiv preprint arXiv:2306.17077(2023)

    2023 arXiv

  15. [23]

    Niko Pajkovic. 2022. Algorithms and taste-making: Exposing the Netflix Recommender System’s operational logics.Convergence28, 1 (2022), 214–235

    2022

  16. [24]

    Doina Precup, Richard S Sutton, and Satinder Singh. 2000. Eligibility traces for off-policy policy evaluation.Computer Science Department Faculty Publication Series(2000), 80

    2000

  17. [25]

    Agn˙e Reklait˙e and Jevgenij Gamper. 2022. Offline assessment of interference effects in a series of AB tests. InProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering. 262–263

    2022

  18. [26]

    Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. 2021. Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation. arXiv:2008.07146 [cs.LG] https://arxiv.org/abs/2008.07146

    2021 arXiv

  19. [27]

    Yuta Saito, Takuma Udagawa, Haruka Kiyohara, Kazuki Mogi, Yusuke Narita, and Kei Tateno. 2021. Evaluating the robustness of off-policy evaluation. InProceedings of the 15th ACM Conference on Recommender Systems. 114–123

    2021

  20. [28]

    Prabhat Kumar Saraswat, Samuel William, and Eswar Reddy. 2021. A Hybrid Approach for Offline A/B Evaluation for Item Ranking Algorithms in Recommendation Systems. InProceedings of the First International Conference on AI-ML Systems. 1–6

    2021

  21. [29]

    Adith Swaminathan and Thorsten Joachims. 2015. The self-normalized estimator for counterfactual learning.Advances in Neural Information Processing Systems28 (2015)

    2015

  22. [30]

    Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlapping experiment infrastructure: More, better, faster experimentation. InProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 17–26

    2010

  23. [31]

    Philip Thomas and Emma Brunskill. 2016. Data-efficient off-policy policy evaluation for reinforcement learning.International Conference on Machine Learning(2016), 2139–2148

    2016

  24. [32]

    Bradley C Turnbull. 2019. Learning Intent to Book Metrics for Airbnb Search. InThe World Wide Web Conference. 3265–3271

    2019

  25. [33]

    Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. 2022. A review of off-policy evaluation in reinforcement learning.arXiv preprint arXiv:2212.06355(2022)

    2022 arXiv

  26. [34]

    Weishi Wang, Yue Wang, Shafiq Joty, and Steven CH Hoi. 2023. Rap-gen: Retrieval-augmented patch generation with codet5 for automatic program repair. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2023

  27. [35]

    Jie JW Wu. 2024. AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement Engineering. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 472–476

    2024

  28. [36]

    Jie JW Wu, Thomas A Mazzuchi, and Shahram Sarkani. 2023. Comparison of multi-criteria decision-making methods for online controlled experiments in a launch decision-making framework.Information and Software Technology155 (2023), 107115

    2023

  29. [37]

    Jie JW Wu, Thomas A Mazzuchi, and Shahram Sarkani. 2023. A multi-objective evolutionary approach towards automated online controlled experiments.Journal of Systems and Software203 (2023), 111703

    2023

  30. [38]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2023. AutoGen: Enabling next-gen LLM applications via multi-agent conversation.arXiv preprint arXiv:2308.08155(2023). Manuscript submitted to ACM 2...

    2023 arXiv

  31. [39]

    Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. 2015. From infrastructure to culture: A/B testing challenges in large scale social networks. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2227–2236. Man...

    2015

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.