Pith. sign in

REVIEW 3 major objections 4 minor 213 references

This paper proves that a proactive, test-space-driven flywheel resolves model errors in K log K flywheel iterations, versus M log M for reactive patching.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:35 UTC pith:SQSEFHLV

load-bearing objection A clean stylized model showing proactive grouping beats reactive patching, but the entire advantage rests on an unvalidated group-removal rule—and the paper's own autonomous-driving example contradicts that rule. the 3 major comments →

arxiv 2607.20532 v1 pith:SQSEFHLV submitted 2026-07-10 cs.LG cs.AI

Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development

Nadine Chang , Maying Shen , Jialiang Wang , Rafid Mahmood , Jose M. Alvarez This is my paper
classification cs.LG cs.AI
keywords AI flywheelreactive patchingproactive test-driven developmenttest spacecoupon collectorlong-tail errorsmodel maintenancegeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that maintaining deployed AI systems by reactively patching each observed error is structurally wasteful, because rare edge cases are hard to collect and fixes don't generalize. It proposes a proactive flywheel that maps each error to a 'test space' of task conditions, so one successful fix removes an entire group of related failure scenarios. The authors prove that this proactive strategy needs on the order of K log K flywheel iterations to find and fix all M scenarios, where K is the number of underlying factor groups, versus M log M for reactive patching. If the grouping is meaningful, this is orders of magnitude fewer cycles and a much smaller backlog. The paper stops short of building the test space; it lays out open research problems for doing so.

Core claim

The central claim is a pair of asymptotic bounds. In a stylized model where M undiscovered scenarios each belong to one of K evenly sized factor groups, a reactive flywheel that patches one scenario at a time with success probability p_R has expected completion time (M/p_R) H_M, scaling as Θ((M/p_R) log M). A proactive flywheel that successfully fixes the whole factor group with probability p_P has expected completion time (K/p_P) H_K, scaling as Θ((K/p_P) log K). Because K is much smaller than M whenever many scenarios share a root cause, the proactive flywheel uses far fewer iteration rounds, and its backlog is provably no worse after a finite crossover time. The authors are explicit that

What carries the argument

The load-bearing object is the 'test space,' an n-dimensional space of task conditions that decomposes high-level requirements into factor groups. Each observed error is mapped to a location in this space; when the developer fixes the factor rather than the single data point, the entire group K(i) is removed from the unresolved set with probability p_P. The proof technique is a coupon-collector analysis: waiting times to remove scenarios are geometric, yielding harmonic sums. The test space is what changes the effective number of coupons from M to K.

Load-bearing premise

The entire advantage rests on one assumption: when the developer succeeds in fixing an observed error, the same fix removes every unseen scenario in the same factor group, with probability p_P; real systems may not have such clean group-level fixes.

What would settle it

Look at any large deployed model with a long tail of logged failures. Compute the actual number of retraining rounds needed to clear all observed error types and compare it to (M/p_R) H_M under reactive patching. Then group the errors by claimed root cause and measure whether a single fix resolves all errors in the group; if the group-level success probability is near zero, the proactive bound Θ((K/p_P) log K) is not achievable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the test space can be built with K much smaller than M, proactive maintenance requires orders of magnitude fewer flywheel iterations than reactive patching for the same set of failure scenarios.
  • The expected backlog under the proactive policy falls below the reactive backlog after a finite crossover time and reaches a far lower maximum, reducing repeated human triage.
  • Even when proactive fixes are two or three times less likely to succeed per pass, they win whenever each success removes a sufficiently large group of related scenarios.
  • The coupon-collector bottleneck disappears: the platform only needs to encounter each factor, not every rare edge case, to cure the long tail.
  • When factors are essentially one-to-one with scenarios (K ≈ M) or proactive fixes are far harder, the paper's Corollary 3.3 condition shows reactive correction remains competitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The mathematical result is indifferent to what the 'test space' actually is; any representation that reliably groups scenarios by fixable root cause would deliver the speedup. The hard engineering problem is discovering that grouping, which the paper leaves open.
  • The model suggests a concrete test for any deployed system: log failures, group them by root cause, and measure the probability that one fix resolves the whole group. If that probability is high, proactive flywheel should win; if not, this argument cannot support it.
  • A natural extension is to apply the same flywheel logic to pre-deployment validation: use the test space to generate the rarest group observations directly, turning proactive maintenance into proactive test generation.
  • The backlog analysis hints that adding evaluation-before-retraining gates, not just more retraining, is what the proactive flywheel needs to avoid catastrophic forgetting, a cost the paper explicitly assumes away.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This position paper argues that deployed AI systems should move from reactive, incident-driven patching to a proactive, test-space-driven flywheel. It defines a stylized model with M usage scenarios partitioned into K equal-sized factors. In the reactive policy, a successful update removes only the observed scenario; in the proactive policy, a successful update removes the entire factor group. The main theoretical results (Theorems 3.1–3.2) give expected completion times of (M/p_R)H_M and (K/p_P)H_K, so proactive scaling is K log K instead of M log M. Theorems 3.4–3.5 and Proposition 3.6 analyze backlog size and show asymptotic advantage for proactive under a stated condition. Section 4 discusses test-space construction, feedback alignment, weakness-pattern recognition, and solution routing, with six open problems. Section 5 acknowledges regimes where reactive correction may be sufficient.

Significance. If the group-removal assumption is accepted, the scaling comparison is mathematically correct and provides a clear, teachable model for why proactive maintenance can reduce flywheel iterations. The paper ships transparent proofs and explicitly lists its assumptions, which is a strength for a position paper; it also honestly discusses alternative views and open problems. However, the external validity of the central claim rests entirely on the assumption that fixing one observed scenario removes every unseen scenario sharing a factor. That assumption is not demonstrated, and the paper's own AD example violates the single-factor model used in the theorems. The abstract's unqualified 'required' and 'better long-term scaling' overstate the formal result, which is conditional on meaningful group structure and on p_P/p_R satisfying Corollary 3.3. The contribution is therefore a useful conceptual framework and a correct analysis of a stylized model, not a demonstrated general result.

major comments (3)
  1. [§3.1, Proactive Flywheel rule] The entire scaling advantage in Theorems 3.1–3.2 is driven by the update rule U_{t+1}=U_t\K(i_t) with probability p_P, i.e., a successful fix of one observed scenario removes the whole factor group. This is a modeling assumption, not a consequence of the test-space concept. The paper states that the assumptions 'do not affect the qualitative analysis' but gives no argument or proof. Since the abstract claims 'mathematically proving' better scaling without qualification, this is load-bearing. Please either prove robustness to overlapping or multi-factor structures, or explicitly state that the theorem applies only when the single-factor group-removal rule holds, and temper the abstract accordingly.
  2. [§4.1 vs. §3.1] The AD example decomposes one error into multiple factors ('yield for public service vehicles', 'flashing lights', 'low visibility'), which contradicts the model's assumption that 'each scenario is associated with a single factor' and that factors partition scenarios evenly. The claim that this simplification 'does not affect the qualitative analysis' is not substantiated. A scenario that maps to several groups may only be partially fixed by addressing one group, and the group-removal probability p_P is not well-defined. Please provide an argument that multi-factor or overlapping assignments preserve the K log K scaling, or restrict the theoretical claim to the single-factor case and clearly separate it from the motivating application.
  3. [Abstract and §5] The abstract says a proactive flywheel 'is required' and 'achieves better long-term scaling', but Corollary 3.3 and Section 5 explicitly identify regimes where reactive correction may be sufficient or preferable: K≈M, very large p_R/p_P, or bounded enumerable error spaces. These statements are internally inconsistent. Please revise the abstract and conclusion to state the conditional nature of the claim, e.g., 'proactive is advantageous when failures share meaningful factors and the group-fix success probability is not too low relative to reactive per-scenario success', and cite the Corollary 3.3 condition.
minor comments (4)
  1. [Appendix B, proof of Theorem 3.4] The definitions of S_{i,s} and F_{i,t} are inconsistent with the later algebra. S_{i,s} is defined as A_{i,s} ∪ {error i was fixed}, which is a union, yet it is called a 'successful resolution'; F_{i,t} is defined as the intersection of S_{i,s}, which is not 'not successfully resolved by time t'. The step 'sixth equality uses S⊆A' does not produce the displayed formula. The final expression M[(1-p_R/M)^t - (1-1/M)^t] is correct if backlog is 'observed at least once and never fixed', but the proof as printed is not valid and should be rewritten.
  2. [Corollary 3.3] The corollary states that proactive 'incurs fewer iterations' if p_P/p_R ≥ K H_K / (M H_M). With equality the expected times are equal, so 'fewer' should be 'at most as many' or the inequality should be strict. Minor rewording.
  3. [§3.2, Figure 2 discussion] The sentence 'reactive paradigm requires more flywheel iterations than proactive and grows in multiple orders of magnitude as the number of scenarios M or proactive success rate p_P increases' is misleading: the gap grows as p_P decreases, not as p_P increases. Please rephrase to 'as M/K increases or p_P decreases'.
  4. [§4.3, Feedback Gathering] There is a stray typographical 'd' at the end of the paragraph: '...automatic proactive flywheel. d' should be removed.

Circularity Check

0 steps flagged

No significant circularity: the proactive scaling result is a direct, explicitly stated consequence of the model's group-removal update rule, not a fitted or self-referential prediction.

full rationale

I walked the derivation chain from Section 3.1 to Theorems 3.1/3.2 and the backlog theorems. The reactive and proactive processes are defined by explicit update rules: reactive removes a single observed scenario with probability pR; proactive removes the entire factor group K(i_t) with probability pP. The expected-completion formulas E[T_R]=(M/pR)H_M and E[T_P]=(K/pP)H_K are then derived (Appendix B) from the waiting-time arguments for those rules; they are not fitted to data and no parameter is post-hoc chosen to reproduce the conclusion. The qualitative advantage of proactive is indeed loaded into the modeling assumption that a successful factor-level fix removes all M/K scenarios in the group, but the paper states this assumption openly ('update the model to try to simultaneously resolve all usage scenarios in the group'), labels it an analytical simplification, and does not conceal the dependence. Whether real test spaces can actually deliver such group-level fixes is an empirical validity question, not a circularity. The only overlapping-author citation is the 'train space' reference (Shen et al., 2025) used as inspiration for the 'test space' concept; it is not used to justify either theorem, so it is not load-bearing. No uniqueness theorem, imported ansatz, or renaming of a known result substitutes for the derivation. The result is conditional and idealized, but self-contained.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

The central claim rests on a stylized model in which all key quantities are given parameters and the scaling advantage follows from the group-removal assumption. There are no fitted constants or hidden empirical inputs; the open problems underscore that the test-space mechanism itself is not demonstrated.

free parameters (4)
  • M (number of undiscovered scenarios)
    Abstract model parameter; varied asymptotically; not measured from any real system.
  • K (number of factors)
    Abstract model parameter; M/K scenarios per factor; no methodology is given to determine K from a real task's requirements.
  • pR (reactive success probability per iteration)
    Model parameter assumed in (0,1]; not estimated. The key comparison treats it as given.
  • pP (proactive group-fix success probability)
    Model parameter assumed in (0,1] with pP < pR; no empirical basis for its magnitude.
axioms (6)
  • domain assumption Each flywheel step draws one scenario uniformly at random from M scenarios.
    Section 3.1: 'let i in [M] be a random scenario drawn uniformly from M scenarios.' This makes discovery a coupon-collector process; the paper later calls this 'optimistic by assuming uniform exposure.'
  • ad hoc to paper Every scenario belongs to exactly one factor, and the factors partition the scenarios evenly (M/K per group).
    Section 3.1: 'we assume that each scenario is associated with a single factor. The factors partition the scenarios evenly.' Stated for analytical clarity; real errors may involve multiple interacting factors.
  • ad hoc to paper A successful proactive update resolves all scenarios in the factor group, not just the observed one.
    Section 3.1 proactive rule: U_{t+1}=U_t \ K(i_t) w.p. pP. This converts M-object coupon collection into K-object coupon collection and is the source of the claimed advantage.
  • domain assumption Errors are finite; each scenario fails at most once; fixes persist (no catastrophic forgetting).
    Section 3.1 sets U0=[M]; Section 3.2 notes the analysis is 'optimistic by assuming ... no catastrophic forgetting, which uniformly affect both policies.'
  • standard math Geometric waiting times and coupon-collector expectations (standard probability).
    Appendix B derives E[T_R] and E[T_P] as sums of geometric waiting times, using harmonic numbers H_M and H_K.
  • standard math For backlog proofs, observed/failure/success counts in a cluster follow a multinomial distribution and S_{i,s} is a subset of A_{i,s}.
    Appendix B Theorems 3.4/3.5 use multinomial counting and the set-inclusion simplification S_{1,s} subset A_{1,s}.
invented entities (2)
  • Test space no independent evidence
    purpose: An n-dimensional space of vectorized task conditions onto which feedback errors are mapped, intended to enable group-level fixes and proactive coverage.
    Defined in Section 4.1 as a conceptual abstraction; no implementation, no empirical validation, and its construction is listed as Open Problems 1-2 and 4.
  • Frequency-weighted atlas no independent evidence
    purpose: A concrete implementation of the test space that specifies which conditions should guide updates and with what weight.
    Mentioned briefly in the Section 3 introduction as an 'atlas-style implementation'; no construction details or evidence are provided.

pith-pipeline@v1.3.0-alltime-deepseek · 18007 in / 14227 out tokens · 150533 ms · 2026-08-02T07:35:12.969345+00:00 · methodology

0 comments
read the original abstract

Many modern AI systems are designed to operate under diverse, open-ended, use-cases. To help generalize deployed systems, many deployed-system maintenance pipelines use a reactive AI flywheel that observes emerging feedback from user behavior (errors) and patches the model accordingly. However, when used as the primary maintenance mechanism, these flywheels often ignore the broader context of these errors within the system's objectives, failing to preempt potential future edge cases, which leads to more unnecessary flywheel iterations. Also, it is statistically increasingly difficult to collect remaining errors due to the long-tail nature of open-world use-cases. This position paper argues that a proactive test-driven flywheel is required to address reactive flywheel's limitations and to approach a generalizable system. We advocate for creating a "test space" to technically map feedback data to task objectives, evolving the flywheel from reactive to proactive. We augment our position by mathematically proving a proactive one achieves better long-term scaling with fewer iterations than the reactive flywheel.

Figures

Figures reproduced from arXiv: 2607.20532 by Jialiang Wang, Jose M. Alvarez, Maying Shen, Nadine Chang, Rafid Mahmood.

Figure 1
Figure 1. Figure 1: Overview of reactive and proactive AI flywheel. In the reactive paradigm, feedback is triaged and specific errors are patched (e.g. ‘pedestrian’ and ‘firetruck’), failing to generalize beyond specific errors. In contrast, the proactive paradigm maps feedback to a test space of required task conditions and addresses the entire task condition (e.g. ‘yield for public service vehicles’ and ‘recognizing people’… view at source ↗
Figure 2
Figure 2. Figure 2: The expected number of flywheel iterations required to correct all scenarios under reactive and proactive flywheel (i.e., Thm 3.1 and 3.2). Reactive iterations can grow to multiple orders of magnitude more as the number of scenarios M or the likelihood of addressing an entire factor group pP increases. 3.2. Reactive paradigm needs more flywheel iterations to correct errors A scenario i is revealed with pro… view at source ↗
Figure 3
Figure 3. Figure 3: Visualizing where proactive flywheel outperforms reac￾tive (in blue) given the difficulty of proactive versus reactive and the number of scenarios per group (i.e., Corollary 3.3). Even if proactive correction is twice as difficult to achieve per iteration, it is attractive if it can address sufficient scenarios. Note log y-axis. when the task is difficult and M/K is small, suggesting that undiscovered scen… view at source ↗
Figure 4
Figure 4. Figure 4: Expected backlog after a certain number of flywheel iter￾ations (i.e., Thm. 3.4 and 3.5). Although proactive flywheel may incur slightly larger backlogs in the early stages, proactive backlog falls rapidly, and the maximum backlog is orders of magnitude fewer than that of reactive. The backlog of any flywheel policy is composed of the gap between how quickly new failure scenarios are discovered (e.g., (1 −… view at source ↗
Figure 5
Figure 5. Figure 5: The paradigm for building a proactive test-driven flywheel. (a) We first create a test space from the task testing conditions, without using any real data. (b) The test space enables a proactive test-driven flywheel. During the loop, gathered feedback is aligned to the test space. Within the test space, weakness patterns associated with the task testing conditions are recognized and later resolved. concret… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

213 extracted references · 2 canonical work pages

  1. [1]

    2023 , title =

    ABC7 San Francisco , author =. 2023 , title =

    2023

  2. [2]

    InsideEVs , author=

    Tesla Model Y Driving On FSD Knocks Down Kid-Sized Dummies , url=. InsideEVs , author=

  3. [3]

    Automated Driving Systems 2.0 , author =

  4. [4]

    2025 , title =

    Mission Local , author =. 2025 , title =

    2025

  5. [5]

    ACM Transactions on Information Systems (TOIS) , volume=

    RLPS: A reinforcement learning--based framework for personalized search , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2021 , publisher=

    2021

  6. [6]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  7. [7]

    2022 , month =

    Lex Clips , title =. 2022 , month =

    2022

  8. [8]

    California Driver’s Handbook , author =

  9. [9]

    Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations , url=

    Tang, Hongyan and Liu, Junning and Zhao, Ming and Gong, Xudong , year=. Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations , url=. doi:https://doi.org/10.1145/3383313.3412236 , journal=

    arXiv

  10. [10]

    RLPS: A Reinforcement Learning–Based Framework for Personalized Search , volume=

    Yao, Jing and Dou, Zhicheng and Xu, Jun and Wen, Ji-Rong , year=. RLPS: A Reinforcement Learning–Based Framework for Personalized Search , volume=. ACM Transactions on Information Systems , publisher=. doi:https://doi.org/10.1145/3446617 , number=

    work page doi:10.1145/3446617

  11. [11]

    Wayve , author=

    Crossing the Pond and Beyond: Generalizable AI Driving for Global Deployment , url=. Wayve , author=. 2025 , month=

    2025

  12. [12]

    IEEE Reliability Magazine , author=

    Lessons From the Cruise Robotaxi Pedestrian Dragging Mishap , volume=. IEEE Reliability Magazine , author=. 2024 , month=. doi:https://doi.org/10.1109/mrl.2024.3412874 , number=

    arXiv 2024

  13. [13]

    arXiv preprint arXiv:2504.13161 , year=

    Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training , author=. arXiv preprint arXiv:2504.13161 , year=

    Pith/arXiv arXiv

  14. [14]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 , pages=

  15. [15]

    arXiv preprint arXiv:1708.00489 , year=

    Active learning for convolutional neural networks: A core-set approach , author=. arXiv preprint arXiv:1708.00489 , year=

    Pith/arXiv arXiv

  16. [16]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Fairdedup: Detecting and mitigating vision-language fairness disparities in semantic dataset deduplication , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  17. [17]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  18. [18]

    2009 IEEE conference on computer vision and pattern recognition , pages=

    Imagenet: A large-scale hierarchical image database , author=. 2009 IEEE conference on computer vision and pattern recognition , pages=. 2009 , organization=

    2009

  19. [19]

    Journal of marketing research , volume=

    What makes online content viral? , author=. Journal of marketing research , volume=. 2012 , publisher=

    2012

  20. [20]

    Trends in Cognitive Sciences , year=

    The psychology of virality , author=. Trends in Cognitive Sciences , year=

  21. [21]

    arXiv preprint arXiv:1712.01815 , year=

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm , author=. arXiv preprint arXiv:1712.01815 , year=

    Pith/arXiv arXiv

  22. [22]

    2025 , url=

    Why benchmarking is hard , author=. 2025 , url=

    2025

  23. [23]

    Stochastic Models , volume=

    The coupon-collector problem revisited—a survey of engineering problems and computational methods , author=. Stochastic Models , volume=. 1997 , publisher=

    1997

  24. [24]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Recent advances in open set recognition: A survey , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2020 , publisher=

    2020

  25. [25]

    2014 , publisher=

    Understanding machine learning: From theory to algorithms , author=. 2014 , publisher=

    2014

  26. [26]

    Proceedings of the 2021 international conference on management of data , pages=

    Production machine learning pipelines: Empirical analysis and optimization opportunities , author=. Proceedings of the 2021 international conference on management of data , pages=

    2021

  27. [27]

    2017 IEEE international conference on big data (big data) , pages=

    The ML test score: A rubric for ML production readiness and technical debt reduction , author=. 2017 IEEE international conference on big data (big data) , pages=. 2017 , organization=

    2017

  28. [28]

    arXiv preprint arXiv:2509.24219 , year=

    ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning , author=. arXiv preprint arXiv:2509.24219 , year=

    arXiv

  29. [29]

    Information fusion , volume=

    Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges , author=. Information fusion , volume=. 2020 , publisher=

    2020

  30. [30]

    ACM Computing Surveys , year=

    Continual learning of large language models: A comprehensive survey , author=. ACM Computing Surveys , year=

  31. [31]

    Proceedings of the IEEE international conference on computer vision , pages=

    Incremental learning of object detectors without catastrophic forgetting , author=. Proceedings of the IEEE international conference on computer vision , pages=

  32. [32]

    Order: Open world object detection on road scenes , author=. Proc. NeurIPS Workshops , volume=

  33. [33]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Towards open world object detection , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  34. [34]

    Neural networks , volume=

    Continual lifelong learning with neural networks: A review , author=. Neural networks , volume=. 2019 , publisher=

    2019

  35. [35]

    Transactions on machine learning research , year=

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year=

  36. [36]

    Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Adaptive testing and debugging of NLP models , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  37. [37]

    International conference on machine learning , pages=

    Tensorfuzz: Debugging neural networks with coverage-guided fuzzing , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  38. [38]

    2018 IEEE Intelligent Vehicles Symposium (IV) , pages=

    Adaptive stress testing for autonomous vehicles , author=. 2018 IEEE Intelligent Vehicles Symposium (IV) , pages=. 2018 , organization=

    2018

  39. [39]

    arXiv preprint arXiv:2209.07858 , year=

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned , author=. arXiv preprint arXiv:2209.07858 , year=

    Pith/arXiv arXiv

  40. [40]

    Advances in neural information processing systems , volume=

    Hard negative mixing for contrastive learning , author=. Advances in neural information processing systems , volume=

  41. [41]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Class-balanced loss based on effective number of samples , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  42. [42]

    Proceedings of the IEEE international conference on computer vision , pages=

    Focal loss for dense object detection , author=. Proceedings of the IEEE international conference on computer vision , pages=

  43. [43]

    Meet Michelangelo: Uber’s Machine Learning Platform , year =

  44. [44]

    Advances in neural information processing systems , volume=

    Adversarial training for high-stakes reliability , author=. Advances in neural information processing systems , volume=

  45. [45]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  46. [46]

    Tesla Autonomy Day , year =

  47. [47]

    European Conference on Computer Vision , pages=

    Improving the intra-class long-tail in 3d detection via rare example mining , author=. European Conference on Computer Vision , pages=. 2022 , organization=

    2022

  48. [48]

    arXiv preprint arXiv:2005.04118 , year=

    Beyond accuracy: Behavioral testing of NLP models with CheckList , author=. arXiv preprint arXiv:2005.04118 , year=

    Pith/arXiv arXiv 2005

  49. [49]

    Advances in neural information processing systems , volume=

    Hidden technical debt in machine learning systems , author=. Advances in neural information processing systems , volume=

  50. [50]

    2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages=

    Failures and fixes: A study of software system incident response , author=. 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages=. 2020 , organization=

    2020

  51. [51]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  52. [52]

    arXiv preprint arXiv:2202.03286 , year=

    Red teaming language models with language models , author=. arXiv preprint arXiv:2202.03286 , year=

    Pith/arXiv arXiv

  53. [53]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  54. [54]

    arXiv preprint arXiv:2501.07238 , year=

    Lessons from red teaming 100 generative ai products , author=. arXiv preprint arXiv:2501.07238 , year=

    Pith/arXiv arXiv

  55. [55]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Training region-based object detectors with online hard example mining , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  56. [56]

    Disengagement Reports , howpublished =

  57. [57]

    2022 , publisher=

    Dataset shift in machine learning , author=. 2022 , publisher=

    2022

  58. [58]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and transferable adversarial attacks on aligned language models , author=. arXiv preprint arXiv:2307.15043 , year=

    Pith/arXiv arXiv

  59. [59]

    and Schneider, Jeff G

    Donmez, Pinar and Carbonell, Jaime G. and Schneider, Jeff G. , title =. Proceedings of the SIAM International Conference on Data Mining (SDM) , year =

  60. [60]

    Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , year =

    Jung, Hee Joon and Park, Yoonsuck and Lease, Matthew , title =. Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , year =

  61. [61]

    Proceedings of the European Conference on Information Retrieval (ECIR) , series =

    Jung, Hee Joon and Lease, Matthew , title =. Proceedings of the European Conference on Information Retrieval (ECIR) , series =

  62. [62]

    , title =

    Bragg, Jonathan and Weld, Daniel S. , title =. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , year =

  63. [63]

    and Oh, Sewoong and Shah, Devavrat , title =

    Karger, David R. and Oh, Sewoong and Shah, Devavrat , title =. Operations Research , volume =

  64. [64]

    Philip and Skene, Allan M

    Dawid, A. Philip and Skene, Allan M. , title =. Journal of the Royal Statistical Society: Series C (Applied Statistics) , volume =

  65. [65]

    Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , booktitle =

    Whitehill, Jacob and Wu, Ting. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , booktitle =

  66. [66]

    , title =

    Welinder, Peter and Branson, Steven and Perona, Pietro and Belongie, Serge J. , title =. Advances in Neural Information Processing Systems (NIPS) 23 , pages =

  67. [67]

    and Kesidis, George , title =

    Kurve, Anand and Miller, David J. and Kesidis, George , title =. IEEE Transactions on Knowledge and Data Engineering , volume =

  68. [68]

    Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , year =

    Mao, Andrew and Kamar, Ece and Horvitz, Eric , title =. Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , year =

  69. [69]

    and Lintott, Chris J

    Mao, Andrew and Chen, Yiling and Horvitz, Eric and Schwamb, Megan E. and Lintott, Chris J. and Smith, Arfon M. , title =. Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , year =

  70. [70]

    Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI) , year =

    Yin, Ming and Chen, Yiling , title =. Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI) , year =

  71. [71]

    , title =

    Xu, Shuling and Hall, Nicholas G. , title =. European Journal of Operational Research , volume =

  72. [72]

    OpenAI’s CEO says the age of giant AI models is already over , url=

    Knight, Will , year=. OpenAI’s CEO says the age of giant AI models is already over , url=. Wired , publisher=

  73. [73]

    Management Science , volume=

    Contracting, pricing, and data collection under the AI flywheel effect , author=. Management Science , volume=. 2022 , publisher=

    2022

  74. [74]

    IEEE Transactions on Information Theory , volume=

    Sequential classification with empirically observed statistics , author=. IEEE Transactions on Information Theory , volume=. 2021 , publisher=

    2021

  75. [75]

    Active sequential hypothesis testing , author=

  76. [76]

    Advances in Neural Information Processing Systems , volume=

    Greedy approximation algorithms for active sequential hypothesis testing , author=. Advances in Neural Information Processing Systems , volume=

  77. [77]

    arXiv preprint arXiv:1810.04805 , year=

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. arXiv preprint arXiv:1810.04805 , year=

    Pith/arXiv arXiv

  78. [78]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  79. [79]

    Amazon , publisher=

    Amazon SageMaker Data Labeling Pricing , url=. Amazon , publisher=

  80. [80]

    Advances in Neural Information Processing Systems , volume=

    Optimizing data collection for machine learning , author=. Advances in Neural Information Processing Systems , volume=

Showing first 80 references.