{"id":"5e215df5-b6e6-4a9a-80d7-93891b65c3ce","arxiv_id":"2607.06195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.","lead":"LogicHunter is a fuzzing framework that finds logic bugs in LLM agent frameworks like LangChain and LlamaIndex by generating spec-compliant test inputs and using an LLM-based agentic oracle to diagnose failures. It found 40 previously unknown bugs (30 confirmed, 26 fixed) where prior tools found none, and its oracle achieves 91.17% precision versus 29.27% for the best passive baseline.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Oracle precision (91.17%) is measured on a self-referential corpus generated, filtered, and labeled by the same team; supplementary external datasets are incomplete (one bugs-only, one non-bugs-only), leaving the headline precision claim under-validated on independently sourced failures.","rationale":"The reader correctly identified the self-referential evaluation loop as the primary structural weakness. The precision claim (91.17%) is measured on a corpus where generator, oracle, and labels all come from the same team, and the two supplementary datasets each cover only one side of the precision/recall tradeoff (bugs-only or non-bugs-only), with no independent labeling.\n\nHowever, I recommend UNCHANGED rather than a harsher verdict for three reasons. First, the 40 bugs found (30 confirmed, 26 fixed by external developers) constitute strong, independently validated evidence for the system's practical effectiveness — this is the most important claim and it holds regardless of the precision measurement issue. Second, the supplementary datasets, while incomplete, do provide partial external validation: 60.65% recall on post-cutoff known bugs and 0.04% FPR on TitanFuzz hard-negatives show the oracle is not purely overfit to its own generator's output. Third, the ablation study is thorough and the system design is well-motivated.\n\nThe CONDITIONAL verdict is appropriate: the paper should be accepted pending release of the labeled corpus for independent verification and/or a combined external corpus evaluation as described in the concrete test. The core contribution is sound; the evaluation methodology needs one additional step to fully validate the precision claim that forms the comparative advantage over baselines.\n\nI note but do not elevate a secondary concern: HCC at temperature 0 may provide less independent consensus than implied, since four sessions with near-deterministic decoding could follow highly correlated paths. This is worth investigating but is not by itself load-bearing — even if HCC's benefit is partly reproducibility rather than consensus, the measured precision on the primary corpus would still be 91.17%; the question is whether that number generalizes, which is already captured by the self-referential concern.","tokens_in":24904,"tokens_out":5328,"duration_ms":376082,"concrete_test":"Construct a single independent evaluation corpus with both bugs and non-bugs: (1) Run a different test generator (e.g., TitanFuzz or Fuzz4All) on the same framework versions, apply LogicHunter's deterministic filtering (Algorithm 1) to extract unique valid L3 failures, and combine with the 31 post-cutoff known bugs. (2) Have 2–3 developers unaffiliated with the paper label all cases using the six-dimensional taxonomy. (3) Evaluate the Agentic Oracle (with HCC) on this combined corpus, reporting both precision and recall. If precision falls below ~80% or recall below ~50%, the self-referential concern lands and the headline precision claim should be qualified. If precision stays above ~85% and recall above ~55%, the concern is substantially mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central precision claim (91.17%, Table 7) is measured on the Realistic Failure Corpus (§4.1.5): 1,000 failures sampled from LogicHunter's own generator output, with ground-truth labels assigned by two co-authors (Cohen's Kappa 0.77). This creates a three-way self-reference: the generator produces the failures, the deterministic pipeline (Algorithm 1) filters them, and co-authors label them. The oracle's precision/recall is then evaluated only on this internally produced distribution.\n\nThe paper provides two supplementary datasets, but neither fully addresses this. The post-cutoff known-bug set (31 bugs) measures only recall (60.65%), not precision — there are no non-bugs to test false positive rate. The hard-negative set (484 TitanFuzz failures) measures only FPR (0.04%), not recall — there are no genuine bugs. Neither set has labels from developers independent of the paper. So we have no single externally-sourced corpus where both precision and recall are measured simultaneously.\n\nA specific mechanism for inflation: LogicHunter's generator produces failures that follow certain patterns (specification-compliant inputs hitting specific API edge cases). The Agentic Oracle is designed to reason about exactly these patterns using code_search, doc_search, and run_code. If the oracle has systematic strengths and weaknesses correlated with the generator's output distribution, the 91.17% precision would not generalize to failures from other sources. The drop from 72.14% recall (primary corpus) to 60.65% recall (post-cutoff known bugs) is consistent with this — the oracle is less effective on bugs outside its generator's distribution.\n\nAdditionally, the HCC mechanism (4/4 consensus at temperature 0) may provide less independent verification than it appears: at temperature 0, the four sessions may follow highly correlated reasoning paths, meaning HCC filters stochastic noise rather than systematic bias. The ablation shows HCC improves precision from 66.67% to 90.91%, but","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents LogicHunter, a fuzzing framework for testing LLM agent frameworks (LangChain, LlamaIndex, CrewAI). The system has two phases: (1) specification-driven test generation that fuses formal type constraints with real-world usage patterns to produce valid-by-construction yet semantically extreme inputs with behavioral probes, and (2) an Agentic Oracle that uses a ReAct-based architecture with dual-layer state management and dual-stream memory to actively retrieve documentation, inspect source code, and execute reproduction scripts to diagnose whether anomalous executions constitute genuine bugs. The evaluation reports 40 previously unknown bugs (30 confirmed, 26 fixed by developers), with state-of-the-art baselines finding zero bugs. The Agentic Oracle achieves 91.17% precision on a Realistic Failure Corpus of 1,000 labeled failures, compared to 29.27% for the best passive baseline. An ablation study isolates the contributions of state management, memory pruning, tool access, and consensus mechanisms.","tokens_in":25073,"tokens_out":1801,"duration_ms":204333,"significance":"The paper addresses a genuine and important problem: the oracle problem for pure-Python agent frameworks where defects manifest as ordinary exceptions or silent semantic failures rather than crashes. The discovery of 40 real bugs with 30 developer confirmations provides strong external validation of practical utility. The ablation study (Table 8) systematically isolates component contributions, and the evaluation includes both GPT-5-mini and GPT-5.2 backbones for oracle baselines. The publicly available artifact (GitHub) and the inclusion of supplementary datasets for recall and false-positive-rate reference checks are commendable. The six-dimensional defect taxonomy and the high-confidence consensus mechanism are well-motivated design choices that address real problems in LLM-based triage.","major_comments":[{"comment":"§4.1.5, Table 7: The headline precision claim (91.17%) is measured on the Realistic Failure Corpus, which is constructed from LogicHunter's own generator output and labeled by two co-authors (Cohen's Kappa 0.77). This creates a self-referential evaluation loop: the generator, the oracle, and the ground truth are all produced by the same team. The two supplementary datasets partially address this but each tests only one dimension — the post-cutoff known-bug set (31 bugs) measures recall only (60.65%, no non-bugs to test FPR), and the hard-negative set (484 TitanFuzz failures) measures FPR only (0.04%, no genuine bugs to test recall). There is no single externally-sourced corpus where both precision and recall are measured simultaneously. This is load-bearing for the central precision claim and should be addressed by either (a) constructing a mixed external corpus with both bugs and non-bs","section":null},{"comment":"§4.1.5: The labeling was performed by 'two developers' who are co-authors of the paper. The paper should explicitly disclose this conflict of interest in the evaluation section and discuss what safeguards were applied to prevent labeling bias (e.g., whether labeling was done blind to the oracle's verdicts, whether any independent reviewer was consulted). Without this disclosure, readers cannot assess the reliability of the ground-truth labels that underpin all oracle accuracy metrics.","section":null},{"comment":"Table 7 vs Table 8: The oracle accuracy metrics differ between the two tables. Table 7 reports P=91.17%, R=72.14%, F1=80.49%, FPR=0.21% for the Agentic Oracle, while Table 8 reports P=90.91%, R=71.43%, F1=80.00%, FPR=0.21% for the 'Default Agentic Oracle.' The paper should clarify whether these are the same experiment with rounding differences, or different experimental setups (e.g., different corpus subsets). If the latter, the relationship between the two evaluations should be stated explicitly.","section":null},{"comment":"§4.5: The paper states that the Agentic Oracle's recall on the post-cutoff known-bug set (60.65%) is lower than passive LLM judges (67-94%) because 'the Agentic Oracle requires stronger evidence from source-code inspection and executable validation before reporting a bug.' This is a plausible explanation, but it also means the oracle's high precision on the primary corpus may come at the cost of missing bugs that do not fit its evidence-gathering patterns. The paper should discuss whether this precision-recall tradeoff is inherent to the design or whether it reflects the self-referential corpus bias (i.e., the oracle is tuned to find exactly the kinds of bugs its generator produces).","section":null}],"minor_comments":[{"comment":"§3.2.3, Eq. for N_mut: The formula ⌊α·C(api)/M_max⌋·M_max simplifies to ⌊α·C(api)/M_max⌋·M_max, which is always a multiple of M_max. If the intent is to allocate budgets proportional to complexity, this quantization seems coarse. Please clarify the values of α and M_max used in the experiments.","section":null},{"comment":"Table 6: The 'Cov.(%)' column header is ambiguous — it is unclear whether this is line coverage, branch coverage, or something else. §4.2 mentions 'Line coverage' but the table should state this explicitly.","section":null},{"comment":"§4.1.4: TELPA is described as evaluated 'only [for] the Pynguin-based variant, as the CodaMOSA-based variant is incompatible.' This is a reasonable exclusion, but the paper should note whether this affects the fairness of the comparison (i.e., whether the CodaMOSA variant might perform better).","section":null},{"comment":"Figure 5: The y-axis label is unclear ('Impact of k_min on oracle performance'). The figure shows multiple metrics but the legend or axis labels should explicitly state which metric each line represents.","section":null},{"comment":"§5.2: The limitations section mentions that 'nondeterminism cannot be fully eliminated despite temperature-zero decoding and HCC.' It would be useful to quantify the residual nondeterminism — e.g., what fraction of oracle verdicts change across the five repeated runs reported in Table 7?","section":null},{"comment":"Table 2: The 'Data Integrity' category description mentions 'LangChain splitters losing info' as an example — this is overly specific for a general taxonomy definition. Consider using a more abstract example.","section":null},{"comment":"§4.3: The paper states '32 root-cause clusters are reported separately in our artifact' but Table 4 reports 40 bugs. The relationship between '40 unique bugs' and '32 root-cause clusters' should be clarified in the main text, not just deferred to the artifact.","section":null},{"comment":"References [34] and [27]: Reference [34] (MirrorFuzz) is dated 2026, which appears to be a future date. Reference [27] duplicates [26] (both cite 'Lost in the Middle'). Please consolidate.","section":null},{"comment":"§4.1.2: The paper uses 'GPT-5-mini' and 'GPT-5.2' as model names. If these are pseudonyms or placeholders for actual models, this should be clarified; if they are real model names, the access dates and API versions should be specified for reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The self-referential evaluation concern is the primary issue. The paper's practical contributions (40 real bugs, 30 confirmed) are strong and independently validated by framework developers, so the core practical contribution is sound. However, the headline precision claim (91.17%) rests on an evaluation corpus that is generated, filtered, and labeled by the same team, and the supplementary datasets are each one-sided. I believe this can be addressed within the manuscript's scope — at minimum through transparent disclosure, discussion of generalizability threats, and ideally through a small mixed external evaluation. The paper is a good fit for the journal's scope (automated testing for emerging software infrastructure). The citation pattern appears normal; the co-author overlap with reference [46] (Xue et al., ASE 2025) is disclosed through author affiliation and is relevant prior work on agent framework bugs."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The comments identify legitimate concerns about evaluation methodology that we will address in revision. Below we respond point by point.","responses":[{"response":"We agree this is a genuine methodological gap. The Realistic Failure Corpus is indeed generated by LogicHunter's own pipeline and labeled by co-authors, and the two supplementary datasets each evaluate only one axis (recall or FPR) in isolation. No single external corpus simultaneously tests both precision and recall. We will address this in two ways. First, we will construct a mixed external corpus by combining the 31 post-cutoff known-bug set with a sample of the 484 hard-negative TitanFuzz failures, creating a single externally-sourced dataset containing both genuine bugs and non-bugs. We will report precision, recall, F1, and FPR on this mixed corpus for the Agentic Oracle and all baselines. Second, we will add an explicit discussion of the self-referential limitation in §4.1.5, acknowledging that the primary corpus reflects LogicHunter's generator distribution and that the mixed external corpus provides a complementary, independent evaluation. We note that the 30 developer-confirmed bugs (§4.3) already provide strong external validation of practical utility, but the referee is correct that the oracle accuracy metrics themselves need an independent mixed corpus. We will add this corpus and report results in the revised manuscript.","revision_made":"yes","referee_comment":"§4.1.5, Table 7: The headline precision claim (91.17%) is measured on the Realistic Failure Corpus, which is constructed from LogicHunter's own generator output and labeled by two co-authors. This creates a self-referential evaluation loop. The two supplementary datasets each test only one dimension. There is no single externally-sourced corpus where both precision and recall are measured simultaneously."},{"response":"We agree. The two labelers are indeed co-authors (the first two authors). We will revise §4.1.5 to explicitly disclose this and describe the safeguards we applied: (1) labeling was performed independently before any comparison or discussion, (2) labelers were given the test code, failure traceback, and official documentation but were not shown the Agentic Oracle's verdict during labeling, (3) disagreements were resolved through consensus discussion with the third author as arbiter when needed, and (4) the Cohen's Kappa of 0.77 indicates substantial inter-rater agreement despite independent labeling. We will also acknowledge that independent external labelers would further strengthen the evaluation and note this as a limitation. Additionally, as part of our response to the first comment, the mixed external corpus will include bugs with developer-confirmed ground truth (from merged PRs), removing the need for author labeling on that subset.","revision_made":"yes","referee_comment":"§4.1.5: The labeling was performed by 'two developers' who are co-authors. The paper should disclose this conflict of interest and discuss safeguards against labeling bias."},{"response":"We thank the referee for catching this. The two tables report the same configuration but under different experimental conditions. Table 7 reports the mean ± standard deviation across five independent runs on the full Realistic Failure Corpus (1,000 cases). Table 8 reports results from a single representative run used as the ablation baseline, conducted on the same corpus but without repeated trials. The small numerical differences (e.g., 91.17% vs 90.91% precision) fall within the reported standard deviation (±4.01%). We acknowledge this is insufficiently explained in the current manuscript. We will revise Table 8 to either report mean ± std across five runs for consistency with Table 7, or add a clear note stating that Table 8 uses a single representative run and that the values are within the variance reported in Table 7. We will also ensure the column headers and captions make the relationship explicit.","revision_made":"yes","referee_comment":"Table 7 vs Table 8: The oracle accuracy metrics differ between the two tables. Table 7 reports P=91.17%, R=72.14%, F1=80.49%, FPR=0.21% for the Agentic Oracle, while Table 8 reports P=90.91%, R=71.43%, F1=80.00%, FPR=0.21% for the 'Default Agentic Oracle.' Clarify whether these are the same experiment or different setups."},{"response":"The referee raises an important point that we should address more thoroughly. We believe both factors contribute. First, the precision-recall tradeoff is partly inherent to the design: the Agentic Oracle requires source-code evidence and executable validation before reporting a bug, which makes it more conservative than passive judges that can classify based on surface-level patterns. Some real-world bugs in the post-cutoff set require complex system construction (e.g., multi-step agent workflows, external service mocking) that the oracle's evidence-gathering tools cannot fully reproduce, leading to missed bugs. Second, we cannot rule out that self-referential corpus bias contributes: the Realistic Failure Corpus reflects LogicHunter's generator distribution, and the oracle's investigation patterns may be better calibrated for the types of failures the generator produces. The fact that recall drops from 72.14% on the primary corpus to 60.65% on the external known-bug set is consistent with this interpretation. We will add a dedicated paragraph in §4.5 discussing both factors, and we will explicitly state that the precision-recall tradeoff on external bugs is an open question that the mixed external corpus (from our response to Comment 1) will help illuminate. We will also note that the 60.65% recall on externally-sourced real-world bugs, while lower than passive judges, still represents meaningful bug-finding capability, and that the Agentic Oracle's dramatically lower FPR (0.04% vs 1.03-2.07% for GPT-5.2 baselines on the hard-negative set) makes it more suitable for automated deployment where false-positive cost dominates.","revision_made":"yes","referee_comment":"§4.5: The Agentic Oracle's recall on the post-cutoff known-bug set (60.65%) is lower than passive LLM judges (67-94%). The paper should discuss whether this precision-recall tradeoff is inherent to the design or reflects self-referential corpus bias."}],"tokens_in":24807,"tokens_out":2154,"duration_ms":133685,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things matter here. First, LogicHunter found 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI — 30 confirmed by developers, 26 fixed. That is real. Second, the Agentic Oracle (a ReAct-based LLM agent that inspects source code, retrieves docs, and runs code before judging) hits 91.17% precision on triaging failures, versus 29% for the best passive LLM judge. The gap is large and the ablation study isolates the components responsible for it fairly convincingly — HCC, dual-stream memory, and the tool suite each contribute measurably (Table 8). The architecture — FSM outer loop wrapping a ReAct inner loop, with pruned observation memory — is a genuine design contribution, not a repackaging of existing ideas. The case study in §5 is concrete and the bug is non-trivial. The artifact is public on GitHub, which matters for a system paper like this. Credit where earned: the engineering is serious and the bug-finding results are externally validated by the framework developers themselves. Now the soft spot, and it is a real one. The 91.17% precision is measured on a corpus of 1,000 failures sampled from LogicHunter's own generator output, labeled by two co-authors (Cohen's Kappa 0.77). The generator, the oracle, and the ground truth are all from the same team. The paper provides two supplementary datasets — a post-cutoff known-bug set (31 bugs, recall 60.65%) and a hard-negative set (484 TitanFuzz failures, FPR 0.04%) — but neither measures both precision and recall simultaneously on externally sourced data. So the headline precision claim is under-validated on independently sourced failures. The recall drop from 72% (primary corpus) to 60.65% (external bugs) is consistent with the oracle being tuned to its own generator's failure distribution. The HCC mechanism (4/4 consensus at temperature 0) may also provide less independent verification than it appears — at temperature 0, the four sessions likely follow correlated reasoning paths, filtering stochastic noise but not systematic bias. The stress-test concern about this is legitimate. That said, the circularity does not invalidate the core contribution. The 30 developer-confirmed bugs are independent of the oracle evaluation entirely. The zero-bug baseline result is partly an artifact of baseline design (crash-only oracles), as the reader notes, but the cross-evaluation of baseline-generated L3 failures (506 cases, 1 bug found) does show the generator matters too, not just the oracle. The unspecified hyperparameters (alpha, M_max, k_min, k_max, HCC threshold) are a minor concern — the ablation on k_min is thorough and the others are standard knobs. This paper is for software engineering researchers and practitioners working on LLM infrastructure testing. A reader gets value from the architecture design, the bug taxonomy, and the empirical evidence that active oracles substantially outperform passive ones for semantic bug triage. It deserves a serious referee. The referee should push hard on the evaluation methodology: request release of the labeled corpus, ask for oracle evaluation on a single externally sourced corpus where both precision and recall are measured, and ask for clarification on HCC independence at temperature 0. The core argument holds up — the bugs are real and the architecture is novel — but the precision claim needs external validation before it should be trusted at face value.","headline":"LogicHunter finds 40 real bugs in LLM agent frameworks using an agentic oracle; the headline precision number is measured on a self-referential corpus but the bugs are independently confirmed.","tokens_in":26043,"tokens_out":798,"would_cite":true,"duration_ms":126626,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Active AI oracle finds 40 real bugs in LangChain, LlamaIndex, CrewAI","keywords":["automated testing","test oracle problem","LLM agent frameworks","fuzzing","agentic oracle","LangChain","LlamaIndex","CrewAI"],"falsifier":"Run the Agentic Oracle on a failure corpus generated by an independent test generator (not LogicHunter) with ground-truth labels assigned by developers unaffiliated with the paper. If precision drops substantially below 91%, the self-referential evaluation loop is inflating the metric.","tokens_in":24847,"feed_emoji":"🔍","tokens_out":1424,"duration_ms":67882,"temperature":0.7,"pith_summary":"This paper argues that the central obstacle in testing LLM agent frameworks is not generating test inputs but deciding whether a failing test reflects a genuine framework defect — the test oracle problem. In pure-Python frameworks like LangChain, LlamaIndex, and CrewAI, bugs rarely crash; they surface as ordinary exceptions (ValueError, KeyError) indistinguishable from API misuse, or as silent semantic failures where execution succeeds but behavior is wrong. The paper introduces LogicHunter, a two-phase system that pairs specification-driven test generation with an Agentic Oracle: an LLM-based agent that, rather than passively classifying a failure as bug-or-not, actively investigates by retrieving documentation, navigating source code via reflection, and executing reproduction scripts in a sandbox. The Agentic Oracle uses a structured reasoning loop (ReAct inside a finite state machine with three phases — initialization, investigation, verdict), pruned memory to avoid context saturation, and a high-confidence consensus mechanism requiring four independent sessions to agree before reporting a bug. The paper claims this design achieves 91.17% precision — meaning roughly 1 in 1.1 flagged failures is a real bug — compared to 29.27% for the best passive LLM judge, and that the system discovered 40 previously unknown bugs across the three frameworks (30 confirmed by developers, 26 fixed), while state-of-the-art baselines found zero.","feed_headline":"Active AI oracle finds 40 real bugs in LangChain, LlamaIndex, CrewAI","feed_subtitle":"An LLM agent that investigates test failures by reading docs and source code achieves 91% precision where passive judges stall at 29% — and基","key_machinery":"Agentic Oracle: a ReAct-based LLM agent with three-phase state management (Initialization, Investigation, Verdict), dual-stream memory (persistent reasoning trace + pruned observation window), four introspection tools (code_search, doc_search, run_code, get_test_info), a six-dimensional defect taxonomy, and high-confidence consensus requiring four independent sessions to agree before flagging a bug. On the generation side: specification-driven seed synthesis fusing Pydantic schemas and type hints with mined real-world usage patterns, a Fixer Agent for seed repair, and a Mutator Agent producing valid-but-extreme inputs with behavioral probes (assertions encoding expected semantic properties).","core_discovery":"The paper's central claim is that replacing a passive test oracle (one that classifies a failure from a fixed snapshot of information) with an active agentic oracle (one that dynamically retrieves documentation, inspects source code, and runs reproduction code to gather evidence before rendering a verdict) transforms the precision of automated bug detection in LLM agent frameworks from impractical (~29%) to deployment-grade (~91%). The mechanism carrying this claim is the Agentic Oracle itself: a ReAct-based agent with dual-layer state management (an inner reasoning-acting loop inside an outer three-state finite state machine), dual-stream memory (persistent reasoning traces plus pruned tool","pith_inferences":[],"forward_implications":["If the Agentic Oracle approach generalizes, the bottleneck in automated testing of complex Python libraries shifts from test generation to oracle design — and active, tool-using LLM agents become a standard component of fuzzing pipelines rather than a research curiosity.","The six-dimensional defect taxonomy and high-confidence consensus mechanism suggest a template for deploying LLM-based oracles in other domains where crash-based oracles fail (e.g., data processing libraries, configuration systems, DSL interpreters).","The finding that passive LLM judges plateau at ~29% precision even with strong models (GPT-5.2) implies that simply scaling model capability does not solve the oracle problem — structured investigation and evidence-gathering architecture matter more than raw model power.","The 32 silent-failure bugs (out of 40 total) indicate that the most dangerous defects in agent frameworks are invisible to crash-based testing, which has implications for how production AI systems should be validated before deployment."],"fun_headline_variants":["Agentic oracle lifts bug-finding precision from 29% to 91% on LLM frameworks","LLM judge that reads docs and source finds 40 bugs where passive oracles find none","ReAct-based oracle hits 91% precision discovering 40 bugs in agent frameworks","Active oracle beats passive classifiers by 61 points in LLM framework fuzzing","LogicHunter: agentic oracle discovers 40 bugs in LangChain, LlamaIndex, CrewAI"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The oracle's precision and recall are measured on a corpus of 1,000 failures generated by LogicHunter's own test generator, with ground-truth labels assigned by two developers who are co-authors of the paper. This means the oracle is evaluated only on the distribution of failures that LogicHunter's generator produces — if the generator systematically misses certain bug classes, the oracle's measured accuracy does not reflect performance on the true bug distribution of thes","fun_headline_variants_meta":{"raw":{"variants":["Agentic oracle lifts bug-finding precision from 29% to 91% on LLM frameworks","LLM judge that reads docs and source finds 40 bugs where passive oracles find none","ReAct-based oracle hits 91% precision discovering 40 bugs in agent frameworks","Active oracle beats passive classifiers by 61 points in LLM framework fuzzing","LogicHunter: agentic oracle discovers 40 bugs in LangChain, LlamaIndex, CrewAI","Agent that investigates its own test failures hits 91% precision on framework bugs","Spec-driven fuzzer with agentic oracle finds 40 bugs, baselines find zero","Dual-layer ReAct oracle turns 29% precision into 91% on agent framework defects","Agentic oracle that retrieves docs and inspects source catches 40 silent framework bugs","Passive oracles stall at 29% precision; agentic oracle reaches 91% and finds 40 bugs"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1147,"prompt_tokens":606,"completion_tokens":541,"prompt_tokens_details":null},"tokens_in":606,"tokens_out":541,"duration_ms":39636,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T13:34:13.004675+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run the Agentic Oracle on a failure corpus generated by an independent test generator (not LogicHunter) with ground-truth labels assigned by developers unaffiliated with the paper. If precision drops substantially below 91%, the self-referential evaluation loop is inflating the metric.","supporting_citations":[],"review_version":1}