{"id":"8ec95dba-e23c-47a2-b617-73f950d4125e","arxiv_id":"2608.07926","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A feedback loop that injects debugger-observed runtime values into LLM prompts improves directed test input generation, reaching target lines in three of four real-world case studies.","lead":"This paper introduces ReDig, a tool that repeatedly asks an AI to generate test inputs, then uses a debugger to pull runtime values from failed runs and feeds those values back to the AI to try again. The authors show three of four case studies on real C/C++ libraries reached the target code lines with this feedback loop, suggesting AI-generated tests can be improved by grounding them in observed behavior.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim that runtime value feedback causes successful directed generation is not yet supported: without a no-feedback baseline or ablation, the observed successes could be due to repeated trials, seed preservation, and restarts.","rationale":"I read the paper as a WIP feasibility study. The framework is clearly described and the motivating example (Section 2.2) shows a plausible mechanism: runtime values such as len=0 and cPrev=0 led the LLM to revise inputs. The case trajectories are consistent with feedback being useful. However, the central claim is causal: runtime value feedback refines directed generation. That requires isolating feedback from the controller's other exploratory machinery. Section 2.4 gives the controller three mechanisms—N repeated attempts, seed preservation/reuse, and restarts—each of which changes the distribution of generated inputs and provides additional opportunities. Because these are always active when feedback is active, the successes in Cases 1–3 do not establish that the gdb observations, rather than extra chances or seed reuse, caused the target hits. The strongest counter-reading is Case 1: success occurred after a restart and with a fresh seed; the first feedback cycle also produced a new-line-covering seed without hitting the target. Section 3.1's sentence 'This result supports the effectiveness of the feedback-guided refinement' is therefore an overstatement for the current evidence. The paper itself defers systematic evaluation to future work (Section 4.2), which supports a conditional rather than reject verdict. I agree with the reader's weakest assumption. The proposed feedback-disabled control is the minimal experiment that would settle attribution; if resource-constrained, even two subjects with repeated runs would help.","tokens_in":8939,"tokens_out":3409,"duration_ms":38252,"concrete_test":"Implement a 'feedback-disabled' control by reusing the ReDig controller and prompts from Section 2.3, but omitting the Feedback Generator and gdb step: each iteration simply performs N=5 LLM generations seeded by the latest seed, updates the seed on new coverage, and restarts on stall, exactly as in Section 2.4. Run this control on the same four subjects with the same LLM (GPT-5.4, temperature 0), same call-path context, and same token/time budget, repeating each condition at least 10 times. If the feedback-disabled control reaches the target in 3 or more subjects, or requires comparable tests/turns, then runtime-value feedback is not demonstrated to be the effective ingredient; if it never or rarely succeeds where ReDig succeeds, the Section 3.1 causal claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states 'This result supports the effectiveness of the feedback-guided refinement' based on four single-run case studies. The controller (Section 2.4) already includes several mechanisms that can improve generation independently of the gdb-derived values: it dispatches N=5 attempts per iteration, preserves any input that covers a previously uncovered line as a seed, instructs the LLM to reuse seed elements, and restarts without a seed when refinement stalls. Case 1's success after restart is especially ambiguous: the eventual target-hitting input was produced in the first refinement of the second seed, so the gdb feedback from that fresh seed may have mattered, but the previous cycle shows that even without target hits, seeding and restart alone produced a new design pattern. No experiment separates the feedback channel from these built-in retries and seed/restart dynamics. Further, each subject was run once, so the 3/4 success rate has no variance estimate. The paper explicitly lists comprehensive evaluation as future work, and the WIP framing is honest, but the causal attribution in Section 3.1 currently outruns the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReDig is a Work-in-Progress framework that wraps an LLM-based directed test-input generator in a feedback control loop. When generated inputs miss a designated target line, a Feedback Generator asks the LLM to propose gdb breakpoint/expression queries, the GDB Driver executes the target-missing input and returns the observed runtime values, and the Directed Input Generator uses these values together with the current seed input to generate refined inputs. A controller runs N=5 generation attempts per round, promotes any input that covers a previously uncovered line to seed, and restarts without a seed when refinement stalls. Four case studies on Poppler and Libsndfile are reported: target lines were reached in three cases and not in one. The paper claims that LLMs can effectively use runtime value feedback to diagnose why previous test scripts failed to reach target lines and to refine subsequent test scripts.","tokens_in":9149,"tokens_out":4703,"duration_ms":50648,"significance":"The proposed mechanism is well motivated and has genuine strengths: the feedback channel is grounded in actual executions via gdb, the LLM is used only to propose queries and interpret values, and the paper reports concrete implementation and cost data. The qualitative trajectories, especially in Cases 2 and 3, are suggestive and provide a useful example of how runtime values can help an LLM revise an incorrect hypothesis. However, the central causal claim, that the observed successes are due to the runtime value feedback, is not yet supported by the reported evaluation. The contribution is a promising framework with preliminary evidence, not a validated technique.","major_comments":[{"comment":"The sentence \"This result supports the effectiveness of the feedback-guided refinement\" is not warranted by the evidence presented. The four case studies are single runs, and there is no baseline that runs the same controller with the feedback channel disabled. The controller's successes could be produced by the N=5 repeated attempts, by seed promotion based on new coverage, or by the restart mechanism, each of which is active independently of the gdb-derived values (Section 2.4). A controlled comparison or an ablation, such as replacing runtime values with a fixed or synthetic feedback message while keeping the same retry, seed, and restart policy, is needed to attribute the outcome to runtime value feedback.","section":"§3.1"},{"comment":"The \"Turns\" column in Table 1 is internally inconsistent with the text. Section 2.2 describes Case 2 as spanning five alternating turns before reaching the target, and Section 3.2 describes Case 1 as taking three turns in total, yet Table 1 reports 2 turns for both cases. Since the table is the quantitative summary of the case studies, this discrepancy must be resolved, for example by clarifying what a turn counts and recomputing the column.","section":"Table 1 / §2.2"},{"comment":"Each subject was run only once, despite substantial nondeterminism in the loop, including random tie-breaking in seed selection and possible LLM nondeterminism. The reported 3/4 success rate therefore has no variance estimate and no repeated-run stability check. At minimum, the paper should report repeated runs per subject with a fixed budget, or explicitly limit the claim to \"one observed trajectory\" and remove causal language such as \"effectiveness\" from Section 3.1.","section":"§3.1"}],"minor_comments":[{"comment":"The phrase \"we found that ReDig effectively derive runtime value feedback\" should be \"ReDig effectively derives\" or \"ReDig effectively uses\".","section":"Abstract"},{"comment":"\"Based to the LLM query result\" should be \"Based on the LLM query result\".","section":"§2.4"},{"comment":"The code snippet contains \"def ogg_crc ( page : bytes } -> int\" with a mismatched brace; the diff markers are also rendered inconsistently and should be formatted as a proper diff or clarified.","section":"Figure 4"},{"comment":"The sentence listing runtime queries has awkward punctuation, especially \"the number of packets remaining in the buffer ,odata->pkt_len - odata->pkt_indx\"; a formatted list would improve readability.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honestly labeled as work in progress, and the authors explicitly list comprehensive evaluation as future work in Section 4.2. This is a strength in transparency, but it also confirms that the current evidence is preliminary. The main path to acceptance is to add a no-feedback baseline or ablation and to report repeated runs; without that, the central causal claim in Section 3.1 outruns the data. The inconsistency in Table 1 should also be fixed. I see no other concerns about novelty or fit with the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReDig is a plausible work-in-progress with a genuinely new assembly: the LLM proposes gdb breakpoints and expressions, the framework extracts the runtime values, and those observations are fed back to the directed input generator along with a seed-and-restart controller. I don't know of prior work that selects which runtime values to observe this way, and the case-study trajectories in Sections 2.2 and 3.3 are genuinely instructive. The implementation details are concrete (4343 LOC, gdb/Python scripts, cost per run), and the WIP framing in Section 4.2 is honest: the authors explicitly list systematic evaluation as future work.\n\nThe soft spot is the one the authors half-acknowledge. Section 3.1 says 'This result supports the effectiveness of the feedback-guided refinement' based on four single-run case studies, three successes. That sentence outruns the evidence. The controller already has several mechanisms that can improve generation without any runtime value feedback: it runs N=5 attempts, keeps any input that covers a new line as a seed, tells the LLM to reuse seed elements, and restarts when refinement stalls. No experiment isolates the feedback channel. Case 1 is especially ambiguous because success came after a restart; Case 4 is an honest failure but the 'diverse path exploration' reading is speculation. There is no baseline that runs the same controller with the feedback loop disabled, no ablation of the query-selection step, and no repeated runs, so the 3/4 success rate carries no variance estimate. The paper would be stronger if the authors presented the case studies as feasibility evidence and dropped the causal attribution until the controlled evaluation is done.\n\nThe citation pattern looks fine. TestWeaver, CoverUp, and Panta are the right neighbors, and the paper makes a credible distinction between coverage/error feedback and value-level feedback. There is a minor self-citation in the future-work paragraph (BugOss), but it is not load-bearing. No code or data is shipped, so reimplementation is possible but not immediate.\n\nWho this is for: people working on LLM-based test generation, especially directed input generation and proof-of-vulnerability construction. They will want to know about ReDig and will probably try the value-feedback idea themselves. I would bring it to a reading group. I would not cite it as evidence for effectiveness until a controlled study appears.\n\nRecommendation: send to peer review as a WIP/vision-style paper, but the referee should insist on a no-feedback baseline and repeated runs before any effectiveness claim is published.","headline":"A new and honest WIP: the gdb-value feedback loop is worth trying, but the effectiveness claim needs a controlled baseline.","tokens_in":9634,"tokens_out":2629,"would_cite":false,"duration_ms":29541,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReDig, a feedback loop that extracts runtime values from failed test executions with gdb and feeds them back to the LLM, reaches three of four target lines in Poppler and Libsndfile case studies.","keywords":["LLM-based test generation","directed test input generation","runtime value feedback","debugger-guided refinement","coverage-guided testing","feedback loop","Poppler","Libsndfile"],"falsifier":"Run ReDig with the same controller and prompts on the same four subjects, but replace the observed runtime values in each feedback prompt with the literal string 'value unavailable' (or with values taken from a different execution). If the target lines are still reached in the same number of turns, or if a broader comparison shows no difference in reachability between the valued and valueless feedback conditions, the central claim that runtime value feedback guides the generation is refuted.","tokens_in":8752,"feed_emoji":"🎯","tokens_out":10611,"duration_ms":105355,"temperature":0.7,"pith_summary":"LLM-based directed test input generation works by asking an LLM to synthesize a script that produces an input meant to reach a designated target line. Its weakness is that the LLM may hold plausible-but-wrong assumptions about the program's runtime state. ReDig closes a control loop around this process: when the generated input misses the target, a Feedback Generator prompts the LLM to propose runtime-value queries, executes those queries with gdb on the failed input, and returns the observed values to the generator with the current seed input. The paper claims this runtime-value feedback lets the LLM correct its hypothesis about the program state, and presents four case studies on Poppler and Libsndfile in which three target lines were reached after refinement and one was not. The work is presented as preliminary, with the central question being whether observed values, rather than merely more attempts, are what makes the loop effective.","feed_headline":"Runtime value feedback guides LLM test inputs to three of four targets","feed_subtitle":"Observed runtime values from failed runs are fed back into the prompt, correcting the LLM's assumptions.","key_machinery":"The Feedback Generator is the central mechanism. Given a failed test input and its line coverage, it asks the LLM to act as a test engineer and list up to five runtime value queries, each a gdb breakpoint location plus an expression. A GDB Driver runs the target binary with the failed input, extracts the observed values for each query (multiple values if the breakpoint is hit several times, an empty list if the line or expression is invalid), and packages these observations into a feedback prompt. The Directed Input Generator then receives the feedback together with the seed input and its coverage, and is instructed to generate a new input by identifying and fixing the misaligned elements of the seed. This loop converts abstract speculation about the target path condition into specific observed numbers that the LLM can reason over.","core_discovery":"The paper's central claim is that runtime value feedback can guide an LLM-based directed input generator to reach target lines that it cannot reach from code context alone. In a reported case study, a generated Ogg file failed to reach the target because the packet buffer was not exhausted; the gdb queries showed exactly that (pkt_indx == pkt_len was false, one packet remaining), and the next generated input used two pages so the second invocation hit the target. The framework's Feedback Generator lets the LLM decide which values to inspect, packaging each query as a breakpoint location plus an expression, and the GDB Driver returns all observed values. The controller uses a seed input that covers new lines and restarts when refinement stalls, so the loop mixes local refinement with global exploration. Three of four case-study target lines were reached in this way, and the paper presents this as feasibility evidence for the refinement loop.","pith_inferences":["The paper's causal claim that the runtime values drive the successes is not yet isolated; a natural next experiment is to run the same controller with the observed values redacted from the feedback, to separate the effect of the values from the effect of extra attempts, seed updates, and restarts.","The reliance on the LLM to propose queries means the loop inherits the LLM's blind spots: if the deciding state lives in a global variable, heap object, or environment setting that the generator never thinks to query, the feedback will be uninformative and the loop will fail even in principle. Inspecting which queries were made in unsuccessful rounds could expose this.","The query-list length (at most five) is a fixed choice; an adaptive budget that raises or lowers the number of queries based on the observed diversity of branch coverage could improve both cost and coverage in harder cases.","The pattern is not limited to test inputs: any LLM artifact synthesis with a runtime oracle (e.g., program repair, config generation, protocol fuzzing) could use the same 'propose a targeted runtime observation, then feed the observed value back' loop."],"forward_implications":["Directed test input generation becomes applicable to target lines whose path conditions depend on runtime state, not just on static code structure, because the loop supplies state observations that static context omits.","The framework is a thin control layer and can wrap any LLM-based directed generator; improvements in the underlying generator should carry through without changing the feedback mechanism.","Because the feedback selects a few values (3–13 per case) rather than dumping a trace, the cost of grounding is low relative to the generation budget and may scale to larger programs.","The seed-update and restart schedule gives the loop a built-in exploration/exploitation trade-off; one case suggests that restarting after a failed refinement can introduce a different seed design that then succeeds.","If the loop's effectiveness is confirmed, the same query-and-feed-back pattern can be reused for reaching code changes, reproducing failure symptoms, or validating patches."],"supporting_citations":[{"why":"Establishes the baseline technique of LLM-based directed test input generation that the framework wraps.","marker":"[7]"},{"why":"Supplies the scientific-debugging prompting pattern that the feedback generator adapts into runtime-value queries.","marker":"[9]"},{"why":"Demonstrates that feeding runtime values from earlier executions into the prompt helps LLM test generation; this work extends that by letting the LLM choose which values to inspect.","marker":"[13]"},{"why":"Provides the open-source C/C++ reference versions (a PDF library and an audio library) used as the four case-study subjects.","marker":"[6]"},{"why":"Motivates the use of execution feedback and program context to iteratively refine LLM-generated inputs toward vulnerable code.","marker":"[3]"}],"fun_headline_variants":["Runtime feedback loops LLM test inputs to hit more targets","LLM test generation learns from runtime values to reach targets","Three of four targets reached by feedback-guided LLM inputs","Failed run values refine LLM prompts to reach target lines","Grounding LLM test generation in runtime behavior"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The successes are attributed to the runtime value feedback, but the controller also repeats generation five times per round, updates the seed whenever any new line is covered, and restarts after failed rounds, so the specific causal contribution of the gdb-observed values — as opposed to the extra attempts and restarts — is not established.","fun_headline_variants_meta":{"raw":{"variants":["Runtime feedback loops LLM test inputs to hit more targets","LLM test generation learns from runtime values to reach targets","Three of four targets reached by feedback-guided LLM inputs","Failed run values refine LLM prompts to reach target lines","Grounding LLM test generation in runtime behavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1362,"prompt_tokens":846,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":462,"tokens_out":516,"duration_ms":5915,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:39:24.049170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReDig with the same controller and prompts on the same four subjects, but replace the observed runtime values in each feedback prompt with the literal string 'value unavailable' (or with values taken from a different execution). If the target lines are still reached in the same number of turns, or if a broader comparison shows no difference in reachability between the valued and valueless feedback conditions, the central claim that runtime value feedback guides the generation is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the baseline technique of LLM-based directed test input generation that the framework wraps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the scientific-debugging prompting pattern that the feedback generator adapts into runtime-value queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the open-source C/C++ reference versions (a PDF library and an audio library) used as the four case-study subjects."}],"review_version":1}