{"id":"51620558-59c7-4fe0-96a4-4df07b5e86fe","arxiv_id":"2412.08281","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Lachesis predicts whether AutoFL's self-consistency vote will rank the true buggy method first, using LSTM and GCN models over reasoning path matrices and graphs.","lead":"This paper trains machine learning models to predict, from the structure of an LLM's reasoning steps, whether the final self-consistency answer will be correct, using the AutoFL fault localization tool as the test bed. It reports up to 81% precision, which could let developers stop expensive repeated LLM sampling early when success looks unlikely.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set peeking inflates the headline numbers: Section III-A4 reports metrics from the epoch with highest test accuracy and no separate validation, so Table II may overstate Lachesis's predictive accuracy.","rationale":"The paper's motivating application is early termination, but the strongest claim as stated is about prediction from complete reasoning paths. For that claim to be supported, Table II must be an unbiased estimate of generalization. The evaluation protocol violates this: choosing the epoch with the highest test accuracy turns the test fold into a validation set. With 10 folds of about 45 samples, the maximum over 50/100 epochs is expected to be optimistic; the magnitude is unknown but could easily account for part of the gap between Lachesis and the majority baseline. The reader's weakest assumption focused on partial-path early termination; that is a real gap for the application, but it is not the load-bearing defect for the central predictive claim. The reader's rationale did note the epoch-selection problem, so my concern is partial agreement. I would keep the CONDITIONAL verdict: the idea is plausible and the S/F/F+A results are independent of the answer token, but the paper needs a corrected evaluation and significance reporting before the headline numbers can be accepted.","tokens_in":6854,"tokens_out":8681,"duration_ms":97413,"concrete_test":"Re-run the 10-fold CV with a nested validation split: hold out 20% of each training fold, stratified on the label, to select the epoch (and, if feasible, the hyperparameters); then evaluate the chosen model on the held-out test fold. Using the same folds, compare the mean ROC-AUC, precision, and recall against Table II. If the corrected mean AUC for GCN-F or GCN-S falls below roughly 0.7, or if any configuration's precision is no longer distinguishable from the 0.6732 majority baseline, the reported predictive advantage is largely an artifact of test-set peeking.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that Lachesis can predict whether self-consistency will rank the buggy method first—rests on the accuracy of Table II. Section III-A4 states that models are trained for 50/100 epochs and 'we report test accuracy ... from the epoch with the highest test accuracy.' Within 10-fold cross-validation, this means the test fold itself is used to select the reporting epoch. With only ~45 bugs per fold, the maximum over 50–100 noisy epochs can be substantially optimistic; the same applies to the ROC-AUC, precision, and recall reported from that epoch. The hyperparameters in Table I are also 'manually fine-tuned' without a nested or held-out selection procedure, so model selection has access to the full dataset. Consequently, even the clean configurations (GCN-S, GCN-F) may not achieve their reported AUCs on truly unseen bugs. The F+A+A result is further suspect because the answer vector is part of the input, making the self-consistency signal partly tautological; however, the S/F/F+A configurations provide independent evidence, so the central claim could survive a corrected evaluation. The problem is that the current paper does not provide an unbiased estimate of how well Lachesis generalizes, and the 'up to' wording cherry-picks the best configuration for each metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Lachesis, a classifier that predicts whether self-consistency voting in AutoFL will rank the true buggy method first, using structural representations of LLM reasoning paths. It proposes two representations: an LLM Inference Matrix (LIM) processed by an LSTM and an LLM Inference Graph (LIG) processed by a GCN. Four input variants are studied: shape only (S), function type only (F), function type plus arguments (F+A), and function type plus arguments plus the final answer (F+A+A). The models are evaluated on 456 bugs from BugsInPy and Defects4J using 10-fold cross-validation, with accuracy, ROC-AUC, precision, and recall reported. The best configuration (GCN with F+A+A) achieves precision 0.8136, while the AutoFL-confidence baseline achieves ROC-AUC 0.8193. The paper motivates early termination of LLM inferences as the intended use case.","tokens_in":7075,"tokens_out":5898,"duration_ms":58924,"significance":"If the results are validated, the paper would provide evidence that reasoning path structure contains reusable signal for forecasting self-consistency outcomes, which is a novel and potentially useful result for cost reduction in LLM-based agents. The paper contributes a new dataset of AutoFL reasoning traces, two structural representations, and an openly available implementation. The clean structural configurations (S, F, F+A) are self-contained in that they do not see the final answer, and the GCN-S result (AUC 0.7791) suggests that path shape alone carries signal. However, the current evaluation protocol is likely to overstate performance, and the headline F+A+A configuration includes the final answer as an input, weakening the claim that the prediction is based on structural properties. The early-termination motivation is not directly tested.","major_comments":[{"comment":"The evaluation protocol reports metrics from the epoch with the highest test accuracy within each fold of the 10-fold cross-validation. Because the test fold itself is used to select the reporting epoch, the accuracy, ROC-AUC, precision, and recall in Table II are optimistically biased; with roughly 45 bugs per fold and 50–100 epochs, the maximum over noisy epochs can be substantial. The manual hyperparameter tuning described in the same section also lacks a nested or held-out selection procedure. Please use an independent validation set (or nested cross-validation) for early stopping and hyperparameter selection, and then report test metrics from the selected epoch; without this correction, the reported numbers cannot be interpreted as unbiased estimates of generalization.","section":"Section III-A4, Table II"},{"comment":"The F+A+A representation includes the final answer of AutoFL as an input feature, while the label is whether that answer is correct. The headline result (GCN with F+A+A, precision 0.8136) therefore depends not only on the structure of reasoning paths but also on the identity of the voted answer, which is inconsistent with the paper's claim of predicting correctness from structural properties. Please make the structural configurations (S, F, F+A) the primary results, and report F+A+A as an ablation or upper bound that includes answer content. To quantify the contribution of answer identity, also report a model that receives only the answer vector.","section":"Section II-C4, Section II-D, Table II"},{"comment":"The paper states that configurations such as F and F+A 'can perform well by leveraging limited information' and that 'predictions based on partial data may be possible,' using this to motivate early termination. However, all experiments are conducted on complete reasoning paths with R=10 and N=10; no experiment measures prediction performance from prefixes of the reasoning steps or from subsets of the R runs. Since the early-termination use case is a central motivation in Sections I and III-B, add experiments with partial traces (e.g., the first k steps, or k of R paths) or explicitly restrict the claims to future work.","section":"Section III-B, final paragraph"},{"comment":"The comparison with AutoFL-confidence is presented as showing comparable performance, but AutoFL-confidence achieves a higher ROC-AUC (0.8193) than every Lachesis configuration (maximum 0.7866), and the fixed threshold of 0.5 used for AutoFL-confidence is acknowledged to be not specifically designed for binary classification. The claim of competitiveness is not supported by significance testing or confidence intervals for the metric differences. Please add statistical tests (e.g., paired tests across folds or bootstrap confidence intervals) to support any comparative claims.","section":"Section III-B, Table II, Fig. 4"}],"minor_comments":[{"comment":"The text says 'Ans example is given in Fig. 2'; this should be 'An example is given in Fig. 2'.","section":"Section II-B2"},{"comment":"The sentence 'This results in a dataset of 456 bugs' would benefit from an explicit statement of the filtering criteria relative to the original 798 bugs, since only single-buggy-method versions and no Closure bugs are retained.","section":"Section III-A1"},{"comment":"The description 'The vector is then appended to the path of the function calls' is ambiguous; please clarify whether the answer embedding is concatenated to every node embedding, added as a separate graph-level feature, or appended in some other way.","section":"Section II-C4"},{"comment":"Table II reports point estimates without standard deviations or confidence intervals; the ROC curves in Fig. 4 show standard deviation bands, but the table would benefit from the same information for all metrics.","section":"Table II"},{"comment":"The phrase 'manually fine-tuned' does not specify the search ranges or the number of configurations explored; please add these details to support reproducibility.","section":"Section III-A4"},{"comment":"The phrase 'precision of up to 0.8136' should be qualified as coming from the GCN F+A+A configuration, which uses the final answer as input, so that readers do not attribute the number to the purely structural configurations.","section":"Abstract and Conclusion"},{"comment":"The score(m) in Equation (1) is not defined in the paper; please define the voting score or cite the AutoFL paper for its computation.","section":"Equation (1)"},{"comment":"The ROC curves are shown only for the F+A+A configuration; including ROC curves for the S, F, and F+A configurations would help the reader assess the structural signal independently of the answer input.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong core idea and a useful dataset, and the code/data availability is a plus. The main concerns are methodological: the epoch-selection protocol and hyperparameter tuning need to be made unbiased, and the F+A+A configuration should be repositioned so that the structural claims rest on S, F, and F+A. The early-termination claim needs experimental support or should be toned down. These are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on arXiv:2412.08281. The thing to know: the paper's core idea is genuinely useful — can you predict, from the structure of the reasoning paths, whether self-consistency voting will land on the right answer? The LIM and LIG representations are new, and the F and F+A configurations give reasonably clean evidence that path shape carries signal. GCN with F+A reaches 0.8022 precision, and the paper is honest that AutoFL's own confidence score does slightly better. Code and data are on figshare, which is more than many SE papers do.\n\nNow the soft spots, and they matter. Section III-A4 explicitly says they pick the test metrics from the epoch with the highest test accuracy. With ~45 bugs per fold and 50–100 epochs, that's substantial test-set peeking; the reported AUCs and precisions are likely optimistic. On top of that, hyperparameters in Table I are manually fine-tuned without a nested or held-out selection, so model selection sees the whole dataset. The headline F+A+A configuration is also partly circular: the final voted answer is one of the input features, and the label is whether that answer is correct. The cleaner F and F+A results are the ones that matter, and they still look decent, but the paper doesn't give an unbiased estimate of them.\n\nThe bigger gap is the motivating use case. The paper argues early termination is possible, but every experiment uses complete reasoning paths, all function calls and, in F+A+A, the final answer. There is no experiment with partial prefixes or subsets of runs. So the early-termination claim is untested. Also, only AutoFL on one filtered benchmark set (single-buggy-method bugs) — generalization to other agents is speculation.\n\nWho is this for? People working on LLM agent efficiency and self-consistency cost reduction. It's a good idea with the right shape of evidence, but the numbers need to be redone. A serious referee should ask for a proper validation split, variance reporting, a clean ablation of the answer token, and at least one partial-path experiment before the early-termination claim is taken seriously. I'd accept it for review; it's not a desk reject. But as it stands, treat the headline numbers as upper bounds.\n\nBest","headline":"Plausible and useful idea with genuinely new representations, but the headline numbers are inflated by test-set peeking and the early-termination use case is never actually measured.","tokens_in":7650,"tokens_out":2564,"would_cite":false,"duration_ms":25614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lachesis predicts whether an LLM-based fault localizer's self-consistency answer will be correct, reaching 0.8136 precision.","keywords":["LLM agents","self-consistency","fault localization","reasoning paths","graph neural networks","LSTM","accuracy prediction","early termination"],"falsifier":"Run Lachesis on truncated reasoning paths (e.g., the first k function calls for k = 1,...,N, with the answer embedding removed) and compare its ROC-AUC against full-path predictions; a sharp drop would contradict the early-termination motivation and the claim that structural signal precedes the final answer.","tokens_in":6617,"feed_emoji":"🔮","tokens_out":9065,"duration_ms":72191,"temperature":0.7,"pith_summary":"This paper asks whether the internal shape of an LLM's reasoning—the sequence of function calls an agent makes before giving an answer—contains enough signal to predict whether self-consistency voting will produce the right answer. The authors build Lachesis, a classifier that turns multiple sampled reasoning paths into either a matrix (LIM) or a graph (LIG) and trains an LSTM or GCN to predict whether the set will produce a correct top-ranked answer. Evaluated on AutoFL, an LLM-based fault-localization technique, over 456 bugs from BugsInPy and Defects4J, Lachesis reaches precision up to 0.8136 and ROC-AUC up to 0.7866, slightly below the AutoFL confidence baseline but using representations that need not include the final voted answer. The motivation is cost: if accuracy can be predicted from partial traces, expensive LLM runs that are unlikely to succeed could be terminated early.","feed_headline":"Classifier predicts LLM self-consistency correctness up to 81%","feed_subtitle":"Lachesis reads LLM tool-call traces to flag when voting will pick the wrong method, enabling early termination.","key_machinery":"The central objects are two representations of a set of reasoning paths. The LLM Inference Matrix (LIM) stacks R paths as columns, with rows holding the per-step embeddings of up to N function calls; the LLM Inference Graph (LIG) is a weighted directed graph whose nodes are specific reasoning steps (tool calls or answers) and whose edge weights count how many paths follow the same subsequence. Four embedding schemes control what a node carries: shape only (a constant vector), function type (one-hot over tool names), function type plus arguments (one-hot over code locations the tool inspected), and function type plus arguments plus answer (appending the voted buggy method). The models are a stacked LSTM that reads the matrix step-by-step across all paths, and a GCN with global mean pooling that reads the graph; both output a binary correctness prediction.","core_discovery":"The central discovery is that the structural properties of multiple reasoning paths—how often the same tools are called, whether paths converge on the same code location, and whether the final answers agree—carry reusable signal about whether self-consistency will rank the true buggy method first. Lachesis encodes each AutoFL reasoning step as an embedding (shape-only, function type, function type plus arguments, or function type plus arguments plus answer), aggregates the R sampled paths into a matrix or a weighted directed graph, and classifies the whole set with an LSTM or GCN. With the fullest representation (F+A+A), the GCN achieves 0.8136 precision and 0.7454 accuracy; even the shape-only graph representation reaches 0.7791 ROC-AUC, showing that convergence structure alone is informative. The paper positions this as an empirical verification that the self-consistency hypothesis—multiple paths lead to the correct answer—leaves footprints in the reasoning process that are visible before the end.","pith_inferences":["A testable extension is to feed Lachesis only the first k of the N reasoning steps and measure how precision changes; if it holds at small k, early termination becomes practical, but the paper does not yet provide this evidence.","The same structural representations could be used for other multi-sample decoding schemes, such as best-of-n or majority voting on code generation tasks, where tool calls are replaced by edit operations or test outcomes.","Lachesis's precision-oriented objective suggests a direct use in cost budgeting: run Lachesis after a few samples, stop inferring when it predicts failure, and spend saved budget on harder bugs."],"forward_implications":["If the pattern holds for other self-consistency-based LLM agents, Lachesis-style classifiers could be attached to any tool-using agent to audit when voting is likely to mislead.","The strong performance of the shape-only graph representation suggests that convergence of tool calls is a general proxy for confidence, independent of the specific domain.","The fact that Lachesis approaches the AutoFL-confidence baseline without using the final voted answer (in F and F+A configurations) implies that correctness signal is present in intermediate tool calls, not just in the final vote.","The precision improvement over the 'always correct' baseline (0.6732 accuracy) shows that the classifier is not just exploiting dataset imbalance; it genuinely separates correct from incorrect inference sets."],"supporting_citations":[{"why":"Supplies the target technique AutoFL, whose reasoning paths and correctness labels form the dataset.","marker":"[6]"},{"why":"States the self-consistency hypothesis that multiple reasoning paths lead to the correct answer, which Lachesis is designed to exploit.","marker":"[8]"},{"why":"Defines the ReAct tool-use framework that determines what counts as a reasoning step in AutoFL.","marker":"[9]"},{"why":"Provides the LSTM architecture used to classify LLM Inference Matrices.","marker":"[13]"},{"why":"Provides the GCN architecture used to classify LLM Inference Graphs.","marker":"[14]"},{"why":"Supplies the BugsInPy benchmark bugs used in the evaluation.","marker":"[15]"},{"why":"Supplies the Defects4J benchmark bugs used in the evaluation.","marker":"[16]"}],"fun_headline_variants":["Structural clues in LLM reasoning predict self-consistency results","Lachesis: reading reasoning paths to forecast LLM answer correctness","81% precision: Lachesis predicts when LLM voting picks the right answer","Graph and LSTM models forecast LLM self-consistency success","Early stop for LLM reasoning when self-consistency will miss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that predictions made from complete reasoning paths, including the final answer, will stay accurate when only a prefix of the reasoning is available, but it never tests partial-path prediction.","fun_headline_variants_meta":{"raw":{"variants":["Structural clues in LLM reasoning predict self-consistency results","Lachesis: reading reasoning paths to forecast LLM answer correctness","81% precision: Lachesis predicts when LLM voting picks the right answer","Graph and LSTM models forecast LLM self-consistency success","Early stop for LLM reasoning when self-consistency will miss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001052,"raw_usage":{"total_tokens":4423,"prompt_tokens":958,"completion_tokens":3465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3372}},"tokens_in":574,"tokens_out":3465,"duration_ms":28291,"temperature":1.0,"reasoning_tokens":3372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:59:53.796036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Lachesis on truncated reasoning paths (e.g., the first k function calls for k = 1,...,N, with the answer embedding removed) and compare its ROC-AUC against full-path predictions; a sharp drop would contradict the early-termination motivation and the claim that structural signal precedes the final answer.","supporting_citations":[{"cited_title":"Long short-term memory,","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM architecture used to classify LLM Inference Matrices."},{"cited_title":"Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies,","cited_arxiv_id":null,"evidence_quote":"Supplies the BugsInPy benchmark bugs used in the evaluation."}],"review_version":1}