{"id":"ed183b98-f063-4e6c-8c2f-2a30f167b7f8","arxiv_id":"2608.04761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Math-only contrastive training of an embedding model yields progress-oriented insight retrieval that improves LLM agents in ALFWorld, WebShop, and ScienceWorld without environment-specific fine-tuning.","lead":"This paper trains a retrieval model on mathematical reasoning problems alone, then uses it to fetch helpful strategy rules for AI agents acting in simulated kitchens, web shops, and science labs. The authors report that this math-only training transfers, improving agent task success over a standard embedding model without retraining on each environment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-attempt utility labels (I+/I-) are the load-bearing assumption; noisy labels could produce the transfer gains without a genuine progress-oriented geometry.","rationale":"The reader's weakest assumption is exactly the single-validation label reliability issue, and the manuscript's own Limitations section confirms it. This is the most load-bearing concern because it attacks the training signal itself: if the I+/I- labels are noisy, the InfoNCE objective in Eq. 3 does not learn the claimed 'progress-conditioned relevance,' and the downstream agentic and retrieval gains could be a byproduct of fine-tuning on any contrastive pairs rather than evidence for the paper's central mechanism. The paper earns credit for honestly flagging the limitation, for extensive robustness checks (GPT-5.2 as insight generator and action model, multiple retrieval budgets, reasoning-oriented retriever baselines, an in-domain ALFWorld-trained comparison, and a no-half-traj ablation), and for the bidirectional math/agentic transfer analysis. These make the empirical phenomenon plausible, but none of them validate the label-generation step. The proposed repeated-validation test is feasible, quantitative, and directly targets the assumption: it measures how much of the transfer gain survives when label noise is reduced. Since the reader already assigned CONDITIONAL with the same core concern, my stress-test does not move the verdict; it sharpens the condition under which the paper should be accepted: the authors should provide multi-attempt label reliability evidence or a no-label-noise ablation. If the test were run and the gains collapsed, the verdict would be REJECT, not because the analysis is dishonest but because the central claim would then rest on an invalidated supervision signal. Until that test is run, CONDITIONAL is the appropriate verdict.","tokens_in":30549,"tokens_out":4248,"duration_ms":47145,"concrete_test":"Rebuild the Stage 1 training pairs with repeated validation: for each candidate insight-problem pair, run N=5 independent solver rollouts with and without the prepended insight, aggregate solve-rate improvement (e.g., majority vote or mean difference), and regenerate I+/I- using the aggregated labels. Retrain InsightEmb from the same Qwen3-Embedding-4B base with identical hyperparameters, then re-run the full evaluation in Tables 2 and 3 (ALFWorld, WebShop, ScienceWorld, SRA-Bench). Also report the label flip rate between the original single-attempt labels and the 5-attempt labels. If the headline gains (e.g., ALF BUNDLE +6.42, WebShop BUNDLE +13.32, SRA-Bench R@10 +8.90) persist, the label-noise concern is largely settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that math-only contrastive training learns a progress-oriented retrieval geometry that transfers to agentic insight retrieval. The only training signal for this geometry is the utility-filtered labels (I+, I-) described in Section 3.3 and Appendix A.2.1: for each candidate insight-problem pair, the insight is labeled positive only if prepending it to the problem improves solve rate, and the Limitations state this relies on 'a single validation attempt.' LLM solver rollouts are stochastic, so a single attempt can classify an insight as helpful or harmful by chance. False positives pull query embeddings toward insights that did not actually resolve the bottleneck; false negatives do the reverse. Because InfoNCE (Eq. 3) directly optimizes separation of these labels, the learned geometry may encode spurious correlations (e.g., lexical overlap between distilled insights and problem text) rather than the claimed action-intent matching. The transfer improvements in Tables 2 and 3 could then arise from a generic fine-tuning effect rather than from progress-oriented matching. The concern is amplified by the paper's second stated limitation: partial-trajectory anchors reuse problem-level (I+, I-) labels instead of state-specific supervision, so a rule that is genuinely useful for the final problem may be mislabeled as positive for a mid-progress state where it is premature or irrelevant. If these labels are unreliable, the central claim lacks a validated training signal, regardless of how consistent the downstream improvements appear.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InsightEmb, a contrastive embedding framework trained exclusively on mathematical reasoning data (MATH) in two stages: situation-to-insight matching with utility-filtered positive/negative labels, and situation-to-experience matching with structurally similar trajectories. At inference, the model retrieves abstract insights for LLM agents in ALFWorld, WebShop, and ScienceWorld under dynamic state-conditioned queries, and is also evaluated on SRA-Bench for static task-to-skill retrieval. The central claim is that math-only contrastive training learns a transferable, progress-oriented retrieval geometry that resolves the current decision bottleneck rather than matching surface semantics, and that this geometry transfers to agentic insight retrieval. The main evidence is consistent gains over the base embedder across the three environments and SRA-Bench, robustness to insight-generator and action-model changes, an in-domain ALFWorld comparison, a no-half-trajectory ablation, and a geometric-diversity analysis. The authors also acknowledge two load-bearing limitations: utility labels rely on a single validation attempt per insight–problem pair, and partial-trajectory anchors reuse problem-level labels rather than state-specific supervision.","tokens_in":30843,"tokens_out":4645,"duration_ms":50364,"significance":"If the central claim holds, the paper makes a useful and non-obvious contribution: it shows that progress-oriented retrieval geometry can be trained from publicly available math reasoning data and transferred to interactive agent tasks without environment-specific supervision. The empirical package is unusually broad: multiple environments, two insight granularities, several retrieval budgets, two action models, two insight generators, an in-domain fine-tuning comparison, and a static benchmark. The no-half-trajectory ablation and the bidirectional transfer experiments (math-to-agent and agent-to-math) are thoughtful attempts to isolate the mechanism, and the geometric-diversity analysis in Appendix C.2 is a concrete step beyond hand-waved notions of transfer. The main reservations are that the training labels are noisy by the authors' own admission, the reported differences often lack statistical assessment, and the state-conditioned interpretation of the partial-trajectory anchors is not directly validated. These issues are fixable and do not by themselves invalidate the approach, but they currently make the central claim less airtight than the text suggests.","major_comments":[{"comment":"The central training signal for the progress-oriented geometry is the utility-filtered label assignment I+/I- described in Section 3.3, and the Limitations state that these labels rely on a single validation attempt per candidate insight–problem pair. Equation (3) directly optimizes separation of these labels, so if LLM inference stochasticity produces false positives or false negatives, the learned geometry may encode spurious correlations (e.g., lexical or topical overlap) rather than the claimed progress-oriented matching. Because this is the only training signal for the transferable property, the central claim is load-bearing on label reliability. I ask for a concrete sensitivity analysis: on a subsample of training pairs, run repeated validation attempts (e.g., 5 or 10), report label agreement, train an ablation on majority-vote labels, and show whether the transfer gains in Tables 2 and 3 persist. Without this, the alternative explanation of a generic fine-tuning effect cannot be ruled out.","section":"Section 3.3 and Limitations"},{"comment":"The partial-trajectory anchors reuse problem-level I+/I- labels rather than state-specific supervision, as the authors acknowledge in the Limitations. This means a rule that is genuinely useful for the final problem may be mislabeled as positive for a mid-progress state where it is premature or irrelevant. This matters especially because the no-half-trajectory ablation in Appendix C.1.1 attributes about 83% of the ALFWorld gain and about 49% of the WebShop gain to the partial-trajectory anchors. If those anchors are not state-conditioned, the mechanism may not be the state-aware retrieval the paper claims. I would like to see either state-specific label construction or a diagnostic that validates the state-conditioned interpretation, for example by annotating a subset of mid-progress states with the correct bottleneck and measuring whether InsightEmb ranks the corresponding insight above the problem-level positive.","section":"Section 3.3 and Appendix C.1.1"},{"comment":"The reported improvements are not accompanied by confidence intervals or significance tests. Some differences are small in absolute terms, notably ScienceWorld top-1 BUNDLE (7.40 vs 8.00) and top-1 ATOMIC (7.40 vs 10.20) over 500 games, and the R@1 gain on SRA-Bench is 1.04 percentage points. Bootstrap confidence intervals across the 500 games (or across task families for SRA-Bench) and a paired test such as McNemar's test for success rates would substantially strengthen the claim that InsightEmb improves over Base 'on all these evaluations.' This is a load-bearing point because the paper's headline is a universal positive transfer result, not just a directional trend.","section":"Tables 2, 3 and Figure 2"},{"comment":"The ScienceWorld result deserves closer scrutiny before it is used as strong evidence of transfer. The no-insight baseline is 2.40%, every insight setting more than triples it, and the Base embedding itself also raises success to 7.40%. The differences between Base and InsightEmb on BUNDLE (7.40 vs 8.00) are within the range of stochastic variation for 500 games without a significance test. The text says InsightEmb 'wins or ties' on all six cells of Table 15, but several cells are very close. I recommend reporting per-game bootstrap intervals and, ideally, the distribution of task-type-level differences, to clarify whether the cross-domain gain is concentrated in a few task types or is a broad effect.","section":"Section 5.3 and Table 2"},{"comment":"Equation (1) defines the target as expected progress given that the agent uses insight I, but the paper does not estimate this expectation in the contrastive loss; instead, it uses a single-pass solve-rate improvement as a proxy. The mismatch between Equation (1) and the actual label-generation process is acknowledged but not quantified. I recommend reporting the distribution of validation outcomes (e.g., how many labels flip if the validation attempt is repeated) and, if possible, the correlation between the proxy labels and a more reliable multi-attempt estimate. This would convert an acknowledged limitation into a measurable property of the training data.","section":"Section 6.1 / Equation (1)"},{"comment":"The SRA-Bench per-family results show a clear negative outlier: MedCalcBench drops from 92.73 R@1 for Base to 63.27 for InsightEmb, which is a substantial regression on a family that is part of the macro-average. The BM25-hybrid remedy recovers most of this gap, but the main-text claim of 'improves over Base on every cutoff' is driven by the macro-average. I recommend reporting the macro-average both with and without MedCalcBench, and stating explicitly that the method is not suitable alone for terminology-dense lexical retrieval. This does not invalidate the paper, but it sharpens the applicability claim.","section":"Section 5.4 and Appendix E.2"},{"comment":"The mechanistic analyses rely heavily on manually selected qualitative examples (e.g., Figure 8, Tables 18-20). These examples are consistent with the authors' interpretation, but they do not by themselves establish that procedural matching is the dominant behavior. I recommend reporting quantitative prevalence counts over all divergent games for each identified pattern (variant-selection awareness, loop prevention, procedural sequencing, etc.), with inter-annotator agreement if manual labeling is used. This would make the mechanistic claim less anecdotal.","section":"Appendix D.2.4 and D.6.7"},{"comment":"There is an apparent inconsistency in the retrieval protocol: Figure 8's caption states 'top-5 retrieval' while the main protocol in Section 4.2 and Table 2 uses top-1 retrieval. Please clarify which setting Figure 8 is from, and ensure all qualitative examples use a consistent protocol or explicitly note when a different budget is used.","section":"Appendix D.2.4 and Figure 8"},{"comment":"The text says the SRA-Bench evaluation is 'over the 636 gold skills and 26,262 distractors,' while Table 21 lists per-family query counts (747, 760, 1,430, 223, 1,100, 1,140) that sum to 5,400. The relationship between the 636 gold-skill candidate set and the per-family query counts should be clarified (e.g., whether queries are shared across families or each family has its own candidate set), so the reader can correctly interpret the macro-average.","section":"Section 5.4 and Table 21"},{"comment":"The robustness check with GPT-5.2-generated corpora uses 700 bundles for each environment, whereas the main DeepSeek-R1 corpora use 2,419 and 501 bundles for ALFWorld and WebShop. The difference in corpus size is not discussed. If the GPT-5.2 corpora are smaller, the comparison may not isolate the insight-generator change from corpus-size effects. Please state whether the corpora were matched in size or explain why the size difference is immaterial.","section":"Section 6.2 and Appendix A.4"},{"comment":"The paper does not mention code, trained model weights, or insight corpus release. Given that the central contribution is a training recipe and a set of corpora, reproducibility would be greatly aided by releasing the training code, the distilled insight corpora, and the evaluation harness. This is a presentation/artifacts issue rather than a technical flaw, but it is important for a methods paper.","section":"Overall reproducibility"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":"The paper is serious and the experimental package is unusually broad. The core concern is that the training labels are noisy by the authors' own admission, and the state-conditioned interpretation of the mid-progress anchors is not directly validated. These are fixable: a repeated-validation analysis on a subsample, majority-vote label training, and confidence intervals for the main tables would substantially strengthen the claim. I would not require a full re-run of all experiments; focused sensitivity analyses and statistical reporting would suffice. The MedCalcBench outlier should be presented more transparently, though the BM25 hybrid is a reasonable remedy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a genuinely useful idea — train the retriever to match situations to bottleneck-resolving rules rather than to topically similar text — and it makes a plausible case that math-only contrastive training transfers to agentic insight retrieval. It is not a paradigm shift, but it is a real step forward for agentic retrieval.\n\nWhat's new: the formulation of action-intent matching, the two-stage curriculum (situation-to-insight and situation-to-experience), and the consistent gains across ALFWorld, WebShop, ScienceWorld, and SRA-Bench. The paper is also honest: the Limitations section names the single-validation-attempt label issue, and the appendices provide per-game statistics, prompts, and a per-family breakdown of SRA-Bench. The qualitative analyses (e.g., ALFWorld step-conditioned retrieval shifting from search to verification) actually support the claimed mechanism rather than just reporting numbers.\n\nSoft spots, in order of importance. First, the training labels are load-bearing and noisy: each (I+, I-) label comes from a single validation attempt, so false positives/negatives are inevitable. The stress-test concern is legitimate — if the labels are unreliable, the learned geometry may reflect spurious cues rather than progress. But it is not fatal: the authors acknowledge it, the transfer is consistent across three very different environments, and the no-half-traj ablation and the in-domain comparison make a generic fine-tuning story less likely. Still, a noise-injection or repeated-validation ablation would substantially strengthen the central claim. Second, no confidence intervals or significance tests anywhere; some differences are small (ScienceWorld 8.0 vs 7.4). Third, the \"without environment-specific training\" framing is a bit generous, since the evaluation corpora are curated with environment-specific solve-rate filtering and the evaluation metric is the same progress criterion used to build training labels — circular in spirit, though not in data. Fourth, no code or data release, which hurts reproducibility.\n\nWho it's for: people working on agent memory, self-improving agents, and reasoning-aware retrieval. It deserves a serious referee — send it to review, but ask for error bars, a label-reliability check, and a release plan before it's done.","headline":"A solid, honest paper on progress-oriented retrieval that deserves a serious referee, held back mainly by noisy single-attempt training labels and missing error bars.","tokens_in":31320,"tokens_out":2846,"would_cite":true,"duration_ms":28449,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A contrastive embedding model trained only on mathematical reasoning problems learns to retrieve the insight an agent needs at its current step, and that skill transfers to embodied, web, and scientific agent tasks without…","keywords":["agentic insight retrieval","action-intent embeddings","contrastive learning","cross-domain transfer","dense retrieval","math reasoning","retrieval-augmented agents"],"falsifier":"Retrain InsightEmb using utility labels that are majority votes over five independent solver attempts instead of one, and compare with the published single-attempt model on the same ALFWorld, WebShop, ScienceWorld, and SRA-Bench protocols; if the two label sets disagree on most pairs and the gains over the base embedder do not survive, the central claim fails. A cheaper check is to measure the disagreement rate between single-attempt and repeated-attempt labels on a sample of training pairs, and a sanity check is to run the trained retriever on a deliberately shuffled insight corpus to confirm that success gains are not coming from corpus artifacts.","tokens_in":30257,"feed_emoji":"🎯","tokens_out":7812,"duration_ms":66351,"temperature":0.7,"pith_summary":"The paper claims that an agent's next-step insight needs can be captured by a vector space: each situation and each abstract rule gets an embedding, and similarity in that space means \"this rule unblocks this situation.\" It further claims this space is domain-independent, so it can be learned entirely from math problems: a contrastive model trained on math-only data retrieves the right procedural insight for embodied, web-shopping, and scientific agents at the moment it is needed. This matters because self-improving agents accumulate libraries of reusable rules, and the deciding bottleneck is which rule applies to the current state, not how to store more rules. The paper reports that this math-only model beats the base embedder, an in-domain retriever trained on the target environment, and two reasoning-oriented retrieval models, and that it can turn a noisy insight pool that would otherwise hurt an agent into a useful one.","feed_headline":"Math-trained embedder beats topical retrieval for agents","feed_subtitle":"Math-only training picks the right next-step insight for agents in ALFWorld, WebShop, and ScienceWorld.","key_machinery":"The central object is the action-intent embedding $f_\\theta$, trained in two contrastive stages on MATH problems only. Stage 1 (situation-to-insight matching) aligns a query—raw problem, partial chain-of-thought, or full chain-of-thought—with a distilled heuristic rule, using InfoNCE with in-batch negatives and temperature $\\tau=0.01$; the positive rule is chosen by a utility filter: prepending it to the problem improves solve rate on validation attempts, tying labels to demonstrated progress. Stage 2 (situation-to-experience matching) contrasts structurally similar versus dissimilar solved problems to sharpen the situation representation itself. At inference the same embedder encodes the agent state with a domain-neutral instruction prefix and retrieves the nearest insight from a pre-encoded corpus.","core_discovery":"The central claim is that action-intent relevance—whether an insight resolves the bottleneck that must be cleared before the agent can make progress—has a transferable geometry. The paper argues that math problems and agentic states instantiate the same situation-to-rule matching operation: a concrete situation (a math problem, or an agent observation with action history) must be matched to an abstract rule that names the next useful step, and six bottleneck categories (search/identify, state-transform, verification, error-recovery, structure/layout, strategy-before-execute) recur in both domains. Training on math-only contrastive triplets, with positives selected by whether prepending the rule actually improves solve rate, produces an embedding that ranks agentic insights by progress rather than topical similarity. The paper demonstrates this by dynamic agent success in ALFWorld, WebShop, and ScienceWorld and by recall on SRA-Bench, with no environment-specific training.","pith_inferences":["If the transfer is as general as the paper suggests, any domain with a verifiable progress signal—code debugging, theorem proving, or tool-use logs—could serve as the training source; this is a testable extension the paper does not run.","The single-validation-attempt labeling is the fragile link; repeating validation with multiple attempts and checking whether the gains grow or shrink would directly measure how much of the effect depends on label quality.","The MedCalcBench exception suggests the learned abstraction can wash out terminology-level cues, so a hybrid lexical-plus-structural retriever may be the practical recipe for skill pipelines that mix procedural and entity-dense skills.","The step-conditioned retrieval shift—search rules early, verification rules late—implies the embedding encodes temporal phase, which could be exploited by agents that abstain from retrieval when the top match is not phase-appropriate; the paper does not implement that abstention."],"forward_implications":["Retrieval can be trained once, in a cheap public domain, and reused across agent environments whose insight corpora are built later; no per-environment retrieval supervision is required.","A progress-blind retriever can actively hurt an agent, and a progress-oriented retriever restores or exceeds the no-retrieval baseline in that regime, so retrieval utility must be measured by end-task delta, not ranking metrics alone.","Training on a broad, structurally diverse source such as math can outperform fine-tuning on the target environment's own pairs, so supervision breadth can matter more than domain match.","The same model improves static task-to-skill retrieval, indicating that the action-intent geometry transfers beyond interactive execution to offline skill selection."],"supporting_citations":[{"why":"Supplies the InfoNCE contrastive objective used in both training stages.","marker":"Chen et al., 2020"},{"why":"The MATH dataset is the sole source of training data for InsightEmb.","marker":"Hendrycks et al., 2021"},{"why":"ALFWorld is the embodied evaluation environment for dynamic insight retrieval.","marker":"Shridhar et al., 2021"},{"why":"WebShop is the web-shopping evaluation environment.","marker":"Yao et al., 2022"},{"why":"ScienceWorld is the scientific-procedure evaluation environment.","marker":"Wang et al., 2022b"},{"why":"SRA-Bench is the static task-to-skill retrieval benchmark used to isolate retrieval quality.","marker":"Su et al., 2026"},{"why":"Qwen3-Embedding-4B is the backbone embedding model fine-tuned by both stages.","marker":"Zhang et al., 2025"},{"why":"Qwen3-8B serves as the action-generating LLM in the dynamic agent protocol.","marker":"Qwen Team, 2025"},{"why":"DeepSeek-R1 distills the main insight corpora used in the agentic evaluations.","marker":"DeepSeek-AI, 2025"},{"why":"ReasonIR is the closest reasoning-retriever baseline that InsightEmb must surpass on agentic retrieval.","marker":"Shao et al., 2025"}],"fun_headline_variants":["Math-trained embeddings rank agent insights by progress, not topic","Agent insight retrieval learns progress geometry from math problems only","InsightEmb: math-only training beats topical retrieval for agent actions","Transferable action-intent geometry learned from math reasoning data","Progress-aware insight embeddings trained on math, tested on agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The contrastive labels that say which insight helps and which hurts come from a single validation attempt per insight–problem pair, so an unlucky or lucky run of the solver could label a helpful rule as unhelpful or vice versa; if those labels are systematically unreliable, the trained geometry may not reflect real progress.","fun_headline_variants_meta":{"raw":{"variants":["Math-trained embeddings rank agent insights by progress, not topic","Agent insight retrieval learns progress geometry from math problems only","InsightEmb: math-only training beats topical retrieval for agent actions","Transferable action-intent geometry learned from math reasoning data","Progress-aware insight embeddings trained on math, tested on agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1316,"prompt_tokens":894,"completion_tokens":422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":510,"tokens_out":422,"duration_ms":4991,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:40:32.920596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain InsightEmb using utility labels that are majority votes over five independent solver attempts instead of one, and compare with the published single-attempt model on the same ALFWorld, WebShop, ScienceWorld, and SRA-Bench protocols; if the two label sets disagree on most pairs and the gains over the base embedder do not survive, the central claim fails. A cheaper check is to measure the disagreement rate between single-attempt and repeated-attempt labels on a sample of training pairs, and a sanity check is to run the trained retriever on a deliberately shuffled insight corpus to confirm that success gains are not coming from corpus artifacts.","supporting_citations":[],"review_version":2}