{"id":"00051a26-3f87-42d5-80f6-04cbc6e63931","arxiv_id":"2606.21024","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A negative knowledge memory layer stores structured records of failed attempts in a shared bank that improves AI agent performance on scientific benchmarks and enables transfer across PDE research tasks.","lead":"The paper proposes a negative knowledge memory layer where a curator agent turns failed AI research attempts into typed records in a shared bank that other agents can adopt or reject before new experiments. A smart generalist might read it to see how treating failures as durable shared assets could make AI research systems more efficient and less wasteful.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Curator-to-record conversion step lacks shown validation that typed failures are accurate and non-noisy before downstream adoption","rationale":"Reader’s weakest assumption directly identifies the same curator-record quality step as the load-bearing point for the empirical claims; full-text details on record schema or curator prompts would be needed to move beyond this concern, but none are supplied in the provided abstract.","tokens_in":1686,"tokens_out":324,"duration_ms":15122,"concrete_test":"Re-run the two nonlinear PDE experiments with an ablated curator that stores only raw failure traces (no typing or bounding) while keeping token budget and agent loop identical; if the success rate on the “new tasks” drops to baseline levels, the typed-record step is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (negative-knowledge agents solve PDE tasks where all baselines fail, plus cross-task transfer) requires that the curator agent produces bounded, typed records whose content is both faithful to the original failure and useful enough to change downstream behavior. The abstract states only that the curator “converts each failed attempt into a bounded, typed record” and that the research agent “explicitly adopts or rejects” them; no verification step, consistency check, or human audit of record quality is described. If the curator introduces systematic mischaracterizations or overly generic records, the measured gains could be artifacts of extra reasoning steps rather than the shared negative-knowledge asset itself. This is the precise condition that must hold for the PDE performance delta to be attributable to the proposed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes a negative-knowledge memory layer for AutoResearch-style AI agents: a curator agent converts each failed attempt into a bounded, typed record stored in a shared bank; downstream research agents explicitly adopt or reject records before proposing the next experiment. The layer is evaluated on same-task retry (ScienceAgentBench) and cross-task PDE research (two nonlinear math-physics problems), with claims that it outperforms vanilla baselines, uses fewer tokens, solves tasks where all baselines fail, and transfers across PDE problems.","tokens_in":1842,"tokens_out":607,"duration_ms":11785,"significance":"If the curator-to-record conversion is shown to be faithful and the performance deltas are causally attributable to the shared negative-knowledge asset rather than extra reasoning steps, the work would establish a concrete mechanism for turning failures into durable, reusable scientific infrastructure. The cross-task transfer result on PDE problems would be particularly noteworthy, as it suggests negative knowledge can function as collective memory beyond single-task debugging.","major_comments":[{"comment":"Abstract and §4 (Evaluation description): the headline claim that negative-knowledge agents solve PDE tasks where all baselines fail rests on the unverified assumption that the curator produces accurate, bounded, typed records whose adoption changes downstream behavior. No validation step, consistency check, human audit, or ablation isolating the record quality is described; without this, the measured gains could be artifacts of additional agent steps rather than the proposed memory mechanism.","section":"Abstract, §4"},{"comment":"§4 and §5 (Results): performance gains, token savings, and cross-task transfer are stated at a high level with no record format, adoption/rejection protocol, number of runs, statistical tests, or error bars supplied. This makes it impossible to assess whether the reported outperformance is robust or reproducible.","section":"§4, §5"},{"comment":"§3 (Method) and transfer experiment: the mechanism by which a prior negative-knowledge bank is loaded and used on a different PDE problem is not specified (e.g., filtering, relevance scoring, or conflict resolution), yet the transfer result is presented as evidence that negative knowledge functions as reusable infrastructure.","section":"§3"}],"minor_comments":[{"comment":"The abstract states 'Code is available at https://github.com/hch-wang/Negative_Knowledge' but the manuscript does not indicate whether the released code includes the exact curator prompts, record schema, or evaluation harness used to generate the reported numbers.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's empirical claims are unusually underspecified for an AI-systems paper; the absence of any quantitative detail on the core conversion step is the primary reason the soundness rating is low. If the authors can supply the missing validation and statistics without changing the experimental scope, the contribution would be worth publishing."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to incorporate the requested clarifications and validations.","responses":[{"response":"We agree that the current manuscript lacks explicit validation of curator output quality. In revision we will add a human audit of sampled records for accuracy, boundedness, and type fidelity, plus an ablation that disables the adopt/reject step while retaining the extra reasoning turns, to isolate the contribution of the shared negative-knowledge asset.","revision_made":"yes","referee_comment":"[Abstract, §4] Abstract and §4 (Evaluation description): the headline claim that negative-knowledge agents solve PDE tasks where all baselines fail rests on the unverified assumption that the curator produces accurate, bounded, typed records whose adoption changes downstream behavior. No validation step, consistency check, human audit, or ablation isolating the record quality is described; without this, the measured gains could be artifacts of additional agent steps rather than the proposed memory mechanism."},{"response":"We accept that the experimental reporting is insufficiently detailed. The revised version will include the precise record schema, the adopt/reject decision protocol, the number of independent runs, the statistical tests performed, and error bars on all metrics.","revision_made":"yes","referee_comment":"[§4, §5] §4 and §5 (Results): performance gains, token savings, and cross-task transfer are stated at a high level with no record format, adoption/rejection protocol, number of runs, statistical tests, or error bars supplied. This makes it impossible to assess whether the reported outperformance is robust or reproducible."},{"response":"The transfer experiment simply loads the entire prior bank into the shared memory for the new PDE task; the research agent then applies the identical adopt/reject protocol with no additional filtering or scoring. We will expand §3 to state this loading procedure explicitly and note that any conflicts are resolved by the agent's own judgment during adoption.","revision_made":"yes","referee_comment":"[§3] §3 (Method) and transfer experiment: the mechanism by which a prior negative-knowledge bank is loaded and used on a different PDE problem is not specified (e.g., filtering, relevance scoring, or conflict resolution), yet the transfer result is presented as evidence that negative knowledge functions as reusable infrastructure."}],"tokens_in":1417,"tokens_out":468,"duration_ms":17662,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a practical addition to AutoResearch-style agents: a curator agent takes each failed run and writes it into a shared bank as a bounded, typed record that the next agent must explicitly accept or reject before trying again. They test this on ScienceAgentBench retries and on two nonlinear PDE problems, reporting that the negative-knowledge version beats the plain baseline, uses fewer tokens, solves some tasks the baselines never crack, and transfers across different PDE instances.\n\nThe transfer result and the code release are the clearest positives. Showing that prior negative records help on a new PDE problem is a concrete step beyond generic memory or logging, and making the code public lets others check the implementation.\n\nThe soft spots sit in the middle of the mechanism. The abstract says the curator “converts each failed attempt into a bounded, typed record” and that agents “explicitly adopt or reject” them, but it gives no example of a record, no description of the typing rules, and no audit or consistency check on whether the curator’s summary matches the actual failure. Without that, the PDE gains could come from the extra reasoning steps rather than from useful negative knowledge. The performance numbers also lack error bars, run counts, or statistical tests, so the size of the improvement is hard to judge.\n\nThe stress-test note about missing validation on the curator step matches what the abstract shows; nothing in the provided description contradicts it.\n\nThis is aimed at groups already running multi-agent scientific pipelines who want to keep failure information from evaporating. It is not yet a finished theory of negative knowledge, but the architectural idea is clear enough that a referee could usefully check the missing details on record construction and the strength of the empirical claims.\n\nI would send it to review rather than desk-reject.","headline":"The paper adds an explicit curator that turns agent failures into typed shared records for later adoption, but the abstract leaves the record quality and validation steps unshown.","tokens_in":2324,"tokens_out":440,"would_cite":false,"duration_ms":17573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A negative knowledge memory layer turns failed AI research attempts into shared records that improve performance on scientific tasks.","keywords":["negative knowledge","shared memory","AutoResearch","failure-aware","PDE problems","AI-assisted research","curator agent","scientific memory"],"falsifier":"If agents equipped with the negative knowledge bank do not solve any new PDE tasks that all vanilla baselines fail to solve, or if the layer uses more tokens without performance gains on ScienceAgentBench or PDE problems.","tokens_in":2573,"feed_emoji":"🧠","tokens_out":660,"duration_ms":18297,"temperature":0.7,"pith_summary":"The paper proposes maintaining structured records of failed experiments as a negative knowledge bank in AI-assisted research systems. A curator agent processes failures into typed records that downstream agents can explicitly adopt or reject before proposing new experiments. This approach is tested on same-task retries in ScienceAgentBench and on cross-task solving of two nonlinear PDE problems in math-physics. It shows better results than standard AutoResearch methods with less token use and enables solving tasks that baselines cannot. The work argues that negative knowledge should be treated as a durable asset alongside positive findings for collective scientific memory.","feed_headline":"Negative knowledge turns AI failures into shared memory for PDE tasks","feed_subtitle":"The layer solves tasks no baseline can and transfers across problems while using fewer tokens.","key_machinery":"negative knowledge memory layer: curator agent converts failed attempts into bounded, typed records in a shared bank, which downstream agents adopt or reject before next experiments","core_discovery":"The central claim is that a failure-aware shared memory layer, where a curator agent converts each failed attempt into a bounded, typed record in a shared bank and downstream research agents explicitly adopt or reject those records before proposing the next experiment, outperforms vanilla AutoResearch baselines on both same-task retry and cross-task scientific research while using fewer tokens; agents with the negative knowledge bank solve new tasks that all baselines fail to solve in PDE systems research, and the previous negative knowledge bank transfers to enhance AutoResearch on different PDE problems.","pith_inferences":["This failure-to-record mechanism could apply to other trial-and-error domains like experimental chemistry or materials discovery where repeated mistakes waste resources.","Treating failures as transferable assets might reduce redundant exploration in multi-agent AI systems across unrelated scientific questions.","Combining the negative bank with positive knowledge stores could form a fuller collective memory infrastructure for AI-driven research teams."],"forward_implications":["The negative knowledge layer outperforms vanilla AutoResearch baselines while using fewer tokens on same-task retry in ScienceAgentBench.","Agents with the negative knowledge bank solve new tasks in PDE systems research that all baselines fail to solve.","The previous negative knowledge bank transfers and enhances AutoResearch on different PDE problems.","Structured negative knowledge is a knowledge asset that should be explicitly maintained in broader AI-engaged scientific research alongside positive findings."],"fun_headline_variants":["Negative knowledge memory boosts AutoResearch on PDE tasks","Failure bank aids agents solving PDE problems baselines miss","Curator turns AI failures into shared records for research","Negative knowledge transfers enhance PDE AutoResearch trials","Shared failure memory cuts tokens in cross-task PDE work"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Failed attempts can be converted by a curator agent into bounded, typed records whose adoption or rejection by downstream agents produces measurable performance gains without introducing harmful bias or noise.","fun_headline_variants_meta":{"raw":{"variants":["Negative knowledge memory boosts AutoResearch on PDE tasks","Failure bank aids agents solving PDE problems baselines miss","Curator turns AI failures into shared records for research","Negative knowledge transfers enhance PDE AutoResearch trials","Shared failure memory cuts tokens in cross-task PDE work"]},"model":"grok-4.3","cost_usd":0.003504,"raw_usage":{"total_tokens":1834,"prompt_tokens":650,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":35037000,"prompt_tokens_details":{"text_tokens":650,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1114,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":650,"tokens_out":70,"duration_ms":9886,"temperature":1.0,"reasoning_tokens":1114,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T14:33:54.294583+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If agents equipped with the negative knowledge bank do not solve any new PDE tasks that all vanilla baselines fail to solve, or if the layer uses more tokens without performance gains on ScienceAgentBench or PDE problems.","supporting_citations":[],"review_version":1}