{"id":"75388487-d482-49ac-9218-7b89cc5de686","arxiv_id":"2608.17247","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding a mandatory memory-state output field does not significantly improve memory-policy classification in two large language models, while exposing the state definitions alone does improve it.","lead":"This paper tests whether forcing a personal AI to first label retrieved user memories as use, ignore, update, or ask actually improves its choices. It finds the added state label does not reliably help on a controlled benchmark, while merely showing the state definitions does, so a popular state-first prompting recipe is not supported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'isolated explicit state-output field' is not what the primary contrast measures: state-output vs policy-only changes both taxonomy exposure and output field, whereas the isolated field effect is state-output vs taxonomy-only (Llama -8.5 pp, GPT-OSS -1.7 pp).","rationale":"The reader's weakest assumption was that rule-derived reference labels may not capture human judgment. That is a real external-validity limitation, and the paper acknowledges it in Section 5 and the Limitations, but it does not threaten the internal validity of the benchmark-relative claim: within a synthetic, intentionally rule-labeled counterfactual set, the rule-derived labels are the definition of the task. The more load-bearing problem is internal: the abstract's phrase 'isolated explicit state-output field' is not what the primary matched contrast tests. The state-output arm differs from the policy-only baseline by two components, state taxonomy exposure and state output. The taxonomy-only arm isolates the taxonomy component, so the isolated output-field effect must be state-output minus taxonomy-only. The paper's own Table 4 shows that this isolated effect is negative for both endpoints, while the abstract reports the combined contrast (+0.6/+3.3) as if it were isolated. This matters because the central claim is specifically about the isolated state-output field; the numbers a reader would cite from the abstract are the combined effect. The paper is otherwise careful: arms were pre-specified, parse failures counted as incorrect, family-cluster statistics used, the label-conditioning diagnostic is correctly refused mechanism status, and the Limitations are honest. The fix is computational and cheap: report the state-output versus taxonomy-only contrast with cluster-aware intervals. Until that contrast is reported, the central numerical claim is not supported by the displayed test, though the broader 'not enough' conclusion appears robust. This is why I keep the verdict CONDITIONAL rather than moving to ACCEPT or REJECT.","tokens_in":24644,"tokens_out":7546,"duration_ms":76108,"concrete_test":"Recompute the matched-ablation contrast as State_Output_No_Rationale minus Taxonomy_No_State_Output on the frozen 160-example, 40-family data with seeds 101-103, using the same family-cluster percentile bootstrap and family-level sign-flip permutation procedures as Table 1. Report the delta, 95% CI, raw and Holm-adjusted p for both Llama-3.3-70B and GPT-OSS-120B. Also report the full 2x2 treatment decomposition (policy-only, taxonomy-only, state-output without rationale, state-output with rationale) so the state-output main effect is separated from the taxonomy exposure main effect. If the state-output-vs-taxonomy delta is negative and significant (e.g., Llama at -8.5 pp), revise the abstract and Section 7 to state that an isolated state-output field actively hurts once state definitions are exposed, rather than reporting a non-significant +0.6/+3.3 against policy-only.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim as worded in the abstract says an 'isolated explicit state-output field' gives +0.6 pp (Llama, p=0.8084) and +3.3 pp (GPT-OSS, p=0.0710). But the primary matched comparison, State_Output_No_Rationale versus Clean_Policy_Only, is not an isolated state-output contrast: the state-output arm also adds the four state definitions, while policy-only contains neither definitions nor state output. The correct control for the output field is Taxonomy_No_State_Output, which supplies the same state definitions but does not require a state field. Appendix F, Table 4 gives the needed numbers: Llama taxonomy-only is 53.1% versus state-output 44.6% (state-output is -8.5 pp relative to taxonomy-only), and GPT-OSS taxonomy-only is 59.8% versus state-output 58.1% (-1.7 pp). Thus the reported +0.6/+3.3 deltas conflate taxonomy exposure with state emission. Section 7 states that the design 'dissociates taxonomy exposure from state emission,' but the dissociation contrast (state-output minus taxonomy-only) is never reported with cluster intervals or p-values. The paper's broader conclusion that explicit state elicitation is not enough would survive a re-analysis, since the isolated field effect is negative in direction, but the abstract's quantitative claim is not the claim that was actually tested. This is an internal mismatch between the headline numbers and the experimental contrast, not merely a missing robustness check.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies memory-policy classification for personalized agents, where a retrieved user memory must be routed to Use, Ignore, Update, or Ask. It first shows that a 480-example synthetic development set is lexically separable and lacks positive Ignore cases, so its apparent gains from a state-structured prompt bundle are treated as motivation rather than evidence. The authors then construct a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies, and run a five-stage audit protocol: dataset shortcuts, prompt isolation, answer-associated label conditioning, decomposed semantic evidence, and provider-level execution failures. On the frozen set, exposing the four state definitions improves accuracy over policy-only prompting, but the paper claims that an isolated explicit state-output field does not significantly improve policy accuracy (Llama +0.6 pp, p=0.8084; GPT-OSS +3.3 pp, raw p=0.0710). A separate label-conditioning diagnostic shows that supplying benchmark-associated state labels shifts routing, and family-level analyses show that complete four-way family success is rare. The paper concludes that explicit state elicitation is not a reliable mechanism for improving memory-policy routing and that development-set bundle gains do not survive isolation.","tokens_in":24832,"tokens_out":4207,"duration_ms":43267,"significance":"If the central result holds, the paper makes a useful negative contribution: it provides a reusable audit protocol for structured intermediate outputs and demonstrates, with unusually transparent execution accounting, that a state-output field is not a reliable route to better policy classification. The design has real strengths: the controlled-set arms were frozen before hosted evaluation; parse failures count as incorrect; uncertainty is quantified with family-cluster bootstrap and permutation tests with Holm adjustment; label-conditioning conditions are explicitly separated from mechanism evidence; and the appendices contain detailed cost, failure, and per-policy reporting. These practices are exactly what the field needs for prompt-intervention claims. However, the headline quantitative claim about an \"isolated\" state-output field is not the contrast that was actually tested, as detailed below. The broader qualitative conclusion is defensible, but the reported numbers and the claimed dissociation need substantive reanalysis before the paper can be accepted.","major_comments":[{"comment":"The abstract and Section 7 describe the state-output result as an \"isolated explicit state-output field\" effect, but the primary matched comparison (State_Output_No_Rationale versus Clean_Policy_Only) changes both taxonomy exposure and state output simultaneously. The correct control for the effect of the output field is Taxonomy_No_State_Output, which supplies the same state definitions without requiring a state field. Appendix F, Table 4 provides the needed numbers: for Llama, state-output accuracy is 44.6% versus taxonomy-only 53.1%, giving -8.5 percentage points; for GPT-OSS, state-output is 58.1% versus taxonomy-only 59.8%, giving -1.7 percentage points. Section 7 states that the design \"dissociates taxonomy exposure from state emission,\" but no cluster interval or p-value is reported for the state-output-minus-taxonomy-only contrast. The abstract's +0.6 and +3.3 figures therefore conflate the taxonomy exposure effect with the state emission effect. This is an internal mismatch between the headline claim and the experimental contrast, not a stylistic issue. The paper should either report the dissociation contrast as the isolated state-output effect or rewrite the abstract and Section 7 so that the tested contrast is described accurately.","section":"Abstract; §7; Table 1; Appendix F (Table 4)"},{"comment":"The reported taxonomy-only gain (+9.17 pp for Llama, +5.00 pp for GPT-OSS) is treated as a finding that \"exposing the four state definitions improves accuracy,\" but Section 8 concedes that the rendered taxonomy prompt lists the four states in the same order as the four policies and closely paraphrases them, so the mapping is recoverable without being stated. Because the benchmark states map deterministically to the reference policies, the taxonomy-only condition may simply restate the answer structure. The paper acknowledges this confound in prose, but the main-text result is still presented as if taxonomy exposure were a separable treatment. A control that permutes or rewords the state definitions, or that uses semantically decomposed evidence fields not isomorphic to the policy labels, would be needed to substantiate the claim that the taxonomy gain is due to useful typed knowledge rather than definitional leakage. At minimum, the taxonomy-only result should be labeled as confounded with definition restatement throughout the abstract and results sections.","section":"§8; §7 (taxonomy-only result)"},{"comment":"Every accuracy number in the paper is measured against rule-derived reference policies generated deterministically from structured semantic slots before rendering. The paper correctly discloses that labels are rule-derived and that annotator agreement \"partly reflects the construction,\" but this limitation is load-bearing for the central null result and for the taxonomy gain: if the rule set mis-encodes the intended abstraction for any of the 40 families, the state-output null, the taxonomy-only gain, and the family-level statistics are all measured against a disputable standard. I do not think this requires rejection, because the paper is transparent about the limitation and the controlled counterfactual design is a reasonable audit standard. However, the authors should add at least one concrete robustness check, such as showing the rule-derived labels for all 40 families in an appendix or reporting results under an alternative annotation or labeling rule, so readers can assess how much of the central result depends on the specific rule encoding.","section":"§5; Limitations; Appendix A"}],"minor_comments":[{"comment":"The phrase \"positive Ignorereference class\" is missing a space between \"Ignore\" and \"reference\".","section":"§1"},{"comment":"The caption's explanation of the Holm adjustment is hard to follow: \"Holm adjustment is over the four arm-versus-clean contrasts shown for taxonomy-only and state-output across the two endpoints; the state-output Holm values are unchanged under this expanded family.\" Please restate exactly which hypotheses form the family and what the expanded family contains.","section":"§7; Table 1 caption"},{"comment":"Table 2 uses \"Policy-only\" for a baseline from a different protocol than the matched ablation in Table 1. The text notes this in passing, but the two tables appear side by side and a reader could easily compare their policy-only rows as if they were the same condition. Please add a clear per-table label, such as \"Policy-only (label-conditioning protocol)\".","section":"§7; Table 2"},{"comment":"The text says \"Exact artifact IDs are stored in the raw-result metadata\" and \"Appendix D maps them to reader-facing names,\" but Appendix D lists only paper-facing names and artifact IDs, not machine IDs. Please state explicitly whether machine IDs are in a separate artifact, and include one concrete example mapping if possible.","section":"Appendix D"},{"comment":"The third-endpoint replication reports a large positive delta for llama-3.1-8b-instant but also 18/480 parse failures in the state-output arm, and the text appropriately warns against reading it as a clean gain. The warning is good, but the phrase \"reproduces the direction of the state-output effect\" could still be misleading; consider calling it a direction-only, high-parse-failure sensitivity check in the main text.","section":"Appendix F.2"},{"comment":"The semantic-evidence follow-up reports grouped word TF-IDF decoders reaching 55.6% accuracy and character TF-IDF 62.5%, which are far above the 25.0% majority baseline, and the text dismisses these as \"design constraints.\" Please clarify in one sentence whether these diagnostics indicate residual lexical separability in the semantic-evidence set or whether they are expected given the template structure, since the current phrasing is open to misinterpretation.","section":"Appendix N"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually transparent about execution details, and the audit protocol is a genuine contribution. The main obstacle is the internal mismatch between the abstract's \"isolated explicit state-output field\" claim and the actual state-output-versus-policy-only contrast; this is fixable with a reanalysis of the dissociation contrast and a rewritten abstract. I would also encourage the editor to verify the availability and accessibility of the many 2026-dated references, since several are cited without obvious archival DOIs or stable URLs, though this is a verification request rather than a substantive criticism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper's negative result is probably correct, but the abstract's headline numbers do not test what they claim to test. The primary matched contrast, state-output versus clean policy-only, changes two things at once: it adds the four state definitions and it adds a required state-output field. The isolated field effect is state-output minus taxonomy-only, which Table 4 gives as -8.5 pp for Llama and -1.7 pp for GPT-OSS. The abstract instead reports +0.6 and +3.3, which are the bundled contrasts, and labels them as the isolated field effect. The correctly isolated deltas are negative in direction, so the broader conclusion that explicit state elicitation is not enough survives, but the quantitative claim as written is not the claim that was tested.\n\nWhat the paper does well: it is a genuinely self-refuting audit. The development-set gain from the state-structured bundle is shown to rest on a lexically separable set with no positive Ignore class, and the controlled follow-up uses matched four-way families, frozen arms, parse failures counted as incorrect, family-clustered bootstrap intervals, and sign-flip permutation tests. The label-conditioning diagnostic is interpreted conservatively, as answer-associated sensitivity rather than mechanism evidence. Family-level exact match is a useful lens: example-level accuracy genuinely overstates how often the model tracks the same scenario across all four counterfactual variants.\n\nSoft spots, in proportion: the abstract mismatch is the main one. The frozen set is synthetic and rule-derived, the annotators' agreement partly reflects construction, and the data/code are not released, so the central empirical claim is not independently checkable. That is why my verdict is conditional rather than accept. The taxonomy-only gain is also confounded with definition restatement; the paper concedes this, but the abstract's positive framing leans on it. The endpoint panel changed across phases, and the GPT-OSS semantic-evidence arm was lost to provider validation. The third-endpoint replication has high parse failures and is appropriately demoted.\n\nWho this is for: anyone working on memory-agent routing or prompt-ablation methodology for structured intermediate outputs. It deserves a serious referee, not a desk reject. My recommendation: send it to review, and require the authors to re-run or re-report the isolated state-output contrast (state-output minus taxonomy-only) with cluster intervals, and to release the frozen set and code.","headline":"The core negative finding is likely right, but the abstract's headline numbers conflate taxonomy exposure with the state-output field, so the primary contrast needs re-framing before acceptance.","tokens_in":850,"tokens_out":807,"would_cite":true,"duration_ms":24003,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asking a large language model to emit an explicit memory-state label does not reliably improve its memory-policy decisions, because the benchmark-defined state is hard to predict and maps deterministically to the answer.","keywords":["memory-policy classification","explicit state elicitation","prompt ablation","counterfactual evaluation","LLM agents","shortcut learning","label conditioning","controlled audit"],"falsifier":"An independent re-annotation of a naturalistic (non-synthetic) memory-policy corpus with the same matched four-way family structure that shows an isolated state-output field producing a large, clustered, statistically significant accuracy gain would overturn the central claim. Short of that, a replication where forced-correct-state routing approaches perfect accuracy while naturally emitted state accuracy stays low would localize the bottleneck to state inference rather than the state-to-policy mapping.","tokens_in":24343,"feed_emoji":"🧠","tokens_out":7494,"duration_ms":59379,"temperature":0.7,"pith_summary":"The paper sets out to establish that making a model output an explicit memory-state label (Active, Stale-or-Irrelevant, Conflicting-or-Superseded, Underspecified) is not a reliable way to improve memory-policy routing — the decision of whether retrieved user memory should be Used, Ignored, Updated, or Asked about. Its initial 480-example benchmark suggested large gains from a state-structured prompt, but the paper argues those gains were artifacts of a lexically separable dataset and a bundle of simultaneous prompt changes. On a controlled counterfactual set with matched four-way families, an isolated state-output field yields +0.6 percentage points for Llama-3.3-70B (p=0.81) and +3.3 points for GPT-OSS-120B (Holm-adjusted p=0.14), while merely exposing the state taxonomy gives significant gains of +9.17 and +5.00 points. The paper's broader aim is to promote an audit protocol that separates dataset shortcuts, bundled prompt components, answer-associated labels, and provider failures before crediting any structured intermediate output with an improvement.","feed_headline":"Adding a state-output field does not improve LLM memory routing","feed_subtitle":"Exposing the state taxonomy helps, but an explicit state field yields null or marginal gains.","key_machinery":"The mechanism that carries the argument is the frozen controlled counterfactual set: 160 synthetic examples organized into 40 scenario families, each with four matched variants whose task text is byte-identical and whose reference policies are derived deterministically from structured semantic slots before any natural-language rendering. The matched prompt ablation varies one component at a time (policy-only, taxonomy-only, state-output, state-output-with-rationale, deterministic state routing) so that accuracy differences can be pinned to a single change; the label-conditioning diagnostic then supplies correct or deliberately wrong state labels to test whether routing is sensitive to answer-associated information. These instruments, plus family-cluster statistics, seed-stability checks, and a rule-based error taxonomy, are what let the paper attribute the development-set gain to dataset shortcuts and bundled changes rather than to the state field itself.","core_discovery":"The paper's central claim is a dissociation: telling a model what the four memory states mean helps its policy decisions, but requiring it to emit a state label in its JSON output does not. On the frozen 160-example controlled set, the matched state-output ablation is statistically null for Llama and non-significant for GPT-OSS, and family-level analysis shows that models almost never solve all four counterfactual variants of a scenario, so example-level accuracy overstates consistency. Supplying benchmark-implied state labels shifts routing, but because the four states deterministically map to the four policies, the paper interprets this as label-conditioning sensitivity rather than evidence of a faithful internal state mechanism. The conclusion is that explicit state elicitation requires more than adding a field to the prompt schema, and that apparent prompt gains must survive a controlled audit before being attributed to the intermediate representation.","pith_inferences":["If the result generalizes, the bottleneck for memory agents is the decision mapping from history and task to policy, not information availability; the paper's own normalized-input probe, which made relations explicit and hurt accuracy, points the same direction.","The five-stage audit protocol transfers to any structured intermediate output — rationales, plans, tags — claimed to improve agent behavior; the same shortcut, isolation, and answer-association checks would apply, although the paper instantiates the protocol only on memory-policy classification.","A clean test of the open question would use evidence fields with no deterministic mapping to the final answer; the paper's semantic-evidence follow-up is negative for Llama but unusable for GPT-OSS because of provider-side JSON validation failures, so that condition remains untested.","The forced-wrong-state result suggests a concrete safety design: if a model routes on supplied state, an erroneous upstream state extractor could silently shift policy decisions by roughly 8 points relative to no state at all, arguing for confidence thresholds or abstention on state outputs."],"forward_implications":["Memory-agent builders should not expect an explicit state-output field to improve policy routing; the isolated-field gain is null for Llama and non-significant for GPT-OSS.","Taxonomy exposure alone gives modest but significant accuracy gains, yet because the state wording mirrors the policy wording, the gain is at least partly a restatement effect rather than typed knowledge.","Supplied state labels can shift routing by up to about 12 points, and a conflicting supplied label costs roughly 15 points relative to a correct one, so deployed systems need state-quality checks, abstention, or clarification rather than state fields alone.","The state field redistributes errors between policies — Llama shifts toward Use (+17.5 pp) and away from Ask (-14.2 pp) — so an overall null result can hide opposing class-level effects.","Family-level counterfactual consistency is the right audit unit: complete four-way family success is rare (0% for Llama; at most 5.0% for GPT-OSS), so marginal accuracy improvements are not the same as tracking the intended semantic relation."],"supporting_citations":[{"why":"Supplies the shortcut-learning concept that motivates treating the lexically separable development set as motivation only, not decisive evidence.","marker":"(Geirhos et al., 2020)"},{"why":"Cautions that intermediate labels can affect an answer without proving they reflect an internal decision process, which frames the label-conditioning diagnostic.","marker":"(Matton et al., 2025)"},{"why":"Establishes that explicit intermediate text can change model behavior, motivating the matched isolation of prompt components.","marker":"(Wei et al., 2022)"},{"why":"Documents prompt privilege — semantically equivalent prompt formulations induce performance differences — reinforcing the need to isolate single prompt changes.","marker":"(Jin et al., 2026)"},{"why":"Documents the Llama-3.3-70B endpoint whose state-output ablation is the paper's primary null result.","marker":"(Grattafiori et al., 2024)"},{"why":"Documents the GPT-OSS-120B endpoint used for the frozen controlled-set experiments.","marker":"(OpenAI, 2025)"}],"fun_headline_variants":["State labels don't boost LLM memory policy routing","Explicit state output fails to aid memory policy accuracy","Asking LLMs to state memory policy yields no gain","Memory policy audit: state field null, definitions help","Eliciting state labels doesn't improve LLM routing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rule-derived reference policies on the frozen 160-example set are the correct ground truth for every accuracy number; the paper concedes these labels 'may not capture every reasonable human judgment' and that annotator agreement 'partly reflects the construction,' so if the rules mis-encode the intended abstraction for any family, the null state-output result, the taxonomy gain, and the family-level statistics are all measured against a disputable standard.","fun_headline_variants_meta":{"raw":{"variants":["State labels don't boost LLM memory policy routing","Explicit state output fails to aid memory policy accuracy","Asking LLMs to state memory policy yields no gain","Memory policy audit: state field null, definitions help","Eliciting state labels doesn't improve LLM routing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1287,"prompt_tokens":996,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":213}},"tokens_in":612,"tokens_out":291,"duration_ms":3298,"temperature":1.0,"reasoning_tokens":213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T20:25:47.244277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent re-annotation of a naturalistic (non-synthetic) memory-policy corpus with the same matched four-way family structure that shows an isolated state-output field producing a large, clustered, statistically significant accuracy gain would overturn the central claim. Short of that, a replication where forced-correct-state routing approaches perfect accuracy while naturally emitted state accuracy stays low would localize the bottleneck to state inference rather than the state-to-policy mapping.","supporting_citations":[{"cited_title":"2026 , month = jun, eprint =","cited_arxiv_id":null,"evidence_quote":"Documents prompt privilege — semantically equivalent prompt formulations induce performance differences — reinforcing the need to isolate single prompt changes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Llama-3.3-70B endpoint whose state-output ablation is the paper's primary null result."}],"review_version":1}