{"id":"d3defe76-7c63-42a4-a981-c747c1968bd6","arxiv_id":"2608.01456","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DunphyBench tests long-horizon, preference-driven house selection in virtual homes; MeMento, a preference-conditioned memory compressor, raises VLM agent accuracy by 7.18% and cuts memory by 85.38%.","lead":"This paper introduces DunphyBench, a benchmark where agents tour virtual houses and must pick the one matching a user's preferences over long interactions. Current agents reach 58% accuracy versus 83% for humans, and a new memory compressor improves accuracy by about 7 points while using far fewer tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MeMento's evaluation conditions are unreported and inconsistent with Table 2, so the 7.18% gain may not be measured on the same DunphyBench split.","rationale":"The reader correctly flags thin empirical support and missing appendices, but the most load-bearing issue is more specific: the MeMento experiment's numbers are inconsistent with the main evaluation table, so the reported gain is not anchored to a fixed benchmark split. This is not a question of theoretical framing or preference-grounding semantics; it directly concerns whether the headline 7.18% improvement exists. The method itself is plausible and the motivation (non-monotonic memory scaling) is interesting, but a central quantitative claim that cannot be reproduced from the stated protocol, and whose supporting details appear in an absent appendix, should not be accepted or conditionally accepted as-is. The paper should be marked unverdictable until the authors release the evaluation split, per-instance outputs, and training/eval separation, or otherwise resolve the discrepancy between Table 2 and Figure 5.","tokens_in":10439,"tokens_out":9680,"duration_ms":98041,"concrete_test":"Request the exact instance IDs and per-instance predictions for every row in Figure 5 and Table 2, plus the train/eval split used for MeMento's stage-1 and stage-2 training data. Then recompute the Figure 5 baselines and MeMento accuracy on the same evaluation set that produced Table 2, and compute a bootstrap 95% confidence interval for the MeMento-minus-text-summarization difference. If the baselines change to the Table 2 values, or if the CI includes 0, the headline 7.18% improvement is not supported on the main benchmark.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central claim in §5.6–Abstract is that MeMento improves accuracy by 7.18% absolute over the strongest baseline while reducing memory by 85.38%. However, §5.6.1 says the MeMento experiments use the same protocol as Section 4, but the Qwen3VL-8B baselines in Figure 5 (Memoryless 26%, Text Mem 39%, Multimodal Mem 30%) do not match the corresponding rows in Table 2 (8.3%, 13.9%, 16.7%). The paper never specifies which instances or split produced Figure 5, and the referenced appendix is absent. If Figure 5 was run on a different, easier, or overlapping subset, the claimed improvement over the 'strongest baseline' is not established on DunphyBench. Compounding this, the visible percentages are consistent with a total test set of roughly 36 instances (18 per track), so a 7.18% absolute difference is about 2.6 instances; without per-instance predictions or error bars, the gain could be sampling noise. The central quantitative claim therefore rests on an unreported and internally inconsistent evaluation condition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DunphyBench, a benchmark for long-horizon embodied decision-making in which an agent explores multiple housing environments and selects the one best matching a set of explicit or implicit user preferences. The authors evaluate several VLM-driven agents, report a substantial gap to human performance, and identify memory management as a key bottleneck via a diagnosis showing non-monotonic accuracy as context history grows. They then propose MeMento, a preference-conditioned multimodal memory compressor based on a Perceiver-style module with learned memory queries, trained in two stages (relevance prediction and alignment with a frozen VLM). The central claimed results are that MeMento improves decision accuracy by 7.18% absolute (15.74% relative) over the strongest baseline on DunphyBench while reducing memory usage by 85.38%. The paper also reports ablations showing all components contribute, a memory-probing QA evaluation, and a preliminary transfer result to web-agent tasks.","tokens_in":10698,"tokens_out":3560,"duration_ms":34669,"significance":"If the central claims hold, the paper makes two useful contributions: a reproducible, automatically verifiable benchmark for preference-driven long-horizon embodied decision-making, and a memory-compression method that improves both accuracy and token efficiency for VLM agents. The benchmark's use of deterministic predicates for automatic verification is a genuine strength, as is the explicit separation of explicit and implicit preference tracks. The diagnostic finding that memory scaling is non-monotonic is also valuable. However, the headline quantitative claims rest on an evaluation figure whose numbers are inconsistent with the main benchmark table, on a very small test set, and on unspecified train/test separation for MeMento's training labels. These issues are load-bearing for the paper's main claims and must be resolved before the results can be assessed.","major_comments":[{"comment":"The central claim that MeMento improves accuracy by 7.18% absolute over the strongest baseline is supported only by Figure 5, which reports Qwen3VL-8B baselines of 26% (Memoryless), 39% (Text Mem), and 30% (Multimodal Mem). These values do not match the corresponding Qwen3VL-8B rows in Table 2, which report 8.3%, 13.9%, and 16.7% for the same configurations. Section 5.6.1 states the experiments use 'same protocol as Section 4,' but no split, difficulty tier, or filtering rule is given for Figure 5. If Figure 5 was run on a different, easier, or overlapping subset, the claimed improvement over the 'strongest baseline' is not established on DunphyBench as defined in Section 4.","section":"§5.6.1, Figure 5 vs. Table 2"},{"comment":"The accuracy percentages in Table 2 are consistent with a total test set of approximately 36 instances (18 per track): for example, 8.3% = 3/36 and 58.3% = 21/36. A 7.18% absolute improvement therefore corresponds to roughly 2.6 instances. The paper reports no confidence intervals, significance tests, or per-instance predictions. The headline gain over the strongest baseline may be sampling noise; at minimum, the statistical uncertainty should be quantified before claiming a robust improvement.","section":"§4.1, Table 2"},{"comment":"MeMento is trained with step-level relevance labels, and those labels are constructed from the same preference-predicate definitions used to determine correctness on DunphyBench. The paper does not specify whether MeMento's training data come from the same instances that are subsequently evaluated, or from a separate training split. Since the main experiment is on DunphyBench and Section 5.6.1 only says 'same protocol as Section 4,' the possibility of training on the evaluation instances is not ruled out. The paper must specify the split, the number of training trajectories, and how the relevance-label supervision avoids overlap with the test set.","section":"§5.4, §5.6.1"},{"comment":"The data-generation pipeline requires that each implicit statement 'uniquely imply a single predicate.' This design choice makes automatic verification possible, but it also means the implicit track reduces to a one-to-one natural-language-to-predicate mapping. Under this mapping, 'implicit preference reasoning' may be keyword-to-predicate matching rather than reasoning about underspecified intent. The paper's claims about implicit-preference capability gaps (e.g., Insight 2) should be tempered or validated with a human study showing that the unique mapping is natural and not an artifact of the generation pipeline.","section":"§3.2, Step 5"},{"comment":"The diagnosis that accuracy improves with history up to 10 turns and then declines is reported only as a qualitative statement ('improves accuracy by 17%') with a figure. No error bars, no per-turn counts, and no significance testing are given. Since this diagnosis motivates the entire MeMento method, it should be presented with the same rigor as the main evaluation, including the number of instances per turn-budget condition and confidence intervals.","section":"§4.2.2, Figure 3(right)"}],"minor_comments":[{"comment":"The manuscript contains many unresolved 'Appendix ??' references (e.g., Sections 3.1, 3.2, 4.1, 5.6.1, 6.1, 6.2, 6.3). The appendix is absent in the submitted version, making several setup details unverifiable.","section":"Throughout"},{"comment":"The text refers to 'Table 5' for the memory-probing results, but the results appear only as a table embedded in Figure 5 (Right). This should be a numbered table or the reference should be corrected.","section":"§6.2"},{"comment":"Typo: 'We then encode encode each explored step'.","section":"§5.1"},{"comment":"Typo: 'an relevance label' should be 'a relevance label'; also 'L stage1 = L rel + λ L attn' is missing the numeral and spacing in the displayed equation.","section":"§5.4"},{"comment":"Typo: 'bassline' should be 'baseline'.","section":"§6.3"},{"comment":"The symbols for memory and tool configurations (e.g., /times, /f⌢nt, ⊷, ♂pen) are not legible in a standard PDF and are not defined in the table caption. Standard textual abbreviations or a legend would greatly improve readability.","section":"Table 2"},{"comment":"'the option that best matches their needs' has a pronoun mismatch ('their' referring to singular 'you'); minor wording issue.","section":"§1, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark idea and a sensible method, but the central quantitative claim is currently unsupported because Figure 5 is inconsistent with Table 2 for the same configurations. Please ask the authors to provide the exact evaluation split for Figure 5, reconcile it with Table 2, and report confidence intervals or per-instance results. Also request explicit clarification of the MeMento training/test split with respect to DunphyBench."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the benchmark, DunphyBench, and the question it asks: can an agent navigate several homes, remember what it saw, and pick the one matching a user's partly implicit preferences? That is a real and under-studied capability, and the benchmark design—explicit vs. implicit tracks, distractor tiers, verifiable predicates—is thoughtful. The finding that raw history helps up to ~10 turns and then hurts is exactly the kind of diagnostic that motivates memory compression, and it is credible.\n\nMeMento is a straightforward preference-conditioned Perceiver compressor. Nothing exotic, but the two-stage training (relevance prediction then VLM alignment) is reasonable, and the probing results (F1 0.719 vs oracle 0.816) suggest the tokens actually retain relevant evidence. The web-task transfer is a nice extra, though only 100 tasks.\n\nThe problem is the evidence for the headline gain. Section 5.6.1 says the MeMento experiments use 'the same protocol as Section 4,' but Figure 5's baselines do not match Table 2. Memoryless is 26% in Figure 5 but 8.3% for the same model in Table 2; text memory is 39% vs 13.9%; multimodal memory is 30% vs 16.7%. That is not a rounding error. Either Figure 5 was run on a different, easier subset, or the protocol changed without being reported. Either way, the claimed 7.18% improvement over 'the strongest baseline' is not anchored to the benchmark's own numbers.\n\nEven if that were fixed, the sample is tiny. The percentages in Table 2 are consistent with about 36 instances total (18 per track), so a 7.18% absolute gain is roughly 2.6 instances. There are no confidence intervals or significance tests. The benchmark and code are not released, and the appendix is missing, so none of the data generation or MeMento training details can be checked.\n\nThere is also a mild circularity: MeMento's relevance labels are built from the same predicate definitions that define correctness on DunphyBench, so the compressor is optimized for the benchmark's own vocabulary. That is not fatal, but it limits what the accuracy gain tells us about general preference grounding.\n\nWho is this for? Embodied agents and memory-compression researchers. The benchmark, once released and cleaned up, could be a useful resource. As it stands, the empirical claims are not established. I would send it to peer review—the idea and benchmark deserve referee time—but with a demand that the authors release data/code, reconcile the numbers, and report significance-aware results. A desk reject would be throwing out a good question along with the bad statistics.","headline":"A genuinely useful benchmark idea and a sensible memory compressor, but the main accuracy claim is undermined by an unreported evaluation inconsistency and a tiny test set.","tokens_in":11201,"tokens_out":4006,"would_cite":false,"duration_ms":35314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that long-horizon embodied decision-making fails from too much unfiltered multimodal history, not too little context, and that preference-conditioned compression of memory into fixed tokens improves accuracy by 7.18% while","keywords":["embodied decision-making","multimodal memory","memory compression","vision-language models","preference alignment","long-horizon reasoning","benchmark evaluation","user preferences"],"falsifier":"Run a larger evaluation (e.g., several hundred instances rather than roughly 18 per track) and check whether MeMento's 7.18% absolute accuracy gain over the strongest baseline shrinks to statistical noise. Additionally, replace MeMento with an ordinary retriever that selects preference-relevant steps and uses them as context; if retrieval matches MeMento's accuracy, the learned compression mechanism is not the causal source of the gain.","tokens_in":10332,"feed_emoji":"🏠","tokens_out":5995,"duration_ms":53182,"temperature":0.7,"pith_summary":"The paper is trying to establish that long-horizon, preference-driven embodied decision-making is bottlenecked not by how much history an agent can hold but by how it compresses that history. On a new benchmark, DunphyBench, agents tour multiple homes, accumulate multimodal observations, and pick the home best matching a user's explicit or implicit preferences; the best agent reaches 58.3% accuracy versus 83.3% for humans. Diagnosing failures, the authors find memory management is a key bottleneck: accuracy rises with history up to about 10 turns and then declines as unfiltered context adds noise. Their proposed MeMento compressor conditions a fixed set of memory tokens on user preferences and compresses per-environment step history, improving VLM-driven agent accuracy by 7.18% absolute while reducing memory usage by 85.38% compared with the strongest baseline.","feed_headline":"Trimming 85% of agent memory lifts accuracy by 7.18%","feed_subtitle":"DunphyBench shows preference-aligned agents choke on raw history; compressing memory by preference fixes it.","key_machinery":"The central object is a preference-conditioned Perceiver-style compressor: a fixed set of learnable memory queries $Q_0$, shifted by a projection of a pooled preference embedding $\\bar{z}_r$ to form $Q_r$, cross-attend over step embeddings of each environment separately, then self-attend to produce environment-level memory tokens, which are concatenated and projected into a frozen VLM's embedding space. The mechanism's job is to make the retained history depend on what the user cares about, so irrelevant or redundant multimodal observations are discarded rather than accumulated.","core_discovery":"MeMento is a preference-conditioned multimodal memory compressor. Each environment's step history is encoded into multimodal embeddings, augmented with step and environment identifiers, then compressed by a fixed set of learned queries that have been shifted by a pooled preference embedding. Cross-attention gathers decision-relevant evidence per environment, self-attention fuses across steps, and an adapter injects the compressed tokens into a frozen VLM as a soft prompt. The compressor is trained in two stages: first to predict step-level preference relevance and align cross-attention with relevance labels, then to answer preference-related memory-probe questions so the compressed tokens re","pith_inferences":["We infer that if the non-monotonic memory curve generalizes, the main practical task for embodied assistants shifts from storing more to deciding what to discard; retrieval alone may be insufficient because it still presents potentially noisy context to the model.","We also infer that the reported 7.18% absolute gain rests on a small test set (about 18 instances per track), so the precise magnitude is fragile; a larger sample is needed before treating the improvement as stable.","We further infer that because the benchmark grounds implicit preferences in deterministic predicates, the implicit track may understate the ambiguity of real human preference; richer commonsense inference would be a harder test for MeMento."],"forward_implications":["Agents should manage memory by preference relevance rather than by recency or raw context length.","Because accuracy degrades beyond roughly 10 turns of unfiltered history, simply extending the context window will not solve long-horizon embodied decision-making.","Compressed, environment-separated memory can match or beat much larger multimodal memory at a fraction of the token cost.","The same preference-conditioned compression transfers to web-agent decision tasks, where MeMento reaches 83% success with 512 tokens."],"supporting_citations":[{"why":"Supplies the ProcTHOR household environments and the structured feature vector from which DunphyBench derives deterministic verification predicates.","marker":"[Deitke et al., 2022]"},{"why":"Supplies the embodied web agent environment used to test whether MeMento's compression transfers beyond housing selection.","marker":"[Hong et al., 2025]"},{"why":"UserBench is the prior user-centric benchmark the paper contrasts with DunphyBench on horizon, memory, and preference aggregation.","marker":"[Qian et al., 2025]"},{"why":"Provides the decision-fatigue evidence that motivates the long-horizon decision-making setting.","marker":"[Pignatiello et al., 2020]"}],"fun_headline_variants":["85% memory cut, 7.18% accuracy lift: MeMento wins","Compress memory 85%, improve embodied decisions 7.18%","DunphyBench: Raw history noise hurts, MeMento fixes it","Preference-aware memory compression: 85% cut, 7.18% gain","MeMento: 85% less memory, 7.18% better choices"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that every implicit natural-language preference can be unambiguously mapped to one deterministic predicate over the structured feature vector; if that mapping is artificial, the measured gains may reflect simplified keyword-to-predicate matching rather than human-aligned preference reasoning.","fun_headline_variants_meta":{"raw":{"variants":["85% memory cut, 7.18% accuracy lift: MeMento wins","Compress memory 85%, improve embodied decisions 7.18%","DunphyBench: Raw history noise hurts, MeMento fixes it","Preference-aware memory compression: 85% cut, 7.18% gain","MeMento: 85% less memory, 7.18% better choices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":2905,"prompt_tokens":754,"completion_tokens":2151,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2045}},"tokens_in":498,"tokens_out":2151,"duration_ms":15796,"temperature":1.0,"reasoning_tokens":2045,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:08:07.109203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a larger evaluation (e.g., several hundred instances rather than roughly 18 per track) and check whether MeMento's 7.18% absolute accuracy gain over the strongest baseline shrinks to statistical noise. Additionally, replace MeMento with an ordinary retriever that selects preference-relevant steps and uses them as context; if retrieval matches MeMento's accuracy, the learned compression mechanism is not the causal source of the gain.","supporting_citations":[{"cited_title":"ProcTHOR : Large-scale embodied ai using procedural generation","cited_arxiv_id":null,"evidence_quote":"Supplies the ProcTHOR household environments and the structured feature vector from which DunphyBench derives deterministic verification predicates."},{"cited_title":"Decision fatigue: A conceptual analysis","cited_arxiv_id":null,"evidence_quote":"Provides the decision-fatigue evidence that motivates the long-horizon decision-making setting."}],"review_version":1}