{"id":"8cca2744-6320-477d-8ec2-3402f143f1e5","arxiv_id":"2505.22006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EHC, a memory-augmented tool-use agent that retrieves category-specific past experiences, outperforms the CLOVA baseline on GQA, NLVR2, MagicBrush editing, and image tagging without updating model parameters.","lead":"Researchers built an AI agent called EHC that improves visual question answering and image editing by saving past experiences in a fast and a slow memory, and by grouping those experiences into task types. The method works without retraining the underlying language model, which makes it cheap to adapt to new tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SOTA claim is internally contradicted: Table 1 lists CFR at 72.1 on GQA while EHC scores 68.4, so EHC does not outperform all cited methods; the abstract overstates the result.","rationale":"I read the paper in good faith and tried to identify the assumption without which the central claim cannot stand. The reader focused on the seven predefined categories, but the category set is an empirical design choice; a failure there would weaken generalization but would not necessarily falsify the reported benchmark results. By contrast, the abstract's 'state-of-the-art' claim is a logical consequence of Table 1, and the table itself contains a baseline (CFR) with a higher GQA score than EHC. This is not a matter of consensus or external theory; it is an internal inconsistency between the headline and the paper's own numbers. The absence of error bars and code means the numerical comparisons cannot be independently checked, which compounds the issue. I therefore recommend moving the verdict from CONDITIONAL to REJECT: the paper can be revised by scoping the claim and completing the promised experiments, but as submitted the central claim is not supported by the evidence given.","tokens_in":6074,"tokens_out":4334,"duration_ms":46989,"concrete_test":"Locate the original CFR paper ([4]) and extract its reported GQA accuracy and evaluation protocol. If CFR's GQA score is truly 72.1 on the same test split used by EHC, then re-rank Table 1: EHC is not first on GQA, and the abstract's 'state-of-the-art' claim is false unless scoped to tool-use methods. If instead the number in Table 1 is typographical (e.g., 62.1) or from a different split, recompute the comparison and check whether the SOTA claim survives. Also verify the HAMMR/CLOVA duplicate rows against [10] and [14].","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that EHC 'outperforms existing methods, achieving state-of-the-art performance' (abstract). The only quantitative evidence is Table 1. In that table, the end-to-end model CFR (Nguyen et al., CVPR 2022) is listed at 72.1 on GQA, while EHC is 68.4. Therefore EHC is not the best method on GQA among the methods the authors themselves chose to compare. Section 3.2 actually acknowledges this by saying EHC 'substantially narrows the performance gap' to end-to-end models, contradicting the abstract's unqualified SOTA wording. Two further table-level anomalies reinforce the concern: HAMMR and CLOVA are both assigned identical GQA (60.2) and NLVR2 (63.8) numbers, suggesting a copy/paste or duplication issue; and RefCOCO/RefCOCO+/RefCOCOg results promised in Section 3.1 never appear. The category-classification concern identified in the reader's verdict is real but secondary: even a perfectly validated category set would not fix the false SOTA claim. To make the central claim true, 'state-of-the-art' must be scoped to tool-use agents, or CFR's row must be corrected, and the abstract revised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EHC, a training-free multi-modal agent that combines a Hierarchical Memory Retrieval (HMR) module with a Task-Category Oriented Experience Learning (TOEL) module. HMR uses a two-pool memory (fast-access in RAM, deep-retrieval in an external database) with LRU-based migration and top-k retrieval, while TOEL classifies stored experiences into seven predefined categories, generates category-specific insights, and retrieves category-matched trajectories at inference. Experiments on GQA, NLVR2, MagicBrush, and an image-tagging task compare EHC with end-to-end models and tool-use baselines, with ablations on the HMR/TOEL components and a second LLM backbone (Mistral-7B). The central claim is that EHC outperforms existing methods and achieves state-of-the-art performance without any parameter updates.","tokens_in":6319,"tokens_out":3200,"duration_ms":36532,"significance":"If the reported results hold, EHC would be a useful practical contribution: it offers a lightweight, explainable mechanism for continual learning in tool-use agents, with external-benchmark evidence and component ablations that support the direction of the design. The paper also reports results on two LLM backbones, which strengthens the generality claim. However, the absence of code, unreported hyperparameters, missing promised benchmark results, and a SOTA claim contradicted by the paper's own table substantially reduce the present confidence in the headline conclusion. The value of the contribution is better established if the claims are scoped to tool-use methods and the missing details are supplied.","major_comments":[{"comment":"The abstract's statement that EHC 'outperforms existing methods, achieving state-of-the-art performance' is contradicted by Table 1: CFR scores 72.1 on GQA while EHC scores 68.4, and Qwen-vl-chat-7B scores 32.54 on tagging while EHC scores 31.92. Section 3.2 itself only claims that EHC 'substantially narrows the performance gap' to end-to-end models. The SOTA claim must either be scoped to tool-use agents or the comparison rows must be corrected and the abstract revised accordingly.","section":"Abstract / Table 1"},{"comment":"The experimental setup promises results on RefCOCO, RefCOCO+, and RefCOCOg evaluated with IoU, but no such results appear anywhere in the paper, including Table 1. Either the experiments were run and should be reported, or the dataset description should be removed; a claim about factual grounding that is never evaluated cannot support the general-agent claim.","section":"Section 3.1"},{"comment":"Key hyperparameters are undefined in the experiments: the fast-access pool capacity C, the retrieval top-k, the confidence threshold θ, the maximum attempts T, and the initial insight weight and update schedule are all introduced in the method but never given numerical values or a sensitivity analysis in Section 3. This makes the reported numbers impossible to reproduce and leaves open the possibility that the results depend on tuning these values to the four evaluated benchmarks.","section":"Sections 2.1 and 2.2"},{"comment":"The predefined category set is asserted to be 'mutually exclusive and collectively exhaustive' but no classification accuracy, confusion analysis, or sensitivity to the category definitions is provided. Because all memory organization, insight generation, and inference-time retrieval depend on this set, the paper should at minimum report the agreement between LLM-proposed labels and the predefined categories, and ideally test robustness to adding or removing categories.","section":"Section 2.2 / Section 3.1"},{"comment":"HAMMR and CLOVA are listed with identical GQA (60.2) and NLVR2 (63.8) scores, which is either a duplication error or a suspicious coincidence that undermines the baseline comparison. The authors should verify the two rows and report the correct values; the same issue affects Table 3 in the CLOVA rows.","section":"Table 1"},{"comment":"No error bars, standard deviations, or number of seeds are reported for any result. The smallest reported improvement, 0.06 on the tagging task (31.92 vs. 31.86), is well within the range of run-to-run variation for LLM-based agents, so without repeated runs the claimed superiority on that task is not statistically supported.","section":"All tables"}],"minor_comments":[{"comment":"The module name is inconsistent: the abstract and Section 2.2 title call it 'Task-Category Oriented Experience Learning,' while the introduction calls it 'Task-Oriented Experiential Learning' and Section 2.2's first paragraph also uses 'Task-Type Oriented.' This should be unified.","section":"Abstract / Introduction / Section 2.2"},{"comment":"The baseline name 'CLOVA' is inconsistently typeset as 'CLOV A' in multiple places, including the introduction and the caption of Table 3.","section":"References"},{"comment":"The definitions of success and failure experiences are not described in enough detail for replication; in particular, what constitutes a 'success' or 'failure' for each benchmark task is never stated, and the self-reflection prompt is not shown.","section":"Section 2.2"},{"comment":"The qualitative examples in Figure 2 are described only briefly; it would help to state whether these examples were selected randomly or as representative best-case demonstrations.","section":"Section 3.4"},{"comment":"The conclusion claims EHC mitigates 'catastrophic forgetting,' but since the method performs no parameter updates, this concept needs a definition in this setting; otherwise the claim is unclear.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the tool-use comparisons are suggestive, but the SOTA claim is internally contradicted and the missing details (RefCOCO results, hyperparameters, code, error bars) are substantial. The referee report focuses on these fixable issues; I do not see a load-bearing error that would require rejection if the authors scope their claims and supply the missing information."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: EHC is a reasonable engineering integration of known ideas—an LRU dual-pool cache for memory and category-conditioned experiential learning—and the ablations are consistent with the design. But the abstract's state-of-the-art claim is false on its face: Table 1 lists CFR at 72.1 on GQA while EHC scores 68.4. That needs to be fixed before this can be taken at face value.\n\nWhat's new: combining hierarchical memory retrieval with category-specific insight generation is a legitimate contribution, and it's distinct from ExpeL, CLOVA, and HAMMR. The results show solid gains over a CLOVA baseline on GQA and NLVR2 with the same backbone. Giving HMR and TOEL separate credit in the ablations is good practice. The design is sensible and the write-up is mostly clear.\n\nSoft spots: The SOTA overclaim is the big one. The paper acknowledges in Section 3.2 that EHC 'substantially narrows the performance gap' to end-to-end models, so the abstract overstates. Also, HAMMR and CLOVA have identical numbers on GQA and NLVR2, which looks like a copy/paste error. RefCOCO results are promised in 3.1 but never appear. No error bars or seeds, no hyperparameter values (C, theta, top-k, T), no code. The category set is asserted to be mutually exclusive and collectively exhaustive without any measurement of classification accuracy or sensitivity analysis. These are not load-bearing flaws in the architecture, but they prevent the paper from being a validated result as written.\n\nCitation pattern looks fine: the relevant prior work is there. This is a paper for researchers working on memory-augmented tool-use agents; they'll find the integration worth reading. It deserves a serious referee. I'd send it out as a revise-and-resubmit, with the evidence completed, the SOTA claim scoped to tool-use agents, and the table corrected.","headline":"EHC is a plausible integration of memory caching and category-conditioned experiential learning, but its abstract's state-of-the-art claim is contradicted by its own Table 1 and the paper lacks the evidence needed to validate the result.","tokens_in":6867,"tokens_out":1408,"would_cite":false,"duration_ms":14120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen-LLM agent learns from experience by sorting memories into seven task categories.","keywords":["hierarchical memory","task-category learning","tool-use agents","multi-modal agents","in-context learning","continual learning","visual question answering","large language models"],"falsifier":"Run EHC on a held-out suite of instructions chosen explicitly to fall outside the seven categories, such as open-ended summarization or multi-step search, and measure accuracy against CLOVA; if the taxonomy is incomplete, category retrieval supplies the wrong context and the advantage disappears. A second, cheaper check is to measure the cosine-similarity classification accuracy of the LLM-suggested labels on a sample of experiences and confirm the argmax picks the true category.","tokens_in":5836,"feed_emoji":"🧠","tokens_out":4949,"duration_ms":47082,"temperature":0.7,"pith_summary":"The paper introduces EHC, a wrapper around a frozen large language model that learns from its own task experiences without updating any parameters. Its aim is to show that an LLM-based tool-use agent can keep improving across diverse multimodal tasks if memories are organized hierarchically and every experience is classified into one of seven predefined task categories. On compositional visual question answering (GQA), multi-image reasoning (NLVR2), instruction-guided image editing (MagicBrush), and image tagging, EHC reports numbers above previous tool-use baselines and close to end-to-end trained models. The claim matters because it suggests agents can adapt and accumulate knowledge in deployment, avoiding the cost and rigidity of retraining.","feed_headline":"A seven-category memory lets a tool-use agent learn without training","feed_subtitle":"GQA accuracy rises from 60.2 to 68.4 with LLaMA2-7B frozen, no gradient updates needed.","key_machinery":"The Hierarchical Memory Retrieval (HMR) module is a two-pool cache: a fixed-capacity fast-access pool in RAM and a scalable external database, with least-recently-used eviction migrating half the pool when full. The Task-Category Oriented Experience Learning (TOEL) module collects success and failure trajectories through trial-and-error, assigns each experience to one of seven categories by matching a BERT embedding of an LLM-suggested label to category embeddings by cosine similarity, then generates category-level insights from intra-category contrast pairs and cross-category comparison groups. Insights are maintained by ADD, EDIT, UPVOTE, and DOWNVOTE operations with weights that can delete a dead insight. Inference uses the category label plus top-k same-category trajectories as in-context examples in a compositional visual program framework.","core_discovery":"On the paper's own terms, the central discovery is that memory structure and task taxonomy, not parameter updates, are what let an LLM-based agent generalize to new multimodal instructions. EHC stores successful and failed trajectories in a fast-access pool plus an external database, moving entries by recency, and organizes them by seven hand-defined categories: judgment, counting, recognition, comparison, addition, removal, and replacement. At inference time the agent sees only same-category insights and trajectories, which the authors argue prevents cross-task interference and yields better programs. The reported headline numbers are 68.4 on GQA and 68.0 on NLVR2 with LLaMA2-7B, beating the CLOVA baseline by 7.8 and 4.0 accuracy points respectively, with smaller gains on MagicBrush editing and image tagging.","pith_inferences":["A direct test implied but not run: compare EHC against a flat memory store holding the same number of retrieved examples without category separation; if category indexing is the active ingredient, the flat variant should lose most of the gain.","If the seven categories generalize beyond visual editing and reasoning, the same HMR-TOEL loop could be dropped onto embodied or web agents, but those domains would likely require re-deriving the category set from scratch.","The paper reports no measurement of how often the BERT-cosine step assigns the wrong category; misclassified experiences would pollute same-category retrieval and could cap the achievable gain."],"forward_implications":["With the same frozen LLaMA2-7B, GQA accuracy moves from 60.2 to 68.4 and NLVR2 from 63.8 to 68.0 when HMR and TOEL are added to the CLOVA baseline.","The same pipeline also lifts Mistral-7B results on all four benchmarks, suggesting the mechanism transfers across backbone LLMs.","Because no parameter access is required, the method applies to closed or hosted LLMs that expose only prompting interfaces.","The bounded fast pool keeps working memory small while external storage records unlimited history, so the approach is designed for open-ended continual deployment."],"supporting_citations":[{"why":"Supplies the compositional visual program generation framework that EHC adopts for inference.","marker":"[6]"},{"why":"CLOVA is the baseline and public toolkit that EHC builds on; all main comparisons and ablations use it.","marker":"[10]"},{"why":"Introduces the experiential learning loop of collecting, reflecting on, and reusing experiences that TOEL extends with categories.","marker":"[13]"},{"why":"HAMMR is the hierarchical multimodal baseline that motivates category-specialized agents and is compared on GQA and NLVR2.","marker":"[14]"},{"why":"GQA is one of the two main benchmark datasets whose top-1 accuracy drives the headline results.","marker":"[15]"},{"why":"NLVR2 is the multi-image reasoning dataset whose top-1 accuracy drives the headline results.","marker":"[16]"},{"why":"MagicBrush provides the instruction-guided image editing task and CLIP-I scoring used for the editing column.","marker":"[19]"},{"why":"ExoViP is a recent tool-use baseline that EHC compares against across all four benchmark columns.","marker":"[20]"}],"fun_headline_variants":["Memory categories teach agents to learn without any training","Zero-training agent learns via hierarchical memory categories","Task-category memory enables learning without parameter updates","Memory categories lift GQA to 68.4 without any training","Frozen LLM agent learns new tasks via categorized memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The seven predefined categories—judgment, counting, recognition, comparison, addition, removal, replacement—are asserted to be mutually exclusive and collectively exhaustive; if real tasks fall outside them or are misclassified, category-filtered retrieval would return irrelevant experience, and the reported gains could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Memory categories teach agents to learn without any training","Zero-training agent learns via hierarchical memory categories","Task-category memory enables learning without parameter updates","Memory categories lift GQA to 68.4 without any training","Frozen LLM agent learns new tasks via categorized memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001205,"raw_usage":{"total_tokens":4938,"prompt_tokens":889,"completion_tokens":4049,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3973}},"tokens_in":505,"tokens_out":4049,"duration_ms":25992,"temperature":1.0,"reasoning_tokens":3973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:17:51.923837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EHC on a held-out suite of instructions chosen explicitly to fall outside the seven categories, such as open-ended summarization or multi-step search, and measure accuracy against CLOVA; if the taxonomy is incomplete, category retrieval supplies the wrong context and the advantage disappears. A second, cheaper check is to measure the cosine-similarity classification accuracy of the LLM-suggested labels on a sample of experiences and confirm the argmax picks the true category.","supporting_citations":[{"cited_title":"Multi- modal foundation models: From specialists to general- purpose assistants,","cited_arxiv_id":null,"evidence_quote":"Supplies the compositional visual program generation framework that EHC adopts for inference."},{"cited_title":"Clova: A closed-loop visual assistant with tool usage and update,","cited_arxiv_id":null,"evidence_quote":"CLOVA is the baseline and public toolkit that EHC builds on; all main comparisons and ablations use it."},{"cited_title":"Efficiently Enhancing General Agents With Hierarchical-categorical Memory","cited_arxiv_id":"2505.22006","evidence_quote":"Introduces the experiential learning loop of collecting, reflecting on, and reusing experiences that TOEL extends with categories."},{"cited_title":"Gqa: A new dataset for real-world visual reasoning and compositional ques- tion answering,","cited_arxiv_id":null,"evidence_quote":"ExoViP is a recent tool-use baseline that EHC compares against across all four benchmark columns."}],"review_version":1}