{"id":"d43e70c1-60bd-42cb-92e0-b54616fa7e9c","arxiv_id":"2504.15263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An LLM agent with short-term and long-term memory improved weighted F1 for construction locomotion prediction from 0.73 to 0.90 on a self-collected dataset of 226 multimodal samples.","lead":"A research team built an AI agent that uses a large language model with short-term and long-term memory to predict what a construction worker will do next from smart glasses video and spoken commands. In a private 226-sample test, the full system raised F1 accuracy from 0.73 to 0.90, but no code or dataset was released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Memory gain may stem from previous-label leakage and hard-coded transition rules in the prompt, not from the memory modules.","rationale":"The reader's verdict was CONDITIONAL, flagging the simulated dataset and self-reported scores. I agree those are concerns, but the more fundamental problem is internal: the memory conditions are not comparable to NoMem. The prompt's Safety Analysis section reads like a hard-coded state machine, and STM supplies the state (previous label). For a dataset collected as continuous trials with frequent transitions, this gives the model the answer for many safety-critical and transition cases. A trivial baseline using the previous label and the same rules would likely reproduce much of the gain. That would not mean the system is useless, but it would mean the paper's central claim that memory-driven LLM agents improve locomotion prediction is not demonstrated by the reported ablation. The paper does provide full prompt text and configuration details, which is a strength: it makes the confound identifiable and the proposed test feasible. The claims about calibration (Brier/ECE) are also suspect because the method for obtaining class probabilities from the LLM is not specified, but I did not pursue that as the primary concern. I recommend keeping the reader's conditional verdict: the paper can be considered only if the authors add the rule-based baseline and/or an ablation with the transition rules removed. If the concern is confirmed, the central claim should be withdrawn or substantially revised.","tokens_in":16523,"tokens_out":9070,"duration_ms":84450,"concrete_test":"Run a non-LLM baseline that, for each of the 226 samples, receives the previous true locomotion label and applies the transition rules stated in Figure 2 (e.g., Ladder Up → Ladder Down; LGN as fallback), and compute the same weighted F1. If this baseline matches or exceeds the STMOnly result (0.81) or approaches the STM+LTM result (0.90), then the reported memory gain is explained by injected previous-label information and hard-coded rules rather than by the proposed memory-driven LLM agent. To isolate the rule effect, also rerun STMOnly with the transition rules deleted from the prompt while keeping STM context; if F1 falls toward NoMem (0.73), the rules are the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is internal validity of the ablation. In the memory conditions, STM stores prior locomotion modes (Section 3.3.1) and the Perception prompt (Figure 2) contains explicit deterministic transition rules, e.g. 'If the most recent memory shows the user climbing a construction ladder, they will 100% climb down the same construction ladder, and vice versa' and a list of safe transitions including 'Ladder Up Climbing' to 'Ladder Down Climbing'. Since the 226 samples form a continuous sequence, STM/LTM effectively provide the immediately preceding (usually correct) true label, and the model is instructed to output the rule-designated successor when the command conflicts with the frames — exactly the safety-critical cases the paper highlights (e.g., 'I'm walking forward' atop a ladder is intended for Construction Ladder Down Climbing). The NoMem baseline lacks the previous label, so it cannot exploit these rules. The reported F1 gain (0.73→0.81→0.90) therefore measures the value of adding the previous label plus hand-authored rules, not the value of learned memory representations or LTM retrieval. No rule-based baseline with the same previous-label input and a transition table is reported, no held-out split is used, and LTM accumulates over the same 226 samples being scored. The paper's limitation discussion (Section 6) does not address this confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based locomotion prediction agent for exoskeleton assistance in construction. The system processes spoken commands and nine egocentric FOV frames through a gpt-4o Perception Module with chain-of-thought prompting, a Short-Term Memory (STM) for recent events, a Long-Term Memory (LTM) with vector retrieval, and a Refinement Module that re-prompts the LLM when a self-reported clarity score is low. On a 226-sample dataset with 12 locomotion modes and clear, vague, and safety-critical commands, the authors report weighted F1 of 0.73 (no memory), 0.81 (STM), and 0.90 (STM+LTM), with Brier Score falling from 0.244 to 0.090 and ECE from 0.222 to 0.044. The central claim is that adding memory systems improves prediction accuracy, calibration, and handling of safety-critical commands.","tokens_in":16829,"tokens_out":8885,"duration_ms":81202,"significance":"If the reported effects were robust, this would be a valuable demonstration that memory-augmented LLM agents can address locomotion intent recognition for exoskeletons in safety-critical, unstructured environments, going beyond the flat-ground walking tasks that dominate prior work. The paper deserves credit for publishing the full perception prompt (Figure 2), covering a multi-class dataset that includes ladders, obstacles, and low-space navigation, and reporting calibration metrics alongside F1. However, the central claim is not yet established: the ablation design does not separate the contribution of the memory modules from the contribution of the previous label and the hand-authored transition rules embedded in the prompt, and the evaluation lacks repeated runs and a held-out split. The paper's value is therefore conditional on a substantial revision of the evaluation protocol.","major_comments":[{"comment":"The ablation confounds memory with the availability of the previous label and with deterministic transition rules. In the memory conditions, STM stores the previous locomotion mode (§3.3.1), and the Perception prompt in Figure 2 instructs the model to apply rules such as 'If the most recent memory shows the user climbing a construction ladder, they will 100% climb down the same construction ladder' and to restrict transitions to a listed safe set. The NoMem condition has neither the previous label nor these rules. Since the 226 samples form a continuous sequence (§5.2), STM effectively provides the immediately preceding prediction (usually the correct ground-truth label) for most samples, so the reported F1 gain from 0.73 to 0.81 to 0.90 measures the joint effect of previous-label availability, hand-authored rules, and memory modules rather than the incremental value of memory. Section 6 discusses limitations for vertical ladders and obstacle classes but does not address this confound. The paper needs a deterministic rule-based baseline that receives the same previous label and the same transition table but no LLM or learned retrieval, and an ablation that removes the prompt rules while keeping memory, to establish the central claim.","section":"§4.4, Figure 2, §5.2"},{"comment":"All reported metrics are point estimates from a single pass over 226 samples, with no error bars, no repeated runs, and no held-out split. This is particularly problematic because the Perception Module samples gpt-4o with temperature 0.7 (Table 3), making the results stochastic, and because the paper does not state whether any validation split was used before reporting the configuration in Table 3. The authors should report mean and standard deviation (or confidence intervals) over multiple independent runs, tune hyperparameters on a development set, and evaluate LTM generalization with memory built only from training samples and tested on unseen held-out samples.","section":"§4.2, §4.4, Tables 4–5"},{"comment":"The refinement trigger and LTM ranking are driven entirely by the LLM's self-reported confidence, vagueness, and discrepancy scores, but these scores are never validated against external ground truth or human judgments. The analysis in §5.3 itself shows that confidence is almost non-discriminative between correct and incorrect predictions (means 0.940 vs 0.920), yet confidence still enters the clarity score with weight 0.2 and the LTM composite score with weight 0.15. The claimed calibration improvement (Brier Score 0.244→0.090, ECE 0.222→0.044) is therefore partly self-referential, because the model's own confidence is used both to decide which samples to refine and to compute the calibration metrics. The authors should calibrate the self-reported scores against an independent measure (e.g., human raters or a supervised classifier), or remove the self-referential score from the decision rule.","section":"§3.4, §5.3"},{"comment":"The attribution of error reduction to LTM in Figure 6 is not cleanly supported. The figure reports, for samples that undergo refinement in the full system, a shift from 17 correct/31 incorrect in NoMem to 39 correct/9 incorrect in STM+LTM. But the Refinement Module uses a different prompt at a different temperature with both STM and LTM context, so this comparison does not isolate LTM; moreover, the subset is selected by the full system's clarity threshold, which introduces selection on the outcome. The paper should report a matched comparison (e.g., STM+LTM with and without the Refinement Module) and full test-set confusion matrices for each condition before claiming that LTM 'reduces incorrect predictions by over two-thirds.'","section":"§5.1, Figure 6"}],"minor_comments":[{"comment":"The phrase 'within a environment tailored to simulate' should be 'within an environment'; the paper should also report the number of participants, the data collection protocol, and whether the spoken commands were scripted or spontaneous.","section":"§4.1"},{"comment":"The prompt display contains several garbled bullet characters and irregular sub-bullet formatting; please clean up the figure so that the exact prompt text is unambiguous.","section":"Figure 2"},{"comment":"The paper repeatedly describes the system as 'interpretable', but no human evaluation of the chain-of-thought explanations is reported; the authors should state precisely what interpretability claim they are making and support it.","section":"§1, §6"},{"comment":"The claim that the score distributions 'justify' the clarity and LTM weights is post hoc, because those weights were already used to generate the scores; please rephrase or provide a validation-based justification.","section":"§5.3"},{"comment":"Some references are incomplete as printed; for example, [23] (MemGPT) lacks a publication venue and page numbers, and a few other entries have inconsistent formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially interesting architecture, but the evaluation as designed cannot distinguish the memory modules' contribution from the previous-label-plus-rules confound, and the point estimates from a single stochastic run over 226 samples are not sufficient. I would require the rule-based baseline, a held-out evaluation protocol, and repeated runs before reconsidering the paper; without those, the central claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea—an LLM agent with short- and long-term memory for locomotion prediction from speech plus egocentric vision—is new for the exoskeleton/construction niche, and the system description is clean. But the headline ablation result is not trustworthy: the memory gain is largely explained by previous-label leakage and hard-coded transition rules in the prompt, not by the memory modules learning anything.\n\nWhat the paper does well: the architecture is clearly laid out, related work on memory-augmented LLM agents is reasonably covered, and the decision to use the LLM's own vagueness/discrepancy/confidence scores to trigger refinement is a pragmatic idea. The class-level error analysis is honest about weak spots like vertical ladder and obstacle classes.\n\nThe soft spot is load-bearing. The perception prompt (Figure 2) includes deterministic rules: if the most recent memory says the user is climbing a construction ladder, the model must output ladder-down next; it also lists allowed safe transitions. Since the 226 samples are a continuous recording, STM gives the model the immediately preceding (usually correct) true label. NoMem doesn't have that, so part of the reported gain is just having the previous label plus a transition table. The paper omits the obvious control: a rule-based baseline that gets the same previous label and the same transition rules. It also lacks a held-out split for LTM (which is tested on the same samples it stores), error bars, and any external validation of the self-reported scores. The calibration improvement may be a side effect of the '100%' language in the prompt.\n\nThese issues don't make the concept worthless, but they mean the 0.73→0.90 F1 story is not established. The right next step is a revised evaluation with a rule-based baseline, a true train/test separation for LTM, and interval estimates. Who should read it: people working on exoskeleton intent recognition or LLM agents for physical tasks will find the application interesting, but should treat the numbers with caution. I'd still send this to a serious referee, because the area is relevant and the architecture is reusable, but with a strong request to fix the evaluation before acceptance.","headline":"A useful application idea undermined by an ablation that lets the previous label leak through hard-coded transition rules; the 0.73-to-0.90 F1 story doesn't hold up as stated.","tokens_in":17281,"tokens_out":3048,"would_cite":false,"duration_ms":27620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding short-term and long-term memory to a language-model agent raises its ability to predict construction locomotion from spoken commands and smart-glasses video, lifting weighted F1 from 0.73 to 0.90 and sharply improving confidence…","keywords":["locomotion prediction","exoskeleton control","intent recognition","LLM agents","short-term memory","long-term memory","chain-of-thought reasoning","multimodal perception"],"falsifier":"Collect a larger dataset of real construction workers performing the same twelve locomotion modes on actual sites, with independent ground-truth labels and human-annotated command-scene agreement, then compare the memory agent's F1 and calibration under the same configuration. If the discrepancy score stops separating correct from incorrect predictions, or if a no-memory baseline matches the memory agent when the command set is realistic rather than simulated, the central claim would be contradicted.","tokens_in":16336,"feed_emoji":"🏗️","tokens_out":7376,"duration_ms":59257,"temperature":0.7,"pith_summary":"This paper claims that a locomotion prediction agent built around a large language model can predict a construction worker's movement mode (walking, climbing a ladder, stepping over an obstacle, etc.) from a spoken command and nine smart-glasses frames, and that the prediction becomes substantially better when the agent is given two kinds of memory: a short-term buffer of the most recent events and a long-term store of similar past situations. On a 226-sample dataset mixing clear, vague, and safety-critical commands, the authors report weighted F1 rising from 0.73 without memory to 0.81 with short-term memory and 0.90 with both memory systems. They also report that the model's self-assessed confidence becomes much better calibrated, with Brier score falling from 0.244 to 0.090 and expected calibration error from 0.222 to 0.044. The practical motivation is exoskeleton control: a wearable robot that knows whether the user is about to ascend a ladder or walk on level ground can assist at the right moment and avoid unsafe actions.","feed_headline":"Memory lifts construction-move prediction from 0.73 to 0.90 F1","feed_subtitle":"Short-term and long-term memory plus chain-of-thought reasoning sharply improve intent recognition and confidence for exoskeleton control.","key_machinery":"The load-bearing mechanism is a clarity-gated refinement loop with discrepancy-weighted retrieval. The perception module first produces a prediction plus four self-reported scores: vagueness, discrepancy, importance, and confidence. A weighted clarity score decides whether the prediction is accepted or sent to a refinement module that re-prompts the model with memories retrieved from the long-term store. The retrieval itself is score-driven: text and image embeddings are blended with weights $1-d$ and $d$, where $d$ is the discrepancy between the spoken command and the visual frames, so that when words and vision conflict, the visual evidence dominates. Retrieved events are ranked by a composite score that prioritizes similarity, importance, and confidence while penalizing vagueness and discrepancy, with safety-critical events decaying more slowly and being penalized less. Short-term memory supplies the immediate safety context, including explicit rules about safe locomotion transitions, such as allowing a ladder descent only after a ladder ascent. Together these components turn the language model from an open-loop classifier into one that reconsiders its answer when its own scores say the input is ambiguous.","core_discovery":"The central discovery is that memory—not more powerful perception—accounts for most of the accuracy gain in this agent. Without memory, the perception module alone, using chain-of-thought prompting, reaches a weighted F1 of only 0.73 on the 226-sample test set. Injecting short-term memory of recent locomotion events (retained for 45 seconds) raises this to 0.81, and adding long-term memory of similar past experiences retrieved from a vector store raises it further to 0.90. The memory modules also drive the calibration improvements: the Brier score drops from 0.244 to 0.090 and ECE from 0.222 to 0.044. The authors further find that on the subset of ambiguous samples that trigger refinement, long-term memory cuts incorrect predictions by more than two-thirds, from 31 to 9 errors, and that safety-critical commands, where the user's words conflict with the visual scene, improve from F1 0.38 without memory to 0.72 with both memories.","pith_inferences":["The same design could be tested on other intent-prediction tasks where users give terse or misleading commands—assistive wheelchairs, teleoperation, or surgical robots—but the authors do not claim this extension.","A natural testable check is whether the improvement survives when retrieved memories are replaced with random past events matched for class and time: if random retrieval performs as well, the benefit would come from the refinement prompt rather than from genuinely similar history.","The discrepancy score is doing double duty (gating refinement, weighting text versus image embeddings, and penalizing LTM ranking); a hidden risk is that if the LLM's discrepancy estimate is biased, all three mechanisms skew together. That is not examined in the paper.","One could ablate the safety-transition rules in STM separately from the memory content; the rules encode domain knowledge, so removing them might separate the contribution of stored events from the contribution of the rule book."],"forward_implications":["On the paper's own ablation, every reported performance gain comes from memory: F1 rises 0.73→0.81 with short-term memory and 0.81→0.90 with long-term memory, so memory is the operative ingredient rather than the base LLM.","Calibration metrics move in the same direction, so the agent not only predicts more often correctly but its stated confidence tracks its accuracy more tightly—a property that matters for safety systems that act on probabilities.","The biggest improvements are on safety-critical commands, where the command contradicts the visual scene; the refinement module corrects most of these errors when long-term memory provides relevant similar events.","Some classes remain hard: vertical ladder climbing (F1 0.62–0.67) and obstacle stepping (0.86–0.92) still confuse with level-ground navigation, especially under vague or safety-critical commands.","The agent can override a misleading command (e.g., saying 'walk' while at the top of a ladder) by combining visual evidence with the recent context held in short-term memory."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting technique the perception module uses to reason step by step.","marker":"[40]"},{"why":"The survey motivates memory integration in LLM agents, grounding the design choice to augment the model with STM and LTM.","marker":"[37]"},{"why":"Introduces episodic memory buffers with verbal self-reflections, the pattern behind the agent's short-term record of recent events.","marker":"[31]"},{"why":"Shows a long-term memory stream of natural-language experiences driving agent behavior, which the LTM store mirrors.","marker":"[24]"},{"why":"Demonstrates paging context between a fixed window and external storage, analogous to LTM vector retrieval.","marker":"[23]"},{"why":"Provides a long-term memory repository with event summaries and user portraits, informing the LTM's event summaries and refinement context.","marker":"[45]"},{"why":"Establishes the multimodal vision-language capability that lets the LLM interpret FOV frames and commands together.","marker":"[22]"},{"why":"Prior work fusing speech commands with exoskeleton motion recognition, the baseline this agent extends to multimodal LLM-based prediction.","marker":"[7]"}],"fun_headline_variants":["Memory boosts exoskeleton intent F1 from 0.73 to 0.90","LLM agent: memory, not perception, raises locomotion F1 to 0.90","Construction-move prediction jumps 17 points with memory modules","Short-term plus long-term memory lift exoskeleton intent to 0.90 F1","Memory lifts safety-critical F1 from 0.38 to 0.72"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that the 226 hand-collected samples, with simulated commands issued in a controlled environment, genuinely represent the variety and noise of real construction locomotion, and that the language model's self-reported vagueness, discrepancy, and confidence scores are accurate enough to drive retrieval and refinement decisions.","fun_headline_variants_meta":{"raw":{"variants":["Memory boosts exoskeleton intent F1 from 0.73 to 0.90","LLM agent: memory, not perception, raises locomotion F1 to 0.90","Construction-move prediction jumps 17 points with memory modules","Short-term plus long-term memory lift exoskeleton intent to 0.90 F1","Memory lifts safety-critical F1 from 0.38 to 0.72"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3666,"prompt_tokens":964,"completion_tokens":2702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2595}},"tokens_in":580,"tokens_out":2702,"duration_ms":16893,"temperature":1.0,"reasoning_tokens":2595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:29:01.781298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a larger dataset of real construction workers performing the same twelve locomotion modes on actual sites, with independent ground-truth labels and human-annotated command-scene agreement, then compare the memory agent's F1 and calibration under the same configuration. If the discrepancy score stops separating correct from incorrect predictions, or if a no-memory baseline matches the memory agent when the command set is realistic rather than simulated, the central claim would be contradicted.","supporting_citations":[{"cited_title":"A survey on large language model based autonomous agents","cited_arxiv_id":null,"evidence_quote":"The survey motivates memory integration in LLM agents, grounding the design choice to augment the model with STM and LTM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces episodic memory buffers with verbal self-reflections, the pattern behind the agent's short-term record of recent events."},{"cited_title":"Memgpt: Towards llms as operating systems","cited_arxiv_id":null,"evidence_quote":"Demonstrates paging context between a fixed window and external storage, analogous to LTM vector retrieval."},{"cited_title":"Memorybank: Enhancing largelanguagemodelswithlong-termmemory","cited_arxiv_id":null,"evidence_quote":"Provides a long-term memory repository with event summaries and user portraits, informing the LTM's event summaries and refinement context."},{"cited_title":"Gpt-4v(ision) system card, 2023","cited_arxiv_id":null,"evidence_quote":"Establishes the multimodal vision-language capability that lets the LLM interpret FOV frames and commands together."},{"cited_title":"Speech-based human-exoskeleton interaction for lower limb motion planning","cited_arxiv_id":null,"evidence_quote":"Prior work fusing speech commands with exoskeleton motion recognition, the baseline this agent extends to multimodal LLM-based prediction."}],"review_version":1}