Pith. sign in

REVIEW 2 major objections 5 minor 30 references

General-purpose memory systems attached to a BIM agent retain too little of the project knowledge discussed in earlier sessions for reliable IFC-grounded retrieval, with the best system answering only 32.4% of probes accurately under deploy

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:57 UTC pith:672CWNAJ

load-bearing objection Useful benchmark construction, honestly reported, but the headline 32.4% is conditional on a synthetic distribution where only user turns carry durable facts — and without a public artifact it is not yet a benchmark you can build on. the 2 major comments →

arxiv 2607.26072 v1 pith:672CWNAJ submitted 2026-07-13 cs.IR cs.AI

IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval

Changyu Du , Alexander Vosseler , Filippo Mazza , Andr\'e Borrmann This is my paper
classification cs.IR cs.AI
keywords Agent memoryLLM agentsBuilding Information ModellingIFCinformation retrievalmemory benchmarkmulti-session tasksdomain transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that long-term memory for LLM agents should be tested where it matters most: a professional workflow in which the agent must combine remembered context with live queries over a structured domain model. To do this it builds IFCMemoryBench, converting incomplete-information questions from IFC-Bench v2 into 143 multi-session tasks with 4,016 prior sessions, and decomposes memory performance into ingestion, retrieval, and utilization. Across vector-, graph-, and file-based memory systems, the strongest achieves only 32.4% answer accuracy under a deployment-realistic ingestion scope, and stays below 60% even with oracle-filtered ingestion or a stronger probe agent. The failure profile is consistent: systems retrieve topically relevant context but store project knowledge as incomplete or fragmented facts, so the bottleneck lies in memory representation and coverage rather than retrieval relevance or final use. The paper concludes that reliable professional agents need domain-aware memory representations that link conversations, project knowledge, and structured model entities.

Core claim

On the paper's own terms, the central finding is a domain-transfer gap: general-purpose memory systems, when attached to a fixed ReAct BIM retrieval agent, preserve too little of the project knowledge needed to answer IFC-grounded questions. In the main comparison, the best system (a vector memory following the Mem0 pattern) reached 32.4% answer accuracy under deployment-realistic ingestion (user messages plus the assistant's final answer), with answer correctness at 44.4% and completeness at 33.1%. The memory judge shows the system retrieves topically relevant content 90.1% of the time but covers the key facts only 49.3% of the time, locating the failure in what is written to memory and how

What carries the argument

The benchmark itself is the mechanism. Each task begins from a Category 4 (incomplete-information) question in IFC-Bench v2. A pipeline extracts the IFC-side information the question needs, labels the question by its dependence on outside facts, synthesises the missing project context and a gold answer, and scatters that context across 25-40 synthetic prior chat sessions, replayed with a real ReAct agent and IFC query tool so the conversation includes assistant and tool turns. At probe time the agent must answer a paraphrased version of the original question using both the attached memory store and live ifcquery tool calls. Performance is decomposed into ingestion, retrieval, and utilization

Load-bearing premise

The results depend on the benchmark seeding all durable project facts exclusively in user messages; if real BIM conversations distribute such facts across assistant and tool turns too, the measured accuracy and the ranking of memory systems could change substantially.

What would settle it

Run a variant of IFCMemoryBench that plants the hidden facts in assistant replies and tool outputs as well as user messages, using the same tasks and systems; if vector or graph memories then reach accuracy comparable to the user-only oracle, the claim that the bottleneck lies in memory representation rather than ingestion scope would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, deploying current general-purpose memory systems in agentic BIM workflows would lose most of the project knowledge needed for measurement-based questions, so professional agents need memory representations that store exact facts, units, and IFC-entity references, not conversation summaries.
  • Evaluations of agent memory should report ingestion, retrieval, and utilization separately, since a high retrieval-relevance score can mask low fact coverage.
  • The probe LLM must be treated as a first-class experimental factor, since the ranking of memory architectures can reverse when the agent changes.
  • A full-context oracle gives a costed upper bound rather than a practical baseline: memory systems trade a one-time ingestion cost for lower per-query context, and the benchmark scales to arbitrarily large histories beyond any context window.
  • The user-only ingestion result (above 50% accuracy) suggests that if deployments could pre-filter which turns carry durable facts, the gap would narrow substantially; this is a design lever, not a deployed condition.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since the paper seeds durable facts only in user turns, the 32.4% figure is partly a benchmark-construction effect; in real workflows where assistant outputs (computed quantities, query routines) also carry memorable content, ingestion-scope comparisons and system rankings could shift materially.
  • The relevance–coverage gap the paper measures is likely to appear in other professional domains that pair conversational memory with structured artifacts (e.g., legal dossiers, medical records, engineering drawings), suggesting the proposed decomposition generalizes as a diagnostic.
  • A testable extension: planting the same hidden facts in assistant and tool turns and measuring whether domain-customised memory (the cited Markdown variant) retains its advantage over the vector store would directly test the claim that representation, not retrieval, is the bottleneck.
  • The full-turn ingestion collapse (accuracy dropping to 16.5% at 167k tokens) suggests that selective memory is not optional as histories grow; long-context prompting degrades with noise, so the practical question is how to learn what to write, which the benchmark can now measure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces IFCMemoryBench, a benchmark for evaluating long-term memory in LLM-based agents performing BIM information retrieval. The benchmark converts 143 Category 4 questions from IFC-Bench v2 into multi-session memory tasks spanning 19 projects and 4,016 prior chat sessions; each task seeds project facts in earlier sessions and asks a probe that requires combining remembered context with live IFC queries. The evaluation framework decomposes memory into ingestion, retrieval, and utilization, using expert-validated LLM judges to measure answer accuracy and memory accuracy. The paper evaluates vector (Mem0), graph (Graphiti), and file (Markdown) memory systems on a fixed ReAct agent, finding that the best system reaches 32.4% answer accuracy under the 'deployment-realistic' ingestion scope (user messages + assistant final answer), with memory coverage as the main bottleneck. Oracle conditions reach 83.2% when all prior user messages are exposed. The paper also studies the effect of ingestion scope, a light domain customisation, and the probe-agent LLM.

Significance. If the results hold, the benchmark fills a gap by testing memory in a tool-grounded professional workflow rather than open-domain conversational recall. The decomposition into ingestion/retrieval/utilization and the use of no-memory and full-context oracle controls provide useful upper and lower bounds. The paper is also unusually transparent: it reports Wilson confidence intervals, validates the judges against a domain expert (95% agreement, κ=0.90 for answer correctness; 100% for memory correctness), and includes full system prompts in the appendix, supporting reproducibility. The paired Bayesian analysis for the probe-LLM comparison is methodologically sound. The main weakness is that the synthetic conversations seed durable facts only in user messages, which is acknowledged by the authors but potentially limits the generality of the headline performance claims.

major comments (2)
  1. [Sec. 3.2, 4.4, 5; Table 3] The benchmark seeds durable project facts only in user messages; assistant and tool turns are generated by replaying the user messages and cannot add new durable facts. Consequently, the finding that user-messages-only ingestion gives the best results (e.g., 54.4% for Markdown cited, Table 3) and the headline 32.4% under the 'deployment-realistic' scope are partly an artifact of this construction. The paper concedes this in Sec. 4.4 and Sec. 5, but the abstract presents 32.4% as a 'deployment-realistic' result without this caveat. In real BIM workflows, assistant replies and tool outputs often carry the exact computed quantities or query routines that need to be remembered; a memory system optimized for such content could change the ranking. Please either add a robustness experiment that seeds durable facts in assistant/tool turns (even a small-scale subset) to show the ranking and bottl
  2. [Sec. 4.2; Sec. 5] All LLM roles (probe agent, memory-system components, and the answer/memory judges) use Grok-4.3. The paper acknowledges the resulting self-preference risk and argues it is uniform across systems, which addresses relative rankings, but the absolute accuracy figures (e.g., the 'below 60%' claim) may still be affected. In particular, the 'stronger probe agent' comparison in Sec. 4.7 uses a Grok judge to score Gemini outputs, which could disadvantage Gemini if the judge favors its own family. The human validation on 40 tasks is reassuring, but a cross-model judge (e.g., a second LLM or human spot-check on the Gemini runs) would make the absolute numbers more robust. At minimum, the abstract should state that all scores come from a single-model judge.
minor comments (5)
  1. [Abstract; Sec. 4.4] The phrase 'oracle-filtered ingestion' is ambiguous; the user-messages-only condition still requires memory extraction and is not a true oracle. The actual oracle is the full-context condition (83.2% in Sec. 4.5). Consider renaming to 'near-oracle user-only ingestion' or similar.
  2. [Sec. 3.3] The 'human validation' is performed by a single domain expert on 40 tasks; 'expert-validated' may overstate consensus. Please clarify that it is one expert and report the exact sample composition and selection procedure.
  3. [Table 3] 'Full turns was run only for the file memory' should read 'Full turns were run...'; also consider noting that the full-turn condition for other systems was omitted for budget reasons, which is acceptable but should be stated in the main text.
  4. [Sec. 4.4] The 180x token increase from user-messages to full turns is reported; it would be helpful to also report retrieval latency or end-to-end query latency, not just ingestion time.
  5. [Fig. 2] The x-axis sums probe-time tokens over all LLM calls; this conflates context length and call count. A per-call token budget or a note on the number of calls would clarify the cost comparison.

Circularity Check

1 steps flagged

Headline ingestion-scope result is partly a benchmark-construction artifact; paper discloses this, so the central performance claim is a measurement with caveats, not a fitted-parameter circularity.

specific steps
  1. self definitional [Section 3.2 (Dataset construction, 'Generating realistic prior chat sessions') and Section 4.4 (Effect of ingestion scope), Table 3]
    "durable facts are seeded only in user messages; assistant and tool turns do not introduce additional durable facts. Wider scopes therefore add mostly distractors... This finding is partly a consequence of the benchmark design, and should therefore be read with that limitation in mind."

    The headline comparison 'user-messages-only ingestion outperforms user+final-answer ingestion' (Table 3: Mem0 52.4% vs 32.4%; Markdown 54.4% vs 24.5%) is determined in large part by the construction rule that all durable facts are planted only in user turns. The ingestion-scope result then follows by construction: including assistant/tool turns can only add noise. The paper explicitly concedes this in Sec. 4.4 and Sec. 5. This does not invalidate the benchmark's other findings, but it makes the scope-ordering claim a definitional consequence rather than an empirical discovery about memory systems.

full rationale

IFCMemoryBench is an empirical benchmark paper, not a derivation, so the classic fitted-parameter circularity patterns do not apply. No parameter is fit to a dataset and then re-predicted; the full-context oracle (83.2% answer accuracy) independently verifies that the tasks are answerable from seeded context, and the scores are measured against expert-validated gold answers and judges. There is no self-citation chain that forces the paper's conclusions: the only author-overlapping citations are BIM retrieval systems used as data sources, not as evidence for the benchmark's claims. The one notable circularity is design-based: the ingestion-scope result (user-only > user+final, and the associated finding that assistant/tool turns are mostly distractors) is partly enforced by Sec. 3.2's construction rule that durable facts are planted only in user messages. The paper itself flags this in Sec. 4.4 ('This finding is partly a consequence of the benchmark design') and Sec. 5 ('assistant replies and tool outputs can themselves carry memorable content... A future version of the benchmark will therefore seed durable facts into assistant and tool turns'). Because the limitation is acknowledged and the central bottleneck finding (R-cov 24-49%, answer accuracy below 60% even under oracle ingestion) does not depend on the scope ordering, I score 4 rather than higher. The headline 32.4% deployment-realistic figure is still a genuine measurement under a stated deployment assumption, with the caveat that the assumption (facts live in user turns) is favorable to user-only ingestion by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on few free parameters: the headline numbers are measurements, not fits. The hand-chosen settings are the answer-judge's 2% numeric tolerance, the 25-40 session range, and unreported retrieval top-k defaults. The axioms are dominated by domain assumptions about the source data (IFC-Bench v2 validity), the synthetic-session design (durable facts in user turns only, conceded as an ecological-validity limit in Sec 4.4/5), judge reliability (one expert on 40 tasks), and the replay agent's leak-freedom. No invented entities.

free parameters (3)
  • Answer-judge numeric tolerance = ±2% of gold value
    Appendix B.1 answer-judge rubric accepts numeric answers within about 2% of the gold value; the paper notes one of the two expert-judge disagreements was a rounding/unit-precision boundary case, so this threshold directly gates correctness labels.
  • Prior-session count per task = 25–40 sessions
    Sec 3.2 fixes 25-40 seeded chat sessions per task; task difficulty scales with the number of sessions and distractor density, so this range is a hand-chosen difficulty control.
  • Memory retrieval top-k limits = unspecified system defaults
    Sec 4.2 runs Mem0 and Graphiti at 'standard default settings' without reporting the retrieval limits exposed through search_project_memory; retrieval-coverage rates depend on these limits.
axioms (5)
  • domain assumption IFC-Bench v2 Category 4 questions, reference answers, and IFC models are correct and representative
    Sec 3.2 builds all 143 tasks from IFC-Bench v2 [9]; source-question errors propagate into IFCMemoryBench, and the paper does not independently re-validate the source questions.
  • ad hoc to paper Durable project facts appear in user chat messages and not in assistant or tool turns
    Sec 3.2 seeds hidden facts only in user messages; Sec 4.4 and Sec 5 concede this makes 'user-only ingestion is best' partly an artifact of construction and limits ecological validity.
  • domain assumption Grok-4.3 as LLM judge approximates expert judgment after 38/40 (answer) and 40/40 (memory) agreement with one expert
    Sec 3.3 validation uses one expert on 40 tasks; the remaining 103 tasks are judged without expert adjudication, and the same model family serves as probe, memory components, and judge (Sec 5 self-preference exposure).
  • domain assumption Session replay enforces no information leakage
    Sec 3.2 restricts the replay agent to the current session only; leakage prevention depends on prompt design plus human review, with no automated leakage audit reported.
  • standard math Wilson score intervals and the Pólya–Gamma Bayesian paired test are appropriate statistical models for the reported rates
    Sec 4.3 and Sec 4.7 use Wilson intervals [17] and a Bayesian paired-proportions sampler [16]; standard methods applied appropriately.

pith-pipeline@v1.3.0-alltime-deepseek · 18745 in / 18397 out tokens · 156391 ms · 2026-08-02T06:57:15.122383+00:00 · methodology

0 comments
read the original abstract

Long-term memory is becoming a core capability of LLM-based agents, but existing evaluations largely test conversational recall in open-domain or persona-grounded settings. We argue that a stronger test is whether an agent can reuse information from prior sessions while acting over a live, structured, domain-specific environment. We study this problem in Building Information Modelling (BIM), a professional engineering workflow where agents must query large IFC models while also relying on project specifications, client decisions, and engineering conventions often discussed in conversation but absent from the model. We introduce IFCMemoryBench, a benchmark for evaluating long-term memory in LLM-based BIM information retrieval. IFCMemoryBench contains 143 multi-session tasks across 19 projects and 4,016 prior sessions, derived from incomplete-information questions in IFC-Bench v2. Each task seeds missing project context across earlier conversations and later asks a probe question that can be answered only by combining remembered context with live IFC queries. Our evaluation framework decomposes memory performance into ingestion, retrieval, and utilization, and measures both answer quality and memory quality with expert-validated LLM judges. We evaluate representative vector-, graph-, and file-based memory systems. The strongest system achieves only 32.4% answer accuracy under a deployment-realistic ingestion scope, and remains below 60% under oracle-filtered ingestion or a stronger probe agent. Analysis shows that current general-purpose memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts. These results reveal a domain-transfer gap in agent memory and suggest that reliable professional agents require domain-aware memory representations linking conversations, project knowledge, and structured model entities.

Figures

Figures reproduced from arXiv: 2607.26072 by Alexander Vosseler, Andr\'e Borrmann, Changyu Du, Filippo Mazza.

Figure 1
Figure 1. Figure 1: IFCMemoryBench construction pipeline. An IFC-Bench v2 Category 4 question is turned into a memory-dependent [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cost–quality trade-off between full-context prompt [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of domain customisation. contain, narrows the coverage gap without changing the archi￾tecture or retrieval. Holding the agent, judges, model, retrieval and ingestion scope (user + final answer) fixed, we apply one min￾imal customisation per system: for Mem0, a domain instruction in its fact-extraction prompt that keeps exact quantities, units and element or material names as self-contained facts; fo… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of the probe-agent LLM, with memory held [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Anthropic. 2025. Claude Code. https://www.anthropic.com/claude-code. Com- puter software

    2025

  2. [2]

    Chhikara, D

    P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. 2025. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.arXiv preprint arXiv:2504.19413(2025). doi:10.48550/arXiv.2504.19413

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.19413 2025

  3. [3]

    Z. Deng, C. Du, S. Nousias, and A. Borrmann. 2025. BIMgent: Towards Autonomous Building Modeling via Computer-Use Agents. InWorkshop on Computer-Use Agents at ICML. doi:10.48550/arXiv.2506.07217

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.07217 2025

  4. [4]

    C. Du, S. Esser, S. Nousias, and A. Borrmann. 2026. Text2BIM: Generating Building Models Using a Large Language Model-Based Multiagent Framework.Journal of Computing in Civil Engineering40, 2 (2026), 04025142. doi:10.1061/JCCEE5. CPENG-6386

    doi:10.1061/jccee5 2026

  5. [5]

    C. Du, S. Nousias, and A. Borrmann. 2024. Towards a Copilot in BIM Authoring Tool Using Large Language Model Based Agent for Intelligent Human-Machine Interaction. InProceedings of the 31st International Workshop on Intelligent Com- puting in Engineering (EG-ICE). https://mediatum.ub.tum.de/node?id=1743921

    2024

  6. [6]

    Y. Gao, F. Hu, C. Chai, Y. Weng, and H. Li. 2026. Multi-Agent Framework for Schema-Guided Reasoning and Tool-Augmented Interaction with IFC Models. Automation in Construction186 (2026), 106888. doi:10.1016/j.autcon.2026.106888

    arXiv 2026

  7. [7]

    P. Guo, H. Xue, J. Ma, and J. C. P. Cheng. 2025. Advancing BIM Information Retrieval with an LLM-Based Query-Domain-Specific Language and Library Code Function Alignment System.Automation in Construction178 (2025), 106374. doi:10.1016/j.autcon.2025.106374

    arXiv 2025

  8. [8]

    Z. He, Y. Wang, C. Zhi, Y. Hu, T.-P. Chen, L. Yin, Z. Chen, T. A. Wu, S. Ouyang, Z. Wang, J. Pei, J. McAuley, Y. Choi, and A. Pentland. 2026. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. arXiv preprint arXiv:2602.16313(2026). doi:10.48550/arXiv.2602.16313

    doi:10.48550/arxiv.2602.16313 2026

  9. [9]

    BIM Information Extraction Through LLM-based Adaptive Exploration

    S. Hellin, S. Jang, S. Fuchs, S. Nousias, and A. Borrmann. 2026. BIM Infor- mation Extraction through LLM-Based Adaptive Exploration.arXiv preprint arXiv:2605.01698(2026). doi:10.48550/arXiv.2605.01698

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.01698 2026

  10. [10]

    Hellin, S

    S. Hellin, S. Nousias, and A. Borrmann. 2025. Natural Language Information Retrieval from BIM Models: An LLM-Based Agentic Workflow Approach. In Proceedings of the European Conference on Computing in Construction (EC3). doi:10. 35490/EC3.2025.265

    2025

  11. [11]

    Y. Hu, Y. Wang, and J. McAuley. 2026. MemoryAgentBench: Evaluating Memory in LLM Agents via Multi-Turn Interactions. InProceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2507.05257

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.05257 2026

  12. [12]

    IfcOpenShell. 2025. IfcOpenShell: The Open Source IFC Toolkit and Geometry Engine. https://ifcopenshell.org/. Computer software

    2025

  13. [13]

    LangChain. 2025. DeepAgents. https://github.com/langchain-ai/deepagents. Computer software

    2025

  14. [14]

    Rasmussen, P

    P. Rasmussen, P. Paliychuk, T. Beauvais, J. Ryan, and D. Chalef. 2025. Zep: A Temporal Knowledge Graph Architecture for Agent Memory.arXiv preprint arXiv:2501.13956(2025). doi:10.48550/arXiv.2501.13956

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.13956 2025

  15. [15]

    T. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths. 2023. Cognitive Architectures for Language Agents.arXiv preprint arXiv:2309.02427(2023). doi:10.48550/arXiv.2309.02427

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.02427 2023

  16. [16]

    Vosseler

    A. Vosseler. 2026. BayesProp: Bayesian A/B Testing for Proportions. Computer software, version 0.1.1.6, Zenodo. doi:10.5281/zenodo.20543002

    work page doi:10.5281/zenodo.20543002 2026

  17. [17]

    Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference.J. Amer. Statist. Assoc.22, 158 (1927), 209–212. doi:10.1080/01621459. 1927.10502953

    arXiv 1927

  18. [18]

    D. Wu, H. Wang, W. Yu, Y. Zhang, K.-W. Chang, and D. Yu. 2025. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. InProceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/ arXiv.2410.10813

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.10813 2025

  19. [19]

    Y. Wu, S. Liang, C. Zhang, Y. Wang, Y. Zhang, H. Guo, R. Tang, and Y. Liu. 2025. From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs.arXiv preprint arXiv:2504.15965(2025). doi:10.48550/arXiv.2504. 15965

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.15965 2025

  20. [20]

    Zheng and M

    J. Zheng and M. Fischer. 2023. Dynamic Prompt-Based Virtual Assistant Frame- work for BIM Information Search.Automation in Construction155 (2023), 105067. doi:10.1016/j.autcon.2023.105067

    arXiv 2023

  21. [21]

    Zhong, L

    W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang. 2024. MemoryBank: Enhancing Large Language Models with Long-Term Memory.Proceedings of the AAAI Conference on Artificial Intelligence38, 17 (2024), 19724–19731. doi:10.1609/aaai. v38i17.29946 A Probe-Agent System Prompt For completeness and reproducibility, we reproduce below the verbatim system prompt used by...

    doi:10.1609/aaai 2024

  22. [22]

    This also reveals units

    **Orient:** Run`ifcquery {ifc_filename} summary`to get schema version, entity counts, and project metadata. This also reveals units

  23. [23]

    For example: `ifcquery {ifc_filename} select'IfcWall, IfcWallStandardCase'`

    **Discover elements:** Use`ifcquery {ifc_filename} select'<IfcClass>'` to find elements of a specific type. For example: `ifcquery {ifc_filename} select'IfcWall, IfcWallStandardCase'`

  24. [24]

    not available in the model

    **Inspect details:** Use`ifcquery {ifc_filename} info <step_id>`to get full attributes, property sets, element type, material, container, placement, and geometry summary for any element. ... ## Important ifcquery Tips - **Always use`{ifc_filename}`verbatim** - your cwd is already the project directory. - **Prefer`--format json`** (default). Use`--format i...

  25. [25]

    Paraphrases are acceptable

    correctness: The system answer's stated facts and conclusions are compatible with the gold answer. Paraphrases are acceptable. Mark correctness false for wrong yes/no conclusions, wrong project/model/scope, contradictory source claims, materially wrong numeric values, incompatible units, wrong list items ↩→, or extra factual claims that materially change ...

  26. [26]

    Require the final requested result and only those qualifiers or source/IFC caveats whose omission would likely mislead the user or change how the answer should be used

    complete: The answer provides enough information to satisfy the user's probe. Require the final requested result and only those qualifiers or source/IFC caveats whose omission would likely mislead the user or change how the answer should be used. Do not require every list item, supporting count, document reference, calculation step, or caveat from the gol...

  27. [27]

    Judge answer accuracy, not writing style or whether the answer mentions agent trace/tool-output details

    relevant: The answer directly addresses the probe question and does not ↩→answer the wrong project, IFC model, element, property, scope, or task. Judge answer accuracy, not writing style or whether the answer mentions agent trace/tool-output details. B.2 Memory Judge You are an expert evaluator. Your task is to judge the quality of memory ↩→retrieval and ...

    2026

  28. [28]

    It may include some noise, but the retrieved content must contain useful memory facts for the task

    retrieval_relevant: The retrieved memory content is relevant to the probe and the target extra information. It may include some noise, but the retrieved content must contain useful memory facts for the task

  29. [29]

    Exact wording is not required

    retrieval_covers_key_facts: The retrieved memory content covers the key facts from the target extra information needed to answer the probe, such as ↩→source schedule/spec names, negative confirmations, classifications, quantities, units, scopes, or inclusion/exclusion rules. Exact wording is not required. ↩→retrieval_covers_key_facts must be false if retr...

  30. [30]

    It should preserve the retrieved key facts without contradiction and should not rely only on IFC lookup when remembered facts are needed

    answer_uses_memory: The system answer uses the retrieved memory accurately. It should preserve the retrieved key facts without contradiction and should not rely only on IFC lookup when remembered facts are needed. Do not penalize concise answers if they correctly use the retrieved memory