Pith. sign in

REVIEW 4 major objections 4 minor 36 references

Current memory-augmented agents can retrieve explicit facts but decline sharply at episodic recall, behavior-pattern aggregation, and personality-trait inference, according to a new synthetic benchmark built from psychological trait profile

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:15 UTC pith:5FUVAHK5

load-bearing objection Genuinely useful benchmark with rigorous generation; headline monotonic-decline claim is undercut by metric mismatch and an unverified recoverability assumption. the 4 major comments →

arxiv 2607.27056 v2 pith:5FUVAHK5 submitted 2026-07-29 cs.AI cs.CL

Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data

Lingyang Zeng , Guangze Chen , Kaichen Yu , Zhicheng Pan , Siyang Weng , Zirui Hu , Xiangyun Du , Hailin He
show 4 more authors
Rong Zhang Chengcheng Yang Kai Huang Xuan Zhou
This is my paper
classification cs.AI cs.CL
keywords personalized agentsmemory benchmarksuser understanding hierarchysemantic memoryepisodic memorybehavior patternspersonality traitsheterogeneous data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that personalized agents need four distinct levels of user understanding—semantic memory, episodic memory, behavior patterns, and personality traits—and that existing memory benchmarks test only the first. It introduces Setoka, a benchmark that generates privacy-preserving, heterogeneous user data from psychological trait profiles and evaluates agents at every level. Across three language models and five memory systems, the best score falls from 0.85 on single-record fact recall to 0.46 on event reconstruction, 0.28 on behavior-pattern aggregation, and 0.24 on personality-trait ranking, which is barely above chance. The authors conclude that simple fact retrieval is insufficient and that memory systems need mechanisms to link, aggregate, and generalize evidence across sources and time.

Core claim

Setoka's central claim is that user understanding is a hierarchy—semantic memory, episodic memory, behavior patterns, personality traits—and that the higher levels are not reducible to fact retrieval. The paper constructs a synthetic user corpus in which every record is traceable to a generated personality profile, then asks memory-augmented agents to answer level-specific queries. On 10 synthetic users, the best system scores 0.85 on semantic recall but 0.46 on episodic linking, 0.28 on behavior-pattern aggregation, and 0.24 on personality-trait ranking, which is marginally above chance. The authors interpret the monotone decline as evidence that current memory systems lack the machinery to

What carries the argument

Setoka's central mechanism is a top-down, psychometrics-guided generation pipeline. Personality trait vectors are sampled jointly from a multivariate Gaussian whose correlations come from meta-analytic psychometric studies; these scores are converted into item-level responses on a validated 60-item personality inventory; responses are compiled into behavior-pattern quotas; and an event-grounded generation tree expands those quotas into a two-month timeline of dated events. Every heterogeneous record—relational table row, document, or graph edge—is derived from one event treated as its single source of truth. This gives the benchmark traceable evidence sets and reference answers at each of th

Load-bearing premise

The benchmark's conclusions rest on the assumption that the synthetic users generated from Gaussian trait vectors and inventory responses behave enough like real people that performance on Setoka transfers to real personalized agents; if the synthetic records do not capture how traits actually manifest in human data, the measured difficulty gradient may not hold outside the benchmark.

What would settle it

Construct a consented corpus of real users' heterogeneous records (messages, calendars, contacts, app usage) and run the same four-level query templates on it; if the monotone drop from semantic to personality-trait accuracy does not reproduce, or the system ranking inverts, then the synthetic generation pipeline—not real user understanding—is what Setoka actually measures.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If Setoka's difficulty gradient reflects real user understanding, then any personalized agent that only retrieves stored facts will systematically fail at tasks requiring event reconstruction, regularity detection, or trait inference, no matter how large its memory store is.
  • The finding that direct database access beats memory systems on single-record lookup but falls behind on linked-event questions implies the bottleneck is not fact storage but the linking and aggregation operations built on top of it.
  • Graph-structured memory systems consistently lead on the two highest abstraction levels, suggesting explicit association between records is part of the mechanism needed for behavior-pattern and trait inference; because no system wins at all levels, representation alone is insufficient.
  • Answer rates must be reported separately from accuracy; a system that answers almost everything may simply be guessing, as the smallest model does here, and accuracy-only scoring conflates calibration with competence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A concrete design consequence the paper leaves implicit: memory systems should maintain at least three representational tiers—verbatim records, linked event summaries, and aggregated behavioral statistics—and route each query to the tier whose operation (select, link, aggregate, generalize) matches the question.
  • Because answer rate rises while accuracy falls with abstraction, any leaderboard or product metric that rewards response rate will reward guessing on deep-user-understanding questions; separating abstention from accuracy should become standard in agent evaluation.
  • The trait-ranking result being near chance suggests current pipelines cannot yet support product claims about personality-aware personalization; a testable next step is to add an explicit trait-inference step that summarizes evidence per trait dimension before answering, and measure whether rank correlation rises above the observed 0.24.
  • Because the generation pipeline is taxonomy-agnostic—any trait inventory with known means, variances, and correlations can be plugged in—Setoka-style instances could be built for workplace, clinical, or cultural trait dimensions, not just the five-factor taxonomy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Setoka is a benchmark for memory-augmented personalized agents that organizes user understanding into four levels — semantic memory (SM), episodic memory (EM), behavior patterns (BP), and personality traits (PT) — grounded in cognitive and personality psychology. The paper contributes a psychometrics-based synthetic-data pipeline: correlation-aware sampling of Big Five trait vectors, BFI-2 response-profile synthesis, an event-grounded generation tree, and heterogeneous record derivation; plus query generation with traceable evidence. It evaluates 3 LLMs × 5 memory systems plus a memory-free DBQuery baseline on 10 synthetic users and reports that accuracy declines from SM to EM to BP, with PT rank correlation near chance. The main conclusions are that current memory systems cannot support deep user understanding and that cross-source integration and abstraction mechanisms are needed.

Significance. If the hierarchy is valid and the synthetic corpus encodes the latent traits in an inferable way, Setoka is a valuable contribution: it is the first benchmark in this space to cover all four levels over heterogeneous data models, and it ships several careful validation ideas — replay match, minimality, true multimodality, run-to-run reproducibility (median |Δ| = 0.004/0.007), and a dedicated DBQuery baseline. The exact behavioral-fidelity guarantee of Theorem 1 and the use of Kendall's τ for PT evaluation are also strengths. However, the central empirical conclusions depend on two checks that are currently missing: whether the latent personality vector is recoverable from the generated records at all, and whether the reported cross-level comparisons use commensurable metrics. These gaps make the headline claims stronger than the evidence supports.

major comments (4)
  1. [§4.2–4.4, Appendix C, Table 2] The benchmark never verifies that the latent trait vector θ is inferable from the generated records. The chain θ → BFI-2 answer sheet → thresholded evidence (Kev=20, τev=0.25) → LLM-proposed behavior patterns → events → records is stochastic and lossy; Theorem 1 only proves that the planned behavioral quotas are realized, not that the records carry statistical information about θ. The DBQuery baseline, which bypasses memory and queries raw records directly, scores at chance on PT (0, −6, −4, Table 2), consistent with the possibility that PT is not answerable from the corpus. Before concluding that memory systems 'cannot handle' PT, add an oracle/upper-bound condition — e.g., provide the answerer with the full raw corpus or with the generation-time behavior-pattern statistics — and show that a system with all available evidence can rank users above chance. If it cannot, the PT-level resul
  2. [§5.2, Table 2] The central monotonic-decline claim conflates metrics. SM, EM, and BP are scored with an LLM-judge similarity on [0,1], while PT uses Kendall's τ on [−1,1]. The sentence 'the best score drops from 0.85 on SM to 0.46 on EM, 0.28 on BP, and 0.24 on PT' and the 'level average' row treat these numbers as commensurable, but they are not; the abstract's 'declines even further' is therefore not established by the reported numbers. The PT result is independently interesting — most systems are near chance — but the paper should either put all levels on a common scale (e.g., report accuracy after dichotomizing PT predictions, or rescale rank correlation to [0,1]) or explicitly restrict the monotonic claim to SM/EM/BP and treat PT as a separate, non-comparable finding.
  3. [§5.2, Fig. F.5] The PT conclusion rests on a very small sample: 10 users, 50 PT items, and a permutation test in Fig. F.5 showing that only 5 of 18 configurations are significantly above chance, with no Gemma configuration significant and the largest pooled τ = 0.24. The claim that current systems 'cannot handle' personality-trait inference is stronger than this evidence permits. Report confidence intervals for the pooled τ, increase the number of synthetic users (the pipeline appears scalable), or soften the conclusion to a preliminary finding. The same small-N concern applies to BP, which has 110 items over 10 users (11 per user) and no reported per-user variance for the BP level.
  4. [§5.1, Appendix G.5] The LLM judge (DeepSeek-V4-Pro) is also the generation backbone. Because the same model family produced the reference answers and scores the responses, the judge may systematically prefer outputs in its own generation style, which could bias level comparisons or system rankings. Validate the judge against a human-annotated subset (e.g., 100–200 items per level) or use an independent judge (different family/version) and report agreement. This is especially important for BP, where 'semantic consistency' with a generated statistical answer is loosely defined and the judge scales carry the entire BP result.
minor comments (4)
  1. [Eq. (6)] The notation overloads E: E^EM is defined as the union of evidence sets from SM-level answers, but each SM answer may have its own evidence set. Make explicit whether E^EM is a set of records or a family of sets, and similarly for E^BP and E^PT.
  2. [Table 2, §5.2] For PT, the 'answer rate' is not defined the same way as for factual queries, since a model can always output a predicted trait score. Clarify what counts as an answer for PT (e.g., non-null prediction) and how abstention is handled.
  3. [Appendix C.3] The capacity bound c_g ≤ cap(a_g)|Δ_g| guarantees individual placeability, but the scheduler can still report joint infeasibility and reschedule. Report how often the pipeline had to resample or reschedule across the 10 released users; this bears on the reproducibility of the generation procedure.
  4. [Fig. F.5] Per-trait Kendall's τ values for N=10 users have very large sampling error. Consider adding error bars, jittered user-level values, or a small simulation-based null band to help readers gauge the uncertainty of each cell.

Circularity Check

0 steps flagged

No circularity: the benchmark's ground truth is generated and then evaluated as a standard construct; no load-bearing reduction to its own inputs.

full rationale

Setoka is a benchmark paper, not a predictive derivation. Its pipeline samples personality vectors, maps them through BFI-2 to behavior patterns, expands patterns into events, and derives heterogeneous records with generation-time ground truth. Evaluating systems against that ground truth is standard benchmark design, not circular reasoning: the reference answers are not fitted parameters of the systems being evaluated, and no fitted input is renamed as a prediction. The formal hierarchy in Eq. (6) defines additional operators (σ, λ, γ, ρ) and broader evidence scopes per level, but it does not force the observed accuracy values; systems could in principle perform better at higher levels, so the reported monotone decline is an empirical outcome rather than a tautology. The paper contains no load-bearing self-citation chain: citations such as Hu et al. 2026 and Wang et al. 2026 are external prior work, and no 'uniqueness theorem' or prior result by the same authors is invoked to force the framework. The use of DeepSeek-V4-Pro both as the generation backbone and as the LLM judge for SM/EM/BP is a possible evaluation bias, but it is not a circular reduction: the judge's scores are not defined in terms of the generator's outputs, and PT uses Kendall's τ against the sampled trait vectors rather than LLM-judge similarity. The skeptical concern that BP/PT may be unrecoverable from the generated corpus is a construct-validity threat, honestly acknowledged in part by the paper's own Appendix A caveat that the hierarchy changes evidence scope and answer type simultaneously; but an unvalidated task or a possible ceiling/floor artifact is not the same as the derivation reducing to its own inputs. No quoted equation, fitted parameter, or self-citation exhibits the required equivalence, so no circular step is identified.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central results depend on a set of hand-chosen pipeline hyperparameters and on the psychometric validity of the trait-to-behavior mapping. The benchmark's realism is an untested assumption—no real user data is involved, so the transfer to real-world personalized agents is a domain assumption, not an established fact.

free parameters (5)
  • Number of synthetic users N = 10
    Chosen by hand; determines statistical power, especially for PT ranking where only 10 users are ranked per trait.
  • Behavior quota rates (ω1, ω2, ω3) = (1/7, 2/7, 4/7) per allowed day
    Hand-set in Appendix C; control how salience bands map to event frequencies, directly shaping BP and PT ground truth.
  • Salience threshold and bands = τ_ev=0.25, bands at 0.4 and 0.7, K_ev=20, M=11, |C|=426
    Hand-set; determine which BFI-2 responses become behavioral evidence and how many behavior patterns are generated.
  • Cap function cap(·) = values in {1,2,3}
    Hand-set; limits within-day recurrence of event categories, affecting the distribution of events and thus query difficulty.
  • Tolerances and jitter (εμ, εσ, εR, η) = not specified in text
    Hand-set in Algorithm B.1 for trait sampling verification; values are not reported, affecting reproducibility of the sampled users.
axioms (4)
  • domain assumption Big Five trait scores follow a multivariate Gaussian distribution with means, standard deviations, and correlations from meta-analytic studies
    Section 4.1; used to sample synthetic user personality vectors. If trait distributions are not Gaussian, the synthetic personas may be unrepresentative.
  • domain assumption BFI-2 item responses and scores validly map personality traits to observable behavior patterns
    Section 4.2; trait vectors are converted to BFI-2 response sheets, which are then translated into behavior patterns. Invalid mapping would break the trait-to-behavior grounding.
  • domain assumption LLM-generated events from structured behavior patterns are realistic simulations of user behavior
    Sections 4.3 and 4.5; all events and records are generated by an LLM (DeepSeek-V4-Pro) given the behavior prior. No real-world user data is used to validate this realism.
  • standard math Standard probability and statistics results (total variation distance, Gaussian tail bounds, Cauchy-Schwarz) hold
    Appendix B; used in propositions on rejection sampling. These are standard mathematical facts.

pith-pipeline@v1.3.0-alltime-deepseek · 29901 in / 11291 out tokens · 303447 ms · 2026-08-04T03:15:08.933617+00:00 · methodology

0 comments
read the original abstract

Personalized agents are increasingly applied to assist users across a wide range of tasks. Effective personalized assistance requires not only retrieving explicit facts from past interactions stored in agent memory, but also inferring abstract personal characteristics. However, existing memory benchmarks primarily evaluate whether an agent can retrieve information explicitly stated in conversational histories, failing to provide an effective assessment of deeper user understanding. In this work, we propose Setoka, a benchmark for evaluating memory-augmented personalized agents with hierarchical user understanding from heterogeneous data. Grounded in theories from cognitive and personality psychology, Setoka defines four levels of user understanding, i.e., semantic memory, episodic memory, behavior pattern, and personality trait. Moreover, to enable realistic yet privacy-preserving evaluation, we design a psychometrics-based pipeline that synthesizes diverse, coherent heterogeneous user data and queries at scale. Finally, we leverage Setoka to evaluate 3 language models combined with 5 memory systems for 10 synthetic users. Our comprehensive evaluation reveals that while existing systems perform well on semantic memory retrieval, their performance declines on episodic memory. Moreover, when dealing with behavior pattern and personality trait understanding tasks that require integrating heterogeneous and fragmented information dispersed over time, performance declines even further. These findings demonstrate that user understanding cannot be handled by simple fact retrieval, motivating the design of memory mechanisms for cross-source integration and abstraction over long-term user behavior.

Figures

Figures reproduced from arXiv: 2607.27056 by Chengcheng Yang, Guangze Chen, Hailin He, Kaichen Yu, Kai Huang, Lingyang Zeng, Rong Zhang, Siyang Weng, Xiangyun Du, Xuan Zhou, Zhicheng Pan, Zirui Hu.

Figure 1
Figure 1. Figure 1: Overview of Setoka. Left: Setoka generates user profiles spanning from abstract personality traits to concrete semantic memory, supporting four levels of user understanding. These profiles are then used to generate heterogeneous user data and queries at each level. Right: Representative questions at each level. As the level of user understanding increases, answering the corresponding questions requires int… view at source ↗
Figure 2
Figure 2. Figure 2: Setoka’s psychometrics-based data generation pipeline. (a) Correlation-aware personality trait sampling draws Big-Five persona vectors jointly from a Gaussian model parameterized by meta-analytic trait correlations. (b) Psychological-scale-based behavior pattern generation converts each persona vector into item-level responses on a validated scale (BFI-2), yielding behavior patterns with associated frequen… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 11 linked inside Pith

  1. [1]

    Cheng, M.; Durmus, E.; and Jurafsky, D. 2023. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models. In ACL, 1504--1532

    2023

  2. [2]

    Chhikara, P.; Khant, D.; Aryan, S.; et al. 2025. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. In arXiv:2504.19413

    Pith/arXiv arXiv 2025

  3. [3]

    DeepSeek-AI . 2026 a . DeepSeek-V4-Flash . https://api-docs.deepseek.com/

    2026

  4. [4]

    DeepSeek-AI . 2026 b . DeepSeek-V4-Pro . https://api-docs.deepseek.com/

    2026

  5. [5]

    Du, Y.; Wang, H.; Zhao, Z.; et al. 2024. PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering. In arXiv:2402.16288

    Pith/arXiv arXiv 2024

  6. [6]

    Fleeson, W. 2001. Toward a Structure- and Process-Integrated View of Personality: Traits as Density Distributions of States. J. Pers. Soc. Psychol., 80(6): 1011--1027

    2001

  7. [7]

    Fleeson, W.; and Jayawickreme, E. 2015. Whole Trait Theory. J. Res. Pers., 56: 82--92

    2015

  8. [8]

    Gemma Team . 2025. Gemma 3 4B IT . https://huggingface.co/google/gemma-3-4b-it

    2025

  9. [9]

    J.; Shu, Y.; Qi, W.; et al

    Guti\'errez, B. J.; Shu, Y.; Qi, W.; et al. 2025. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models. In arXiv:2502.14802

    Pith/arXiv arXiv 2025

  10. [10]

    J.; Viechtbauer, W.; et al

    Haslam, N.; McGrath, M. J.; Viechtbauer, W.; et al. 2020. Dimensions over categories: A meta-analysis of taxometric research. Psychol. Med., 50(9): 1418--1432

    2020

  11. [11]

    Hu, Y.; Liu, S.; Yue, Y.; et al. 2026. Memory in the Age of AI Agents. In arXiv:2512.13564

    Pith/arXiv arXiv 2026

  12. [12]

    B.; and Rossi, D

    Huet, A.; Houidi, Z. B.; and Rossi, D. 2025. Episodic Memories Generation and Evaluation Benchmark for Large Language Models. In arXiv:2501.13121

    Pith/arXiv arXiv 2025

  13. [13]

    Jiang, B.; Hao, Z.; Cho, Y.-M.; et al. 2025 a . Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. In arXiv:2504.14225

    arXiv 2025

  14. [14]

    Jiang, B.; Yuan, Y.; Shen, M.; et al. 2025 b . PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. In arXiv:2512.06688

    arXiv 2025

  15. [15]

    Kendall, M. G. 1938. A New Measure of Rank Correlation. Biometrika, 30(1/2): 81--93

    1938

  16. [16]

    Li, Y.; Wen, H.; Wang, W.; et al. 2024. Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security. In arXiv:2401.05459

    Pith/arXiv arXiv 2024

  17. [17]

    Liu, A.; Diab, M.; and Fried, D. 2024. Evaluating Large Language Model Biases in Persona-Steered Generation. In Findings of ACL, 9832--9850

    2024

  18. [18]

    Maharana, A.; Lee, D.-H.; Tulyakov, S.; et al. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. In ACL, 13851--13870

    2024

  19. [19]

    Markovic, V.; Obradovic, L.; Hajdu, L.; et al. 2025. Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning. In arXiv:2505.24478

    Pith/arXiv arXiv 2025

  20. [20]

    McAdams, D. P. 1995. What Do We Know When We Know a Person? J. Pers., 63(3): 365--396

    1995

  21. [21]

    Mistral AI . 2026. Ministral 3 14B Instruct . https://huggingface.co/mistralai/Ministral-3-14B-Instruct

    2026

  22. [22]

    Packer, C.; Wooders, S.; Lin, K.; et al. 2023. MemGPT: Towards LLMs as Operating Systems. In arXiv:2310.08560

    Pith/arXiv arXiv 2023

  23. [23]

    P.; Rodriguez, A.; Spritzer, K

    Reise, S. P.; Rodriguez, A.; Spritzer, K. L.; et al. 2018. Alternative Approaches to Addressing Non-Normal Distributions in the Application of IRT Models to Personality Measures. J. Pers. Assess., 100(4): 363--374

    2018

  24. [24]

    J.; and John, O

    Soto, C. J.; and John, O. P. 2017. The next B ig F ive I nventory ( BFI-2 ): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. J. Pers. Soc. Psychol., 113(1): 117--143

    2017

  25. [25]

    Sui, Y.; Zhou, M.; Zhou, M.; et al. 2024. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. In WSDM, WSDM '24, 645–654. ISBN 9798400703713

    2024

  26. [26]

    Tan, H.; Zhang, Z.; Ma, C.; et al. 2025. M em B ench: Towards More Comprehensive Evaluation on the Memory of LLM -based Agents. In Findings of ACL, 19336--19352. ISBN 979-8-89176-256-5

    2025

  27. [27]

    Tulving, E. 1972. Episodic and Semantic Memory. In Organization of Memory, 381--403

    1972

  28. [28]

    C.; and Christal, R

    Tupes, E. C.; and Christal, R. E. 1992. Recurrent personality factors based on trait ratings. J. Pers., 60(2): 225--251

    1992

  29. [29]

    van der Linden, D.; te Nijenhuis, J.; and Bakker, A. B. 2010. The General Factor of Personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study. J. Res. Pers., 44(3): 315--327

    2010

  30. [30]

    Wang, S.; Yu, E.; Love, O.; et al. 2026. MemMachine: A Ground-Truth-Preserving Memory System for Personalized AI Agents. In arXiv:2604.04853

    Pith/arXiv arXiv 2026

  31. [31]

    Wu, D.; Wang, H.; Yu, W.; et al. 2025 a . LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. In arXiv:2410.10813. ICLR 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Wu, T.; Chen, Z.; Weng, Z.; et al. 2026. KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions. In arXiv:2601.04745

    Pith/arXiv arXiv 2026

  33. [33]

    Wu, X.; Yang, J.; Chai, L.; et al. 2025 b . TableBench: a comprehensive and complex benchmark for table question answering. In AAAI, AAAI'25/IAAI'25/EAAI'25, 25497--25506. ISBN 978-1-57735-897-8

    2025

  34. [34]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; et al. 2023. Judging LLM -as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS (Datasets and Benchmarks)

    2023

  35. [35]

    L.; and Su, F

    Gibbs, A. L.; and Su, F. E. 2002. On Choosing and Bounding Probability Metrics. Int. Stat. Rev., 70(3): 419--435

    2002

  36. [36]

    Higham, N. J. 2002. Computing the Nearest Correlation Matrix---A Problem from Finance. IMA J. Numer. Anal., 22(3): 329--343

    2002