REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services

Kexin Zhu; Yang Han

REVIEW 4 major objections 7 minor 20 references

REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark called REAL measures whether large language models can do real estate work in Beijing, and its results show today's best public models still average only about 50 percent on memory, comprehension, and reasoning questions.

desk verdict First housing-transactions benchmark for Chinese LLM agents; genuinely useful, honestly limited, and worth a serious referee despite the missing human baseline and closed data. read the letter →

arxiv 2507.03477 v1 pith:IIQFDVBP submitted 2025-07-04 cs.AI

Kexin Zhu , Yang Han This is my paper

classification cs.AI

keywords LLMbenchmarkrealestatehousingtransactionshallucinationevaluationnaturallanguageinferencedomain-specificBeijingagent

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REAL is a new evaluation suite, built from Beike's Beijing housing data, that tests large language models on the work a real-estate agent does: remembering facts about neighborhoods, understanding house descriptions, reasoning about what a client will buy, and not inventing places that do not exist. The paper uses 5,316 expert-checked entries to evaluate eleven models and finds the best public model averages only about 50 percent on the multiple-choice topics, with every model below 50 percent on reasoning. When queries contain fictional names, models hallucinate at 61 to 93 percent rates. The authors conclude that LLMs still have significant room for improvement before they can be applied in the real estate field.

What carries the argument

The load-bearing object is the REAL benchmark itself, made of 5,316 entries in 14 categories across four topics. Memory, comprehension, and reasoning are multiple-choice: memory asks for factual attributes and surroundings of real Beijing blocks, comprehension asks to match natural-language descriptions to blocks or client demands to block recommendations, and reasoning asks to predict which block a client actually bought from their browsing history, with the real transaction as ground truth. Hallucination entries are query-premise pairs built by injecting fictional names (blocks, areas, surrounding facilities, traffic names) into real client queries; a GPT-4 judge classifies each model response against the premise using natural-language-inference categories (entailment, contradiction, neutral). The data pipeline collects real records from Beike, filters for privacy and quality, rewrites typos, generates fictional names with style templates, and has every entry verified by human annotators.

What would settle it

One way to test the central claim: take a random sample of the 4,302 multiple-choice entries, have independent real-estate experts relabel them without seeing the original answers, and compare; if agreement on reasoning-by-behavior questions is at or near chance, the benchmark's conclusion that LLMs still have room for improvement would not be supported by a reliable ground truth.

Watch

Extended reading notes

Core claim

The paper claims REAL is the first evaluation suite designed specifically for housing transactions and services, and that on this suite the most advanced public or API models fall well short of usable performance: GPT-4o reaches 50.64 percent average accuracy across memory, comprehension, and reasoning, while the best overall system is an internal model at 71.73 percent. Reasoning proves hardest, with all models scoring below 50 percent, which the authors read as evidence that prediction from behavior data is still unresolved. In the hallucination topic, models confidently describe fictional blocks, areas, surroundings, and roads, with hallucination rates of 60.91 to 93.00 percent when a fictional entity is present, but below 5.58 percent when the entity is real. The paper validates the GPT-4 judge used for hallucination scoring against human annotators on 1,500 sampled pairs, reporting Pearson correlations above 0.9 and accuracy above 95 percent.

Load-bearing premise

The strongest results rest on the assumption that the benchmark's ground-truth labels are correct: the reasoning questions use the block a client actually bought as the right answer, which the paper itself concedes is uncertain (Appendix A.8), and hallucination rates depend on a GPT-4 judge whose agreement with humans was measured on a sample of 300 questions per model (Section 4.4).

Editorial extensions

If this is right

A production real-estate chatbot relying on today's LLMs would need heavy domain-specific grounding or human oversight: on roughly half of the multiple-choice questions the best public model would give the wrong answer.
All models fall below 50 percent on reasoning-by-behavior, so browsing-history-driven predictions are not yet a safe feature for agent systems.
Because hallucination rates explode when names are unfamiliar, any deployment should pair the model with a verified knowledge base and reject or flag queries mentioning entities outside it.
REAL's design shows a viable template for building professional benchmarks from real business data plus expert annotation, which could be extended to other cities and other transaction-heavy services.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

As an editorial inference: the hallucination measurement may understate the problem, because the judge only sees a single response per query; a model could acknowledge uncertainty in one answer and then fabricate in a follow-up, which the benchmark would not detect.
As an editorial inference: the reasoning-by-behavior ground truth (the actual purchased block) is the fragile part of the suite; the paper itself flags uncertainty in section A.8, so comparisons on that topic should be read as approximate.
As a testable extension: adding an explicit 'None of the above / this block does not exist' option to the hallucination queries would let the benchmark distinguish models that refuse to answer from models that fabricate, which the current binary rate does not.
As an editorial inference: because REAL covers only Beijing, its difficulty curve may not transfer to other cities; re-running the same pipeline on another city's data would test whether the current model ranking reflects general real-estate competence or memorization of Beijing-specific facts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, read this one if you follow domain benchmarks for LLM agents. It's the first real estate transactions/services evaluation suite I know of, and it's not a toy: 5,316 expert-checked items across Beijing housing, split into memory, comprehension, reasoning, and hallucination. The design follows the LawBench/FinEva playbook, which is fine, and the domain extension is genuinely new. What's done well: the hallucination topic is the most original part. They inject fictional block/area/road names into realistic client queries, use GPT-4 as an NLI judge, and validate the judge against human annotators on 1,500 samples, hitting >95% agreement and correlation above 0.9. That's real evidence for the judge pipeline, even though the dataset itself isn't released. The headline result — public LLMs hover around 50% on this material, and all models sit below 50% on reasoning — is plausible and internally consistent.

Soft spots, in proportion. First, there is no human or expert ceiling anywhere in Table 4; the only baseline is random guessing at 25%. The stress-test note is right about the reasoning topic: Section A.8 concedes that the Reasoning by Behavior labels are real transaction outcomes with "a certain degree of uncertainty." If the recorded label is one of several jointly plausible choices, even a perfect reasoner could miss it, so the sub-50% scores there may partly be label noise. That said, the broader "significant room for improvement" claim still holds for memory and comprehension, where questions are factual and manually verified; no competent human agent would score at the 40-70% level these models show. A human baseline is needed, but its absence doesn't sink the paper. Second, no confidence intervals or multiple seeds; with 45-201 items per category, some model differences are probably noise. Third, the internal Beike models are evaluated on data derived from Beike's own systems — an overlap risk they disclose, but it limits comparisons against those models. Fourth, the data is not released, so nothing is independently checkable. These are fixable. Minor: English typos like "blosk" and some loose appendix phrasing.

Verdict: solid first benchmark with honest limitations. I'd send it to peer review, conditional on releasing at least a public sample, adding error bars, inter-annotator agreement, and a human expert baseline. If those land, it becomes a useful reference for Chinese real estate LLM deployment. I'd cite it in a domain-benchmark comparison. Reading group: maybe, since the breadth is good but the depth is thin.

Referee Report

4 major / 7 minor

Summary. The paper introduces REAL, a benchmark suite for evaluating large language models (LLMs) on housing transactions and services, containing 5,316 entries across four topics: memory, comprehension, reasoning, and hallucination, organized into 14 categories. The benchmark is constructed from Beike Inc. data and expert-annotated options, and the paper evaluates 11 LLMs, including two internal Beike models, GPT-4 variants, and several open-weight Chinese models. Reported results show internal models leading memory and comprehension, all models scoring below 50% on reasoning, and high hallucination rates when fictional entity names are injected. The paper concludes that LLMs still have significant room for improvement in this domain.

Significance. If the benchmark is valid, it addresses a real gap in domain-specific LLM evaluation, and the construction pipeline (Section 3.4) with expert involvement, manual verification, and a hallucination judge validated against human annotators (Table 6) are notable strengths. The hallucination topic is particularly innovative in testing whether models challenge fictional premises. However, the central claim of significant room for improvement is underdetermined because no human or expert baseline is provided, and the reasoning labels are conceded to be uncertain (Section A.8). These issues must be addressed before the benchmark's main conclusion can be accepted.

major comments (4)

[Section 4.2, Table 4] The central conclusion that 'LLMs still have significant room for improvement' (Abstract and Section 5) rests on comparing model accuracies around 50% with a 25% random baseline, but no human or expert ceiling is reported anywhere. Without knowing how a competent human agent performs on the same items, the observed scores underdetermine the conclusion: they could indicate hard items, label noise, or a genuine ability gap. I recommend reporting human/expert baselines on a representative sample of each topic, or at least on the reasoning topic, and using these to calibrate the claim.
[Section A.8, Table 4] The Reasoning by Behavior items use the real transaction outcome as the correct answer, and Section A.8 itself concedes that 'the results also have a certain degree of uncertainty.' If several blocks are jointly plausible given the behavior context, a rational agent cannot be expected to select the recorded label reliably; the 45-51% accuracies on this topic may therefore reflect label noise rather than a genuine reasoning deficiency. The paper should provide evidence on label determinism, for example inter-annotator agreement on a subsample, and either restrict the benchmark to items with a uniquely identifiable answer or report accuracy conditioned on label confidence.
[Section 4.2, Table 3] The two internal Beike models are evaluated on a benchmark built from Beike's own data (Section 3.3). The paper notes their memory advantage 'because of the internal data' but does not state whether the benchmark entries or the underlying data were used in training these models. If so, the internal models' performance advantage is at least partly a training-data overlap effect, and the comparison with public API models is not apples-to-apples. Please disclose this or provide a decontamination analysis, such as testing internal models on items or data that were demonstrably excluded from training.
[Sections 4.3 and 4.4, Tables 5 and 6] The hallucination rates depend entirely on GPT-4 as the judge, and the agreement validation (Table 6) covers 300 questions per model for five models. Although the reported accuracy (95.33-99.00%) is strong, no confidence intervals, no inter-annotator agreement among human annotators, and no analysis of the judge's stability over repeated runs or API versions are provided. Without information about human label noise and judge variability, the hallucination rates in Table 5 could be systematically biased. Please add these statistics or explicitly discuss the limitation.

minor comments (7)

[Various sections] There are several typos and inconsistencies: 'blosk' in Appendix A.11, 'TopicHallucination topic' in Section 3.2, 'Descirption' in Figure 2, and inconsistent use of digits for numbers below ten (e.g., 'The other 4 categories' in Section 3.3).
[Section 4.4] The agreement evaluation states that 1,500 pairs of queries and responses are obtained from 300 questions; please clarify that this is 5 models × 300 questions, and describe how the 300 questions were sampled (e.g., stratified by category).
[Table 5] The 'total' column appears to be the unweighted average of the 'with' and 'without' fictional entity columns; please state this explicitly, as the 'total' is not the hallucination rate over all 1,014 items.
[Equation (2)] The Pearson correlation formula uses a 'P' symbol that is not standard and the summation subscripts are garbled; please typeset it properly as a sum over items.
[Section 3.4] The description of P and Q sets says they are 'different for each category' but the example only compares the number of styles in the Q set; please rephrase to avoid confusion.
[Data and code availability] No link to the dataset or evaluation code is provided. For a benchmark paper, please indicate whether and where the dataset and prompts will be made available, both for reproducibility and for use by the community.
[Section 4.3] The mapping from NLI decisions (entailment, contradiction, neutral) to hallucination counts is not defined; please state which decision(s) are counted as hallucination (e.g., contradiction only, or contradiction plus neutral).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: REAL is an empirical benchmark with externally validated labels and judge, and its conclusions rest on measured accuracies rather than on any self-referential derivation.

full rationale

The paper contains no derivation chain whose output is equivalent to its input. REAL is a dataset-construction and evaluation paper: questions are built from Beike transaction data and expert annotations, and model performance is measured by accuracy or by a GPT-4 hallucination judge. The hallucination judge is not adopted as an unexamined premise; Section 4.4 validates it against human annotations on 1,500 sampled judgments, with Pearson correlations above 0.9 and accuracy above 95%, which is independent grounding. The Reasoning by Behavior labels are conceded in Appendix A.8 to have uncertainty because the correct answer is the actual transaction outcome, but this is an acknowledged limitation of label reliability, not a circular step: the label is not derived from the models being evaluated, nor is any model score used to define the label. The use of Beike-owned data and Beike internal models raises possible training-data overlap, but that is a contamination risk rather than a reasoning loop, and the internal models are compared against external API and open-weight models. No load-bearing self-citation, uniqueness theorem, fitted parameter renamed as prediction, or ansatz smuggled in via citation appears in the manuscript. The central claim that models have room for improvement is an empirical reading of Table 4, which is underdetermined without a human ceiling, but underdetermination is not circularity. No circular steps are present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark's claims rest on the quality of Beike's data, the correctness of expert labels, and the generalization of the judge model; none of these are fully verifiable from the paper alone.

assumptions (3)

domain assumption Beike's proprietary data accurately represents the relevant facts about Beijing residential blocks.
All memory, comprehension, and reasoning questions are built from this data source (Section 3.3), so data errors or bias directly affect the benchmark.
domain assumption Expert-written answers and annotations are correct and unambiguous.
No inter-annotator agreement is reported, and Reasoning by Behavior is admitted to have uncertainty (Section A.8).
ad hoc to paper The GPT-4 judge's NLI decisions are reliable on all hallucination entries, not only the validated sample.
Agreement was checked on 1,500 query-response pairs (300 per model), but the full 1,014-entry hallucination set is not independently verified (Section 4.4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services." pith.science (2026). https://pith.science/paper/IIQFDVBP

@misc{pith2026250703477,
  author       = {Pith},
  title        = {Pith review of: REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIQFDVBP}},
  note         = {Machine review of arXiv:2507.03477}
}

read the original abstract

The development of large language models (LLMs) has greatly promoted the progress of chatbot in multiple fields. There is an urgent need to evaluate whether LLMs can play the role of agent in housing transactions and services as well as humans. We present Real Estate Agent Large Language Model Evaluation (REAL), the first evaluation suite designed to assess the abilities of LLMs in the field of housing transactions and services. REAL comprises 5,316 high-quality evaluation entries across 4 topics: memory, comprehension, reasoning and hallucination. All these entries are organized as 14 categories to assess whether LLMs have the knowledge and ability in housing transactions and services scenario. Additionally, the REAL is used to evaluate the performance of most advanced LLMs. The experiment results indicate that LLMs still have significant room for improvement to be applied in the real estate field.

Figures

Figures reproduced from arXiv: 2507.03477 by the authors.

**Figure 1.** Abilities that chatbot should have in housing transactions and services scenario. LLMs and cannot reflect the performance in some specific fields. On the other hand, building benchmarks for specific fields requires professional knowledge from domain expertise. This makes the process of building the benchmark extremely time-consuming and labor-intensive. To evaluate whether LLMs have the abilities of memory, compreh… view at source ↗

**Figure 2.** Overview of the REAL. REAL to evaluates the capabilities of mainstream LLMs for housing transactions and services. The main contributions are as follows: • We propose a professional benchmark specifically for the housing transactions and services. The benchmark contains three topics about Memory, Comprehension and Reasoning. There are 4,302 high-quality evaluation entries across these three topics. • We construc… view at source ↗

**Figure 3.** Pyramid Structure of REAL. and network security event theoretical questions. It provides effective evaluation capabilities for the research and development of security. The Flames (Huang et al., 2024) is highly adversarial and is dedicated to evaluate the performance of models in three dimensions such as fairness, morality and legality. The establishment of these benchmarks for specific fields requires experienced e… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: REAL Data Pipeline. vanced and complex skill and is built upon knowledge and comprehension skills. Reasoning topic contains tasks that reasoning what type of house the client is likely to purchase based on their past behavior data. The data of behavior involve browsin…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

[1]

01.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. 2024. Yi: Open foundation models by 01.ai

work page 2024
[2]

AntGroup and SUFE. 2023. Fin-eva. https://github.com/alipay/financial_evaluation_dataset

work page 2023
[3]

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

work page 2024
[4]

EastMoney and ShanghaiAILab. 2024. Openfindata. https://github.com/open-compass/OpenFinData

work page 2024
[5]

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. 2024. L aw B ench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 7933--7962, Miami, Florida, USA, November. Association...

work page 2024
[6]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools

work page 2024
[7]

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems , volume 36, pages 62991--63010. Curran Ass...

work page 2023
[8]

Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, and Dahua Lin. 2024. Flames: Benchmarking value alignment of LLM s in C hinese. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

work page 2024

Show all 20 references

[9]

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU : Measuring massive multitask language understanding in C hinese. In Findings of the Association for Computational Linguistics: ACL 2024 , pages 11260--11285, Bang...

2024
[10]

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. A lign B ench: Benchmarking C hinese alignment ...

2024
[11]

Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9004--9017, Singapor...

2023
[12]

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1906--1919, Online, July. Association for Compu...

2020
[13]

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381--2391, Brussels...

2018
[14]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. 2024. Gpt-4 technical report

2024
[15]

Qwen Team . 2024. Qwen2.5: A party of foundation models, September

2024
[16]

Tencent. 2024. Secbench. https://secbench.org/

2024
[17]

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models

2023
[18]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report

2024
[19]

Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su, Han-Sen Zhong, and Yuqiang Li. 2024. Chemllm: A chemical large language model

2024
[20]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

01.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. 2024. Yi: Open foundation models by 01.ai

work page 2024

[2] [2]

AntGroup and SUFE. 2023. Fin-eva. https://github.com/alipay/financial_evaluation_dataset

work page 2023

[3] [3]

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

work page 2024

[4] [4]

EastMoney and ShanghaiAILab. 2024. Openfindata. https://github.com/open-compass/OpenFinData

work page 2024

[5] [5]

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. 2024. L aw B ench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 7933--7962, Miami, Florida, USA, November. Association...

work page 2024

[6] [6]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools

work page 2024

[7] [7]

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems , volume 36, pages 62991--63010. Curran Ass...

work page 2023

[8] [8]

Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, and Dahua Lin. 2024. Flames: Benchmarking value alignment of LLM s in C hinese. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

work page 2024