Benchmarking pre-trained text embedding models in aligning built asset information

Ali Motamedi; Mehrzad Shahinmoghadam

REVIEW 4 major objections 4 minor 13 references

Benchmarking pre-trained text embedding models in aligning built asset information

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read General-purpose text-embedding benchmarks do not reliably pick models for aligning built asset information; the paper builds six domain-specific datasets from IFC and Uniclass showing that model rankings shift across tasks and datasets.

desk verdict Useful domain benchmark for embedding alignment, but the rankings are shaped by the very model used to select the evaluation instances; treat specific scores as provisional. read the letter →

arxiv 2411.12056 v1 pith:63VUPPMO submitted 2024-11-18 cs.CL cs.AIcs.IRcs.LG

Mehrzad Shahinmoghadam , Ali Motamedi This is my paper · ORCID

classification cs.CLcs.AIcs.IRcs.LG

keywords textembeddingbuiltassetinformationdataalignmentIFCUniclassclusteringretrievalreranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated alignment of built asset information to classification systems like IFC and Uniclass is hard because the text is technical, multi-sourced, and inconsistently phrased. This paper argues that general-purpose text-embedding benchmarks, such as MTEB, cannot be trusted to pick the right pre-trained embedding model for that alignment task, and it builds the largest domain-specific benchmark to date to test the models directly. The benchmark covers six datasets—clustering, retrieval, and reranking, each at short-name (S2S/S2P) and paragraph (P2P) lengths—over a corpus of 4,234 Uniclass products and 977 IFC entities. Across 24 models, the results show large task-to-task and dataset-to-dataset variability: the top generalist model wins only some tasks, a 33-million-parameter model keeps pace with models ten times its size, and datasets that look thematically similar to built-asset text still order the models differently. The authors conclude that domain-specific evaluation is a necessary step before deploying embedding models for built asset data, and they release the datasets and benchmark code publicly.

What carries the argument

The machinery is a six-dataset benchmark construction pipeline. Two corpora are assembled: 4,234 Uniclass products with three levels of hierarchy labels, and 977 IFC entities (IfcElement subclasses plus type enumerations) with descriptions. Since Uniclass has no product descriptions, descriptions are synthesized by concatenating the product's hierarchical category titles and then paraphrased with GPT-4, followed by two-expert manual review. Sampling is driven by cosine similarity under a reference embedding model (mxbai-embed-large-v1): diversity-based selection for positive samples and hard-negative selection for negatives, with clustering subsets admitted only if an upper-threshold model scores below 0.8 and a lower-threshold model scores above random. Evaluation uses V-measure for clustering, nDCG@10 for retrieval, and MAP for reranking.

What would settle it

Compile a held-out set of real built asset texts with known IFC or Uniclass classes—for example maintenance work orders, equipment submittals, or specification excerpts—run them through the models ranked top and middle in this benchmark, and check whether the model ordering on alignment accuracy matches the synthetic-dataset ordering. If real-text alignment rankings diverge materially from the benchmark's rankings, the central transferability claim is refuted.

Watch

Extended reading notes

Core claim

The central claim of the paper is that no general ranking of pre-trained embedding models transfers reliably to the built asset domain; model choice must be informed by domain-specific evaluation. The evidence is a benchmark of 24 models on six datasets derived from the IFC 4.3 schema and the Uniclass Pr 1.33 product table. The headline results are variability: NV-Embed-v2, first on the general MTEB leaderboard, finishes first overall but not on every task; gte-small, the smallest model at 33M parameters, nearly matches the average performance of models ten times larger; and relative model rankings on the built-asset clustering datasets differ noticeably from rankings on StackExchange and Reddit despite high embedding-similarity between those datasets. The paper reads this as evidence that general-purpose benchmarks fail to capture the semantic complexity of built asset technical terminology, and as motivation for domain adaptation research.

Load-bearing premise

The load-bearing premise is that the benchmark's synthetic Uniclass descriptions—hierarchical labels concatenated and then paraphrased by GPT-4—faithfully represent how built asset text actually appears in practice; if real technical documents use vocabulary, abbreviations, and phrasing that neither Uniclass labels nor the paraphrase model reproduce, the measured model rankings will not predict real alignment performance.

Editorial extensions

If this is right

Model selection for built asset alignment should be made per task and per dataset, not from an overall leaderboard, because the top overall model (NV-Embed-v2) is beaten on reranking by text-embedding-3-large and on some clustering subsets by smaller models.
Deployment need not default to the largest model: gte-small at 33M parameters is competitive with models an order of magnitude larger, so efficiency can be bought without giving up much accuracy.
Instruction-tuned models generally outscore non-instruction-tuned models across most tasks, indicating that instruction tuning, rather than raw scale, is a promising lever for domain adaptation.
General-purpose benchmarks such as MTEB can be actively misleading for this domain: even thematically similar datasets (StackExchange, Reddit) produce different model orderings, so domain-specific evaluation should be a prerequisite for model choice.
The released open-source benchmark provides a reproducible standard for future domain-adapted embedding models and a technical text corpus for other built-asset NLP tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the variability finding is robust, it implies that similarity of average embeddings is not enough to certify benchmark transferability; the paper's own heatmap shows high similarity to StackExchange and Reddit where rankings nonetheless diverge, so benchmark designers should test ranking agreement, not just vector similarity.
A direct next experiment the paper does not run is fine-tuning a small instruction-tuned model on a slice of these datasets and checking whether it overtakes the 7B-parameter generalist on held-out built-asset text; that would operationalize the paper's call for domain adaptation.
The benchmark's sampling depends on one reference embedding model (mxbai-embed-large-v1) to define both positive diversity and hard negatives, so dataset difficulty is partly conditioned on that model's geometry; rebuilding the subsets with a different anchor model or an ensemble could change which models pass the quality thresholds.
Retrieval and reranking ground truth inherit any inaccuracies in the published IFC-Uniclass mappings used as relevance labels; the paper does not quantify label noise in those mappings.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: useful contribution for the construction informatics crowd, but the headline rankings come with a built-in bias. The authors use mxbai-embed-large-v1 to select hard negatives, diversity-sample positives, and filter clustering subsets—and that model is one of the 24 evaluated. That doesn't kill the paper, but the task instances already encode one participant's geometry, and the paper doesn't acknowledge it or test for it.

What's new: six datasets from IFC 4.3 and Uniclass 1.33, with real effort on the data pipeline—manual review of GPT-4 paraphrased descriptions by two domain experts, careful handling of IFC enumeration types, public GitHub repo. Evaluating 24 models on clustering/retrieval/reranking is the most comprehensive thing in this niche, and the finding that MTEB rankings don't transfer is plausible and useful.

Soft spots. First, the reference-model circularity. Clustering subsets are only included if mxbai scores them below 0.8, and negatives are chosen because mxbai thinks they're similar to the query. That can systematically favor models whose similarity geometry agrees with mxbai. A simple ablation—rebuild a subset with a different embedding model and see how rankings move—would go a long way. Second, the Uniclass descriptions are synthetic: concatenated category labels rephrased by GPT-4. That's a real limitation for any claim about real asset documents, and the paper's language gets strong when it says general-purpose benchmarks are inadequate based on these artificial texts. Third, Table 2 has no variance or significance testing; a one-run difference of one point between adjacent models is noise.

The qualitative conclusion—task-to-task and model-to-model variability is real, and domain-specific evaluation is needed—survives these issues. But the specific rankings should be read as provisional.

Who it's for: anyone building or selecting embedding models for construction/BIM/asset management text. It deserves a serious referee. My bottom line: send out, accept with revisions that address the reference-model dependence and add error bars.

Referee Report

4 major / 4 minor

Summary. The paper introduces a benchmark of pre-trained text embedding models for aligning built asset information, constructed from IFC and Uniclass classification sources. It defines six datasets covering clustering, retrieval, and reranking tasks over sentence- and paragraph-level inputs, evaluates 24 models with standard MTEB-style metrics, and reports substantial performance variability across tasks and models. The authors conclude that general-purpose benchmarks are inadequate for capturing built asset semantics and that domain-specific evaluation and adaptation are needed. The benchmark resources are released as an open-source library.

Significance. If the benchmark construction is valid, this is a useful contribution: it is, to my knowledge, the broadest public evaluation of text embedding models for built asset terminology, with a transparent pipeline, standard metrics, and released code and data. The paper also makes a credible practical point that model rankings differ across tasks and that domain-specific evaluation matters. However, the central claim depends on the benchmark datasets being representative and neutral samples of built asset alignment tasks, and that neutrality is not currently established. The reference-model dependence in dataset construction and the synthetic nature of the Uniclass descriptions are the main threats, and the lack of variance reporting weakens the fine-grained model comparisons.

major comments (4)

[Section 2.4 and Section 3.1.1] The dataset construction uses mxbai-embed-large-v1, which is itself one of the 24 evaluated models, for three load-bearing steps: semantic diversity sampling of positives, hard negative sampling, and the clustering subset inclusion thresholds. Because every task instance is selected according to this model's embedding geometry, models whose similarity judgments correlate with mxbai can be systematically advantaged and those that disagree disadvantaged. The paper does not report an ablation with a different reference model nor exclude mxbai from the evaluation. Without such a robustness check, the observed task-to-task and model-to-model variability cannot be cleanly attributed to domain-specific semantic complexity rather than to the dataset construction mechanism.
[Section 2.3] The Uniclass descriptions are synthetic: they are produced by concatenating hierarchical category labels and then paraphrasing them with gpt-4-turbo, followed by expert review. All clustering, retrieval, and reranking evaluations are computed on these artificial texts rather than on naturally occurring built asset documents such as maintenance manuals, specifications, or submittals. Since the central claim is that general-purpose benchmarks fail to capture the semantic complexity of built asset text, the benchmark's conclusions require evidence that the synthetic paraphrases preserve the lexical and terminological properties of real project documents. At minimum, the claims should be scoped to synthetic descriptions, or a small validation set of real-world texts should be added to test whether model rankings transfer.
[Table 2 and Section 3.1.1] Table 2 reports single score values without variance or repeated runs, while the clustering task trains a mini-batch k-means model, which is stochastic. Several adjacent models differ by less than one point (e.g., clustering-s2s: gte-small 44.31 vs. gte-base 45.96; reranking-s2p: bge-base-en-v1.5 67.01 vs. bge-large-en-v1.5 68.44). Without multiple runs, seeds, standard deviations, or significance tests, the fine-grained rankings and the 'notable variation' narrative are not fully supported. This is a load-bearing issue for any claim that specific models are meaningfully better or worse in this domain.
[Section 4, Table 3] The claim that general-purpose benchmarks are inadequate is partly supported by Table 3, which compares model rankings across only three datasets and reports no correlation statistic or significance test. The rank differences are visually suggestive but are not quantified; Spearman or Kendall correlations with confidence intervals, or a permutation test, are needed to establish that the transfer of rankings from general-purpose datasets to this domain is statistically unreliable. As it stands, Table 3 is anecdotal evidence rather than a demonstrated result.

minor comments (4)

[Section 4, Figure 2] The main text says the thematic similarity heatmap was generated using stella-en-400M-v5, while the Figure 2 caption states mxbai-embed-large-v1 was used. This inconsistency must be resolved because it affects reproducibility and also bears on the reference-model concern.
[Section 2.3] There is a typo in 'Unicalss' in the data enrichment paragraph; it should be 'Uniclass'.
[Section 3.1.3] The reranking task description does not specify how the positive-to-negative ratio (1:3) is implemented across queries, nor whether the same negative pool is reused across queries; adding this detail would improve reproducibility.
[Section 4] The discussion of model size would benefit from a quantitative correlation or regression between parameter count and average score, rather than relying on selected examples such as gte-small.

Circularity Check

1 steps flagged · score 3.0 of 10

Mild benchmark-construction circularity: task instances are selected using mxbai-embed-large-v1's own embeddings, and that same model is then ranked on those instances.

self definitional [Section 2.4 (Sampling), Section 3.1.1 (Clustering), and Table 2]
"Embeddings are generated using a state-of-the-art text embedding model ("mxbai-embed-large-v1"[Li and Li, 2023]). ... For negative sampling, we prioritize the selection of product samples that yield closer semantic similarity to a given query ... using the same embedding model used in the semantic diversity sampling. ... To ensure the quality of the subsets, we evaluate the baseline scores using two embedding models ... A subset is included in the dataset only if its score with the upper threshold model is below 0.8 and greater than 1/N with the baseline model."

mxbai-embed-large-v1 is not an external reference: it is one of the 24 models ranked in Table 2. The positive samples are chosen because mxbai finds them mutually dissimilar, the hard negatives because mxbai finds them close to the query, and the clustering subsets are retained only when mxbai's V-measure is below 0.8. Every reported task score, including the relative ordering of all 24 models, is therefore computed on a dataset whose difficulty is partly defined by one participant's embedding geometry. The paper presents the resulting variability as evidence about domain-specific semantic complexity, but the observed variability could also arise from conditioning on mxbai's geometry.

full rationale

The benchmark has a real but bounded self-reference. Section 2.4 builds the clustering and reranking task instances using mxbai-embed-large-v1's cosine-similarity geometry: positives are chosen for mxbai-dissimilarity, negatives for mxbai-similarity, and Section 3.1.1 keeps clustering subsets only when mxbai's V-measure is below 0.8. Since mxbai appears in Table 2 as one of the 24 ranked models, the dataset is not an independent sample of built asset alignment difficulty, and the rankings and cross-task variability could be partly an artifact of conditioning on one model's geometry. I do not score this higher than 3 because no equation forces any particular score or ranking, the reference model itself lands mid-table rather than being guaranteed a win, and the paper's main observations (task-to-task variability, instruction-tuned models doing better, general-purpose ranking instability) have content beyond the construction loop. A robustness check with a different reference embedding model would resolve the concern. The synthetic GPT-4 Uniclass descriptions and the text/caption inconsistency about the model used for Figure 2 are validity/reporting issues rather than circularity; there is no load-bearing self-citation chain, and the central benchmarking claim is not reduced to a fitted parameter or to a self-cited uniqueness theorem.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim depends on treating classification systems, crosswalk mappings, and synthetic descriptions as faithful proxies for real built asset semantics. No new physical or theoretical entities are introduced. The benchmark's quantitative scores depend on several hand-set thresholds and ratios, which are transparently reported but not varied or justified with sensitivity analysis.

free parameters (2)

Clustering subset inclusion thresholds = 0.8 upper threshold and 1/N lower threshold
Hand-set thresholds determine which clustering subsets are retained in the benchmark (Section 3.1.1). They influence all clustering scores and are chosen by the authors to keep task difficulty in a target range.
Positive-to-negative sampling ratio = 1:3
Applied to create the reranking evaluation sets (Section 4). This hand-chosen ratio affects reranking difficulty and therefore the reported MAP scores.

assumptions (5)

domain assumption IFC schema and Uniclass product table are valid ground-truth sources for built product semantics.
Section 2.1 relies on these classification systems as the source of product names, descriptions, and labels; if their categorization is not semantically coherent, all tasks inherit that.
domain assumption NBS IFC-Uniclass crosswalk mappings are correct relevance ground truth for retrieval and reranking.
Sections 3.1.2 and 3.1.3 derive query-document labels from these published mappings without independent validation.
domain assumption GPT-4 paraphrasing and manual review preserve factual content of synthesized descriptions.
Section 2.3 states prompts were designed to prevent fact changes and experts reviewed, but no systematic evaluation of semantic fidelity is provided.
standard math Cosine similarity of embedding vectors is an adequate semantic similarity measure for ranking and retrieval.
Used throughout Section 3 for sampling and evaluation; assumes embedding space geometry reflects domain semantics.
standard math Clustering quality is appropriately measured by V-measure with k set to the number of labels.
Section 3.1.1; relies on knowing true labels and k, a simplification compared with unsupervised discovery.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking pre-trained text embedding models in aligning built asset information." pith.science (2026). https://pith.science/paper/63VUPPMO

@misc{pith2026241112056,
  author       = {Pith},
  title        = {Pith review of: Benchmarking pre-trained text embedding models in aligning built asset information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63VUPPMO}},
  note         = {Machine review of arXiv:2411.12056}
}

read the original abstract

Accurate mapping of the built asset information to established data classification systems and taxonomies is crucial for effective asset management, whether for compliance at project handover or ad-hoc data integration scenarios. Due to the complex nature of built asset data, which predominantly comprises technical text elements, this process remains largely manual and reliant on domain expert input. Recent breakthroughs in contextual text representation learning (text embedding), particularly through pre-trained large language models, offer promising approaches that can facilitate the automation of cross-mapping of the built asset data. However, no comprehensive evaluation has yet been conducted to assess these models' ability to effectively represent the complex semantics specific to built asset technical terminology. This study presents a comparative benchmark of state-of-the-art text embedding models to evaluate their effectiveness in aligning built asset information with domain-specific technical concepts. Our proposed datasets are derived from two renowned built asset data classification dictionaries. The results of our benchmarking across six proposed datasets, covering three tasks of clustering, retrieval, and reranking, highlight the need for future research on domain adaptation techniques. The benchmarking resources are published as an open-source library, which will be maintained and extended to support future evaluations in this field.

Figures

Figures reproduced from arXiv: 2411.12056 by the authors.

**Figure 2.** Thematic similarity heatmap between our proposed clustering tasks and those from MTEB. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[2]

buildingSmart International

Accessed: 2024-06-24. buildingSmart International. buildingsmart data dictionary (bsdd). https://www.buildingsmart.org/ users/services/buildingsmart-data-dictionary/ , 2024a. Accessed: 2024-06-24. buildingSmart International. Ifc 4.3 documentation. https://standards.buildingsmart.org/IFC/ RELEASE/IFC4_3/, 2024b. Accessed: 2024-06-24. X. Cao and M. Kosinsk...

work page 2024
[3]

Cappuzzo, P

R. Cappuzzo, P. Papotti, and S. Thirumuruganathan. Creating embeddings of heterogeneous relational datasets for data integration tasks. In Proceedings of the 2020 ACM SIGMOD international conference on management of data, pages 1335–1349,

work page 2020
[5]

Forth, P

K. Forth, P. Berggold, and A. Borrmann. Domain-specific fine-tuning of llm for material matching of bim elements and material passports. In Proc. of 2024 ASCE International Conference on Computing in Civil Engineering,

work page 2024
[6]

C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024a. J. Lee, Z. Dai, X. Ren, B. Chen, D. Cer, J. R. Cole, K. Hui, M. Boratko, R. Kapadia, W. Ding, et al. Gecko: Versatile text embeddings distilled from large language ...

arXiv
[7]

Accessed: 2024-10-20. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26,

work page 2024
[11]

Reimers and I

11 N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

work page 2019
[2018]

Rosenberg and J

A. Rosenberg and J. Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 410–420,

work page 2007
[2019]

URL http://arxiv.org/abs/1908.10084. C. J. Roberts, E. A. Pärn, D. J. Edwards, and C. Aigbavboa. Digitalising asset management: concomitant benefits and persistent challenges. International Journal of Building Pathology and Adaptation, 36(2):152–173,

arXiv 1908

Show all 13 references

[2020]

Enevoldsen, M

K. Enevoldsen, M. Kardos, N. Muennighoff, and K. L. Nielbo. The scandinavian embedding benchmarks: Com- prehensive assessment of multilingual and monolingual text embedding. arXiv preprint arXiv:2406.02396,

arXiv
[2021]

Pennington, R

J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543,

2014
[2022]

National building specification

NBS. National building specification. https://www.thenbs.com/, 2024a. Accessed: 2024-06-24. NBS. Uniclass. https://uniclass.thenbs.com/, 2024b. Accessed: 2024-06-24. M. Ostendorff, E. Ash, T. Ruas, B. Gipp, J. Moreno-Schneider, and G. Rehm. Evaluating document represen- tation...

2024
[2023]

Muennighoff, N

N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316,

arXiv
[2024]

10 Ifcopenshell

Accessed: 2024-06-24. 10 Ifcopenshell. https://github.com/IfcOpenShell/IfcOpenShell/,

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [2]

buildingSmart International

Accessed: 2024-06-24. buildingSmart International. buildingsmart data dictionary (bsdd). https://www.buildingsmart.org/ users/services/buildingsmart-data-dictionary/ , 2024a. Accessed: 2024-06-24. buildingSmart International. Ifc 4.3 documentation. https://standards.buildingsmart.org/IFC/ RELEASE/IFC4_3/, 2024b. Accessed: 2024-06-24. X. Cao and M. Kosinsk...

work page 2024

[2] [3]

Cappuzzo, P

R. Cappuzzo, P. Papotti, and S. Thirumuruganathan. Creating embeddings of heterogeneous relational datasets for data integration tasks. In Proceedings of the 2020 ACM SIGMOD international conference on management of data, pages 1335–1349,

work page 2020

[3] [5]

Forth, P

K. Forth, P. Berggold, and A. Borrmann. Domain-specific fine-tuning of llm for material matching of bim elements and material passports. In Proc. of 2024 ASCE International Conference on Computing in Civil Engineering,

work page 2024

[4] [6]

C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024a. J. Lee, Z. Dai, X. Ren, B. Chen, D. Cer, J. R. Cole, K. Hui, M. Boratko, R. Kapadia, W. Ding, et al. Gecko: Versatile text embeddings distilled from large language ...

arXiv

[5] [7]

Accessed: 2024-10-20. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26,

work page 2024

[6] [11]

Reimers and I

11 N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

work page 2019

[7] [2018]

Rosenberg and J

A. Rosenberg and J. Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 410–420,

work page 2007

[8] [2019]

URL http://arxiv.org/abs/1908.10084. C. J. Roberts, E. A. Pärn, D. J. Edwards, and C. Aigbavboa. Digitalising asset management: concomitant benefits and persistent challenges. International Journal of Building Pathology and Adaptation, 36(2):152–173,

arXiv 1908