REVIEW 4 major objections 5 minor 20 references
An end-to-end pipeline classifies real-estate questionnaires and extracts 35 fixed property attributes from selectable-text documents at scale with an LLM.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-08 19:38 UTC pith:A6YJCE7R
load-bearing objection Working proptech pipeline that extracts 35 fixed attributes from selectable-text questionnaires at a few-thousand scale, but reliability rests on internal consistency and a weak silhouette rather than ground-truth field accuracy. the 4 major comments →
Structured Data Extraction from Real Estate Documents using Clustering, Classification, and Large Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Automated extraction of 35 predefined property attributes from selectable-text real-estate questionnaire documents is feasible and reliable at scale: all 2781 submitted documents were processed successfully into 2766 unique records, cosine-similarity matching reaches a Jaccard consistency of 0.82, and K-Means on the extracted features yields interpretable market segments with silhouette 0.2088.
What carries the argument
A three-way structural classifier (text_only / scanned / special_char) that routes only selectable-text documents into a structured-output prompt for DeepSeek R1, forcing the model to emit a fixed 35-field JSON schema of property attributes.
Load-bearing premise
That a Jaccard consistency of 0.82 from cosine matching and a K-Means silhouette of 0.2088 are adequate proof of extraction reliability, and that success on the selectable-text subset alone supports the paper’s broader claims about heterogeneous questionnaires.
What would settle it
A hand-labeled gold set of several hundred selectable-text questionnaires on which field-level exact or fuzzy match rates against the LLM JSON fall well below the reported consistency scores, or where many of the 35 attributes systematically disagree with the source text.
If this is right
- Listing platforms can expose legal status, heating, utilities and structural condition as searchable structured fields instead of buried PDFs.
- Market-segmentation and valuation models can incorporate attributes that previously required manual document reading.
- The same classify-then-extract pattern can be reapplied to other listing inventories that attach selectable-text questionnaires.
- Cosine-similarity and K-Means checks can serve as automated, label-free quality gates on the extracted JSON records.
Where Pith is reading between the lines
- Because scanned and special-character layouts are only classified, not extracted, the reliability claim currently covers only the selectable-text slice of the heterogeneous population the introduction highlights.
- A modest hand-annotated evaluation set would convert the indirect Jaccard and silhouette numbers into direct field-level precision and recall that can be compared across models.
- Once reverse-engineered API access is replaced by official partner endpoints, the same pipeline could run continuously on new listings rather than as a one-shot crawl.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents an end-to-end pipeline that acquires real-estate questionnaire documents via reverse-engineered REST APIs, classifies them into three layout categories (text_only, scanned, special_char), and extracts 35 predefined property attributes from the selectable-text subset using DeepSeek R1 prompted for structured JSON. Applied to 3965 documents from a live platform, the pipeline successfully processes all 2781 text_only documents into 2766 unique property records. Downstream checks report cosine-similarity matching with Jaccard consistency 0.82 and K-Means market segments with silhouette score 0.2088; the authors conclude that extraction is feasible and reliable at this scale.
Significance. If the reliability claim holds, the work supplies a practical, scalable route to unlock rich property-level attributes (legal status, structural condition, utilities, heating) that currently remain trapped in heterogeneous questionnaire attachments and are invisible to listing APIs. Processing nearly three thousand documents end-to-end with an open LLM and a fixed JSON schema is a concrete engineering contribution for real-estate data platforms and related PropTech applications. The layout taxonomy and the explicit restriction of extraction to selectable text are useful design choices that other practitioners can reuse. The absence of machine-checked proofs or fully parameter-free derivations is expected for an applied systems paper; the main scientific value therefore rests on the strength of the empirical reliability evidence.
major comments (4)
- [Abstract results paragraph; validation section] The central claim that extraction is 'feasible and reliable at this scale' is supported only by (i) 100 % JSON parse success on the 2781 text_only documents and (ii) two internal consistency metrics (Jaccard 0.82 from cosine-similarity matching, silhouette 0.2088 from K-Means). Neither metric measures field-level correctness against the source questionnaire text. Processing success shows only that the LLM returned parseable JSON conforming to the 35-attribute schema; systematic extraction errors (hallucinated values, missed checkboxes transcribed as free text, swapped fields) would still produce valid JSON. Without a held-out human annotation set reporting precision/recall or exact-match accuracy per attribute, the reliability conclusion remains under-supported.
- [Abstract; K-Means results] A silhouette coefficient of 0.2088 indicates only weak cluster structure. In the standard interpretation of silhouette scores this value is far below the conventional threshold for 'reasonable' separation and cannot by itself underwrite the claim of 'interpretable market segments' that confirm extraction fidelity. If the authors wish to retain the clustering result as supporting evidence, they should either (a) report additional internal validation (e.g., stability under bootstrap resampling, comparison against a null model) or (b) demote the silhouette figure to a secondary exploratory observation rather than a pillar of the reliability argument.
- [Problem statement; pipeline description; results] The problem statement emphasises heterogeneous document types—digitally generated selectable text, scanned physical forms, and checkbox-heavy layouts that defeat conventional OCR. The pipeline correctly classifies all three categories, yet extraction is performed exclusively on the text_only subset. Consequently the headline reliability claim does not cover the very document classes that motivate the work. The manuscript should either (a) extend extraction (or a calibrated OCR+LLM path) to a non-trivial sample of scanned and special_char documents and report the same accuracy metrics, or (b) explicitly scope the reliability claim to selectable-text documents and reframe the contribution accordingly.
- [Downstream validation / cosine-similarity matching] The Jaccard consistency of 0.82 is obtained via cosine-similarity matching of the extracted attribute vectors. If the matching is performed against other extracted records or against overlapping platform metadata rather than against independent human labels of the source documents, the score largely reflects self-agreement of the extraction process (or agreement with already-structured API fields) and can remain high even under systematic bias. The manuscript must clarify the exact reference set used for the cosine matching and, ideally, replace or augment it with a ground-truth comparison on a manually annotated sample.
minor comments (5)
- [Abstract] The abstract states 'all 2781 submitted documents were processed successfully, producing a final dataset of 2766 unique property records.' The 15-document discrepancy is never explained; a brief note on de-duplication criteria would remove ambiguity.
- [Methods / schema definition] The 35 predefined attributes are never listed in the provided text. A table or appendix enumerating the JSON schema (attribute name, type, allowed values) would make the extraction target fully reproducible.
- [LLM extraction subsection] Prompt engineering details for DeepSeek R1 (system prompt, few-shot examples, temperature, retry logic) are omitted. Because the reliability claim rests on the LLM stage, these details belong in the main text or a supplementary repository.
- [Results] The silhouette score is reported to four decimal places (0.2088) while the Jaccard score is given to two (0.82). Consistent precision and, preferably, confidence intervals or bootstrap standard errors would improve presentation.
- [Data acquisition] Reverse-engineering of the live platform's REST APIs raises reproducibility and ethical questions. A short statement on data-use permissions, rate-limiting, and whether the collected documents can be shared under a research licence would strengthen the paper.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The four major comments correctly identify that our reliability argument rests too heavily on parse success and internal consistency metrics, that the silhouette score is weak evidence of extraction fidelity, that the headline claim is scoped only to selectable-text documents, and that the cosine/Jaccard procedure needs a clearer reference set. We accept these points. In revision we will (i) add a manually annotated held-out sample with per-attribute exact-match / precision-recall, (ii) demote K-Means to an exploratory observation and soften related wording, (iii) explicitly restrict the reliability claim to text_only documents and reframe the contribution accordingly, and (iv) clarify the cosine-matching reference set and augment it with the new ground-truth comparison. We believe these changes address the under-supported reliability conclusion while preserving the engineering contribution of the end-to-end pipeline.
read point-by-point responses
-
Referee: The central claim that extraction is 'feasible and reliable at this scale' is supported only by (i) 100% JSON parse success on the 2781 text_only documents and (ii) two internal consistency metrics (Jaccard 0.82, silhouette 0.2088). Neither measures field-level correctness against source text. Without held-out human annotation reporting precision/recall or exact-match per attribute, the reliability conclusion remains under-supported.
Authors: We agree. Parse success and schema conformance show only that DeepSeek R1 returned valid JSON for all 2781 text_only documents; they do not rule out systematic field-level errors (hallucinations, missed checkboxes, swapped attributes). Internal consistency metrics are likewise insufficient as primary evidence of correctness. In revision we will: (1) draw a stratified held-out sample of source questionnaires, obtain independent human labels for the 35 attributes, and report per-attribute exact-match accuracy and precision/recall; (2) soften the abstract and conclusion so that 'reliable' is tied to these ground-truth metrics rather than to parse success alone; and (3) retain 100% parse success as an engineering result (schema adherence at scale) but not as a substitute for field-level evaluation. This directly supplies the missing evidence the referee requests. revision: yes
-
Referee: A silhouette coefficient of 0.2088 indicates only weak cluster structure, far below conventional thresholds for 'reasonable' separation, and cannot underwrite 'interpretable market segments' that confirm extraction fidelity. Authors should either report additional internal validation (stability, null model) or demote the silhouette figure to a secondary exploratory observation.
Authors: The referee’s reading of the silhouette score is correct: 0.2088 indicates weak structure and is inadequate as a pillar of the reliability argument. We will not treat clustering as confirmatory evidence of extraction fidelity. In revision we will (a) demote the K-Means result to a brief exploratory observation in the discussion, (b) remove or rephrase claims that the segments 'confirm' data quality or extraction fidelity, and (c) note the modest silhouette explicitly so readers are not misled. We do not plan to elevate clustering with bootstrap stability or null-model tests, because that would still not measure field-level correctness against the source documents; the human annotation study (Comment 1) is the appropriate primary validation path. revision: yes
-
Referee: The problem statement emphasises heterogeneous types (selectable text, scanned forms, checkbox-heavy layouts), yet extraction is performed only on text_only. The headline reliability claim therefore does not cover the document classes that motivate the work. Either extend extraction (or OCR+LLM) to a non-trivial sample of scanned/special_char documents with the same metrics, or explicitly scope the reliability claim to selectable-text documents and reframe the contribution.
Authors: We accept this scoping criticism. Classification covers all three layout classes, but structured extraction—and thus the reliability claim—applies only to the 2781 text_only documents. Extending a calibrated OCR+LLM path to scanned and special_char documents with the same accuracy metrics is valuable future work and is beyond what we can rigorously complete for this revision without introducing a second, under-evaluated subsystem. We will therefore take option (b): explicitly scope the reliability and feasibility claims to selectable-text (text_only) documents in the abstract, problem statement, results, and conclusion; state clearly that scanned and special_char remain classified but not extracted; and reframe the contribution as (i) large-scale acquisition and three-way layout classification, and (ii) schema-constrained LLM extraction demonstrated on the selectable-text subset, with OCR-based extension left as future work. This removes the overclaim without misrepresenting what was done. revision: yes
-
Referee: The Jaccard consistency of 0.82 is obtained via cosine-similarity matching of extracted attribute vectors. If matching is against other extracted records or overlapping platform metadata rather than independent human labels, the score largely reflects self-agreement and can remain high under systematic bias. Clarify the exact reference set and, ideally, replace or augment it with ground-truth comparison on a manually annotated sample.
Authors: The referee is right that the reference set must be stated unambiguously and that agreement with other extractions or with already-structured API fields cannot establish field-level correctness. In the current manuscript the cosine/Jaccard procedure is insufficiently specified and is not a substitute for human labels. In revision we will: (1) document exactly which vectors are compared (extracted records vs. which reference fields, any filters, and how Jaccard is derived from the matches); (2) treat the 0.82 figure as an internal-consistency / cross-source agreement statistic only, not as accuracy; and (3) augment—and where appropriate replace—this metric with the held-out human annotation study described under Comment 1, reporting attribute-level exact-match and precision/recall against the source questionnaire text. That ground-truth comparison is the appropriate primary measure of extraction quality. revision: yes
Circularity Check
No circular derivation found; pipeline results and internal consistency metrics are empirical outcomes, not reductions of inputs by construction.
full rationale
This is an applied systems paper describing an end-to-end engineering pipeline (API acquisition, structural classification into text_only/scanned/special_char, DeepSeek R1 JSON extraction of 35 attributes, then cosine-similarity and K-Means). The load-bearing claims—100% processing success on 2781 selectable-text documents yielding 2766 records, Jaccard consistency 0.82, silhouette 0.2088—are reported empirical measurements after running the pipeline, not quantities defined in terms of each other or forced by a fitted parameter renamed as a prediction. There are no uniqueness theorems, no self-citation chains that forbid alternatives, no ansatz smuggled via prior author work, and no renaming of a known empirical law. Downstream cosine matching and clustering operate on the extracted feature vectors and may constitute weak (non-independent) evidence of extraction fidelity if they lack human ground-truth labels against source text; that is a correctness/validation-strength concern, not circularity by construction. Per the analyzer rules, weak external grounding or internal-consistency metrics do not raise the circularity score when no equation or claim reduces to its own inputs. The derivation is therefore self-contained as an empirical systems result; score 0 with empty steps is the warranted finding.
Axiom & Free-Parameter Ledger
free parameters (3)
- 35 predefined property attributes (JSON schema) =
35 fields (names not listed in abstract)
- document layout taxonomy (text_only / scanned / special_char) =
3 classes
- K-Means configuration for market segments
axioms (4)
- domain assumption DeepSeek R1 prompted for structured JSON can extract the 35 attributes from selectable-text questionnaires with accuracy high enough to call the pipeline reliable.
- domain assumption Reverse-engineered REST API access yields a representative corpus of questionnaire documents from the live platform.
- ad hoc to paper Cosine-similarity matching yielding Jaccard 0.82 and K-Means silhouette 0.2088 are appropriate proxies for extraction data quality.
- standard math Standard document classification and clustering methods (unspecified algorithms beyond K-Means) behave as expected on this corpus.
read the original abstract
Real estate property listings expose structured metadata through the API. Still, the richest property-level information (i.e., legal status, structural condition, utility supplies, heating systems) sits in attached questionnaire documents that no automated system currently processes at scale. These documents are heterogeneous. Some are digitally generated with selectable text, others are scanned physical forms. There are even more complex layouts that contain checkbox annotations that defeat conventional text extraction. In this paper, we present an end-to-end pipeline for acquiring, classifying, and extracting structured data from selectable text documents. The pipeline was applied to 3965 questionnaire documents collected from a live property platform via reverse-engineered REST APIs. First, we classified each document into one of three structural categories (text_only, scanned, and special_char), then extracted 35 predefined property attributes from eligible documents using DeepSeek R1 as the Large Language Model, prompted to return a structured JSON object. All 2781 submitted documents were processed successfully, producing a final dataset of 2766 unique property records. Downstream validation confirmed the data quality. Cosine similarity matching achieves a Jaccard consistency score of 0.82, and K-Means clustering produces interpretable market segments with a silhouette score of 0.2088. Results show that the proposed extraction from each property document is both feasible and reliable at this scale.
Figures
Reference graph
Works this paper leans on
-
[1]
The Law Society: Transaction (TA) forms – Property Information Form (TA6).https://www.lawsociety.org.uk/topics/property/transaction-forms, last accessed 2025/04/01
work page 2025
- [2]
-
[3]
Nature Communications15, 1418 (2024).https://doi.org/10
Dunn, A., et al.: Structured information extraction from scientific text with large language models. Nature Communications15, 1418 (2024).https://doi.org/10. 1038/s41467-024-45563-x
work page 2024
-
[4]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI: DeepSeek-R1: Incentivizing reasoning capability in LLMs via rein- forcement learning. arXiv preprint arXiv:2501.12948 (2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[5]
Groq Inc.: Groq API documentation.https://console.groq.com/docs/, last ac- cessed 2025/04/01
work page 2025
-
[6]
Freedman, J.: pdfplumber: Plumb a PDF for detailed information about each char, rectangle, and line.https://github.com/jsvine/pdfplumber, last accessed 2025/04/01
work page 2025
-
[7]
readthedocs.io, last accessed 2025/04/01
Artifex Software: PyMuPDF – Python bindings for MuPDF.https://pymupdf. readthedocs.io, last accessed 2025/04/01
work page 2025
-
[8]
In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp
Bast, H., Korzen, C.: A benchmark and evaluation for text extraction from PDF. In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 99–108. IEEE, Toronto (2017).https://doi.org/10.1109/JCDL.2017.7991564
-
[9]
A Survey of Deep Learning Approaches for OCR and Document Understanding
Subramani, N., Matton, A., Greaves, M., Lam, A.: A survey of deep learning ap- proaches for OCR and document understanding. arXiv preprint arXiv:2011.13534 (2020)
work page internal anchor Pith review Pith/arXiv arXiv 2011
-
[10]
In: Ninth International Con- ference on Document Analysis and Recognition (ICDAR 2007), vol
Smith, R.: An overview of the Tesseract OCR engine. In: Ninth International Con- ference on Document Analysis and Recognition (ICDAR 2007), vol. 2, pp. 629–633. IEEE (2007)
work page 2007
-
[11]
In: Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1192–
-
[12]
ACM, New York (2020)
work page 2020
-
[13]
In: Proceedings of the 30th ACM International Conference on Multimedia, pp
Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: Pre-training for docu- ment AI with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 4083–4091. ACM, New York (2022)
work page 2022
-
[14]
arXiv preprint arXiv:2510.10138 (2025)
Hybrid OCR-LLM framework for enterprise-scale document information extrac- tion. arXiv preprint arXiv:2510.10138 (2025)
-
[15]
BMJ Health & Care Informatics32(1), e101139 (2025)
Large language models for data extraction from unstructured and semi-structured electronic health records. BMJ Health & Care Informatics32(1), e101139 (2025). https://doi.org/10.1136/bmjhci-2024-101139
-
[16]
In: ICDAR 2023 Workshops, LNCS, vol
Yu, W., et al.: ICDAR 2023 competition on structured text extraction from visually-rich document images. In: ICDAR 2023 Workshops, LNCS, vol. 14193, pp. 1–18. Springer, Heidelberg (2023)
work page 2023
-
[17]
Utilizing Large Language Models for Information Extraction from Real Estate Transactions
Zhao, Y., et al.: Utilizing large language models for information extraction from real estate transactions. arXiv preprint arXiv:2404.18043 (2024)
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[18]
chrome.com/docs/devtools/network/reference, last accessed 2025/04/01
Google: Chrome DevTools – Network features reference.https://developer. chrome.com/docs/devtools/network/reference, last accessed 2025/04/01
work page 2025
-
[19]
Reitz, K.: Requests: HTTP for Humans.https://docs.python-requests.org, last accessed 2025/04/01 Real Estate Clustering 17
work page 2025
-
[20]
International Journal of Advances in Soft Computing & Its Ap- plications13(3) (2021)
Khder, M.A.: Web scraping or web crawling: State of art, techniques, approaches and application. International Journal of Advances in Soft Computing & Its Ap- plications13(3) (2021)
work page 2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.