{"id":"ab04ea7e-1899-4d95-b33e-c4e723969ee2","arxiv_id":"1909.00704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An automated valuation model for Italian homes works best when it uses official OMI price bands and nearby comparable advertised prices as features.","lead":"A machine learning system estimates home values by combining professional appraisals, official price zones, and advertised prices of nearby homes. It reports about 21,000 euros average error on Turin homes, but the claim that it works across Italy is tested on only 58 properties.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 17,000-euro out-of-sample result requires that OMI bands and comparable-property prices were computed as of each property's appraisal date; the paper does not document this temporal alignment, so the nationwide generalization claim is not yet supported.","rationale":"The paper's strongest independent evidence for nationwide generalization is the 58-property out-of-sample result in Section 5.3. I examined the feature construction in Sections 3.2.1 and 3.3 and found no statement of temporal alignment for this test. OMI values are explicitly time-varying, and comparable advertised prices are market snapshots; using post-appraisal values would encode the future into features. This is a correctness risk, not merely a missing detail, because the model is intended to produce valuations at the time of request. The same issue is less severe for the Turin test set because train and test periods overlap, but the out-of-sample set spans 2012-2016 and is described as 'totally new and uncorrelated.' Without the temporal control, the 17k error cannot be distinguished from an artifact of lookahead. The proposed test is straightforward: re-run the 58-property evaluation with date-stamped features. This does not make the paper's approach unsound; it makes the current evidence insufficient, supporting the reader's CONDITIONAL verdict. I agree with the reader's weakest_assumption that the comparables pipeline for out-of-sample data is the key unverified step.","tokens_in":11655,"tokens_out":4284,"duration_ms":54546,"concrete_test":"Reproduce the out-of-sample evaluation with explicit temporal controls: for each of the 58 properties, (1) use the OMI min/max band in force for the semester of the appraisal date, (2) compute comparables only from listings/appraisals active on or before that date, excluding the target property itself and any appraisal in the Turin training set, and (3) record the crawl date and query parameters. Recompute the 17,000-euro error under these constraints and also report the error using OMI+surface without comparables. If the error rises materially or if comparables no longer improve accuracy, the published out-of-sample result is contaminated by lookahead.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 reports a Mean Error of 17,000 euros on 58 properties appraised between 2012 and 2016, and uses it to support the claim that the Turin-trained model generalizes to the whole Italian territory. For this claim to be valid, two features must be available at the valuation date: the OMI min/max price band for the property's OMI area and the average price per square meter of comparable properties. Section 3.2.1 explains that OMI ranges are updated every six months, yet Section 5.3 never states which OMI release was used for the 58 properties or whether it matched the semester of each appraisal. Section 3.3 describes the comparable-property crawler but gives no crawl date, query radius, filters, or de-duplication rule for the out-of-sample set, and it notes only that the valuation date of previous appraisals 'should be taken into account' without confirming that web comparables were restricted to listings active before the appraisal date. If OMI values or comparable prices were crawled at a later date, the model was given information not available at valuation time, making the 17,000-euro error optimistic rather than a prospective measure. This is the single most load-bearing gap because the whole nationwide applicability claim rests on this 58-property experiment; the Turin test set alone cannot support generalization to other cities and time periods.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated valuation model (AVM) for Italian residential real estate, integrating three data sources: professional appraisal records, open geographic data from the Italian Revenue Agency's OMI observatory and points-of-interest databases, and advertised prices of comparable properties obtained by web crawling. After cleaning, 3983 Turin appraisals are used to train several regressors, with three feature sets: a full hedonic set, a reduced OMI-centered set (OMI min/max price bands plus surface), and the same set augmented with the average price per square meter of comparable properties. The best models achieve an RMSE of roughly 21,000 euros (R2 about 0.96) on a held-out Turin test set, and the paper reports a mean error of about 17,000 euros on 58 out-of-sample properties located in different Italian cities and appraised in 2012-2016. The central claim is that a model trained only on Turin data can be used for real-time valuations across the whole Italian territory.","tokens_in":11986,"tokens_out":4191,"duration_ms":48171,"significance":"If the out-of-sample result is valid, the paper offers a practically relevant result: a small set of features, combining official OMI price bands with web-crawled comparable prices, predicts professional appraisals at an accuracy that domain experts reportedly find acceptable. The use of external benchmarks (OMI bands and advertised comparables) rather than a refit of the target variable reduces, though does not eliminate, circularity concerns, and the use of a separate test set is good practice. The main strengths are the realistic appraisal dataset, the explicit comparison of feature sets, and the attempt to support spatial and temporal generalization with an out-of-sample test. However, the generalization claim currently rests on an underdocumented 58-property experiment, so the significance is conditional on closing that gap.","major_comments":[{"comment":"The nationwide generalization claim rests on the 58-property out-of-sample result, but the paper does not document that the OMI and comparable-property features were computed as of each property's appraisal date. Section 3.2.1 states that OMI price ranges are updated every six months, and Section 5.3 covers appraisals from 2012 to 2016, yet the paper never states which OMI release was used for the 58 properties. Similarly, Section 3.3 describes the comparable crawler for the Turin data but gives no crawl date, query radius, or rule for excluding listings posted after the appraisal date for the out-of-sample set. If these features were obtained from later data, the 17,000-euro error is optimistic and not a prospective measure. This is the most load-bearing gap in the paper.","section":"Section 5.3 and Sections 3.2.1/3.3"},{"comment":"The feature selection process is not described as a fully independent procedure. Section 4.1 explains that the reduced OMI-centered feature set was adopted after examining feature importances from models trained on the data, and Section 5.2 then reports test-set results for that selected feature set. Because the feature set was chosen using knowledge of the same data (and potentially the same test set), the reported errors may be optimistic. The paper should state explicitly that feature selection was performed only on the training partition, or use a nested/outer holdout procedure, and should report results over multiple random splits to quantify variance.","section":"Section 4.1 and Sections 5.1-5.2"},{"comment":"What the paper calls 'Mean Error' (ME) is in fact the root mean square error: Eq. (3) defines ME as sqrt((1/n) sum (y_i - yhat_i)^2). This is not a mean error in the standard sense (which would be the mean absolute or mean signed error). The terminology propagates to the abstract, Section 5.2, and Section 5.3, where the out-of-sample 'Mean Error of 17,000 Euros' should be called 'RMSE of 17,000 Euros'. The mislabeling makes the reported accuracy appear to be a bias measure rather than an error magnitude and should be corrected throughout.","section":"Section 5.1, Eq. (3), and Tables 2-4"},{"comment":"The claim that the model is 'predictive and practically effective for the whole Italian territory' is stronger than the evidence. The out-of-sample set contains only 58 properties from unspecified Italian cities, and no information is given about their distribution across cities, market segments, or time periods. Even with perfect temporal alignment, 58 properties are a thin basis for a nationwide applicability claim, and the paper should either add details about the composition of this set or temper the conclusion to a preliminary indication.","section":"Section 5.3 and Conclusions"}],"minor_comments":[{"comment":"The appraisal years are stated inconsistently: Section 3.1 says the original data set contains valuations performed between 2011 and 2016, while Section 5.3 says the Turin valuations used to train the model were performed between 2015 and 2016. Please clarify which years were used after cleaning and how the discrepancy arises.","section":"Section 3.1 vs Section 5.3"},{"comment":"The reported partition sizes (2789 + 596 + 596 = 3981) do not sum to the stated 3983 appraisals. Please check the counts and explain how the 70/15/15 split was applied.","section":"Section 4.2"},{"comment":"The units of MSE should be stated explicitly. Since ME is expressed in thousands of euros, MSE is in (thousands of euros)^2, which is an unusual unit and may confuse readers.","section":"Tables 2-4"},{"comment":"The statement that an error around 21,000 euros has been 'considered acceptable by the domain experts' would be more convincing if the acceptability criterion were defined, for example as a percentage of mean property value or as a comparison with expert disagreement between appraisers.","section":"Section 5.2"},{"comment":"The claim of outperforming the state of the art in [16] is not supported by a direct comparison table or by re-evaluation of the same benchmark data. Please either include such a comparison or rephrase the claim as an indicative comparison.","section":"Section 2 and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a preprint-style submission with no code or data release, which makes independent verification difficult. The central issue is that the out-of-sample generalization result, which carries the paper's main claim, depends on temporal alignment of the OMI and comparable-property features; the manuscript does not document this. If the authors can supply the missing details or re-run the experiment with clearly contemporaneous features, the paper would be considerably stronger. Otherwise, the nationwide claim should be removed or substantially qualified. The recommendation is therefore major_revision rather than reject, because the methodological approach is sound and the missing pieces appear addressable in principle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper has a genuinely useful empirical finding — for Italian residential appraisals, OMI official price bands plus surface and average comparable prices beat a large hedonic feature set, with RMSE around €21k on a held-out Turin test set. That is a plausible, practically relevant result for the AVM world. The second thing: the nationwide generalization claim rests on a 58-property out-of-sample set, and the paper never documents whether OMI bands and comparable listings were as of each appraisal date. That gap is load-bearing, and the stress-test note is right to flag it.\n\nWhat is actually new: the OMI-centered feature set is a nice local insight, and using professional appraisals as the target (rather than transaction or advertised prices) is a sensible design choice that matches how appraisal companies work. The paper also does the right thing in comparing several regressors and showing that the compact feature set is robust across them. The cleaned data pipeline and the PoI features are standard, but applied carefully.\n\nSoft spots, in proportion. The out-of-sample temporal alignment is the big one: section 5.3 gives appraisal years for the 58 properties but no OMI release semester and no crawl dates or filters for comparables. If those features came from a later date, the €17k error is optimistic. That is an addressable problem — describe the pipeline as-run. Second, the evaluation has no error bars or repeated splits; a single random split plus feature selection informed by the same data leaves the reported R² slightly exposed. Minor: the paper calls RMSE 'Mean Error' — that name is wrong but the formula is correct. Minor: no code or data, so the results are not independently checkable. The literature review is adequate and the claims about outperforming De Nadai & Lepri are not backed by a direct comparison on the same benchmark, but that is a side comment.\n\nWho this is for: applied researchers and practitioners building AVMs, especially in Italy. The OMI insight is worth knowing even if the generalization claim needs work. I think it deserves a serious referee: the core finding is interesting and the flaws are fixable. I would send it to review with a request for temporal alignment details, repeated cross-validation, and ideally data/code.\n\nBest.","headline":"Useful applied finding: OMI price bands plus comparables predict Turin appraisals well, but the out-of-sample generalization claim is not yet supported because the paper doesn't document feature timing.","tokens_in":12460,"tokens_out":2250,"would_cite":false,"duration_ms":25451,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a three-feature automated valuation model can reproduce professional real-estate appraisals to within about €21,000 using OMI price bands, surface, and comparable-property prices.","keywords":["automated valuation models","real estate appraisal","machine learning","OMI price bands","comparable properties","web crawling","open data","feature selection"],"falsifier":"Re-run the out-of-sample test with date-restricted features: for each of the 58 properties, crawl comparables and fetch OMI bands only from the period on or before the appraisal date, and check whether the error stays near €17,000. The paper's Section 5.3 reports the result but does not describe the comparables pipeline for these properties, so this is where the generalization claim can be tested.","tokens_in":11471,"feed_emoji":"🏠","tokens_out":8725,"duration_ms":206555,"temperature":0.7,"pith_summary":"This paper argues that a practical automated valuation model for Italian residential property can be built from heterogeneous web and open data rather than from detailed on-site inspection. Its central claim is that once three numbers are known—the official OMI price band for the property's zone, the surface in square meters, and the average advertised or appraised price per square meter of nearby comparable properties—the remaining structural and neighborhood features add almost nothing. Trained on professional appraisals in Turin and evaluated on a held-out test set, the best regressors reach a root mean square error of about €21,000 with $R^2\\approx 0.96$, and about €17,000 on 58 additional properties elsewhere in Italy. If the claim holds, banks and appraisal companies could obtain near-expert valuations in real time over the whole country, with the web crawler supplying the local market signal.","feed_headline":"Three features match professional appraisals to €21,000","feed_subtitle":"A model trained in Turin predicts home values across Italy from just three numbers: OMI band, surface, comparable listings.","key_machinery":"The load-bearing mechanism is the constructed feature \"average price per square meter of comparable properties,\" combined with the official OMI price band and the property's surface. Comparables are obtained by simulating appraiser practice: a web crawler searches nearby advertised properties and previous appraisals near the target, filters by distance and similarity, and averages their unit prices into a single number. This feature encodes the local market directly, which is why adding it drops test error from about €35,000 to about €21,000. Tree-ensemble regressors—extra trees, random forests, and bagging—are the learning machinery, and their feature-importance rankings were the tool that revealed the OMI-centered feature set.","core_discovery":"The key discovery is that a reduced feature set—OMI minimum and maximum price per square meter for the zone, property surface, and the average unit price of comparable properties—reproduces professional appraisals more accurately than a rich hedonic model with dozens of structural, point-of-interest, and geographic features. Feature-importance analysis using tree ensembles showed the OMI values dominating the ranking, which led the authors to drop the OMI area name, distance from center, and all PoI counts. With the comparables feature added, bagging, extra trees, and random forests all give a mean error around €21,000 and $R^2$ around 0.96 on the Turin test set. The same trained model applied to 58 out-of-sample appraisals in different Italian cities and different years gives a mean error of about €17,000, which the paper presents as evidence that the model is predictive beyond the training city.","pith_inferences":["Because the comparables feature is computed from advertised asking prices rather than transaction prices, it likely captures list-price levels; a natural extension is to test whether replacing it with transaction prices from deeds, where publicly available, changes accuracy.","The transferability of the approach to other countries depends on finding an administrative analog of OMI price bands; in markets without such public zoning valuations, the model would need a different spatial prior to achieve similar accuracy.","The 58-property out-of-sample set is too small to certify a national claim; a stronger test would hold out entire cities or time periods from training.","The paper's target is expert appraisal rather than market price, so the model may reproduce systematic deviations in expert judgment; comparing predictions to actual transaction prices would separate these two effects."],"forward_implications":["A national Italian automated valuation service can be trained on one city and reused elsewhere, because the OMI band and comparables carry the local market signal.","Appraisal companies can use the model as an automated validation tool: the paper reports that the error level of about €21,000 on a mean value of about €189,000 was considered acceptable by the domain experts involved.","Lenders could issue real-time draft mortgage valuations from a web crawler plus three features, deferring the full expert visit to later in the process.","Collecting hedonic details such as elevator, floor, and maintenance status may no longer be necessary for a first-cut valuation when OMI values are available, reducing the cost of appraisal data acquisition."],"supporting_citations":[{"why":"Supplies the OMI price-band values and zone polygons that form the two dominant features of the reduced model.","marker":"[28]"},{"why":"Supplies the Extremely Randomized Trees regressor used in the best-performing experiments and in feature-importance analysis.","marker":"[24]"},{"why":"Provides the urban-environment valuation baseline that the paper compares against and reports outperforming.","marker":"[16]"},{"why":"Shows advertised big housing data can be used as a price signal, the precedent for the crawled comparables feature.","marker":"[13]"}],"fun_headline_variants":["Three numbers predict home values across Italy with €17k error","Three features outperform dozens in home value estimation","Turin-trained model prices homes across Italy from just 3 features","Fewer inputs, better predictions: €21k error on home values"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The nationwide claim rests on the 58 out-of-sample appraisals being measured with the same price-band and comparable-property data that were available on or before each appraisal's date; if later information leaked into those features, the €17,000 error is too optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Three numbers predict home values across Italy with €17k error","Three features outperform dozens in home value estimation","Turin-trained model prices homes across Italy from just 3 features","Fewer inputs, better predictions: €21k error on home values"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2169,"prompt_tokens":873,"completion_tokens":1296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1226}},"tokens_in":489,"tokens_out":1296,"duration_ms":10866,"temperature":1.0,"reasoning_tokens":1226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:38:31.812662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the out-of-sample test with date-restricted features: for each of the 58 properties, crawl comparables and fetch OMI bands only from the period on or before the appraisal date, and check whether the error stays near €17,000. The paper's Section 5.3 reports the result but does not describe the comparables pipeline for these properties, so this is where the generalization claim can be tested.","supporting_citations":[{"cited_title":"Manuale della banca dati dell’osservatorio del mercato immobiliare, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the OMI price-band values and zone polygons that form the two dominant features of the reduced model."},{"cited_title":"The economic value of neighborhoods: Predicting real estate prices from the urban environment","cited_arxiv_id":null,"evidence_quote":"Provides the urban-environment valuation baseline that the paper compares against and reports outperforming."},{"cited_title":"The potential of big housing data: An application to the italian real-estate market","cited_arxiv_id":null,"evidence_quote":"Shows advertised big housing data can be used as a price signal, the precedent for the crawled comparables feature."}],"review_version":1}