{"id":"6e116b88-cfe6-45b1-a562-183a498705ad","arxiv_id":"2412.12183","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using XGBoost and SHAP on simulated Tehran blocks, the study ranks 30 urban morphology parameters and finds building shape, window-to-wall ratio, and commercial floor share dominate energy demand, while neighbor heights and distances drive cooling and solar access.","lead":"The study combined building-energy simulations for 2,400 synthetic urban blocks in Tehran with machine learning and SHAP explanations to rank which urban form features, such as building shape, window area, and neighbor building height, most affect cooling, heating, lighting, solar access, and photovoltaic output.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WWR's top ranking may be an artifact of confounding: Table 3 fixes WWR per building archetype, so SHAP cannot separate window-to-wall ratio from building-use effects.","rationale":"The most load-bearing concern is internal rather than external: the parametric design in Section 3.1 deliberately holds WWR constant per archetype, so WWR has no independent variation and its SHAP importance is inseparable from building-use effects. This directly undermines a headline finding (WWR as a critical design lever) and is testable by inspecting the dataset or rerunning a small deconfounded simulation. The reader's weakest assumption about representativeness of the synthetic corpus is also legitimate, but it is a different concern about generalization, not identifiability. The rest of the paper's contribution (UBEM + ML + SHAP) and other dominant features such as building shape may still hold, so the verdict remains conditional rather than reject or accept. No judgment about author intent is implied; this is a standard collinearity issue in sensitivity analysis.","tokens_in":24052,"tokens_out":4911,"duration_ms":55097,"concrete_test":"Obtain the generated dataset or scripts and verify whether WWR is constant within each building-use archetype and whether 'commercial ratio' is a deterministic function of 'residential ratio'. Then create a small augmented simulation set in which WWR is varied independently within each building use (e.g., 20-60% in 10% steps) while keeping other parameters fixed, retrain XGBoost, and recompute SHAP rankings. If WWR ceases to be dominant or its rank changes substantially, the original ranking is a confound artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that window-to-wall ratio (WWR) is among the most critical urban-morphology parameters for energy performance is not identifiable from the reported experimental design. In Section 3.1, Table 2 lists the parametric design variables; WWR is not among them. Table 3 fixes WWR by building use: 35% residential, 40% office, 50% commercial. The same section states that 'some parameters, such as WWR, construction materials, and internal loads, were held constant across scenarios to focus on urban morphology variables.' Thus within the generated corpus, WWR is perfectly collinear with building function/archetype. Nevertheless, Sections 4.2.2-4.2.4 and the abstract rank WWR as a dominant driver (80.7% cooling, 74.6% heating, 77.7% lighting). SHAP attributes to WWR any effect of the archetype/use dimension, including occupancy, equipment loads, and lighting density, all of which differ across the three uses in Table 3. The separate 'commercial ratio' feature, derived from the 'Residential ratio' input in Table 2, is also tied to the same use dimension, making the two top-ranked features redundant proxies. Consequently, the headline finding on WWR is an artifact of confounding rather than evidence that varying window size independently changes performance.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a workflow that combines parametric urban building energy modeling with seven machine-learning algorithms and SHAP-based feature attribution to rank 30 urban morphology parameters for six performance metrics (cooling, heating, lighting demand, sunlight hours, PV generation, and sky view factor) at the urban-block scale, using Tehran's dry-arid climate as a case study. The authors generate 2,400 synthetic regular-grid blocks (yielding 36,060 parcels), select XGBoost as the best trade-off between accuracy (R² reported up to 0.97) and training time (3.64 s), and then use SHAP to identify dominant, influential, and negligible variables, concluding that building shape, window-to-wall ratio, and commercial ratio most strongly affect energy demand, while neighboring-building heights and distances drive cooling demand and solar access.","tokens_in":24237,"tokens_out":5657,"duration_ms":57740,"significance":"If the results are valid, the paper offers a useful and relatively scalable pipeline for urban-scale sensitivity analysis, and its comparison of seven machine-learning models on a large synthetic simulation corpus is a practical contribution. The inclusion of environmental metrics (SVF, solar hours, PV) alongside energy metrics is a strength, as is the use of UWG-modified weather to approximate urban microclimate. However, the headline finding on window-to-wall ratio is confounded by the experimental design, and the model evaluation lacks an out-of-sample validation protocol. These issues currently limit the reliability of the quantitative rankings and of the claimed generalizability to other dry-arid regions.","major_comments":[{"comment":"Window-to-wall ratio is not an independent design variable in the corpus: Table 2 lists parametric variables without WWR, while Table 3 fixes WWR at 35%, 40%, and 50% for residential, office, and commercial archetypes respectively. Since the same table also fixes plug/equipment loads, lighting density, and occupancy-related assumptions per archetype, WWR is perfectly collinear with building function. The SHAP attributions in §§4.2.2–4.2.4 therefore cannot separate the effect of window size from the effect of building use; a model fitted on these data will assign to WWR any variation that is actually due to archetype. The abstract's claim that WWR is among the most critical parameters is not identifiable from the reported experimental design. The authors should either introduce independent variation of WWR across archetypes or re-frame the finding as a property of the archetype dimension.","section":"§3.1, Tables 2 and 3"},{"comment":"Equation (1) defines R² as SSR/SST − 1, which is algebraically inverted. If SSR denotes residual sum of squares, the standard definition is R² = 1 − SSR/SST; if SSR denotes regression sum of squares, R² = SSR/SST. As written, the equation would give negative R² for any plausible fit and contradicts the reported values (e.g., R² = 0.97 for XGBoost on PV). In addition, no train/test split or cross-validation is described in §3.2 or §4.1; the reported accuracy metrics appear to be computed on the training data. An out-of-sample evaluation is needed to support the choice of XGBoost and to ensure that the SHAP rankings reflect generalizable structure rather than overfitting.","section":"§3.2, 'Models' evaluation', Eq. (1)"},{"comment":"The numerical results are internally inconsistent. For example, commercial ratio is reported as 81.27% for heating in §4.2.3, as 47.4% for lighting in §4.2.4, and as 27.4% for cooling and 37.1% for lighting in §5; building shape is 94.1% for cooling in §4.2.2 but the lighting section reports an 85.71% effect for 'building shape and the number of commercial floors.' Figure 7's dominance categories do not match the percentages in the text (e.g., the sunlight-hours panel lists three dominant variables while the text names four with values above 50%). Because the central contribution is a variable ranking, these discrepancies must be resolved with a single reproducible SHAP importance table and a consistent set of magnitudes.","section":"§4.2 and §5, Figure 7"},{"comment":"The synthetic corpus is built on a regular grid, two building typologies, fixed construction archetypes, and UWG-modified weather, but no part of the energy or solar results is compared with measured data from Tehran or any other dry-arid city. The conclusion that the findings offer 'generalizable insights applicable to other dry-arid regions' therefore goes beyond the evidence. I recommend either adding an external validation case (e.g., a measured block or a published benchmark) or explicitly limiting the claim to the simulated domain.","section":"§3.1 and §6"}],"minor_comments":[{"comment":"Equation (3) appears garbled: the factorial numerator should be |S|! (n − |S| − 1)! / n!, but the printed expression includes a stray 'i' and is missing the second factorial.","section":"§3.3, Eq. (3)"},{"comment":"The SHAP normalization is performed separately for each category, which makes the percentages comparable only within a category; the thresholds of 50% and 20% are arbitrary and should be justified or tested for robustness.","section":"§3.3"},{"comment":"The arrow and color coding in Figure 5 is described in the caption but is not legible in the preprint; please enlarge the figure or split it into per-metric panels.","section":"Figure 5"},{"comment":"There are typos and terminology inconsistencies, e.g., 'Enegy efficiancy' in reference [62], and the text interchangeably uses 'commercial ratio' and 'number of commercial floors' when describing the same feature.","section":"Throughout"},{"comment":"The abstract reports R²: 0.92 for XGBoost, but §4.1 reports per-metric XGBoost R² values of 0.97 (PV), 0.82 (cooling), 0.85 (heating), 0.88 (lighting), and ≥0.9 for sunlight/SVF; the abstract should state which metric or average the value refers to.","section":"Abstract and §4.1"}],"recommendation":"major_revision","confidential_remarks":"The WWR confounding is the main barrier to acceptance; even with careful rewriting, the paper should either re-simulate with independently varied WWR or substantially soften the claim. The internal numerical inconsistencies in the SHAP results also require a clean, reproducible importance table, ideally with code or data release for verification. The paper's scope fit with Advances in Building Energy Research is appropriate, but it needs these technical issues resolved before it can be relied upon for design guidance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I'd tell you about arXiv:2412.12183. It's a decent UBEM workflow: 2,400 parametric blocks, 36,060 parcels, 30 features, six metrics, Tehran dry-arid context. They compare seven ML models, pick XGBoost, and use SHAP to rank features. The strength is the scale and the separate treatment of neighbor heights and distances, which gives practical guidance about shading and solar access. The SHAP rankings for building shape, density, and neighbor geometry are plausible and consistent with prior work.\n\nThe soft spot is load-bearing. The abstract's claim that window-to-wall ratio (WWR) is a critical design parameter is not identifiable from the experimental design. In Table 2, WWR is not a variable. Table 3 fixes it by archetype: 35% residential, 40% office, 50% commercial. The text even says WWR was 'held constant across scenarios'—which is only true within each archetype. So in the generated corpus, WWR is perfectly collinear with building use. The SHAP analysis then attributes to WWR whatever effect comes from the archetype dimension, including occupancy, equipment, and lighting loads. The 'commercial ratio' feature is tied to the same use axis, so the two top-ranked features are redundant proxies for function. The paper cannot separate window size from building use. That is not a minor flaw: the headline finding about WWR is an artifact.\n\nOther issues are smaller: Equation 1 defines R² as SSR/SST - 1, which is wrong (should be 1 - SSR/SST). No train/test split or cross-validation is described for the reported R² and RMSE values. No code or data are released. And the generalization claim to other dry-arid regions goes beyond what a synthetic parametric corpus can support. The authors do acknowledge some of these limitations in the conclusion.\n\nNone of this kills the paper. If the WWR language is removed or reframed as 'building use and fenestration combined,' and the authors re-run with WWR actually varied (or clearly state the limitation), the rest of the ranking—building shape, density, neighbor heights/distances—stands. The workflow is useful for urban planners in dry-arid climates.\n\nI'd send it to review: it's substantial, relevant, and the methodological fix is within reach. But I'd want the confound addressed before publication. For my own work, I wouldn't cite the WWR result; I'd cite the workflow after the fix.","headline":"Solid workflow and scale, but the headline WWR finding is an artifact of confounding: WWR is not varied independently in the simulation design, so the paper's top-ranked 'design parameter' is really building use.","tokens_in":24876,"tokens_out":3126,"would_cite":false,"duration_ms":31819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A surrogate model of 2,400 simulated blocks ranks which urban morphology parameters drive energy and solar performance.","keywords":["urban morphology","urban building energy modeling","XGBoost","SHAP","sensitivity analysis","dry-arid climate","solar access","sky view factor"],"falsifier":"Simulate or monitor a set of real Tehran blocks with geometries, materials, and orientations outside the paper's grid (for example, courtyard typologies, towers over ten floors, or non-orthogonal plots), run the same XGBoost-and-SHAP pipeline on those outputs, and check whether building shape, window-to-wall ratio, and commercial ratio remain the top energy drivers and whether southwest neighbor height and distance still lead cooling and solar access. If the rankings reorder, the paper's generalizable claim fails; if they hold, the method transfers.","tokens_in":23791,"feed_emoji":"🏙️","tokens_out":7428,"duration_ms":83805,"temperature":0.7,"pith_summary":"The paper tries to establish that a machine-learning surrogate trained on parametric urban-block simulations can replace slow city-scale energy simulation long enough to rank design levers. It uses 2,400 simulated blocks, 30 morphology variables, and six performance metrics, and claims that XGBoost plus SHAP give a trustworthy ranking of which variables matter in dry-arid Tehran. The specific ranking is that building shape, window-to-wall ratio, and commercial ratio dominate energy demand, while neighboring building heights and distances control cooling demand and solar access. A sympathetic reader would care because the ranking is actionable: designers and planners can focus on a few high-impact form parameters instead of treating all thirty as equal.","feed_headline":"Shapley values rank what drives Tehran block energy use","feed_subtitle":"A surrogate model over 36,060 simulated parcels says building form and neighbor heights matter most for cooling and solar access.","key_machinery":"The machinery is a two-stage surrogate pipeline: first, parametric urban-block simulation generates 36,060 parcel-level outcomes (29,560 buildings and 6,500 parks) over a regular grid of blocks; then an XGBoost regression model is fitted to those outcomes and explained with SHAP values. The Shapley-value formula from cooperative game theory is what turns the fitted surrogate into a ranked, signed list of which of the 30 morphology parameters matter for each metric. SHAP values are normalized to a 0–100 scale and grouped into dominant, influential, and negligible categories so that the ranking can be read directly as design priorities.","core_discovery":"On a synthetic corpus of 2,400 urban blocks modeled with typical Tehran configurations, geometries, and microclimate-adjusted weather, the paper shows that an XGBoost surrogate predicts six block-level outputs—cooling, heating, and lighting demand, sunlight hours on facades, PV generation, and sky view factor—with an aggregate R² around 0.92 and a training time of 3.64 seconds. SHAP analysis then attributes each output to 30 morphology parameters and ranks them. The central claim is the ranking: building shape, window-to-wall ratio, and commercial ratio are the most critical parameters for energy demand, while the heights and distances of neighboring buildings, especially in the southwest, strongly influence cooling demand and solar access, and street width dominates sky view factor.","pith_inferences":["Editorial inference: the paper reports each metric separately, so a planner cannot yet see whether the layout that minimizes cooling demand also sacrifices PV output; a Pareto-style multi-objective study is the natural next step and is not claimed here.","Editorial inference: the eight-directional coding of neighbor heights and distances implies orientation-specific zoning rules (taller southwest neighbors shade afternoon sun, for instance), but the paper does not translate its ranking into regulatory target values.","Editorial inference: because all blocks are regular-grid synthetic layouts, transfer to real irregular dry-arid cities is untested; an out-of-sample test on measured city data would be the direct validation."],"forward_implications":["Early-stage urban design in dry-arid climates should treat building shape and window-to-wall ratio as first-order levers for cooling, heating, and lighting demand.","Block-level cooling and solar access can be steered by regulating the heights of and distances to southwest and southeast neighbors, not only by each building's own envelope design.","Because the XGBoost surrogate predicts all six metrics in seconds, a city-scale screening tool could explore thousands of block layouts that would take months with physics-based simulation alone.","Street width is the dominant lever for sky view factor, so urban canyon geometry should be set separately from density targets in block-scale planning."],"supporting_citations":[{"why":"Provides the XGBoost algorithm, the surrogate model that carries the predictions after model selection.","marker":"[66]"},{"why":"Supplies the Shapley-value definition used in Equation 3 for feature attribution.","marker":"[74]"},{"why":"Motivates modeling neighboring buildings as separate variables by showing that ignoring urban context can produce 8–31% simulation error.","marker":"[22]"},{"why":"Supplies prior evidence that building height, floor-area ratio, and coverage are influential factors whose ranking this study tests at block scale.","marker":"[26]"},{"why":"Grounds the choice of compact versus detached typologies and the interpretation of density effects on heat-energy demand.","marker":"[28]"},{"why":"Argues that non-optimal building arrangements can raise energy use by up to 30%, framing morphology as a design variable worth ranking.","marker":"[3]"},{"why":"Provides the early-design parameter-sensitivity method that this study extends from single buildings to urban blocks.","marker":"[20]"},{"why":"Supplies the comparison point for building footprint as the dominant input for solar performance in a different climate.","marker":"[76]"}],"fun_headline_variants":["SHAP ranks building shape and windows as top Tehran energy drivers","Neighbor building heights decide Tehran cooling and solar access","XGBoost surrogate explains block-level energy patterns in Tehran","Urban form's energy impact ranked by explainable AI in dry climates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings are only as representative as the 2,400 synthetic regular-grid blocks built from the paper's discrete design values, so the central premise is that those blocks stand in for real Tehran and other dry-arid cities well enough that the SHAP rankings carry over to actual urban fabric.","fun_headline_variants_meta":{"raw":{"variants":["SHAP ranks building shape and windows as top Tehran energy drivers","Neighbor building heights decide Tehran cooling and solar access","XGBoost surrogate explains block-level energy patterns in Tehran","Urban form's energy impact ranked by explainable AI in dry climates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000976,"raw_usage":{"total_tokens":4152,"prompt_tokens":954,"completion_tokens":3198,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3128}},"tokens_in":570,"tokens_out":3198,"duration_ms":30174,"temperature":1.0,"reasoning_tokens":3128,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:28:19.954275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate or monitor a set of real Tehran blocks with geometries, materials, and orientations outside the paper's grid (for example, courtyard typologies, towers over ten floors, or non-orthogonal plots), run the same XGBoost-and-SHAP pipeline on those outputs, and check whether building shape, window-to-wall ratio, and commercial ratio remain the top energy drivers and whether southwest neighbor height and distance still lead cooling and solar access. If the rankings reorder, the paper's generalizable claim fails; if they hold, the method transfers.","supporting_citations":[{"cited_title":"Development of a Machine-Learning Framework for Overall Daylight and Visual Comfort Assessment in Early Design Stages,","cited_arxiv_id":null,"evidence_quote":"Supplies the Shapley-value definition used in Equation 3 for feature attribution."},{"cited_title":"Parametric energy simulation in early design: High-rise residential buildings in urban contexts,","cited_arxiv_id":null,"evidence_quote":"Motivates modeling neighboring buildings as separate variables by showing that ignoring urban context can produce 8–31% simulation error."},{"cited_title":"Cities and energy: Urban morphology and residential heat-energy demand,","cited_arxiv_id":null,"evidence_quote":"Grounds the choice of compact versus detached typologies and the interpretation of density effects on heat-energy demand."},{"cited_title":"Simulating the Impact of Urban Morphology on Energy Demand - A Case Study of Yuehai, China,","cited_arxiv_id":null,"evidence_quote":"Argues that non-optimal building arrangements can raise energy use by up to 30%, framing morphology as a design variable worth ranking."},{"cited_title":"Sensitivity of design parameters on energy, system and comfort performances for radiant cooled office buildings in the tropics,","cited_arxiv_id":null,"evidence_quote":"Provides the early-design parameter-sensitivity method that this study extends from single buildings to urban blocks."}],"review_version":1}