{"id":"f02e1c87-881e-4c78-8552-c09a2707713d","arxiv_id":"2607.08554","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A mmaction2 + YOLOv10 + random-forest pipeline maps street video to a 1–5 Degree of Informal Behavior score and plots its fluctuation over time.","lead":"CommuniWave is a video pipeline that scores how much temporary informal activity (vending, dancing, gatherings) appears on city streets. Urban managers could use the scores to adjust rules without constant field surveys, if the numbers prove reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Negative R² and tiny external test leave the DIB-quantification claim unsupported by the reported numbers.","rationale":"The reader correctly isolates the ground-truth and evaluation weakness as the load-bearing soft spot. The negative R^{2} is the most concrete numerical evidence that the BEM does not yet quantify DIB better than chance; the 10-clip external test is too small to rescue it. No other flaw (narrow label set, cultural locality, missing code) is as immediately decisive for the strongest claim. The recommended test is a minimal, standard check that would either rehabilitate the numbers or confirm that the pipeline remains a methods sketch. Verdict therefore stays CONDITIONAL; no stronger rejection is warranted because the systems idea itself is coherent and the feature-importance discussion is still useful once accuracy is fixed.","tokens_in":9863,"tokens_out":529,"duration_ms":5648,"concrete_test":"Re-train BEM with an explicit baseline (mean predictor and a linear model on the same PCA features) and report R^{2}, Spearman rank correlation and decision-threshold accuracy (e.g., DIB ≥ 3.5) on both the original 20 % hold-out and an expanded external set of ≥ 50 clips spanning all three road classes and three times of day. If R^{2} remains negative or rank correlation falls below 0.5, the quantification claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CommuniWave produces a usable Degree of Informal Behavior (DIB) score from street video that is close enough to human judgment to support refined urban decisions. Section 3.1 reports BEM test metrics of MSE 0.9599, RMSE 0.9798 and R^{2} = -0.1681 on the held-out portion of the 180-clip set, plus MAD = 0.709 versus 10 new volunteer means on only 10 re-shot clips. A negative R^{2} means the random-forest regressor explains less variance than simply predicting the mean score; on a 1–5 scale an RMSE near 1 and MAD of 0.7 are large relative to the dynamic range. The paper never shows that these errors still preserve rank order or decision-relevant thresholds across street classes or times of day (Figure 4). Consequently the claim that the output charts can guide management rests on an unvalidated mapping from surface detections to a stable DIB.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents CommuniWave, an integrated ML pipeline for detecting and scoring temporary informal resident behaviors (e.g., street vending, square dancing, gatherings) from street-level video. BCN combines mmaction2 spatiotemporal action detection with a custom YOLOv10 model (YLX) trained on six Chinese informal labels; BEM is a PCA-reduced random-forest regressor that maps the resulting feature counts plus street class to a continuous Degree of Informal Behavior (DIB) score (1–5). Ground truth is the MAD-filtered mean of 10 volunteer ratings on 180 ten-second clips from three road classes in one southern Chinese city. The system outputs DIB fluctuation charts intended to support dynamic monitoring and refined urban management for territorial resilience.","tokens_in":10183,"tokens_out":1150,"duration_ms":19264,"significance":"If the DIB scores were demonstrably reliable and generalizable, the work would supply urban designers with a practical, video-based alternative to labor-intensive POE/PSPL surveys and fragmented mobile-data methods, enabling continuous, spatially resolved monitoring that balances top-down planning with bottom-up vitality. The modular design (open YLX/BEM retraining interfaces, privacy de-identification, SHAP explanations, multi-class street conditioning) is a genuine engineering contribution that could be adapted by local managers. The reported negative R^{2} and limited external test, however, leave the quantification claim unsupported, so the significance remains prospective rather than demonstrated.","major_comments":[{"comment":"Section 3.1 reports BEM test-set metrics MSE = 0.9599, RMSE = 0.9798 and R^{2} = −0.1681 on the held-out portion of the 180-clip set. A negative coefficient of determination means the random-forest regressor explains less variance than simply predicting the mean DIB; on a 1–5 scale an RMSE near 1 is large relative to the dynamic range. These numbers directly contradict the claim that BEM produces a usable quantitative DIB for management decisions.","section":"§3.1 Model Evaluation"},{"comment":"The only external check is MAD = 0.709 between model predictions and new volunteer means on 10 re-shot clips. No confidence intervals, rank-correlation, threshold-agreement, or baseline (mean/median predictor) comparisons are supplied, yet the text describes the result as “overall close alignment.” With only ten samples the statistic cannot establish that the mapping from surface detections to DIB is stable across street classes or times of day (Figure 4).","section":"§3.1 Model Evaluation"},{"comment":"Ground-truth DIB is defined solely as the mean of ten volunteers’ 1–5 ratings (after MAD outlier removal) on 180 short clips filmed in a single medium-sized southern Chinese city, using six hand-chosen informal labels. No inter-rater reliability (ICC, Krippendorff’s α) is reported, nor is any external behavioral or economic validation offered. Consequently the claim that the resulting scores support “refined decisions to enhance \\ldots territorial resilience” rests on an unvalidated, geographically narrow proxy.","section":"§2.1–2.2, §3.1"},{"comment":"Feature-importance (SHAP, Figure 3) and temporal-fluctuation (Figure 4) analyses are generated by the same under-performing BEM. Because the model fails to outperform a constant predictor, statements about which informal behaviors dominate each road class and about morning-versus-evening DIB patterns lack statistical support and cannot be used for the management recommendations advanced in §4.3.","section":"§3.2–3.3, §4.1"}],"minor_comments":[{"comment":"“Jan Geh” (p. 2) should be “Jan Gehl”; several other proper-name and citation formatting inconsistencies appear (e.g., “Anon., 2018”).","section":"§1 Introduction"},{"comment":"Figure 1 and Figure 2 captions are terse; axis labels and color legends for the DIB fluctuation charts (Figure 4) are missing or hard to read in the supplied text.","section":"Figures 1–4"},{"comment":"The precise feature vector passed from BCN to BEM (counts, durations, spatial extents, street-class encoding) is never enumerated; a short table or equation would aid reproducibility.","section":"§2.4–2.5"},{"comment":"Hyper-parameter search ranges for the random forest and the exact PCA variance threshold retained are not stated, only that RandomizedSearchCV and mlflow were used.","section":"§2.5.2"}],"recommendation":"major_revision","confidential_remarks":"The negative R^{2} is a load-bearing failure of the central quantification claim; if the authors cannot improve the regressor or reframe the contribution as a detection pipeline with an exploratory scoring module, the paper is better suited to a workshop or methods note than a full journal article. The urban-resilience framing is ambitious relative to the empirical support."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that CommuniWave is a domain-specific video-to-score pipeline for temporary informal street behaviors (vendors, square dancing, gatherings, etc.), conditioned on three street classes, and the authors ship concrete fluctuation charts. The components are off-the-shelf (mmaction2, YOLOv10, random forest + PCA + SHAP), so novelty sits in the integration and the volunteer-defined DIB construct for Chinese community streets, not in new algorithms.\n\nWhat they do well: the problem framing is honest about top-down planning gaps, the data-collection protocol (high-angle 10 s clips, de-identification, street-class stratification) is practical, and the SHAP discussion of how different behaviors matter on different road classes is the most useful part of the results. They also flag cultural locality and the need for re-training, which is responsible.\n\nThe soft spot is load-bearing. Section 3.1 reports BEM R^{2} = −0.1681, MSE ≈ 0.96, RMSE ≈ 0.98 on the held-out split of the 180-clip set. That is worse than predicting the mean on a 1–5 scale. The external check is MAD 0.709 on only 10 new clips with no intervals or rank-order analysis. Figure 4’s time-series charts therefore rest on an unvalidated mapping. Label ontology is narrow (six hand-chosen behaviors), ground truth is the mean of 10 volunteers after MAD filtering, and no code or data are released. These are ordinary supervised-learning limitations, not circularity, but they undercut the claim that the scores are ready for refined management decisions.\n\nThis is for computational urbanists and municipal tech groups who want a starting workflow, not for readers seeking a validated metric or a methods advance. The evaluation numbers are visible in the text, so the stress-test concern holds. I would still send it to peer review as a methods sketch that needs accuracy baselines, larger external tests, and artifacts; it is not desk-reject material. Engage if you work on street vitality sensing; otherwise skim the SHAP and limitations sections and move on.","headline":"Applied pipeline for informal street activity is clear and useful as a sketch, but the negative R^{2} and tiny external test leave the DIB quantification claim unsupported.","tokens_in":10774,"tokens_out":548,"would_cite":false,"duration_ms":5354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"CommuniWave turns street video into a Degree of Informal Behavior score so managers can track spontaneous community activity over time.","keywords":["machine learning","spatiotemporal action detection","urban communities","Degree of Informal Behavior","refined design","territorial resilience","street video analysis"],"falsifier":"Record a new set of street videos from a different city or cultural setting, obtain fresh volunteer DIB ratings under the same protocol, and check whether CommuniWave’s mean absolute deviation stays near 0.7 or rises sharply; a large rise would show the score is not transferable.","tokens_in":10744,"feed_emoji":"📹","tokens_out":637,"duration_ms":11786,"temperature":0.7,"pith_summary":"Top-down community plans often clash with the spontaneous, bottom-up activities residents create—street vending, gatherings, square dancing, and the like—because managers lack a practical way to measure those informal behaviors. CommuniWave is a machine-learning pipeline that watches ordinary street video, detects those activities, and converts them into a single Degree of Informal Behavior (DIB) score that can be plotted as a fluctuation chart. The system combines an action-detection network, a custom object detector for six common informal behaviors, and a random-forest evaluator that also accounts for street type. On held-out clips the scores sit close to human raters, and multi-hour street recordings produce clear morning-to-evening patterns that differ by road class. The authors argue that this continuous, video-based metric lets managers regulate informal activity more fairly and flexibly, supporting territorial resilience without extinguishing neighborhood vitality.","feed_headline":"Street video becomes a live score of informal community behavior","feed_subtitle":"CommuniWave’s DIB charts let managers track vending, gatherings and more without constant surveys","key_machinery":"The Degree of Informal Behavior (DIB) pipeline: Behavior Capture Net (mmaction2 spatiotemporal detection plus a YOLOv10-based YLX detector for six informal labels) feeds feature bundles plus street-class labels into a PCA-reduced random-forest Behavior Eval Model that outputs a continuous 1–5 DIB score and its time-series chart.","core_discovery":"A three-stage machine-learning model (Behavior Capture Net + YLX detector + random-forest Behavior Eval Model) can extract visual features from short street videos and produce a Degree of Informal Behavior score that tracks human volunteer ratings closely enough (mean absolute deviation 0.709 on unseen clips) to generate usable fluctuation charts for first-, second-, and third-class roads across different times of day.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["ML pipeline scores temporary informal acts from short street videos","CommuniWave turns urban street clips into Degree of Informal Behavior charts","Three-stage model matches volunteer ratings of community informal behavior","Street video yields DIB fluctuation scores across road classes and times","BCN plus YOLOv10 and random forest quantify informal community activity"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That the average of ten local volunteers’ 1–5 ratings on 180 short clips, after outlier removal, is a stable ground-truth DIB that surface visual features alone can predict for other streets and times.","fun_headline_variants_meta":{"raw":{"variants":["ML pipeline scores temporary informal acts from short street videos","CommuniWave turns urban street clips into Degree of Informal Behavior charts","Three-stage model matches volunteer ratings of community informal behavior","Street video yields DIB fluctuation scores across road classes and times","BCN plus YOLOv10 and random forest quantify informal community activity"]},"model":"grok-4.5","effort":"low","cost_usd":0.003372,"raw_usage":{"total_tokens":1090,"prompt_tokens":696,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":33720000,"prompt_tokens_details":{"text_tokens":696,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":306,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":696,"tokens_out":88,"duration_ms":3462,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T05:26:57.151158+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Record a new set of street videos from a different city or cultural setting, obtain fresh volunteer DIB ratings under the same protocol, and check whether CommuniWave’s mean absolute deviation stays near 0.7 or rises sharply; a large rise would show the score is not transferable.","supporting_citations":[],"review_version":1}