{"id":"ad5b8323-3428-4af9-a4b0-e9eb73f2f1c6","arxiv_id":"2501.10514","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-layer fully connected neural network predicts next-stop departure time deviations on 151 Boston MBTA routes with a root-mean-square error of 77.8 seconds.","lead":"This paper trains a fully connected neural network that predicts, for a Boston MBTA bus currently at one stop, how early or late it will depart the next stop. It reports reducing the average schedule deviation from about four minutes to a test error of about 78 seconds across 151 routes, which is the kind of signal passenger apps and smart bus-stop displays would use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random row-level split and missing persistence baseline leave the reported sub-80-second RMSE unsupported as a real-time generalization claim.","rationale":"The reader's weakest assumption is the random split in Section 5.3, and I agree that this is the most load-bearing issue. The paper's central contribution is an empirical accuracy claim, and that claim is only meaningful if the test set represents future, unobserved departures. Because the segmentation step creates multiple rows per trip, a random row-level split permits same-trip leakage, which can substantially inflate accuracy when the input includes the current-stop deviation. I also flagged the missing persistence baseline: without comparing to 'predict next deviation equals current deviation,' the reported RMSE does not demonstrate improvement over a trivial rule. These concerns are closely related and both bear on the validity of the headline number. A temporal grouped holdout with a persistence baseline would settle the matter. I am not recommending rejection because the model could still perform well under a proper evaluation; the paper simply has not provided that evidence. The verdict remains CONDITIONAL, consistent with the reader's assessment.","tokens_in":12706,"tokens_out":4591,"duration_ms":48347,"concrete_test":"Retrain the selected 512-128-64 FCNN under a temporal grouped holdout: train on January-February data, validate on the last week of February, and test on March data, while also enforcing that no Half-Trip ID appears in more than one split. On the same test set, compute a persistence baseline d_pred(i+1) = d_actual(i) and the linear-regression baseline. If the temporal-split RMSE is materially above 77.8 s, or if the persistence RMSE is within a few seconds of the FCNN's, the abstract's under-80-second real-time claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result is a test RMSE of 77.83 s (MAE 55.63 s) on the preprocessed MBTA data, but the evaluation in Section 5.3 uses a 70/20/10 random row-level split. After the Section 4.2 segmentation, each Half-Trip ID yields multiple segment-level examples from consecutive stops in the same trip, sharing route, vehicle, day, weather, and nearby stop context. A random split can therefore place examples from the same trip in both training and testing, and because the model's input includes the current-stop departure-time deviation, the test examples are nearly repeated instances of training examples rather than unseen future departures. The resulting 77.8 s RMSE does not measure the real-time generalization the abstract claims. The concern is compounded by the absence of a persistence baseline: the comparison in Section 5.3 is only against a linear regression (RMSE 161.44 s). Since predicting the next-stop deviation as the current-stop deviation is a natural and potentially very strong baseline for this task, the reported error alone does not establish 'significant improvement' even under a leak-free split. The paper is internally consistent and the FCNN training setup is plausible, but the central claim is an evaluation claim, and the current evaluation protocol does not support it as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a fully connected neural network (FCNN) that predicts the departure time deviation at the next bus stop for MBTA buses, using a one-stop lookback of the current-stop deviation together with time, distance, weather, and route features. The dataset integrates three public sources: MBTA 2023 departure records (restricted to the first quarter), Visual Crossing weather data, and MBTA stop geodata. After cleaning and outlier removal, trips are segmented into consecutive stop pairs, and the model is trained with MSE loss. An architecture search over one to five hidden layers selects a three-layer model (512, 128, 64 neurons) with test RMSE 77.83 s and MAE 55.63 s, compared against a linear regression with RMSE 161.44 s. The paper also reports per-route RMSE, MAPE with a discussion of its limitations, and a model-level inference time of 28.7 microseconds.","tokens_in":12960,"tokens_out":2186,"duration_ms":24654,"significance":"If the reported evaluation were sound, the paper would be a useful applied contribution: it addresses a practical real-time prediction task on a large public dataset, provides a compact model suitable for IoT-style deployment, and includes a sensible feature engineering pipeline and an architecture ablation. The strengths are the concrete, reproducible data source, the clear single-output regression setup, the explicit treatment of computational complexity (FLOPs/parameters) for IoT constraints, and the honest discussion of MAPE's interpretability problems. However, the central claim—\"significant improvement,\" predicting deviations to under 80 seconds (abstract and Section 6)—is an evaluation claim, and the current evaluation protocol does not establish it. The reported accuracy is best interpreted as interpolation accuracy on a random sample of segments rather than as a measure of real-time predictive generalization to future, unseen departures.","major_comments":[{"comment":"The evaluation uses a random 70/20/10 split of the row-level preprocessed data. After the segmentation in Section 4.2, each Half-Trip ID produces multiple consecutive stop-pair examples that share route, vehicle, date, weather, and nearby stop context. A random split can therefore place examples from the same trip in both training and testing. Since the input includes the current-stop departure deviation, test examples become near-duplicates of training examples rather than unseen future departures. This does not measure the real-time generalization claimed in the abstract and conclusion. Please re-evaluate with a temporal holdout (e.g., train on January-February, validate and test on March) or at least a grouped split by Half-Trip ID and by Service Date, and report the resulting RMSE/MAE.","section":"Section 5.3"},{"comment":"No persistence or trivial baseline is reported. Because the current-stop departure deviation is a key input and the target is the next-stop deviation, the natural baseline \"predict the next deviation equals the current deviation\" is likely very strong for this task. The paper compares only against linear regression (RMSE 161.44 s). Without the persistence baseline, the claim that the FCNN achieves a \"significant improvement\" is unsupported even under a leak-free split. Please add this baseline, ideally alongside route-level and time-of-day stratified results.","section":"Section 5.3"},{"comment":"The architecture is selected using the test RMSE: Table 2 reports test RMSE for every candidate architecture and the three-layer model is chosen because it balances accuracy and complexity. This makes the reported 77.83 s test RMSE an optimistically biased estimate of the selected model's error. Please separate model selection from evaluation—for example, use the validation split for architecture choice and report test error only for the final model—or use nested cross-validation.","section":"Section 5.1 and Table 2"},{"comment":"The outlier removal with k=2 in Equations (1)-(2) discards 406,969 points (about 6.4% of the data), including many long delays that are precisely the events a real-time prediction system must handle. The thresholds are computed on the full dataset before any train/test separation, so information from the test period influences the preprocessing. Please justify the choice of k, examine sensitivity to k, and ensure that outlier detection is fit only on the training portion in any temporal evaluation.","section":"Section 4.1"},{"comment":"No error bars, repeated runs, or statistical significance tests are reported. The differences among some architectures in Table 2 are very small (e.g., 76.8411 vs. 76.8323 seconds for four- and five-layer models), and without multiple seeds or confidence intervals it is unclear whether these differences are meaningful. Please report mean and standard deviation over several random seeds, or at least state whether the reported numbers are from a single run.","section":"Sections 5.2 and 5.3"}],"minor_comments":[{"comment":"The abstract and conclusion say the average deviation is \"nearly 4 minutes,\" but the preprocessed dataset mean is 211.866 s (3 min 31 s); the original dataset mean is 261.84 s (4 min 22 s). The text should disambiguate which dataset the average refers to.","section":"Abstract and Section 4.1"},{"comment":"The description of \"current stop lateness status\" is ambiguous: the bullet says it is derived from the \"status of the previous stop\" but then says it is determined by whether the deviation at the current stop is positive or negative. Please clarify which stop is used.","section":"Section 4.2"},{"comment":"The RMSE equation uses i as both the sample index and the stop index; the indexing of m and the summation over samples should be distinguished from the stop index i in Section 4.2.","section":"Equation (4)"},{"comment":"The reported inference time of 28.7 microseconds omits the hardware and software stack; without specifying the device (the experiments otherwise use Tesla V100 GPUs), this number does not support the IoT-deployment claim. Please state the benchmarking environment.","section":"Section 5.3"},{"comment":"Several citations are incomplete or malformed (e.g., \"axi,\" \"apt,\" \"str,\" \"tsg,\" and the Visual Crossing URL), and the reference \"Sto\" is used for both a dataset and a definition of rush hour. Please clean up the reference list and unify citation keys.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is internally consistent and the modeling choices are plausible, but the headline evaluation claim is not yet supported. The fixes needed—a temporal or grouped split, a persistence baseline, and a clean model-selection protocol—are all within the scope of a revision and would make the contribution publishable if the results hold. I do not see a fundamental circularity problem: the target is next-stop deviation and current-stop deviation is a legitimate input feature. The main risk is that the measured advantage over the persistence baseline may shrink substantially under a proper temporal split, so I recommend asking for those experiments before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a legitimate extension of the authors' own FCNN arrival-prediction work to departure-time deviation, and the numbers are internally consistent, but the headline \"under 80 seconds\" is not supported as a real-time generalization claim because the evaluation leaks trip context and there is no persistence baseline.\n\nThe genuinely new piece is the one-stop lookback FCNN applied across 151 MBTA routes with weather and stop-spacing features, plus the FLOPs/parameter comparison. I believe the RMSE/MAE values are reproducible in-sample; the paper is transparent about its preprocessing and reports the MAPE caveat honestly. Credit where due: the inference-time measurement and the discussion of why MAPE is misleading are useful.\n\nThe soft spots are load-bearing. Section 5.3 uses a random 70/20/10 row-level split even though Section 4.2 segments every Half-Trip into consecutive stop pairs. A random split puts adjacent examples from the same trip on both sides of the train/test boundary, and since the input includes the current-stop deviation, the test examples are close to repeated instances of training examples. That alone invalidates the abstract's claim of real-time prediction. Worse, there is no persistence baseline: predicting the next-stop deviation as the current-stop deviation is the natural naive model here, and the paper only compares against linear regression. So we do not know whether the FCNN beats the trivial predictor even under a clean split. The architecture search also selects on test RMSE (Table 2), and there are no error bars or repeated runs. These are fixable, but they matter.\n\nThe citation pattern is acceptable; self-citation is to the authors' prior arrival-prediction work, which is the direct predecessor. The paper would benefit from a temporal holdout, a persistence baseline, and code or data artifacts.\n\nWho this is for: researchers in applied transit ML who want a Boston case study and a baseline FCNN pipeline. It deserves a serious referee because the evaluation design is fixable and the engineering claims are concrete. My recommendation: send it out, but require a temporal split and the persistence baseline before acceptance.","headline":"A clean FCNN case study for bus departure deviation whose headline RMSE is undercut by random train/test splitting and a missing persistence baseline.","tokens_in":13507,"tokens_out":2371,"would_cite":false,"duration_ms":23022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bus departure times predicted to within 80 seconds across 151 routes","keywords":["bus departure prediction","fully connected neural network","feature engineering","real-time transit information","MBTA Boston","IoT smart transit","deep learning"],"falsifier":"Train the same model on the first two months of MBTA 2023 data and test on the third month, holding out a contiguous future period, and compare RMSE with the reported 77.8 seconds; if the temporal-split RMSE exceeds roughly 120 seconds or degrades by more than 50 percent, the central real-time prediction claim is not supported.","tokens_in":12522,"feed_emoji":"🚌","tokens_out":4486,"duration_ms":39885,"temperature":0.7,"pith_summary":"This paper claims that a compact fully connected neural network can predict a bus's departure-time deviation at the next stop well enough for real-time passenger information, using only data available at the current stop plus route, weather, and stop-location features. Trained on the first three months of 2023 MBTA Boston bus records and tested across all 151 routes, the model achieves a test RMSE of 77.8 seconds and a mean absolute error of 55.6 seconds, cutting the network-wide average deviation of roughly 211.9 seconds (about 3.5 minutes) by more than half. The authors frame this as an enabler for smart IoT transit systems, where buses, cloud servers, and passenger apps exchange departure predictions. The claim matters because schedule unreliability is a leading cause of bus ridership decline, and a model deployable on resource-limited devices could improve both passenger information and operator decision-making.","feed_headline":"Bus departure times predicted to within 80 seconds across 151 routes","feed_subtitle":"Boston MBTA test across 151 routes cuts schedule deviation error from about 4 minutes to 78 seconds.","key_machinery":"The central mechanism is the FCNN with a one-bus-stop lookback: each training example corresponds to the segment between two consecutive stops of a trip, with 173 input features describing the current stop (deviation from schedule, lateness status, day type, rush hour, stop-to-stop distance, weather one-hots, route and direction one-hots, and coordinates) and a single output neuron giving the predicted departure-time deviation at the next stop. The predicted deviation is added to the scheduled next-stop time to produce an actual departure estimate. This design lets one network serve all 151 routes, captures nonlinear feature interactions, and keeps the model small enough to run on IoT-class hardware.","core_discovery":"On the authors' own terms, the discovery is that a single FCNN with three hidden layers (512, 128, 64 neurons), 173 input features, and a one-bus-stop lookback window predicts the next stop's departure time deviation across the entire MBTA network with 77.83 seconds RMSE on a held-out test set, compared with a 161.44 seconds RMSE for linear regression. The model uses Min-Max scaled features grouped into time, distance, weather, and traffic categories, and is trained with MSE loss and the Adam optimizer. The paper also shows diminishing returns beyond three hidden layers: four- and five-layer variants improve RMSE by less than 1 percent while multiplying parameter count and FLOPs, supporting the choice of a compact architecture suitable for IoT deployment. Reported inference time is 28.7 microseconds per prediction, excluding network latency.","pith_inferences":["The random 70/20/10 split likely overstates real-world accuracy, because trips from the same route, day, or vehicle can appear in both training and test sets; a temporal split would give a fairer estimate for truly future departures.","Since the model predicts only one stop ahead, errors accumulate over a multi-stop trip; a route with 14 stops could see several times the per-stop RMSE by journey's end.","High-error route 32 (RMSE 240.6s) suggests stop-level features such as passenger dwell time are missing; adding automated passenger counters could be a direct test of that hypothesis.","The same pipeline should transfer to other GTFS-compliant transit agencies, but the weather, rush-hour, and stop-spacing thresholds would need recalibration from local data."],"forward_implications":["A single citywide model, rather than per-route models, can provide next-stop departure predictions with under-80-second RMSE for every route in the network.","Because the model needs only the current stop's deviation plus static features, it can be updated in real time as each bus reaches a stop, supporting countdown displays and mobile apps.","The compact 163K-parameter architecture makes edge deployment feasible: per-request inference takes about 28.7 microseconds, excluding communication latency.","Replacing a linear regression baseline (RMSE 161.4s) with the FCNN more than halves prediction error, indicating that nonlinear feature interactions carry much of the signal."],"supporting_citations":[{"why":"Supplies the MBTA Bus Departure Times 2023 dataset, which defines the prediction task with scheduled and actual departure times across 151 routes.","marker":"[MBT]"},{"why":"Provides historical meteorological data from the Visual Crossing API, used to construct the weather-related input features.","marker":"[wea]"},{"why":"Supplies bus stop geographical coordinates from the MBTA Bus Routes and Stops dataset, enabling distance and location features.","marker":"[sto]"},{"why":"Presents the earlier fully connected neural network approach for New York bus arrival prediction that this work extends to departure times.","marker":"Rashvand et al. [2024a]"},{"why":"Motivates the reliability-focused, data-driven optimization of transit schedules and the use of feature selection for prediction accuracy.","marker":"Basak et al. [2019]"},{"why":"Demonstrates the effectiveness of ANN and SVM models for multi-route bus arrival prediction, informing the modeling choices here.","marker":"Yin et al. [2017]"},{"why":"Provides an earlier hybrid scheme for real-time bus departure-time prediction, serving as a baseline against which the neural approach is positioned.","marker":"Fadaei Oshyani and Cats [2014]"}],"fun_headline_variants":["AI predicts bus departures to within 80 seconds","Neural net cuts bus delay error to 78 seconds","Compact model predicts bus times, error under 80s","Boston bus AI: departure deviations below 80 seconds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes each bus-stop record is independent, so randomly splitting the data into training and test sets does not let information from the same trip, route, or day leak between the two; if that assumption fails, the reported 77.8-second RMSE overstates how well the model predicts future, unseen departures.","fun_headline_variants_meta":{"raw":{"variants":["AI predicts bus departures to within 80 seconds","Neural net cuts bus delay error to 78 seconds","Compact model predicts bus times, error under 80s","Boston bus AI: departure deviations below 80 seconds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1378,"prompt_tokens":950,"completion_tokens":428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":566,"tokens_out":428,"duration_ms":4919,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:09:37.769188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on the first two months of MBTA 2023 data and test on the third month, holding out a contiguous future period, and compare RMSE with the reported 77.8 seconds; if the temporal-split RMSE exceeds roughly 120 seconds or degrades by more than 50 percent, the central real-time prediction claim is not supported.","supporting_citations":[],"review_version":1}