{"id":"c6fb5d91-5cf2-4d0e-81b6-938bee8da1e0","arxiv_id":"2412.00753","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper defines ecological forecast limits as the first lead time at which a scoring function exceeds a tolerance, and classifies them into potential, absolute, and relative predictability.","lead":"This paper proposes a unified framework for computing forecast limits in ecological forecasting, the time horizon up to which a model's predictions remain acceptable. It distinguishes potential, absolute, and relative limits and demonstrates them on population, forest, and land-surface models.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 6 and Eq. 10 do not define the first crossing of the tolerance: argmin(S−ϱ) returns the minimum score, not the crossing, and the skill-ratio direction in §2.3 is inverted.","rationale":"The reader's CONDITIONAL verdict identifies the verification-reference issue, especially the yield-table circularity in the iLand case study. That is a real limitation, and the authors acknowledge it in §4.6. But the more load-bearing problem is internal to the formalization: Eqs. 6 and 10 are the definitions on which all three forecast-limit types rest, and they do not express the first-crossing time the text describes. This is not a matter of external consensus or a hidden assumption; it is a mathematical inconsistency in the central construction. The applied case studies use the standard CRPSS definition (Eq. 15), which is correct, so the empirical demonstrations are not necessarily invalidated. However, the paper's contribution is explicitly a formal framework, and a framework whose central definitions are ill-posed cannot support the claim that forecast limits are defined for practically any ecological forecast. The fix is localized: replace argmin of the signed margin with the first hitting time of the tolerance, or the first time the skill score is non-positive, and correct the direction of Eq. 9. Because the intended recipe is clear and the reported calculations appear to follow the standard skill-score convention, this warrants revision rather than rejection. Hence the reader's CONDITIONAL verdict stands, with the formalization defect as the primary reason.","tokens_in":31221,"tokens_out":4670,"duration_ms":47085,"concrete_test":"Evaluate Eq. 6 with S(εt) = (1, 2, 3) and ϱ = 2: the condition S ≤ ϱ holds at t = 1 and first fails at t = 3, but argmin_t(S − ϱ) returns t = 1. For the relative limit, use a non-monotone skill series such as (0.5, 0.1, −0.2, 0.3): Eq. 10 returns the most negative value (t = 3), while a first-crossing definition would give t = 3 only if monotone; with series (0.5, −0.2, 0.3) Eq. 10 gives t = 2, and the meaning of 'first crossing' is ambiguous. Recompute the Ricker and aiLand forecast limits using the corrected hitting time h = min{t : S(εt) > ϱ} (or the analogous skill-score definition) and compare the reported limits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that forecast limits can be computed from a verification reference, a scoring function, and a predictive-error tolerance, with Eqs. 5–10 providing the formal definition. As written, these equations do not define the quantity they describe. In §2.2, the acceptable region is S(εt) ≤ ϱ and h is called the first time the condition fails. But h = argmin_t (S(εt) − ϱ) minimizes the signed margin; before the crossing S − ϱ ≤ 0, so the argmin is typically the smallest lead time rather than the crossing lead time. Eq. 7, with γ = 1 only when S − ϱ = 0, also cannot mark the transition from 0 to 1. §2.3 has the same argmin problem and a sign error: for an error score, S(εt)/S(εRt) > 1 means the forecast model is worse than the reference, yet the text says the more the ratio exceeds 1, the better the forecast performs; and hr = argmin_t(1 − S/SR) again selects the smallest skill value, not the first time skill drops to zero. All three limit types inherit Eqs. 6 and 10, so the formal framework, as written, does not yet produce the forecast limit it claims. This is separate from the acknowledged observation-error caveat and should be repaired before the framework is used.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified framework for defining empirical forecast limits in ecological forecasting, distinguishing potential, absolute, and relative forecast limits. The framework rests on three ingredients: a verification reference Y_t, a scoring function S, and a predictive-error tolerance ϱ. The authors formalize the forecast limit as the first time the score crosses the tolerance, provide recipes and a decision tree, and demonstrate the three limit types with a stochastic Ricker population model, the iLand forest model, and the aiLand land-surface emulator. The central claim is that, within this framework, forecast limits can be defined for practically any ecological forecast.","tokens_in":31539,"tokens_out":3832,"duration_ms":38499,"significance":"The taxonomic distinction among potential, absolute, and relative forecast limits, and the attempt to unify existing practices under common formal definitions, is a useful contribution to ecological forecasting. The paper is commendably concrete: it provides a decision tree, worked case studies at three ecological scales, and code/data availability statements, and it candidly acknowledges several limitations, including model-conditionality of limits and observation error. However, the central formalization in Sections 2.2 and 2.3 is internally inconsistent as written: the defining equations do not return the first crossing of the tolerance, and one sign argument is inverted. Because the equations are the paper's principal claimed contribution, they need to be corrected before the framework can be accepted.","major_comments":[{"comment":"Equation (6), h = argmin_t (S(ε_t) − ϱ), does not define the first time the acceptable condition S(ε_t) ≤ ϱ fails. Before the crossing, S(ε_t) − ϱ is negative, so the argmin over t will typically be the lead time with the smallest score, not the crossing time. Equation (7), with γ = 1 only when S(ε_t) − ϱ = 0, cannot represent the transition from acceptable to unacceptable unless the score exactly equals the tolerance at the boundary. The correct definition would be h = min{t ∈ T_i : S(ε_t) > ϱ}, with a corresponding step function that switches from 0 to 1 at that time. Because all three limit types inherit this definition, this error is load-bearing.","section":"§2.2, Eq. (6)–(7)"},{"comment":"The relative-skill formulation contains a sign error and an invalid argmin. For an error score, S(ε_t)/S(ε_R_t) > 1 means the forecast model is worse than the reference, yet the text states that the more the ratio exceeds 1, the better the forecast performance. Equation (10), h_r = argmin_t (1 − S(ε_t)/S(ε_R_t)), selects the time where the skill expression is most negative, i.e. where the forecast model is worst relative to the reference, not the first time skill drops to zero. The intended object is the first lead time at which S(ε_t) > S(ε_R_t), equivalently the first time the skill score drops below zero.","section":"§2.3, Eqs. (9)–(10)"},{"comment":"The statement 'By definition, the expression in the brackets of 10 is monotonic' is not correct for the quantities used in this paper, and it is contradicted by the paper's own results. The CRPSS time series shown in Figures 5B, 6B, 7B, and 8B oscillate around zero and cross the threshold more than once; for example, the surface-layer soil temperature series in Figures 5B and 7B repeatedly leaves and re-enters the skilful region. The 'first crossing' of a non-monotonic curve is still definable, so the framework does not require monotonicity, but the claim as written should be removed or replaced by a statement about the first crossing.","section":"§2.3, monotonicity claim"},{"comment":"The iLand case study demonstrates the absolute forecast limit, but both the verification Y_t and the tolerance ϱ are derived from the same yield-table model: Y_t is reconstructed from yield tables, and the tolerance is the neighbouring yield class k ± 1. Consequently, the computed absolute limit partly measures the distance to an alternative yield-table trajectory rather than the distance to the observed ecosystem state. The manuscript acknowledges this in Sections 4.2 and 4.6, and the acknowledgment is appreciated, but the issue should also be stated where the framework's generality is claimed, because the trustworthiness of the verification is a precondition for the central claim that forecast limits are defined for any ecological forecast.","section":"§3.2 and §4.6, iLand verification"}],"minor_comments":[{"comment":"Equation (12) defines S_AE_t = ϱ − |ε_t| and says the forecast limit is reached when AE_t < 0, which is consistent with the tolerance crossing, but the preceding text says 'the forecast limit is reached when AEt < 0' while Eq. (5) frames acceptability as S(ε_t) ≤ ϱ; the use of strict versus non-strict inequalities should be harmonized throughout.","section":"§2.2, Eq. (12)"},{"comment":"Equation (8), S(ε_t) ≤ ϱ ≤ S(ε_t^R), is stated as a chain of inequalities, but ϱ is introduced as a scalar threshold; if the reference score is below the threshold, the chain cannot hold. The intended meaning is presumably that the reference defines the threshold at each lead time, i.e. ϱ(t) = S(ε_t^R).","section":"§2.3, Eq. (8)"},{"comment":"In the second paragraph of Section 3.3, 'CPRPSS' is a typo for 'CRPSS'.","section":"§3.3, results text"},{"comment":"The caption begins 'The resulting heat maps in Figures 7 and 7 A reveal...' and later refers to 'Figures 7 and 7 A'; the duplicated reference should be corrected to point once to the figure.","section":"Figure 7 caption"},{"comment":"In the station list, the soil type for LaGrandCombe is given as 'LaGrandCombe', which appears to be a placeholder or typo; the soil type should be completed or omitted.","section":"Appendix B, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's conceptual taxonomy and case studies are likely useful to the ecological forecasting community, but the formal definitions in Section 2 need correction before the central claim is substantiated. The issues identified are fixable within the manuscript's scope, so I do not recommend rejection; the authors should revise Equations (6), (7), (9), and (10), remove the monotonicity claim, and reconsider how the iLand verification caveat is reflected in the framing. I would also suggest asking the authors to state explicitly that their computed forecast limits in the case studies were obtained by the intended crossing definition, not by the literal argmin formulas."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful synthesis: it gives ecological forecasters a clear vocabulary (potential, absolute, relative forecast limits), recipe boxes, and three worked examples with code and data links. The aiLand relative-limit case uses independent station observations, which is the cleanest of the three. Second, the formal definitions of the forecast limit are wrong as written. Eq. 6 defines h as argmin(S−ϱ), which before the crossing selects the earliest lead time rather than the crossing. Eq. 7's step function, which fires only at exact equality, will never fire for continuous scores. In §2.3 the skill-ratio direction is inverted: S(εt)/S(εR_t)>1 means the forecast is worse than the reference, not better, and the claim that 1−S/S_R is monotonic by definition is contradicted by their own CRPSS curves. These are load-bearing because the paper says the framework defines forecast limits for practically any ecological forecast, but the three-type distinction and the case studies survive the bad equations—the fix is to replace argmin with something like min{t : S(εt)>ϱ}.\n\nWhat is genuinely new: the three-way distinction and the decision tree are a reasonable repackaging of existing forecast-horizon and skill-horizon concepts from Petchey et al. and Buizza and Leutbecher. That is not a flaw for a synthesis; the novelty is modest but real in packaging and terminology. The case studies demonstrate the framework concretely, and the limitations section is unusually honest, especially on observation error and the iLand yield-table reconstruction. The iLand absolute limit is partly a consistency check against yield tables, as the authors concede, not an independent verification.\n\nWho this is for: applied ecologists and forecasting practitioners who want a checklist for reporting how far ahead their models are useful. It deserves a serious referee because the framework is standardizable and the errors are repairable. I would not cite the current version in my own work. If they fix the equations and make the yield-table tolerance framing explicit, I'd likely cite it.","headline":"Useful synthesis and case studies, but the defining equations for forecast limits are wrong as written and need repair before this can be adopted.","tokens_in":32078,"tokens_out":3531,"would_cite":false,"duration_ms":30708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P12","62M20"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that forecast limits can be computed for practically any ecological forecast using just a verification reference, a scoring function, and a predictive error tolerance, and unifies potential, absolute, and relative limits…","keywords":["ecological forecasting","forecast limit","predictability","verification","scoring functions","CRPS","benchmark models","MAE"],"falsifier":"Take a synthetic forecast system with known ground truth, compute its forecast limit using the true state as verification, then recompute it using a degraded, noisy observation of that state; if the two limits differ substantially, the framework's reliance on verification fidelity is the source of error and the claim that limits are defined for any forecast fails for realistic observations.","tokens_in":31024,"feed_emoji":"🌿","tokens_out":8659,"duration_ms":69497,"temperature":0.7,"pith_summary":"The paper argues that every ecological forecast can be assigned a forecast limit, the moment in time beyond which the forecast's predictive error is no longer acceptable, and that this limit is computable from three ingredients: a verification reference, a scoring function, and a predictive error tolerance. It distinguishes three kinds of limits: the potential limit using simulated verification in a perfect-model setting, the absolute limit using an ad-hoc error tolerance, and the relative limit using a benchmark or null model such as climatology. The authors test the framework on three case studies, a stochastic Ricker population model, the iLand forest ecosystem model, and the aiLand machine-learning land-surface emulator, and conclude that forecast limits are defined for practically any ecological forecast. A sympathetic reader would care because forecast limits convert vague claims about model skill into an interpretable statement of how far into the future a prediction can be trusted, which is what decision-makers need.","feed_headline":"Three ingredients fix the trust limit of any ecological forecast","feed_subtitle":"A verification, a scoring rule and an error tolerance say how far ahead model statements can be trusted.","key_machinery":"The load-bearing machinery is the inequality $S(\\varepsilon_t) \\leq \\varrho$ combined with the crossing-time identity $h = \\mathrm{argmin}_t\\,(S(\\varepsilon_t) - \\varrho)$. Here $S$ is any monotonic scoring function of the predictive error $\\varepsilon_t = \\hat{Y}_t - Y_t$, and $\\varrho$ is the scoring tolerance, which can be a reference model's score, a skill-score threshold, or an ad-hoc error bound. For the relative limit the machinery becomes the skill-score ratio $1 - S(\\varepsilon_t)/S(\\varepsilon_t^R)$ with a benchmark $R$ such as climatology, and for the potential limit both verification and reference are drawn from the forecast model's own ensemble, so the framework needs no observations at all. The same pair $(S, \\varrho)$ carries all three case studies: MAE for the iLand point forecast, CRPS for the Ricker and aiLand ensembles, and either a neighbour yield-class or the climatological distribution as the tolerance.","core_discovery":"The paper's central claim is that the forecast limit, the lead time at which predictive error grows beyond acceptable tolerance, can be defined for essentially any ecological forecast, and that all existing definitions collapse into one formal scheme. The scheme requires exactly three ingredients: a verification reference Y_t, which may be observations or model simulations; a scoring function S that measures the discrepancy between forecast and verification, with the authors using mean absolute error for point forecasts and the continuous ranked probability score for ensembles; and a scoring tolerance ϱ that sets the acceptable error, either as a benchmark model or as an ad-hoc threshold. Depending on the choice of verification and tolerance, the same equation yields the relative forecast limit, where the forecast is no better than a reference model such as climatology; the potential forecast limit, a perfect-model upper bound with both verification and reference simulated from the model; and the absolute forecast limit, where the error exceeds an ad-hoc standard. The authors demonstrate all three in case studies and conclude that within this framework forecast limits are defined for practically any ecological forecast.","pith_inferences":["If the framework is correct, forecast limits could be reported as routinely as skill scores in ecological forecasting, giving every issued forecast an explicit expiry date in lead time.","A large gap between a model's potential and relative forecast limits would diagnose that model structure or observation error, not intrinsic system unpredictability, is the binding constraint, telling modellers where improved data or process representation would buy the most lead time.","The absolute limit's dependence on ad-hoc tolerances implies the same forecast can be adequate for one user and inadequate for another, so formalising the tolerance choice would turn forecast evaluation into a stakeholder-driven quantity.","Applying the framework to hindcasts with different verification types, direct observations versus model reconstructions, would quantify how much the reported limit is an artefact of the verification, testing the paper's reliance on verification fidelity."],"forward_implications":["Forecast limits give ecological forecasts an interpretable trust horizon instead of a raw skill score at arbitrary lead times.","The relative forecast limit ranks models by how long they stay ahead of a benchmark such as climatology, making model comparison a statement about lead time.","The potential forecast limit estimates the upper bound of system predictability from within the model, requiring no observations at all.","The absolute forecast limit couples forecast evaluation to user-defined error tolerances, so adequacy becomes decision-specific.","The same three-requirement recipe works for population, ecosystem, and Earth-system models, so forecast limits become comparable across ecological scales."],"supporting_citations":[{"why":"Introduces the ecological forecast horizon that this paper formalises into three distinct forecast-limit types.","marker":"Petchey et al. (2015)"},{"why":"Defines the forecast skill horizon against a climatological reference, the basis of the relative forecast limit.","marker":"Buizza and Leutbecher (2015)"},{"why":"Provides the statistical formalism of model forecasts and process error on which the error definitions rest.","marker":"Berger and Smith (2019)"},{"why":"Supplies the theory of strictly proper scoring rules used to justify the scoring function S.","marker":"Gneiting and Raftery (2007)"},{"why":"Establishes benchmarks and skill scores in hydrological ensemble prediction, the template for the relative limit.","marker":"Pappenberger et al. (2015)"},{"why":"Defines predictability as the difference between forecast and climatological distributions, grounding the potential limit.","marker":"DelSole and Tippett (2007)"},{"why":"Demonstrates the forecast limit as a function of ad-hoc tolerance, the template for the absolute limit in the iLand case.","marker":"Massoud et al. (2018)"},{"why":"Is the process-based forest model iLand used to demonstrate the absolute forecast limit.","marker":"Seidl et al. (2012a)"},{"why":"Is the machine-learning land surface emulator aiLand used to demonstrate the relative forecast limit.","marker":"Wesselkamp et al. (2025)"}],"fun_headline_variants":["Three rules cap trust in any ecological forecast","One framework sets the limit for every ecological forecast","How far can forecasts be trusted? Three ingredients decide","Ecological forecast limits: a universal recipe in three steps","Verification, scoring, tolerance: the forecast limit trio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework treats the verification reference as a faithful stand-in for the true ecosystem state, ignoring observation error; if that reference is noisy or itself model-derived, the computed forecast limit can describe the reference rather than the system.","fun_headline_variants_meta":{"raw":{"variants":["Three rules cap trust in any ecological forecast","One framework sets the limit for every ecological forecast","How far can forecasts be trusted? Three ingredients decide","Ecological forecast limits: a universal recipe in three steps","Verification, scoring, tolerance: the forecast limit trio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1288,"prompt_tokens":956,"completion_tokens":332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":572,"tokens_out":332,"duration_ms":3583,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:02:17.290435+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic forecast system with known ground truth, compute its forecast limit using the true state as verification, then recompute it using a degraded, noisy observation of that state; if the two limits differ substantially, the framework's reliance on verification fidelity is the source of error and the claim that limits are defined for any forecast fails for realistic observations.","supporting_citations":[{"cited_title":"font-variant:small-caps","cited_arxiv_id":null,"evidence_quote":"Establishes benchmarks and skill scores in hydrological ensemble prediction, the template for the relative limit."}],"review_version":1}