Unbiased determination of the proton structure function F_2^p with faithful uncertainty estimation

Worth your time if you care about how uncertainty bands on functions get constructed; less so if you just want the F2 number. This is the NNPDF F2 paper, and it is the template for everything that came after, so it is worth knowing what it does and does not nail down. What is genuinely new relative to their 2002 paper: the genetic-algorithm epoch that minimizes the full correlated chi-squared (eq. 12), the dynamical weighted training of eq. 15, HERA data folded in, and a shipped artifact — a FORTRAN routine plus 1000 parameter sets at the URL in the conclusion. That last part matters; you can actually check their band against your own fit. The statistical bookkeeping is careful: replica generation (eq. 9), normalization handled as an overall rescaling (eqs. 13-14), D'Agostini symmetrization of asymmetric errors (eqs. 7-8). They are also upfront about the things that went wrong — NMC's internal inconsistencies, the post-hoc ZEUS94 exclusion argued through kinematic overlap and E3 deterioration, and the caveat that extrapolated errors become unreliable far from the data. That honesty is worth something. Where I would push: the abstract's 'bias-free' (abstract:s1) is stronger than the body supports. The width of the band is set by the GA training length, fixed at 4x10⁴ cycles by requiring the central fit to flatten at chi2 ~ 1.2 (fig. 3) — the same central values the ensemble is later said to describe. Figure 4 shows the net error falling monotonically with training length, so the band is a bias-variance knob tuned for self-consistency, not an independently validated quantity. The only safeguard is the R-estimator, computed on the same replicas and data, and footnote 2's claim that the R ~ 0.5 calibration survives the switch from E2 to E3 is asserted ('easy to see') rather than derived or checked numerically. Since the whole point of this paper over the 2002 one is that E3 minimization matters, that is exactly the step that needed support. The stress-test note lands here; the reader's circularity score of 4.0 is a touch harsh, because fig. 4 is a real argument, just not an independent one. None of this sinks the method — the 3-4x error reduction is plausibly information-combining — but the abstract should have said 'controlled' rather than 'bias-free'. Send it to referees; cite it.

The abstract's claim [abstract:s1] is that the ensemble gives a bias-free probability measure with faithful errors. The width of that measure is set almost entirely by one hyperparameter: the GA training length. Section 4 fixes it at 4×10^4 cycles by requiring the fit to the central data to flatten at χ²≈1.2 (fig. 3), i.e. the stopping point is chosen using the same experimental central values that the ensemble is later said to describe. Figure 4 then shows ⟨σ(net)⟩ falling monotonically with training length and tracking the χ² improvement. So the reported error band is the value of a bias–variance knob that was tuned until the fit looked good against the data — a self-consistency condition, not an independent validation of 'faithful' errors. The only stated safeguard against overlearning is the R-estimator (eq. 34), and it has two weaknesses the paper does not close. First, R is computed from the same replicas and the same data used in training, so it cannot detect a bias that is common to all replicas (e.g. a systematically over-shrunk spread). Second, its calibration — R≈0.5 clean, R≳1 biased — is imported from ref. [4], where the nets minimized the diagonal E2; footnote 2 states that 'it is easy to see' the calibration survives under E3 'provided R is redefined accordingly', but no derivation or numerical check of the redefined R is given. Since the whole point of this paper relative to ref. [4] is that E3 minimization (via GA) is now essential — the paper itself shows E2 minimization gives no BCDMS error reduction — the calibration transfer is exactly the step that needs support and does not get it. The 3–4× reduction of ⟨σ(net)⟩ below ⟨σ(exp)⟩ (table 5) is therefore asserted to be information-combining rather than over-shrinking on the strength of an unverified diagnostic. This is a narrower and more checkable concern than the reader's, which correctly flags the internality of R but does not locate the training-length tuning as the mechanism that sets the band

The paper is a careful methodological work with a shipped artifact (FORTRAN routine plus 1000 parameter sets) and unusually candid disclosure of its choices, including the admission in section 5 that extrapolated errors become unreliable far from the data. Much of what could be criticized is already acknowledged: the NMC and ZEUSSVX95 residuals are discussed, the ZEUS94 exclusion is argued through kinematic overlap and E3 deterioration, and the asymmetric-error treatment is delegated to refs. [21,22]. I do not treat any of those as load-bearing, and I do not treat the reliance on published results as a defect. The concern I do raise is narrower than the reader's and, I think, sharper. The reader correctly identifies that the R-estimator is an internal diagnostic computed on the training data, and that the bias-freedom claim inherits whatever R fails to detect. I agree with that framing but would locate the mechanism more precisely: the ensemble width is governed by the GA training length, which section 4 selects by requiring χ²≈1.2 on the central data, and figure 4 shows the width falling in lockstep with that χ². That makes 'faithful uncertainty' a self-consistency statement about a tuned hyperparameter. The second, independent gap is footnote 2: the R calibration is carried over from ref. [4]'s E2-trained nets to the present E3-trained nets by assertion. Because the paper's own argument is that E3 minimization changes the outcome materially (it is what produces the BCDMS error reduction absent in ref. [4]), the calibration transfer is precisely the step that cannot be assumed. Neither gap is fatal, and neither is concealed; both are the kind of thing a closure test settles in an afternoon with the released code. That is why I would leave the verdict at CONDITIONAL rather than move it: the method is sound, the artifact is available, and the claim is plausible, but the word 'faithful' currently outruns the internal evidence by exactly one unvalidated calibration step. If the closure test in the concrete_test field shows the recovered band matches the true replica spread, the paper's central claim is fully supported and I would move to ACCEPT; if the band is over-shrunk, the abstract's 'faithful uncertainty estimation' needs qualification to 'uncertainty estimated under a training-length prescription validated on the fitted data'.

If the central claim holds, this is a valuable methods paper: it delivers a usable, publicly downloadable determination of F2(x,Q2) with an uncertainty band, and it develops the machinery (GA training on the full correlated chi-squared, dynamical weighting, probabilistic selection) that the collaboration subsequently needs for parton distributions. The paper deserves credit for releasing the full ensemble rather than only a central fit, which makes the result reproducible and independently checkable, and for the explicit pull comparison to the pre-HERA fit of ref. [4] (Fig. 8), which is a falsifiable, quantitative statement rather than a qualitative assertion. The authors are also commendably explicit that extrapolated errors grow rapidly and become unreliable far from the data (region (c) discussion in §5), which is exactly the kind of limitation such a paper should state. The technical core is carefully documented and, in my reading, sound.

The paper tries to establish that one can put a trustworthy uncertainty band on a function — here the proton structure function $F_2(x,Q^2)$ — rather than only on a handful of fitted parameters. Its strategy is to generate many fake copies of the experimental data that reproduce the measured central values, errors and correlations, then train one neural network on each copy; the spread of the resulting network ensemble is presented as the probability distribution of the function itself. If that is right, the band is not an artifact of choosing a particular fitting form, and the same machinery can be carried over to parton distributions, where reliable errors are needed for collider predictions. The paper also supplies a concrete public product: a set of 1000 trained networks that return $F_2(x,Q^2)$ and its uncertainty at any point in the covered kinematic region.

What carries the argument. The machinery is a three-stage training scheme for an ensemble of neural networks. First, many replicas of the data are drawn as a multigaussian distribution, with one random number per independent error source, so that each replica carries the experimental central values, statistical errors, correlated systematics and normalization uncertainty. Each network is then trained on one replica: two back-propagation epochs first minimize an unweighted deviation and then the diagonal chi-squared, and a final genetic-algorithm epoch minimizes the full correlated chi-squared $E_3$, which is nonlocal and expensive but captures the correlations. The ensemble average gives the central structure function, and the ensemble variance gives the uncertainty; the $R$ estimator compares the average full chi-squared of individual networks to the chi-squared of the ensemble mean and is used as a check that the error reduction is not caused by a restrictive fitting form.

Core claim. The central claim is that a Monte Carlo ensemble of neural networks, trained on replicas of all available charged-lepton deep-inelastic scattering data, provides a bias-free determination of the probability measure in the space of structure functions. The authors argue that the ensemble retains the experimental errors and correlations, including correlated systematics and normalization uncertainties, while interpolating between measured points and extrapolating under a smoothness requirement. They support the bias-freedom claim with an internal diagnostic, the ratio $R$ of two chi-squared measures, which comes out near $0.5$ for essentially all experiments, and with the observation that the network error shrinks as training improves rather than staying fixed as a parametrization bias would. The final data set contains 1698 points from NMC, BCDMS, E665, ZEUS and H1, with ZEUS94 excluded because its inclusion degraded the fit without adding kinematic coverage.

The premise it rests on. The load-bearing premise is that the network ensemble converges to the underlying physical curve rather than to noise in the replicated data, and that the paper's internal $R$ diagnostic — near $0.5$ when no fitting-form bias is present — reliably distinguishes those two outcomes.

The method gives a usable uncertainty band for $F_2^p(x,Q^2)$ across the measured region and a public set of 1000 networks that can be evaluated at arbitrary kinematic points.

Because the training handles many disparate data sets with correlated systematics, the same replica-plus-network approach can be extended to a full set of parton distributions, which is the stated next target.

The paper's data-selection procedure — training on subsets and keeping experiments that lower the total chi-squared — offers a way to automate decisions about which data sets to include in a global fit.

The comparison with the earlier BCDMS/NMC-only fit shows that adding HERA data sharpens the band at small $x$ and large $Q^2$, while also showing that extrapolated errors grow rapidly and become unreliable far outside the data.

A convincing public test would be to release closure-test coverage plots alongside the 1000 networks; if the band is a true probability measure, repeated synthetic experiments should cover the known truth at the nominal rate, not merely produce a smooth spread.

The paper's own old-versus-new comparison hints at a practical limit: the ensemble can be precise where data constrain it and still be confidently wrong where it extrapolates, so users should treat the band as conditional on the training region rather than as a global error.

The $R$ diagnostic could be stress-tested by deliberately training on a restrictive parametrization; if $R$ remains near $0.5$ there, it is not sensitive to the bias it is meant to detect.

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review