Pith.

Machine Learning · arXiv:2611.02461 v2 · submitted 2 Nov 2026

Training Autoregressive Language Models on Rounded, Bounded, and Censored Numerical Labels via Exact Likelihood

Jingquan Yan, Yuwei Miao, Yang Xie, Junzhou Huang

The paper says

“Numerical labels often specify a set of possible values: a measurement is rounded, a laboratory test reports only a bound, or a program is stopped at a timeout.”

more of the abstract

We introduce boundary marginalization, the first method that trains an autoregressive language model on the exact likelihood of such observations through its existing digit tokens. A fixed-width, order-preserving numeral representation groups consecutive grid values under shared digit prefixes. An interval is a union of such groups, so its exact probability and gradient follow from at most two boundary paths, the successive digit prefixes of its endpoints, each of length logarithmic in the number of grid values. The resulting objective unifies precise, rounded, bounded, and censored supervision and trains digit decisions at the resolution supplied by each observation. The same computation supports exact distributional queries and unbiased gradients for the continuous ranked probability score on exact labels. Experiments show recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and more accurate code latency distributions on two backbones. Exact interval updates remain tractable as the numerical support grows to one million values.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 3 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Boundary marginalization is a real, clean algorithmic contribution for training LMs on interval/censored numerical labels; the math holds up, the main open question is whether the real hERG reports satisfy the paper's own ignorability assumption. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §6.1 (hERG), Eq. (5), Table 2

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The hERG experiment is the only naturally censored real-data demonstration, and the manuscript presents its event-NLL numbers as exact report likelihoods. That interpretation holds only under conditionally ignorable reporting, q(A|k,x)=c(A,x)1{k in A}. The paper states this assumption in Section 3.3 and Appendix A, but the hERG reporting process itself selects the equality subset; if the equality-versus-bound decision or the bound …

more of objection 1

… threshold correlates with potency within the compatible set, the 0.408-nat full-NLL gain and the 60.3% censored-NLL reduction in Table 2 measure a proxy objective rather than the likelihood of the observed reports. The manuscript should either explicitly reframe the hERG results as comparisons of event NLL under the CAR assumption, or supply empirical support for (5) on this cohort, for example by showing threshold and equality-selection distributions conditional on molecule or assay features, or by giving a sensitivity analysis under a nonignorable reporting model.

The paper says, at §3.1, §6.1 (hERG conversion), Appendix E

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

Proposition 1 states exactness for the discrete grid distribution (3), but the hERG objective is constructed after mapping continuous ChEMBL measurements to grid events: equality records with undocumented precision become the nearest grid value, and the declared support is a 10^4-point pIC50 grid.

more of objection 2

The training loss is therefore the exact probability of the declared grid event, not the exact likelihood of the original continuous report. Since the abstract and Section 1 claim exact likelihood of such observations, the paper should qualify the claim as exact event likelihood on the declared grid and either quantify the discretization error or report sensitivity of Table 2 to the grid resolution.

The paper says, at §6.1 (rho=0), Appendix A

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.79):

“Atρ = 0, the controlled rounding and threshold design leaves some individual grid probabilities in the tails unidentified.”

Pith says: Objection 3 of 3 · major

The rho=0 controlled result is summarized as recovery of a conditional distribution from rounded and censored reports alone, but Appendix A states that this reporting design leaves some individual grid probabilities in the tails unidentified because the threshold distribution is confined to [-1.5, 1.5].

more of objection 3

The ISE metric integrates over the whole support, so the reported mean ISE of 6.801 is not by itself evidence of full distributional recovery; the claim should be scoped to the identified report probabilities, for example the CDF over the threshold-covered region, and the comparison with imputed-label CE at rho=0 should be interpreted accordingly.

4 minor objections

The paper says, at §4.2, Eq. (10)

Pith says: minor objection 1 of 4

The tuple-concatenation notation (double vertical bar) in equation (10) is hard to parse; an explicit expanded form for a carry example such as 0999 to 1000 would improve readability.

The paper says, at Figure 2 and Table 8

Pith says: minor objection 2 of 4

The ISE unit notation is inconsistent between the figure axis (x10^3) and the text (in units of 10^-3); please unify the convention.

show 2 more

The paper says, at §6 and Appendix F

Pith says: minor objection 3 of 4

The pinned revision hashes and dataset manifests are excellent for reproducibility, but a public code repository link would make the 112 fits and the precision benchmark easier for readers to verify.

The paper says, at §5.4, Algorithm 2

Pith says: minor objection 4 of 4

Algorithm 2 should state explicitly that the proposal distribution pi is fixed before training and re-used across model updates, since the unbiasedness in Proposition 4 depends on pi being independent of theta.

What would settle it

Test hERG ignorability directly. From ChEMBL, collect molecules with multiple hERG records, including at least one exact IC50 and one bound report. Use the exact record as a proxy for the underlying pIC50, and among records whose proxy value lies above the bound threshold, fit a logistic model for P(bound report | proxy pIC50, molecular weight/solubility descriptors, assay maximum concentration) and for the chosen bound threshold. If these covariates predict bound status or threshold after conditioning on membership in the compatible set, condition (5) fails and the hERG event-likelihood objective is not the report likelihood; if they do not, the ignorability concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 3 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 6 min 24 s · 25,820 tokens read, 21,740 written
  3. Editor wrote the verdict above 18 s · 27,451 tokens read, 900 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 53 s · 26,178 tokens read, 15,695 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The core of this paper is solid and genuinely new. Boundary marginalization — computing the exact probability of an interval event in an autoregressive digit decoder from at most two teacher-forced boundary paths — is a clean idea, and the derivation in Section 4 checks out. The FP64 verification against exhaustive enumeration on small grids is the right kind of evidence, and the complexity claim, O(b log_b M), is exactly what the algorithm delivers. Proposition 2's characterization of which prefixes actually receive gradient is also useful and not just a corollary. This is a real technical contribution, not a repackaging of imputation or auxiliary-head tricks.

The controlled experiments are the strongest part. The reporting mechanism is explicitly constructed to satisfy conditionally ignorable reporting (5), paired seeds are reported, and the comparisons at rho = 0, 0.1, 0.2, 0.5 are meaningful. The five-seed follow-up at rho = 0.2 is a nice check. The KBSS experiments on two backbones with fixed splits and timeouts are competently done. The paper is honest about what it cannot claim: it notes that predictive accuracy across grid depths is untested, that the two backbones differ in both size and generation, and that seed variation on fixed splits does not quantify data-sampling uncertainty. That level of candor earns credit.

The soft spots are real but not fatal. The main one is the hERG application. The full-NLL gain on naturally censored molecular data is the paper's most practically interesting result, and it depends on assumption (5): that for a fixed input, all compatible values are equally likely to produce the report, up to a constant independent of the model. The authors state this assumption and acknowledge in Appendix A that the assay reporting process selects the hERG equality subset. The concern is not that bounds exist — deterministic threshold censoring would satisfy (5) — but that the equality-versus-bound choice might be driven by unmodeled factors correlated with potency. If so, the hERG objective is the likelihood of a proxy event, not the likelihood of the observed report. This is an empirical question the paper does not resolve, and it deserves to be addressed head-on, ideally with a sensitivity analysis or an explicit argument about the ChEMBL reporting process.

Two smaller items. At rho = 1, the comparison uses one shared full-label reference fit per seed, so that point is descriptive rather than an independent comparison; the main gains at lower rho do not depend on it. And the paper ships no code, despite describing saved manifests and hashes — that is an addressable reproducibility gap, not a scientific flaw. Also worth noting: on hERG equality reports, event likelihood is 0.071 nats worse than complete-case CE, which is a minor but honest blemish in the table.

I would send this to peer review. The math is worth publishing even if the hERG application remains partly unresolved. A serious referee should push on the ignorability assumption and on code/data availability, but the central contribution survives that scrutiny. I'd cite the algorithm in my own work on numerical LM training, and I'd bring it to the reading group.

The strongest objection

The central algorithm is internally sound: with probabilities defined by (2)–(3), equations (7)–(11) and the gradient claims follow, and the FP64 checks in Appendix D support the implementation. The application claim, however, carries the extra premise of conditionally ignorable reporting, equation (5). Boundary marginalization computes pθ(A|x); this is the likelihood of the observed report only when q(A|k,x)=c(A,x)1{k∈A} with c independent of θ. The paper states this assumption and Appendix A concedes that nonignorable reporting requires modeling, so the unresolved question is empirical. The hERG experiment is the only naturally censored real-data demonstration, and the paper itself notes that "the assay reporting process selects the hERG equality subset." Deterministic threshold censoring would satisfy (5), so the concern is not that bounds exist; it is that the equality-versus-bound choice or the chosen threshold may be driven by unmodeled molecule- or assay-level factors that correlate with potency within the compatible set. If so, the event-NLL objective is misspecified, and the reported 0.408-nat full-NLL gain and 60.3% censored-NLL reduction are evidence about a proxy loss, not about the true report likelihood.

The reader's weakest assumption is also the most load-bearing condition I find. The mathematical core survives scrutiny: Proposition 1's partition argument is correct, Proposition 2's local-gradient form follows from posterior weighting, and the CRPS estimator proofs are standard and sound. The paper's own Appendix A acknowledges that nonignorable reporting requires modeling, so I do not count the assumption itself as a hidden flaw. What remains unresolved is whether it holds in the one natural real-data setting, hERG, where the report process determines equality versus bound status. That is empirically checkable, and the proposed test would settle it. Full-label reference sharing at ρ=1 and the absence of a code repository are secondary practical issues; neither changes the verdict independently. Since the reader already made the verdict CONDITIONAL on exactly this type of concern, no adjustment is needed.

The referee's report

The contribution is practically significant: it converts rounded, bounded, and censored numerical supervision into an exact likelihood objective using existing digit tokens, with no auxiliary head, imputation, or rollout, and the same boundary computation yields exact tail and quantile queries and unbiased CRPS gradients. The paper's strengths are its clean derivation, the FP64 verification against exhaustive enumeration in Appendix D, the explicit construction of controlled reporting mechanisms that satisfy the ignorability condition (5), the large number of paired fits (112, Table 7), and an unusual level of reproducibility detail with pinned revisions, manifests, and fixed splits. The consistency result Proposition 3 and the CRPS variance-reduction result Proposition 5 are useful beyond the specific application. The main caveat is that the exact-likelihood reading of the hERG results depends on conditionally ignorable reporting; this is acknowledged by the authors but not resolved. If that claim is scoped or supported, the paper would be a solid contribution.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that an autoregressive language model can be trained directly on the exact likelihood of imprecise numerical observations — rounded measurements, one-sided bounds, censored timeouts — through its existing digit tokens, rather than discarding such reports or replacing them with representative numerals. It claims a computation called boundary marginalization evaluates and differentiates the total probability of any interval of grid values using at most two teacher-forced boundary paths, each of length logarithmic in the number of grid values, so the cost does not depend on how many compatible values the report contains. If true, this unifies precise, rounded, bounded, and censored supervision into a single objective and lets each observation supervise the model at its own resolution, which matters because real numerical labels are often incomplete.

What carries the argument. The central object is the prefix-subtree decomposition of an ordered, fixed-width digit grid. With M = b^d grid values indexed by their base-b digit strings, the model's probability of a numeral is the product of digit conditionals, so the mass of a prefix subtree is just the prefix probability P(z_{<j}), and the suffix probabilities within any reached subtree sum to one. Boundary marginalization reads digit distributions only along the two boundary paths z(l) and z(u) — each path a single teacher-forced forward pass in a causal transformer — and writes the interval log-probability as a log-sum-exp over disjoint groups: the two boundary numerals, sibling subtrees choosing a middle digit at the first differing position, and subtrees departing from the lower or upper boundary at a later position. This decomposition carries the argument: it turns interval likelihood, gradients, CDF queries, and CRPS training into O(b·log_b M) scalar arithmetic, independent of the interval's width. Proposition 2 shows the resulting gradient supervises only prefix nodes whose subtrees straddle a boundary, weighted by the posterior probability of that prefix given the observed event.

Core claim. In a fixed-width numeral written most-significant digit first, every numeral strictly below a threshold first departs from the threshold's digit string at some position by choosing a smaller digit, after which all suffixes are allowed and their conditional probabilities sum to one. Numerals sharing a prefix therefore form a subtree whose total probability equals the probability of that prefix. An interval of values is a disjoint union of such subtrees, so its exact probability and gradient can be assembled from the two boundary digit strings — the successive prefixes of the interval's endpoints — together with sibling next-digit probabilities read from teacher-forced paths. Proposition 1 states this exactness: the specified equations compute exact event probabilities using at most 2d prefix conditionals and O(b·log_b M) scalar arithmetic, with no enumeration of the support. The same computation yields exact cumulative distribution and quantile queries and unbiased, threshold-sampled gradients for the continuous ranked probability score. Experiments report recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and improved code-latency distributions on two backbones, with interval updates staying tractable as the support grows to one million values.

The premise it rests on. The entire likelihood interpretation rests on conditionally ignorable reporting: for a fixed input, every value compatible with a report must have the same chance of producing it, with that chance independent of the model parameters; if real reports are informative about the underlying value, the training objective is misspecified.

If the paper is right (Pith's inferences, not the authors')

A single loss covers precise labels, rounding intervals, and one-sided bounds, and a singleton observation reduces exactly to standard sequence cross-entropy, so one decoder trains on mixed report types without imputed targets.

Each observation supervises digit decisions at the resolution it actually supplies: subtrees fully inside the event contribute their whole mass with no target digit, and only prefixes straddling a boundary receive a direct gradient.

Trained models answer exact tail, interval, and quantile queries without sampling, and the same boundary computation gives unbiased CRPS training gradients on exact labels.

Training cost and memory stay nearly flat as the grid grows from 10^2 to 10^6 values, with interval updates using roughly 43–68% less peak memory than a dense per-value output.

With zero precise labels, rounding and threshold reports alone recover the conditional distribution with mean ISE 6.801×10^-3, an 80.1% reduction over imputed-label cross-entropy, and event likelihood improves censored-report NLL on the molecular data and code-latency CRPS on two backbones.

A direct extension would couple boundary marginalization with an explicit model of the reporting mechanism: when the chance of a report depends on the underlying value (non-ignorable reporting), the objective is no longer the likelihood of the observed report, and the equality-versus-bound selection in the molecular assay is a plausible real-world case where this matters.

The identity relies only on the tree structure of an ordered autoregressive distribution, not on the transformer architecture, so the same two-path computation should transfer to hierarchical softmax over ordered bins or any tree-structured decoder over a totally ordered support.

Because exact CDF values and quantiles are available in one or two forward passes, the method enables decision-theoretic decoding — optimal point predictions under asymmetric losses, or calibrated interval forecasts — without sampling rollouts, an opportunity the paper mentions only in passing.

The identifiability condition suggests a testable design rule: overlapping reports, such as mixed rounding widths or interval-plus-bound reports, identify within-bin probabilities, so one could quantify how much distributional information a given reporting scheme can ever reveal.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Numerical labels often specify a set of possible values: a measurement is rounded, a laboratory test reports only a bound, or a program is stopped at a timeout. We introduce boundary marginalization, the first method that trains an autoregressive language model on the exact likelihood of such observations through its existing digit tokens. A fixed-width, order-preserving numeral representation groups consecutive grid values under shared digit prefixes. An interval is a union of such groups, so its exact probability and gradient follow from at most two boundary paths, the successive digit prefixes of its endpoints, each of length logarithmic in the number of grid values. The resulting objective unifies precise, rounded, bounded, and censored supervision and trains digit decisions at the resolution supplied by each observation. The same computation supports exact distributional queries and unbiased gradients for the continuous ranked probability score on exact labels. Experiments show recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and more accurate code latency distributions on two backbones. Exact interval updates remain tractable as the numerical support grows to one million values.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-03

The authors' main claim, as the classifier picked it out 0.64

“Experiments show recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and more accurate code latency distributions on two backbones.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.94 · states limitation 0.12 · human participants 0.05 · controlled comparison 0.14 · randomized assignment 0.02 · states quantitative result 0.73 · compares with prior results 0.27 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Numerical labels often specify a set of possible values: a measurement is rounded, a laboratory test reports only a bound, or a program is stopped at a timeout. background 1.00 0.06 0.02 asserted 0.96
We introduce boundary marginalization, the first method that trains an autoregressive language model on the exact likelihood of such observations through its existing digit tokens. contribution 1.00 0.95 0.49 asserted 1.00
A fixed-width, order-preserving numeral representation groups consecutive grid values under shared digit prefixes. method 0.99 0.52 0.04 asserted 1.00
An interval is a union of such groups, so its exact probability and gradient follow from at most two boundary paths, the successive digit prefixes of its endpoints, each of length logarithmic in the number of grid values. method 0.97 0.82 0.11 asserted 1.00
The resulting objective unifies precise, rounded, bounded, and censored supervision and trains digit decisions at the resolution supplied by each observation. contribution 0.65 0.87 0.32 asserted 1.00
The same computation supports exact distributional queries and unbiased gradients for the continuous ranked probability score on exact labels. contribution 0.81 0.85 0.10 asserted 1.00
Experiments show recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and more accurate code latency distributions on two backbones. result 1.00 0.95 0.34 asserted 1.00
Exact interval updates remain tractable as the numerical support grows to one million values. result 0.99 0.91 0.13 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.78

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-03: 1 request, 6,235 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Boundary marginalization for 237 ≤ Y ≤ 684 on the three-digit grid vk = k, where ∗ denotes an arbitrary digit. Every numeral in the event either is a boundary numeral or first departs from a boundary path at one digit and then continues with an arbitrary suffix. Since the suffix probabilities of a subtree sum to one, the event probability is a sum of prefix masses times sibling digit probabilities, all evaluated at prefixes on the two boundary paths (colored). The model is never evaluated inside a shaded subtree, so the cost is independent of the 448 values in the event. inference…
Figure 2 of the paper
Figure 2. Distribution recovery across precise-label probabilities. Large markers show mean ISE and
Figure 3 of the paper
Figure 3. Conditional CDFs at ρ = 0.1 for the test input nearest x = 0 (x = −0.000416), with one panel per training seed. Curves use checkpoints selected by validation event NLL and connect evaluations at 512 common grid thresholds. The dashed curve is the true grid CDF. hERG: natural reports KBSS: exact test outcomes Method Full NLL Equality NLL CRPS (10−3 ms) CRPS (log10 ms) Event likelihood (ours) 4.675±0.012 6.857±0.049 8.262±0.299 0.0878±0.0013 Complete-case CE 5.083±0.055 6.786±0.062 8.870±0.235 0.0980±0.0024 Imputed-label CE 4.958±0.032 6.998±0.086 10.229±0.135 0.1205±0.0026…
Figure 4 of the paper
Figure 4. Precision scaling without dense output growth. Depths
Figure 5 of the paper
Figure 5. Individual fits behind Figure 2. Each marker is one training run, identified by its fine-tuning
Figure 6 of the paper
Figure 6. Paired differences across training seeds. (a) ISE differences, event likelihood minus each of
Figure 7 of the paper
Figure 7. Validation CRPS during the exact-label score study, shown from update 100 through the
Figure 8 of the paper
Figure 8. Training update time for the same workloads as Figure 4. The event implementation

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment