REVIEW 2 major objections 2 minor
Deciding which tokens small language models should learn versus delegate requires grammatical signals beyond loss alone.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-05-21 12:50 UTC pith:UUIAAANH
load-bearing objection Loss alone misses delegation cues for factual errors in SLMs, and spaCy grammar adds a workable but imperfect fix. the 2 major comments →
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
LaCy augments the standard language modeling loss with factuality signals derived from grammatical information to determine which tokens an SLM should predict itself and which ones should be replaced by a <CALL> token during pretraining. This combination allows the model to learn appropriate delegation behavior, resulting in higher FactScores during generation in a cascade with a larger model and outperforming SLMs trained with Rho or LLM-judge methods.
What carries the argument
The LaCy pretraining objective, which combines per-token loss with grammatical factuality signals to decide between self-prediction and delegation via a <CALL> token.
Load-bearing premise
Grammatical information reliably signals whether a token continuation is factually or semantically valid in the target domain.
What would settle it
Training an SLM with only the loss signal and measuring whether its FactScores in a cascade match or exceed those of a LaCy-trained model on the same data.
If this is right
- SLMs trained this way produce higher FactScores when generating in a cascade with a larger model.
- Delegation decisions become more accurate by distinguishing acceptable high-loss alternatives from invalid ones.
- The method requires no external judges during training and remains simpler and cheaper than alternatives.
- The model learns to insert the delegation token at appropriate points rather than always defaulting to loss-based triggers.
Where Pith is reading between the lines
- Pretraining for capacity-limited models may need domain-adapted validity signals rather than uniform loss objectives.
- Similar augmentation could be tested in non-Wikipedia text by substituting other lightweight structural cues for the grammatical parser.
- If the approach generalizes, it suggests delegation can be learned as part of the core pretraining objective without separate post-training stages.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that deciding which tokens small language models should learn during pretraining versus delegate via a <CALL> token cannot be based on loss alone, as loss predicts token mismatch but not whether a substitution yields factually or semantically invalid text. It proposes LaCy, which augments loss with grammatical signals (POS, dependencies) from a spaCy parser in Wikipedia-like domains to improve delegation, yielding higher FactScores in cascades with larger models and outperforming Rho/LLM-judge baselines.
Significance. If the empirical results hold, LaCy offers a lightweight, parser-based alternative to pure loss or LLM-judge delegation for SLMs, potentially improving efficiency and factuality without heavy external resources; the reproducible training procedure and falsifiable delegation policy are strengths.
major comments (2)
- [Abstract and method description] The central claim that spaCy grammatical features reliably proxy for factuality or semantic validity of high-loss token continuations is load-bearing but unsupported by direct evidence; no correlation analysis between spaCy tags and human factuality judgments on mismatched tokens is described, leaving open that the parser may capture only surface acceptability.
- [Pretraining procedure] The <CALL> token insertion during pretraining is presented as non-interfering, yet no ablation or analysis shows it preserves the core language modeling objective or ensures the learned policy transfers to normal inference without distribution shift.
minor comments (2)
- [Experiments] Clarify the exact combination rule for loss and spaCy features (e.g., weighting, feature vector construction) and report statistical significance of FactScore gains.
- [Evaluation] Add details on data splits, exact baselines (Rho, LLM-judge), and domain specificity of Wikipedia-like text to allow replication.
Simulated Author's Rebuttal
We thank the referee for their constructive and detailed feedback. We address each major comment point by point below, outlining how we will strengthen the manuscript.
read point-by-point responses
-
Referee: [Abstract and method description] The central claim that spaCy grammatical features reliably proxy for factuality or semantic validity of high-loss token continuations is load-bearing but unsupported by direct evidence; no correlation analysis between spaCy tags and human factuality judgments on mismatched tokens is described, leaving open that the parser may capture only surface acceptability.
Authors: We agree that a direct correlation analysis would provide stronger support for using spaCy signals as a proxy. The current results rely on downstream FactScore improvements in cascades as indirect evidence that the grammatical signals help identify tokens whose substitution affects semantic validity. In the revised manuscript we will add a new subsection with human annotations on a sample of high-loss mismatched tokens, reporting correlation coefficients between spaCy tags (POS, dependencies) and human factuality/semantic validity judgments to address this gap. revision: yes
-
Referee: [Pretraining procedure] The <CALL> token insertion during pretraining is presented as non-interfering, yet no ablation or analysis shows it preserves the core language modeling objective or ensures the learned policy transfers to normal inference without distribution shift.
Authors: This is a fair observation; the manuscript does not contain explicit ablations on this point. We will add an ablation study comparing standard language modeling metrics (perplexity, next-token loss) on held-out Wikipedia data for models trained with and without <CALL> insertion. We will also include inference-time experiments measuring generation quality and delegation frequency under standard prompting (no explicit <CALL> cues) to demonstrate that the learned policy transfers without substantial distribution shift. revision: yes
Circularity Check
No significant circularity in empirical method
full rationale
The paper describes an empirical pretraining procedure (LaCy) that augments per-token loss with lightweight grammatical features extracted by an external spaCy parser to improve delegation decisions via a <CALL> token. No equations, derivations, or fitted parameters are presented that reduce the claimed improvement in FactScore or delegation accuracy to a quantity defined in terms of itself. The central insight—that loss alone is insufficient and must be supplemented by domain-specific signals—is justified by experimental comparisons rather than by construction from the inputs or by load-bearing self-citations. The method remains self-contained against external benchmarks such as human factuality judgments and comparisons to Rho or LLM-judge baselines.
Axiom & Free-Parameter Ledger
read the original abstract
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their parameter size. Especially the capacity of Small Language Models (SLMs) is limited, leading to factually incorrect generations. This problem is often mitigated by giving the SLM access to an outside source: the ability to query a larger model, documents, or a database. Under this setting, we study the fundamental question of \emph{which tokens an SLM can and should learn} during pretraining, versus \emph{which ones it should delegate} via a \texttt{<CALL>} token. We find that this is not simply a question of loss: although the loss is predictive of whether a predicted token mismatches the ground-truth, it is insufficient for identifying which predictions would actually lead to factual or semantically invalid continuations. Some high-loss tokens correspond to \emph{acceptable} alternative continuations of a pretraining document and therefore should not trigger a \texttt{<CALL>}. This suggests that learnability cannot be characterized from loss alone, but requires additional domain-specific signals about the role of a token in the sentence. In Wikipedia-like domains, we show that augmenting the loss signal with lightweight grammatical information from a spaCy parser substantially improves delegation decisions. Based on this insight, we propose LaCy, a novel pretraining method that combines loss with factuality signals to decide which tokens an SLM should learn. Our experiments demonstrate that LaCy models successfully learn which tokens to predict and when to call for help. This results in higher FactScores when generating in a cascade with a bigger model and outperforms Rho or LLM-judge trained SLMs, while being simpler and cheaper.
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
LaCy modifies the standard negative log-likelihood objective by replacing ground-truth targets... CLaCy(xi) = C_spaCy(xi) · I[i is in the top n% of L(B;θ)]
-
IndisputableMonolith/Foundation/AlphaCoordinateFixation.leanJ_uniquely_calibrated_via_higher_derivative unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
augmenting the loss signal with lightweight grammatical information from a spaCy parser substantially improves delegation decisions
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.