Computation and Language · arXiv:2610.35654 v2 · submitted 26 Oct 2026
Improving JEV Entailment Reranking with Span Readouts and Selective Adaptation
The paper says
“JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options.”
more of the abstract
Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. We combine a mean-span readout with selective backbone adaptation. Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The no-FFN adaptation result is the solid half; the mean-span pooling gain is real only under the default NLI template, and the paper deserves review with that condition made explicit. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Appendix B.4 / Table 2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.71):
“The sign changes show why the default-template gain should be interpreted alongside this sensitivity to how the NLI relations are described.”
Pith says: Objection 1 of 3 · major
The pooling increment in the headline result is contingent on a single NLI template, and the paper's own sensitivity analysis does not cover the configuration behind the headline. The +0.91±0.35 pp no-FFN pooling gain (Table 2, b107) is measured only under the default three-option descriptions of Appendix A.1.
more of objection 1
Appendix B.4 (b479) reports that on PIQA, averaging label orders within each description set gives pooling advantages of +6.31, −1.80, and +5.03 pp across the three description sets, with the smallest fixed-template mean at −5.51 pp, and the default set is the most pooling-favorable of the three. That sensitivity analysis was run on the higher-rate all-module checkpoints, and B.4 explicitly states that the final no-FFN pooling configuration is evaluated only on MC11, JevBench, and Kev. The demonstrated cross-template spread (±3.27 pp, with a sign reversal) is therefore several times the size of the claimed +0.91 pp gain, and the template robustness of the headline configuration is untested. The authors acknowledge the sensitivity in Section 4 and B.4, but the abstract's statement that mean-span pooling yields further consistent gains is not supported in this template-independent form. I request either an evaluation of the no-FFN models under the three description sets on at least a subset of MC11 tasks, or a clear scoping of the pooling claim to the default template.
The paper says, at Section 5.4 / Table 4 / A.6
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The one-epoch, lower-learning-rate recipe was selected on the same MC11 benchmark used to report the headline gains. The rate and budget diagnostics in Table 4 and the schedule comparisons in Table 10 are MC11 results, and A.6 states that model development used the primary evaluation suites and fold-development data.
more of objection 2
Consequently, the absolute accuracy levels and the gains relative to the untuned baselines are partly a product of tuning on the evaluation benchmark. The within-recipe paired contrasts (readout, scope) remain matched and are the right primary evidence, and the paper partially mitigates the selection issue by replicating the individual factors on out-of-benchmark sources (Table 5); nevertheless, the ~3.1 pp headline and the consistent-gains wording should be accompanied by an explicit statement of which choices were tuned on MC11, ideally with a recipe-validation run on a held-out configuration.
The paper says, at Table 11 / Appendix D / A.6
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The size of the pooling effect is comparable to the documented run-to-run execution noise. Table 11 reports the repeated-pointer control at +0.17±0.74 pp on MC11, and Appendix D reports that using the repeated pointer as the reference changes the all-module pooling effect from +0.73±1.07 pp to +0.55±1.11 pp; the reference-dependence of the no-FFN pooling effect (+0.91±0.35 pp) is not reported because no repeated no-FFN pointer was run.
more of objection 3
Given that CUDA determinism was disabled (A.6) and dropout trajectories were not saved for the rate experiments, the paper should either state the no-FFN pooling effect relative to this execution-noise floor or report a repeated no-FFN pointer before presenting +0.91 pp as a robust consistent gain.
5 minor objections
The paper says, at Equations (1), (4), A.3
Pith says: minor objection 1 of 5
The summation/product glyphs in Eq. (4) (e.g., '˝𝑗𝑒𝑠𝑗') and in the training loss of A.3 ('−˝𝑖𝑡𝑖 log 𝑝𝑖') are garbled rendering artifacts; the mathematical content is clear from context, but the typesetting must be corrected.
The paper says, at Section 5.1 / Appendix D
Pith says: minor objection 2 of 5
The additivity claim in Section 5.1 ('readout-by-scope interaction is small ... consistent with approximately additive gains') should cite Appendix D's finding that omitting chess reverses the MC11-weighted interaction mean, so that the additivity statement is read with that sensitivity in view.
show 3 more
The paper says, at Table 1 / B.4
Pith says: minor objection 3 of 5
The fixed-checkpoint interface comparison in Table 1 varies not only the scoring interface but also the request wording between direct selection and reranking (e.g., 'Select the solution...' versus 'The correct solution is:...'); a sentence noting this residual confound would strengthen the interface interpretation.
The paper says, at Section 6.2 / Conclusion
Pith says: minor objection 4 of 5
The Qwen3-4B aggregate null (−0.47±0.59 pp for pooling on MC11) is reported prominently, which is commendable; the conclusion would be better balanced if it stated explicitly that the pooling gain is backbone- and template-dependent rather than a general property of entailment reranking.
The paper says, at Table 2
Pith says: minor objection 5 of 5
The Mean-span row of Table 2 runs the two accuracies together ('51.00±0.2153.39±0.24'); the values are recoverable from context, but the table needs a visible column separator.
What would settle it
Re-run the eight no-FFN checkpoints from Table 2 (four fold-seed settings × pointer and mean-span readouts) with description sets 2 and 3 from Appendix B.4 on MC11, retrieving entailment by label identity while keeping the default label order and then the six orders. If the pooling-minus-pointer paired mean stays positive under both alternative sets, the template concern is settled; if it turns negative under set 2 (the set that reverses the comparison on PIQA at -1.80 pp), the headline 53.39% claim is default-template-specific and the design guidance should say so. This requires fresh backbone forward passes because alternative templates change hidden states and stored candidate probabilities cannot be rescored, but it is inference-only on existing checkpoints.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Honest take: the selective-adaptation half is solid; the pooling half is real but template-dependent. I agree with the conditional verdict, and the stress-test note lands.
What is new: the fixed-checkpoint observation that the preferred readout flips with the scoring interface (pointer under direct selection, mean-span pooling under NLI reranking) is a clean result, and I do not see it in the cited work. The no-FFN adaptation result is the stronger contribution: +2.21 pp on MC11 with a 55% parameter reduction, and it replicates on PIQA and CommonsenseQA, under a higher learning rate, and under every leave-one-source-out omission. That is real evidence.
The methodology is better than typical: four fold-seed settings, within-setting paired differences, descriptive SEs, task-level decomposition, budget controls, and unusually candid limitations. The mean-pooling identity in Eq. 6 is straightforward linear algebra, but the way they use it to explain the training-gradient spread is fine.
The weak point is the pooling increment. Appendix B.4 shows the readout comparison is wording-dependent: averaging label orders within description sets gives +6.31, −1.80, and +5.03 pp, and the smallest fixed-template mean is −5.51 pp. The MC11 claim uses a single default template, and the sensitivity study was run on the higher-rate all-module checkpoints, not on the no-FFN models that produce the headline +0.91 pp. So the abstract's "further consistent gains" overstates the evidence. The honest claim is: consistent under the default wording, not across wordings. The authors do flag the sign sensitivity, so this is a condition to address, not a misrepresentation.
Two more soft spots, in proportion. The one-epoch lower-rate recipe was selected on MC11, so the headline 3.1 pp is partly a tuned operating point; the no-FFN direction survives off-benchmark tests, while the pooling increment may not. And anonymous reproduction is blocked by unavailable framework forks and nondeterministic CUDA execution, though the saved predictions for rescoring and the explicit accounting in A.6 are to their credit.
Who this is for: people building reranking interfaces on typed decision models, and anyone studying readout and template sensitivity. It deserves a serious referee, with a request for either a template-robust evaluation of the final configuration or a revised claim, plus independent data for recipe selection.
The strongest objection
The central claim is a joint ~3.1 pp gain on NeoHorse-1-4B from two increments: removing FFN adapters (+2.21±0.66 pp) and mean-span pooling (+0.91±0.35 pp), both positive in all four fold-seed settings (Table 2; b24, b107). The no-FFN increment is the larger and more robust half: it replicates under both readouts, on PIQA (+5.21 pp) and CommonsenseQA (+9.56 pp), at the higher learning rate (+3.67 pp, Table 10), and under every leave-one-source-out omission (Appendix D). The pooling increment is the load-bearing weak point. Every headline comparison scores candidates with one fixed NLI template, and Appendix B.4 shows the readout comparison is wording-dependent: on the same PIQA checkpoints, averaging label orders within each description set gives pooling advantages of +6.31, -1.80, and +5.03 pp, and the smallest fixed-template mean is -5.51 pp (b479). The default template is the most pooling-favorable of the three. The authors acknowledge this sensitivity, but the sensitivity study was run on the higher-rate all-module checkpoints, never on the no-FFN MC11 models that produce the headline +0.91 pp, so whether the claimed gain survives alternative wordings is untested. A 0.91 pp effect whose demonstrated cross-template spread (±3.27 pp; sign reversal between description sets) is several times its size cannot underwrite the template-independent statement that mean-span pooling yields further consistent gains. The concern is a condition, not a refutation: pooling helps on average over the tested templates, and the selective-adaptation and parameter-reduction contribution stands on its own.
This pass agrees with the reader's verdict and with the specific weakest assumption. The paper's central claim is the joint ~3.1 pp MC11 gain from removing FFN adapters and switching to mean-span pooling on NeoHorse-1-4B (Table 2; b24, b107). For that claim to hold as stated, both increments must be real at the chosen operating point, and the pooling increment must not be an artifact of the NLI option wording. The no-FFN increment survives scrutiny: it is consistent across both readouts, replicates on PIQA and CommonsenseQA, holds under the higher-rate and head-first recipes, and remains positive under every LOSO omission; the rank-28 budget control showing 52.40% versus 52.49% indicates much of the benefit is capacity reduction rather than FFN-specific harm, which matches the paper's own reducing-adaptation-capacity framing. The pooling increment is where the argument is least secure. Appendix B.4 (b479) is an in-scope, author-supplied limitation that directly demonstrates the readout comparison flipping sign across natural description sets on the same backbone, with the default template being the most pooling-favorable of the three. The authors are explicit about the sensitivity, but the sensitivity evidence covers only the higher-rate all-module PIQA checkpoints, leaving the headline no-FFN MC11 comparison untested under alternative wordings. That gap is the load-bearing concern, and the proposed inference-only test on the existing eight no-FFN checkpoints would settle it. Because the concern is acknowledged, partial, and conditional rather than disqualifying, the reader's CONDITIONAL verdict stands unchanged; the claim should be reported as default-template-conditional until the test is run.
The referee's report
If the results hold, the paper delivers a clean and practically useful empirical finding: for this JEV reranking pipeline, omitting FFN adapters improves accuracy while cutting trainable parameters by 55.2%, and the no-FFN increment replicates across readouts, tasks, learning rates, and source-omission analyses. The matched four-setting design with paired differences and descriptive SEs is a genuine strength, as are the exact analytic identity in Eq. (6), the fixed-checkpoint interface comparison in Table 1, and the availability of saved predictions for 72 checkpoints. The paper's transparency about its own failure modes — template sensitivity, nondeterministic training, and unresolved reproducibility assets — is exemplary and should be credited.
Pith's reading of the paper (a machine's summary, not the authors')
This paper asks whether two design choices tuned for single-request multiple-choice scoring still help when the same model is used as a cross-request entailment reranker, where each candidate answer is scored in its own natural-language-inference (NLI) request against fixed relation options. It argues that the designs do not transfer unchanged: averaging the contextualized hidden states across each NLI option's label-description span (mean-span pooling) beats the usual option-ending pointer under reranking, and restricting low-rank adapters to attention and gated DeltaNet projections while removing feed-forward adapters improves accuracy while cutting trainable parameters by more than half. On a 4-billion-parameter backbone and an eleven-configuration benchmark (MC11), the two changes together move accuracy from 50.28% to 53.39%, with improvement in every fold-seed setting. If the paper is right, a reranking scorer can be both cheaper to train and more accurate, provided the model is not also needed for its original direct-selection use, where the same choices cost a few points.
What carries the argument. Two mechanisms carry the argument. Mean-span pooling replaces the readout state at each option's closing delimiter with the average $\bar{h}_i = \frac{1}{|S_i|}\sum_{j\in S_i} h_j$ of the contextualized states over the option's interior label and description tokens, excluding both delimiters. Because the shared bilinear head score $s_i = (W_q h_d + b_q)^\top (W_k \bar{h}_i + b_k)/\sqrt{d_{\text{head}}}$ is affine in the option state, the pooled score equals the exact average of the pre-softmax token-level scores, and the direct gradient $\partial s_i^{\text{pool}}/\partial h_j = W_k^\top q/(|S_i|\sqrt{d_{\text{head}}})$ reaches every interior token, at zero added head parameters. Selective adaptation keeps rank-64 LoRA adapters on the attention projections (q/k/v/o) and the gated DeltaNet projections while removing the three feed-forward LoRA targets (gate/up/down), leaving the pretrained FFN weights in the forward pass; this removes 72.35M of the 131.17M trainable parameters. The two choices are crossed in a 2$\times$2 design under a tuned one-epoch, lower-learning-rate recipe, with each configuration trained afresh in four matched fold-seed settings.
Core claim. The central claim is that the scoring interface decides which option readout is better, and that reducing adaptation capacity helps rather than hurts cross-request reranking. On NeoHorse-1-4B under a matched one-epoch, lower-learning-rate recipe, the all-module pointer reference scores 50.28% on MC11; selective adaptation (no-FFN) raises this to 52.49% while cutting trainable parameters from 131.17M to 58.82M, and adding mean-span pooling reaches 53.39% at the same parameter count. Both steps are positive in all four fold-seed settings, with a small readout-by-scope interaction that the paper reads as approximately additive gains. The interface effect is shown directly: with checkpoints and questions held fixed on PIQA and SocialIQA, the pointer leads on average under direct selection while pooling leads under entailment reranking, with the difference on PIQA moving from $-0.12$ to $+8.07$ pp. The paper also reports the limits of its own claim: pooling's advantage over the pointer depends on the wording of the NLI relation options (Appendix B.4 gives description-set means of $+6.31$, $-1.80$, and $+5.03$ pp), the aggregate readout gain does not survive on Qwen3-4B, and the combined model loses 3.32 and 3.25 pp on the JevBench and Kev direct-decision suites.
The premise it rests on. All headline numbers rest on one fixed wording of the three NLI relation options on a single backbone, and the paper's own template-sensitivity test shows the pooling advantage over the pointer can fall to $-1.80$ pp (averaged over label orders) and to $-5.51$ pp for one fixed template under alternative descriptions, so the claimed readout gain depends on the default template being representative of how practitioners phrase entailment options.
If the paper is right (Pith's inferences, not the authors')
A practitioner building an entailment-reranking scorer on a 4B-class backbone can expect a roughly 3.1 pp gain on the eleven-configuration MC11 average while training 55.2% fewer parameters, compared with the all-module pointer recipe.
Readout choice should be matched to the deployment interface: the same fixed checkpoints show the pointer ahead under direct selection and pooling ahead under reranking.
The benefit is not task-uniform: HellaSwag and synthetic chess carry most of the net gain, while GSM8K and GPQA decline under the final model, so adopting the design means accepting task-level trade-offs.
Capacity location matters more than total budget: spreading the same ~58M parameters across all projections at rank 28 gives about the same accuracy as the no-FFN rank-64 allocation, so the no-FFN choice is not merely a parameter-count effect.
The reranking improvements do not transfer back to direct selection: the final combined model scores 73.79% on JevBench and 78.12% on Kev, 3.32 and 3.25 pp below the all-module pointer.
My reading: the template sensitivity the paper itself reports suggests the pooling-versus-pointer ranking is entangled with how the entailment labels are phrased, so treating the relation descriptions as a tunable hyperparameter and measuring MC11 over several phrasings would be a direct next test.
My reading: because the pooling step is negative overall on Qwen3-4B yet positive on NeoHorse, the readout gain may depend on the attention-plus-DeltaNet mixture; ablating DeltaNet adapters on NeoHorse while keeping attention adapters would isolate which projection family makes pooling pay off.
My reading: head-first staged adaptation combined with mean-span pooling remains untested, and since head-first plus no-FFN already reaches 53.54% at a higher learning rate, combining all three is a natural configuration to try, though the negative scope-by-schedule interaction the paper reports warns that gains may not add.
My reading: the one-epoch, low-rate recipe is itself load-bearing for every reported contrast, which suggests that readout and adapter-scope evaluations should report recipe sensitivity, since standard multi-epoch fine-tuning could reverse the observed ordering.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options. Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. We combine a mean-span readout with selective backbone adaptation. Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 1.00
“On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.94 · randomized assignment 0.02 · states quantitative result 0.89 · compares with prior results 0.94 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options. | background 0.99 | 0.08 | 0.02 | asserted 1.00 |
| Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. | background 0.99 | 0.08 | 0.03 | asserted 1.00 |
| We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. | gap 0.48 | 0.30 | 0.03 | unclear 0.55 |
| We combine a mean-span readout with selective backbone adaptation. | method 0.69 | 0.91 | 0.08 | asserted 0.94 |
| Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. | method 0.99 | 0.79 | 0.04 | asserted 1.00 |
| Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. | method 0.99 | 0.71 | 0.04 | asserted 0.95 |
| On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count. | result 1.00 | 0.95 | 0.79 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.89 · evidence from observation or experiment 0.95
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 5,557 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment