A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints

Bar Weiner; David Konopnicki; Haggai Roitman; Or Rivlin; Shai Erera; Yosi Mass

REVIEW 3 major objections 4 minor 18 references

A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning BERT with a 256-token input window yields the best passage re-ranking for non-factoid questions.

desk verdict The BERT-for-non-factoid-re-ranking results are probably sound, but the '256 tokens is optimal' claim is selected on the test set and needs significance testing. read the letter →

arxiv 1908.06780 v1 pith:FSYN6HVT submitted 2019-08-19 cs.IR cs.CL

Yosi Mass , Haggai Roitman , Shai Erera , Or Rivlin , Bar Weiner , David Konopnicki This is my paper · ORCID

classification cs.IRcs.CL

keywords BERTpassagere-rankingnon-factoidquestionansweringlearningtoranksequencelengthsegmentationpoint-wiserankingpair-wise

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether BERT's fixed input-length limit can be navigated for non-factoid question answering, where answers are whole passages rather than short factoids. By fine-tuning BERT in three learning-to-rank setups—point-wise, pair-wise triplet, and pair-wise cross-entropy—on three standard datasets, it finds that truncating each (query, passage) pair to 256 tokens gives the best re-ranking accuracy. At that window, its BERT rankers improve over the best previous deep-learning method by 31% to 120% depending on dataset and metric. The paper also shows that passages longer than BERT's 512-token cap can be re-ranked by splitting them into chunks and aggregating the chunk-level BERT scores with an attention mechanism, at a moderate cost in accuracy.

What carries the argument

The central mechanism is BERT's pooled [CLS] representation of a (query, passage) pair, produced by the last Transformer layer and fine-tuned on labeled relevance pairs. Three learning-to-rank heads are attached to that representation: a point-wise cross-entropy classifier, a triplet network with hinge loss over a learned scoring vector, and a pair-wise softmax cross-entropy variant. Passage length is controlled through the sequence-length input window, and for long passages the paper divides the passage into equal chunks, computes BERT(q, chunk), and combines the chunk representations with an attention mechanism before scoring.

What would settle it

Train the same BERTlets model on WebAP with an alternative negative-sampling rule, such as uniformly sampling passages from the collection or sampling negatives from the BM25 top list without a normal-length constraint, and compare P@1. If the results change materially, the normal-length sampling assumption is load-bearing; if not, it is not. A second check: measure the actual length distribution of the negatives produced by the paper's sampling procedure and verify that it matches the assumed normal distribution.

Watch

Extended reading notes

Core claim

The paper claims that BERT can be fine-tuned for non-factoid passage re-ranking with three learning-to-rank heads—a point-wise classifier, a pair-wise triplet network, and a pair-wise cross-entropy model—and that the best results occur when each (query, passage) pair is truncated to 256 tokens. At that length, the BERT-based rankers outperform the best prior deep-learning baseline by 31% on MAP for WikiPassageQA and by 120% on P@1 for nfL6. The authors further report that segmenting longer passages into equal chunks and combining chunk-level BERT representations with an attention mechanism handles passages that exceed BERT's 512-token limit, with only a moderate drop in accuracy. The central empirical claim is therefore that a mid-sized input window, not the longest possible window, is optimal for this task.

Load-bearing premise

The experiments on the WebAP dataset assume that irrelevant passages used as negative training examples have lengths drawn from a normal distribution whose mean and standard deviation are taken from the relevant passages; if this sampling assumption is wrong, the WebAP training data is mis-specified and the reported gains may not generalize.

Editorial extensions

If this is right

Practitioners re-ranking non-factoid passages should set BERT's input window to 256 tokens, since both shorter windows (64, 128) and a longer one (384) reduce P@1 or MAP on all three tested datasets.
Passages longer than 512 tokens can still be handled by splitting them into equal chunks, scoring each (query, chunk) with BERT, and combining scores via attention; this costs only a few points of accuracy.
The choice between point-wise and pair-wise learning-to-rank has little effect on final ranking quality, so the simpler point-wise classifier can be used without expecting a significant drop.
The improvement over prior deep-learning baselines is large enough to suggest that the ranking head is not the main bottleneck; the input representation carries most of the signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: The 256-token optimum is likely tied to the average passage lengths in these datasets; on collections with systematically longer answers the optimum could shift, a dependence the paper does not examine.
Inference: The chunking-plus-attention approach only combines independent (query, chunk) scores; a testable extension would feed chunk representations through another attention layer to let BERT model cross-chunk dependencies explicitly.
Inference: Because WebAP contains only 80 queries, its reported P@1 differences are sensitive to how negative passages are sampled; re-running with alternative negative distributions would show how much of the WebAP gain depends on the paper's normal-length sampling assumption.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The practical take: the paper's core message—fine-tuned BERT helps a lot for non-factoid passage re-ranking—is probably right, but the specific '256 tokens is optimal' headline is not secured. They picked 256 off the test-set curves in Figure 2, then reused that same test data to report the headline numbers. That is selection on the test set, and with only 80 WebAP queries the 128/256/384 differences could easily be noise. No significance tests anywhere.

What's actually new: the BERTlets triplet variant (score as dot product of CLS representation with a learned vector, hinge loss), a chunk-and-attention method for long passages, and a systematic passage-length sweep on three datasets. Nogueira & Cho already had BERT re-ranking; this paper adds the length analysis and the triplet setup. The gains over the prior DL baselines are large and consistent across datasets, so I'm confident the general conclusion survives. They also report segmentation results showing only moderate degradation, which is useful for practitioners.

Soft spots, in order of severity. First, the SeqLen=256 claim: selected on test data, no validation split, and no significance testing. This is load-bearing for the paper's central and novel finding, so it needs fixing—either a proper dev-based selection or at least a paired test across folds. Second, the 'DL' baseline is never named. They say 'previously best known DL method' but don't say which model, so the 31–120% improvement figures can't be independently checked. That should be trivial to fix. Third, the WebAP negative sampling assumes passage lengths are normally distributed with mean and std from the relevant passages. They disclose this, but it's an arbitrary assumption and could bias the WebAP training data; a sensitivity check would help. Fourth, no code or data release is mentioned, though the datasets are public.

The citation pattern looks fine; they build on the relevant BERT-for-retrieval work and credit it. The writing is clear and honest; they explicitly note the omitted dev set and the sampling assumption, which I appreciate.

Who this is for: anyone working on non-factoid QA or long-passage retrieval with transformers, especially people deciding on SeqLen or whether to chunk. It's a solid empirical study, not a breakthrough.

Recommendation: send to peer review. The flaws are real but repairable, and the paper deserves referee time. With a proper validation split, significance tests, and a named baseline, it would be a useful contribution.

Referee Report

3 major / 4 minor

Summary. The paper studies the use of fine-tuned BERT for non-factoid question-answering, specifically the passage re-ranking task, under the constraint that BERT can process only a limited number of tokens per (query, passage) pair. The authors propose and compare three learning-to-rank methods: a point-wise classifier (BERT[PW]) and two pair-wise methods (BERTlets and BERT[CE]). Using BERT-Base, they fine-tune on three datasets (nfL6, WebAP, WikiPassageQA) and report improvements in P@1 and MAP over BM25 and earlier deep-learning baselines. A central analysis varies the BERT sequence length (SeqLen) from 64 to 384 tokens and concludes that SeqLen=256 gives the best results. The paper also investigates splitting long passages into chunks and aggregating chunk-level representations with an attention mechanism, finding moderate degradation compared with using the full passage at SeqLen=256.

Significance. If the main findings are properly validated, the paper would provide practical guidance for applying BERT to non-factoid passage re-ranking, particularly the recommendation that a sequence length of 256 tokens is the best operating point. The comparison of point-wise versus pair-wise fine-tuning, and the chunking strategy for long passages, are useful contributions to a practitioner-oriented literature. However, the central length recommendation is currently based on selecting the best value on test data, and the reported 'significant' improvements are not supported by statistical tests. The explicit statement of the WebAP negative-sampling assumption in Section 4.1 is a sign of transparency, though the assumption itself needs scrutiny. Because the empirical claims are not yet fully substantiated, the paper is not publishable in its current form; the issues are fixable and the underlying approach is reasonable.

major comments (3)

[§4.3.1, Figure 2] The central claim that SeqLen=256 is the optimal passage length is derived by evaluating several SeqLen values on the same test folds that are later used to report the headline results in Figure 1. Since Section 4.2 states that the development set was omitted, the sweep in Section 4.3.1 is effectively selecting the model on the test data. This makes the 'best P@1 and MAP at 256 tokens' a test-set maximum rather than a validated general finding. Please report the result with a validation split (e.g., using the existing WikiPassageQA development set or a held-out fold) or provide per-fold results with paired significance tests across the length values. Without this, the paper's primary conclusion about mid-sized sequence length is not secured.
[§4.1, WebAP dataset description] For WebAP, the paper states: 'we assumed passage lengths are normally distributed with mean and standard deviation estimated using the relevant passages.' This assumption directly defines the negative training examples for one of the three datasets, and it is stated without empirical justification. If the true distribution of negative passage lengths differs from the fitted normal, the training signal for WebAP is mis-specified and the reported gains on this dataset may not generalize to other settings. Please validate the assumption against the data (e.g., a histogram or a normality test) and include a sensitivity analysis, such as re-running the experiments with differently sampled negatives, or clearly caveat the results as contingent on this assumption.
[§4.3, Figure 1] The paper asserts that the BERT-based methods 'significantly outperform' the BM25 and deep-learning baselines, but no significance tests or confidence intervals are reported. With only 80 WebAP queries and a modest WikiPassageQA test split, the observed improvements, especially between the BERT variants, could be within noise. Please provide per-fold or per-query results and use a paired statistical test (e.g., Wilcoxon signed-rank or bootstrap) for the P@1 and MAP comparisons. This is particularly important because the selection of SeqLen=256 in Section 4.3.1 further inflates the apparent performance of the reported configuration.

minor comments (4)

[Throughout] The manuscript contains numerous typographical artifacts, such as 'Pa ssage', 'Y osi Mass', 'V aswani et al.', and 'BERT(q,p) is the BERT pooled representation of the CLS token of the last layer'. Please run spell-check and copy-editing to clean up these issues.
[Table 1] The number of nfL6 queries is given as 87,361 in Section 4.1 but as 87,362 in Table 1. Please reconcile these numbers and confirm the correct dataset statistics.
[§3, BERT[CE] description] The description of BERT[CE] says it uses the negative log-likelihood of the positive example instead of the Hinge loss, but the exact normalization (whether over a minibatch or a single triplet) is not specified. Please clarify the loss formulation so that the method is reproducible.
[§4.3.2] The attention size of 192 is introduced without any justification or sensitivity analysis. Since this hyperparameter affects the chunk aggregation, please state whether it was chosen by prior tuning or report results for a couple of values.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical evaluations against external baselines; its length-optimality finding is a test-set observation, not a derivation from its own inputs.

full rationale

The paper's claims are empirical: fine-tuned BERT point-wise and pair-wise rankers are evaluated with held-out test folds (5-fold for nfL6 and WebAP; fixed train/dev/test split for WikiPassageQA) and compared to external baselines (BM25 and previously published DL methods). The BERT architecture, [CLS] representation, and fine-tuning recipe are taken from Devlin et al. (2018), an external source, and the pairwise losses are implemented from Liu et al. (2019), also external. There are no self-citations by the present authors, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The SeqLen=256 choice is an empirical observation from Figure 2, not a fitted parameter disguised as a prediction; the paper states it 'was found to be the optimal size' in the same section where the graphs are presented. Selecting a hyperparameter on the test folds (the paper says 'Similar to Devlin et al., we used a fixed number of three train epochs, hence we omitted the development set') is a statistical validity concern that could make the reported margin over 128/384 tokens less reliable, but it is model selection on observed data, not a reduction of a predicted quantity to its own input. Similarly, the WebAP negative-passage sampling assumption affects training-data construction and generalizability, but it is not used as evidence for the headline results. No Eq. X = Eq. Y by construction, no fitted parameter renamed as a prediction, and no load-bearing self-citation are present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The experimental claims rest on standard BERT fine-tuning plus a few dataset-specific choices: the in-sample selection of SeqLen=256, the WebAP negative-sampling normality assumption, and hand-set hyperparameters. No new entities are postulated.

free parameters (3)

SeqLen = 256 tokens
The paper found 256 to be optimal by comparing values 64, 128, 256, and 384 on the test sets; this selection is in-sample and not held out.
margin m = 0.2
Hinge loss margin hyperparameter for the BERTlets triplet loss, chosen by hand.
attention size = 192
Used for combining chunk representations in the segmentation experiments, chosen by hand.

assumptions (3)

domain assumption WebAP negative passage lengths follow a normal distribution with parameters estimated from relevant passages.
Stated in Section 4.1 as an assumption because the original paper lacked details.
domain assumption Adding a correct passage at position k of the BM25 list is a valid evaluation protocol.
The paper follows Cohen and Croft 2018 and assumes this protocol does not bias the re-ranking evaluation.
domain assumption The [CLS] pooled representation of BERT captures enough query-passage interaction for scoring.
The scoring methods in Section 3 rely on the BERT [CLS] representation as the sole passage-query embedding.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints." pith.science (2026). https://pith.science/paper/FSYN6HVT

@misc{pith2026190806780,
  author       = {Pith},
  title        = {Pith review of: A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSYN6HVT}},
  note         = {Machine review of arXiv:1908.06780}
}

read the original abstract

We study the use of BERT for non-factoid question-answering, focusing on the passage re-ranking task under varying passage lengths. To this end, we explore the fine-tuning of BERT in different learning-to-rank setups, comprising both point-wise and pair-wise methods, resulting in substantial improvements over the state-of-the-art. We then analyze the effectiveness of BERT for different passage lengths and suggest how to cope with large passages.

Figures

Figures reproduced from arXiv: 1908.06780 by the authors.

**Figure 2.** Effect of SeqLen (n) on BERTlets 4.3.2 Effect of passage segmentation To overcome the limitations of BERT on passagelength, we further segmented each passage into equal number of chunks. Given a query q and passage p, we derived BERT(q, c), for chunks c ∈ p, and combined their representation through an attention mechanism (Yang et al., 2016). We tried to break to two and three chunks and pass them with SeqLen= 12… view at source ↗

**Figure 1.** Comparison between methods (P@1 and MAP). SeqLen=256 4.3.1 Effect of Passage Length As was mentioned above, BERT imposes a limit of maximum 512 tokens10 (SeqLen) in each pair when using a GPU [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Bruce Croft

Daniel Cohen and W. Bruce Croft. 2018. A hybrid embedding approach to noisy answer passage retrieval. In 40th European Conference on IR Research, ECIR 2018, pages 127--140

work page 2018
[4]

Bruce Croft

Daniel Cohen, Liu Yang, and W. Bruce Croft. 2018. Wikipassageqa: A benchmark collection for research on non-factoid answer passage retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, pages 1165--1168

work page 2018
[5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In CoRR, abs/1810.04805

arXiv 2018
[6]

Mathias G\' e ry and Christine Largeron. 2012. Bm25t: A bm25 extension for focused information retrieval. Knowl. Inf. Syst., 32(1):217--241

work page 2012
[7]

Elad Hoffer and Nir Ailon. 2018. Deep metric learning using triplet network. CoRR, abs/1412.6622

work page Pith review arXiv 2018
[8]

Hang Li. 2011. A short introduction to learning to rank. IEICE Transactions, 94-D:1854--1862

work page 2011

Show all 18 references

[9]

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. CoRR, abs/1901.11504

2019 arXiv
[10]

Bhaskar Mitra and Nick Craswell. 2019. An updated duet model for passage re-ranking. CoRR, abs/1903.07666

2019 arXiv
[11]

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. CoRR, abs/1901.04085

2019 arXiv
[12]

Bruce Croft

Harshith Padigela, Hamed Zamani, and W. Bruce Croft. 2019. Investigating the successes and failures of bert for passage re-ranking. CoRR, abs/1905.01758

2019 arXiv
[13]

Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the behaviors of bert in ranking. CoRR, abs/1904.07531

2019 arXiv
[14]

Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. 2004. Are loss functions all the same? Neural Comput., 16(5):1063--1076

2004
[15]

Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Lstm-based deep learning models for non-factoid answer selection. CoRR, abs/1511.04108

2015 arXiv
[16]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000--6010

2017
[17]

Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. CoRR, abs/1902.01718

2019 arXiv
[18]

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT, pages 1480--1489

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Bruce Croft

Daniel Cohen and W. Bruce Croft. 2018. A hybrid embedding approach to noisy answer passage retrieval. In 40th European Conference on IR Research, ECIR 2018, pages 127--140

work page 2018

[4] [4]

Bruce Croft

Daniel Cohen, Liu Yang, and W. Bruce Croft. 2018. Wikipassageqa: A benchmark collection for research on non-factoid answer passage retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, pages 1165--1168

work page 2018

[5] [5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In CoRR, abs/1810.04805

arXiv 2018

[6] [6]

Mathias G\' e ry and Christine Largeron. 2012. Bm25t: A bm25 extension for focused information retrieval. Knowl. Inf. Syst., 32(1):217--241

work page 2012

[7] [7]

Elad Hoffer and Nir Ailon. 2018. Deep metric learning using triplet network. CoRR, abs/1412.6622

work page Pith review arXiv 2018

[8] [8]

Hang Li. 2011. A short introduction to learning to rank. IEICE Transactions, 94-D:1854--1862

work page 2011