{"id":"efd485a5-4b87-47a2-9c59-b044ef9db43d","arxiv_id":"2411.16985","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Smaller language models can generalize to unseen compositional questions when trained and evaluated with retrieval-augmented contexts, and combining Wikipedia retrieval with LLM-generated rationales improves accuracy.","lead":"This thesis shows that smaller language models, around 400 million to 1 billion parameters, can answer novel multi-step questions by reasoning over retrieved text rather than by memorizing training examples. It introduces retrieval-augmented training data and a rationale-ranking method that combine Wikipedia retrieval with explanations from bigger models, improving offline question answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Anti-memorisation claim rests on hand-calibrated T=60 and a most-similar-only answer filter; if unmemorisable subsets still contain memorisable pairs, reported gains could be inflated.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern in the thesis: the Unmemorisable subset construction in Section 4.2.3. The anti-memorisation conclusion in Chapter 4 is the foundation for the first stated contribution, and it rests on the T=60 threshold and the most-similar-only answer-overlap filter. My reading confirms the reader's concern and adds a sharper point: the thesis explicitly limits the answer-overlap check to the single most similar training sample, leaving open the possibility that a different, still-similar training sample provides the memorised answer. A concrete test can settle this by broadening the filter to all training samples and observing whether the reported gains persist. If they do not, the central claim of generalisation beyond memorisation is weakened, though other contributions (RATD, RR, baselines) have independent support from external benchmarks and finetuned results, so the overall verdict remains conditional rather than rejected. The reader's CONDITIONAL verdict is appropriate; my analysis does not change that assessment.","tokens_in":51040,"tokens_out":3345,"duration_ms":33038,"concrete_test":"Recompute the Unmemorisable subsets of DROP and ROPES (Table 4.2) checking answer-term overlap against all UQA+TDND training samples, not just the single most similar sample; then re-run the UQA versus UQA+TDND comparison on the resulting stricter subset. If the 9.0% and 25.7% gains shrink below statistical significance, the anti-memorisation conclusion is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the 440M BART model generalises beyond memorisation (Section 1.3, Chapter 4) depends on the Unmemorisable subset being genuinely unanswerable from memory. The subset is defined by a similarity threshold T=60 (Equation 4.1) that was hand-calibrated by inspecting the ten most similar eval-train pairs per dataset, and by removing only samples whose answer has no word overlap with the single most similar training sample (Section 4.2.3, Table 4.2). Two gaps follow. First, sentence embeddings from stsb-roberta-large can miss paraphrased or discontinuous overlaps, so a memorisable pair could score below T=60. Second, the answer-overlap filter only checks the most similar training sample; an evaluation sample whose answer appears in a different, less-similar training sample remains in the Unmemorisable subset, and the model could have memorised that other sample. The thesis itself acknowledges this ('we are unable to eliminate evaluation samples that have answer overlap with any training sample'). If even a modest fraction of the 652 DROP or 197 ROPES 'unmemorisable' samples are actually memorisable, the reported 9.0% and 25.7% improvements (Table 4.5) could be inflated by memorisation, and the 'beyond simple memorisation' contribution would be overstated. The 9.6% DROP-CS gain is already non-significant, suggesting sample-size sensitivity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD thesis studies whether a 440M-parameter BART model, trained with multitask question-answering datasets, can answer compositional questions that are unseen during training when provided with retrieved or generated context. Chapter 4 proposes a semantic-similarity method to partition evaluation samples into memorisable and unmemorisable subsets, and an intervention experiment (UQA vs UQA+TDND) that claims the model generalises beyond memorisation on DROP and ROPES. Chapter 5 introduces the Iterator multi-hop dense retrieval system and retrieval-augmented training datasets (RATD), reporting significant improvements over baselines on several unseen datasets. Chapter 6 adds LLM-generated rationales as a second knowledge source and proposes Rationale Ranking (RR) to score and combine contexts; the thesis claims significant gains from combining sources and from the RR method, and shows that the small model can outperform direct LLM prompting given the same context.","tokens_in":51413,"tokens_out":9956,"duration_ms":87812,"significance":"If the findings hold, the practical contribution is substantial: it would demonstrate that a locally deployable 440M model can approach or exceed much larger models on unseen multi-hop and commonsense QA when given the same contextual information. The thesis is strong in empirical breadth: multiple unseen datasets, paired bootstrap and Nemenyi significance tests, ablations for training-regime components, and public code/data links. The external evaluation of the RR model on TruthfulQA (Table 6.3) is a valuable sanity check that the model learns truthfulness, not just relevance. However, two methodological issues (the calibration of the memorisation filter in Chapter 4 and the selection of the RR threshold on the evaluation data in Chapter 6) currently limit the strength of the two headline contributions.","major_comments":[{"comment":"The anti-memorisation claim (Contribution 1, Section 1.3) depends on the 'Unmemorisable' subsets being genuinely unanswerable from memory. The subset is defined by a hand-calibrated similarity threshold T=60 and by removing only samples whose answer has no word overlap with the single most similar training sample. The thesis explicitly acknowledges in Section 4.2.3 that it cannot eliminate evaluation samples whose answer overlaps with any training sample. As a result, a DROP or ROPES evaluation sample whose answer appears in a second-most-similar training sample remains in the subset, and the model could have memorised that sample. In addition, the stsb-roberta-large embeddings used in Eq. (4.1) may miss paraphrased or discontinuous overlaps, so a memorisable pair could score below T=60. If even a modest fraction of the 652 DROP or 197 ROPES 'Unmemorisable' samples are memorisable, the 9.0% and 25.7% gains in Table 4.5 would be inflated, and the conclusion in Section 4.4 ('the improvement is not attributable to memorisation') would be unsupported. To make the claim load-bearing, the filter should be applied against all training samples (e.g. via a nearest-neighbour scan), and the sensitivity of the gains to T should be reported.","section":"§4.2.3, Table 4.2, Table 4.5"},{"comment":"The 'Generally best RR combo' is selected per Reasoning Model as the combination method with the highest unweighted macro-average over the unseen evaluation datasets (Section 6.3.2, text immediately before Table 6.5). These same datasets are then used to compute the +RR -RATD versus -RR -RATD difference in Table 6.6 (45.5 vs 42.7) and to run the Nemenyi significance tests. This is selection on the evaluation data: the threshold t has been chosen to maximise the very quantity being tested, so the reported improvement is an optimistically biased estimate of performance on arbitrary unseen questions. While the thesis correctly states that this method is not usable for a truly unseen question, it nevertheless uses this result as evidence for Contribution 5 ('the RR method significantly outperforms...'). The authors should either (a) choose t on a held-out validation fold and report the resulting test performance, or (b) restrict the RR-value claim to threshold-free evidence such as the Naïve Concatenation result, which also shows a significant combined-context benefit (47.2 vs 42.7).","section":"§6.3.2, Table 6.6"},{"comment":"The Chapter 5 conclusion states: 'training on RATD datasets improves performance on all unseen evaluation datasets with retrieved contexts'. This is internally contradicted by Table 5.9, where MusiqueR results significantly degrade from 24.3 F1 (Base) to 22.2 F1 (Base+RATD), while ARC-DA improves. The post-hoc ablative model (excluding Musique RATD, or using only unique-label Musique samples) recovers and improves performance, which is informative, but it does not make the original claim true. The conclusion should be amended to state that RATD improves performance on all retrieved-context datasets except Musique, where the training-distribution bias causes degradation.","section":"§5.4, Table 5.9"}],"minor_comments":[{"comment":"'accessable' should be 'accessible' in the sentence 'All versions of our evaluation (and training) datasets are accessable at github.com/timhartill/unseen_questions.'","section":"§2.3"},{"comment":"The method is called 'Rational Ranking' in the first bullet point of the introduction but 'Rationale Ranking' everywhere else; the terminology should be consistent.","section":"§6.1"},{"comment":"The Base model's SQAR score of 48.4 (below random) is accompanied by a footnote that prepending 'Yes or no -' improves it to 54.9. This prompt adjustment should be presented as the primary result for transparency, since reporting the below-random score is likely to be misinterpreted.","section":"§5.3.2.1, Table 5.3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the two main issues are fixable in revision. I would not recommend reject; the empirical contribution is real. The thesis format is appropriate for arXiv; the TMLR publication of Chapter 5 is disclosed. Please ask the authors to either add a held-out validation for threshold selection in Chapter 6 or remove the claim that RR significantly outperforms baselines; and to strengthen the memorisation filter in Chapter 4 or soften the conclusion. The Section 5.4 overstatement is a simple fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this thesis delivers a genuinely useful empirical package for resource-constrained question answering, but the \"beyond simple memorisation\" headline should be read with caution. The core retrieval-and-rationale results in Chapters 5 and 6 stand up reasonably well; the memorisation analysis in Chapter 4 is the softest part, and the stress-test note lands.\n\nWhat's new: the Iterator with Evidence Set Scoring extends MDR/Baleen to arbitrary hop counts; the RATD training datasets are a practical way to teach a small model to reason over noisy partial evidence; and the Rationale Ranking model adds a truthfulness filter on top of relevance scoring. The evaluation is unusually broad: multiple unseen datasets, paired bootstrap tests, Nemenyi tests, ablations, and the code/data are public. That is real work, and it shows.\n\nThe soft spot is exactly where the reader and stress-test point. The unmemorisable subsets are built on a T=60 semantic-similarity threshold that was hand-calibrated by inspecting the ten most similar pairs per dataset, and the answer-overlap filter only removes samples whose answer overlaps with the single most similar training sample. So a memorisable pair that is below threshold, or that matches a less-similar training sample, stays in the subset. The thesis acknowledges this openly, but the consequence is that the reported 9–25% gains on unmemorisable DROP and ROPES are upper bounds, not clean estimates. The DROP-CS gain is already non-significant, which fits that reading. For the central claim, this is a moderate blow, not a fatal one: the contributions in Chapters 5 and 6 do not depend on this subset being perfectly clean.\n\nThere are a couple of smaller wrinkles: the RR combination threshold is selected on the evaluation sets, so the \"Generally best RR combo\" number is optimistic; and the Musique RATD failure required post-hoc ablations. But both are handled explicitly, and the ablations are informative rather than hidden.\n\nWho should read this: anyone building small-model QA pipelines, especially with retrieval and rationale distillation. The thesis is long, but the core chapters are self-contained. I'd cite it and I'd send it out for review — a serious referee can push for a cleaner memorisation analysis without needing to reject the whole thesis.\n\nRecommendation: accept for review, conditional on toning down the anti-memorisation claim or tightening the contamination filter.\n\nBest.","headline":"A thorough, candid thesis on training small models for unseen compositional QA; the anti-memorisation claim needs a pinch of salt, but the retrieval-plus-rationale results are solid and worth refereeing.","tokens_in":51846,"tokens_out":2808,"would_cite":true,"duration_ms":28763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 440M-parameter BART model, trained to reason over retrieved contexts and LLM-generated rationales, can generalise to unseen compositional questions rather than answer from memorised training samples, and can match or exceed much larger…","keywords":["smaller language models","compositional questions","retrieval-augmented training datasets","rationale ranking","multi-hop dense retrieval","memorisation vs generalisation","unseen evaluation","question answering"],"falsifier":"One could audit the 'Unmemorisable' subsets by retrieving, for each sample, the nearest training neighbours below the threshold and checking whether the model's correct prediction coincides with a neighbour's gold answer; finding even one such pair would show the filter is incomplete and the generalisation gain is partially memorisation.","tokens_in":50840,"feed_emoji":"🧠","tokens_out":8405,"duration_ms":75918,"temperature":0.7,"pith_summary":"Working in an offline setting with only local workstation compute, this thesis tries to establish that a small language model (a 440M-parameter BART sequence-to-sequence model) can answer unseen compositional questions by reasoning over a provided context, rather than by reproducing memorised answers from training data. The author argues that two ingredients make this work: retrieval-augmented training datasets (RATD) built from noisy, partially evidential Wikipedia contexts, and a Rationale Ranking (RR) model that scores both LLM-generated rationales and retrieved paragraphs for relevance and truthfulness before they are combined. The thesis reports that RATD training and RR scoring each significantly improve performance on unseen evaluation datasets, that combined contexts beat single-source contexts, and that the small reasoner outperforms direct chain-of-thought prompting of the LLM that generated the rationales. If these claims hold, capable question answering no longer requires a huge always-online model; a modest local model with the right training and context sources would suffice.","feed_headline":"440M model matches much larger LLMs on unseen QA with same context","feed_subtitle":"Retrieval-augmented training and rationale ranking let a small offline model reason over noisy contexts instead of memorising.","key_machinery":"The load-bearing machinery is the context-acquisition and context-scoring pipeline. The Iterator extends multi-hop dense retrieval to an arbitrary number of hops (four here): at each hop it retrieves candidate paragraphs, reranks paragraphs and sentences, and uses an Evidence Set Scorer to decide whether the accumulated sentences are sufficient to answer the question, finally returning a title-prefixed paragraph-fragment context. The Rationale Ranking (RR) model is a smaller Transformer trained, with shared normalisation, to give one score to a question-context pair that reflects both relevance and truthfulness, allowing it to select and filter components from two knowledge sources (LLM-generated rationales and retrieved Wikipedia paragraphs). The Reasoning Model is a multitask-trained BART model; the RATD datasets give it practice weighing partially evidential facts in long noisy contexts of the same form it will see at test time.","core_discovery":"The central claim is that a smaller Language Model is capable of performance beyond simple memorisation in deriving correct answers to challenging compositional questions. To support this, the thesis introduces a method that scores each evaluation question together with its answer against every training sample using sentence-embedding cosine similarity, keeps only samples below a similarity threshold with no answer-term overlap, and then measures the effect of adding new training data on that 'unmemorisable' subset; improvements there (for DROP and ROPES) cannot be attributed to memorisation. The thesis further claims that the Iterator, an $n$-hop dense retrieval, reranking and evidence-set scoring system, can supply useful contexts for arbitrary unseen questions from Wikipedia, and that training on RATD datasets built from those noisy contexts teaches the 440M BART model to reason toward plausible answers from partial evidence. Adding a Rationale Ranking model that scores contexts for truthfulness as well as relevance lets the small reasoner exploit combined LLM-rationale and retrieved-paragraph contexts, with the combined-context model matching or exceeding much larger models when both receive the same knowledge.","pith_inferences":["Editorial inference: the embedding-based memorisability audit could be reused as a general contamination check for QA benchmarks, since scoring both question and answer catches paraphrase and discontinuous overlap that n-gram filters miss.","Editorial inference: the Rationale Ranking approach points toward a cheap, general route to truthfulness filtering — a small model trained on positive/negative pairs from diverse datasets may substitute for much larger factuality detectors in resource-limited pipelines.","Editorial inference: the recipe 'give a small reasoner noisy multi-source context and train it on similar noise' could transfer to other tasks such as fact verification, summarisation, or open-domain reading comprehension, reducing the need for specialised retrieval architectures.","Editorial inference: since the best global RR combination defaulted to naive concatenation for most samples, the untried step the thesis leaves implicit is a per-question selector that predicts which combination strategy will work, rather than applying one strategy globally."],"forward_implications":["A single workstation with one consumer GPU could run a question-answering system that handles unseen multi-hop and commonsense questions at levels previously associated with models orders of magnitude larger.","Retrieval-augmented training that includes partial, irrelevant, or missing evidence teaches the reasoner to use noisy combined contexts, so it can exploit a naive concatenation of rationale and retrieved paragraphs without special scoring.","Scores that combine relevance with truthfulness can filter false LLM rationales and improve mean performance, meaning small models can act as practical truthfulness filters in constrained settings.","The two knowledge sources are complementary: LLM rationales help commonsense reasoning, while multi-hop retrieval helps questions that need facts from two or more documents, so combining them raises performance beyond either source alone.","A smaller model given the same knowledge that a LLM generates for itself can beat the LLM's own direct answer, suggesting the bottleneck for small models is knowledge access rather than reasoning capacity."],"supporting_citations":[{"why":"Supplies the UnifiedQA multitask training format and dataset mixture from which the UQA baseline and much of the Reasoning Model training derive.","marker":"(Khashabi et al., 2020b)"},{"why":"Defines BART, the 440M encoder-decoder architecture used as the Reasoning Model throughout.","marker":"(Lewis et al., 2020a)"},{"why":"Provides the TDND synthetic numerical datasets used in the Chapter 4 intervention to separate memorisation from generalisation.","marker":"(Geva et al., 2020)"},{"why":"Supplies the sentence-embedding model whose cosine similarity scores define memorisable versus unmemorisable evaluation samples.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Provides the multi-hop dense retrieval training regime and learnable-path sampling that the Iterator extends beyond two hops.","marker":"(Xiong et al., 2021)"},{"why":"Introduces the two-stage sentence/paragraph reranking and evidence-set idea that the Iterator's reranker and Evidence Set Scorer build on.","marker":"(Khattab et al., 2021)"},{"why":"Shows smaller language models fail on unmemorised open-domain questions, motivating the thesis's context-reasoning setting and its memorisation analysis.","marker":"(Lewis et al., 2021)"},{"why":"Supplies the chain-of-thought prompting method used to generate rationales from locally run LLMs as a second knowledge source.","marker":"(Wei et al., 2022)"}],"fun_headline_variants":["440M model matches larger LLMs on unseen QA with context","Retrieval-augmented training aids small model on unseen questions","Rationale ranking boosts small model's unseen QA performance","Small model reasons over noisy contexts to match larger LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The anti-memorisation conclusion rests on the assumption that the similarity threshold of 60 plus the no-answer-term-overlap filter really leaves only evaluation samples that cannot be answered from memory; if a memorisable paraphrase or a discontinuous token overlap falls below that threshold, the reported gains on 'unmemorisable' subsets could be inflated by memorisation.","fun_headline_variants_meta":{"raw":{"variants":["440M model matches larger LLMs on unseen QA with context","Retrieval-augmented training aids small model on unseen questions","Rationale ranking boosts small model's unseen QA performance","Small model reasons over noisy contexts to match larger LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3792,"prompt_tokens":1072,"completion_tokens":2720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2651}},"tokens_in":688,"tokens_out":2720,"duration_ms":20565,"temperature":1.0,"reasoning_tokens":2651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:39:20.320066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One could audit the 'Unmemorisable' subsets by retrieving, for each sample, the nearest training neighbours below the threshold and checking whether the model's correct prediction coincides with a neighbour's gold answer; finding even one such pair would show the filter is incomplete and the generalisation gain is partially memorisation.","supporting_citations":[],"review_version":1}