{"id":"6bea18a4-3082-4a79-aa6c-bc182559051b","arxiv_id":"2411.19360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DENIAHL, a new synthetic benchmark, shows that LLM needle-in-a-haystack recall is influenced by data type, item length, and pattern structure, not just context length.","lead":"This paper introduces DENIAHL, a synthetic benchmark that tests how well large language models recall a specific item buried in long lists of key-value pairs while the size, type, and pattern of the data are varied. The findings show that recall depends not only on context length, but also on data characteristics such as numbers versus letters and item length.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-type finding may be confounded with tokenization: 32-character numeric vs letter items are not shown to be token-equivalent, so 'lost-in-the-middle' vs 'lost-in-the-end' could reflect effective context length, not data type.","rationale":"I read the paper as asserting that NIAH recall depends on data features beyond context length, with the sharpest evidence being the number-vs-letter curve shapes. That assertion requires the data-type manipulation to be unconfounded with length. The manuscript controls sizes by characters/KV pairs, not tokens, and never reports token counts, yet the models' context limit is specified in tokens. Since tokenization is a known source of length differences in LLaMA-2, this is a concrete internal-validity threat. The reader's external-validity concern is real but secondary: even if random strings are a good proxy, the type effect would still be uninterpretable if token length varies. The benchmark construction is clear, code is provided, and the manual ROUGE recheck is good practice; those do not resolve the confound. I recommend keeping the conditional verdict but making the required condition token-matched replication of the data-type condition, not merely additional models or real-world data.","tokens_in":9427,"tokens_out":5055,"duration_ms":46434,"concrete_test":"Use the LLaMA-2 tokenizer to count tokens in the full prompts for the 'All Numbers', 'All Letters', and 'Mixed' conditions at item length 32 (and at 48/56/64 if rerun). If the per-prompt token counts differ across types, construct new datasets with item lengths adjusted so that for each target position the total prompt length in tokens and the target's token index are matched across the three data types (e.g., shorten letter strings or lengthen numeric strings). Rerun the Fig. 6 sweep on LLaMA-2 7B with matched token positions and compare the curve shapes. If the lost-in-the-middle vs lost-in-the-end distinction disappears under token matching, the data-type claim is an artifact of tokenization; if it persists, the claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim (Figs. 1 and 6; §4.2.3) is that changing data type from numbers to letters changes the shape of LLaMA-2 7B's NIAH recall curve. The manipulation is implemented by fixing key/value length at 32 characters while varying whether the characters are digits, letters, or mixed. But DENIAHL's size limits are defined in token space (§4.2: '4096 input token constraints'), and no token counts are reported for the three data-type conditions. LLaMA-2's SentencePiece tokenizer can represent digit strings and letter strings with different numbers of tokens; if one type yields shorter prompts, then at every item index the target sits at a different token depth, and the 'end' of context falls at a different token position. The observed U-shape vs L-shape could therefore be produced by effective token-length differences rather than by data type per se. The paper also scores exact-match on generated strings; tokenization differences in the gold values can change exact-match difficulty independently of retrieval. Because the paper's own frame is about factors 'beyond context length,' this confound directly threatens the strongest claim. Similar reasoning applies to the item-length comparison across types in Fig. 6.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DENIAHL, a synthetic benchmark for needle-in-a-haystack (NIAH) evaluation that ablates factors beyond context length: data size (number of key-value pairs and item length), pattern structure (simple/complex numerical and letter patterns, with and without broken patterns), and data type (numbers, letters, mixed). The authors evaluate LLaMA-2 7B and GPT-3.5 and report that data type and item length affect recall accuracy curves, with number-type data exhibiting a 'lost-in-the-middle' pattern and letter-type data exhibiting a 'lost-in-the-end' pattern for LLaMA-2 7B. The paper also compares against the Kamradt (2023) and Liu et al. (2023) benchmarks and discusses implications for long-context model evaluation.","tokens_in":9614,"tokens_out":5438,"duration_ms":46194,"significance":"If its central claims survive scrutiny, DENIAHL would be a valuable contribution because it moves NIAH evaluation beyond the common context-length axis and highlights that data characteristics shape model recall behavior. The benchmark is synthetic and released on GitHub, and the paper makes falsifiable, model-specific predictions (e.g., LLaMA-2 7B shows data-type-dependent curve shapes, GPT-3.5 degrades only on long mixed-type items). However, the headline claim that data type per se changes the shape of the recall curve is currently not cleanly established because the data-type manipulation is confounded with token count. The paper also relies on visual inspection of curves without statistical uncertainty quantification. These issues are fixable and do not invalidate the benchmark concept, but they must be addressed before the specific findings can be trusted.","major_comments":[{"comment":"The data-type comparison is confounded with tokenization. In §4.2.3, the task fixes item length at 32 characters for numbers, letters, and mixed data, but the size limits are defined in tokens (§4.2: '4096 input token constraints'). No token counts are reported for the three data-type conditions. The paper itself provides direct evidence that character length does not correspond to token length: §5.2.3 states that the 64-character numerical condition was omitted for LLaMA-2 7B 'due to the input length constraint,' while the 64-character letter condition was run. This means numeric and letter strings tokenize differently, so in the 32-character conditions the target key-value pair sits at a different token depth and the total context token length differs. Thus the observed 'lost-in-the-middle' for numbers versus 'lost-in-the-end' for letters could be an artifact of effective token context length, not data type. The authors must report token statistics and either match conditions in token space or otherwise control for token position before making the central data-type claim.","section":"§4.2.3 and §5.2.3, Equation/tokenization confound"},{"comment":"The key shape claims—'lost-in-the-middle' versus 'lost-in-the-end' for different data types—are based on visual inspection of plotted mean accuracy curves without confidence intervals, error bars, or significance tests. Since each condition is averaged over 50 datasets, standard errors or bootstrap confidence intervals are straightforward to compute. Without such uncertainty quantification, the specific labels (U-shape versus L-shape) are not statistically supported, and this is load-bearing because the paper's central contribution is precisely that data type changes the shape of the recall curve.","section":"§5.2.3 (Figures 1 and 6)"}],"minor_comments":[{"comment":"The name of the framework is inconsistent: the Abstract and Introduction call it 'Data-oriented Evaluation of NIAH for LLM's,' while the header of §3 reads 'Data-centric Evaluation of Needle-in-a-haystack for LLM's.' Please unify the terminology.","section":"Title and §3"},{"comment":"The apostrophe in 'LLM's' is used for the plural; the correct plural is 'LLMs' (e.g., Abstract, §1, §5.2).","section":"Throughout"},{"comment":"The text states that 'GPT-3.5 effectively retrieves over its entire context window,' but Figure 2 only shows LLaMA-2 7B. Please clarify where the GPT-3.5 key-value retrieval results are presented, or restrict the claim to the data shown.","section":"§5.1.1"},{"comment":"The manual verification of GPT-3.5's responses that supports the '100% accuracy' conclusion is described only in prose; a brief protocol (e.g., number of responses inspected, criteria for correctness) would improve reproducibility.","section":"§5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a course project and is modest in scope, but the benchmark itself is a useful resource for the long-context evaluation community. The tokenization confound is the primary technical obstacle; the fact that the authors had to omit 64-char numeric data for LLaMA-2 7B while running 64-char letter data strongly suggests token-count differences that undermine the headline data-type claim. If the authors re-run the experiments with token-length-matched conditions or at minimum report and control for token positions, the paper could become acceptable. The lack of confidence intervals is a second issue that should be straightforward to fix. I also note that the paper does not compare against Ruler (Hsieh et al., 2024), which is the most closely related benchmark; a quantitative comparison would strengthen the positioning of DENIAHL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: DENIAHL is a genuinely useful benchmark, but the headline finding about data type is confounded with tokenization, so the specific shape claims should be treated with caution until the authors control for token counts.\n\nWhat's new: the paper systematically varies data type (numbers, letters, mixed), item length, and pattern integrity in a key-value NIAH setup, with clean synthetic data and released code. The broken-pattern condition is a smart manipulation — it tests whether models rely on global regularities rather than local retrieval. The finding that GPT-3.5 is largely robust while LLaMA-2 7B is not, and that pattern-breaking can sometimes help rather than hurt, adds useful texture to the NIAH literature.\n\nThe paper also does several things right: it builds on Liu et al. and Ruler, is honest in its limitations section, and its central qualitative claim — that NIAH performance depends on data features beyond context length — is plausible and worth taking seriously.\n\nThe main soft spot is the data-type comparison. The paper fixes item length at 32 characters, not 32 tokens. LLaMA-2's tokenizer is byte-pair; digit strings and letter strings can easily tokenize to different lengths. If numeric items yield longer or shorter token contexts, the needle sits at a different token depth for each type, and the observed 'lost-in-the-middle' vs 'lost-in-the-end' difference could come from effective position rather than data type. That directly threatens the paper's strongest claim. The fix is straightforward: report token counts per condition and re-run with matched token lengths.\n\nOther issues are minor by comparison: no confidence intervals for the curve shapes, only two models, one dropped condition, and the acknowledged ROUGE limitations for long-form NIAH. These don't sink the paper but do limit how much weight the shape claims can carry.\n\nIn sum, this is a legitimate empirical contribution with a testable central idea, and the tokenization confound is addressable. I'd send it to peer review, expecting the authors to add token-length controls before acceptance. If I were writing about long-context evaluation, I'd cite it, and it's worth a reading group slot.","headline":"DENIAHL is a worthwhile NIAH benchmark, but its data-type shape claims need token-length controls.","tokens_in":10180,"tokens_out":2429,"would_cite":true,"duration_ms":23455,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Data type, not just length, shifts where LLMs lose the needle","keywords":["needle-in-a-haystack","long-context language models","key-value retrieval","lost-in-the-middle","data type effects","synthetic benchmark","LLM recall","context length"],"falsifier":"Re-running DENIAHL's 50-pair all-letters and all-numbers conditions at 4k tokens on LLaMA-2 7B and checking accuracy at the final needle positions would settle the claim: the paper predicts a letter-specific end-of-context drop that is absent for numbers, and a numeric lost-in-the-middle dip; a model with flat letter recall would falsify the proposed type effect.","tokens_in":9194,"feed_emoji":"🔍","tokens_out":5553,"duration_ms":44776,"temperature":0.7,"pith_summary":"This paper argues that a language model's ability to find a needle in a haystack depends on the shape of the data in the haystack, not only on how long the haystack is. To show this, the authors build DENIAHL, a synthetic key-value benchmark that varies data type (numbers, letters, mixed), item size, and whether values follow a pattern. On LLaMA-2 7B they find the classic lost-in-the-middle curve for numbers but a lost-in-the-end curve for letters, and longer items and more items generally hurt recall. GPT-3.5 stays near-perfect on most variants and only degrades on long mixed-type items. The point is that NIAH results are data-dependent, so measuring context length alone is not enough.","feed_headline":"Numbers and letters shift where LLMs lose the needle","feed_subtitle":"LLaMA-2 7B shows lost-in-the-middle for numeric data but lost-in-the-end for letters, so context size alone won't predict recall.","key_machinery":"The carrying object is DENIAHL, a synthetic benchmark of string-serialized JSON key-value pairs in which the dataset features are ablated in three directions: data size (number of items and item length), pattern (arithmetic or letter-case patterns, sometimes broken at the queried position), and data type (numbers, letters, mixed). The task is to extract the value for a queried key, scored by exact-match accuracy. Because the same retrieval task is run while only the data's statistical surface changes, the benchmark attributes changes in the recall-position curve to data type and size rather than to the number of tokens alone.","core_discovery":"On DENIAHL, the recall curve's shape is not fixed: LLaMA-2 7B exhibits typical lost-in-the-middle behavior with numeric key-value pairs, where accuracy is highest at the beginning and end and dips in the middle, but lost-in-the-end behavior with letter key-value pairs, where accuracy stays high at the start and falls near the end; mixed data resembles letters. The paper also reports that recall drops when item length increases, and that LLaMA-2 7B sometimes retrieves a deliberately broken pattern's true value better than it retrieves a value that follows the pattern, which the authors read as evidence against reliance on global pattern inference. GPT-3.5 is largely unaffected except by long mixed-type items, so per the paper the combination of data features, not any one feature alone, can degrade even a stronger model.","pith_inferences":["If the letter-versus-number difference generalizes, it may partly reflect tokenization: digit sequences are compressed into fewer tokens than random letters, so the effect could be reproduced or altered by changing tokenization rather than model architecture, a testable extension using hex-encoded digits versus random letters.","A practical consequence the authors leave implicit is that document-preprocessing choices, such as converting tables to prose or changing numeric identifiers to alphanumeric strings, may systematically move where a deployed system's recall failures occur.","The same ablation could be extended to newer models with different positional encodings or extended training, which would clarify whether these type-dependent curves are a general property of transformers or a artifact of specific training data."],"forward_implications":["NIAH evaluations that report only average accuracy across needle positions can hide that a model's failures concentrate at the end of context for alphabetic data and in the middle for numeric data.","Increasing a model's context window does not by itself guarantee recall, because position biases shift with item length and data type, so long-context capacity should be benchmarked across data features.","Combined features matter: GPT-3.5's recall degrades only when long item length and mixed data type occur together, which suggests feature-interaction tests should be part of long-context evaluations.","Models can retrieve a deliberately broken pattern more accurately than an intact pattern, so exact-match retrieval tasks should break patterns to test fine-grained recall rather than pattern completion."],"supporting_citations":[{"why":"Supplies the original key-value retrieval benchmark and the lost-in-the-middle phenomenon that DENIAHL extends and contrasts.","marker":"Liu et al. (2023)"},{"why":"Provides the needle-in-a-haystack test and the Paul Graham essays benchmark used as a natural-language comparison.","marker":"Kamradt (2023)"},{"why":"Ruler is the most similar synthetic key-value NIAH benchmark and serves as the closest prior baseline.","marker":"Hsieh et al. (2024)"},{"why":"Represents the prior focus on input token count that this paper expands beyond.","marker":"Levy et al. (2024)"},{"why":"Defines ROUGE-1, the metric used to score the essay-based NIAH responses.","marker":"Lin (2004)"},{"why":"Supports the broader claim that NIAH performance is influenced by properties of the input or training data.","marker":"Machlab and Battle (2024)"}],"fun_headline_variants":["Numbers dip mid-context, letters dip at end for LLaMA-2","DENIAHL shows data type reshapes LLM's needle recall","Item size and type, not just context length, hurt recall","GPT-3.5 unfazed unless mixed-type items get long","LLaMA-2's recall shape flips when data type changes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that synthetic random key-value strings behave like real-world long-context data, so the type- and size-dependent recall patterns will show up in practical uses such as document QA; the paper itself notes its tests may not reflect real-world extractive needs.","fun_headline_variants_meta":{"raw":{"variants":["Numbers dip mid-context, letters dip at end for LLaMA-2","DENIAHL shows data type reshapes LLM's needle recall","Item size and type, not just context length, hurt recall","GPT-3.5 unfazed unless mixed-type items get long","LLaMA-2's recall shape flips when data type changes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1395,"prompt_tokens":920,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":536,"tokens_out":475,"duration_ms":4313,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:14:28.547543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running DENIAHL's 50-pair all-letters and all-numbers conditions at 4k tokens on LLaMA-2 7B and checking accuracy at the final needle positions would settle the claim: the paper predicts a letter-specific end-of-context drop that is absent for numbers, and a numeric lost-in-the-middle dip; a model with flat letter recall would falsify the proposed type effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ROUGE-1, the metric used to score the essay-based NIAH responses."}],"review_version":1}