{"id":"ed67fc47-ff87-4425-b8bb-f16f5a3b4503","arxiv_id":"2412.13377","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DateLogicQA evaluates 12 LLMs on 190 date-reasoning questions and claims separate representation-level and logical-level temporal biases, but the Semantic Integrity Metric is undefined.","lead":"DateLogicQA is a 190-question benchmark that tests how well large language models handle dates in seven different formats. The authors evaluate 12 models with human judges and claim to find two distinct failure types: problems with how dates are represented, and problems with reasoning about dates even when the date is correct.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed representation-vs-logic bias split is unsupported because the Semantic Integrity Metric is never defined and the bias labels are inferred from final outputs rather than from tokenizer evidence.","rationale":"The paper's central claim is not merely that LLMs are imperfect at date reasoning (which is amply supported) but that errors decompose into two named, separable biases tied to distinct mechanisms. That claim requires separate measurement of tokenization/embedding quality and reasoning quality. The only stated instrument for the former, the Semantic Integrity Metric, is absent: no definition, formula, or code appears in Sections 3-5; the conclusion references it, and the appendix contains one illustrative table and a correlation figure without axes or data. Human annotators, who are also the paper's authors, then label outputs. Since final outputs are a joint product of tokenization, decoding, and reasoning, the labels cannot identify the mechanism with the confidence the taxonomy requires. This is a correctness risk, not merely a presentation issue: a benchmark user cannot compute SIM on new examples, and the headline result, two distinct bias types with different recommended fixes (tokenization strategies versus post-training), is not derivable from the reported evidence. The underlying idea is plausible, and the paper cites relevant prior work on tokenization and temporal reasoning, but the evidence here does not establish the separation. The proposed test would either recover the metric and validate the contingency or fail; if it fails, the reader's REJECT verdict stands. Because the reader already identified the undefined metric and the output-inference problem, my analysis does not change the verdict, though my emphasis differs slightly from the reader's stated weakest assumption.","tokens_in":15321,"tokens_out":5887,"duration_ms":56863,"concrete_test":"Obtain or reconstruct the SIM formula from the authors or derive it from Table 4. For all 190 questions and the 12 models, compute SIM for each date string using the model's actual tokenizer (open models: HuggingFace tokenizers; API models: tiktoken or official tokenizers). Then test the contingency: does the human 'wrong-date' label (Light Orange) predict a low SIM independent of final-answer correctness, and does 'correct-date-but-wrong-logic' (Light Teal) predict a high SIM? If the authors cannot state the SIM formula, or if the contingency is not significant, the representation-logical split is not supported by the reported evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that 'Representation-Level Bias' (wrong token/embedding) and 'Logical-Level Bias' (correct token, wrong reasoning) are separable in observed errors. Two conditions must hold: (C1) tokenization quality is measured directly, and (C2) the human rubric's assignment of a response to one camp is not confounded by final-answer correctness. Both fail. The Semantic Integrity Metric is named in the abstract and conclusion but never defined in Section 4 or elsewhere in the main text; the only trace is the appendix table (Table 4, 'Generated by Spread-LaTeX') listing SI values (1.00, 0.66, 0.60, 0.45, 0.40) with tokenized outputs, plus Figure 5, which plots 'semantic integrity score against token count' but gives no formula, code, or axis details. Consequently, the claimed correlation between SI and output type cannot be checked. Separately, Section 4.1's rubric labels Light Orange and Light Teal by reading off whether the model's output contains a wrong date or wrong inference, but for API models tokenization is invisible; a malformed date string can arise from decoding, sampling, prompting, or answer formatting rather than suboptimal embeddings, and a correct date in the answer does not prove the model tokenized or interpreted it correctly internally. If C1 and C2 fail, the two categories collapse into 'final answer wrong for some reason,' and the paper's headline contribution, a separable two-level bias analysis with distinct mitigations, is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DateLogicQA, a human-curated benchmark of 190 questions spanning seven date formats, three temporal contexts, and four reasoning types, and reports a human evaluation of 12 LLMs. The authors propose the Semantic Integrity Metric (SI) to assess tokenization quality and claim to identify two distinct temporal biases: Representation-Level Bias (faulty tokenization/embeddings of dates) and Logical-Level Bias (correct date tokens but flawed reasoning). The results are presented as accuracy breakdowns by date format, temporal context, and question type, followed by qualitative discussion of mitigation strategies. The central claim is that LLMs exhibit separable representation-level and logical-level temporal reasoning failures.","tokens_in":15638,"tokens_out":3859,"duration_ms":36251,"significance":"If the central claim were established, the paper would offer a useful diagnostic lens for temporal reasoning in LLMs, and the two-level bias taxonomy could guide targeted interventions in tokenization and post-training. The use of human evaluation with a reported Cohen's kappa of 0.80, the coverage of uncommon Julian and compact date formats, and the breadth of models tested are positive features. However, the paper's key analytical tool, the Semantic Integrity Metric, is never formally defined, and the bias categories are inferred from final answer text rather than from direct tokenization evidence. The benchmark itself is not released. As presented, the paper's main claims are not verifiable, and the proposed representation-versus-logic distinction is not supported by the reported methodology.","major_comments":[{"comment":"The Semantic Integrity Metric (SI) is never defined. The abstract and conclusion name it as the measure of tokenization quality, and Table 4 lists SI values (1.00, 0.66, 0.60, 0.45, 0.40) together with tokenized outputs, while Figure 5 plots an unnamed correlation between 'semantic integrity score' and token count. No formula, computational procedure, axis labels, or coefficient are provided anywhere. Because SI is the basis for the representation-level bias claim, this missing definition is a load-bearing gap that makes the central analysis impossible to evaluate or reproduce.","section":"Abstract; Section 4.1; Appendix Table 4; Figure 5"},{"comment":"The assignment of responses to Representation-Level Bias versus Logical-Level Bias is made from the final answer text: Light Orange is defined as 'the model tokenizes dates inaccurately but reaches the correct answer through logical reasoning,' and Light Teal as 'the model tokenizes correctly but misapplies logic.' For API models, tokenization is not observable, so a wrong date string in the output could stem from decoding, sampling, prompting, or answer formatting rather than from suboptimal embeddings, while a correct date in the output does not verify that the model tokenized or interpreted the date correctly internally. The two bias categories are therefore not separable from final-answer content, and the headline claim of distinct representation-level and logical-level biases is not supported by the reported evidence.","section":"Section 4.1 (rubric definitions, Light Orange and Light Teal)"},{"comment":"The ground-truth answer for the question 'Which of the following famous people died on 23041616?' is listed as 'William Shakespeare OR Miguel de Cervantes.' This is an ambiguous label: both individuals died in April 1616 depending on the calendar convention, and the table shows models receiving credit for either B or D. A benchmark question with an inclusive ground truth of this kind cannot serve as a reliable gold label for measuring tokenization or reasoning quality, and it also compromises the human rubric labels derived from such answers.","section":"Table 3, Factual row"},{"comment":"The DateLogicQA dataset is not released or linked, and only five example questions are shown in Table 1. There is no stated availability plan, and the paper does not report the number of questions per date format, temporal context, or question type. This makes the percentage breakdowns in Section 5 uninterpretable in terms of sample sizes and prevents independent verification of the benchmark's composition and the reported model rankings.","section":"Section 3; Section 5"},{"comment":"The correlation plot between 'semantic integrity score' and token count is presented without a correlation coefficient, confidence interval, or axis descriptions. If the SI values are computed from the same tokenized outputs used by the human rubric in Section 4.1, the claimed relationship would be circular; because the metric is undefined, the reader cannot rule out this possibility.","section":"Figure 5"}],"minor_comments":[{"comment":"The caption 'Generated by Spread-LaTeX' appears to be an artifact of the LaTeX table-generation tool rather than a meaningful scientific caption and should be replaced or removed.","section":"Table 4 caption"},{"comment":"The sentence 'These results evidence two critical areas where LLMs shine and their struggles' is vague; the text should state on which formats, contexts, or question types the models specifically succeed or fail.","section":"Section 5.1"},{"comment":"Several references are duplicated or appear under two different keys: Su et al. (2024a) and (2024b), Tan et al. (2023a) and (2023b), and Xiong et al. (2024a) and (2024b) cite the same works with different labels.","section":"References"},{"comment":"The paper reports four annotators and Cohen's kappa of 0.80 but does not describe the annotation interface, the exact instructions given beyond the color rubric, or the procedure for resolving disagreements; additional detail would strengthen the reliability claim.","section":"Section 5.1, human evaluation"},{"comment":"The definitions of the four color categories are given only informally in the text; a formal decision tree or pseudocode describing how a response is assigned to Dark Orange, Light Orange, Light Teal, or Dark Teal would improve reproducibility.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The paper reads as an early draft. The undefined Semantic Integrity Metric, the unobservable tokenization attribution for API models, the ambiguous ground truth in Table 3, and the unreleased dataset jointly undermine the central claims. The missing definition could in principle be added, but the fundamental problem is that the representation-versus-logic distinction is not identified by the experimental design; it is inferred from final outputs. Rejection with an opportunity to resubmit after a substantially revised methodology and released benchmark seems appropriate. The small 190-question benchmark is not compared to existing temporal reasoning benchmarks such as TempReason or TimeQA, so the incremental contribution over prior work is unclear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: DateLogicQA is a reasonable small benchmark idea with a systematic sweep of date formats, but the paper's headline claim—that LLM temporal errors split into separable Representation-Level and Logical-Level biases—is not supported as written. The Semantic Integrity Metric that supposedly measures tokenization quality is never defined anywhere; it appears only as numbers in an appendix table and a correlation plot with no formula, code, or axis details. Without that definition, the central correlation in Figure 5 is uncheckable.\n\nWhat is genuinely useful: the dataset design covers seven date notations including the rare day-of-year style, crosses four reasoning types and three time periods, and evaluates twelve models with human labeling at kappa 0.80. The descriptive results—models struggle most with the unusual day-of-year formats and with numerical questions—are plausible and consistent with prior work on tokenization and arithmetic. If the benchmark were released with precise labels and the metric properly specified, it would be a usable instrument for the temporal-reasoning community.\n\nThe soft spots go beyond the missing formula. The bias taxonomy is inferred from final answers rather than direct tokenizer observations: a wrong date string in the output is taken as evidence of faulty tokenization, and a correct date with wrong logic as evidence of faulty reasoning. For API models, tokenization is invisible, so the same output error could come from decoding, sampling, or response formatting. The two categories may collapse into 'wrong answer for some reason.' The ground truth also contains at least one ambiguous question—the Shakespeare/Cervantes death-date item where two options are arguably correct. The annotators are the paper's authors, which is a conflict worth disclosing in the limitations. And the 'Julian calendar' labels actually describe day-of-year notation, a minor but telling terminology slip.\n\nBottom line: this is a promising benchmark with an unfulfilled methodological core. A serious editor should send it to review, but the authors need major revision: define SI, release the dataset and rubric, and either obtain tokenizer-level evidence for the bias split or soften the claim. I wouldn't cite it in its current form.","headline":"A useful small benchmark idea undermined by an undefined core metric and a bias taxonomy inferred from final outputs rather than tokenizer evidence.","tokens_in":16133,"tokens_out":3282,"would_cite":false,"duration_ms":29522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DateLogicQA's human evaluation of twelve LLMs finds temporal reasoning failures split into two distinct biases: misreading the date and misreasoning about a correctly read date.","keywords":["DateLogicQA","temporal reasoning","LLM bias","tokenization","representation-level bias","logical-level bias","Semantic Integrity Metric","date formats"],"falsifier":"Run each model's tokenizer on the benchmark's date strings and inspect token boundaries and embeddings for a sample of wrong-date answers. If a substantial fraction of wrong-date outputs come from inputs whose date tokens are correctly segmented and embedded, the Representation-Level Bias category is not supported, and the two bias types collapse into a single output-error category.","tokens_in":15124,"feed_emoji":"📅","tokens_out":6280,"duration_ms":52314,"temperature":0.7,"pith_summary":"DateLogicQA is a 190-question benchmark for temporal reasoning in large language models, covering seven date formats, past/present/future contexts, and four reasoning types. The paper claims that LLM temporal errors split into two distinct categories: Representation-Level Bias, where the model mis-encodes the date itself, and Logical-Level Bias, where the date is read correctly but the reasoning about it goes wrong. A human evaluation of twelve models, supported by a Semantic Integrity Metric for tokenization quality, finds both biases across models. The result matters because fixing each failure mode requires different remedies, from tokenization and pretraining data to post-training reasoning methods.","feed_headline":"Benchmark splits LLM date failures into misreading and misreasoning","feed_subtitle":"190 human-scored questions show date-token and reasoning errors are separable failure modes.","key_machinery":"The key mechanism is the four-colour human evaluation rubric combined with the Semantic Integrity Metric. The rubric labels each model response as incorrect, representation-level biased (wrong date, compensated reasoning), logical-level biased (right date, wrong reasoning), or correct. The Semantic Integrity Metric scores how faithfully a tokenizer preserves date components such as day, month, and year; the paper uses it to connect tokenization quality to performance. Together these tools attribute each error to either the encoding stage or the reasoning stage of the model.","core_discovery":"On the paper's own terms, the central discovery is that LLMs exhibit two separable temporal biases rather than one undifferentiated error rate. Representation-Level Bias shows up when a model outputs a faulty date, for example 'March 273, 2040' for a Julian-format input, yet may still land on the right answer through compensating reasoning. Logical-Level Bias appears when the model reproduces the date tokens correctly but then misapplies arithmetic or world knowledge, as when a model computes October 2040 instead of July 1454 for a numerical date addition. The four-colour human rubric, which classifies each response as incorrect, representation-biased, logical-biased, or correct, is what separates these categories. Aggregate results show Julian calendar formats are hardest, numerical questions hardest, and future dates easier than historical or present dates.","pith_inferences":["The two-bias split implies that a single accuracy score hides which stage fails; two models with identical accuracy may need opposite fixes.","A direct test of the representation-level claim is to inspect tokenizer outputs for wrong-date answers: if date tokens are intact in those cases, the bias is not truly at the representation level.","The same rubric could transfer to other structured entities such as times, quantities, or identifiers to ask whether representation-level versus logical-level biases are a general phenomenon.","Because the benchmark is small and human-scored, the reported bias rates likely carry wide uncertainty; a larger validated version would make the comparisons between formats and models more precise."],"forward_implications":["Improving tokenization and pretraining coverage for rare date formats should shrink Representation-Level Bias.","Post-training approaches such as Direct Preference Optimization and chain-of-thought prompting can be benchmarked specifically against Logical-Level Bias on DateLogicQA.","The format and context axes provide model developers with a checklist of which date notations and temporal periods need more training data.","Future dates being easier than historical or present ones suggests pretraining data coverage, not reasoning capacity, is the main bottleneck."],"supporting_citations":[{"why":"Supplies the prior temporal reasoning benchmark (Test of Time) that DateLogicQA extends and contrasts with.","marker":"Fatemi et al. (2024)"},{"why":"Establishes that numerical tokenization affects counting and arithmetic, motivating the tokenization-bias hypothesis for dates.","marker":"Zhang et al. (2024)"},{"why":"Shows tokenization impacts arithmetic in frontier LLMs, supporting the representation-level mechanism.","marker":"Singh and Strouse (2024)"},{"why":"Identifies temporal biases such as nostalgia and neophilia in LLMs, which the paper refines into representation-level and logical-level categories.","marker":"Zhu et al. (2024)"},{"why":"Shows temporal misalignment hampers accuracy, motivating the temporal-context analysis.","marker":"Zhao et al. (2024)"},{"why":"Documents inductive biases of GPT-3.5 and GPT-4 on temporal data, supporting the logical-level bias concept.","marker":"Kishore and He (2024)"},{"why":"Proposes task-agnostic temporal reasoning enhancement, which the mitigation discussion builds on.","marker":"Su et al. (2024a)"}],"fun_headline_variants":["LLMs fail dates two ways: misreading tokens, misreasoning logic","DateLogicQA: Two distinct temporal biases found in LLMs","Separating date-token errors from reasoning errors in LLMs","New benchmark reveals dual failure modes in LLM date handling","LLM temporal reasoning split: representation vs logic bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a wrong date in the output proves the date was tokenized or embedded incorrectly, and a right date with a wrong answer proves reasoning failed; this attribution is inferred from final answers without direct observation of tokenizer internals.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail dates two ways: misreading tokens, misreasoning logic","DateLogicQA: Two distinct temporal biases found in LLMs","Separating date-token errors from reasoning errors in LLMs","New benchmark reveals dual failure modes in LLM date handling","LLM temporal reasoning split: representation vs logic bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1095,"prompt_tokens":760,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":376,"completion_tokens_details":{"reasoning_tokens":249}},"tokens_in":376,"tokens_out":335,"duration_ms":4038,"temperature":1.0,"reasoning_tokens":249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:10:51.803420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each model's tokenizer on the benchmark's date strings and inspect token boundaries and embeddings for a sample of wrong-date answers. If a substantial fraction of wrong-date outputs come from inputs whose date tokens are correctly segmented and embedded, the Representation-Level Bias category is not supported, and the two bias types collapse into a single output-error category.","supporting_citations":[{"cited_title":"Unveiling Divergent Inductive Biases of LLMs on Temporal Data","cited_arxiv_id":"2404.01453","evidence_quote":"Documents inductive biases of GPT-3.5 and GPT-4 on temporal data, supporting the logical-level bias concept."}],"review_version":1}