{"id":"b20f6e15-c8d8-4d0b-a679-f2d65a19bd13","arxiv_id":"2509.19376","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A half-life recency prior appears to improve freshness in RAG, but the paper's abstract and body conflict, and the topic-evolution failure is misattributed to the clusterer.","lead":"This paper proposes a recency-weighted scoring rule for RAG retrieval and a K-means-based topic tracker on cybersecurity logs. It claims the recency prior fixes missed freshness, but the abstract and the main text report different results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Perfect Latest-Set@10 is plausibly an artifact of an underspecified, trivially satisfiable evaluation; the abstract's hard NVD test is absent from the body.","rationale":"The reader's weakest assumption is that the evaluation setup is not demanding or representative and that the perfect scores may be artifacts of a trivial retrieval problem. My reading identifies the same load-bearing concern and makes it more precise: the Latest-Set@10 metric, combined with the lack of reported query/candidate statistics and the CERT dataset's terminal-timestamp concentration, permits a trivial success. The abstract's NVD CVE results—the only non-perfect, more credible numbers—are entirely absent from the body, which further undermines the central claim. I therefore agree with the reader's REJECT verdict, and no adjustment is needed.","tokens_in":8830,"tokens_out":3369,"duration_ms":30917,"concrete_test":"Run src/eval_metrics_login.py with added instrumentation that prints, per query, the number of candidate documents before re-ranking and the size of the newest-document set. Add a negative-control baseline that ranks purely by recency (score = 0.5^{age/h}) and a harder decoy set: for each query, append 100 synthetically generated messages from the same product/event_type with similar text but timestamps 30–300 days older. If the recency-only baseline also achieves 1.00, or if the fused score drops below 1.00 after adding decoys, the reported perfect accuracy is an artifact of a candidate pool that already excludes older relevant items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a half-life recency prior solves freshness, reaching 1.00 Latest-Set@10 on CERT and 1.00 Latest@10 on synthetic data (§5.4–§5.5, Tables 4–6). This claim is load-bearing for the paper's main contribution, but the evaluation does not rule out a trivial success. The metric in §5.3 counts a query as successful if any document from the 'set of newest relevant documents' appears in the top 10. The paper never reports the number of queries, the candidate pool size per query, the distribution of timestamps, or how queries were constructed. If queries are generated from a document's own text and the candidate pool is filtered to a single product/entity, the newest relevant set can dominate the pool, making even a recency-only ranking score 1.0. The α-sensitivity results (Tables 4, 6) show only a drop to 0.667 at α=0.9, which is consistent with a tiny, easy query set. The abstract's 'hard NVD CVE' test (0.60 vs 0.20) is not in the body; the only real-data evaluation is CERT logon, which is described as having 'multiple equally valid events often shared the exact same terminal timestamp' (§5.3), a property that makes the 'newest set' large and easy to hit. Section 6 acknowledges that 'a perfect accuracy score can suggest an overly simplistic evaluation,' but the sensitivity analysis only varies α; it does not test the difficulty of the retrieval problem. Thus the central freshness claim is unsupported as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'temporal memory layer' for RAG: a four-stage pipeline (normalization, embedding, weekly clustering/matching, time-aware retrieval) and a fused ranking score score = α·cos(q,d) + (1−α)·0.5^(age_days/h) (Eq. 1). It reports two empirical contributions: (i) the recency prior achieves Latest@10 / Latest-Set@10 of 1.00 vs 0.00 for a cosine-only baseline on synthetic and CERT datasets; (ii) a K-means trend tracker yields macro-F1 0.08, which the body interprets as demonstrating the insufficiency of simple heuristics for topic evolution. The abstract, however, claims a hard NVD CVE test with Latest@10 0.60 vs 0.20 and attributes the low F1 to the labeling rule rather than the clusterer. The body lacks the NVD CVE experiment and the HDBSCAN/rule-fixing ablations, and reports no query counts, query construction details, candidate pool sizes, or statistical tests for the perfect accuracy scores.","tokens_in":9230,"tokens_out":4383,"duration_ms":36499,"significance":"If the results were fully substantiated, the work would be a useful lightweight baseline for freshness in RAG and a cautionary negative result on heuristic trend detection. The paper is readable, the method is simple, and a reference implementation is promised. However, the internal contradictions between the abstract and the body, combined with the absence of the abstract's hard tests from the body, make the empirical contribution unreliable as reported. The perfect accuracy scores are not supported by sufficient evaluation detail to rule out a trivially easy retrieval setup, and the negative topic-evolution result is directly challenged by the abstract's own attribution of the low F1 to the labeling rule. These are load-bearing issues, not presentation concerns.","major_comments":[{"comment":"The abstract reports an NVD CVE experiment with Latest@10 0.60 (fused) vs 0.20 (semantic-then-newest) and states that the method 'stays partial and parameter-sensitive.' The body contains no NVD CVE test; §5.4 and Table 5 report perfect 1.00 Latest@10 on synthetic data, and §5.5/Table 6 report perfect 1.00 Latest-Set@10 on CERT. These are incompatible characterizations of the same method's performance. The central freshness claim is ambiguous: is the contribution the perfect scores of the body or the partial scores of the abstract? The missing experiment must be included and the contradiction resolved.","section":"Abstract vs §5.4–5.5"},{"comment":"The body's central negative result is that a K-means heuristic tracker fails with macro-F1 0.08, 'rigorously demonstrating the insufficiency of simple heuristics' (§5.4) and attributing the failure to K-means assumptions (§6, Table 7). The abstract states that the low 0.08 F1 'is driven by the labeling rule, not the clusterer,' citing HDBSCAN: 0.10 and fixing the rule alone reaching 0.49, and 0.96 without clustering noise. These ablations do not appear in the body. If the abstract is correct, the paper's interpretation is reversed: the clusterer is not the bottleneck. As reported, the evidence cannot support the stated conclusion.","section":"Abstract vs §5.4 and §6"},{"comment":"The perfect accuracy scores are not interpretable without knowledge of query construction. No number of queries is given; no candidate pool size; no timestamp distribution; no breakdown of how queries were generated. 'Latest-Set@10' counts success if any document from the newest relevant set appears in the top 10; if that set is large (as the paper acknowledges for CERT: 'multiple equally valid events often shared the exact same terminal timestamp', §5.3) or if the candidate pool is small, a trivial ranking can score 1.0. The sensitivity analyses vary only α and report point estimates with no variance, confidence intervals, or significance tests; they cannot establish robustness.","section":"§5.3, Tables 4–6"},{"comment":"The key parameters α=0.7 and h=14, as well as the trend-label thresholds, are hand-set and then evaluated on the same datasets that motivated them. The paper acknowledges that 'a perfect accuracy score can suggest an overly simplistic evaluation' (§6), but the only mitigation offered is the α sensitivity analysis; there is no held-out validation, no alternative query generation, and no variation of half-life or trend thresholds. The claim that the default parameters are generally effective is therefore unsupported.","section":"§2.4, §3.2, Table 2"}],"minor_comments":[{"comment":"Typos and formatting issues: 'Thisatemporalview' (§1), 'Pptionally' (§2.3), 'clarifed' (§7), and missing spaces in 'per-slice;cluster' (§2.3).","section":"Throughout"},{"comment":"The two sensitivity tables report identical numbers; consider merging or noting explicitly that the results are duplicated across datasets.","section":"Tables 4 and 6"},{"comment":"The relationship between 'Latest@10' and 'Latest-Set@10' is not defined precisely; the abstract uses Latest@10 while the body switches to Latest-Set@10. Define both and state which is used for each result.","section":"§5.3"},{"comment":"In Eq. (1), age_days(t)/h appears in the exponent; specify that h is in days and clarify the reference time (query time) used for age_days(t).","section":"Eq. (1)"}],"recommendation":"reject","confidential_remarks":"The discrepancy between the abstract and the body is severe enough that the manuscript should not be published in its current form. The authors may have a useful idea, but the current version does not support its central claims: the promised NVD CVE test and HDBSCAN/rule-fixing ablations are absent from the body, and the reported perfect accuracies are not accompanied by enough evaluation detail to rule out trivial retrieval settings. A resubmission would need substantially new experiments and a reconciled narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: the paper’s core freshness result is unverifiable as reported because the abstract and body disagree, and the perfect scores look like artifacts of an easy evaluation. That’s a shame, because the framing is useful and the implementation is simple and reproducible.\n\nWhat’s actually new: applying a half-life recency prior from temporal IR to RAG is a minor adaptation, but the paper shows it can beat a cosine-only baseline on latest-item retrieval in two controlled settings. The sensitivity analysis around α is a nice touch: performance degrades when semantic weight dominates, which supports the temporal component. The topic-evolution negative result is potentially interesting, but as written it’s undermined by the abstract.\n\nNow the soft spots, and they are substantial. First, the abstract reports an NVD CVE experiment and HDBSCAN/labeling-rule ablations that are entirely absent from the body. On the NVD test, Latest@10 is claimed to be 0.60 with the prior and 0.20 with a semantic-then-newest baseline. The body has no such test. The abstract says the low trend F1 is driven by the labeling rule, not the clusterer, with HDBSCAN at 0.10 and fixed-labeling scores of 0.49 and 0.96. The body attributes the low F1 to the clusterer and uses it to conclude simple heuristics are insufficient. These are contradictory scientific statements.\n\nSecond, the evaluation is too thin. No query counts, candidate pool sizes, or timestamp distributions are reported. The Latest-Set@10 metric counts a success if any document from the newest relevant set appears in the top 10. If queries are generated from the newest document’s text, a recency-only ranker can achieve 1.00. The paper itself acknowledges that a perfect score can suggest an overly simplistic evaluation, but then only varies α, not task difficulty.\n\nThird, α=0.7 and the 14-day half-life were hand-tuned and tested on the same datasets, with no holdout. That’s circularity, though not fatal for a heuristic paper—but it limits the strength of the freshness claim.\n\nWhat’s good: the code is promised, the limitations section is honest, and the paper is clearly written. The method is a legitimate engineering pattern.\n\nWho this is for: practitioners building lightweight time-aware RAG layers will find a starting point, but they should not trust the numbers as evidence of robustness. The paper needs major revision to reconcile the abstract with the body, add the missing experiments or remove them, and report basic evaluation statistics. I would not desk-reject it outright—there is a real kernel—but I would send it back for major revision before peer review. As it stands, I wouldn’t cite it.\n\nBest.","headline":"The core freshness result is unverifiable as reported—the abstract and body tell different stories, and the perfect scores look like artifacts of an easy evaluation.","tokens_in":9682,"tokens_out":5142,"would_cite":false,"duration_ms":42545,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A recency prior restores freshness in RAG where semantic-only retrieval scores zero.","keywords":["temporal RAG","recency prior","half-life decay","freshness","topic evolution","trend detection","as-of queries","clustering heuristic"],"falsifier":"A single freshness query where the newest relevant item is not in the initial semantic top-K candidate list—so no re-ranking score can surface it—would falsify the claim that the fusion method alone fixes freshness; likewise, a benchmark where a cosine-only retriever already finds the newest relevant item whenever it is semantically closest would show the 1.00-versus-0.00 gap is an artifact of task design.","tokens_in":8712,"feed_emoji":"🕒","tokens_out":4072,"duration_ms":37565,"temperature":0.7,"pith_summary":"This paper tries to separate two temporal problems in retrieval-augmented generation: finding the newest relevant item and tracking how topics evolve. For freshness, it argues that adding a half-life recency prior to cosine similarity is enough to surface the newest relevant item; on its synthetic and real-world log streams, the fused score reaches perfect latest-item accuracy while cosine-only retrieval misses every target. For topic evolution, the paper claims the opposite: a simple weekly clustering and label-matching heuristic scores only 0.08 macro-F1, showing why this harder task needs richer models. A reader should care because the temporal layer is model-agnostic and requires no retraining, so if the freshness claim holds it is a drop-in fix for a common failure mode.","feed_headline":"Half-life recency score turns RAG freshness misses into hits","feed_subtitle":"A tiny timestamp-decay term rescues stale retrieval on security-log queries; heuristic trend detection remains far too weak.","key_machinery":"The load-bearing object is the fused retrieval score score(q,d,t)=α cos(q,d)+(1−α)·0.5^(age_days(t)/h), with default α=0.7 and half-life h=14 days. It re-ranks an initial semantic candidate list so that on-topic documents within roughly a half-life of the query time can outrank older near-duplicates. The secondary machinery is the weekly topic tracker: per-week clusters matched greedily to the previous week by centroid similarity, with hard thresholds converting matched clusters into emergence, growth, decay, drift, or stable labels. That tracker's failure is measured by macro-F1 against scripted labels.","core_discovery":"The central discovery is a clean asymmetry in temporal RAG. A lightweight score that blends semantic similarity with a half-life temporal decay—score = α cos(q,d) + (1−α)·0.5^(age/h)—turns a 0.00 latest-item retrieval baseline into 1.00 on both a scripted 12-week synthetic security-log stream and a noisy 71-week public logon stream, with performance degrading only when the semantic weight α exceeds 0.7. The same pipeline's heuristic topic tracker, built from weekly clusters matched across weeks and labeled emergence, growth, decay, drift, or stable, attains only 0.08 macro-F1 on scripted trends, which the paper reads as evidence that trend detection is a genuinely harder problem that simple","pith_inferences":["If the asymmetry generalizes, freshness in RAG may not need learned or architecture-level temporal models; a calibrated half-life and the right candidate pool could be sufficient, leaving topic evolution as the main open temporal problem.","A direct test would vary the half-life h per domain (breaking news versus technical documentation) and measure whether the perfect accuracy holds outside the two log datasets; the paper only reports default h=14 days.","Because re-ranking only works on candidates that survive the initial semantic top-K list, the method's guarantees are bounded by the recall of the first-stage retrieval; a deliberately constructed query whose newest answer is absent from that candidate set would expose the limit.","The perfect scores could partly reflect a retrieval task where the newest document is also a near-duplicate of the query; on heterogeneous corpora with many contemporaneous but semantically distinct items, the gap between the fused score and the cosine baseline may shrink."],"forward_implications":["If the freshness result is right, adding timestamps and a two-line decay term to an existing RAG retriever fixes the common stale-context failure without retraining.","The sensitivity sweep implies the method is robust across a range of recency weightings, but collapses once semantics dominate (α≥0.9), defining an operational boundary.","As-of queries are handled correctly by filtering to documents at or before the requested date, so time-scoped answers need no model changes.","The 0.08 macro-F1 result establishes that heuristic trend labels are unreliable for answering 'what changed' questions, motivating dynamic topic models or density-based alternatives."],"fun_headline_variants":["Half-life recency rescues RAG freshness; heuristic trends stay weak","Tiny decay term flips RAG retrieval from zero to 1.00 on test logs","RAG freshness fixed by adding age; trend detection fails at 0.08 F1","Asymmetric temporal RAG: recency works, heuristic trends don't","Fixing RAG trend labels, not clusters, jumps F1 from 0.08 to 0.49"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the evaluation tasks actually demand freshness: the newest relevant document is the right answer, it is present in the initial candidate pool, and the chosen defaults (α=0.7, half-life=14 days) are not tuned to the two test streams.","fun_headline_variants_meta":{"raw":{"variants":["Half-life recency rescues RAG freshness; heuristic trends stay weak","Tiny decay term flips RAG retrieval from zero to 1.00 on test logs","RAG freshness fixed by adding age; trend detection fails at 0.08 F1","Asymmetric temporal RAG: recency works, heuristic trends don't","Fixing RAG trend labels, not clusters, jumps F1 from 0.08 to 0.49"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3416,"prompt_tokens":700,"completion_tokens":2716,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":2602}},"tokens_in":444,"tokens_out":2716,"duration_ms":20053,"temperature":1.0,"reasoning_tokens":2602,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T16:03:55.143826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single freshness query where the newest relevant item is not in the initial semantic top-K candidate list—so no re-ranking score can surface it—would falsify the claim that the fusion method alone fixes freshness; likewise, a benchmark where a cosine-only retriever already finds the newest relevant item whenever it is semantically closest would show the 1.00-versus-0.00 gap is an artifact of task design.","supporting_citations":[],"review_version":1}