{"id":"21ef88e4-249c-4a5e-b5a9-647df623d3bd","arxiv_id":"2507.22927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PRGB introduces a placeholder-based, fine-grained RAG benchmark that evaluates LLMs on filtering, combination, and multi-hop reasoning, with English and Chinese datasets.","lead":"PRGB is a new benchmark that tests how well language models use retrieved documents in RAG systems, with filtering, combination, and reasoning tasks. It uses placeholder substitution to hide true answers, aiming to stop models from simply recalling memorized facts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central decoupling claim rests on an unvalidated assumption: no experiment compares original values with placeholders, so 'solely relies on external information' is unsupported; a leakage control would settle it.","rationale":"The load-bearing condition for PRGB's validity is that placeholders remove parametric knowledge. The reader identified the same assumption as weakest, and I agree. The paper contains no test of this condition; the ablation in the 'Placeholders' section examines answer stability across placeholder values but not whether a model would have answered from memory. The abstract and Section 'Dynamic Placeholder Substitution' make a strong causal claim, so the missing control is a correctness risk rather than a scope limitation. Other issues (GPT-4o as judge when GPT-4o is also evaluated; code link in abstract vs. 'code and data will be released later' in the conclusion; small inconsistencies in Table 1 and no dataset release) reinforce the need for revision but are secondary. The proposed leakage experiment is standard, feasible with the authors' existing pipeline, and would directly validate or refute the central claim. Since the concern is addressable and the benchmark's multi-level task structure remains useful, the appropriate verdict is CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":10555,"tokens_out":4810,"duration_ms":53316,"concrete_test":"Stratified sample of ~500 English items across filtering, combination, and reasoning. For each item, build two versions: (A) original answer value v; (B) one placeholder candidate p_v from the benchmark candidate set. Keep documents identical except for the target value. Run the Table 1 models on both versions. Measure (1) exact-match accuracy per version; (2) leakage rate: fraction of (B) responses containing v (the value no longer in the document) instead of p_v; (3) rank correlation between A and B accuracies across models. If B accuracy is substantially above chance, leakage >5%, or A accuracy strongly predicts B accuracy, the placeholders are not decoupling parametric knowledge. Report leakage per task and per model.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that dynamic placeholder substitution makes evaluation robust by ensuring model performance 'solely relies on their ability to utilize external information' (abstract and Section 'Dynamic Placeholder Substitution'). The section asserts two mechanisms: (a) it 'reduces bias from the model's internal knowledge during evaluation' and (b) repeated testing 'lowers the probability of models guessing correct answers.' However, no experiment in the paper compares the same items with original values vs. placeholder values, and no leakage metric is reported. The later 'Placeholders' subsection shows that changing placeholders changes outcomes for about 30% of partially correct cases, but sensitivity to placeholder changes is not evidence that parametric knowledge is suppressed; it only shows the model sometimes follows the document. Without a control, correct answers could still be produced by parametric priors over candidate values, especially because candidates are generated by GPT-4o and Qwen2.5-MAX and may be distributionally biased. The claim is therefore internally unsupported, not merely outside consensus. The missing control is concrete and addressable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PRGB, a bilingual (English/Chinese) benchmark for evaluating retrieval-augmented generation (RAG) with three fine-grained dimensions—multi-level filtering, combination, and reference reasoning—built from a triplet-based metadata graph. Its central methodological proposal is a dynamic placeholder substitution algorithm that replaces answer values in golden documents with type-consistent placeholders and candidate values, intended to suppress LLMs' parametric knowledge so that evaluated performance 'solely relies on their ability to utilize external information.' The authors report construction statistics (3,887 English and 3,387 Chinese samples), a GPT-4 and Qwen-assisted synthesis pipeline with manual validation, and experiments on a range of closed- and open-source LLMs, concluding that reasoning models handle noisy contexts better while non-reasoning models are prone to misleading answers. The paper also includes ablation-style analyses of noise levels and placeholder sensitivity.","tokens_in":10789,"tokens_out":5074,"duration_ms":53646,"significance":"If the decoupling claim were validated, PRGB would be a useful contribution to RAG evaluation: it targets LLM-specific document-utilization abilities rather than end-to-end RAG scores, provides a finer-grained task taxonomy than RGB or RAGBench, and ships public code and data. The explicit triplet-to-task construction, the bilingual scale, and the manual validation step are genuine strengths, and the observation that model rankings differ across filtering, combination, and reasoning is interesting. However, the central methodological claim—that placeholder substitution removes parametric-knowledge bias—is asserted but not directly tested, and the evaluation pipeline has a self-referential component because GPT-4o is both a data generator/judge and an evaluated model. These issues must be addressed before the benchmark's headline conclusions can be accepted.","major_comments":[{"comment":"The paper's core claim that placeholder substitution makes performance 'solely rely on their ability to utilize external information' (Abstract and Section 'Dynamic Placeholder Substitution') is not directly tested. The section asserts that replacing values (a) reduces bias from internal knowledge and (b) lowers the probability of guessing, but no experiment compares the same items with original values against placeholder versions, and no leakage metric is reported. The 'Placeholders' discussion and Figure 2 only show that changing placeholders changes outcomes for about 30% of partially correct cases; sensitivity to placeholder changes is consistent with models partially following the document and does not establish that parametric knowledge is suppressed. Please add a control condition (for example, original-value items versus placeholder items, stratified by entity popularity or recency) and report agreement between placeholder-based scores and scores on held-out original items.","section":"Dynamic Placeholder Substitution"},{"comment":"GPT-4o is used to generate placeholder documents and candidate values ('Placeholder-based Candidate Values Generation'), to judge answers ('Evaluation Metrics': 'we directly let GPT4o judge whether the answer is correct'), and is itself one of the evaluated models (Table 1, GPT4o-1120 row). This creates a self-referential evaluation loop: the benchmark's lead model may be rewarded for answers that match GPT-4o's preferred phrasing or its own generated candidate distributions. Please either exclude the judge model from the evaluated set, use a judge not involved in data generation, or report per-model agreement between Covered Exact Match and GPT-4o judgment to show that the judgment is not a GPT-4o self-preference artifact.","section":"Evaluation Metrics / Table 1"},{"comment":"The abstract claims that experiments demonstrate limitations 'particularly in error resilience and context faithfulness,' but no context-faithfulness metric is defined anywhere in the paper. The only metrics described in 'Evaluation Metrics' are covered exact-match accuracy and GPT-4o correctness judgment; no faithfulness or error-resilience measure is specified or reported. Please either add and report the missing faithfulness metrics or revise the abstract and conclusion to state precisely which capabilities were measured.","section":"Abstract / Conclusion / Evaluation Metrics"},{"comment":"Algorithm 1 takes n < N candidate placeholders as input but computes Score = AVG(metric(result_{1,...,N}, GT)) over N, while the main experiments set three placeholders despite the datasets averaging 4.39–4.68 candidate values per sample (Benchmark Statistics). This inconsistency makes the exact scoring procedure ambiguous. Please clarify how the averaging is computed and whether using n=3 introduces sampling variance, and report standard errors or bootstrap intervals for the headline scores in Table 1, especially for models separated by only a few points.","section":"Algorithm 1 / Experimental Setup / Benchmark Statistics"},{"comment":"The manual validation step starts from samples that 'multiple models fail to answer correctly' under the simplest condition and then corrects approximately 30% of the dataset. Because the same model families are later evaluated on the corrected data, this creates a model-dependent data-selection step; the paper gives no evidence that the corrections do not systematically favor or penalize particular architectures. Please report the fraction of corrected samples per model family and check whether headline rankings are stable after excluding corrected samples.","section":"Data Quality Validation"}],"minor_comments":[{"comment":"The text has several typographical errors, including 'toevaluate' and 'isinsufficient' in the Introduction; these should be corrected.","section":"Introduction"},{"comment":"In the Related Work section, REALM is cited as '(Lewis et al. 2020)', but the reference list correctly attributes REALM to Guu et al. (2020); please fix this citation.","section":"Related Work"},{"comment":"The abstract and conclusion state different emphases: the abstract mentions 'error resilience and context faithfulness' while the conclusion emphasizes 'mitigating the interference of knowledge stored in the model's internal parameters'; please align these statements with the actual metrics used.","section":"Benchmark Statistics"},{"comment":"Model names are written inconsistently (for example, 'GPT4o' versus 'GPT-4o' and 'Qwen' versus 'Qwen3'); please standardize the notation throughout.","section":"Experimental Setup"},{"comment":"The 'Covered Exact Match' metric is described informally; a formal definition, including how the 'or' and 'and' operators are applied to keyword sets, would improve reproducibility.","section":"Evaluation Metrics"},{"comment":"The 'Placeholders' subsection reports that about 30% of partially correct answers are affected by placeholder changes, but this result is presented without confidence intervals or a description of how many samples underlie Figure 2; please add this information.","section":"Placeholders"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a benchmarking venue, but the central decoupling claim needs experimental validation before the work can be accepted. The missing original-versus-placeholder control and the GPT-4o judge/evaluated-model overlap are load-bearing and should be handled in revision. I would not recommend rejection because the benchmark design is otherwise concrete and reproducible, and these issues are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid, well-scoped benchmark paper for RAG evaluation, but the central claim about placeholder substitution needs a control experiment before I'd trust the decoupling story.\n\nWhat's genuinely new: the three-way split of document-utilization skills (multi-level filtering, combination, reference reasoning) is more granular than RGB or RAGBench, and the placeholder substitution idea—replacing answer values with type-matched arbitrary values so models can't rely on memorized facts—is a clever workaround for the parametric-knowledge confound. The construction pipeline is explicit: triplet seeds from Wikipedia, propagated child entities, per-task noise levels, manual validation of a chunk of the data. Two languages, 7k+ samples, and a reasonable set of ablations on noise ratios and placeholder counts. That's real work, and it's reproducible in principle.\n\nThe soft spots are concentrated around the main claim. There is no experiment that compares the same items with original values versus placeholders. The placeholder section shows that changing placeholders changes outcomes for ~30% of partially-correct cases, but that only shows models sometimes follow the document—it doesn't show parametric knowledge is suppressed. Without a control, a model could still be guessing from a distributionally-biased candidate set (generated by GPT-4o and Qwen2.5-MAX). Second, GPT-4o both generates data, judges answers, and is itself evaluated; that's a self-reference loop that should at least be acknowledged with an external judge or agreement analysis. Third, no head-to-head comparison against existing benchmarks (RGB, RAGBench, RAGAS) is provided, so it's hard to tell whether the new dimensions measure something other benchmarks miss. Also, the abstract says code is available, the conclusion says it will be released later, and the numbers in Table 1 look off for Gemma3-12b (89.92 for combination on ZH?); needs a pass.\n\nThese are all addressable. The core architecture is sensible, the data is probably useful, and the benchmark could become a solid evaluation tool if the authors add the original-vs-placeholder control, switch to an external judge or at least report agreement, fix the inconsistencies, and release the dataset.\n\nWho it's for: people building or evaluating RAG systems, especially those who care about separating what the model knows from what it reads. I'd give it a serious referee rather than desk reject, but I'd push for a major revision.\n\nRecommendation: send it to review, conditional on the control experiment being added.","headline":"A well-built RAG benchmark whose headline 'decoupling' claim needs a control experiment before it fully lands.","tokens_in":11318,"tokens_out":1750,"would_cite":true,"duration_ms":16726,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a placeholder-based benchmark that replaces answer values in retrieved documents with type-consistent placeholders, so that a model's score reflects its ability to use external information rather than its memorized…","keywords":["retrieval-augmented generation","benchmark","placeholder substitution","parametric knowledge decoupling","multi-level filtering","combination ability","reference reasoning","LLM evaluation"],"falsifier":"Take a sample of benchmark questions and run them twice: once with the original answers in the documents and once with placeholder substitutions. If models answer the original versions correctly even when no document is supplied, or if accuracy on the placeholder versions shifts when the candidate values are chosen to be common versus rare in pretraining data, then the decoupling is incomplete.","tokens_in":10356,"feed_emoji":"🔍","tokens_out":5598,"duration_ms":51986,"temperature":0.7,"pith_summary":"This paper is trying to establish a way to measure whether an LLM actually uses retrieved documents in a RAG system, rather than falling back on knowledge memorized during pre-training. It introduces a benchmark whose questions are built from Wikipedia entity triplets and graded along three progressive dimensions: multi-level filtering of noise, combination of scattered facts, and reference reasoning across documents. The central move is a dynamic placeholder substitution: the answer value in a document is replaced with an arbitrary but type-consistent placeholder, and each question is repeated with several candidate values. The authors argue this makes evaluation robust because the model cannot rely on having seen the answer before; its score then reflects genuine document utilization. If this works, it gives the community a reproducible way to compare RAG generation models on their actual grounding ability rather than on memorization.","feed_headline":"Placeholders hide answers to expose how well RAG models really read","feed_subtitle":"A new benchmark swaps answer values for placeholders, forcing models to use retrieved documents, not memorized facts.","key_machinery":"The central object is the dynamic placeholder substitution: for each golden triplet $(e, p, v)$, the answer value $v$ is replaced by a placeholder to form $(e, p, \\text{Placeholder})$, documents are generated around that placeholder, and multiple candidate values of the same datatype are produced. The evaluation pipeline repeats each question with several placeholder values and averages the scores, which is what blocks a model from guessing the correct answer from memory. The benchmark's task taxonomy—multi-level filtering (weak, moderate, hard noise), combination (explicit, multi-value, multi-scenario), and reference reasoning (comparative, deductive, comparative-deductive)—is built on triplet-based metadata from parent and child entities. Together, the placeholder mechanism and the graded task dimensions are what the paper claims makes the benchmark a fine-grained and robust test of document utilization.","core_discovery":"The paper's central claim is that a placeholder-based benchmark can decouple an LLM's parametric knowledge from the external knowledge in retrieved documents, so that measured performance 'solely relies on their ability to utilize external information.' To support this, it constructs 3,887 English and 3,387 Chinese evaluation samples from 224 parent entities, 2,272 child entities, and 16,033 triplets, and organizes tasks into multi-level filtering, combination, and reference reasoning. It reports experiments on a range of state-of-the-art models showing consistent limitations in error resilience and context faithfulness, and it shows that filter ability does not scale monotonically with model size while combination and reasoning do favour larger models.","pith_inferences":["If placeholder substitution truly removes parametric memory, the same trick could be adapted to test knowledge contamination over time: a benchmark whose answers are periodically replaced with fresh placeholders should stay valid even as models absorb the original data.","The method implicitly assumes candidate values are neutral; a practical extension would be to measure and correct for distributional bias, for example by checking whether models prefer candidate values that are statistically more frequent in their training data.","The approach could transfer to non-textual RAG settings, such as retrieval over code or tabular data, wherever a 'value' can be swapped while preserving the surrounding structure."],"forward_implications":["If the placeholder decoupling works, any LLM can be retested as new models appear, and scores will continue to measure grounding rather than memorization.","The benchmark's noise library can be reused to evaluate retrieval models, since the same noise types (similar entities, generalizing parent cases) stress retrieval as well as generation.","The finding that filter ability does not scale with model size suggests that smaller models can be competitive in needle-in-a-haystack settings, which bears on model selection for RAG deployment.","Reasoning-mode and non-reasoning-mode differences indicate that chain-of-thought style reasoning changes how models handle noisy documents, a factor RAG evaluation should report explicitly."],"supporting_citations":[{"why":"Defines basic RAG abilities such as noise robustness and document integration; this benchmark refines those dimensions into a finer-grained taxonomy.","marker":"Chen et al. 2023"},{"why":"RAGBench is the most direct prior framework for evaluating LLM generation within RAG, which the paper argues is too coarse and topically skewed.","marker":"Friel, Belyi, and Sanyal 2025"},{"why":"POPQA motivates the need to avoid parametric knowledge by using less popular facts; the placeholder approach extends this idea to arbitrary entities and values.","marker":"Mallen et al. 2023"},{"why":"Supplies evidence that retrieval quality alone does not guarantee generation accuracy, supporting the need for a generation-focused benchmark.","marker":"Min et al. 2020"},{"why":"RAGAS demonstrates LLM-based evaluation metrics for context relevance and faithfulness, which the paper positions as complementary but less granular.","marker":"Es et al. 2024"},{"why":"ARES provides an alternative automated NLI-based evaluation framework for RAG, used as a point of comparison for the proposed approach.","marker":"Saad-Falcon et al. 2023"}],"fun_headline_variants":["Benchmark hides answers to reveal if RAG models truly use docs","Placeholder-based benchmark exposes RAG models' weak reading habits","New test masks values to show when RAG leans on memory, not context","PRGB: Hiding answers in benchmarks to force real document use","Benchmark decouples memorized facts from retrieved knowledge in RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that replacing an answer with an arbitrary but type-consistent placeholder value removes the model's memorized knowledge without changing the difficulty of the question.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark hides answers to reveal if RAG models truly use docs","Placeholder-based benchmark exposes RAG models' weak reading habits","New test masks values to show when RAG leans on memory, not context","PRGB: Hiding answers in benchmarks to force real document use","Benchmark decouples memorized facts from retrieved knowledge in RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1687,"prompt_tokens":897,"completion_tokens":790,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":513,"tokens_out":790,"duration_ms":8797,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:43:44.246174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of benchmark questions and run them twice: once with the original answers in the documents and once with placeholder substitutions. If models answer the original versions correctly even when no document is supplied, or if accuracy on the placeholder versions shifts when the candidate values are chosen to be common versus rare in pretraining data, then the decoupling is incomplete.","supporting_citations":[],"review_version":1}