{"id":"eedf2d61-0d14-4fd4-a9d7-c0e3d36321c3","arxiv_id":"2506.17645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Retrieval-augmented in-context learning with nearest-neighbor examples, category guidelines, and GPT-4o feedback improves BLEU/METEOR/ROUGE-L scores for histopathology report generation on HistGen.","lead":"PathGenIC adds retrieval-based in-context learning to a vision-language model for generating histopathology reports from whole slide images. It reports state-of-the-art scores on the HistGen benchmark by feeding the model similar prior cases, category-level guidelines, and feedback.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an asymmetric evaluation: PathGenIC receives GPT-4o-derived feedback at test time that no baseline receives, so the reported gains may reflect privileged information rather than the method's ICL components.","rationale":"The reader's weakest assumption is exactly the fairness asymmetry in the evaluation protocol: PathGenIC queries GPT-4o at test time with retrieved training reports and feedback while no baseline is given equivalent context. My reading of §3.2 and Table 1 confirms this is the load-bearing condition for the central 'state-of-the-art' claim. The concern is concrete: the base model already scores 0.411 BLEU-1 versus HistGen's 0.413, so the published claim depends on a small margin (+0.018 BLEU-1, +0.012 BLEU-4) that is not accompanied by error bars or significance tests, and the paper states in §4.2 that factENT values for all other methods are unavailable, so the largest reported gap is not even comparable. The GPT-4o feedback mechanism (§3.2, Appendix A) transfers knowledge of the training ground-truth reports into the test prompt, which is a genuine protocol asymmetry, not merely a difference in model architecture. I do not allege any misrepresentation; the paper describes its components honestly. The check is therefore to rerun a matched baseline with the same retrieved context and to compute bootstrap confidence intervals, either of which would settle whether the claimed state-of-the-art margin survives a fair comparison. Since the concern is about the strength of the empirical claim rather than the soundness of the method, CONDITIONAL is the right verdict: the paper's core idea is coherent and the ablation is self-consistent, but the published comparisons must be made symmetric and statistically grounded before the headline claim can be taken at face value. I agree with the reader's identification of the weakest assumption and do not see a different, more fundamental flaw in the argument.","tokens_in":8078,"tokens_out":2236,"duration_ms":19223,"concrete_test":"Reproduce Table 1's comparison under a matched protocol: give exactly the same retrieved nearest-neighbor WSI tokens and retrieved ground-truth report to HistGen (or to the base model without GPT-4o feedback) and to PathGenIC. If the base model with retrieved context alone closes the 0.018 BLEU-1 gap to PathGenIC, then the reported SOTA margin is explained by retrieval/document-context rather than by the GPT-4o feedback or category guideline components. Also report the 769-sample 95% bootstrap confidence intervals for the Table 1 BLEU-1, BLEU-4, METEOR, and ROUGE-L entries; if the PathGenIC-versus-HistGen intervals overlap, the 'state-of-the-art' wording should be relaxed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is 'state-of-the-art results' on HistGen (Table 1: BLEU-1 0.431, BLEU-4 0.196, METEOR 0.197, ROUGE-L 0.357, factENT 0.462). The most load-bearing weakness is that the comparison protocol is not apples-to-apples. PathGenIC is the only method that (a) retrieves a nearest-neighbor WSI-report pair, (b) injects GPT-4o-generated category guidelines, and (c) injects GPT-4o-generated corrective feedback computed from each training WSI's generated-versus-ground-truth report comparison. Baselines receive none of this context, and Table 1 reports no base-model-equipped baselines receiving equivalent retrieved information. The base model alone already reaches 0.411 BLEU-1, slightly below HistGen's 0.413, so the absolute headroom over published baselines is small (0.018 BLEU-1, 0.012 BLEU-4 over HistGen). The GPT-4o feedback is created by comparing the base model's generated reports to ground-truth training reports; at test time the model gets feedback for retrieved training WSIs whose 'perfect counterpart' was used during training, meaning test prompts contain supervision-derived knowledge that no baseline sees. This makes 'state-of-the-art' true only under an asymmetric evaluation protocol, and the paper explicitly notes in §4.2 that other methods' factENT values are 'not available', further weakening the one comparison where PathGenIC shows the largest gap. The absence of error bars, significance tests, and released prompts/code makes the 0.018-0.012 margin unverifiable, and the ablation in Table 3 shows the whole ICL stack adds only +0.020 BLEU-1 over the base, i.e., the 'state-of-the-art' claim is partially inherited from the strong base model. This is not an internal inconsistency; it is a fairness and reproducibility risk in the evaluation protocol that directly undercuts the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PathGenIC, a framework for histopathology report generation from whole slide images. The method builds on Quilt-LLaVA with HistGen's DINOv2 ViT-L feature extractor and introduces three in-context learning components: nearest-neighbor retrieval of WSI-report pairs, GPT-4o-generated category guidelines, and GPT-4o-generated corrective feedback derived by comparing base-model outputs against ground-truth training reports. The framework is evaluated on the HistGen benchmark, and the paper claims state-of-the-art results, reporting BLEU-1 0.431, BLEU-4 0.196, METEOR 0.197, ROUGE-L 0.357, and factENT 0.462 for the full PathGenIC model, with ablations over components, the number of retrieved neighbors, report length, and disease categories.","tokens_in":8509,"tokens_out":4515,"duration_ms":51529,"significance":"If the reported results are robust, the paper makes a useful empirical contribution: it demonstrates that retrieval-based in-context learning, category-level guidelines, and feedback derived from training-set comparisons can improve vision-language report generation on whole slide images. The clinical motivation is well grounded, the method description is clear, and the appendices provide the exact prompts used. The ablations and the analysis across disease categories are also valuable. However, the core claim of state-of-the-art performance is weakened by an asymmetric evaluation protocol and the absence of statistical significance measures. PathGenIC is the only method receiving retrieved reports, GPT-4o-generated guidelines, and feedback derived from ground-truth comparisons, so the reported gains may partly reflect additional test-time information rather than the method itself. The improvement margins over HistGen are small, and no error bars or significance tests are reported. These issues prevent the paper from currently supporting its strongest claim, but they are addressable with additional controls and analysis.","major_comments":[{"comment":"The comparison in Table 1 is not apples-to-apples. PathGenIC is the only system that receives (i) retrieved WSI tokens and their associated reports, (ii) GPT-4o-generated category guidelines, and (iii) GPT-4o feedback produced by comparing the base model's output with the ground-truth training reports (Eq. (5)). None of the baselines is given an equivalent retrieval/reference/feedback prompt, so the reported improvements (BLEU-1 0.431 vs 0.413, BLEU-4 0.196 vs 0.184) may come from the extra test-time information rather than from the ICL mechanism. In particular, feedback B_j is derived from ground-truth reports of the retrieved training WSIs and can state which entities the generated report lacks; exposing this to the VLM at test time is a form of privileged information. I do not claim mathematical circularity, since no test ground truth is used, but the 'state-of-the-art' claim is only valid under this asymmetric protocol. Please add controls where the same retrieved reports, guidelines, and feedback are provided to the baselines (e.g., HistGen with identical ICL context), and include a random-retrieval condition to quantify the contribution of retrieval quality.","section":"§3.2, Eq. (5), Table 1"},{"comment":"No measure of variability is reported. All numbers appear to be from single runs; the margins over HistGen are small (BLEU-1 +0.018, BLEU-4 +0.012, METEOR +0.015, ROUGE-L +0.013), and Table 3 shows that adding the guideline and feedback components to nearest-neighbor retrieval changes BLEU-1 by 0.003 and ROUGE-L by 0.003. Without multiple seeds, bootstrap confidence intervals, or significance tests, the claim of 'significant improvements across BLEU, METEOR, and ROUGE-L' in the abstract is unsupported. Additionally, factENT is given only for the two proposed models, with the text noting that values for other methods are 'not available' in §4.2, so the factual-completeness advantage cannot be verified. Please provide standard deviations or significance tests and either obtain factENT for the baselines or remove the implied comparative claim.","section":"§4.1, §4.2, Table 1 and Table 3"},{"comment":"The paper states that the results in Table 1 are obtained 'based on only the first 100 tokens of the generated reports.' It is unclear whether this truncation was applied to the reported baseline numbers taken from HistGen or only to the proposed models. If the baselines were evaluated on full generated reports, the comparison is biased because BLEU, METEOR, and ROUGE-L are sensitive to report length. Please clarify explicitly whether the same truncation was applied to every method, and ideally report results on full generated reports as well.","section":"§4.2, Figure 3 and Table 1"}],"minor_comments":[{"comment":"The category guideline procedure is described as using the K most similar WSIs to determine the majority category, but the prompt in Appendix A lists 'Report 1 ... Report 20'. The relationship between K and the number of reports used to generate the guideline is not described; please clarify how the 20 reports are selected.","section":"§3.2 and Appendix A"},{"comment":"The bar plot in Figure 4 shows BLEU scores across 32 disease categories, but no error bars or sample sizes per category are provided, which makes it hard to determine which cross-category differences are meaningful.","section":"Figure 4"},{"comment":"The motivation cites 'Doe and Smith, 2023' with a webpage reference; a more standard peer-reviewed source on pathology peer review would strengthen the introduction. In addition, the paper does not report the GPT-4o sampling parameters or the computational cost of the feedback and guideline generation, which would aid reproducibility.","section":"References and Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and likely worth publishing after the evaluation protocol and statistical rigor are improved. The main risk is the framing: calling the method state-of-the-art on the basis of a protocol in which only PathGenIC receives retrieved reports, GPT-4o guidelines, and feedback derived from ground-truth comparisons is likely to be contentious. I recommend conditioning acceptance on the proposed controls and on reporting variability. Releasing code or detailed retrieval prompts would also help verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on this paper.\n\nThe useful part is real: the authors take Quilt-LLaVA, swap in HistGen's ViT-L encoder, and show that nearest-neighbor retrieval of WSI-report pairs improves BLEU scores on the HistGen benchmark. That specific combination is not in the cited literature, and the paper is transparent about its prompts and ablations. The ablation in Table 3 is particularly clear: the retrieval component delivers most of the improvement (0.411 to 0.428 BLEU-1), and the guideline and feedback components each add about a thousandth. That's honest and reproducible in spirit.\n\nThe soft spots are mostly about the evaluation protocol. The \"state-of-the-art\" claim rests on a comparison where PathGenIC is the only system that receives retrieved reports, GPT-4o category guidelines, and GPT-4o feedback at test time. No baseline gets an equivalent retrieval prompt or extra context, so the comparison measures a full system against stripped baselines, not the method against comparable methods. The feedback component is the trickiest: it is generated by asking GPT-4o to compare the base model's training outputs to ground-truth reports, and at test time the model sees feedback for retrieved training WSIs. That is not test-set leakage, but it is a supervision-derived hint that no baseline receives, and it makes the \"fair comparison\" claim hard to swallow. Also, the margins over HistGen are tiny——0.018 BLEU-1 and 0.012 BLEU-4——and there are no error bars or significance tests, so I'm not confident those differences are meaningful. The factENT column is reported only for their own models, which weakens the one metric where the gap looks large. Finally, no code is released, though the appendix does include the exact prompts.\n\nNone of this is fatal if the claim is softened. The paper is a coherent engineering study, and the results are likely reproducible. But as written, \"state-of-the-art\" overstates what is demonstrated. A referee should ask for a baseline that also receives retrieved examples and guidelines, repeated runs with error bars, and a clear statement about the privileged information in the feedback.\n\nRecommendation: this deserves a serious referee. The core idea is sound and the issues are identifiable and fixable. I would send it to review with the expectation of major revision, not rejection.","headline":"Clear writing and a sensible ablation, but the SOTA claim depends on an evaluation that gives PathGenIC context no baseline gets; the measured gains are small and unquantified.","tokens_in":9041,"tokens_out":3178,"would_cite":false,"duration_ms":33855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PathGenIC claims top HistGen report scores by adding retrieved examples, guidelines, and feedback to a vision-language model.","keywords":["histopathology report generation","multimodal in-context learning","vision-language model","whole slide image","retrieval-augmented generation","GPT-4o feedback","HistGen benchmark","LoRA fine-tuning"],"falsifier":"Run the Table 1 comparison again but append the same top-1 nearest-neighbor ground-truth report to the prompt of every baseline, including HistGen and the base model, while keeping all other settings fixed; if PathGenIC's BLEU-4 advantage over HistGen shrinks to roughly zero, the state-of-the-art claim is explained by the added retrieval context rather than by the method itself.","tokens_in":7835,"feed_emoji":"🩺","tokens_out":5635,"duration_ms":51686,"temperature":0.7,"pith_summary":"The paper claims that histopathology report generation improves substantially if, before writing a report for a whole slide image, the model is shown similar past cases the way a pathologist would consult them. It proposes PathGenIC, which retrieves the nearest-neighbor WSI-report pair from the training set, obtains a disease-category guideline and corrective feedback produced by GPT-4o, and folds all three into the prompt of a fine-tuned vision-language model. On the HistGen benchmark, the full system reports BLEU-1 0.431, BLEU-4 0.196, METEOR 0.197, ROUGE-L 0.357, and factENT 0.462, the strongest published numbers among the compared methods. The relevance is that report writing, not just image classification, is a bottleneck in pathology workflows, and this is a concrete recipe for extracting more from training data at inference time.","feed_headline":"Retrieved examples push pathology report BLEU-4 to 0.196","feed_subtitle":"PathGenIC feeds a vision-language model similar past cases, disease guidelines, and GPT-4o feedback to draft whole-slide reports.","key_machinery":"The load-bearing mechanism is multimodal in-context learning with three context signals: nearest-neighbor retrieval, where the test WSI's contextualized tokens are concatenated with the tokens of the most similar training WSI and its ground-truth report; category guidelines, where GPT-4o summarizes report-writing habits from the top-K nearest disease category; and feedback, where GPT-4o critiques base-model drafts against truth and the critiques of the K nearest training cases are fed in. These context cues are injected through the text prompt and token stream of a LoRA-adapted Quilt-LLaVA, whose visual front end is HistGen's DINOv2 ViT-L with learnable query tokens.","core_discovery":"The central claim is that in-context learning over the training set, rather than a larger architecture or more parameters, is what lifts report generation quality. Starting from Quilt-LLaVA, the authors replace its visual encoder with HistGen's DINOv2 ViT-L, add learnable query tokens and transformer blocks to make fixed-length WSI tokens, and LoRA-tune the VLM. On top of this base, three context signals are appended: the image tokens and report of the most cosine-similar training WSI, a GPT-4o-generated guideline distilled from up to 20 reports in the nearest disease category, and GPT-4o feedback that compares base-generated training reports against ground truth. Each component alone helps; together they produce the reported state-of-the-art numbers, and the paper shows the gain is not a quirk of report length or disease category.","pith_inferences":["Editorial inference: The reported comparison is not apples-to-apples, because PathGenIC receives GPT-4o-generated text at test time while the baselines receive only the test image; a control that gives every baseline the same retrieved report would determine how much of the gain comes from retrieval versus from GPT-4o's prompt-writing ability.","Editorial inference: The same recipe could transfer to other dense-report domains such as radiology, dermatopathology, or surgical pathology notes, provided a retrieval index and a source of supervisory feedback exist.","Editorial inference: The category guideline prompt asks GPT-4o to expose report-writing biases, which suggests the method could amplify systematic wording or coding biases present in a hospital's historical reports; measuring entity-level calibration rather than only BLEU overlap would test this.","Editorial inference: Because evaluation truncates reports at 100 tokens, the metric gains may not reflect full-report clinical completeness; a clinician-rated study on full-length reports would be the natural next test."],"forward_implications":["If the reported numbers hold, any report generation system on HistGen can be improved by retrieving similar cases at test time without retraining the visual encoder.","The K=3 retrieval setting gives the best tradeoff, so the method is robust to the number of retrieved neighbors rather than requiring exact tuning.","The per-category analysis suggests the gains are consistent across disease types, so the method is not riding on one easy subset.","A direct corollary of the ablations is that each context signal contributes incrementally, with the full combination giving the largest improvement.","Because the base model already outperforms HistGen, part of the state-of-the-art result comes from the improved backbone, while the in-context components provide the additional gains in Table 3."],"supporting_citations":[{"why":"Supplies the HistGen benchmark dataset, the DINOv2 ViT-L feature extractor, and the HistGen baseline numbers that PathGenIC must beat.","marker":"(Guo et al., 2024)"},{"why":"Provides Quilt-LLaVA, the base vision-language model that is fine-tuned into PathGenIC.","marker":"(Seyfioglu et al., 2025)"},{"why":"Establishes in-context learning as the mechanism the framework adapts for medical report generation.","marker":"(Brown et al., 2020)"},{"why":"Motivates retrieval-augmented generation, the retrieval component PathGenIC uses to select similar cases.","marker":"(Lewis et al., 2020)"},{"why":"Supplies the LoRA method used to adapt the VLM efficiently during fine-tuning.","marker":"(Hu et al., 2022)"},{"why":"Provides the factENT entity-match metric and the BioBERT-based evaluation used to measure report completeness.","marker":"(Miura et al., 2021)"},{"why":"Motivates the feedback component of the framework through the diagnosis learning cycle.","marker":"(Branson et al., 2021)"}],"fun_headline_variants":["PathGenIC: in-context learning lifts pathology report BLEU-4 to 0.196","Similar cases and AI feedback sharpen histopathology text","Retrieve, guide, and correct: new method for report generation","ICL with retrieval and feedback boosts WSI reports to SOTA","Pathology reports get better with multimodal in-context examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is only meaningful if giving PathGenIC retrieved example reports, GPT-4o disease guidelines, and GPT-4o feedback at test time does not hand it an advantage that the comparison models were never offered.","fun_headline_variants_meta":{"raw":{"variants":["PathGenIC: in-context learning lifts pathology report BLEU-4 to 0.196","Similar cases and AI feedback sharpen histopathology text","Retrieve, guide, and correct: new method for report generation","ICL with retrieval and feedback boosts WSI reports to SOTA","Pathology reports get better with multimodal in-context examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1209,"prompt_tokens":881,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":235}},"tokens_in":497,"tokens_out":328,"duration_ms":4653,"temperature":1.0,"reasoning_tokens":235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:29:05.402804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Table 1 comparison again but append the same top-1 nearest-neighbor ground-truth report to the prompt of every baseline, including HistGen and the base model, while keeping all other settings fixed; if PathGenIC's BLEU-4 advantage over HistGen shrinks to roughly zero, the state-of-the-art claim is explained by the added retrieval context rather than by the method itself.","supporting_citations":[{"cited_title":"Histgen: A local-global encoding framework for pathology report generation","cited_arxiv_id":null,"evidence_quote":"Supplies the HistGen benchmark dataset, the DINOv2 ViT-L feature extractor, and the HistGen baseline numbers that PathGenIC must beat."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Motivates retrieval-augmented generation, the retrieval component PathGenIC uses to select similar cases."},{"cited_title":"Improving factual completeness and consistency of image-to-text radiology report generation","cited_arxiv_id":null,"evidence_quote":"Provides the factENT entity-match metric and the BioBERT-based evaluation used to measure report completeness."},{"cited_title":"Improving diagnostic accuracy through feedback: The diagnosis learning cycle","cited_arxiv_id":null,"evidence_quote":"Motivates the feedback component of the framework through the diagnosis learning cycle."}],"review_version":1}