ReportLogic: Evaluating Logical Quality in Deep Research Reports

Jujia Zhao; Jun Zhou; Suzan Verberne; Xiaolu Zhang; Zhaochun Ren; Zhaoxin Huan; Zihan Wang

REVIEW 3 major objections 5 minor 9 references

ReportLogic: Evaluating Logical Quality in Deep Research Reports

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the logical quality of deep-research reports is a measurable property, and that a judge trained on rubric-guided preferences — LogicJudge — matches human expert verdicts better than 17 off-the-shelf LLM judges.

desk verdict ReportLogic is a genuinely useful benchmark for report-level logical quality, and LogicJudge is a credible trained judge, but the headline numbers lack error bars and the LLM-generated rubric deserves an external validity check. read the letter →

arxiv 2602.18446 v2 pith:HVTRCYUB submitted 2026-01-27 cs.CL cs.AI

Jujia Zhao , Zhaoxin Huan , Zihan Wang , Xiaolu Zhang , Jun Zhou , Suzan Verberne , Zhaochun Ren This is my paper

classification cs.CLcs.AI

keywords logicevaluationdeepresearchreportsauditabilityrubric-guidedannotationLLM-as-judgepairwisepreferenceadversarialrobustnessbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep-research reports can be fluent and factually accurate yet still fail readers because claims are unsupported, sections do not connect, or conclusions outrun the evidence. This paper argues that this 'logical quality' is a distinct, measurable property, and defines it through the idea of auditability: whether a reader can trace, understand, and verify the report's analytical process. To make that measurable, it builds ReportLogic, a benchmark that decomposes logical quality into a three-layer taxonomy — organization, exposition, and claim–support structure — and instantiates each dimension into instance-specific rubrics that guide human annotators. It then trains an open-source judge, LogicJudge, on distilled pairwise preferences; across three domains, LogicJudge agrees with human expert verdicts (73–75%) more often than 17 off-the-shelf LLM judges or ensemble baselines. If correct, this gives the field a scalable, diagnostic way to evaluate whether LLM-generated reports can be trusted as a basis for decisions.

What carries the argument

The load-bearing mechanism is the auditability lens, which turns an abstract quality into three traceable reader requirements — organization, flow, and argument support — each broken into eight fine-grained dimensions. The second mechanism is context-aware rubric generation: for each query and report pair, a rubric generator produces instance-specific comparison questions, span-level cues, and paired good/bad examples, making the decision boundary explicit. The third mechanism is the judge training protocol: LogicJudge is trained on pairwise preferences distilled from three frontier LLM teachers, kept only when the teachers agree unanimously and when swap-consistency holds, then aligned thro

What would settle it

On a held-out set, take logically weaker reports and make them wordier with outline-style headers and labeled 'evidence' without adding claims; if LogicJudge prefers the padded version about as often as the off-the-shelf judges it claims to beat, its robustness claim fails. A second decisive check: have independent annotators label the same pairs without seeing the LLM-generated rubrics; chance-level agreement with the rubric-guided labels would show the rubric encodes model preferences rather than reader judgment.

Watch

Extended reading notes

Core claim

ReportLogic's central claim is that report-level logical quality — distinct from factual accuracy and surface fluency — can be operationalized and measured. Logical quality is defined as auditability: a reader can trace the report's organization, understand how ideas progress, and verify conclusions against explicit support. This yields a three-layer, eight-dimension taxonomy (Macro-, Expositional-, and Structural-Logic), instantiated by context-aware rubrics that guide expert annotations. LogicJudge, trained on distilled pairwise preferences, reaches 74.5%, 75.0%, and 73.0% agreement with expert verdicts across DeepResearch, Zhihu, and Quora, beating 17 off-the-shelf LLM judges and ensemble

Load-bearing premise

The load-bearing premise is that the human preference labels — gathered under rubrics written by a frontier LLM and trained on labels distilled from three frontier LLM teachers — are a valid ground truth for logical quality; if those LLM-generated rubrics and consensus labels carry systematic blind spots, both benchmark and judge inherit them, and the paper offers no independent audit of rubric quality.

Editorial extensions

If this is right

ReportLogic provides a human-annotated benchmark where logical quality is diagnosed per dimension rather than as a single holistic score.
LogicJudge can act as a scalable automated evaluator, enabling community-wide comparison of deep-research report generators.
Context-aware rubrics raise human inter-annotator agreement from 0.37 (no rubric) to 0.71, so instance-specific criteria are needed for reliable logical judgment.
Off-the-shelf LLM judges are often swayed by verbosity, outline-style scaffolding, and surface 'evidence' labeling, so their logical-quality scores need robustness checks.
Reasoning-optimized judges may fill in missing warrants and thereby hide broken support relations; more test-time reasoning alone does not make a judge more reliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper leaves implicit is using LogicJudge's preferences as a training reward for report generators, turning logical-quality evaluation into an optimization signal.
The taxonomy and bias-type attack suite could transfer to other high-stakes long-form writing, such as policy briefs or regulatory analyses, where explicit warrants are practically required.
Because the rubrics and distilled labels both originate from LLMs, the benchmark's ground truth may encode what frontier models treat as logical rather than an independent reader standard; comparing rubric-guided labels with first-principles annotations would test this.
The bias-type attack categories (length, structure, qualifier wording, evidence illusion, causal display) form a reusable robustness protocol for any LLM-as-judge deployment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it ships a genuinely new benchmark for report-level logical quality, and the trained judge is a credible attempt. Second, the central table has no confidence intervals, and on the smallest test set the headline margin is basically noise. That's the paper in one breath.

What's actually new: nobody has operationalized report-level logical quality as auditability with a three-layer, eight-dimension taxonomy before. The context-aware rubric protocol—instance-specific inspection items generated by Claude-4.5-Sonnet—moves the field forward. It raises human inter-annotator agreement from κ=0.37 with no rubric to κ=0.71. That's a real empirical finding, and it's the strongest piece of evidence in the paper. The bad-case analysis in the appendix is also excellent: the three failure modes (numerical accounting violations, smooth but checklist-like transitions, missing warrants) are concrete and the suggested minimal fixes are useful.

LogicJudge itself is a reasonable distillation effort. The consensus and swap-consistency filtering makes sense, and the two-stage SFT+GRPO with format-first rewards is a prudent design. The adversarial attack study correctly shows that off-the-shelf judges are vulnerable to length bias and evidence illusion—that's a useful warning for the community.

Now the soft spots, in proportion.

1. No error bars anywhere. Table 1 reports agreement accuracies without confidence intervals or significance tests. DeepResearch has only 102 human-labeled test pairs. LogicJudge's 74.5% is within a typical 95% CI of qwen3-max's 73.5%, and the margins on Zhihu and Quora are also thin. Without statistical tests, the paper cannot actually claim to beat the baselines—it can only show a direction.

2. Silent exclusions. The appendix says unparsable baseline outputs are dropped, but doesn't report how often that happens. If a baseline fails to produce a parseable preference on a nontrivial fraction of cases, its accuracy is computed on a biased subset. That's fixable, but it's an omitted detail.

3. The circularity worry about LLM-generated rubrics is legitimate but not fatal. The humans are guided by Claude-4.5-Sonnet rubrics, and the distilled training labels come from three other frontier models. If the rubric carries systematic blind spots, those blind spots could propagate into the gold labels. But the humans are experts and can overrule the rubric; the rubric just makes their decisions more consistent. Still, an external validity check—say, a subset annotated with human-written rubrics or no rubrics at all—would strengthen the construct validity claim substantially. As it stands, the paper shows reliability but not independent validity.

4. Release: the ethical statement says they release ReportLogic and LogicJudge, but the version I read has no link or commit hash. That needs to be resolved before anyone can build on it.

Who should read this: anyone building deep-research systems, evaluation pipelines, or long-form generation. It deserves a serious referee. My recommendation: accept with revisions. Require confidence intervals, exclusion rates, a direct release link, and ideally a small rubric-validity study. The core contribution is solid and will be cited.

Referee Report

3 major / 5 minor

Summary. ReportLogic introduces a human-annotated benchmark for evaluating the logical quality of deep-research-style reports, operationalizing logical quality as auditability through a three-layer, eight-dimension taxonomy (Macro-Logic, Expositional-Logic, Structural-Logic). The construction pipeline uses Claude-4.5-Sonnet to generate instance-specific rubrics that guide three expert annotators in pairwise preference labeling. The paper additionally trains LogicJudge, an open-source Qwen-3-30B-A3B-based judge, via SFT and GRPO on distilled labels produced by unanimous o3/GPT-5/Gemini-2.5-Pro consensus with swap-consistency filtering. Experiments report that LogicJudge achieves the highest agreement with human labels on three domains (DeepResearch 74.5%, Zhihu 75.0%, Quora 73.0%) against 17 off-the-shelf LLM judges and ensemble baselines. Ablations show context-aware rubrics improve inter-annotator agreement, and adversarial attacks reveal that many judges are sensitive to verbosity and other surface cues. The paper positions ReportLogic and LogicJudge as a reusable diagnostic evaluation tool for deep-research report logic.

Significance. If the benchmark is valid, it fills a real gap: existing long-form evaluation focuses on factuality, citation, and fluency, not on whether a report's claims and arguments can be traced, understood, and verified. The paper's strengths include a clearly specified taxonomy grounded in argumentation theory, a careful annotation protocol with adjudication, a strict position-controlled agreement metric, swap-consistency filtering for distilled training data, an open-source judge model, and a manually screened adversarial attack suite. These are substantive contributions. However, the central claims are conditional on two issues: the absence of statistical support for the headline ranking, and the lack of an external validity check on the LLM-generated rubrics that anchor both the human gold labels and the distilled training signal. The paper is transparent about its limitations and intended use, which is commendable.

major comments (3)

[§5.2, Table 1, Table 3] The headline claim that LogicJudge achieves the highest agreement is not statistically supported. The DeepResearch human-labeled test set contains only 102 pairs (Table 3). LogicJudge's 74.5% equals 76 correct; qwen3-max's 73.53% equals 75 correct—a one-instance margin. Several baselines (e.g., gemini-2.5-pro at 68.63%) are within a few instances. No confidence intervals, significance tests, or bootstrap estimates are reported for any domain. Please report exact test counts, CIs (e.g., Wilson) or bootstrap intervals, and pairwise significance tests for the leading entries. Without this, the ranking in Table 1 should be treated as provisional.
[§3.2, §4.2, Table 2] The validity of the human gold labels is not externally anchored. Annotators judge using rubrics generated by Claude-4.5-Sonnet (Fig. 10), and LogicJudge is trained on unanimous labels from o3/GPT-5/Gemini-2.5-Pro after swap-consistency filtering. The rubric ablation (Table 2) shows that context-aware rubrics increase inter-annotator reliability (κ=0.71 vs 0.37 no-rubric), but reliability does not demonstrate that the rubric items capture logical quality rather than surface preferences induced by the LLM generator. If the rubric has systematic blind spots, they propagate to both the gold labels and the distilled training signal, so the reported agreement may reflect alignment with LLM-induced preferences. Please add an external audit, e.g., compare against human-authored rubrics or independent holistic expert judgments on a sample, or validate against known injected logical defects, and
[§5.1, Table 3, Appendix B.1] The dataset statistics are internally inconsistent and must be reconciled. §5.1 states 1,319 DeepResearch queries; Table 3 reports 1,204 queries but the train/val/test split sums to 1,089; Appendix B.1 says the original benchmark contains only 102 queries and that synthetic variants were used only in training. The test set has 102 human-labeled pairs, consistent with 17 test queries, but the paper should state clearly how many of the 1,204/1,319 queries are original vs synthetic, and give the exact human-labeled test count per domain in Table 1, since the n=102 figure is critical to the significance claim above.

minor comments (5)

[Figure 2] Typographical errors: "Tracability" and "Understanbility" in the taxonomy labels should be corrected.
[§2.1] Minor grammar: "primary focuses" should be "primary focus." Also, the related-work description of DeepResearch benchmarks would benefit from exact version/source identifiers.
[Appendix B.2, Table 1, Figure 4] Baseline model names are inconsistent across text and figures: B.2 mentions Claude-3.5-Sonnet and Sonnet-3.5-Think, while Table 1 lists claude-4-sonnet and claude-4.5-sonnet; Figure 4 uses "gemini-3-pro-preview" and "claude-sonnet-4". Please unify model naming throughout.
[Table 1] For Ensemble Consensus, the paper says it abstains when not unanimous, but does not state how abstentions are treated in the reported percentage (excluded from the denominator or counted incorrect). This makes the 49.02% value ambiguous.
[§5.4, Figure 4] The text says "Quora omitted for brevity" but Figure 4 shows only two panels; clarify whether the Quora result appears in an appendix or the claim is based on the two shown domains only.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LogicJudge is validated against held-out human labels; LLM-rubric provenance is a validity caveat, not a construction-level reduction.

full rationale

ReportLogic's central claim is that LogicJudge aligns with human expert preferences on a held-out test split. The human gold labels are produced by three expert annotators with majority voting and are reserved exclusively for the test split (Table 3); LogicJudge is trained on a separate distilled corpus (§4.2) and never fitted to the test labels. The distillation teachers (o3, GPT-5, Gemini 2.5 Pro) and the rubric generator (Claude-4.5-Sonnet) are LLMs, so the benchmark's construct is not fully independent of LLM-generated artifacts; however, the test labels are not by construction equal to the training labels or to the rubrics. The taxonomy draws on external frameworks (Toulmin, Halliday, Doshi-Velez & Kim), and no self-citation carries the argument. The absence of an external audit of rubric validity is a legitimate construct-validity caveat, but it is not an equivalence-by-construction between prediction and input. Therefore no circular step is established.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fitted: this is a benchmark-and-evaluation paper rather than a parametric derivation. The load-bearing choices are domain assumptions about what counts as logical quality and about the validity of LLM-generated rubrics and distilled labels. No new physical or conceptual entities are postulated.

assumptions (4)

domain assumption Logical quality of a Deep Research report is adequately operationalized by reader-centric auditability, decomposed into Macro-, Expositional-, and Structural-Logic with eight dimensions.
This is the definitional grounding of the benchmark; if a reader-relevant logical dimension is missing, ReportLogic under-measures. Stated in §3.1 and used throughout.
domain assumption Claude-4.5-Sonnet-generated context-aware rubrics provide unbiased, instance-specific decision boundaries for both human annotators and LLM judges.
These rubrics guide the human gold labels and are also fed to LogicJudge; no independent audit of rubric-generator blind spots is provided. Introduced in §3.2.
domain assumption Majority vote of three trained annotators, with expert adjudication, yields reliable gold preferences for logical quality.
Inter-annotator agreement (κ=0.71, 81.7% pairwise) supports reliability but does not prove it; the annotator pool is small and shares a professional background. Described in §3.2, §5.4, Appendix E.
domain assumption Frontier-model consensus labels, after consensus and swap-consistency filtering, are a valid training signal for logical preferences.
If the three teacher models share surface or stylistic biases, LogicJudge inherits them. Used for SFT and GRPO training in §4.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ReportLogic: Evaluating Logical Quality in Deep Research Reports." pith.science (2026). https://pith.science/paper/HVTRCYUB

@misc{pith2026260218446,
  author       = {Pith},
  title        = {Pith review of: ReportLogic: Evaluating Logical Quality in Deep Research Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HVTRCYUB}},
  note         = {Machine review of arXiv:2602.18446}
}

read the original abstract

Users increasingly rely on Large Language Models (LLMs) for Deep Research, using them to synthesize diverse sources into structured reports that support understanding and action. In this context, the practical reliability of such reports hinges on logical quality: whether the report's claims and arguments are explicitly supported and can be trusted as a basis for downstream use, rather than merely appearing fluent or informative. However, current evaluation frameworks largely overlook this requirement. To bridge this gap, we introduce ReportLogic, a benchmark that quantifies report-level logical quality through a reader-centric lens of auditability. Specifically, ReportLogic adopts a hierarchical taxonomy that evaluates whether readers can (1) trace an on-topic report structure with a unified analytical arc (Macro-Logic), (2) understand the progression with necessary context (Expositional-Logic), and (3) verify conclusions via explicit claim--support (Structural-Logic). Based on this taxonomy, we construct a human-annotated rubric-guided dataset and train an open-source LogicJudge for scalable evaluation. We further evaluate judge robustness via adversarial attacks, showing that off-the-shelf LLM judges are frequently influenced by superficial cues (e.g., verbosity), and reasoning modes can mask broken support relations. Overall, our results provide actionable guidance for building more robust logic evaluators and improving the logical reliability of LLM-generated reports.

Figures

Figures reproduced from arXiv: 2602.18446 by the authors.

**Figure 1.** Comparison between existing evaluation views and ReportLogic on a Deep Research report. action (Shi et al., 2025; Huang et al., 2025). In this context, the utility of the output report extends beyond producing correct facts or well-formed passages. For example, a report may cite accurate statistics and remain well written, yet still be unhelpful when central claims are not adequately supported, leaving readers un… view at source ↗

**Figure 2.** ReportLogic framework. We define logical quality as auditability and decompose it into a three-layer [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** ReportLogic Leaderboard. Heatmap of win-rates for 16 frontier models across three domains. Darker [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Ablation Study on Rubric Effectiveness. ventions and topics. (2) Longer inference does not reliably improve logical quality. Reasoningstyle (“think”) variants do not consistently outperform their base counterparts and sometimes underperform. A plausible explanation …

**Figure 5.** Figure 5: Attack analysis of judge robustness. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Bad-case visualization for Deep Research-style reports. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: Isolation Ratio (IR) under targeteddimension attacks. C.4 Attack Analysis Details This appendix provides additional details for the attack-based evaluation in RQ3, including adversarial construction protocols, metrics, and extended interpretations. The main paper em…

**Figure 8.** Figure 8: Prompts used to filter open-domain queries for Deep Research suitability. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

**Figure 9.** Figure 9: Prompt used for Deep Research style long-form report generation. [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

**Figure 10.** Figure 10: Prompt used to generate context-aware, instance-specific rubric for eight-dimensional logical evalua [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: Prompt used to distill rubric-guided, dimension-level pairwise preferences for training LogicJudge. [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]

**Figure 12.** Figure 12: LogicJudge Training Prompt [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 2 linked inside Pith

[1]

Task alignment & claim clarity
[3]

Internal consistency
[4]

Concept introduction & logical transition
[6]

Evidence sufficiency & relevance
[7]

Warrants & causal reasoning
[8]

aspect_evaluations

Qualifiers & counterpoints Input. QUERY:{QUERY} TEXT A:{TEXT A} TEXT B:{TEXT B} Logic Rubric (JSON):{RUBRIC JSON} Output Format (STRICT).The model must outputexactly one valid JSON objectfollowing the schema below. This structured format explicitly supervises both dimension-level reasoning and the final preference decision, enabling fine- grained analysis...
[2017]

aspect_evaluations

Computational argumentation quality assess- ment in natural language. InProceedings of the 15th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Volume 1, Long Papers, pages 176–187. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, and 1 others. 202...

arXiv 2024
[2024]

InProceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pages 8211–8227

Towards verifiable text generation with evolv- ing memory and self-reflection. InProceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pages 8211–8227. Stephen E Toulmin. 2003.The uses of argument. Cam- bridge university press. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Al- ber...

2024

Show all 9 references

[2025]

Finale Doshi-Velez and Been Kim

Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models.arXiv preprint arXiv:2501.14851. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. Mingxuan Du, ...

2017 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Task alignment & claim clarity

[2] [3]

Internal consistency

[3] [4]

Concept introduction & logical transition

[4] [6]

Evidence sufficiency & relevance

[5] [7]

Warrants & causal reasoning

[6] [8]

aspect_evaluations

Qualifiers & counterpoints Input. QUERY:{QUERY} TEXT A:{TEXT A} TEXT B:{TEXT B} Logic Rubric (JSON):{RUBRIC JSON} Output Format (STRICT).The model must outputexactly one valid JSON objectfollowing the schema below. This structured format explicitly supervises both dimension-level reasoning and the final preference decision, enabling fine- grained analysis...

[7] [2017]

aspect_evaluations

Computational argumentation quality assess- ment in natural language. InProceedings of the 15th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Volume 1, Long Papers, pages 176–187. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, and 1 others. 202...

arXiv 2024

[8] [2024]

InProceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pages 8211–8227

Towards verifiable text generation with evolv- ing memory and self-reflection. InProceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pages 8211–8227. Stephen E Toulmin. 2003.The uses of argument. Cam- bridge university press. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Al- ber...

2024