{"id":"e80a1dbe-2cb6-45f9-84bc-e0e4f8d82e6b","arxiv_id":"2602.18446","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An auditability-based benchmark with three logic layers and eight dimensions shows a distilled judge agrees with human experts ~74-75% versus ~62-74% for frontier LLM judges.","lead":"This paper introduces ReportLogic, a benchmark that scores AI-generated deep-research reports on whether readers can trace, understand, and verify their arguments, using eight dimensions grouped into three levels. It also trains an open judge called LogicJudge that agrees with human experts more often than off-the-shelf models and is less fooled by verbosity and formatting tricks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human gold labels are generated under Claude-4.5-Sonnet rubrics (§3.2) and LogicJudge is trained on o3/GPT-5/Gemini consensus (§4.2); without external audit of the rubrics, the reported agreement may be circular.","rationale":"I read the paper in good faith: ReportLogic is a serious benchmark effort, the taxonomy is well-motivated, the adversarial study is thoughtful, and LogicJudge's results are internally consistent. I also credit the rubric ablation (Table 2) and the multi-annotator/adjudication protocol as real evidence that the protocol reduces noise. However, those strengths do not address the construct-validity concern. The central empirical claim is that LogicJudge aligns with human expert preferences better than 17 baselines. For that claim to mean what it says, the human preferences must be independent evidence of logical quality, not preferences manufactured by the rubric generator. The paper's own ablation shows the rubrics change human judgments enough to move inter-annotator agreement from 0.37 to 0.71; that is strong evidence of influence, which makes the absence of an external validity check more important. I do not think this is a fatal flaw: the concern is concrete and testable, and it could be remedied with an independent annotation study and by releasing artifacts with a stable commit hash. The small DeepResearch test set (102 pairs) and silent exclusion of unparsable baseline outputs are additional reasons for caution, but the rubric provenance is the single load-bearing issue because it threatens the benchmark's interpretation rather than just its error bars. Hence I keep the reader's CONDITIONAL verdict; no adjustment beyond what the reader already recommended.","tokens_in":26933,"tokens_out":5664,"duration_ms":64628,"concrete_test":"Independently annotate 150–200 ReportLogic test pairs (stratified across DeepResearch/Zhihu/Quora) with a fresh panel of experts using a human-authored, pre-registered rubric for the same eight dimensions, with no exposure to the Claude-4.5-Sonnet rubrics. Compare these labels with (a) the published gold labels and (b) LogicJudge's predictions using preference-match rate and kappa. If the independent labels disagree with the published human labels substantially (e.g., >10 percentage points) or if LogicJudge's agreement with the independent labels drops materially below its reported 74.5–75.0%, the LLM-generated rubrics are shaping the ground truth and the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that ReportLogic's gold labels are valid measures of report-level logical quality and that LogicJudge's high agreement reflects alignment with that quality. The weakest point is the provenance of the labels. Every human annotation is guided by instance-specific rubrics produced by Claude-4.5-Sonnet (§3.2, Fig. 10), and LogicJudge's training supervision is produced by unanimous o3/GPT-5/Gemini-2.5-Pro consensus plus swap-consistency filtering (§4.2). Both the gold standard and the training signal therefore pass through LLM-generated artifacts. The paper shows that the context-aware rubric raises inter-annotator agreement (Table 2: κ=0.71 vs 0.37 no-rubric, 0.67 general-rubric), but agreement gains demonstrate reliability, not validity: they show the LLM-generated rubric makes decisions more consistent, not that those decisions are correct. If Claude-4.5-Sonnet has systematic blind spots—e.g., rewarding explicit surface markers of support, penalizing implicit but valid reasoning, or preferring a particular discourse template—those blind spots are inherited by the annotators who use its rubrics and then by LogicJudge via distilled teacher labels. The reported 74.5%/75.0%/73.0% agreement with human experts may then reflect agreement with preferences induced by the same LLM pipeline rather than with independently established logical quality. The paper provides no external audit of rubric validity: no comparison against human-authored rubrics, no annotation of a sample without LLM-generated rubrics, and no analysis of cases where annotators disagreed with the rubric. This is the load-bearing assumption: if it fails, the benchmark's construct validity collapses even though all reported numbers are internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReportLogic introduces a human-annotated benchmark for evaluating the logical quality of deep-research-style reports, operationalizing logical quality as auditability through a three-layer, eight-dimension taxonomy (Macro-Logic, Expositional-Logic, Structural-Logic). The construction pipeline uses Claude-4.5-Sonnet to generate instance-specific rubrics that guide three expert annotators in pairwise preference labeling. The paper additionally trains LogicJudge, an open-source Qwen-3-30B-A3B-based judge, via SFT and GRPO on distilled labels produced by unanimous o3/GPT-5/Gemini-2.5-Pro consensus with swap-consistency filtering. Experiments report that LogicJudge achieves the highest agreement with human labels on three domains (DeepResearch 74.5%, Zhihu 75.0%, Quora 73.0%) against 17 off-the-shelf LLM judges and ensemble baselines. Ablations show context-aware rubrics improve inter-annotator agreement, and adversarial attacks reveal that many judges are sensitive to verbosity and other surface cues. The paper positions ReportLogic and LogicJudge as a reusable diagnostic evaluation tool for deep-research report logic.","tokens_in":27203,"tokens_out":8456,"duration_ms":96564,"significance":"If the benchmark is valid, it fills a real gap: existing long-form evaluation focuses on factuality, citation, and fluency, not on whether a report's claims and arguments can be traced, understood, and verified. The paper's strengths include a clearly specified taxonomy grounded in argumentation theory, a careful annotation protocol with adjudication, a strict position-controlled agreement metric, swap-consistency filtering for distilled training data, an open-source judge model, and a manually screened adversarial attack suite. These are substantive contributions. However, the central claims are conditional on two issues: the absence of statistical support for the headline ranking, and the lack of an external validity check on the LLM-generated rubrics that anchor both the human gold labels and the distilled training signal. The paper is transparent about its limitations and intended use, which is commendable.","major_comments":[{"comment":"The headline claim that LogicJudge achieves the highest agreement is not statistically supported. The DeepResearch human-labeled test set contains only 102 pairs (Table 3). LogicJudge's 74.5% equals 76 correct; qwen3-max's 73.53% equals 75 correct—a one-instance margin. Several baselines (e.g., gemini-2.5-pro at 68.63%) are within a few instances. No confidence intervals, significance tests, or bootstrap estimates are reported for any domain. Please report exact test counts, CIs (e.g., Wilson) or bootstrap intervals, and pairwise significance tests for the leading entries. Without this, the ranking in Table 1 should be treated as provisional.","section":"§5.2, Table 1, Table 3"},{"comment":"The validity of the human gold labels is not externally anchored. Annotators judge using rubrics generated by Claude-4.5-Sonnet (Fig. 10), and LogicJudge is trained on unanimous labels from o3/GPT-5/Gemini-2.5-Pro after swap-consistency filtering. The rubric ablation (Table 2) shows that context-aware rubrics increase inter-annotator reliability (κ=0.71 vs 0.37 no-rubric), but reliability does not demonstrate that the rubric items capture logical quality rather than surface preferences induced by the LLM generator. If the rubric has systematic blind spots, they propagate to both the gold labels and the distilled training signal, so the reported agreement may reflect alignment with LLM-induced preferences. Please add an external audit, e.g., compare against human-authored rubrics or independent holistic expert judgments on a sample, or validate against known injected logical defects, and","section":"§3.2, §4.2, Table 2"},{"comment":"The dataset statistics are internally inconsistent and must be reconciled. §5.1 states 1,319 DeepResearch queries; Table 3 reports 1,204 queries but the train/val/test split sums to 1,089; Appendix B.1 says the original benchmark contains only 102 queries and that synthetic variants were used only in training. The test set has 102 human-labeled pairs, consistent with 17 test queries, but the paper should state clearly how many of the 1,204/1,319 queries are original vs synthetic, and give the exact human-labeled test count per domain in Table 1, since the n=102 figure is critical to the significance claim above.","section":"§5.1, Table 3, Appendix B.1"}],"minor_comments":[{"comment":"Typographical errors: \"Tracability\" and \"Understanbility\" in the taxonomy labels should be corrected.","section":"Figure 2"},{"comment":"Minor grammar: \"primary focuses\" should be \"primary focus.\" Also, the related-work description of DeepResearch benchmarks would benefit from exact version/source identifiers.","section":"§2.1"},{"comment":"Baseline model names are inconsistent across text and figures: B.2 mentions Claude-3.5-Sonnet and Sonnet-3.5-Think, while Table 1 lists claude-4-sonnet and claude-4.5-sonnet; Figure 4 uses \"gemini-3-pro-preview\" and \"claude-sonnet-4\". Please unify model naming throughout.","section":"Appendix B.2, Table 1, Figure 4"},{"comment":"For Ensemble Consensus, the paper says it abstains when not unanimous, but does not state how abstentions are treated in the reported percentage (excluded from the denominator or counted incorrect). This makes the 49.02% value ambiguous.","section":"Table 1"},{"comment":"The text says \"Quora omitted for brevity\" but Figure 4 shows only two panels; clarify whether the Quora result appears in an appendix or the claim is based on the two shown domains only.","section":"§5.4, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction is careful and the paper is likely to be a useful community resource, but the central ranking claim needs statistical backing and the rubric-validity concern needs an external anchor. I see a clear path to acceptance if the authors add these analyses. No concerns about research integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it ships a genuinely new benchmark for report-level logical quality, and the trained judge is a credible attempt. Second, the central table has no confidence intervals, and on the smallest test set the headline margin is basically noise. That's the paper in one breath.\n\nWhat's actually new: nobody has operationalized report-level logical quality as auditability with a three-layer, eight-dimension taxonomy before. The context-aware rubric protocol—instance-specific inspection items generated by Claude-4.5-Sonnet—moves the field forward. It raises human inter-annotator agreement from κ=0.37 with no rubric to κ=0.71. That's a real empirical finding, and it's the strongest piece of evidence in the paper. The bad-case analysis in the appendix is also excellent: the three failure modes (numerical accounting violations, smooth but checklist-like transitions, missing warrants) are concrete and the suggested minimal fixes are useful.\n\nLogicJudge itself is a reasonable distillation effort. The consensus and swap-consistency filtering makes sense, and the two-stage SFT+GRPO with format-first rewards is a prudent design. The adversarial attack study correctly shows that off-the-shelf judges are vulnerable to length bias and evidence illusion—that's a useful warning for the community.\n\nNow the soft spots, in proportion.\n\n1. No error bars anywhere. Table 1 reports agreement accuracies without confidence intervals or significance tests. DeepResearch has only 102 human-labeled test pairs. LogicJudge's 74.5% is within a typical 95% CI of qwen3-max's 73.5%, and the margins on Zhihu and Quora are also thin. Without statistical tests, the paper cannot actually claim to beat the baselines—it can only show a direction.\n\n2. Silent exclusions. The appendix says unparsable baseline outputs are dropped, but doesn't report how often that happens. If a baseline fails to produce a parseable preference on a nontrivial fraction of cases, its accuracy is computed on a biased subset. That's fixable, but it's an omitted detail.\n\n3. The circularity worry about LLM-generated rubrics is legitimate but not fatal. The humans are guided by Claude-4.5-Sonnet rubrics, and the distilled training labels come from three other frontier models. If the rubric carries systematic blind spots, those blind spots could propagate into the gold labels. But the humans are experts and can overrule the rubric; the rubric just makes their decisions more consistent. Still, an external validity check—say, a subset annotated with human-written rubrics or no rubrics at all—would strengthen the construct validity claim substantially. As it stands, the paper shows reliability but not independent validity.\n\n4. Release: the ethical statement says they release ReportLogic and LogicJudge, but the version I read has no link or commit hash. That needs to be resolved before anyone can build on it.\n\nWho should read this: anyone building deep-research systems, evaluation pipelines, or long-form generation. It deserves a serious referee. My recommendation: accept with revisions. Require confidence intervals, exclusion rates, a direct release link, and ideally a small rubric-validity study. The core contribution is solid and will be cited.","headline":"ReportLogic is a genuinely useful benchmark for report-level logical quality, and LogicJudge is a credible trained judge, but the headline numbers lack error bars and the LLM-generated rubric deserves an external validity check.","tokens_in":27787,"tokens_out":5469,"would_cite":true,"duration_ms":63143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the logical quality of deep-research reports is a measurable property, and that a judge trained on rubric-guided preferences — LogicJudge — matches human expert verdicts better than 17 off-the-shelf LLM judges.","keywords":["logic evaluation","deep research reports","auditability","rubric-guided annotation","LLM-as-judge","pairwise preference","adversarial robustness","benchmark"],"falsifier":"On a held-out set, take logically weaker reports and make them wordier with outline-style headers and labeled 'evidence' without adding claims; if LogicJudge prefers the padded version about as often as the off-the-shelf judges it claims to beat, its robustness claim fails. A second decisive check: have independent annotators label the same pairs without seeing the LLM-generated rubrics; chance-level agreement with the rubric-guided labels would show the rubric encodes model preferences rather than reader judgment.","tokens_in":26774,"feed_emoji":"📋","tokens_out":6627,"duration_ms":73040,"temperature":0.7,"pith_summary":"Deep-research reports can be fluent and factually accurate yet still fail readers because claims are unsupported, sections do not connect, or conclusions outrun the evidence. This paper argues that this 'logical quality' is a distinct, measurable property, and defines it through the idea of auditability: whether a reader can trace, understand, and verify the report's analytical process. To make that measurable, it builds ReportLogic, a benchmark that decomposes logical quality into a three-layer taxonomy — organization, exposition, and claim–support structure — and instantiates each dimension into instance-specific rubrics that guide human annotators. It then trains an open-source judge, LogicJudge, on distilled pairwise preferences; across three domains, LogicJudge agrees with human expert verdicts (73–75%) more often than 17 off-the-shelf LLM judges or ensemble baselines. If correct, this gives the field a scalable, diagnostic way to evaluate whether LLM-generated reports can be trusted as a basis for decisions.","feed_headline":"LogicJudge beats 17 LLM judges at spotting report logic","feed_subtitle":"Trained on human rubric preferences, it matches expert verdicts across DeepResearch, Zhihu, and Quora.","key_machinery":"The load-bearing mechanism is the auditability lens, which turns an abstract quality into three traceable reader requirements — organization, flow, and argument support — each broken into eight fine-grained dimensions. The second mechanism is context-aware rubric generation: for each query and report pair, a rubric generator produces instance-specific comparison questions, span-level cues, and paired good/bad examples, making the decision boundary explicit. The third mechanism is the judge training protocol: LogicJudge is trained on pairwise preferences distilled from three frontier LLM teachers, kept only when the teachers agree unanimously and when swap-consistency holds, then aligned thro","core_discovery":"ReportLogic's central claim is that report-level logical quality — distinct from factual accuracy and surface fluency — can be operationalized and measured. Logical quality is defined as auditability: a reader can trace the report's organization, understand how ideas progress, and verify conclusions against explicit support. This yields a three-layer, eight-dimension taxonomy (Macro-, Expositional-, and Structural-Logic), instantiated by context-aware rubrics that guide expert annotations. LogicJudge, trained on distilled pairwise preferences, reaches 74.5%, 75.0%, and 73.0% agreement with expert verdicts across DeepResearch, Zhihu, and Quora, beating 17 off-the-shelf LLM judges and ensemble","pith_inferences":["A natural next step the paper leaves implicit is using LogicJudge's preferences as a training reward for report generators, turning logical-quality evaluation into an optimization signal.","The taxonomy and bias-type attack suite could transfer to other high-stakes long-form writing, such as policy briefs or regulatory analyses, where explicit warrants are practically required.","Because the rubrics and distilled labels both originate from LLMs, the benchmark's ground truth may encode what frontier models treat as logical rather than an independent reader standard; comparing rubric-guided labels with first-principles annotations would test this.","The bias-type attack categories (length, structure, qualifier wording, evidence illusion, causal display) form a reusable robustness protocol for any LLM-as-judge deployment."],"forward_implications":["ReportLogic provides a human-annotated benchmark where logical quality is diagnosed per dimension rather than as a single holistic score.","LogicJudge can act as a scalable automated evaluator, enabling community-wide comparison of deep-research report generators.","Context-aware rubrics raise human inter-annotator agreement from 0.37 (no rubric) to 0.71, so instance-specific criteria are needed for reliable logical judgment.","Off-the-shelf LLM judges are often swayed by verbosity, outline-style scaffolding, and surface 'evidence' labeling, so their logical-quality scores need robustness checks.","Reasoning-optimized judges may fill in missing warrants and thereby hide broken support relations; more test-time reasoning alone does not make a judge more reliable."],"fun_headline_variants":["New benchmark measures auditability of LLM reports","LogicJudge: AI that spots broken logic in reports","ReportLogic: quantifying trust in AI-generated reports","Auditability: the key to trusting deep research reports","LLM reports judged by logic, not fluency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the human preference labels — gathered under rubrics written by a frontier LLM and trained on labels distilled from three frontier LLM teachers — are a valid ground truth for logical quality; if those LLM-generated rubrics and consensus labels carry systematic blind spots, both benchmark and judge inherit them, and the paper offers no independent audit of rubric quality.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark measures auditability of LLM reports","LogicJudge: AI that spots broken logic in reports","ReportLogic: quantifying trust in AI-generated reports","Auditability: the key to trusting deep research reports","LLM reports judged by logic, not fluency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1343,"prompt_tokens":773,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":517,"tokens_out":570,"duration_ms":6032,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:32:52.353822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set, take logically weaker reports and make them wordier with outline-style headers and labeled 'evidence' without adding claims; if LogicJudge prefers the padded version about as often as the off-the-shelf judges it claims to beat, its robustness claim fails. A second decisive check: have independent annotators label the same pairs without seeing the LLM-generated rubrics; chance-level agreement with the rubric-guided labels would show the rubric encodes model preferences rather than reader judgment.","supporting_citations":[],"review_version":1}