{"id":"e72ee42e-e9ab-49cb-aedf-8c752c56e7de","arxiv_id":"2505.17104","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"P2P is a multi-agent framework that automatically generates HTML-rendered academic posters from papers, backed by a 30k instruction dataset and a 121-pair evaluation benchmark.","lead":"This paper introduces P2P, a multi-agent LLM system that turns research papers into HTML academic posters, plus a large instruction dataset and a benchmark for evaluating poster quality. The work is a systems contribution with new resources, but its quality claims rest heavily on LLM-based scoring that needs closer validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on LLM-as-judge scores; the Universal calibration excludes most compared models and FineGrain has no human validation on generated posters, so rankings may reflect judge bias rather than human quality.","rationale":"In good faith, the paper contributes a reproducible multi-agent system, a 30K instruction dataset, a 121-pair benchmark with human-annotated checklists, and ablations supporting the modular design. These are real contributions independent of the headline quality claim. What would have to be true for the central claim is that the evaluation metrics faithfully track human quality across all compared methods. That condition is least secure at the LLM-as-judge interface. The paper's own §3.2.1 limits human calibration to original posters and P2P outputs; all other models are scored without human grounding. FineGrain is defined purely as GPT-4o's checklist adherence, with no reported agreement between GPT-4o and human raters on generated posters. Cross-validation R²=0.92 cannot establish external validity, since the validation folds come from the same two populations. This is not a disagreement with consensus, but an internal gap between the evidence and the generalization claim. The pairwise human study (Table 2) is helpful but narrow: it compares only P2P/YuanBao/original and reports aggregate preference percentages without protocol details, confidence intervals, or per-poster agreement. A targeted human re-scoring of a stratified sample would settle whether the LLM judge is biased; if correlations hold, the central claim would be substantially strengthened. Given that the concern is concrete and fixable, the conditional verdict stands unchanged.","tokens_in":17330,"tokens_out":4775,"duration_ms":40781,"concrete_test":"Run a held-out human validation: select 25-35 posters stratified across Table 1's score range and model families (e.g., Claude-3.7-Sonnet, GPT-4o, Qwen3-P2P-8B, Deepseek-R1, InternVL3-8B, and original posters). Have three annotators who were not involved in P2PEval annotation score each poster on the ten Universal criteria (0-50) and independently check a random subset of FineGrain checklist items. Compute per-model Spearman correlation and mean signed error between these human scores and the reported Universal/FineGrain scores. If within-family correlation is below ~0.7 or the error is non-uniform across families, Table 1's rankings and the human-level quality conclusion are not supported. The same study should report the exact protocol and N for Table 2's pairwise preference to confirm the 35.59% strict-preference figure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—'posters that approach or sometimes exceed the quality of human-created examples'—rests on scores from GPT-4o-as-judge. The Universal evaluation in §3.2.1 is calibrated with an XGBoost model (reported R²=0.92) trained on 1,701 human ratings, but the paper explicitly states that this human evaluation 'covers all original posters and our P2P outputs, excluding multi-agent approaches.' That means the fitted model is validated only on two poster populations, then applied to all 33 systems in Table 1; 10-fold cross-validation on that restricted set does not test generalization to other pipelines. The FineGrain evaluation in §3.2.2 is even less protected: the checklists are human-annotated, but the per-item scores s_i are produced directly by GPT-4o, and no human scoring of generated posters against those checklists is reported. Because P2P always emits HTML/CSS with explicit layout constraints (Appendix D), a systematic judge preference for well-formed HTML, LLM-typical wording, or specific color schemes would inflate P2P and text-only reasoning models uniformly. The individual human preference data in Table 2 does show P2P strictly preferred over the original in 35.59% of comparisons, so the claim is not baseless; but the broad model ranking and the 'approach human quality' wording are dominated by unvalidated LLM scores. Thus the load-bearing assumption—GPT-4o scores are a reliable proxy for human preference on out-of-distribution posters—is unverified precisely where it matters.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents P2P, a multi-agent LLM-based framework that generates academic posters as HTML from research papers, along with P2PInstruct, a 30k-example instruction dataset, and P2PEval, a benchmark of 121 paper-poster pairs with dual (Universal and Fine-Grained) LLM-based evaluation methodologies. The authors evaluate 33 models, report that P2P with reasoning-enhanced LLMs approaches or sometimes exceeds human-created posters, and include ablation studies, format comparisons, and human pairwise preference results.","tokens_in":17617,"tokens_out":4387,"duration_ms":38546,"significance":"If the evaluation is sound, this is a useful contribution: a practical poster-generation system, the first large-scale instruction dataset for this task, a benchmark with human-annotated checklists, and open-source fine-tuned models. The paper also provides machine-checkable artifacts (released code and dataset) and a serious attempt at structured evaluation with XGBoost calibration and fine-grained checklists. The central quality claim, however, rests on LLM-as-judge scores whose validity for out-of-distribution generated posters is not established by the reported validations.","major_comments":[{"comment":"The Universal score is calibrated with XGBoost (reported R²=0.92) on 1,701 human ratings, but the text states these ratings cover only original posters and P2P outputs, excluding other multi-agent approaches. The resulting model is then applied to all 33 systems in Table 1, including direct MLLM outputs, text-only LLMs, and poster images from Tencent YuanBao. Ten-fold cross-validation on that restricted set does not test generalization across different generation pipelines, so the Universal ranking in Table 1 is not human-validated for most compared models. Please either calibrate/validate on a sample that spans all model families represented in Table 1, or explicitly restrict the Universal-score-based claims to the original-vs-P2P comparison.","section":"§3.2.1 and Table 1"},{"comment":"The FineGrain score is computed by GPT-4o directly, using human-annotated checklists from official posters, but no human scoring of generated posters against those checklists is reported. The formula S_fine = (Σs_i / ΣM_i) × 100 is only as reliable as the per-item scores s_i, and with no agreement data (e.g., correlation or item-level accuracy against human judges on generated output), the FineGrain values in Table 1 cannot support the conclusion that P2P 'approaches or sometimes exceeds the quality of human-created examples.' Please conduct a human-judge validation on a subset of generated posters (from at least the top-performing systems per Table 1) and report the agreement with FineGrain.","section":"§3.2.2 and Fig. 3"},{"comment":"The pairwise human preference study is limited to P2P vs. YuanBao, P2P vs. Original, and YuanBao vs. Original. This is valuable evidence for those specific comparisons, but it does not justify the general claim in the conclusion that P2P 'produces posters that approach or sometimes exceed the quality of human-created examples, particularly when employing reasoning-enhanced LLMs,' because the broader 33-model comparisons in Table 1 rely on the unvalidated LLM scores discussed above. Please either add human pairwise evaluations for the models whose high LLM-based scores underpin the conclusion, or soften the conclusion to state that the quality claim is based on the limited human study plus LLM-based metrics without implying full human validation across models.","section":"§4.3, Table 2, and §6"}],"minor_comments":[{"comment":"The sentence 'We utilize both powerful models like GPT-4o and lighter models such as Qwen-VL-2.5-32B, ensuring the trained annotators are exposed to diverse samples to enhance generalizability' is confusing: it seems to mix the LLMs used as judges with human annotator training. Please rephrase to clarify which models are used for evaluation and which population the human annotators rated.","section":"§3.2.1"},{"comment":"The footnotes defining 'R' and 'T' are ambiguous; for example, 'Deepseek-R1RT 37/671B' and 'Qwen3RT 3/30B' are hard to parse. Please state explicitly for each row whether the output is from a reasoning/thinking mode, whether the model is text-only with Claude-provided figure descriptions, and what the slash-separated sizes mean.","section":"Table 1"},{"comment":"The 'Judge' metric is described as 'how frequently VLLM-based automated evaluators prefer P2P's posters over original author-created versions,' but in Table 1 a Judge value is reported for every model, presumably as that model's generated posters versus originals. Please clarify the definition and specify the VLLM used, the prompt, and the number of evaluation samples.","section":"§4.2"},{"comment":"No inter-annotator agreement is reported for the human overall scores (0–50) used to train XGBoost. Reporting agreement (e.g., Krippendorff's alpha or pairwise correlation) would strengthen the claim that the calibration target is reliable.","section":"§3.2.1"},{"comment":"Figure 6 is presented at very low resolution, making the three poster examples difficult to compare. Please provide higher-resolution images or zoom-ins for the key comparison.","section":"Appendix A / Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is suitable for the venue's scope and makes concrete contributions (dataset, benchmark, system). The main risk is that the central quality claims are supported by LLM-based evaluation whose human validation covers only a narrow subset of the compared systems. This is fixable with additional human validation on generated posters and a calibrated evaluation across model families, so I recommend major revision rather than rejection. I would also urge the authors to include the dataset and benchmark URLs in the main text and to consider reporting uncertainty across evaluation runs, as LLM judge scores can be noisy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Sun et al. build a three-agent pipeline for turning papers into HTML posters, release a 30k instruction dataset and a 121-pair benchmark with human-annotated checklists, and evaluate 33 models. The systems contribution is real and the resources are likely to be used: the dataset and benchmark fill a genuine gap, the ablation shows the component choices matter, and the pairwise human study gives direct evidence that P2P beats original posters in a third of comparisons.\n\nThe soft spots are concentrated in evaluation. The Universal score is an XGBoost model trained on human ratings for original posters and P2P outputs only. Applying that model to all 33 systems assumes it generalizes to other pipelines, which is unverified. The FineGrain score is computed by GPT-4o directly, with no human validation on generated posters. There are no confidence intervals or significance tests anywhere. And the original poster is missing from Table 1, so the headline claim about approaching human quality leans on LLM-as-judge scores rather than the human data. These are fixable: expand the calibration set to include held-out systems, add human ratings on generated posters for FineGrain, report error bars, and put the original poster in Table 1.\n\nThe citation pattern is adequate: it covers poster generation, HTML code generation, and LLM-as-judge work; self-citations are not excessive. The paper is not circular: the benchmark uses official posters as ground truth and the calibration is an independent regressor.\n\nThis paper is for people working on document summarization, layout generation, and LLM-as-judge evaluation. The dataset and benchmark are the durable parts. I'd accept it for peer review with the expectation of major revision on the evaluation, because the contribution is new and the resources ship. I'd cite the dataset and benchmark if I were working on document-to-poster generation.","headline":"A genuinely useful dataset and benchmark for paper-to-poster generation, with an evaluation section that needs a serious revision before the quality claims are trustworthy.","tokens_in":18216,"tokens_out":2546,"would_cite":true,"duration_ms":20557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated multi-agent pipeline can turn research papers into posters approaching human quality.","keywords":["paper-to-poster generation","multi-agent LLM","academic posters","LLM-as-a-Judge","P2PInstruct dataset","P2PEval benchmark","HTML rendering","fine-grained evaluation"],"falsifier":"Have a set of fresh human annotators who were not involved in the benchmark rate a sample of P2P-generated and author-created posters side by side without knowing which is which; if their strict preferences substantially disagree with the order predicted by the Universal or FineGrain scores, the paper's central quality claim falls.","tokens_in":17105,"feed_emoji":"🖼️","tokens_out":6696,"duration_ms":54178,"temperature":0.7,"pith_summary":"This paper claims that an LLM-based multi-agent framework can turn a research paper directly into a finished, HTML-rendered academic poster, and that the result approaches or sometimes beats posters made by the paper's own authors. The authors argue that splitting the work into three specialist agents—visual-element processing, content generation, and final assembly—each with a checker that triggers iterative revision, is what makes automatic poster generation practical. To support training and evaluation, they release a 30,000-plus instruction dataset and a benchmark of 121 paper–poster pairs with two LLM-as-judge scoring methods. If the claim holds, it means a major time-consuming part of scholarly communication can be automated, and it gives the community a standardized way to measure progress in that task.","feed_headline":"Three-agent LLM pipeline makes posters near human quality","feed_subtitle":"A self-checking multi-agent system turns papers into HTML posters that rival author-made ones, with a new 121-poster benchmark.","key_machinery":"The load-bearing mechanism is the three-agent pipeline with per-agent checker modules: the Figure Agent, Section Agent, and Orchestrate Agent each produce an intermediate artifact and then verify it, looping back through 'reflection' when the checker flags a problem. The Section Agent's use of Markdown-style figure references and the Orchestrate Agent's HTML/CSS rendering with content-structure decoupling are what let the system integrate visuals and text coherently. The paper also treats the P2PEval benchmark as a key device: a Universal score (ten 0–5 criteria calibrated to human ratings through an XGBoost model, reported R²=0.92) and a FineGrain score (normalized checklist adherence computed by GPT-4o against 1,738 human-annotated items) are the instruments that convert 'looks good' into a measurable claim.","core_discovery":"On its own terms, the paper's central discovery is that academic poster generation is an automatable document-design task if it is decomposed into three cooperating agents whose outputs are repeatedly checked and revised. The Figure Agent extracts figures and tables with DocLayout-YOLO and generates semantic descriptions, the Section Agent turns the paper into structured poster text with references to those visuals, and the Orchestrate Agent renders the result as HTML/CSS with layout rules; each agent has a dedicated checker and reflection loop. The authors report that the full P2P system, especially with reasoning-capable LLMs such as Claude-3.7-Sonnet, receives FineGrain and Universal judge scores that approach or exceed the original author-created posters, and that the release of P2PInstruct and P2PEval gives the field both a training resource and a benchmark with human-annotated checklists.","pith_inferences":["We infer that the same three-agent decomposition—extract visual elements, draft content, assemble layout—could transfer to other document-to-visual generation tasks, such as slides or infographics, where the checker-and-reflection loop would serve the same quality-control role.","A testable extension of the benchmark is to use the FineGrain checklist scoring beyond posters: if the normalized checklist score reflects human domain priorities here, it may generalize to faithfulness evaluation in other paper-to-visual translations, and that hypothesis can be checked directly.","The reported Universal-score calibration (R²=0.92) was trained on original posters and P2P outputs only; a readier test the authors leave open is whether that R² holds on posters from unrelated generator systems, which would show whether the judge calibration overfits to the paper's own pipeline."],"forward_implications":["If the central claim is right, generating a conference-ready poster can become a one-click step from a paper's PDF, saving researchers hours of layout work.","The P2PInstruct dataset implies that smaller open models can be fine-tuned to beat their base versions on poster generation, and the paper shows Qwen3-P2P-8B achieving the highest ROUGE scores among all tested models.","The benchmark gives a standardized yardstick, so future poster generators can be compared on the same 121 paper–poster pairs rather than on ad hoc examples.","The paper's format comparison suggests HTML is a better output medium than SVG or LaTeX for automatic poster generation, which points toward web-native academic posters as the practical choice."],"supporting_citations":[{"why":"Supplies DocLayout-YOLO, the layout detector used by the Figure Agent to extract figures and tables from papers.","marker":"[65]"},{"why":"Postdoc is the prior template-based MLLM poster generator that P2P contrasts with and aims to outperform.","marker":"[20]"},{"why":"SciPostLayout provides many of the 121 paper-poster pairs in P2PEval and is the layout dataset P2P builds on.","marker":"[50]"},{"why":"ROUGE is one of the objective metrics used to measure lexical overlap between generated and reference poster text.","marker":"[28]"},{"why":"BERTScore supplies the semantic similarity metric used alongside ROUGE to evaluate poster text.","marker":"[64]"},{"why":"The Claude 3 model family is the reasoning-capable LLM whose variants produce the strongest P2P outputs in the experiments.","marker":"[2]"},{"why":"Prior poster-generation work that decomposed the task into content extraction and layout generation, providing the lineage P2P extends.","marker":"[39]"}],"fun_headline_variants":["LLM agents auto-design academic posters from papers","Self-checking AI agents produce near-human posters","Paper to poster: multi-agent LLM pipeline with benchmark","Automated poster creation from papers via LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that P2P approaches human-quality posters rests on GPT-4o-as-judge scores being reliable proxies for what human viewers actually prefer; the XGBoost calibration was trained only on original posters and P2P outputs, so a judge bias toward LLM-generated text or HTML layouts would invalidate the comparison.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents auto-design academic posters from papers","Self-checking AI agents produce near-human posters","Paper to poster: multi-agent LLM pipeline with benchmark","Automated poster creation from papers via LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1387,"prompt_tokens":953,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":569,"tokens_out":434,"duration_ms":4588,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:20:02.391459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a set of fresh human annotators who were not involved in the benchmark rate a sample of P2P-generated and author-created posters side by side without knowing which is which; if their strict preferences substantially disagree with the order predicted by the Universal or FineGrain scores, the paper's central quality claim falls.","supporting_citations":[{"cited_title":"Scipostlayout: A dataset for layout analysis and layout generation of scientific posters","cited_arxiv_id":null,"evidence_quote":"SciPostLayout provides many of the 121 paper-poster pairs in P2PEval and is the layout dataset P2P builds on."},{"cited_title":"The claude 3 model family: Opus, sonnet, haiku","cited_arxiv_id":null,"evidence_quote":"The Claude 3 model family is the reasoning-capable LLM whose variants produce the strongest P2P outputs in the experiments."},{"cited_title":"Learning to generate posters of scientific papers by probabilistic graphical models.Journal of Computer Science and Technology, 34:155–169, 2019","cited_arxiv_id":null,"evidence_quote":"Prior poster-generation work that decomposed the task into content extraction and layout generation, providing the lineage P2P extends."}],"review_version":1}