Pith. sign in

REVIEW 4 major objections 6 minor 24 references

GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation

T0 review · 4 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Long-form article writing can be made inspectable and self-improving by treating writing, retrieval, evaluation, and rule updates as named modular skills rather than a fixed multi-agent pipeline.

desk verdict Clean systems engineering for long-form writing: modular skills plus a real evaluate-and-permanent-patch loop, with modest gains that are partly tied to the same LLM judge. read the letter →

arxiv 2607.11503 v1 pith:NS7D4DKD submitted 2026-07-13 cs.CL

Jiale Zhang , Juntao Hu , Zhijian Ou This is my paper
classification cs.CL
keywords long-formgenerationagentskillsevaluation-guidedimprovementLLM-as-a-judgeWikipedia-stylearticlesmulti-agentwritingskillpatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long Wikipedia-style articles demand long context, long instructions, and long outputs at once, so single-pass language models lose structure, invent unsupported claims, and lack mid-course quality checks. Multi-agent systems such as STORM improve coverage by simulating specialized roles, but those roles are usually tangled inside prompts and fixed procedures that are hard to inspect, reuse, or fix. This paper argues that the right unit of design is the named, declarative skill: one skill writes through Request-Plan-Draft-Audit-Refine-Deliver, others collect browser evidence and images, render diagrams, score PDFs pairwise, and map recurring weaknesses into permanent rule patches. On twenty Featured-Article topics, the skill composition beats a default writer by eight points on a 100-point PDF rubric and beats STORM on structure and content; after eight patches derived from the same evaluation reports, average scores rise from 82.90 to 86.95 with seventeen topics improved. The claim is that process modularity and evaluation-guided skill evolution can matter as much as the underlying model for long-form generation.

What carries the argument

GEIS: a Generation-Evaluation-Improvement loop of named declarative skills, with the core article-writer running Request-Plan-Draft-Audit-Refine-Deliver, a pairwise PDF-comparison-evaluation skill producing structured reports, and an improvement skill that maps recurrent authoring findings into permanent patches on the writing skill.

What would settle it

Run the same twenty topics with a different independent judge (human experts or a stronger separate model) and an absolute factuality check; if the patched skill no longer beats the unpatched skill or STORM on structure and content, the improvement-loop claim fails.

Watch

Extended reading notes

Core claim

The paper claims that long-form generation can be reframed from a fixed workflow into an inspectable, modular, evaluation-guided improvement loop of agent skills. Under the same generation backend, the skill composition improves over a default writer by 8.0 points on a 100-point PDF quality rubric, outperforms STORM on structural and content quality, and permanent patches to the writing skill raise average scores from 82.90 to 86.95 with 17 of 20 topics improved, mainly on content.

Load-bearing premise

The central claim rests on treating an LLM-as-a-judge pairwise PDF rubric as a reliable measure of real article quality, so that score gains from patches derived from those same reports count as genuine writing improvement rather than judge-aligned style fitting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GEIS, a skill-based Generation–Evaluation–Improvement loop for Wikipedia-style long-form article generation, implemented in Tasi Harness. Writing is factored into named declarative skills (article-writer with Request–Plan–Draft–Audit–Refine–Deliver; browser evidence/images; diagram rendering; PDF pairwise evaluation; and rule-level skill improvement). On 20 Featured Article topics, with GPT-5.4 for generation and Qwen 3.5 Plus as judge, GEIS scores +8.0 over the harness default on a 100-point PDF rubric, leads STORM on comparable structure and content dimensions, and after permanent patches derived from evaluation reports raises average writing-focused scores from 82.90 to 86.95 (17/20 topics improved, mainly content). The authors conclude that long-form generation can be reframed as an inspectable, modular, evaluation-guided skill loop rather than a fixed multi-agent pipeline.

Significance. If the results hold under independent quality checks, the work is a useful systems contribution to long-form generation: it makes capability boundaries explicit, adds mid-course audit/refine gates, and shows a lightweight path from pairwise quality reports to permanent writing-skill patches without retraining the base model. Strengths include a clear skill decomposition, separation of generator and judge models, deterministic visual anchors, and three reported experiments with transparent tables (Tables 3–6) and an honest discussion of topic-level regressions. The improvement loop idea is practically relevant for professional writing systems where model weights cannot be changed. Significance is currently limited by reliance on a single LLM-as-a-judge family for both diagnosis and measurement of the improvement claim, and by evaluation confined to 20 topics on one harness.

major comments (4)
  1. §4.2, §6.3, Table 5: The improvement experiment is load-bearing for the paper’s title claim, but the same evaluation skill/rubric family that produces findings also re-scores post-patch articles. The eight permanent patches (sources, inline citations, depth budgets, bibliography, conclusions, etc.) align closely with the observed content-only gain (+3.85/50 structure +0.20). Without a held-out judge, human ratings, or a frozen pre-patch evaluation protocol that cannot see the patch targets, the 82.90→86.95 lift and 17/20 wins can be explained as judge-aligned style fitting. Please add at least one independent quality signal (human raters on a subset, a different judge model, or factuality metrics such as citation support / FActScore-style checks) and report whether the pre/post ranking still holds.
  2. §5.3, §6.1–6.2, Tables 3–4: All primary scores are single-pass LLM-as-a-judge aggregates with no variance, multi-seed generation, or significance tests on n=20. The +8.0 harness gap and +5.6 structure gap vs STORM are large enough to be interesting, but the manuscript treats them as definitive. Report judge agreement (e.g., repeated pairwise calls with order swap), generation variance if any, and simple paired tests or bootstrap CIs so readers can assess stability of the system-level claims.
  3. §6.2, Table 4: The STORM comparison is only partially controlled. STORM is scored only on A+B because it is not run through GEIS’s visual-anchor and PDF pipeline; GEIS also benefits from browser/image/diagram skills that STORM is not given in an equivalent form. Either (i) evaluate both systems on a pure text/Markdown writing profile with matched retrieval budgets, or (ii) clearly reframe Table 4 as a process/delivery comparison rather than a head-to-head writing-quality win, and avoid aggregating “comparable total” as if the systems were equally equipped.
  4. §5.2, §7–8: Wikipedia Featured Articles are used as topics and as high-quality peer references in pairwise diagnosis. That is reasonable for open-ended writing, but the paper should quantify how much of the GEIS advantage is “looks more like a Featured Article under this rubric” versus external usefulness (e.g., expert preference, coverage of required subtopics from a fixed checklist, or citation correctness against sources). A small expert or checklist study on 5–10 topics would substantially strengthen the central claim that the skill loop improves authoring quality, not only rubric fit.
minor comments (6)
  1. Table 1 and surrounding text: several layout artifacts appear in the manuscript (“T able 1”, “thesetcovershistory”); clean table captions and spacing for camera-ready.
  2. §3.2–3.3: The six-stage process and anchor syntax are clear, but a short example of one Plan outline and one Audit checklist item (even in the appendix) would make the skill contract more concrete for readers who will not see the full SKILL.md until open-sourcing.
  3. §5.3: Specify model endpoints/versions and decoding settings for GPT-5.4 and Qwen 3.5 Plus as fully as possible; “GPT-5.4” and “Qwen 3.5 Plus” alone will age poorly for replication.
  4. Table 6: Three regressions (Autonomous agent, Knowledge graph, Microservices) are discussed well in §7; consider adding a one-sentence note in the table caption that negative Δ values are intentional evidence of uniform-patch limits.
  5. Related work §2.3: Reflexion and skill-RL citations are appropriate; a brief contrast with other prompt/skill self-edit systems (beyond naming them) would help position the permanent-patch design.
  6. Appendix A / Table 7: State explicitly whether full SKILL.md assets and evaluation scripts will be released with the camera-ready or only “upon acceptance,” and whether the 20-topic outputs will be archived for audit.

Circularity Check

1 steps flagged · score 4.0 of 10

Improvement gains partly reduce to same-judge, same-topic fitting: patches are written from the pairwise rubric findings, then re-scored by that same judge family on the same 20 topics.

  1. fitted input called prediction [§4.2, §6.3, Table 5; also Abstract / §9 improvement claim]
    "In the 20-topic improvement run, recurrent findings produced eight improvement items: minimum source requirements, inline citations for factual claims, stable image sources, deeper section budgets, bridging transitions, formal bibliography formatting, topic completeness checks, and mandatory conclusions. ... The average score increases from 82.90 to 86.95, a gain of 4.05 points. Content quality accounts for almost all of the improvement (+3.85), while structure changes only slightly (+0.20). This is consistent with the actual patches..."

    The improvement skill is fitted to findings from the same pairwise LLM-as-a-judge rubric that later certifies success, on the same 20 topics with no held-out set. Patch targets (citations, depth budgets, bibliography, conclusions, completeness) are exactly the content-dimension levers the judge rewards; post-patch content rises +3.85 while structure is nearly flat. The reported “authoring improvement” is therefore partly statistically forced alignment to the judge’s preferred surface features rather than an independently measured quality gain. Not fully by-construction (3 topics decline; generation still must execute the rules), but the pre/post delta is not an external prediction.

full rationale

This is an engineering systems paper, not a first-principles derivation, so most of the architecture is non-circular: the six-stage writing skill, skill separation, and the GEIS-vs-default / GEIS-vs-STORM comparisons are independent empirical claims under a fixed judge. The only load-bearing circular step is the 20-topic improvement experiment. There, article-writer-improving reads the pairwise evaluation reports, maps recurrent findings into permanent writing-skill patches (sources, inline citations, depth budgets, bibliography, conclusions, etc.), regenerates the same 20 topics, and re-scores with the same writing-focused structure+content rubric from the same evaluation skill family. Content rises by +3.85 while structure barely moves (+0.20), matching the patch targets exactly; there is no held-out topic set and no independent human or held-out-judge check. That makes the 82.90→86.95 lift and 17/20 wins partly a fitted-input-called-prediction: the metric that generated the rules is the metric that certifies the rules. The paper itself notes judge variance and that uniform patches can regress some topics, which shows the loop is not fully tautological—but the central improvement claim still lacks an external quality signal. No self-definitional equations, uniqueness theorems, or load-bearing self-citation chains drive the architecture claims. Score 4 reflects partial circularity confined to the improvement loop, not the whole paper.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The central claims rest on engineering design choices and an LLM-judge evaluation protocol rather than free physical parameters. Load-bearing premises are: modular declarative skills are the right unit of abstraction; the six writing stages and four-dimension PDF rubric capture quality; Wikipedia Featured topics are a fair testbed; recurrent judge findings should become permanent writing rules; and Tasi Harness faithfully implements the skills. No new physical entities; free parameters are rubric weights and hand-chosen patch rules.

free parameters (2)
  • PDF rubric dimension weights (A/B/C/D = 35/35/15/15; improvement profile 50/50)
    Hand-chosen scoring weights define the reported totals; different weights would change which system wins and how large the +8.0 / +4.05 gains appear.
  • Eight permanent improvement patches (sources, citations, image stability, depth budgets, transitions, bibliography, comp
    Rule set selected from recurrent findings in the 20-topic run; composition is author-chosen and drives the post-improving scores.
assumptions (5)
  • domain assumption LLM-as-a-judge pairwise comparison under a structured PDF rubric is a valid proxy for long-form article quality.
    All main scores in §5–§6 come from Qwen 3.5 Plus pairwise reports; no human gold ratings.
  • domain assumption Wikipedia Featured Articles are appropriate open-ended evaluation targets and high-quality reference documents without being gold answers.
    Dataset construction and improvement diagnosis in §5.2 rest on this.
  • domain assumption Authoring issues can be cleanly separated from PDF-export/tooling artifacts so only the former should patch writing skills.
    Core of article-writer-improving in §4.2.
  • ad hoc to paper Progressive disclosure of named skills preserves context and capability boundaries better than fixed multi-agent prompt pipelines.
    Design goal stated in §3.1 and Table 1; not independently proven outside this system comparison.
  • standard math Standard LLM tool-use and generation assumptions (models can follow multi-stage skill documents and use browser/diagram tools).
    Background capability assumed throughout implementation in Tasi Harness.
invented entities (2)
  • GEIS skill loop (article-writer + evaluation + improving skills composition)
    purpose: Package long-form generation as named declarative skills with a closed generate–evaluate–permanent-patch cycle.
    The paper’s primary proposed system; evidence is internal experiments only, not an external physical entity.
  • Deterministic visual anchors ([[DIAGRAM:...]], [[IMAGE:...]])
    purpose: Decouple prose generation from image/diagram rendering and PDF delivery.
    Engineering construct introduced in §3.3; validated only inside GEIS outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation." pith.science (2026). https://pith.science/paper/NS7D4DKD

@misc{pith2026260711503,
  author       = {Pith},
  title        = {Pith review of: GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NS7D4DKD}},
  note         = {Machine review of arXiv:2607.11503}
}
read the original abstract

Long-form article generation remains difficult for large language models because it combines long context, long instructions, and long outputs. Existing multi-agent pipelines such as STORM improve information coverage by simulating role-specialized agents, but their capabilities are often entangled in prompts and fixed procedures, making them hard to inspect, reuse, or iteratively improve. This paper presents GEIS (Generation-Evaluation-Improvement loop of agent Skills), a loop of named and declarative skills for Wikipedia-style long-form article generation. Implemented and evaluated in Tasi Harness, GEIS composes skills for article writing, browser-based evidence and image collection, diagram rendering, PDF-aware pairwise evaluation, and rule-level skill improvement. Its core writing skill follows Request, Plan, Draft, Audit, Refine, and Deliver; the pairwise evaluation skill produces structured quality reports; and the improvement skill maps recurrent findings into permanent patches to the writing skill in our 20-topic experiment. We evaluate GEIS on 20 Wikipedia Featured Article topics. Under the same generation backend, GEIS improves over the Tasi Harness default writer by 8.0 points on a 100-point PDF quality rubric and outperforms STORM on the two comparable writing dimensions, structural quality and content quality. In the 20-topic improvement experiment, the patched writing skill raises the average score from 82.90 to 86.95, with 17 out of 20 topics improved and the gain mainly coming from content quality. These results show that long-form generation can be reframed from a fixed workflow into an inspectable, modular, and evaluation-guided improvement loop.

Figures

Figures reproduced from arXiv: 2607.11503 by the authors.

Figure 1
Figure 1. The six-stage article-writer process. Request normalizes the user task, Plan constructs the article outline, Draft writes the full article, Audit checks quality, Refine revises weak parts, and Deliver emits the final text or document. Our contributions are threefold. First, we formulate a skill-based decomposi￾tion for long-form generation, separating writing, retrieval, image and diagram handling, evaluation, and i… view at source ↗
Figure 2
Figure 2. Closed-loop optimization. The writing skill generates an article, the pairwise evaluation skill produces a report, and the improvement skill converts recurrent report findings into permanent writing-skill patches in the reported experiment. In the 20-topic improvement run, recurrent findings produced eight improve￾ment items: minimum source requirements, inline citations for factual claims, stable image sources, dee… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2024)

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2024)

    arXiv 2024

  2. [2]

    Transactions of the Association for Computational Linguistics 12, 157–173 (2024)

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173 (2024)

    2024

  3. [3]

    ACM Transactions on Information Systems 43(2) (2024).https://doi.org/10.1145/3703155

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: Prin- ciples, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43(2) (2024).https://doi.org/10.1145/3703155

    doi:10.1145/3703155 2024

  4. [4]

    In: Proceedings of NAACL-HLT 2024, pp

    Shao, Y., Jiang, Y., Kanell, T., Xu, P., Khattab, O., Lam, M.: Assisting in writing Wikipedia-like articles from scratch with large language models. In: Proceedings of NAACL-HLT 2024, pp. 6252–6278 (2024).https://doi.org/10.18653/v1/2024. naacl-long.347

    doi:10.18653/v1/2024 2024

  5. [5]

    arXiv preprint arXiv:2208.11663 (2022)

    Schick, T., Dwivedi-Yu, J., Jiang, Z., Petroni, F., Lewis, P., Izacard, G., You, Q., Nalmpantis, C., Grave, E., Riedel, S.: PEER: A collaborative language model. arXiv preprint arXiv:2208.11663 (2022)

    arXiv 2022

  6. [6]

    arXiv preprint arXiv:2308.08155 (2023)

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A.H., White, R.W., Burger, D., Wang, C.: AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155 (2023)

    arXiv 2023

  7. [7]

    arXiv preprint arXiv:2502.00640 (2025)

    Wu, S., Galley, M., Peng, B., Cheng, H., Li, G., Zhu, Y., Leskovec, J., Gao, J.: CollabLLM: From passive responders to active collaborators. arXiv preprint arXiv:2502.00640 (2025)

    arXiv 2025

  8. [8]

    arXiv preprint arXiv:2408.07055 (2024)

    Bai, Y., Zhang, J., Lv, X., Zheng, L., Zhu, S., Hou, L., Dong, Y., Tang, J., Li, J.: LongWriter: Unleashing 10,000+ word generation from long context LLMs. arXiv preprint arXiv:2408.07055 (2024)

    arXiv 2024

Show all 24 references
  1. [9]

    arXiv preprint arXiv:2409.02076 (2024) GEIS for Long-Form Article Generation 15

    Wu, Y., Hee, M.S., Hu, Z., Lee, R.K.-W.: LongGenBench: Benchmarking long-form generation in long context LLMs. arXiv preprint arXiv:2409.02076 (2024) GEIS for Long-Form Article Generation 15

    2024 arXiv

  2. [10]

    In: Text Summarization Branches Out, pp

    Lin, C.-Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)

    2004

  3. [11]

    In: Proceedings of EMNLP 2023, pp

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P.W., Iyyer, M., Zettle- moyer, L., Hajishirzi, H.: FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In: Proceedings of EMNLP 2023, pp. 12076– 12100 (2023).https://doi.org/10...

    2023 doi

  4. [12]

    In: Advances in Neural Information Processing Systems (2023)

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In: Advances in Neural Information Processing Systems (2023)

    2023

  5. [13]

    Science China Information Sciences 68(2) (2025).https://doi.org/10

    Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al.: The rise and potential of large language model based agents: A survey. Science China Information Sciences 68(2) (2025).https://doi.org/10. 1007/s11432-024-4222-0

    2025

  6. [14]

    In: International Conference on Learning Representations (2023)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: International Conference on Learning Representations (2023)

    2023

  7. [15]

    arXiv preprint arXiv:2302.04761 (2023)

    Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., Scialom, T.: Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023)

    2023 arXiv

  8. [16]

    In: Advances in Neural Information Processing Systems (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W.-t., Rocktaschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems (2020)

    2020

  9. [17]

    In: Proceedings of ISCSLP (2024)

    Zhao, Y., Cao, H., Zhao, X., Ou, Z.: An empirical study of retrieval augmented generation with chain-of-thought. In: Proceedings of ISCSLP (2024)

    2024

  10. [18]

    arXiv preprint arXiv:2508.18168 (2025)

    Cao, H., Wu, Y., Cai, Y., Zhao, X., Ou, Z.: Improving end-to-end training of retrieval-augmented generation models via joint stochastic approximation. arXiv preprint arXiv:2508.18168 (2025)

    2025 arXiv

  11. [19]

    GitHub repository

    TasiTech: Tasi Harness. GitHub repository. https://github.com/TasiTech/ tasi-harness(2026), last accessed 4 July 2026

    2026

  12. [20]

    Anthropic: Agent Skills.https://www.anthropic.com/news/agent-skills (2025), last accessed 28 May 2026

    2025

  13. [21]

    In: Inter- national Conference on Learning Representations (2024)

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., et al.: DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In: Inter- national Conference on Learning Representations (2024)

    2024

  14. [22]

    In: Advances in Neural Information Processing Systems (2023)

    Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S.: Reflex- ion: Language agents with verbal reinforcement learning. In: Advances in Neural Information Processing Systems (2023)

    2023

  15. [23]

    arXiv preprint arXiv:2602.08234 (2026)

    Xia, P., Chen, J., Wang, H., Liu, J., Zeng, K., Wang, Y., Han, S., Zhou, Y., Zhao, X., Chen, H., Zheng, Z., Xie, C., Yao, H.: SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234 (2026)

    2026 arXiv

  16. [24]

    arXiv preprint arXiv:2605.09359 (2026)

    Vishe, Y., Surana, R., Jiang, X., Huang, Z., Li, X., Kuang, N.L., Yu, T., Rossi, R.A., Shang, J., McAuley, J., Wu, J.: Skill-R1: Agent skill evolution via reinforcement learning. arXiv preprint arXiv:2605.09359 (2026)

    2026 arXiv

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.