Pith. sign in

REVIEW 3 major objections 7 minor 24 references

GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that evaluating multi-agent AI systems by final-answer accuracy alone misses large differences in how those agents collaborate, and that two graph-based metrics — information diversity and unnecessary-path ratio —…

desk verdict Useful diagnostic direction, but UPR as implemented re-expresses accuracy rather than redundancy; the headline claim needs reworking. read the letter →

arxiv 2507.13190 v1 pith:RH2QZQTD submitted 2025-07-17 cs.CL

Jisoo Lee , Raeyoung Chang , Dongwook Kwon , Harmanpreet Singh , Nikhil Verma This is my paper · ORCID
classification cs.CL
keywords multi-agentsystemsgraph-basedevaluationprocess-levelmetricsinformationdiversityredundantreasoningLLMdirectedacyclicgraphcollaborationefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the usual way of grading multi-agent language-model systems — by whether the final answer is correct — hides how well the agents actually collaborate. To make that visible, it models the full interaction trace as a directed acyclic graph and defines two structural scores: Information Diversity Score (IDS), which measures semantic variation in inter-agent messages, and Unnecessary Path Ratio (UPR), which measures the share of reasoning paths that do not contribute. On GSM8K, systems whose accuracies differ by only 2.1 percentage points differ by 12.8% in IDS and by 80% in UPR, and similar gaps appear on other benchmarks. The paper concludes that outcome-only evaluation is insufficient and that process-level metrics can guide design of more interpretable and token-efficient multi-agent systems.

What carries the argument

The machinery is a graph encoding of the agent conversation: each node is an agent with its prompt-response pair, each edge is information flow, and spatial and temporal adjacency matrices encode who can talk and who actually depends on whom. On this DAG, the Information Diversity Score computes a weighted average of $1 - SS_{\text{total}}[i,j]$ over connected agent pairs, where $SS_{\text{total}}$ blends TF-IDF and BERT cosine similarity, so high values mean agents say different things. The Unnecessary Path Ratio enumerates all subpaths and counts a path as necessary when the fraction of agents on it whose extracted answer equals the ground-truth answer is at least 0.5; UPR is then $1 - |P_{\text{necessary}}|/|P_{\text{all}}|$. What the two metrics do is convert an unstructured conversation log into comparable numbers about diversity and redundancy, which is what lets the paper compare architectures like Vanilla-AD, AgentDropout, AgentPrune, and G-Designer on the same scale.

What would settle it

Build a linear chain of agents in which each agent copies the previous message verbatim and the final agent outputs the correct answer alpha. Algorithm 2 will score every path 1.0, so UPR = 0 even though all reasoning is redundant; observing UPR = 0 on such a maximally redundant trace would falsify the claim that UPR quantifies unnecessary paths.

Watch

Extended reading notes

Core claim

The central claim is that the quality of a multi-agent LLM system resides partly in the structure of its communication graph, not just in its final answer. The paper argues that modeling agent interactions as a DAG with spatial and temporal adjacency lets you compute process-level diagnostics: IDS for semantic uniqueness and UPR for redundant reasoning. It reports that structurally different configurations with nearly equal accuracy — for instance Vanilla-AD at 85.6% vs G-Designer at 87.4% on GSM8K with Qwen2.5-7B-Instruct — can have very different UPR (0.40 vs 0.08), and that on MultiArith two systems at 99.4% accuracy differ in IDS (0.43 vs 0.57) and UPR (0.16 vs 0.00). The paper takes these cases as evidence that accuracy alone understates differences in collaboration efficiency.

Load-bearing premise

The load-bearing assumption is that a reasoning path should count as unnecessary exactly when fewer than half the agents on it produced an answer matching the ground-truth answer; if stating the correct answer is not the same as contributing new information, then UPR measures per-agent accuracy rather than redundant reasoning.

Editorial extensions

If this is right

  • On GSM8K, G-Designer and Vanilla-AD differ by only 2.1 percentage points in accuracy but by a factor of five in UPR (0.08 vs 0.40), so accuracy-based rankings can rate very different collaboration processes as near-tied.
  • Configurations that combine high IDS with low UPR, such as AgentPrune on MultiArith (IDS 0.57, UPR 0.00 at 99.4% accuracy), deliver high accuracy with little redundant communication, making them more attractive under token budgets.
  • A topology that scores well on one benchmark can fail on another: AgentPrune reaches UPR 0.00 on MultiArith but 0.97 on SV AMP, so process metrics are needed to validate pruning decisions per task.
  • Reporting IDS and UPR alongside accuracy gives system designers two concrete knobs — increase semantic diversity and cut unnecessary paths — rather than a single accuracy number.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because computing UPR needs only the graph and the known answer, it could be turned into a runtime stopping signal: stop a run when the current trace's UPR exceeds a threshold, saving tokens at a predictable accuracy cost. The paper does not develop this use.
  • The 0.5 necessity threshold treats a path as useful if half its agents merely state the correct answer; a natural variant would define necessity by whether a message changes the downstream reasoning, which would separate 'correct repetition' from 'new information' more cleanly.
  • IDS mixes lexical and semantic similarity, so it may reward rephrasing as much as genuinely new content; an extension would compare IDS against human novelty judgments or against downstream answer improvement to isolate epistemic contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces GEMMAS, a graph-based evaluation framework for multi-agent LLM systems, modeling agent interactions as a directed acyclic graph. It proposes two process-level metrics: Information Diversity Score (IDS), which measures semantic dissimilarity between agent outputs weighted by graph adjacency, and Unnecessary Path Ratio (UPR), which is intended to quantify the fraction of reasoning paths that are redundant. The framework is evaluated on five math benchmarks using Llama 3.1-8B-Instruct and Qwen 2.5-7B-Instruct across Vanilla-AD, AgentDropout, AgentPrune, and G-Designer configurations. The paper's central claim is that GEMMAS reveals substantial internal collaboration differences among systems with similar final accuracy, thereby showing that outcome-only metrics are insufficient.

Significance. If valid, the framework addresses a real and timely need: evaluating multi-agent LLM systems beyond final-answer accuracy, with attention to communication efficiency and reasoning redundancy. The use of graph structure to define process-level metrics is a promising direction, and the IDS metric, combining TF-IDF and BERT similarity in a structure-aware way, is a reasonable and moderately novel proposal. The paper also includes a sensitivity analysis for IDS weights. However, the central contribution is undermined by the definition of UPR. As implemented, UPR does not measure redundancy at all; it is a path-level aggregate of per-agent answer correctness. This makes the headline empirical claims about UPR unsupported, and the conclusion that outcome-only metrics are insufficient rests on this flawed metric. No code or repository is provided, and no external validation of UPR against an independent notion of redundancy is offered.

major comments (3)
  1. [Section 3.2, Eq. (3), Algorithm 2] Algorithm 2, lines 7-15, defines the path contribution score as the fraction of agents on the path whose extracted answer equals the ground-truth answer alpha, and a path is 'necessary' when this fraction is at least 0.5. This is a path-level measure of per-agent answer correctness, not a measure of information redundancy. A path in which several agents all repeat the same correct answer is labeled necessary even if it perfectly duplicates another path, while a path of agents producing incorrect-but-informative intermediate reasoning (e.g., adversarial challenges) is labeled unnecessary simply because their extracted answers do not match alpha. The text in Section 3.2 promises a 'contribution function defined over message impact,' but Algorithm 2 never examines downstream impact, mutual information, or any notion of duplicated versus novel information. Consequently, UPR does not quantify redundant reasoning paths, and the abstract's statement that UPR quantifies 'redundant reasoning paths' is not supported by the implementation.
  2. [Section 4.1, Tables 1 and 2] The headline empirical claim, that systems with only a 2.1% accuracy difference differ by 80% in UPR, is not evidence of hidden process-level inefficiency. Since UPR is defined through equality with the gold answer alpha, differences in UPR across configurations are largely mechanical consequences of how many individual agents happen to output the gold answer under each communication topology. For example, on GSM8K with Qwen2.5-7B, G-Designer's higher accuracy (87.4% vs. 85.6%) and lower UPR (0.08 vs. 0.40) may simply reflect that the optimized graph causes more agents to produce the correct answer, rather than indicating less redundant reasoning. The paper provides no partial-correlation analysis, no regression controlling for accuracy, and no validation against an external notion of redundancy (e.g., token overlap, mutual information between messages, or ablation of supposedly unnecessary paths). Without such evidence, the stated conclusion that 'outcome-only metrics are insufficient' is not established by the data.
  3. [Section 3.2, Eq. (3), Algorithm 2, line 1] The definition of Pall as 'all subpaths in G' is not well-specified. The number of subpaths in a directed graph is generally exponential, and it is unclear whether paths of length zero, length one, or repeated vertices are included; no enumeration procedure or complexity analysis is given. Additionally, the fixed threshold of 0.5 in Algorithm 2 is arbitrary, and no sensitivity analysis is provided for UPR, in contrast to the IDS weight-sensitivity analysis in Appendix A.1. This makes the metric difficult to reproduce and raises concerns about whether the reported UPR values are stable under reasonable variations in path definition or threshold.
minor comments (7)
  1. [Abstract and Section 4.1] The abstract and Section 4.1 state that the GSM8K systems differ by 'only 2.1%' in accuracy, but Table 2 shows a difference of 1.79 percentage points (85.63% vs. 87.42%); please reconcile the numbers and report exact values with error bars.
  2. [Algorithm 2, line 7] The ExtractAnswer function is not defined. For free-form mathematical reasoning, answer extraction is non-trivial and can materially affect the UPR computation; the paper should specify the extraction procedure and, ideally, validate it against a gold-standard set.
  3. [Section 3.2, Eq. (2)] The weights w_ij are described as encoding 'spatial and temporal proximity,' but Eq. (2) only uses direct adjacency entries; no path distance or indirect influence is captured. Calling this 'proximity' is misleading and should be clarified.
  4. [Tables 1 and 2] No variance estimates or statistical tests are reported for accuracy, IDS, or UPR. Since the central argument relies on comparing small differences between configurations, the paper should provide standard errors or confidence intervals, especially given the modest benchmark sizes (e.g., 180 MultiArith examples).
  5. [Section 4.1] The phrase 'fivefold improvement in redundant reasoning reduction' is imprecise: a UPR decrease from 0.40 to 0.08 is a ratio of 5, but it is not a 'fivefold reduction' in a well-defined sense, and the sentence should report the absolute change and its uncertainty.
  6. [Throughout] The benchmark 'SV AMP' is inconsistent with the standard spelling 'SVAMP'; please correct this for consistency with the cited work.
  7. [Appendix A.1] The appendix states that lambda_2 = 1 - lambda_1, while the main text says lambda_1 = lambda_2 = 0.5; please make the parametrization explicit in the main text to avoid confusion.

Circularity Check

1 steps flagged · score 6.0 of 10

UPR's 'necessary path' is defined by per-agent answer correctness against the ground truth α, so the claimed process-level redundancy metric reduces by construction to an aggregation of outcome correctness.

  1. self definitional [Section 3.2, Eq. (3) and Algorithm 2 (lines 6–15)]
    ""A path is deemed necessary if it facilitates the production of correct or informative responses by downstream agents, based on a contribution function defined over message impact." ... Algorithm 2 defines the contribution score as: "a ← ExtractAnswer(output(v)); if a == α then c ← c + 1 ... score ← c/t if t >0 else 0 {contribution score}; if score ≥ 0.5 then Pnecessary ← Pnecessary ∪ {p}"."

    The paper's central claim is that UPR quantifies redundant reasoning paths and that GEMMAS evaluates process beyond final accuracy. But by construction, path necessity is not about redundancy or downstream message impact: a path is 'necessary' whenever at least half of its agents individually produce an extracted answer equal to the ground truth α. A path whose agents all repeat the same correct answer is labeled necessary, while a path containing unique but wrong intermediate reasoning is labeled unnecessary.

full rationale

No load-bearing self-citation is present; the framework and Algorithm 1 (IDS) are self-contained and outcome-independent. The circularity is concentrated in UPR, the metric carrying the paper's strongest process-level claim. Equation 3 defines UPR as 1 − |Pnecessary|/|Pall|, and Algorithm 2 operationalizes 'necessary' via a 0.5 threshold on the fraction of agents on a path whose extracted answer equals the ground-truth α. This is a direct definitional reduction: the supposedly process-level concept of 'redundant reasoning path' is defined in terms of the outcome-level concept of answer correctness, rather than in terms of message impact, information overlap, or downstream contribution as Section 3.2 promises. Consequently, the headline result that systems with similar accuracy differ by 80% in UPR is not independent evidence that outcome-only metrics are insufficient; it is partly a restatement of per-agent correctness differences. Because IDS retains independent content, the paper is not entirely circular, but the central UPR-based claim is undermined by this construct-validity failure.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or mechanistic entities. The load-bearing assumptions are the DAG model of communication and, crucially, the correctness-based definition of UPR path necessity. The latter is an ad hoc design choice that directly affects the central claim. Two free parameters (lambda_1 and the UPR threshold) are chosen by hand without sensitivity analysis on UPR.

free parameters (3)
  • lambda_1 = 0.5
    Equal weight for TF-IDF and BERT similarity in IDS. Appendix A.1 shows IDS values vary with lambda_1, so this is a tunable design choice rather than a derived constant.
  • UPR threshold = 0.5
    A path is necessary if at least 50% of its agents output the gold answer (Algorithm 2, line 14). No sensitivity analysis or justification is provided.
  • BERT embedding configuration = not stated
    The paper says 'BERT embeddings' without specifying the checkpoint, layer, or pooling, and IDS results depend on this choice.
assumptions (3)
  • domain assumption The agent interaction graph is a directed acyclic graph.
    Section 3.1 models the system as G=(V,E,F) and assumes a DAG, which excludes cyclic or iterative communication patterns that occur in many multi-agent designs.
  • ad hoc to paper An individual agent outputting the gold answer is a valid proxy for path necessity.
    Algorithm 2 uses ExtractAnswer(output(v)) == alpha as the contribution signal, conflating outcome accuracy with information contribution.
  • domain assumption Cosine similarity of TF-IDF and BERT embeddings measures semantic diversity in agent messages.
    IDS relies entirely on this similarity; no validation against human judgments of redundancy or diversity is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems." pith.science (2026). https://pith.science/paper/RH2QZQTD

@misc{pith2026250713190,
  author       = {Pith},
  title        = {Pith review of: GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RH2QZQTD}},
  note         = {Machine review of arXiv:2507.13190}
}
read the original abstract

Multi-agent systems built on language models have shown strong performance on collaborative reasoning tasks. However, existing evaluations focus only on the correctness of the final output, overlooking how inefficient communication and poor coordination contribute to redundant reasoning and higher computational costs. We introduce GEMMAS, a graph-based evaluation framework that analyzes the internal collaboration process by modeling agent interactions as a directed acyclic graph. To capture collaboration quality, we propose two process-level metrics: Information Diversity Score (IDS) to measure semantic variation in inter-agent messages, and Unnecessary Path Ratio (UPR) to quantify redundant reasoning paths. We evaluate GEMMAS across five benchmarks and highlight results on GSM8K, where systems with only a 2.1% difference in accuracy differ by 12.8% in IDS and 80% in UPR, revealing substantial variation in internal collaboration. These findings demonstrate that outcome-only metrics are insufficient for evaluating multi-agent performance and highlight the importance of process-level diagnostics in designing more interpretable and resource-efficient collaborative AI systems.

Figures

Figures reproduced from arXiv: 2507.13190 by the authors.

Figure 1
Figure 1. Overview of the GEMMAS evaluation framework. The process begins with input mathematical problems, which are solved collaboratively by a multi-agent system composed of specialized agents. Their interactions are represented as a DAG, capturing both communication flow and reasoning structure. From this DAG, GEMMAS computes structural metrics—Information Diversity Score (IDS) and Unnecessary Path Ratio (UPR)—to evaluate… view at source ↗
Figure 2
Figure 2. illustrates the structural evolution of the DAG topology, comparing the initial structure at Iteration 1 with the optimized structure at Iter￾ation 10 produced by the G-Designer method. To analyze the effect of structural optimization, we implement and compare three state-of-the-art meth￾ods called 1) AgentPrune prunes communication links with low marginal impact, 2) AgentDropout dynamically removes underperforming … view at source ↗
Figure 3
Figure 3. Sensitivity of Information Diversity Score (IDS) to syntactic-semantic weight balance across five [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sensitivity of Information Diversity Score (IDS) to syntactic-semantic weight balance across five [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 13 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, and 1 others. 2023. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2(4):6

    arXiv 2023

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

    2019

  6. [6]

    Yubo Dong, Xukun Zhu, Zhengzhe Pan, Linchao Zhu, and Yi Yang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.964 V illager A gent: A graph-based multi-agent framework for coordinating complex task dependencies in M inecraft . In Findings of the Association for Computational Linguistics: ACL 2024, pages 16290--16314, Bangkok, Thailand. Association for...

    doi:10.18653/v1/2024.findings-acl.964 2024

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

Show all 24 references
  1. [9]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055

    2015 arXiv

  2. [10]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://doi.org/10.18653/v1/P17-1015 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  3. [11]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  4. [12]

    Sungjin Park, Xiao Liu, Yeyun Gong, and Edward Choi. 2024. Ensembling large language models with process reward-guided tree search for better complex reasoning. arXiv preprint arXiv:2412.15797

    2024 arXiv

  5. [13]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080--2094

    2021

  6. [14]

    Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743--1752

    2015

  7. [15]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154--38180

    2023

  8. [16]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11--21

    1972

  9. [17]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  10. [18]

    Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. 2025. Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration. arXiv preprint arXiv:2503.18891

    2025 arXiv

  11. [19]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  12. [20]

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2024 a . Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506

    2024 arXiv

  13. [21]

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2024 b . G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782

    2024 arXiv

  14. [22]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097--1100

    2018

  15. [23]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.