Pith. sign in

REVIEW 4 major objections 5 minor 44 references

This paper claims that iterative refinement in LLMs is not uniformly useful: gains arrive early in ideas and code, while math can improve late—but only when prompted to elaborate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:07 UTC pith:VZUMOOGT

load-bearing objection Useful domain fingerprints of iterative prompting, but the headline early-vs-late gains are measured under a task-forgetting protocol and need a grounded-context check before they become practical guidance. the 4 major comments →

arxiv 2509.06770 v2 pith:VZUMOOGT submitted 2025-09-08 cs.AI cs.HC

Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting

Shashidhar Reddy Javaji , Bhavul Gauri , Zining Zhu This is my paper
classification cs.AI cs.HC
keywords iterative refinementmulti-turn promptingprompt specificitysemantic driftlexical noveltyLLM evaluationdomain-dependent behaviorelaboration prompting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish when multi-turn "improve it" prompting helps and when it hurts, by running controlled 12-turn conversations across 50 ideation, code, and math problems with four large language models, under vague feedback and targeted steering. Its central finding is domain-dependence: ideas and code improve early or not at all, whereas math can yield late breakthroughs, but only when the prompt asks for elaboration rather than exploration. A second claim is that after a few turns, vague feedback plateaus or reverses quality, while specific prompts reliably move the quality axis they name. The paper contributes per-turn metrics—semantic drift, turn-to-turn volatility, lexical novelty, and output growth—that make these dynamics measurable and can signal when to steer, stop, or switch strategy.

Core claim

Iterative refinement is not a single uniform tool. Under a memoryless 12-turn protocol where each model sees only its previous output plus a short instruction, the paper finds that in ideation and code the useful iterations happen early: correct code paths are found in the first few turns or vague refinement fails, and continuing to ask for improvement mainly inflates code size or drifts ideas without restoring feasibility. In math, by contrast, the default is logical fixation, but asking the model to elaborate each step can break that fixation late in the conversation, producing substantially higher correctness, with one weak model rising from 6.9% to 40.5%. Vague instructions behave like a

What carries the argument

The framework's central object is the turn-wise trace: every response in a 12-turn conversation is logged and scored separately. It combines domain-appropriate outcome checks (unit tests for code; answer-equivalence plus reasoning soundness for math; LLM-judge scores for ideation and code quality) with three families of behavioral metrics: semantic drift from the first response and turn-to-turn volatility (cosine distances in embedding space), lexical novelty (percentage of new bigrams and trigrams per turn), and growth factor (word count or lines of code normalized to turn 1). These metrics do the work of exposing whether iteration is meaning-changing, size-inflating, or stuck.

Load-bearing premise

The results come from a memoryless setting where, from turn 2 on, the model sees only its previous output and a short instruction—never the original task—so the observed early collapses and late math gains may not hold when the original prompt stays in context.

What would settle it

Re-run the 12-turn protocol with the original task prompt re-inserted at every turn. If code correctness stops collapsing after turn 3–4 or the late math gains disappear, the memoryless context is the cause of the paper's domain patterns; if trajectories are unchanged, the patterns are robust to re-anchoring.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If a correct code path is not found in the first 3–4 turns, continued vague refinement is unlikely to succeed: stop or restart rather than push more 'improve it' turns.
  • In math, late turns are not wasted: asking for step-by-step elaboration can uncover correct solutions late in the conversation, so an early wrong answer should not end the loop.
  • In ideation, the loop should be separated into widening (novelty-seeking) then tightening (feasibility-seeking), because a single vague prompt tends to repeat itself or drift into infeasibility.
  • Vague feedback is an unreliable control: near-synonyms like 'improve', 'make it better', and 'refine' produce measurably different trajectories, so workflow designers should use explicit quality axes.
  • The behavioral metrics can serve as early signals to steer, stop, or switch strategy before quality collapses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The memoryless protocol, which hides the original task from turn 2 onward, is likely harsher than real workflows; re-anchoring the original prompt each turn is a cheap, testable modification that may reduce drift and bloat and shift where gains appear.
  • The domain fingerprints suggest that turn-level metric trajectories alone may be enough to classify task type and automatically select a prompt strategy—something the paper does not implement.
  • The math elaboration result implies that 'explain each step in more detail' acts as a compute-like expansion of the reasoning path; one could test whether it transfers to other formal domains such as code reasoning or theorem proving.
  • The paper's proposed multi-role designs—generator then refiner for ideas, elaborator with depth for math, early-stopper for code—are designs it did not run; testing them directly is the natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a turn-wise evaluation framework for iterative LLM refinement across three domains: ideation (LiveIdeaBench), code (DS-1000), and math (OmniMath). For each task it runs controlled 12-turn conversations with four models, using three vague prompts and two domain-specific targeted prompts, logging every turn. It computes behavioral dynamics metrics (drift from origin, turn-to-turn volatility, lexical novelty, growth factor) and domain-specific outcome scores (unit tests for code; Gemini-judged answer correctness and reasoning soundness for math; Gemini-judged originality/feasibility/pragmatism/readability for ideation and code). The central claims are that gains are domain-dependent—early in ideas and code, late in math when guided by elaboration—and that vague feedback plateaus or reverses after a few turns while targeted prompts shift the intended quality axis. The paper also proposes practical stop/steer/switch rules and a multi-agent recipe.

Significance. If the claims hold, the paper would provide a useful measurement toolbox for iterative prompting and concrete guidance on when iteration helps or hurts, which is a timely topic. The design has real strengths: explicit metric definitions, controlled prompt palettes, per-turn logging, unit-test-based correctness for code, a cross-model/cross-domain comparison, and concrete worked examples in the appendix. However, the central empirical conclusions currently rest on a single memoryless protocol and on unquantified statistical uncertainty, and several headline results depend on LLM-as-a-judge scores without validation. The significance is therefore conditional on those issues being resolved; the framework itself is a valuable contribution.

major comments (4)
  1. [§3.2] From turn 2 onward, the model is presented with only its previous output plus a short instruction; the original task prompt is never re-shown. All headline claims—early gains in ideas/code, late math gains under elaboration, vague-prompt plateau/reversal—are measured exclusively under this memoryless setting. The paper explicitly calls the protocol a stress test (§3.2) but does not compare it with a grounded-context condition in which the original task remains in context. The code collapse and the late math breakthroughs could therefore be driven by loss of task grounding (e.g., forgetting the 100–200 word limit, the unit-test problem statement, or the OmniMath problem) rather than by intrinsic properties of iteration. This is not a circularity problem: the metrics are defined independently. It is an external-validity problem that is load-bearing for the practical stop/steer/switch recom
  2. [§4.2–4.3, Fig. 19/20] No confidence intervals, significance tests, or multiple seeds are reported. Section 3.4 fixes temperature=0.7 and §3.2 says each task-model-prompt combination is 'run independently,' but no number of seeds is given and the reported numbers appear to be from a single run per cell. With n=50 tasks, binomial noise is substantial: Llama math accuracy rising from 6.9% to 40.5% has approximate standard errors of 3.6% and 7.0%, respectively, and many heatmap cells (e.g., 0.34 vs 0.38 in Fig. 2(d)) are within such noise. The large elaboration-vs-vague contrast may be real, but the finer 'early vs late' and 'vague vs targeted' timing claims need per-turn confidence intervals, seed-level variance, or permutation/significance tests. Without these, the headline quantitative distinctions are not statistically supported.
  3. [§3.5.1, §4.3] Math correctness and reasoning soundness are assigned by Gemini 2.5 Pro through an LLM judge prompt, not by a deterministic verifier: the prompt asks the judge to check answer equivalence and to score reasoning. The paper's central late-turn math discovery claim depends entirely on this evaluator, yet no validation against human annotation, no agreement statistics, and no error analysis are provided. The appendix's own worked example (Table 3) shows that the judge's per-turn correctness follows a correct streak at turns 3–6 and then a regression at turn 7, while the underlying reasoning scores fluctuate—exactly the kind of behavior that could be sensitive to judge inconsistency. Please report a human-validated sample of judge decisions, and where feasible use a symbolic/rule-based answer-equivalence check as a cross-check.
  4. [§3.5.3, §4.1–4.2] The claims that targeted prompts 'reliably shift the intended quality axis' (novelty vs feasibility in ideation; speed vs readability in code) rely on Gemini 2.5 Pro ratings for originality/feasibility/pragmatism/readability. These ratings are used as the only evidence that the intended axis moves while other qualities are preserved. No human agreement study or second-judge consistency check is reported. Since the entire paper's 'targeted steering works' message rests on these subjective scores, a small human-annotation validation (e.g., 50–100 turns per domain) should be added or the claims should be weakened accordingly.
minor comments (5)
  1. [Figure 1 caption] The caption lists 'Specific ... Math: Speed, Readability; Code: Elaborate, Alternate', but §3.3 defines math as Elaboration/Alternate method and code as Performance/Readability. The two domains are swapped and should be corrected.
  2. [§3.3 vs Appendix A] The vague prompt wording differs between the main text ('This [idea/code/solution] can be better. Improve it.') and Appendix A ('This {subject} is good, improve it.'). One canonical set of exact strings should be used.
  3. [§3.5.2] There are typographical issues: 'V olatility' appears as a broken token, and the equation for Drift_from_Origin is missing a space before the Volatility definition. Please also define the embedding model version explicitly.
  4. [Abstract/§3.5.1] The abstract calls math evaluation 'answer-equivalence plus reasoning-soundness' and groups it with 'ground-truth checks,' but the math correctness is an LLM-judge score. This is misleading; either describe it as judge-based or report evidence that it agrees with ground truth.
  5. [§3.2] 'To ensure statistical robustness, each task-model-prompt combination is run independently' is not a statement of robustness if there is only one run per cell. Please either report the number of seeds and show variation, or reword the sentence.

Circularity Check

0 steps flagged

No significant circularity: headline claims are empirical measurements over external benchmarks; the protocol's memoryless setting is an external-validity limitation, not a derivation-level circularity.

full rationale

The paper is an observational study rather than a derivation. Its central claims—that gains are domain-dependent, that vague feedback plateaus, and that elaboration drives late math gains—are obtained by running fixed prompting protocols and measuring outputs with unit tests, ground-truth answer equivalence, and LLM-assisted quality ratings. These measurements are not equal by construction to the conclusions. Drift, volatility, lexical novelty, and growth factor are defined independently of the headline findings and are computed directly from model outputs; the paper does not fit any parameter to data and then rename that fit as a prediction. The judge model (Gemini 2.5 Pro) is separate from the generator models, and the correctness checks for code and math are grounded in external benchmarks (DS-1000 unit tests and Omni-MATH ground-truth answers). The only mild concern is semantic overlap: the 'specific steering' prompts name a quality axis (novelty, feasibility, readability, elaboration) and the judge rubrics rate the same axis. However, the paper's claim is not that the axis is shifted by definition but that models measurably shift scores along that axis, which is an empirical and falsifiable result—a model could fail to comply. Section 3.2's memoryless protocol (the model sees only its previous output plus an instruction from turn 2 onward) is explicitly described as a deliberate stress test, and it is a legitimate external-validity limitation rather than a circular step; it does not make any result true by construction. The Limitations section also acknowledges unvalidated proposals, and no load-bearing self-citation or imported uniqueness theorem appears. Overall, the empirical chain is self-contained with respect to its benchmarks, so circularity is minimal.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

All numbers and assumptions the central claims depend on. No parameters are fitted to data, but several design choices (12 turns, temperature 0.7, judge model) are load-bearing for the reported dynamics. The paper's conclusions rely on unvalidated assumptions that embedding distance tracks semantic drift, that an LLM judge scores quality axes faithfully, and that a memoryless 12-turn loop represents real iterative use.

free parameters (3)
  • conversation length = 12 turns
    Fixed at 12 for every conversation; the math late-turn finding (gains concentrated in Turns 8-12) is only observable within this horizon, and no sensitivity analysis is reported.
  • sampling temperature = 0.7
    Single temperature across all models and runs; trajectories may differ under greedy or other sampling temperatures, and no averaging over seeds is provided.
  • max_tokens = 10,000
    Caps generation length; growth-factor measurements for long outputs could be truncated at this bound.
axioms (3)
  • domain assumption Cosine distance in Qwen3-Embedding-0.6B space is a valid proxy for semantic drift and volatility across turns.
    Section 3.5.2 defines Drift from Origin and Volatility using embedding cosine distance; no human validation or correlation with perceived semantic change is reported.
  • domain assumption LLM-as-a-judge (Gemini 2.5 Pro) ratings for originality, feasibility, pragmatism, readability, soundness, and clarity match human judgment on these scorecards.
    Section 3.5.3 cites Zheng et al. 2023 for judge-human agreement in general, but no per-axis agreement is measured for this study's specific prompts and domains.
  • ad hoc to paper A memoryless 12-turn loop, in which the model sees only its previous output and not the original task after turn 1, is an informative model of iterative refinement dynamics.
    Section 3.2 states the protocol feeds 'only its own output from the previous turn'; the paper frames this as a stress test, but the generalizability of the observed collapse patterns to real, context-preserving workflows is unexamined.

pith-pipeline@v1.3.0-alltime-deepseek · 19540 in / 15660 out tokens · 151930 ms · 2026-08-04T23:07:04.158688+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting." pith.science (2026). https://pith.science/paper/VZUMOOGT

@misc{pith2026250906770,
  author       = {Pith},
  title        = {Pith review of: Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZUMOOGT}},
  note         = {Machine review of arXiv:2509.06770}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) are now used in multi-turn workflows, but we still lack a clear way to measure when iteration helps and when it hurts. We present an evaluation framework for iterative refinement that spans ideation, code, and math. Our protocol runs controlled 12-turn conversations per task, utilizing a variety of prompts ranging from vague ``improve it'' feedback to targeted steering, and logs per-turn outputs. We score outcomes with domain-appropriate checks (unit tests for code; answer-equivalence plus reasoning-soundness for math; originality and feasibility for ideation) and track turn-level behavior with three families of metrics: semantic movement across turns, turn-to-turn change, and output size growth. Across models and tasks, gains are domain-dependent: they arrive early in ideas and code, but in math late turns matter when guided by elaboration. After the first few turns, vague feedback often plateaus or reverses correctness, while targeted prompts reliably shift the intended quality axis (novelty vs. feasibility in ideation; speed vs. readability in code; in math, elaboration outperforms exploration and drives late-turn gains). We also observe consistent domain patterns: ideation moves more in meaning across turns, code tends to grow in size with little semantic change, and math starts fixed but can break that path with late, elaborative iteration. Together, the framework and metrics make iteration measurable and comparable across models, and signal when to steer, stop, or switch strategies.

Figures

Figures reproduced from arXiv: 2509.06770 by Bhavul Gauri, Shashidhar Reddy Javaji, Zining Zhu.

Figure 1
Figure 1. Figure 1: An overview of our experimental framework for studying iterative LLM refinement. We test four [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Llama-3.1-8B (Math): (a) accuracy rises 6.9% [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ideas domain — Turn-to-turn Drift (mean) [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ideas domain — Turn-to-turn volatility (mean) [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ideas domain — Growth factor (mean) 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-model trajectories (Turns 1–12) for originality, feasibility, and clarity with techniques distinguished [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-model trajectories (Turns 1–12) for originality, feasibility, and clarity with techniques distinguished [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Coding domain — Drift from origin (mean) [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Coding domain — Lexical novelty (mean) 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Coding domain — Turn-to-turn volatility (mean) [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Coding domain — Growth factor (mean) 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Turn-wise success heatmap. Each cell shows the percentage of tasks that pass at turn t (columns 1–12) under each prompt variant (rows; label shows n tasks). Warmer colors indicate higher pass rates (0–100%). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Per-model trajectories (Turns 1–12) for pragmatism and readability; techniques are color/marker [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Math domain — Drift from origin (mean) [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Math domain — Lexical novelty (mean) 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Math domain — Turn-to-turn volatility (mean) [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Math domain — Growth factor (mean) 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Per-model trajectories (Turns 1–12) for correctness and reasoning quality; techniques are color/marker [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Turn-wise analysis for (a) Claude-Sonnet-4.0 and (b) GPT-3.5-Turbo. Each plot shows: average answer correctness, average reasoning soundness (mean ± s.d.), cumulative percent solved, and a technique×turn accuracy heatmap. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Turn-wise analysis for GPT-OSS-20B: (a) average answer correctness by turn; (b) average reasoning soundness (mean ± s.d.); (c) cumulative percent solved; (d) technique×turn accuracy heatmap. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 12 canonical work pages

  1. [1]

    A Systematic Survey of Prompt Engineering in Large Language Models : Techniques and Applications , March 2025

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A Systematic Survey of Prompt Engineering in Large Language Models : Techniques and Applications , March 2025. URL http://arxiv.org/abs/2402.07927. arXiv:2402.07927

    Pith/arXiv arXiv 2025

  2. [2]

    Beyond Single - Turn : A Survey on Multi - Turn Interactions with Large Language Models , May 2025

    Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. Beyond Single - Turn : A Survey on Multi - Turn Interactions with Large Language Models , May 2025. URL http://arxiv.org/abs/2504.04717. arXiv:2504.04717

    Pith/arXiv arXiv 2025

  3. [3]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    work page 2022

  4. [4]

    IMPROVE : Iterative Model Pipeline Refinement and Optimization Leveraging LLM Experts , June 2025

    Eric Xue, Ke Chen, Zeyi Huang, Yuyang Ji, Yong Jae Lee, and Haohan Wang. IMPROVE : Iterative Model Pipeline Refinement and Optimization Leveraging LLM Experts , June 2025. URL http://arxiv.org/abs/2502.18530. arXiv:2502.18530

    arXiv 2025

  5. [5]

    SELF - REFINE : iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. SELF - REFINE : iterative refinement with self-feedback. In Proceedings of the 37th International Con...

    work page 2023

  6. [6]

    Reflexion: Language Agents with Verbal Reinforcement Learning , October 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning , October 2023. URL http://arxiv.org/abs/2303.11366. arXiv:2303.11366

    Pith/arXiv arXiv 2023

  7. [7]

    Phase Transitions in the Output Distribution of Large Language Models , May 2024

    Julian Arnold, Flemming Holtorf, Frank Schäfer, and Niels Lörch. Phase Transitions in the Output Distribution of Large Language Models , May 2024. URL http://arxiv.org/abs/2405.17088. arXiv:2405.17088

    Pith/arXiv arXiv 2024

  8. [8]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models . In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/foru...

    work page 2022

  9. [9]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems , NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc. event-place: New Orleans, LA, USA

    work page 2023

  10. [10]

    On the Effectiveness of Large Language Models in Domain - Specific Code Generation

    Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. On the Effectiveness of Large Language Models in Domain - Specific Code Generation . ACM Trans. Softw. Eng. Methodol., 34 0 (3), February 2025. ISSN 1049-331X. doi:10.1145/3697012. URL https://doi.org/10.1145/3697012

    doi:10.1145/3697012 2025

  11. [11]

    Reinforce LLM Reasoning through Multi - Agent Reflection

    Yurun Yuan and Tengyang Xie. Reinforce LLM Reasoning through Multi - Agent Reflection . In Forty-second International Conference on Machine Learning , 2025. URL https://openreview.net/forum?id=6k3oFS3Lbl

    work page 2025

  12. [12]

    LLMs Get Lost In Multi - Turn Conversation , May 2025

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs Get Lost In Multi - Turn Conversation , May 2025. URL http://arxiv.org/abs/2505.06120. arXiv:2505.06120

    Pith/arXiv arXiv 2025

  13. [13]

    Understanding the effects of iterative prompting on truthfulness

    Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Understanding the effects of iterative prompting on truthfulness. In Proceedings of the 41st International Conference on Machine Learning , ICML '24. JMLR.org, 2024. event-place: Vienna, Austria

    work page 2024

  14. [14]

    Survey of Hallucination in Natural Language Generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation . ACM Computing Surveys, 55 0 (12): 0 1--38, December 2023. ISSN 0360-0300, 1557-7341. doi:10.1145/3571730. URL https://dl.acm.org/doi/10.1145/3571730

    doi:10.1145/3571730 2023

  15. [15]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle : How Language Models Use Long Contexts . Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024. doi:10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9/

    doi:10.1162/tacl_a_00638 2024

  16. [16]

    LLM as a Broken Telephone : Iterative Generation Distorts Information

    Amr Mohamed, Mingmeng Geng, Michalis Vazirgiannis, and Guokan Shang. LLM as a Broken Telephone : Iterative Generation Distorts Information . In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 7493-...

    doi:10.18653/v1/2025.acl-long.371 2025

  17. [17]

    Critical Phase Transition in Large Language Models , October 2024

    Kai Nakaishi, Yoshihiko Nishikawa, and Koji Hukushima. Critical Phase Transition in Large Language Models , October 2024. URL http://arxiv.org/abs/2406.05335. arXiv:2406.05335

    Pith/arXiv arXiv 2024

  18. [18]

    Know When To Stop : A Study of Semantic Drift in Text Generation

    Ava Spataru, Eric Hambro, Elena Voita, and Nicola Cancedda. Know When To Stop : A Study of Semantic Drift in Text Generation . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies ( Volume 1: Long Papers ) , pages 3656--3671, Mexico City, Mexico, 2024. Associatio...

    doi:10.18653/v1/2024.naacl-long.202 2024

  19. [19]

    Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms, 2025

    Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms, 2025. URL https://arxiv.org/abs/2501.17399

    Pith/arXiv arXiv 2025

  20. [20]

    Evolving llms' self-refinement capability via iterative preference optimization, 2025

    Yongcheng Zeng, Xinyu Cui, Xuanfa Jin, Guoqing Liu, Zexu Sun, Dong Li, Ning Yang, Jianye Hao, Haifeng Zhang, and Jun Wang. Evolving llms' self-refinement capability via iterative preference optimization, 2025. URL https://arxiv.org/abs/2502.05605

    arXiv 2025

  21. [21]

    Magicore: Multi-agent, iterative, coarse-to-fine refinement for reasoning, 2024

    Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. Magicore: Multi-agent, iterative, coarse-to-fine refinement for reasoning, 2024. URL https://arxiv.org/abs/2409.12147

    arXiv 2024

  22. [22]

    Learning to clarify: Multi-turn conversations with action-based contrastive self-training

    Maximillian Chen, Ruoxi Sun, Tomas Pfister, and Sercan O Arik. Learning to clarify: Multi-turn conversations with action-based contrastive self-training. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=SIE6VFps9x

    work page 2025

  23. [23]

    Overconfidence in llm-as-a-judge: Diagnosis and confidence-driven solution, 2025

    Zailong Tian, Zhuoheng Han, Yanzhe Chen, Haozhe Xu, Xi Yang, Richeng Xuan, Houfeng Wang, and Lizi Liao. Overconfidence in llm-as-a-judge: Diagnosis and confidence-driven solution, 2025. URL https://arxiv.org/abs/2508.06225

    Pith/arXiv arXiv 2025

  24. [24]

    Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models, 2025

    Prateek Chhikara. Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models, 2025. URL https://arxiv.org/abs/2502.11028

    arXiv 2025

  25. [25]

    Taming overconfidence in llms: Reward calibration in rlhf, 2025

    Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in llms: Reward calibration in rlhf, 2025. URL https://arxiv.org/abs/2410.09724

    Pith/arXiv arXiv 2025

  26. [26]

    AI Models Collapse When Trained on Recursively Generated Data

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. AI Models Collapse When Trained on Recursively Generated Data . Nature, 631 0 (8022): 0 755--759, jul 2024. doi:10.1038/s41586-024-07566-y. URL https://doi.org/10.1038/s41586-024-07566-y

    doi:10.1038/s41586-024-07566-y 2024

  27. [27]

    Donoho, and Sanmi Koyejo

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. In First Conference on Lan...

    work page 2024

  28. [28]

    When models don't collapse: On the consistency of iterative mle, 2025

    Daniel Barzilai and Ohad Shamir. When models don't collapse: On the consistency of iterative mle, 2025. URL https://arxiv.org/abs/2505.19046

    arXiv 2025

  29. [29]

    Evaluating llm-based agents for multi-turn conversations: A survey, 2025

    Shengyue Guan, Haoyi Xiong, Jindong Wang, Jiang Bian, Bin Zhu, and Jian guang Lou. Evaluating llm-based agents for multi-turn conversations: A survey, 2025. URL https://arxiv.org/abs/2503.22458

    arXiv 2025

  30. [30]

    Self iterative label refinement via robust unlabeled learning, 2025

    Hikaru Asano, Tadashi Kozuno, and Yukino Baba. Self iterative label refinement via robust unlabeled learning, 2025. URL https://arxiv.org/abs/2502.12565

    arXiv 2025

  31. [31]

    Liveideabench: Evaluating llms' divergent thinking for scientific idea generation with minimal context, 2025

    Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, and Hao Sun. Liveideabench: Evaluating llms' divergent thinking for scientific idea generation with minimal context, 2025. URL https://arxiv.org/abs/2412.17596

    arXiv 2025

  32. [32]

    Ds-1000: A natural and reliable benchmark for data science code generation, 2022

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022. URL https://arxiv.org/abs/2211.11501

    Pith/arXiv arXiv 2022

  33. [33]

    Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024. URL https:/...

    Pith/arXiv arXiv 2024

  34. [34]

    Primack, Summer Yue, and Chen Xing

    Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, and Chen Xing. M ulti C hallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLM s. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar...

    2025

  35. [35]

    MT -eval: A multi-turn capabilities evaluation benchmark for large language models

    Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT -eval: A multi-turn capabilities evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p...

    doi:10.18653/v1/2024.emnlp-main.1124 2024

  36. [36]

    A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023. URL https://arxiv.org/abs/2303.10420

    Pith/arXiv arXiv 2023

  37. [37]

    System card: Claude opus 4 & claude sonnet 4

    Anthropic. System card: Claude opus 4 & claude sonnet 4. https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf, May 2025. Accessed: YYYY-MM-DD

    work page 2025

  38. [38]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  39. [39]

    gpt-oss-120b & gpt-oss-20b Model Card

    OpenAI . gpt-oss-120b & gpt-oss-20b Model Card . Technical report, OpenAI, August 2025. URL https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf. Model card, version posted Aug 5, 2025

    work page 2025

  40. [40]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    MTEB : Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB : Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014--2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics...

    doi:10.18653/v1/2023.eacl-main.148 2014

  42. [42]

    A diversity-promoting objective function for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 110--...

    doi:10.18653/v1/n16-1014 2016

  43. [43]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595--46623, 2023. URL https://papers.nips.cc/pap...

    work page 2023

  44. [44]

    Gemini 2.5 pro: Model card, June 2025

    Google . Gemini 2.5 pro: Model card, June 2025. URL https://storage.googleapis.com/model-cards/documents/gemini-2.5-pro.pdf. Last updated June 27, 2025

    work page 2025