GraphDancer: Training LLMs to Explore and Reason over Graphs via Two-Stage Curriculum Post-Training

Jianwen Xie; Ping Nie; Yuyang Bai; Yu Zhang; Zhuofeng Li

REVIEW 4 major objections 4 minor 13 references

GraphDancer: Training LLMs to Explore and Reason over Graphs via Two-Stage Curriculum Post-Training

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Curriculum-based reinforcement learning lets a 3B model outpace 14B backbones on unseen graph reasoning tasks.

desk verdict Plausible curriculum-RL idea for graph tool-use, but the difficulty-labeling step is underspecified and the two-stage claim in the abstract doesn't match the body. read the letter →

arxiv 2602.02518 v2 pith:OY645JZ7 submitted 2026-01-24 cs.LG cs.AIcs.CL

Yuyang Bai , Zhuofeng Li , Ping Nie , Jianwen Xie , Yu Zhang This is my paper

classification cs.LGcs.AIcs.CL

keywords graphreasoningcurriculumreinforcementlearningheterogeneousgraphsLLMagentstoolusemulti-hopcross-domaingeneralizationinformationseeking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GraphDancer argues that the obstacle to LLM reasoning over graph-structured knowledge is not model scale but the order in which a model meets training questions. It trains a 3B language model with reinforcement learning to interleave natural-language reasoning with deterministic graph function calls, using a rule-based reward that credits correct final answers and well-formed interaction traces. The core move is a graph-aware curriculum that classifies questions by structural difficulty — single-round lookups versus multi-round neighborhood expansions — and samples them in an easy-to-hard, biased mixture during RL. Trained only on an academic graph domain, the resulting policy achieves higher average answer accuracy across four unseen domains and out-of-distribution question types than prompting baselines built on 14B backbones or a stronger closed-source model. If right, this shows that small models can internalize portable tool-use skills when the training signal is organized by interaction complexity.

What carries the argument

The central object is the graph-aware curriculum built on an information-seeking round decomposition. A round is a maximal set of graph function calls that depend only on the state at the start of the round, and therefore could in principle execute in parallel; its 'scope' is the set of node identifiers returned. Rounds divide into singleton lookups (exactly one node) and neighborhood expansions (multiple nodes). Question difficulty is then structural: Easy requires one round, Medium multiple rounds but at most one expansion, Hard at least two expansions. The training signal during RL is a rule-based reward combining exact-match answer correctness, a mild penalty for correct answers with mal

What would settle it

An empirical control would be to train the same RL pipeline with identical rewards and hyperparameters on the same ACADEMIC data under three conditions: uniform sampling (no curriculum), curriculum labels shuffled, and curriculum labels reversed. If the curriculum variant does not consistently beat both controls on the four unseen domains, then ordering difficulty is not the source of the gains. Reproducibility also requires the authors to release the round-decomposition labeling procedure, so a reader can check whether the labels are computed from gold answers alone.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a moderate-sized LLM can internalize generalizable graph-exploration skills through curriculum-based RL, provided difficulty is defined structurally rather than by content. GraphDancer models graph question answering as an episodic MDP in which the agent emits reasoning blocks and graph function calls, and the environment injects observations; only agent tokens receive gradients. Difficulty is computed from the round decomposition of an information-seeking trajectory: each round is classified as a singleton lookup (one node identifier surfaced) or an expansion (several node identifiers surfaced), and questions are labeled Easy (one round),

Load-bearing premise

The load-bearing premise is that each training question can be reliably labeled Easy/Medium/Hard from an information-seeking round decomposition, but the paper never specifies how that decomposition is computed for the training set; if the labels are arbitrary or come from the model's own behavior, the curriculum — and therefore the claimed cause of the improvement — collapses.

Editorial extensions

If this is right

A 3B model trained this way reaches higher average accuracy than prompting baselines with a 14B backbone or a stronger closed-source model on four unseen domains.
Curriculum order reduces format-validity collapse: GraphDancer keeps much higher format-valid multi-round interaction rates than vanilla RL on Medium and Hard subsets across domains.
Loop/timeout episodes nearly halve relative to vanilla RL and the base model, indicating the learned policy navigates more purposefully.
Format-validity gains on OOD questions do not translate into higher answer quality (evidence-hit is zero by design), suggesting the curriculum mainly strengthens structural tool use, not open-ended answer generation.
Although the curriculum helps in most domains, transfer is not uniform: the paper reports a performance drop on LITERATURE relative to vanilla RL, indicating domain-specific limitations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The round decomposition that generates Easy/Medium/Hard labels for the training set is stated but never specified; if the labels come from an unstated oracle or from the model's own rollouts, the curriculum effect could be confounded. Publishing the labeling algorithm or releasing the labeled data is a necessary reproducibility step.
A cleaner control would be to train vanilla RL on the same data with identical rewards and a shuffled or reversed curriculum; if uniform sampling matches the curriculum variant, the claimed benefit is really a data-filtering or reward-shaping effect, not the ordering.
Because difficulty is defined purely by round and expansion counts, the same curriculum schedule should transfer to other structured tool-use environments (SQL databases, web APIs, search engines), where multi-step evidence gathering is the bottleneck; this is a testable extension beyond graphs.
The abstract's 'two-stage' description (first rule-based interaction, then preference for efficient trajectories) does not match the single RL loop in Algorithm 1; readers should interpret the stages as optional SFT warmup plus curriculum RL, or request clarification.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: GraphDancer trains a 3B Qwen model to call graph functions via RL with a difficulty curriculum based on a new structural measure (S-rounds vs E-rounds). The cross-domain results are suggestive, but as written the paper's central mechanism is not reproducible because the paper never says how the difficulty labels are computed, and the abstract promises a two-stage framework the body doesn't implement.

What's actually new: the graph-aware difficulty metric is a reasonable way to schedule tool-use training, and the biased-mixture sampler is a sensible addition to avoid brittle pure easy-to-hard schedules. The paper deserves credit for evaluating on unseen domains and OOD question types, and the qualitative case study plus behavioral diagnostics (format validity, loop reduction) give concrete evidence that the curriculum changes behavior beyond just score improvements. The ablation against Vanilla RL is present and mostly favorable.

The soft spots, in order:

1. Load-bearing: Algorithm 1 line 4 calls for a 'round decomposition' but no procedure is given. Section 2.4 defines difficulty in terms of rounds and E-rounds, but never says where those rounds come from. There's no gold trace, no rollout-based estimator, no mention of using ground-truth answer paths. The entire justification for the curriculum and the comparison to Vanilla RL hinges on this. If the labels are computed from the model's own rollouts, the curriculum is confounded with the policy being trained, and Table 3's difficulty-wise splits are not comparable across models. This is an internal specification gap, and it's the main reason the paper can't be accepted as is.

2. The abstract describes two stages (first learning to interact, second preferring grounded/efficient trajectories) but Section 2 and Algorithm 1 describe a single RL loop with a curriculum. The title even differs between the abstract and the full text. This suggests either the framework was simplified late in the writing or the abstract is aspirational.

3. Reward coefficients lambda_struct and lambda_final are defined in Eq. 5 but never given values. A reader can't reproduce the reward.

4. The evaluation is a single run. The average gains over the strongest baseline are around 0.8 Rouge-L and 0.2 GPT4Score, and LITERATURE is worse. The claim 'outperforms' is technically true but fragile without variance or seeds.

5. The closest RL baselines (Graph-R1, Structure-R1) are cited in related work but not evaluated. That weakens the comparison.

My take: the method is promising, the thinking is clear, but the missing labeling step is a real obstacle. This paper deserves a serious referee, and with the experiment details filled in and the abstract fixed it could be a solid contribution. Without that, it's an interesting but unverifiable recipe.

Recommendation: send it to review, but expect a request for the missing specification and multi-seed results.

Referee Report

4 major / 4 minor

Summary. The paper proposes GraphDancer, an RL post-training method that teaches a Qwen2.5-3B model to interact with heterogeneous graph environments through typed function calls, interleaving reasoning blocks with executable graph actions. The central novelty is a graph-aware curriculum that organizes training examples by the structural difficulty of their information-seeking trajectories (Easy/Medium/Hard based on the number of rounds and of neighborhood-expansion rounds). The authors train only on the ACADEMIC domain of GRBench and test on four unseen domains plus out-of-distribution question types, reporting that the 3B GraphDancer model outperforms stronger baselines, including Graph-CoT with Qwen3-14B and GPT-4o-mini. The paper also presents behavioral diagnostics (format validity, call validity, evidence hit) and a qualitative case study to argue that the curriculum improves multi-round graph tool use.

Significance. If the empirical claims hold, the result would be valuable: it would show that curriculum-based RL can internalize graph-tool-use skills in a small model and transfer them across domains and schemas, which is directly relevant to making LLM agents more grounded. The strengths are the clean rule-based reward, the deterministic graph environment, the one-domain-to-many-domains evaluation protocol, and the inclusion of behavioral diagnostics that go beyond answer accuracy. The public code/model links also aid reproducibility. However, the paper's key methodological component—the round-decomposition procedure that defines the curriculum—is not specified, and the empirical support for the headline claim is thin (single run, small average margins, and non-uniform gains). The significance is therefore conditional on resolving these issues.

major comments (4)

[§2.4; Algorithm 1 line 4] The graph-aware curriculum is the paper's central novelty and the only difference from the Vanilla RL ablation, but the 'round decomposition' that assigns each training question to Easy/Medium/Hard is never defined. Section 2.4 defines S-rounds and E-rounds and the difficulty categories once a trajectory is decomposed, but it does not state how this decomposition is obtained for training examples: there is no gold interaction trace, no oracle, no rollout-based estimator, and no description of using ground-truth answer paths. If the decomposition is computed from the model's own rollouts, the difficulty labels are policy-dependent and the Easy/Medium/Hard splits in Table 3 are not comparable across methods; if it is computed from an external reference procedure, that procedure is omitted. This must be specified precisely for the central claim to be reproducible.
[Abstract; §2.6/Algorithm 1] The abstract and the title describe a 'two-stage' framework—stage one teaches graph interaction under rule-based rewards, and stage two teaches more grounded and efficient trajectories—but the body does not implement or even define a second stage. Section 2.6 and Algorithm 1 describe a single RL loop with one reward function (Eq. 5) and one curriculum sampler. If the intended second stage is the biased-mixture curriculum or a separate phase, it is not written down; if not, the abstract is misleading. The complete training recipe should be stated in one place so readers know exactly what is being evaluated.
[Table 2; Limitations] The headline claim that a 3B model 'outperforms' 14B or GPT-4o-mini baselines is supported by very small average differences: the average Rouge-L advantage over Graph-CoT-Qwen3-14B is 0.77 points and the GPT4Score advantage is 0.24 points. No variance, number of seeds, or significance tests are reported, and the Limitations section explicitly says a systematic multi-seed ablation is left to future work. Given the single-run setup and the fact that GraphDancer is worse than both Vanilla RL and Graph-CoT-Qwen3-14B on LITERATURE, the unqualified 'outperforms' claim is not statistically supported. Report per-seed mean and standard deviation, or soften the claim accordingly.
[Table 3; §3.3] The claim of 'robust cross-domain generalization' is weakened by the difficulty-wise results. On LITERATURE Hard, GraphDancer drops to 2.53 Rouge-L versus 15.28 for Vanilla RL and 6.59 for Graph-CoT; on HEALTHCARE Hard it is 0.00. On LEGAL OOD it is also worse than Vanilla RL (16.11 vs. 19.33). The paper acknowledges non-uniformity in prose, but the abstract and conclusion still state robust generalization. If the curriculum actively causes these regressions, that needs analysis; if the regressions are small-sample noise, that reinforces the need for variance reporting.

minor comments (4)

[Appendix A.3] The reward coefficients λ_struct and λ_final in Eq. (5) are never given numerical values. Also, the Gaussian curriculum schedule is described as having β=3 and σ=0.75, but β is not defined in Section 2.5; it is also used as the KL penalty coefficient in Algorithm 1. Please disambiguate and provide the missing hyperparameters.
[Appendix B.1; Table 8] The text says 'EH is 0 across all three OOD domains,' but Table 8 shows EH=45.00 for LEGAL for all three methods. This internal inconsistency should be corrected.
[§2.2; Appendix prompt template] Section 2.2 says an action block '<graph>...</graph>' may contain one or more function calls, but the prompt template in Table 9 states 'issue EXACTLY ONE function per step.' Please reconcile these specifications, as they affect both the round definition and the reproducibility of the interaction protocol.
[§2.3; Appendix A.3] Equation (1) is a KL-regularized expected-reward objective, but the training details mention policy clip ratio and value clip range, suggesting a PPO-style surrogate. The actual update objective should be stated explicitly rather than left as the abstract optimization in Eq. (1).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GRAPHDANCER is an empirical training recipe with direct benchmark evaluations; the underspecified round-decomposition step is a reproducibility gap, not a self-referential derivation.

full rationale

The paper's claimed results are measured outcomes of an RL training procedure evaluated on an external public benchmark (GRBench). There is no fitted parameter that is later renamed as a prediction, and no equation reduces the reported Rouge-L/GPT4Score numbers to the curriculum's inputs by construction. The curriculum's difficulty labels (Easy/Medium/Hard) are defined structurally (Section 2.4) in terms of S-rounds and E-rounds, and the same definition is used for both training sampling and evaluation stratification; this is a consistent experimental design rather than a circular derivation. The genuine weakness is Algorithm 1 line 4, which calls for a 'round decomposition' of each training instance without specifying how it is obtained from D=(x,G,y*), and Section 2.4 defines rounds only after a trajectory exists. This is an omitted algorithm/reproducibility gap and a potential confound, but the paper does not claim to derive the curriculum's benefit from those labels, so it does not meet the bar for circularity. Citations to Parashar et al. (2025) and Jin et al. (2024) involve an overlapping author (Yu Zhang), but they support a standard Gaussian sampling formula and provide the benchmark/baseline, not an unverified uniqueness theorem; independent foundations (e.g., Bengio et al. 2009) are also cited. Under the rule that only explicit reductions count as circularity, this paper receives 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The ledger shows the paper contributes a training recipe rather than a law-like derivation. The main hand-set quantities are the reward coefficients (unreported) and the curriculum parameters (reported). The difficulty decomposition is an ad hoc operationalization, not a foundational quantity.

free parameters (7)

λ_struct (reward structural penalty coefficient)
Coefficient in Eq. 5 penalizing correct answers with malformed interaction traces; value never reported in Section A.1 or A.3.
λ_final (reward final-answer shaping coefficient)
Coefficient in Eq. 5 rewarding well-formed incorrect attempts; value never reported.
Gaussian curriculum variance σ = 0.75
Hand-set in Section A.3; controls the spread of the easy-to-hard Gaussian schedule.
η_start (mixture start weight) = 0.2
Hand-set in Section A.3; initial mixing strength of the bias prior.
η_end (mixture end weight) = 0.8
Hand-set in Section A.3; final mixing strength.
bias prior q = [0.5, 0.5, 0]
Hand-set in Section A.3; fixed sampling weights over Easy/Medium/Hard.
max_turns T = 10
Interaction budget; hand-set in Section A.3.

assumptions (4)

domain assumption Graph functions (RetrieveNode, NodeFeature, NeighborCheck, NodeDegree) are deterministic and sufficient to answer all training questions.
Section 2.2 assumes the environment is complete and noise-free, which may not hold for real-world graphs.
ad hoc to paper The structural difficulty (number of E-rounds) is a valid proxy for question difficulty.
Section 2.4 asserts this correlation without independent validation against other difficulty measures.
domain assumption Exact-match on the final answer is a sufficient training reward.
Section A.1 assumes the gold answer string is uniquely defined; this may not hold for open-ended OOD questions.
domain assumption A fixed interaction budget of 10 rounds is enough to reach optimal answers.
Section A.3 sets T=10 without evidence that longer horizons could improve performance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GraphDancer: Training LLMs to Explore and Reason over Graphs via Two-Stage Curriculum Post-Training." pith.science (2026). https://pith.science/paper/OY645JZ7

@misc{pith2026260202518,
  author       = {Pith},
  title        = {Pith review of: GraphDancer: Training LLMs to Explore and Reason over Graphs via Two-Stage Curriculum Post-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OY645JZ7}},
  note         = {Machine review of arXiv:2602.02518}
}

read the original abstract

Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many real-world knowledge sources are organized as heterogeneous graphs rather than plain text. Reasoning over such graphs requires models to follow schema-defined relations through precise function calls and to aggregate evidence across multiple rounds of interaction. We propose GraphDancer, a two-stage post-training framework that teaches LLMs to reason over graphs by interleaving natural-language reasoning with graph function execution. The first stage teaches the model how to interact with the graph under rule-based rewards, while the second stage further teaches it to prefer more grounded and efficient interaction trajectories. The key novelty of GraphDancer is a graph-aware curriculum that organizes both stages by the structural complexity of information-seeking trajectories, progressively increasing task difficulty during training. We evaluate GraphDancer on a multi-domain benchmark by training on one domain only and testing on unseen domains and out-of-distribution question types. Despite using only a 3B backbone, GraphDancer outperforms baselines equipped with larger/stronger backbones, demonstrating robust cross-domain generalization of graph exploration and reasoning skills. Our code can be found at https://github.com/leopoldwhite/GraphDancer.

Figures

Figures reproduced from arXiv: 2602.02518 by the authors.

**Figure 1.** Overview of GRAPHDANCER framework. LLM reasoning (Guo et al., 2025) and information seeking (Jin et al., 2025), but also by the natural formulation of graph interaction as a decisionmaking process: graphs can be viewed as environments, function calls as actions, and the learning of graph interaction as exploring the environment through actions. Meanwhile, to make this solution applicable to open-source, moderat… view at source ↗

**Figure 3.** Outcome breakdown over all evaluation episodes. Each bar decomposes episodes into Correct and three failure modes: Invalid Format, Loop / Timeout, and Premature Stop. GRAPHDANCER increases the overall success rate and noticeably reduces Loop / Timeout, while the remaining errors are more concentrated in Premature Stop. which reflects a trade-off between exploration depth and timely termination, especially for mult… view at source ↗

**Figure 4.** Original reasoning trajectories for the case study discussed in Section [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 linked inside Pith

[2]

Do NOT include any other text in <graph>

Inside <graph>...</graph>, issue EXACTLY ONE function per step. Do NOT include any other text in <graph>
[3]

Do NOT fabricate <information>; it is ONLY produced by the environment immediately after your <graph> step
[4]

InICLR’25

Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InICLR’25. Youssef Mroueh. 2025. Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynam- ics, and success amplification.arXiv preprint arXiv:2503.06639. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll W...

arXiv 2025
[5]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn

Curriculum reinforcement learning from easy to hard tasks improves llm reasoning.arXiv preprint arXiv:2506.06632. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn

arXiv
[6]

InNeurIPS’23

Direct preference optimization: Your language model is secretly a reward model. InNeurIPS’23. Adam Santoro, David Raposo, David G Barrett, Ma- teusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural net- work module for relational reasoning. InNeurIPS’17, pages 4967–4976. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì,...

2017
[7]

10 Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and 1 others

Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599. 10 Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and 1 others. 2024. Freshllms: Re- freshing large language models with search engine augmentation. InFindings of ACL’24, pages 13697– 13720. Junlin Wu,...

arXiv 2024
[8]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 others

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 others

arXiv
[12]

Do NOT put a graph call inside <think>, and do NOT put thoughts inside <graph>

Keep thoughts concise and ONLY inside <think>. Do NOT put a graph call inside <think>, and do NOT put thoughts inside <graph>

Show all 13 references

[13]

The final output MUST contain ONLY one <answer>...</answer> block with the requested node main features (e.g., names), not node IDs. In-context examplesHere are some examples: {examples} (END OF EXAMPLES) Graph schemaDefinition of the graph: {graph_definition} QuestionQuestion...

2024
[2005]

InIJCNN’05, pages 729–734

A new model for learning in graph domains. InIJCNN’05, pages 729–734. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseek- r1 incentivizes reasoning in llms through reinforce- ment ...

2025 arXiv
[2023]

Junqi Gao, Xiang Zou, Ying Ai, Dong Li, Yichen Niu, Biqing Qi, and Jianxing Liu

Simteg: A frustratingly simple approach improves textual graph learning.arXiv preprint arXiv:2308.02565. Junqi Gao, Xiang Zou, Ying Ai, Dong Li, Yichen Niu, Biqing Qi, and Jianxing Liu. 2025. Graph counselor: Adaptive graph exploration via multi-agent synergy to enhance llm re...

2025 arXiv
[2024]

feature" key NodeDegree[Node, neighbor_type] ## returns the number of

Qwen2.5 technical report.arXiv preprint arXiv:2412.15115. Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. 2024. Language is all a graph needs. InFindings of EACL’24, pages 1955–1973. Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, ...

2024 arXiv
[2025]

arXiv preprint arXiv:2505.00949

Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949. Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadom- ski, Piotr Nyczyk, and 1 others. 2024. Graph of tho...

2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Do NOT include any other text in <graph>

Inside <graph>...</graph>, issue EXACTLY ONE function per step. Do NOT include any other text in <graph>

[2] [3]

Do NOT fabricate <information>; it is ONLY produced by the environment immediately after your <graph> step

[3] [4]

InICLR’25

Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InICLR’25. Youssef Mroueh. 2025. Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynam- ics, and success amplification.arXiv preprint arXiv:2503.06639. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll W...

arXiv 2025

[4] [5]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn

Curriculum reinforcement learning from easy to hard tasks improves llm reasoning.arXiv preprint arXiv:2506.06632. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn

arXiv

[5] [6]

InNeurIPS’23

Direct preference optimization: Your language model is secretly a reward model. InNeurIPS’23. Adam Santoro, David Raposo, David G Barrett, Ma- teusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural net- work module for relational reasoning. InNeurIPS’17, pages 4967–4976. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì,...

2017

[6] [7]

10 Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and 1 others

Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599. 10 Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and 1 others. 2024. Freshllms: Re- freshing large language models with search engine augmentation. InFindings of ACL’24, pages 13697– 13720. Junlin Wu,...

arXiv 2024

[7] [8]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 others

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 others

arXiv

[8] [12]

Do NOT put a graph call inside <think>, and do NOT put thoughts inside <graph>

Keep thoughts concise and ONLY inside <think>. Do NOT put a graph call inside <think>, and do NOT put thoughts inside <graph>