Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents

Alice Oh; Eunsu Kim; Juhyun Oh

REVIEW 4 major objections 5 minor 12 references

Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Single-turn scores hide multi-turn planning failures

desk verdict A genuinely new benchmark for multi-turn travel planning, undermined by single-run evaluations that leave the headline findings within noise. read the letter →

arxiv 2506.04649 v1 pith:VBKDFNJS submitted 2025-06-05 cs.CL

Juhyun Oh , Eunsu Kim , Alice Oh This is my paper

classification cs.CL

keywords flexibleplanningmulti-turnevaluationconstraintprioritizationtravelbenchmarklargelanguagemodelsdynamicintroductionpassrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flex-TravelPlanner is a benchmark for testing whether language models can revise travel plans as new requirements arrive over multiple turns, rather than from a single static prompt. The paper's central claim is that a model's performance on single-turn planning says little about how it will handle multi-turn adaptation: GPT-4o and Llama 3.1 70B both show order-dependent success, with global constraints like budget degrading when local constraints are added later. The paper also claims that models systematically mishandle priorities, satisfying newly introduced soft preferences (cuisine or restaurant rating) at the expense of the hard budget constraint. If these findings hold, planning evaluations need to measure dynamic constraint handling and priority maintenance, not just static correctness.

What carries the argument

The benchmark's central object is the multi-turn constraint-adaptive scenario: the same query is presented all at once, in two turns, or in three turns, with constraints split into global (budget, affecting the whole itinerary) and local (room rule, room type, cuisine, rating). The scoring mechanism is the Constraint Pass Rate, the ratio of satisfied constraints to total constraints across all generated plans. The priority-aware setting constructs conflicts by setting a budget too low to satisfy both the hard budget and a soft preference, so the correct behavior is defined as keeping the budget and dropping the preference. That design lets the authors measure not only whether constraints are met, but whether the hierarchy between hard and soft constraints is preserved when new information arrives.

What would settle it

A direct test is to present the same 134 priority-aware scenarios to human judges and ask whether they would prefer a budget-compliant plan that ignores the stated cuisine preference or a plan that overspends to honor it; if most judges choose the overspending plan, the benchmark's ground-truth priority rule is not a valid measure of correct behavior. A complementary check is to swap the order in which the budget and the preference are introduced and see whether the budget-violation rate changes.

Watch

Extended reading notes

Core claim

The paper's central claim is that flexible planning—revising an itinerary as constraints arrive across turns—is not predictable from static planning ability. Using 120 TravelPlanner queries in all-at-once, two-turn, and three-turn formats, it finds that the two models' relative ranking flips between all-at-once and multi-turn presentation, and both models lose grip on the budget when local constraints are introduced afterward. In a separate set of 134 conflicting-constraint scenarios, 59–62 percent of plans violate the hard budget in order to honor a newly stated cuisine or rating preference. The paper concludes that current LLMs lack reliable constraint-hierarchy maintenance in dynamic settings.

Load-bearing premise

The load-bearing premise is that the hard budget constraint should always outrank a soft cuisine or rating preference, so a plan that overspends for a preferred food is judged a failure; if users would rather overspend, the benchmark's 'correct prioritization' is not actually correct.

Editorial extensions

If this is right

Single-turn planning benchmarks should not be used alone to certify an agent's readiness for interactive planning tasks.
Evaluators should control and report the order in which constraints are revealed, because order alone can shift global-constraint pass rates by roughly fifteen percentage points in two-turn settings.
Models need explicit mechanisms to track constraint priority across turns, since neither GPT-4o nor Llama 3.1 70B reliably keeps budget constraints intact when preferences are added later.
Sequential introduction of constraints may be a useful prompt-level strategy for weaker models, since Llama improves in multi-turn settings compared with all-at-once presentation.
The benchmark's released code and dataset give other researchers a direct way to reproduce the order and priority effects before applying fixes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension of the paper's findings would swap the order in which the budget and the soft preference are introduced; if budget violations drop, the failure is recency-driven rather than a true priority-reasoning failure.
The same conflict structure is likely to appear in other evolving-requirement domains such as project scheduling or logistics, since nothing in the result depends on travel specifics.
The benchmark's ground-truth priority rule could be validated by a human preference study; if many users accept overspending for a favored cuisine, the scoring would need to weigh subjective utility rather than treating every budget breach as an error.
Giving models explicit labels of which constraints are hard and which are soft would isolate whether the deficit is attentional (losing track of priorities) or inferential (not understanding the hierarchy).

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on Flex-TravelPlanner. The benchmark is a genuine step forward: no one else has combined sequential constraint introduction with explicit priority conflicts for travel planning, and the global/local constraint distinction is a useful lens. The authors also release their code and dataset, which is real value.

Strongest finding, to me, is the budget pass-rate erosion: when local constraints are added after the budget, both models often let the budget slip. That pattern is visible in Figure 4 and is not dependent on a single arbitrary metric. The order effects and the single-turn vs multi-turn disconnect are more fragile.

The problem is that all headline results come from a single run per query. One generation, no temperature reported, no seeds, no confidence intervals, no significance tests. With 120 queries, the standard error on a pass rate is roughly 4 percentage points, so the reported differences—e.g., GPT-4o 0.62 vs Llama 0.63 in 2-turn +L, or the 0.52-to-0.67 order effect for Llama—could be noise. The 'disconnect' claim rests on a rank reversal between models across conditions; that reversal is within a few points and needs a test.

The priority-aware ground truth is a softer issue. The authors assume the budget should outrank a cuisine preference; they deliberately construct budgets that are too low, so it is a reasonable operationalization, but it is an assumption. A quick user study or at least a sensitivity analysis would help. Minor: the query selection from the validation set is not justified, and the citations for GPT-4o and Llama 3.1 are wrong (they point to the GPT-4 and Llama 3 technical reports).

In sum, this is a well-designed benchmark paper with a real hole in the statistical rigor. The benchmark deserves to be used and cited, but the empirical claims should be read as preliminary. Send it to peer review, with the expectation that the authors add multiple runs or at least significance tests, and clarify the prioritization assumption.

Referee Report

4 major / 5 minor

Summary. The paper introduces Flex-TravelPlanner, a benchmark derived from TravelPlanner that evaluates language models in two dynamic planning settings: (1) constraint-adaptive plan revision, where constraints are introduced sequentially over 1, 2, or 3 turns, and (2) priority-aware plan revision, where soft cuisine/rating preferences conflict with a hard budget constraint. The authors evaluate GPT-4o and Llama 3.1 70B zero-shot on these settings and report three findings: single-turn performance does not predict multi-turn performance, constraint introduction order affects pass rates, and both models often violate hard budget constraints to accommodate low-priority preferences. The dataset and code are publicly released.

Significance. If the findings hold, the benchmark would be a useful complement to static planning benchmarks by operationalizing constraint introduction and priority conflicts. The paper's main contributions are the two evaluation settings and a public implementation, which can support future work on dynamic planning. The central empirical claims, however, rest on single-run pass rates over a small set of queries without confidence intervals or significance tests, and the priority-aware ground truth is an assumption rather than a validated user preference model. The benchmark itself is a reasonable starting point, but the paper needs stronger statistical grounding and clearer dataset construction details before the findings can be taken as established.

major comments (4)

[§3.2, Figures 2–3] The three headline findings are all based on a single generation per query (120 or 134 queries) with no reported temperature, seeds, confidence intervals, or significance tests. Several of the differences that support the rank-reversal claim are 1–2 percentage points (e.g., 2-turn +G local pass rates 0.62 for both GPT-4o and Llama; 2-turn +L global 0.47 vs. 0.45). With binary outcomes over 120 queries, the standard error is roughly 4–5 percentage points, so the observed cross-over is within sampling noise. The order effect (Llama 2-turn global 0.52 vs. 0.67) and the priority failure rates (59% vs. 61.9%) also need error bars and paired significance tests (e.g., McNemar or bootstrap over queries) across multiple runs.
[§2.2] The construction of the 120-query multi-turn set is not justified: no criteria are given for how these queries were selected from TravelPlanner's validation set, whether the selection was random, stratified, or filtered for solvability, and no distribution of query attributes (group size, number of local constraints, constraint types) is reported. Without this, it is unclear whether the observed effects are specific to this subset or general to TravelPlanner queries. The same applies to the 134 priority-aware scenarios.
[§3.2 / Figure 5 / Appendix A.1] In the priority-aware experiments, the budget is deliberately set so that the cuisine/rating preference cannot be satisfied together with all other local constraints at their minimum options. The paper then treats satisfying the budget as the uniquely correct behavior because the preference is phrased 'if possible'. This ordering is an assumption, not a validated ground truth; no user study or preference elicitation supports the claim that users would never accept overspending to satisfy a stated cuisine preference. At minimum, the paper should report how often model plans satisfy the preference when the budget is violated, and should discuss the sensitivity of the 'prioritization failure' finding to the hard/soft ordering.
[§2.2 / Eq. (1)] The evaluation metric counts a constraint as passed or failed, but the paper does not describe how pass/fail is computed from the model's generated plan string, nor whether the rule-based validator from TravelPlanner is used or reimplemented. To make the benchmark reproducible, include the parser/validator description and the exact set of constraints checked per query.

minor comments (5)

[§3.1] The references for the two models are swapped: GPT-4o should cite the OpenAI GPT-4 technical report, and Llama 3.1 70B should cite Dubey et al.
[§3.1] The text states experiments are conducted 'without any prompting', but the appendix includes a prompt template; what is meant is zero-shot (no in-context examples), which should be clarified.
[Figures 2–3] The legend and table use 'GPT-4' while the text uses 'GPT-4o'; use a single consistent model name.
[Figure 1] The figure illustrates a budget revision ('budget is changed to $1,500'), but Section 2.2 states that only constraint addition, not revision, is tested in this study; this discrepancy should be fixed in the figure or text.
[Eq. (1)] The notation is ambiguous about whether multi-turn plans contribute all intermediate plans or only the final plan to the set P; clarify this in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's findings are empirical pass-rate measurements against an externally defined ground truth, with no fitted parameters renamed as predictions and no load-bearing self-citation chain.

full rationale

Flex-TravelPlanner is an evaluation benchmark; its central claims are empirical pass-rate measurements on fixed query sets, not derivations from fitted parameters or self-citations. The constraint-adaptive results compare performance across presentation orders on identical constraint sets, and the pass rates are computed directly from model outputs via the stated Constraint Pass Rate formula, so the order effects and the single-turn/multi-turn disconnect are measured, not constructed. The priority-aware scenarios deliberately make full satisfaction infeasible (Appendix A.1), and the claim that models 'incorrectly' favor preferences imports a normative judgment that the budget should outrank soft preferences; this is an evaluative assumption about ground truth, not a circular derivation, and it does not force the observed frequencies (e.g., 59% vs. 61.9%) because models could in principle choose budget-first more often. The only self-citation (Kim et al., 2024) supports a background claim about existing interactive-evaluation work and is not load-bearing. No fitted input is relabeled as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled via citation. Statistical concerns about single runs and lack of significance tests are correctness and robustness risks, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted in this benchmark paper. The central claims rest on domain assumptions about the base dataset, the constraint checker, and the priority ordering.

assumptions (3)

domain assumption TravelPlanner validation set queries with both global and local constraints are representative of planning tasks.
The paper selects 120 queries from TravelPlanner's validation set without specifying selection criteria beyond having both constraint types.
domain assumption Constraint satisfaction can be reliably and completely checked by the TravelPlanner evaluation pipeline.
The paper relies on the underlying TravelPlanner validator to compute Constraint Pass Rate but does not describe or audit this checker.
domain assumption Language models should infer that 'if possible' preferences are soft relative to hard budget constraints.
The prompt template does not explicitly state the priority; the paper assumes models should treat budget as higher priority.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents." pith.science (2026). https://pith.science/paper/VBKDFNJS

@misc{pith2026250604649,
  author       = {Pith},
  title        = {Pith review of: Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBKDFNJS}},
  note         = {Machine review of arXiv:2506.04649}
}

read the original abstract

Real-world planning problems require constant adaptation to changing requirements and balancing of competing constraints. However, current benchmarks for evaluating LLMs' planning capabilities primarily focus on static, single-turn scenarios. We introduce Flex-TravelPlanner, a benchmark that evaluates language models' ability to reason flexibly in dynamic planning scenarios. Building on the TravelPlanner dataset~\citep{xie2024travelplanner}, we introduce two novel evaluation settings: (1) sequential constraint introduction across multiple turns, and (2) scenarios with explicitly prioritized competing constraints. Our analysis of GPT-4o and Llama 3.1 70B reveals several key findings: models' performance on single-turn tasks poorly predicts their ability to adapt plans across multiple turns; constraint introduction order significantly affects performance; and models struggle with constraint prioritization, often incorrectly favoring newly introduced lower priority preferences over existing higher-priority constraints. These findings highlight the importance of evaluating LLMs in more realistic, dynamic planning scenarios and suggest specific directions for improving model performance on complex planning tasks. The code and dataset for our framework are publicly available at https://github.com/juhyunohh/FlexTravelBench.

Figures

Figures reproduced from arXiv: 2506.04649 by the authors.

**Figure 2.** Local, global and average pass rates of the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Local (L), Global (G), and Average (Avg.) constraint pass rates across conditions. 0 25 50 75 100 125 150 175 Count LLaMA (2-turn) GPT (2-turn) LLaMA (3-turn) GPT (3-turn) 25 (13.2%) 21 (11.1%) 59 (31.2%) 84 (44.4%) 31 (16.6%) 19 (10.2%) 63 (33.7%) 74 (39.6%) 47 (24.9%) 7 (3.7%) 50 (26.5%) 85 (45.0%) 35 (18.5%) 18 (9.5%) 53 (28.0%) 83 (43.9%) Budget Constraint Changes after Adding New Local Constraint True2False Fal… view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Preference and global constraint pass rate after conflicting preference constraint introduction in the second turn [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

[1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024
[3]

Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation

Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, and Alice Oh. Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation. arXiv preprint arXiv:2412.10424, 2024

arXiv 2024
[4]

MT -eval: A multi-turn capabilities evaluation benchmark for large language models

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT -eval: A multi-turn capabilities evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp....

doi:10.18653/v1/2024.emnlp-main.1124 2024
[5]

On the planning abilities of large language models-a critical investigation

Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36: 0 75993--76005, 2023

2023
[6]

Mint: Evaluating llms in multi-turn interaction with tools and language feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024

work page 2024
[7]

Travelplanner: A benchmark for real-world planning with language agents

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622, 2024

arXiv 2024
[8]

Natural plan: Benchmarking llms on natural language planning

Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520, 2024

arXiv 2024

Show all 12 references

[9]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[10]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[11]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[12]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

[3] [3]

Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation

Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, and Alice Oh. Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation. arXiv preprint arXiv:2412.10424, 2024

arXiv 2024

[4] [4]

MT -eval: A multi-turn capabilities evaluation benchmark for large language models

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT -eval: A multi-turn capabilities evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp....

doi:10.18653/v1/2024.emnlp-main.1124 2024

[5] [5]

On the planning abilities of large language models-a critical investigation

Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36: 0 75993--76005, 2023

2023

[6] [6]

Mint: Evaluating llms in multi-turn interaction with tools and language feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024

work page 2024

[7] [7]

Travelplanner: A benchmark for real-world planning with language agents

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622, 2024

arXiv 2024

[8] [8]

Natural plan: Benchmarking llms on natural language planning

Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520, 2024

arXiv 2024