{"id":"e3b8e0b4-ee00-411f-9925-f195cd96e2ba","arxiv_id":"2507.07203","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A prompt framework that makes LLM game merchants follow a six-state trading flow achieves over 97% state compliance, over 95% item accuracy, and 99.7% price accuracy in simulated dialogues.","lead":"Researchers at SayBerryGames show that a specially structured prompt can stop large language models from making up items or miscalculating prices when role-playing a game shopkeeper. The method, called State-Inference-Based Prompting, gets the model to infer whether a chat is casual, a price offer, or a purchase confirmation before replying, and it reports over 97% adherence in 100 simulated dialogues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline metrics are computed from the LLM's own self-reported JSON fields (context_subtype, items, prices) with no independent check against dialogue text or game state; high STCR/SIRR/price accuracy may reflect self-consistency rather than true rule adherence.","rationale":"I read the paper as making a practical claim: SIBP makes LLM NPC trading reliable enough for commercial deployment, evidenced by >97% STCR, >95% SIRR, and 99.7% price precision. For that claim to hold, the metrics must measure actual rule-following rather than the model's internal consistency. The weakest point is therefore the evaluation design, not the prompting method itself: every headline metric is computed from the structured JSON fields that the same LLM generates (Appendix A), and no independent verification against dialogue text or game state is provided. STCR (Sec. 4.2) counts self-reported state transitions; a model that wrongly labels an OFFER_SELL turn as CHECK_CONFIRMATION would pass. SIRR (Sec. 4.3) inspects only the items array, so hallucinations in the natural-language response are invisible to the metric. Price accuracy (Sec. 4.4) is effectively a property of the post-processor under placeholder substitution, since the system computes the sum from the LLM's own item details; the metric does not check that those item details match the player's requested purchase. The virtual-player limitation (Sec. 4.1) is real but secondary, because it threatens generalization rather than the internal validity of the reported numbers. My concrete test re-scores STCR from raw dialogue text; if that rate drops materially, the central claim is not supported. Because the reader's CONDITIONAL verdict already conditions on external verification of the metrics, my concern does not move the verdict.","tokens_in":14937,"tokens_out":6826,"duration_ms":69780,"concrete_test":"Re-score STCR from the raw dialogue text independently of the JSON: for every CONFIRM_SELL transition in the 100 scenario-1 dialogues with SIBP+PPP, determine from the immediately preceding NPC utterance whether it actually contained an explicit purchase-confirmation question (e.g., a Korean equivalent of 'So, will you buy it?'). Compute the text-based compliance rate (confirmed transitions / total CONFIRM_SELL transitions). If this rate is materially lower than the reported 97.73% (e.g., by more than 5 percentage points), the headline claim is inflated by self-reporting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the reported metrics actually measure rule adherence. As defined, they do not. STCR (Sec. 4.2) is the proportion of dialogues where the self-reported state sequence ends with CONFIRM_SELL after CHECK_CONFIRMATION, with both states taken from the NPC's own last_trade_context/context_subtype JSON fields (Appendix A). There is no verification that the raw npc_dialogue asked the required confirmation question before finalization; a model that mislabels the previous state as CHECK_CONFIRMATION passes despite never seeking confirmation. SIRR (Sec. 4.3) checks only the items array in the JSON, not the natural-language response; an NPC could mention a hallucinated item in npc_dialogue while keeping the items array clean and SIRR would not catch it. Price accuracy (Sec. 4.4) is defined as agreement between the stated total and the item details in the same LLM response; with placeholder post-processing the system itself computes that sum, so the 99.7% figure is a property of the arithmetic post-processor, not of the LLM's reasoning, and it does not validate quantities or prices against the player's request. All three headline metrics are thus self-referential: they cannot distinguish correct rule-following from an LLM that produces internally consistent but factually wrong state labels. The virtual-player issue (Sec. 4.1) is a genuine external-validity concern, but the self-reporting issue undermines the internal validity of the reported numbers themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes State-Inference-Based Prompting (SIBP), a prompt-engineering framework for LLM-driven NPCs in game trading. The method decomposes trading into five sub-states (SHOW_INVENTORY, OFFER_SELL, NEGOTIATE_PRICE, CHECK_CONFIRMATION, CONFIRM_SELL) under a TRADE context, adds a directive for the LLM to identify and report the previous trading state, and uses state-specific referencing plus placeholder-based price post-processing. The authors evaluate SIBP against several baselines using a virtual player LLM across 100 dialogues per condition, reporting >97% state transition compliance (STCR), >95% sellable-item response rate (SIRR), and 99.7% price accuracy. They also report computational-efficiency comparisons across model sizes and structured-output variants.","tokens_in":15237,"tokens_out":4832,"duration_ms":48729,"significance":"If the reported results were trustworthy, SIBP would be a practically useful prompt-design recipe for rule-governed LLM dialogue, with plausible extensions to other transactional domains. The prompt design ideas—especially the explicit previous-state identification and state-specific post-processing—are reasonable and worth studying. However, the evaluation as presented does not support the headline claims: the main metrics are computed from the LLM's own JSON state labels and item arrays, on a selectively chosen subset of dialogues, and the price-accuracy metric for the post-processed condition is circular by construction. The paper would be significantly strengthened by externally verified metrics and a clear statement of the effective sample size.","major_comments":[{"comment":"STCR is not computed across 100 dialogues as the abstract states. The text says STCR was calculated based on the first 88 dialogues for each method where the CONFIRM_SELL state was reached, and this 88 is the minimum number of such occurrences across methods. This outcome-dependent selection excludes dialogues that never reached the final state, which can inflate compliance rates and makes the comparison unfair across methods with different rates of reaching CONFIRM_SELL. Please report the number of excluded dialogues per method and re-run the analysis with a fixed denominator (e.g., all 100 dialogues, treating failure to reach CONFIRM_SELL as non-compliance when a purchase was intended), or provide a clear statistical justification for conditioning on reaching the final state.","section":"Section 4.2, Table 1"},{"comment":"STCR is scored from the NPC LLM's self-reported JSON state fields (context_subtype and last_trade_context), with no verification against the actual natural-language utterance. A dialogue is counted as compliant if the JSON label sequence ends with CHECK_CONFIRMATION followed by CONFIRM_SELL, regardless of whether the npc_dialogue text actually contained a confirmation question. Thus the metric measures the LLM's consistency in labeling states, not its adherence to the rule that an explicit confirmation must be obtained before finalizing a purchase. Please add a manual or programmatic check of the dialogue text (e.g., detecting a confirmation question before the final transition) or re-define the metric accordingly.","section":"Section 4.2, STCR definition"},{"comment":"The Sellable Item Response Rate is computed from the items array in the JSON response, not from the natural-language npc_dialogue. A response that narratively offers a hallucinated item (e.g., mentioning a non-inventoried item in the spoken text) would be scored as correct if the items field contains only sellable items. Since the paper's stated goal is to prevent item hallucination in the player-visible dialogue, SIRR should be evaluated on the full response, including the free-form npc_dialogue, or a separate hallucination rate on the natural-language output should be reported.","section":"Section 4.3, SIRR"},{"comment":"The 100% price accuracy in OFFER_SELL for SIBP+PPP is by construction: the system computes the total from the item details and substitutes it for the __PRICE__ placeholder, so the metric only checks that the placeholder was emitted, not that the LLM performed the calculation. Moreover, the price-accuracy metric compares the stated total to the item details within the same LLM response, not to the ground-truth inventory prices; a response that lists a wrong unit price but a consistent total would still be counted as accurate. Please report agreement with the actual inventory price list, and separately report (i) the placeholder-emission rate and (ii) the arithmetic-consistency rate after post-processing.","section":"Section 4.4, Table 2"}],"minor_comments":[{"comment":"The abstract claims evaluation across 100 trading dialogues, but STCR uses only 88 per method; please align the wording or the analysis.","section":"Abstract and Section 4.2"},{"comment":"The state-transition heatmaps are presented without confidence intervals or statistical tests; the difference between baseline4 (94.32%) and SIBP (97.73%) may not be significant given the sample size. Please report uncertainty or perform a significance test.","section":"Section 4.2, Figure 2"},{"comment":"The paper refers to 'six states' in several places but lists three main contexts (NONE, TRADE, END_CONVERSATION) with five trade subcontexts; the taxonomy should be stated consistently.","section":"Section 3.2.1"},{"comment":"The use of a single virtual-player LLM (gemini-2.5-flash, temperature 0.7) is a limitation for external validity; the paper should explicitly discuss potential differences from real player behavior and consider human evaluation or additional backbone models.","section":"Section 4.1"},{"comment":"The PDF contains rendering artifacts in Table 1 and its caption (e.g., '/uni00000036/...' strings) that obscure the figure; also, reference [16] contains an unusual citation format 'arXiv preprint [6, 7]'. Please fix these formatting issues.","section":"Appendix and Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style paper with a plausible prompt-design idea, but the evaluation metrics are self-referential and the headline numbers are not supported by the current analysis. The authors should be encouraged to release their dialogue corpus and evaluation scripts so that external verification is possible. The response-time and token measurements are useful, but they do not compensate for the circular price metric. The paper would benefit from a clearer discussion of the estimand for STCR and from validation of the state labels against the actual dialogue text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you want to know about: SIBP is a plausible, well-written prompt-engineering recipe for keeping LLM merchant NPCs inside trading rules, and it shows one genuinely useful trick—forcing the model to write down its inferred previous state in the JSON response—which the ablation suggests is doing real work. The headline numbers, though, should not be taken at face value until the evaluation is tightened.\n\nWhat's new: the specific package—six trade states, transition conditions keyed to the previous state, an explicit identify-and-report directive, state-dependent inventory references, and __PRICE__ placeholder post-processing—is a sensible integration that I don't see in the MART or DST papers they cite. The four-element ablation is the most useful part: Element 4 alone gets 94.3% STCR versus 79.6% with none, and the full combination reaches 97.7%. That is a real empirical signal, assuming the metric is trustworthy.\n\nThe soft spots are real. STCR is computed from the NPC's own JSON context_subtype/last_trade_context fields, with no independent check that the raw dialogue actually contained the confirmation question. A model that mislabels the previous state as CHECK_CONFIRMATION passes. The paper also reports STCR on the first 88 dialogues per method that reached CONFIRM_SELL, not on the full 100; dialogues that never reached that state are excluded, so the denominator flatters the method. Second, the 99.7% price accuracy in SIBP+PPP is mostly a property of the deterministic sum-and-replace post-processor, not the LLM's arithmetic; it validates placeholder recovery, not price correctness. SIRR likewise checks the JSON items array, not the natural-language utterance, so a hallucinated item in npc_dialogue could slip through. On the external side, the virtual player is a reasonable controlled stand-in, and the paper is transparent about not testing with humans.\n\nNone of this kills the contribution. It is a useful engineering pattern, and the ablation is worth publishing. But the paper should report full-100 metrics, verify state compliance against raw dialogue or game state, ideally with a deterministic checker or second annotator, release the dialogues and data, and moderate the commercial-deployment claims. I would send it to review—a serious referee can help fix the evaluation—but I would not currently quote the headline numbers.","headline":"SIBP is a useful prompt-engineering recipe with a sharp ablation, but the headline compliance numbers rely on self-reported state labels and a filtered 88-dialogue denominator.","tokens_in":15776,"tokens_out":2602,"would_cite":false,"duration_ms":28641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Natural-language NPC trading stays rule-compliant when the LLM must infer and report its previous dialogue state before answering.","keywords":["large language models","game NPCs","natural language trading","prompt engineering","dialogue state inference","state transition compliance","item hallucination","price calculation"],"falsifier":"Run the same SIBP prompt with 100 human players instead of the virtual player and have an independent judge or the game server record whether every purchase reaches CONFIRM_SELL only after a CHECK_CONFIRMATION turn. If human dialogues show a skip rate meaningfully above the 2.3 percent failure seen in simulation, or if a server-side state replay disagrees with the self-reported last_trade_context, the central deployment claim is weakened.","tokens_in":14716,"feed_emoji":"🎮","tokens_out":9385,"duration_ms":92680,"temperature":0.7,"pith_summary":"Natural-language trading with a game NPC is hard for LLMs because the player can change topic or intent mid-dialogue, and the NPC must know which game rule applies at that moment. This paper proposes State-Inference-Based Prompting (SIBP), which decomposes the dialogue into a fixed set of contexts—general conversation, conversation end, and five trade sub-states such as showing inventory, offering a price, negotiating, and confirming—puts the definitions and transition rules in one prompt, and forces the LLM to state its inferred previous trade state in every structured reply. The paper argues that this self-reported previous state lets one utterance like \"That sounds good!\" resolve differently in different contexts, blocks premature purchase finalization, and makes state-specific item referencing and placeholder-based price calculation possible. In 100 simulated purchase dialogues, SIBP configurations reach over 97 percent state-transition compliance, over 95 percent sellable-item referencing, and 99.7 percent price accuracy without added computational cost. The contribution, read sympathetically, is a reusable prompt-engineering recipe for steering LLMs through rule-governed interaction flows.","feed_headline":"LLM NPC merchants trade reliably by reporting their own state","feed_subtitle":"Reporting the previous trade state pushes rule compliance past 97% and price accuracy to 99.7%.","key_machinery":"The load-bearing mechanism is the \"identify and report the previous state\" directive embedded in a unified prompt, together with the state-specific rules stacked on it. In each JSON response the model must fill last_trade_context, context_type, context_subtype, and an items array, which makes the model's state inference inspectable and usable by the game system. The same field drives conditional transitions—for example, CONFIRM_SELL is permitted only when last_trade_context is CHECK_CONFIRMATION and the player gives positive confirmation. Two auxiliary mechanisms carry the reliability results: state-specific data referencing, which uses the full game item list for general chat but only the merchant inventory for trade, and placeholder-based price post-processing, where the LLM writes __PRICE__ and the system computes the arithmetic.","core_discovery":"The central discovery is that reliable rule adherence does not require an external state manager: the LLM can be made to act as its own state tracker. SIBP's prompt tells the model to identify the most recent trading sub-context from dialogue history before responding, and to write that identification into the JSON field last_trade_context. The response format then decides whether a player's confirmation counts as accepting an item description (so the NPC offers a price) or accepting a price (so the NPC asks for final confirmation). With all four prompt elements active, 97.73 percent of dialogues in both scenarios follow the mandatory CHECK_CONFIRMATION-to-CONFIRM_SELL sequence, and the reporting element alone lifts compliance to 94.32 percent when the identification directive is absent. Inside TRADE state, item references draw only from the NPC's inventory, while the OFFER_SELL stage emits a __PRICE__ placeholder that the system later replaces with a computed total, yielding 100.0 percent price accuracy at the offer and 99.7 percent across later states.","pith_inferences":["A natural next test is to score every pairwise state transition rather than only the CHECK_CONFIRMATION-to-CONFIRM_SELL gate, since the paper's one-gate metric leaves the full transition graph unmeasured.","Recruiting human players or a second player-model family would test whether the simulated-player results hold under noisier inputs; the current evaluation uses one model for both roles.","Because placeholder pricing replaces arithmetic with deterministic post-processing, the accuracy gain should transfer across models and languages as long as the placeholder token stays short; the paper's smaller-model result is the first hint.","A production variant could persist last_trade_context as game state, allowing the NPC to resume a mid-transaction after disconnects or context-window truncation."],"forward_implications":["If SIBP works as reported, game studios can deploy LLM merchant NPCs without an external dialogue-state manager or tool-use pipeline; the prompt itself supplies the control.","The previous-state reporting element is the main lever: enabling it alone nearly closes the compliance gap, and combining it with explicit transition explanations gives the best result.","Placeholder post-processing removes arithmetic from the model, so accuracy gains appear even with smaller models: SIBP+PPP on 2.0-flash reaches 100.0 percent price accuracy at the offer stage versus 49.4 percent for basic SIBP on the same model.","State-specific referencing keeps NPCs able to discuss any game-world item in general conversation while restricting trade proposals to sellable inventory, which is the paper's proposed cure for item hallucination.","The same prompt recipe can be lifted to other rule-governed LLM interactions such as interactive kiosks and customer-support workflows, where the model must infer which workflow step is active."],"supporting_citations":[{"why":"Prior active-merchant-NPC work that identified item hallucination and price-calculation failures, which SIBP's post-processing directly targets.","marker":"[12]"},{"why":"Dialogue-state-tracking literature that establishes inferring conversational state from dialogue, which SIBP turns into a control mechanism.","marker":"[18]"},{"why":"Generative-agent work showing LLM-driven NPCs can carry believable open-ended interaction, the line this paper extends to rule-governed transactions.","marker":"[3]"},{"why":"Earlier haggling-focused framework that covers price negotiation but not the full multi-state trading flow SIBP formalizes.","marker":"[17]"},{"why":"LLM negotiation benchmark that frames the evaluation landscape for LLM commercial interactions beyond game NPCs.","marker":"[16]"}],"fun_headline_variants":["LLM NPCs self-track state for 97% rule adherence","Self-inferred state lifts game NPC trade accuracy to 99.7%","State-inference prompting makes NPC merchants follow pricing rules","NPCs trade by self-declared state: 99.7% price precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation assumes that dialogues generated by a simulated player LLM represent real player behavior, and that the NPC's own self-reported state fields are a trustworthy measure of rule compliance; if either assumption fails, the headline percentages may not transfer to actual players.","fun_headline_variants_meta":{"raw":{"variants":["LLM NPCs self-track state for 97% rule adherence","Self-inferred state lifts game NPC trade accuracy to 99.7%","State-inference prompting makes NPC merchants follow pricing rules","NPCs trade by self-declared state: 99.7% price precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3506,"prompt_tokens":870,"completion_tokens":2636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":2558}},"tokens_in":486,"tokens_out":2636,"duration_ms":20969,"temperature":1.0,"reasoning_tokens":2558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:46:51.395467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SIBP prompt with 100 human players instead of the virtual player and have an independent judge or the game server record whether every purchase reaches CONFIRM_SELL only after a CHECK_CONFIRMATION turn. If human dialogues show a skip rate meaningfully above the 2.3 percent failure seen in simulation, or if a server-side state replay disagrees with the self-reported last_trade_context, the central deployment claim is weakened.","supporting_citations":[{"cited_title":"Leveraging Large Language Models for Active Merchant Non-player Characters","cited_arxiv_id":"2412.11189","evidence_quote":"Prior active-merchant-NPC work that identified item hallucination and price-calculation failures, which SIBP's post-processing directly targets."},{"cited_title":"item_id\",","cited_arxiv_id":null,"evidence_quote":"Dialogue-state-tracking literature that establishes inferring conversational state from dialogue, which SIBP turns into a control mechanism."},{"cited_title":"LLM-Stakeholders Interactive Negotiation: A Benchmark for Cooperation, Competition, and Maliciousness","cited_arxiv_id":null,"evidence_quote":"LLM negotiation benchmark that frames the evaluation landscape for LLM commercial interactions beyond game NPCs."}],"review_version":1}