Pith.

Computation and Language · arXiv:2610.22980 v1 · submitted 16 Oct 2026 · 46 references

KiranaBench: Can a Language Model Run a Neighbourhood Shop?

Bhuvan Arora, Ankita Karmakar, Chaitanya Jain, Dilesh Churiwal, Harshit Jain, Shivansh Manish Dubey, Manish Prasad, Raja Piyush

show 5 more authors

Kshitij Tayal, Dhruv Kumar, Jagat Sesh Challa, Raunak Manekar, Yash Sinha

The paper says

“Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly.”

more of the abstract

We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 19 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. KiranaBench is a genuinely novel benchmark for multi-agent, multilingual LLM operation in an Indian retail context, and the results are stark, though the demand parameters deserve a sensitivity analysis before the quantitative claims are taken at face value. This line is wrong

1 major objection · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 46 references say.

The paper says, at Section I (Demand and Customers, especially sentences b262, b263, b279, b286)

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

The demand model relies on three categories of assumed parameters—household purchase shares (50%/70%/80%), festival multipliers (e.g., 2–3× for fasting ingredients, 4× for gifting dry fruit), and loyalty-magnitude estimates—that are all explicitly labeled as assumptions derived from industry reporting rather than from a published, reproducible source.

more of objection 1

The authors note that the survey 'cannot settle them' and that no research-grade festival demand series exists. While this transparency is commendable, the headline profit gap (best LLM earns one-eighth of the scripted shopkeeper) is a direct function of these parameters. The paper does not include any sensitivity analysis over plausible ranges of these assumptions, nor does it bound how much the scripted shopkeeper's profit or the LLM ordering could shift under alternative demand elasticities, loyalty responses, or purchase-location splits. I do not consider the qualitative conclusion ('not yet reliable') fragile—even if the scripted baseline profit were halved, the best LLM would still be far below—but the quantitative claim of 'one-eighth' and the fine-grained ranking claims (Section 5.1) would be strengthened by even a one-factor or two-factor sensitivity sweep. The authors should either add such an analysis or discuss in the Limitations section the likely magnitude of the uncertainty and why it does not affect the main ordering.

4 minor objections

The paper says, at Section L (Language in Detail)

Pith says: minor objection 1 of 4

The rule for classifying an e-mail as English, Hindi, or Hinglish is described in detail in Appendix L, but the choice of the 'one-tenth of words' threshold for romanised Hindi appears arbitrary. While the rule is a reasonable heuristic, a brief justification (even a qualitative one) would help readers assess whether borderline cases could materially affect the language cost figures reported in Table 7.

The paper says, at Figure 2 and Table 2

Pith says: minor objection 2 of 4

The profit curves in Figure 2 average over three seeds; the seed-by-seed Figure 3 reveals very high variance for Gemma 4 (range from +Rs37,226 to -Rs40,762). The paper correctly reports this variance and notes that the pairwise ordering between model families is strict, but a short sentence explicitly restating that the comparison across Qwen and Gemma (the top two) is not statistically significant would be helpful to prevent readers from overinterpreting their ranks.

show 2 more

The paper says, at Appendix A (Table 3)

Pith says: minor objection 3 of 4

The reference shopkeeper profit is reported as +Rs94,919 in Table 3 and +Rs1,03,434 in Table 2 for the same policy. This discrepancy arises because Table 3 averages over five seeds with no language model, while Table 2 averages over the three seeds used for model runs. Please reconcile these values in the main text or add a footnote explaining the difference.

The paper says, at Appendix R (Prompts, 'How a call is made')

Pith says: minor objection 4 of 4

The prompts are reproduced in full, which is excellent for reproducibility. However, the instructions for the munshi include a file_gst tool that takes 15 minutes, but the tool list in Table 6 also lists file_gst at 15 minutes—this is consistent. A small typo: 'Devanagari elided' in the prompt text appears multiple times without the script actually being elided in the printed version; this is a typesetting artifact but should be noted for the camera-ready copy.

What would settle it

Re-run the scripted shopkeeper and the best-performing LLM (Qwen3.8) on seed 0 with two extreme parameter sets: (A) all purchase shares set to 100% and festival multipliers set to 1 (no festival effect), and (B) shares halved (25%, 35%, 40%) and festival multipliers doubled (fasting 4–6×, gifting 8×). For each set, compute the scripted shopkeeper’s profit and the LLM’s profit. If the ratio (scripted_profit / LLM_profit) changes by more than a factor of 2 relative to the original ratio (≈7.6), then the demand parameters are influential on the headline conclusion; if the ratio stays within ±50%, the concern is mitigated.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 19 Oct 2026

  1. Reader holds up confidence moderate
  2. Skeptic agrees 42 s · 37,939 tokens read, 3,330 written
  3. Editor wrote the verdict above 16 s · 38,872 tokens read, 1,238 written
  4. Referee minor revision: 1 major objection, 4 minor 33 s · 38,287 tokens read, 2,869 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The real contribution here is the benchmark design, not the specific profit numbers. KiranaBench is the first to combine multi-agent delegation (manager who only sees the shop through staff reports), report fidelity scoring, and demand calibrated to a national survey, all in an Indian retail context with three languages. That's a genuinely new evaluation axis. The authors also deserve credit for transparency: every run is reported, every failure traced to a named action, and the audit found no money errors across 360 days. The scripted baselines are sensible and bracket the models cleanly.

The results are striking—best model at one-eighth of scripted profit, two models below doing nothing—but the headline numbers rest on demand parameters (household purchase shares, festival multipliers, loyalty magnitudes) that the authors themselves call "assumptions" and "estimates" with no published source. The pressure test is correct that without sensitivity analysis over these parameters, we don't know how much of the gap is intrinsic to LLM limitations versus artifacts of the simulator's difficulty tuning. The authors acknowledge this in the Limitations ("The shop is a single Delhi kirana…"), but they don't bound the effect. That's a soft spot, and it's the main thing I'd ask for in revision.

Minor soft spots: the language results for Hindi and Hinglish are based on small e-mail counts (e.g., Qwen3.8 sent only 3 Hindi e-mails), and the three-seed per model design leaves model-level variance poorly estimated for Qwen vs. Gemma, which overlap. None of these undercut the central claim that current models are far from competent, but they limit the precision of the ordering.

Who is this for? Anyone working on LLM agents for long-horizon, multi-agent, or multilingual settings. The benchmark itself is reusable and the failure taxonomy (Appendix O) is useful. I'd accept this for review: the design is sound, the claims are appropriately scoped (the paper says "evidence that current LLM agents are not yet reliable"—that's supported), and the missing sensitivity analysis is fixable. I would bring it to a reading group.

The strongest objection

The paper models household demand using purchase location shares (50% for milk, 70% for eggs/bread, 80% for others) and festival multipliers (e.g., fasting ingredients 2–3×, gifting dry fruit 4×) that are explicitly labeled as assumptions from industry reporting, not from a published, reproducible source. The authors state (Section I, sentences 262–263, 279) that the survey cannot settle the shares and that no research-grade festival retail demand series exists. The loyalty magnitude estimates are likewise called estimates (sentence 286). The central claim that the best LLM earns only one-eighth of the scripted shopkeeper’s profit depends on these parameters shaping the difficulty of the environment. If real kirana demand exhibits different seasonal elasticity, lower loyalty responsiveness, or different purchase-location splits, the profit gap might shrink or widen significantly, changing the conclusion about LLM reliability. Because the paper provides no sensitivity analysis over these parameters, the robustness of the headline result to plausible variations is unknown. This is the most load-bearing concern because it directly undercuts the quantitative anchor (the scripted shopkeeper’s profit) against which all LLM performance is measured, and because the authors acknowledge the uncertainty but do not bound its effect on the comparison.

The reader's weakest_assumption correctly identified the demand parameters as unverified assumptions. I share this concern and consider it the most load-bearing because the paper's central quantitative claim—the one-eighth profit ratio—rests on these parameters. However, the paper is transparent about the assumptions, provides detailed appendices, and includes independent audits showing that the cash book balances perfectly and failures are traceable to concrete model errors. A sensitivity test (as described) would settle whether the parameters drive the result; until that is done, the current evidence supports the benchmark's validity and the overall story that current LLMs fail at this long-horizon multi-agent task. The paper's honest reporting of limitations and the strength of the behavioral measures (report fidelity, credit decisions, etc.) reinforce acceptance. Therefore, the reader's ACCEPT verdict remains appropriate, with the caveat that the profit magnitude is parameter-dependent.

The referee's report

If the results hold, the paper makes a timely and empirically grounded contribution. It extends business-operation agent benchmarks to a multi-agent, multilingual, long-horizon setting that mirrors real-world small retail in India, where most prior work is single-agent, English-only, and Western-context. The transparent audit of all 360 days, the release of all traces and code, and the inclusion of scripted baselines that span from competent operation to abandonment give the evaluation unusual rigor. The finding that two commercially relevant open-weight models lose money by actively trading (rather than simply doing nothing) is a concrete caution for practitioners. The benchmark provides a clear and reproducible testbed for measuring improvements in agentic coherence, delegation fidelity, and multilingual tool use.

Pith's reading of the paper (a machine's summary, not the authors')

This paper introduces KiranaBench, a simulation of an Indian neighborhood store where four LLM agents (manager, cashier, helper, bookkeeper) must cooperate across languages (English, Hindi, Hinglish). Demand is calibrated to India's household consumption survey. Evaluating four open-weight LLMs, the best model earns only Rs13,683 average profit against a scripted shopkeeper's Rs1,03,434 — about one-eighth. Two models finish below an unmanaged shop. The results suggest that current LLM agents are not yet reliable for autonomously operating small businesses in India, and the benchmark provides a testbed for improving long-horizon, multi-agent, multilingual agent systems.

What carries the argument. KiranaBench, a simulated Indian general store where one LLM fills four roles (manager, cashier, helper, bookkeeper). The manager cannot observe the shop directly; it learns the day's events only through reports from the other agents, which may be inaccurate. Demand is derived from India's Household Consumption Expenditure Survey, with customer and supplier interactions in English, Hindi, or Hinglish generated by a separate fixed language model. The simulation models detailed retail operations including shelf management, supplier ordering, credit, spoilage, and GST.

Core claim. All four tested open-weight LLMs fall far short of a simple scripted shopkeeper baseline in a realistic Indian retail simulation. The best model (Qwen3.8) averaged Rs13,683 profit over 30 simulated days, compared to the scripted shopkeeper's Rs1,03,434. Two models (gpt-oss and LFM2.5) performed worse than a shop with no management at all, losing more money by continuing to trade unprofitably. Additionally, agent performance degraded significantly when communicating with suppliers in Hindi or Hinglish versus English, even for models that otherwise performed well. The failures are traced to specific errors: writing to nonexistent suppliers, duplicate orders, misreporting sales, and incorrectly handling deliveries.

The premise it rests on. The simulation's demand parameters (household purchase shares, festival multipliers, loyalty magnitudes) are educated guesses from industry reports rather than empirical measurements, and if these misrepresent real kirana shop dynamics, the profit numbers may not generalize to actual stores.

If the paper is right (Pith's inferences, not the authors')

If the results hold for other models and seeds, LLM agents cannot yet replace human kirana shop operators in real-world settings.

The large performance gap suggests that long-horizon coherence and multi-agent coordination remain unsolved for realistic retail contexts.

The degradation in Hindi/Hinglish supplier communication indicates that model development must prioritize Indic languages for deployment in Indian retail.

The specific failure modes identified (invented suppliers, duplicate orders, false reports) point to concrete aspects of agent architecture that need improvement.

The benchmark could be extended to other retail formats (e.g., street vendors, online delivery) and more languages to test generality of the findings.

Because the scripted shopkeeper reads stock directly while agents rely on staff reports, part of the gap may reflect the cost of delegation itself; a future version with a scripted manager using staff reports could isolate this delegation cost.

The demand model parameters (festival multipliers, loyalty magnitudes) are assumptions from industry reports; different values could change the profit ordering and need empirical validation.

Traceable failures like duplicate orders suggest that simple rule-based guardrails in the agent loop (e.g., checking for existing orders) could close much of the gap without requiring better reasoning.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 0.98

“KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.”

How the main claim is stated: asserted 0.86

What the abstract itself states, as probabilities: claims first 0.19 · states limitation 0.60 · human participants 0.06 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.97 · mentions code or data availability 0.07

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. background 1.00 0.05 0.02 asserted 0.99
Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. gap 0.96 0.05 0.02 asserted 1.00
We study how these results transfer to a multi-agent Indic setting. contribution 0.67 0.37 0.02 unclear 0.60
We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. contribution 0.73 0.90 0.03 asserted 1.00
Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. method 0.97 0.76 0.02 asserted 0.98
Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. result 1.00 0.97 0.56 asserted 1.00
Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. result 1.00 0.93 0.07 asserted 1.00
KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual. interpretation 0.97 0.81 0.94 asserted 0.86

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.59

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 6,303 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (16)
Figure 1 of the paper
Figure 1. KiranaBench. The shop has four jobs and one model does all of them (left), each with its own tools and clock. Staff write only to the manager, so the day reaches it through them, and what they say about their own work can be wrong. Code computes every outcome (centre) and the truth those reports are scored against. A separate fixed model gives customers and suppliers their words in English, Hindi or Hinglish (right) and decides nothing. How the agent works What the shop models Benchmark Domain Lang. through staff reports scored counterparts speak credit on the book staff time budget…
Figure 2 of the paper
Figure 2. Profit by simulated day, averaged over the three seeds, one line per model. The scripted shops bracket
Figure 3 of the paper
Figure 3. Profit by simulated day for every run, line colour denoting the model and line style the seed. Both scripted
Figure 4 of the paper
Figure 4. E-mails that the supplier could not read, by the
Figure 5 of the paper
Figure 5. Live-customer outcomes by the language the customer speaks: the share of served customers who bought,
Figure 7 of the paper
Figure 7. The reply names the valid identifiers; the
Figure 8 of the paper
Figure 8. No such payment appears anywhere in the cash book, and no role has a tool that could make one. Example 3: An e-mail the supplier cannot read. Gemma 4 31B, seed 0, day 1 MANAGER, WRITING TO MANDI_EN, MANDI_HINGLISH, MANDI_HI [08:29] Namaste, please provide your cur￾rent rates per pack for: atta_1kg, atta_5kg, atta_10kg, chawal_1kg, chawal_5kg, dal_1kg, dal_500g, besan_1kg, besan_500g, cheeni_1kg, cheeni_5kg, tel_1l, tel_5l, ghee_1l, ghee_500ml. MANDI DIRECT SUPPLIES, WHICH WRITES HINGLISH Maaf kijiye bhaiya, aapka message samajh nahin aaya. Hinglish mein likhiye na
Figure 11 of the paper
Figure 11. The reply lists every identifier that does exist.
Figure 12 of the paper
Figure 12. The munshi reads the real cash book, so a fig
Figure 13 of the paper
Figure 13. The tool asks how much of the load to keep.
Figure 14 of the paper
Figure 14. A single simulated day (Qwen3.8-27B, seed 1, day 5). Each bar is a turn by one of the four roles, as long
Figure 15 of the paper
Figure 15. How often each kind of failure occurs, per simulated day, on a logarithmic scale. The failures are not
Figure 16 of the paper
Figure 16. What the shop is made of over the month, for one seed per model: cash in the till, and stock valued at

Showing 13 of the paper's 16 figures. Figures are the authors', reproduced from the paper.

References (46)
  1. [2] -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author =. arXiv preprint arXiv:2406.12045 , year = on PithPith's reviewarXiv
  2. [3] Decoupling Strategy and Generation in Negotiation Dialogues , author =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , year = on Pith
  3. [4] Deal or No Deal? End-to-End Learning of Negotiation Dialogues , author =. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages =. 2017 , url = on Pith
  4. [5] How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis , author =. arXiv preprint arXiv:2402.05863 , year = on PithPith's reviewarXiv
  5. [6] Cur. Modelling handling operations in grocery retail stores: An empirical analysis , doi =. Journal of the Operational Research Society , volume = on Pith
  6. [8] Annual Survey of Unincorporated Sector Enterprises (ASUSE) 2021-22 and 2022-23: Factsheet , author = on Pith
  7. [9] Household Consumption Expenditure Survey 2022--23: Report No. 591 , author =. 2024 , url = on Pith
  8. [10] Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context , author =. arXiv preprint arXiv:2607.25375 , year = on PithPith's reviewarXiv
  9. [11] Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier , author =. arXiv preprint arXiv:2412.04261 , year = on PithPith's reviewarXiv
  10. [14] CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts , author =. arXiv preprint arXiv:2505.05063 , year = on PithPith's reviewarXiv
  11. [20] DukaanBench , author =. 2026 , howpublished = on Pith
  12. [22] Are Your Agents Upward Deceivers? , author =. Proceedings of the 43rd International Conference on Machine Learning (ICML) , pages =. 2026 , note = on Pith
  13. [24] Sinha, Stuti and Kumar, Himanshu and Mandapati, Aryan Raju and Sakhuja, Rakshit and Kumar, Dhruv , journal =. The Language of Bargaining: Linguistic Effects in. 2026 , url = on Pith
  14. [25] Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL) , pages =. 2026 , url = on Pith
  15. [26] Peng, Kuang and Fan, Yuchun and Li, Jiangnan and Wu, Minghao and Tang, Jialong and Wei, Haoran and Wang, Weixuan and Tu, Jianhong and Yang, Baosong and Xiao, Tong , journal =. BabelArena: A Large-Scale Multilingual Benchmark for. 2026 , url = on Pith
  16. [28] Wang, Yifan and Shi, Jiaxin and Oldfield, James and Marro, Samuele and Yu, Philip and Torr, Philip and Du, Yali and Bibi, Adel , journal =. BazaarBench: Delegation Safety in Decentralized. 2026 , url = on Pith
  17. [29] Kulkarni, Mayank and Mazzia, Vittorio and Gaspers, Judith and Hench, Chris and FitzGerald, Jack , booktitle =. 2025 , doi = on Pith
  18. [30] Caciolai, Andrea and others , journal =. OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier. 2026 , url = on Pith
  19. [31] Li, Hongliang and others , journal =. PolyWorkBench: Benchmarking. 2026 , url = on Pith
  20. [34] 2026 , howpublished = on Pith
show 26 more
  1. [35] 2025 , howpublished = on Pith
  2. [36] Axel Backlund and Lukas Petersson. 2025. https://arxiv.org/abs/2502.15840 Vending-bench: A benchmark for long-term coherence of autonomous agents . arXiv preprint arXiv:2502.15840 on PithPith's reviewarXiv
  3. [37] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. https://arxiv.org/abs/2506.07982 ^2 -bench: Evaluating conversational agents in a dual-control environment . arXiv preprint arXiv:2506.07982 on PithPith's reviewarXiv
  4. [38] Anca Cur s eu, Tom van Woensel, Jan Fransoo, Karel van Donselaar, and Rob Broekmeulen. 2009. https://doi.org/10.1057/palgrave.jors.2602553 Modelling handling operations in grocery retail stores: An empirical analysis . Journal of the Operational Research Society, 60(2):200--214 on Pithdoi:10.1057/palgrave.jors.2602553
  5. [39] Vijay Devane, Mohd Nauman, Bhargav Patel, Aniket Wakchoure, et al. 2025. https://arxiv.org/abs/2510.25409 Bhashabench v1: A comprehensive benchmark for the quadrant of indic domains . arXiv preprint arXiv:2510.25409 on PithPith's reviewarXiv
  6. [40] Omar El Housni, Qing Feng, and Huseyin Topaloglu. 2024. https://arxiv.org/abs/2411.10310 Assortment optimization under the multinomial logit model with covering constraints . arXiv preprint arXiv:2411.10310 on PitharXiv
  7. [41] Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang, Baohua Dong, Yangqiu Song, and Dayiheng Liu. 2026. https://arxiv.org/abs/2608.30730 E-commerce bench: Evaluating llm agents on long-horizon autonomous business operation . arXiv p... on PithPith's reviewarXiv
  8. [42] Gemma Team . 2026. https://arxiv.org/abs/2607.02770 Gemma 4 technical report . arXiv preprint arXiv:2607.02770 on PithPith's reviewarXiv
  9. [43] Dadi Guo, Qingyu Liu, et al. 2026. https://proceedings.mlr.press/v306/guo26k.html Are your agents upward deceivers? In Proceedings of the 43rd International Conference on Machine Learning (ICML), pages 38052--38092. ArXiv:2512.04864 on PitharXiv
  10. [44] Mayank Kulkarni, Vittorio Mazzia, Judith Gaspers, Chris Hench, and Jack FitzGerald. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.1099 MASSIVE -agents: A benchmark for multilingual function-calling in 52 languages . In Findings of the Association for Computational Ling... on Pithdoi:10.18653/v1/2025.findings-emnlp.1099
  11. [45] Liquid AI . 2026. https://huggingface.co/LiquidAI/LFM2.5-8B-A1B LFM2.5-8B-A1B . Model release. Model card, accessed 2026-10-14 on Pith
  12. [46] Zheng Luo, T. Pranav Kutralingam, Ogochukwu N. Okoani, Wanpeng Xu, Hua Wei, and Xiyang Hu. 2026. https://aclanthology.org/2026.acl-long.2039/ Lost in execution: On the multilingual robustness of tool calling in large language models . In Proceedings of the 64th Annual Meeting ... on Pith
  13. [47] Ministry of Statistics and Programme Implementation . 2024. https://mospi.gov.in/sites/default/files/publication_reports/Report_591_HCES_2022-23New.pdf Household consumption expenditure survey 2022--23: Report no. 591 . Technical report, Government of India on Pith
  14. [48] Mistral AI . 2025. https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 Mistral Small 3.2 . Model release. Model card, accessed 2026-10-14 on Pith
  15. [49] OpenAI . 2025. https://arxiv.org/abs/2508.10925 gpt-oss-120b & gpt-oss-20b model card . arXiv preprint arXiv:2508.10925 on PithPith's reviewarXiv
  16. [50] Kuang Peng, Yuchun Fan, Jiangnan Li, Minghao Wu, Jialong Tang, Haoran Wei, Weixuan Wang, Jianhong Tu, Baosong Yang, and Tong Xiao. 2026. https://arxiv.org/abs/2609.23490 Babelarena: A large-scale multilingual benchmark for LLM agents . arXiv preprint arXiv:2609.23490 on PithPith's reviewarXiv
  17. [51] Qwen Team . 2026. https://huggingface.co/Qwen/Qwen3.8-27B Qwen3.8-27B . Model release. Model card, accessed 2026-10-14 on Pith
  18. [52] Daksh Raghuvanshi, Ved Vedere, and Yifan Wang. 2026. https://arxiv.org/abs/2610.10942 Storebench: A live-commerce environment for evaluating and training autonomous operator agents . arXiv preprint arXiv:2610.10942 on PithPith's reviewarXiv
  19. [53] Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu, Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, and Chengfu Huo. 2026. https://arxiv.org/abs/2607.28956 Merchantbench: Benchmarking llm agents for long-term coherence in e-commerce o... on PithPith's reviewarXiv
  20. [54] Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. https://arxiv.org/abs/2404.16816 Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages . arXiv preprint arXiv:2404.16816 on PithPith's reviewarXiv
  21. [55] Stuti Sinha, Himanshu Kumar, Aryan Raju Mandapati, Rakshit Sakhuja, and Dhruv Kumar. 2026. https://arxiv.org/abs/2601.04387 The language of bargaining: Linguistic effects in LLM negotiations . arXiv preprint arXiv:2601.04387 on PithPith's reviewarXiv
  22. [56] Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2024. https://arxiv.org/abs/2411.02538 Milu: A multi-task indic language understanding benchmark . arXiv preprint arXiv:2411.02538 on PithPith's reviewarXiv
  23. [57] Yifan Wang, Jiaxin Shi, James Oldfield, Samuele Marro, Philip Yu, Philip Torr, Yali Du, and Adel Bibi. 2026. https://arxiv.org/abs/2610.06748 Bazaarbench: Delegation safety in decentralized C2C marketplaces run by LLM agents . arXiv preprint arXiv:2610.06748 on PithPith's reviewarXiv
  24. [58] Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, et al. 2024. https://arxiv.org/abs/2412.14161 Theagentcompany: Benchmarking llm agents on consequential real world tasks . arXiv preprint arXiv:2412.14161 on PithPith's reviewarXiv
  25. [59] Linghua Zhang, Jun Wang, Jingtong Wu, and Zhisong Zhang. 2026 a . https://arxiv.org/abs/2603.16453 Retailbench: Evaluating long-horizon autonomous decision-making and strategy stability of llm agents in realistic retail environments . arXiv preprint arXiv:2603.16453 on PithPith's reviewarXiv
  26. [60] Xinran Zhang, Pengrui Lu, Lyumanshan Ye, and Pengfei Liu. 2026 b . https://arxiv.org/abs/2609.04667 Erpbench: Evaluating llm agents for enterprise decision-making across competitive market ecologies . arXiv preprint arXiv:2609.04667 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment