Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

LLMs build valid spreadsheets but fall short on function and style, and adjusting for spreadsheet features compresses the leaderboard sharply.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:03 UTC pith:VP7OFQOK

load-bearing objection A real new benchmark with a serious, fixable identifiability problem in its central feature-adjustment analysis. the 4 major comments →

arxiv 2603.10002 v2 pith:VP7OFQOK submitted 2026-02-16 cs.CL cs.AIcs.LG

SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

Srivatsa Kundurthy , Clara Na , Michael Handley , Zach Kirshner , Chen Bo Calvin Zhang , Manasi Sharma , Emma Strubell , John Ling This is my paper
classification cs.CL cs.AIcs.LG
keywords spreadsheet generationLLM evaluationarena benchmarkBradley-Terry modelpreference decompositionfinancial modeling conventionsstructured artifact generationhuman preference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SpreadsheetArena, a platform and dataset for evaluating end-to-end spreadsheet generation through blind pairwise human votes over 4,357 battles among 16 LLMs. It argues that spreadsheet generation is a challenging, understudied task: models usually produce well-formed workbooks with valid formulas, but practical functional utility and adherence to stylistic guidelines are much less reliable. Using a feature-augmented paired-comparison model, the paper shows that observable spreadsheet features—text density, fills, numeric content, formula error rates, and layout—explain a large share of preference, compressing Elo differences and shifting rankings; which features matter varies dramatically by domain. A finance-domain expert evaluation shows that even top-ranked models often violate professional modeling conventions, and expert ratings agree with crowd preferences only modestly. The authors' central claim is that preferences over structured spreadsheet artifacts do not uniformly reward all quality dimensions, so aggregate leaderboards need to be decomposed.

Core claim

End-to-end spreadsheet generation is presented as a distinct and challenging benchmark task. The paper claims that current LLMs produce syntactically valid spreadsheet workbooks with valid formulas more often than not, but that practical functional utility and adherence to stylistic guidelines are much less reliable. In blind pairwise evaluations, formatting and structural features (text density, background fills, number formatting, layout width) are significantly associated with win likelihood, while formula-sophistication features (lookups, conditionals, embedded constants) are not; adjusting for 29 spreadsheet features compresses the Elo leaderboard substantially, with top models losing 1

What carries the argument

The feature-augmented Bradley-Terry model is the central mechanism: win probability is modeled as sigma(theta_A - theta_B + sum_k beta_k (X_Ak - X_Bk)), where theta are per-model strength parameters and beta_k are coefficients on pairwise differences of 29 programmatically extracted spreadsheet features. This decomposes preference into model identity versus observable output features, yielding feature-adjusted Elo ratings. Supporting machinery includes the SheetSpec JSON schema for deterministic spreadsheet rendering, a 7-bucket LLM-judged failure taxonomy, and a finance-expert rubric covering errors, formula conventions, color coding, structure, modeling conventions, and practical utility.

Load-bearing premise

The leaderboard is estimated only from battles in which both models produced syntactically valid spreadsheets; if models differ in how often they generate valid output at all, the Elo ranking does not measure the probability that a model satisfies a user.

What would settle it

Compute each model's invalid-output rate on the released prompts and re-estimate the paired-comparison model treating an invalid output as a loss (or weighting battles by output validity); if the re-ranked leaderboard differs materially from the published one, the conditional-ranking assumption fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If formatting and layout features, not formula sophistication, drive most preference votes, then spreadsheet arena leaderboards should be read as style-and-structure preference rankings rather than correctness rankings.
  • Feature adjustment compresses the leaderboard substantially, so a model's raw Elo advantage is partly attributable to measurable output features; post-training on naive pairwise preferences may reward surface polish over substance.
  • Domain-specific feature effects (number formatting hurts in academic prompts, finance color conventions help in finance prompts) imply that a single global preference signal is insufficient for structured generation evaluation and alignment.
  • The modest agreement between finance experts and crowd votes suggests that crowd preferences alone are not a proxy for professional best practices; expert rubrics supply a distinct, non-redundant quality signal.
  • The failure taxonomy shows presentation deficiencies dominate across all models, while top models fail more often on integrity and numerical computation errors, indicating that blind preference may underweight 'looks fine but is structurally misleading' failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because battles with invalid outputs are discarded, the published Elo leaderboard estimates preference conditional on syntactic validity; a model that frequently fails to produce a valid workbook is neither penalized on the leaderboard nor matched against the full competition, so the ranking may not reflect the probability that a model satisfies a real user.
  • The feature-adjusted scores are explicitly not counterfactual estimates; a natural testable extension is to manipulate features directly (for example, strip formatting from a winning spreadsheet) and check whether win probability shifts as the fitted coefficients predict.
  • The 'integrity failure' bucket—hardcoded checks, drivers not linked to outputs, models that do not respond to input changes—points to a spreadsheet-specific failure mode that scalar feature extraction likely misses; a targeted auditability metric (e.g., mutate an input and verify all dependent outputs update) could catch it automatically.
  • The domain-specific sign reversals suggest that vertical or per-category leaderboards would rank models differently from a global arena; releasing per-category preference data would let others test whether specialized rankings better predict real-world task success.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces SpreadsheetArena, a live pairwise-preference platform for evaluating end-to-end LLM spreadsheet generation. The authors collect 4,357 blind votes over 16 models, compute Bradley–Terry/Elo ratings, augment the BT model with 29 programmatically extracted spreadsheet features to 'decompose' preference, build a data-driven failure taxonomy via an LLM judge, and report a finance-domain expert evaluation. The main claims are that spreadsheet generation is a challenging, understudied task; that feature-adjusted Elo scores compress the leaderboard; that significant preference-relevant features vary substantially by domain; and that crowd preferences align only weakly with expert finance judgments.

Significance. If the statistical decomposition were identified, this would be a useful benchmark contribution: spreadsheet artifacts are an underexplored structured-generation domain, the live arena and planned dataset release are valuable, the failure taxonomy is a practical tool, and the expert finance study provides an external check that most arena evaluations lack. The paper is also honest about the regression-adjustment interpretation. However, the central 'decomposing preference' analysis is compromised by an identifiability problem in the feature-augmented BT model, and several secondary claims lack proper uncertainty quantification and multiple-testing controls. The descriptive benchmark, failure taxonomy, and expert study are likely salvageable, but the headline feature-decomposition results need to be re-estimated and re-interpreted.

major comments (4)
  1. [§3.3, Eq. (1)] The covariate X_ik is defined as the mean value of feature k across outputs generated by model i. In a BT model with per-model intercepts θ_i, model-level covariates are collinear with those intercepts. For any vector c, the transformation θ_i' = θ_i + cᵀ(X_i − X_m0), β' = β − c leaves every predicted P(A≻B) in Eq. (1) unchanged and preserves the GPT-4o anchor θ_m0 = 1000. The likelihood therefore has a flat direction; the paper reports no identifiability constraint, no regularization, and no rank/degeneracy check. This makes the feature-adjusted Elo values in Table 1, the win-probability shifts in Fig. 3, and the domain-specific coefficients in Tables 3/4/10/13 unidentifiable as specified. Re-specify X_ik as the feature value of the specific output in each battle, or impose and justify an identifiable constraint, and re-estimate all affected results.
  2. [§4.3, Algorithm 1] Line 10 discards any pair in which at least one model generates an invalid output and replaces it with another pair. This conditions the Elo ranking on both models producing schema-valid workbooks. If invalid-output rates differ across models—which the failure taxonomy in Table 5 suggests is plausible for weaker models—the published leaderboard does not estimate the probability that a model satisfies a user, and models that fail often are never penalized for those failures. The global claim in §1 that models produce 'well-formed spreadsheets with valid formulas more often than not' is not supported without reporting invalid-output rates per model. Please quantify these rates and either include invalid battles as losses in a sensitivity analysis or explicitly state and justify the conditional interpretation.
  3. [§5.2.2, Table 10] The domain-specific analysis tests 29 features across five prompt categories, i.e., approximately 145 tests, but reports no multiple-testing correction. At the 0.05 level, roughly seven false positives are expected by chance. The highlighted findings are individually near the threshold: e.g., Academic & Research pct_number_format p = 0.041 and pct_fill p = 0.040; Finance pct_number_format p = 0.046, finance_color_convention p = 0.022, largest_table_pct p = 0.030, has_border p = 0.013. These may not survive correction. Report adjusted p-values or validate the domain-specific feature effects on a held-out or pre-registered set of comparisons.
  4. [Tables 1, 4, 13 and Fig. 3] No confidence intervals, standard errors, or bootstrap intervals are reported for any Elo rating or BT coefficient. With 4,357 votes spread over 16 models, the 'stable ranking' claim is not quantitatively supported, and several ranking shifts after feature adjustment are small (e.g., GPT-5.2 dropping two positions in Table 1). Report uncertainty estimates for both the baseline and feature-adjusted ratings, and use them to assess whether the rank changes are distinguishable from noise.
minor comments (4)
  1. [§5.2.1] Typo: 'parwise' should be 'pairwise'.
  2. [Appendix F] The sentence 'merging professional finance and professional finance categories' contains an obvious typo; it should read 'Professional Finance and Corporate Finance & FP&A.'
  3. [Table 4] The Academic & Research ranking table lists only 10 of the 16 models, without explaining the selection criterion. State whether the remaining models had insufficient votes in that category or were omitted for another reason.
  4. [References] Several author names contain mojibake/encoding artifacts, e.g., 'Roziôlre', 'DÃl'fossez', 'AkyÃijrek'. These should be fixed to proper UTF-8 or LaTeX escapes.

Circularity Check

1 steps flagged

Feature-adjusted Elo in Eq. (1) reduces to vanilla Bradley–Terry by construction: covariates are model-level means, so β and θ are collinear and the leaderboard compression is unidentifiable.

specific steps
  1. other [§3.3 Eq. (1); applied in §5.2.1, §5.2.2, Table 1, Tables 3/10]
    "The augmented Bradley-Terry model expresses win probability as: P(A≻B)=σ(θA−θB+ΣKk=1 βk(XAk−XBk)) (1) where θi again denotes the latent skill of model i, βk is the coefficient for differenced feature k, and Xik is the mean value of feature k across outputs generated by model i."

    Because Xik is a model-level mean, XAk−XBk is constant across every battle between a fixed pair of models. The covariate sum equals (ΣβkXAk)−(ΣβkXBk), so with θ̃i = θi + ΣβkXik, Eq. (1) is algebraically identical to the vanilla Bradley–Terry logit θ̃A−θ̃B. The per-model θi are free parameters, so β is unidentified: for any c, θi′ = θi + c(Xi−Xm0), β′ = β−c preserves every predicted probability and the GPT-4o anchor. The likelihood is flat along this direction. Thus the 'feature-adjusted' θ and β reported in Table 1 and Tables 3/10 are arbitrary reparameterizations of the same preference data, not quantities identified by the votes; the claimed leaderboard compression and per-domain feature effects are not estimable as specified.

full rationale

The raw arena pipeline—4,357 blind votes, vanilla Elo estimation, the failure taxonomy, and the finance expert study—is self-contained and does not depend on a circular self-citation chain; the expert study is a partly external check and the paper's raw rankings remain meaningful. The circularity is localized to the preference-decomposition model, which is the paper's headline contribution. In Eq. (1), the covariates are defined as model-level feature means, so the feature-difference term is a linear combination of the model intercepts already present in the model. Substituting θ̃_i = θ_i + Σβ_kX_{ik} reproduces the vanilla Bradley–Terry likelihood exactly, and no identifiability constraint or regularization is stated. The reported 'feature-adjusted' Elo compression and per-domain significant feature coefficients are therefore an arbitrary reparameterization of the same votes rather than an identified decomposition of preference. This is a mathematical equivalence by construction, not a speculation about author intent. The paper's disclaimer that the adjustments are 'not an estimate of counterfactual performance' does not cure the collinearity. Because the decomposition analysis is central but other contributions (dataset, vanilla leaderboard, expert evaluation) are independent, the score is 6 rather than higher.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 3 invented entities

The paper's central rankings and feature decompositions are estimated from the same 4,357 votes with no external ground-truth benchmark for spreadsheet quality. The evaluation assumes decisive votes are representative, invalid outputs are non-informative, and single-pass JSON generation stands for the full task.

free parameters (3)
  • BT strength coefficients theta_i for 16 models = Anchored Elo values in Table 1 (e.g., Claude Opus 4.5=1550, Qwen3 30B=692, GPT-4o=1000)
    Estimated by MLE from the 4,357 votes and converted to Elo; every ranking claim in Sections 5.1-5.2 depends on these estimates, which carry no confidence intervals and are conditioned on discarding invalid-output battles.
  • Feature coefficients beta_k for 29 spreadsheet features = Table 8 (e.g., pct_text +1.562, compute_error_rate -1.338, pct_fill +1.150)
    Fitted in the augmented Bradley-Terry model; the feature-adjusted leaderboard and domain-specific analyses are direct functions of these coefficients. No multiple-testing correction is applied, and multicollinearity is not diagnosed.
  • GPT-4o anchor (theta_m0 = 1000) = 1000
    Hand-selected reference rating (Section 3.2); does not affect relative ordering but fixes the Elo scale and all reported point deltas.
axioms (6)
  • standard math Bradley-Terry specification P(A≻B)=σ(θA−θB) and MLE consistency
    Used to convert votes into rankings (Section 3.1); if preferences are non-transitive or context-dependent, the single-parameter model mis-summarizes them.
  • domain assumption Decisive votes are representative of all evaluations
    Only 87.5% of battles had a clear preference; ties and both-bad votes are excluded from most analyses (Section 5.1), which can bias rankings if non-decisive outcomes correlate with models or prompt categories.
  • domain assumption Blind pairwise preference is a valid proxy for spreadsheet quality
    Core measurement of the arena (Section 4.3); voters are unvetted and may not audit files, so votes can reward visual polish rather than functional correctness.
  • domain assumption Single-pass SheetSpec@2 JSON generation is representative of end-to-end spreadsheet generation
    Iterative/agentic generation is explicitly left to future work (Section 4.2); conclusions about model capability are therefore scoped to one-shot generation.
  • ad hoc to paper Invalid-output battles are non-informative
    Algorithm 1 discards pairs where at least one model produces invalid output and replaces them; this treats invalidity as missing data rather than as a failure, a choice that directly affects the leaderboard.
  • domain assumption LLM-judge failure taxonomy is valid at scale
    Judge alignment is checked on only 50 stratified samples with 78% agreement (Section 5.3); the taxonomy is then applied to all decisive battles without per-battle human verification.
invented entities (3)
  • SpreadsheetArena platform no independent evidence
    purpose: Collects blind pairwise preference votes over LLM-generated workbook pairs and computes Elo rankings
    The platform is introduced by this paper; its validity as a quality measure rests on assumptions not yet externally validated or replicated.
  • SheetSpec@2 JSON schema no independent evidence
    purpose: Serialization format for generated spreadsheet workbooks, including cells, formulas, styles, named ranges, and conditional formatting
    New specification defined in Appendix B; no external standard or interoperability validation is provided.
  • 7-bucket failure taxonomy no independent evidence
    purpose: Multi-label classification of dispreferred spreadsheets in head-to-head battles
    Derived via clustering and hand-curation; only internal expert validation (78% agreement on 50 items) is reported, with no external benchmark.

pith-pipeline@v1.3.0-alltime-deepseek · 25913 in / 15788 out tokens · 163726 ms · 2026-08-02T23:03:35.348127+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks." pith.science (2026). https://pith.science/paper/VP7OFQOK

@misc{pith2026260310002,
  author       = {Pith},
  title        = {Pith review of: SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VP7OFQOK}},
  note         = {Machine review of arXiv:2603.10002}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We consider the task of end-to-end spreadsheet generation, where language models produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language. We introduce SpreadsheetArena, a platform for evaluating models' performance on the task via blind pairwise preference votes of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary greatly across use cases, often in ways that are difficult to formalize. Compared to general dialogue or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex interactivity and layout considerations. We observe that stylistic, structural, and functional features of preferred spreadsheets vary meaningfully across prompts. Expert evaluations of spreadsheets for finance prompts suggest that even highly ranked models do not reliably produce spreadsheets aligned with domain-specific best practices. We host a live arena and release a dataset of prompts, generated spreadsheets, and preference votes, which we hope will facilitate further study of tasks operating over spreadsheets as a challenging and interesting class of complex, open-ended tasks for LLMs.

Figures

Figures reproduced from arXiv: 2603.10002 by Chen Bo Calvin Zhang, Clara Na, Emma Strubell, John Ling, Manasi Sharma, Michael Handley, Srivatsa Kundurthy, Zach Kirshner.

Figure 1
Figure 1. Figure 1: Elo ratings for 16 models ranked in SPREAD￾SHEETARENA. Standard Elo scores are anchored on GPT-4o at 1000. Overall, Claude models are often preferred. In §5 we con￾textualize these global rankings with observable feature-adjusted scores, category-specific analysis across prompts, characterization of failure modes in dispreferred spreadsheets, and expert evalua￾tions in financial modeling use cases. models … view at source ↗
Figure 2
Figure 2. Figure 2: In SPREADSHEETARENA, users submit a prompt and are shown four pairwise battles between LLM-generated spreadsheet workbooks. Votes are blind, and users can indicate that one spreadsheet is preferred over the other, or that both are equally satisfactory or unsatisfactory. Workbooks can contain multiple sheets, and sheets often contain a mixture of text, values, and formulas, where cells may contain stylistic… view at source ↗
Figure 3
Figure 3. Figure 3: presents the pairwise win probability changes. We find that feature adjustment redistributes estimated compet￾itive advantage; for example, Claude Opus 4.5’s average win probability against all opponents decreases by 11.2 percentage points on average. Claude Opus 4.5 Gemini 3 Pro Claude Opus 4.1 Claude Sonnet 4.5 Gemini 2.5 Flash Gemini 2.5 Pro GPT-5.2 GPT-5 Grok 4.1 Fast Grok 4 GPT-5.1 Grok Code Fast 1 Ki… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of expert ratings across six evaluation di￾mensions for finance-domain spreadsheets (n = 134 evaluations). Color Coding and Formatting stands out as the weakest dimension, with 77.6% of evaluations scoring 2 or below. Limited alignment between expert evaluations and arena preferences suggests that generalized arena preferences may not fully capture finance domain-specific quality require￾ments… view at source ↗
Figure 6
Figure 6. Figure 6: A model response to the “Academic & Research” prompt in Appx. D. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Elo ratings trend inwards after feature adjustment. E.2. Corporate Finance & FP&A [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: A model response to the “Corporate Finance & FP&A” prompt in Appx. D. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A model response to the “Creative & Generative” prompt in Appx. D [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: A model response to the prompt “draw a spongebob fully colored and not exceeding a 50 x 50 size”. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A model response to the “Operations & Supply Chain” prompt in Appx. D. E.5. Professional Finance [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: A model response to the “Professional Finance” prompt in Appx. D. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: A model response to the “SMB & Personal” prompt in Appx. D. F. Feature Coefficients [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Expert ratings for arena winners vs. losers across 52 finance-domain battles. Points above the diagonal indicate expert agreement with arena outcomes; points below indicate disagreement. L. Finance Expert Evaluation Scoring Rubric and Instructions In this section, we include the full rater instructions and scoring anchors. L.1. Evaluator Instructions For each assigned task, evaluators completed the follow… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BlueFin: Benchmarking LLM Agents on Financial Spreadsheets

    cs.SE 2026-05 unverdicted novelty 6.0

    BlueFin is a new benchmark for LLM agents on financial spreadsheets showing frontier models score below 50% with weaknesses in dynamic correctness.

Reference graph

Works this paper leans on

6 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [2]

    magic numbers

    Download and open the Excel file.Review it as you would any financial model—check formulas, structure, formatting. 3.Rate on 6 criteria (1–5 scale).Score each dimension using the detailed rubric below. 4.Add notes (optional but helpful).Brief explanations of scores help us understand the reasoning. 5.Review and submit.The overall rating is calculated auto...

  2. [5]

    r”, black = “b

    URL https://showdown.scale.com/ assets/SEAL_Showdown_Tech_Report.pdf. Accessed: 2026-01-21. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https: //arxiv.org/abs/1707.06347. Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin- son, D., Authur, R.,...

    Pith/arXiv arXiv 2026

  3. [9]

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C

    URL https://www.sciencedirect.com/ science/article/pii/S0167923610000461. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728–53741, 2023. Roziôlre, B., Gehring, J., Gloeckle, F., Sootl...

    Pith/arXiv arXiv 2023

  4. [2022]

    cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference

    URL https://proceedings.neurips. cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference. 12 SpreadsheetArena: Decomposing Preference in LLM Spreadsheet Generation html. Also available as arXiv preprint arXiv:2203.02155. Panko, R. R. and Aurigemma, S. Revising the panko- halverson taxonomy of spreadsheet errors.Deci- sion Supp...

    Pith/arXiv arXiv 2022

  5. [2024]

    URL https: //arxiv.org/abs/2406.04770

    doi: 10.48550/arXiv.2406.04770. URL https: //arxiv.org/abs/2406.04770. Ma, Z., Zhang, B., Zhang, J., Yu, J., Zhang, X., Zhang, X., Luo, S., Wang, X., and Tang, J. Spreadsheet- bench: Towards challenging real world spreadsheet manipulation. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances in Neural...

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.04770 2024

  6. [2025]

    Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L

    URL https://openreview.net/forum? id=DIELgiqdvJ. Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, X., Gu, Y ., Malik, S., Graf, V ., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y ., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing fr...

    Pith/arXiv arXiv 2025