Pith. sign in

REVIEW 3 major objections 5 minor 12 references

An encoder model can predict Magic: the Gathering draft deck strength from pick sequences alone, outperforming linear baselines and approaching the data's noise ceiling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 13:36 UTC pith:F34JEGSX

load-bearing objection Solid first outcome-prediction benchmark for MTG Draft: real gains over linear baselines, honest noise ceiling, code shipped; transfer is the soft spot. the 3 major comments →

arxiv 2607.04782 v1 pith:F34JEGSX submitted 2026-07-06 cs.LG cs.AI

Predicting Drafted Deck Strength for "Magic: the Gathering"

Tomas Rigaux , Hisashi Kashima This is my paper
classification cs.LG cs.AI
keywords Magic: the Gatheringdraftdeck strength predictiontransformer encoderset-contextualized embeddingscombinatorial synergiesoutcome predictioncard games
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Magic: the Gathering Draft forces players to build a 40-card deck through 39–45 sequential picks under partial information, where each card's value depends on combinatorial synergies with earlier and later choices rather than fixed rules. This paper claims that an encoder architecture can turn those pick sequences into set-contextualized card embeddings and, from them, predict the resulting deck's win rate. Trained on large-scale real Arena draft logs, the model consistently beats linear and heuristic baselines, nearly doubles their rank correlation after the first few picks, and approaches the estimated 0.23 Spearman ceiling imposed by sampling noise. The result matters because it shows that outcome prediction is possible without simulating games, giving a first learned benchmark for a setting where cards themselves rewrite the rules and the card pool continually expands.

Core claim

DraftEncoder produces set-contextualized card embeddings that encode the full draft decision sequence and yields a consistent improvement over linear baselines on large-scale real-world data, establishing a first learned benchmark for outcome prediction in MTG Draft that approaches the estimated Spearman noise ceiling of roughly 0.23.

What carries the argument

DraftEncoder: a two-stage transformer encoder that first contextualizes each card against the rest of the expansion (and player rank via FiLM), then iteratively updates pack-card and draft-state embeddings so that later picks condition on earlier combinatorial choices before a final MLP predicts win probability.

Load-bearing premise

Censored match records from self-selected Arena players (stop at 7 wins or 3 losses, average observed win rate inflated to about 55 percent) are clean enough proxies for true deck strength that combinatorial synergies can be learned from draft sequences alone, without any gameplay simulation.

What would settle it

On a held-out expansion with matched rank and meta-feature controls, if DraftEncoder's Spearman correlation with observed win rates stays at or below the linear baseline even after four days of fine-tuning, the claim that the architecture captures transferable combinatorial draft structure would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies outcome prediction for Magic: the Gathering Draft, isolating sequential card selection (39–45 picks under partial information) from gameplay. It introduces DraftEncoder: frozen EmbeddingGemma text embeddings of oracle card strings (optionally concatenated with 17Lands META win-rate statistics), FiLM-conditioned on player rank, refined by a set-level Transformer encoder (NE layers) into expansion-contextualized card embeddings, then a draft-state encoder (ND layers with RoPE and pack/pick attention) that produces per-pick win-rate predictions. Training uses a pick-weighted NLL that respects the 7-win/3-loss censoring rule. On temporally split 17Lands logs the model consistently outperforms a heuristic (top-23 in-deck win rates) and linear regression baselines in Spearman ρ (Fig. 3), approaches the authors’ simulated noise ceiling of ~0.23, and shows limited zero-shot transfer to a held-out expansion that improves after four days of fine-tuning (Table II). Code is released.

Significance. If the reported gains hold, the work supplies the first public learned benchmark for predicting MTG Draft deck strength from draft sequences alone, together with a reusable architecture for set-contextualized combinatorial decisions and a carefully designed loss for censored match outcomes. The large real-world scale, temporal splits, multi-seed reporting, explicit noise-ceiling calibration, and open code are concrete strengths that make the benchmark immediately usable by the game-AI and structured-prediction communities. The contribution is modest—primarily empirical and domain-specific—yet fills a genuine gap left by prior pick-prediction work and demonstrates that non-linear synergy modeling can extract measurable signal even under heavy label noise and self-selection bias.

major comments (3)
  1. [§IV-A / §V-A] §IV-A and §V-A (META features): The paper never states whether the per-card META statistics (average win rates when in opening hand, drawn, etc.) are computed exclusively on the training temporal split or on the full 17Lands dump. If the latter, the +Meta variants in Fig. 3 and Table II contain direct label leakage; the claimed “consistent improvement” of DraftEncoder+Meta would then be partly artifactual and the comparison to pure text embeddings would be unfair. This must be clarified and, if necessary, recomputed.
  2. [§VI / Table II] §VI / Table II (generalization): Zero-shot transfer without META features is no better than linear regression and worse than the heuristic that already encodes full-set meta statistics. The fine-tuning result is encouraging but rests on only four days of FDN data and three seeds; the paper should report whether the same rapid adaptation occurs for additional held-out expansions (or at least quantify variance across more than one target set) before claiming that “transferable draft representations imes adapt quickly.”
  3. [§V-B] §V-B (baselines): The only non-linear comparator is the authors’ own model. A simple bag-of-picks MLP or a standard Transformer over the sequence of selected cards (without the pack-attention and draft-state machinery) is missing. Without it, it remains unclear how much of the Spearman gain in Fig. 3 is due to the proposed set-contextualization versus ordinary non-linearity.
minor comments (5)
  1. [Fig. 3] Fig. 3: the y-axis label “Spearman” is truncated; expand to “Spearman ρ” and state explicitly that the shaded bands are ±1 std over three seeds.
  2. [§IV-C] §IV-C: the binomial coefficients in the censored NLL are written with non-standard notation (e.g., 6+L choose 6); a short derivation or reference to the 7/3 stopping rule would help readers verify the likelihood.
  3. [Table I] Table I caption and §II: “sample drafts” should be “drafts”; also clarify whether the listed draft counts already exclude incomplete or bot-assisted logs.
  4. Throughout: inconsistent capitalization and quotation of the game title (“Magic: the Gathering” vs. “Magic: The Gathering”); pick one style.
  5. [§VII] §VII: the claim that the model “approaches the estimated noise ceiling of 0.23” is slightly overstated for the in-distribution curves (Fig. 3 peaks near 0.20); soften or show the exact peak value.

Circularity Check

1 steps flagged

No significant circularity: DraftEncoder is trained to predict held-out censored win rates from draft sequences; optional META features share a data source but are ablated and do not force the claimed gains by construction.

specific steps
  1. fitted input called prediction [Section IV-A (Card Features / META) and Results (Fig. 3, Table II)]
    "we also compute statistical features E(N)c for each card, following the META features of Bertram et al. [3] (i.e. the average card win rate when it is in the opening hand, drawn, tutored, in the deck, in the sideboard, or seen during the game). In our results, we indicate when a model concatenates these META features to the textual features. ... Adding meta features consistently improves performance"

    The optional META features are empirical average win rates computed from the same 17Lands ecosystem that supplies the (W, L) labels. Concatenating them gives the model direct access to quantities correlated with the target, so part of the +Meta lift is statistically forced rather than purely learned from draft sequences. This is minor: the paper ablates META, still reports gains for DraftEncoder without it (especially after fine-tuning), and does not claim the META-augmented numbers as a first-principles prediction.

full rationale

The paper's central claim is an empirical modeling result: an encoder architecture that contextualizes card embeddings and draft states yields higher Spearman ρ than linear and heuristic baselines on 17Lands draft logs (Fig. 3, Table II), approaches a simulated noise ceiling of ~0.23, and is released with code. The learning objective (Section II) maps draft sequences D≤k to p_win via a weighted NLL that accounts for the 7-win/3-loss censoring; this is ordinary supervised learning on held-out temporal splits, not a quantity defined by the model outputs. The optional META features (average card win rates from the same 17Lands source) introduce a mild shared-source risk, but the paper reports both +Meta and −Meta variants, shows that the architecture still improves without them after fine-tuning, and does not treat META as a first-principles derivation. The heuristic baseline also uses those stats, so the comparison is transparent rather than circular. No self-definitional equations, fitted-input-as-prediction, load-bearing self-citation uniqueness claims, or renamed known results appear. The result is self-contained against external benchmarks (real draft logs, public code). Score 1 reflects only the minor optional META overlap already flagged by the reader; the derivation chain does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard supervised learning plus domain modeling choices for MTG Draft and 17Lands outcomes. Free parameters are ordinary neural hyperparameters and the FiLM scale; axioms are domain facts about draft rules and the use of censored win rates as labels; invented entities are architectural constructs (contextualized card embeddings, draft-state tokens) without independent physical existence.

free parameters (6)
  • inner embedding dimension = 64
    Chosen as 64 for both linear baselines and DraftEncoder; not derived from theory.
  • NE (card encoder layers) = 10
    Number of residual attention layers over expansion cards; set by authors.
  • ND (draft encoder layers) = 5
    Number of pack/state update blocks; set by authors.
  • FiLM gamma scale (0.1 tanh) = 0.1
    Hand-chosen scale on rank-conditioned gamma to keep modulation mild.
  • peak learning rate / training schedule = 1e-4
    AdamW peak LR 1e-4, 100 epochs, batch 256; fitted for validation loss.
  • loss pick-weighting i/K = i/K
    Later picks weighted higher by construction; design choice, not derived.
axioms (4)
  • domain assumption Draft is a sequence of K∈{39,42,45} picks from packs of size 13–15 with alternating pass direction; deck strength can be scored from post-draft match outcomes alone.
    Section II formalizes the draft and isolates selection from gameplay; the learning objective depends on this isolation.
  • domain assumption Observed (W,L) under a 7-win/3-loss stop rule is a usable (if censored and biased) estimate of true deck win probability p_win.
    Outcome definition and NLL loss in Sections II and IV-C; average win rate ~55% noted as self-selection bias.
  • ad hoc to paper Frozen EmbeddingGemma text embeddings of oracle card strings, optionally concatenated with META stats, are sufficient initial features for learning synergies via attention.
    Section IV-A; no proof that text embeddings capture rules interactions—assumed and tested empirically.
  • standard math Standard transformer attention, residual MLP blocks, RoPE, and FiLM are valid building blocks for set- and sequence-contextualization.
    Invokes Vaswani et al., Su et al. (RoPE), Perez et al. (FiLM) as background machinery.
invented entities (2)
  • DraftEncoder (set-contextualized card embeddings + draft-state tokens) no independent evidence
    purpose: Encode expansion context and sequential draft decisions into per-pick win-rate predictions.
    Architectural construct introduced in Section IV; no existence claim beyond the model.
  • Draft state token s_k no independent evidence
    purpose: Summarize information after k picks for the predictor head.
    Defined in IV-B as an evolving embedding updated by attention over pack cards; internal model state only.

pith-pipeline@v1.1.0-grok45 · 12079 in / 3302 out tokens · 29758 ms · 2026-07-11T13:36:03.486366+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Predicting Drafted Deck Strength for "Magic: the Gathering"." pith.science (2026). https://pith.science/paper/F34JEGSX

@misc{pith2026260704782,
  author       = {Pith},
  title        = {Pith review of: Predicting Drafted Deck Strength for "Magic: the Gathering"},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F34JEGSX}},
  note         = {Machine review of arXiv:2607.04782}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Many real-world games do not admit a fixed, compact rule set: instead, their dynamics are defined by interactions among a large and often evolving collection of game pieces, making general-purpose policy learning impractical. Magic: the Gathering (MTG) exemplifies this setting, where the cards themselves define and alter gameplay rules, strategic constraints, and long-term outcomes, while the pool of available cards is ever-changing. We study Draft, a constrained deck-building format of MTG in which eight players make 39-45 sequential selections from semi-random packs to construct a 40-card deck under partial information. By isolating the card selection process from gameplay, Draft provides a tractable yet non-trivial setting for studying decision-making driven by combinatorial card synergies. We propose an encoder-based model that produces set-contextualized card embeddings to encode the draft decision sequence, with a consistent improvement over linear baselines on large-scale real-world data, establishing a first learned benchmark for outcome prediction in MTG Draft. Our code is available at github.com/akulen/MtGDraftEncoder.

Figures

Figures reproduced from arXiv: 2607.04782 by Hisashi Kashima, Tomas Rigaux.

Figure 1
Figure 1. Figure 1: Overview of DraftEncoder’s full architecture. The main block [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: To prepare the oracle string of a card, we concatenate the applying [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Spearman ρ across pick number, on BLB, FDN, and TDM. Shaded bands show ± standard deviation over 3 seeds. We train the linear regression and our model for 100 epochs, with an inner dimension of 64, NE = 10, ND = 5, 2 attention heads, a batch size of 256, optimizing with AdamW [10], and a peak learning rate of 1e-4. VI. RESULTS We report, as mean ± standard deviation over 3 random seeds for the epoch with t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 1 linked inside Pith

  1. [1]

    A Large-Scale Dataset of MTG Drafts and Games,

    17Lands, “A Large-Scale Dataset of MTG Drafts and Games,” 2026

    2026

  2. [2]

    AI solutions for drafting in Magic: the Gathering,

    H. N. Ward, B. Mills, D. J. Brooks, D. Troha, and A. S. Khakhalin, “AI solutions for drafting in Magic: the Gathering,” in2021 IEEE Conference on Games (CoG). IEEE, 2021, pp. 1–8

    2021

  3. [3]

    Learning With Generalised Card Representations for “Magic: The Gathering

    T. Bertram, J. Fürnkranz, and M. Müller, “Learning With Generalised Card Representations for “Magic: The Gathering”,” in2024 IEEE Conference on Games (CoG). IEEE, 2024, pp. 1–8

    2024

  4. [4]

    Mastering Strategy Card Game (Hearthstone) with Improved Techniques,

    C. Xiao, Y . Zhang, X. Huang, Q. Huang, J. Chen, and c. Sun, “Mastering Strategy Card Game (Hearthstone) with Improved Techniques,” in2023 IEEE Conference on Games (CoG), 2023, pp. 1–8

    2023

  5. [5]

    EmbeddingGemma: Powerful and Lightweight Text Representations,

    H. S. Vera, S. Dua, B. Zhang, D. Salz, R. Mullins, S. R. Panyamet al., “EmbeddingGemma: Powerful and Lightweight Text Representations,” CoRR, vol. abs/2509.20354, 2025, arXiv: 2509.20354

    Pith/arXiv arXiv 2025

  6. [6]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez et al., “Attention is All you Need,” inAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. v. Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanathanet ...

    2017

  7. [7]

    A Continuously Updated MtG Card Dataset,

    Scryfall, “A Continuously Updated MtG Card Dataset,” 2026

    2026

  8. [8]

    FiLM: Visual Reasoning with a General Conditioning Layer,

    E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. Courville, “FiLM: Visual Reasoning with a General Conditioning Layer,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018

    2018

  9. [9]

    RoFormer: En- hanced transformer with Rotary Position Embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “RoFormer: En- hanced transformer with Rotary Position Embedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  10. [10]

    Decoupled Weight Decay Regularization,

    I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” in7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019

    2019

  11. [11]

    JAX: composable transformations of Python+NumPy programs,

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, Y . Katariya, C. Learyet al., “JAX: composable transformations of Python+NumPy programs,” 2018. [Online]. Available: http://github.com/jax-ml/jax

    2018

  12. [12]

    Equinox: neural networks in JAX via callable PyTrees and filtered transformations,

    P. Kidger and C. Garcia, “Equinox: neural networks in JAX via callable PyTrees and filtered transformations,”Differentiable Program- ming workshop at Neural Information Processing Systems 2021, 2021

    2021