Pith. sign in

REVIEW 4 major objections 5 minor 15 references

PromptPack shows that packing twenty ad creatives into one LLM prompt preserves retrieval quality while cutting token costs by 89%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:40 UTC pith:XN2HMPDG

load-bearing objection Credible offline evidence that batching works; the live-quality claim rests on a self-cited correlation the paper does not re-establish under batching. the 4 major comments →

arxiv 2607.20528 v1 pith:XN2HMPDG submitted 2026-07-10 cs.AI

PromptPack: Scaling LLM Annotation Agents for Online Recommendation

Sebastian Koralewski , Merwan Barlier , Yulia Stolin , Bla\v{z} \v{S}krlj This is my paper
classification cs.AI
keywords LLM annotationin-context batchingad creative feature extractioncost reductionCTR predictionXML promptingretrieval qualityprompt engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PromptPack is an annotation-agent architecture for online advertising that processes multiple ad creatives in a single LLM call. The paper argues that the three-part design—a shared system prompt evaluated once per batch, a strict XML envelope separating items, and a deterministic correction layer—eliminates the cost bottleneck that made per-creative LLM feature extraction too expensive. On the production model, batch size 20 matches the single-call baseline's AUC (0.609 vs 0.608) while cutting token costs by 89% and running 2.5x faster. The same quality preservation holds across three other small/fast models within small margins. If correct, the result lets ad platforms roll out LLM-derived features at scale without surrendering ranking quality.

Core claim

The central claim is that markup-guided in-context batching can amortize a large shared taxonomy prompt without causing context bleeding between items. The paper demonstrates this by batching 20 creatives per request and showing downstream retrieval AUC is statistically indistinguishable from the unbatched baseline on gpt-4.1-nano, while token cost falls 89% and wall-clock time per 1,000 creatives drops from 70.7s to 28.2s. The design pairs a one-time system prompt with an XML <item> envelope and row-id alignment, plus a correction layer that repairs JSON and re-issues failed items; together they preserve per-item feature fidelity. The paper also introduces VWAL, a diagnostic that weights pe

What carries the argument

The load-bearing object is the XML structural envelope: a <batch> containing <item id="N"> blocks with escaped title text, combined with a batch-mode addendum in the system prompt instructing the model to copy row ids into each JSON record. It enforces item boundaries, prevents semantic cross-talk, lets out-of-order or partial responses be re-joined, and—for a small model like gpt-4.1-nano—is necessary to keep BPE ensembling from collapsing AUC. The shared system prompt (≈8,500 tokens) is what makes batching cost-effective, and the correction layer (JSON repair, tag validation, row-id alignment, bounded retries) makes output deterministic.

Load-bearing premise

The bridge between the offline benchmark and the production claim is that AUC on a balanced 10,000-title sample strictly tracks live production RIG; the paper cites a prior deployment for this correlation but provides no data in this paper, so a batch-induced change that preserves AUC may not preserve live CTR.

What would settle it

Run the same batching recipe in the live production scorer with a randomized A/B test: if PromptPack at bs=20 shows a statistically significant drop in RIG or CTR relative to the single-call baseline, the central claim is refuted. A cheaper offline check: measure per-item tag distributions at bs=20 vs bs=1 and test whether any tag shifts are correlated with click label in a way AUC is insensitive to.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If valid, LLM-driven feature enrichment can be scaled to full inventory without the cost gate; a platform that could only afford a fraction of creatives can now annotate all of them at the same budget.
  • The 89% reduction is not an upper bound: the cost equation shows further savings by increasing batch size or using cache-enabled models, with additional 52–57% off already-cached baselines at bs=20.
  • The AUC×VWAL quadrant view gives operators a concrete way to tell whether a batching change degrades genuine signal or just shifts tag volume, informing when a non-linear ranker is worth adopting.
  • The finding that XML is essential for small-model stability under ensembling but not for larger models gives a practical rule for when structural envelopes matter.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The cost savings generalize beyond ad creatives: any pipeline that sends a large static prompt with many independent items (e.g., document classification, review moderation) could adopt the same XML-envelope recipe, provided item isolation is testable.
  • The paper's offline-to-online bridge rests on a single self-cited prior deployment; a live A/B test of PromptPack at bs=20 against the single-call agent would be the direct confirmatory experiment and would also reveal whether batching changes tag distribution in ways that affect CTR beyond AUC.
  • VWAL's fractional-attribution scheme assumes clicks and views distribute uniformly across tags in a cell; at high batch sizes with many co-occurring tags, this could under- or over-state a lone decisive tag, so VWAL should be read as diagnostic, not causal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. PromptPack is an LLM annotation agent that batches multiple ad-creative titles into a single API call by combining a shared system prompt, an XML envelope, and a deterministic correction layer. The paper evaluates the agent offline on 10,000 production-derived, click-balanced titles with a downstream logistic-regression ranker, reporting ROC-AUC and a new diagnostic metric, Volume-Weighted Absolute Lift (VWAL), across four small/fast LLMs and batch sizes 1-20. The main claim is that, on the production model gpt-4.1-nano, batch size 20 preserves the unbatched baseline's AUC while cutting token costs by 89% and improving throughput by 2.5x. The authors also compare against BatchPrompt+BPE and a simulated BatchLLM baseline, and identify claude-haiku-4.5 and gemini-2.5-flash as promising higher-accuracy alternatives.

Significance. If the headline result holds, PromptPack is a practically valuable engineering contribution: it amortizes a large fixed taxonomy prompt (about 8,500 tokens, roughly 94% of billed input tokens at bs=1) without measurable degradation in offline retrieval AUC, and the method is API-only, model-agnostic, and does not require local inference stacks. The paper's main AUC claim is supported by a cross-validated logistic-regression benchmark with reported confidence intervals, and the multi-model panel plus the XML/no-XML and caching ablations are useful. The explicit cost model in Eq. (4) is a strength, and the authors are transparent about the N=500 BPE cells and the lack of prompt caching on the production model. However, the bridge from offline LR AUC to live production RIG is assumed rather than demonstrated, and the 89% cost figure is not reproducible from the parameters given. The paper therefore requires revision before the production claim can be accepted at face value.

major comments (4)
  1. [Section 4 and Section 5] The offline-to-online bridge is load-bearing and unsupported. Section 4 states that "this offline AUC strictly correlates with the RIG observed in live production" and Section 5 asserts "solid offline->online generalization," with the only cited support being self-cited prior deployment [13]. The paper provides no fresh evidence that this correlation survives batching. The paper's own diagnostics show batching changes tag distributions: Section 6.3 reports gpt-4.1-nano VWAL rising up to 43% while AUC stays flat, and claude-haiku-4.5 entering the L-AUC/L-VWAL quadrant at bs=20. A logistic-regression AUC on a balanced 10k sample can be insensitive to exactly those tag-distribution changes. The abstract's "fully preserving AUC" is a true statement about the offline benchmark, but it does not by itself justify the production comparison. Please provide an online A/B result, a production-deriv
  2. [Section 6.2, Eq. (4)] The headline 89% cost saving is not reproducible from the reported parameters. Eq. (4) depends on S, T, O, p_ic, p_i, and p_o, but the text gives only S≈8,500. Even using the 94%-redundancy statement to infer T≈540, the output-token term p_o*O is unmeasured, and the 89% figure can be recovered only for a specific, unreported ratio of output to input token prices. Please report the measured average T and O per item, the applicable price parameters (or actual billed costs), and a sensitivity analysis of Δ(b,R) to output-token cost and batch size. Without this, the cost claim is an unverified parameterization.
  3. [Section 5 and Table 4] The BPE comparisons are run on an N=500 subset and then linearly rescaled to N=10,000. The Table 4 note discloses this, but the rescaling is not statistically valid for the way the numbers are used. VWAL is a sum over rows, so linearly rescaling it assumes the 500-item subset is perfectly representative; "adjusted" AUC confidence intervals for N=500 do not make those cells comparable to the full-sample baselines in the same table. These BPE cells are used in Section 6.1 to claim a statistically significant BPE peak for gpt-4.1-nano (0.654 vs. 0.611) and to inform deployment recommendations. Please either evaluate BPE at the full N, or clearly separate and label pilot-only cells and refrain from combining them with full-sample numbers in the main benchmark table.
  4. [Section 4, Figure 3, Section 6.3] The quadrant analysis anchors relative AUC and VWAL at "BatchLLM bs=1," but BatchLLM bs=1 is defined in Section 5(d) as direct_concurrency = 16 concurrent requests, not as the unbatched single-call baseline described in Section 4. This conflation affects all relative values and H/L labels in Figure 3 and Section 6.3. If the intended anchor is the production unbatched baseline, use PromptPack bs=1 at concurrency=1; if BatchLLM bs=1 with 16-way concurrency is the anchor, relabel it and adjust the interpretation of the quadrants.
minor comments (5)
  1. [Section 3.3] The sentence "Tag becomes none;0.2" appears to be a formatting artifact; please clarify the exact fallback rule for tags below the confidence floor.
  2. [Table 2 and Section 4] The metric name is inconsistently typeset as "VW AL" in Table 2 and elsewhere; unify to VWAL.
  3. [Section 5] The evaluation uses a single randomized 3-fold stratified CV. With 10,000 samples, 3-fold CIs are coarse; consider repeated or 5-fold CV to tighten the comparison of near-identical AUC values.
  4. [Section 6.2] The section discusses asynchronous batch API discounts and prompt caching constraints, but the relationship between those discounts and the 89% figure is not made explicit. A short worked example with assumed price parameters would help.
  5. [Figure 3] Model names are inconsistent between the text ("claude-haiku-4.5") and figure labels ("claude-haiku-4-5").

Circularity Check

0 steps flagged

No circular derivation; the central AUC-preservation result is an empirical measurement. The only self-citation is the offline-to-live RIG bridge from prior deployment [13], which is not definitionally tied to the current experiment.

full rationale

The paper's main claim is an empirical measurement: PromptPack (XML) at bs=20 maintains downstream AUC 0.609 vs 0.608 at bs=1 (Table 4), while cost and throughput are computed from an explicit token-cost model (Eqs. 2-4). No fitted parameter is renamed as a prediction, and no equation reduces to the paper's own inputs. VWAL is introduced as a defined diagnostic, not as a derived result, and the AUC measurement is anchored directly to the unbatched bs=1 baseline. The one self-citation relevant to the live framing is Section 4's statement that 'this offline AUC strictly correlates with the RIG observed in live production,' attributed to the previous single-call deployment [13]. This is a self-citation (co-author Škrlj) and it supports the abstract's live-production comparison, but it is a prior external deployment result, falsifiable from production logs, not a theorem or fitted value. The core AUC-preservation comparison is measured in this paper against its own bs=1 anchor, so it stands independently. Concerns that batching changes tag distributions in ways LR AUC may miss (Section 6.3: VWAL rises up to 43% while AUC stays flat; claude-haiku-4.5 enters L-AUC/L-VWAL at bs=20) are correctness risks for the offline-to-live bridge, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central claim rests on the offline benchmark faithfully representing production, the cost model using unstated token/price parameters, and several hand-set thresholds (C=10, tag cap 6, confidence floor 0.10, retry cap 5%) that are not sensitivity-analyzed.

free parameters (4)
  • VWAL smoothing prior C = 10
    Hand-selected in Section 4 to balance bias-variance; no search or sensitivity analysis shown.
  • Per-feature tag cap = 6
    Wire-format constant (Section 3) that limits output sparsity; affects downstream one-hot features.
  • Confidence floor = 0.10
    Tags below 0.10 confidence are dropped (Section 3), shaping feature volume.
  • Retry cap = 5%
    Correction layer re-issues at most 5% of items (Section 3.3); if drop rates differ between methods, AUC comparison could be biased.
axioms (5)
  • domain assumption Offline AUC on a balanced 10k-sample with a logistic-regression ranker strictly correlates with live production RIG.
    Asserted via self-cited [13]; this is the bridge from offline benchmark to production claim.
  • domain assumption The 10,000-title balanced sample is representative of the production creative distribution.
    Section 5 states alignment with production, but titles are unique by content hash and balanced on click label, which may not match live imbalance.
  • domain assumption The cost model (Eq. 2-4) with S≈8500 and unspecified per-item tokens T/O and list prices describes actual API billing.
    89% cost saving is computed, not measured; T, O, and prices are not reported.
  • domain assumption API outputs at temperature 0 are sufficiently deterministic across batch sizes for the correction layer to guarantee reproducible feature rows.
    The paper itself notes closed APIs exhibit slight variation under concurrency; central claim assumes this variation does not bias AUC.
  • domain assumption XML item boundaries prevent semantic cross-talk between batched creatives.
    Section 3.2 argues this; it is supported only indirectly by the observed AUC preservation, not verified directly.
invented entities (1)
  • Volume-Weighted Absolute Lift (VWAL) no independent evidence
    purpose: Diagnostic metric to interpret AUC changes in terms of per-tag signal mass.
    VWAL is defined in the paper with a hand-set smoothing prior C=10 and no external validation or falsifiable prediction; it is a post-hoc interpretation tool.

pith-pipeline@v1.3.0-alltime-deepseek · 14801 in / 16864 out tokens · 153742 ms · 2026-08-02T07:40:14.861338+00:00 · methodology

0 comments
read the original abstract

Online recommendation platforms increasingly use Large Language Models (LLMs) to extract structured features from ad creatives. While deploying a single-call LLM annotation agent yields significant Click-Through Rate (CTR) improvements in our live production environment, per-creative prompting is prohibitively expensive to scale. The redundant system instructions sent in every request account for 94% of billed input tokens. To break this cost bottleneck, we introduce PromptPack, a scalable, high-throughput LLM annotation agent. PromptPack achieves this scale via in-context batching, combining a shared system prompt, a strict XML structural envelope, and an output correction layer to ensure deterministic, pipeline-ready feature extraction across multiple creatives simultaneously. We evaluate PromptPack via an offline retrieval benchmark using a downstream logistic-regression ranker. To deeply profile the agent's behavior, we measure AUC and introduce Volume-Weighted Absolute Lift (VWAL), a novel metric capturing the signal quality of the generated features. Compared to our live, unbatched production baseline, PromptPack at batch size 20 cuts our LLM costs by 89% and accelerates throughput by 2.5x while fully preserving AUC.

Figures

Figures reproduced from arXiv: 2607.20528 by Bla\v{z} \v{S}krlj, Merwan Barlier, Sebastian Koralewski, Yulia Stolin.

Figure 2
Figure 2. Figure 2: The three layers of a PromptPack call: a shared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: PromptPack annotation agent inside the ad [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-model results across the full (method, batch-size) sweep. Rows are the four LLMs; columns are downstream [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

  1. [1]

    Faruk Alpay and Taylan Alpay. 2025. XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocols.arXiv preprint arXiv:2509.08182(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Anthropic. 2024. Prompt Engineering Interactive Tutorial: Use XML Tags. https://docs.anthropic.com/en/docs/build-with-claude/prompt- engineering/use-xml-tags. Accessed: 2026-06-21

    2024

  3. [3]

    Andrei Z. Broder. 2008. Computational advertising and recommender systems. InProceedings of the 2008 ACM Conference on Recommender Systems(Lausanne, Switzerland)(RecSys ’08). Association for Computing Machinery, New York, NY, USA, 1–2. doi:10.1145/1454008.1454009

    arXiv 2008

  4. [4]

    Zaharia, and James Y

    Lingjiao Chen, Matei A. Zaharia, and James Y. Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv preprint arXiv:2305.05176(2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch Prompting: Efficient Inference with Large Language Model APIs. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, Mingx- uan Wang and Imed Zitouni (Eds.). Association for Computational Linguistics, Singapore, 792–810. doi:10.18653/v1/2023.emnlp-industry.74

    doi:10.18653/v1/2023.emnlp-industry.74 2023

  6. [6]

    Kushal Dave and Vasudeva Varma. 2014. Computational Advertising: Techniques for Targeting Relevant Ads.Foundations and Trends in Information Retrieval8, 4-5 (2014), 263–418. doi:10.1561/1500000045

    work page doi:10.1561/1500000045 2014

  7. [7]

    Zhaoxuan Ji, Xinlu Wang, Zhaojing Luo, Zhongle Xie, and Meihui Zhang. 2025. Optimized Batch Prompting for Cost-effective LLMs.Proceedings of the VLDB Endowment18, 7 (2025), 2172–2184. doi:10.14778/3734839.3734853

    arXiv 2025

  8. [8]

    Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. 2024. Batch- Prompt: Accomplish more with less. InInternational Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 21590–21612. https://proceedings.iclr.cc/paper_files/paper/ 2024/file/5d8c01de2dc698c54201c1c7d0b86974-Pa...

    2024

  9. [9]

    Jiayi Liu, Tinghan Yang, and Jennifer Neville. 2024. CliqueParcel: An Approach for Batching LLM Prompts that Jointly Optimizes Efficiency and Faithfulness. arXiv preprint arXiv:2402.14833(2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Tongtong Liu, Zhaohui Wang, Meiyue Qin, Zenghui Lu, Xudong Chen, Yuekui Yang, and Peng Shu. 2025. Real-time Ad Retrieval via LLM-generative Com- mercial Intention for Sponsored Search Advertising. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Chris- tos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and V...

    work page doi:10.18653/v1/2025.emnlp-main.1473 2025

  11. [11]

    Daniele Micci-Barreca. 2001. A Preprocessing Scheme for High-Cardinality Categorical Attributes in Classification and Prediction Problems.ACM SIGKDD Explorations Newsletter3, 1 (2001), 27–32. doi:10.1145/507533.507538

    arXiv 2001

  12. [12]

    2006.Credit Risk Scorecards: Developing and Implementing Intelli- gent Credit Scoring

    Naeem Siddiqi. 2006.Credit Risk Scorecards: Developing and Implementing Intelli- gent Credit Scoring. John Wiley & Sons. doi:10.1002/9781119201731

    work page doi:10.1002/9781119201731 2006

  13. [13]

    Blaž Škrlj, Benoît Guilleminot, and Andraž Tori. 2026. Agent0: Leveraging LLM Agents to Discover Multi-value Features from Text for Enhanced Recommenda- tions. InAI Agent for Information Retrieval: Generating and Ranking, Qingsong Wen, Yongfeng Zhang, Zhiwei Liu, Julian McAuley, Hua Wei, Linsey Pang, Wei Liu, and Philip S. Yu (Eds.). Springer Nature Switz...

    2026

  14. [14]

    Ruiming Tang, Chenxu Zhu, Bo Chen, Weipeng Zhang, Menghui Zhu, Xinyi Dai, and Huifeng Guo. 2025. LLM4Tag: Automatic Tagging System for Information Retrieval via Large Language Models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2(Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, ...

    arXiv 2025

  15. [15]

    Zhou, Chuanjie Liu, and Gang Peng

    Zhen Zheng, Xin Ji, Taosong Fang, F. Zhou, Chuanjie Liu, and Gang Peng. 2026. BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching. InProceedings of Machine Learning and Systems

    2026