{"id":"881a1fc4-bfa0-4455-b59d-a118006456c3","arxiv_id":"2607.20528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In-context batching of LLM ad-tagging with XML item delimiters and a client-side correction layer preserves downstream AUC at batch size 20 while cutting token cost by 89%.","lead":"PromptPack batches many ad titles into one LLM call—sharing an 8,500-token taxonomy prompt, using XML to separate items, and repairing output client-side. At batch size 20, it reports equal offline ranking accuracy to per-ad calls while cutting token cost by 89%, making LLM feature extraction affordable at ad scale.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The offline AUC-to-live RIG bridge is load-bearing and unsupported: batching changes tag distributions in ways LR AUC can miss.","rationale":"I agree with the reader's weakest assumption, and I think it is indeed the load-bearing one. I considered whether the cost-accounting assumptions in Eq. (2)-(4) (unstated per-item token counts and price ratios) or the BPE 500-to-10,000 rescaling are more fundamental. Those are checkable numerical issues: they could change the size of the cost saving, but even a corrected 80% saving would not sink the central engineering contribution. The quality-transfer assumption is different: if offline LR AUC does not track live RIG under batching, then the central claim \"fully preserving AUC\" is irrelevant to the production claim it is attached to. The paper deserves credit for the architecture, the cross-model sweep, and the self-consistent cost formula; the problem is not internal inconsistency but missing external validation of the one correlation that connects the offline benchmark to the headline. A live A/B comparing bs=1 vs bs=20 RIG, or even running the production scorer on the same offline titles, would settle it. I would keep the reader's CONDITIONAL verdict: the condition is precisely the production-scorer/RIG validation. I also note a minor internal inconsistency between Figure 3's \"Sequential bs=1, conc=1: 1269s\" and Table 3's 70.7s for gpt-4.1-nano; this could affect the 2.5x throughput claim if it is not a caption/labeling error, but it is secondary relative to the quality-transfer bridge.","tokens_in":15080,"tokens_out":8015,"duration_ms":79855,"concrete_test":"Have the authors run the exact bs=1 and bs=20 PromptPack outputs through the production CTR scorer, not the LR proxy, on a held-out sample, or run a live A/B on matched traffic that measures RIG lift over no-LLM features for both batch sizes. Concretely: compute RIG(bs=1) and RIG(bs=20) relative to the no-LLM feature set on the same production traffic; if RIG(bs=20) is inside the 95% CI of RIG(bs=1), the correlation bridge is valid. If the production scorer cannot be exposed, a preregistered offline replication using the production feature store and scorer on the same 10,000 titles would still settle the direction of the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing step is the transfer from the offline LR-AUC benchmark to the live production claim. Section 4 asserts that \"this offline AUC strictly correlates with the RIG observed in live production\" and Section 5 asserts \"solid offline->online generalization,\" but the only cited support is the self-cited prior deployment [13]; no fresh evidence in this paper shows that this correlation survives batching. Table 4's AUC is a 3-fold cross-validated AUC of a logistic-regression ranker on 10,000 balanced titles, while the abstract's baseline is a live CTR/RIG production system. The paper's own diagnostics show batching materially changes the tag distribution: Section 6.3 reports gpt-4.1-nano VWAL rising up to 43% while AUC stays flat, and claude-haiku-4.5 entering the L-AUC/L-VWAL quadrant at bs=20. If the LR ranker is saturated or insensitive to those changes, identical offline AUC does not imply identical live RIG. The headline \"fully preserving AUC\" would then be true, but would not justify the \"compared to live production baseline\" cost/quality claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PromptPack is an LLM annotation agent that batches multiple ad-creative titles into a single API call by combining a shared system prompt, an XML envelope, and a deterministic correction layer. The paper evaluates the agent offline on 10,000 production-derived, click-balanced titles with a downstream logistic-regression ranker, reporting ROC-AUC and a new diagnostic metric, Volume-Weighted Absolute Lift (VWAL), across four small/fast LLMs and batch sizes 1-20. The main claim is that, on the production model gpt-4.1-nano, batch size 20 preserves the unbatched baseline's AUC while cutting token costs by 89% and improving throughput by 2.5x. The authors also compare against BatchPrompt+BPE and a simulated BatchLLM baseline, and identify claude-haiku-4.5 and gemini-2.5-flash as promising higher-accuracy alternatives.","tokens_in":15357,"tokens_out":7383,"duration_ms":77337,"significance":"If the headline result holds, PromptPack is a practically valuable engineering contribution: it amortizes a large fixed taxonomy prompt (about 8,500 tokens, roughly 94% of billed input tokens at bs=1) without measurable degradation in offline retrieval AUC, and the method is API-only, model-agnostic, and does not require local inference stacks. The paper's main AUC claim is supported by a cross-validated logistic-regression benchmark with reported confidence intervals, and the multi-model panel plus the XML/no-XML and caching ablations are useful. The explicit cost model in Eq. (4) is a strength, and the authors are transparent about the N=500 BPE cells and the lack of prompt caching on the production model. However, the bridge from offline LR AUC to live production RIG is assumed rather than demonstrated, and the 89% cost figure is not reproducible from the parameters given. The paper therefore requires revision before the production claim can be accepted at face value.","major_comments":[{"comment":"The offline-to-online bridge is load-bearing and unsupported. Section 4 states that \"this offline AUC strictly correlates with the RIG observed in live production\" and Section 5 asserts \"solid offline->online generalization,\" with the only cited support being self-cited prior deployment [13]. The paper provides no fresh evidence that this correlation survives batching. The paper's own diagnostics show batching changes tag distributions: Section 6.3 reports gpt-4.1-nano VWAL rising up to 43% while AUC stays flat, and claude-haiku-4.5 entering the L-AUC/L-VWAL quadrant at bs=20. A logistic-regression AUC on a balanced 10k sample can be insensitive to exactly those tag-distribution changes. The abstract's \"fully preserving AUC\" is a true statement about the offline benchmark, but it does not by itself justify the production comparison. Please provide an online A/B result, a production-deriv","section":"Section 4 and Section 5"},{"comment":"The headline 89% cost saving is not reproducible from the reported parameters. Eq. (4) depends on S, T, O, p_ic, p_i, and p_o, but the text gives only S≈8,500. Even using the 94%-redundancy statement to infer T≈540, the output-token term p_o*O is unmeasured, and the 89% figure can be recovered only for a specific, unreported ratio of output to input token prices. Please report the measured average T and O per item, the applicable price parameters (or actual billed costs), and a sensitivity analysis of Δ(b,R) to output-token cost and batch size. Without this, the cost claim is an unverified parameterization.","section":"Section 6.2, Eq. (4)"},{"comment":"The BPE comparisons are run on an N=500 subset and then linearly rescaled to N=10,000. The Table 4 note discloses this, but the rescaling is not statistically valid for the way the numbers are used. VWAL is a sum over rows, so linearly rescaling it assumes the 500-item subset is perfectly representative; \"adjusted\" AUC confidence intervals for N=500 do not make those cells comparable to the full-sample baselines in the same table. These BPE cells are used in Section 6.1 to claim a statistically significant BPE peak for gpt-4.1-nano (0.654 vs. 0.611) and to inform deployment recommendations. Please either evaluate BPE at the full N, or clearly separate and label pilot-only cells and refrain from combining them with full-sample numbers in the main benchmark table.","section":"Section 5 and Table 4"},{"comment":"The quadrant analysis anchors relative AUC and VWAL at \"BatchLLM bs=1,\" but BatchLLM bs=1 is defined in Section 5(d) as direct_concurrency = 16 concurrent requests, not as the unbatched single-call baseline described in Section 4. This conflation affects all relative values and H/L labels in Figure 3 and Section 6.3. If the intended anchor is the production unbatched baseline, use PromptPack bs=1 at concurrency=1; if BatchLLM bs=1 with 16-way concurrency is the anchor, relabel it and adjust the interpretation of the quadrants.","section":"Section 4, Figure 3, Section 6.3"}],"minor_comments":[{"comment":"The sentence \"Tag becomes none;0.2\" appears to be a formatting artifact; please clarify the exact fallback rule for tags below the confidence floor.","section":"Section 3.3"},{"comment":"The metric name is inconsistently typeset as \"VW AL\" in Table 2 and elsewhere; unify to VWAL.","section":"Table 2 and Section 4"},{"comment":"The evaluation uses a single randomized 3-fold stratified CV. With 10,000 samples, 3-fold CIs are coarse; consider repeated or 5-fold CV to tighten the comparison of near-identical AUC values.","section":"Section 5"},{"comment":"The section discusses asynchronous batch API discounts and prompt caching constraints, but the relationship between those discounts and the 89% figure is not made explicit. A short worked example with assumed price parameters would help.","section":"Section 6.2"},{"comment":"Model names are inconsistent between the text (\"claude-haiku-4.5\") and figure labels (\"claude-haiku-4-5\").","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry short paper whose central offline AUC result is plausible and fairly presented, but the abstract makes a production claim that goes beyond the evidence in the manuscript. The offline-to-online RIG bridge and the unverified cost parameterization are the key blockers. Both are fixable either by adding validation data or by narrowing the claims to what the offline benchmark supports, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a decent industrial paper, not a scientific breakthrough. The real contribution is the evaluation: four commercial LLMs, 10k titles, cross-validated AUC, and a batching recipe that preserves offline retrieval quality at batch size 20 while cutting cost by ~89% on their production model. That is genuinely useful data for anyone running LLM feature extraction. The VWAL diagnostic is a small but sensible addition.\n\nWhat the paper does well: the ablation design is thoughtful—XML vs. no XML, BatchLLM simulation, and BPE with K=3/5. The finding that XML matters most for the small model under ensembling is concrete. The cost model is explicit even if parameters are hidden. The authors are clear about what they measured offline.\n\nWhere it gets soft: the bridge from offline AUC to live RIG is load-bearing and unsupported. Section 4 says the correlation was established in a prior self-cited deployment; no fresh evidence shows it survives batching. The stress-test is right: batching changes the tag distribution (VWAL rises up to 43% on gpt-4.1-nano while AUC stays flat; claude-haiku-4.5 drops into the L-AUC/L-VWAL quadrant at bs=20). A linear ranker can be saturated, so identical offline AUC does not imply identical live retrieval quality. The abstract's wording—'compared to our live, unbatched production baseline … fully preserving AUC'—runs ahead of what offline evaluation shows. Section 3 even concedes the setting is 'aligned with deployed production environment, yet not exact mirror due to required disclosure policies'; that caveat should be in the abstract.\n\nAlso: BPE cells are evaluated on 500 items and rescaled to 10k; the 89% cost saving depends on unstated token/price parameters; VWAL has no uncertainty interval; 'identical AUC' is really 'within CI.' None of this kills the result, but it makes the quantitative headline less firm than it looks.\n\nThis paper is for practitioners building annotation pipelines; a theorist won't get much. The authors are thinking clearly and cite prior work honestly. I'd send it to peer review—the method is plausible, the data is real, and the weaknesses are addressable in revision. I would not cite it in my own work without first verifying the cost model and the live bridge.\n\nRecommendation: engage with it, ask for the missing parameters and a live test or a clear downgrade of the live claim.","headline":"Credible offline evidence that batching works; the live-quality claim rests on a self-cited correlation the paper does not re-establish under batching.","tokens_in":15861,"tokens_out":7624,"would_cite":false,"duration_ms":70528,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PromptPack shows that packing twenty ad creatives into one LLM prompt preserves retrieval quality while cutting token costs by 89%.","keywords":["LLM annotation","in-context batching","ad creative feature extraction","cost reduction","CTR prediction","XML prompting","retrieval quality","prompt engineering"],"falsifier":"Run the same batching recipe in the live production scorer with a randomized A/B test: if PromptPack at bs=20 shows a statistically significant drop in RIG or CTR relative to the single-call baseline, the central claim is refuted. A cheaper offline check: measure per-item tag distributions at bs=20 vs bs=1 and test whether any tag shifts are correlated with click label in a way AUC is insensitive to.","tokens_in":14969,"feed_emoji":"📦","tokens_out":3263,"duration_ms":33364,"temperature":0.7,"pith_summary":"PromptPack is an annotation-agent architecture for online advertising that processes multiple ad creatives in a single LLM call. The paper argues that the three-part design—a shared system prompt evaluated once per batch, a strict XML envelope separating items, and a deterministic correction layer—eliminates the cost bottleneck that made per-creative LLM feature extraction too expensive. On the production model, batch size 20 matches the single-call baseline's AUC (0.609 vs 0.608) while cutting token costs by 89% and running 2.5x faster. The same quality preservation holds across three other small/fast models within small margins. If correct, the result lets ad platforms roll out LLM-derived features at scale without surrendering ranking quality.","feed_headline":"Batching 20 ads per call cuts LLM costs 89%, same AUC","feed_subtitle":"PromptPack packs multiple ad creatives into one LLM request without loss of retrieval quality across four small/fast models.","key_machinery":"The load-bearing object is the XML structural envelope: a <batch> containing <item id=\"N\"> blocks with escaped title text, combined with a batch-mode addendum in the system prompt instructing the model to copy row ids into each JSON record. It enforces item boundaries, prevents semantic cross-talk, lets out-of-order or partial responses be re-joined, and—for a small model like gpt-4.1-nano—is necessary to keep BPE ensembling from collapsing AUC. The shared system prompt (≈8,500 tokens) is what makes batching cost-effective, and the correction layer (JSON repair, tag validation, row-id alignment, bounded retries) makes output deterministic.","core_discovery":"The central claim is that markup-guided in-context batching can amortize a large shared taxonomy prompt without causing context bleeding between items. The paper demonstrates this by batching 20 creatives per request and showing downstream retrieval AUC is statistically indistinguishable from the unbatched baseline on gpt-4.1-nano, while token cost falls 89% and wall-clock time per 1,000 creatives drops from 70.7s to 28.2s. The design pairs a one-time system prompt with an XML <item> envelope and row-id alignment, plus a correction layer that repairs JSON and re-issues failed items; together they preserve per-item feature fidelity. The paper also introduces VWAL, a diagnostic that weights pe","pith_inferences":["The cost savings generalize beyond ad creatives: any pipeline that sends a large static prompt with many independent items (e.g., document classification, review moderation) could adopt the same XML-envelope recipe, provided item isolation is testable.","The paper's offline-to-online bridge rests on a single self-cited prior deployment; a live A/B test of PromptPack at bs=20 against the single-call agent would be the direct confirmatory experiment and would also reveal whether batching changes tag distribution in ways that affect CTR beyond AUC.","VWAL's fractional-attribution scheme assumes clicks and views distribute uniformly across tags in a cell; at high batch sizes with many co-occurring tags, this could under- or over-state a lone decisive tag, so VWAL should be read as diagnostic, not causal."],"forward_implications":["If valid, LLM-driven feature enrichment can be scaled to full inventory without the cost gate; a platform that could only afford a fraction of creatives can now annotate all of them at the same budget.","The 89% reduction is not an upper bound: the cost equation shows further savings by increasing batch size or using cache-enabled models, with additional 52–57% off already-cached baselines at bs=20.","The AUC×VWAL quadrant view gives operators a concrete way to tell whether a batching change degrades genuine signal or just shifts tag volume, informing when a non-linear ranker is worth adopting.","The finding that XML is essential for small-model stability under ensembling but not for larger models gives a practical rule for when structural envelopes matter."],"fun_headline_variants":["20 ads per LLM call: 89% cheaper, same AUC","Batch 20 ads, cut LLM costs 89%, keep AUC","PromptPack: 89% cost cut, 2.5x speed, no AUC loss","One call, 20 ads: LLM cost down 89%, quality intact","LLM annotation on autopilot: 89% off, AUC unchanged"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The bridge between the offline benchmark and the production claim is that AUC on a balanced 10,000-title sample strictly tracks live production RIG; the paper cites a prior deployment for this correlation but provides no data in this paper, so a batch-induced change that preserves AUC may not preserve live CTR.","fun_headline_variants_meta":{"raw":{"variants":["20 ads per LLM call: 89% cheaper, same AUC","Batch 20 ads, cut LLM costs 89%, keep AUC","PromptPack: 89% cost cut, 2.5x speed, no AUC loss","One call, 20 ads: LLM cost down 89%, quality intact","LLM annotation on autopilot: 89% off, AUC unchanged"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1205,"prompt_tokens":749,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":493,"tokens_out":456,"duration_ms":4439,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:40:14.861338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same batching recipe in the live production scorer with a randomized A/B test: if PromptPack at bs=20 shows a statistically significant drop in RIG or CTR relative to the single-call baseline, the central claim is refuted. A cheaper offline check: measure per-item tag distributions at bs=20 vs bs=1 and test whether any tag shifts are correlated with click label in a way AUC is insensitive to.","supporting_citations":[],"review_version":1}