{"id":"9a530628-68bb-4b33-97d1-902fdfb9f0ef","arxiv_id":"2607.09965","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Form-level test assembly can be cast as stratified multi-armed bandits with Fisher-information rewards so new uncertain items are calibrated while blueprint constraints and precision targets are met.","lead":"This paper proposes a multi-armed-bandit method (SCH) that builds full linear test forms from item pools that keep receiving new AI-generated items with uncertain quality. It matters for large-scale digital testing programs that must refresh item banks continuously without expensive separate pilot studies.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Top-s truncation by multi-objective utility may systematically exclude high-information items that the bandit reward actually optimizes, so the reported ATI gains may not survive a larger candidate set.","rationale":"The Reader correctly isolated the top-s truncation plus cell independence as the weakest assumption. That assumption is load-bearing for the strongest claim: without it the arm space is intractable, yet the paper never shows that the multi-objective pre-filter preserves near-optimal information arms. The concrete ablation on s and on pure-information ranking would settle whether the reported ATI/CS/NIC numbers are robust or are artifacts of the pragmatic reduction. Because the rest of the simulation design is transparent and no internal contradiction appears, the appropriate verdict remains CONDITIONAL pending that check (and the real-pool validation already noted by the Reader). No stronger concern (e.g., an algebraic error in the delta-method variance or a contradiction with Proposition 1) was found.","tokens_in":8222,"tokens_out":658,"duration_ms":7171,"concrete_test":"Re-run the S1 and S5 simulations with s∈{8,12,16,20} and with an alternative ranking that uses only the information term of Eq. 2 (i.e., ignore λ2,λ3,λ4 when forming Qσ). If ATI for SCH-MAB drops by more than ~5% relative to ST-Linear, or if the CS/NIC advantages disappear once the candidate set is enlarged or re-ranked by pure information, the truncation step is load-bearing and the headline claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on SCH-MAB matching ST-Linear ATI (0.656 vs 0.642) while also improving CS under drift and NIC. That claim is only demonstrated after a pragmatic reduction (Section 4.1) that first ranks every cell by the multi-objective utility ui (Eq. 2) containing exposure, novelty and uncertainty terms, keeps only the top s=8 items, and then runs independent Thompson sampling whose reward is pure expected information R(a). Proposition 1 justifies cell-wise additivity of information, but supplies no guarantee that the truncated candidate set Qσ still contains the information-maximizing feasible subsets once the non-information terms in ui have reordered the ranking. Because λ4 and λ2 can demote high-a items that are already well-calibrated or heavily exposed, the bandit never sees those arms; the observed ATI advantage over LOFT-IW and the near-parity with unconstrained ST-Linear could therefore be an artifact of the particular (λ,s) pair rather than a property of the MAB formulation itself. The paper only asserts that s=8 is 'pragmatically chosen' and satisfies s≥2nσ; it reports no ablation on s or on pure-information ranking.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes the Stochastic Constrained Hybrid (SCH) framework for assembling linear test forms under blueprint constraints when item pools continuously receive AI-generated items with uncertain IRT parameters. Form assembly is recast as a multi-armed bandit problem whose reward is expected Fisher information; a multi-objective utility (Eq. 2) ranks items, a pragmatic top-s=8 truncation per stratum cell reduces the arm space to ≤70, and Thompson sampling (with a full three-parameter delta-method variance) selects forms. Two variants (SCH-MAB, SCH-EXP) are compared in a five-scenario simulation (N=1000, n=40, T=500, J=500, R=30) against SR, LOFT-SH, a new LOFT-IW baseline, and ST-Linear. The central claim is that SCH-MAB matches ST-Linear precision (ATI 0.656 vs 0.642) while improving constraint satisfaction under drift (CS 0.963 vs ~0.88) and accelerating new-item calibration (NIC 0.072 vs 0.039) without a separate pilot phase.","tokens_in":8608,"tokens_out":1151,"duration_ms":9391,"significance":"If the simulation results hold under more realistic pools and larger candidate sets, the work supplies a practical, principled route for operational programs that must field AI-generated items immediately. The additive decomposition (Proposition 1), the full delta-method variance that corrects an 88% under-estimate, the feasibility filter that removes learning bias, and the newly identified LOFT-IW intermediate regime are concrete technical contributions. The demonstration that pool concentration is tunable via λ4 rather than an algorithmic inevitability is also useful. The manuscript is transparent about free parameters and simulation design, which aids reproducibility.","major_comments":[{"comment":"Section 4.1 and Eq. (2): the central ATI claim rests on a pragmatic reduction that first ranks every cell by the multi-objective utility ui (containing exposure, novelty and uncertainty terms), retains only the top s=8 items, and then runs independent Thompson sampling whose reward is pure expected information R(a). Proposition 1 justifies cell-wise additivity of information but supplies no guarantee that the truncated set Qσ still contains the information-maximizing feasible subsets once the non-information terms have reordered the ranking. Because λ4 and λ2 can demote high-a items that are already well-calibrated or heavily exposed, the bandit never sees those arms; the reported near-parity with ST-Linear (Table 2: ATI 0.656 vs 0.642) and the advantage over LOFT-IW could therefore be an artifact of the particular (λ,s) pair. An ablation on s (or on pure-information ranking) is needed t","section":null},{"comment":"Section 5 and Table 2: all free parameters (utility weights, s=8, β, ρmax, novelty decay α, jump-start SE range) are fixed a priori and never varied except for a brief λ4 sensitivity note. The ranking of methods is therefore conditional on this single hyper-parameter point. At minimum the paper should report whether the Regime-3 separation and the NIC advantage remain when s is increased or when the utility weights are re-tuned under a pure-information objective.","section":null}],"minor_comments":[{"comment":"Abstract and Introduction: the phrase 'a sequential decision-making problem under uncertainty' is repeated almost verbatim; tighten for length.","section":null},{"comment":"Eq. (3): the partial derivatives ∂R/∂ai, ∂R/∂bi, ∂R/∂ci are left symbolic; a short appendix derivation or reference would help readers implement the full delta-method variance.","section":null},{"comment":"Table 2 caption: the dagger and double-dagger footnotes are useful but the 'Cached production cost ≈4 ms' claim for LOFT-IW is not explained in the text; clarify how caching is applied.","section":null},{"comment":"Section 6: IPS ≈ 0.93 is described as '≈120 items receive all usage'; a brief histogram or Lorenz curve in the supplement would make the concentration claim more concrete.","section":null},{"comment":"References: Sharpnack et al. (2026) is listed as 'In Proceedings of AIME-Con 2026'; confirm status or supply a preprint link so readers can access the SAC baseline.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an honest extension of the author's own BanditCAT/SAC line; the form-level construction, LOFT-IW baseline and full delta-method variance are genuine increments, not forced by the prior citations. The main risk is that the pragmatic top-s truncation is load-bearing for the ATI numbers; if the authors cannot supply even a modest ablation, the central claim remains under-supported for a methods journal. Scope is appropriate for stat.AP or a measurement journal that accepts simulation-heavy work."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean engineering extension of Sharpnack et al.’s item-level bandit CAT work to blueprint-constrained linear forms. It is not a new theory of bandits; it is a practical way to put jump-start items into operational linear assembly without a pilot phase, and the simulation shows three cleanly separated operating regimes.\n\nWhat is actually new is the form-level construction: additive TIF decomposition over stratum cells (Proposition 1 is correct), the top-s=8 candidate reduction that makes Thompson sampling feasible (~70 arms per cell), the full three-parameter delta-method variance (they correctly note that b-only underestimates by ~88%), the feasibility filter that removes the post-repair learning bias, and the new LOFT-IW baseline that sits in a previously undescribed intermediate regime (ATI 0.35, IPS 0.42). SCH-MAB matches ST-Linear on ATI while improving CS under drift and NIC. The λ4 sensitivity analysis is honest: concentration is tunable, not inevitable. Citation pattern is appropriate; the prior BanditCAT/SAC papers are the natural base, not circular padding.\n\nThe soft spot the stress-test flags is real but proportionate. Ranking by multi-objective ui then sampling on pure information can demote high-a items that are already well-calibrated or exposed, so the reported ATI near-parity with ST-Linear could partly depend on the particular (λ,s) pair. The paper only calls s=8 “pragmatic” and never ablates pure-information ranking or larger s. That is a genuine limitation of the evidence, not an internal contradiction. Everything else (simulation design, metrics, regime separation) is transparent and internally consistent. No code or real-pool data is shipped; that is the usual simulation-only caveat for this literature.\n\nThis is for people who build or evaluate automatic test assembly systems for large digital programs (Duolingo-scale, high-stakes CAT/LOFT shops). A serious referee should see it. I would bring it to reading group if we are talking operational ATA or bandit psychometrics; I would cite the LOFT-IW regime and the cell-reduction trick. Send it out.","headline":"Solid form-level extension of BanditCAT that works in simulation and introduces a useful LOFT-IW baseline; the top-s truncation is a real but fixable soft spot, not a collapse of the claim.","tokens_in":9286,"tokens_out":553,"would_cite":true,"duration_ms":4585,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Form assembly under AI item inflow can be cast as a multi-armed bandit that maximizes precision while calibrating uncertain new items without a pilot.","keywords":["test assembly","multi-armed bandit","Thompson sampling","item response theory","Fisher information","item calibration","exposure control","AI-generated items"],"falsifier":"Re-run the same simulation design but enlarge the per-cell candidate set far beyond s=8 (or remove the truncation) and check whether average test information, constraint satisfaction under drift, and new-item calibration rates remain essentially unchanged; a large drop would falsify the pragmatic reduction.","tokens_in":9005,"feed_emoji":"🎯","tokens_out":658,"duration_ms":6662,"temperature":0.7,"pith_summary":"Traditional test assembly treats form construction as a static optimization problem: pick items that meet a content blueprint and maximize measurement precision. In AI-enabled systems, however, new items continually enter the pool with poorly known psychometric parameters, and forms must be delivered on demand. The paper claims that this setting is better handled as a sequential decision problem under uncertainty. It introduces the Stochastic Constrained Hybrid (SCH) framework, which treats each blueprint-feasible form as a bandit arm whose reward is expected Fisher information. By decomposing the form into independent content-by-difficulty cells, restricting each cell to a short candidate list, and using full three-parameter variance estimates inside Thompson sampling, SCH can assemble linear forms that simultaneously maximize precision, satisfy constraints, control item exposure, and route uncertain new items to examinees for calibration. Simulation evidence is offered that SCH matches classical information-maximizing methods on precision while improving constraint satisfaction under parameter drift and accelerating new-item calibration.","feed_headline":"Bandits assemble tests that calibrate new AI items on the fly","feed_subtitle":"SCH turns form construction into a multi-armed bandit, matching precision while routing uncertain items without a pilot.","key_machinery":"The SCH framework: a multi-objective utility that ranks items by information, parameter uncertainty, novelty, and exposure; a stratified cell decomposition that reduces each cell's arm set to at most 70 candidate subsets; and Thompson sampling driven by a three-parameter delta-method variance approximation of expected test information.","core_discovery":"The paper establishes that form-level test assembly under continuous item inflow and parameter uncertainty can be solved as a multi-armed bandit whose reward is expected test information. By exploiting the additive decomposition of Fisher information across blueprint stratum cells, the intractable space of all feasible forms reduces to independent per-cell bandits of manageable size; Thompson sampling with a full delta-method variance then balances precision against active calibration of jump-start items, without a separate pilot phase.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Bandits turn form assembly into sequential decisions under item uncertainty","SCH reduces blueprint forms to per-cell bandits with Fisher-info rewards","Thompson sampling assembles tests while calibrating jump-start AI items","Form-level multi-armed bandits balance precision and pool sustainability","Additive Fisher info lets independent cell bandits handle evolving item pools"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That keeping only the top eight items per content-difficulty cell, then running independent Thompson sampling on information rewards, still approximately maximizes the global multi-objective objective that also includes exposure, novelty, and uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Bandits turn form assembly into sequential decisions under item uncertainty","SCH reduces blueprint forms to per-cell bandits with Fisher-info rewards","Thompson sampling assembles tests while calibrating jump-start AI items","Form-level multi-armed bandits balance precision and pool sustainability","Additive Fisher info lets independent cell bandits handle evolving item pools"]},"model":"grok-4.5","effort":"low","cost_usd":0.003202,"raw_usage":{"total_tokens":1103,"prompt_tokens":758,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":32020000,"prompt_tokens_details":{"text_tokens":758,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":253,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":758,"tokens_out":92,"duration_ms":2894,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T14:15:57.126747+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same simulation design but enlarge the per-cell candidate set far beyond s=8 (or remove the truncation) and check whether average test information, constraint satisfaction under drift, and new-item calibration rates remain essentially unchanged; a large drop would falsify the pragmatic reduction.","supporting_citations":[],"review_version":1}