REVIEW 3 major objections 6 minor 2 references
Alternates, Assemble! Selecting Optimal Alternates for Citizens' Assemblies
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Selecting alternates for citizens' assemblies reduces to PAC learning, and polynomial dropout samples suffice for near-optimal representation.
desk verdict A genuinely new problem and a clean sample-complexity result, but the advertised robustness guarantee for linear loss is proved only under a relaxation the paper itself concedes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the ERM-Alts reduction together with its dimension bounds. Formally, the hypothesis class for an instance is all $a$-subsets of the pool; a hypothesis scores a dropout set by the best achievable quota deviation after replacement. Bounding the VC-dimension of the binary class and the pseudodimension of the linear class by $\Theta(a\log n)$—where pseudodimension is the real-valued analogue of VC-dimension, measuring how many points a class can separate with thresholds—turns the exponential $2^k$ evaluation problem into a polynomial-size sampling problem, and keeps the integer program that selects alternates small enough for off-the-shelf solvers. The stability proof is carried by a telescoping and coupling argument that moves from true to estimated dropout probabilities one panelist at a time, showing that each panelist's mis-estimation contributes at most $\gamma$ times the number of feature-values to the linear loss, which sums to the $2\gamma|\mathit{FV}|$ bound.
What would settle it
Measure the joint dropout distribution in a real or simulated assembly under a shared external shock and compute its total-variation distance to the product distribution formed by each panelist's marginal dropout rate; if that distance is material, the sample-complexity and stability guarantees of Corollaries 3.6-3.7 and Theorem 4.2 no longer bound realized loss.
Extended reading notes
Core claim
The paper's discovery is a reduction: alternate selection is agnostic PAC learning. Each alternate set $A$ is a hypothesis $h_A$ that maps a dropout set $D\subseteq K$ to the smallest deviation achievable by replacing the dropped panelists with at most $|D|$ members of $A$; the risk is the expectation of $h_A(D)$ when $D$ is drawn from the product distribution induced by independent biased coins with dropout probabilities $\rho_i$. The hypothesis classes for binary and linear deviation have VC-dimension and pseudodimension $\Theta(a\log n)$, respectively, so known uniform-convergence bounds give Corollary 3.7: with $s=O((a\log n+\log(1/\varepsilon)+\log(1/\delta))/\varepsilon^2)$ samples, ERM-Alts returns an alternate set whose expected linear loss is within $\varepsilon$ of the optimal alternate set with probability at least $1-\delta$. The companion stability claim is that if the dropout probabilities are mis-estimated by at most $\gamma$, the linear-loss variant's expected loss increases by at most $2\gamma|\mathit{FV}|+\varepsilon$, while the binary variant can suffer near-maximal loss increase for arbitrarily small $\gamma$.
Load-bearing premise
The load-bearing premise is that each panelist's dropout event is independent of every other panelist's, because the sampling procedure and all guarantees are built on drawing dropout sets by independent biased coins.
Editorial extensions
If this is right
- Practitioners can solve alternate selection with an off-the-shelf ILP over a few hundred sampled dropout sets, instead of enumerating all $2^k$ possible dropout patterns.
- Imperfect dropout predictions are usable: with the linear-loss variant, an estimation error of $\gamma$ adds at most $2\gamma|\mathit{FV}|+\varepsilon$ to the expected quota deviation, so organizers can budget for prediction error.
- The binary-loss variant is unsafe when dropout probabilities are estimated rather than known, because arbitrarily small estimation errors can make the chosen alternates nearly useless.
- On real assemblies, ERM-Alts with linear loss matches the loss of the unattainable 'choose from the whole pool after seeing dropouts' benchmark while using only about half the panel size in alternates, and beats quota-based and greedy heuristics.
- The same ERM construction extends, with the same $O(a\log n)$ sample complexity, to alternates that themselves drop out, to selecting extra panelists, and to jointly choosing the panel and its alternates.
Reading between the lines
- An implication the authors leave implicit is that the independence assumption in Section 2 is load-bearing: correlated dropouts (a transit strike, a weather event, or social ties among invitees) move the real distribution away from the product distribution, and the $O(a\log n)$ guarantee no longer applies to realized loss.
- The stability theorem suggests a calibration rule not stated in the paper: before trusting dropout-probability estimates, compare ERM-Alts with linear loss against the equal-probabilities variant on held-out assemblies; the empirical crossing point around $\gamma=0.4$–$0.6$ indicates when predictions stop helping.
- A testable extension is to simulate correlated dropout sets with a shared shock and measure the gap between ERM-Alts's realized linear loss and the independence-based bound; the gap should track the total-variation distance between the true joint distribution and the product of its marginals.
- The paper's deterministic alternate selection creates a trade-off with sortition's randomization ideals; randomizing over near-optimal alternate sets could restore equal selection chances, at a quantifiable cost in expected representation that the paper does not compute.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ERM-Alts formalizes the problem of preselecting a set of a alternates for a quota-constrained citizens' assembly panel so as to minimize the expected post-dropout deviation from the quotas, and proposes an empirical-risk-minimization algorithm that draws dropout sets by independent biased coins and solves an ILP over the empirical support. Modeling panelists as dropping out independently with estimated probabilities, the paper reduces alternate selection to agnostic PAC learning, proves that the binary- and linear-deviation hypothesis classes have VC/pseudo-dimension Θ(a log n) (Theorem 3.4), and derives sample-complexity bounds of O(a log n/ε²) for ERM-Alts0/1 and ERM-Altsℓ1 (Corollaries 3.6-3.7). Section 4 analyzes robustness to dropout-probability estimation error γ: Theorem 4.1 shows binary loss is arbitrarily non-robust, while Theorems 4.2-4.3 give linear-in-γ bounds for the linear loss—but for a relaxed loss that drops the constraint |R|≤|D| (footnote 3). Section 5 evaluates the pipeline on US and Canadian assembly data against quota-based and greedy benchmarks and against real practitioner alternates; Section 6 discusses extensions and the fairness of deterministic alternate selection.
Significance. The sample-complexity analysis is the paper's strongest and most complete contribution: the dimension bounds are tight (matching lower-bound constructions), the PAC reduction is clean, and the O(a log n) sample size directly controls the ILP size. The paper is also commendably transparent about its caveats: footnote 3 concedes that Theorem 4.2 has not been shown for the constrained loss of Eq. (3), and Section 5.2 acknowledges that ERM-Altsℓ1's dominance in Figure 4 is by construction (the theoretical results do not depend on that figure, and the concern is not one of circularity). The empirical work is grounded in real dropout records, includes calibration checks (Appendix E.4), and contains a perturbation experiment (Figure 5) that is not by construction. Section 6 identifies a genuine open problem: deterministic alternate selection conflicts with the fairness and strategyproofness goals of randomized sortition. If the robustness claim is repaired or explicitly qualified, the paper is a solid contribution to algorithmic sortition; as written, the headline robustness claim outruns the theorem that supports it.
major comments (3)
- [Abstract; §1; §4 (Theorems 4.2-4.3)] The abstract's claim that the theoretical bounds cover 'loss due to dropout probability mis-estimation' and Section 1's statement that 'the linear loss is extremely robust, its loss growing linearly with the prediction error' are presented without qualification, but Theorems 4.2-4.3 are proved only for the relaxed loss L defined at the top of Section 4, in which the replacement set R is optimized over all R⊆A and the constraint |R|≤|D| of Eq. (3) is dropped. Footnote 3 concedes that Theorem 4.2 has not been shown for the original loss. The constraint is not a harmless technicality: Eq. (2), the deployed ILP (Appendix B.1), and the evaluations in Figures 3-5 all enforce it, and it expresses the practical requirement that the final panel not exceed its budgeted size. Please either extend the proof to the constrained loss or state the relaxation prominently in the abstract, the introduction, and the statements of Theorems 4.2-4.3, and make explicit that the empirical robustness observations in Section 5.3 are not covered by the theorem.
- [§4 (shorthand definitions); Appendix D.2 (proof of Theorem 4.2)] The proof of Theorem 4.2 conflates the constrained and relaxed optima. The shorthand at the start of Section 4 defines OPT-Â and ERN-Â through the Section 2 Opt ILP, which enforces |R|≤|D|, and Corollary 3.7 (invoked in the chain) is for that constrained loss. However, the chain requires OPT̃-Â to be optimal for the relaxed loss on D̃ρ ('the first inequality is by the optimality of OPT̃-Âℓ1 for D̃ρ') and similarly requires OPT-Â's relaxed loss to be no larger than ERN-Â's; neither follows from constrained optimality, since the relaxed loss admits strictly more replacement sets. A second obstruction is the common replacement set R_X in the coupling: it is taken from one of the unconstrained optimizers R(Y(X)) or R(Y′(X)), and since D(Y(X))△D(Y′(X))={i}, the two dropout sets differ in size by one, so R_X need not satisfy |R|≤|D| for both coupled draws. Please (i) prove the theorem for the constrained loss with a feasible coupling, or (ii) restate Section 4 entirely for the relaxed problem: define the relaxed ERM algorithm, prove the analogue of Corollary 3.7 for it, and state the theorem as being about that algorithm, clearly separated from the deployed constrained version.
- [§2 (independence assumption), Corr. 3.6-3.7, Eq. (6)] The sample-complexity guarantees and the empirical estimator (Eq. (6)) both rely on the assumption that the dropout indicators X_i are independent Bernoulli variables. The paper justifies this once (Section 2), but several dropout drivers it cites—transport disruption, childcare failure, severe weather—are naturally shared shocks, and the data-cleaning notes in Appendix E.1 describe external shocks that affected specific assemblies. If dropouts are positively correlated, drawing s dropout sets by independent coin flips does not sample the realized process, and the PAC bound no longer controls loss under the true distribution. Because this assumption is load-bearing for the central sample-complexity claim, please add a sensitivity discussion (e.g., the effect of a common-shock component or mixture model) and, if the raw data permit, a quick empirical check of cross-panelist dropout correlation in the US and Canadian datasets.
minor comments (6)
- [§4 (definition of L)] The sentence 'Note that this relaxation does not affect our results from Section 3' needs a one-line justification. The Pdim upper bound survives because Theorem 3.4's upper bound uses only the class size |A|, and the lower-bound construction shatters singleton dropout sets on which |R|≤|D| is inactive.
- [Appendix D.2] In the display following the definition of S(X) and S′(X), the expectation reads E_X[|dev(S′(X))−dev(S′(X))|]; the first S′ should be S(X). Also, the five-case analysis bounding |dev_{f,f(i)}(S)−dev_{f,f(i)}(S′)| states 'we show only the first such case'; since Appendix D.2 purports to be the full proof, the remaining four cases should be written out or the passage explicitly marked as a proof sketch.
- [Appendix C.3 (Lemma C.2)] The pseudo-dimension uniform-convergence bound is cited to a set of course lecture notes (Fetaya, 2016). Please cite a standard reference instead (e.g., Anthony and Bartlett, Neural Network Learning: Theoretical Foundations, 1999), and check whether the stated bound is the tightest available form.
- [§5.2 ('Loss Estimation')] The evaluation estimates Lℓ1(A;Dρ) with 300 fresh samples, the same count used for training the ERM algorithms; a larger evaluation sample (500, as in the Appendix E.2 convergence plots) would reduce evaluation noise in Figures 4 and 5.
- [§1; §5.1; Appendix D.3] Minor typos: 'isitself NP-hard' in §1; 'algortihms' in §5.1; and the sample-complexity display in the proof of Theorem 4.3 ('Θ(k/2·log k + log(k/γ^k) + log(1/δ) / γ^{2k}/k²)') is garbled and should be cleaned up.
- [Eq. (6); §5] The multiplicative dropout model has |FV|+1 free parameters fitted on very small per-cluster datasets; a brief note on parameter identifiability (e.g., absorbing multiplicative constants among the βs) and on the stability of the MLE at these sample sizes would strengthen the empirical section beyond the data-limitation acknowledgment in footnote 4.
Circularity Check
No significant circularity: the PAC/ERM derivation is self-contained, and the one by-construction empirical benchmark is explicitly acknowledged by the paper.
full rationale
The paper's theoretical derivation chain is not circular. The sample complexity results (Corollaries 3.6 and 3.7) follow from standard agnostic PAC-learning uniform-convergence bounds applied to the hypothesis classes of alternate sets, with VC-dimension and pseudodimension bounds proven directly from the cardinality and construction of the classes; none of these bounds assume the target loss guarantee. The robustness results (Theorems 4.1-4.3) use a coupling/telescoping argument to bound changes in the relaxed loss L, and the proof does not presuppose the theorem statement. The only empirical comparison that is literally by construction is Figure 4, where the paper itself states: "ERM-Alts1's dominance here is by construction, so we repeat this analysis for five other performance metrics." The algorithm optimizes exactly the L_l1 objective used for evaluation in that figure, so its dominance there is an acknowledged sanity check, not a hidden prediction. The independent empirical support comes from Figure 3 (evaluation on realized dropout sets using dropout probabilities estimated only from the other assemblies) and Figure 5 (evaluation under perturbed dropout probabilities), which do not reduce to the algorithm's own objective. One scope limitation should be noted, though it is a correctness caveat rather than circularity: Section 4 proves the linear-loss robustness bound for the relaxed loss L that drops the replacement-size constraint |R|≤|D|, and footnote 3 admits "we have not been able to determine whether Theorem 4.2 does" hold without the relaxation; the abstract's unqualified "extremely robust" claim is therefore stronger than what is proven. This is an overstatement in scope, not a derivation that feeds its own conclusion back in as an input.
Assumptions & free parameters
free parameters (2)
- beta_0, beta_{f,v} (multiplicative dropout model parameters) =
not reported in paper
- sample size s = 300 =
300
assumptions (6)
- domain assumption Panelists' dropout events are mutually independent
- domain assumption Quota parameters and feature-values are exactly known for every pool and panel member
- domain assumption The original panel satisfies the quotas
- standard math Agnostic PAC learning uniform convergence bounds (Theorems 6.7/6.8 of Shalev-Shwartz and Ben-David; pseudodimension bound of Fetaya)
- domain assumption ILP solvers return exact optima for the Opt programs
- domain assumption The multiplicative 'simple independent action' model correctly represents dropout propensity
Cite this review
Pith. "Pith review of Alternates, Assemble! Selecting Optimal Alternates for Citizens' Assemblies." pith.science (2026). https://pith.science/paper/WZ6YGXX4
@misc{pith2026250615716,
author = {Pith},
title = {Pith review of: Alternates, Assemble! Selecting Optimal Alternates for Citizens' Assemblies},
year = {2026},
howpublished = {\url{https://pith.science/paper/WZ6YGXX4}},
note = {Machine review of arXiv:2506.15716}
}
read the original abstract
Citizens' assemblies are an increasingly influential form of deliberative democracy, where randomly selected people discuss policy questions. The legitimacy of these assemblies hinges on their representation of the broader population, but participant dropout often leads to an unbalanced composition. In practice, dropouts are replaced by preselected alternates, but existing methods do not address how to choose these alternates. To address this gap, we introduce an optimization framework for alternate selection. Our algorithmic approach, which leverages learning-theoretic machinery, estimates dropout probabilities using historical data and selects alternates to minimize expected misrepresentation. Our theoretical bounds provide guarantees on sample complexity (with implications for computational efficiency) and on loss due to dropout probability mis-estimation. Empirical evaluation using real-world data demonstrates that, compared to the status quo, our method significantly improves representation while requiring fewer alternates.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Thus, it can be simplified to only consider the case when𝑋𝑖∈( min(𝜌𝑖, ˜𝜌𝑖), max(𝜌𝑖, ˜𝜌𝑖)], which (due to the distribution of𝑋𝑖) occurs with probability max(𝜌𝑖, ˜𝜌𝑖)− min(𝜌𝑖, ˜𝜌𝑖)≤ 𝛾: ≤ E𝑿 |𝑑𝑒𝑣ℓ1(𝑆′(𝑿))− 𝑑𝑒𝑣ℓ1(𝑆′(𝑿))| 𝑋𝑖∈( min(𝜌𝑖, ˜𝜌𝑖), max(𝜌𝑖, ˜𝜌𝑖)] ·𝛾 Angelos Assos, Carmel Baharav, Bailey Flanigan, and Ariel D. Procaccia Now expanding by𝑓,𝑣 and applying ...
work page 2024
-
[2]
The quotas require exactly one of each of the 𝑚 unique panelists, and anywhere from 𝑘−𝑚 to𝑘 of the 0 panelists. Now assume that all of the panelists with a non-zero feature-value have a high probability of dropping out, and the panelists with feature-value 0 have a dropout probability of 0. Assume that the pool has at least 𝑎 of each type of agent, so our...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.