{"id":"98029fe1-1e96-4bb7-8335-07677cae7c1f","arxiv_id":"1908.02723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Advocacy learning, with class-conditional attention maps from per-class advocates and a judge network, yields small classification accuracy improvements on some image benchmarks.","lead":"This paper introduces advocacy learning, a training scheme where one neural network per class produces an attention map arguing for its class and a judge network weighs these arguments to classify the input. It reports small but consistent accuracy gains on MNIST and Fashion-MNIST, with mixed results on CIFAR-10 and a clinical time-series task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed advantage of deceptive advocacy depends on a fragile judge/advocate capacity balance that the paper only finds by post hoc test-set search, so the central 'can improve' claim is not yet shown to be reproducible under an honest model-selection protocol.","rationale":"The reader's CONDITIONAL verdict and weakest assumption identify the same load-bearing issue: the Judge must be able to use the Advocates' evidence without being deceived, and the paper demonstrates this balance is fragile. My stress test focuses the concern more sharply on the absence of a valid model-selection procedure: Section 4.3's capacity search is evaluated on test accuracy, and the CIFAR-10 collapse in Section 4.1 shows that the training scheme is not robust to even a single capacity change. This does not refute the modest empirical claim on MNIST/Fashion-MNIST, but it means the paper has not shown that the gains are obtainable under a reproducible, non-test-peeking protocol. Since the paper's own abstract limits the claim to 'small improvements' and 'in some cases,' a conditional acceptance with a demand for validation-based capacity selection and reported hyperparameters is the appropriate outcome. I agree with the reader's assessment rather than proposing a stronger rejection, because the underlying idea is novel and the reported Table 1 results are internally plausible; the weakness is in the evidence base, not in a demonstrated logical inconsistency.","tokens_in":9803,"tokens_out":4136,"duration_ms":53137,"concrete_test":"Run a pre-registered model-selection experiment on MNIST and Fashion-MNIST: split the training data into a training set and a validation set, fix the L1 attention penalty coefficient to a reported value, then grid-search judge residual blocks (1-5) and advocate residual blocks (1-3) for both the Advocacy Net and the Multi-Attention Net, selecting each method's configuration by best validation accuracy across 5 seeds. Report the selected configurations and test accuracies. If the validation-selected Advocacy Net does not beat the validation-selected Multi-Attention Net on both datasets, or if the selected capacity ratio is not stable across seeds, then the claimed consistent improvement is an artifact of test-set-based capacity selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that advocacy learning yields small but consistent accuracy gains over an identical supervised baseline. For that claim to hold in general, there must be a reliable way to keep the Judge able to use evidence without being deceived by the Advocates. The paper's own experiments show this balance is load-bearing and unstable: Section 4.1 reports that replacing the Judge with a ResNet-110 collapses Advocacy Net accuracy to 30.54% versus 92.01% for the Multi-Attention Net, while Honest Advocacy Learning still reaches 92.68%. Section 4.3 then searches over judge capacity (1-5 residual blocks) and advocate capacity (1-3 blocks) and reports that the best Advocacy Net beats the best Multi-Attention Net only when judge capacity is high and advocate capacity is low. This capacity rule is derived after seeing test-set accuracies, with no validation split and no error bars on the search, so it is vulnerable to selection on test performance. The concern is not that disagreement with the baseline is impossible; it is that the paper does not supply a principled, out-of-sample procedure for finding the required balance. If the balance point differs across datasets, seeds, or L1 penalty settings, then the observed Table 1 gains may reflect a fortunate capacity choice rather than a general property of competitive class-conditional attention. The missing L1 penalty coefficient (Section 3.2) and the absence of CIFAR error bars further obscure whether the advertised improvements are robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces advocacy learning, a supervised training scheme for attention-based classification in which N class-specific Advocate subnetworks each produce an attention map (\"argument\") and a Judge network predicts the class from the element-wise products of these maps with the input. Advocates are trained with the deceptive objective of maximizing the Judge's probability for their own class even on inputs from other classes; an honest variant trains each Advocate only on its own class's examples. Experiments compare Advocacy Nets against an Attention Net and a Multi-Attention Net on MNIST, Fashion-MNIST, CIFAR-10, MIMIC-III mortality prediction, and imbalanced/binary MNIST variants. The main positive result is a small but consistent accuracy improvement on MNIST (99.42% vs 99.16% Attention Net, 99.33% Multi-Attention Net) and Fashion-MNIST (91.62% vs 89.71% Attention Net, 90.11% Multi-Attention Net), with means and standard deviations over five seeds. Results on CIFAR-10, MIMIC, and Binary MNIST are mixed or negative, and the paper concludes that advocacy learning is beneficial only in some circumstances.","tokens_in":10154,"tokens_out":4414,"duration_ms":44396,"significance":"If the MNIST and Fashion-MNIST results are reproducible, advocacy learning is a conceptually novel way to combine class-conditional attention with competition among subnetworks. The paper's strengths include the public code release, the use of architecture-matched baselines (Multi-Attention Net) to isolate the effect of the training objective, the honest-advocacy ablation, and the five-seed mean/standard-deviation reporting on the main image benchmarks. The central limitation is that the method's success appears to depend on a judge/advocate capacity balance that is identified post hoc on test-set accuracy, and the CIFAR-10 collapse with a ResNet-110 judge shows the balance is not automatically achieved. This limits the strength of the general claim until a principled, out-of-sample model-selection procedure is demonstrated.","major_comments":[{"comment":"The capacity analysis in Section 4.3 selects the number of residual blocks by comparing test-set accuracies (best Advocacy Net 99.46% vs best Multi-Attention Net 99.34%) with no validation split and no multiple-seed error bars reported for the search. Because the conclusion that the Judge should have higher capacity than the Advocates is drawn from this post hoc test-set selection, the paper does not provide a reproducible model-selection protocol. Given that Section 4.1 shows a high-capacity Judge (ResNet-110) collapses the Advocacy Net to 30.54% on CIFAR-10, the balance is load-bearing for the central claim; please re-run the capacity search using a held-out validation split and report means and standard deviations over seeds for the selected configuration.","section":"Section 4.3"},{"comment":"CIFAR-10 results are reported as single accuracies without standard deviations: 83.47 vs 79.73 for the lower-capacity setup and 30.54 vs 92.01/92.68 for the ResNet-110 judge setup. The collapse of the deceptive Advocacy Net under a high-capacity Judge is a serious instability that conflicts with an unqualified reading of the abstract's claim that advocacy learning leads to improvements. Please report repeated-seed statistics and specify the exact architecture and hyperparameters used for the CIFAR-10 experiments, or explicitly restrict the claim to the settings in which the improvement is statistically stable.","section":"Section 4.1"},{"comment":"The L1 attention regularization penalty coefficient is mentioned but its value is never given, and no sensitivity analysis is provided. Since the sparsity of the attention maps directly controls the evidence available to the Judge, an unspecified coefficient makes the experiments not fully reproducible and leaves open the possibility that the observed gains depend on a particular regularization strength. Please report the coefficient and, ideally, include a small robustness sweep over this hyperparameter.","section":"Section 3.2"},{"comment":"Table 2 shows that the Advocacy Net underperforms the Attention Net on MIMIC (AUROC 77.73 vs 83.29) and on Binary MNIST (98.72 vs 99.23), and only matches the Honest Advocacy Net on Imbalanced MNIST (99.17 vs 99.17). The paper's conclusion appropriately says 'in some circumstances,' but the abstract and introduction's phrasing ('can lead to small improvements') should be scoped to balanced multi-class image data, and these negative results should be treated as boundary conditions of the method rather than as a general demonstration of improvement.","section":"Section 4.4 / Table 2"}],"minor_comments":[{"comment":"The phrase 'Though a series of follow-up experiments' should be 'Through a series of follow-up experiments'.","section":"Abstract"},{"comment":"In the sentence reporting the shared-encoder ablation, 'FMIST' should be 'FMNIST'.","section":"Section 4.3"},{"comment":"Reference [Adam et al., 2017] is misattributed; the PyTorch paper should be cited as Paszke et al., 2017.","section":"References"},{"comment":"The honest-advocate update condition in line 13 ('if not honest or i = y') is easy to misread; consider rewriting as 'if (not honest) or (i == y)' and adding a sentence clarifying that honest advocates receive no gradient for classes other than the true class.","section":"Algorithm 1"},{"comment":"The abstract's phrase 'identical supervised baseline' is ambiguous; the Multi-Attention Net is the architecture-matched baseline, while the Attention Net removes all but one attention module. Please clarify this distinction in the text.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a competent empirical paper with a modest central claim. The main risk is that the positive MNIST/Fashion-MNIST results are robust, but the broader claim depends on a judge/advocate capacity balance that is found post hoc on test data. A major revision that adds a validation-based model-selection procedure, repeated-seed CIFAR-10 results, and the missing L1 penalty value would address the load-bearing concerns and, in my view, make the paper publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Advocacy learning is a real, if modest, new idea: per-class attention maps trained to convince a Judge, plus a Judge trained not to be fooled. I do not know of prior work doing exactly this combination, and the related-work section draws honest contrasts with AI debate, capsules, and mixture-of-experts. The paper earns credit for that, and for shipping code and reporting mean plus standard deviation over five seeds on the main benchmarks.\n\nWhat it does well: on MNIST and Fashion-MNIST the gains are small but consistent — 99.42 vs 99.16 and 91.62 vs 89.71 against the attention baseline, with the Honest Advocacy Net as a clean control separating class-conditional attention from deception. The confusion-matrix analysis and the qualitative attention maps are thoughtful. The finding that deception itself adds something beyond class-conditional attention is the most interesting part.\n\nWhere it is soft, in proportion: the abstract's claim that advocacy learning leads to improvements across several tasks is too strong. The paper's own CIFAR-10 experiment is a red flag, not a side detail: swapping in a ResNet-110 Judge collapses Advocacy Net accuracy to 30.54% while Multi-Attention gets 92.01%, and Honest Advocacy still works. That means the advertised benefit depends on a judge/advocate capacity balance the paper does not yet know how to find reliably. Section 4.3 searches capacities and reports that high Judge capacity and low Advocate capacity works best, but the search appears to be evaluated on test accuracies, without a validation split or error bars. That is selection on the test set, and it weakens the general claim. Missing pieces reinforce the concern: no CIFAR error bars, and the L1 attention penalty coefficient is never reported. These are correctable, but they need to be corrected before the 'can lead to improvements' claim can be taken as robust.\n\nThe MIMIC result is honestly reported and useful: advocacy learning does worse there, and the discussion of why is plausible. The citation pattern is fine. This is not a paper with a load-bearing logical flaw; it is a paper with a load-bearing fragility in the method and an over-broad summary of its own evidence.\n\nWho is this for? People working on attention mechanisms, adversarial or competitive training, and interpretability-by-attention. They will find it a useful, citable data point even if they do not adopt the method. I would bring it to a reading group as a good case study in capacity sensitivity and test-set selection.\n\nRecommendation: send to review. A serious referee should require CIFAR error bars, the L1 penalty value, and either a validation-based capacity selection protocol or an explicit caveat that the gains require capacity tuning and do not transfer to all settings. With those changes, the paper would be a fair conditional accept.","headline":"Advocacy learning is a genuinely new training trick with modest, real gains on two benchmarks, but the paper's own experiments show the method is fragile in ways the abstract understates; it still deserves a careful referee.","tokens_in":10640,"tokens_out":2143,"would_cite":true,"duration_ms":27278,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that attention-based classifiers improve when each class gets a dedicated advocate whose argument a judge learns to weigh, and reports small consistent gains on MNIST and Fashion-MNIST.","keywords":["advocacy learning","class-conditional attention","attention maps","competitive training","deception","supervised classification","MNIST","Fashion-MNIST"],"falsifier":"Run a capacity sweep on Fashion-MNIST varying the number of residual blocks in the judge and advocates: the paper's claim predicts a middle-capacity zone where the deceptive Advocacy Net beats the best Multi-Attention Net. If no such zone exists, or if it appears at a capacity balance different from the MNIST result, the capacity-balance explanation is wrong.","tokens_in":9585,"feed_emoji":"⚖️","tokens_out":9153,"duration_ms":90493,"temperature":0.7,"pith_summary":"This paper proposes a supervised training scheme called advocacy learning for attention-based classifiers. Instead of one attention map, the network contains one advocate per class; each advocate produces an attention map meant to convince a judge network that the input belongs to that advocate's class, even when it does not. The judge sees all the class-specific arguments and must decide the true label. On MNIST and Fashion-MNIST, the scheme beats an identical baseline with a single supervised attention map (99.42% vs. 99.16% and 91.62% vs. 89.71%), and a non-deceptive honest variant helps on imbalanced and clinical data. The broader claim is that competition among class-conditional representations can improve discriminative performance, but only when judge and advocate capacities are balanced.","feed_headline":"Per-class advocate training tops single-attention baseline","feed_subtitle":"A judge weighs one attention map per class; the scheme beats standard supervision on MNIST and Fashion-MNIST.","key_machinery":"The central object is the advocacy net: $N$ advocate subnetworks (one per class) share an encoder and output attention maps $a_i \\in [0,1]^d$; each evidence is $e_i = a_i \\odot x$, and the judge network receives $E = [e_1, \\ldots, e_N]$ and outputs $\\hat{y}$. Training alternates two objectives: the judge minimizes the standard classification loss on the true label, while advocate $i$ minimizes $-\\log \\hat{y}[i]$, the loss that would be correct if the input belonged to class $i$. The honest variant updates advocate $i$ only when $i$ is the true class. This asymmetric training is the mechanism that creates competition; the shared encoder and the L1 sparsity penalty on attention maps are what keep the advocates from becoming incoherent.","core_discovery":"The central claim is that training per-class attention maps competitively, with each advocate maximizing the judge's probability for its assigned class, can yield small but consistent accuracy gains over identical networks trained end-to-end. The result is shown on MNIST and Fashion-MNIST across five seeds, with the deceptive version outperforming both the attention baseline and the honest variant (99.42 vs. 99.32 on MNIST; 91.62 vs. 90.81 on Fashion-MNIST). The paper also reports that the gains are conditional: with a high-capacity judge on CIFAR-10, advocacy collapses to 30.54% accuracy while the honest variant stays competitive, and on MIMIC-III deceptive advocacy underperforms baselines while honest advocacy slightly improves AUPR. The discovery is that competition and deception, normally avoided in supervised learning, can be useful if the judge is powerful enough to evaluate evidence but not so powerful that it trains slowly and gets deceived.","pith_inferences":["As an extension not tested in the paper, competition should transfer to other argument formats: if an input can be edited to resemble a wrong class, advocates could argue with additive or textual edits rather than pixel masks, and the same inverted-U judge-capacity curve should appear.","As an extension, the 8/9 asymmetry suggests a testable predictor: deception helps most for class pairs where one class can be visually morphed into the other, and confusions should shrink mainly in the morphable direction.","As an extension, a curriculum that anneals the fraction of deceptive advocate updates from zero to the full advocacy objective might stabilize high-capacity judges, combining the robustness of honest advocacy with the accuracy of deception."],"forward_implications":["On balanced image tasks, replacing a single supervised attention map with $N$ class-conditional maps improves accuracy even before competition is added.","The deception objective contributes beyond class-conditionality: the Advocacy Net outperforms the Honest Advocacy Net on MNIST and Fashion-MNIST.","Gains are conditional: the scheme degrades sharply when the judge is much higher capacity than the advocates (CIFAR-10) and when inputs cannot be morphologically transformed between classes (MIMIC-III).","Honest advocacy is the stable variant: it retains accuracy in the high-capacity-judge setting and improves AUPR on clinical mortality prediction."],"supporting_citations":[{"why":"Supplies the Fashion-MNIST benchmark used for the main accuracy comparisons.","marker":"[Xiao et al., 2017]"},{"why":"Provides the MIMIC-III electronic health record data used in the generalization experiments.","marker":"[Johnson et al., 2016]"},{"why":"Defines the clinical time-series mortality-prediction benchmark used to evaluate advocacy learning on non-image data.","marker":"[Harutyunyan et al., 2017]"},{"why":"Provides the residual blocks used to vary judge and advocate capacity in the balancing experiments.","marker":"[He et al., 2016]"},{"why":"Supplies the adversarial-competition framework that motivates the idea of having subnetworks compete.","marker":"[Goodfellow et al., 2014]"},{"why":"Provides the optimizer used to train all models in the experiments.","marker":"[Kingma and Ba, 2014]"},{"why":"Supplies the encoder-decoder architecture used to generate the advocate attention maps.","marker":"[Badrinarayanan et al., 2017]"}],"fun_headline_variants":["Advocacy learning: deceptive attention maps edge out standard training","Per-class advocates beat baseline when judge is powerful but not too smart","Teaching networks to argue: a win for adversarial attention","Competitive attention maps improve accuracy on MNIST and Fashion-MNIST"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a usable balance between judge and advocate capacities can be found: the judge must be strong enough to weigh evidence well and weak enough not to be fooled by deceptive advocates.","fun_headline_variants_meta":{"raw":{"variants":["Advocacy learning: deceptive attention maps edge out standard training","Per-class advocates beat baseline when judge is powerful but not too smart","Teaching networks to argue: a win for adversarial attention","Competitive attention maps improve accuracy on MNIST and Fashion-MNIST"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000118,"raw_usage":{"total_tokens":1073,"prompt_tokens":925,"completion_tokens":148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":76}},"tokens_in":541,"tokens_out":148,"duration_ms":2891,"temperature":1.0,"reasoning_tokens":76,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:36:17.072029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a capacity sweep on Fashion-MNIST varying the number of residual blocks in the judge and advocates: the paper's claim predicts a middle-capacity zone where the deceptive Advocacy Net beats the best Multi-Attention Net. If no such zone exists, or if it appears at a capacity balance different from the MNIST result, the capacity-balance explanation is wrong.","supporting_citations":[{"cited_title":"Pollard, Lu Shen, H","cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-III electronic health record data used in the generalization experiments."},{"cited_title":"Gen- erative adversarial nets","cited_arxiv_id":null,"evidence_quote":"Supplies the adversarial-competition framework that motivates the idea of having subnetworks compete."},{"cited_title":"Kingma and Jimmy Ba","cited_arxiv_id":null,"evidence_quote":"Provides the optimizer used to train all models in the experiments."},{"cited_title":"Segnet: A deep convolu- tional encoder-decoder architecture for image segmenta- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the encoder-decoder architecture used to generate the advocate attention maps."}],"review_version":1}