{"id":"6aae6b4e-6fc3-49e0-af54-1bb65849137d","arxiv_id":"2508.08126","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OFAL improves an MNIST classifier from 93.0% to 95.7% test accuracy without oracle labels, by generating uncertain synthetic samples from confident seeds using a VAE and dropout uncertainty.","lead":"This paper introduces OFAL, a method that trains a neural network to be more accurate without any human labeling oracle. It does so by finding images the model is already confident about, then generating synthetic versions of those images that the model finds confusing, and retraining on them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"THU's stopping rule does not enforce label preservation (Alg. 1 lines 9-11); Fig. 4d shows a seed predicted 9 transformed into samples predicted 4, so generated pseudo-labels are likely wrong and OFAL's mechanism is not established.","rationale":"The reader's weakest assumption was precisely label preservation, and I agree. The paper's own Figure 4d makes the problem concrete rather than hypothetical: the visualization shows the predicted class flipping from 9 to 4 during THU, yet Section IV-A assigns the seed's label to the output. Since Algorithm 1's stopping rule is based solely on top-1/top-2 margin and not class identity, there is no mechanism preventing this. If pseudo-labels are wrong, the central claim that OFAL performs active learning by generating correctly labeled uncertain samples collapses; the observed accuracy gain could be due to the concurrently added confident real samples or to noise from mislabeled synthetic data. The proposed test (human labeling of generated xnew) directly measures the failure rate and would settle the concern. Because the reader already conditioned acceptance on missing ablations and error bars, and this concern is an instance of the same weak assumption, the verdict remains conditional rather than being upgraded to rejection: a focused experiment could either confirm the flaw or show that mismatch is negligible. I therefore recommend no change to the reader's conditional verdict.","tokens_in":12578,"tokens_out":6492,"duration_ms":77067,"concrete_test":"Use the 1000 labeled MNIST training samples (or a labeled subset) as seeds. Run Algorithm 1 unchanged (Tconf=0.99, Tstop=0.4, stepmax as in paper) and have a human annotator label each generated xnew image. Compute the mismatch rate between the seed's true label and the human label of xnew. If the mismatch rate is above, say, 5%, the label-preservation assumption fails. As a secondary check, retrain with only Xconf added per iteration (no THU samples) and compare final accuracy to 95.70%; if it matches, the generated samples contribute nothing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of OFAL is that walking in VAE latent space toward higher model uncertainty preserves the class of the confident seed, so the seed's pseudo-label Yconf can be assigned to the generated xnew (Section III-C, Algorithm 1, and Section IV-A: 'we use their baseline labels for them'). This label-preservation assumption is load-bearing: if xnew crosses into another class's latent region, training on it with Yconf introduces wrong labels, and any measured improvement cannot be attributed to correctly labeled informative samples. Algorithm 1's stopping criterion (lines 9-11) only requires max1 - max2 < Tstop, i.e., the top-1 and top-2 softmax probabilities become close. It never checks that the top-1 class of x' matches the seed's class. The paper's own Figure 4d contradicts the assumption: a seed predicted as 9 is transformed through intermediate steps into samples predicted as 4 while the method would assign the baseline label 9. Thus the generated sample is not 'still carrying the same label' as promised in Section I. Even if the seed's true label is 9, the final xnew predicted 4 is almost certainly a different digit. A separate but compounding issue is that each iteration also adds 200 confident real samples (Xconf,Yconf) alongside the 200 generated samples, so the 2.7% improvement may come entirely from pseudo-labeled real data (standard self-training) rather than from THU-generated samples. No ablation isolates the contribution of xnew. Together, these issues mean the paper's central interpretation—that oracle-free active learning via generated uncertain samples drives the gain—is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OFAL, an oracle-free active learning scheme. After initial training on a small labeled set, the method selects high-confidence unlabeled samples, assigns them pseudo-labels, and then uses a variational autoencoder (VAE) to walk in latent space toward regions of higher model uncertainty, generating new synthetic samples. Both the confident real samples and the generated uncertain samples are added to the training set with the seed pseudo-labels. Experiments on MNIST report that the model improves from 93.00% to 95.70% test accuracy after 12 acquisition iterations without an oracle. The paper also compares and integrates OFAL with uniform, uncertainty, margin, and entropy sampling methods.","tokens_in":12948,"tokens_out":4137,"duration_ms":48838,"significance":"If the central mechanism were validated, OFAL would be an interesting contribution: it proposes a way to create informative synthetic samples without oracle labels, potentially reducing labeling cost in active learning. The authors provide a clear algorithmic description and release code, which supports reproducibility. However, the current evidence is insufficient to support the core claim. The label-preservation assumption of the THU algorithm is unverified and appears contradicted by the paper's own Figure 4d, and the reported improvement is not isolated from standard pseudo-label self-training because confident real samples are added alongside generated samples. The evaluation also relies on a single MNIST run with no error bars. These issues are load-bearing for the central claim, so the paper requires major revision.","major_comments":[{"comment":"The core assumption that THU-generated samples retain the seed's label is neither enforced nor verified. Algorithm 1's stopping criterion (lines 9-11) only checks that the top-1 and top-2 softmax probabilities become close (max1 - max2 < Tstop); it never checks that the top-1 class of x' matches the seed's class. The paper's own Figure 4d shows a seed predicted as 9 transformed into samples predicted as 4, while Section IV-A states that the generated samples use their baseline labels. If the top-1 class changes, the pseudo-label assigned to x_new is wrong, and training on it can degrade the model. The authors must either modify the algorithm to reject samples whose predicted class changes, or provide an offline evaluation on ground-truth labels to quantify the label-preservation rate.","section":"Section III-C, Algorithm 1, Figure 4d"},{"comment":"The 2.7% improvement is not attributable to THU generation. In each acquisition iteration, the method adds 200 confident real samples (Xconf, Yconf) and 200 generated samples (Xuncr, Yconf) to the training set. A baseline that adds only Xconf with pseudo-labels (i.e., standard self-training) is necessary to isolate the contribution of the generated uncertain samples. Without this ablation, the improvement could be entirely due to pseudo-labeling of already-confident real samples, which is a known technique. Furthermore, Figure 6a shows a single run with no error bars; the random selection of confident samples makes the procedure stochastic, so multiple seeds are required to establish reliability.","section":"Section IV-A, Figure 6a, Table I"},{"comment":"The hyperparameters Tconf=0.99 and Tstop=0.4 are chosen heuristically: the text states that 'after experiments with several values' and 'by experimenting with different values for Tstop and observing the generated samples,' these values were selected. No quantitative sensitivity analysis is provided. Since these thresholds directly control pseudo-label quality and the degree of latent-space transformation, the paper should report how final accuracy varies with Tconf and Tstop (e.g., a table or figure) to demonstrate that the result is not sensitive to ad-hoc choices.","section":"Section IV-A"},{"comment":"The comparison with uniform and uncertainty sampling is confounded. In Figure 7, the OFAL curve includes both confident real pseudo-labeled samples and THU-generated samples, while the sampling baselines use only oracle-labeled real samples. The claim that OFAL reaches the same accuracy with one-tenth of the labeled data does not account for the fact that OFAL additionally uses a large number of unlabeled samples (with pseudo-labels) and adds 400 samples per iteration versus 200 for the baselines. The authors should compare methods under matched training-set sizes or at least explicitly separate the contribution of the generated samples from the pseudo-labeled real samples.","section":"Section IV-B and IV-C"}],"minor_comments":[{"comment":"Typo: 'wth' should be 'with'. The phrase 'still carrying the same label' is a central assertion but is not supported by any evidence at that point.","section":"Section I"},{"comment":"The variable name is inconsistent: 'Xcont' and 'Xconf' are used interchangeably. Please standardize.","section":"Section IV-A"},{"comment":"The caption and labels show transformed samples with different predicted classes than the seed (e.g., 9 to 4). Given the label-preservation assumption, this is confusing and should be explicitly discussed.","section":"Figure 4"},{"comment":"Line 6, 'grad <- apply gradient descent(loss)', is underspecified. The learning rate, optimizer, and number of gradient steps per iteration are not given; these are needed for reproducibility.","section":"Algorithm 1"},{"comment":"The restriction term is defined generically in Eq. (9) but Algorithm 1 uses MSE(z, z0). Please clarify whether the actual implementation uses MSE in latent space and whether any other distance was tested.","section":"Equation (9) and Algorithm 1"},{"comment":"Several captions contain typos: 'Magin' should be 'Margin', 'Unifo%m' should be 'Uniform', 'Unce%taint)' should be 'Uncertainty'. Please proofread the figure captions.","section":"Figures 7 and 8"},{"comment":"The phrase 'we active 2.7% improvement' should be 'we achieve a 2.7% improvement'.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is more in the style of a short workshop paper than a full journal article: it evaluates on a single dataset (MNIST) with one run and no ablations. The central mechanism (THU label preservation) is not validated, and the key experimental claim is confounded by the simultaneous addition of confident pseudo-labeled real samples. The authors should be asked to provide a substantially stronger empirical evaluation, including label-preservation checks, an ablation against self-training, and multiple seeds, before the paper could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on active learning under low-label budgets. What's genuinely new: instead of querying an oracle, pick high-confidence unlabeled samples, use their predicted labels as pseudo-labels, and then walk the VAE latent space toward higher epistemic uncertainty (MC dropout) to synthesize new training points. The THU algorithm is clearly specified, and code is provided. That combination isn't in the GAN-based generative AL papers they cite.\n\nWhat it does well: the framing is honest about being oracle-free; the MNIST setup is simple enough to reproduce; and they show integrations with uniform/uncertainty/margin/entropy sampling that make sense. The writing is readable.\n\nSoft spots, in order of seriousness. First, the load-bearing assumption that the latent walk preserves the seed's class is contradicted by their own Figure 4d: a seed predicted 9 ends up predicted 4, and they would label it 9. Algorithm 1's stopping rule checks only the margin between top-1 and top-2 softmax probabilities; it never verifies that the top-1 class of x' matches the seed. So some generated samples are almost certainly mislabeled, and the claimed mechanism—uncertainty-guided generation of correctly labeled informative samples—is not established. Second, every iteration adds 200 confident real pseudo-labeled samples alongside 200 generated ones. The 2.7% gain (93.0 to 95.7) may come entirely from self-training on those real confident samples. There is no ablation that trains on Xconf alone, or on generated samples only, so the contribution of THU generation is not isolated. Third, evaluation is a single MNIST run, no error bars, with Tconf and Tstop chosen after looking at validation performance. That makes the headline number weaker than it looks.\n\nThe paper is not incoherent and the method could work; the missing controls are fixable. But as written, the central claim is unsupported. I'd send it to a workshop or a venue with a reproducibility-focused review, asking for multi-seed runs, the self-training ablation, and a class-match check during THU. With those, this could be a useful paper. Without them, the empirical section doesn't demonstrate what the title promises.","headline":"OFAL is a plausible oracle-free active learning idea with code, but the paper's own figures undermine its core label-preservation assumption and no ablation isolates the generative mechanism from plain pseudo-labeling.","tokens_in":13460,"tokens_out":3575,"would_cite":false,"duration_ms":38510,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OFAL claims that active learning can proceed without an oracle by generating new, uncertain training samples from the model's own confident predictions.","keywords":["active learning","oracle-free","epistemic uncertainty","Monte Carlo dropout","variational autoencoder","mutual information","pseudo-labeling","MNIST"],"falsifier":"Decode the THU-generated samples from any acquisition round and ask a separate classifier or human annotator to label them; if more than a small fraction disagree with the seed's predicted label, the pseudo-label premise of OFAL fails and the observed accuracy gain cannot be attributed to correctly labeled synthetic data.","tokens_in":12473,"feed_emoji":"🤖","tokens_out":5074,"duration_ms":53063,"temperature":0.7,"pith_summary":"OFAL claims that active learning can be made oracle-free: instead of paying a labeler for the most uncertain unlabeled samples, the model starts from samples it already classifies with 99% or higher confidence, walks their representations in a variational autoencoder's latent space toward regions of high epistemic uncertainty, and treats the decoded results as new training samples carrying the seed's predicted label. On MNIST, after initial training to 93.00% test accuracy, twelve acquisition rounds of 400 such pseudo-labeled samples each raise accuracy to 95.70%. The framework also augments standard sampling methods, so an oracle-based method reaches a given accuracy with roughly one-tenth of the labeled samples when preceded by OFAL. The sympathetic reading is that, at least in this setting, uncertainty-guided synthesis plus confident-model pseudo-labels can substitute for an oracle.","feed_headline":"No oracle needed: active learning lifts MNIST to 95.7%","feed_subtitle":"Synthesized uncertain samples replace an oracle's labels, raising accuracy from 93.0% to 95.7% in 12 rounds.","key_machinery":"The Toward Higher Uncertainty (THU) algorithm carries the argument: it fixes the model weights, treats the VAE decoder as a differentiable generator, and performs gradient descent on the latent code z so that the decoded image maximizes the model's epistemic uncertainty (BALD mutual information, estimated via Monte Carlo dropout) subject to an MSE constraint keeping z near its starting point z0. The stopping criterion based on the top-two class probability gap is what keeps the generated sample close enough to the seed's class to justify the pseudo-label. The variational autoencoder's latent space is the central geometric object: it encodes all data into a meaningful low-dimensional manifold","core_discovery":"OFAL's central claim is that a model can improve itself from a pool of unlabeled data without any oracle by converting its own confident predictions into newly generated, uncertain training samples. The paper defines epistemic uncertainty as BALD mutual information estimated with Monte Carlo dropout, trains a VAE on all (labeled plus unlabeled) data, and then, for each unlabeled sample whose predicted class probability exceeds Tconf=0.99, walks the sample's latent code toward regions of higher uncertainty by gradient descent on L = -Uncertainty + α·MSE(z, z0). The stop rule halts the walk when the gap between the top two class probabilities drops below Tstop=0.4. The decoded sample xnew is a","pith_inferences":["Editorial inference: the paper never verifies that a generated sample xnew actually belongs to the seed's class; a straightforward test is to decode the 200 generated samples from any round and compare a human or held-out classifier's label against the seed's predicted label. If a meaningful fraction cross class boundaries, the pseudo-labels are corrupted and the accuracy gain would need a differe","Editorial inference: the reported 2.7% gain is on MNIST with a heavily over-trained CNN; on datasets with less separated classes, the VAE latent walk may not find same-class uncertain regions, so the gain could shrink or reverse.","Editorial inference: the hyper-parameter trade-off described (Tconf=99.99% yields too few confident samples, Tconf=99% gives near-perfect labels) suggests that the method implicitly relies on a model that is already well-calibrated on the unlabeled pool; on poorly calibrated models the confidence threshold would not guarantee label correctness."],"forward_implications":["OFAL can be stacked on top of any oracle-based sampling method: each acquisition round adds 200 model-selected confident samples and 200 synthesized uncertain samples to the 200 oracle-labeled samples, improving accuracy under the same oracle budget.","When OFAL runs before uncertainty sampling, the same accuracy is reached after one acquisition iteration instead of ten, i.e., with 200 labeled samples instead of 2000.","The improvement is achieved with zero oracle labels for the 200 generated samples, since their labels are the seed's predicted labels; the paper reports these seed labels are almost 100% accurate at Tconf=0.99.","The method's effectiveness depends on VAE latent geometry: confident samples sit in low-uncertainty regions and uncertain regions sit near class borders, so walks stay on the same-class side when Tstop=0.4."],"supporting_citations":[{"why":"Supplies the Monte Carlo dropout estimator of predictive entropy and mutual information used by THU to measure uncertainty.","marker":"[19]"},{"why":"Defines BALD mutual information as the epistemic uncertainty measure that the loss maximizes.","marker":"[5]"},{"why":"Provides the variational autoencoder whose latent space is the domain of the walk.","marker":"[8]"},{"why":"Introduces generative active learning, the paradigm OFAL extends by removing the oracle.","marker":"[23]"},{"why":"Provides the MNIST dataset on which the method is evaluated.","marker":"[10]"},{"why":"Separates aleatoric from epistemic uncertainty, justifying why the generated uncertain samples are informative rather than noise.","marker":"[7]"}],"fun_headline_variants":["No oracle, no problem: AI teaches itself to 95.7%","Self-generated samples push MNIST to 95.7%","Active learning goes oracle-free, gains 2.7%","Model's own uncertainty becomes its teacher: 95.7%","Drop the oracle: AI synthesizes its own training data"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The new sample generated by walking in VAE latent space carries the same class label as the confident seed sample it came from, and OFAL never checks that label preservation actually holds.","fun_headline_variants_meta":{"raw":{"variants":["No oracle, no problem: AI teaches itself to 95.7%","Self-generated samples push MNIST to 95.7%","Active learning goes oracle-free, gains 2.7%","Model's own uncertainty becomes its teacher: 95.7%","Drop the oracle: AI synthesizes its own training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3412,"prompt_tokens":700,"completion_tokens":2712,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":2623}},"tokens_in":444,"tokens_out":2712,"duration_ms":23124,"temperature":1.0,"reasoning_tokens":2623,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:37:24.309648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode the THU-generated samples from any acquisition round and ask a separate classifier or human annotator to label them; if more than a small fraction disagree with the seed's predicted label, the pseudo-label premise of OFAL fails and the observed accuracy gain cannot be attributed to correctly labeled synthetic data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the variational autoencoder whose latent space is the domain of the walk."},{"cited_title":"LeCun, L","cited_arxiv_id":null,"evidence_quote":"Provides the MNIST dataset on which the method is evaluated."},{"cited_title":"Kendall and Y","cited_arxiv_id":null,"evidence_quote":"Separates aleatoric from epistemic uncertainty, justifying why the generated uncertain samples are informative rather than noise."}],"review_version":1}