{"id":"fcbd69c6-8908-49bb-bf88-1156d2b7ad1b","arxiv_id":"2412.18105","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"High-confidence unknown samples extracted from the target domain are used as hard negatives to tighten OVANet's open-set boundaries, improving unknown accuracy at a small known-accuracy cost.","lead":"This paper adds a penalty term to OVANet that forces high-confidence unknown target samples to remain outside all known classes, tested with direct, augmented, and GAN-generated negatives. It reports H-scores close to existing methods on Office-31, Office-Home, and VisDA while often improving unknown-category accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The extraction threshold's purity is validated only on Office-31 (Fig. 4); without measuring it on Office-Home/VisDA, Lneg_ova may push known samples into the unknown region, inflating Acc_Lunk at the cost of Acc_Ls.","rationale":"The reader's weakest assumption is exactly the condition I identify as load-bearing: the threshold 1 - po(yhat|x) > 0.9 must select almost only true unknown samples, and this is only validated on Office-31. I agree with that assessment. The mechanism in Eq. (1) is a hard negative constraint on all one-vs-all classifiers; if known-class samples enter X, the loss actively suppresses the correct known-class output, so known accuracy degrades. The empirical pattern in Table 5, where Acc_Ls drops are not negligible on Office-Home and VisDA, is consistent with contamination, though it could also be an intended trade-off. The paper provides no per-dataset measurement of X purity or size for the two larger benchmarks, so the concern cannot be dismissed from the reported numbers. The provided code is a positive factor, but was not inspected and does not by itself settle the calibration-transfer question. I considered other issues, such as the lambda selection being done on the same benchmark used for final evaluation and the lack of error bars, but those are secondary protocol concerns; the threshold-purity assumption is the one on which the method's validity hinges. Since the reader already issued a CONDITIONAL verdict and my analysis supports exactly that condition, I recommend no change to the verdict rather than a new rejection or acceptance.","tokens_in":14157,"tokens_out":3800,"duration_ms":39207,"concrete_test":"For each task in Office-Home (12 tasks) and VisDA (1 task), run the unmodified OVANet reproducibility baseline up to the breakpoint (iteration 1000), apply the extraction rule X = {x : 1 - po(yhat|x) > 0.9} to the target training set, and then use the target labels (available for evaluation, not for training) to compute the precision of X, i.e., the fraction of extracted samples whose true label is not in Ls. Report precision per task and the fraction of target samples extracted. If precision is below 90% on any task, the negative constraint in Eq. (1) is likely suppressing known accuracy through contamination, and the central claim is not supported. A complementary check, if resources permit, is to rerun the Original strategy on Office-Home with an oracle X built from true labels and compare Acc_Ls and H-score against the threshold-based X.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism requires that the set X extracted with threshold 1 - po(yhat|x) > 0.9 contains almost only true unknown samples. This is asserted from Figure 4, which is shown only for Office-31 (Section 4.5.2), and then assumed to transfer to Office-Home and VisDA. On those datasets the domain shift is stronger and the number of known classes differs (|Ls|=25 and |Ls|=6 versus |Ls|=10), so the one-vs-all confidence scores may be miscalibrated. If X contains known samples, the loss in Eq. (1) penalizes all binary classifiers in O, including the classifier corresponding to the sample's true known class, driving po toward 0 for that class and directly suppressing known accuracy. The reported trade-offs are consistent with this risk: on Office-Home, Original improves Acc_Lunk by 4.5 points while losing 1.7 points of Acc_Ls; on VisDA, Generation improves Acc_Lunk by 4.9 points while losing 3.6 points of Acc_Ls (Table 5). The paper does not report the size or the label-based purity of X for Office-Home or VisDA, so the reader cannot tell whether these Acc_Ls losses are the intended price of a stricter boundary or the result of contaminated negative supervision. This is the most load-bearing assumption because the method's benefit, and the interpretation of every reported result, depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to improve OVANet for Open Set Domain Adaptation by extracting high-confidence unknown samples from the target domain (set X) using a threshold of 1 - po > 0.9, and then adding a negative-constraint loss Lneg_ova (Eq. 1) that compels the one-vs-all binary classifiers in O to reject those samples. Three variants are evaluated: using the pristine negatives (Original), using augmented negatives (Augmentation), and using GAN-generated negatives (Generation), with an additional Generation++ variant that uses OVANet's backbone in the GAN discriminator and interleaves GAN training with OVANet training. Experiments on Office-31, Office-Home, and VisDA report H-score, Acc_Ls, and Acc_Lunk; the results show that the variants often improve Acc_Lunk over the OVANet reproducibility baseline while keeping H-score comparable on some tasks.","tokens_in":14472,"tokens_out":4785,"duration_ms":43707,"significance":"If the central claim holds, the paper contributes a simple, method-agnostic idea—exploiting high-confidence unknown samples as negative constraints—that can be plugged into existing OSDA methods. The authors release code, reproduce OVANet baselines on all three benchmarks, and provide a fairly extensive set of ablations. The reporting of Acc_Ls and Acc_Lunk separately is more informative than H-score alone, and the paper is honest that gains in Acc_Lunk come with reductions in Acc_Ls. However, the mechanism's validity rests on the purity of the extracted set X, and the evidence for that purity is currently limited to one dataset, which is a load-bearing gap.","major_comments":[{"comment":"The threshold 0.9 used to extract X is validated only on Office-31, where the boxplots show that most unknown samples receive 1 - po > 0.9. The paper then assumes the same threshold transfers to Office-Home (|Ls|=25) and VisDA (|Ls|=6), where the one-vs-all confidence scores may be miscalibrated. If X contains a non-negligible fraction of known samples, Eq. (1) penalizes all binary classifiers in O, including the one corresponding to the sample's true known class, driving po toward 0 for that class and suppressing Acc_Ls. The observed trade-offs in Table 5 (e.g., Original on Office-Home: +4.5 Acc_Lunk, -1.7 Acc_Ls; Generation on VisDA: +4.9 Acc_Lunk, -3.6 Acc_Ls) are consistent with contaminated X. The authors should report the label-based purity of X on Office-Home and VisDA (labels are available for post-hoc analysis) or at least show the same distribution plot as Figure 4 for those datasets.","section":"Section 4.5.2, Figure 4"},{"comment":"The hyperparameters λ and the extraction threshold are tuned on Office-31 only and then applied to Office-Home and VisDA without a held-out validation set. Table 2 shows that λ = 0.20 is marginally better on Office-31, but it does not establish that this value is appropriate for datasets with different |Ls| and stronger domain shift. Since the magnitude of Lneg_ova directly controls how aggressively known-sample probabilities are suppressed, the authors should either report a sensitivity analysis on all benchmarks or justify the transfer of λ and the threshold with a principled calibration argument.","section":"Sections 4.5.1 and 4.5.2"},{"comment":"The set X is defined by OVANet's own open-set confidence (1 - po > 0.9) and then used to train the same binary classifiers that define that confidence. This creates a potential self-reinforcing loop: Eq. (1) can make already-high-confidence unknown samples even more confident without necessarily tightening the boundary between known and unknown regions. To separate genuine boundary tightening from self-reinforcement, the authors should report, for instance, the accuracy on known samples that lie near the decision boundary (e.g., 0.5 ≤ po ≤ 0.9) before and after applying Lneg_ova, or the fraction of X that was already correctly rejected before the negative-constraint training began.","section":"Section 3 and Eq. (1)"}],"minor_comments":[{"comment":"The conclusion states that the approach increases Acc_Lunk 'in exchange for a small amount of Acc_Lunk'; the second Acc_Lunk should be Acc_Ls.","section":"Section 5"},{"comment":"The paper acknowledges that the generated negative image in the third row is 'not meaningful at all'; this undercuts the motivation for the Generation approach. The authors should discuss whether the entropy and agreement losses in Eqs. (2) and (3) can produce semantically meaningful negatives, or whether the empirical benefit from Generation arises from a regularizing effect rather than from semantic fidelity.","section":"Section 4.6.1, Figure 7(a)"},{"comment":"The claim that the strategies are 'method-agnostic' is speculative because all experiments are conducted on OVANet; a single additional comparison on another OSDA method would make this claim more concrete.","section":"Section 4.6, Table 6"},{"comment":"The paper states that the standard deviation is up to 1 and therefore reports only means. It would be more informative to include the standard deviations at least in a supplementary table, particularly because the reported differences between some variants are around 1 percentage point.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of Pattern Recognition Letters and the idea is timely, but the load-bearing assumptions—threshold transferability and the lack of held-out hyperparameter selection—need strengthening. The authors should provide a purity analysis of X on Office-Home and VisDA, or otherwise demonstrate that the negative constraint does not suppress known-class accuracy via contamination. The circularity concern is also worth addressing with an additional experiment. I would support acceptance after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the full text. The core idea is simple and clearly stated: after a short warm-up, extract high-confidence unknowns from the target domain (threshold 1 - po > 0.9), then use them as negatives for OVANet's one-vs-all binary classifiers, with three flavors (pristine, augmented, GAN-generated). The GAN variants add real work: equations for entropy and agreement losses, discriminator backbone, interleaved training. That is new relative to the cited works and honestly positioned as an extension of OVANet.\n\nWhat the paper does well: the experiments are reproducible in spirit. Code is public, the OVANet baseline is reproduced, three benchmarks are used, and the authors report AccLs, AccLunk, and H-score separately rather than hiding the trade-off. The main empirical claim is modest: you give up a few known-accuracy points and gain several unknown-accuracy points, with H-score near state-of-the-art. Tables 5 and 6 support that claim. On Office-31, the Original variant reaches H=92.0, slightly above the reproducibility baseline; on Office-Home and VisDA the gains and losses are of the stated magnitude.\n\nSoft spots, in order. First, the extraction threshold 0.9 is validated only on Office-31 (Fig. 4) and carried over to Office-Home and VisDA without checking purity. That is a genuine gap. If contaminated, Lneg_ova suppresses true known classes. It does not sink the paper, because the reported AccLs drops are consistent with contaminated negatives too, but the authors should measure X's label purity on all datasets or at least show known-class accuracy degradation patterns. Second, the text says OVANet is trained for \"half\" of the iterations before the breakpoint, but Table 1 lists the breakpoint at 1000 out of 10000 (or 25000). Sloppy and misleading. Third, no error bars despite claiming 3 runs; the text says standard deviation stays up to 1, but the tables show only means. For Pattern Recognition Letters that may be acceptable, but a serious referee should ask for the numbers.\n\nThe conclusion contains a typo (\"small amount of Acc Lunk\" should be AccLs). This is minor but shows rushed proofreading.\n\nFor OSDA researchers, especially those working with OVANet-style baselines, this offers a cheap trick worth knowing: unknown recall improves at a modest known-accuracy cost, and the code lets you verify quickly. It is not a state-of-the-art breakthrough, and the paper does not claim it is. With fixes to threshold validation and the schedule description, it fits a letters venue. I would send it to peer review rather than desk-reject.","headline":"A modest but honest OVANet extension: negative-sample constraints buy unknown-recall at a small H-score cost, though threshold transfer and training-schedule mismatch need fixing.","tokens_in":15013,"tokens_out":2317,"would_cite":false,"duration_ms":22611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a negative-constraint loss on high-confidence unknown samples tightens OVANet's known/unknown boundaries and raises accuracy on unknown categories.","keywords":["open set domain adaptation","unknown exploration","negative constraint loss","OVANet","one-vs-all classifiers","GAN-generated negatives","H-score","unknown category accuracy"],"falsifier":"Measure the fraction of true unknown-class samples in the extracted set X for each Office-Home and VisDA task; if that precision is substantially below the Office-31 level (for example, below 80 percent), the mechanism's reported benefit should degrade, since known samples in X would push the penalty to reject known categories.","tokens_in":13954,"feed_emoji":"🎯","tokens_out":7080,"duration_ms":56232,"temperature":0.7,"pith_summary":"This paper tries to establish that unknown target samples, which most Open Set Domain Adaptation methods either ignore or compress into a single extra logit, can be actively used as negative supervision to tighten the known/unknown classification boundary. The authors extract a set X of target instances for which OVANet's open-set confidence says 'unknown' with probability above 0.9, then add a loss Lneg_ova that drives the one-vs-all binary classifiers to reject those samples. They test three ways of using X: directly, with data augmentation, and with GAN-synthesized negatives, plus an improved Generation++ variant. Across Office-31, Office-Home, and VisDA, the strategies keep H-score close to or above the OVANet reproducibility baseline while consistently increasing accuracy on unknown categories. If correct, this is a method-agnostic plug-in that strengthens a well-known universal domain adaptation method on open-set tasks.","feed_headline":"Using unknown samples as negatives lifts OVANet's unknown accuracy","feed_subtitle":"Three strategies—direct, augmented, and GAN-generated negatives—raise unknown-class accuracy while H-score holds steady.","key_machinery":"The central object is the negative-constraint loss Lneg_ova (Equation 1) together with the extraction set X defined by the rule 1 - po(y_hat|x) > 0.9 on target samples at the breakpoint. This loss is the average over the known classes of log(1 - po(y_k|x)) for x in X, and it works as a hard constraint that compels OVANet's open-set head O to reject the selected high-confidence unknowns, tightening the known/unknown decision boundary. For the Generation strategy, the paper adds two generator losses, Lgen_ent (entropy over the closed-set classifier's outputs, pushing synthetic samples toward a known class) and Lgen_agree (agreement that the open-set head sees them as known), so the GAN produces adversarial negatives that the open-set head must learn to reject.","core_discovery":"The paper's central claim is the hypothesis stated in Section 3: incorporating knowledge from negative samples can enhance OVANet's learning capability and tighten the classification boundaries between known and unknown categories. The mechanism is the negative-constraint loss Lneg_ova, which averages log(1 - po(y_k|x)) over known classes for each sample in X and is added to OVANet's loss after a breakpoint at half the training iterations. The authors evaluate three usage strategies for X and report that the Original, Augmentation, and Generation/Generation++ approaches all beat the OVANet reproducibility baseline on at least one of the three benchmarks, with the best average H-score of 92.0 on Office-31 (Original), 64.5 on Office-Home (Augmentation), and 61.0 on VisDA (Generation++). The reported pattern is a consistent trade-off: unknown accuracy rises (for example, 4.5 points on Office-Home for Original) at the cost of small drops in known accuracy (1.7 points), which the authors frame as tightening boundaries at the expense of a little closed-set precision.","pith_inferences":["The threshold oracle is validated only on Office-31, so a natural test is to measure the precision of X (true-unknown fraction) on Office-Home and VisDA; if it is much lower, the reported gains should shrink accordingly.","Because the GAN-generated images look semantically meaningless in the paper's own visualization, the Generation advantage may come from extra training iterations or the entropy/agreement constraints rather than from realistic synthetic negatives; an ablation using random noise as negatives could separate these effects.","The consistent pattern of higher Acc_Lunk at the cost of Acc_Ls suggests the method widens the rejection region rather than learns richer features; a decision-boundary distance analysis in feature space would test this interpretation.","If the approach is truly method-agnostic, applying the same Lneg_ova logic to a newer baseline such as MLNet should transfer the gain; that is a direct testable extension."],"forward_implications":["On Office-Home, the Original strategy raises average unknown accuracy by 4.5 percentage points while giving up 1.7 points of known accuracy, keeping H-score essentially unchanged.","On VisDA, the Generation strategy lifts unknown accuracy by 4.9 points over the OVANet reproducibility baseline, and Generation++ achieves the best average H-score (61.0) among the proposed variants.","Because the extraction and loss are defined on top of OVANet only through its open-set confidence outputs, the same negative-constraint scheme could be applied to any OSDA method that produces per-class open-set scores.","The ablation shows robustness to the penalty weight lambda across {0.01, 0.05, 0.1, 0.2}, with lambda=0.20 marginally best, and the openness analysis on Office-31 shows roughly stable metrics as the number of unknown classes varies.","The UMAP visualization indicates that Generation++ pushes sparsely distributed unknown samples toward known-class boundaries, which is the geometric effect the loss is designed to produce."],"supporting_citations":[{"why":"Supplies the base method OVANet, its one-vs-all open-set head, and the loss that Lneg_ova extends.","marker":"(Saito & Saenko, 2021)"},{"why":"Supplies the threshold-based extraction of high-confidence unknown samples from the target domain.","marker":"(Rakshit et al., 2020)"},{"why":"Provides the idea of using a GAN to generate negative samples for open-set domain adaptation, which the Generation strategy builds on.","marker":"(Baktashmotlagh et al., 2022)"},{"why":"Motivates that unknown samples are informative and can be used to improve OSDA learning.","marker":"(Liu et al., 2022)"},{"why":"Provides the closed-set/open-set relationship hypothesis that unknown exploitation can tighten boundaries.","marker":"(Vaze et al., 2022)"},{"why":"Introduces the H-score metric used to evaluate the known versus unknown accuracy trade-off.","marker":"(Bucci et al., 2020)"},{"why":"Supplies the MLNet results and comparison protocol used for state-of-the-art positioning.","marker":"(Lu et al., 2024)"},{"why":"Supplies the NCAL results used as a recent state-of-the-art comparison.","marker":"(Su et al., 2023)"},{"why":"Preliminary version of this work that established the Original and Augmentation strategies.","marker":"(Silva et al., 2023)"}],"fun_headline_variants":["Unknown samples as negatives sharpen OSDA class boundaries","Three negative strategies lift OVANet's unknown-class accuracy","GAN negatives and augmentation boost OSDA unknown recognition","Tighten boundaries with negatives, trade known for unknown accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that OVANet's own open-set confidence threshold, 1 - po(y_hat|x) > 0.9, selects almost only true unknown samples from the target domain, a property demonstrated only on Office-31 and assumed to transfer to Office-Home and VisDA.","fun_headline_variants_meta":{"raw":{"variants":["Unknown samples as negatives sharpen OSDA class boundaries","Three negative strategies lift OVANet's unknown-class accuracy","GAN negatives and augmentation boost OSDA unknown recognition","Tighten boundaries with negatives, trade known for unknown accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1576,"prompt_tokens":1017,"completion_tokens":559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":633,"tokens_out":559,"duration_ms":5614,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:02:01.717075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the fraction of true unknown-class samples in the extracted set X for each Office-Home and VisDA task; if that precision is substantially below the Office-31 level (for example, below 80 percent), the mechanism's reported benefit should degrade, since known samples in X would push the penalty to reject known categories.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base method OVANet, its one-vs-all open-set head, and the loss that Lneg_ova extends."},{"cited_title":"S., Banerjee, B., Roig, G., & Chaudhuri, S","cited_arxiv_id":null,"evidence_quote":"Supplies the threshold-based extraction of high-confidence unknown samples from the target domain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the idea of using a GAN to generate negative samples for open-set domain adaptation, which the Generation strategy builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates that unknown samples are informative and can be used to improve OSDA learning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the closed-set/open-set relationship hypothesis that unknown exploitation can tighten boundaries."},{"cited_title":"R., & Tommasi, T","cited_arxiv_id":null,"evidence_quote":"Introduces the H-score metric used to evaluate the known versus unknown accuracy trade-off."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Preliminary version of this work that established the Original and Augmentation strategies."}],"review_version":1}