{"id":"1ea2d470-3c00-44b7-b5e2-3e349d6e8f71","arxiv_id":"2509.05564","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An active-learning loop that uses GPT-4o-mini to label the item pairs a classifier is most unsure about boosts out-of-distribution accuracy by up to 37% but barely helps or even hurts in-distribution.","lead":"KARL is an active-learning loop that uses GPT-4o-mini to label the item pairs a classifier is most unsure about, expanding a small expert-labeled dataset. In out-of-distribution tests it raised macro-F1 by up to 37%, while in-distribution gains stayed below 0.5% and extra rounds could reduce accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD test pairs are not shown to be excluded from the unlabeled pool P_U, so the 37% gain may be label leakage; this is the load-bearing assumption.","rationale":"The paper's strongest claim is the OOD improvement, and the causal story is that diversity from uncertainty sampling expands knowledge into an unexplored feature space. The most direct way this story can be false is if the OOD test set is not actually unseen: the unlabeled pool from which KARL samples is defined over the same item categories and the same ASKUL source as D_H_ood, and the paper never excludes D_H_ood pairs from P_U. Since the OOD test pairs are constructed by the same one-query-per-category pairing procedure used in Step 1, the default assumption should be overlap until shown otherwise. This is not a disagreement with the field's consensus; it is an unstated and untested data-hygiene assumption. The reader identified exactly this as the weakest assumption, and I agree. The CONDITIONAL verdict is appropriate: the paper should be accepted only if the overlap check comes back empty or the results survive deletion of overlapping pairs. I therefore recommend UNCHANGED rather than moving the verdict. A secondary concern is the absence of error bars and code, but that is not needed to decide the main threat.","tokens_in":8875,"tokens_out":5477,"duration_ms":47595,"concrete_test":"Compute the overlap between the accumulated LLM-labeled set D_L (all pairs adopted across 20 rounds, for each sampling method) and D_H_ood. Because P_U is all possible same-category pairs from the same source, report the intersection size. If non-empty, rerun the OOD evaluation after deleting every overlapping pair from D_L (or excluding them from P_U before sampling) and report macro-F1 after 20 rounds for Random, QBC, and Margin. If the up-to-37% gain drops substantially, the headline OOD effect is contamination; if it persists, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central OOD claim (up to 37% macro-F1 gain) is secure only if the LLM-labeled training pairs added by KARL are disjoint from the 2,790 OOD test pairs. The paper does not state or enforce this. Section 3 Step 1 defines P_U as all possible unlabeled item pairs within the same categories, and Section 4.1 says P_U is drawn from the same ASKUL e-commerce source as the FBL datasets. D_H_ood was built by sampling one query item per fine-grained category and pairing it with another item based on BBLs; Step 1 uses the same category-bounded pairing procedure. Thus every OOD test pair is a member of the natural universe P_U unless explicitly removed. If any such pair is selected by uncertainty sampling and LLM-annotated into D_L, the OOD test set is directly seen during training, inflating the reported improvement and the diversity-accuracy correlation. The ID result is less vulnerable to this particular leak because ID test folds come from D_H_id and the observed degradation is inconsistent with large-scale leakage, but the OOD numbers are fully consistent with contamination. This is a load-bearing threat to the paper's central empirical contrast.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KARL, a framework that iteratively expands a small human-annotated function-based-label (FBL) dataset by selecting uncertain item pairs via active learning, labeling them with GPT-4o-mini under a three-way consistency protocol, and retraining a bagged logistic-regression classifier on the augmented data. The authors evaluate KARL on two real-world e-commerce datasets: in-distribution (ID) accuracy via nested cross-validation on the human-annotated set D_H_id, and out-of-distribution (OOD) accuracy by training on D_H_id and testing on D_H_ood. They report marginal ID gains (below 0.5% macro-F1) with eventual degradation, and OOD gains of up to 37% macro-F1, with uncertainty-based sampling (QBC, Margin) outperforming random sampling. They attribute the contrast to training-data diversity, analyzed via scatter plots of diversity gain versus accuracy gain.","tokens_in":9119,"tokens_out":5400,"duration_ms":51830,"significance":"If the empirical claims are sound, the paper makes a useful practical contribution: it shows a relatively cheap LLM-based active-learning loop can substantially improve complementary-relation classification in an unfamiliar item feature space, while also documenting an important failure mode in ID settings. The public FBL datasets and the comparison of three sampling strategies are strengths, and the OOD/ID contrast is a falsifiable and interesting finding. However, the central OOD claim rests on data-hygiene assumptions that are not verified, and the supporting analyses lack statistical safeguards. The significance is therefore conditional on a clean re-evaluation.","major_comments":[{"comment":"The OOD evaluation is not shown to be leak-free, and this is load-bearing for the 37% improvement claim. Section 3 defines P_U as all possible unlabeled item pairs within the same categories, and Section 4.1 states that P_U consists of item pairs from the same e-commerce source as the FBL datasets. D_H_ood was built by sampling one query item per fine-grained category and pairing it with another item based on BBLs, which is exactly the kind of category-bounded pairing used to construct the candidate pool in Step 1. The paper never states that the 2,790 D_H_ood pairs are removed from P_U before uncertainty sampling. If any LLM-labeled pair coincides with a test pair, the OOD macro-F1 gain is inflated by direct label leakage. The same concern applies to the ID nested-cross-validation folds: P_U could contain D_H_id test-fold pairs, although the observed ID degradation makes large-scale contamination there less plausible. The authors must explicitly state and enforce disjointness of P_U from all test pairs, and rerun the experiments if the exclusion was not already in place.","section":"Section 3, Step 1; Section 4.1"},{"comment":"The quality of LLM labels on the actively selected uncertain pairs is not validated against human ground truth. Step 3 adopts only pairs for which three independent LLM labels agree; this is a self-consistency check, not a correctness check. The prior validation of GPT-4o-mini as an FBL annotator cited from [32] was performed on a different sample and does not cover the low-confidence, decision-boundary pairs that KARL actively selects. Because these are precisely the pairs on which the classifier is most uncertain, the LLM may be least reliable there. The paper should report agreement between LLM labels and human labels on a held-out subset of the actively selected pairs, or otherwise quantify the label-error rate. Without this, the claim that KARL 'augments knowledge' rather than injecting biased labels is not fully supported.","section":"Section 3, Step 3; Section 4.1"},{"comment":"The diversity-accuracy analysis does not provide the statistical support needed for the paper's causal narrative. The text states that OOD diversity gain exhibits a 'strong and consistent positive correlation' and that ID diversity is 'counterproductive', but no correlation coefficients, confidence intervals, or significance tests are reported. Moreover, the 200 points in each panel are not independent: they arise from five outer folds, twenty loop rounds, and ten bagging models, so a naive Pearson correlation over all points would overstate evidence. A hierarchical or blocked analysis is needed to account for this dependence. The authors should report effect sizes with intervals and, ideally, a test of whether the ID and OOD slopes differ.","section":"Section 4.4, Figures 4 and 5"},{"comment":"The main experimental figures report single averaged curves without error bars or repeated-run variance. Random sampling and bagging with random undersampling are stochastic, and the claimed 'up to 37%' improvement and the relative ordering of QBC/Margin versus Random could change across seeds. The paper should report means and standard deviations over multiple independent runs, or at least include confidence bands for the macro-F1 curves. This is particularly important because the central message is the size of the OOD gain and the ID degradation, not just the sign of the effect.","section":"Figures 2 and 3; Section 4.2-4.3"}],"minor_comments":[{"comment":"The phrase 'with prolonged learning could degrade accuracy' is grammatically incomplete; it should be rephrased, for example as 'and prolonged learning degraded accuracy'.","section":"Abstract"},{"comment":"The sentence 'ambiguous samples from uncertainty sampling acted as “noise” that disrupted the model’s stable distribution of the model' contains a duplicated phrase; remove 'of the model'.","section":"Section 4.2"},{"comment":"The comparison 'final accuracy achieved by the uncertainty-based methods was up to 6.6% higher than that of Random' should state whether this is an absolute or relative difference in macro-F1.","section":"Section 4.3"},{"comment":"The text says the diversity gain produces an improvement 'of up to approximately 50%', while the abstract reports up to 37% in OOD macro-F1. These numbers should be reconciled or the different metrics explicitly distinguished.","section":"Section 4.4"},{"comment":"The number of pairs added per active-learning round is not stated precisely; Step 2 selects one pair per fine-grained category, but the actual number of categories with available candidates is not reported. This makes the total annotation cost difficult to assess.","section":"Section 3, Step 1"},{"comment":"The notation for the diversity metric writes rho as R^d x R^d -> [-1,1], but the formula applies rho to pairs of feature vectors within the sample; the domain/signature should be clarified.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is short and the central empirical claim is interesting, but the evaluation hygiene issue in Section 3/Section 4.1 is the kind of problem that a workshop reviewer might miss. If the authors can demonstrate that P_U is disjoint from the test sets and provide the requested statistical and label-quality validations, the contribution could be acceptable. I also note that the paper relies heavily on the authors' own prior datasets and prompts [24,32]; this is not inappropriate, but the novelty relative to those papers should be made clearer in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the paper reports a 37% OOD gain from LLM-based active learning for complementary recommendation labels, but that number is not trustworthy because the unlabeled pool P_U is defined as all possible item pairs in the same categories, and the OOD test set is drawn from the same source and categories without any stated exclusion. If any test pair got LLM-labeled and added to training, the gain is leakage. This is the load-bearing issue.\n\nWhat's new: the framework KARL is a straightforward but reasonable combination: uncertainty sampling (QBC, margin) from an unlabeled pair pool, LLM annotation with a three-consistency protocol, and retraining with bagged logistic regression. The novel empirical claim is the asymmetry: adding diverse LLM-labeled pairs helps in OOD (unfamiliar feature space) but hurts in ID. That is worth examining.\n\nWhat's good: the paper is clearly written, uses a real-world dataset (ASKUL), builds on the authors' prior FBL datasets, and the baseline setup is sensible. The ID degradation result, even if not the headline, is a useful caution about over-annotation.\n\nSoft spots: (1) The leakage risk is real and unaddressed. Section 4.1 says P_U comes from the same e-commerce source as the FBL datasets; Section 3 Step 1 says P_U is all possible item pairs in the same categories. The OOD test set was built by sampling one query item per fine-grained category and pairing it with another item—exactly the kind of pair in P_U. Without an explicit deduplication step, the 37% improvement could be inflated by direct label leakage. (2) No error bars, no statistical tests for the diversity-accuracy correlation; the paper shows scatter plots and asserts a \"strong and consistent positive correlation\" without reporting correlation coefficients or significance. (3) No code released for the KARL pipeline, only the FBL datasets. (4) The paper frames the diversity effect as causal, but it only observes correlation.\n\nThe math itself is fine—logistic regression with bagging is straightforward. The citation pattern is appropriate: the active learning + LLM work is cited, and the self-citations to the authors' FBL/LLM annotation papers are justified since those are the dataset and prompt source.\n\nWho this is for: the complementary-recommendation subfield, especially researchers interested in label-efficient training with LLMs. A reasonable workshop paper if the leakage concern is resolved.\n\nRecommendation: send to peer review, but with a major-revision requirement: the authors must explicitly state that D_H_ood (and D_H_id) test pairs are excluded from P_U, or re-run with that exclusion. If the results hold, the paper earns its place. As is, the central claim is unsupported.","headline":"The 37% OOD gain is likely inflated because test pairs are not shown to be excluded from the unlabeled pool P_U; the framework is sensible, but the central claim needs a deduplication guarantee.","tokens_in":9644,"tokens_out":3070,"would_cite":false,"duration_ms":28033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-labeling active-learning loop lifts out-of-distribution complementary-recommendation accuracy by up to 37 percent.","keywords":["complementary recommendation","function-based labels","active learning","large language models","LLM annotation","uncertainty sampling","out-of-distribution generalization","e-commerce"],"falsifier":"Take the item pairs KARL selected and LLM-labeled during all 20 rounds, intersect them with the 2,790 out-of-distribution test pairs, and rerun the OOD evaluation with any overlapping pairs removed from the training labels. The 37 percent improvement should survive that removal; if it shrinks substantially, the reported gain is partly label leakage.","tokens_in":8688,"feed_emoji":"🛒","tokens_out":12604,"duration_ms":101013,"temperature":0.7,"pith_summary":"Complementary recommendation asks whether two products belong together, and the most reliable training labels for that question—function-based labels built on expert descriptions of item functions—are too expensive to produce for a broad catalog. This paper claims that a small expert-labeled seed can be expanded cheaply by an active-learning loop: the classifier proposes the item pairs it finds hardest to classify, and a large language model writes labels for those pairs. On an out-of-distribution test set, item categories and feature combinations the classifier had not been trained on, the loop improved the average per-category F1 score by up to 37 percent over the unaugmented baseline. On an in-distribution test set, drawn from the same feature space the model already knew, the gain was less than half a percent and continued training eventually degraded accuracy. The paper attributes the split to training-data diversity: uncertainty-driven expansion adds useful new knowledge in unfamiliar spaces but acts as noise that disturbs a well-learned distribution.","feed_headline":"LLM active learning lifts accuracy on unfamiliar items by 37%","feed_subtitle":"The classifier's hardest item pairs get LLM labels, helping new categories but not familiar ones.","key_machinery":"The machinery is KARL's four-step loop. Starting from a logistic-regression classifier trained on human function-based labels, the loop (1) samples candidate item pairs from a large unlabeled pool with category-stratified two-stage sampling; (2) scores each candidate pair by uncertainty, using either uniform random scores, query-by-committee variance across ten bagged classifiers, or the margin between the top two predicted classes, and selects the most uncertain pair per fine-grained category; (3) asks a large language model to classify each selected pair into the nine function-based categories, accepts the pair only if three independent LLM runs agree, and maps the nine categories to the three classes complementary, substitute, and unrelated; and (4) retrains the bagged logistic-regression ensemble on the human labels plus all accepted LLM labels. The explanatory metric is training-data diversity, defined as one minus the mean absolute Pearson correlation between the feature vectors of training pairs, which the paper plots against accuracy gain to make the ID/OOD contrast visible.","core_discovery":"KARL's central empirical claim is that the value of LLM-expanded training data depends on whether the model already knows the item feature space. In the out-of-distribution setting, KARL's uncertainty-based variants improved macro-F1 by up to 37 percent over the baseline, reached a given accuracy in fewer rounds than random sampling, and peaked 6.6 percent higher than random sampling. In the in-distribution setting, the same procedure improved accuracy by at most 0.5 percent and then degraded it, with the degradation most severe for the uncertainty-based samplers. The paper shows a consistent positive correlation between the diversity of the training pairs and accuracy gain in the out-of-distribution case, and a flat or negative relationship in the in-distribution case, and concludes that diversity is a knowledge-expansion mechanism in unexplored feature spaces and a distribution-disruption mechanism in learned ones.","pith_inferences":["A dual-mode policy could automate the ID/OOD distinction: estimate distribution shift from classifier confidence or feature-space distance and invoke the LLM expansion loop only outside the learned region, the adaptive strategy the paper sketches as future work.","Because diversity, not uncertainty, was the measured driver of OOD gains, an explicit diversity-maximizing sampler such as farthest-point selection in feature space is a natural alternative to uncertainty sampling that could be tested on the same two datasets.","The three-consistency label gate could be supplemented with a small human audit on newly added categories, turning the LLM's self-agreement into measured agreement against expert labels.","Before the 37 percent figure is adopted as a benchmark, a split audit should verify that no LLM-labeled training pair coincides with an out-of-distribution test pair; the paper does not report that exclusion."],"forward_implications":["Out-of-distribution generalization of a complementary-recommendation classifier can be improved by up to 37 percent without new expert annotation, by adding LLM labels for the pairs the current model finds least certain.","Uncertainty sampling is more cost-efficient than random sampling in unexplored item spaces: it reaches a given accuracy in fewer LLM-annotation rounds and ends at a higher peak accuracy.","Running the same expansion loop on an already well-covered item space is counterproductive, so a deployment should stop adding LLM-labeled pairs once in-distribution accuracy peaks.","The positive diversity–accuracy correlation in the OOD setting indicates that data diversity itself, not simply more data, carries the generalization gain.","The baseline classifier's 0.44 OOD macro-F1 shows that a model trained on one curated category set cannot be assumed to transfer to new item features without knowledge augmentation."],"supporting_citations":[{"why":"Introduces function-based labels and the nine functional categories that KARL's prompt and label mapping are built on.","marker":"[24]"},{"why":"Supplies the FBL datasets, the Bayesian-optimized logistic-regression setup, the 424-dimensional features, and the LLM-as-judge prompt KARL reuses.","marker":"[32]"},{"why":"Documents the GPT-4o family model used as the LLM annotator in Step 3.","marker":"[18]"},{"why":"Grounds the active-learning loop and the uncertainty-sampling principle.","marker":"[22]"},{"why":"Defines query-by-committee, one of the two uncertainty samplers compared in Step 2.","marker":"[23]"},{"why":"Defines margin-based sampling, the other uncertainty sampler compared.","marker":"[26]"},{"why":"Motivates the three-run consistency protocol that filters LLM labels before they enter training.","marker":"[19]"},{"why":"Supplies the repeated nested cross-validation design used for ID accuracy evaluation.","marker":"[8]"},{"why":"Supplies bagging, the ensemble method used to handle class imbalance in the accumulated LLM labels.","marker":"[3]"}],"fun_headline_variants":["LLM labels boost unfamiliar item accuracy 37%, but hurt familiar ones","Active learning with LLMs: 37% gain on new items, tiny loss on known","Why LLM-expanded training data helps unseen categories, not seen ones","Complementary recommendation: LLM labels lift OOD accuracy 37%, degrade ID","KARL: LLM active learning yields 37% OOD boost, ID stagnation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 37 percent OOD gain assumes that the LLM-labeled training pairs never include an out-of-distribution test pair; the paper does not state that its candidate pool excluded the 2,790 test pairs.","fun_headline_variants_meta":{"raw":{"variants":["LLM labels boost unfamiliar item accuracy 37%, but hurt familiar ones","Active learning with LLMs: 37% gain on new items, tiny loss on known","Why LLM-expanded training data helps unseen categories, not seen ones","Complementary recommendation: LLM labels lift OOD accuracy 37%, degrade ID","KARL: LLM active learning yields 37% OOD boost, ID stagnation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1477,"prompt_tokens":979,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":595,"tokens_out":498,"duration_ms":4678,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:22:41.038761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the item pairs KARL selected and LLM-labeled during all 20 rounds, intersect them with the 2,790 out-of-distribution test pairs, and rerun the OOD evaluation with any overlapping pairs removed from the training labels. The 37 percent improvement should survive that removal; if it shrinks substantially, the reported gain is partly label leakage.","supporting_citations":[{"cited_title":"Yamasaki, K","cited_arxiv_id":null,"evidence_quote":"Supplies the FBL datasets, the Bayesian-optimized logistic-regression setup, the 424-dimensional features, and the LLM-as-judge prompt KARL reuses."},{"cited_title":"2009.Active Learning Literature Survey","cited_arxiv_id":null,"evidence_quote":"Grounds the active-learning loop and the uncertainty-sampling principle."},{"cited_title":"Tong and D","cited_arxiv_id":null,"evidence_quote":"Defines margin-based sampling, the other uncertainty sampler compared."},{"cited_title":"Filzmoser, B","cited_arxiv_id":null,"evidence_quote":"Supplies the repeated nested cross-validation design used for ID accuracy evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies bagging, the ensemble method used to handle class imbalance in the accumulated LLM labels."}],"review_version":2}