{"id":"a72c64c6-1bea-4678-bd07-1688c98b0202","arxiv_id":"1908.08142","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Conditional entropy between label sequences predicts task transferability and hardness without training any models, with strong correlations on CelebA, AwA2, and CUB.","lead":"Researchers show how hard a classification task is and how well models transfer between tasks can be estimated just from the labels of the training data, without training any models. Their estimates use conditional entropy between label sequences and are verified on 437 real-world attribute classification tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transferability results are not controlled for target marginal entropy H(Y); since H(Y|Z)=H(Y)-I(Y;Z), reported correlations may be driven by target hardness rather than by the source-target relationship, leaving the central transferability claim unproven.","rationale":"The paper's two headline claims are intertwined: H(Z|C) estimates hardness and H(Y|Z) estimates transferability. The hardness estimate is literally the marginal entropy H(Z), because C is constant. The transferability estimate decomposes as H(Y)-I(Y;Z). In the fixed-source protocol, if target-task errors are driven mostly by class balance/difficulty, then H(Y|Z) inherits that signal and the source-specific part may be irrelevant. The paper does not report the relevant baselines or ablations, so the strongest interpretation - that conditional entropy between two tasks' labels measures their relationship - is not yet established. This is not an internal inconsistency, but it is a correctness risk for the paper's central novelty. The reader's concern about the training-loss-to-test-error bridge and the mismatch between the theoretical classifier k-bar and the experimental lSVM is also real and secondary; the theoretical bound does not directly apply to the measured test errors. However, even if that bridge were repaired, the missing H(Y) control would remain. The proposed test is inexpensive because all quantities can be recomputed from the training label tables and the reported test errors. If the control shows that H(Y) alone predicts as well, the paper becomes a hardness-prediction paper, not a transferability-prediction paper. If the control shows that I(Y;Z) adds significant predictive power, the central claim is supported. Given this uncertainty, the conditional verdict is appropriate.","tokens_in":30298,"tokens_out":7231,"duration_ms":73636,"concrete_test":"For each source-target pair in Figs. 2, 7-10 and Table 3, compute (i) the Pearson correlation between target test error and H(Y) alone, and (ii) the partial correlation between H(Y|Z) and test error controlling for H(Y). Equivalently, compare the reported H(Y|Z) correlations against those obtained with source labels randomly permuted, so that only H(Y) remains in the conditional entropy. If the H(Y)-only or shuffled-label correlations match the reported H(Y|Z) correlations within sampling error, the transferability-specific component I(Y;Z) is not doing the work, and the central claim should be weakened to a hardness claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that H(Y|Z) estimates transferability from a fixed source Z to targets Y (Sec. 5.1, Figs. 2, 7-10). But H(Y|Z) = H(Y) - I(Y;Z). For binary attribute pairs, I(Y;Z) is often small relative to H(Y), so H(Y|Z) is dominated by the target marginal entropy H(Y) - which is exactly the paper's own hardness estimate: for a constant C, Eq. (14) gives H(Z|C) = H(Z). The paper separately shows that H(Z) correlates strongly with dedicated test error (Fig. 6, r=0.58-0.96). No baseline or partial correlation controlling for H(Y) is reported in the transferability experiments. Without such a control, the high correlations in Fig. 2 could be reproduced by a source-agnostic predictor using only target label entropy; the results would then validate hardness prediction, not transferability. The identity-to-attribute case study (Sec. 5.2) provides one clear example where CE differs from H(Y) (e.g., 'male' has high hardness but low CE from identity), but this does not quantify how much of the overall correlation is source-specific.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-theoretic, solution-agnostic way to estimate the transferability and hardness of supervised classification tasks defined over the same input instances. Transferability from a source task T_Z to a target task T_Y is estimated by the conditional entropy H(Y|Z) of the two training label sequences, and task hardness is estimated by H(Z|C), the conditional entropy from a constant-label trivial task. The main theoretical result, Theorem 1, lower-bounds the target training log-likelihood of a transferred representation by the source training log-likelihood minus H(Y|Z). The paper validates these estimates on 437 attribute classification tasks from CelebA, AwA2, and CUB, reporting strong Pearson correlations between the proposed estimates and empirically measured test errors, plus a case study transferring a face recognition network to CelebA attribute prediction.","tokens_in":30576,"tokens_out":6803,"duration_ms":69507,"significance":"If the central claim holds, the paper offers a remarkably cheap and model-agnostic predictor of transferability and hardness, computable from label counts alone, in contrast to model-zoo approaches such as Taskonomy or Task2Vec. The proof of Theorem 1 in Appendix A is elementary and correct, and the empirical scale is substantial: all 437 transferability scatter plots in the appendix show high correlations, and the identity-to-attribute case study is a practically relevant demonstration. The main risk is interpretive: the proposed transferability measure H(Y|Z) contains the target marginal entropy H(Y), which is the paper's own hardness measure, and the theoretical bound concerns training log-likelihood of an augmented classifier space, while the experiments measure test error of lSVMs. The paper therefore needs additional controls and a clearer statement of what is proven versus what is empirically suggested.","major_comments":[{"comment":"Theorem 1 is a correct lower bound on the training log-likelihood lY(wZ,kY) of the transferred model, but the experiments in §5.1 measure target test error. The only bridge is the informal statement in §3.1 that 'if the trained model does not overfit, the log-likelihood on the training set ... provides a good indicator of Eq. (4)', and Discussion 3 extends this to test log-likelihood only under an unproven bounded-difference assumption. Moreover, the theorem's bound holds for kY chosen from K after augmenting with the constructed classifier \\bar{k}, whereas the experiments use a linear SVM; no argument is given that the lSVM's log-likelihood attains at least the bound. Consequently, the analytical result as stated does not cover the evaluation protocol used to support the paper's main empirical claim. Please prove a test-error analogue under explicit assumptions, or present the empirical results as heuristic validation and add a direct check, such as comparing lSVM training log-likelihood with the right-hand side of Eq. (10), to show that the missing link holds in practice.","section":"§3.1, Eq. (5), §5.1"},{"comment":"The quantity being correlated with transferred test error is H(Y|Z) = H(Y) - I(Y;Z). For the binary attributes used in these experiments, I(Y;Z) is often small relative to H(Y), so H(Y|Z) is dominated by the target marginal entropy H(Y), which is exactly the paper's own hardness estimate (for a constant C, H(Z|C) = H(Z) in Eq. (14); Fig. 6 shows that H(Y) already strongly predicts dedicated test error). The high correlations in Fig. 2 may therefore reflect target hardness rather than the source-target relationship. No source-agnostic baseline, such as predicting transferred error from H(Y) alone, and no partial correlation controlling for H(Y), are reported. The identity-to-attribute case study in Sec. 5.2 provides one example where CE and H(Y) diverge (male has high hardness but low CE from identity), but it does not quantify the fraction of the overall correlation that is source-specific. Please add the baseline and partial-correlation analysis; if CE does not add predictive value over H(Y), the central transferability claim is not supported.","section":"§5.1, Fig. 2, Eq. (7)"},{"comment":"Eq. (14) is an upper bound on hardness, not an equality or a proof of monotonicity: Hard(T_Z) ≤ H(Z|C) does not by itself imply that a task with larger H(Z|C) is harder. The empirical correlation in Fig. 6 is therefore the only support for using H(Z|C) as a hardness ordering, and the paper should state this explicitly. Additionally, because H(Z|C) equals the marginal label entropy H(Z) for a constant C, the measure is a label-balance statistic that ignores the input domain entirely. The text should acknowledge this and compare against a simple majority-class or label-entropy baseline for the dedicated errors, to separate the contribution of the information-theoretic framing from this well-known cue.","section":"§4, Eq. (14), Fig. 6"}],"minor_comments":[{"comment":"The derivation of Eq. (14) is only sketched; please add the short argument that optimality of (wZ,hZ) for T_Z gives Hard(T_Z) ≤ -lZ(wC,kZ), which combined with Theorem 1 and zero training loss of the trivial source yields the bound.","section":"Appendix B"},{"comment":"The paper reports correlations for selected source tasks in Fig. 2 and says full results are in the appendix, but it does not give a summary statistic across all source tasks. Please report the mean, median, minimum, and maximum correlation over the 437 transferability conditions.","section":"§5.1"},{"comment":"The p-values treat the target tasks as independent observations, but the 39 or 84 target tasks share the same images and are not statistically independent. A permutation test or a mixed-effects model would be more appropriate for assessing significance.","section":"§5.1, Fig. 2"},{"comment":"The sentence 'We then compare these transferability scores to the corresponding CE estimates ... using an existing correlation analysis [44]' cites a continual-learning preprint; the standard Pearson correlation analysis should be described directly rather than through that reference.","section":"§5.1"},{"comment":"The notation 'CE↑(decreasing transferability)' and 'Conditional Entropy↑' is unclear; please state that attributes are sorted by ascending CE and explain the arrow convention in the caption.","section":"Table 1 and Appendix D"},{"comment":"There are minor typos and formatting issues, including 'tranferability' in Sec. 3.2 and inconsistent capitalization of Softmax; these should be corrected in a revision.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the machine learning and computer vision communities if the source-specificity of the transferability measure is established. The key concern is that H(Y|Z) may be a proxy for target hardness H(Y), and the current experiments do not rule this out. I do not see evidence of misconduct, but the framing of the empirical results as validating Theorem 1 overstates the logical connection between the theorem and the lSVM test-error protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new and cheap label-statistics measure for transferability, with a correct bound behind it, but the experiments do not rule out the possibility that the correlations are mostly about target hardness rather than the source-target relation.\n\nThe novel bit is conditional entropy between label sequences as a transferability estimate. That is a clean idea and distinct from Taskonomy and Task2Vec, which need trained models. Theorem 1 is a correct lower bound on the target training log-likelihood of a transferred model; the proof is straightforward. The empirical scale is real: 437 tasks across CelebA, AwA2, and CUB, and the correlations in Fig 2 are high. The identity-to-attribute case study (Fig 3, Table 1) is a practical demonstration that the measure can rank attribute transferability from a face recognition model without training attribute models.\n\nNow the soft spots. The stress-test concern is legitimate: H(Y|Z) = H(Y) - I(Y;Z). When the source is fixed and targets vary, H(Y|Z) is dominated by the target marginal entropy H(Y), which is exactly the paper's own hardness estimate. The paper shows H(Y) correlates strongly with dedicated test error (Fig 6). So the high correlations in Fig 2 could largely be explained by target hardness, not by any source-specific relationship. No partial correlation or baseline using H(Y) alone is reported. Without that control, the central claim—that CE measures transferability from a specific source—is not fully proven. The case study helps, because there CE differs sharply from H(Y) (male has low CE from identity but high hardness), but it does not quantify how much of the overall correlation is source-specific.\n\nSecond, the theory bounds training log-likelihood, while experiments use test error. The paper explicitly assumes non-overfitting, which is reasonable for the lSVM protocol but is not part of the theorem. Third, the hardness component reduces to label entropy; that is not a new result, and calling it an estimate of hardness via Theorem 1 is an upper bound treated as a predictor.\n\nNone of this is fatal. The measure is cheap and the large-scale correlations show it carries real signal. The paper just overclaims when it says it predicts transferability without showing that it does better than target hardness alone. A referee should ask for a partial correlation controlling for H(Y), or a comparison of CE against H(Y) as a predictor, and the paper would be the stronger for it.\n\nBottom line: this is a useful, honest paper that deserves serious peer review. I would bring it to reading group and would cite it if I worked on source-task selection. With the confound addressed, it becomes a solid practical contribution.","headline":"A label-only conditional entropy measure for task transferability is a genuinely useful idea, but the experiments need to control for target hardness before the core claim is proven.","tokens_in":31059,"tokens_out":2632,"would_cite":true,"duration_ms":25638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Label entropy bounds transferability between tasks","keywords":["conditional entropy","transferability","task hardness","information theory","transfer learning","label statistics","classification","model-free estimation"],"falsifier":"Take two source-target task pairs with identical empirical joint label distributions (the same counts of each $(z,y)$ combination) but different assignments of those label pairs to individual images—for example, swap which images carry the $(z=0,y=1)$ label between two tasks. The conditional entropy estimate is identical, but actual transferability can diverge because the representation learned from $Z$ may separate the input space along the wrong boundary for one target task. An experiment showing large transfer-error differences between such paired tasks, despite equal $H(Y|Z)$, would refute the claim that the entropy alone orders transferability.","tokens_in":30109,"feed_emoji":"🔁","tokens_out":8129,"duration_ms":71153,"temperature":0.7,"pith_summary":"This paper claims that how well a model trained on one task transfers to another can be predicted in advance from the training labels alone, without training any model. The key quantity is the conditional entropy $H(Y|Z)$ of the target labels given the source labels, which lower-bounds the target log-likelihood of a transferred model (Theorem 1). The same idea estimates task hardness: a task is hard to the extent that its labels have high conditional entropy given a trivial constant label. If correct, this gives a cheap, model-free way to rank tasks and plan transfer learning. The claim is tested on 437 attribute-classification tasks across three image benchmarks, where the entropy estimates correlate strongly with measured transfer errors and hardness.","feed_headline":"Label entropy bounds transferability between tasks","feed_subtitle":"Validated on 437 classification tasks, the label-only estimate ranks transferability and hardness without training any model.","key_machinery":"The central object is the empirical conditional entropy $H(Y|Z)$ of two label sequences, computed by counting how often each source label $z$ co-occurs with each target label $y$ in the training set. It measures, in bits, how much uncertainty about the target label remains after the source label is known. The proof of the bound uses a specially constructed classifier $\\bar{k}$ that converts the source softmax output into a target distribution by taking the conditional expectation $\\sum_z \\hat{P}(y|z) p_Z(z)$; because this classifier is available to the optimization, the target log-likelihood of the actual best classifier can only be higher, yielding the inequality.","core_discovery":"The paper's central claim is Theorem 1: for two classification tasks $T_Z$ and $T_Y$ sharing the same training inputs, the transferability of $T_Z$ to $T_Y$—measured as the target training log-likelihood after transferring the representation trained on $Z$ and retraining a classifier on $Y$—is at least $l_Z(w_Z, h_Z) - H(Y|Z)$, where $l_Z$ is the source task's own log-likelihood and $H(Y|Z)$ is the conditional entropy of the label sequences. Because $l_Z$ is fixed when the source is fixed, the conditional entropy alone orders target tasks by transferability. The paper also claims that task hardness is estimated by $H(Z|C)$, the conditional entropy of the task's labels given a constant label sequence, which upper-bounds the negative log-likelihood of the optimal model. Empirically, these label-only estimates correlate strongly with measured test errors on 437 attribute tasks, and a face-recognition representation transferred to attributes ranked as most transferable matches or beats dedicated classifiers.","pith_inferences":["Editorial inference: the same conditional-entropy quantity could serve as a task-ordering criterion in continual learning, choosing the next task to minimize information loss from previously learned labels; the paper mentions continual learning only as a potential application.","Editorial inference: the bound implies that mutual information $I(Y;Z) = H(Y) - H(Y|Z)$ is the model-free component of task relatedness; this connects the entropy measure to a broader information-geometric view of task similarity.","Editorial inference: a natural stress test is to apply the measure to structured or continuous labels (segmentation maps, regression outputs) by discretizing them; the paper notes the extension but does not test it.","Editorial inference: the theory may also predict when fine-tuning the full network is necessary: targets with high $H(Y|Z)$ require changing the representation itself, not just retraining the last layer."],"forward_implications":["Transferability between any two discrete-label classification tasks sharing an input domain can be estimated by counting label co-occurrences alone, in milliseconds, without training any model or maintaining a zoo of trained networks.","Task hardness can be ranked before training, so budgets for model training and data collection can be allocated to tasks predicted to be hardest.","With a fixed source task, the conditional entropy orders all candidate target tasks before any transfer, enabling selection of targets worth transferring to and early stopping of unpromising transfers.","The estimates cover multi-class and multi-label settings, since they are computed directly on the label sequences, not on model outputs.","Following the case study, a single representation trained for a rich source task (such as face recognition) can be transferred to many target attributes, with accuracy comparable to dedicated models on targets ranked highly transferable."],"supporting_citations":[{"why":"Supplies the definition of conditional entropy used to define $H(Y|Z)$ from label sequences.","marker":"[14]"},{"why":"CelebA dataset; provides the 40 facial attribute classification tasks and identity labels used in the main transferability and hardness experiments.","marker":"[35]"},{"why":"Caltech-UCSD Birds 200 dataset; supplies 312 attribute classification tasks for hardness and transferability correlation.","marker":"[66]"},{"why":"Animals with Attributes 2 dataset; supplies 85 attribute classification tasks for empirical validation.","marker":"[67]"},{"why":"Supports the non-overfitting assumption that training log-likelihood tracks test accuracy for properly trained networks, bridging the bound to test error.","marker":"[72]"}],"fun_headline_variants":["Label entropy predicts task transferability and hardness","No models needed: label entropy ranks transfer and difficulty","Transfer loss bounded by label entropy alone","Conditional entropy on labels estimates task difficulty","Hardness and transferability from label statistics only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bound is proven for training log-likelihood, while the experiments measure test error, and the paper assumes that a well-trained model's training log-likelihood tracks its test accuracy, and that the linear SVM used in practice behaves like the specially constructed classifier in the proof.","fun_headline_variants_meta":{"raw":{"variants":["Label entropy predicts task transferability and hardness","No models needed: label entropy ranks transfer and difficulty","Transfer loss bounded by label entropy alone","Conditional entropy on labels estimates task difficulty","Hardness and transferability from label statistics only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1245,"prompt_tokens":957,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":573,"tokens_out":288,"duration_ms":3324,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:48:21.812570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two source-target task pairs with identical empirical joint label distributions (the same counts of each $(z,y)$ combination) but different assignments of those label pairs to individual images—for example, swap which images carry the $(z=0,y=1)$ label between two tasks. The conditional entropy estimate is identical, but actual transferability can diverge because the representation learned from $Z$ may separate the input space along the wrong boundary for one target task. An experiment showing large transfer-error differences between such paired tasks, despite equal $H(Y|Z)$, would refute the claim that the entropy alone orders transferability.","supporting_citations":[{"cited_title":"Elements of informa- tion theory","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of conditional entropy used to define $H(Y|Z)$ from label sequences."},{"cited_title":"Deep learning face attributes in the wild","cited_arxiv_id":null,"evidence_quote":"CelebA dataset; provides the 40 facial attribute classification tasks and identity labels used in the main transferability and hardness experiments."},{"cited_title":"Welinder, S","cited_arxiv_id":null,"evidence_quote":"Caltech-UCSD Birds 200 dataset; supplies 312 attribute classification tasks for hardness and transferability correlation."},{"cited_title":"Zero-shot learning-a comprehensive evalu- ation of the good, the bad and the ugly","cited_arxiv_id":null,"evidence_quote":"Animals with Attributes 2 dataset; supplies 85 attribute classification tasks for empirical validation."},{"cited_title":"Understanding deep learning re- quires rethinking generalization","cited_arxiv_id":null,"evidence_quote":"Supports the non-overfitting assumption that training log-likelihood tracks test accuracy for properly trained networks, bridging the bound to test error."}],"review_version":1}