{"id":"bbcee00d-8409-4629-bbf8-0e1c2e4c8e57","arxiv_id":"2501.19316","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Frozen embeddings from simple task-tuned BERT models, especially paraphrase detection, transfer useful knowledge to coreference resolution, and attention-based combination of several models outperforms single models.","lead":"BERT models fine-tuned on simple tasks (paraphrase detection, relation extraction, and others) can serve as frozen building blocks for training a coreference resolution system; some tasks transfer much better than others, intermediate layers help, and combining several frozen models with an attention layer gives the best results. This suggests a way to reuse existing task-tuned models instead of training everything from scratch.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-set selection in §4.4–§4.6 likely inflates reported transfer gains; held-out test evaluation is required to support the intermediate-layer and combination claims.","rationale":"The reader's weakest assumption captures the most load-bearing issue. The paper is transparent about its protocol and provides source code, and the core observation that frozen task embeddings can drive a coreference head is plausible; however, the validation-bias issue directly affects the headline comparisons. Every headline number in Tables 1–3 is either itself selected on the validation set (best truncation layer per model, normalization on/off, mean vs attention, 4/6/12-layer concatenation) or built from such selections. Since the paper's contribution is precisely a ranking of source tasks, layers, and aggregation strategies, those rankings need to be stable on data not used for model selection. Early stopping adds another selection step on the same split. A secondary concern, acknowledged by the authors in Section 4.3, is that the attention aggregator's gains over mean are confounded with extra trainable projections; this weakens the specific 'attention is the right aggregator' claim but not the general transfer finding. The proposed held-out test would settle the selection-bias question: if the test F1s reproduce the rankings, the conditional verdict can be upgraded, and if not, the relevant claims should be treated as exploratory. Because the reader already conditioned the verdict on this issue, the appropriate verdict remains CONDITIONAL, i.e., no change from the reader's verdict.","tokens_in":11574,"tokens_out":5267,"duration_ms":54242,"concrete_test":"Run the identical protocol but evaluate on the CoNLL-2012 test split (or a previously unused held-out partition): for each source model and each aggregation setting, select the best layer/truncation/normalization using only the development split, freeze those choices, and report the selected configurations' F1 on the test split. Compare the resulting test F1 and layer ranking with Tables 1–3. If the best-on-validation configurations lose a substantial margin on the test split, or if the ordering of layers or source tasks changes, the intermediate-layer superiority and combination-gain claims are selection artifacts. A useful secondary check is to report the maximum-over-grid validation F1 alongside the test F1 of that same grid point so that the selection bias can be quantified explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claims—that intermediate layers (e.g., layer 9 for MRPC, layer 6 for NER) outperform final layers, that truncation and normalization help, and that multi-task combinations with attention improve over single models—are supported by F1 numbers computed on the same OntoNotes validation split used to choose those settings. Section 4.4 states that after comparing layers 6 through 12, the best performing layer was selected for each source model; Tables 1–3 report the resulting 'trunc' scores with a subscript marking the chosen layer, and Section 4.1 uses early stopping on the same validation split. This is a selection-on-validation protocol: the reported score is effectively the maximum over a grid of layer choices, normalization options, and aggregation methods, rather than the expected performance of a fixed configuration. The small seed-to-seed standard deviations (0.2–3 F1) do not account for variance across the configurations considered, so the apparent superiority of a particular intermediate layer, or the +26.2% gain from truncating NER, could be partly an artifact of choosing the best of many options on the evaluation set. Because the paper's conclusions are about which embeddings transfer best, the ranking of source tasks and layers must be demonstrated on data not used for model selection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Reverse Probing introduces a probing-style framework in which frozen BERT embeddings from models fine-tuned on simpler source tasks (MRPC, NER, RE, QA) are used as token representations for coreference resolution. The paper systematically varies the extraction layer (full vs. truncated), optionally applies L2 normalization, concatenates multiple hidden layers, and compares mean versus attention-based aggregation. All experiments are evaluated on the OntoNotes validation split, with the coreference head trained on top of the frozen embeddings. The central claims are that semantic-similarity source tasks (MRPC) transfer best, intermediate layers often outperform final layers, and combining multiple source models, especially with attention aggregation, improves coreference F1 up to 69.56 versus 73.75 for a fully trained model.","tokens_in":11823,"tokens_out":4138,"duration_ms":41226,"significance":"If the empirical claims are robust, this is a useful and clearly presented contribution: it extends probing methodology toward knowledge transfer from simpler to more complex tasks, provides a systematic comparison of source tasks and embedding manipulations, and the authors publicly release their code. The paper also reports seed-averaged scores with standard deviations and includes a limitations section. However, the headline quantitative claims rest on a selection-on-validation protocol, so the reported gains and rankings are not unbiased estimates of transfer performance and need verification on a held-out test set before they can be accepted at face value.","major_comments":[{"comment":"The central claims about intermediate layers and truncation gains are based on selecting the best truncation layer on the OntoNotes validation set and then reporting F1 on that same validation set. Section 4.4 states that after comparing layers 6 through 12, the best performing layer was selected for each source model, and Tables 1–3 report the resulting 'trunc' scores with the chosen layer as a subscript. Early stopping in §4.1 also uses the validation split. This is selection-on-validation: the reported numbers are effectively maxima over a grid of layer choices, normalization choices, and aggregation methods, not the expected performance of a fixed configuration. The +26.2% NER gain from truncation and the conclusion that intermediate layers outperform final layers could therefore be inflated. The authors should evaluate the selected configurations on the held-out CoNLL-2012 test set, or use nested validation, for at least the central single-model and combination comparisons.","section":"§4.4, Tables 1–3"},{"comment":"The claim that 'combinations of two models always outperform single models' and the reported combination gains (e.g., +13.23% for NER+MRPC) are computed from configurations that were themselves selected on the validation set. The seed-level standard deviations (roughly 0.2–3 F1) capture only variation across runs of a fixed configuration, not the variance introduced by choosing among many configurations on the same data. Consequently, the reported ranking of single vs. combined models, and the ordering among combinations, may not generalize to unseen data. A held-out test evaluation, or at least a nested-validation estimate, is needed to support the combination claims.","section":"§4.6, Tables 2–3"},{"comment":"The comparison between mean and attention aggregation confounds the aggregation mechanism with added trainable parameters: attention introduces a learned projection while mean aggregation does not. The paper acknowledges this in the text ('we get by simply adding additional projections'), but the conclusion that 'attention-based aggregation yields substantial gains' is still stated as an effect of the aggregation method. To make this claim load-bearing, the authors should include a control condition with matched parameters, such as a linear projection followed by mean pooling, or explicitly frame the comparison as 'attention with extra parameters vs. mean without extra parameters.'","section":"§4.3 and §4.4"}],"minor_comments":[{"comment":"There are typos: the abstract contains 'tasks tasks' and §3.2 contains 'quesion answering'.","section":"Abstract and §3.2"},{"comment":"The text reads 'NE+MRPC' in the normalization discussion; this should be 'NER+MRPC'.","section":"§4.4"},{"comment":"The text reports BERT and MRPC scores of 64.01 and 64.32 F1 from Figure 2, while Table 1's 'full mean' column reports 64.95 and 61.16 for the same settings. The discrepancy should be clarified, for example by stating which configuration Figure 2 uses.","section":"§4.2 and Table 1"},{"comment":"The mean aggregation formula is missing the averaging factor in the displayed equation, and the notation H_s in the text omits the layer superscript used elsewhere.","section":"§2"},{"comment":"The text refers to Figures 5, 6, 7, and 8, but the figure numbering in the submission is inconsistent and some captions are incomplete; the authors should ensure all figures are numbered and captioned consistently.","section":"Figures 5–8"},{"comment":"The sentence 'RE+QA pair gains +8.17% F1 with mean aggregation and +2.84% with attention aggregation when both models are truncated' should state the baseline to which these gains are relative.","section":"§4.4"},{"comment":"The Limitations section lists model architecture, source-task coverage, and efficiency limitations, but it does not mention the validation-set selection issue; adding an explicit statement about this would help readers calibrate the reported results.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is genuine and fixable: adding a held-out test evaluation, or nested validation, for the central comparisons would substantially strengthen the paper. The current version's headline numbers should not be treated as unbiased. There are no concerns about novelty disclosure or code availability; the code release is a positive feature. With the evaluation protocol tightened, the paper could be a solid contribution to an NLP venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper asks a practical question—can you freeze BERT models fine-tuned on simpler tasks and use their embeddings for coreference, and which ones work? The answer, based on a systematic set of experiments, is that MRPC (paraphrase detection) embeddings help most, intermediate layers often beat the final layer, and attention-based aggregation of multiple models gives the best results. The work is careful: three seeds, public code, standard deviations reported, and a nice cosine-similarity check against the target model. That is real evidence and the qualitative ordering of source tasks is probably right.\n\nThe soft spot is the evaluation protocol. In §4.4, the authors pick the best truncation layer (out of layers 6–12) per source model using the validation set, and then report validation F1 for that chosen layer. The same validation set is used for early stopping. So the numbers in Tables 1–3 are effectively maxima over a grid of layer and normalization choices, not the expected performance of a fixed configuration. The seed-to-seed standard deviations don't capture that. The result is that the intermediate-layer claim and the gains from truncation (e.g., +26.2% for NER) are likely optimistic. If the authors evaluated on a held-out test set, the differences would probably shrink—though the ranking of source tasks might survive, since the gaps are large and cosine similarity agrees.\n\nOther minor concerns: the 'reverse probing' framing is fresh but the operations are standard feature extraction; and the scope is narrow—BERT only, one target task, four main sources. The paper acknowledges these limits.\n\nAll in all, this is a solid empirical study that deserves a serious referee. I'd send it to review with a request for held-out test evaluation or a corrected selection protocol. For someone thinking about freezing BERT embeddings for transfer, the qualitative trends are worth knowing, but I wouldn't rely on the exact F1 numbers until they survive a proper test-set check.","headline":"A useful empirical map of which frozen task embeddings help coreference, but the headline layer and combination gains are likely inflated by selecting configurations on the validation set.","tokens_in":12349,"tokens_out":3142,"would_cite":true,"duration_ms":30325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that frozen embeddings from simple source tasks, especially paraphrase detection, can be repurposed for coreference resolution, with attention-based combination giving the best transfer.","keywords":["reverse probing","coreference resolution","task embeddings","transfer learning","attention aggregation","BERT","probing classifiers","OntoNotes"],"falsifier":"Take the same frozen source embeddings and the same trainable coreference head, split the data into train/validation/test instead of using one validation set for both selection and reporting, and compare configurations chosen on validation against their test F1. The central claim collapses if the best validation configuration is not the best on the test set, or if MRPC no longer beats NER once selection bias is removed.","tokens_in":11377,"feed_emoji":"🧩","tokens_out":6474,"duration_ms":58671,"temperature":0.7,"pith_summary":"This paper tries to establish that a complex target task can be solved largely with frozen representations from simpler source tasks, a 'reverse' of classical probing. Using coreference resolution as the target, the authors show that embeddings from a BERT model fine-tuned on paraphrase detection transfer best, while named-entity-recognition embeddings transfer worst. They also find that representations from intermediate layers of fine-tuned source models are often more useful than final-layer outputs, and that combining several frozen embedding streams with a learned attention aggregator consistently improves performance. On the OntoNotes validation split, the best frozen-embedding setup reaches 69.56 F1, close to the 73.75 F1 of a fully trained coreference model. This matters because it maps which task knowledge transfers where and suggests a cheaper path to building task-capable representations.","feed_headline":"Frozen task embeddings hit 69.56 F1 on coreference","feed_subtitle":"Paraphrase detection transfers best; attention pooling and intermediate layers add the biggest gains.","key_machinery":"The central mechanism is a reverse-probing pipeline over frozen, fine-tuned BERT encoders. For each source task, hidden-layer outputs $H_s^{(l)}$ are extracted, optionally truncated to a chosen intermediate layer or formed by concatenating the last 4, 6, or 12 layers, and optionally L2-normalized; the streams are then merged token-wise either by the mean $E_T = \\frac{1}{k}\\sum_i H_{s_i}$ or by attention $E_T = \\sum_i \\alpha_i H_{s_i}$ with $\\alpha_i = \\mathrm{softmax}(W \\cdot H_{s_i})$. The aggregated embeddings feed a trainable coreference head trained on OntoNotes while all source weights stay frozen. The attention aggregator is the load-bearing component: it learns which source and layer to trust, which explains why combinations and low-scoring sources gain most from it.","core_discovery":"Frozen embeddings from simple source tasks can be repurposed for coreference resolution if extracted and combined correctly. Across single-model probes, paraphrase detection (MRPC) and vanilla BERT give the highest F1 (64.32 and 64.01), relation extraction and question answering sit in the middle (52.43 and 47.51), and NER is lowest (36.03). Truncating each source model to a non-final layer, layer 9 for MRPC and RE, layer 8 for QA, layer 6 for NER, improves every model, with NER gaining up to 26.2 points. Attention-based token-wise aggregation outperforms simple averaging, especially for low-scoring sources such as NER (+19.7) and QA (+9), and combining source tasks with attention plus truncation gives the best overall validation score of 69.56 F1 from MRPC+NER+QA. The authors interpret this as evidence that semantic-similarity knowledge is most aligned with coreference, while entity-span knowledge encoded by NER is less transferable unless reweighted by attention.","pith_inferences":["A natural extension the authors did not run is to apply the same protocol to other complex targets such as semantic role labeling or discourse relations; if the MRPC-first ranking repeats, semantic similarity may be a general-purpose predictor of transferability.","The observation that QA transfers better than NER despite lower cosine similarity with the target suggests representation-distance measures are incomplete; a learned attention projection may be capturing a more useful notion of task relatedness.","The validation-set dependency flagged in the paper could be tested by re-running layer and aggregation selection on a training split and reporting test F1; that would separate genuine transfer from selection on the evaluation set.","Because the authors note the added overhead of combining layers and models, a practical follow-up is to prune the attention aggregator to a single per-source weight after training, reducing inference cost while keeping most of the gain."],"forward_implications":["If these results hold, task selection for embedding reuse is not neutral: semantic-similarity training (paraphrase detection) transfers to coreference far better than named-entity recognition.","Intermediate layers of fine-tuned transformers should be treated as reusable assets; final task-specialized layers can hurt a downstream target, so layer truncation is a cheap and effective adjustment.","Attention-based aggregation of frozen embeddings from several tasks is a viable way to approach a fully trained model, reaching 69.56 F1 against a 73.75 upper bound.","Combining several hidden layers (last 4 or 6) is helpful, but pooling all 12 layers is not, so more context does not always transfer better.","The reverse-probing setup gives a low-cost way to compare source tasks for a target before committing to full finetuning."],"supporting_citations":[{"why":"Supplies the BERT-base-cased backbone and pretraining that all source models and the target coreference model start from.","marker":"(Devlin et al., 2019)"},{"why":"Supplies the end-to-end coreference model implementation and the finetuning recipe used as the target task baseline.","marker":"(Xu and Choi, 2020)"},{"why":"Provides the OntoNotes CoNLL-2012 dataset used to train and evaluate the target coreference model and the NER source model.","marker":"(Pradhan et al., 2012)"},{"why":"Defines the MRPC paraphrase corpus whose embeddings turn out to transfer best to coreference.","marker":"(Dolan and Brockett, 2005)"},{"why":"Defines SQuAD 2.0, the question-answering source task that transfers better than its cosine similarity would suggest.","marker":"(Rajpurkar et al., 2016)"},{"why":"Defines TACRED, the relation-extraction source task used as an intermediate-transfer comparison.","marker":"(Zhang et al., 2017)"},{"why":"Establishes the classical probing paradigm that the paper reverses by probing simple source embeddings on a complex target.","marker":"(Conneau et al., 2018)"}],"fun_headline_variants":["Attention-based embedding mixing scores 69.56 F1 on coreference","Simple task embeddings, cleverly combined, hit 69.56 coreference F1","Paraphrase knowledge transfers best, but attention pooling wins coreference","Truncated layers plus attention: coreference F1 jumps to 69.56","Reverse probing reveals semantic similarity as top coreference transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the OntoNotes validation set is a fair place to both choose the best layer, normalization, and aggregation settings and to report the final F1; if the best settings on validation do not transfer to unseen test data, the ranking of source tasks and the claimed gains from attention and truncation are overstated.","fun_headline_variants_meta":{"raw":{"variants":["Attention-based embedding mixing scores 69.56 F1 on coreference","Simple task embeddings, cleverly combined, hit 69.56 coreference F1","Paraphrase knowledge transfers best, but attention pooling wins coreference","Truncated layers plus attention: coreference F1 jumps to 69.56","Reverse probing reveals semantic similarity as top coreference transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1411,"prompt_tokens":959,"completion_tokens":452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":575,"tokens_out":452,"duration_ms":4678,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:36:01.782862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same frozen source embeddings and the same trainable coreference head, split the data into train/validation/test instead of using one validation set for both selection and reporting, and compare configurations chosen on validation against their test F1. The central claim collapses if the best validation configuration is not the best on the test set, or if MRPC no longer beats NER once selection bias is removed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OntoNotes CoNLL-2012 dataset used to train and evaluate the target coreference model and the NER source model."}],"review_version":1}