Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Transductive Auxiliary Task Self-Training for Neural Multi-Task Models

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding predicted auxiliary-task labels for the test sentences to multi-task training improves main-task tagging accuracy, with absolute gains up to 9.56 points for dependency relations and 13.03 points for semantic…

desk verdict Solid large-scale empirical study of a cheap self-training trick, but the transductive claim is not isolated from an inductive silver-label baseline. read the letter →

arxiv 1908.06136 v2 pith:F3ABSF5O submitted 2019-08-16 cs.CL

Johannes Bjerva , Katharina Kann , Isabelle Augenstein This is my paper · ORCID
classification cs.CL
keywords multi-tasklearningself-trainingtransductivesequencelabelingpart-of-speechtaggingdependencyrelationsemanticlow-resourceNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a cheap way to improve low-resource sequence taggers: train a single-task model on an easier auxiliary task (part-of-speech tagging), use it to label the main task's test sentences, and add those automatically generated auxiliary labels to the multi-task training set. The central claim is that this transductive auxiliary task self-training beats a plain multi-task model, with macro-averaged absolute accuracy gains up to 9.56 percentage points for dependency relation tagging and 13.03 for semantic tagging across 86 task-language combinations. The appealing consequence, if the claim holds, is that models in low-resource settings can improve without spending money on annotation, and the largest gains occur when the main-task training set is smallest.

What carries the argument

The mechanism is a three-step loop: train an auxiliary single-task tagger, label every main-task test sentence with it, and retrain a multi-task model with those silver labels mixed into the auxiliary training data. The load-bearing assumptions are that the auxiliary task is easier than the main one, so single-task predictions are reliable, and that transduction matters, so labels for the test instances help more than the same number of labels elsewhere. The model itself is a bidirectional character-and-word LSTM with hard parameter sharing across tasks, which lets the auxiliary labels shape a shared encoder used for the main task.

What would settle it

Run the same procedure but generate silver auxiliary labels for a matched set of held-out non-test sentences instead of the test sentences, keeping all other factors fixed; if test-set accuracy is not higher when the test sentences themselves are labeled, then transduction is not the active ingredient and the reported gains are just extra auxiliary data.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the most useful auxiliary supervision is not more gold auxiliary labels from the training distribution but predicted labels for the exact test instances: the final multi-task model is trained on the main task plus an auxiliary task whose training set has been augmented with single-task predictions for the test input sentences. Because the auxiliary task is assumed easy, those predictions are accurate enough that the added signal improves the shared hidden representation, and because the labels come from the test set, the model is adapted to the specific inputs it must tag. The paper reports that this procedure outperforms the multi-task baseline in all tested data sizes for both main tasks, outperforms adding random gold auxiliary data in the hardest low-resource settings, and in some cases outperforms adding random gold main-task data.

Load-bearing premise

The results stand or fall on whether the auxiliary task is easy enough that a single-task model's predicted labels for the test sentences are accurate enough to improve the shared representation rather than add noise.

Editorial extensions

If this is right

  • Low-resource sequence taggers can be improved at near-zero annotation cost whenever a reliable auxiliary tagger exists for the same data.
  • The benefit increases as the main training set shrinks; the largest reported gains appear at the smallest training sizes tested.
  • Automatic test-set auxiliary labels can substitute for manual auxiliary-task annotation in the low-data regime, and can even beat adding random gold main-task examples in some settings.
  • The method leaves the test set's main-task labels unused, so it is compatible with transductive scenarios where only unlabeled test inputs are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run a control where the same silver auxiliary labels are generated for non-test sentences; such a control would separate the effect of transduction from the effect of simply having more auxiliary data.
  • Because the gains come from predicted labels on the test inputs, the method may be sensitive to domain shift between training and test: if test sentences come from a different distribution, the single-task tagger's confidence may fall and the benefit may shrink.
  • A natural, testable extension would use the same procedure with a stronger auxiliary signal, such as morphological features, or with a pretrained transformer encoder, where the marginal value of extra silver labels may differ.
  • The reported comparisons treat extra gold data as random samples; a baseline that selects the most informative gold sentences to annotate could close or reverse the gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a method called transductive auxiliary task self-training (Aux-ST). A single-task POS tagger is trained on auxiliary-task training data and used to label the inputs of the main-task test set; these silver auxiliary labels are appended to the auxiliary training set, and a hard-parameter-sharing bi-LSTM multi-task model is trained on main and auxiliary data. The authors evaluate on two main tasks (dependency relation tagging and semantic tagging) with POS as auxiliary task, across 41 languages and training-set sizes from 10k down to 0.1k sentences. They report average absolute accuracy gains over a multi-task baseline of up to 9.56 percentage points for DepRel and 13.03 for SemTag, and compare against adding gold auxiliary labels for test sentences (Aux-ST ceiling), adding extra gold auxiliary data (Extra Aux), and adding extra gold main data (Extra Main). They also report a FreqBin unsupervised auxiliary baseline.

Significance. If the reported gains are reliable, the method is an attractive cheap boost for low-resource sequence labeling: no manual annotation is required, and the gains are larger at smaller training sizes, which is exactly where annotation is most costly. The paper's main strengths are the breadth of the evaluation (86 language–task combinations), the consistency of the gains direction, the Aux-ST ceiling as a sanity check of label quality, and the inclusion of a FreqBin control. The central weakness is that the paper does not isolate the transductive mechanism from the generic effect of adding silver-labeled auxiliary data for arbitrary unlabeled sentences, and the evaluation set is the same set used for generating the silver labels. These issues affect the interpretation of the contribution, not the existence of the measured gains.

major comments (4)
  1. [Section 3, Algorithm 1, and Table 1] The unique transductive component of the method is not isolated. Every self-training condition in the paper adds auxiliary labels for the test sentences, and there is no condition that adds predicted auxiliary labels for non-test unlabeled sentences. Consequently, the observed gains are consistent with the hypothesis that any silver-labeled auxiliary data helps the multi-task model, rather than with the paper's stated claim in Section 3 that 'auxiliary task labels for the test instances [are] most useful.' The FreqBin comparison does not address this confound, because it changes the auxiliary task itself rather than the selection of unlabeled instances. I would require an inductive control: apply Aux-ST to a held-out set of unlabeled sentences from the same domain, with evaluation on a test set distinct from the sentences used for labeling.
  2. [Section 3 and Table 1] Because the method adds auxiliary labels for the exact test sentences and is evaluated on that same test set, the measured improvements are partly transductive fit to the evaluation set rather than evidence of generalization to new inputs. The paper discloses this in principle, but it does not quantify how much of the reported 9.56% and 13.03% gains transfer to inputs not seen during the labeling step. A separate held-out evaluation set, or a cross-validation design in which the evaluation sentences are never used for generating silver labels, is needed to support the broader claim that the method improves a model for future test instances.
  3. [Section 4.2 and Table 1] The significance testing is underdescribed. The table marks differences with asterisks for 'Significant (p < 0.05)' but the paper does not state the test statistic, the unit of analysis (treebank, language, or run), the number of random seeds, or whether any multiple-comparison correction was applied across the 86 combinations and four data sizes. Given that several differences are small in absolute terms (for example, Aux-ST at 1.97% versus Extra Main at 0.19% in the 10k DepRel condition), the significance claim needs a precise statistical protocol before the reader can assess whether the improvements are reliable.
  4. [Section 4.2] The sentence 'Our comparison with the FreqBin task does not yield substantial improvements, with mean differences compared to standard MTL at -0.001% (stdev. 0.022)' lacks the necessary aggregation details: how many comparisons, across which languages and data sizes, and whether the differences are per-treebank or per-language. This makes it difficult to interpret the claim that FreqBin rules out domain-adaptation effects as an explanation for the Aux-ST gains.
minor comments (4)
  1. [Section 2.1] The paragraph describing the three tasks states 'In all cases, POS is the auxiliary task, and either POS tagging or DepRel tagging is the main task.' This should read 'either DepRel tagging or semantic tagging is the main task,' since the experiments use POS as the auxiliary task throughout.
  2. [Algorithm 1] Line 5 of Algorithm 1 contains a typo: 'for sentence in testinpmain' should be 'for sentence in testinpmain' with a space, or 'test_input_main' for readability.
  3. [Section 4] The paper does not report the full hyperparameter configuration (hidden size, embedding size, learning rate, batch size, dropout) or the number of runs per condition, which would be useful for reproducibility even if these choices are standard for the DyNet hierarchical bi-LSTM.
  4. [Abstract and Section 1] The phrase 'Drawing heavily on ideas from those two approaches' is informal; a more precise phrasing such as 'Building on multi-task learning and self-training' would better match the style of the rest of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical transductive self-training method with no fitted constants, no self-citation chain, and no derivation step that reduces to its inputs.

full rationale

The paper makes no first-principles derivation: its central claim is an empirical accuracy improvement from transductive auxiliary task self-training. The procedure (Algorithm 1) trains a single-task auxiliary model, labels test inputs with auxiliary tags, and retrains a multi-task model on those silver labels plus gold training data; the reported gains are measured on the same test set. This is disclosed transductive learning (Vapnik, 1998), not a hidden reduction: the predicted auxiliary labels are auxiliary-task outputs, not the main-task predictions being evaluated, and no model parameter is fitted to the main-task test labels. The comparison baselines Extra Aux and Extra Main add gold labels for other sentences, and no condition adds silver auxiliary labels for non-test unlabeled sentences; that missing control is a potential experimental confound (the transductive component is not isolated from generic self-training on arbitrary unlabeled sentences), but it is not circularity by construction. There is no fitted input renamed as a prediction, no load-bearing self-citation, and no uniqueness theorem imported from the authors' prior work. The weak auxiliary-task premise in Section 3 is an assumption, not a result derived from the target outcome. Under the requested definition, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical objects or fitted constants. Its central claim depends on the standard MTL assumption that shared parameters transfer between tasks, on the self-training assumption that the auxiliary model is accurate enough, and on an untested ordering assumption that test-set auxiliary labels are more useful than labels for other unlabeled sentences. The lack of reported hyperparameters is a reproducibility gap.

free parameters (1)
  • bi-LSTM hyperparameters (hidden size, embedding size, learning rate, batch size, dropout) = not reported
    The central empirical results depend on these choices, but the paper only reports Adam, a maximum of 10 epochs, and early stopping.
assumptions (4)
  • domain assumption The auxiliary task is easier than the main task and the single-task auxiliary model is sufficiently accurate.
    Stated in Section 3 as the basis for self-training; if the single-task POS model were inaccurate, silver labels would inject noise.
  • domain assumption Hard parameter sharing lets auxiliary task training signal, including silver labels on test instances, improve main task predictions.
    This is the multi-task learning assumption, inherited from Caruana (1993) and Plank et al. (2016).
  • ad hoc to paper Auxiliary labels for test instances are more useful than auxiliary labels for other unlabeled instances.
    Assumed in Section 3 to justify the transductive choice, but not directly tested against a non-transductive silver-label baseline.
  • domain assumption Evaluation on the same test set used for transductive labeling is a valid measure of the method's value.
    The paper explicitly chooses a transductive viewpoint, so using test inputs in training is intentional; however, this makes comparisons with non-transductive baselines less clean.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transductive Auxiliary Task Self-Training for Neural Multi-Task Models." pith.science (2026). https://pith.science/paper/F3ABSF5O

@misc{pith2026190806136,
  author       = {Pith},
  title        = {Pith review of: Transductive Auxiliary Task Self-Training for Neural Multi-Task Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3ABSF5O}},
  note         = {Machine review of arXiv:1908.06136}
}
read the original abstract

Multi-task learning and self-training are two common ways to improve a machine learning model's performance in settings with limited training data. Drawing heavily on ideas from those two approaches, we suggest transductive auxiliary task self-training: training a multi-task model on (i) a combination of main and auxiliary task training data, and (ii) test instances with auxiliary task labels which a single-task version of the model has previously generated. We perform extensive experiments on 86 combinations of languages and tasks. Our results are that, on average, transductive auxiliary task self-training improves absolute accuracy by up to 9.56% over the pure multi-task model for dependency relation tagging and by up to 13.03% for semantic tagging.

Figures

Figures reproduced from arXiv: 1908.06136 by the authors.

Figure 1
Figure 1. POS tags, DepRel labels and semantic tags for an example sentence. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Results for UD treebanks [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  5. [5]

    Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations . In Proceedings of EACL

    work page 2017

  6. [6]

    Isabelle Augenstein, Sebastian Ruder, and Anders S gaard. 2018. Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces . In Proceedings of NAACL

    work page 2018

  7. [7]

    Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2015. Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs . In Proceedings of EMNLP

    work page 2015

  8. [8]

    Yoshua Bengio, J\' e r\^ o me Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of ICML

    work page 2009

Show all 33 references
  1. [9]

    Johannes Bjerva. 2017. One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis. Ph.D. thesis, University of Groningen

    2017

  2. [10]

    Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic Tagging with Deep Residual Networks . In COLING, pages 3531--3541

    2016

  3. [11]

    Rich Caruana. 1993. Multitask Learning: A Knowledge-Based Source of Inductive Bias . In Proceedings of ICML

    1993

  4. [12]

    Guoqing Chao and Shiliang Sun. 2012. Semi-supervised Multitask Learning via Self-training and Maximum Entropy Discrimination . In Proceedings of International Conference on Neural Information Processing

    2012

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  6. [14]

    Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization . arXiv preprint arXiv:1412.6980

    2014 arXiv

  7. [15]

    Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-Sequence Models for Parsing and Generation . In Proceedings of ACL

    2017

  8. [16]

    Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing Transfer Languages for Cross-Lingual Learning . arXiv:1905.12688

    2019 arXiv

  9. [17]

    Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Lu\' i s Marujo, and Tiago Lu\' i s

    Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Lu\' i s Marujo, and Tiago Lu\' i s. 2015. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation . In Proceedings of EMNLP

    2015

  10. [18]

    H \' e ctor Mart \' nez Alonso and Barbara Plank. 2017. When is multitask learning effective? Semantic sequence prediction under varying data conditions . In EACL

    2017

  11. [19]

    Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980

    2017 arXiv

  12. [20]

    Kamal Nigam and Rayid Ghani. 2000. Analyzing the effectiveness and applicability of co--training. In Proceedings of Information and Knowledge Management

    2000

  13. [21]

    Manning, Ryan T

    Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan T. McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A Multilingual Treebank Collection . In ...

    2016

  14. [22]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations . In NAACL, pages 2227--2237. Association for Computational Linguistics

    2018

  15. [23]

    Barbara Plank, Anders S gaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss . In Proceedings of ACL (Short Papers)

    2016

  16. [24]

    Ellen Riloff, Janyce Wiebe, and Theresa Wilson. 2003. Learning Subjective Nouns using Extraction Pattern Bootstrapping . In Proceedings of CoNLL

    2003

  17. [25]

    Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders S gaard. 2019. Latent Multi-task Architecture Learning . In Proceedings of AAAI

    2019

  18. [26]

    Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian Optimization . In Proceedings of EMNLP

    2017

  19. [27]

    Anders S gaard and Yoav Goldberg. 2016 a . Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of ACL (Short Papers)

    2016

  20. [28]

    Anders S gaard and Yoav Goldberg. 2016 b . Deep multi-task learning with low level tasks supervised at lower layers. In ACL

    2016

  21. [29]

    Milan Straka, Jan Hajic, and Jana Strakov \'a . 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing . In Proceedings of LREC

    2016

  22. [30]

    Vladimir N Vapnik. 1998. Statistical Learning Theory. John Wiley

    1998

  23. [31]

    David Yarowsky. 1995. Unsupervised Word-Sense Disambiguation Rivaling Supervised Methods . In Proceedings of ACL

    1995

  24. [32]

    Zhi-Hua Zhou and Ming Li. 2005. Tri-training: exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17:1529--1541

    2005

  25. [33]

    Xiaojin Zhu. 2005. Semi-Supervised Learning Literature Survey . Technical Report 1530, Computer Sciences, University of Wisconsin-Madison

    2005

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.