Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding low-level linguistic supervision (part-of-speech tagging, chunking, dependency parsing) to a shared encoder helps multi-task learning more than orthogonality constraints, adversarial learning, gating, or…

desk verdict Useful first systematic comparison of five MTL mechanisms, but the headline ranking is undermined by a shared-encoder depth confound the authors admit to, and the 'improves all datasets' claim is contradicted by their own tables. read the letter →

arxiv 1908.07820 v2 pith:HWQTWGAN submitted 2019-08-16 cs.CL cs.LG

Jianquan Li , Xiaokang Liu , Wenpeng Yin , Min Yang , Liqun Ma , Yaohong Jin This is my paper · ORCID
classification cs.CLcs.LG
keywords multi-tasklearningdeepneuralnetworksnaturallanguageprocessinglinguistichierarchiesorthogonalityconstraintsadversarialgatemechanismlabelembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to answer which of five widely used multi-task learning (MTL) mechanisms—exploring linguistic hierarchies, orthogonality constraints, adversarial learning, gate mechanisms, and label embedding—actually helps deep neural networks on representative NLP tasks. It claims that all five beat single-task models, but linguistic hierarchical information performs best individually, and combining linguistic hierarchies with gate mechanism and label embedding achieves the best results on most datasets. The practical stake is a design guide: rather than relying on isolated state-of-the-art claims, practitioners can use a systematic comparison across text classification, semantic textual similarity, and natural language inference on both high- and low-relevance datasets to choose how to structure an MTL system.

What carries the argument

The central object is a basic MTL framework: a shared encoder made of M-layer BiLSTMs (M=3 in experiments), task-specific private BiLSTM extractors, pooling (mean and max), and a softmax output layer per task. Each of the five mechanisms is integrated into this framework as a plug-in. For the winning ELH mechanism, auxiliary losses from POS tagging, chunking, and dependency parsing are injected at successively deeper shared layers (lower-level tasks at lower layers), and the final loss is a linear combination of the auxiliary losses and the main task loss. This shared-encoder-plus-auxiliary-supervision design is what the paper credits for the best individual performance, and it is also the component that the hybrid ABC builds on.

What would settle it

Re-run the same evaluation holding the shared encoder at the same depth for all five mechanisms (e.g., three BiLSTM layers for all, and also one layer for all) and check whether linguistic hierarchies still beats orthogonality constraints and adversarial learning; if the gap shrinks or reverses, the depth asymmetry is the cause.

Watch

Extended reading notes

Core claim

In the authors' own run of experiments, the single-task baseline averages 83.92 on six Amazon review domains and 72.55 on the six low-relevance tasks (CoLA, SST-2, MRPC, STS-B, MNLI, QNLI). Each of the five MTL mechanisms improves on this, but exploring linguistic hierarchies (ELH)—supervising POS tagging, chunking, and dependency parsing on progressively deeper shared BiLSTM layers—yields the largest average improvement (3.36 and 3.81 points, respectively). Ablations show that the best ELH variant does not always use all three auxiliary tasks: dependency parsing alone is best on high-relevant datasets, while chunking plus dependency parsing is best on low-relevant ones. The paper also reports that combining ELH, gate mechanism, and label embedding (ABC) gives the best results on most datasets, and that gains from different mechanisms do not add up: the full five-mechanism ensemble performs no better than the three-mechanism hybrid.

Load-bearing premise

The comparison assumes that the five MTL mechanisms are on an equal footing, but the paper's setup gives ELH a deeper shared encoder (three BiLSTM layers) than OC and AL (one layer), so capacity, not the mechanism, could explain the ranking.

Editorial extensions

If this is right

  • If the findings hold, a practical MTL recipe for text classification, similarity, and NLI is to add low-level linguistic auxiliary tasks (POS, chunking, dependency parsing) to the shared encoder rather than to rely on orthogonality constraints, adversarial training, or gating alone.
  • Non-additivity of improvements implies that stacking every known MTL mechanism is not a sound default; the three-mechanism hybrid ABC is the paper's recommended configuration over the full ensemble.
  • Because higher-level auxiliary tasks (dependency parsing) outperformed lower-level ones in ablations, task selection for ELH matters: including all three basic tasks is not always optimal.
  • All five MTL methods mitigate overfitting on STS-B compared to the single model, with adversarial learning and label embedding reducing the drop most, so MTL can serve as a regularization strategy even when gains on a specific benchmark are small.
  • The comparison of ELH training strategies suggests that optimizing all layer losses simultaneously is at least as good as the original bottom-to-top loss injection, which is relevant for reproducing the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own experimental setup gives ELH a three-layer shared BiLSTM while OC and AL use a single-layer shared BiLSTM when used alone, so the headline ranking could reflect shared-encoder capacity rather than the mechanisms themselves; rerunning with matched shared depth would settle this.
  • Because the best ELH variant never uses all three auxiliary tasks, a direct follow-up could search over auxiliary-task subsets per target task rather than fixing the three-layer hierarchy.
  • The SLH result implies that part of ELH's gain on low-relevance tasks may be a regularization effect from hierarchical supervision rather than multi-task transfer, a distinction the paper does not separate.
  • The non-additivity result suggests that future MTL design should treat mechanisms as interacting and search over subsets rather than concatenating all known components.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an empirical comparison of five multi-task learning mechanisms—linguistic hierarchies (ELH), orthogonality constraints (OC), adversarial learning (AL), gate mechanism, and label embedding (LE)—integrated into a shared BiLSTM framework. It evaluates these methods on six high-relevant Amazon sentiment datasets and six low-relevant datasets (SST-2, CoLA, MNLI, MRPC, QNLI, STS-B), reporting individual and combinatorial results. The central claims are that MTL improves over single-task models on all datasets, that ELH is the best individual method, and that the combination ELH+Gate+LE (ABC) achieves the best results on most datasets. The paper also includes an ablation of the auxiliary tasks used in ELH and an overfitting analysis.

Significance. If the comparative ranking were established, the paper would offer useful practical guidance for designing MTL architectures in NLP. The auxiliary-task ablation in Tables 4 and 5 is a valuable piece of analysis, and the paper's effort to evaluate several mechanisms under a common framework is well motivated. However, the central ranking is compromised by a confound in shared-encoder depth, the headline claims are contradicted by the paper's own tables, and no statistical support is provided. As it stands, the paper does not deliver reliable empirical conclusions.

major comments (4)
  1. [Section 6 and Tables 2-4] The comparison of individual MTL mechanisms is confounded by shared-encoder depth. Section 4.1 fixes M=3 for the shared BiLSTM, and ELH is evaluated with this three-layer stack, while Section 6 states that OC and AL use a single-layer shared BiLSTM when used alone or without ELH. Table 4's 'No Auxiliary Tasks' row, which uses the three-layer shared stack without auxiliary losses, reaches 86.90 on high-relevant data, already 2.98 points over Single (83.92) and within 0.38 of ELH's 87.28. This makes model capacity a plausible explanation for ELH's advantage over OC (86.74) and AL (87.06). The claim that linguistic hierarchical information performs better than other individual MTL methods is therefore not established; the authors need to rerun OC and AL with the same three-layer shared backbone and ideally report parameter counts.
  2. [Section 1 and Tables 2-3] The claim that 'Multi-task learning methods have significant improvements over the single models on all the datasets' is contradicted by the paper's own results. For example, Table 2 shows A (ELH) at 82.75 and D (OC) at 82.58 on DVDs, both below the single-task 83.00; Table 3 shows D at 66.03 on STS-B, below the single-task 67.47. The wording should be corrected to reflect the actual pattern, and the term 'significant' should be backed by a statistical test rather than used descriptively.
  3. [Section 7 and Tables 2-3] No standard deviations, confidence intervals, or significance tests are reported, although the paper states that each experiment was repeated three times. The differences that support the ranking of individual methods are small: on high-relevant data, A averages 87.28 versus E at 87.06 and D at 86.74; on low-relevant data, A averages 76.36 versus E at 74.81 and D at 73.51. Without variance estimates, these margins cannot be distinguished from noise, and the conclusion that ELH 'performs better' is not statistically supported.
  4. [Section 1 and Tables 2-3] The claim that 'combining linguistic hierarchies, gate mechanism, and label embedding methods can achieve best results on most of the datasets' overstates the evidence. ABC has the best average on both dataset groups, but on high-relevant datasets ABC is best on only one of six domains (Apparel, 91.00), and on low-relevant datasets ABC is best or tied on three of six (CoLA, QNLI, STS-B) and not on SST-2, MNLI, or MRPC. The supported conclusion is that ABC achieves the best average, not that it is best on most datasets.
minor comments (4)
  1. [Section 8] The conclusion refers to 'all six individual MTL methods', but only five mechanisms (A-E) are defined and evaluated.
  2. [Section 7.1, Orthogonality constraint paragraph] The sentence beginning 'bu In the case of low-relevance' is incomplete and should be rewritten.
  3. [Section 6] The sentence 'the share part of diff and adversarial are lighter in the paper comparing linguistic hierarchie' is grammatically unclear and should be rephrased for precision.
  4. [Tables 2 and 5] The reported single-task average on low-relevant data is 72.55 in Table 3 but 72.58 in Table 5; these values should be reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark, not a derivation; the ELH-vs-OC/AL comparison is confounded by shared-encoder depth, but no claim reduces to its own inputs or to a self-citation.

full rationale

This paper reports a systematic empirical comparison of five MTL mechanisms on standard NLP benchmarks. There is no claimed first-principles derivation, no fitted parameter later relabeled as a prediction, and no uniqueness theorem imported from prior author work. The headline findings ('using linguistic hierarchical information performs better than other individual MTL methods' and 'combining linguistic hierarchies, gate mechanism, and label embedding methods can achieve best results') are post-hoc summaries of the tables, not predictions derived from the model equations. The one passage that could raise concern is Section 6: 'the share part of diff and adversarial are lighter in the paper comparing linguistic hierarchie, we use a single-layer bi-LSTM in share part when diff and adversarial are used alone or combining with the other component without exploring linguistic hierarchies.' That is an uncontrolled confound: ELH always uses the three-layer shared BiLSTM (Section 4.1 fixes M=3) while OC and AL run with a one-layer shared stack, so the reported advantage could reflect encoder capacity rather than the mechanism. However, this is a correctness/validity problem about unequal comparison conditions, not circularity: the conclusion does not reduce by definition to the input, and no equation is shown to be equivalent to another by construction. The paper is also self-contained with respect to external benchmarks (GLUE-family datasets and CoNLL tasks), and no load-bearing self-citation was found. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The central ranking depends on two hand-set loss weights (lambda_adv, lambda_OC), on an ad hoc choice to give OC and AL a lighter shared encoder than ELH, and on the unexamined assumption that three-run averages are stable. No new theoretical entities are introduced.

free parameters (3)
  • Adversarial loss weight lambda_adv = 0.05
    Set manually in Section 6.3 and not tuned; the AL method's ranking could shift with this weight.
  • Orthogonality constraint weight lambda_OC = 0.01
    Set manually in Section 6.3; affects OC behavior and the ensemble results.
  • Shared-encoder depth for OC and AL = 1 BiLSTM layer (3 for ELH)
    Chosen by the authors (Section 6) to be lighter for OC/AL, which confounds the method comparison with model capacity.
assumptions (2)
  • domain assumption Observed accuracy differences between methods are due to the MTL mechanism, not to differences in shared-encoder depth or hyperparameters.
    The paper varies shared-encoder depth across methods (Section 6) and tunes hyperparameters per dataset, so the assumption is violated.
  • domain assumption The average of three runs provides a stable ranking without reported variance.
    No standard deviations or significance tests are given, so small gaps (around 1 point) may not be reliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing." pith.science (2026). https://pith.science/paper/HWQTWGAN

@misc{pith2026190807820,
  author       = {Pith},
  title        = {Pith review of: Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWQTWGAN}},
  note         = {Machine review of arXiv:1908.07820}
}
read the original abstract

Multi-Task Learning (MTL) aims at boosting the overall performance of each individual task by leveraging useful information contained in multiple related tasks. It has shown great success in natural language processing (NLP). Currently, a number of MLT architectures and learning mechanisms have been proposed for various NLP tasks. However, there is no systematic exploration and comparison of different MLT architectures and learning mechanisms for their strong performance in-depth. In this paper, we conduct a thorough examination of typical MTL methods on a broad range of representative NLP tasks. Our primary goal is to understand the merits and demerits of existing MTL methods in NLP tasks, thus devising new hybrid architectures intended to combine their strengths.

Figures

Figures reproduced from arXiv: 1908.07820 by the authors.

Figure 1
Figure 1. The overview of the basic MTL framework. The shared extractor is composed of several BiLSTM layers that [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Integrating the ELH method into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Integrating the OC method into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Integrating the AL method into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Integrating the gate mechanism into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Integrating the label embedding method into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The ensemble model that integrates all the five MTL algorithms into the basic MLT framework. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Overfitting on STS-B(Taking an average of every five values) [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages

  1. [1]

    Z. Yu , Y . Qiang,A survey on multi-task learning, Arxiv Preprint arXiv:1707.08114., (201),

    arXiv

  2. [2]

    Rich, Multitask learning, Machine Learning., 28, , 41–75, (1997),

    C. Rich, Multitask learning, Machine Learning., 28, , 41–75, (1997),

    work page 1997

  3. [3]

    Sebastian, An overview of multi-task learning in deep neural networks , Arxiv Preprint arXiv:1706.05098., (201),

    R. Sebastian, An overview of multi-task learning in deep neural networks , Arxiv Preprint arXiv:1706.05098., (201),

    arXiv

  4. [4]

    Liqiang, Z

    X. Liqiang, Z. Honglun , C. Wenqing, Gated Multi-Task Network for Text Classification, Proceedings Of The 2018 Conference Of The North American Chapter Of The Association For Computational Linguistics: Human Language Technologies, V olume 2 (short Papers)., pp. 726–731, (201),

    work page 2018

  5. [5]

    Ishan, S

    M. Ishan, S. Abhinav, G. Abhinav , H. Martial, Cross-stitch networks for multi-task learning, Proceedings Of The Ieee Conference On Computer Vision And Pattern Recognition., pp. 3994–4003, (201),

    work page

  6. [6]

    Sebastian, B

    R. Sebastian, B. Joachim, A. Isabelle , S. Anders, Learning what to share between loosely related tasks, Arxiv Preprint arXiv:1705.08142., (201), 12

    arXiv

  7. [7]

    Xiaodong, H

    L. Xiaodong, H. Pengcheng, C. Weizhu , G. Jianfeng, Multi-Task Deep Neural Networks for Natural Language Understanding, Arxiv Preprint arXiv:1901.11504., (201),

    arXiv 1901

  8. [8]

    Pengfei, Q

    L. Pengfei, Q. Xipeng , H. Xuanjing, Recurrent neural network for text classification with multi-task learning, Arxiv Preprint arXiv:1605.05101., (201),

    arXiv

Show all 34 references
  1. [9]

    Pengfei, Q

    L. Pengfei, Q. Xipeng , H. Xuanjing, Adversarial multi-task learning for text classification , Arxiv Preprint arXiv:1704.05742., (201),

    arXiv

  2. [10]

    Anders , G

    S. Anders , G. Yoav,Deep multi-task learning with low level tasks supervised at lower layers, Proceedings Of The 54th Annual Meeting Of The Association For Computational Linguistics (volume 2: Short Papers)., 2, pp. 231–235, (201),

  3. [11]

    Kazuma, X

    H. Kazuma, X. Caiming, T. Yoshimasa , S. Richard, A joint many-task model: Growing a neural network for multiple nlp tasks, Arxiv Preprint arXiv:1611.01587., (201),

    arXiv

  4. [12]

    Sepp , S

    H. Sepp , S. Jürgen, Long short-term memory, Neural Computation., 9, , 1735–1780, (1997),

    1997

  5. [13]

    Jeffreyl, Finding structure in time, Cognitive Science., 14, , 179–211, (1990),

    E. Jeffreyl, Finding structure in time, Cognitive Science., 14, , 179–211, (1990),

    1990

  6. [14]

    C. Qian, Z. Xiaodan, L. Zhenhua, W. Si, J. Hui , I. Diana, Enhanced lstm for natural language inference, Arxiv Preprint arXiv:1609.06038., (201),

    arXiv

  7. [15]

    Konstantinos, T

    B. Konstantinos, T. George, S. Nathan, K. Dilip , E. Dumitru, Domain separation networks, Advances In Neural Information Processing Systems., pp. 343–351, (201),

  8. [16]

    Isabelle, R

    A. Isabelle, R. Sebastian , S. Anders, Multi-task learning of pairwise sequence classification tasks over disparate label spaces, Arxiv Preprint arXiv:1802.09913., (201),

    arXiv

  9. [18]

    Brianw, Comparison of the predicted and observed secondary structure of T4 phage lysozyme, Biochimica Et Biophysica Acta (bba)-protein Structure., 405, , 442–451, (1975),

    M. Brianw, Comparison of the predicted and observed secondary structure of T4 phage lysozyme, Biochimica Et Biophysica Acta (bba)-protein Structure., 405, , 442–451, (1975),

    1975

  10. [19]

    Richard, P

    S. Richard, P. Alex, W. Jean, C. Jason, M. Christopherd, N. Andrew , P. Christopher, Recursive deep models for semantic compositionality over a sentiment treebank, Proceedings Of The 2013 Conference On Empirical Methods In Natural Language Processing., pp. 1631–1642, (201),

    2013

  11. [20]

    Williamb , B

    D. Williamb , B. Chris, Automatically constructing a corpus of sentential paraphrases, Proceedings Of The Third International Workshop On Paraphrasing (iwp2005)., (200),

  12. [21]

    Daniel, D

    C. Daniel, D. Mona, A. Eneko, L. Inigo , S. Lucia, Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation, Arxiv Preprint arXiv:1708.00055., (201),

    2017 arXiv

  13. [22]

    Adina, N

    W. Adina, N. Nikita , B. Samuelr, A broad-coverage challenge corpus for sentence understanding through inference, Arxiv Preprint arXiv:1704.05426., (201),

    arXiv

  14. [23]

    Pranav, Z

    R. Pranav, Z. Jian, L. Konstantin , L. Percy,Squad: 100,000+ questions for machine comprehension of text, Arxiv Preprint arXiv:1606.05250., (201),

    arXiv

  15. [24]

    W. Alex, S. Amapreet, M. Julian, H. Felix, L. Omer , B. Samuelr, Glue: A multi-task benchmark and analysis platform for natural language understanding, Arxiv Preprint arXiv:1804.07461., (201),

    arXiv

  16. [25]

    W. Alex, S. Amanpreet , B. Samuelr,Neural network acceptability judgments, Arxiv Preprint arXiv:1805.12471., (201),

    arXiv

  17. [26]

    Erikf , D

    S. Erikf , D. Fien, Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, Arxiv Preprint cs/0306050., (200),

    2003 arXiv

  18. [27]

    Zhilin, S

    Y . Zhilin, S. Ruslan , C. William, Multi-task cross-lingual sequence tagging from scratch , Arxiv Preprint arXiv:1603.06270., (201),

    arXiv

  19. [28]

    Samuelr, A

    B. Samuelr, A. Gabor, P. Christopher , M. Christopherd,A large annotated corpus for learning natural language inference, Arxiv Preprint arXiv:1508.05326., (201),

    arXiv

  20. [29]

    Tushar, S

    K. Tushar, S. Ashish , C. Peter,Scitail: A textual entailment dataset from science question answering, Thirty-second Aaai Conference On Artificial Intelligence., (201),

  21. [30]

    Mathieu, E

    S. Mathieu, E. Carlhenrik, U. Raquel , D. Trevor, Factorized orthogonal latent spaces , Proceedings Of The Thirteenth International Conference On Artificial Intelligence And Statistics., pp. 701–708, (201),

  22. [31]

    D. Long, C. Trevor, B. Steven , C. Paul, Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser, Proceedings Of The 53rd Annual Meeting Of The Association For Computational Linguistics And The 7th International Joint Conference On Natural L...

  23. [32]

    Yongxin , H

    Y . Yongxin , H. Timothym,Trace norm regularised deep multi-task learning, Arxiv Preprint arXiv:1606.04038., (201),

    arXiv

  24. [33]

    Mitchell, S

    M. Mitchell, S. Beatrice , M. Maryann, Building a large annotated corpus of English: The Penn Treebank,

  25. [34]

    C. Michael, Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms, Proceedings Of The Acl-02 Conference On Empirical Methods In Natural Language Processing- volume 10., pp. 1–8, (2002),

    2002

  26. [35]

    Nikita , K

    K. Nikita , K. Dan, Constituency parsing with a self-attentive encoder, Arxiv Preprint arXiv:1805.01052., (201), 14

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.