Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For Hindi-English code-mixed humour and sarcasm detection, multi-task learning with a hate-detection auxiliary task and reusing monolingual task samples yields the largest F1-score gains, while prompting or instruction-finetuning very…

desk verdict Useful extension of the group's native-mixing idea to humor/sarcasm, but the headline MTL gains are maxima over a big grid with no variance estimates or selection details. read the letter →

arxiv 2412.12761 v2 pith:HJWTH2OC submitted 2024-12-17 cs.CL cs.AI

Debajyoti Mazumder , Aakash Kumar , Jasabanta Patro This is my paper · ORCID
classification cs.CLcs.AI
keywords code-mixedhumourdetectionsarcasmHindi-Englishcode-mixingnativesamplemixingmulti-tasklearninghateauxiliarytaskmultilinguallanguagemodelssyntheticHindidata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hindi-English code-mixed tweets switch between Hindi and English inside a single utterance, and labelled data for detecting humour and sarcasm in them is scarce. This paper tests whether that scarcity can be bypassed by reusing existing resources: adding monolingual English and machine-translated Hindi humour or sarcasm samples to the code-mixed training set, jointly training with a semantically related hate-detection task, and prompting or instruction-finetuning very large multilingual models. Its central finding is that multi-task learning gives the largest improvement, with F1-score gains up to 10.67% for humour and 12.35% for sarcasm, followed by native sample mixing at up to 6.76% and 8.64%, while the very large models show no F1 improvement. The paper therefore argues that transfer from monolingual task data and a related task can substitute for expensive code-mixed annotation, at least when the base models are smaller multilingual language models.

What carries the argument

The argument is carried by two mechanisms. The first is native sample mixing: augmenting the code-mixed training set with monolingual task samples, in this case English datasets and synthetic Hindi versions produced by machine translation, so that a multilingual encoder can draw on linguistic knowledge from non-code-mixed text. The second is the multi-task framework, a 12-layer BERT-style architecture with a shared bottom module, separate task-specific top modules for humour, sarcasm, and hate detection, and a gating mechanism that blends the shared representation with each task representation; a regularizer softly pulls the task-specific weight matrices together. The gating mechanism is what lets the model exploit the hate-detection task on short and misspelled code-mixed inputs, and the paper's ablation shows that removing it lowers scores.

What would settle it

Re-run the three strategies with a fixed, validation-only model-selection protocol on the same Khandelwal et al. (2018) and Swami et al. (2018) test splits, and report per-seed F1 distributions or confidence intervals; if the MTL and native-mixing configurations selected without test-set peeking no longer beat the code-mixed-only baseline by the reported margins, the central claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Hindi-English code-mixed humour and sarcasm detectors can be improved by training with data that is not code-mixed, and improved further by learning a related task at the same time. When monolingual English samples and translated Hindi samples are added to the code-mixed training set, the best multilingual models reach 0.79 F1 for humour and 0.88 for sarcasm, gains of up to 6.76% and 8.64%. When the same models are trained in a multi-task framework with hate detection as an auxiliary task, the best scores rise to 0.83 for humour and 0.91 for sarcasm, gains of up to 10.67% and 12.35%. The paper also finds that prompting or instruction-finetuning very large multilingual models with native samples does not improve them, largely because they over-assign the positive label, and that statistical classifiers are hurt rather than helped by native sample mixing.

Load-bearing premise

The load-bearing premise is that the reported F1 gains are not artifacts of configuration selection: the paper averages over three seeds but reports no variance or explicit test-selection protocol, so if the best configurations were chosen after looking at test-set scores, the headline improvements could be optimistic.

Editorial extensions

If this is right

  • Practitioners can improve code-mixed humour and sarcasm detectors without new code-mixed annotation by reusing English humour and sarcasm datasets and an existing hate-detection corpus.
  • The multi-task configuration with hate detection is the strongest of the three strategies, with the largest gains on short, misspelled inputs, and the gating mechanism is a load-bearing component of that gain.
  • Machine-translated Hindi samples are not a reliable source of humorous or sarcastic data, because wordplay and ironic tone are largely lost in translation; gold Hindi data or better context-aware translation would be needed.
  • Very large multilingual models prompted with native or code-mixed examples, or instruction-finetuned, do not match the smaller multilingual models on these tasks under the tested setups, due to a tendency to favour humour and sarcasm labels.
  • Statistical classifiers degrade when native samples are mixed in, so the benefit of native sample mixing is specific to pre-trained multilingual representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its observation about low KL divergence could be turned into a selection rule: rank candidate native datasets by overlap between positive and negative word distributions, and prefer lower-divergence datasets before mixing.
  • A natural testable extension is to replace machine-translated Hindi with native Hindi humour and sarcasm data or with context-aware LLM translation; the paper's own error analysis predicts that gold Hindi data would push native sample mixing closer to the multi-task gains.
  • The VMLMs' tendency to over-predict humour and sarcasm suggests that calibration or explicit negative-example prompting, neither of which the paper tries, is the next lever for making large models competitive in this setting.
  • The hate-related errors in the multi-task models indicate that task interference is real; varying the strength of the soft-parameter-sharing regularizer or the hate task weight could trade off the observed gains and losses, a test the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports experiments on Hindi-English code-mixed humour and sarcasm detection. Three strategies are compared: (i) mixing native monolingual (English and translated Hindi) task samples into the code-mixed training set, (ii) multi-task learning with hate detection as an auxiliary task, and (iii) prompting and instruction-finetuning of very large multilingual language models (VMLMs). The central empirical claim is that native sample mixing and multi-task learning improve positive-class F1 relative to code-mixed-only training, with MTL giving the largest gains ('up to 10.67%' for humour and 'up to 12.35%' for sarcasm), while VMLMs do not improve. The paper includes baseline reproductions, ablations, error analysis, and a public code repository.

Significance. If the claims hold, the paper provides useful evidence that scarce code-mixed annotations can be supplemented by monolingual data and a related auxiliary task, and it documents negative results for VMLM prompting in this code-mixed setting. The release of code and the reproduction of prior baselines are concrete strengths, as is the qualitative error analysis that connects MTL gains to hate-related contexts. However, the headline magnitudes are not yet supported by the reported statistics: the absence of variance estimates, an undefined significance test, and the lack of a stated model-selection protocol make it impossible to distinguish a genuine MTL advantage from selection over a large configuration grid.

major comments (3)
  1. [Section 4, Tables 3 and 4] The paper reports F1-scores 'averaged over three random seeds' and marks p<0.05 results with '*', but it never describes the significance test, reports no standard deviations or confidence intervals, and does not indicate whether any multiple-comparison correction was applied. Because the headline claims 'up to 10.67%' and 'up to 12.35%' are maxima over a large configuration space (three MLMs, four native dataset choices, seven auxiliary-task combinations, gated and ungated variants, plus the hyperparameter grid in Table 10), the reported gains may be selection artifacts rather than stable effects. Please report per-configuration variance and a complete significance-test procedure, or weaken the claims accordingly.
  2. [Appendix E.1, Tables 3 and 4] It is never stated whether the best configurations in Tables 3 and 4 were chosen on the validation split or after inspecting test-set outcomes. If test-set selection occurred, the 'up to' improvements are optimistic by construction; even if validation was used, the number of configurations makes uncorrected per-configuration p-values unreliable. Please state the exact selection protocol, including whether the reported best configurations were fixed before the test set was used.
  3. [Appendix G vs. Section 3.2 and Table 3] Appendix G concludes that Hindi translations 'didn't preserve the humorous and sarcastic context' and that the authors 'decided to use only English samples for further experiments,' but Section 3.2 and Table 3 present CM+Hi and CM+Hi+En as main experimental conditions, and the title and abstract emphasize 'synthetic native samples.' This is an internal inconsistency about what was actually evaluated. Please clarify which reported results use translated Hindi samples, whether the Hindi-augmented columns in Table 3 are included in the main claims, and whether translated Hindi samples were used in the MTL experiments.
minor comments (5)
  1. [Throughout] The phrase 'upto' should be 'up to' in the abstract, contributions, and conclusion; also 'Futhermore' in Section 4.4 is a typo.
  2. [Table 1, Sarcasm row] The citation 'Swami et al. (1805)' is a date typo for 2018; please correct it.
  3. [Section 7, Limitation bullet] The phrase 'refer Appendix 5' should be 'refer Section 5', since the error analysis is in Section 5 of the main text.
  4. [Table 2] The reproduced F1 for Pandey and Singh (0.85) differs substantially from the originally reported 0.92; a brief explanation, such as a different evaluation split or metric conversion, would help readers interpret the baseline comparison.
  5. [Table 10] The table lists hyperparameter ranges, but the paper does not state which specific values (learning rate, batch size, sequence length, regularization strength) were used for the best configurations reported in Tables 3 and 4; please report the selected values for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark results evaluated against external datasets and public baselines, with no prediction that reduces to its inputs by construction.

full rationale

This is an empirical study, not a derivation. The three strategies (native sample mixing, MTL, and VMLM prompting/finetuning) are evaluated on the fixed code-mixed test splits of Khandelwal et al. (2018) and Swami et al. (2018), with baselines from prior non-self work. The central claim that native sample mixing and MTL improve humor and sarcasm F1-scores is established by comparing model configurations on these external benchmarks; the reported gains are not obtained by defining the evaluation metric in terms of the fitted training data. The paper does cite the authors' prior work (Mazumder et al., 2024) to motivate why native sample mixing might help, but that prior result is used only as motivation and is externally falsifiable; the current paper independently tests the strategy on new tasks and datasets. The MTL framework is attributed to Rotman and Reichart (2022), not to the authors' own prior work. The post-hoc observation that low-KL native datasets appear in the best configurations, and the error-analysis claim that samples related to hate improved in the MTL setting, are interpretations of the experimental results rather than inputs that force those results. The limitations section explicitly acknowledges missing gold Hindi data and possible task/domain interference, which further indicates the findings are contingent empirical observations. Statistical concerns about selecting 'up to' maxima over many configurations and the absence of variance estimates are correctness risks, not circularity: they do not make the reported result equivalent to its input by definition. No equation in the paper defines a predicted quantity in terms of the fitted value, and no self-citation is invoked to forbid alternatives or to supply a uniqueness theorem. The paper is self-contained against external benchmarks, so a non-circular finding with score 0 is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central claims rest on dataset-transfer and task-transfer assumptions, plus several hand-chosen training quantities such as augmented sample counts, regularization strength, and the number of unfrozen layers. The synthetic Hindi samples are generated data, not a newly postulated entity.

free parameters (4)
  • MTL regularization strength lambda = searched over {0, 5e-1, 5e-2, 5e-3, 5e-4}; final per-model value not reported
    Used in the joint loss for soft parameter sharing in Section E.1; the MTL results in Table 4 depend on this choice.
  • Augmented native sample count per dataset = 1,180 per class for humour datasets; 2,100 per class for NHD and SC-V2; 1,067 for iSarc
    Hand-chosen counts used to balance classes in Table 9; they directly determine the native-mixing training sets in Tables 3 and 4.
  • Number of unfrozen layers during MLM fine-tuning = last four layers unfrozen
    Stated in Section 3.2 as a fixed choice; affects all fine-tuned MLM results.
  • Clustering parameters for few-shot selection = not specified
    Appendix D.1 says few-shot examples were selected by clustering but does not give cluster count or distance metric, so the VMLM prompting results are not uniquely reproducible.
assumptions (4)
  • domain assumption Native English humour and sarcasm samples share enough linguistic and cultural context with Hindi-English code-mixed text for transfer to improve the code-mixed task.
    This is the transfer premise behind R1 and Section 3.2; it is motivated by Mazumder et al. (2024) but not independently proven for humour and sarcasm.
  • domain assumption Hate detection is a semantically related auxiliary task whose joint training improves humour and sarcasm detection.
    Section 3.3 justifies the choice of hate detection by semantic relatedness and the presence of hateful keywords in humour and sarcasm samples, but this is an assumption about task transfer.
  • domain assumption The test split is identical to that of Khandelwal et al. (2018) and Swami et al. (2018) and is not used for configuration selection.
    Appendix E.1 states the test set follows the original splits, but the paper does not describe the model selection protocol, so fair comparison depends on this assumption.
  • ad hoc to paper Google Translate Hindi translations preserve enough of the humorous and sarcastic meaning to serve as synthetic native samples.
    Section 3.2 uses Google Translate to create Hindi samples, but Appendix G reports that humour and sarcasm were lost in translation and that the authors decided to use only English samples for further experiments, partly contradicting this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection." pith.science (2026). https://pith.science/paper/HJWTH2OC

@misc{pith2026241212761,
  author       = {Pith},
  title        = {Pith review of: Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJWTH2OC}},
  note         = {Machine review of arXiv:2412.12761}
}
read the original abstract

In this paper, we reported our experiments with various strategies to improve code-mixed humour and sarcasm detection. Particularly, we tried three approaches: (i) native sample mixing, (ii) multi-task learning (MTL), and (iii) prompting and instruction finetuning very large multilingual language models (VMLMs). In native sample mixing, we added monolingual task samples to code-mixed training sets. In MTL learning, we relied on native and code-mixed samples of a semantically related task (hate detection in our case). Finally, in our third approach, we evaluated the efficacy of VMLMs via few-shot context prompting and instruction finetuning. Some interesting findings we got are (i) adding native samples improved humor (raising the F1-score up to 6.76%) and sarcasm (raising the F1-score up to 8.64%) detection, (ii) training MLMs in an MTL framework boosted performance for both humour (raising the F1-score up to 10.67%) and sarcasm (increment up to 12.35% in F1-score) detection, and (iii) prompting and instruction finetuning VMLMs couldn't outperform the other approaches. Finally, our ablation studies and error analysis discovered the cases where our model is yet to improve. We provided our code for reproducibility.

Figures

Figures reproduced from arXiv: 2412.12761 by the authors.

Figure 2
Figure 2. Prompt template for k-shot prompting uti￾lized for humor detection. English parts are marked in red and the Hindi parts are marked in blue. D.2 LoRA-adapter based finetuning: For the second scenario, we performed instruc￾tion finetuning on the VMLMs using the de￾fault parameters of LoRA adapter based su￾pervised finetuning given in LLaMA-Factory (Zheng et al., 2024). We utilized the same prompt template for giving i… view at source ↗
Figure 3
Figure 3. MTL architecture when number of tasks is two, i.e., T = 2. posting about issues. The indirect criticism and taunting nature make it a clear case of sarcasm.’ • VMLMs showed a tendency to pick key￾words, like in the non-sarcastic exam￾ple ‘Yeh bhee cricket kee deewani hain....’ (Gloss: ‘She is also crazy about cricket...’), Llama-3.1 predicted it sarcastic and gave the reason as ‘The phrase “Yeh bhee cricket kee deew… view at source ↗
Figure 4
Figure 4. Restructured dataset considered for MTL￾based experiments. E.2 Model configurations: We conducted all our experiments on a single NVIDIA A100 GPU card. We presented our considered set of VMLMs and their respective versions in [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Performance analysis with increasing context length. Here, the corresponding pie-chart represents the distribution of context length. 2. Shorter context: For example, in the humorous sample: “Sir @arvindkejriwal AAP karen to chamatkaar, BJP kare to balatkaar.”(Gloss: S…
Figure 6
Figure 6. Figure 6: Translated samples of Hindi humor and sarcasm directly from the native English dataset. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

  1. [1]

    @flypigmk uski g**d mein dum hai.. agar kisi aur ke g**d mein nahi hai to uske baap ka kya jaat hai... #sarcasm with #g**d

    Spelling errors:For instance, consider the sarcastic statement:“@flypigmk uski g**d mein dum hai.. agar kisi aur ke g**d mein nahi hai to uske baap ka kya jaat hai... #sarcasm with #g**d”(Gloss: @flypigmk, he has strong a**... if some- one else doesn’t have the a**, what does that say about his father’s caste... #sar- casm with #a**). Here, the MTL model ...

    work page

  2. [2]

    Sir @arvindkejriwal AAP karen to chamatkaar, BJP kare to balatkaar

    Shorter context:For example, in the humorous sample:“Sir @arvindkejriwal AAP karen to chamatkaar, BJP kare to balatkaar.”(Gloss:Sir @arvindkejriwal, If AAP does it, it’s a miracle, if BJP does it, it’s a rape.), the gating mechanism over- took the model without gating by detect- ing humorous contrast using the rhyming words“chamatkaar”(Gloss:miracle) and ...

    work page

  3. [4]

    A Survey of Code-switched Speech and Language Processing

    “I know who you are”: Character-based features for conversational humor recognition in Chinese. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 2927–2932, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Vaibhav Shukla, Manjira Sinha, and Tirthankar Dasgupta. 2019. Automatic humor detection from co...

    work page Pith review arXiv 2022

  4. [5]

    A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection

    A corpus of english-hindi code-mixed tweets for sarcasm detection.arXiv preprint arXiv:1805.11869. Leonard Tang, Alexander Cai, Steve Li, and Jason Wang. 2022. The naughtyformer: A transformer understands offensive humor. 15 Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2023. MVP: Multi-task super- vised pre-training for natural language genera-...

    work page Pith review arXiv 2022

  5. [7]

    A dataset for detecting irony in hindi- english code-mixed social media text. InProceed- ings of 4th Workshop on Sentic Computing, Sen- timent Analysis, Opinion Mining, and Emotion Detection (EMSASW 2018) Co-located with the 15th Extended Semantic Web Conference 2018 (ESWC 2018), Heraklion, Greece, June 4, 2018, volume 2111 ofCEUR Workshop Proceedings, pa...

    work page Pith review arXiv 2018

  6. [8]

    Later on, deep learning-based approaches gained pace and demonstrated promising results

    and shift of sentiment and various incon- gruities (Joshi et al., 2015). Later on, deep learning-based approaches gained pace and demonstrated promising results. The RNN, LSTM and CNN (or a combination of them) networks claimed to show strengths of semantic modelling (Ghosh and Veale, 2016; Zhang et al., 2016). Further, the combination of contextual pretr...

    work page 2020

  7. [12]

    Is the intention of this text to be humorous?

    comprised of 10k samples. Meaney et al. (2021) created this dataset with 5http://www.punoftheday.com Twitter posts (80%) and Kaggle Short Jokes samples6 (20%). While the hu- morous tweets were collected from hu- morous Twitter accounts (e.g. @hu- murous1liners and @conanobrien), and the non-humorous tweets were collected from some celebrity accounts (e.g....

    work page 2021

  8. [15]

    What does a sheep in denial say? Nahhhh002 Hindi T ranslation: इनकार म ें भेड़ क्या003 कहती ह ै? नहह004

    work page

Show all 18 references
  1. [16]

    Their phones have been blowing006 up.007 Hindi T ranslation: जब से सैमसंग क े बारे में008 खबर सामने आई ह ै

    Ever since the news came out about Sam-005 sung.... Their phones have been blowing006 up.007 Hindi T ranslation: जब से सैमसंग क े बारे में008 खबर सामने आई ह ै.... उनक े फोन धड़ाम हो रह े हैं।009 Sarcasm translations:010

  2. [17]

    You’re never going to have a white011 boyfriend, are you Jess? Nah, don’t think012 so

    “You’re never going to have a white011 boyfriend, are you Jess? Nah, don’t think012 so. ”013 Hindi T ranslation: “आप कभी भी एक श्वेत014 प्रेमी नहीं होने जा रह े हैं आप जेस ह ैं? नाह ऐसा नहीं015 सोचते”016

  3. [18]

    babe stop i’m about to gleek017 Hindi T ranslation: बेब स्टॉप मैं ग्लीक क े बारे018 में हूँ019 1 Figure 6:Translated samples of Hindi humor and sarcasm directly from the native English dataset. 29 HumorPrecision NLD→ Col POTD HaHa 16000 Model↓ CM CM+Hi+En CM+En CM+HiCM+Hi+EnCM...

  4. [2005]

    What’s the tallest building in your city? the library, because it has the most stories

    datasets. This is because they are rela- tively balanced and widely used in past stud- ies(Kenneth et al., 2024). On the contrary, we couldn’t find any appropriate Hindi humour dataset. Although Kumar et al. (2023) re- cently proposed one, it comprised dialogues taken from a f...

    2023

  5. [2010]

    G Richard Tucker

    Icwsm — a great catchy name: Semi- supervised recognition of sarcastic sentences in online product reviews.Proceedings of the Inter- national AAAI Conference on Web and Social Media, 4(1):162–169. G Richard Tucker. 1999. A global perspective on bilingualism and bilingual educa...

    1999

  6. [2015]

    The non-humorous samples were scraped from APnews, theNewYorkTimes, Yahoo! An- swer and Proverb websites

    (MIT licensed) dataset consists of humorous samples directly collected from the pun-of-the-day jokes website5. The non-humorous samples were scraped from APnews, theNewYorkTimes, Yahoo! An- swer and Proverb websites. Their authors performed a curated sampling of negative sampl...

    2021

  7. [2018]

    In Proceedings of the Second Workshop on Compu- tational Modeling of People’s Opinions, Person- ality, and Emotions in Social Media, pages 36–41, New Orleans, Louisiana, USA

    A dataset of Hindi-English code-mixed social media text for hate speech detection. In Proceedings of the Second Workshop on Compu- tational Modeling of People’s Opinions, Person- ality, and Emotions in Social Media, pages 36–41, New Orleans, Louisiana, USA. Association for Com...

    2021 arXiv

  8. [2021]

    and contextual attention mechanism for multi-modal Hindi-English code-mixed sarcasm detection (Bedi et al., 2023). A.5 Multi-task learning: Multi-task learning has been used widely in the field of natural language processing for the past few years (Xie et al., 2024; Li et al.,...

    2023

  9. [2022]

    Association for Computational Lin- guistics

    co-located with ICNLSP 2022, pages 8–13, Trento, Italy. Association for Computational Lin- guistics. Sripriya N, Thenmozhi Durairaj, Nandhini K, Bharathi B, Kishore Kumar Ponnusamy, Char- mathi Rajkumar, Prasanna Kumar Kumare- san, Rahul Ponnusamy, Subalalitha Chinnau- dayar N...

    2022 arXiv

  10. [2023]

    Frances Adriana Laureano De Leon, Harish Tay- yar Madabushi, and Mark Lee

    Humourhindinet: Humour detection in hindi web series using word embedding and con- volutional neural network.ACM Transactions on Asian and Low-Resource Language Information Processing. Frances Adriana Laureano De Leon, Harish Tay- yar Madabushi, and Mark Lee. 2024. Code- mixed...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.