Pith. sign in

REVIEW 3 major objections 4 minor 56 references

MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that data poisoning attacks on deep-learning code search models can be exposed at inference time by comparing a query's rankings with those of a semantically equivalent rewritten query, flagging large divergences as…

desk verdict Genuinely novel angle, but the evaluation never runs a backdoor-triggered query through the model, so the reported detection numbers don't support the central claim. read the letter →

arxiv 2507.11092 v1 pith:WKOVQWI3 submitted 2025-07-15 cs.SE

Gong Chen , Wenjie Liu , Xiaoyuan Xie , Xunzhu Tang , Tegawendé F. Bissyandé , Songqiang Chen This is my paper
classification cs.SE
keywords metamorphictestingdatapoisoningattackbackdoordetectioncodesearchsemanticequivalenceHybridSimilarityVariationtarget-triggermatchdeeplearningsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MT4DP claims that data poisoning attacks on deep-learning-based code search models can be detected by metamorphic testing at inference time, without access to training data. The Semantically Equivalent Metamorphic Relation demands that a query and a semantically equivalent rewritten query produce consistent search rankings, and a sufficiently large divergence flags the backdoor. On the CodeSearchNet-Python evaluation with three poisoning attacks and three code search models, the paper reports an average F1 of 27.32% and precision of 19.45%, improving on the best baseline ONION by 191% in F1 and 265% in precision. If this holds, it offers a model-agnostic, training-free way to warn users that a query may have triggered a poisoned code search model.

What carries the argument

The load-bearing object is the Semantically Equivalent Metamorphic Relation (SE-MR), instantiated by two follow-up query generators (synonym replacement and '[MASK]' replacement) and a divergence metric called Hybrid Similarity Variation (HSV). HSV combines an absolute similarity variation based on rank movement and a relative similarity variation based on semantic-score changes, weighted and normalized, and a query whose HSV exceeds the mean threshold is flagged. This machinery converts backdoor detection into a consistency check between outputs of related inputs, which is the standard metamorphic testing move.

What would settle it

Run MT4DP on a code search model that was actually trained on poisoned data and check its per-query flags against queries known to contain the target word: if the true-positive rate is no better than the rate measured when triggers are inserted at rank 25 into a clean model's results, the central detection claim collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the key discovery is that the backdoor in a poisoned code search model is not a property of the poisoned code snippet alone: it is a target-trigger match spanning the query and the snippet. Existing outlier- and representation-based detectors analyze code alone and therefore miss the attack; MT4DP instead breaks the match by replacing the suspected high-frequency target word in the source query with a low-frequency synonym or with the '[MASK]' token, re-ranks the same retrieved list by the follow-up query, and measures divergence with a hybrid similarity variation. When the divergence exceeds a threshold, the SE-MR is violated and the source query is labeled poisoned. The paper further reports that statistical analysis of violations across queries sharing a word can identify the actual attack target, and removing the suspected trigger changes rankings enough to confirm the target-trigger pair.

Load-bearing premise

The load-bearing assumption is that inserting the attack trigger into the code snippet at rank 25 of a query's result list (Section 5.1.1) produces search-result behavior equivalent to what a model genuinely poisoned during training would exhibit, so the positive labels in the evaluation are realistic rather than artifacts of dataset construction.

Editorial extensions

If this is right

  • A user query can be checked for backdoor triggering at inference time using only the model's own ranking and similarity scores, with no need for training data, labels, or model internals.
  • Because the framework is model-agnostic, it can be applied to any code search model that produces query–code similarity scores, including models beyond the three evaluated.
  • Statistical aggregation of violations per candidate word can expose the attack's target word, and trigger-removal cross-validation can expose the trigger itself.
  • Detection works across three different poisoning mechanisms (dead code injection, identifier renaming, constant unfolding), so it is tied to the target-trigger structure rather than to one trigger syntax.
  • The paper's reported accuracy, precision, and F1 improvements over ONION, if correct, indicate that code-target-only detectors leave most poisoned queries undetected.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's dataset labels a query as poisoned only when the trigger is inserted into the snippet at rank 25 of that query's result list; whether the measured F1 transfers to a model whose poison was learned during real training is an open question that Section 5.1.1 leaves unvalidated.
  • An attacker who chooses a mid-frequency word as the target, rather than a top-10 high-frequency word, could evade the frequency-based candidate selection; the paper does not test this case.
  • The self-referential threshold, namely the mean HSV over the same queries, makes the detector calibration-free but also means its false-positive rate depends on the distribution of clean queries in the current query batch.
  • The paper itself notes that a missing comparison against CodeDetector and a single-language evaluation on CSN-Python limit the breadth of the results; those are acknowledged limitations rather than parts of the central contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MT4DP, a metamorphic-testing-based framework for detecting data poisoning backdoors in DL-based code search models. Following the proposed SE-MR relation, MT4DP identifies high-frequency query words as suspicious attack targets, generates synonym-based and mask-based semantically equivalent follow-up queries, obtains a source ranking list, re-ranks that list under the follow-up queries, and computes an HSV score from absolute and relative similarity variations. A source query is flagged as poisoned when its HSV exceeds a threshold. The evaluation reports large F1, precision, and accuracy improvements over AC, SS, and ONION on CodeBERT, CodeT5, and a BiRNN-based model under DCI, IR, and CU attacks.

Significance. If the central claim were supported, this would be a meaningful step: it applies metamorphic testing to a two-part target-trigger backdoor setting rather than to classification-only defenses, and it reports comparisons across three models and three attack types with an ablation of the two follow-up generation strategies. The paper also articulates a clear motivation for why representation-based and outlier-based defenses miss the target-trigger match. However, the evaluation as constructed does not actually exercise the learned backdoor, and the decision rule uses the test set for both threshold and weight selection; these issues are load-bearing, so the reported state-of-the-art numbers do not establish the paper's claim as it stands.

major comments (3)
  1. [Section 5.1.1, Table 3, Section 6.1] The positive samples do not test the claimed backdoor effect. The detection dataset is built by first retrieving each source query's top-50 list, then inserting the attack trigger into the snippet that already sits at rank 25 and labeling the query as poisoned; the model is never run on a query/snippet pair in which the trigger is present during retrieval. The HSV therefore measures the generic effect of modifying a snippet's text and re-ranking under paraphrased queries, not the target-trigger association learned during poisoning. The case study in Figure 6 confirms this: the source list is captured before the trigger is considered, and the follow-up ranks 34 and 36 are produced by re-scoring an already-modified snippet. The reported F1 and precision values are consequently not evidence that MT4DP detects data poisoning at inference time.
  2. [Algorithm 1, line 29; Section 5.5] The threshold is computed as the mean HSVfinal of the same 1,000 queries being classified, and W1=0.7 is selected in RQ-4 as the value maximizing average F1 on the same evaluation set. This is test-set leakage: both the decision boundary and the metric-optimized weight are chosen using the labeled test queries, so the reported F1, precision, and accuracy are in-sample numbers. A comparison against baselines that do not receive this tuning is not informative; a held-out threshold or a validation-based selection procedure is required.
  3. [Section 5.1.1, Tables 5-10] MT4DP's metrics are computed at the query level while the baselines' metrics are computed at the code-snippet level. Adjusting the poisoning ratio to 10% in both settings does not make F1 and precision comparable, because the unit of classification differs and the positive base rates over the two instance sets are different. The headline 191% and 265% improvements therefore compare quantities that are not commensurable.
minor comments (4)
  1. [Section 5.1.6] The text states that experiments use PyTorch 3.7; PyTorch is not versioned this way, and the authors likely mean Python 3.7, which should be corrected.
  2. [Section 6.2] The citations in 'For DCI [43] and CU [38]' and 'For IR [25]' are inconsistent with the references used elsewhere; in this manuscript CU is [25] and IR is [38].
  3. [Section 5.1.1, Table 3] The relation between the 10 suspicious words, the 1,000 source queries, and the 100 poisoned queries is not stated precisely; the text says 100 source queries are retained per target, which would imply 1,000 total queries, but the distribution of the 100 positive labels across target words is left unclear.
  4. [Tables 8 and 10] Several cells in Tables 8 and 10 carry asterisks with no explanatory note; the note under Table 10 does not define the asterisk.

Circularity Check

3 steps flagged · score 7.0 of 10

Evaluation protocol makes the central detection claim circular: positive labels are constructed by post-hoc trigger insertion, and threshold and weight W1 are fitted on the same test queries.

  1. self definitional [Section 5.1.1 (Detection dataset); Section 4.4-4.5 / Algorithm 1]
    "We collected the search results of all queries and checked the code snippets through text analysis to ensure that all code snippets did not contain triggers. Then, we inserted the trigger into the code snippet at the 25th ranking in each rank list of these queries."

    The ground-truth 'poisoned query' is the query whose rank list received a trigger insertion (the queries containing the attack target). The source rank list was captured before the trigger was inserted, so the model never ranks the trigger-containing snippet under the source query; the alleged backdoor promotion effect is never observed. The detector's HSV compares the untouched source list with re-ranked lists computed on the same snippets after trigger insertion. Hence the detected signal is the trigger insertion itself, not the learned target-trigger association, and the positive labels are artifacts of dataset construction. Reported F1/precision therefore do not test the claimed backdoor-activation detection capability.

  2. fitted input called prediction [Algorithm 1 lines 28-32; Section 4.5]
    "As discussed in Section 5.1.1, due to the imbalance in the detection dataset, directly applying a fixed threshold may lead to biased decisions toward the majority class. To mitigate this issue, we take the mean value of all source queries' final HSV score as the threshold t (line 29). ... if it exceeds t, the source query is detected as poisoned; otherwise, the source query is considered clean."

    The decision threshold is computed as the mean of the final HSV scores of exactly the same 1,000 queries being classified, and the same scores are then compared with this threshold to produce the labels. This in-sample, data-dependent threshold means the evaluation metrics are computed on the data that set the decision boundary; no held-out or calibrated threshold is used. The 'detection' thus partially fits the evaluation set rather than providing an out-of-sample prediction, inflating the accuracy/F1 numbers reported as MT4DP's performance.

1 more flagged steps
  1. fitted input called prediction [Section 5.5 (RQ-4) and Section 5.1.6 (Implementation Details)]
    "we calculated the average F1 score of MT4DP under different W1 and W2 settings for each poisoning attack method and took the W1 setting with the highest average F1 score as the optimal setting. ... We follow the hyperparameter experimental results in RQ-4 and set W1 = 0.7 and W2 = 0.3."

    The weight coefficient W1=0.7 is selected by maximizing the F1 score on the same detection test sets whose F1 is then reported as the method's detection performance. This is test-set hyperparameter tuning: the reported F1 and precision are the result of fitting the HSV combination weights to the evaluation labels. The 191% and 265% improvements over baselines are therefore partly the product of in-sample selection rather than a prediction of detection quality on new, unseen poisoned queries.

full rationale

The paper's core metamorphic-testing idea is not inherently circular: the SE-MR states that semantically equivalent queries should return consistent rank lists, and breaking a suspected target-trigger match could in principle expose a backdoor. However, the central empirical claim is validated in a circular way. In Section 5.1.1 the positive samples are constructed by inserting the attack trigger into the rank-25 snippet of a query's pre-computed source rank list, after that list was produced; the detector therefore never sees the backdoor's actual effect (promotion of the poisoned snippet in response to the target query). The 'poisoned query' label is assigned by query-word identity plus trigger insertion, and the HSV signal measures the consequence of that insertion rather than a learned target-trigger association. This means the reported F1/precision values are, to a substantial degree, artifacts of dataset construction. Compounding this, Algorithm 1 sets the detection threshold to the mean HSV of the very queries being classified, and Section 5.5 selects W1=0.7 by maximizing F1 on the same test sets; both choices make the reported metrics in-sample fits. No load-bearing self-citation chain is present: attacks [38,43], baselines [3,31,42], and the code-search models are external prior work. Because the reported 191%/265% improvements reduce in part to the construction of the positive labels and to test-set fitted threshold/weights, the paper warrants a circularity score of 7: the evaluation protocol, not the MT4DP concept itself, forces much of the reported detection performance.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central claim depends on several hand-chosen or fitted values: the threshold and weight are tuned on the evaluation set, and the trigger insertion rank, list size, and candidate word count are arbitrary design choices. The SE-MR and HSV constructs are new but lack external validation, and the key evaluation assumption that rank-25 trigger insertion emulates real poison attacks is ad hoc to this paper.

free parameters (5)
  • Threshold t = mean of all final HSV scores in the detection dataset
    Set in Algorithm 1 line 29 as the average HSV over all queries under test; this makes the decision boundary depend on the same data being classified and is effectively fitted to the evaluation set.
  • Weight W1 = 0.7 (W2 = 0.3)
    Selected in RQ-4 by comparing average F1 on the test set for different values; this is test-set hyperparameter tuning.
  • Trigger insertion rank = 25
    Positive queries are constructed by inserting the attack trigger into the snippet at rank 25 of the search result list; performance depends on this arbitrary position.
  • Suspicious word list size = 10
    Top 10 high-frequency words are treated as candidate attack targets without a principled cutoff.
  • Rank list length = 50
    Only top 50 snippets per query are retained for re-ranking; results may change with a different cutoff.
assumptions (4)
  • domain assumption A clean code search model should return consistent rankings for a query and its semantically equivalent rewritten query.
    Section 4.2. The entire method rests on this relation; the paper itself shows even a clean model produces average rank change 86.9 (Table 1), so consistency is only approximate and depends on a threshold.
  • domain assumption High-frequency words are the only plausible attack targets and low-frequency words cannot be part of the trigger match.
    Section 3.1 and Section 4.3.1. Used to justify replacing high-frequency words with low-frequency synonyms and [MASK]; prior attacks support it, but it bounds the method's scope.
  • domain assumption Synonym and [MASK] replacement preserves query intent and breaks the target-trigger match.
    Section 4.3.2. No human evaluation or external benchmark verifies the semantic equivalence of the generated follow-up queries; the selection of synonyms itself uses the model under test.
  • ad hoc to paper Inserting a trigger into the snippet at rank 25 of the search results creates a realistic poisoned query.
    Section 5.1.1. This is the central evaluation assumption; it does not use the actual ranking behavior of the trained poisoned model, so the constructed positives may not match real attack effects.
invented entities (2)
  • SE-MR (Semantically Equivalent Metamorphic Relation)
    purpose: Defines the relation between source and follow-up query search results used as the detection oracle.
    The relation is novel to this paper but is not validated against any external benchmark; it is operationalized only through a threshold and the HSV metric.
  • HSV (Hybrid Similarity Variation), ASV, RSV
    purpose: Metrics to quantify ranking and similarity changes between source and follow-up rank lists for detection.
    These metrics are introduced in the paper and their W1/W2 weighting is tuned on the test set; no independent falsifiable handle is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing." pith.science (2026). https://pith.science/paper/WKOVQWI3

@misc{pith2026250711092,
  author       = {Pith},
  title        = {Pith review of: MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WKOVQWI3}},
  note         = {Machine review of arXiv:2507.11092}
}
read the original abstract

Recently, several studies have indicated that data poisoning attacks pose a severe security threat to deep learning-based (DL-based) code search models. Attackers inject carefully crafted malicious patterns into the training data, misleading the code search model to learn these patterns during training. During the usage of the poisoned code search model for inference, once the malicious pattern is triggered, the model tends to rank the vulnerability code higher. However, existing detection methods for data poisoning attacks on DL-based code search models remain insufficiently effective. To address this critical security issue, we propose MT4DP, a Data Poisoning Attack Detection Framework for DL-based Code Search Models via Metamorphic Testing. MT4DP introduces a novel Semantically Equivalent Metamorphic Relation (SE-MR) designed to detect data poisoning attacks on DL-based code search models. Specifically, MT4DP first identifies the high-frequency words from search queries as potential poisoning targets and takes their corresponding queries as the source queries. For each source query, MT4DP generates two semantically equivalent follow-up queries and retrieves its source ranking list. Then, each source ranking list is re-ranked based on the semantic similarities between its code snippets and the follow-up queries. Finally, variances between the source and re-ranked lists are calculated to reveal violations of the SE-MR and warn the data poisoning attack. Experimental results demonstrate that MT4DP significantly enhances the detection of data poisoning attacks on DL-based code search models, outperforming the best baseline by 191% on average F1 score and 265% on average precision. Our work aims to promote further research into effective techniques for mitigating data poisoning threats on DL-based code search models.

Figures

Figures reproduced from arXiv: 2507.11092 by the authors.

Figure 1
Figure 1. The detection principles comparison of MT4DP and baselines. Secondly, there is a gap between detection methods and data poisoning attacks on DL￾based code search models. Several studies have been applied to detect data poisoning attacks on DL-based code search models [25, 38, 43]. However, these detection methods are typically designed for classification tasks. These detection methods typically fall into two categor… view at source ↗
Figure 2
Figure 2. The process of data poison attack on DL-based code search models. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The process of MT. 3.3 Principle of MT4DP In this work, we detect data poisoning attacks on DL-based code search models via MT. Our intuition is that the essence of the data poisoning attack is to change the search results, so it is straightforward that poisoning attacks can be detected by analyzing changes in the search results. Therefore, we conducted a preliminary experiment to explore the correlation between the… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The overview of MT4DP. 4.2 Semantically Equivalent Metamorphic Relation The MR is the core of MT. In this study, we propose a semantically equivalent metamorphic relation, named SE-MR. We adopt the SE-MR to guide the generation of follow-up queries and poisoning attack…
Figure 5
Figure 5. Figure 5: The average 𝐹 1 scores with different 𝑊1. Combined with the detection performance of baselines shown in [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: A poison detection sample of MT4DP. This case study illustrates the workflow of MT4DP. It also effectively demonstrates the effective￾ness of MT4DP. 6.2 Target-Trigger Analysis Target Analysis. After the detection, we can obtain the detection result for each query. How…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 18 canonical work pages

  1. [1]

    2025. Github. Retrieved from https://github.com/

    work page 2025

  2. [2]

    Muhammad Hilmi Asyrofi, Zhou Yang, Imam Nur Bani Yusuf, Hong Jin Kang, Ferdian Thung, and David Lo. 2022. BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems. IEEE Transactions on Software Engineering 48, 12 (2022), 5087–5101. https://doi.org/10.1109/TSE.2021.3136169

    work page arXiv 2022

  3. [3]

    Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. 2018. Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering. arXiv:1811.03728 [cs.LG] https://arxiv.org/abs/1811.03728

    arXiv 2018

  4. [4]

    Gong Chen, Wenjie Liu, and Xiaoyuan Xie. 2025. RFMC-CS: A Representation Fusion Based Multi-View Momentum Contrastive Learning Framework for Code Search. Automated Software Engineering 32, 1 (Jan. 2025), 16. https: //doi.org/10.1007/s10515-025-00487-8

    work page doi:10.1007/s10515-025-00487-8 2025

  5. [5]

    Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. 2021. BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models. arXiv:2110.02467 [cs.CL] https://arxiv.org/ abs/2110.02467

    arXiv 2021

  6. [6]

    Songqiang Chen, Shuo Jin, and Xiaoyuan Xie. 2021. Testing Your Question Answering Software via Asking Recursively. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 104–116. https: //doi.org/10.1109/ASE51524.2021.9678670

    arXiv 2021

  7. [7]

    Songqiang Chen, Shuo Jin, and Xiaoyuan Xie. 2021. Validation on Machine Reading Comprehension Software without Annotated Labels: A Property-Based Method. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ACM, Athens Greece, 590–602. https://doi.org/10.1145/34...

    arXiv 2021

  8. [8]

    Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. ACM Comput. Surv. 51, 1, Article 4 (Jan. 2018), 27 pages. https://doi.org/10.1145/3143561

    doi:10.1145/3143561 2018

Show all 56 references
  1. [9]

    Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang

  2. [10]

    Yuchen Chen, Weisong Sun, Chunrong Fang, Zhenpeng Chen, Yifei Ge, Tingxu Han, Quanjun Zhang, Yang Liu, Zhenyu Chen, and Baowen Xu. 2025. Security of Language Models for Code: A Systematic Literature Review. arXiv:2410.15631 [cs.SE] https://arxiv.org/abs/2410.15631

    2025 arXiv

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2019 doi

  4. [12]

    Zuohua Ding, Qingfen Zhang, and Mingyue Jiang. 2020. Metamorphic Testing of Code Search Engines. In 2020 International Symposium on Theoretical Aspects of Software Engineering (TASE) . 177–184. https://doi.org/10.1109/ TASE49443.2020.00032

    2020

  5. [13]

    Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder. 2018. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In Proceedings of the 27th ACM SIGS...

    2018

  6. [14]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics . Assoc...

    2020 doi

  7. [15]

    Ranasinghe, and Surya Nepal

    Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. 2019. STRIP: a defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference (San Juan, Puerto Rico, USA) (ACSAC ’19)....

    2019

  8. [16]

    Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733 [cs.CR] https://arxiv.org/abs/1708.06733

    2019 arXiv

  9. [17]

    Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep Code Search. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden). Association for Computing Machinery, New York, NY, USA, 933–944. https://doi.org/10.1145/3180155.3180167

    2018

  10. [18]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational , Vol. 1, No. 1, Article . Publication date: September ...

    2022 doi

  11. [19]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Rep...

    2021

  12. [20]

    Pinjia He, Clara Meister, and Zhendong Su. 2020. Structure-invariant testing for machine translation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 9...

    2020

  13. [21]

    Pinjia He, Clara Meister, and Zhendong Su. 2021. Testing Machine Translation via Referential Transparency. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . 410–422. https://doi.org/10.1109/ICSE43902. 2021.00047

    2021

  14. [22]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  15. [23]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs/1909.09436 (2019). arXiv:1909.09436 [cs.LG]

    2019 arXiv

  16. [24]

    Mingyue Jiang, Tsong Yueh Chen, and Shuai Wang. 2022. On the Effectiveness of Testing Sentiment Analysis Systems with Metamorphic Testing. Information and Software Technology 150 (Oct. 2022), 106966. https://doi.org/10.1016/j. infsof.2022.106966

    2022

  17. [25]

    Jia Li, Zhuo Li, Huangzhao Zhang, Ge Li, Zhi Jin, Xing Hu, and Xin Xia. 2024. Poison Attack and Poison Detection on Deep Source Code Processing Models. 33, 3, Article 62 (March 2024), 31 pages. https://doi.org/10.1145/3630008

    2024 doi

  18. [26]

    Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. 2021. DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 263–274. https://doi.org/10....

    2021

  19. [27]

    Erik Linstead, Sushil Bajracharya, Trung Ngo, Paul Rigor, Cristina Lopes, and Pierre Baldi. 2009. Sourcerer: Mining and Searching Internet-Scale Software Repositories. Data Mining and Knowledge Discovery 18, 2 (April 2009), 300–336. https://doi.org/10.1007/s10618-008-0118-x

    2009 doi

  20. [28]

    Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning Attack on Neural Networks. In Proceedings 2018 Network and Distributed System Security Symposium . Internet Society, San Diego, CA. https://doi.org/10.14722/ndss.2018.23291

    2018

  21. [29]

    Fei Lv, Hongyu Zhang, Jian-guang Lou, Shaowei Wang, Dongmei Zhang, and Jianjun Zhao. 2015. CodeHow: Effective Code Search Based on API Understanding and Extended Boolean Model. In2015 30th IEEE/ACM International Conference on Automated Software Engineering. 260–270. https://do...

    2015 doi

  22. [30]

    Collin McMillan, Mark Grechanik, Denys Poshyvanyk, Qing Xie, and Chen Fu. 2011. Portfolio: finding relevant functions and their usage. In 2011 33rd International Conference on Software Engineering . 111–120. https://doi.org/10. 1145/1985793.1985809

    2011

  23. [31]

    Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Lin...

    2021

  24. [32]

    Yubin Qu, Song Huang, and Yongming Yao. 2024. A survey on robustness attacks for deep code models. Automated Software Engineering. 31, 2 (Aug. 2024), 40 pages. https://doi.org/10.1007/s10515-024-00464-7

    2024 doi

  25. [33]

    Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You Autocomplete Me: Poisoning Vulnera- bilities in Neural Code Completion. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 1559–1575. https://www.usenix.org/conference/usenixs...

    2021

  26. [34]

    Sanchez, and Antonio Ruiz-Cortés

    Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cortés. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9 (Sept. 2016), 805–824. https://doi.org/10.1109/TSE.2016.2532875

    2016

  27. [35]

    Jinyang Shao. 2021. Testing Object Detection for Autonomous Driving Systems via 3D Reconstruction. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . 117–119. https://doi.org/10.1109/ICSE-Companion52605.2021.00052

    2021

  28. [37]

    Jacob Steinhardt, Pang Wei Koh, and Percy Liang. 2017. Certified defenses for data poisoning attacks. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA...

    2017

  29. [38]

    Weisong Sun, Yuchen Chen, Guanhong Tao, Chunrong Fang, Xiangyu Zhang, Quanjun Zhang, and Bin Luo. 2023. Backdooring Neural Code Search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers, Jordan Boyd...

    2023 doi

  30. [39]

    Weisong Sun, Chunrong Fang, Yuchen Chen, Guanhong Tao, Tingxu Han, and Quanjun Zhang. 2022. Code Search Based on Context-Aware Code Translation. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania). Association for Computing Ma...

    2022

  31. [40]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks (ICML’17). JMLR.org, 10 pages

    2017

  32. [41]

    Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-Neural-Network- Driven Autonomous Cars. InProceedings of the 40th International Conference on Software Engineering. ACM, Gothenburg Sweden, 303–314. https://doi.org/10.1145/3180155.3180220

    2018

  33. [42]

    Brandon Tran, Jerry Li, and Aleksander Mądry. 2018. Spectral signatures in backdoor attacks. InProceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 8011–8021

    2018

  34. [43]

    Yao Wan, Shijie Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Dezhong Yao, Hai Jin, and Lichao Sun. 2022. You see what I want you to see: poisoning vulnerabilities in neural code search. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium...

    2022

  35. [44]

    Wenxuan Wang, Jen-tse Huang, Weibin Wu, Jianping Zhang, Yizhan Huang, Shuqing Li, Pinjia He, and Michael R. Lyu

  36. [45]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder- Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for...

    2021 doi

  37. [46]

    Xiaoyuan Xie, Shuo Jin, Songqiang Chen, and Shing-Chi Cheung. 2024. Word Closure-Based Metamorphic Testing for Machine Translation. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 203 (Nov. 2024), 46 pages. https://doi.org/10. 1145/3675396

    2024

  38. [47]

    Xiaoyuan Xie, Xingpeng Li, and Songqiang Chen. 2024. Metamorphic Testing of Image Captioning Systems via Image-Level Reduction. IEEE Transactions on Software Engineering 50, 11 (2024), 2962–2982. https://doi.org/10.1109/ TSE.2024.3463747

    2024

  39. [48]

    Chang Xu, Jun Wang, Yuqing Tang, Francisco Guzmán, Benjamin I. P. Rubinstein, and Trevor Cohn. 2021. A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning. In Proceedings of the Web Conference 2021 (WWW ’21) . ACM, 3638–3650. https://doi.org/10...

    2021

  40. [49]

    Ling Xu, Huanhuan Yang, Chao Liu, Jianhang Shuai, Meng Yan, Yan Lei, and Zhou Xu. 2021. Two-Stage Attention-Based Model for Code Search with Textual and Structural Features. In2021 IEEE International Conference on Software Analysis, Evolution and Reengineering. 342–353. https:...

    2021

  41. [50]

    Zhang, Hong Jin Kang, Jieke Shi, Junda He, and David Lo

    Zhou Yang, Bowen Xu, Jie M. Zhang, Hong Jin Kang, Jieke Shi, Junda He, and David Lo. 2024. Stealthy Backdoor Attack for Code Models. IEEE Transactions on Software Engineering 50, 4 (2024), 721–741. https://doi.org/10.1109/TSE. 2024.3361661

    2024

  42. [51]

    Zhang, Mark Harman, Lei Ma, and Yang Liu

    Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering 48, 1 (Jan. 2022), 1–36. https://doi.org/10.1109/TSE.2019.2962027

    2022

  43. [52]

    Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) . 132...

    2018

  44. [53]

    Quan Zhang, Yifeng Ding, Yongqiang Tian, Jianmin Guo, Min Yuan, and Yu Jiang. 2021. AdvDoor: adversarial backdoor attack of deep learning system. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, Denmark) (ISSTA 2021). As...

    2021

  45. [54]

    Zhiyi Zhang, Pu Wang, Hongjing Guo, Ziyuan Wang, Yuqian Zhou, and Zhiqiu Huang. 2021. DeepBackground: Metamorphic Testing for Deep-Learning-driven Image Recognition Systems Accompanied by Background-Relevance. Information and Software Technology 140 (Dec. 2021), 106701. https:...

    2021

  46. [2021]

    In Annual Computer Security Applications Conference

    BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements. In Annual Computer Security Applications Conference. ACM, Virtual Event USA, 554–569. https://doi.org/10.1145/3485832.3485837

  47. [2023]

    In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    MTTM: Metamorphic Testing for Textual Content Moderation Software. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . 2387–2399. https://doi.org/10.1109/ICSE48619.2023.00200

    2023

  48. [9566]

    https://doi.org/10.18653/v1/2021.emnlp-main.752

    2021 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.