Pith. sign in

REVIEW 3 major objections 5 minor 32 references

This paper claims that defining metamorphic relations directly over the source code a transpiler emits—rather than over the runtime behavior of compiled binaries—lets testers find faults that pure fuzzing cannot.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:49 UTC pith:IU4447AK

load-bearing objection A genuinely new metamorphic-testing angle for transpilers, with an honest but under-validated case study; worth refereeing. the 3 major comments →

arxiv 2607.29247 v1 pith:IU4447AK submitted 2026-07-31 cs.SE

Metamorphic Testing of Transpilers via Mutation Consistency of Programs

Enea Raffaele Ilario Papaleo , Luca Guglielmo , Giovanni Denaro This is my paper
classification cs.SE
keywords transpiler testingmetamorphic testingmutation consistencysource-to-source compilertest oracledomain-specific languageprogram mutation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Transpilers convert domain-specific language (DSL) programs into general-purpose source code, and their faults can silently corrupt deployed systems. Existing compiler-testing oracles assume the generated program can be executed, which is often impossible for transpiler output that requires special hardware or toolchains. This paper proposes to check the structure of the generated source instead: when a small mutation is applied to the input DSL program, the output source should change in a predictable, localized way. The authors instantiate nine such 'mutation consistency' relations for an industrial control-logic DSL, implement a tool called MCP-Tester, and seed faults into the transpiler's bytecode. In ten 12-hour sessions, the metamorphic oracles flagged 6 to 43 injected faults per session that produced no crash, while a pure-fuzzing baseline only saw crashes.

Core claim

The central claim is that mutation consistency of transpiled programs is a viable oracle for transpiler testing. For each mutation operator on the input DSL (adding a configuration parameter, an actuator command, an enum value, etc.), there are expected code-level equivalences and differences in the output C sources: equivalence checks assert that certain regions remain unchanged, and difference checks assert that specific additions appear, such as a struct field and a getter. The paper's case study shows that these source-level oracles catch silent miscompilation faults—cases where the transpiler produces compilable but semantically wrong output—while a fuzzer that applies the same mutation

What carries the argument

An MCP metamorphic relation is a pair (mutation operator, consistency checkers). The mutation operator makes a small grammar-valid change to the DSL source, such as adding an enum value or a configuration parameter. The consistency checkers are hand-written predicates over diffs of the transpiler's output: equivalence checks verify that unaffected code regions are unchanged, and difference checks verify that expected structural additions appear in the correct files and forms. MCP-Tester iterates over seed DSL programs, relations, and mutation spots, runs the transpiler on base and follow-up inputs, and reports a failure whenever the output diff violates the checkers.

Load-bearing premise

The soundness of the approach rests on the hand-written consistency checkers: if a checker encodes a wrong expectation about what a correct transpiler should emit, then the 'oracle failures' it reports are false alarms rather than real faults.

What would settle it

Run MCP-Tester with the nine relations against a transpiler known to be correct for the DSL constructs under test—for example, a reference implementation or a manually audited version. Any reported oracle failure on that correct transpiler would demonstrate that a checker is wrong. Alternatively, take the confirmed oracle failures from the paper, translate the base DSL program with a trusted tool, and check whether the flagged diff genuinely contradicts the expected structural change; a single false positive among the confirmed failures would show the checker is unsound.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MCP-Tester exposes silent miscompilation faults in transpilers—faults that yield compilable but semantically wrong output—not just crashes.
  • The oracle overhead is negligible: the fuzzer baseline executed nearly the same number of transpiler runs, so checking source-code diffs costs little.
  • The technique works in settings where compiled binaries cannot be executed, such as hardware-in-the-loop or simulator-dependent outputs.
  • Because the relations are derived from the DSL grammar and documented requirements, extending the approach to new DSL constructs only requires writing additional mutation-consistency relations.
  • Ablating the oracles (pure fuzzing) did not increase crash detection, confirming that the metamorphic oracles are the source of the added value.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The consistency-checker idea could be automated: expected diff templates might be inferred from a DSL grammar together with a small set of manually verified correct translations, reducing manual effort and the risk of human error in checker design.
  • MCP testing could generalize beyond transpilers to any source-to-source transformation tool—code generators, minifiers, or formatters—whenever the output cannot be readily executed.
  • The experimental confirmation step re-runs the same checker to confirm a failure; a stronger confirmation would independently check each reported failure against a manually inspected expected output, which would also estimate the false-positive rate of the hand-written checkers.
  • The seed programs were grammar-generated; mixing in real-world DSL programs could exercise different constructs and yield a different fault-detection profile.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MCP (Mutation Consistency of transpiled Programs), a metamorphic testing technique for transpilers in which metamorphic relations are defined directly over the generated source code rather than over runtime behavior. Each relation is a pair of a mutation operator on the input DSL program and a set of consistency checkers on the output code, prescribing structural equivalences and differences between the outputs for the original and mutated inputs. The MCP-Tester tool instantiates this workflow, mutating DSL seeds, running the transpiler on both inputs, and checking the output-source diffs against the hand-written checkers. The evaluation is an industrial case study: nine MCP relations for an anonymized control-logic DSL, 297 grammar-generated valid seeds, seeded branch faults injected into the transpiler bytecode, and ten 12-hour sessions comparing MCP-Tester with a no-oracle fuzzer version using the same mutations and random seeds. The paper reports a median of 39 confirmed oracle-detected failures versus 13 crashes for MCP-Tester, while the fuzzer sees essentially the same crash counts and no oracle failures, and concludes that the approach reveals faults that pure fuzzing misses.

Significance. If the consistency checkers are semantically valid, the MCP idea is a genuine and practical contribution: it provides an oracle for silent miscompilation in source-to-source translators when the generated code cannot be executed with available toolchains, hardware, or inputs. This extends metamorphic testing beyond the compiled-binary assumption that dominates compiler testing. The paper is also honest about its scope, limiting claims to the studied transpiler and relations, and the artifact is promised as open source. The main strength is the ablation design: the fuzzer baseline uses the same mutations, the same seeds, and the same random sequence, so the only difference is the presence of the metamorphic oracles. However, the empirical backbone depends entirely on the correctness of nine hand-written consistency checkers, and the validation of those checkers is currently only self-confirmation by re-execution. That gap, together with an error in the reported median, prevents the current version from fully supporting the central claim.

major comments (3)
  1. [§4.1 and §4.3; Fig. 3 and Fig. 4] The entire advantage of MCP-Tester over the fuzzer lies in the 'Oracle failures' column of Fig. 4, and those failures are produced by the hand-written consistency checkers of Fig. 3. The paper states that the checkers are derived from 'the DSL grammar and the documented requirements of the transpiler,' but those requirements are confidential and the checkers for eight of the nine relations are omitted. The confirmation step in §4.3 only re-executes the same pair of test cases with the same metamorphic relation; it verifies reproducibility, not whether the expected output diff is a semantic requirement. If a checker encodes an implementation convention, for example the specific getter name get_<NAME> in the 'Add Configuration Parameter' relation, rather than a semantic requirement, then the reported oracle failures could be false positives. The paper does not report how many oracle failur
  2. [§4.4, Fig. 4] The reported summary statistics do not match the table. The ten oracle-failure counts are 33, 40, 6, 20, 28, 40, 39, 40, 27, 43; the median of these ten values is 36, not 39. Similarly, the crash counts 13, 15, 5, 5, 12, 16, 12, 14, 12, 14 have median 12.5, not 13. Since these medians are the paper's headline quantitative evidence, they must be corrected and the conclusions restated accordingly.
  3. [§4.2 and §4.5] The abstract and RQ2 use the phrase 'pure fuzzing' for the baseline, but the baseline is MCP-Tester with the oracles removed and the same DSL-mutation operators. This is a legitimate ablation for assessing the cost and benefit of the metamorphic oracles, and it is explicitly acknowledged in §4.5. The wording nevertheless overstates the comparison: the authors do not compare with a coverage-guided or grammar-based fuzzer such as AFL. The conclusion should be reframed as 'fuzzing with the same mutations and no oracle misses these faults,' and the 'pure fuzzing' phrasing in the abstract and Section 1 should be adjusted.
minor comments (5)
  1. [§4.1] Typo: 'matamorphic relations' should be 'metamorphic relations'.
  2. [§4.3] Typo: 'we remark tat this confirmation step' should be 'we remark that'.
  3. [§2.1 and Fig. 2] The description of Figure 2 says the enumeration bug manifests 'between lines 0–0 of trafficlight.h'; the line numbers should be concrete, and the duplicated sentence 'The code is still compilable and executable, but it is no more compliant...' should be removed.
  4. [§3.2] In the enumeration of possible spots for inserting NEWVAL, the text says 'between the Green and Blue identifiers,' but the worked example has states Red, Green, Yellow, with no Blue. The wording should refer to Yellow.
  5. [§4.3] The explanation of the fault-injection heuristic is dense; it would help to state explicitly whether the 'Crashes' and 'Oracle failures' columns count unique injected faults or individual failing test cases, since Fig. 4 also reports unique 'Injected faults' separately.

Circularity Check

0 steps flagged

No significant circularity: the metamorphic checkers are specification-derived and the fault injections are independent, so the central claim does not reduce to its inputs.

full rationale

The paper's central derivation—MCP metamorphic relations over transpiled source code—is not built from the data it claims to predict. The consistency checkers in Fig. 3 are stated to be derived from 'the DSL grammar and the documented requirements of the transpiler' (§4.1), i.e., an independent specification source, not from the transpiler under test or from the injected faults. The evaluation's injected faults are branch-opcode mutations seeded into the transpiler bytecode (§4.3), and the oracle failures are produced by applying the pre-defined checkers to the outputs; no parameter was fitted to make the checkers match the fault set. The confirmation step (§4.3) re-executes the same pair to check reproducibility, which validates stability but not checker correctness—that is an internal-validity threat, not a circular reduction. The comparison against pure fuzzing is meaningful because the fuzzer uses the same mutation operators and seeds and differs only in omitting the oracles; crash counts are comparable, and the oracle failures are exactly the silent miscompilations the approach is designed to expose. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled via prior work. Thus the central claim has independent empirical content and no step reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The technique has no fitted scientific constants; the free parameters are testing-budget hyper-parameters. The main assumptions are about the validity of the hand-authored checkers and the representativeness of injected faults.

free parameters (3)
  • max_spots = 4
    Number of mutation applications per seed per MR; manually set in Section 4.3, bounds the number of test cases and affects the counts.
  • max_time = 12 hours per session
    Testing budget; manually chosen; RQ2 compares MCP and fuzzer under the same budget.
  • max_mr per instance = 3, 2, 2, 2
    Distributes the nine MRs across four parallel instances so each is picked exactly once per seed; manual configuration.
axioms (3)
  • domain assumption MCP metamorphic relations can be defined a priori from the DSL grammar and documented requirements.
    Section 4.1 states MRs are derived from the grammar and documented requirements; if those requirements are wrong, oracle failures are meaningless.
  • domain assumption Structural equivalence/difference of generated source text is a valid proxy for semantic correctness of a transpiler.
    The whole technique compares text-level diffs (Section 3.1); a transpiler could be structurally consistent yet semantically wrong, and this is not tested.
  • domain assumption Injected bytecode branch mutations are representative of real transpiler faults.
    Fault injection targets ifeq/ifne/ifnonnull/ifnull opcodes and discards crash-inducing faults (Section 4.3); representativeness is asserted, not demonstrated.

pith-pipeline@v1.3.0-daily-deepseek · 16483 in / 10957 out tokens · 104105 ms · 2026-08-03T10:49:34.283835+00:00 · methodology

0 comments
read the original abstract

Transpilers are increasingly used for software development, especially in industrial domains that rely on domain-specific languages (DSLs), to allow engineers to work with familiar concepts and appropriate abstractions. Ensuring the correctness of these instruments is therefore critical in many industrial settings. This paper observes that existing approaches for compiler testing hardly generalize to transpilers. Differential testing approaches are hindered as multiple equivalent implementations of the transpiler under test are seldom available in practice. The approaches based on metamorphic testing assume the ability to execute the compiled binaries, an assumption that cannot be always made for transpilers, which oftentimes produce results expressed as source code, requiring complex toolchains, hardware-in-the-loop setups, and depending on non trivial inputs. This paper introduces a novel metamorphic testing technique tailored to transpilers. Instead of reasoning about the runtime behavior of compiled programs, our approach defines metamorphic relations directly over the source code produced by the transpiler. These relations capture a property that we call mutation consistency of the (transpiled) programs: mutation-style changes in the input DSL program must induce predictable and structurally consistent changes in the generated output. We implemented this idea in a tool, MCP-Tester, and evaluated it through a case study conducted in the context of a technology-transfer project. Our current empirical results indicate that the proposed approach can effectively reveal faults that would remain undetected with pure fuzzing.

Figures

Figures reproduced from arXiv: 2607.29247 by Enea Raffaele Ilario Papaleo, Giovanni Denaro, Luca Guglielmo.

Figure 1
Figure 1. Figure 1: A sample traffic-light controller in a custom DSL [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The output of the execution of a (bugged) transpiler [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Eventually, it might add the identifier 𝑁 𝐸𝑊𝑉𝐴𝐿 as the first value of the Enumeration in the program, and observe the failure of the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The MCP metamorphic relations instantiated in our case study transpiler has added a proper getter in the produced C code. For instance, if the mutation operator of the metamorphic relation Add configuration parameter is applied to add a line as: c on f i gu r a t i on parameter MCPTEST_a26c5 : B o olea n at the end of the declarations section in the sample code of Fig￾ure 1, then the difference checks veri… view at source ↗
Figure 4
Figure 4. Figure 4: Results of MCP-Tester and the Fuzzer version per testing session causes. First, some transpiler runs may not traverse any yet-non￾executed fault. Second, since transpiler runs are done in parallel, synchronization delays may lead them to activate the same injected fault. Furthermore, no data on oracle failures is available for the fuzzer version of the tool, as the fuzzer does not check oracles. The result… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 2 canonical work pages

  1. [1]

    Jon Ayerdi, Valerio Terragni, Gunel Jahangirova, Aitor Arrieta, and Paolo Tonella

  2. [2]

    Andrés Bastidas Fuertes, María Pérez, and Jaime Meza Hormaza. 2023. Tran- spilers: A Systematic Mapping Review of Their Usage in Research and Industry. Applied Sciences13, 6 (2023). doi:10.3390/app13063667

    work page doi:10.3390/app13063667 2023

  3. [3]

    Patryk Chaber and Maciej Ławryńczuk. 2016. Effectiveness of PID and DMC control algorithms automatic code generation for microcontrollers: Application to a thermal process. In2016 3rd Conference on Control and Fault-Tolerant Systems (SysTol). IEEE, 618–623

    2016

  4. [4]

    Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing.Acm Computing Surveys (Csur)53, 1 (2020), 1–36

    2020

  5. [5]

    Alastair F Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated testing of graphics shader compilers.Proceedings of the ACM on Programming Languages1, OOPSLA (2017), 1–29. Metamorphic Testing of Transpilers via Mutation Consistency of Programs

    2017

  6. [6]

    Chris Hawblitzel, Shuvendu K Lahiri, Kshama Pawar, Hammad Hashmi, Sedar Gokbulut, Lakshan Fernando, Dave Detlefs, and Scott Wadsworth. 2013. Will you still compile me tomorrow? static cross-version compiler validation. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 191–201

    2013

  7. [7]

    Matthias Hirzel and Herbert Klaeren. 2016. Code coverage for any kind of test in any kind of transcompiled cross-platform applications. InProceedings of the 2nd International Workshop on User Interface Test Automation. 1–10

    2016

  8. [8]

    Paul Hudak. 1997. Domain-specific languages.Handbook of programming languages3, 39-60 (1997), 21

    1997

  9. [9]

    René Just. 2014. The Major mutation framework: Efficient and scalable mutation analysis for Java. InProceedings of the 2014 international symposium on software testing and analysis. 433–436

    2014

  10. [10]

    Tomaž Kosar, Sudev Bohra, and Marjan Mernik. 2016. Domain-specific languages: A systematic mapping study.Information and Software Technology71 (2016), 77–91

    2016

  11. [11]

    Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equiv- alence modulo inputs. InProceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation(Edinburgh, United Kingdom) (PLDI ’14). Association for Computing Machinery, New York, NY, USA, 216–226. doi:10.1145/2594291.2594334

    arXiv 2014

  12. [12]

    Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. InProceedings of the 2015 ACM SIGPLAN In- ternational Conference on Object-Oriented Programming, Systems, Languages, and Applications(Pittsburgh, PA, USA)(OOPSLA 2015). Association for Computing Machinery, New York, NY, USA, 386–399. doi:10.1145...

    arXiv 2015

  13. [13]

    Shaohua Li and Zhendong Su. 2023. Finding unstable code via compiler-driven differential testing. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume

    2023

  14. [14]

    Huai Liu, Xuan Liu, and Tsong Yueh Chen. 2012. A new method for constructing metamorphic relations. In2012 12th international conference on quality software. IEEE, 59–68

    2012

  15. [15]

    Yu-Seung Ma, Jeff Offutt, and Yong-Rae Kwon. 2006. MuJava: a mutation system for Java. InProceedings of the 28th international conference on Software engineering. 827–830

    2006

  16. [16]

    Michaël Marcozzi, Qiyi Tang, Alastair F Donaldson, and Cristian Cadar. 2019. Compiler fuzzing: How much does it matter?Proceedings of the ACM on Pro- gramming Languages3, OOPSLA (2019), 1–29

    2019

  17. [17]

    Johannes Mayer and Ralph Guderlei. 2006. An empirical study on the selection of good metamorphic relations. In30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 1. IEEE, 475–484

    2006

  18. [18]

    Aravind Nair, Karl Meinke, and Sigrid Eldh. 2019. Leveraging mutants for auto- matic prediction of metamorphic relations using machine learning. InProceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation(Tallinn, Estonia)(MaLTeSQuE 2019). Association for Computing Machinery, New York, NY, USA...

    arXiv 2019

  19. [19]

    Kazuhiro Nakamura and Nagisa Ishiura. 2016. Random testing of C compilers based on test program generation by equivalence transformation. In2016 IEEE Asia pacific conference on circuits and systems (APCCAS). IEEE, 676–679

    2016

  20. [20]

    Georg Ofenbeck, Tiark Rompf, and Markus Püschel. 2016. RandIR: differential testing for embedded compilers. InProceedings of the 2016 7th ACM SIGPLAN Symposium on Scala. 21–30

    2016

  21. [21]

    Md Shaik Sadi, Fei-Ching Kuo, Joshua WK Ho, Michael A Charleston, and Tsong Yueh Chen. 2011. Verification of phylogenetic inference programs using metamorphic testing.Journal of Bioinformatics and Computational Biology9, 06 (2011), 729–747

    2011

  22. [22]

    Prashanta Saha and Upulee Kanewala. 2019. Fault detection effectiveness of metamorphic relations developed for testing supervised classifiers. In2019 IEEE International conference on artificial intelligence testing (AITest). IEEE, 157–164

    2019

  23. [23]

    Emin Gün Sirer and Brian N Bershad. 1999. Using production grammars in software testing.ACM SIGPLAN Notices35, 1 (1999), 1–13

    1999

  24. [24]

    Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and analyzing compiler warning defects. InProceedings of the 38th International Conference on Software Engineering. 203–213

    2016

  25. [25]

    Chang-ai Sun, Yiqiang Liu, Zuoyi Wang, and Wing Kwong Chan. 2016. 𝜇MT: a data mutation directed metamorphic relation acquisition methodology. In Proceedings of the 1st International Workshop on Metamorphic Testing. 12–18

    2016

  26. [26]

    tachuris. 2024. tachuris/rmutt.js. https://github.com/tachuris/rmutt.js original- date: 2015-05-14T06:54:15Z

    2024

  27. [27]

    Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In2010 Asia Pacific Software Engineering Conference. 270–279. doi:10.1109/APSEC.2010.39

    work page doi:10.1109/apsec.2010.39 2010

  28. [28]

    2023.Domain-Specific Languages

    Andrzej Wąsowski and Thorsten Berger. 2023.Domain-Specific Languages. Springer

    2023

  29. [29]

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and under- standing bugs in C compilers. InProceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation. 283–294

    2011

  30. [30]

    Takahide Yoshikawa, Kouya Shimura, and Toshihiro Ozawa. 2003. Random program generator for Java JIT compiler test system. InThird International Conference on Quality Software, 2003. Proceedings.IEEE, 20–23

    2003

  31. [31]

    Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal program enu- meration for rigorous compiler testing. InProceedings of the 38th ACM SIGPLAN conference on programming language design and implementation. 347–361

    2017

  32. [2024]

    Genmorph: Automatically generating metamorphic relations via genetic programming.IEEE Transactions on Software Engineering50, 7 (2024), 1888–1900

    2024