REVIEW 3 major objections 7 minor 28 references
MergirafSemi: A Language-Agnostic Semistructured Merge Tool
T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that a language-agnostic semistructured merge tool, MergirafSemi, can reduce spurious conflicts compared with line-based merging while matching the accuracy of a language-specific semistructured tool at far lower runtime.
desk verdict Solid, honest engineering paper on language-agnostic semistructured merge, with real artifacts and a believable trade-off result; needs revision on S3M subset analysis, test-oracle assumptions, and one data inconsistency, but deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the truncated Concrete Syntax Tree (CST) controlled by a language profile. Tree-sitter parses the source into a CST; the profile marks certain node types (e.g., method bodies) as unstructured, so those subtrees are collapsed into leaf nodes containing raw text. The merge pipeline runs in three phases: an autotuned diff3-style pass on the full file as a fast path; if that yields conflicts or duplicate signatures, local semistructured merging that parses only the conflicting regions into CSTs, aligns nodes with GumTree, and unifies unmatched nodes that share the same signature; and, if problems persist, the same semistructured process applied to the whole file. This design does the structural work only where it is needed, which is why the median runtime stays low, and it is what lets the tool be configured for a new language in hours once a Tree-sitter parser exists.
What would settle it
Re-run the evaluation on the same 21,615 scenarios but with an independent correctness oracle: after each tool's merge, run both the project's own test suite and a strict type checker or linter, and inspect every conflict-free output that newly fails the independent check. If the number of MergirafSemi outputs that compile and pass project tests but fail the independent checks is comparable in magnitude to its added false-negative count (27 in Java, 65 in Go, 172 in Python), the paper's central trade-off claim would be called into question.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a merge tool does not need a full parse of the program to get most of the benefit of structure-aware merging. MergirafSemi uses Concrete Syntax Trees in which chosen node types, such as Java method bodies, are truncated into text leaves; those leaves are merged with diff3, while the surrounding structure resolves what diff3 would treat as overlapping edits. Across the evaluation, this configuration yields an automatic resolution rate of 80.0% against 84.1% for the fully structured Mergiraf, but with far fewer added false negatives (for example, 65 versus 143 in Go and 19 versus 90 in Python), and it cuts added false positives relative to diff3 (for example, from 33 to 3 in Java on the full dataset). Against the Java-specific semistructured tool S3M, restricted to the 852 scenarios S3M completes, MergirafSemi has fewer added false positives (6 vs 11) and more added false negatives (27 vs 13), with a median runtime of 159.4 ms versus S3M's roughly 260,180 ms. The authors read this as evidence that language-agnostic semistructured merging can be a viable drop-in alternative to both unstructured and language-specific semistructured tools.
Load-bearing premise
The classification of every merge as correct or incorrect rests on the assumption that a tool output that compiles and passes the project's test suite is a correct integration; if a project's tests are weak, a genuinely broken merge that happens to compile and pass will be counted as a correct resolution, understating false negatives and potentially reversing the claimed trade-off.
Editorial extensions
If this is right
- A team adopting MergirafSemi in a multilingual repository could expect fewer hand-resolved false conflicts than with plain git merge, at the cost of occasionally missing a conflict that diff3 would have flagged.
- Because the fast path is a plain diff3 merge, the common case of non-overlapping edits pays almost no structural overhead; the tree machinery engages only when conflicts or duplicate signatures actually appear.
- The configurable truncation rules mean the false-positive/false-negative trade-off can be tuned per language by choosing which node types stay structured, without reimplementing the tool.
- The unification step that merges nodes with identical signatures gives the tool a built-in mechanism to detect duplicate declarations that line-based tools silently combine.
- The Java comparison suggests that replacing a language-specific semistructured tool with MergirafSemi would cut merge latency from minutes to milliseconds on typical scenarios while keeping accuracy in the same range.
Reading between the lines
- We infer that the same truncation idea could be specialized further per language, e.g., keeping Python function bodies structured and Java method bodies as text, to shift the trade-off toward fewer false positives or fewer false negatives as a project prefers.
- The paper's correctness oracle (repository merge plus compile-and-test) is only as strong as the project's test suite; in repositories with sparse tests, the reported false-negative counts are almost certainly understated, so the true trade-off between MergirafSemi and diff3 could be less favorable than it appears.
- The unification step's signature-based matching could be exported as a standalone duplicate-detection service, useful for detecting redefinitions in dynamically typed languages where declaration order matters.
- If the performance pattern generalizes beyond the five languages studied, a production version could decide at runtime, based on file size or conflict density, whether to use the fast path or the full semistructured pipeline, further improving the observed median speedup over Mergiraf.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. MergirafSemi is a language-agnostic semistructured merge tool built on Tree-sitter Concrete Syntax Trees and the existing Mergiraf infrastructure. Rather than merging fully structured trees, it truncates configured node types (such as method bodies) into text leaves that are merged with a diff3-style algorithm, while structural matching and signature-based unification handle higher-level declarations. The tool operates in three phases: an autotuned unstructured fast path, a local semistructured merge of conflicting regions, and a global semistructured merge of the entire file. The empirical evaluation uses 21,615 real-world merge scenarios from 513 repositories across Java, JavaScript, Python, Go, and Rust, comparing MergirafSemi with diff3, fully structured Mergiraf, the Java-specific semistructured tool S3M, and a commutative-context variant MergirafSemi+. Accuracy is measured through resolution rate and through added false positives (aFPs) and added false negatives (aFNs) derived from pairwise tool comparisons, validated against the developer-accepted repository merge and by whether outputs compile and pass the project's tests. The paper's central claims are that semistructured merging reduces spurious conflicts relative to diff3 with many fewer false negatives than fully structured Mergiraf, and that MergirafSemi achieves effectiveness comparable to S3M on the 852 Java scenarios where S3M completes, with more robust execution and lower runtime.
Significance. If the empirical claims are supported, the paper makes a useful practical case that semistructured merging can be implemented language-agnostically and cheaply: the tool could serve as a drop-in replacement for diff3 that eliminates a substantial share of spurious conflicts, and as a far cheaper alternative to a language-specific tool such as S3M. The study has genuine strengths. The dataset is large and multilingual, and the methodology follows the established evaluation pipeline of prior work, including compile-and-test validation. The RQ1 comparison with Mergiraf is well designed, since both tools share the same algorithmic foundation and therefore isolate the effect of structural granularity. RQ3 is a clean ablation of one configuration choice, and its finding that relaxing commutativity has a marginal effect is informative. The artifact package ships the exact tool binaries, dataset, and scripts, making the results reproducible. The trade-off pattern observed, that more structural granularity increases resolution and reduces aFPs but raises aFNs, is consistent with prior semistructured-versus-structured studies, which lends credibility to the qualitative conclusions.
major comments (3)
- [§4.2.2 and §6 (Threats to Validity)] The validation oracle that classifies aFP and aFN outcomes is load-bearing for every accuracy number in Tables 1–4, and its strength is never quantified. Under the rules in §4.2.2, an output that differs from the repository merge is judged correct if it compiles and passes the project's tests, and an aFN is counted only when the non-identical output fails to compile or breaks a test. Because MergirafSemi's distinctive behavior is to auto-resolve conflicts that diff3 or S3M flag, its non-identical outputs are exactly the ones judged by this oracle; if the test suites are weak, semantically wrong resolutions that happen to compile and pass tests are silently counted as correct. This would understate the aFN counts that support the central trade-off claim (e.g., 27 vs S3M's 13 in Table 2, and 55 vs diff3's 1 on the 852-scenario subset in §5.2). Section 6 concedes that 'tests are only as strong as the project's test suite' and that build/test validation runs only on disagreement scenarios, but the manuscript never reports the oracle's coverage. Please add: (a) a breakdown of how many scenarios were validated by syntactic identity with the repository merge, by compile-only success, and by compile-and-test success; (b) the number of tests actually executed per scenario or per language; and (c) a manual inspection of a random sample of tool-resolved outputs that differ from the repository merge, with the agreement rate. Without this evidence, the reported accuracy trade-off is not firmly established.
- [§5.2, Table 2] The MergirafSemi-versus-S3M comparison is restricted to the 852 of 1,479 Java scenarios in which S3M completes; S3M timed out or crashed on the other 627 (approximately 42%). The paper is transparent about this restriction, but it never examines selection bias. If S3M's failures correlate with scenario difficulty (file size, number of textual conflicts, structural complexity, build configuration), then the conclusion that the language-agnostic tool achieves 'comparable effectiveness' on the subset need not generalize to the full Java scenario set. Please characterize the 852-scenario subset against the excluded 627 (for example, distributions of file size, diff3 conflict counts, and resolution rates of the other tools on each subset), and report MergirafSemi's aFP/aFN counts on the excluded scenarios. If the excluded scenarios differ systematically, the corresponding conclusion and the abstract's wording should be qualified.
- [§5.1–§5.3, abstract] No significance testing is reported anywhere in Section 5, although the discussion of Table 2 states that 'differences are significant.' The abstract's claim of 'comparable effectiveness' rests on Table 2, where MergirafSemi has 27 aFNs against S3M's 13 (more than double) while having 6 aFPs against 11; this is plausibly a trade-off, but 'comparable' needs quantitative support or more cautious wording. A paired test such as McNemar's over the 852 scenarios would clarify whether the aFP and aFN differences in Table 2 are distinguishable from chance, and the same test would strengthen the RQ1 claims in Table 1. If the counts are too small for meaningful tests, the qualitative claims should be downgraded accordingly.
minor comments (7)
- [§5.1 vs Table 1] The sentence 'in Python, aFPs drop from 224 in MergirafSemi to 10 in Mergiraf' contradicts the Python row of Table 1, which reports 244 aFPs for MergirafSemi; one of the two is wrong and should be corrected.
- [Table 2] The caption 'Merge performance' for Table 2 is misleading because the table reports accuracy metrics (aFPs and aFNs), not runtime; rename the caption and state explicitly that the counts are computed on the 852-scenario subset in which S3M completes.
- [References [5] and [6]] References [5] and [6] are the same paper (identical title, authors, and DOI) and should be merged into a single entry.
- [Acknowledgements] The acknowledgements thank 'the participants of our survey and interviews,' but no survey or interview is described anywhere in the manuscript; the sentence should be removed or supported.
- [§3.3] The claim that supporting a new language 'typically requires only a few hours' is presented without evidence; it should be labeled an informal estimate or substantiated with examples from the five evaluated languages.
- [§3.3, §4.3.1] The manuscript should state explicitly that the language profiles (truncation rules, signatures, and commutative contexts) were fixed before the evaluation was run and were not tuned on the reported scenarios; otherwise the hand-set configuration leaves the 'comparable effectiveness' conclusion exposed to selection bias.
- [§5.2] The restricted-subset figures reported in prose (MergirafSemi 1 aFP and 55 aFNs versus diff3's 13 aFPs and 1 aFN on the 852 Java scenarios) should also appear as a row in Table 3 or as a small separate table, so readers can verify the comparison without recomputing from the text.
Circularity Check
No significant circularity: the paper's empirical evaluation is anchored to external benchmarks, and its central trade-off claim is measured rather than derived from its own definitions or self-citations.
full rationale
The paper proposes MergirafSemi and evaluates it against diff3, Mergiraf, S3M, and a variant. The central claims (balanced trade-off, fewer false positives than diff3, comparable accuracy to S3M) are supported by measured aFP/aFN counts and runtimes on a large externally sourced dataset. The aFP/aFN definitions in Section 4.2.2 are standard pairwise-comparison metrics validated with repository merges, compilation, and tests; the paper explicitly acknowledges in Section 6 that tests are only as strong as the project's test suite, which is a validity limitation rather than a circular step. No fitted parameter is renamed as a prediction: the language profiles and truncation rules are design choices with a stated configuration, and the MergirafSemi+ comparison is a controlled configuration change. Self-citations (S3M, LastMerge, MiningFramework) provide tools and related work, but the load-bearing results are measured in this study, not imported from those citations. There is no uniqueness theorem, ansatz smuggled in via citation, or renaming of a known result as a new derivation. The derivation chain is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (3)
- Tree-truncation node types (language profile) =
hand-configured per language; e.g., method bodies in Java
- Signature definitions =
hand-configured per language
- Commutative contexts =
Java defaults non-commutative; MergirafSemi+ treats methods and variable declarations as commutative
assumptions (4)
- domain assumption Tree-sitter produces correct CSTs for Go, Java, JavaScript, Python, and Rust, and parsing failures are confined to fallback phases.
- domain assumption GumTree's node matching across base, left, and right revisions is accurate enough to support the merge and unification decisions.
- domain assumption A merged file that is not identical to the repository merge but compiles and passes the project's tests is a correct merge.
- domain assumption The developer-accepted repository merge is a valid reference outcome.
Cite this review
Pith. "Pith review of MergirafSemi: A Language-Agnostic Semistructured Merge Tool." pith.science (2026). https://pith.science/paper/UMZCJO3Q
@misc{pith2026260811345,
author = {Pith},
title = {Pith review of: MergirafSemi: A Language-Agnostic Semistructured Merge Tool},
year = {2026},
howpublished = {\url{https://pith.science/paper/UMZCJO3Q}},
note = {Machine review of arXiv:2608.11345}
}
read the original abstract
Developers frequently face merge conflicts when integrating concurrent changes. Most merge tools rely on unstructured, line-based comparisons, often producing spurious conflicts and missing actual conflicts. To address these limitations, structure-aware merge tools have been proposed, which leverage syntactic representations to improve merge accuracy. However, fully structured tools may incur higher computational cost, and language-specific tools require significant development and maintenance effort. To balance these trade-offs, we propose MergirafSemi, a language-agnostic semistructured merge tool that captures structural information without requiring full structural modeling or language-specific implementations, and applies line-based merging within specific program regions, such as method bodies in Java. Our tool leverages lightweight Concrete Syntax Trees to guide merging decisions while preserving flexibility and efficiency across languages. We evaluate our tool through an empirical study on real-world merge scenarios across multiple programming languages, comparing it with unstructured, semistructured, and structured tools. Our results show that increasing structural granularity improves automatic conflict resolution but can also lead to more aggressive merge decisions, increasing the number of missed actual conflicts. In contrast, MergirafSemi achieves a more balanced trade-off, reducing spurious conflicts while maintaining competitive accuracy and better runtime performance in most common scenarios. Compared to an unstructured tool, it substantially reduces spurious conflicts, and when compared to a semistructured language-specific tool, it achieves comparable effectiveness while exhibiting more robust execution behavior and significantly lower runtime overhead.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Sven Apel, Olaf Leßenich, and Christian Lengauer. 2012. Structured merge with auto-tuning: balancing precision and performance. InProceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering(Essen, Germany)(ASE ’12). Association for Computing Machinery, New York, NY, USA, 120–129. doi:10.1145/2351676.2351694
-
[2]
Sven Apel, Jörg Liebig, Benjamin Brandl, Christian Lengauer, and Christian Kästner. 2011. Semistructured Merge: Rethinking Merge in Revision Control Systems. InSIGSOFT/FSE 2011 - Proceedings of the 19th ACM SIGSOFT Symposium on Foundations of Software Engineering. doi:10.1145/2025113.2025141
arXiv 2011
-
[3]
Felipe Araujo, Paulo Borba, and Guilherme Cavalcanti. 2024. Refinando a Precisão da Detecção de Conflitos: Uma Análise do CSDiff com Abordagem Focalizada. In Anais do XXXVIII Simpósio Brasileiro de Engenharia de Software(Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 246–256. doi:10.5753/sbes.2024.3395
-
[4]
2025.tree-sitter/tree-sitter: v0.25.3
Max Brunsfeld, Amaan Qureshi, Andrew Hlynskyi, Patrick Thomson, Ob- serverOfTime, Will Lillis, Josh Vera, dundargoc, Phil Turnbull, Timothy Clem, Douglas Creager, Andrew Helwer, Rob Rix, Daumantas Kavolis, Hendrik van Antwerpen, Michael Davis, Christian Clason, Ika, Amin Ya, Riley Bruins, Tuan- Anh Nguyen, Stafford Brunk, Matt Massicotte, bfredl, Niranjan...
-
[6]
Guilherme Cavalcanti, Paulo Borba, and Paola Accioly. 2017. Evaluating and improving semistructured merge.Proc. ACM Program. Lang.1, OOPSLA, Article 59 (Oct. 2017), 27 pages. doi:10.1145/3133883
doi:10.1145/3133883 2017
-
[7]
Guilherme Cavalcanti, Paulo Borba, Leonardo dos Anjos, and Jonatas Clementino
-
[8]
Guilherme Cavalcanti, Paulo Borba, Georg Seibt, and Sven Apel. 2019. The Impact of Structure on Software Merging: Semistructured Versus Structured Merge. In2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1002–1013. doi:10.1109/ASE.2019.00097
arXiv 2019
-
[9]
Jônatas Clementino, Paulo Borba, and Guilherme Cavalcanti. 2021. Textual merge based on language-specific syntactic separators. InAnais do XXXV Simpósio Brasileiro de Engenharia de Software(Joinville). SBC, Porto Alegre, RS, Brasil. https://sol.sbc.org.br/index.php/sbes/article/view/18820
work page 2021
Show all 28 references
-
[10]
Jinhao Dong, Qihao Zhu, Zeyu Sun, Yiling Lou, and Dan Hao. 2023. Merge Conflict Resolution: Classification or Generation?. In2023 38th IEEE/ACM In- ternational Conference on Automated Software Engineering (ASE). 1652–1663. doi:10.1109/ASE56229.2023.00155
2023
-
[11]
Joao Pedro Duarte, Paulo Borba, and Guilherme Cavalcanti. 2025. LastMerge: A language-agnostic structured tool for code integration. arXiv:2507.19687 [cs.SE] https://arxiv.org/abs/2507.19687
2025 arXiv
-
[12]
Lopes et al. 2026. MergirafSemi: A Language-Agnostic Semistructured Merge Tool. Zenodo. doi:10.5281/zenodo.21422483
2026 doi
-
[13]
Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. InPro- ceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering(Vasteras, Sweden)(ASE ’14). Assoc...
2014
-
[14]
Gleiph Ghiotto, Leonardo Murta, Marcio Barros, and Andre van der Hoek. 2020. On the Nature of Merge Conflicts: A Study of 2,731 Open Source Java Projects Hosted by GitHub .IEEE Transactions on Software Engineering46, 08 (Aug. 2020), 892–915. doi:10.1109/TSE.2018.2871083
2020
-
[15]
GitHub, Inc. 2020. Greatest Hits — GitHub Archive Program. https:// archiveprogram.github.com/greatest-hits/. Accessed: 2026-05-05
2020
-
[16]
Software Productivity Group. 2026. MiningFramework. https://github.com/ predohenr/miningframework. Accessed in: 29/04/2026
2026
-
[17]
Hunt and W.F
J.J. Hunt and W.F. Tichy. 2002. Extensible language-aware merging. InIn- ternational Conference on Software Maintenance, 2002. Proceedings.511–520. doi:10.1109/ICSM.2002.1167812
2002 arXiv
-
[18]
Sanjeev Khanna, Keshav Kunal, and Benjamin C. Pierce. 2007. A Formal In- vestigation of Diff3. InFSTTCS 2007: Foundations of Software Technology and Theoretical Computer Science, V. Arvind and Sanjiva Prasad (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 485–496
2007
- [19]
-
[20]
T. Mens. 2002. A State-of-the-Art Survey on Software Merging.IEEE Trans. Softw. Eng.28, 5 (May 2002), 449–462. doi:10.1109/TSE.2002.1000449
2002 arXiv
-
[21]
Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating GitHub for engineered software projects.Empirical Softw. Engg.22, 6 (Dec. 2017), 3219–3253. doi:10.1007/s10664-017-9512-6
2017 doi
-
[22]
Benedikt Schesch, Ryan Featherman, Kenneth J Yang, Ben Roberts, and Michael D. Ernst. 2024. Evaluation of Version Control Merge Tools. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA)(ASE ’24). Association for ...
2024
-
[23]
Georg Seibt, Florian Heck, Guilherme Cavalcanti, Paulo Borba, and Sven Apel
-
[24]
Alberto Trindade Tavares, Paulo Borba, Guilherme Cavalcanti, and Sérgio Soares
-
[25]
Ohlsson, Björn Regnell, and Anders Wesslén
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012.Analysis and Interpretation. Springer Berlin Heidelberg, Berlin, Heidelberg, 123–151. doi:10.1007/978-3-642-29044-2_10
2012 doi
-
[26]
Fengmin Zhu, Fei He, and Qianshan Yu. 2019. Enhancing Precision of Structured Merge by Proper Tree Matching. In2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSECompanion). 286–287. doi:10.1109/ICSE-Companion.2019.00117
2019
-
[2020]
InProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering(San Diego, California)(ASE ’19)
Semistructured merge in JavaScript systems. InProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering(San Diego, California)(ASE ’19). IEEE Press, 1014–1025. doi:10.1109/ASE.2019.00098
2019
-
[2021]
doi:10.1109/TSE.2021.3123143
Leveraging Structure in Software Merge: An Empirical Study.IEEE Trans- actions on Software Engineering48 (01 2021), 1–1. doi:10.1109/TSE.2021.3123143
2021
-
[2024]
In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft- ware Engineering(Sacramento, CA, USA)(ASE ’24)
Semistructured Merge with Language-Specific Syntactic Separators. In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft- ware Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machinery, New York, NY, USA, 1032–1043. doi:10.1145/3691...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.