{"id":"237e533c-c444-4375-8708-9c9a4eff2f7e","arxiv_id":"2507.19687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LastMerge, a Tree-sitter-based generic structured merge tool, achieves accuracy and runtime performance comparable to Java-specific tools jDime and Spork on a dataset of 5,229 Java merge scenarios.","lead":"This paper introduces LastMerge, a generic structured merge tool that combines code changes using language-independent syntax trees configured through a thin interface. It presents evidence that such generic tools can match language-specific merge tools in accuracy and speed, potentially lowering the barrier to adopting structured merging across many languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy comparison is circular: LastMerge is both the tool under test and the oracle for syntactic equivalence, so its reported advantage over jDime may be an artifact of matcher bias.","rationale":"I focused on the circular oracle because it directly threatens the quantitative evidence for the central claim. The reader's weakest assumption identified the same mechanism; I agree that it is the most load-bearing. The alternative concern about evaluating only Java is real and is acknowledged in Section V-C as a threat to validity, but it is a limitation on external validity rather than a flaw in the internal comparison; even if the Java results are perfect, generalization to other languages would require additional experiments. The oracle bias, in contrast, could make the Java comparison itself unsound. The small manual sample (40 scenarios) is a secondary issue: it affects the causal attribution of differences to implementation details, but the overall accuracy comparison would still be interpretable if the oracle were unbiased. Thus I would keep the reader's conditional verdict, pending the oracle validation.","tokens_in":14232,"tokens_out":8071,"duration_ms":92521,"concrete_test":"Recompute the aFP/aFN classification for all 400 disagreement scenarios between jDime and LastMerge using an independent syntactic equivalence checker: parse the candidate output and the merge commit with a different Java parser (e.g., Eclipse JDT or JavaParser) and compare the resulting ASTs for exact structural isomorphism (same node types, child order, and token values, ignoring whitespace and comments). Keep the rest of the decision procedure (test execution) unchanged. If the resulting aFP and aFN counts differ from Table II by more than 20 scenarios (5% of 400), then LastMerge's matcher introduces a bias, and the reported accuracy advantage is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that generic structured merge tools achieve accuracy similar to language-specific ones rests on the aFP/aFN counts in Tables II and III. The classification procedure in Section IV-C decides each disagreement by checking whether the non-conflicting tool's output is 'syntactically equivalent' to the developer's merge commit. This check is performed by LastMerge itself: 'we use LAST MERGE to parse the files into trees and later rely on its matching algorithm to verify whether their program root nodes match entirely.' Thus, whenever LastMerge is one of the tools in the pair, its own output is validated with its own parser and matcher, which implement the same matching heuristics (unordered nodes, identifiers, Yang/GumTree-style matching) used in its merge engine. There is no independent evidence that this matcher is a sound and complete syntactic equivalence relation. If the matcher is systematically permissive in a way that aligns with LastMerge's merge decisions, then LastMerge's outputs will be judged equivalent to the merge commit more often than jDime's, deflating LastMerge's aFPs and inflating jDime's. The reported 15% difference in false positives is of the same order as the possible bias. The formatting normalization (running jDime on LastMerge outputs) does not remove the circularity, because the final equivalence is still computed in LastMerge's tree representation. The 40-scenario manual analysis only inspects conflict causes, not the equivalence check, so it cannot detect this bias. Without an independent oracle, the central accuracy comparison is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LastMerge, a language-agnostic structured merge tool built on Tree Sitter's concrete syntax trees and a thin language configuration interface. It reports an experiment comparing LastMerge and another generic tool, Mergiraf, against Java-specific tools jDime and Spork on 5,229 merge scenarios from an existing dataset. The evaluation uses relative added false positive (aFP) and added false negative (aFN) metrics, supplemented by test execution and a manual analysis of 40 scenarios. The authors report no evidence that generic structured merge significantly impacts accuracy, note that LastMerge has 15% fewer aFPs than jDime while Mergiraf has 42% fewer aFNs than Spork, and observe comparable or better runtime performance. They conclude that generic structured merge tools can effectively replace language-specific ones.","tokens_in":14482,"tokens_out":6121,"duration_ms":67869,"significance":"If the central claim holds, the paper would be an important contribution: it would show that the high cost of building language-specific structured merge tools can be avoided, enabling structured merge for a wide range of languages. The study has real strengths: it uses a large, publicly available dataset of merge scenarios; compares two independent generic tools against two language-specific baselines; measures runtime with repeated executions; and provides an online appendix with scripts and data. The paper is also honest about several threats to validity, including the Java-only evaluation and the reliance on test passing as an approximation of correctness. However, the accuracy comparison has a load-bearing circularity concern, the manual attribution relies on a small sample, and the broad conclusion overstates what the relative metrics and single-language evaluation can support.","major_comments":[{"comment":"The syntactic equivalence check that decides aFPs and aFNs uses LastMerge's own parser and matching algorithm: 'we use LAST MERGE to parse the files into trees and later rely on its matching algorithm to verify whether their program root nodes match entirely.' For the LastMerge vs. jDime comparison, LastMerge is therefore both the tool under test and the oracle. If this matcher is systematically permissive toward LastMerge's output representation, the aFP/aFN counts in Table II would be skewed in LastMerge's favor. The paper provides no independent validation of the matcher as a sound and complete equivalence relation. Please add an independent equivalence oracle (e.g., a separate parser or a normalized text diff) and report agreement on a sample, or manually evaluate a sample of equivalence decisions.","section":"Section IV-C and Table II"},{"comment":"The aFP and aFN metrics count only scenarios where the two tools disagree on conflict existence; scenarios in which both tools make the same mistake are invisible. This relative design is appropriate for comparing tools, but it cannot support the abstract's claim that generic tools 'achieve similar levels of accuracy' in an absolute sense. Please either add an absolute accuracy assessment on a sample with known conflict status, or explicitly restrict the accuracy claim to relative differences between tools.","section":"Section IV-A and Tables II-III"},{"comment":"The conclusion that most observed differences stem from 'implementation details and configuration choices' is based on manually analyzing only 5 aFPs and 5 aFNs per tool (40 scenarios total), drawn from 400 and 601 disagreement scenarios for the two tool pairs. This sample is too small to support the general claim, and no confidence intervals or inter-rater reliability are reported. Please enlarge the manual sample, report the uncertainty of the manual classification, or soften the generalization to the analyzed scenarios.","section":"Section IV-C and Section V-A"},{"comment":"The experiment instantiates both generic tools only for Java, as Section V-C acknowledges. The abstract and conclusion, however, claim that generic structured merge tools can effectively replace language-specific ones and 'pav[e] the way for broader adoption of structured merge in industry.' This is an unsupported generalization beyond the Java evidence. Please either scope the claims to Java or provide an evaluation on at least one additional language.","section":"Sections IV-B and V-C"},{"comment":"The statement 'our results show no evidence that generic structured merge significantly impacts merge accuracy' is not backed by any statistical test or confidence interval, and the observed relative differences are substantial: LastMerge has 85 aFNs versus jDime's 29, and Mergiraf has 290 aFPs versus Spork's 150. Please add significance tests or confidence intervals, or rephrase the conclusion to say that meaningful differences cannot be ruled out.","section":"Section V-A and Abstract"}],"minor_comments":[{"comment":"Please fix the typos: 'auto tunnig' should be 'auto-tuning' (Section V-A), 'wether' should be 'whether' (Sections V-A and V-C), and 'to to an intermediate representation' should be 'to an intermediate representation' (Section V-B).","section":"Sections V-A and V-B"},{"comment":"The tool name is written inconsistently as 'LAST MERGE', 'LASTMERGE', 'LastMerge', and 'LASTMerge' (e.g., Figure 8 caption). Please choose a single notation and apply it consistently.","section":"Throughout"},{"comment":"The footnote to Table I says 'The total sum in each column can vary because not all scenarios were successfully integrated by each tool.' This is unclear, since the sums exceed the 5,229-scenario sample size. Please clarify which scenarios are excluded from each column and report the effective number of scenarios.","section":"Table I"},{"comment":"The phrase 'explained in detail latter' should be 'explained in detail later'.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The circularity of using LastMerge as its own equivalence oracle is a serious concern, but it appears addressable by adding an independent oracle or a manual validation sample. The paper's contribution is valuable if the authors can also strengthen the statistical support and scope the claims more carefully. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LastMerge is a real tool and the paper is worth reading, but the headline comparison has a circularity problem that needs to be fixed before the accuracy numbers can be trusted. Section IV-C uses LastMerge itself to parse and match the outputs of all tools when deciding whether a merged file is syntactically equivalent to the developer's merge commit. When LastMerge is one of the tools in the pair, its own output is validated with its own parser and matcher—the same matching heuristics used in its merge engine. The reported 15% reduction in false positives over jDime is exactly the kind of difference that matcher bias could produce. The normalization step (running jDime on LastMerge's output) doesn't remove the circularity; the final equivalence is still decided in LastMerge's tree representation. This makes the central claim 'generic can replace language-specific' not established for the LastMerge/jDime comparison. The Mergiraf/Spork comparison is less affected because LastMerge is not one of the tools, but the oracle is still LastMerge's matcher, so its permissiveness or strictness could bias that comparison in either direction.\n\nWhat the paper does well: it ships a new generic structured merge tool with a thin configuration interface, built on Tree-sitter's 350+ language ecosystem. That is a genuine contribution. The empirical setup is honest and transparent—they report relative added false positives/false negatives, acknowledge the oracle heuristic in the threats to validity, and provide an online appendix. The paired design (each generic tool matched to its language-specific ancestor) is thoughtful. The manual analysis of 40 scenarios is small but useful: it shows that many differences trace to configuration choices rather than language independence, which is informative even if not conclusive.\n\nOther soft spots: there are no statistical significance tests or confidence intervals, so the statement 'no evidence that generic structured merge significantly impacts merge accuracy' is not backed by a formal test. The aFP/aFN metrics are relative, so we learn how the tools differ from each other, not whether either of them is actually good at detecting real conflicts. Cases where both tools are wrong are invisible. The paper acknowledges this, but it means the conclusion should be read as 'generic tools are in the same ballpark as language-specific ones' rather than 'they achieve similar levels of accuracy.'\n\nWho should read it: anyone working on merge tools, and researchers interested in evaluation methodology for refactoring/merging. The circular ground truth issue is a good discussion point.\n\nMy recommendation: send it to peer review. The tool is real, the dataset is large, and the circularity can be addressed—for example, by using an independent parser or hand-checking a sample of equivalence decisions. This is a solid paper that needs a revision, not a desk reject.","headline":"Solid tool paper whose central accuracy comparison is undermined by a circular ground-truth oracle: LastMerge judges its own outputs.","tokens_in":15026,"tokens_out":3088,"would_cite":true,"duration_ms":33711,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generic structured merge tools can achieve accuracy and speed comparable to language-specific tools, according to a replay of 5,229 real merge scenarios.","keywords":["software merge","structured merge","language-agnostic merge","concrete syntax tree","tree-sitter","merge accuracy","false positives","false negatives"],"falsifier":"Recompute the aFP/aFN counts with an independent ground-truth mechanism—for example, human-annotated conflict labels on a random sample of the 400 and 601 disagreement scenarios—and check whether LastMerge's 15% fewer false positives and Mergiraf's 42% fewer false negatives persist; expanding the manual analysis from 40 scenarios to the full disagreement set would also confirm or overturn the claim that most differences are configuration details.","tokens_in":14026,"feed_emoji":"🔀","tokens_out":6397,"duration_ms":66976,"temperature":0.7,"pith_summary":"This paper tries to establish that structured merge—resolving conflicts by comparing code as syntax trees rather than text lines—can go generic without losing accuracy or speed. It introduces LastMerge, a merge tool whose core engine works on concrete syntax trees produced by the Tree-sitter parser framework, configured for each language through a thin interface. In an experiment replaying 5,229 real-world Java merge scenarios, LastMerge and a second generic tool, Mergiraf, matched the accuracy and runtime of the language-specific tools jDime and Spork; the roughly 10% behavioral difference is traced mostly to configuration and implementation details, not to genericity. If this holds, structured merge could become available for hundreds of languages at a small fraction of the current per-language implementation cost. That matters because line-based merge tools are universal but frequently report spurious conflicts or miss real ones.","feed_headline":"Generic merge tools match language-specific accuracy","feed_subtitle":"A Tree-sitter-based generic tool and its counterpart matched jDime and Spork on 5,229 real merge scenarios.","key_machinery":"The load-bearing mechanism is LastMerge's core merge engine, which operates on concrete syntax trees (CSTs) produced by Tree-sitter, a parser framework with grammars for more than 350 languages. A thin configuration interface supplies language-specific aspects: which nonterminal nodes are unordered, how to extract node identifiers using Tree-sitter queries, and optional parsing handlers that restructure trees, such as grouping imports. Matching adapts jDime's algorithms to these generic trees, using Yang's algorithm for ordered children and a linear-programming maximum matching for unordered children; the merge phase reports conflicts only when differing changes affect corresponding tree nodes. Mergiraf, a second generic tool, adapts Spork's algorithm with GumTree matching and auto-tuning, providing evidence that the result generalizes beyond one particular design.","core_discovery":"The paper claims that generic structured merge tools can effectively replace language-specific ones, achieving similar levels of accuracy and efficiency. Concretely, it argues that a merge engine operating over language-independent concrete syntax trees, configured by a thin per-language description, reproduces the behavior of Java-specific tools: LastMerge reports 15% fewer added false positives than jDime, Mergiraf misses 42% fewer added false negatives than Spork, and both generic tools run at least as fast. The 7.53% and 12.22% disagreement rates with their specific counterparts stem mostly from configuration choices and implementation details that could be adjusted, not from the language-independent design.","pith_inferences":["If the finding generalizes beyond Java, the economics of structured merge change: the fixed cost per language is small, so covering many languages becomes a configuration effort rather than a research and engineering effort.","The accuracy verdict leans on LastMerge itself to judge whether another tool's output matches the merge commit; an independent ground-truth check would strengthen or revise the reported aFP/aFN ratios.","Auto-tuning, which Mergiraf already uses, could be added to LastMerge-style tools to get conflict-free speed when no conflict exists, a combination the paper notes is easy to implement.","Since jDime and Spork only support Java, the same paired comparison for other languages is the natural next test; until then, the Java-only evidence leaves open how much of the result is language-specific."],"forward_implications":["Structured merge support for a new language reduces to providing a Tree-sitter grammar plus configuration queries and handlers, instead of implementing a full language-specific merge engine.","Developers working in multi-language projects could get structured merge accuracy everywhere, without maintaining a separate tool per language.","The observed differences between generic and specific tools would shrink if configurations were aligned, since most disagreements are attributed to adjustable configuration details.","Generic structured merge does not impose a runtime penalty over the studied specific tools, so adopting it does not cost performance.","The same configuration interface can emulate semistructured merge behavior by stopping the tree at an intermediate level, a flexibility language-specific tools lack."],"supporting_citations":[{"why":"Supplies the 5,983-merge-scenario dataset, filtered to 5,229 scenarios, that the replay experiment runs on.","marker":"[12]"},{"why":"Defines jDime, the language-specific baseline paired with LastMerge, and the structured merge with auto-tuning approach LastMerge's algorithm adapts.","marker":"[10]"},{"why":"Defines Spork, the language-specific baseline paired with Mergiraf, whose merge algorithm Mergiraf adapts.","marker":"[9]"},{"why":"Provides the Tree-sitter parser framework and its grammar ecosystem, which gives the generic trees the claim of language coverage rests on.","marker":"[13]"},{"why":"Supplies Yang's algorithm used for maximum matching of ordered children in LastMerge's matching phase.","marker":"[14]"},{"why":"Supplies the GumTree differencing algorithm that Mergiraf uses for its tree matchings.","marker":"[15]"},{"why":"Introduces the added false positive and added false negative relative comparison metrics that structure the accuracy evaluation.","marker":"[16]"}],"fun_headline_variants":["Generic merge tools match language-specific accuracy","Language-agnostic merge rivals Java-specific tools","Structured merge without language lock-in","LastMerge: generic merge matches Java tools","No accuracy loss for generic merge tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy counts depend on using LastMerge itself to parse and tree-match tool outputs against the repository merge commit as ground truth; if LastMerge's parsing or matching is systematically biased, the relative false-positive and false-negative numbers for every tool are skewed.","fun_headline_variants_meta":{"raw":{"variants":["Generic merge tools match language-specific accuracy","Language-agnostic merge rivals Java-specific tools","Structured merge without language lock-in","LastMerge: generic merge matches Java tools","No accuracy loss for generic merge tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1258,"prompt_tokens":932,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":548,"tokens_out":326,"duration_ms":3885,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:07:50.833521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the aFP/aFN counts with an independent ground-truth mechanism—for example, human-annotated conflict labels on a random sample of the 400 and 601 disagreement scenarios—and check whether LastMerge's 15% fewer false positives and Mergiraf's 42% fewer false negatives persist; expanding the manual analysis from 40 scenarios to the full disagreement set would also confirm or overturn the claim that most differences are configuration details.","supporting_citations":[{"cited_title":"Evaluation of version control merge tools,","cited_arxiv_id":null,"evidence_quote":"Supplies the 5,983-merge-scenario dataset, filtered to 5,229 scenarios, that the replay experiment runs on."},{"cited_title":"Structured merge with auto- tuning: balancing precision and performance,","cited_arxiv_id":null,"evidence_quote":"Defines jDime, the language-specific baseline paired with LastMerge, and the structured merge with auto-tuning approach LastMerge's algorithm adapts."},{"cited_title":"Spork: Struc- tured Merge for Java With Formatting Preservation ,","cited_arxiv_id":null,"evidence_quote":"Defines Spork, the language-specific baseline paired with Mergiraf, whose merge algorithm Mergiraf adapts."},{"cited_title":"tree-sitter/tree-sitter: v0.25.3,","cited_arxiv_id":null,"evidence_quote":"Provides the Tree-sitter parser framework and its grammar ecosystem, which gives the generic trees the claim of language coverage rests on."},{"cited_title":"Identifying syntactic differences between two programs,","cited_arxiv_id":null,"evidence_quote":"Supplies Yang's algorithm used for maximum matching of ordered children in LastMerge's matching phase."},{"cited_title":"Fine-grained and accurate source code differencing,","cited_arxiv_id":null,"evidence_quote":"Supplies the GumTree differencing algorithm that Mergiraf uses for its tree matchings."},{"cited_title":"Evaluating and improving semistructured merge,","cited_arxiv_id":null,"evidence_quote":"Introduces the added false positive and added false negative relative comparison metrics that structure the accuracy evaluation."}],"review_version":1}