{"id":"393d721b-a57a-4b85-96f7-ecb19e6e5944","arxiv_id":"2608.11345","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MergirafSemi, a language-agnostic semistructured merge tool built on truncated syntax trees, reduces spurious conflicts relative to diff3 and matches a Java-specific semistructured tool's accuracy at far lower runtime, at the cost of more missed conflicts.","lead":"MergirafSemi is a new merge tool that combines structural and line-based merging without needing language-specific code. It reduces the number of false conflicts developers see compared with plain line merging, runs much faster than language-specific semistructured tools, and accepts a small number of missed real conflicts in exchange.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation oracle in §4.2.2 equates 'compiles and passes tests' with correctness; weak test suites could undercount false negatives and reverse the claimed trade-off.","rationale":"The reader's weakest_assumption identified the same load-bearing concern: the accuracy metrics in Section 4.2.2 rely on 'compiles and passes tests' as sufficient evidence of correctness. I agree that this is the weak point of the evaluation. The central claim depends on aFPs and aFNs being reliable, and the oracle is most likely to fail exactly in the scenarios that distinguish MergirafSemi from conservative tools—cases where it automatically resolves a conflict that diff3 or S3M would flag. A semantic conflict that does not break the build or test suite is invisible to the metric. The paper's own threat-to-validity discussion acknowledges this limitation, but the effect on the headline numbers is unquantified. I do not see a stronger internal inconsistency that would overturn the conditional verdict. The Python aFP discrepancy (244 in Table 1 vs '224' in the text) and the lack of statistical tests are real issues, but they are less decisive than the classification oracle. Because the reader already reached CONDITIONAL and my concern points to the same condition, the appropriate judgment is to leave the verdict unchanged.","tokens_in":15229,"tokens_out":7508,"duration_ms":69036,"concrete_test":"On the 852-scenario Java subset used for the S3M comparison, identify all outputs where MergirafSemi or S3M produced a conflict-free merge that differs from the repository merge and passed the automated build/tests. Randomly sample at least 100 such outputs and have two independent human evaluators, blinded to tool identity and to the automated classification, judge whether each merged output is semantically correct, incorrect-but-compiles, or unclear. Adjudicate disagreements with a third evaluator and compute inter-rater agreement. Then recompute the aFP/aFN counts using the human labels. If the reclassified aFN count for MergirafSemi increases by more than the current 27-vs-13 gap to S3M, or the gap to diff3 narrows materially, the reported trade-off is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the classification oracle used for every aFP and aFN count in Tables 1–4. In §4.2.2, a tool output is treated as correct if it is syntactically identical to the repository merge, or, when different, if it compiles and passes the project's tests. An output that differs from the repository merge is counted as a false negative only if it fails to compile or causes test failures. This has a direct asymmetric effect on the paper's central claim. MergirafSemi's contribution is precisely that it auto-resolves conflicts that diff3 or S3M would flag; when those resolutions compile and pass tests but are semantically wrong, they are silently counted as correct. The reported aFN counts that support the 'comparable effectiveness' conclusion—27 vs S3M's 13 on the 852-scenario Java subset, and 172–292 vs diff3 across languages—could therefore be substantially underestimated. The paper itself concedes in §6 that 'tests are only as strong as the project's test suite,' but it never reports how many scenarios actually executed non-trivial tests. In a dataset spanning 513 repositories with generic CI workflows, some projects may have few or no tests, making 'passes tests' vacuous. If a meaningful fraction of test-passing merges are actually incorrect, the central trade-off claim—that semistructured merging is safer than fully structured merging while remaining competitive with S3M—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MergirafSemi is a language-agnostic semistructured merge tool built on Tree-sitter Concrete Syntax Trees and the existing Mergiraf infrastructure. Rather than merging fully structured trees, it truncates configured node types (such as method bodies) into text leaves that are merged with a diff3-style algorithm, while structural matching and signature-based unification handle higher-level declarations. The tool operates in three phases: an autotuned unstructured fast path, a local semistructured merge of conflicting regions, and a global semistructured merge of the entire file. The empirical evaluation uses 21,615 real-world merge scenarios from 513 repositories across Java, JavaScript, Python, Go, and Rust, comparing MergirafSemi with diff3, fully structured Mergiraf, the Java-specific semistructured tool S3M, and a commutative-context variant MergirafSemi+. Accuracy is measured through resolution rate and through added false positives (aFPs) and added false negatives (aFNs) derived from pairwise tool comparisons, validated against the developer-accepted repository merge and by whether outputs compile and pass the project's tests. The paper's central claims are that semistructured merging reduces spurious conflicts relative to diff3 with many fewer false negatives than fully structured Mergiraf, and that MergirafSemi achieves effectiveness comparable to S3M on the 852 Java scenarios where S3M completes, with more robust execution and lower runtime.","tokens_in":15442,"tokens_out":15954,"duration_ms":116707,"significance":"If the empirical claims are supported, the paper makes a useful practical case that semistructured merging can be implemented language-agnostically and cheaply: the tool could serve as a drop-in replacement for diff3 that eliminates a substantial share of spurious conflicts, and as a far cheaper alternative to a language-specific tool such as S3M. The study has genuine strengths. The dataset is large and multilingual, and the methodology follows the established evaluation pipeline of prior work, including compile-and-test validation. The RQ1 comparison with Mergiraf is well designed, since both tools share the same algorithmic foundation and therefore isolate the effect of structural granularity. RQ3 is a clean ablation of one configuration choice, and its finding that relaxing commutativity has a marginal effect is informative. The artifact package ships the exact tool binaries, dataset, and scripts, making the results reproducible. The trade-off pattern observed, that more structural granularity increases resolution and reduces aFPs but raises aFNs, is consistent with prior semistructured-versus-structured studies, which lends credibility to the qualitative conclusions.","major_comments":[{"comment":"The validation oracle that classifies aFP and aFN outcomes is load-bearing for every accuracy number in Tables 1–4, and its strength is never quantified. Under the rules in §4.2.2, an output that differs from the repository merge is judged correct if it compiles and passes the project's tests, and an aFN is counted only when the non-identical output fails to compile or breaks a test. Because MergirafSemi's distinctive behavior is to auto-resolve conflicts that diff3 or S3M flag, its non-identical outputs are exactly the ones judged by this oracle; if the test suites are weak, semantically wrong resolutions that happen to compile and pass tests are silently counted as correct. This would understate the aFN counts that support the central trade-off claim (e.g., 27 vs S3M's 13 in Table 2, and 55 vs diff3's 1 on the 852-scenario subset in §5.2). Section 6 concedes that 'tests are only as strong as the project's test suite' and that build/test validation runs only on disagreement scenarios, but the manuscript never reports the oracle's coverage. Please add: (a) a breakdown of how many scenarios were validated by syntactic identity with the repository merge, by compile-only success, and by compile-and-test success; (b) the number of tests actually executed per scenario or per language; and (c) a manual inspection of a random sample of tool-resolved outputs that differ from the repository merge, with the agreement rate. Without this evidence, the reported accuracy trade-off is not firmly established.","section":"§4.2.2 and §6 (Threats to Validity)"},{"comment":"The MergirafSemi-versus-S3M comparison is restricted to the 852 of 1,479 Java scenarios in which S3M completes; S3M timed out or crashed on the other 627 (approximately 42%). The paper is transparent about this restriction, but it never examines selection bias. If S3M's failures correlate with scenario difficulty (file size, number of textual conflicts, structural complexity, build configuration), then the conclusion that the language-agnostic tool achieves 'comparable effectiveness' on the subset need not generalize to the full Java scenario set. Please characterize the 852-scenario subset against the excluded 627 (for example, distributions of file size, diff3 conflict counts, and resolution rates of the other tools on each subset), and report MergirafSemi's aFP/aFN counts on the excluded scenarios. If the excluded scenarios differ systematically, the corresponding conclusion and the abstract's wording should be qualified.","section":"§5.2, Table 2"},{"comment":"No significance testing is reported anywhere in Section 5, although the discussion of Table 2 states that 'differences are significant.' The abstract's claim of 'comparable effectiveness' rests on Table 2, where MergirafSemi has 27 aFNs against S3M's 13 (more than double) while having 6 aFPs against 11; this is plausibly a trade-off, but 'comparable' needs quantitative support or more cautious wording. A paired test such as McNemar's over the 852 scenarios would clarify whether the aFP and aFN differences in Table 2 are distinguishable from chance, and the same test would strengthen the RQ1 claims in Table 1. If the counts are too small for meaningful tests, the qualitative claims should be downgraded accordingly.","section":"§5.1–§5.3, abstract"}],"minor_comments":[{"comment":"The sentence 'in Python, aFPs drop from 224 in MergirafSemi to 10 in Mergiraf' contradicts the Python row of Table 1, which reports 244 aFPs for MergirafSemi; one of the two is wrong and should be corrected.","section":"§5.1 vs Table 1"},{"comment":"The caption 'Merge performance' for Table 2 is misleading because the table reports accuracy metrics (aFPs and aFNs), not runtime; rename the caption and state explicitly that the counts are computed on the 852-scenario subset in which S3M completes.","section":"Table 2"},{"comment":"References [5] and [6] are the same paper (identical title, authors, and DOI) and should be merged into a single entry.","section":"References [5] and [6]"},{"comment":"The acknowledgements thank 'the participants of our survey and interviews,' but no survey or interview is described anywhere in the manuscript; the sentence should be removed or supported.","section":"Acknowledgements"},{"comment":"The claim that supporting a new language 'typically requires only a few hours' is presented without evidence; it should be labeled an informal estimate or substantiated with examples from the five evaluated languages.","section":"§3.3"},{"comment":"The manuscript should state explicitly that the language profiles (truncation rules, signatures, and commutative contexts) were fixed before the evaluation was run and were not tuned on the reported scenarios; otherwise the hand-set configuration leaves the 'comparable effectiveness' conclusion exposed to selection bias.","section":"§3.3, §4.3.1"},{"comment":"The restricted-subset figures reported in prose (MergirafSemi 1 aFP and 55 aFNs versus diff3's 13 aFPs and 1 aFN on the 852 Java scenarios) should also appear as a row in Table 3 or as a small separate table, so readers can verify the comparison without recomputing from the text.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The work comes from a group that has published extensively on semistructured and structured merge, and the position of MergirafSemi relative to the group's own LastMerge and Mergiraf is handled transparently. The novelty is incremental (a configured semistructured mode on top of an existing tool), so the paper's value rests almost entirely on the empirical study; the reported accuracy figures are currently the weakest point because the validation oracle is unquantified and the S3M comparison is restricted to a subset whose representativeness is unexamined. The abstract overstates 'comparable effectiveness' relative to the evidence in Table 2. The manuscript also needs a polish pass (duplicate reference, leftover survey/interviews acknowledgement, and the 224/244 inconsistency). These issues are fixable within the paper's scope, so I am not recommending rejection, but the revision needs real additional analysis, not just copyediting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a credible engineering paper that delivers what it promises—a language-agnostic semistructured merge that actually runs on five languages, with a large dataset and a replication package. The central trade-off finding, that more structure buys higher resolution but more false negatives, is consistent with prior work and is supported by the broad pattern of numbers. The paper deserves a serious referee.\n\nWhat is actually new: prior work had semistructured merging in language-specific tools (FSTMerge, S3M, SESAME) and language-agnostic structured merging (Mergiraf, LastMerge). MergirafSemi combines the two by truncating CSTs at configurable node types and falling back to line-based merging inside those regions. That combination, plus the five-language empirical study, is the contribution. The design is sensible: a three-phase pipeline with a diff3 fast path, then local semistructured merging, then global fallback. The authors use Tree-sitter and GumTree, and they ship exact tool versions and the dataset. That is real evidence and worth crediting.\n\nThe soft spots are real but not fatal. First, the S3M comparison runs on only 852 of 1,479 Java scenarios because S3M times out or crashes on the rest. The paper is transparent about this and recomputes the diff3 comparison on the same subset, but it means the \"comparable effectiveness to S3M\" claim rests on a biased subset. That is a moderate weakness, not a dealbreaker.\n\nSecond, there is an internal inconsistency in the Python aFP count: Table 1 shows 244 for MergirafSemi, the text says 224. One of these is a typo, but the discrepancy needs to be fixed before publication.\n\nThird, the validation oracle equates \"compiles and passes tests\" with correctness. The stress-test note worries this could undercount false negatives, and that worry is legitimate. The paper concedes tests are only as strong as the suite, but it never reports how many scenarios actually ran non-trivial tests. This is the load-bearing assumption behind every aFN count. It does not reverse the main trade-off—the pattern is consistent across languages—but it does mean the absolute numbers should be read with caution.\n\nFourth, the paper claims differences are \"significant\" without any statistical test. The effect sizes are large in many comparisons, so the conclusion probably holds, but \"significant\" needs a formal basis or the word should be dropped.\n\nOn balance: the central claim, that a language-agnostic semistructured merge can reduce spurious conflicts while staying competitive with a language-specific tool, holds up. The paper is honest about its own limitations. The right outcome is major revision, not rejection. The authors should resolve the aFP inconsistency, report test coverage or at least the number of scenarios with executed tests, and either add significance testing or soften the language.\n\nWho is this for? People working on merge tools and collaborative software engineering. I'd bring it to a reading group and cite it. I'd send it to review.","headline":"Solid, honest engineering paper on language-agnostic semistructured merge, with real artifacts and a believable trade-off result; needs revision on S3M subset analysis, test-oracle assumptions, and one data inconsistency, but deserves peer review.","tokens_in":16065,"tokens_out":1886,"would_cite":true,"duration_ms":14645,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a language-agnostic semistructured merge tool, MergirafSemi, can reduce spurious conflicts compared with line-based merging while matching the accuracy of a language-specific semistructured tool at far lower runtime.","keywords":["Merge tools","Semistructured merge","Concrete Syntax Trees","Tree-sitter","Conflict resolution","False positives","False negatives","Empirical software engineering"],"falsifier":"Re-run the evaluation on the same 21,615 scenarios but with an independent correctness oracle: after each tool's merge, run both the project's own test suite and a strict type checker or linter, and inspect every conflict-free output that newly fails the independent check. If the number of MergirafSemi outputs that compile and pass project tests but fail the independent checks is comparable in magnitude to its added false-negative count (27 in Java, 65 in Go, 172 in Python), the paper's central trade-off claim would be called into question.","tokens_in":14954,"feed_emoji":"🔀","tokens_out":8737,"duration_ms":78371,"temperature":0.7,"pith_summary":"MergirafSemi is a new merge tool that treats source code as a lightweight tree: high-level elements like classes and method signatures are structured, while regions such as method bodies are kept as raw text and merged line by line. The paper claims this language-agnostic semistructured design gives a better balance than the two extremes: it produces substantially fewer spurious conflicts than the line-based diff3 tool, and far fewer missed conflicts than the fully structured Mergiraf tool, while roughly matching the accuracy of S3M, a Java-specific semistructured tool, at a tiny fraction of S3M's runtime. On a dataset of 21,615 real-world merge scenarios in Go, Java, JavaScript, Python, and Rust, MergirafSemi resolved 80.0% of scenarios conflict-free, never timed out or crashed, and ran with a median of 44.7 ms per merge. If this holds up, a single configurable tool could replace both line-based and language-specific semistructured merging in multilingual projects, with new languages added in hours rather than months.","feed_headline":"Semistructured merge tool matches language-specific rivals","feed_subtitle":"MergirafSemi cuts spurious conflicts vs diff3 with accuracy close to S3M at a fraction of the runtime.","key_machinery":"The carrying mechanism is the truncated Concrete Syntax Tree (CST) controlled by a language profile. Tree-sitter parses the source into a CST; the profile marks certain node types (e.g., method bodies) as unstructured, so those subtrees are collapsed into leaf nodes containing raw text. The merge pipeline runs in three phases: an autotuned diff3-style pass on the full file as a fast path; if that yields conflicts or duplicate signatures, local semistructured merging that parses only the conflicting regions into CSTs, aligns nodes with GumTree, and unifies unmatched nodes that share the same signature; and, if problems persist, the same semistructured process applied to the whole file. This design does the structural work only where it is needed, which is why the median runtime stays low, and it is what lets the tool be configured for a new language in hours once a Tree-sitter parser exists.","core_discovery":"On the paper's own terms, the central discovery is that a merge tool does not need a full parse of the program to get most of the benefit of structure-aware merging. MergirafSemi uses Concrete Syntax Trees in which chosen node types, such as Java method bodies, are truncated into text leaves; those leaves are merged with diff3, while the surrounding structure resolves what diff3 would treat as overlapping edits. Across the evaluation, this configuration yields an automatic resolution rate of 80.0% against 84.1% for the fully structured Mergiraf, but with far fewer added false negatives (for example, 65 versus 143 in Go and 19 versus 90 in Python), and it cuts added false positives relative to diff3 (for example, from 33 to 3 in Java on the full dataset). Against the Java-specific semistructured tool S3M, restricted to the 852 scenarios S3M completes, MergirafSemi has fewer added false positives (6 vs 11) and more added false negatives (27 vs 13), with a median runtime of 159.4 ms versus S3M's roughly 260,180 ms. The authors read this as evidence that language-agnostic semistructured merging can be a viable drop-in alternative to both unstructured and language-specific semistructured tools.","pith_inferences":["We infer that the same truncation idea could be specialized further per language, e.g., keeping Python function bodies structured and Java method bodies as text, to shift the trade-off toward fewer false positives or fewer false negatives as a project prefers.","The paper's correctness oracle (repository merge plus compile-and-test) is only as strong as the project's test suite; in repositories with sparse tests, the reported false-negative counts are almost certainly understated, so the true trade-off between MergirafSemi and diff3 could be less favorable than it appears.","The unification step's signature-based matching could be exported as a standalone duplicate-detection service, useful for detecting redefinitions in dynamically typed languages where declaration order matters.","If the performance pattern generalizes beyond the five languages studied, a production version could decide at runtime, based on file size or conflict density, whether to use the fast path or the full semistructured pipeline, further improving the observed median speedup over Mergiraf."],"forward_implications":["A team adopting MergirafSemi in a multilingual repository could expect fewer hand-resolved false conflicts than with plain git merge, at the cost of occasionally missing a conflict that diff3 would have flagged.","Because the fast path is a plain diff3 merge, the common case of non-overlapping edits pays almost no structural overhead; the tree machinery engages only when conflicts or duplicate signatures actually appear.","The configurable truncation rules mean the false-positive/false-negative trade-off can be tuned per language by choosing which node types stay structured, without reimplementing the tool.","The unification step that merges nodes with identical signatures gives the tool a built-in mechanism to detect duplicate declarations that line-based tools silently combine.","The Java comparison suggests that replacing a language-specific semistructured tool with MergirafSemi would cut merge latency from minutes to milliseconds on typical scenarios while keeping accuracy in the same range."],"supporting_citations":[{"why":"Supplies the Tree-sitter parser framework that produces the Concrete Syntax Trees at the core of the tool.","marker":"[4]"},{"why":"LastMerge, the prior language-agnostic structured merge tool whose base implementation and evaluation approach MergirafSemi builds on.","marker":"[11]"},{"why":"S3M, the Java-specific semistructured tool used as the accuracy baseline in RQ2.","marker":"[5]"},{"why":"GumTree, the tree-matching algorithm used to align nodes across the base, left, and right revisions.","marker":"[13]"},{"why":"The dataset-construction and build-validation methodology that the evaluation pipeline adapts to five languages.","marker":"[22]"},{"why":"jDime, the structured merge tool whose auto-tuning idea motivates the fast-path diff3-first phase.","marker":"[1]"},{"why":"FSTMerge, the original semistructured merge approach whose structural-plus-textual idea MergirafSemi extends.","marker":"[2]"},{"why":"The empirical comparison of semistructured versus structured merge whose observed trade-off the paper's RQ1 results align with.","marker":"[8]"}],"fun_headline_variants":["MergirafSemi: fewer false merge conflicts, minimal structural parsing","Language-agnostic merge tool cuts spurious conflicts vs diff3","Structure-aware merge with less parse, faster than language-specific tools","Partial syntax merge: near-structured accuracy, low runtime"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The classification of every merge as correct or incorrect rests on the assumption that a tool output that compiles and passes the project's test suite is a correct integration; if a project's tests are weak, a genuinely broken merge that happens to compile and pass will be counted as a correct resolution, understating false negatives and potentially reversing the claimed trade-off.","fun_headline_variants_meta":{"raw":{"variants":["MergirafSemi: fewer false merge conflicts, minimal structural parsing","Language-agnostic merge tool cuts spurious conflicts vs diff3","Structure-aware merge with less parse, faster than language-specific tools","Partial syntax merge: near-structured accuracy, low runtime"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001267,"raw_usage":{"total_tokens":5249,"prompt_tokens":1075,"completion_tokens":4174,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":4101}},"tokens_in":691,"tokens_out":4174,"duration_ms":28780,"temperature":1.0,"reasoning_tokens":4101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:29.562893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on the same 21,615 scenarios but with an independent correctness oracle: after each tool's merge, run both the project's own test suite and a strict type checker or linter, and inspect every conflict-free output that newly fails the independent check. If the number of MergirafSemi outputs that compile and pass project tests but fail the independent checks is comparable in magnitude to its added false-negative count (27 in Java, 65 in Go, 172 in Python), the paper's central trade-off claim would be called into question.","supporting_citations":[{"cited_title":"2025.tree-sitter/tree-sitter: v0.25.3","cited_arxiv_id":null,"evidence_quote":"Supplies the Tree-sitter parser framework that produces the Concrete Syntax Trees at the core of the tool."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"jDime, the structured merge tool whose auto-tuning idea motivates the fast-path diff3-first phase."}],"review_version":1}