{"id":"0e889999-d758-4223-9da2-847a5df0f349","arxiv_id":"2411.15502","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors argue that cross-language maintainability comparison requires relative, language-agnostic indicators, consistent rule sets, and a simple explainable weighted aggregation.","lead":"This paper describes practical challenges in comparing software maintainability across different programming languages and proposes a lightweight aggregation framework based on relative indicators. It is a position paper grounded in 15 years of code audits in Wallonia, useful for teams deciding between applications written in different languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'successful' comparison claim rests on an unvalidated, hand-picked weighting and threshold scheme; without sensitivity analysis or a validation dataset, the reported cross-language comparison is currently an assertion.","rationale":"I agree with the reader that the hand-picked weights and thresholds are the weakest assumption underpinning the central claim. The paper is transparent about its practical context and explicitly lists validation as a next step, so there is no internal inconsistency; the gap is evidentiary. The proposed aggregation's output is sensitive to the chosen parameters, and without a sensitivity analysis or external validation the 'successful comparison and explainability' claim is not yet supported. The reader's CONDITIONAL verdict is appropriate because the approach may be reasonable but needs either scaled-back framing or demonstrated robustness before adoption. My read does not change that verdict.","tokens_in":3625,"tokens_out":2880,"duration_ms":26137,"concrete_test":"Apply the proposed aggregation to the authors' database (or a public multi-language benchmark) and run a full weight/threshold sensitivity analysis: vary each of the four weights in steps of ten percentage points (renormalized to sum 100) and vary the comment and duplication thresholds by ±5 points, then compute the rank of candidate applications under each parameter set. If the relative ordering of candidates changes under any plausible perturbation, the 'successful comparison' claim is not robust. A complementary test is to check against independent maintainer-effort or change-request-based ground truth for at least ten projects across at least three languages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the slide statement that the approach is 'successful for comparison and explainability on a set of languages from legacy 4GL to current mainstream languages.' The only support offered is the example table in Section 2 with weights 15% comment, 15% duplication, 45% technical debt, 25% volumetry, and thresholds such as comment ratio 15–40% and duplication 15–5%, described as 'simple explainable weights.' The paper's own key principle says to 'consider sensitivity analysis,' but no sensitivity analysis or validation result is reported. Because the final score is a linear weighted sum of relative indicators, candidate rankings are directly determined by these parameters; different plausible weight choices could change which application is selected. The paper also does not specify the exact TDR rule sets used across languages or quantify how 'keep similar rules at intersection of rule sets' affects coverage. Consequently, the success claim is interchangeable with a position statement: no evidence is given that the aggregation yields maintainability assessments that track actual maintenance effort.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is an experience/position report from CETIC on comparing code maintainability across programming languages. It reviews the Maintainability Index, the SIG model, and SQALE, identifies challenges arising from language paradigms, metric coverage, and language-specific rule sets, and proposes a practical approach based on relative indicators, consistent rule sets, and a simple weighted aggregation. The approach is illustrated with a hand-specified mapping table for a functional selection scenario, and the paper concludes that the approach is 'successful for comparison and explainability on a set of languages from legacy 4GL to current mainstream languages', with validation planned on the authors' database.","tokens_in":3806,"tokens_out":3503,"duration_ms":31625,"significance":"If substantiated, the proposed framework would be useful for practitioners who need transparent, explainable cross-language maintainability comparisons, particularly in portfolio prioritization and application selection. The paper correctly identifies real limitations of existing approaches, especially the opacity and arbitrary weights of the Maintainability Index and the rule-set comparability problems in SQALE. It also deserves credit for making its own assumptions explicit and for listing sensitivity analysis as a desirable step. However, the contribution is currently a position statement: no dataset, no quantitative results, no comparison against ground truth, and no sensitivity analysis are provided. The paper references 15 years of code analysis in Wallonia but does not present any results from that work. The central success claim is therefore not yet supported by evidence, and the paper's main value at this stage is as a concise formulation of practical challenges and a candidate methodology.","major_comments":[{"comment":"The assertion 'Approach successful for comparison and explainability on a set of languages from legacy 4GL to current mainstream languages' is the central claim of the paper, but the manuscript provides no dataset, no quantitative results, no comparison against a ground truth or an independent method, and no error or sensitivity analysis. The only illustration is the hand-specified mapping table. This claim should either be backed by evidence from the authors' database or retracted in favor of a statement that the approach is proposed and illustrated as a practical method whose validation is future work.","section":"§2, 'Current status and next steps'"},{"comment":"The weights (comment 15%, duplication 15%, technical debt 45%, volumetry 25%) and thresholds (comment ratio 15%–40%, duplication 15%–5%, TDR 10–20) are presented without justification. Because the final score is a linear combination, candidate rankings are directly determined by these parameters; different plausible choices could change a selection decision. The paper's own key principles state 'compare indicators separately and together, consider sensitivity analysis,' but no sensitivity analysis is reported. Please either report a robustness analysis over plausible weights and thresholds or explicitly mark the table as an illustrative example whose parameters require calibration.","section":"§2, example table"},{"comment":"The advice to 'keep similar rules at intersection of rule sets' is not operationalized. The paper does not state how many rules are common to the languages considered (e.g., 4GL vs C#/Java), how rule-set coverage differences affect the TDR, or how the 'same estimation method' for production effort is defined across languages with different verbosity. Without this detail, the TDR component of the comparison may reflect rule-set coverage rather than maintainability. Please specify the rule intersection, the coverage, and the production-effort estimation method, or explicitly state these as assumptions with caveats.","section":"§2, technical debt ratio comparability"}],"minor_comments":[{"comment":"Reference [1] contains a typo: 'Gloassary' should be 'Glossary'.","section":"References"},{"comment":"The slide text contains 'acro ss' instead of 'across'.","section":"Slides, 'Context'"},{"comment":"The spelling of 'McCabe' is inconsistent: 'Mc Cabe' appears in the SIG limitations, while 'McCabe' appears elsewhere.","section":"Throughout"},{"comment":"The expansion 'Software QuALity Enhancement model (SQALE)' is nonstandard; the usual expansion is 'Software Quality Assessment based on Lifecycle Expectations'.","section":"§2, SQALE description"},{"comment":"The paper says 'Our work is based on a set of code analysis carried out in Wallonia over the past 15 years,' but no anonymized illustrations or summary statistics from that work are actually shown; adding a sentence clarifying that the paper is a position paper without empirical validation would set reader expectations accurately.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"This is more a workshop position statement than a journal article. The main issue is that the central 'successful' claim is unsupported by the evidence in the manuscript, although it can be fixed either by adding a small validation study or by explicitly reducing the claim to a proposed approach. No concerns about novelty disclosure or citation practices beyond the normal expectation that the authors provide the data behind their 15-year experience if they cite it as a basis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a workshop paper that reads as an honest experience report, and its explicit aggregation scheme is a useful practical artifact, but the central claim—that the approach is 'successful' for cross-language comparison—goes beyond what it shows. The evidence is a few anonymized projects and a hand-picked weighting table, not a validation study.\n\nWhat is actually new: not the models, which are standard, but the concrete recipe for making them comparable across languages. The principles—use relative indicators, keep rule sets at the intersection, count each attribute once, use simple transparent weights—are sensible and actionable. The paper is also candid about the limitations of MI, SIG, and SQALE, and the references are appropriate. The example weights (15/15/45/25) are explicit, which is more than most maintainability tools offer, and the mapping table with thresholds is a concrete starting point for a practitioner.\n\nSoft spots: the success claim is the real issue. No sensitivity analysis, no comparison to actual maintenance effort, no dataset with error bars. Because the final score is a weighted sum, rankings can flip under plausible alternative weights; the paper itself lists sensitivity analysis as a principle but does not report it. The TDR rule sets across languages are specified only vaguely, and 'keep similar rules at intersection of rule sets' is not quantified. For a position paper these omissions are acceptable; for the claim as written, they are not. The paper would be stronger if it either presented a validation dataset or framed the framework as a hypothesis to test rather than a proven method.\n\nMinor: the 'Current status' slide is the abstract's worst sentence because it overstates what the paper demonstrates. But the review of MI/SIG/SQALE is accurate and balanced, and there is no circular reasoning here—the paper is not fitting anything to its own conclusions.\n\nWho gets value: practitioners comparing multi-language portfolios or selecting among candidate applications, and researchers who want a concise checklist of the challenges. It is not a new empirical result, but it is a clear and honest statement of the problem.\n\nRecommendation: I would take this to a reading group as a discussion piece. For peer review, I would send it out rather than desk reject, because the topic is practical and the authors' experience is real. The referee should push for either a modest reframing or a genuine validation effort.","headline":"A useful practitioner checklist and explicit weighting scheme, but the 'successful' claim is ahead of the evidence.","tokens_in":4307,"tokens_out":2369,"would_cite":false,"duration_ms":23960,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transparent weights and relative indicators make maintainability scores comparable across programming languages.","keywords":["software maintainability","cross-language comparison","technical debt ratio","relative indicators","static code analysis","software modernization","code audit"],"falsifier":"Recompute the proposed aggregation on the authors' project database with weight vectors varied across a plausible grid (for example, moving the technical-debt weight from 45% down to 30% and reallocating the difference to duplication or volumetry); if the relative ranking of projects written in different languages changes, then the claimed cross-language comparability is an artifact of the chosen weights rather than of the indicators.","tokens_in":3410,"feed_emoji":"⚙️","tokens_out":10459,"duration_ms":84340,"temperature":0.7,"pith_summary":"This paper argues that maintainability can be compared across very different programming languages if the comparison is built from relative indicators, a rule set kept consistent across languages, and a transparent weighted aggregation. It identifies why standard composite indices and raw technical-debt ratios fail in that setting: their weights are opaque, their metrics depend on language verbosity, their rule sets differ, and the production-effort estimate is unstable. The proposed protocol maps each indicator onto a common 0–100 scale, counts each quality attribute once, and combines four indicators with fixed, explainable weights (comment ratio 15%, duplication ratio 15%, technical-debt ratio 45%, volumetry 25%). The authors report that this approach was successful for comparison and explainability on a set of languages from a legacy 4GL to C# and Java, and that it revealed caveats in earlier methods such as tooling bias and duplication estimation.","feed_headline":"Transparent weights make maintainability scores comparable across languages","feed_subtitle":"Portfolios and app choices spanning a legacy 4GL to Java can be ranked on explainable, comparable scores.","key_machinery":"The load-bearing device is a comparison protocol built on relative indicators. Each quality dimension is expressed as a ratio—comment ratio, duplication ratio, technical-debt ratio (the estimated cost of fixing rule violations divided by total production effort), and volumetry relative to a minimum—and mapped through agreed bounds onto a common 0–100 scale. The protocol then combines these scores with fixed weights (15%, 15%, 45%, 25%) that are deliberately simple and visible, counts each attribute once, aligns language-specific rule sets by taking their intersection, and uses the same production-effort estimation for all languages.","core_discovery":"The central claim is that comparable maintainability assessments across languages are achievable through a disciplined comparison protocol, not a new metric. The protocol requires using relative indicators rather than raw counts, applying the same production-effort estimation method to every language, restricting the rule set to the intersection of language-specific rule sets, avoiding object-oriented-only metrics when procedural languages are included, and combining indicators with fixed, explicit weights after mapping each one to a 0–100 scale. The paper's instantiation of this protocol uses four indicators—comment ratio (15%), duplication ratio (15%), technical-debt ratio (45%), and volumetry (25%)—with stated acceptable bounds such as a comment ratio between 15% and 40%. On a set of languages ranging from a legacy 4GL to current mainstream languages, the authors report that the approach succeeded for comparison and explainability and exposed caveats in previous methods, including tooling bias and inconsistencies in duplication estimation.","pith_inferences":["A natural next test is to vary the weights across a plausible grid and check whether cross-language rankings of projects stay stable on the authors' database; the paper does not report such a sensitivity analysis.","If the protocol generalises, the same relative-indicator logic could be extended to architecture-level indicators such as coupling and cohesion, and to testing coverage, which the paper hints at as future work.","Because the protocol takes the intersection of language-specific rule sets, the comparison is only as strong as the language with the least complete rule coverage; a language with richer rules could receive a systematically lower technical-debt score.","A stronger benchmark than the authors' own experience would compare the protocol's scores against actual maintenance effort or expert rankings on the same corpus; the paper leaves this as future validation."],"forward_implications":["A portfolio containing many languages can be ranked on one transparent maintainability scale instead of separate tool-specific scores.","Application selection between functionally similar candidates written in different languages can be done with explainable per-indicator comparisons.","The same protocol can monitor a system during modernisation, since the scale remains stable as the implementation language changes.","Using a common production-effort estimate makes technical-debt ratios comparable across languages, exposing differences that raw rule-violation counts hide.","Reporting indicator-level scores alongside the weighted total lets reviewers see which dimension drives a low rating, avoiding cancellation effects seen in composite indices."],"supporting_citations":[{"why":"Supplies the comparative study of composite-index, quality-model, and technical-debt-ratio methods that the paper uses to enumerate cross-language limitations.","marker":"[6]"},{"why":"Cited for the deficiencies of composite maintainability indices, in particular arbitrary weights and missing quality dimensions such as duplication.","marker":"[7]"},{"why":"Defines the product quality model whose maintainability sub-characteristics the protocol's indicators are meant to cover.","marker":"[3]"},{"why":"Introduces the technical-debt concept that underlies the technical-debt-ratio indicator and its A-to-E scale.","marker":"[4]"},{"why":"Documents the authors' fifteen years of code audits, the empirical base the paper draws on and plans to use for generalisation.","marker":"[5]"}],"fun_headline_variants":["A protocol for fair maintainability comparisons across languages","How to compare code maintainability across languages fairly","Fair maintainability scores: a cross-language comparison protocol","Comparing maintainability across languages? Use this disciplined protocol","No new metric, just a protocol for cross-language maintainability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that hand-chosen weights and thresholds—comments 15%, duplication 15%, technical debt 45%, volumetry 25%, with a comment ratio between 15% and 40% considered acceptable—are valid across languages and that a linear weighted sum of these relative indicators faithfully represents maintainability.","fun_headline_variants_meta":{"raw":{"variants":["A protocol for fair maintainability comparisons across languages","How to compare code maintainability across languages fairly","Fair maintainability scores: a cross-language comparison protocol","Comparing maintainability across languages? Use this disciplined protocol","No new metric, just a protocol for cross-language maintainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1445,"prompt_tokens":834,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":450,"tokens_out":611,"duration_ms":5654,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:12:05.505100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the proposed aggregation on the authors' project database with weight vectors varied across a plausible grid (for example, moving the technical-debt weight from 45% down to 30% and reallocating the difference to duplication or volumetry); if the relative ranking of projects written in different languages changes, then the claimed cross-language comparability is an artifact of the chosen weights rather than of the indicators.","supporting_citations":[{"cited_title":"Strečanskỳ, S","cited_arxiv_id":null,"evidence_quote":"Supplies the comparative study of composite-index, quality-model, and technical-debt-ratio methods that the paper uses to enumerate cross-language limitations."},{"cited_title":"Nugroho, J","cited_arxiv_id":null,"evidence_quote":"Cited for the deficiencies of composite maintainability indices, in particular arbitrary weights and missing quality dimensions such as duplication."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the product quality model whose maintainability sub-characteristics the protocol's indicators are meant to cover."},{"cited_title":"Ponsard, J","cited_arxiv_id":null,"evidence_quote":"Documents the authors' fifteen years of code audits, the empirical base the paper draws on and plans to use for generalisation."}],"review_version":1}