Challenges in Comparing Code Maintainability across Different Programming Languages

Christophe Ponsard; Denis Darquennes; Gustavo Ospina

REVIEW 3 major objections 5 minor 10 references

Challenges in Comparing Code Maintainability across Different Programming Languages

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Transparent weights and relative indicators make maintainability scores comparable across programming languages.

desk verdict A useful practitioner checklist and explicit weighting scheme, but the 'successful' claim is ahead of the evidence. read the letter →

arxiv 2411.15502 v1 pith:D54RRBPW submitted 2024-11-23 cs.SE

Christophe Ponsard , Gustavo Ospina , Denis Darquennes This is my paper · ORCID

classification cs.SE

keywords softwaremaintainabilitycross-languagecomparisontechnicaldebtratiorelativeindicatorsstaticcodeanalysismodernizationaudit

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that maintainability can be compared across very different programming languages if the comparison is built from relative indicators, a rule set kept consistent across languages, and a transparent weighted aggregation. It identifies why standard composite indices and raw technical-debt ratios fail in that setting: their weights are opaque, their metrics depend on language verbosity, their rule sets differ, and the production-effort estimate is unstable. The proposed protocol maps each indicator onto a common 0–100 scale, counts each quality attribute once, and combines four indicators with fixed, explainable weights (comment ratio 15%, duplication ratio 15%, technical-debt ratio 45%, volumetry 25%). The authors report that this approach was successful for comparison and explainability on a set of languages from a legacy 4GL to C# and Java, and that it revealed caveats in earlier methods such as tooling bias and duplication estimation.

What carries the argument

The load-bearing device is a comparison protocol built on relative indicators. Each quality dimension is expressed as a ratio—comment ratio, duplication ratio, technical-debt ratio (the estimated cost of fixing rule violations divided by total production effort), and volumetry relative to a minimum—and mapped through agreed bounds onto a common 0–100 scale. The protocol then combines these scores with fixed weights (15%, 15%, 45%, 25%) that are deliberately simple and visible, counts each attribute once, aligns language-specific rule sets by taking their intersection, and uses the same production-effort estimation for all languages.

What would settle it

Recompute the proposed aggregation on the authors' project database with weight vectors varied across a plausible grid (for example, moving the technical-debt weight from 45% down to 30% and reallocating the difference to duplication or volumetry); if the relative ranking of projects written in different languages changes, then the claimed cross-language comparability is an artifact of the chosen weights rather than of the indicators.

Watch

Extended reading notes

Core claim

The central claim is that comparable maintainability assessments across languages are achievable through a disciplined comparison protocol, not a new metric. The protocol requires using relative indicators rather than raw counts, applying the same production-effort estimation method to every language, restricting the rule set to the intersection of language-specific rule sets, avoiding object-oriented-only metrics when procedural languages are included, and combining indicators with fixed, explicit weights after mapping each one to a 0–100 scale. The paper's instantiation of this protocol uses four indicators—comment ratio (15%), duplication ratio (15%), technical-debt ratio (45%), and volumetry (25%)—with stated acceptable bounds such as a comment ratio between 15% and 40%. On a set of languages ranging from a legacy 4GL to current mainstream languages, the authors report that the approach succeeded for comparison and explainability and exposed caveats in previous methods, including tooling bias and inconsistencies in duplication estimation.

Load-bearing premise

The load-bearing premise is that hand-chosen weights and thresholds—comments 15%, duplication 15%, technical debt 45%, volumetry 25%, with a comment ratio between 15% and 40% considered acceptable—are valid across languages and that a linear weighted sum of these relative indicators faithfully represents maintainability.

Editorial extensions

If this is right

A portfolio containing many languages can be ranked on one transparent maintainability scale instead of separate tool-specific scores.
Application selection between functionally similar candidates written in different languages can be done with explainable per-indicator comparisons.
The same protocol can monitor a system during modernisation, since the scale remains stable as the implementation language changes.
Using a common production-effort estimate makes technical-debt ratios comparable across languages, exposing differences that raw rule-violation counts hide.
Reporting indicator-level scores alongside the weighted total lets reviewers see which dimension drives a low rating, avoiding cancellation effects seen in composite indices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next test is to vary the weights across a plausible grid and check whether cross-language rankings of projects stay stable on the authors' database; the paper does not report such a sensitivity analysis.
If the protocol generalises, the same relative-indicator logic could be extended to architecture-level indicators such as coupling and cohesion, and to testing coverage, which the paper hints at as future work.
Because the protocol takes the intersection of language-specific rule sets, the comparison is only as strong as the language with the least complete rule coverage; a language with richer rules could receive a systematically lower technical-debt score.
A stronger benchmark than the authors' own experience would compare the protocol's scores against actual maintenance effort or expert rankings on the same corpus; the paper leaves this as future validation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a workshop paper that reads as an honest experience report, and its explicit aggregation scheme is a useful practical artifact, but the central claim—that the approach is 'successful' for cross-language comparison—goes beyond what it shows. The evidence is a few anonymized projects and a hand-picked weighting table, not a validation study.

What is actually new: not the models, which are standard, but the concrete recipe for making them comparable across languages. The principles—use relative indicators, keep rule sets at the intersection, count each attribute once, use simple transparent weights—are sensible and actionable. The paper is also candid about the limitations of MI, SIG, and SQALE, and the references are appropriate. The example weights (15/15/45/25) are explicit, which is more than most maintainability tools offer, and the mapping table with thresholds is a concrete starting point for a practitioner.

Soft spots: the success claim is the real issue. No sensitivity analysis, no comparison to actual maintenance effort, no dataset with error bars. Because the final score is a weighted sum, rankings can flip under plausible alternative weights; the paper itself lists sensitivity analysis as a principle but does not report it. The TDR rule sets across languages are specified only vaguely, and 'keep similar rules at intersection of rule sets' is not quantified. For a position paper these omissions are acceptable; for the claim as written, they are not. The paper would be stronger if it either presented a validation dataset or framed the framework as a hypothesis to test rather than a proven method.

Minor: the 'Current status' slide is the abstract's worst sentence because it overstates what the paper demonstrates. But the review of MI/SIG/SQALE is accurate and balanced, and there is no circular reasoning here—the paper is not fitting anything to its own conclusions.

Who gets value: practitioners comparing multi-language portfolios or selecting among candidate applications, and researchers who want a concise checklist of the challenges. It is not a new empirical result, but it is a clear and honest statement of the problem.

Recommendation: I would take this to a reading group as a discussion piece. For peer review, I would send it out rather than desk reject, because the topic is practical and the authors' experience is real. The referee should push for either a modest reframing or a genuine validation effort.

Referee Report

3 major / 5 minor

Summary. The paper is an experience/position report from CETIC on comparing code maintainability across programming languages. It reviews the Maintainability Index, the SIG model, and SQALE, identifies challenges arising from language paradigms, metric coverage, and language-specific rule sets, and proposes a practical approach based on relative indicators, consistent rule sets, and a simple weighted aggregation. The approach is illustrated with a hand-specified mapping table for a functional selection scenario, and the paper concludes that the approach is 'successful for comparison and explainability on a set of languages from legacy 4GL to current mainstream languages', with validation planned on the authors' database.

Significance. If substantiated, the proposed framework would be useful for practitioners who need transparent, explainable cross-language maintainability comparisons, particularly in portfolio prioritization and application selection. The paper correctly identifies real limitations of existing approaches, especially the opacity and arbitrary weights of the Maintainability Index and the rule-set comparability problems in SQALE. It also deserves credit for making its own assumptions explicit and for listing sensitivity analysis as a desirable step. However, the contribution is currently a position statement: no dataset, no quantitative results, no comparison against ground truth, and no sensitivity analysis are provided. The paper references 15 years of code analysis in Wallonia but does not present any results from that work. The central success claim is therefore not yet supported by evidence, and the paper's main value at this stage is as a concise formulation of practical challenges and a candidate methodology.

major comments (3)

[§2, 'Current status and next steps'] The assertion 'Approach successful for comparison and explainability on a set of languages from legacy 4GL to current mainstream languages' is the central claim of the paper, but the manuscript provides no dataset, no quantitative results, no comparison against a ground truth or an independent method, and no error or sensitivity analysis. The only illustration is the hand-specified mapping table. This claim should either be backed by evidence from the authors' database or retracted in favor of a statement that the approach is proposed and illustrated as a practical method whose validation is future work.
[§2, example table] The weights (comment 15%, duplication 15%, technical debt 45%, volumetry 25%) and thresholds (comment ratio 15%–40%, duplication 15%–5%, TDR 10–20) are presented without justification. Because the final score is a linear combination, candidate rankings are directly determined by these parameters; different plausible choices could change a selection decision. The paper's own key principles state 'compare indicators separately and together, consider sensitivity analysis,' but no sensitivity analysis is reported. Please either report a robustness analysis over plausible weights and thresholds or explicitly mark the table as an illustrative example whose parameters require calibration.
[§2, technical debt ratio comparability] The advice to 'keep similar rules at intersection of rule sets' is not operationalized. The paper does not state how many rules are common to the languages considered (e.g., 4GL vs C#/Java), how rule-set coverage differences affect the TDR, or how the 'same estimation method' for production effort is defined across languages with different verbosity. Without this detail, the TDR component of the comparison may reflect rule-set coverage rather than maintainability. Please specify the rule intersection, the coverage, and the production-effort estimation method, or explicitly state these as assumptions with caveats.

minor comments (5)

[References] Reference [1] contains a typo: 'Gloassary' should be 'Glossary'.
[Slides, 'Context'] The slide text contains 'acro ss' instead of 'across'.
[Throughout] The spelling of 'McCabe' is inconsistent: 'Mc Cabe' appears in the SIG limitations, while 'McCabe' appears elsewhere.
[§2, SQALE description] The expansion 'Software QuALity Enhancement model (SQALE)' is nonstandard; the usual expansion is 'Software Quality Assessment based on Lifecycle Expectations'.
[Abstract and §1] The paper says 'Our work is based on a set of code analysis carried out in Wallonia over the past 15 years,' but no anonymized illustrations or summary statistics from that work are actually shown; adding a sentence clarifying that the paper is a position paper without empirical validation would set reader expectations accurately.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper makes no fitted prediction and its weight/threshold scheme is explicitly illustrative, with validation deferred.

full rationale

The paper is a practical experience/position paper, not a derivation or prediction chain. Its central claim that the proposed approach is 'successful for comparison and explainability' is supported by illustrative example weights (comment ratio 15%, duplication 15%, technical debt 45%, volumetry 25%) and thresholds, but these are presented as 'simple explainable weights,' not as fitted parameters or as outputs of a derivation. No quantity is predicted from fitted inputs, and no result is shown to be equivalent to its own assumptions by construction. The one self-citation, to the authors' prior database of Walloon code audits [4], appears only as background context and as the planned target for future generalisation ('Planning to generalise on our database [4]'), not as load-bearing evidence for the present claims. The paper explicitly frames the next step as validation ('Planning to generalise on our database'), which is the opposite of presenting a fitted result as an independent prediction. Weaknesses such as the unvalidated weighting scheme, missing sensitivity analysis, and unsupported success assertion are correctness/validity risks, not circularity. Under the hard rules, no circular step can be exhibited with a quote and a specific reduction, so the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities, but its central claim depends on several domain assumptions about the validity of static metrics and the cross-language comparability of normalized indicators. The example weights are free parameters chosen for illustration and are not fitted to data, but they are nevertheless arbitrary and untested.

free parameters (4)

Weight for comment ratio = 15%
Chosen by hand in the example aggregation table; no justification or sensitivity analysis is provided.
Weight for duplication ratio = 15%
Chosen by hand in the example aggregation table; the inverse scale (15% to 5%) is also arbitrary.
Weight for technical debt ratio = 45%
Chosen by hand and set to three times the individual indicator weights; no empirical basis is given.
Weight for volumetry = 25%
Chosen by hand in the example aggregation table; the threshold 'Min to 1.5 x Min' is unstated in its derivation.

assumptions (3)

domain assumption Static code analysis metrics are valid proxies for software maintainability.
The entire paper assumes that metrics like complexity, duplication, and comment ratio reflect maintainability, following the SQuaRE model, without questioning the underlying construct validity.
domain assumption Relative indicators, normalized to a common scale, are comparable across different programming languages.
The proposed method relies on the belief that scaling indicators (e.g., comment ratio 15%-40%) removes language-specific verbosity and paradigm differences, but no evidence is provided for this comparability.
ad hoc to paper A linear weighted sum of the chosen indicators is a meaningful measure of maintainability.
The paper proposes a specific linear aggregation with chosen weights, but this assumes additivity and no interaction effects between metrics, which is not justified by theory or data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Challenges in Comparing Code Maintainability across Different Programming Languages." pith.science (2026). https://pith.science/paper/D54RRBPW

@misc{pith2026241115502,
  author       = {Pith},
  title        = {Pith review of: Challenges in Comparing Code Maintainability across Different Programming Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D54RRBPW}},
  note         = {Machine review of arXiv:2411.15502}
}

read the original abstract

Comparing the quality of software written in different computer languages is required in a variety of scenarios, e.g. multi-language projects or application selection process among candidates in different languages. We focus on the challenges related to comparing the maintainability quality typically through a maintainability index or technical debt approaches. We identify and discuss how to manage a number of challenges to produce comparable maintainability assessments across languages related to the programming paradigm (purely procedural vs OO vs multi-paradigm), the coverage of key quality dimensions, and the use of generic metrics vs more languages specific rules. Our work is based on a set of code analysis carried out in Wallonia over the past 15 years.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages

[1]

IEEE, Standard Gloassary of Software Engineering Terminology, 1990

work page 1990
[2]

Christa, et al., Software maintenance: From the perspective of effort and cost requirement, in: Proc

S. Christa, et al., Software maintenance: From the perspective of effort and cost requirement, in: Proc. of the International Conference on Data Engineering and Communication Technology, 2017

work page 2017
[3]

ISO, 25010:2023 - Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model, https://www.iso.org/standard/78176.html, 2023

work page 2023
[4]

Cunningham, The wycash portfolio management system, SIGPLAN OOPS Mess

W. Cunningham, The wycash portfolio management system, SIGPLAN OOPS Mess. 4 (1992) 29–30. URL: https://doi.org/10.1145/157710.157715. doi:10.1145/157710.157715

arXiv 1992
[5]

Ponsard, J

C. Ponsard, J. Deprez, Helping smes to better develop software: experience report and challenges ahead, in: Proc. of the 40th International Conference on Software Engineering:(SEIP), 2018

work page 2018
[6]

Strečanskỳ, S

P. Strečanskỳ, S. Chren, B. Rossi, Comparing maintainability index, SIG method, and SQALE for technical debt identification, in: Proc. of the 35th Annual ACM Symp. on Applied Computing, 2020

work page 2020
[7]

Nugroho, J

A. Nugroho, J. Visser, T. Kuipers, An empirical model of technical debt and interest, in: Proc. of the 2nd Workshop on Managing Technical Debt, Association for Computing Machinery, 2011. Challenges in Comparing Code Maintainability acro ss Different Programming Languages Christophe Ponsard, Gustavo Ospina, Denis Darquennes - CETIC - christophe.ponsard@cet...

work page 2011
[8]

Stre čansk ỳ, S

P. Stre čansk ỳ, S. Chren, B. Rossi, Comparing maintainability index, SIG method, and SQALE for technical debt identification, in: Proc. 35th Annual ACM Symp. on Applied Computing, 2020

work page 2020

Show all 10 references

[9]

Nugroho, J

A. Nugroho, J. Visser, T. Kuipers, An empirical model of technical debt and interest, in: Proc. of the 2nd Workshop on Managing Technical Debt, Association for Computing Machinery, 2011

2011
[10]

Ponsard, J

C. Ponsard, J. Deprez, Helping smes to better develop software: experience report and challenges ahead, in: Proc. of the 40th International Conference on Software Engineering:(SEIP), 2018. Existing approaches (based on [2]) and limitations Maintainability Index (Oman and Hagem...

2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

IEEE, Standard Gloassary of Software Engineering Terminology, 1990

work page 1990

[2] [2]

Christa, et al., Software maintenance: From the perspective of effort and cost requirement, in: Proc

S. Christa, et al., Software maintenance: From the perspective of effort and cost requirement, in: Proc. of the International Conference on Data Engineering and Communication Technology, 2017

work page 2017

[3] [3]

ISO, 25010:2023 - Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model, https://www.iso.org/standard/78176.html, 2023

work page 2023

[4] [4]

Cunningham, The wycash portfolio management system, SIGPLAN OOPS Mess

W. Cunningham, The wycash portfolio management system, SIGPLAN OOPS Mess. 4 (1992) 29–30. URL: https://doi.org/10.1145/157710.157715. doi:10.1145/157710.157715

arXiv 1992

[5] [5]

Ponsard, J

C. Ponsard, J. Deprez, Helping smes to better develop software: experience report and challenges ahead, in: Proc. of the 40th International Conference on Software Engineering:(SEIP), 2018

work page 2018

[6] [6]

Strečanskỳ, S

P. Strečanskỳ, S. Chren, B. Rossi, Comparing maintainability index, SIG method, and SQALE for technical debt identification, in: Proc. of the 35th Annual ACM Symp. on Applied Computing, 2020

work page 2020

[7] [7]

Nugroho, J

A. Nugroho, J. Visser, T. Kuipers, An empirical model of technical debt and interest, in: Proc. of the 2nd Workshop on Managing Technical Debt, Association for Computing Machinery, 2011. Challenges in Comparing Code Maintainability acro ss Different Programming Languages Christophe Ponsard, Gustavo Ospina, Denis Darquennes - CETIC - christophe.ponsard@cet...

work page 2011

[8] [8]

Stre čansk ỳ, S

P. Stre čansk ỳ, S. Chren, B. Rossi, Comparing maintainability index, SIG method, and SQALE for technical debt identification, in: Proc. 35th Annual ACM Symp. on Applied Computing, 2020

work page 2020