REVIEW 3 major objections 7 minor 35 references
Methodological Advances and Challenges in Indirect Treatment Comparisons: A Review of International Guidelines and HAS TC Case Studies
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Only 13.3% of indirect treatment comparisons submitted to the French Transparency Committee from 2021 to 2023 influenced reimbursement decisions, with acceptance ranging from 34.4% in genetic diseases to 4.2% for network meta-analyses.
desk verdict Useful French HTA dataset, but the 13.3% 'influenced' headline loses the paper's own distinction between accepted and merely not-rejected ITCs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central instrument is an acceptability classification built from the Transparency Committee's published summaries. Each ITC was tagged as acceptable, not rejected, not acceptable, or unclear depending on whether and how the decision summary mentioned the ITC, with the ITC rather than the drug opinion as the unit of analysis. A 41-variable extraction grid and a three-branch taxonomy of limitations (data, methodology, uncertainty) let the authors quantify acceptance by therapeutic area, by statistical method, and by whether the ITC was the primary evidence source.
What would settle it
Look for a sample of the 169 ITCs classified as not influencing decisions in the committee's internal records or in interviews with committee members: if a substantial share were actually discussed and used in the SMR or ASMR reasoning, the 13.3% estimate is an artifact of written reporting. A simpler check would re-run the same dataset coding influence from whether the ITC changed the final rating relative to the manufacturer's requested rating and compare the two acceptance rates.
Extended reading notes
Core claim
Across all French Transparency Committee opinions from 2021 to 2023, 138 contained at least one indirect comparison, yielding 195 ITCs for analysis. Only 13.3% were considered in the ASMR decision; 86.7% were not. When an ITC was the primary source of comparative effectiveness, the proportion of important clinical benefit fell to 60.9% versus 73.4% when randomized controlled trials anchored the comparison, and the proportion of insufficient benefit rose from 9.6% to 18.8%. The committee's most frequent criticisms were heterogeneity or risk of bias (59%), lack of or unclear data (48%), statistical-methodology problems (29%), study-design concerns (27%), small sample size (25%), and variability in outcome definition or timing (20%). The authors read the low acceptance of network meta-analyses and Bucher comparisons (4.2%) as a sign that strict homogeneity and consistency conditions are often unmet, while the higher acceptance of unadjusted comparisons reflects settings of extreme unmet medical need rather than methodological preference.
Load-bearing premise
The classification of an ITC as influencing the decision depends entirely on whether the Transparency Committee's written ASMR summary mentions it, so an ITC that was weighed but not cited is counted as not influencing the decision.
Editorial extensions
If this is right
- Manufacturers submitting ITCs to French reimbursement reviews should expect most to be ignored in the final ASMR decision, even when the ITC is the only comparative evidence.
- The method gradient means network meta-analyses and Bucher comparisons are unlikely to influence decisions unless homogeneity and consistency are convincingly demonstrated, whereas IPD-based or unadjusted comparisons in rare and genetic diseases have a better chance.
- Relying on an ITC as the primary source of comparative evidence is associated with a lower probability of an important clinical benefit rating and a higher probability of an insufficient rating.
- Because international guidelines largely agree, a single well-conducted ITC dossier could in principle serve multiple European HTA bodies, with the main national divergence centered on population-adjusted methods.
Reading between the lines
- The 13.3% figure is only as good as the written summaries: a natural validation would compare the summary-based classification against the committee's internal deliberation records or manufacturers' own accounts of what was discussed.
- The higher acceptance of unadjusted comparisons is probably confounded with disease context, since companies tend to submit them only where unmet need is extreme and treatment effects are dramatic; matching on disease area could separate method quality from context.
- If the 2023 HAS doctrine changes ITC requirements, the acceptance rate among opinions from 2024 onward would be a direct test of whether clearer guidance actually raises the influence of indirect comparisons.
- The guideline comparison suggests that a common European ITC reporting template could reduce duplication, a policy implication the paper notes but does not test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a pragmatic review of indirect treatment comparison (ITC) guidelines from seven HTA and multistakeholder bodies, together with a systematic analysis of all French Transparency Committee (TC) opinions published between 2021 and 2023 that mention at least one ITC. The authors extracted 195 ITCs from 138 TC opinions, classified each ITC's acceptability on the basis of how it was treated in the TC's ASMR summary, and reported acceptance rates by disease area, by statistical method, and by whether the ITC was the primary source of evidence. The paper's central quantitative claim is that only 13.3% of submitted ITCs influenced TC decision-making, with higher acceptance in genetic diseases and for IPD-based methods, and it also describes the most common limitations identified by the TC.
Significance. If the classification issues were resolved, this would be a valuable and comprehensive descriptive study of ITC use in French HTA decision-making. Its strengths include a census-like sampling frame (all TC opinions in a three-year window), a structured 41-variable extraction template, manual review with a documented 10% quality-assurance check, and a transparent list of the most frequently cited limitations. The guideline comparison in Table 1 is a useful synthesis. The paper is, however, descriptive rather than methodological, and the headline influence rate is not currently supported by the paper's own category definitions. The dataset and extraction approach could serve as a reproducible baseline for future evaluations if the terminological and classification ambiguities are corrected.
major comments (3)
- [Abstract; Methods (Overall acceptability); Results (paragraph beginning 'Among the 195 indirect comparisons')] The headline 'Only 13.3% of these ITCs influenced TC decision-making' conflates the category 'Acceptable' with 'Not rejected.' The Methods define three distinct outcomes: 'Acceptable,' 'Not rejected' (considered for the decision-making but with clear stated issues), and 'Not acceptable.' The Results report 26/195 (13.3%) as 'considered,' and the text around Figure 1 implies 6 accepted ITCs (3.1%) and 20 not rejected ITCs (10.3%). If 'influence' is reserved for ITCs the TC positively relied on, the rate is 3.1%; if it includes ITCs described as having 'clear stated issues,' the abstract's wording needs an explicit definition. The abstract, Results, and Discussion should report the three categories separately and use consistent terminology such as 'considered,' 'accepted,' and 'not rejected' rather than using 'influenced' for the combined category.
- [Results, paragraph after Figure 1] The sentence 'the percentage of accepted indirect comparisons is higher when the ITC is the main source for comparing effectiveness (21.6% vs 5.8%)' directly contradicts the preceding sentence reporting 3% acceptance in both the main-source and non-main-source groups. The values 21.6% (20/92) and 5.8% (6/103) correspond to the combined 'accepted or not rejected' category, not to 'accepted' alone. This incorrect label propagates to the discussion of IPD-based versus NMA-based acceptance rates and should be corrected, with the accepted and not-rejected subcategories reported separately throughout.
- [Methods (Overall acceptability definition)] The acceptability classification depends entirely on whether and how the ITC is mentioned in the TC's ASMR summary. This assumes that the written summary fully captures the role of the ITC in the actual decision. A TC could weigh an ITC without citing it, or mention it without substantive influence. This measurement-validity assumption should be acknowledged as a limitation and, if possible, tested by comparing a sample of classifications against the full TC dossiers rather than only the opinion summaries.
minor comments (7)
- [Throughout] The terms 'accepted,' 'considered,' and 'influenced' are used interchangeably; please standardize the terminology to match the definitions in the Methods.
- [Abstract and Results] The abstract refers to 'important clinical benefit' while the Results report 'important SMR'; use one consistent term for this outcome.
- [Methods, first sentence] The sentence 'This article are based on two complementary works' should read 'This article is based on two complementary works.'
- [Methods, TC opinion screening] The automated keyword algorithm is described only as 'based on key words' with a reference to Supplementary Methods; please provide the exact search strategy and validation results in the main text or a fully available supplement.
- [Table 1 legend] The table uses both 'X' and 'O' but the legend explains only 'X' and 'o'; unify the case and clearly define all symbols.
- [Figure 2 and accompanying text] Some acceptance-rate comparisons are based on very small denominators (e.g., unadjusted comparisons 3/8); please report exact counts with confidence intervals or explicit cautions about small-sample comparisons.
- [Discussion] The statement 'This is in line with HTA recommendations... (HAS, 2019)' cites a year not present in the reference list; either add the reference or correct the citation.
Circularity Check
No significant circularity: this is a descriptive coding review, not a derivation; the 13.3% headline is an operational category issue, not a logical loop.
full rationale
This paper is a descriptive, empirically coded review rather than a mathematical derivation. It reviews HTA guidelines and manually codes 195 ITCs from 138 TC opinions using a predefined extraction grid. The central statistics (13.3% considered, acceptance by disease area and method, limitation frequencies, SMR comparisons) are direct tabulations of the coded data, not outputs of a model fitted to those same data. No parameter is fitted and then redesignated as a prediction, and no equation is used whose definition pre-implies the result. The only adjacent concern is that 'influenced' is operationalized as 'considered' in the TC summary, with 'Not rejected' (considered but with stated issues) counted alongside 'Acceptable'; this is a coding and interpretation limitation, not a circular derivation, because the paper does not derive the category from the outcome it is measuring. The guideline comparison relies on standard HTA documents and published literature, with no load-bearing self-citation or imported uniqueness theorem. Therefore there is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption The ASMR summary text in each TC opinion accurately reflects the role of each ITC in the decision-making process.
- domain assumption The automated keyword search of HAS opinions captured all opinions containing at least one ITC.
- domain assumption The manual extraction and categorization of ITC limitations is reproducible across reviewers.
Cite this review
Pith. "Pith review of Methodological Advances and Challenges in Indirect Treatment Comparisons: A Review of International Guidelines and HAS TC Case Studies." pith.science (2026). https://pith.science/paper/QJNQFHGC
@misc{pith2026250611587,
author = {Pith},
title = {Pith review of: Methodological Advances and Challenges in Indirect Treatment Comparisons: A Review of International Guidelines and HAS TC Case Studies},
year = {2026},
howpublished = {\url{https://pith.science/paper/QJNQFHGC}},
note = {Machine review of arXiv:2506.11587}
}
read the original abstract
To evaluate methodological challenges and regulatory considerations of indirect treatment comparisons (ITCs) with the analysis of international health technology assessment guidelines and French Transparency Committee (TC) decisions. We conducted a pragmatic review of ITC guidelines from major health technology assessment (HTA) bodies and multistakeholder organizations. Then, we analyzed TC opinions published between 2021-2023. We extracted data on ITC methodology, therapeutic areas, acceptability, and limitations expressed by the TC. The targeted review of the main guidelines showed mainly agreements between HTA bodies and multistakeholder organizations, with some specificities. 138 TC opinions containing 195 ITCs were analyzed. Only 13.3% of these ITCs influenced TC decision-making. ITCs were more frequently accepted in genetic diseases (34.4%) compared to oncology (10.0%) and autoimmune diseases (11.1%). Methods using individual patient data showed higher acceptance rates (23.1%) than network meta-analyses (4.2%). Main limitations included heterogeneity/bias risk (59%), lack of data (48%), statistical methodology issues (29%), study design concerns (27%), small sample size (25%), and outcome definition variability (20%). When ITCs were the primary source of evidence, the proportion of important clinical benefit was lower (60.9% vs. 73.4%) than when randomized controlled trials were available. While ITCs are increasingly submitted, particularly where direct evidence is impractical, their influence on reimbursement decisions remains limited. There is a need for clear and accessible guides so manufacturers can produce clearer and more robust ITCs that follow regulatory guidelines, from the planning phase to execution.
Reference graph
Works this paper leans on
-
[1]
Use of External Comparators for Health Technology Assessment Submissions Based on Single-Arm Trials
Patel D, Grimson F, Mihaylova E, Wagner P, Warren J, van Engen A, et al. Use of External Comparators for Health Technology Assessment Submissions Based on Single-Arm Trials. Value Health. 1 août 2021;24(8):1118‑25
work page 2021
-
[2]
Pooley N, Kisomi M, Embleton N, Langham S. CO163 The Increasing Use of Population- Adjusted Indirect Comparisons in the NICE Health Technology Assessment (HTA) Submission Process and the Response to These Methods. Value Health. 1 déc 2022;25(12):S49
work page 2022
-
[3]
FDA, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Data Standards for Drug and Biological Product Submissions Containing Real-World Data - Guidance for Industry. https://www.fda.gov/regulatory-information/search-fda-guidance- documents/data-standards-drug-and-biological-product-submissions-containing-real-world-dat...
work page 2023
-
[4]
FDA, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), Oncology Center of Excellence (OCE). Real-World Data: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products. déc 2023
work page 2023
-
[5]
Indirect comparisons: Methods and validity
Haute Autorité de Santé. Indirect comparisons: Methods and validity. 2009
work page 2009
-
[6]
Guide méthodologique : Choix méthodologiques pour l’évaluation économique à la HAS
Haute Autorité de Santé. Guide méthodologique : Choix méthodologiques pour l’évaluation économique à la HAS. https://www.has-sante.fr/upload/docs/application/pdf/2020- 07/guide_methodologique_evaluation_economique_has_2020_vf.pdf. Accessed October 18, 2023
work page 2020
-
[7]
Haute Autorité de Santé. Guide méthodologique « Etudes en vie réelle pour l’évaluation des médicaments et dispositifs médicaux ». https://www.has- sante.fr/upload/docs/application/pdf/2021-06/guide_etude_en_vie_reelle_medicaments__dm.pdf. Accessed September 18, 2024
work page 2021
-
[8]
Rapid access to innovative medicinal products while ensuring relevant health technology assessment
Vanier A, Fernandez J, Kelley S, Alter L, Semenzato P, Alberti C, et al. Rapid access to innovative medicinal products while ensuring relevant health technology assessment. Position of the French National Authority for Health. BMJ Evid-Based Med. 14 févr 2023;bmjebm-2022- 112091
work page 2023
Show all 35 references
-
[9]
Doctrine de la commission de la transparence (CT) Principes d’évaluation de la CT relatifs aux médicaments en vue de leur accès au remboursement
Haute Autorité de Santé. Doctrine de la commission de la transparence (CT) Principes d’évaluation de la CT relatifs aux médicaments en vue de leur accès au remboursement. https://www.has-sante.fr/upload/docs/application/pdf/2021-03/doctrine_ct.pdf. Accessed December 1, 2023
2021
-
[10]
Guideline on Clinical Trials in Small Populations
EMA. Guideline on Clinical Trials in Small Populations. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-clinical-trials-small- populations_en.pdf. Accessed December 1, 2023
2023
-
[11]
Guideline on the clinical evaluation of anticancer medicinal products
EMA. Guideline on the clinical evaluation of anticancer medicinal products. EMA; 2020
2020
-
[12]
Reflection paper on establishing efficacy based on single-arm trials submitted as pivotal evidence in a marketing authorisation - Considerations on evidence from single-arm trials
EMA/CHMP. Reflection paper on establishing efficacy based on single-arm trials submitted as pivotal evidence in a marketing authorisation - Considerations on evidence from single-arm trials. https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-establish...
2023
-
[13]
Methods Guideline-D4.3.2-Methodological-Guideline-on-Direct-and-indirect- comparisons-V1.0.pdf
EUnetHTA. Methods Guideline-D4.3.2-Methodological-Guideline-on-Direct-and-indirect- comparisons-V1.0.pdf. https://www.eunethta.eu/wp-content/uploads/2022/08/EUnetHTA-21- Deliverable-D4.3.2-Methodological-Guideline-on-Direct-and-indirect-comparisons-V1.0.pdf. Accessed December 18, 2023
2022
-
[14]
EUnetHTA 21 - Individual Practical Guideline Document D4.3.1: DIRECT AND INDIRECT COMPARISONS [Internet]
EUnetHTA. EUnetHTA 21 - Individual Practical Guideline Document D4.3.1: DIRECT AND INDIRECT COMPARISONS [Internet]. https://www.eunethta.eu/wp- content/uploads/2022/12/EUnetHTA-21-D4.3.1-Direct-and-indirect-comparisons-v1.0.pdf. Accessed October 10, 2023
2022
-
[15]
Practical Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons
HTA CG. Practical Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons. 2024
2024
-
[16]
Methodological Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons
HTA CG. Methodological Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons. 2024
2024
-
[17]
Use of indirect and mixed treatment comparisons for technology assessment
Sutton A, Ades AE, Cooper N, Abrams K. Use of indirect and mixed treatment comparisons for technology assessment. PharmacoEconomics. sept 2008;26(9):753‑67
2008
-
[18]
Checking consistency in mixed treatment comparison meta-analysis
Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatment comparison meta-analysis. Stat Med. 30 mars 2010;29(7‑8):932‑44
2010
-
[19]
Evidence Synthesis of Treatment Efficacy in Decision Making: A Reviewer’s Checklist
Ades AE, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. Evidence Synthesis of Treatment Efficacy in Decision Making: A Reviewer’s Checklist. (NICE Decision Support Unit Technical Support Documents). http://www.ncbi.nlm.nih.gov/books/NBK395872/. Accessed January 10, 2023
2023
-
[20]
NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression,bias and bias-adjustment
Dias S, Sutton AJ, Welton NJ, Ades A. NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression,bias and bias-adjustment. https://www.sheffield.ac.uk/nice-dsu/tsds/evidence-synthesis. Accessed May 20, 2023
2023
-
[21]
Guide to the methods of technology appraisal
NICE. Guide to the methods of technology appraisal. 2013; https://www.nice.org.uk/process/pmg9/chapter/foreword. Accessed May 20, 2023
2013
-
[22]
NICE DSU Technical Support Document 4: Inconsistency in networks of evidence based on randomised controlled trials
Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades A. NICE DSU Technical Support Document 4: Inconsistency in networks of evidence based on randomised controlled trials. https://www.sheffield.ac.uk/nice-dsu/tsds/evidence-synthesis. Accessed May 20, 2023
2023
-
[23]
Appendix K: Network meta-analysis reporting standards | Tools and resources | Developing NICE guidelines: the manual | Guidance | NICE
NICE. Appendix K: Network meta-analysis reporting standards | Tools and resources | Developing NICE guidelines: the manual | Guidance | NICE. https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual- appendices-2549710189/chapter/appendix-k-network...
2023
-
[24]
NICE DSU Technical Support Document 2 : A general linear modelling framework for pair-wise and network meta-analysis of randomised controlled trials
Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 2 : A general linear modelling framework for pair-wise and network meta-analysis of randomised controlled trials. https://www.sheffield.ac.uk/nice-dsu/tsds/evidence-synthesis. Accessed May 20, 2023
2023
-
[25]
Assessing evidence inconsistency in mixed treatment comparisons
Lu G, Ades A. Assessing evidence inconsistency in mixed treatment comparisons. J Am Stat Assoc. juin 2006;101 (474):447‑59
2006
-
[26]
Methods for Population- Adjusted Indirect Comparisons in Health Technology Appraisal
Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. Methods for Population- Adjusted Indirect Comparisons in Health Technology Appraisal. Med Decis Mak Int J Soc Med Decis Mak. févr 2018;38(2):200‑11
2018
-
[27]
Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals
Phillippo DM, Dias S, Elsada A, Ades AE, Welton NJ. Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals. Int J Technol Assess Health Care. janv 2019;35(3):221‑8
2019
-
[28]
real-world evidence framework
NICE. real-world evidence framework. www.nice.org.uk/corporate/ecd9. Accessed October 18, 2023
2023
-
[29]
Meta-Analyses of Randomized Controlled Clinical Trials to Evaluate the Safety of Human Drugs or Biological Products Guidance for Industry
FDA. Meta-Analyses of Randomized Controlled Clinical Trials to Evaluate the Safety of Human Drugs or Biological Products Guidance for Industry. U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for B...
2024
-
[30]
Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products (Draft guidance)
FDA. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products (Draft guidance). https://www.fda.gov/regulatory-information/search-fda- guidance-documents/considerations-design-and-conduct-externally-controlled-trials-drug-and- ...
2024
-
[31]
Conducting indirect- treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2
Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, et al. Conducting indirect- treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health J Int Soc Pha...
2011
-
[32]
Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Val...
2011
-
[33]
Jansen JP, Trikalinos T, Cappelleri JC, Daw J, Andes S, Eldessouki R, et al. Indirect Treatment Comparison/Network Meta-Analysis Study Questionnaire to Assess Relevance and Credibility to Inform Health Care Decision Making: An ISPOR-AMCP-NPC Good Practice Task Force Report. Va...
2014
-
[34]
ISPOR | International Society For Pharmacoeconomics and Outcomes Research
Willke RJ. ISPOR | International Society For Pharmacoeconomics and Outcomes Research. 2017. Comparative-Effectiveness Research: New Progress in Assessing the Evidence in Today’s Health Care Marketplace. https://www.ispor.org/publications/journals/value-outcomes- spotlight/abst...
2017
-
[35]
Yes”,”No
IQWIG. General Methods - Version 7.0. Gen Methods. https://www.iqwig.de/en/about- us/methods/methods-paper/. Accessed September 19, 2023. SUPPLEMENTARY MATERIAL Appendix 1 – Variables extracted - Common by opinion: o Automatically extracted: Review ID Evamed Number link,...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.