The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance

Csaba Kiss; Gabriele Pergola; Roland Molontay

REVIEW 2 major objections 2 minor 16 references

The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Domain-specific pre-training gives BioBERT the highest accuracy and best match to traditional signals in causal adverse drug event detection.

desk verdict BioBERT beats the other three models on accuracy and signal concordance in these two specific tasks, but the work stays narrow. read the letter →

arxiv 2606.17113 v1 pith:ZW57ONFU submitted 2026-06-15 cs.LG cs.CL

Csaba Kiss , Roland Molontay , Gabriele Pergola This is my paper · ORCID

classification cs.LGcs.CL

keywords causalinferencepharmacovigilanceadversedrugeventstransformermodelsmodelselectiondomain-specificpre-trainingBioBERTcalibration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines how the underlying classification model affects performance inside a system that combines transformers with do-calculus to separate causal adverse drug events from spurious associations. It runs repeated cross-validation on two pharmacovigilance benchmarks, comparing a tree baseline, the original transformer, a biomedical transformer, and a fine-tuned medical large language model while tracking accuracy, calibration error, and agreement with disproportionality metrics. BioBERT records the top accuracy scores and strongest concordance, whereas the larger model trails and calibration produces mixed downstream effects. These patterns indicate that targeted pre-training on biomedical text outweighs both simpler baselines and increases in model scale for this task.

What carries the argument

The framework that integrates a transformer-based classifier with do-calculus operations to perform causal inference on adverse drug event reports.

What would settle it

A larger, independent pharmacovigilance collection in which the medical large language model or the tree baseline equals or exceeds the biomedical transformer in accuracy and in Jaccard overlap with the three traditional signals.

Watch

Extended reading notes

Core claim

Within the framework that pairs transformer classifiers with do-calculus to identify causal adverse drug events, BioBERT attains the highest accuracy on both the analgesics-induced liver failure and tramadol mortality benchmarks while also delivering the largest Jaccard overlap with PRR, ROR, and EBGM signals. The biomedical transformer outperforms the original model, the tree-based baseline, and the medical large language model despite the latter's greater size and parameter-efficient adaptation. Isotonic calibration lowers expected calibration error yet shows inconsistent influence on accuracy and on the set of terms recovered as causal.

Load-bearing premise

The two selected benchmarks together with accuracy, calibration error, and Jaccard concordance against PRR, ROR, and EBGM are representative enough to rank model performance for causal adverse event detection in general.

Editorial extensions

If this is right

Domain-specific pre-training produces better causal term recovery than scaling model size alone.
Post-hoc calibration reliably reduces expected calibration error but does not guarantee gains in accuracy or in concordance with established signals.
Simpler baselines remain usable yet consistently lag the domain-adapted transformer on both accuracy and signal agreement.
Traditional disproportionality metrics continue to serve as external validators for new causal methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the observed ranking persists across additional drug classes, resources for pharmacovigilance systems may be better allocated to biomedical fine-tuning than to ever-larger general models.
The mixed calibration results imply that separate metrics for calibration and for causal discovery may be needed rather than relying on accuracy alone.
Extending the comparison to multi-task or ensemble variants of the top-performing model could reveal whether further gains remain available without increasing parameter count.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main thing to know is that BioBERT came out ahead of XGBoost, ALBERT, and Med-LLaMA on both AILF and TRAM for accuracy and overlap with traditional PRR/ROR/EBGM signals, while the larger Med-LLaMA lagged. Domain pre-training looked more useful than scale.

The paper runs a clean head-to-head inside the existing InferBERT setup. It uses 5-fold cross-validation repeated 20 times, paired t-tests, accuracy, ECE before and after isotonic calibration, and Jaccard overlap with the classic disproportionality metrics. That gives a concrete ordering on these two drug-event pairs and shows calibration helps the error score but has mixed effects elsewhere.

What is new is the direct comparison of these four models on exactly these benchmarks with the added concordance check. The abstract reports the numbers clearly enough to see BioBERT on top.

The soft spot is the scope. Both datasets are narrow—one on analgesics and liver failure, the other on tramadol and mortality—so the claim that domain pre-training gives a general advantage rests on limited ground. The paper does not show why the pattern should hold for other ADE types or larger pharmacovigilance collections.

This is for readers already working with InferBERT or similar causal setups in computational pharmacovigilance who want a data point on model choice. It is not broad enough to change practice on its own.

It deserves a serious referee. The methods are standard and the results are falsifiable, even if reviewers will likely ask for more datasets or a clearer argument on generalizability.

Referee Report

2 major / 2 minor

Summary. The paper evaluates the impact of classification model choice within the InferBERT framework for distinguishing causal adverse drug events (ADEs) from spurious correlations in pharmacovigilance. Using two benchmarks (Analgesics-induced Acute Liver Failure (AILF) and Tramadol-related Mortalities (TRAM)), it compares XGBoost, ALBERT, BioBERT, and Med-LLaMA via 5-fold cross-validation repeated 20 times, reporting accuracy, Expected Calibration Error (ECE) before/after isotonic regression, and Jaccard concordance with traditional signals (PRR, ROR, EBGM). Paired t-tests are used for significance. The central claim is that BioBERT yields the highest accuracy and strongest concordance, demonstrating that domain-specific pre-training provides a decisive advantage over simpler baselines and larger LLMs, while calibration has mixed effects.

Significance. If the empirical comparisons hold, the work provides concrete evidence favoring domain-adapted transformers over scaling or generic models for causal inference tasks in pharmacovigilance. Strengths include the multi-metric evaluation (accuracy, calibration, concordance with established signals), repeated cross-validation with statistical testing, and direct comparison against external baselines. This could inform practical model selection in computational pharmacovigilance, though the narrow benchmark scope limits immediate generalizability.

major comments (2)

[Abstract / Results] Abstract and results sections: The headline conclusion that 'domain-specific pre-training provides a clear advantage ... for computational pharmacovigilance' is load-bearing yet rests solely on performance patterns observed on the two specific datasets AILF (analgesics-liver failure) and TRAM (tramadol-mortality). No additional datasets, cross-domain experiments, or argument for representativeness are provided to support extrapolation beyond these narrow drug-event pairs.
[Methods / Experimental Setup] Methods (implied in abstract): The description of 5-fold CV repeated 20 times with paired t-tests and Jaccard concordance is given, but without explicit details on data splits, exclusion criteria for drug-event pairs, how causal terms are extracted for concordance, or handling of class imbalance, it is impossible to verify whether post-hoc modeling choices affect the reported superiority of BioBERT.

minor comments (2)

[Abstract] Abstract: The phrase 'Investing in manageable, domain-aware models is more effective ... than simply scaling model size' is a strong generalization; qualify it as holding on the evaluated tasks.
[Methods] Notation: Define 'causal terms' and the exact procedure for computing Jaccard concordance with PRR/ROR/EBGM more explicitly.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below, indicating where revisions to the manuscript are planned.

read point-by-point responses

Referee: [Abstract / Results] Abstract and results sections: The headline conclusion that 'domain-specific pre-training provides a clear advantage ... for computational pharmacovigilance' is load-bearing yet rests solely on performance patterns observed on the two specific datasets AILF (analgesics-liver failure) and TRAM (tramadol-mortality). No additional datasets, cross-domain experiments, or argument for representativeness are provided to support extrapolation beyond these narrow drug-event pairs.

Authors: We acknowledge that the empirical results are based on the AILF and TRAM benchmarks. These datasets were selected to represent distinct pharmacovigilance scenarios (liver failure signals versus mortality associations) with different drugs and outcome types. The consistent advantage of BioBERT across both supports the role of domain-specific pre-training for causal ADE detection tasks. We agree that explicit discussion of representativeness is warranted. In revision we will expand the Discussion to argue for the clinical relevance and diversity of these benchmarks and add a Limitations subsection noting that broader validation on additional datasets is needed for stronger generalizability claims. revision: partial
Referee: [Methods / Experimental Setup] Methods (implied in abstract): The description of 5-fold CV repeated 20 times with paired t-tests and Jaccard concordance is given, but without explicit details on data splits, exclusion criteria for drug-event pairs, how causal terms are extracted for concordance, or handling of class imbalance, it is impossible to verify whether post-hoc modeling choices affect the reported superiority of BioBERT.

Authors: We agree that additional methodological detail is required for full reproducibility and verification. The revised Methods section will explicitly describe: the inclusion/exclusion criteria applied to drug-event pairs, the stratification and random seeds used for the repeated 5-fold splits, the exact procedure for extracting causal terms prior to Jaccard computation, and the approach taken to class imbalance (including any weighting or resampling). These details will be moved from supplementary material into the main text where appropriate. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical model comparisons on external benchmarks

full rationale

The paper reports direct empirical results from 5-fold CV (repeated 20 times) on two fixed benchmarks (AILF, TRAM), measuring accuracy, ECE, and Jaccard overlap with independent signals (PRR/ROR/EBGM). No equations, fitted parameters renamed as predictions, self-definitional steps, or load-bearing self-citations appear. All performance claims rest on observable outcomes against external baselines, satisfying the self-contained criterion.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Review based solely on abstract; no free parameters, new entities, or complex axioms are visible. The central empirical claim rests on the representativeness of the two chosen benchmarks and the adequacy of the selected evaluation metrics.

assumptions (1)

domain assumption The AILF and TRAM datasets adequately represent the broader problem of distinguishing causal ADEs from spurious correlations
The study draws general conclusions about model selection from performance on these two benchmarks alone.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance." pith.science (2026). https://pith.science/paper/ZW57ONFU

@misc{pith2026260617113,
  author       = {Pith},
  title        = {Pith review of: The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZW57ONFU}},
  note         = {Machine review of arXiv:2606.17113}
}

read the original abstract

Distinguishing causal adverse drug events (ADEs) from spurious correlations remains a central challenge in pharmacovigilance. The InferBERT framework integrates transformer models with Do-calculus, but its success hinges on the underlying classification model. This study evaluates the impact of model choice in InferBERT, assessing whether simpler models suffice, if domain-specific pre-training helps, whether scaling to LLMs improves causal detection, and the effect of post-hoc calibration. We performed a comparative study on two benchmarks: Analgesics-induced Acute Liver Failure (AILF) and Tramadol-related Mortalities (TRAM). Four models were evaluated-XGBoost (baseline), ALBERT (original InferBERT), BioBERT (biomedical transformer), and Med-LLaMA (medical LLM)-using 5-fold cross-validation repeated over 20 runs. We measured accuracy, Expected Calibration Error (ECE) pre- and post-isotonic regression, and Jaccard concordance of causal terms with PRR, ROR, and EBGM; significance was tested with paired t-tests. BioBERT achieved the highest accuracy on both datasets, while Med-LLaMA underperformed despite its size and parameter-efficient fine-tuning. Domain-specific pre-training was decisive. Calibration improved ECE but had mixed effects on accuracy and causal discovery. BioBERT's superiority also yielded the strongest concordance with traditional pharmacovigilance signals. These results show that domain-specific pre-training provides a clear advantage over simpler baselines and larger LLMs. Investing in manageable, domain-aware models is more effective for computational pharmacovigilance than simply scaling model size.

Figures

Figures reproduced from arXiv: 2606.17113 by the authors.

**Figure 1.** Schematic overview of the framework and the experimental setup. The upper panel depicts the full causal inference framework, flowing from FAERS data extraction to causal term identification via Docalculus. The lower panel illustrates the specific experimental setup for the different models, contrasting the tabular data processing for XGBoost with the template-based sentence generation used for the transformer model… view at source ↗

**Figure 2.** Distribution of classification accuracy across 20 runs for each model. The central mark indicates the median, the box extends to the 25th and 75th percentiles, and the whiskers show the range of non-outlier data. BioBERT demonstrates consistently higher accuracy and lower variance. 3.2 The Nuanced and Often Limited Impact of Probability Calibration Next, we investigated whether post-hoc calibration reliably improves… view at source ↗

**Figure 3.** Causal trees generated by XGBoost, ALBERT, and BioBERT (from left to right in each panel). These trees are derived from a single, representative cross-validation fold to provide a qualitative illustration of the models’ inferred causal hierarchies. Note BioBERT’s identification of clinically plausible factors like ‘alcohol‘ (A) and ‘female gender‘ (B), which are missed by the other models. 4 Discussion Our systemati… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages

[1]

Muaed Alomar, Ali M Tawfiq, Nageeb Hassan, and Subish Palaian. Post marketing surveillance of suspected adverse drug reactions through spontaneous reporting: current status, challenges and the future.Therapeutic Advances in Drug Safety, 11:2042098620938595, 2020

2020
[2]

XGBoost: A Scalable Tree Boosting System

Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), pages 785–794. ACM, 2016

2016
[3]

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

YuGu,RobertTinn,HaoCheng,MichaelLucas,NaotoUsuyama,XiaodongLiu,TristanNaumann,JianfengGao, and Hoifung Poon. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans. Comput. Healthcare, 3(1), October 2021

2021
[4]

Causal inference, 2010

Miguel A Hernán and James M Robins. Causal inference, 2010

2010
[5]

Gradient boosting trees and large language models for tabular data few-shot learning.arXiv preprint arXiv:2411.04324, 2024

Carlos Huertas. Gradient boosting trees and large language models for tabular data few-shot learning.arXiv preprint arXiv:2411.04324, 2024

work page arXiv 2024
[6]

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2020

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2020

2020
[7]

The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints

Thanh-Dung Le, Ti Ti Nguyen, Vu Nguyen Ha, Symeon Chatzinotas, Philippe Jouvet, and Rita Noumeir. The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints. IEEE Access, 13:109365–109377, 2025

2025
[8]

Biobert: a pre-trained biomedical language representation model for biomedical text mining.Bioinformatics, 36(4):1234–1240, 2020

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining.Bioinformatics, 36(4):1234–1240, 2020

2020

Show all 16 references

[9]

Causal Inference in Statistics: An Overview.Statistics Surveys, 3:96–146, 01 2009

Judea Pearl. Causal Inference in Statistics: An Overview.Statistics Surveys, 3:96–146, 01 2009

2009
[10]

Estimating Expected Calibration Errors

Nicolas Posocco and Antoine Bonnefoy. "Estimating Expected Calibration Errors". In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter, editors,Artificial Neural Networks and Machine Learning – ICANN 2021, pages 139–150, Cham, 2021. Springer International Publishing

2021
[11]

Emeri Potter, Melissa Reyes, Jennifer Naples, and Gerald Dal Pan. FDA Adverse Event Reporting System (FAERS) Essentials: A Guide to Understanding, Applying, and Interpreting Adverse Event Data Reported to FAERS.Clinical Pharmacology & Therapeutics, 118(3):567–582, 2025

2025
[12]

InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance.Frontiers in Artificial Intelligence, Volume 4 - 2021, 2021

Xingqiao Wang, Xiaowei Xu, Weida Tong, Ruth Roberts, and Zhichao Liu. InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance.Frontiers in Artificial Intelligence, Volume 4 - 2021, 2021

2021
[13]

TransTab:LearningTransferableTabularTransformersAcrossTables

ZifengWangandJimengSun. TransTab:LearningTransferableTabularTransformersAcrossTables. InS.Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 2902–2915. Curran Associates, Inc., 2022

2022
[14]

Pharmacovigilance: ensuring the safe use of medicines

World Health Organization. Pharmacovigilance: ensuring the safe use of medicines. Technical report, World Health Organization, 2004

2004
[15]

Me-LLaMA: Foundation Large Language Models for Medical Applications.arXiv preprint arXiv:2402.12749, 2024

Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Xinyu Zhou, Huan He, Lucila Ohno-Machado, Yonghui Wu, Hua Xu, and Jiang Bian. Me-LLaMA: Foundation Large Language Models for Medical Applications.arX...

2024
[16]

Machine learning in causal inference: application in pharmacovigilance.Drug Safety, 45(5):459–476, 2022

Yiqing Zhao, Yue Yu, Hanyin Wang, Yikuan Li, Yu Deng, Guoqian Jiang, and Yuan Luo. Machine learning in causal inference: application in pharmacovigilance.Drug Safety, 45(5):459–476, 2022

2022

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.

[1] [1]

Muaed Alomar, Ali M Tawfiq, Nageeb Hassan, and Subish Palaian. Post marketing surveillance of suspected adverse drug reactions through spontaneous reporting: current status, challenges and the future.Therapeutic Advances in Drug Safety, 11:2042098620938595, 2020

2020

[2] [2]

XGBoost: A Scalable Tree Boosting System

Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), pages 785–794. ACM, 2016

2016

[3] [3]

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

YuGu,RobertTinn,HaoCheng,MichaelLucas,NaotoUsuyama,XiaodongLiu,TristanNaumann,JianfengGao, and Hoifung Poon. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans. Comput. Healthcare, 3(1), October 2021

2021

[4] [4]

Causal inference, 2010

Miguel A Hernán and James M Robins. Causal inference, 2010

2010

[5] [5]

Gradient boosting trees and large language models for tabular data few-shot learning.arXiv preprint arXiv:2411.04324, 2024

Carlos Huertas. Gradient boosting trees and large language models for tabular data few-shot learning.arXiv preprint arXiv:2411.04324, 2024

work page arXiv 2024

[6] [6]

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2020

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2020

2020

[7] [7]

The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints

Thanh-Dung Le, Ti Ti Nguyen, Vu Nguyen Ha, Symeon Chatzinotas, Philippe Jouvet, and Rita Noumeir. The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints. IEEE Access, 13:109365–109377, 2025

2025

[8] [8]

Biobert: a pre-trained biomedical language representation model for biomedical text mining.Bioinformatics, 36(4):1234–1240, 2020

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining.Bioinformatics, 36(4):1234–1240, 2020

2020