Performance metrics for models designed to predict treatment effect

BMC Medical Research Methodology

Research

H. F. Lingsma¹  C. C. H. M. Maas¹  D. van Klaveren²  R. Dekker³  M. C. Hughes⁴  D. M. Kent⁴ 

[1] Department of Public Health, Erasmus University Medical Center, Doctor Molewaterplein 40, 3015 GD, Rotterdam, Netherlands;Department of Public Health, Erasmus University Medical Center, Doctor Molewaterplein 40, 3015 GD, Rotterdam, Netherlands;Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, USA;Erasmus School of Economics, Erasmus University Rotterdam, Rotterdam, Netherlands;Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, USA;

关键词: Heterogeneous treatment effect;    Prediction models;    Logistic regression;    Causal forest;   

DOI  :  10.1186/s12874-023-01974-w

 received in 2022-11-18, accepted in 2023-06-10,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundMeasuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs).MethodsSimilar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately “perturbed models” were compared to those of the data-generating model, i.e., “optimal model”. To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest.ResultsAs desired, performance metric values of “perturbed models” were consistently worse than those of the “optimal model” (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package “HTEPredictionMetrics”.ConclusionThe proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

41116_2023_38_Article_IEq269.gif

Fig. 1

41116_2023_38_Article_IEq263.gif

41116_2023_38_Article_IEq259.gif

41116_2023_38_Article_IEq251.gif

41116_2023_38_Article_IEq247.gif

Fig. 3

41116_2023_38_Article_IEq243.gif

41116_2023_38_Article_IEq235.gif

41116_2023_38_Article_IEq229.gif

41116_2023_38_Article_IEq228.gif

41116_2023_38_Article_IEq227.gif

41116_2023_38_Article_IEq226.gif

41116_2023_38_Article_IEq225.gif

41116_2023_38_Article_IEq224.gif

41116_2023_38_Article_IEq223.gif

41116_2023_38_Article_IEq222.gif

41116_2023_38_Article_IEq221.gif

41116_2023_38_Article_IEq220.gif

12974_2023_2839_Article_IEq25.gif

41116_2023_38_Article_IEq217.gif

41116_2023_38_Article_IEq211.gif

41116_2023_38_Article_IEq210.gif

41116_2023_38_Article_IEq209.gif

41116_2023_38_Article_IEq208.gif

41116_2023_38_Article_IEq195.gif

Fig. 3

41116_2023_38_Article_IEq185.gif

41116_2023_38_Article_IEq165.gif

41116_2023_38_Article_IEq164.gif

41116_2023_38_Article_IEq159.gif

Fig. 1

41116_2023_38_Article_IEq146.gif

41116_2023_38_Article_IEq145.gif

41116_2023_38_Article_IEq143.gif

41116_2023_38_Article_IEq118.gif

Fig. 1

41116_2023_38_Article_IEq78.gif

Fig. 9

Fig. 8

Fig. 1

Fig. 6

Fig. 3