inv-prediction-modelling: 1
Clinical applications of predictions under interventions
Nan van Geloven
Leiden University Medical Center, Leiden, The Netherlands
Prognostic models (or prognostic algorithms) are increasingly used to inform medical treatment decisions. Typically, individuals with high risks of adverse outcomes are advised to start (or intensify) treatment while those at low risk are advised more conservative treatment. Regular prediction models do not always provide risks that are relevant to inform such decisions: for example, an individual may be estimated to be at low risk because similar individuals in the past received a treatment which lowered their risk. To overcome these limitations, new proposals focus on predicting outcomes under specified treatment options. These are known as counterfactual predictions or predictions under interventions. Estimating and evaluating predictions under interventions using observational data comes with additional requirements such as causal assumptions, confounding adjustment, as well as suitable data.
In this talk, I will illustrate several clinical applications where prediction under interventions were used, and contrast them to the information one could get from regular prediction models. I will start with a simple point treatment setting where confounding variables are already part of the predictor set and explain why even here regular ways of predictive model development and evaluation may not be sufficient. Following applications will build up in complexity, including settings with time-varying treatments, time-varying confounding as well as sequential (i.e. repeated) decision making. For each application, I will focus on why predictions are needed under certain (or multiple) intervention strategies and point out the additional data requirments and steps needed during analysis. Clinical applications include fertility, cardiology, transplantation and transfusion medicine.
inv-prediction-modelling: 2
Dynamic Prediction of Survival Benefit to Inform Liver Transplant Decisions in Hepatocellular Carcinoma Patients
Pedro Miranda Afonso1, Hau Liu2, Michele Molinari3, Dimitris Rizopoulos1
1Department of Biostatistics, Erasmus MC, The Netherlands; 2Starzl Transplant Institute, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania, US; 3J.C. Walter Jr Transplant Center, Houston Methodist Hospital, Houston, Texas, US
Background/Introduction
Liver transplantation (LT) is the only curative treatment for selected patients with unresectable hepatocellular carcinoma (HCC). However, due to organ scarcity, patients must often wait for a suitable graft, during which they may become ineligible due to tumour progression or clinical deterioration. A predictive model identifying patients at the highest risk of waitlist dropout and those who would benefit most from LT could improve organ allocation.
Transplant-related survival benefit, defined as the additional survival time gained from LT compared to waitlist survival, provides a comprehensive metric to guide allocation. Estimating this causal effect requires addressing the observational nature of transplant data and time-varying confounders. To address these challenges, we developed a joint model for longitudinal and time-to-event data that dynamically predicts individualised transplant-related survival benefit in HCC patients. Unlike alternative approaches, such as the G-formula, structural marginal models and targeted maximum likelihood estimation, our model makes stronger assumptions about the biomarker measurement process but remains non-parametric for competing processes like censoring and visit times.
Methods
We analysed data from 7,471 HCC patients listed in the US Scientific Registry for Transplant Recipients (SRTR) between 2012 and 2022, of whom 4,786 received a liver. We developed a Bayesian joint model to associate the pre-transplant trajectories of three well established predictors—the serum level of tumour α-fetoprotein (AFP) level, the tumour burden score (TBS), and the model for end-stage liver disease (MELD) score—with the risk of death before and after transplantation. We defined the assumptions necessary to obtain unbiased estimates of the causal effect of transplantation using observational data. Our model predicts a patient's survival probabilities with and without transplantation, which are then used to estimate liver transplant survival benefit. Dynamic updates enable real-time refinement of the predictions and identification of the patients most likely to benefit from transplantation. The model is implemented in the freely available R package JMbayes2.
Results
Our model reveals distinct forms of association between AFP, TBS, and MELD score and the risk of death before and after transplantation. It provides unbiased estimates of the causal effect of transplantation on individual survival using observational SRTR data without explicitly requiring a model for the transplant assignment mechanism.
Conclusion
This prediction model represents an advancement in optimizing liver transplant decisions, promoting fairer organ allocation, and improving overall survival for waitlisted HCC patients.
inv-prediction-modelling: 3
From prediction to treatment decision: aligning development, evaluation and monitoring
Wouter A.C. van Amsterdam
University Medical Center Utrecht, Netherlands, The
From sepsis prediction to heart-attack risk and cancer prognosis, the medical literature is full of models predicting future patient health. Many of these models are motivated by the goal of supporting clinical decisions, yet few are designed or evaluated with this goal in mind.
Prediction models are typically assessed based on their predictive accuracy, but strong predictive performance does not automatically translate into better clinical decision-making. To truly optimize patient outcomes, we must align model development, evaluation, and monitoring with the goal of informing treatment decisions.
In this talk, I will outline a framework for this alignment, discussing key pitfalls in traditional predictive model evaluation and how to assess real-world decision impact. I will also address the challenge of monitoring predictive models in clinical practice—an area where EMA and FDA regulations demand action but offer little guidance. Standard metrics like discrimination and calibration can be misleading when applied to deployed models, particularly when not recognizing the effect of the deployed model on treatment decisions and patient outcomes.
Finally, I will discuss prediction-under-intervention models—models designed to estimate patient outcomes under different treatment scenarios. These models directly connect predictions to treatment decisions, making evaluation and monitoring conceptually straightforward. Though conceptually appealing, these models come with their own challenges—both methodological and practical.
Mapping out approaches to predictive model evaluation and monitoring—and addressing the lack of clear guidance—will be essential to ensuring that predictive models truly support clinical decision-making.
|
