Conference Agenda

The two-trials rule in drug regulation requires independent statistically significant results from two pivotal trials to demonstrate drug efficacy. However, it is unclear how effect estimates from two trials should be combined to quantify the drug effect. Combination with meta-analysis can lead to situations where the two-trials rule is not satisfied, but the meta-analytic confidence interval excludes the value of no effect. Here we show how the two-trials rule and meta-analysis can be cast in the framework of combined p-value functions, where they are variants of Wilkinson's and Stouffer's combination methods, respectively. We show how compatible combined p-values, effect estimates, and confidence intervals can be obtained, and derive them in closed-form. We also investigate Edgington's, Fisher's, Pearson's, and Tippett's p-value combination methods. We find that when both trials have the same true underlying effect, all methods can consistently estimate it, although some methods show bias. When the true trial effects differ, Fisher's and Tippett's methods are asymptotically anti-conservative (converging to the more extreme effect), the two-trials rule and Pearson's method are conservative (converging to the less extreme effect), and Edgington's method and meta-analysis are balanced (converging to a weighted average). However, Edgington's method asymptotically always includes the individual effect estimates in its confidence interval while the meta-analytic confidence interval converges to a point. We conclude that all these methods may be appropriate depending on the estimand of interest.

In-vitro diagnostics (IVDs) are increasingly important in modern healthcare. The use of effective IVDs can substantially improve patient outcomes and there is opportunity for research and design innovation and investment. However, the pathway between the development of a new IVD and widespread adoption in the UK is complex and there are multiple points along the innovation pathway where bottlenecks may occur. Using the Roche Elecsys sFlt-1/PlGF ratio test to diagnose pre-eclampsia as an exemplar, we illustrate the full pathway from evidence development to adoption and highlight the challenges and barriers to widespread implementation and patient access in the UK. We will then discuss potential solutions to support more timely access to innovations.

In 2008, a Health Technology Assessment (HTA) outlined the need for accurate biomarkers to diagnose pre-eclampsia. However, it was not until 2016 that there was enough evidence for the National Institute for Health and Care Excellence (NICE) to recommend use of PlGF-based testing to rule out pre-eclampsia. Following the recommendation, there were significant implementation issues in the National Health Service due to funding and procurement barriers. Additional funding was obtained to reimburse the cost of the test, but barriers to implementation persisted until key stakeholders worked together on a national level to prioritise pathway transformation resource. A review of the 2016 NICE guidance began in 2020, and new NICE guidelines published in 2022 continue to endorse use of the PlGF-based testing for both rule-in and rule-out indications.

The evidence generation pathway for an IVD is more complex than for pharmaceuticals and should follow a dynamic, cyclical approach. However, more research is needed to inform the methods for doing so. In particular, further guidance is needed to allow for effective combination of clinical performance and clinical effectiveness data, which takes into account the full value proposition of the diagnostic technology aside from improvements in accuracy. Mandated funding following NICE approval of IVDs should be considered, and key stakeholders should collaborate to identify barriers to adoption, especially given the projected growth of the IVD market in upcoming years.

1．Introduction

Many research of AI application for healthcare have been reported. However, in order to use these techniques in practice it is necessary to get regulatory approval from the authority in each country according to the regulation or the act at first [1]. We will review the latest status so called SaMD: Software as Medical Device in regulation for AI diagnosis with Gastrointestinal endoscopy as an example.

2. Methods

We identified the SaMD with AI/ML which has been already approved both Japan and US.

Then we review the data package including the design;

• Endpoints
• Sample size
• Control group, or any comparison with humans
• Prospective or Retrospective
• Intervention or Observational

3. Result

So far thirteen SaMD have been approved by Pharmaceuticals and Medical Devices Agency in Japan as of February 14, 2025 while five SaMD have been approved by US Food and Drug Agency as of October 1, 2024. There is only one SaMD which was approved in both US and Japan.

According to the package inserts which are available at the site of Japanese Pharmaceutical and Medical Device Agency, test data were collected retrospectively and there was no prospective study conducted in all of thirteen SaMD. Performance evaluation tests were conducted while primary endpoint was sensitivity and secondary endpoint was specificity and 95% confidence intervals were reported. There are two SaMD that were evaluated in the comparison with human experts or non-experts, however the package insert for them do not show how many clinicians were attended for this performance test. Other than the above two SaMD, there were no simultaneous control group in their performance evaluation tests.

According to the summaries of 510(k) Premarket Notification which are available at the site of US Food and Drug Agency, test data includes both retrospective performance evaluation tests concerning sensitivity and specificity and prospectively randomized controlled trials which primary endpoints are detection rate in terms of superiority and positive predictive rate as non-inferiority.

4. Conclusion

There is NO universally common regulation to get approval as well as what to do in post marketing. The regulation for AI diagnosis is different in each country and there is no harmonization.

[1] Walradt T, Glissen Brown JR, Alagappan M, Lerner HP, Berzin TM.J. Regulatory considerations for artificial intelligence technologies in GI endoscopy. Gastrointest Endosc. 2020;92(4):801-806.

Introduction

Physiologically-based pharmacokinetic (PBPK) models are becoming increasingly popular for model-informed drug development (MIDD), prompting the development of several evaluation frameworks and regulatory guidance documents. This guidance is, by design, often of a general and non-specific nature: it is clear what steps should be carried out, but not necessarily how. For example, one of the necessary steps is validation, where the predictions of the model are compared to an external dataset. However, how to carry out this comparison remains unclear and is therefore the topic of this work.

Methods

We propose a validation step based on the popular Continuous Ranked Probability Score (CRPS). Contrary to the current standard of only evaluating if the model is on average correct, this metric explicitly measures how well the model captures the distribution of the observed data. The CRPS is applicable to both individual-level predictions and virtual population simulations, making it well-suited for PBPK modeling. Additionally, we demonstrate that the average CRPS for the entire dataset can be computed with only a single integral, greatly increasing computational efficiency and thereby facilitating the computation of bootstrap confidence intervals. Lastly, the average CRPS of an individual model can be compared with the average CRPS of a naïve one-dimensional model to obtain a skill score, an intuitive measure of model performance.

Results

We applied the CRPS-based validation approach to compare two PBPK models, showing that it gives much more insightful and robust validation than the current standard. Additionally, we showed that the skill score offers a clear interpretation of the obtained CRPS scores. Surprisingly, both PBPK models produced a negative score.

Conclusion

The proposed validation method offers a more quantitative and interpretable approach to PBPK model evaluation. While our focus is on PBPK models, this framework is broadly applicable to other modeling scenarios that involve virtual populations. To facilitate adoption, we provide an accessible online tool that implements the CRPS-based validation method.

Background/Introduction:

Conditional marketing authorisation (CMA) is a way of allowing early market access of new drugs addressing an unmet medical need, especially for serious or life-threatening diseases (European Union, 2006). For justifying a CMA, the benefit-risk profile has to be positive and the applicant must be able to provide comprehensive data post-authorisation for converting it to a full marketing authorization (MA). Initial attempts of basing a CMA on promising data from a single-arm trial, complemented by a randomised clinical trial post-authorisation, were notoriously difficult. A recent proposal is to plan a randomised clinical trial with interim analyses and basing the decision for CMA and full MA on an intermediate and the final primary endpoint, respectively. To control the study-wise type I error, the dual primary endpoint concept, which is essentially a Bonferroni-split of the study-wise type I error between the intermediate and final primary endpoint, is applied. We argue that the resulting statistical definition of study success is not in line with clinical and regulatory assessment of the overall trial outcome (Großhennig & Thomas, 2023).

Methods:

Consistent with European Regulatory Guidance on multiplicity issues in clinical trials (EMA, 2002), we propose to define the intermediate and final primary endpoint as co-primary – each assessed with a standard group sequential design testing procedure. We illustrate our proposal using an oncological phase III clinical trial with complete remission as intermediate and overall survival as final primary endpoint.

Results/Conclusion:

Based on the example, we demonstrate that our proposal is a valid and flexible alternative that would not increase costs in terms of sample size or timing of the interim analysis for applying a CMA but clearly improves the interpretation of the overall trial outcome.

References

• ­European Medicines Agency. Committee for proprietary medicinal products. (2002). Points to consider on multiplicity issues in clinical trials. https://www.ema.europa.eu/en/documents/scientific-guideline/points-consider-multiplicity-issues-clinical-trials_en.pdf
• ­European Union (2006, March 30). Commission regulation (EC) No 507/2006 of 29 March 2006 on the conditional marketing authorisation for medicinal products for human use falling within the scope of Regulation (EC) No 726/2004 of the European Parliament and of the Council. Official Journal of the European Union. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32006R0507
• ­Großhennig, A., Thomas, N. H., Brannath, W., & Koch, A. (2023). How to avoid concerns with the interpretation of two primary endpoints if significant superiority in one is sufficient for formal proof of efficacy. Pharmaceutical Statistics, 22(5), 836-845. https://doi.org/10.1002/pst.2314