Generative KI-Systeme, insbesondere Large Language Models (LLMs), bieten vielfältige Möglichkeiten zur Automatisierung komplexer textbasierter Aufgaben. Ihre inhärente Unsicherheit stellt jedoch eine Herausforderung für den produktiven Einsatz dar – insbesondere in sensiblen Anwendungsfeldern wie dem Gesundheitswesen. In diesem Vortrag zeigen wir am Beispiel der Techniker Krankenkasse, wie mithilfe eines sogenannten LLM-as-a-Judge-Ansatzes die Unsicherheit von KI-generierten Ausgaben abgeschätzt und für Menschen verständlich gemacht werden kann.

Dabei beurteilt ein zweites LLM kritisch die Ausgabe eines ersten Modells im Kontext von Eingabe und Aufgabe und weist auf potenzielle Probleme, Unstimmigkeiten oder Unsicherheiten hin. Wir teilen unsere praktischen Erfahrungen und beantworten unter anderem folgende Fragen:

- Wie entwickelt man ein verlässliches LLM-as-a-Judge-System?

- Wie lässt sich Unsicherheit für Anwender:innen verständlich ausdrücken?

- Wann ist der Einsatz solcher Systeme sinnvoll – in der Entwicklungsphase oder im Produktivbetrieb?

- Wie kann man die metakognitive Bewertungsaufgabe so gestalten, dass das LLM sie möglichst effizient und treffsicher lösen kann?

Darüber hinaus diskutieren wir methodische und ethische Implikationen sowie mögliche Erweiterungen für andere Einsatzgebiete.

Multistate life tables (MSLTs), or multistate survival models, have become a widely used analytical framework among medical researchers, epidemiologists, social scientists, and demographers. MSLTs can be cast in continuous time or discrete time. While the choice between the two approaches depends on the concrete research question and available data, discrete-time models have a number of appealing features: They are easy to apply; the computational cost for point estimates is typically low; and today's empirical studies are frequently based on regularly spaced longitudinal data, which naturally suggests modelling in discrete time. Up to now, explicit formulas for the covariance matrices of the outcome statistics of discrete-time multistate models (DTMS) have only been developed to a limited extent, which is why many research papers have to resort to costly bootstrap procedures.

This presentation lays out several new asymptotic inference results for DTMS, which substantially cut the computational burden and open new possibilities for the combination and presentation of model outcome statistics. First, we derive asymptotic covariance matrices for the outcome statistics of conditional and/or state expectancies, mean age at first entry, and lifetime risk. We then discuss group comparisons of these outcome measures, which require the calculation of a joint covariance matrix of two or more results. Finally, new procedures are presented for the estimation of multistate models over a partial age range, and how these subrange calculations relate to the result that is obtained from the full age range of the model. All newly derived expressions are checked against bootstrap results in order to verify correctness of results and to assess performance.

Discrete-time multistate models (DTMS) have become a widely used analytical framework among epidemiologists, social scientists, and demographers. Markov Chains with rewards (MCWR) have been shown to be a useful modelling extension to discrete-time multistate models. In this paper, we substantially improve and extend the possibilities that MCWR holds for DTMS. We make several contributions. First, we develop a system of creating and naming different rewards schemes, so-called "standard rewards". While some of these schemes are of interest in their own right, several new possibilities emerge when dividing one rewards result by another, the result of which we call "composite rewards". In total, we can define at least ten new useful outcome statistics based on MCWR that have not yet been used in the literature. Secondly, we derive expressions for asymptotic covariance matrices that are applicable for any standard rewards definition. Thirdly, we show how joint covariance matrices of any number of rewards results can be obtained, which leads to expressions for the joint covariance matrices of (any number of) composite rewards. Lastly, expressions for point estimates and covariance matrices of partial age ranges are derived. We confirm correctness of results by comparisons to simulation-based results (point estimates) and by comparisons to bootstrap-based results (covariance matrices).

Non-probability sampling remains a popular method for collecting survey data at low cost. However, statistical inference based on non-probability relies heavily on assumptions about the unknown sampling design. A common way to address this problem is to pair the non-probability sample with a probability sample, which allows for unbiased point and variance estimation.

We propose to improve the precision of estimates based on a probability sample by borrowing strength from a larger non-probability sample.

The goal is to estimate a regression model that holds for the population of interest, which can be tested using the probability sample.

The basic assumption is that the observations in the non-probability sample come from a finite mixture, some of which are drawn from the distribution in the population of interest, that is, for them the regression model holds, for others a regression model with possibly quite different parameters holds. One can borrow strength from the non-probability sample by identifying which observations can be considered as observations from the regression model that is being investigated. The essential requirement is that a probability sample is available. Only then is it possible to evaluate whether or not observations from the non-probability sample can contribute.

An EM algorithm is used to estimate the propensities that an element in the non-probability sample is part of the target population. The method is applied to the pooled sample, where we know with certainty that the elements from the probability sample are part of the target population.