Conference Agenda

Effective policies based on previous experience and a profound understanding of problems is an ideal that politicians and their staff generally strive towards. Formulating EU policy can be challenging considering the heterogeneity of problems and plausible solutions among its members states. The European Commission, which has the sole right of initiative, does not always have the necessary expertise internally, and therefore look to external actors for insights and information in the early stages of the policy process. Since 2003, Impact Assessments has been used by the Commission as a tool for ensuring the quality of legislation by summarizing the existing knowledge, showing evidence for the claims made about the problem, and formulating potential solutions. Parallel to the Commissions own analysis, an ambition to hold broad consultations before finalizing a draft proposal is another important part of the interinstitutional agreement. The Impact Assessments are part of the early stages of policy formulation to evaluate the potential impact of available policy options. However, it is unclear what kind of information reaches officials within the Commission, how this varies between policy domains and in turn how it influences policy output. Policy domains are signified by different levels of discreteness and maturity, enabling an analysis of how information is used to justify policy options in domains with distinct characteristics. Using a quantitative approach, this article gives an overview of the information referenced in Commission Impact Assessments and examines whether the nature of EU policy domains impacts what information is referenced. By examining references and citations in more than 1000 Impact Assessment spanning all policy domains over 20 years, the role of information and evidence is examined. The references in the Impact Assessments are systematically categorized. The results show that the European Commission mostly reference internal documents and reports when formulating policy options, even in more discrete policy domains. Meanwhile, peer-reviewed research articles are seldom utilized. This, in combination with the high status of evidence-based policymaking, risk creating biases in the Impact Assessments. Something that could be problematic, since these documents are frequently referenced as a baseline throughout the succeeding policy process when designing effective EU policy.

The growing understanding of the pluralization and transformation of policy advisory systems is accompanied by a rising demand for novel analytical frameworks capable of capturing and assessing their evolving impact (Christensen, 2023; Mavrot et al., 2024). Although the literature is still developing, it already offers well-defined experiments and pilot case studies that encourage application to contexts such as Italy (Bandera et al., 2024), where the evolution of the advisory system is multifaceted, ambiguous, and raises questions about its policy capacity.

This paper contributes to this stream of literature by proposing an original tool—an analytical framework —to assess the effectiveness of policy advice in the Italian context. The framework integrates recent theoretical developments and is tailored to the specific characteristics of the national case.

The paper has a twofold objective. First, it submits for discussion a theoretical and methodological framework which aims to test, through process tracing, three pairs of hypotheses concerning: the reputation of selected advisors, the use of evidence-based arguments within the advisory process, and the impact on the decision-making. Second, it showcases the results of its application to three government-level advisory committees established in Italy between 2020 and 2024. The cases were selected based on the following criteria: promotion by different governments (Conte II, Draghi, and Meloni), ministerial bodies, and policy areas; formal appointment of members; and conclusion of their advisory activity at the time of the research. The selection follows a most different systems design and aims to analyse how advisory committees operated under varying political and administrative conditions.

This research is the final phase of a broader study on Italian policy advisory system promoted in 2020 and carried out by the National School of Administration (SNA) that is the body - part of the Cabinet Office - tasked with training and research on public administration.

In this paper, we examine the use of evaluations within four different Swedish public agencies. The purpose of the comparison is to identify similarities and differences in implementation, and to contribute with an updated understanding of the varying use of evaluations within different types of administrative activities. The results indicates that the various forms of use are based on agency-specific assignments and areas of responsibility, and that evaluations are used “upwards” in the political system – in preparation for political re-prioritization and re-decisioning. Evaluations also diffuse and contribute “downwards” with new knowledge among project owners, policy specific street-level professionals, as well as within the networks in which they operate. However, national funders rarely ask what learning the evaluations result in – and the evaluation reception looks very different in different policy contexts.

Experimental policy—understood as the small-scale testing of innovative policies to evaluate their effects—has gained a lot of interest in the recent past. Policy experimentations aim at measuring interventions' impact in a real life setting, to assess their risks and impact through a robust instrumentation. A consequent body of literature has developed on the topic, achieving a high level of methodological sophistication on how to design and assess policy experimentation. However, while most scholarly attention is devoted to the effects of policy experiments, their interactions with real-world contingencies have been less subject to systematic theorization.

In this paper, we focus on the constraints that weigh on experimental policy design at the example of current pilot-trials for the regulation of cannabis in Switzerland. Since 2021, the Narcotic Act has been revised to allow cities to experiment with various experimental models of recreational cannabis regulation. In this context and provided they meet framework conditions, cities are allowed to test regulation model at a local scale, under the supervision of the federal state and for a limited period of time. These policy trials run in parallel and are compared in the frame of a policy evaluation, to identify the best policies with regard to several dimensions: youth protection, individual and public health, public safety, social justice, shrinking of the illegal market, etc. The results are meant to inform the political and public opinion for future debates on a national regulation framework. Such a cautious and evidence-based process is in line with Switzerland's experimental attitude on drug issues that led the country to innovative harm reduction policies in the context of the HIV epidemics in the eighties.

However, several factors affect the basic principles of policy experimentation as it should be in theory. First, the politicization of this topic has a major impact on policy designing: the trials adopted sub-optimal policy design for reasons of political feasibility. This has an impact on the kind of evidence produced for subsequent policy decision-making. Second, the experimental policies face an issue of political timing: the political agenda is pushing the issue at a rapid pace, with the risk of a political decision being taken before the end of the trials. This leads to produce data in a shortened frame, away from the requirements of scientific caution and hindsight. Third, there is a problem of upscaling from the current trials at the local level to the future national regulation: all stakeholders involved in the designing and implementation of the experimental policies are located at the city level, while a future regulation will grant those competences to the cantonal level. This issue has not been anticipated, and the current capitalization of knowledge and know-how at the local level is at risk of getting lost in the future. In addressing these tensions, the paper theorizes various categories of factors affecting experimental design at the policy-politics nexus.

Learning is viewed as a key goal of policy experiments. However, policy learning from experiments is not guaranteed or automatic. We still have only limited knowledge of which factors facilitate and hinder learning from experiments. Our paper seeks to address this gap in scholarly literature. In the theoretical part of our paper, we draw on the literatures of experimental policymaking, policy learning, and evidence-based policymaking to outline three key lenses for viewing instrumental policy learning from experiments: the research lens, the managerial lens, and the politics lens. Each of these lenses are characterized by sets of factors that are considered to affect policy learning from experiments.

In the empirical part of our study, we draw on 66 semi-structured interviews conducted with public officials in Finland and Estonia (2022-2023) to explore how these three lenses illuminate policy learning by examining the perceptions of policy actors who have been involved in public sector experiments. Our case selection proceeded from the following considerations. Despite being geographically close, the two countries represent different cases in terms of their historical-social background and experience with policy experimentation. Finland has had extensive experience with policy experiments - including with large-scale strategic experiments - going back several decades. In Estonia, policy experimentation has emerged only recently and has tended to be bottom-up. The experiments conducted in Estonia so far have mostly been small-scale design experiments and policy pilots.

Our comparative analysis shows that the research lens adopted by Finnish public officials tends to favour randomized-controlled trials (RCTs) over other designs, whereas in Estonia, RCTs are considered less crucial for policy learning. We find that both in Estonia and Finland, managerial factors like goal setting, explicit metrics, and collaboration are viewed as critical for ensuring learning from experiments. Our analysis indicates that the politics lens (i.e. how political factors affect learning from experience) plays a critical role in the perceptions of Finnish officials about experimental policymaking. The Finnish interviews point to various ways how electoral considerations, ideology, and political time-frames have undermined policy learning from experiments. We find that political considerations are perceived to influence the design, management and evaluation of experiments - all of which, in turn, are viewed to affect policy learning from experiments. In Estonia however, public sector experiments tend to take place under the political radar, limiting the emergence of the politics lens in the perceptions of public officials.