Programme de la conférence

In the Netherlands, the national government introduced the so-called Policy Compass (hereafter abbreviated to PC) as an instrument to guarantee and enhance the quality of law and policymaking for all ministries (Ministry of Justice and Security, 2022). This instrument consists of a website that presents the key questions that, by answering them, legislation and policy-making professionals can use to formulate new laws and/or policies. The website offers an overview of requirements for quality assurance that new laws and regulations and policies must adhere to, as well as a broad array of tools and tests to formulate answers to the key questions. Almost one and a half years after its introduction the use of and the practical experiences with the PC have been evaluated. Our evaluation design was based on a reconstruction of the policy theory (Bongers, 2023; Leeuw, 1991) behind the design and implementation of the PC. What were the intentions behind this instrument? What kind of problem(s) does this instrument needed to solve with regard to new law and policymaking? How were its use and outcomes perceived? The reconstructed policy theory was then used as a basis for data collection and analysis on behalf of the first evaluation round (the baseline measurement) of the use of the PC.

The evaluation of the PC was conducted using a mixed methods approach (Pluye, 2023; Mertens, 2018). It composed of desk research (documents and websites regarding rationales behind the introduction of the PC), a survey among legislation and policy-making professionals and their supervisors (managers) at Dutch ministries (regarding the actual use of an practical experiences with the PC) (N= 664), in-depth interviews with representatives of these professionals and supervisors (N=16), an automated analysis of the content of the executed PC-implementation for new laws and policies (using AI-based software program) and two focus group meeting with members of the coordinating implementation working group (monitoring the implementation of the PC).

Hence, by collecting data through multiple quantitative and qualitative methods we applied triangulation (Campbell et al., 2028; Patton, 1987). In this way, we are able to illuminate and substantiate our findings from multiple angles and perspectives.

Based on the data collection and analysis we concluded that the PC is known and recognized by a small majority of both legislation and policy professionals as well as their supervisors. However, actual use is still limited to almost one fifth of them. Also, the quality of the actual use is not without difficulties either. Not all key questions are equally valuated as useful and meaningful by the professionals involved, with regard to its support for ‘writing good quality laws and policies’. Having said this, in general the content of the PC – its key questions, the supporting tools and tests and the requirements for quality assurance, are evaluated positively, indicating that professionals acknowledge its practical value.

Due to the complex, diverse and necessary nature of sustainability transformation, a critical and rigorous evaluation framework is urgently needed. Evaluation is a key activity in sustainability transitions, to ensure that programmes and policies striving for a more sustainable society are meeting the targets to reduce social inequalities and to ensure development within planetary boundaries. Problematically, there is little thorough research on the topic, which is reflected in the lack of clear theoretical framing of complexity thinking for evaluation in the sustainability context. This systematic literature review explores the application of complexity theory as a critical lens for understanding and evaluating sustainability transitions across social, ecological, and technological systems.

Employing a multi-stage search strategy across 3 academic databases (Web of Science, Scopus, Greenfile), we analyzed a large number of peer-reviewed publications from 2000 to 2024. The analysis focuses on examining the role of complexity thinking in reframing traditional approaches to evaluating and assessing sustainability challenges in peer-reviewed scientific literature. Our analysis explores the potential of complexity-based perspectives for the evaluation practice, challenging linear, reductionist models of change by emphasizing emergent properties, non-linear dynamics, and adaptive interconnected systems. By mapping the theoretical and practical applications, the review synthesizes how complexity perspective has been utilized in evaluating sustainability transformation at policy, programming and local level.

The review identifies key theoretical contributions, including novel frameworks for analyzing and evaluating sustainability transformation, as well as methodological tools for carrying out complexity-informed evaluation. The findings contribute to the future designs of policy evaluations and adaptive policy frameworks. In addition, key research gaps from the literature are identified to facilitate and support the development of the evaluation of sustainability transitions. Specifically, systematic analysis of complexity-informed evaluation frameworks and approaches enhances the overall soundness of sustainability assessments and contributes to the implementation of environmental sustainability objectives as such.

Evaluation is generally understood as the assessment of the worth or value of something, e.g. a policy, that has happened or is ongoing to provide guidance for future actions. Underlying this view is an assumption that a policy that has previously been effective (or ineffective) in addressing a policy problem will continue to be so in the future. It is also common to look at evaluation as something that mainly aims to improve what is already being done. This becomes problematic in the volatile and uncertain moment in which we find ourselves where policy problems seem to grow increasingly complex, involve multiple interdependent actors and external influences that make it difficult to predict the outcomes of different solutions.

Complex policy problems require policymakers and public administrators to rethink problem definitions, adopt new approaches to problem solving, even reassess which problems should fall under the public sector’s domain. For evaluators, there may thus be a need to contribute with reasoned assessments of problems that may happen and designs of policies to meet these before an action is decided and implemented. This shift in evaluative thinking is already underway. Schwandt (2019), referencing Shakespeare’s phrase “the game is afoot” (meaning that something new and exciting is about to happen), suggests that traditional evaluation can be complemented by “post-normal evaluation” in decision-making contexts where facts are uncertain, values are contested, stakes are high, and decisions are urgent.

Schwandt is surely not alone in these reflections. Over the past decade, researchers in evaluation, policy analysis, futures studies and planning have emphasized the need for creative and strategic thinking about the future and, not the least, for cross-fertilization between these fields. Titles such as “Look to the future, evaluators” (Ruedy & Clark, 2024), “Bridging foresight and evaluation” (Gardner et al., 2024), and “Thinking outside the box?” (Considine, 2012)are a few examples that illustrate this growing trend. The European Commission’s (2023) regulatory impact assessments for future-proof legislation, and the creation of design laboratories for public sector innovation further reflect this development (Peters, 2022).

My paper builds upon these discussions. First, I briefly describe a handful of approaches that may be relevant for future-oriented evaluation (needs assessment, policy design, what’s the problem represented to be, program theory evaluation, evaluation in planning and foresight). The paper concludes by problematizing these approaches through some key evaluation issues, specifically the temporal dimension of the object of evaluation, epistemology, valuation, and the role of the evaluator.

Welfare privatisation has become an increasing trend in countries facing shifting demographic structures and mounting fiscal pressures, and China is no exception. Over the past decade, China has strived to bridge public and private welfare provision by promoting government-backed, privately-operated welfare schemes, most notably the city-based supplementary medical insurance scheme and private pension scheme. However, implementing these initiatives at the municipal level remains highly challenging due to substantial regional disparities and the lack of comprehensive national guidelines, thereby requiring local governments to develop their own localised strategies.

In addressing these challenges, policy learning is widely regarded as an effective tool for identifying feasible solutions and mitigating potential failures. Although much is known about the actors who learn and the varying lessons learned across policy contexts, much less is known about whether and how such learning contributes to policymaking. This study explores these questions in the context of China’s city-based welfare privatisation initiatives. The presentation will offer preliminary reflections from fieldwork based on elite interviews and document analysis, along with theoretical insights from Dunlop and Radaelli’s (2013) policy learning framework and the Advocacy Coalition Framework. Upon completion, the study aims to deepen understanding of policy learning in welfare policymaking, shed light on the adaptation of Western policy process frameworks in Chinese settings, and inform similar initiatives in China and beyond.

Public international organizations (PIOs), such as the World Health Organization (WHO) and the Organization for Economic Cooperation and Development (OECD), play an increasingly important role in global governance and policymaking, particularly through undertaking policy design and evaluation functions. Central to the ability of such organisations to properly undertake such functions is how well they systematically engage in policy learning, a process in which policy actors track and process knowledge about emerging policy problem, aiming to update their understandings and beliefs regarding viable solutions (Dunlop & Radaelli, 2013; Zaki, 2022). While policy learning within national public organisations is already complex, it is even more challenging and substantively different in PIOs. This can be attributed to three reasons inherent to the nature of PIOs and the context in which they operate. First, PIOs operate in complex environments with a broad range of external stakeholders, leading to exacerbated heterogeneity of actor interests and limited consensus, which complicates learning. Second, PIOs’ internal structures, often constrained by compliance mandates and special statutes, can limit their capacity for reflexivity and adaptation, fostering conditions that amplify barriers to learning, such as joint decision traps, increased affinity for bargaining, political deadlocks, the instrumentalization of knowledge, and power imbalances (Powell,2002; Putnam,1988; Zaki,2022). Third, PIOs face significant legitimacy challenges, particularly in terms of democratic deficit (Montpetit, 2009; Buchana, 2009). This renders them highly dependent on learning to achieve consensus, making the need for robust learning processes even more critical. Within these contexts, effective learning can help mitigate such challenges, whereas inadequate learning risk undermining the very foundation of PIOs and threaten their long-term viability (Tallberg et al., 2014; Kamkhaji, 2022; Zaki, 2024).

These factors have stimulated research on policy learning within PIOs, whereby literature has grown considerably over the years. However, several challenges persist: First, literature on learning in PIOs is highly fragmented and remains in need of conceptual, theoretical, methodological synthesis. Second, very little is known as the PIO understandings of policy learning, specifically their policy design and evaluation functions. Namely, how factors inherent to PIO arrangements influence learning processes and outcomes. Accordingly, this article addresses these challenges by conducting a systematic review of 118 studies on policy learning in PIOs. We are guided by the following key questions:

- What is the current conceptual, theoretical, methodological, and empirical landscape of policy learning in PIOs?

- How do PIOs contribute to policy design and evaluation through policy learning processes?

- What factors shape learning processes and outcomes in PIOs?

- What analytical frameworks have been used to assess policy learning in PIOs, and how can these be adapted to compare learning within and across PIOs, considering variations in their structures, mechanisms, political contexts, and values?

In doing so, this article makes three main contributions. First, it advances theoretical understanding of policy learning in PIOs by providing a comprehensive synthesis of the factors that shape learning processes and outcomes therein. Second, it offers a framework for analyzing learning in PIOs, and third, it develops an evidence-based agenda for future research.