The role played by evidence in the policy-making process is hotly disputed. The recent COVID-19 crisis led to further debate about different types of evidence and how they are used. There is no general agreement over how evidence is defined and understood. What might count as evidence for one policy actor may not be the same for another. Different contexts might also lead to divergent definitions.

This research contributes to the literature on evidence use by examining how evidence is defined and understood by policy actors at the local and national levels in a country. Using Q methodology, a mixed method developed in psychology, we survey 66 policy actors from government, parliament, academia, local government, health organisations, the voluntary sector, and the wider policy community at the local and national level in Wales. Participants were asked to rank forty statements selected from a wide range of sources on what evidence is and its role in policy making.

The findings allow us to develop five profiles which describe attitudes towards evidence – Evidence Based Policy Making (EBPM) Idealist, Pragmatist, Political, Practical, and Inclusive. The results demonstrate how different cultures (including informal ways of working and patterns of behaviour) play a role in determining how evidence is perceived and mobilised in the policy process. We find that different factors – career paths, time in a role, extent of academic training, and policy area – influence how evidence is defined and mobilised at different policy making levels.

The profiles which emerge from local and national policy communities in Wales are different with a significantly greater leaning towards a more practical and inclusive view of evidence at the local level. At this level, there is stronger support for evidence being anything that helps to draw a rich picture of an issue, such as individual stories, than perceptions of evidence at the national level. There is also less emphasis on EBPM (such as the use of Randomised Controlled Trials (RCTs) and hierarchies of evidence) at the local level. The differences found between the local and national levels of government suggest that national government should pay better attention to how evidence is understood at the local level, the risk being of an implementation and policy gap.

The results show that what counts as evidence depends on the context, the nature of the research question, and the evidence available to answer that question. We also find that a policy community, and even a single organisation such as government, can harbour multiple and sometimes contradictory cultures of evidence.

In the broader literature in political science as well as public administration and policy, we are consistently inundated with research programs and questions about the role of culture and institutions in the behavior of social actors and how these shapes the ways by which we understand society and the competing interests it embodies. But in the quest to understand what these terms represent, we also tend to see them through various discordant and oftentimes conflicting conceptual frames. In consideration of the importance of these two phenomena in understanding the building blocks of social systems, processes, and prevailing normative structures, it is important (for both practical and epistemological reasons) to explore the extent to which culture and institutions are relevant (or even efficacious) in matters dealing with government effectiveness, and how they evolve as social processes whose manifest functions can be measured or ascertained to a reasonable degree.

It is also important to note that the administrative enterprise takes place in a highly fluid and complex social system, in which many variables may pose different sets of challenges and obligations on both the administrative agenda and the mechanisms by which administrative processes are carried out. For this reason alone, it would make sense to argue that some administrative failures may not be because public administrators or bureaucrats have failed in their duties or are ill-equipped to deal with their assigned responsibilities; rather it could be because the larger environment within which public administration takes place is very complex, uncertain, and lack discrete indicators that can accurately be used to make a credible judgement between success and failure. In any effort to find ways of adapting to, or at least, managing this complexity, it is therefore important that we understand the relational properties that characterize the phenomenon of interest and in what ways it could be relevant in shaping specific policy and administrative outcomes.

The aim of this chapter is to do four things:

 Provide a conceptual definition of culture and institutionalization in the context of the public sector.

 Provide a systematic research design that illustrates essential qualitative and quantitative approaches that can be used in future research in public administration and public policy, particularly as it concerns the measurement of abstract and complex phenomena such as culture and institutionalization.

 Demonstrate how abstract non-empirical phenomena can be reduced to scale as measurable constructs for making causal estimations and predications in public administration research.

Globally, jurisdictions responded with varying degrees of coordination and acumen to the COVID-19 pandemic. Alike, there is a myriad of policies designed to address global warming. Immigration and refugee policies vary too, despite similar inputs, politics, administrative or other constraints. While some studies have touched on the affect of the cultural bias of social units on policy making, the authors are not aware of any social sciences study addressing the affect of cultural bias as an independent variable, on the process of policy analysis as a dependent variable. With policy analysis being a widely adopted process aiming to extend evidence-based reliable and transparent policy advice in a world of complex policy problems, such a study is more pertinent than ever.

A recent study by Drake and Geva-May applies for the first time, Geva-May’s Model of Policy Analysis by Cultural Bias, which initially, appeared in a Special Issue of the Journal of Comparative Policy analysis Research & Practice (2002, vol.4, no. 3), titled “Cultural Theory’s Gift for Policy Analysis.” The findings from this study have established at a high degree of significance, that there are major causal relationships between the contextual cultural bias of a social unit and the degree of utilization, or lack thereof, of policy analysis methods. This causal evidence moves the dialogue beyond typologies of policy analysts (Jenkins-Smith, 1982; Mayer Van Daalen & Bots, 2001; Meltsner, 1982). The study is based on a comparative multi case research design and the objects of analysis are Canadian provincial and territorial governments-level social units where policy analysis takes place, i.e., the cultural bias context and the analysts performing policy analyses, whose procedures are assumingly affected.

Beyond ascertaining the causal relationship between types of cultural bias and degrees of utilization of policy analysis methods, and which methods (approaches, strategies or techniques) are more or less susceptible, another key contribution of this study includes the establishment of a reliable Battery of Tools, sensitive enough to identify and assess dimensions of cultural bias (as per Dame Mary Douglas’s Grid/Group-Cultural Theory) and degrees of use of policy analysis methods (as aligned with the “Stages Approach” for Policy Analysis (Lindbloom, 1946 ,de Leon 1988, Geva-May 2023, Dunn, 2015; Geva-May & Wildavsky, 1997; MacRae & Whittington, 1997; Majone & Quade, 1980).

A further contribution concerns the use of the research design operationalizing the Geva-May Model in delivering a strategic, diagnostic barometer capable to assess cultural bias per core systematic methods of policy analysis. This barometer has implications for further studies concerning cultural bias and other phases of policy making.