Conference Agenda

Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).

Session Overview
Evaluation - Different perspectives
Thursday, 12/Sep/2019:
3:50pm - 5:30pm

Session Chair: Theres Paulsen
Location: Europa
Conference Centre Wallenberg

Show help for 'Increase or decrease the abstract text size'

Evaluative and enabling infrastructures: Supporting the ability of urban co-production processes to contribute to societal change

David Simon1, Henrietta Palmer1, Merritt Polk2

1Chalmers University of Technology,; 2Gothenburg University

As widely attested in the literature, the evaluation of co-production is complex and unsuited to the use of conventional quality, monitoring and evaluation indicators. This reflects the uncertainties, co-contributory factors and time lags involved, particularly when seeking to assess institutional and wider societal effects of multi-stakeholder participatory processes and deliberative fora. The most widely assessed effects include the immediate or direct outcomes of a project or activity (so-called first order effects) while wider societal or third order impacts continue to be the most difficult to capture and consequently, the least well studied. Because of this difficulty, the intermediate, second order, effects of organisational transformation and policy implementation constitute a growing challenge for evaluation. This is our focus here. After 10 years of transdisciplinary co-productive research practice, Mistra Urban Futures, as an interstitial research space bridging academia and practice, has reached a phase where some of these effects are becoming distinguishable. However, they remain patchy and uneven. Accordingly, we discuss the prerequisites for co-production practitioners to engage their respective organisations in transitional and incremental experimentation in order to achieve relevant institutional changes. This requires enabling infrastructures that support training, facilitation and the creation of ‘safe’ spaces to promote trust and legitimacy. These are needed to underpin the long-lasting personal and organisational commitments which are crucial to achieve transformative organisational effects.

Simon-Evaluative and enabling infrastructures-235.docx

What do review panels do when they take funding decisions about transdisciplinary research?

Antonietta Di Giulio, Rico Defila

University of Basel,

The paper will present results from the project "Civil society and research for sustainable development: demanding and fostering transdisciplinarity" (ZiFoNE). The project is the accompanying research project to the funding program "Research for sustainable development" (WfNE) in the "Vorab, Lower Saxony", managed by the Volkswagen Foundation. The project is funded by the Ministry of Science and Culture of Lower Saxony. The project leaders are Rico Defila, Antonietta Di Giulio, and Claudia Binder (EPPFL Lausanne, Switzerland). The project investigates three questions, one of them is devoted to the appropriate evaluation of transdisciplinary research. The results that will be presented focus on this research question.

In this research program, we had the opportunity of observing the discussions and funding decisions by an interdisciplinary review panel for three periods of funding. The review panel had, in a first step, to decide which of the consortia that had submitted a project proposal should be invited to present their projects. These presentations and the questioning by the review panel took place in public. In a second step, and after the presentations, the panel decided which projects would get funding. In its funding decisions, the panel had to consider the transdisciplinary quality of the projects as well as their potential contributions to sustainability. The criteria the panel applied had been collaboratively developed by the funding agency and the review panel (we accompanied this process). The majority of the members of the review panel remained in the panel for four years. We observed, tape recorded and transcribed the entire discussions of the review panel.

Taking funding decisions in such a panel is a collaborative act of decision-making. This act requires mutual understanding and learning, it requires developing a common understanding of both the criteria used to assess the project proposals and of how these criteria should be weighted, and it requires building trust in the other members of the panel. The analysis of such a process can focus on the social process and the (interdisciplinary) group dynamics, that is, on how the individuals interact and organise their collaboration. We pursue another approach: We perceive the collaborative decision-making by the review panel as a collaborative speech act, that is, as an act that is enacted by speech. This allows us to focus on the cognitive structure of the deliberations.

In the paper, we will present the results of our data analysis. In analysing the data, we adopt a linguistic approach (interactional linguistics), a novel approach in this field of research, that is, we analyse the speech acts that are performed by the reviewers, how these acts interact and build upon each other, and how they lead to the final decisions by the panel. In the discussion, we will draw conclusion with a view to assessing and funding transdisciplinary research for societal transformation.

Di Giulio-What do review panels do when they take funding decisions-201.docx

Uncovering the perspective of participants of a transdisciplinary dialogue – The case of Tertúlias do Montado, Alentejo, Portugal

Mª Helena Guimarães1, Cristian Pohl2

1Institute of Mediterranean Agricultural and Environmental Sciences (ICAAM) Évora University,; 2Transdisciplinarity Lab USYS TdLab, Department of Environmental Systems Science, ETH Zurich

Understanding the perspectives about transdisciplinarity of the participants of such processes is not a common field of study. Most of the literature focus on the evaluation of a transdisciplinary project by external evaluators either at the stages of applying for funding or at the stage of understanding the project impact. The present study is related to both evaluation efforts but the focus is distinct. Our goal is to attain the perspectives of the participants of a transdisciplinary dialogue about the process itself. Therefore, participants are asked to undertake an evaluation of what they have experience. The evaluation is done in the form of a questionnaire built taking into consideration the dedicated literature and adapting it to the needs of the present study. The questionnaire was developed by telephone and assisted by an interviewer that recorded the replies attained. The case study focus on a transdisciplinary dialogued, named Tertúlias do Montado: This case study is not a transdisciplinary research project but an ongoing dialogue between researchers and other societal members around a common problem of interest. The Tertúlias do Montado started in 2016 and after 3 years of initiative it is important to understand how the process is understood by those that are participating on it. The questionnaire will be applied in April 2019 to the full range of 153 participants in the 18 sessions that occurred so far. This implies that the questionnaire will be reply by frequent participants and also by those that so far participated only once and in different years. One top of this data, we statistically explore the outcomes of 250 evaluation questionnaires attained at the end of each session.

Guimarães-Uncovering the perspective of participants of a transdisciplinary dialogue – The case of Tertúlias d.docx

Transforming complex policy evaluation through co-production: innovating for change

Amy Louise Proctor1, Adam Hejnowicz2, Frances Rowe1, Jeremy Phillipson1

1Newcastle University,; 2University of York

Policies designed to intervene across water, energy, food and environment sectors are not only intervening in a complex system but are themselves also highly complex and part of a wider complex policy landscape. This poses great difficulties for evaluating the effectiveness, impacts and successes of policies, because both the systems and the policies are constantly open to change and are changing. Very often they comprise multiple interacting elements operating across-scales in a highly dynamic way with multiple feedbacks and non-linear behaviours. This dynamism together with the capacity for newly emergent properties to arise overtime means that the influence of polices is often highly contingent and uncertain, and so their impacts and outcomes can often be unpredictable. So how might you transform the practice of policy evaluation to make it fit for a complex world? Drawing upon a suite of UK policy case studies from across the water-energy-food-environment nexus, we report on work conducted by the Centre for the Evaluation of Complexity Across the Nexus (CECAN) which implemented a transdisciplinary mode of working between academics, policy-makers and evaluation practitioners in order to foster a new praxis designed to lead towards more complexity appropriate evaluation approaches. This paper reflects on the processes of co-production that underpinned this work including capacity building and expertise exchange. Co-production is complex and non-linear and requires all parties to adjust to different ways of working including navigating organisational cultures and boundaries. The research highlights how this social innovation was essential for driving the methodological innovation in evaluation approaches and methods.

Proctor-Transforming complex policy evaluation through co-production-140.docx

Contact and Legal Notice · Contact Address:
Privacy Statement · Conference: ITD 2019
Conference Software - ConfTool Pro 2.6.129
© 2001 - 2019 by Dr. H. Weinreich, Hamburg, Germany