evaluation-conference.de - Panel Discussion 1

Panel Discussion 1

From left to right:
Richard Hummelbrunner:Senior associate of OEAR Regionalberatung; practitioner, trainer and author.
Bob Williams:Freelance consultant, specializing in empirical field research and evaluation technique, systems theory and practice, organizational development and action research
Daniela Körppen:Researcher at the Berghof Foundation for Peace Support in Berlin
Reinhard Stockmann:Founder and director of the Center for Evaluation (CEval) at the Saarland University
Esteban Tapella:Lecturer and researcher at Programa de Estudios del Trabajo, el Ambiente y la Sociedad, National University of San Juan

On the panel, different perspectives were represented.

There is no such thing as the systemic evaluation

Reinhard Stockmann stressed that it needed to be clarified what is understood by systemic approaches in evaluation. There was an agreement among the panelists that there is no such thing as the systemic evaluation. Instead, one should talk about applying systems ideas in evaluations. Bob Williams compared a good evaluator to a thief who steals appropriate tools and methods from different fields including the systems field.

Systemic approaches and "traditional" evaluation approaches: Antithetic or complementary?

The five panelists discussed common aspects of and differences between "traditional" and rigorous evaluation approaches on the one hand and systemic approaches on the other. All panelists advocated multi-method approaches. Richard Hummelbrunner und Bob Williams stressed that systemic approaches are not always needed. They can be particularly useful in highly complex situations - but not every setting is complicated or complex. Daniela Körppen stressed that it is important to integrate systems ideas into more "traditional" evaluation approaches instead of creating the image of a completely new evaluation approach that replaces the others.

Reinhard Stockmann pointed out, however, that there are some fundamental epistemological differences between the constructivist systems thinking and evaluation approaches that are based on neo-positivist/critical rationalist assumptions. In his opinion, development programs and their environment are not as chaotic and unpredictable as complexity science suggests. He also noted that "traditional" evaluation approaches are not as naïve as some proponents of systemic thinking argue. Context and different perspectives are usually taken into consideration also in "traditional" evaluations.

Process vs. impact orientation; learning vs. accountability

Daniela Körppen and Esteban Tapella put great emphasis on the process orientation of evaluations that are based on systems thinking. Explaining the South American evaluation approach "sistematización", Esteban Tapella stressed its focus on learning and understanding processes from different perspectives rather than on providing accountability.
On the other hand, Reinhard Stockmann warned against focusing too much on processes while losing sight of the impact of development interventions. When it comes to providing accountability, he argued, rigorous methods are needed.
According to Richard Hummelbrunner, there is no contradiction between a focus on processes and a focus on outcomes and impacts. Bob Williams proposed that we rethink our notion of accountability. According to him, in complex settings accountability is an illusion and only a sense of responsibility can be created.

Cultural appropriateness of systemic thinking in partner countries

Daniela Körppen argued that in the Western discussion, one might get the impression that systems ideas are very theoretical. According to her experience, systems thinking is part of everyday life in many cultures and often easier to understand for partners than result chains. From a Latin American perspective, Esteban Tapella agreed to this.

Systemic approaches in evaluation as the latest fashion

Systemic approaches in evaluation are "fashionable". Bob Williams considered this fact to be the movement's greatest help and biggest hinderer at the same time. He pointed out that  many evaluations of poor quality are labelled "systemic" in order to get attention. He requested that systems thinking be respected as a field of knowledge and urged not just to use it to tag a fashionable name on evaluations.


Pictures & Illustrations

Man from the audience asking a question
R. Stockmann on the panel
R. Stockmann and E. Tapella on the panel
R. Hummelbrunner and B. Williams on the panel
Panelists of panel discussion 1
Visual Facilitation: Finding the right tool for a specific situation
Visual Facilitation: The perfect evaluator: A thief
Visual Facilitation: In development cooperation systems thinking is one of the pillars
Visual Facilitation: The world is not that unpredictable
Visual Facilitation: What happens beyond the result chain?