DESIGNING & EVALUATING SYSTEM CHANGE - Dr David Rees Founding Partner Dr Sarah Appleton Director
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
DESIGNING &
EVALUATING
SYSTEM CHANGE
Dr David Rees Dr Sarah Appleton
Founding Partner Director
david.rees@synergia.co.nz sarah.appleton@synergia.co.nz
+64 21 800 266 +64 21 302 276
Level 2, 318 Lambton Quay,
Wellington 6011
AGENDA
• Key Concepts in Systems Thinking
• Contribution vs Attribution
• Designing & Evaluating System Change
“While programmatic interventions help people beat the odds,
systemic interventions change their odds.”
Karen Pittman (2015)
CEO of the Forum on Youth Investment
2
Systems Tribes
CRITICAL SYSTEM HEURISTICS
SOFT SYSTEMS METHODOLOGY
COMPLEX SYSTEMS
VIABLE SYSTEMS
SYSTEMS THINKING
AGENT BASED MODELLING
COMPLEX ADAPTIVE SYSTEMS
3
Key Concepts
Organisation
Feedback
Time
Structure
Center for Ecoliteracy
https://www.ecoliteracy.org/article/teaching-children-see-systems-all-around-us
6
Photo by Greg Nunes
Feedback
Take Account of Time
The Economist
December 200680 Million Years
The 80 million year
perspective shows quite
clearly that the earth is
cooling
Peter Barrett
Prof. Geology, Victoria University
Policy Quarterly, 2006400,000 Years
The 400,000 year
perspective shows a
period of climatic
instability over cycles
of around 100,000
years.
Peter Barrett
Prof. Geology, Victoria University
Policy Quarterly, 20061,000 Years
The 1000 year
perspective shows a
long period of stability
up until the latter years
of the 20th century.
Peter Barrett
Prof. Geology, Victoria University
Policy Quarterly, 2006Structure
EVENTS
Increasing Leverage
PATTERNS
STRUCTURE
MENTAL MODELS
VALUESMoving from Behavior to Structure:
(Modelling Accumulations & Feedback Responses)
The Event The Pattern The Structure
Photo by Natasha Kapur
15Is the glass half-full or half-empty?
The Static View The Dynamic View
Depends if you are an Depends if you are ….
optimist or pessimist
pouring drinking
PESSIMIST
OPTIMIST
Photo by Thomas Lipke Photo by Natasha Kapur Photo by Hayes Potter
16Model Structure
17Model Equations
Current_Level_in_the_Glass(t) = Current_Level_in_the_Glass(t - dt) + (beer_flow) * dt
INIT Current_Level_in_the_Glass = 0
UNITS: ml
INFLOWS:
beer_flow = maximum_flow*actual_flow
UNITS: ml/seconds
actual_flow = MAX(0, MIN(flow_decision, perceived_gap/maximum_flow))
desired_level_in_glass = 60
UNITS: ml
flow_decision = 1
FULL_GLASS = 100
gap = desired_level_in_glass-Current_Level_in_the_Glass
maximum_flow = 10
UNITS: ml/sec
perceived_gap = DELAY(gap, time_to_perceive_gap, 0)
time_to_perceive_gap = 1
UNITS: sec
18Contribution vs Attribution
19The challenge of
designing and
evaluating system
level change is that
there is unlikely to be
one single factor that
dominates.
2016-new-zealand-innovation-awards-
excellence-social-innovation-innovation-education-training-development-
manaiakalani-education-trust 20Contribution v Attribution
Attribution Contribution
Direct attribution, in the positivist sense that The challenge therefore becomes one of
“if some factor X occurred - such as an providing a logical argument, supported by
intervention - then there would be the sufficient evidence, that the intervention
observed result Y” is not possible in complex made an important contribution to the
social situations, which characterise many of observed results.
the issues of concern
21Evaluating Complex Systems
Context:
• Multi-site programmes
• Competing interests amongst stakeholders
• Demands from programme managers and funders
• Limited resources
The Challenge
• Balancing these pressures in creating a successful resolution is difficult
• It can be difficult untangling the different web of interactions to understand WHY
the programmes worked, or not
• Therefore, evaluators need to tools that help them to map the web of interactions
to examine the context within which programmes operateSystem Change: Key Questions
i. Is there a reasoned theory of change for the intervention
ii. Have the activities of the intervention been implemented as set
out in the theory of change
iii. Is the theory of change supported by and confirmed by evidence
of observed results and underlying assumptions
iv. Have other influencing factors been assessed and either shown
not to have made a significant contribution, or their relative role
in contributing to the desired result has been recognised.
23Designing & Evaluating System
Change
24Step 1: Describe the ‘Reference Mode’
Scale
Feared Path
Historical path
Preferred Path
2016 2017 2018 2019 2020 2021 2022 Time
today
25Step 1: Describe the ‘Reference Mode’
Scale
2016 2017 2018 2019 2020 2021 2022 Time
today
2627
Step 2: Develop the Theory of Change
28Evaluators Understanding of ‘Theory of Change’
“A ‘theory of change’ explains how activities are
understood to produce a series of results that
contribute to achieving the final intended
impacts. It can be developed for any level of
intervention – an event, a project, a programme,
a policy, a strategy or an organization.”
Rogers, P., (2014), Theory of Change, UNICEF. Retrieved
from:http://devinfolive.info/impact_evaluation/img/download
s/Theory_of_Change_ENG.pdf
29Programme Theory of Change: Absenteeism
30System Theory of Absenteeism 31
System Theory of Absenteeism 32
System Theory of Absenteeism 33
System Theory of Absenteeism 34
System Theory of Absenteeism 35
System Theory of Absenteeism 36
System Theory of Absenteeism 37
System Theory of Absenteeism 38
System Theory of Absenteeism 39
FAMILY TEACHERS
SCHOOL
System Theory of
Absenteeism 40A Theory of Wellbeing in Schools
1 Create a new life-skill class/seminar series
2 Provide a course to help Improve peoples’
time management
3 Build a nurturing school environment
5 6
4 Change the canteen menu
5 Develop activities to support positive relationships 3 4
2
6 Change canteen business model
7 Change nutrition guidelines for school events 7
1
Mahurangi College
Version: 28 June 2018
41Step 3: Test the Theory of Change
Evidence:
• the observed results,
• each of the links in the CLD,
• other influencing factors, and
• rival explanations
Simulation:
• explore ‘what if’,
• test assumptions,
• explore change over time
• sensitivity analysis
42Testing Theories with Simulation
43Not Every Problem Needs a Hammer
You will need other tools if the project you are designing/evaluating….. • is embedded in social settings and therefore subject to numerous exogenous factors that influence the targeted outcomes • aims to change behaviour • will involve feedback loops that generate unintended activities and strategies • may involve emergent outcomes
You will need other tools if the project you are designing/evaluating….. • does not allow for experimenting, either with the implementation of the interventions as a whole, or with other influencing factors • will be made up of a number of multiple project-level interventions that will be implemented over time • will involve multiple levels of national, regional and local organisations.
The very simple key point I am making is
that you need to have a range of methods
and tools, because if you only have a
hammer you’re going to find an awful lot of
nails.THANK YOU
48You can also read
