Chameleonic knowledge: a study of ex ante analysis in large infrastructure policy processes
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Chameleonic knowledge: a study of ex ante analysis in
large infrastructure policy processes
Lars Dorren*
Politics and Public Governance, University of Antwerp and Leiden University
l.dorren@fgga.leidenuniv.nl
Wouter Van Dooren
Politics and Public Governance, University of Antwerp
wouter.vandooren@uantwerpen.be
Citation: Dorren, L, and Van Dooren, W. (2021). Chameleonic knowledge: a study of ex ante
analysis in large infrastructure policy processes. Policy Sciences, 1–24.
Abstract
Using ex ante analysis to predict policy outcomes is common practice in the world of infra- structure
planning. However, accounts of its uses and merits vary widely. Advisory agencies and government think
tanks advocate this practice to prevent cost overruns, short-term decision-making and suboptimal choices.
Academic studies on knowledge use, on the other hand, are critical of how knowledge can be used in decision
making. Research has found that analyses often have no impact at all on decision outcomes or are mainly
conducted to provide decision makers with the confidence to decide rather than with objective facts. In this
paper, we use an ethnographic research design to understand how it is possible that the use of ex ante analysis
can be depicted in such contradictory ways. We suggest that the substantive content of ex ante analysis plays
a limited role in understanding its depictions and uses. Instead, it is the process of conducting an ex ante
analysis itself that unfolds in such a manner that the analysis can be interpreted and used in many different
and seemingly contradictory ways. In policy processes, ex ante analysis is like a chameleon, figuratively
changing its appearance based on its environment.
I. Introduction ex ante analyses can improve decision making
by providing hard evidence on costs, benefits,
L
arge infrastructure projects are often con- and effects.
tested for being over budget or over time Policy research on knowledge use, however,
because, or being a product of politicians’ tells a different story. For example, knowledge
‘monument complex’ (Flyvbjerg et al., 2003; ends up not being used or not used for their
Hall, 1980; Leijten, 2017; Wegrich et al., 2016). intended purpose (Feldman, 1989; Weiss,1979).
In response, international organizations and If it is used, it is because it fit the preconceived
govern- ment think tanks have suggested that positions of dominant actors rather than be-
decision making should be based on the results cause of their evidential value (Stevens, 2007,
of ex ante analyses, such as cost–benefit anal- 2011). Moreover, policy research has shown
yses and environmental effect studies (Minis- how relying on scientific knowledge tends to
terie van Infrastructuur en Milieu, 2016; OECD, depoliticize policy choices that are essentially
2015a, 2015b; World Bank, 2014a, 2014b; World political. Depoliticization occurs, for instance,
Economic Forum, 2012). The expectation is that when studies demarcate which project dimen-
sions are up for discussion and which dimen-
* Corresponding author sions are out of bounds (Flinders Wood, 2015;
1Dorren & Van Dooren • Chameleonic knowledge • April 2021
Jasanoff, 1990). From this perspective, studies section first describes how the practice of ex
such as the ex ante analysis appear to be at best ante analysis use does indeed reflect many of
a ‘boundary object’ (Gieryn, 1995) or a ‘useful the theoretical perspectives mentioned in our
myth’ (Boswell, 2017) to overcome policy con- theoretical framework. This section goes on to
troversy. Moreover, they are thought to often explain the characteristics of ex ante analysis
make policy processes messier instead of pro- that allow it to be so ‘chameleonic.’ We found
viding the promised clarity; it can introduce that ex ante analysis is able to be chameleonic
complex technical procedures that ultimately because of three main characteristics: (1) it em-
distract from the political core of the policy ploys methods that make an analysis’ outcome
process (Parsons, 2002). In some cases, ex ante relative to its inputs, while its outcomes seem
analysis may even spark the very policy con- very robust and undisputable, (2) there is no
flicts it is trying to prevent (Wolf Van Dooren, definitive authority deciding which interpreta-
2017), or result in a situation in which actors tion of an ex ante analysis is correct, (3) trust in
keep producing analyses as counterarguments ex ante analysis is based on a varied set of cri-
to the analyses by other actors, resulting in a teria. These three characteristics allow ex ante
‘dialogue of the deaf’ where actors no longer analysis to facilitate a wide variety of discus-
listen to each other’s actual arguments (van sions without there being a definitive arbiter
Nispen, 2003). deciding which interpretation of the analysis
In other words, policy advice and policy re- is correct. Because trust in ex ante analysis is
search paint widely varying pictures of analy- based on a number of different criteria, trust
sis use. On top of that, policy theories of anal- in analysis is generally high, and people are
ysis use differ to a great extent among each unlikely to set its results aside easily. As such,
other. The aim of this paper is to explain how ex ante analysis can be used in many different
knowledge us can be depicted in many seem- ways, while continuing to appear credible, as
ingly contradictory ways. The wide variety is reflected in the variance among theories on
of conceptions of knowledge use could mean knowledge use.
one of two things: first, it could mean that
some of these conceptions are incorrect, and II. Knowledge in policy processes
second, it could mean that pieces of knowledge
have certain properties that allow it to facilitate Knowledge use has been studied extensively
many different uses simultaneously. This pa- in public administration, policy studies, and
per explores this question by studying ex ante organization studies. This has led to a plu-
analysis as a popular form of knowledge in rality of perspectives, which oftentimes offer
policy processes. conflicting and sometimes even diametrically
In the first section of this paper, we show opposed views of knowledge use. In this sec-
the wide range of theories of analysis use by tion, we group theories of knowledge use in
drawing examples from three key perspectives three prominent perspectives. Based on the
in the academic debate on the uses of knowl- work of Weiss (1979), we distinguish between
edge in policy processes. Then, we describe linear perspectives, nonlinear perspectives, and
how we employed ethnographic methods to ex- critical perspectives.
plain how ex ante analysis are able to reflect all
these different theories of use. We immersed
i. Linear perspectives
ourselves in the preparatory processes of three
large infrastructure projects: two in Flanders Linear perspectives perceive the relationship
and one in the Netherlands, during which we between study outcomes and the policy pro-
paid particular attention to the use of ex ante cess as direct. In the linear perspective, the re-
analysis—a form of analyses which plays a sults of a study can be directly implemented in
central role in these processes. The ‘Results’ the policy process. Weiss (1979) distinguishes
2Dorren & Van Dooren • Chameleonic knowledge • April 2021
between two linear approaches: the knowledge- lem. The second model Weiss discusses is the
driven approach and the problem-solving ap- political model, which assumes that knowledge
proach. The knowledge-driven approach as- is only used when it aligns with a political pur-
sumes that the mere presence of knowledge pose. In this model, a decision maker will only
will lead to its use. In the problem- solv- refer to the results of an ex ante analysis if it
ing approach, knowledge is produced in re- supports their preferred decision. The polit-
sponse to a specific problem. The problem ical model is also employed by (Huff, 1991)
drives the research question, and the research and Barker and Guy Peters (1993a), who ana-
shows how the problem can be solved. The lyze how facts are framed and employed in a
problem-solving approach to the use of evi- power struggle between different actors with
dence in infrastructure decision-making pro- competing goals in such a way that each actor
cesses is prevalent among advisory reports (An- achieves their goals. Barker and Guy Peters
dres et al., 2015; Andres et al., 2007; Brown et (1993b), for instance, have edited a set of chap-
al., 2006; European Conference of Ministers ters that describe how scientific expertise is
of Transport, 2005; OECD, 2012, 2015a, 2015b; used to legitimize policy. In this case, what fits
World Bank, 2014a, 2014b; World Economic Fo- the policy goal appears to be more important
rum, 2012), government guidelines (‘Decreet than the quality of the science and the consen-
betreffende complexe projecten,’ 08-27-2014; In- sus among scientists (Topf, 1993). Finally, the
frastructure and Projects Author- ity, 2017; Min- third model is the tactical model, which can be
isterie van Infrastructuur en Milieu, 2016), and applied to situations where research is being
management handbooks (Klaassen Hakvoort, used for reasons that have ‘little relation to the
2015; Priemus van Wee, 2013; Priemus et al., substance of the research’ (Weiss, 1979, p. 429).
2008; Sowden et al., 2011; Taylor, 1947; We- In this model, research can be used to do things
grich et al., 2016). Handbooks typically discuss such as delay action or deflect criticism by us-
knowledge as a means to an end, the ‘end’ ing a research outcome as the reasoning be-
being an optimal decision. By directly show- hind a policy decision. Boswell (2009) presents
ing the solution that is allegedly best, ex ante what appears to be a variation on Weiss’ tacti-
analysis promises to take over politicians’ re- cal and political models, where policy makers
sponsibility for choosing an alternative. and politicians refer to research to substantiate
policy preferences or to steer a discussion in
a particular way. However, Boswell also finds
ii. Nonlinear perspectives that policy makers actually were reluctant rely
too much on scientific research in their decision
Many policy makers do not appear to believe
making, because of the epistemic uncertainties
that the linear perspective is an adequate rep-
that come with a scientific study. As such, the
resentation of the practice of policy making
role of knowledge simultaneously is substan-
(Dorren, 2018; Mouter, 2014, 2016). In a self-
tial and marginal.
criticism, Weiss (1979) presents a second group
of perspectives which describe the relation- Weiss’ last and arguably most referred to
ship between facts and policy processes as any- model is called the enlightenment model. This
thing but linear. Weiss (1979) distinguishes model holds that research does not directly in-
between three nonlinear perspective models. fluence policies; rather, it ‘diffuses circuitously
First, the interactive model argues that knowl- through various channels such as professional
edge is used in a complex arena of interactions journals, the mass media, conversations with
between many different actors among which colleagues and over time the variables it deals
knowledge-producing scientists are only one with and the generalizations it offers provide
of many. Research informs policy, but research decision makers with ways of making sense out
findings seldom produce conclusive evidence of a complex world’ (Weiss, 1979, p. 430). In-
that points toward the right solution to a prob- stead of influencing decisions directly, research
3Dorren & Van Dooren • Chameleonic knowledge • April 2021
indirectly affects the frame of reference that whether a certain punishment is fair could be
policy makers use to make decisions. While it reframed into a debate about the societal costs
is often regarded as most accurately represent- of that punishment. Stevens also notes that
ing the practice of policy making (Marra, 2000), these evidence-based narratives serve the es-
the enlightenment model lacks true explana- sential function of ‘reduc[ing] the role of un-
tory capacity. It merely states that research certainty as a barrier to action’ (Stevens, 2011,
reaches policy makers a variety of ways, after p. 234). Policy makers need to make decisions,
which some of it ends up being used and some but they are faced with uncertainty and lack
of it does not (Stevens, 2007). The common of predictability. Evidence builds confidence.
observation that this model is the most repre- The idea of linear perspectives of knowledge
sentative of policy making in practice demon- use is therefore what Boswell (2017) calls a
strates the challenge of moving beyond superfi- ‘useful myth.’ Even though research does not
cial description without also delving deep into present policy makers with objective truths, it
the specifics of the case studied. The represen- is still useful because it increases confidence
tativeness of the enlightenment model shows and brings people together because it is based
that the ways knowledge influences policy pro- on principles most process participants will
cesses is largely case by case. It is likely that, support (Boswell, 2017).
for this reason, other nonlinear perspectives on
Other critical perspectives describe the role
knowledge use (such as Cairney (2017, 2018);
of knowledge in policy processes as a continu-
Head (2008); Marmot (2004); Mulgan (2005);
ous battle to determine the boundary between
Strassheim and Kettunen (2014); Young et al.
science and politics. Whether something is
(2002)) do not replace old typologies, but rather
labeled as science or not determines who is
they add to them.
able to discuss it (Jasanoff, 1990, p. 236). In
order for a critique of something scientific to
iii. Critical perspectives be taken seriously, it generally has to come
from a member of the scientific community.
The critical perspective is another way to un- However, the boundary between what is and
derstand knowledge use. In contrast to the is not considered science is not set in stone
previous perspectives, the critical perspective (Gieryn, 1995; Hoppe, 2005). By engaging in
does not primarily concern itself with describ- what Jasanoff calls ‘boundary work,’ political
ing the complexities of policy practice. Rather, actors and scientist directly or indirectly nego-
it focuses on the underlying power structures tiate the boundary between policy and science.
and discourse that shapes knowledge use (Tri- Subjects of this negotiation process include
antafillou, 2017, pp. 9–25). It could be ar- methods, standards for evidence, and the valid-
gued that the critical project is a project of ity of the interpretation of evidence. Making a
re-politicization, laying bare power structures similar argument, Grundmann (2009) describes
and putting them up for debate. how science plays the role of a referee in policy
Stevens (2007), for example, uses the con- processes, removing impure —that is, unscien-
cept of ‘survival of the fittest’ to explain which tific— elements from a policy process. Parsons
pieces of research influence decisions and (2002) claims that the evidence-based policy
which do not (see also: Monaghan, 2009). Pol- movement has made policy processes messier
icy makers select evidence to craft ‘policy sto- rather than more focused. Policy studies are
ries’ that fit the dominant narrative within gov- not isolated from policy processes, but instead
ernment Stevens (2011). The critical perspec- are part of it. Demanding that policy processes
tive for using research could transform a ques- have to adhere to evidence-based standards
tion about ethics into a financial dilemma that frustrates the deliberative processes by neglect-
fits a new public management context. Sim- ing values involved in decision-making and
ilarly, using this perspective, a debate about decreasing flexibility. In a sense, these critical
4Dorren & Van Dooren • Chameleonic knowledge • April 2021
perspectives are complementary to linear and lowed us to move beyond participant recon-
nonlinear perspectives. It is possible for a pol- struction. What policy makers present in inter-
icy maker to literally follow a study outcome, views is often a summarized or idealized and
while simultaneously engaging in what can coherent version of their actual practice (Czar-
be understood as story-crafting or enforcing niawska, 1997; Dorren, 2018; Portelli, 1991).
a boundary between science and nonscientific Being present for the analysis enabled us to
knowledge. capture the complexity of interactions between
It may seem unlikely that ex ante analy- an analysis and a policy process. We maxi-
ses could accommodate multiple, sometimes mized context sensitivity (Ybema et al., 2009,
seemingly opposing perspectives on knowl- pp. 6–7), which is so important according to
edge use. Those with a linear perspective claim the enlightenment model by Weiss (1979).
that knowledge gleaned from research can be
applied directly to policy processes, but those
with a nonlinear or critical perspective prob- i. Case selection
lematize this notion. Some may wonder how it We selected three infrastructure policy pro-
is possible that people simultaneously partici- cesses on three projects as cases. In infrastruc-
pate in the continuation of a ‘useful myth’ and ture policy making, the use of ex ante analysis
also use research and analyses to fill knowl- is relatively highly formalized and extensive
edge gaps. Based on the fact that all three (‘Decreet betreffende complexe projecten,’ 2014;
perspectives appear equally popular, we argue Ministerie van Infrastructuur en Milieu, 2016).
that instead of some perspectives being wrong For instance, in the Netherlands in 2008, the av-
and others being right, ex ante analysis has erage infrastructure project undertaken by the
certain properties which let it reflect theories national government took five years of study-
from all three perspectives. Therefore, when ing before work on the final design started
following a policy process over time and fo- (Elverding et al., 2008).
cusing on the messiness of policy practice, one
We selected cases in the Netherlands and
will likely see uses which resemble different
Flanders. Although the Netherlands neighbors
theories occur.
the Flanders region of Belgium, both areas dif-
fer significantly in terms of administrative cul-
III. Methods and case selection ture. The Netherlands is ‘commonly regarded
as one of the strongholds of policy analysis’
Because our goal was to gather insights on the (van Nispen Scholten, 2015) with a longstand-
development of interactions with ex ante anal- ing tradition of integrating analyses into policy
ysis over time without losing sight of the rich- processes; it also has a range of independent
ness of day-to-day policy practice, we decided agencies that evaluate policies. Flanders has a
to use an ethnographic study design. Instead culture in which policy decisions are ‘primar-
of focusing on a particular person or group ily the results of political bargaining,’ which
that deals with ex ante analysis often, we de- hinders their ability to develop a strongly insti-
cided to use particular sets of analyses as our tutionalized culture of policy analysis (Fobé et
research subject and follow them throughout al., 2017, p. 51). Including cases from these two
a policy process for a period of approximately diverging con- texts reduced the likeliness that
one year. This allowed us to see how different the mechanisms we observed, were specific to
perspectives on the use of analysis interacted a particular policy context. Table 1 gives an
and how interpretations of analyses changed overview of the cases in this study at the start
over time. The ethnographic design enabled of the observation process.
us to witness analysis used in many different When studying these policy process cases,
situations by a broad range of actors. we paid particular attention to the ways in
Furthermore, the ethnographic design al- which process participants discussed ex ante
5Dorren & Van Dooren • Chameleonic knowledge • April 2021
analysis. An ex ante analysis aims to predict. interests.
It departs from a reference situation, mostly
consisting of the situation ‘which exists at the
time the option will be realized, possibly com-
ii. Data Collection
bined with the situation which would arise if Observations were recorded in field notes. Be-
the option is not realized’ (Klaassen Hakvoort, cause the meeting settings were often static,
2015) and then predicts the expected develop- the primary focus of these field notes was on
ment of certain indicators as opposed to the what people said and how they interacted. We
reference situation. Ex ante analyses are gener- paid attention to tone of voice and nonverbal
ally used to compare policy options to inform communication. In addition, our field notes
policy choices. They can come in many shapes contained details on setting. They described
and sizes: in this study, we encountered every- the look and feel of the rooms in which meet-
thing from computer simulations to qualitative ings were held, their location, and any other
interviews with experts (reported as ‘expert circumstances of note.
judgments’), all identified as ex ante analyses. Because of the technical nature of infras-
One thing that all ex ante analyses had in com- tructure projects and the decision-making pro-
mon was that they involved quantification in cesses that surround them, we engaged in
some capacity. For example, a cost–benefit anal- ‘pre-member checking’ through small conversa-
ysis predicted the economic costs and benefits tions with participants during meeting breaks.
of a project in euros, and an environmental im- This enabled us to test our understanding of
pact assessment predicted the environmental what was going on in the meetings. In addition
impact of a project on a scale from 3 to + 3. to the meetings, we also had access to a por-
We conducted observations of these cases be- tion of the e-mail communica- tions for each
tween November 2017 and July 2019. At the be- project case. We were included in the email
ginning of our observation period, all projects list that was used to circulate the documents
were in an explorative stage. In the explorative that were up for discussion in the meetings.
stage, the project aims are broadly laid out. The Even though these communications were not
goal of this phase is to explore and compare explicitly included in our analysis, they did
several ways of reaching the project’s aims. We enrich our understanding of the con- tent of
had access to two types of meetings: publicly the meetings.
accessible meetings and internal meetings. All
cases had publicly accessible meetings. These
consisted of, for instance, information markets iii. Coding
about the project or more focused participa- In order to be able to process the data we gath-
tion sessions in which a wide range of actors ered and arrive at an overview of the uses of ex
were asked to give their input. For two of the ante analysis in infrastructure decision-making
three cases, we also had access to internal meet- processes, we devised an inductive cod- ing
ings. These consisted of, for example, regular strategy based on Miles and Huberman’s (1994)
meetings between members of the project staff, strategy for a cross-case comparison. We used
one-time presentations by experts, or meetings NVivo 12 as our coding software. Our strategy
with the political steering committee of the consisted of two main analytical steps. First,
project. In total, we conducted approximately we conducted a within-case analysis of each
200 h of observations in 74 meetings. case, coding each case separately. This first
A wide variety of actors attended these meet- round of coding began by assigning chunks
ings, from interested citizens to members of of data to descriptive codes containing little
action groups and from representatives of lo- interpretation (Miles Huberman, 1994, p. 57).
cal business associations to NGO’s. Table 2 These codes allowed us to distinguish between
contains an overview of these actors and their cases, people, and topics of conversation or
6Dorren & Van Dooren • Chameleonic knowledge • April 2021
Table 1: Cases and characteristics
Project Country Phase Budget Types of analyses (ob- Analysts Themes of participation ses-
type or region served) sions (observed)
Highway Netherlands Start of the ‘exploration phase’, ± †500 Environmental effects report, Contract research conducted Getting an overview of local
project in which several viable alter- million cost–benefit analysis, traffic by several private firms. Stud- situation. Making sure stud-
natives are compared quanti- studies,including railway traf- ies were supervised by the ies are as complete as pos-
tatively and qualitatively to fic. Predicting and monitor- project management team, sible. Gathering public sup-
reach an optimal solution ing the progress of behavioral who had access to in-house port for the project. Inform-
and infrastructural interven- expertise for feedback ing citizens
tions in terms of traffic reduc-
tion
Maritime Flanders End of the exploration phase; ± † 900 Environmental effects report, Contract research conducted Informing citizens
transport start of the phase in which million cost–benefit analysis by a private firm, supervised
infras- the ‘optimal’ alternative is ex- by the project management
tructure plored further in order to pro- team, who had access to in-
duce a definitive decision house expertise for feedback
Multimodal Flanders Exploration phase ± † Environmental effects report, Contract research conducted Informing citizens, collecting
transport 900–1.500 cost–benefit analysis by a private firm, supervised feedback on preliminary ver-
with a million by the project management sions of the analyses, build-
strong (depend- team, who had access to in- ing public support
maritime ing on house expertise for feedback
compo- solution)
nent
observation. This way, we were able to check iv. From codes to perspectives on
whether certain observations only held for cer- knowledge use
tain groups of people, or whether there was
a difference in the way in which people dis- We used our codes to qualitatively assess
cussed different aspects of an analysis. Second, whether the theoretical perspectives on knowl-
we analyzed the content of these descriptive edge use occurred in our data. For example,
codes in a second round of coding, creating we associated the code ‘absolutizing’ with lin-
analytical codes. Analytical codes have a more ear perspectives on knowledge use. When as-
interpretive character (Miles Huberman, 1994, sessing whether the observations coded under
p. 57). In this particular study, the analytical ‘absolutizing’ contained examples of people
codes mainly were meant to convey actors’ use employing a linear perspective on knowledge
and interpretation of ex ante analysis. To be use, we did indeed find statements such as
able to explain how ex ante analyses are able ‘this cost–benefit analysis shows us what the
to reflect many different models of use, we best trajectory for this bike path is’ or ‘we will
devised a data- driven coding scheme based not continue pursuing [this policy option], as
on the way in which people discussed specific our studies have shown it is unlikely to have
parts of ex ante analysis. This way of coding any effect.’ We looked at contextual informa-
allowed us to observe patterns to analyses use tion to assess whether these statements actu-
beyond the applicability of theoretical perspec- ally most closely resembled a linear perspec-
tives on knowledge use. Our coding resulted tive, or—for example—a situation more closely
in codes such as ‘downplaying,’ for instances resembling a political perspective in which a
in which actors tried to convince other actors linear-seeming statement was made.
to not take analysis outcomes to seriously, or
‘positive,’ capturing a set of reactions in which By comparing our coding to our theoretical
actors conveyed positive sentiments about an framework in this way, we were able to ver-
analysis. Other examples of codes were ‘abso- ify whether it is indeed the case that a broad
lutizing’ and ‘relativizing,’ for reactions that, range of theories applies to analysis use, rather
respectively, took study outcomes as a given or than some of these theories being wrong and
as relative to a study’s input. others being right. By studying patterns oc-
curring across cases in this light, we ultimately
distilled a set of properties which allowed ex
ante analysis to be used in such varying ways.
7Dorren & Van Dooren • Chameleonic knowledge • April 2021
Table 2: Actor types encountered during observations and their interest in the project in question
Actor type Interest with regard to the project
Audience in public meet-
This category covers a wide range of other actors attending public meetings, most of
ings (nongovernmental)
them citizens. Some joined public meetings out of interest. Others were concerned
because the project would directly affect them, for example because they owned
a house or business near a potential project site. Sometimes, the interests of these
people aligned with those of local action groups, but sometimes they held opposite
interests. For example, in the Flemish maritime infrastructure project, some people
resented the action group for ‘unnecessarily delaying the project,’ prolonging the
state of uncertainty they found themselves in. Overall, the interest of these actors
appeared more closely related to their personal situation
Expert (not part of the During public meetings, we encountered a different type of expert. These were
project team) university professors or former government employees who often attended public
meetings because they were opposed to the project, based on their expertise. We
encountered one instance of such an expert who was in favor of the project in ques-
tion and was invited to join the public fora by members of the project management
team
Government employee Sometimes, project management teams would consult government experts, for
(not part of the project example when meeting with analysts or deciding on a communication strategy.
team) These experts often did not display a vested interest in the project. During public
meetings, these government employees would occasionally help out by moderating
or providing factual information about the project
Representative of an in- Public fora were attended by many interest groups, with varying attitudes toward
terest group the project. Some, such as the Dutch Cyclists’ Union or an organization representing
local small- and medium-sized enterprises, evaluated the project from their specific
perspective. Their goal was to ensure the interests of the specific cause they
represented, were covered in the project. A rather specific type of interest group
we encountered was the local action group. In the Flemish maritime infrastructure
project, a group was established to specifically protest this project out of concern for
the livability of the surrounding area. In the Netherlands, we encountered a local
environmental action group protesting the project out of concern for air quality.
In both cases, these groups were strongly opposed to the project. In the second
Flemish case, several action groups established to protest another project, joined
the public meetings. As the project they were established to protest was a highway
project, these groups paid particular attention to traffic effects
Members of the project In the Flemish cases, members from the project management team were employed
team by the regional government’s department for Mobility and Public Works and the
department for Environment. In the Dutch case, team members were employed
by the Ministry of Infrastructure and Water Management, as well as regional
governments who participated in the project. Even though members’ views of the
project would sometimes differ from those of their superiors, team members would
always support the project as a consequence of their assignment to complete it
Chair of a public meet- These actors chaired public meetings. In the Dutch project, this function was
ing fulfilled by either the project manager or the head of the consortium of private
parties conducting the analysis. In the Flemish maritime transport infrastructure
case, the project manager chaired the public meetings we attended. In the Flemish
multimodal project, this function was either fulfilled by a prominent member of the
project management team or someone from a private firm contracted to facilitate
public meetings. Depending on their affiliation, these actors had more or less of an
interest in defending the project against challenges
Analyst These actors were the ones conducting the actual ex ante analysis or managing that
process. Their interests were largely determined by their assignment: to conduct a
solid analysis which compared different policy options against each other
8Dorren & Van Dooren • Chameleonic knowledge • April 2021
IV. Results and can, in a way, replace politicians’ input
when it comes to making decisions. In addition
In this section, we exemplify how the differ- to the linear perspective, we also encountered
ent perspectives on knowledge use explored all nonlinear perspectives on knowledge use
above occurred in our data. Then, we explain during our observations. For instance, in one
what specific characteristics of ex ante analy- of the project cases, a new policy option was
sis allows it to reflect such a wide variety of added to an already existing set which had
perspectives without it resulting in so much been heavily debated in a deliberation trajec-
ambiguity that leads to the ex ante analysis los- tory of almost two years. The new option was
ing credibility or being abandoned all together. announced by a minister and came as a sur-
The first characteristic of ex ante analysis we prise to most participants in the process. The
discuss is its built-in ambiguity, which allows civil servant presiding the participation pro-
the analysis to reflect a multitude of different cess was not able to pinpoint where the idea
ideas for its use. The second characteristic we came from, and they were only able to say that
discuss is lack of full ownership of the analysis it emerged from ‘new insight,’ which would
by any one person, which also means that no be an example of Weiss’ (1979) enlightenment
one is completely responsible for it. The third model.
characteristic is trustworthiness: people do not We also encountered critical perspectives in
seem to lose trust in ex ante analysis results our observations. Even though these perspec-
because of, among other things, its complexity tives are of a more analytical nature—meaning
and ability to seemingly separate facts from one would not expect them to be directly de-
values. ployed by process participants—we did occa-
sionally observe them in action. For example,
the idea of the ex ante analysis as a story-
i. Facilitating many different perspec-
crafting device was mentioned explicitly by
tives members of one of the project management
We observed all perspectives of knowledge teams. When discussing the design of an ex
use during our fieldwork, albeit some were ante analysis, these people wanted the analy-
more clearly present than others. A substan- sis being able to tell ‘the story of the project.’
tial amount of the discussion about analysis In a presentation for project managers in the
we observed expressed a linear perspective on same case, an analyst proclaimed they wanted
knowledge use. This perspective suggests that their measurements to ‘enable politicians to
a study outcome could directly and exclusively tell a story’ rather than just having them speak
inform parts of a policy outcome. The fre- about the benefits of the project in broad terms.
quency with which linear perspectives were Other examples of how differ- ent theories of
expressed can be at least partially attributed to knowledge use appeared in our data, can be
legal rules and procedures. For instance, Flan- found in Table 3.
ders has air quality norms that limit the nega- It is remarkable that none of the perspectives
tive impact of a project on air quality. When the can be put aside based on our observations,
expected amount of pollution exceeds a certain despite some of them appearing to be mutu-
level, the government is legally required to take ally exclusive. This means that apparently, ex
measures to decrease pollution. For many peo- ante analysis possesses certain qualities that
ple involved in policy processes, such a linear enable an ex ante analysis to reflect theories
application of knowledge is ideal. In Flanders, from all these perspectives, rather than it being
an often-heard claim was that people wanted the case that some perspectives are more right
the policy process to be ‘objective’ instead of than others. In the next section, we describe
‘political.’ This implies the opinion that anal- how certain features of ex ante analysis allow
yses can point to an objectively right solution, it to reflect different theories of use and enable
9Dorren & Van Dooren • Chameleonic knowledge • April 2021
Table 3: Perspectives on analysis use and examples from data
Category Type of Use Example
Linear Problem solv- There are situations in which legal norms dictate a linear application of study
perspec- ing outcomes. Also, some of the attendants of public meetings seem to desire that the
tives government take a linear approach, as opposed to having a politician decide on a
policy
Knowledge Sometimes, at the public meetings we observed, government officials invited univer-
driven sity professors to reflect on the analysis. One of them was a professor who would
suggest using more complex models, simply because they were available and thus
should be used
Nonlinear Interactive Most clearly present in private meetings where discussion on the outcomes of
perspec- studies is a balancing act between the outcome of the study, and, for example,
tives experience with previous projects. For example, in one of the cases, an expert from
one of the participating governments in the project caused confusion by sending
the project team an e-mail with an attached feasibility study that had a negative
outcome. How- ever, in the e-mail he said he was feeling positively about the
project and gave it a high chance of succeeding. It turned out that he drew this
conclusion based on his own experiences with this type of project, and said he saw
some opportunities that were not adequately reflected in the study. This is also
how the use of analysis was presented by government officials at public meetings:
analysis will aid decision makers, but the decision makers are not obliged to follow
the outcome of the studies
Political per- In some of the public meetings we observed, a member of an action group voiced
spective the suspicion that the analyses serve the political goal of making it seem like there
are no real differences between building the project and not building it in terms of
traffic generation
Tactical per- n the meetings we observed, members of project teams sometimes mentioned
spective benefits of analyses not directly related to their contents. For example, someone
mentioned the expectation that by studying thoroughly, the chances of losing court
appeals against the project diminished
Enlightenment This perspective is difficult to observe, as the enlightenment perspective is all
perspective about how studies influence a broader frame of reference that people use to make
decisions. See the text of ‘Facilitating many different perspectives’ section for
examples
Critical Story crafting This is a recurring theme in the private meetings of one of the project cases we
perspec- observed. This project had some people on the board who encouraged a person
tives who was designing the analysis to take into account that the analysis has to ‘tell the
story of the project’
A useful myth According to our observations, people appeared to universally agree that analysis is
important. No one questioned the need for studies, even though they have widely
varying opinions on the way those studies should be designed. Furthermore, analy-
ses seemed to enable proponents and opponents of a project to have a constructive
dialogue
A boundary When comparing private meetings to public meetings, it is quite visible how the
object scope of studies determines what is discussed and which things get noticed. For
instance, after the project starts, the project aim is ‘locked in’ in the studies. This
means the project aim is no longer up for discussion and also limits discussions on
what variables should and should not be included in the analysis
10Dorren & Van Dooren • Chameleonic knowledge • April 2021
actors in policy processes to switch between groups made statements in both categories. In
different uses. fact, when zooming in on specific individuals
within groups, we see that individuals regu-
larly absolutized and then relativized analysis
ii. Like a chameleon: how ex ante
outcomes in the same discussion.
analysis reflects many different theo- This pattern of behavior is a result of the way
ries of use simultaneously in which people react to a specific set of proper-
We found that ex ante analysis is able to reflect ties of ex ante analysis. An ex ante analysis can
many different theories of use because of three be relativized because it is a simplified model
key characteristics. First, ex ante analysis can of reality. Decisions about what to include in
be relativized, yet deliver absolute- seeming the analysis and what to exclude from it must
results. Second, the responsibility for ex ante be made. In all policy processes we observed,
analysis is split up over different actors, up to there was room for the process participants to
the extent that outcomes are ascribed to the provide input about what should be included
analysis itself, rather than the people involved in and excluded from the analysis, even though
in its construction. Lastly, ex ante analysis some analyses were based on standardized per-
draws from many different sources of trust. spectives and indicators. If not already present,
inviting actors to provide input for the analysis
appeared to spark awareness of the simplifying
iii. Facilitating debate: soft methods, nature of ex ante analysis.
hard facts Where the simplifying nature of ex ante anal-
ysis allows for relativizing comments, the out-
The first characteristic that allows an analysis
comes of the analyses have an exact-seeming
to reflect many different perspectives stems
nature. Outcomes presented appear as solid
from the way the analysis process is structured
facts with that are difficult to argue with. They
and the format in which outcomes are pre-
appear precise and not at all the product of a
sented. As a feature of their design, analyses’
limited set of methods, as is illustrated by the
models can be relativized, yet their outcomes
following observation:
have a very absolute character. This causes peo-
ple to simultaneously relativize and absolutize The analyst tells the group that they
analyses’ outcomes. have done computer simulations to
This observation came about as follows. In determine the optimal shape of the
our analysis, we coded for the level of certainty dock. In this simulation a digital ship
with which people make statements about anal- enters a digital dock under certain
yses or based on analyses. Statements coded conditions to judge the operational
as ‘relative certainty’ implicitly or explicitly quality of each policy option. The an-
pointed toward the margining of uncertainty alyst tells us that in this case, the tug-
that comes with the outcomes of an analysis. boats pulled over 80 tons and used x
For example, responses pointing at the limi- amount of power. They go on, telling
tations of a model were coded as expressing us how many times the tugboats had
relative certainty. Statement coded as convey- to use which engines and how strong
ing ‘absolute certainty’ presented analysis out- the winds were during these manoeu-
comes as if this margin of uncertainty was vres. On the projector screen, a ship-
not a factor. These statements mostly con- shaped figure moves down a river on
cluded that ‘the analysis shows that x causes what looks like a Google Maps satel-
y,’ without addressing the uncertainties con- lite image. It makes a sweeping move
tained in this outcome. Table 4 shows that across the [river] and parks itself on a
when analyzing statements about the analysis couple of fields, in the midst of which
made by different groups of people, all of these resides what appears to be a small
11Dorren & Van Dooren • Chameleonic knowledge • April 2021
Table 4: Distribution of the level of certainty over statements made by different groups of participants
Role in the meeting Absolute certainty Relative certainty
(%) (%)
Audience in public meetings (nongovernmental) 40 60
Expert (not part of the project team) 46 54
Government employee (not part of the project team) 65 35
Representative of an interest group 47 53
Members of the project team 53 47
Chair of a public meeting 78 20
Analyst 57 43
house.1 pragmatic model, based on what is feasible,’
are valid arguments in the same discussion pre-
The analyst quoted above talks as if they ob- sented minutes after one another by the same
served real-world tugboats tugging a ship over person. The conception of ex ante analysis be-
a river in certain wind conditions. In another ing simultaneously relative and absolute allows
presentation on the same study, effects on high it to facilitate two types of discussions simul-
tide levels were presented in terms of centime- taneously. The analysis as something relative
ters. A large results table presenting the out- allows for a discussion on what Bertolini (2017)
comes of a cost–benefit analysis in another case describes as a dilemma. A dilemma involves a
displayed the cost differences between project choice between incommensurables, which are
options in eurocents, even though much of often difficult to quantify. What is more im-
its input came from interviews with transport portant, economic growth or reducing traffic
companies. A positive effect on inland water- nuisance? Who deserves better protection, peo-
way shipping in the same project is represented ple or birds? 2 These and more dilemmas oc-
by an absolute number (+ 1), despite it being curred in discussions on what perspectives or
an estimation based on conversations with gov- indicators to use in policy processes, but they
ernment employees and transport companies. also served to relativize analysis outcomes. For
The language used to present the outcomes example, displeasing results were attributed
of an analysis often has a confidence and cer- often to a narrow scope.
tain tone. They include phrases such as ‘we Where the relative side of analysis allows
see that these effects are negligible’ (as if the for a discussion on dilemmas, the absolute-
analyst has seen true evidence of this) or ‘the seeming outcomes of an analysis facilitate a
effects are similar for all alternatives’ (as if the problem-solving approach. Problems are dif-
alternatives have been built in reality and the ferent from dilemmas in that they require find-
analyst was able to observe them) rather than ing a ‘best’ solution (Bertolini, 2017) instead
phrases such as ‘the model shows...’ or ‘’we of presenting a choice between incommensu-
expect that...’. rables. Questions linked to problems include
These shifting levels of certainty allow pro- ‘How do we optimize traffic flow on this road?’
cess actors to shift between uses of an analysis and ‘How do we build with a minimal impact
which would fit under different academic the- on the environment?’.
ories on knowledge use. ‘The analysis shows In the early stages of the policy process,
that traffic effects will be null’ and ‘this is a when the ex ante analysis is still being de-
1 All field notes, including quotations, are originally in 2 A question someone asked one of the members of a
Dutch. All material included in this paper is our own trans- project team on an information evening when com- ment-
lation. Some details have been altered to ensure anonymity ing on the impact nature conservation guidelines have on
of the respondents. the validity of certain policy options.
12Dorren & Van Dooren • Chameleonic knowledge • April 2021
signed, there is ample time to discuss dilem- ity. During our observations, we witnessed
mas. In all cases that we observed, stake- hold- the conception of several analyses. Usually,
ers were asked to provide input for the design this involved a group of civil servants stand-
of the analysis by listing what they felt was ing around a flipchart and using thick, col-
important and should be taken into consider- ored markers to write down things they believe
ation. However, once the analysis starts, the should be in the analysis. This input was to
analysis design becomes fixed. Now, the ex be considered by the analysts who conduct the
ante analysis has become a problem solver. 3 actual ex ante analysis. Despite their essential
Its outcome—a cost–benefit ratio per alterna- role in the construction of the ex ante analy-
tive—does not necessarily facilitate a debate on sis, analysts did not feel that they had own-
dilemmas but instead allows a decision-making ership of the analysis they were conducting.
process to move forward. Reflecting on both Instead, analysts tended to portray themselves
relativity and absoluteness, ex ante analysis as people who merely execute. Oftentimes,
allowed two discussions to run simultaneously. they got their standards and input for mod-
The existence of two simultaneous discussions els elsewhere. For instance, they often based
led to tensions. Where actors interested in their work on models developed by the de-
completing the project wanted the process to partment of transport. These models can be
move from optimizing the study to choosing found in books and manuals, which are on
a project alternative to be built, those critical the websites of the Flemish and Dutch govern-
of the project or a particular study wanted to ments. In Flanders, these standards have been
keep discussing the scope of the study. This provided by consultants. In the Nether- lands,
sometimes led to situations in internal meet- they come from a combination of government
ings where people were forced to make deci- in-house experts and private sec- tor consul-
sions based on analyses whose scope they did tants. They reside in different places and are
not agree on. informed by different sources (Departement
To sum up, ex ante analysis appears to have Omgeving; Ministerie van Infrastructuur en
built-in ambiguity. This is because the meth- Milieu, 2016, pp. 101–136). In the process of
ods used allow the analysis to be relativized, conducting an analysis, analysts also got input
but the way outcomes are presented simulta- from other sources. They organized citizen con-
neously gives the analysis a sense of absolute- sultations, interviewed important stakehold-
ness. This ambiguousness allows analyses to ers, retrieved data from government agencies,
reflect perspectives that present them as a solid proposed analytical frameworks to a project’s
ground for decision-making, such as linear per- political steering committee, and asked what
spectives and perspectives that emphasize that they called ‘experts’ to estimate the effects of
the use of analysis in policy making depends planned interventions. The substantial number
on one’s interpretation of the content of the of parties influencing ex ante analysis makes
analysis. The built-in ambiguity of ex ante it impossible to assign ownership of the anal-
analysis is, in other words, the first character- ysis to a specific actor. This feature of ex ante
istic that enables it to be a chameleon in the analyses caused actors to generally experience
policy process. the ownership of the analyses as shared. When
analysts received questions about the analy-
sis they were working on, they regularly redi-
iv. Split-up responsibility
rected those questions to other parties. When
A second characteristic of ex ante analysis that asked questions about the reasoning behind
allows it to reflect many perspectives of knowl- certain models, they referred to the govern-
edge use at once is that it does not appear to ment agency providing these standards. Most
fully fall under anyone person’s responsibil- often, though, analysts—and others working
3 Based
with them—referred to the analysis itself as a
on Bertolini (2017).
13Dorren & Van Dooren • Chameleonic knowledge • April 2021
source of authority. It was not the analysts, but ‘If we would not build this project, we
the analysis that ‘showed’ or ‘indicated’ some- would see an immediate decrease of
thing. Despite many people being involved in container traffic’ ‘Whilst the problems
the construction of the analyses we observed, are not pressing as we speak, they
the outcome of an analysis appeared to be seen will be in the future’ ‘All alternatives
as a product of the analysis itself rather than come with some risk, but—and this is
something created or influenced by analysts. important—all risks are con- trollable’
When looking at where our code ‘absolute cer- a: ‘There are also people living in this
tainty’ overlaps with our codes ‘conclusions area, you know! You do not think
based on outcomes’ and ‘discussing outcomes,’ about that!’ b: ‘We do, but it has been
we see how an analysis which actors construct proven that the effects on traffic are
themselves produces outcomes that are dis- nil’.
cussed as if they are inevitable or undisputable:
This scene demonstrates how, by accepting
‘If we would not build this project, we outcomes as a given, one also accepts the in-
would see an immediate decrease of puts of an analysis as a given. Consequently,
container traffic’ ‘Whilst the problems actors wanting to criticize these inputs have to
are not pressing as we speak, they make an extra effort. Actors responsible for the
will be in the future’ ‘All alternatives inputs, on the other hand, are obscured behind
come with some risk, but—and this is the agency that an analysis exudes.
important—all risks are controllable’ At the same time, the agency of ex ante anal-
a: ‘There are also people living in this ysis also allowed it to be used more flexibly.
area, you know! You do not think The fact that there is no ultimate authority dic-
about that!’ b: ‘We do, but it has been tating what is and is not a correct interpretation
proven that the effects on traffic are of an ex ante analysis allows ex ante analysis to
nil’. be used and interpreted in contradictory ways.
Because there is no definitive interpretation,
While no one is fully in control of the out- people are relatively free to use ex ante analy-
come of an analysis, the analysis does have sis how they want. This freedom, which is a
control over the decisions a government makes. result of the way ex ante analysis is structured,
The fact that no one seems to be completely constitutes its second chameleonic quality.
in control of the analysis means that it is dif-
ficult to challenge, but also that there is no
v. Trust
one to respond to challenges. This had an im-
pact on the power balance in policy processes. In the previous section, we explained that even
When outcomes are presented as a product of though analyses are recognized as being de-
analytical models and are discussed as if they pendant on assumptions that are openly being
are indisputable, that acts as a shield between discussed, they do convey a sense of authority.
an analysis’ critics and the analytical choices After reading these sections, one might still
that produced these outcomes. This is demon- have questions about the popularity of ex ante
strated in the following scene, in which two analysis. An analysis with built-in ambiguity
members of a project management team have for which no one appears to be fully responsi-
a discussion about the interpretation of the re- ble does not necessarily invoke a sense of trust.
sults of an analysis. One of the team members Nevertheless, in our observations we did not
pushes for stricter management of a set of sub- encountered anyone who dismissed the idea of
projects, as they fear these projects might be conducting analyses altogether. Ex ante analy-
underperforming based on an ex ante analysis. sis appears to be trusted, despite the ambiguity
The other team member aims to challenge this surrounding it. In this section, we will explore
position by critiquing the analysis: what drives trust in ex ante analysis.
14You can also read
