Evidence-Based Management: Concept Cleanup Time?
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
2009 Briner, Denyer, and Rousseau 19
E X C H A N G E
Evidence-Based Management: Concept Cleanup
Time?
by Rob B. Briner, David Denyer, and Denise M. Rousseau
Executive Overview
The term evidence-based management (EBMgt) is relatively new, though the idea of using research evidence
to help make managerial decisions is not. In this paper we identify and clarify a number of common
misconceptions about EBMgt. Rather than a single rigid method, EBMgt is a family of approaches that
support decision making. It is something done by practitioners, not scholars, although scholars have a
critical role to play in helping to provide the infrastructure required for EBMgt. Properly conducted
systematic reviews that summarize in an explicit way what is known and not known about a specific
practice-related question are a cornerstone of EBMgt.
T
he virtues of using research evidence to inform Like most multifaceted new ideas, EBMgt is un-
management practice have permeated manage- derdeveloped, misunderstood, misapplied, and im-
rial writings and organizational research over plemented inconsistently. That’s why new learn-
the past 50 or more years, as the lead article by ing and developments in EBMgt must continue.
Reay, Berta, and Kohn (RBK, this issue) points In that spirit, our rejoinder to the RBK article
out. Evidence-based management (EBMgt) as a makes four key points:
concept in itself is new and can be defined as
1. EBMgt is something done by practitioners, not
follows:
scholars. We’ll show the key ideas behind the
Evidence-based management is about making decisions concept of EBMgt and highlight and clarify
through the conscientious, explicit, and judicious use of sources of confusion and opportunities.
four sources of information: practitioner expertise and
judgment, evidence from the local context, a critical
2. EBMgt is a family of practices, not a single rigid
evaluation of the best available research evidence, and the formulaic method of making organizational de-
perspectives of those people who might be affected by the cisions. This notion is critical to understanding
decision. both how it might be implemented and what
should be evaluated.
3. Scholars, educators, and consultants can all
We acknowledge our debt to Denny Organ for appropriating the title play a part in building the essential supports for
from his seminal article on citizenship behavior for our own purposes.
David Denyer’s work was supported by the Advanced Institute of Manage-
the practice of EBMgt. To effectively target
ment Research and the Economic and Social Research Council (grant critical knowledge and related resources to
number: RES033102700045). Denise Rousseau’s work was supported by the practitioners, an EBMgt infrastructure is re-
H. J. Heinz II University Professorship.
Rob B. Briner (r.briner@bbk.ac.uk) is Professor of Organizational Psychology at Birkbeck College, University of London.
David Denyer (david.denyer@cranfield.ac.uk) is Senior Lecturer in Organizational Studies at Cranfield School of Management, Cranfield
University.
*Denise M. Rousseau (denise@cmu.edu) is H. J. Heinz II University Professor of Organizational Behavior and Public Policy at Carnegie
Mellon University.
Copyright by the Academy of Management; all rights reserved. Contents may not be copied, e-mailed, posted to a listserv, or otherwise transmitted without the copyright holder’s express written
permission. Users may print, download, or e-mail articles for individual use only.20 Academy of Management Perspectives November
quired; its development depends on the dis- ment research could and should inform practice is
tinctive knowledge and skills found in each of old, as are many of the texts cited by RBK. How-
these communities. Scholars, as the originators ever, the EBMgt concept was born in the 1990s, a
of research evidence, have a particularly im- spin-off of the evidence-based medicine that had
portant role in the process. Nonetheless, sup- emerged around that time (cf. Sackett et al., 1996,
porting EBMgt is only a part of their profes- 2000). Third, it is difficult to practice EBMgt
sion’s activities. thoroughly without accessible systematic reviews
4. Systematic reviews (SRs) are a cornerstone of of evidence. As few of these currently exist in
EBMgt practice and its infrastructure, and they management or organization studies, even practi-
need to possess certain features if they are to be tioners who wanted to could not fully practice
informative and useful. EBMgt. Given all this, it is highly unlikely that
there would be much formal research into the
Let’s start with RBK’s key ideas in order to place
efficacy of EBMgt, since practitioners may not
our four points in context.
even be aware they are doing it, the idea is too
recent, and the basic tools it requires are not yet
Reay, Berta, and Kohn’s Article available.
R
BK assert that there is no research evidence for We share RBK’s concerns about EBMgt and
the effectiveness of EBMgt, and given the indeed have many more of our own. One of the
manner in which their question is framed and dangers is the privileging of research evidence
the review conducted, we would broadly agree. over other forms of evidence, the local context
Their conclusions are based largely on an apparent (e.g., Johns, 2006), and insight from other sources,
absence in the literature they reviewed of evi- especially professional experience. A second dan-
dence about the practice of EBMgt. An absence of ger is blind adoption of a “Big Science” perspec-
evidence tells nothing about whether something tive on EBMgt that prizes randomized control
is effective or otherwise. Indeed, we believe there trials and meta-analyses above all other kinds of
is evidence about the practice of EBMgt, but they research evidence. A third danger is the top-down
are looking in the wrong place. approach where scholars tell practitioners what
EBMgt fundamentally is something performed they should do, thus imposing “our” evidence on
by practitioners, not scholars. Searching for evi- “them.” Instead, as we will discuss, EBMgt is prac-
dence of EBMgt in scholarly texts seems to us akin tice-focused and starts with the questions, prob-
to searching for one’s lost keys under the street lems, and issues faced by managers and other
light as it’s the only place on the whole block organizational practitioners. It is not the hypoth-
where there’s light. But if RBK are looking in the eses, research problems, or theoretical puzzles that
wrong place, where else should one look for evi- are the primary focus of scholarship. A fourth
dence on EBMgt? This question has several concern is that research evidence replaces rather
answers. than complements other forms of data and knowl-
There are three main reasons a large body of edge that go into making quality decisions. We do,
research on something called EBMgt does not however, believe that a fit-for-purpose approach
exist. First, EBMgt is a family of approaches, and to evidence-based management would benefit
much of the work in the realm of EBMgt doesn’t both the scholarly and practitioner communities.
carry the label. Practitioners often practice some- Given that there are now several publications
thing close to EBMgt, when, for example, they and conference events regarding EBMgt, some
take a basic principle developed from employee scholars appear to be at least somewhat aware of
selection research and incorporate it into their it. However, with partial awareness comes confu-
regular decision making, problem solving, and sion, as demonstrated by the RBK review. Clari-
practice. But in doing so they are likely not to fication is needed regarding the nature and mean-
even know they have practiced EBMgt. Second, ings of EBMgt; we attempt to provide such
the term EBMgt is new. The idea that manage- clarification in Table 1.2009 Briner, Denyer, and Rousseau 21
Table 1
What Is Evidence-Based Management?
Evidence-Based Management Is . . . Evidence-Based Management Is Not . . .
● Something managers and practitioners do ● Something management scholars do
● Something practitioners already do to some extent ● A brand-new way of making decisions
● About the practice of management ● About conducting particular types of academic research
● A family of related approaches to decision making ● A single decision-making method
● A way of thinking about how to make decisions ● A rigid, one-size-fits-all decision-making formula
● About using different types of information ● About privileging evidence from academic research
● About using a wide range of different kinds of research evidence ● About using only certain types of research evidence irrespective of the problem
depending on the problem
● Practitioners using research evidence as just one of several sources of ● Scholars or research evidence telling practitioners what they should do
information
● A means of getting existing management research out to practitioners ● About conducting research only about management practices
● Likely to help both the process and outcome of practitioner ● The solution to all management problems
decision making
● About questioning ideas such as “best practice” ● About identifying and promoting “best practice”
We believe that the concerns RBK raise and cesses can aid in achieving a quality decision even
the findings of their review largely reflect a set of under considerable uncertainty.
misconceptions about EBMgt. In doing or thinking about EBMgt, it is essen-
tial to consider all four of its elements (see Figure
Point #1: Evidence-Based Practice Is Something 1). EBMgt takes place at the intersection of all
Practitioners Do (or Not) four, but crucially the size of each circle—and
Each time a practitioner attempts to make a de- hence the strength of its influence—varies with
cision, solve a problem, or institute a set of prac- each decision. In some circumstances, the opin-
tices is distinct and unique. Nonetheless, a general ions of stakeholders or ethical considerations may
approach or way of thinking can apply across be judged by the decision makers to be much more
decisions; this is what we refer to when we speak important than the external research evidence
of EBMgt. Within this general approach, EBMgt and thus be given much greater emphasis in the
incorporates, among other things, the best avail- decision. In other circumstances, there may be
able scientific evidence in making managerial de- little internal evidence available and thus its in-
cisions. Like its counterparts in medicine (e.g., fluence on the decision would be relatively minor.
Sackett et al., 2000) and education (Thomas & In all cases, though, the choice to place more or
Pring, 2004), EBMgt is informed by practitioner less emphasis on various elements should be made
judgment regarding experience, contextual cir- in a mindful, conscious fashion.
cumstances, and ethical concerns. Peter Drucker, Another consequence of the newness of the
a seminal management thinker, was perhaps the EBMgt concept is that various writers place dif-
first to assert that most business issues—from mo- ferent emphases on each of the four elements. In
rale problems to strategic implementation—are some accounts, notably that of the EBMgt Col-
generic, “repetitions of familiar problems cloaked laborative (2009), the primary though not exclu-
in the guise of uniqueness” (quote from Lowen- sive concern is to find ways of facilitating practi-
stein, 2006; Drucker, 1966). If a problem is ge- tioners’ use of research evidence. They draw
neric, effective managers can benefit from under- inspiration from global organizations such as the
standing the principles underlying it as a guide to Cochrane (2009) and Campbell (2009) Collabo-
action. If the problem is novel, awareness of ef- rations, which produce and disseminate systematic
fective decision-making and problem-solving pro- research reviews in medical, educational, and so-22 Academy of Management Perspectives November
Figure 1 ment about the evidence itself and its applicabil-
The Four Elements of EBMgt ity to the situation.
These are all aspects of EBMgt, and a signifi-
cant amount of work has been conducted in each
of these areas. There are pockets of activity
Evaluated external evidence around the world, most notably in health care
management (Kovner, Elton, & Billings, 2005;
Lemieux-Charles & Champagne, 2004). But, as
yet, little coordination or integration exists, and
Praconer Stakeholders (e.g.,
experience and employees), thus EBMgt has not reached a tipping point into
Decision
judgments preferences,
or values
expanded consensus and adoption as it has in
other fields (Sackett et al., 1996, 2000; Sherman,
2002). As has been argued elsewhere:
Context, organizaonal The absence of a critical mass of evidence-based managers
actors, circumstances today translates into pressures to conform to more ad hoc
and experience-based approaches, especially in settings
where critical organizational positions are held by man-
agers without evidence-based training. Indeed an entire
generation of evidence-based managers may be needed
cial policy research. Evidence-based education before behavioral science evidence is widely used (Rous-
(Rousseau, 2005, 2006; Rousseau & McCarthy, seau & McCarthy, 2007, p. 99).
2007) can also facilitate this, and the training of
managers has, in some institutions such as Cran-
Point #2: EBMgt Represents a Family of
field, Carnegie Mellon, and Case Western Re-
Approaches
serve universities, come to place more emphasis
than previously on the skills involved in accessing Any decision-making process is likely to be en-
and making use of research findings. hanced through the use of relevant and reliable
By contrast, other accounts of EBMgt, most evidence, whether it’s buying someone a birthday
notably Pfeffer and Sutton (2006), emphasize the present or wondering which new washing ma-
importance of collecting and analyzing internal chine to buy. We use evidence quite automatically
organizational evidence and pay less attention to and unconsciously for even the smallest of deci-
the role of external research and systematic re- sions. To the extent that it is based on complex
views. Still others call for tools and techniques domain-relevant schemas, even intuition (Dane
promoting effective decision making and the ex- & Pratt, 2007) draws on the evidence of experi-
ercising of judgment (Yates, 1990). In these ac- ence. But EBMgt, like evidence-based practice,
counts, judgment is an essential management generally is taking what can be a fairly automatic
skill, especially where time is short or the circum- approach and making it more explicit, mindful,
stances sensitive. Moreover, individual judgment critical, and systematic.
and critical appraisal of evidence are essential Mundane everyday examples of evidence use
when using research evidence to inform a specific abound. Jane is booking a holiday and wants to go
decision in a particular setting. Is the evidence somewhere that’s not too hot. Fred wants to
valid and reliable? The meanings and importance choose a route that isn’t too hilly for his weekend
of validity and reliability will depend on the prob- hike. Costas is trying to book a restaurant for a
lem and decision. Is the evidence relevant here visitor from out of town who is very keen on food.
and to this context? It’s unlikely that available In each case, these decision makers may actively
evidence is gathered from exactly the same con- seek out evidence online or in print to obtain
text or setting. Using research evidence, like using information such as the average temperature in a
any other form of evidence, requires critical judg- given month at a vacation spot, the topography of2009 Briner, Denyer, and Rousseau 23
a region’s hiking trails, or the comments of a local EBMgt approaches. Examples of some of these
newspaper’s restaurant critic. (some also described by Pfeffer & Sutton, 2006)
The logical rather than empirical analysis of a are described below.
situation can also act as a form of evidence. For First is the use of logical analysis and disci-
example, there are no randomized controlled trials plined imagination (Weick, 1989) to answer
of parachute use (Smith & Pell, 2003). However, “what if?” questions. What is likely to happen if
would we wish to conclude in the absence of some practice or technique is introduced or some
evidence that parachute use is ineffective in pre- potential solution implemented? In cases where
venting death or injury? Similarly, the absence of external evidence from research or internal evi-
certain kinds of evidence for the application of dence is not available or perceived as too difficult
EBMgt does not mean that incorporating relevant to interpret and utilize, it is still possible to apply
and reliable evidence will not enhance decision logic and reasoning (itself based of course on ev-
making. idence, experience, and theory). In these cases,
What might a general EBMgt process look like? explicit reasoning can be applied to consider the
Although no single or agreed-upon process exists, questions of how and if a particular managerial
we suggest that the following steps might consti- intervention is likely to have the desired benefits
tute one approach to EBMgt: and what, if any, costs may be incurred. A second
● The start of the process is the practitioner’s or related approach involves making the justification
manager’s problem, question, or issue. The ac- for the decision explicit and transparent. Even
curacy of the problem statement would be dis- where evidence is limited or ambiguous, identify-
cussed and challenged so that it could be artic- ing what is believed about that evidence, the
ulated as clearly and explicitly as possible. context, and how each piece of evidence plays a
● Next, internal organizational evidence or data role in the decision allows for a more critical
about the problem or issue would be gathered appraisal of the available evidence and the as-
and examined to check its relevance and valid- sumptions held by decision makers.
ity. At this stage the problem may be restated or Third, specifying what might, in principle, con-
reformulated or made more specific. stitute relevant evidence about the problem or
● External evidence from published research question represents another approach. This is an
about the problem would also be identified and essential initial stage in conducting systematic
critically appraised in the form of something reviews (see below) and can, even without con-
like a systematic review or a rapid evidence ducting the review, help shape understanding of
assessment (a quicker and more tightly con- the problem and suggest likely consequences of
strained version of a systematic review, which different decisions. This also applies to internal
similarly adopts an explicit and systematic evidence within the organization, where, again,
method). the process of identifying what sort of data might
● The views of stakeholders and those likely to be be relevant can give insight about the apparent
affected by the decision would be considered, problem and potential solutions. This process may
along with ethical implications of the decision. also help identify relevant organizational evidence
● When all these sources of information had been that can be gathered fairly readily.
gathered and critically appraised, a decision The approach organizations sometimes adopt
would be made that attempts to take account of in dealing with consultants provides another ex-
and integrate these four sources of information. ample of the ways in which elements of EBMgt are
Although this is adequate as a general descrip- already practiced. For instance, when organiza-
tion of what practicing EBMgt might entail, any tions question the evidence behind consultants’
approach to decision making that involves a more recommendations, services, and products and con-
conscious use of these four sources of evidence can sider how the effects of such may be measured and
be considered to be a member of the family of evaluated, they gain a better understanding of24 Academy of Management Perspectives November
their problems and the nature of the evidence teaching, and administrative demands. Further,
required to make an informed decision. early-career scholars in particular have to focus on
Last, the approach taken in some management getting as many journal articles as possible in the
education programs provides a foundation for best journals. Hence, contributing to EBMgt may
EBMgt. For example, classes that focus on the be more possible for senior rather than junior
methodological strengths and weaknesses of a par- scholars. However, for those inclined toward de-
ticular empirical study and how it might apply veloping EBMgt, there are exciting roles to play.
practically to organizations provide students with So what does EBMgt ask of those scholars in-
some of the analytical tools they need to practice terested in making a contribution? There are
EBMgt. Similarly, management education that many possible roles, and just some examples are
critically analyzes management fads and fashions considered here. Scholars could play a role in
and questions their novelty, supporting evidence, systematically collating the available evidence
and applicability is likewise preparing students to about a given, specific question or problem and
contemplate whether particular practices are then developing ways of judging the quality of
likely to be effective in the contexts in which they pertinent evidence. They might also play a role in
work. More directly relevant are the relatively few “consensus building” by devising and applying ex-
programs that teach students how to conduct sys- plicit methods for developing an agreed interpre-
tematic reviews or rapid evidence assessments. tation and synthesis of evidence when it is equiv-
The process of conducting a review not only pro- ocal, as is often the case. Scholars could learn to
vides in-depth knowledge about the problem or be better knowledge brokers, feeding relevant and
question addressed by the review but also gives critically appraised evidence into organizations,
wider insight into how to make decisions using government policy, and wider political issues such
evidence, critical thinking, and a mindful ap- as those around CEO compensation (Kaplan,
proach to practice. A common misperception is 2008) and the environmental performance of
that systematic reviews provide “the answer.” Al- firms (Ambec & Lanoie, 2008). To practice EBMgt,
though such reviews certainly do provide a much practitioners may need to acquire, assess, adapt,
clearer picture of what is known and not known and apply research evidence to their decisions;
and the boundary conditions of that knowledge, and we note that academic skills and knowledge
undertaking such reviews also provides much can aid this acquisition and use.
deeper insight into the practical problem.
EBMgt therefore represents a way of thinking Point # 4: Managers and Scholars Need to Better
about or approaching organizational problems and Understand How to Conduct and Use Systematic
decisions. As indicated above, this way of think- Reviews
ing or at least elements of it can be found in a wide A general consensus across all fields interested in
range of existing decision-making and analytical evidence-based practice is that a synthesis of evi-
processes. EBMgt is an attempt to incorporate and dence from multiple studies is better than evi-
integrate each of these elements in a conscious, dence from a single study. Often producing erro-
explicit, and mindful way. neous conclusions, single studies almost never
matter in themselves (Hunter & Schmidt, 2004).
Point #3: Academics as Scholars and Educators Are It is the collective body of evidence we need to
Needed to Build EBMgt Supports understand.
The knowledge and skills scholars possess are Systematic reviews have become fundamental
needed to produce, appraise, synthesize, and cre- to evidence-based practice and represent a key
ate access to research evidence. We recognize and methodology for locating, appraising, synthesiz-
have heard many times the concerns of scholars ing, and reporting “best evidence.” It should be
thinking about getting involved in EBMgt. The noted that other fields are very active in system-
average management scholar is already struggling atic reviewing— discussing methods (e.g., Boaz et
to get tenure or promotion and balance research, al., 2006; Greenhalgh & Peacock, 2005; Petti-2009 Briner, Denyer, and Rousseau 25
crew, 2001), providing training (Centre for Re- topics outside medicine. Drawing on the work of
views and Dissemination, 2009; Critical Appraisal Pawson (2006), Denyer and Tranfield (2009) ar-
Skills Programme, 2009; EPPI-Centre, 2009), and gue that well-formulated review questions in man-
developing related techniques such as rapid evi- agement and organization studies need to take
dence assessments (REA Toolkit, 2009) and best into account why or how the relationship occurs
evidence topics (Best BETs, 2009). and in what circumstances. They reformulate
Methodological developments in systematic re- PICO into CIMO for use in the social sciences:
views in medicine have been significant, and the
approach promoted by the Cochrane Collabora- C — Context. Which individuals, relationships,
tion, the worldwide community of clinicians and institutional settings, or wider systems are being
research scientists who conduct systematic re- studied?
views of medical evidence to answer questions I — Intervention. The effects of what event, ac-
critical to the practice of medicine, is widely re- tion, or activity are being studied?
garded as a benchmark. A systematic review is a M — Mechanisms. What are the mechanisms that
replicable, scientific, and transparent approach explain the relationship between interventions
that differs greatly from a traditional literature and outcomes? Under what circumstances are
review in several important ways. these mechanisms activated or not activated?
First, a systematic review rigorously addresses a O — Outcomes. What are the effects of the in-
clearly specified answerable question usually de- tervention? How will the outcomes be mea-
rived from a policy or practice problem: “[T]he sured? What are the intended and unintended
question guides the review by defining which effects?
studies will be included, what the search strategy
to identify the relevant primary studies should be, Denyer and Tranfield (2009, p. 682) provide an
and which data need to be extracted from each example of a question framed with these compo-
study. Ask a poor question and you will get a poor nents: “Under what conditions (C) does leader-
review” (Counsell, 1997, p. 381). ship style (I) influence the performance of project
Systematic reviews in medical science are often teams (O), and what mechanisms operate in the
structured according to the PICO approach: influence of leadership style (I) on project team
performance (O)?”
P — Patient or Problem. For which group is evi-
Second, a broad range of stakeholders often
dence required?
contributes to the development of review ques-
I — Intervention. The effects of what event, ac-
tions and processes (Tranfield et al., 2003) and to
tion, or activity are being studied?
the effective dissemination of review findings to
C — Comparison. What is the alternative to the
appropriate audiences (Petticrew, 2001, p. 100).
intervention (e.g., placebo/different interven-
For example, reviews by the Evidence for Policy
tion)?
and Practice Information Centre (EPPI Centre)
O — Outcomes. What are the effects of the in-
on education topics frequently have short state-
tervention?
ments from teachers, principals, or school gover-
An example of a well-formulated systematic re- nors, providing interpretations of the findings and
view question for a medical problem is provided in suggestions for how these might be implemented.
the Cochrane Handbook: “[Is] a particular anti- Third, extensive searches are conducted of
platelet agent, such as aspirin, [intervention] . . . both published and unpublished studies. The aim
effective in decreasing the risks of a particular is to find all studies relating to the question.
thrombotic event, stroke [outcome] in elderly per- Greenhalgh and Peacock (2005) demonstrated
sons with a previous history of stroke [popula- the limitations of search strategies that focus
tion]?” (Higgins & Green, 2006, p. 62). solely on citation databases. For complex ques-
Clearly this approach is less appropriate to the tions, a systematic search should always use sev-
study of complex questions and multidisciplinary eral methods, including searching electronic data-26 Academy of Management Perspectives November
bases, hand-searching known journals, soliciting assessing the consistency among previous studies;
expert recommendations, and cross-referencing. these tasks are clearly not unique to medicine”
Fourth, criteria for the inclusion of studies are (2001, p. 99).
explicitly determined before the review com- We believe it is unfeasible and undesirable for
mences. This helps ensure that reviews are impar- management research to simply adopt the bench-
tial and balanced, preventing reviewers from in- mark of the Cochrane model or any other field’s
cluding only those studies supporting their approach toward the review process or the hierar-
particular argument. Systematic reviews are not chy of evidence. All academic fields are different.
restricted to papers published in the “top” jour- Which evidence is “best” depends entirely on its
nals. “Gray literature” such as unpublished papers appropriateness to the question being asked (Boaz
and conference reports are often included. This is & Ashby, 2003). If the question is “what effect
deemed necessary and appropriate to help over- does intervention X have on outcome Y?” then a
come publication bias and the file drawer problem meta-analysis of randomized trials may indeed be
(where researchers file away studies with negative the best possible evidence. Similarly, if the ques-
or neutral outcomes as they are more difficult to tion is about tools for personnel selection, meta-
publish). Every study included in the review must analysis of predictive validity studies is likely to be
meet the predetermined criteria specified for the appropriate and may provide relevant evidence for
particular review. practitioners (e.g., Le, Oh, Shaffer, & Schmidt,
Fifth, systematic reviews summarize the find- 2007). For other questions, longitudinal studies or
ings of all the individual studies in a transparent quasi-experiments might be the best evidence
and accessible format. Findings from individual available.
studies are often presented so that “other research- If, on the other hand, the question is “how do
ers, decision makers and other stakeholders can women interpret their role on male-dominated
look behind an existing review, to assure them- boards?” then qualitative data will form the best
selves of its rigor and validity, reliability and ver- evidence. If the question is “why or how does goal
ifiability of its findings and conclusions” (Pawson, setting result in higher team performance?” then
2006, p. 79). Systematic reviews have processes we need theory as well as evidence from which we
for synthesizing multiple studies in order to pro- can infer processes. For other, more complex ques-
vide results that are more than the sum of the tions, of the sort common in management and
parts. As with any method, the approaches chosen organizational studies, we may need to integrate
should be appropriate to the purpose of the study different forms of evidence. Best evidence can be
and the nature of the available data (cf. Noblit & quantitative, qualitative, or theoretical. Quantita-
Hare, 1988; Pawson, 2006; Rousseau et al., 2008). tive and qualitative contributions need to be ap-
Sixth, in relation to final review outcomes, the praised separately in a systematic review using
summarized or synthesized findings are often con- criteria that are relevant to the particular methods
densed into a set of practical conclusions. Where employed in the original studies. We do not be-
numerous studies provide consistent results, sys- lieve that it is possible to judge qualitative re-
tematic reviews might provide reasonably clear search using criteria designed to evaluate quanti-
conclusions about what is known and not known. tative research, and vice versa.
If, on the other hand, the review identifies gaps or We also believe that there are alternative ap-
inconsistent findings, practical conclusions are proaches to meta-analysis as a mode of synthesis.
more nuanced or circumspect and raise questions Elsewhere (Rousseau, Manning, & Denyer, 2008)
for future research. we have argued that synthesis can involve aggre-
Given its success in medicine, the systematic gation, integration, interpretation, or explana-
review methodology has been adopted in many tion. The most appropriate method of synthesis
fields. As Petticrew argued, “Systematic review is depends on the types of evidence reviewed, which
an efficient technique for hypothesis testing, for in turn depend on the review question. It seems
summarizing the results of existing studies, and for likely, given the idiosyncratic features of manage-2009 Briner, Denyer, and Rousseau 27
ment and organization studies, that a range of profit organizations (see Table 3 for examples of
approaches will be required. systematic reviews, rapid evidence assessments,
In terms of practical utility, it is important to and other forms of synthesis produced by col-
note that systematic reviews never provide “an- leagues at our own institutions). This list is not
swers.” What they do is report as accurately as representative or comprehensive, but it does show
possible what is known and not known about the that practitioners in a range of organizations are
questions addressed in the review. The Cochrane attempting to incorporate evidence into their de-
Handbook (Higgins & Green, 2006, p. 167) is cisions. Reviewing the outcomes of the manage-
careful to point out that although “the primary rial decisions based on such reviews exemplifies a
purpose of the review should be to present infor- more appropriate source of evidence than the ac-
mation, rather than offer advice, the discussion ademic literature used exclusively in the RBK
and conclusions should be to help people to un- review.
derstand the implications of the evidence in rela- Alternatively, an evaluation of EBMgt might
tionship to practical decisions.” investigate the outcomes resulting from applica-
Academic research is only one sort of evidence, tions of specific established research-based princi-
but it has the advantages of greater rigor and ples (e.g., Locke, 2009). For example, one might
independence. Yet EBMgt is about drawing on a consider the uses to which the well-established
range of sources and types of evidence such as principles of goal setting have been applied
financial information, monitoring data, surveys, (Locke & Latham, 2002) and evaluate the body of
public opinion, practical experience, consultants, evidence (published and unpublished) that bears
anecdotes, and internal organizational research. on the effectiveness of practitioner applications of
Any of these sources of information can be useful goal-setting principles. Thus, one can examine
and valid in making a decision, depending on the the effectiveness of EBMgt in the general sense,
decision and its circumstances. following the commissioning of a core knowledge
We argue above that systematic reviews in product related to EBMgt practice, the systematic
management and organizational studies have a set review. Or one can examine the effectiveness of a
of features that can be used to assess their quality. particular practice that is purported to be evi-
We now apply our six quality criteria to the RBK dence-based (that is, subject to managerial re-
article (see Table 2). It should be apparent that search, such as 360-degree feedback or pay-for-
fundamental problems exist with the RBK article performance). Such global or particular
as a systematic review. From the need to reformu- assessments constitute two of many useful ap-
late its question and its limited, unsystematic se- proaches to assessing the effectiveness of practices
lection criteria to its omission of essential docu- related to EBMgt.
mentation, the RBK review falls short of the
requirements for an informative systematic re- Conclusion
E
view. As we note in our evaluation of RBK’s BMgt is already happening in a variety of ways.
review, framing the initial question itself is diffi- Yet, as a new concept, its uptake and evolution
cult, requiring thoughtful vetting and often the are in fits and starts. Its new forms may be at
involvement of various stakeholders differing in first unrecognizable, or resemble the proverbial
expertise and perspective (Rousseau et al., 2008). old wine in new bottles and vice versa. Careful
Nonetheless, there are several alternative ways discernment is needed to appraise where it stands.
of answering RBK’s question about the evidence As has happened in other fields, there will be
for EBMgt. First and foremost is to examine how more, new, and different ways in which it will be
existing systematic reviews have been put to use. explored and practiced. Empirical work is also
As already noted, systematic reviews are a key part required to address the key question raised in the
of evidence-based practice, and a number of sys- RBK review: Does practicing EBMgt improve the
tematic reviews have been commissioned by com- process and outcome of decision making in orga-
mercial firms, public sector agencies, and not-for- nizations?28 Academy of Management Perspectives November
Table 2
An Evaluation of the RBK Systematic Review
Suggested Ways of Overcoming Limitations Based on
Systematic Review Checklist Some Limitations of the Reay et al. Approach to Systematic Review Standard Systematic Review Methodology
Does the review explore a Three questions are addressed in the review: Rather than tackling this topic with a single overarching
clearly specified, 1. Is there a substantial body of evidence in the management literature concerning review, it would be more effective to conduct several
answerable question, the concept of evidence-based management? systematic reviews that would adopt different
which is usually derived 2. What is the quality of the evidence (where it exists) regarding evidence-based methods to address different and more specific
from a policy or practice management? questions about EBMgt.
problem? 3. Is there evidence that employing evidence-based management will improve The question could be delimited by intervention type.
organizational performance? For example, How do managers (at all levels and in
The review incorporates contributions from all management study settings, all industries) [context] access, make sense of and
regardless of industry and inclusive of all levels of management. utilize [mechanisms] the findings of reviews of
The review covers the entire range of work in the area of evidence-based research evidence (systematic or otherwise)
management, where the range is anchored by theoretical contributions on one [intervention], and does this lead to more effective
end and applied/intervention studies on the other. The breadth of the review decision making [outcomes]?
makes comparison difficult.
It is not clear, for example, what would constitute a “substantial” body of
evidence, what “employing” evidence-based management means, how an
“improvement” would be identified, or the how the “organizational
performance” is defined. Each term is open to multiple interpretations, making
both searching for and interpreting evidence and ultimately answering the
questions very difficult.
Were a broad range of The review questions, procedures, and report were developed exclusively by the For the question outlined, a review consultation group
stakeholders involved in authors. comprising scholars working in evidence-based
the review? management, research utilization, and other related
fields could be created. It would also include
managers and policy makers who had commissioned
and used reviews of research evidence. Librarians
and information scientists would also be included. It
is likely that the initial question outlined above
would be further refined by the group.
Were extensive searches Searches were conducted in EBSCO, ABI Inform, and Web of Science (excluding For the second question outlined above a set of search
conducted of both medicine) for the search terms “evidence-based management” and its variants. terms would be devised, such as:
published and unpublished The search terms do not include all members of the family of approaches that (reviews OR evidence assessments OR syntheses OR
studies? comprise evidence-based management and would not have picked up any systematic review OR meta analyses, etc.)
evidence about these. In addition, terms were included that were not relevant to (knowledge OR research OR ideas OR evidence, etc.)
EBMgt and, in the case of the search term “best practice,” are in fact the (utilization OR transfer OR adoption OR
antithesis of EBMgt. dissemination OR exploitation OR commercialization
The paper excludes evidence-based practice in related fields. For example, there is OR assimilation OR absorption OR implementation, etc.)
a significant amount of published material in the field of education that (decision OR judgment OR recommendation, etc.)
addresses management issues such as school leadership. Similarly, because the Please note: This is just a small selection of the terms
medicine database was excluded, papers in health care management may have that should be used.
been missed. Simple operators such as truncation characters or the core
The paper makes no reference to some of the key papers and debates published components of keywords would be used to ensure that
on the subject in journals such as the British Journal of Management, Journal of we covered all the alternatives; “review*” would cover
Management Studies, and Organization Studies (see references). It is unclear reviews OR reviewing, etc.
whether these papers were identified and not used or not found in the search. Boolean logic operators would be used to combine the
Only five articles were found from cross-referencing and citation searches. Very terms, such as (review* OR evidence assessment* OR
often in systematic reviews this is a key way of finding relevant evidence. synthes* OR systematic review* OR meta analys*, etc.)
Book chapters, working papers, commercial research, government publications, AND (utili*ation OR transfer OR adopt* OR disseminat*
and other possible sources of evidence were not included. OR exploit* OR commerciali* OR assimilate* OR
absorption OR implement*, etc.).
Using citation search and cross-referencing is essential.
A search of the “gray” literature would be crucial to
locating the large number of systematic reviews on
management issues that have been commissioned by
organizations (see Table 3).2009 Briner, Denyer, and Rousseau 29
Table 2
Continued
Suggested Ways of Overcoming Limitations Based on
Systematic Review Checklist Some Limitations of the Reay et al. Approach to Systematic Review Standard Systematic Review Methodology
Were prespecified It is unclear whether or how the prespecified selection criteria were applied. In A table of inclusion and exclusion criteria should be
relevance and quality other words, there was no attempt to establish in principle what would included to delimit the review.
criteria for the selection/ constitute research evidence relevant to the three review questions. A fit-for-purpose quality appraisal approach would be
inclusion of studies The authors largely identify conceptual papers that explicitly mention EBMgt (e.g., adapted from the freely available checklists (e.g.,
created and made explicit Rousseau, 2006). It is not clear how such papers provide evidence relevant to CASP, 2009). Some journals also publish
before the review any of the three research questions. comprehensive sets of quality appraisal questions
commenced? The authors also include papers that do not explicitly mention EBMgt but are on that could be adapted for use in a systematic review.
research utilization (e.g., Beyer and Trice, 1982). However, they exclude the See, for example, the evaluation criteria for
rest of the body of literature on research utilization. They include a review qualitative papers produced by the Journal of
article by Ployhart (2006) on staffing best practice and again exclude the whole Occupational and Organizational Psychology.
body of knowledge on this topic. The rationale for these decisions is not made Different sets of criteria would be used depending on
explicit. the nature of the data and also the review question.
The selection criteria do not allow the inclusion of articles that are about evidence-
based management but do not use that label.
The paper adopts a hierarchy of evidence used in one setting (medicine) to judge
the quality of research in relation to specific types of medical systematic review
questions. This is inappropriate.
Does the review The evidence base is not made accessible to the reader. Very few of the 144 coded A descriptive analysis including specific examples and
summarize the findings articles are referred to in the manuscript. detailing the study characteristics would be
from all of the individual Some of the analyses performed (e.g., publications by management area or year conducted, along with an audit trail explicitly
studies in a transparent of study) are not relevant to the review questions. demonstrating the link between the review findings
and accessible format, and and the review conclusions.
are the findings Key emerging themes could be highlighted relating to
synthesized into a the ways in which managers access, make sense of,
coherent whole? and utilize the findings of reviews of research
evidence. Whatever overarching categories emerge
for the tabulation, these should also have a detailed
audit trail back to the core findings to justify and
ground the conclusions.
Where possible, details of both the quantitative and
qualitative evidence would be tabulated showing
what links, if any, exist between the use of reviews
of research evidence and the effectiveness of
decision making.
Does the study identify The review appears to fall into the trap of reverting to a traditional literature The results to the question posed above would provide:
what is known and what review. The authors employ a narrative approach to describe and comment on a (1) A summary explaining how managers (at all levels
is not known relating to set of papers that they have selectively chosen to support their argument. and in all industries) [context] access, make sense
the questions posed, and For example, the first review question is “Is there a substantial body of of, and utilize [mechanisms] the findings of reviews
does it provide a set of evidence in the management literature concerning the concept of evidence- of research evidence (systematic or otherwise).
practical conclusions for based management?” Given this, the review should focus on the meaning of (2) The evidence showing the effect of reviews of
policy, practice, and future “substantial” and then say whether or not the body of evidence found is research evidence (systematic or otherwise) on
research? substantial. The review fails to do this. More fundamentally, it is not made clear decision making [outcomes].
why substantiality of a body of evidence is relevant. The aim would be help people to understand the
implications of the findings in relation to practical
decisions. For example, the review would be of value
to an organization considering commissioning a
systematic review or a scholar wondering if
conducting a systematic review will have any effect
on organizational decision making.
EBMgt will help focus management research on such a focus. Instead we note that a wealth of valu-
addressing the questions, problems, and challenges able and relevant evidence exists to be mined for
that managers and other practitioners face. We do practical use. The use of systematic reviews will also
not contend that all management research requires help management scholars identify in a more robust30 Academy of Management Perspectives November
Table 3
A Small Sample of Research Reviews Commissioned by Organizations and Government Agencies
Commissioning
Title Date Authors Organization Focus and Impact Publications
External Knowledge: A Review of 2005 Bessant, J., Phelps, B.,
Department for Trade Presented to the U.K. government’s Leadership Available from the Advanced
the Literature Addressing the & Adams, R. and Industry (U.K.) and Management Advisory Panel, which was Institute of Management
Role of External Knowledge and particularly interested in using it to identify the (www.aimresearch.org)
Expertise at Key Stages of distinctive phases of organizational growth and Phelps et al. (2007)
Business Growth and factors that might affect different approaches
Development to leadership and management.
No Going Back: A Review of the 2005 Buchanan, D. A., National Health Authors were members of the Research Into Buchanan et al. (2005)
Literature on Sustaining Ketley, D., Gollop, R.,Service Practice team within the U.K.’s NHS Buchanan et al. (2007)
Organizational Change Jones, J. L., Lamont, Modernization Modernization Agency. It explores factors
S. S., Neath, A., & Agency (U.K.) affecting the diffusion and sustainability of
Whitby, E. new working practices. A diagnostic called the
“sustainability wheel” was created to help
clinical and managerial teams identify factors
threatening improvement sustainability.
Internal briefings were run for service
improvement leads based on the findings.
Networking and Innovation: A 2004 Pittaway, L., Department for Trade The review synthesizes research linking the Available from the Advanced
Systematic Review of the Robertson, M., Munir, and Industry (U.K.) networking behavior of firms with their Institute of Management
Evidence K., Denyer, D., & innovative capacity. It presents evidence on the (www.aimresearch.org)
Neely, A. principal benefits of networking as identified in Pittaway et al. (2004)
the literature.
Human Capital Management: A 2008 Parry, E. Ceridian Consulting The review output was a model of factors Report owned by the
Systematic Review of the affecting human capital and its impact on commissioning organization.
Literature performance and a list of possible measures.
The model was used to develop a diagnostic
tool to maximize human capital.
Review of the Effectiveness and 2009 Hillage, J., Rick, J., National Institute for The review led to guidance for primary care Available from NICE
Cost Effectiveness of Pilgrim, H., Jagger, Health and Clinical workers and employers on the management of (www.nice.org.uk)
Interventions, Strategies, N., Carroll, C., & Excellence (U.K.) long-term sickness and incapacity.
Programmes and Policies to Booth, A.
Reduce the Number of
Employees Who Move From
Short-Term to Long-Term
Sickness Absence and to Help
Employees on Long-Term
Sickness Absence Return to Work
Mitigating Risks to Health and 2009 Parry, E., & Paddock, Ministry of Defence This review examined the practices used to Report owned by the
Well Being: Provision of Welfare S. (U.K.) alleviate the negative impacts of separation on commissioning organization.
for Families During Separation the families of personnel in the armed forces
and compared practice in the U.K. armed
forces with that in overseas militaries and in
civilian organizations. This review was used as
a basis for future empirical work and for policy
development in this area.
way what is known and not known about a given kinds of evidence in the decisions they make. It is
problem or question. At its core, EBMgt helps man- fundamentally a process of informed practitioner
agers focus on the need for professional reflection judgment. As such, EBMgt can aid critical thinking
and judgment about the validity and value of various and the appraisal of all forms of evidence.2009 Briner, Denyer, and Rousseau 31
References complex evidence: Audit of primary sources. British Med-
ical Journal, 331, 1064 –1065.
Ambec, S., & Lanoie, P. (2008). Does it pay to be green? A Higgins, J. P. T., & Green, S. (updated September 2006).
systematic overview. Academy of Management Perspec- Cochrane handbook for systematic reviews of interventions.
tives, 22, 45– 62. The Cochrane Library, vol. 4.2.6. Available from http://
Best BETs (2009). Retrieved October 5, 2009, from http:// www.cochrane.org/
www.bestbets.org/ Hunter, J. E. & Schmidt, F. L. (2004). Methods of meta-
Boaz, A., & Ashby, D. (2003). Fit for purpose? Assessing analysis: Correcting error and bias in research findings.
research quality for evidence-based policy and practice. Newbury Park, CA: Sage.
ESRC UK Centre for Evidence-Based Policy and Prac- Johns, G. (2006). The essential impact of context on orga-
tice (Working Paper 11). Available from: http:// nizational behavior. Academy of Management Review, 31,
kcl.ac.uk/content/1/c6/03/46/04/wp11.pdf 386 – 408.
Boaz, A., Ashby, D., Denyer, D., Egan, M., Harden, A., Kaplan, S. (2008). Are U.S. CEOs overpaid? Academy of
Jones, D. R., Pawson, R., & Tranfield, D. (2006). A Management Perspectives, 22, 5–20.
multitude of syntheses: A comparison of five approaches Kovner, A. R., Elton, J. J., & Billings, J. D. (2005). Evi-
from diverse policy fields. Evidence and Policy, 2, 479 – dence-based management. Frontiers of Health Services
502. Management, 16, 3–24.
Briner, R. B. (1998). What is an evidence-based approach to Le, H., Oh, I., Shaffer, J., & Schmidt, F. (2007). Implica-
practice and why do we need one in occupational psy- tions of methodological advances for the practice of
chology? Proceedings of the 1998 British Psychological So- personnel selection: How practitioners benefit from
ciety Occupational Psychology Conference, 39 – 44. meta-analysis. Academy of Management Perspectives, (21)
Buchanan, D. A., Fitzgerald, L., & Ketley, D. (Eds.). (2007). 3, 6 –15.
The sustainability and spread of organizational change: Mod- Lemieux-Charles, L., & Champagne, F. (2004). Using
ernizing healthcare. London: Routledge. knowledge and evidence in healthcare: Multidisciplinary per-
Buchanan, D., Fitzgerald, L., Ketley, D., Gollop, R., Jones, spectives. Toronto: University of Toronto Press.
J. L., Saint Lamont, S., Neath, A., & Whitby, E. (2005). Locke, E. A. (Ed.). (2009). Handbook of principles of organi-
No going back: A review of the literature on sustaining zational behavior: Indispensable knowledge for evidence-
organizational change. International Journal of Manage- based management. New York: Wiley.
ment Reviews, 7, 189 –205. Locke, E. A., & Latham, G. P. (2002). Building a practically
Campbell Collaboration. (2009). Retrieved October 5, useful theory of goal setting and task motivation: A
2009, from http://www.campbellcollaboration.org/. 35-year odyssey. Annual Review of Psychologist, 57, 705–
Centre for Reviews and Dissemination. (2009). Retrieved 717.
October 5, 2009, from http://www.york.ac.uk/inst/crd/ Lowenstein, R. (2006, January 22). When business has ques-
methods.htm tions, Drucker still has answers. New York Times, Bu 7.
Cochrane Collaboration. (2009). Retrieved October 5, Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography:
2009, from http://www.cochrane.org/ Synthesizing qualitative studies. London: SAGE Publica-
Counsell, C. (1997). Formulating questions and locating tions.
primary studies for inclusion in systematic reviews. An- Pawson, R. (2006). Evidence-based policy: A realist perspec-
nals of Internal Medicine, 127, 380 –387. tive. London: SAGE Publications.
Critical Appraisal Skills Programme. (2009). Retrieved Oc- Petticrew, M. (2001). Systematic reviews from astronomy to
tober 5, 2009, from http://www.phru.nhs.uk/Pages/PHD/ zoology: Myths and misconceptions. British Medical Jour-
CASP.htm nal, 322, 98 –101.
Dane, E., & Pratt, M. G. (2007). Exploring intuition and its Pfeffer, J., & Sutton, R. I. (2006). Hard facts, dangerous half-
role in managerial decision making. Academy of Manage- truths, and total nonsense: Profiting from evidence-based man-
ment Review, 32, 33–54. agement. Cambridge, MA: Harvard Business School Press.
Denyer, D., & Tranfield, D. (2009). Producing a systematic Phelps, R., Adams, R., & Bessant, J. (2007). Life cycles of
review. In D. A. Buchanan & A. Bryman (Eds.), The growing organizations: A review with implications for
SAGE handbook of organizational research methods (pp. knowledge and learning. International Journal of Manage-
671– 689). London: SAGE Publications Ltd. ment Reviews, 9, 1–30.
Denyer, D., Tranfield, D., & van Aken, J. E. (2008). De- Pittaway, L., Robertson, M., Munir, K., Denyer, D., &
veloping design propositions through research synthesis. Neely, A. (2004). Networking and innovation: A sys-
Organization Studies, 29, 393– 413. tematic review of the evidence. International Journal of
Drucker, P. F. (1966). The effective executive. New York: Management Reviews, 56(34), 137–168.
HarperCollins. REA Toolkit. (2009). Retrieved October 5, 2009, from http://
EBMgt Collaborative. (2009). Retrieved October 5, 2009, www.civilservice.gov.uk/networks/professional/gsr/resources/
from http://wpweb2.tepper.cmu.edu/rlang/ebm_conf/ gsr-rapid-evidence-assessment-toolkit.aspx
EPPI-Centre. (2009). Retrieved October 5, 2009, from Rousseau, D. M. (2005). Evidence-based management in
http://eppi.ioe.ac.uk/cms/ health care. In C. Korunka & P. Hoffmann (Eds.),
Greenhalgh, T., & Peacock, R. (2005). Effectiveness and Change and quality in human service work. Munich:
efficiency of search methods in systematic reviews of Hampp Publishers.You can also read
