Evidence-Based I-O Psychology: Not There Yet
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Industrial and Organizational Psychology, 4 (2011), 3–22.
Copyright © 2011 Society for Industrial and Organizational Psychology. 1754-9426/11
FOCAL ARTICLE
Evidence-Based I–O Psychology:
Not There Yet
ROB B. BRINER
University of London
DENISE M. ROUSSEAU
Carnegie Mellon University
Abstract
Evidence-based practice is now well established in several fields including medicine, nursing, and social policy.
This article seeks to promote discussion of whether the practice of industrial–organizational (I–O) psychologists
is evidence based and what is needed to make I–O psychology an evidence-based discipline. It first reviews
the emergence of the concept of evidence-based practice. Second, it considers the definitions and features of
evidence-based practice, including evidence-based management. It then assesses whether I–O psychology is
itself an evidence-based discipline by identifying key characteristics of evidence-based practice and judging the
extent these characterize I–O psychology. Fourth, some key strategies for promoting the use of evidence in
I–O psychology are considered: practice-oriented research and systematic reviews. Fifth, barriers to practicing
evidence-based I–O psychology are identified along with suggestions for overcoming them. Last is a look to the
future of an evidence-based I–O psychology that plays an important role in helping consultants, in-house I–O
psychologists, managers, and organizations become more evidence based.
Most industrial–organizational (I–O) psy- in journal articles or workforce facts and
chologists would confidently claim that metrics collected through practice.
their practice is based on evidence—in Look a bit deeper, however, and things
particular, psychological research findings. aren’t quite so straightforward. Practitioners
And, in a general sense, that’s more or less of all stripes—from the snake-oil salesper-
true. The founders of our field, including son with a cure-all remedy to the fam-
James McKeen Cattell and C. S. Myers, ily physician or the personal trainer to
originated the application of systematic the psychic who communicates with the
research to workplace issues and in doing dead—claim to have evidence for what
so established psychological science itself they do. Claims to use evidence are there-
in both the United States and Britain. Most fore meaningless in themselves. However,
of what we do in I–O psychology draws on the hallmark of any profession is the exis-
or is at least informed by some type of evi- tence of an agreed-upon core of knowledge
dence whether that be research published and means for its continued generation and
refinement (Friedson, 1986). Unlike com-
mon sense or general knowledge claims,
Correspondence concerning this article should be it’s the nature and quality of the par-
addressed to Rob B. Briner.
E-mail: r.briner@bbk.ac.uk ticular evidence a profession’s practition-
Address: Department of Organizational Psychol- ers use that distinguishes them from the
ogy, Birkbeck College, University of London, Malet layperson—or the snake-oil salesperson. In
Street, London WC1E 7HX, United Kingdom.
We are very grateful to three anonymous reviewers the case of professions such as I–O psychol-
for their comments and suggestions. ogy, medicine, education, or engineering,
34 R.B. Briner and D.M. Rousseau much of their value and legitimacy depends majority of respondents (who were largely on the extent a scientific evidence base practitioners) held the opinion that prac- informs and is used in practice. tice is ahead of research in 14 content In many areas of I–O psychology, large areas, including coaching, talent manage- bodies of reasonable quality evidence are ment, and employee relations, whereas continually refined and critically evaluated. science was ahead in only two (measure- The expanded use of meta-analysis in I–O ment and statistics, and job analysis). In five psychology indicates the value it places on areas, practitioners and nonpractitioners systematic evidence, cumulative research held opposite views. Practitioners saw prac- findings, and critical analyses (e.g., Judge, tice ahead and nonpractitioners saw science Heller, & Mount, 2002; Judge & Illies, ahead in recruitment, performance manage- 2002). Similarly, the value of the structured ment, organizational culture, training and interview over its unstructured counterpart development, and employee engagement. has been affirmed repeatedly over decades A plurality shared the view that science is (McDaniel, Whetzel, Schmidt, & Maurer, behind practice in its choice of research 1994); the I–O psychology practitioner topics and that practice-relevant research is probably one of the more consistent was sparse. Finding perceived gaps on both voices in industry to buck the populist sides is not so surprising. In I–O psy- tide that continues to favor unstructured chology scientists and practitioners each techniques. On the other hand, I–O prize their own knowledge sources over psychology does not always resist more the other’s, raising concern regarding the ‘‘pop-psychology’’ topics such as emotional quality of the interface between the two. In intelligence. Although introduced to social turn, these findings raise questions about science through work by Mayer and the extent the science and practice of Salovey (1997), the type of claims made I–O psychology is synergistic in a fash- in Daniel Goleman’s (1998) popular book ion that would promote evidence-based and other popular accounts of the topic are practice. clearly questionable (e.g., Cherniss, 2010; Another expression of this concern can Locke, 2005). be found in many of this journal’s focal In other areas, the evidence familiar to articles, as they are precisely about exam- I–O psychologists may appear more static, ining the concepts and evidence under- taking the form of received wisdom. For lying our practice in a range of areas example, consider the notion that accurate including the identification of discrimi- feedback is good (i.e., at least if task nation in workplace evaluations (Landy, related; cf., Kluger & DeNisi, 1996). Still, 2008), executive coaching (McKenna & uncritical reliance on received wisdom Davis, 2009), job performance ratings is problematic: Cognitive and educational (Murphy, 2008), employee engagement research demonstrates that feedback too (Macey & Schneider, 2008), executive early in the learning process can undermine selection (Hollenbeck, 2009), and leader- the mental processes needed to acquire ship development (McCall, 2010). These certain skills (Merrill, Reiser, Merrill, & articles have created lively, important, and Landes, 1995; VanLehn, Siler, Murray, healthy debate. Yamauchi, & Baggett, 2003). Given that Many professions have expressed such new information replaces or refines existing concerns and pursued ways that evidence information, the evidence base we use is can better inform practice. One of the inevitably a work in progress. most recent and widespread ideas used I–O psychologists have expressed con- to frame solutions to this problem is the cerns about the existence, relevance, and subject of this focal article: evidence-based use of evidence in the profession. A recent practice. For some years, evidence-based survey of SIOP members (Silzer, Cober, practice has evolved as a way of identifying Erickson, & Robinson, 2008) revealed that a and developing techniques and processes
Evidence-based I–O psychology 5
that practitioners can use to incorporate medicine and its allied disciplines, such
evidence from various sources into their as nursing, have engaged the challenge of
everyday work. Our purpose here is to pro- using scientific evidence to better inform
mote discussion and further development practice.
of the practice of I–O psychology as an In medicine and nursing, the notion of
evidence-based discipline. being evidence based is now well estab-
This article is structured as follows. lished. It forms part of the basic training
It reviews the emergence of evidence- of nurses, physicians, and other profes-
based practice in medicine, social pol- sions allied to medicine. Medical research
icy, and more recently management, and and its funders, in addition to producing
considers the definitions and features of new basic science, also put considerable
evidence-based practice, including that of resources into research on effective prac-
evidence-based management (EBMgt). It tice, including how to best treat specific
assesses whether I–O psychology is itself types of patients, as well as in conduct-
an evidence-based discipline by identify- ing systematic reviews of research litera-
ing key characteristics of evidence-based ture to answer practice-relevant questions.
practice and gauging the extent these char- Systematic reviews are essentially litera-
acterize the practice of I–O psychology. By ture reviews that address a very specific
identifying the key contributions of practice- review question using an explicit, system-
oriented research and systematic reviews to atic methodology to identify, select, and
evidence-based practice, this article details pull together findings from a range of studies
how these applications promote evidence to draw conclusions about what is known
use in practice, and describes barriers to and not known about the question. (Meta-
evidence-based approaches to I–O psy- analyses are one type of systematic review.)
chology practice along with suggestions to Systematic reviews are in essence pieces of
overcome them. Last, we look to the future research on existing research.
and consider the prospects for an evidence- The findings of both patient-oriented
based I–O psychology that helps managers studies and systematic reviews are then
and organizations themselves become more translated into tools and decision-making
evidence based. aids such as checklists and patient-care
protocols used by medical clinicians. Later,
Emergence of Evidence-Based we will talk about how the ‘‘functional
Practice and Management I–O psychology equivalents’’ of these can
The notion of using scientific evidence to promote our own evidence-based practice.
inform professional practice is not new. At In the late 1990s, the term ‘‘evidence
the same time, neither is the observation based’’ became paired with other nonmedi-
that from medicine to management much cal disciplines and practice areas, including
practice is not related to evidence in education, social work, criminology, and
any significant way. This tension found government policy making. Over a decade
expression in a British Medical Journal ago, Briner (1998) made what appears to
editorial reporting, ‘‘only about 15% of be the first attempt to suggest that some
medical interventions are supported by ideas from evidence-based medicine could
solid scientific evidence’’ (Smith, 1991, be adapted to the practice of organizational
p. 798).1 This article marked a turning psychology. These same ideas have been
point in the development of evidence-based applied more recently to the practice of
medicine. More than other professions, management. In 2006, two key publications
(Pfeffer & Sutton, 2006; Rousseau, 2006)
stimulated discussions of EBMgt and how it
1. It may be possible and potentially useful to might be used in business school teaching
make a similar estimate of the percentage of
I–O psychology practices that are based on solid (Rousseau & McCarthy, 2007), its limita-
scientific evidence. tions and potential dangers (Learmonth &6 R.B. Briner and D.M. Rousseau
Harding, 2006; Morrell, 2008), the evi- What Is Evidence-Based Practice?
dence for EBMgt (Reay, Berta, & Kohn,
As mentioned, all practitioners claim
2009), and its meaning and definition
to have evidence for their practice.
(Briner, Denyer, & Rousseau, 2009).
Nonetheless, evidence-based practice is a
In the last 2 decades, considerable soul
particular approach or more accurately
searching by management scholars over
a set of approaches to incorporate evi-
the research–practice ‘‘gap’’ or ‘‘divide’’
dence into practice decisions. In medicine,
has raised difficult questions about why for example, this means ‘‘integrating indi-
the gap exists, how to bridge it, and the vidual clinical expertise with the best
value and purpose of management research available external clinical evidence from
itself (Hambrick, 1994; Rynes, Brown, & systematic research’’ (Sackett, Richardson,
Colbert, 2002). Although I–O psychol- Rosenburg, & Haynes, 1997, p. 2) in making
ogy is not immune to such concerns, in decisions about patient care. Three aspects
other respects I–O psychology would seem of this definition need to be highlighted.
to have avoided the criticisms levied on First, evidence-based practice integrates
other social sciences (Anderson, Herriot, & the practitioner’s expertise and external
Hodgkinson, 2001). Anderson and col- evidence from research. Both sources of
leagues used the label ‘‘pedantic science’’ knowledge are vital. Second, it is about
to describe those research domains driven trying to obtain and use the best available
by theoretical concerns and fastidious ana- evidence even if ultimately determined to
lytics while ignoring real-world issues. be inconsistent or rejected as irrelevant.
I–O psychology might seem an unlikely Using evidence does not mean slavishly
candidate for the pedantic label. Even when following it, acting only when there is
conducted by university-based scholars, good evidence or doing nothing if there is
I–O psychology research is rooted in issues none. Rather, it is a more active process of
and problems arising in organizational examining and evaluating the best of what
settings. Given that its graduates work in is there and applying it along with other
industry as consultants or in-house I–O sources of information, such as situational
psychologists, our discipline would seem facts, to decision making. Some research
to be an exemplar of ‘‘evidence-based’’ evidence may be more readily converted
practice. The research-intensive training of into practical knowledge, although both
I–O psychologists and our often close-to- basic scholarly and more applied evidence
practice research should make reliance on can contribute to evidence-based practice.
research evidence almost second nature. Third, it uses systematic reviews to assess
Indeed, it is probably no coincidence that all available and relevant evidence rather
many advocates of EBMgt (including the than relying on single studies.
present authors) are also I–O psychologists EBMgt has been defined in several
(e.g., Wayne Cascio, Edwin Locke, and Ed different ways but most definitions draw
Lawler). on ideas of evidence-based practice found
So how can we explore the truth of I–O in medicine and elsewhere. One recent
psychology’s claim to engage in evidence- definition of EBMgt (Briner et al., 2009,
based practice? On one hand, I–O psy- p. 19) is as follows:
chology seems to embody Anderson et al.’s
(2001) ideal of a ‘‘pragmatic science,’’ Evidence-based management is about
addressing questions of theoretical and making decisions through the consci-
practical relevance in a methodologically entious, explicit, and judicious use of
sound manner. Yet, if we use the nature four sources of information: practitioner
of evidence-based practice as the starting expertise and judgment, evidence from
point in specifying our criteria, we would the local context, a critical evaluation
draw a different conclusion. of the best available research evidence,Evidence-based I–O psychology 7
and the perspectives of those people who sources, it may be judged to be of poor
might be affected by the decision. quality and therefore not that usable or
relevant. Third, even where the quantity
The conscientious use of the four and quality of information is relatively bal-
sources of information means that an anced across sources, it may be that one
EBMgt approach involves paying careful highly pertinent piece ‘‘trumps’’ others. In
and sustained attention to sources of what a safety-critical organization, for example,
can be potentially different, conflicting, even where nearly all the information sug-
and sometimes difficult-to-interpret infor- gests that a particular intervention may be
mation. Being explicit means using informa- effective, a very small piece of information
tion from each source in a clear, conscious, implying that it also increases errors may
and methodical way such that the roles be enough to push the decision away from
played by all the information in the final what most of the evidence would suggest.
decision are understood. And being judi- Evidence-based practice across various
cious involves using reflective judgment to fields uses similar approaches intended to
evaluate the validity and relevance of the improve the process and consequences of
information from each source. Evidence and decision making by collecting and critically
information is critically evaluated in relation analyzing evidence from a number sources
to the practice context and problem. and then integrating it to make a practical
Take the example of a senior HRM decision or solve a problem (e.g., Gough,
practitioner in a large firm who believes Kiwan, Sutcliffe, Simpson, & Houghton,
the organization has a problem with high 2003; Soukup, 2000).
absence levels and wants to intervene to
reduce the absence rate. Table 1 provides
How Evidence-Based Is
examples of the sorts of information
I–O Psychology?
that the practitioner may compile and
critically evaluate to decide on what kind So, how evidence based are we? To be
of intervention, if any, is likely to be asked this question can feel a bit unnerv-
most effective. After this information and ing or even insulting. I–O psychology is a
evidence is compiled, the next stage is to science after all. We all know how to read
integrate the different sources of evidence. journals and do research. We understand
The decision takes place at the intersection scientific principles and can distinguish
of these four information sources. Exactly good research from bad. We can interpret
how these sources of information are and apply findings from studies. We eval-
integrated depends on the context and who uate, measure, and assess what we do as
is making the decision. It is likely that they we go along. Although all these statements
will vary in several ways. may be broadly true, they don’t reveal how
First, varying quantities of informa- evidence based we actually are as profes-
tion are available from each source. For sionals. Instead, these statements express
example, little may be known about the some of our principles and aspirations. Such
perspectives of those who may be affected statements may be our expectations, but
by the intervention, but the practitioners they can differ from what we actually do.
involved may have much expertise in and We now turn to examine the extent
experience with the problem. Very strong to which characteristics of evidence-based
asymmetries may lead to decisions biased practice identified above are present in
toward those sources of information of the practice of I–O psychology. Note that
which there simply are more. A second no systematic study exists on the actual
way in which these sources of informa- practice of I–O psychologists, that is, the
tion will vary is in relation to their quality, consultants, in-house psychologists, and
validity, and reliability. Although there may others working in industry who hold I–O
be plenty of information from one of these psychology degrees either at the master’s or8 R.B. Briner and D.M. Rousseau
Table 1. Examples of Information From Each of the Four Sources Relevant to Intervening
in the Presenting Problem of High Absence
Practitioner expertise and judgment Evidence from the local context
• Have I seen this before? • What actually is the absence rate?
• What happened? • What type of absences and where?
• What are my beliefs about the causes of • What are local explanations for
absence? absence?
• What’s worked in the past and why? • Internal research (e.g., surveys)
• What are my hunches? • What absence management is currently
• What do I think are the causes and in place, and is it working?
possible solutions? • What do managers think is going on?
• Is this situation occurring elsewhere? • What are the possible costs and benefits
• How relevant and applicable is my of interventions? Is it worth intervening
experience? here?
• What is happening or what is going to
Critical evaluation of best available research happen in the organization or outside
evidence it that might be affecting absence?
• What are the average rates of absence
in my sector and location—Is the Perspectives of those who may be affected
absence rate here ‘‘high’’? by intervention decision
• What does systematically reviewed • How do employees feel about the
research evidence suggest to be the proposed interventions?
major causes of absence? • Do they see downsides or unintended
• How relevant and applicable is that negative consequences?
evidence here? • How do managers feel about these
• What does research evidence from interventions?
systematic reviews suggest as effective • How practical or workable do those
interventions? responsible for implementing the
• How well might the interventions the interventions feel?
research describes work here? • What alternative explanations and
proposed solutions do others have?
doctoral level. So, our starting point is this to obtain continuing education credit), our
question: If I–O psychology were strongly interpretations of results from the recent
evidence based in the ways defined and surveys of the I–O psychology profession
described above, what would we expect to and practitioner–research connections dis-
observe? cussed briefly above (Silzer et al., 2008),
Table 2 describes some of the charac- and also decades of teaching in I–O psy-
teristics we would expect in an area of chology programs. Our assessments of I–O
practice if it were also evidence-based psychology practice are discussed below,
per the above definitions. It describes our each accompanied by an explanation.
current judgment of the extent these char-
acteristics are observable in I–O psychol- 1. The term ‘‘evidence based’’ is used or
ogy. Others will have different experi- known. Although the general notion
ences and observations, which we hope of using evidence is well established,
will be expressed in the responses to this the specific term ‘‘evidence-based’’
focal article. Our judgments are based on and what it entails is not widely used
observations from a number of sources, or well known. It is rare to find the
including our recent participation in the term ‘‘evidence based’’ paired with
first evidence-based practice workshop at I–O psychology or organizational
SIOP (attended by dozens of practitioners psychology. However, the 2009 SIOPEvidence-based I–O psychology 9
Table 2. Some Key Characteristics of Evidence-Based Practice and an Estimate of the
Extent to Which They Are Found in I–O Psychology
Found in I–O
Some characteristics of evidence-based practice psychology?
1. The term ‘‘evidence based’’ is used or well known. Given To a very limited
evidence-based practice exists in many other fields and the term is extent?
widely used, it is unlikely that any field adopting an evidence-based
approach would not know of and use the term, even though
definitions might vary across practice fields.
2. The latest research findings and research summaries are accessible. To a very limited
It is difficult to do evidence-based practice without access to evi- extent?
dence in research journals and research summaries. A fundamental
principle of evidence-based practice is that systematic reviews of the
best available external evidence need to be available.
3. Articles reporting primary research and traditional literature reviews To some extent?
are accessible to practitioners. For many reasons systematic reviews
may not be available or produced in an area of interest. When this is
the case, primary research and traditional reviews published in
journals can be used.
4. ‘‘Cutting-edge’’ practices, panaceas, and fashionable new ideas are To some extent?
treated with healthy skepticism. One characteristic of areas of
practice that are not particularly evidence based is the constant
search for and promotion of the newest solution or cure. This
characteristic is found in popular books on topics such as dieting,
self-help, and indeed management.
5. There is a demand for evidence-based practice from clients and To some extent?
customers. If the clients or customers of a particular practice do not
want or even reject evidence-based practice then it is simply
impossible to practice in an evidence-based way.
6. Practice decisions are integrative and draw on the four sources of To some extent?
information and evidence described above. As emphasized, (Difficult to judge.)
evidence-based decision making is more than looking at external
published evidence. Rather, it is about combining evaluated external
evidence, the perspectives of those who may be affected by the
decision, information from the local context, and practitioner
experience and expertise.
7. Initial training and continuing professional development (CPD) adopt To a very limited
evidence-based approaches. From an evidence-based perspective, extent?
initial training and CPD focus on developing evidence-based
approaches to practice. This involves training practitioners to identify
and critically evaluate external and contextual evidence relevant to a
specific practice problem to help inform a practice decision. This
approach creates an active need to obtain and use relevant evidence,
as it is being used directly to help solve a problem.
annual conference did have EBMgt as systematic reviews in I–O psychol-
one of its themes. ogy. SIOP is currently developing its
2. Systematic reviews are produced and Science You Can Use series, which
made accessible. Although we have will contain reviews that are not
plenty of traditional reviews and systematic as such but will go some
meta-analyses, there are very few of the way to summarizing research10 R.B. Briner and D.M. Rousseau
findings that I–O practitioners can else will. However, as I–O psycholo-
use. gists, we may at the same time try
3. Articles reporting primary research to rework the fad into something
and traditional literature reviews are closer to our own practice and to
accessible to practitioners. We have established and evidence-based tech-
found that this is a difficulty for niques.2
many I–O psychology practition- 5. There is a demand for evidence-based
ers unless they are still attached in practice from clients and customers.
some way to a university. Although Many of our major clients are those
abstracts are easily accessible, pur- working at mid to senior levels in
chasing single articles from publishers HRM. HRM is not a field that has
can be costly. In addition, to build embraced the notion of evidence-
up even limited knowledge of the based practice in any significant way.
evidence in a particular area can Although, of course, managers do
require access to dozens of arti- not actively seek to purchase inef-
cles, which may be prohibitively fective I–O psychology products or
expensive. It also appears to be services, they are under pressure to
the case that not all I–O psychol- meet certain shorter term goals. They
ogy practitioners, depending some- may therefore come to depend on
what on where they trained, are a general impression that particu-
highly skilled in reading and digesting lar products or techniques ‘‘work’’
(sometimes rather indigestible) jour- rather than whether they will work
nal articles. in their specific context given the
4. ‘‘Cutting-edge’’ practices, panaceas, problem they are trying to solve.
and fashionable ‘‘new’’ ideas are HRM departments may also lean on
benchmarking or mimicry by adopt-
treated with healthy skepticism. As
ing the same I–O psychology prac-
a consequence of our training as psy-
tices already used by their more
chologists, we should be inclined
successful competitors. The authors
to be quite skeptical or at least
have also heard many times from I–O
are inclined to ask about evidence
psychologists who say they wish to
and empirical support. At the same
practice in an evidence-based way
time, however, we are also somewhat
but that clients have often already
drawn to what might be called man-
decided what they want (e.g., assess-
agement fads and fashions. Some of
ment centers, training programs, and
the recent focal articles in this jour-
employee attitude surveys) and are
nal demonstrate that we are some- asking the I–O psychologist as a
times inclined to pick up and run technical specialist to deliver it. This
with the Next Big Thing even where situation suggests that our clients are
evidence does not yet exist or is ques- not demanding an evidence-based
tionable. In addition, next time you approach in the sense discussed here,
attend the SIOP annual conference, although they are of course interested
check out the products and services in adopting practices they believe to
for sale in the exhibit hall. In our be effective.
experience, many of these feel more 6. Practice decisions are integrative and
like fads than evidence-based inter- draw on the four sources of informa-
ventions (and often no supporting tion and evidence described above.
evidence is presented). One reason
we pay attention to fads is that clients
often demand the latest thing, and if 2. We thank an anonymous reviewer for these
we don’t deliver it, then some one observations.Evidence-based I–O psychology 11
This is the most difficult character- passive way to learn and retain infor-
istic of evidence-based practice to mation. We note that in both the
assess without access to numerous United States and Britain, the major-
observations of what practicing I–O ity of practicing I–O psychologists
psychologists actually do. In addition, have terminal master’s degrees. The
it may be that as discussed above, we typical master’s program in the field
are not the decision makers; instead, has no required supervised intern-
we play an advisory role, provid- ships and does not train its stu-
ing information and interpretations to dents to conduct literature searches
the decision makers. However, tak- on practice topics let alone system-
ing each source of information in atic reviews. The forms of CPD used
turn, I–O psychologists do, as dis- by SIOP and other I–O psychol-
cussed above, draw to some extent ogy professional bodies also tend to
on evaluated external evidence when be fairly traditional. In fact, some
making decisions, even though few of these forms of CPD reverse the
systematic reviews are available and approach adopted by evidence-based
access to primary research can be dif- practice in that they present par-
ficult. The perspectives of those who ticipants with recent research find-
may be affected by the decision are ings, evidence, or new techniques
likely to be taken into account at and then discuss how they might
least to some extent because of the be used in practice rather than start-
APA’s Ethics Code (2002; assuming
ing with practice problems and then
we are complying with the code)3 and
searching for and evaluating the
also because of the broader aware-
evidence that may help solve the
ness we should have as psychologists
problem.
about our responsibilities to organi-
zational and individual clients. We
So, what is it possible to conclude from
are likely to look for and use evi-
this analysis of the extent to which I–O
dence from the local context and
psychology shares some of the character-
attempt some initial assessment of
istics of evidence-based practice? First, we
the problem or situation and seek
out organizational data that might suggest that I–O psychology is not strongly
help with problem diagnosis. The evidence based in the sense that the term
use of practitioner experience and is used in other professions. But, we can
judgment seems highly likely, partic- say with some confidence that we are as
ularly if the problem or technique is a profession extremely well positioned to
one we have encountered frequently adopt, should we wish to do so, many
before. of these characteristics. Next, I–O psy-
7. Initial training and continuing pro- chologists in many instances are not the
fessional development (CPD) in evi- key decision makers but, rather, sources of
dence-based approaches. Training in information and advice to managers mak-
I–O psychology master’s degrees and ing the decision (see below). Last, it is
doctorates tends to be of the fairly clear that there are many barriers to the
traditional academic variety where adoption of evidence-based practice, some
students are expected in a relatively within and others outside our control. Hav-
ing evaluated I–O psychology as falling
somewhat short on evidence-based prac-
3. Ethical I–O psychology and evidence-based I–O tice and supports for it, we turn to two
psychology are similar in other respects including important means for bridging I–O psychol-
the focus on being concerned and explicit about
the evidence for the benefits of interventions and ogy’s own research–practice gap: practice-
the evidence that interventions are not harmful. oriented evidence and systematic reviews.12 R.B. Briner and D.M. Rousseau
Key Strategies for Promoting Anderson et al. (2001) noted that between
Evidence-Based Practice 1949 and 1965, practitioners authored a
full 36% of JAP articles (31% by prac-
I–O psychologists are not one seamless
titioners alone). From 1990 to 2000 (the
community of comembers of the same
terminal year of their survey), practitioners
discipline. Practitioners and scholars in I–O
authored only 4% of JAP articles (1% by
psychology are largely distinct communities
practitioners alone). The other I–O journals
of interest, knowledge, and social ties.
manifest a similar decline. Many factors
In promoting evidence-based practice, it
may account for this decline in practitioner
is advantageous in such circumstances
research publication in our field’s journals,
to design ways of communicating and
including a shift in journal focus to more
sharing ideas that serve the interests of
academic topics of rigor, greater corporate
both. Although evidence-based practice
concern for protecting intellectual property,
involves the better use and integration of
as well as ramped-up global competition
evidence and information from all four
and its accompanying time and resource
sources described above, we focus here
crunch, which in turn limited practitioner
on improving the use of critically evaluated
opportunity for research let alone publi-
research evidence.
cation. One conclusion is apparent: I–O
Between the research and practice psychology’s academics and practitioners
domains of I–O psychology, we need are not mingling with each other in our jour-
devices for translating back and forth infor- nals. Regardless of the underlying reasons,
mation and knowledge, promoting better there is one serious consequence of the
quality communication and learning. We decline in practitioner conducted research
note that in the SIOP practitioner survey publication: Academics are the ones asking
described above, a frequent practitioner the research questions and interpreting the
request to SIOP was for clarification of answers.
standards for I–O practice and better ways If the gap between research and prac-
of differentiating I–O psychologists from tice in I–O psychology is at least partly
other practitioners in the marketplace. Such attributable to lower participation by prac-
clarification and professional differentiation titioners in research, the problem may
can come from creating the evidence- be exacerbated by omission in current
oriented products and associated processes research of the kinds of complex prob-
proposed here. Such products can simulta- lems in complex settings faced by prac-
neously meet the needs of I–O psychology’s titioners in their work. An antidote to
practitioners and scholars, adding value the latter has been suggested in the form
to both. The two products we suggest as of engaged scholarship (Van de Ven,
critical to evidence-based practice in I–O 2007) and scholar–practitioner collabora-
psychology are practice-oriented evidence tion (Lawler & Mohrman, in press), where
and systematic reviews. academics and practitioners work together
to formulate research questions, investigate
them, and draw conclusions.
Practice-Oriented Evidence
If indeed I–O psychology research is
Most research published in I–O psychol- now academic centric, the gap between
ogy’s premier journals, including Journal research and practice entails problems in
of Applied Psychology (JAP ), Person- knowledge transfer. It takes two to tango,
nel Psychology, and Journal of Occupa- and difficulties transferring knowledge can
tional and Organizational Psychology is be because of communication issues on
theory-oriented investigations authored by both sides (what’s readable, accessible,
academy-based I–O psychologists answer- understandable, and interesting?). Barriers
ing questions of interests to other aca- to transfer may also reside in the nature of
demics. This was not always the case. the knowledge itself. As a case in point,Evidence-based I–O psychology 13
meta-analysis and literature reviews in I–O average and college reputation. Another
psychology have led to the formulation reason for caution in using tests of GMA is
of general knowledge principles based on concern over adverse impact, even though
scientific evidence (The Handbook of Prin- some methods have relatively low adverse
ciples of Organizational Behavior: Indis- impact (Schmidt, 2009).
pensable Knowledge for Evidence-Based This debacle may be tied to what Van
Management ; Locke, 2009). The Handbook de Ven and Johnson (2006) refer to as
has over 30 chapters summarizing several a knowledge production problem, argu-
hundred evidence-based principles, the vast ing that the research–practice gap is best
majority of them derived from I–O psy- bridged by producing practice-oriented sci-
chology research and all intended to guide entific knowledge via research approaches
the practice of current and future managers engaging both academics and researchers
and other practitioners. This book provides collaboratively. This approach calls for
many examples and is written in plain combining the knowledge of practition-
English. It represents, perhaps at its best, ers and the knowledge of academics at
knowledge (principles) based on what might all stages of the research process. This
be called ‘‘researcher-oriented evidence,’’ knowledge production problem has been
that is, evidence from rigorous tests of the- encountered, and addressed, in other
ory, replicated and found to be relatively evidence-based practice fields.
generalizable over time and context. Medical researchers and clinicians dis-
So, what’s wrong with that? In many tinguish between two kinds of research
ways, these research-based principles evidence: disease oriented and patient ori-
achieve our ideal as scientists: to under- ented. ‘‘Disease-oriented evidence’’ (DOE)
stand the world well and disseminate this focuses on the causes of disease providing
knowledge. The downside is this: It is evidence of pathology and ways of altering
not always obvious to practitioners, cer- the condition (e.g., drugs and surgery). In
tainly not to the least experienced or less I–O psychology, our version of DOEs might
reflective, how exactly to apply the prin- take the form of phenomena-oriented evi-
ciples identified in such research. Let’s dence, such as the origins of job stress or job
take the classic example of the finding satisfaction. The second kind of research
that general mental ability (GMA) is pos- evidence in medicine is ‘‘patient-oriented
itively related to individual performance evidence that matters’’ (POEMs), evidence
(Schmidt, 2009). A well-established finding gathered from studies of real patients about
over several decades, one practice impli- issues such as mortality, morbidity, and
cation is that if an organization seeks to quality of life. An example of a POEM
improve the quality of its workforce and is a study comparing one antihyperten-
the performance of individual members, it sive drug to another to determine which
should select on intelligence. For a host of reduced mortality from hypertension over a
reasons, this principle is not widely pro- 10- to 20-year period. In I–O psychology,
moted by I–O psychology practitioners and our version of POEMs might take the form
is soundly rejected by even experienced HR of studies contrasting two interventions to
managers (Rynes et al., 2002). Practitioners reduce job stress that assess the types of
think about the educated idiot who is book individuals, work settings, and job strains
smart, tests fantastically well, and can’t they best ameliorate.
match socks. Managers fear being labeled There is a growing trend in the prac-
elitist and perhaps wonder whether they tice of medicine to value patient-oriented
would have gotten their own jobs if their data more highly than DOE. However,
company used IQ tests. Or, they use a test because practice-oriented evidence does
of GMA and find that it doesn’t improve per- not yet exist to inform every clinical
formance levels over the biographical infor- need, practitioners must use other ways
mation they already rely on like grade point of making decisions too, including relying14 R.B. Briner and D.M. Rousseau
on their knowledge of basic physiological of GMA. Practice-oriented research could
processes. We expect much the same look into whether performance criteria in
alternative forms of decision making in use affected the value and usefulness practi-
an evidence-informed practice of I–O tioners attach to indicators of GMA. Perhaps
psychology. settings where innovation and creativity are
An example of practice-oriented evi- important performance metrics place more
dence in I–O psychology are the reports by value on mental ability than those where
Robert Pritchard and his team, developing more routine performance is involved. Aca-
and investigating the use of the Produc- demically oriented evidence indicates that
tivity Measurement and Enhancement Sys- GMA is likely to predict performance in
tem (ProMES) system for job analysis and either case. Practitioners may only find
strategic planning (e.g., Pritchard, Harrell, GMA useful where mental ability is an
DiazGranados, & Guzman, 2008). Differ- organizationally valued contributor to per-
ences identified between studies in how formance.
thoroughly the ProMES system was applied
suggested that several implementation-
related factors, including the extent users
adhered to the ProMES process and the
Systematic Reviews
quality of the feedback provided, affected Systematic reviews are fundamental to
the overall productivity gains associated evidence-based practice. As such, much is
with ProMES. Pritchard and colleagues then written about them in other fields (Gough
address the circumstances under which et al., 2003; Soukup, 2000; Tranfield,
there are differences in implementation or Denyer, & Smart, 2003). Experience indi-
compliance with standard practices and cates that it is impossible to fully engage
the sensitivity of outcomes to these vari- in evidence-based practice without them:
ations (see Pritchard et al., 2008). In the Such reviews provide one of the four
context of widespread variation in orga- sources of information required when mak-
nizational and management practice (from ing evidence-based decisions. Applied to
performance appraisals to quality programs) the example in Table 1, a systematic
as well as in individual implementers, evi- review’s purpose would be to search for,
dence regarding the effects of such variabil- collect, critically appraise, and pull together
ity on outcomes has considerable practical research evidence relevant to the causes
and scholarly value. and possible solutions to the problem
Consider what practice-oriented evi- of high absence. I–O psychologists are
dence might mean for some of the stickier
avid producers and consumers of literature
problems in I–O psychology. We know
reviews. However, systematic reviews are
that GMA is predictive of individual per-
very different from nearly all those pub-
formance, but organizations are reluctant
lished in I–O psychology.
to accept or act on this knowledge (Rynes
et al., 2002), often preferring intuitive selec-
tion methods (Highhouse, 2008). Practice- Systematic reviews are literature reviews
oriented evidence could be developed from that adhere closely to a set of scientific
investigations into conditions making use methods that explicitly aim to limit sys-
of GMA as a selection criterion more read- tematic error (bias), mainly by attempting
ily useful. Looking into conditions of use to identify, appraise and synthesize all
could identify, for example, whether the relevant studies (of whatever design) in
backlash is to written IQ-type tests, where order to answer a particular question (or
structured interview questions that tapped set of questions). In carrying out this task
GMA would be more acceptable, or how they set out their methods in advance,
the concerns over adverse impact could be and in detail, as one would for any piece
better balanced with the predictive validity of social research. In this respect . . . theyEvidence-based I–O psychology 15
are quite unlike most ‘‘traditional’’ narra- would attempt to find evidence about the
tive reviews (Petticrew & Roberts, 2006, relative effectiveness of different forms of
pp. 9–10). absence management interventions given
the current and desired absence rates and
Systematic reviews are, essentially, a taking into account as much as possible
way of analyzing existing research using aspects of the context such as the type of
explicit and replicable methods, allowing employees involved, the sector, and the
conclusions to be drawn about what is existing work arrangements and absence
known and what is not known in relation policies. In the context of evidence-based
to the review question (and within the practice, systematic reviews can take forms
limitations of the method). Similar, but not akin to phenomena-oriented evidence or
identical to the traditional meta-analysis, practice-oriented evidence, depending on
systematic reviews are studies of studies. the review questions and their intended
Meta-analyses are a type of systematic use as well as the kinds of research avail-
review but one that uses only quantitative able. In evidence-based fields, an important
data and statistical synthesis and focuses result of systematic reviews is guidelines for
on a question repeatedly addressed in the practice.
same way by researchers rather than a Systematic reviews can be useful for
practice question or problem. As with meta- purely academic research purposes too.
analyses, systematic reviews are conducted We may, for example, be interested in
out of recognition that single empirical collecting all available evidence about
studies, although useful and sometimes absence-management interventions to pro-
informative, should not be emphasized
vide a more general overview about what
because their biases and limitations cannot
is known, and not known, about the effi-
be fully accounted for. Looking at all
cacy of such interventions. In this respect,
relevant studies, systematically gathered,
a systematic review might differ from the
constitutes more reliable evidence.
traditional meta-analysis in that it would
Thus, in the context of evidence-
also consider qualitative information and
based practice, neither traditional literature
descriptions, not being limited to effect
reviews nor meta-analyses are especially
sizes alone depending on the review ques-
useful. First, traditional literature reviews
are open to many forms of bias. For tion (Rousseau, Manning, & Denyer, 2008).
example, reviewers do not make clear how All systematic reviews follow a process of
they have selected the studies they have clearly specified stages. One example of this
included, do not critically appraise them process (adapted from Petticrew & Roberts,
in an explicit or systematic way, and do 2006) contains seven stages.
not usually pull them together or synthe-
size findings across studies. Second, tra- 1. Identify and clearly define the ques-
ditional reviews do not usually focus on tion the review will address. The ques-
a specific research, practice question, or tion needs to be sufficiently specific so
problem. It is this latter point that also that it is clear, in principle, what types
differentiates a systematic review from the of data would be relevant. Aspects of
quantitative meta-analysis used tradition- the context (e.g., population, sector,
ally in I–O psychology. The process of and organizational type), the inter-
making evidence-based decisions requires ventions (what qualifies as a relevant
more focused and tailored reviews of evi- intervention?), the mechanisms link-
dence where both a practice question or ing intervention to outcomes (e.g.,
problem and the conditions to which the processes, mediators, and modera-
evidence might be applied are taken into tors), and the outcomes themselves
account. Returning to the case of high (which data are the outcomes of inter-
absence in Table 1, a systematic review est) are also clearly specified.16 R.B. Briner and D.M. Rousseau
2. Determine the types of studies and Synthesis may be quantitative, qual-
data that will answer the question. itative, or both. Review findings are
The criteria used to decide which often described in terms of the overall
studies will be selected or excluded number of studies found, the quality
from the review are identified. For profile of this group of studies, and
example, a review that addresses a the number of studies that obtained
causal question might exclude studies particular results.
with cross-sectional designs. The aim 7. Disseminate the review findings. A
is to increase the chances that all full report of a systematic review can
relevant studies are included and all be quite large. In addition, shorter
those that are irrelevant are excluded. journal article length versions or even
3. Search the literature to locate relevant shorter summaries may be produced.
studies. A clear search strategy is used Dissemination is often planned at the
specifying, for example, key words, outset of a systematic review given
the databases to be searched, and the aim is often to inform practice.
how, and whether unpublished data
will be found and included. Although systematic review is rare in
4. Sift through all the retrieved studies to I–O psychology at present, I–O psychol-
identify those that meet the inclusion ogists are certainly familiar with its general
criteria (and need to be examined approach. The underlying logic of system-
further) and those that do not and atic review is similar to that of many psy-
chological research methods and is similar
should be excluded. Each study is
to meta-analyses. Meta-analysis’ exclusive
examined, usually by two review
use of quantitative data and statistical rather
team members, and checked against
than other forms of synthesis sets it apart
the inclusion and exclusion criteria.
from a systematic review, which often uses
Where agreement is not possible, a
different types of data and different forms
third reviewer assesses the study. At
of synthesis. In addition, meta-analyses
this stage, it is not uncommon to find
can only address questions that have been
that only a fraction of the initial pool
addressed many times in more or less the
of studies can be included. same way by researchers (e.g., the corre-
5. Critically appraise the studies by lation between job satisfaction and per-
assessing the study quality deter- formance) rather than questions that arise
mined in relation to the review ques- from practice problems, where an array of
tion. The quality of each study is data types may be required to formulate an
critically appraised or evaluated in answer.
relation to the review question as An example of a structured abstract from
research quality can only be judged in a systematic review particularly relevant to
relation to the question. Even where I–O psychology is presented in Table 3.
studies meet the inclusion criteria, This demonstrates much of the underly-
they are still likely to vary in terms ing methodology and shows the explicit
of quality. Assessing quality allows and systematic nature of the method.
the review conclusions to clearly state Joyce, Pabayo, Crichley, & Bambra’s (2010)
how many of the studies included review clearly states the review objectives,
were, for example, of very high, search strategy, criteria for including stud-
medium, and low quality. ies, method of analysis (in this case, a
6. Synthesize the findings from the stud- narrative synthesis as the studies were dis-
ies. A key part of any systematic similar), the number of studies found, and
review is the pulling together of find- the findings of each.
ings from across the studies to repre- A few aspects of systematic reviews
sent what is known and not known. and their differences from traditional I–OEvidence-based I–O psychology 17 Table 3. Example of a Systematic Review Abstract Flexible working conditions and their effects on employee health and well-being (Joyce et al., 2010) Background: Flexible working conditions are increasingly popular in developed countries, but the effects on employee health and well-being are largely unknown Objectives: To evaluate the effects (benefits and harms) of flexible working interventions on the physical, mental, and general health and well-being of employees and their families Search strategy: Our searches (July 2009) covered 12 databases including the Cochrane Public Health Group Specialized Register, CENTRAL, MEDLINE, EMBASE, CINAHL, PsycINFO, Social Science Citation Index, ASSIA, IBSS, Sociological Abstracts, and ABI/Inform. We also searched relevant Web sites, hand searched key journals, searched bibliographies, and contacted study authors and key experts Selection criteria: Randomized controlled trials, interrupted time series, and controlled before and after studies (CBA), which examined the effects of flexible working interventions on employee health and well-being. We excluded studies assessing outcomes for less than 6 months and extracted outcomes relating to physical, mental, and general health/ill-health measured using a validated instrument. We also extracted secondary outcomes (including sickness absence, health service usage, behavioral changes, accidents, work–life balance, quality of life, health and well-being of children, family members, and coworkers) if reported alongside at least one primary outcome Data collection and analysis: Two experienced review authors conducted data extraction and quality appraisal. We undertook a narrative synthesis as there was substantial heterogeneity between studies Main results: Ten studies fulfilled the inclusion criteria. Six CBA studies reported on interventions relating to temporal flexibility: self-scheduling of shift work (n = 4), flexitime (n = 1), and overtime (n = 1). The remaining four CBA studies evaluated a form of contractual flexibility: partial/gradual retirement (n = 2), involuntary part-time work (n = 1), and fixed-term contract (n = 1). The studies retrieved had a number of methodological limitations, including short follow-up periods, risk of selection bias, and reliance on largely self-reported outcome data. Four CBA studies on self-scheduling of shifts and one CBA study on gradual/partial retirement reported statistically significant improvements in either primary outcomes (including systolic blood pressure and heart rate; tiredness; mental health, sleep duration, sleep quality, and alertness; and self-rated health status) or secondary health outcomes (coworker social support and sense of community), and no ill-health effects were reported. Flexitime was shown not to have significant effects on self-reported physiological and psychological health outcomes. Similarly, when comparing individuals working overtime with those who did not, the odds of ill-health effects were not significantly higher in the intervention group at follow-up. The effects of contractual flexibility on self-reported health (with the exception of gradual/partial retirement, which when controlled by employees improved health outcomes) were either equivocal or negative. No studies differentiated results by socioeconomic status, although one study did compare findings by gender but found no differential effect on self-reported health outcomes Authors’ conclusions: The findings of this review tentatively suggest that flexible working interventions that increase worker control and choice (such as self scheduling or gradual/partial retirement) are likely to have a positive effect on health outcomes. In contrast, interventions that were motivated or dictated by organizational interests, such as fixed-term contract and involuntary part-time employment, found equivocal or negative health effects. Given the partial and methodologically limited evidence base, these findings should be interpreted with caution. Moreover, well-designed intervention studies are needed to delineate the impact of flexible working conditions on health, well-being, and health inequalities
You can also read
