What Is Evidence-Based Behavior Analysis?
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The Behavior Analyst 2013, 36, 7–33 No. 1 (Spring)
What Is Evidence-Based Behavior Analysis?
Tristram Smith
University of Rochester Medical Center
Although applied behavior analysts often say they engage in evidence-based practice, they
express differing views on what constitutes ‘‘evidence’’ and ‘‘practice.’’ This article describes a
practice as a service offered by a provider to help solve a problem presented by a consumer.
Solving most problems (e.g., increasing or decreasing a behavior and maintaining this change)
requires multiple intervention procedures (i.e., a package). Single-subject studies are invaluable
in investigating individual procedures, but researchers still need to integrate the procedures into
a package. The package must be standardized enough for independent providers to replicate yet
flexible enough to allow individualization; intervention manuals are the primary technology for
achieving this balance. To test whether the package is effective in solving consumers’ problems,
researchers must evaluate outcomes of the package as a whole, usually in group studies such as
randomized controlled trials. From this perspective, establishing an evidence-based practice
involves more than analyzing the effects of discrete intervention procedures on behavior; it
requires synthesizing information so as to offer thorough solutions to problems. Recognizing the
need for synthesis offers behavior analysts many promising opportunities to build on their
existing research to increase the quality and quantity of evidence-based practices.
Key words: evidence-based practice, clinical trials, treatment effectiveness evaluation,
behavior analysis, behavior modification
A Google search for evidence-based based, and delineate categories such
behavior analysis yields 16,500,000 as ‘‘probably’’ or ‘‘possibly’’ effica-
hits. Most of the top hits contain cious (Chambless & Hollon, 1998;
affirmations that behavior analysis is Kratochwill & Stoiber, 2002; Odom
indeed evidence based. Similar state- et al., 2005; West et al., 2002). Many
ments appear regularly in traditional guidelines in medicine specify that the
media (e.g., Smith, 2012). Despite gold standard for identifying evi-
this refrain, many behavior analysts dence-based interventions is a sys-
express misgivings about published tematic, statistical analysis of data
operational definitions of the term from multiple randomized controlled
evidence based (Green, 2008; O’Don- trials (RCTs), in which investigators
ohue & Ferguson, 2006). Such de- assign large numbers of participants
finitions usually include guidelines by chance to treatment or control
for rating the quality of individual groups (Guyatt, Oxman, et al., 2008).
studies, aggregating findings across By examining how closely the avail-
studies, and classifying the overall able research approaches this stan-
level of empirical support for an dard and how favorable the results
intervention. For example, guidelines are, reviewers can rank or grade the
in education and psychology list intervention (United States Preven-
specific indicators of high-quality tive Services Task Force, 2007).
studies, set a threshold for the Behavior analysts have exposed
number of such studies needed to many limitations of RCTs (Johnston,
establish an intervention as evidence 1988; Johnston & Pennypacker, 1993)
and statistical methods for combining
Correspondence concerning this article data from different studies (Baron
should be addressed to Tristram Smith, & Derenne, 2000). One practical
Division of Neurodevelopmental and Behav- problem is that individuals who con-
ioral Pediatrics, Department of Pediatrics, sent to enroll in RCTs and risk being
University of Rochester Medical Center, 601
Elmwood Ave., Box 671, Rochester, New
assigned to the control group may
York 14642 (e-mail: Tristram_Smith@URMC. differ from individuals who decline
Rochester.edu). to enroll (Christenson, Carlson, &
7
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:20 7 Cust # MS TBA12-158 TRISTRAM SMITH Valdez, 2002). Quasiexperimental display of the data, obviating the studies, in which treatment and con- need for statistical analyses. Replica- trol groups are matched on partici- tions of intervention procedures with- pant characteristics but are assigned in and across participants will eluci- nonrandomly (e.g., according to the date how and when the relation availability of providers to deliver the occurs (Johnston, 1996). experimental treatment), may mini- In relation to evidence-based prac- mize this problem. Nevertheless, a tice, behavior analysts advocate for more fundamental difficulty remains: the recognition of SSED studies as Both RCTs and quasiexperimental important sources of data (Detrich, studies test the average change within 2008; Green, 2008) and have pro- an intervention group instead of the posed criteria for classifying an inter- more clinically relevant analysis of vention as evidence based on the change made by each individual strength of findings from such studies participant as a function of a specific (Horner et al., 2005; Kratochwill et intervention (Johnston, 1988). al., 2010; National Autism Center Statistical metrics that merge data [NAC], 2009). These guidelines in- across studies compound the prob- clude indicators of high-quality re- lems in RCTs (Salzberg, Strain, & ports, notably clearly described inter- Baer, 1987; Strain, Kohler, & Gresh- vention procedures and multiple am, 1998). Basically, these metrics replications of intervention effects, represent an average of the average assessed by direct, reliable observa- change reported in different studies. tions of the target behavior (Horner The studies almost always vary from et al., 2005; NAC, 2009). Reviews of one another in terms of their inter- SSED studies find numerous applied vention protocols, outcome mea- behavior-analytic (ABA) procedures sures, and criteria for enrolling par- that meet criteria for classification as ticipants. Thus, the metrics are evidence based (e.g., Kurtz, Boelter, derived from an amalgamation of Jarmolowicz, Chin, & Hagopian, data obtained by an assortment of 2011; Lee, 2005; NAC, 2009; Sulzer- methods. False precision is apt to be Azaroff & Austin, 2000; Vegas, the result. Jenson, & Kircher, 2007). Because of these limitations of From such work, some behavior group studies (RCTs and quasiex- analysts conclude that we have per- perimental research), behavior ana- suasively rebutted standard defini- lysts favor studies with single-subject tions of evidence-based practice experimental designs (SSEDs). SSED (Keenan & Dillenburger, 2011), re- studies involve conducting repeated placed them with definitions of our observations to compare an individ- own (e.g., Horner et al., 2005), and ual’s behavior during a baseline documented that many of our inter- period when the individual receives ventions merit this designation (e.g., no intervention to the behavior in NAC, 2009). However, such conclu- one or more intervention phases. sions are too hasty, in this writer’s They are intended to detect changes judgment. Inspection of the ABA that occur as soon as an intervention interventions that are described as begins. Thus, SSED studies are much evidence based and the nature of better suited than group studies empirical support adduced for them for establishing a functional relation reveal that some basic questions between a particular intervention and remain unanswered. change of a specific behavior (John- First of all, what is a practice? Lists ston, 1988; Johnston & Pennypacker, of evidence-based ABA practices are 1993). If the functional relation is an incongruous lot. For example, robust, the change should be evident after a meticulous literature search from visual inspection of a graphical and coding process, the National The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:20 8 Cust # MS TBA12-15
EVIDENCE-BASED BEHAVIOR ANALYSIS 9
Standards Project (NAC, 2009) on What is evidence based? Going
interventions for individuals with beyond our longstanding contention
autism spectrum disorders (ASD) that SSED studies have advantages
classified 11 practices as evidence over between-groups studies (Sidman,
based: 1960), behavior analysts assert that
such studies can stand alone as the sole
1. Antecedent package source of empirical support for a
2. Behavioral package practice (Horner et al., 2005). Is this
3. Comprehensive behavioral treat- more expansive assertion tenable?
ment for young children Going still further, some writers argue
4. Joint attention intervention that many ABA intervention ap-
5. Modeling proaches (e.g., differential reinforce-
6. Naturalistic teaching strategies ment and shaping) are so well estab-
7. Peer training package lished and are monitored so closely for
8. Pivotal response treatment each individual who receives them that
9. Schedules they can be considered evidence based
10. Self-management even without reference to a particular
11. Story-based intervention pack- set of SSED studies (Keenan &
age (NAC, p. 43) Dillenburger, 2011). Does this line of
Two items in this list, modeling and reasoning withstand scrutiny? Also,
schedules, are specific cuing proce- many systems have been proposed for
dures that have a wide range of classifying practices as evidence based
applications for individuals with and on account of findings from either
without ASD. Most other items are SSED or between-groups studies
combinations of procedures that vary (Chambless & Hollon, 1998; NAC,
from study to study. However, three 2009; National Secondary Transition
items refer to more standardized Technical Assistance Center, 2010;
packages. Comprehensive behavioral Odom et al., 2005; Reichow, Volkmar,
treatment originated in the work of & Cicchetti, 2008). How can two such
Lovaas (1987) on early intensive disparate methodologies, often char-
behavioral intervention (EIBI) for acterized as separate research tradi-
toddlers and preschoolers with ASD. tions by behavior analysts (e.g., John-
It has evolved into many different ston, 1988), both be used to identify
models (Handleman & Harris, 2001), evidence-based practices?
all of which combine a multiplicity of These questions are not merely
intervention procedures that are de- about semantics. The purpose of
livered 20 to 40 hr per week for 2 to identifying evidence-based practices
3 years (Smith, 2011). Pivotal re- is to help consumers and providers
sponse treatment (Koegel & Koegel, choose among intervention approach-
2006) is one approach to using es (Detrich, 2008). Essentially, a
naturalistic teaching strategies, with practice is a service offered by provid-
priority given to behaviors identified ers to consumers, and representing a
by the developers as especially impor- service as evidence based is a form
tant for promoting children’s devel- of endorsement or recommendation,
opment. Story-based intervention is backed by an appeal to science.
derived from a trademarked pro- Insurers and other third-party payers
gram, Carol Gray’s Social Stories increasingly use information about
(Gray, 2004). As acknowledged in evidence-based practices to determine
an incisive commentary, the only which services to cover (Green, 2008).
common element of all 11 of these The pliable definitions and criteria
items is that each is comprised of for evidence-based ABA practices
intervention procedures or combina- now in use are troublesome because
tions that seem to share core charac- they put us at risk for overestimating
teristics (NAC, 2009). our accomplishments and helpfulness
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:21 9 Cust # MS TBA12-1510 TRISTRAM SMITH
to consumers and providers while we to require a program’’ (p. 310). Azrin
overlook topics in need of more (1977) added that a package or
study. The goal of the present article program should contain strategies
is to help behavior analysts provide for deploying techniques effectively,
useful guidance to consumers, pro- efficiently, and credibly, even when
viders, and third-party payers by confronted with real-world con-
clarifying what is or is not a practice straints such as resource limitations,
and what is or is not suitable variations in skill level and enthusi-
evidence. Accordingly, the article asm of interventionists, and competi-
examines what constitutes a practice tion from alternative programs. In
(a procedure or package of proce- other words, a package should help
dures, standardized or not) and providers make decisions on how to
persuasive evidence for the practice use procedures in their practice.
(findings from SSED or group stud- The need for intervention packages
ies). In so doing, prior discussions of led Azrin (1977) to recommend mov-
these issues by behavior analysts are ing quickly to conducting research
reconsidered in light of the much on packages. However, Birnbrauer
larger literature currently available, (1979) countered that it is premature
and ABA interventions that behavior to evaluate packages unless their
analysts have identified as evidence- components have been thoroughly
based practices are critiqued. These investigated. Johnston (1996) added
considerations are relevant for ABA that evaluations of components
providers, who need to recognize should go through a sequence begin-
which services they offer have ade- ning with investigations in controlled
quate scientific support and which settings that allow rigorous experi-
do not. Moreover, they raise critical mental analyses and proceeding to
questions to be addressed by ABA field settings that might require com-
researchers, who must determine promises (e.g., reliance on inexperi-
what is known and unknown about enced interventionists or introduction
interventions and set priorities for of an intervention after a baseline
future programs of inquiry. that was too short to establish a
stable rate of behavior).
PROCEDURES OR PACKAGES? Although they warned against
forming packages comprised of in-
In their formative description of completely understood procedures,
ABA as a discipline, Baer, Wolf, and both Birnbrauer (1979) and Johnston
Risley (1968) emphasized that ABA (1996) agreed that movement from
is technological, meaning that its procedures to packages is essential.
intervention techniques or proce- Birnbrauer averred that ‘‘no one
dures are precise, detailed, and repli- can argue with’’ (p. 19) the value
cable. A key aspect of ABA research of packages that improve outcome,
ever since has been to discover such minimize costs, and are acceptable to
procedures and understand how and consumers. He added that SSED
why they work. However, Baer (1975, studies on an individual procedure
2004) pointed out that, as research are ‘‘at best, slower’’ and ‘‘more
progresses, it may become appropri- difficult to sell to consumers’’ (Birn-
ate to combine procedures into a brauer, p. 19) than research on
package or program and test the packages. Likewise, Johnston (1993,
combination. A single intervention 1996) pointed out that the goal of
procedure (i.e., a format for teaching ABA research is ‘‘fully identifying all
a new skill or tactic to decrease a of the individuals and behaviors of
behavior) is seldom enough to fix a interest’’ and ‘‘understanding the web
problem by itself; as Baer (2004) of social contingencies’’ that influ-
remarked, ‘‘People’s problems seem ence these individuals and behaviors
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:21 10 Cust # MS TBA12-15EVIDENCE-BASED BEHAVIOR ANALYSIS 11
(Johnston, 1993, p. 329). He further willing and able to adopt. Even more
observed, strikingly, they acknowledged a dis-
connect between what appears in an
The challenge is to design the bits and pieces outlet such as the Journal of Applied
of a procedure into an integrated and often Behavior Analysis (JABA) and what
complex whole. … At some point, however, happens in practice: ‘‘Although some
everything that has been learned should
suggest a reasonable combination of proce- of the features of JABA research may
dural elements as a focus for further analytic certainly be found in routine practice,
attention. (p. 330) the pages of that journal do not
represent the full range of practices
In short, analysis of procedures yields in ABA’’ (Johnston et al., 2006,
the information needed to form a p. 61).
coherent package that can itself be The exchanges between Azrin
studied. (1977) and Birnbrauer (1979) and
Nevertheless, controversies over between Carr et al. (2002) and
efforts to create intervention packag- Johnston et al. (2006), along with
es show that behavior analysts con- the comments by Baer (1975, 2004),
tinue to struggle with how and when suggest that a broad consensus has
to do so. For example, Carr et al. emerged on what constitutes a prac-
(2002) reproached behavior analysts tice. Formulating and evaluating
for overrating studies of isolated practices, however, remain conten-
procedures: tious. Thus, it is worth exploring
the consensus on what constitutes a
For decades, applied behavior analysts have practice and then to consider research
prided themselves on the publication of many approaches that would increase suc-
successful research demonstrations that in-
volve the application of single interventions. cess in developing practices.
These demonstrations have made for great There is general accord among
science but ineffective practice. A comprehen- behavior analysts that a practice is
sive approach involving multicomponent in- a service aimed at solving people’s
tervention is necessary to change the many
facets of an individual’s living context that are problems. Further, although no one
problematic. (p. 9) has ruled out the possibility that the
solution might consist of a single
In a rejoinder, Johnston, Foxx, Ja- procedure in some cases, behavior
cobson, Green, and Mulick (2006) analysts concur that the solution is
disputed the charge that ABA proce- more likely to involve a combination
dures look impressive in a research of procedures, along with strategies
context yet fail in practice. They also for deploying the practice in typical
criticized many aspects of the frame- service settings (i.e., a package). For
work advocated by Carr et al. for example, a practice aimed at reduc-
creating packages to address problem ing a problem behavior is likely to
behavior (positive behavior interven- include a functional analysis or as-
tion and support; PBIS). It is impor- sessment, followed by the use of
tant to note, however, that they too function-based interventions that in-
described effective intervention for volve altering both antecedents and
problem behavior as involving a consequences of the behavior while
combination of procedures and in- strengthening replacement behaviors
deed worried that advocates of PBIS (Johnston et al., 2006). A practice
have downplayed some key proce- aimed at overcoming skill deficits is
dures, especially consequence-based likely to include a careful evaluation
strategies such as differential rein- of the individual’s current skill level
forcement. Moreover, they admitted and the skills that would be most
that behavior analysts have much to helpful for the individual to learn, as
learn about how to package inter- well as a task analysis of those skills,
ventions that nonspecialists would be followed by the implementation of a
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:21 11 Cust # MS TBA12-1512 TRISTRAM SMITH
range of teaching strategies to help service setting and the individual’s
the individual acquire, generalize, life circumstances (i.e., a package).
and maintain the skills (Greer, Definitions of evidence-based prac-
2002). Either kind of practice proba- tice by behavior analysts use different
bly also involves considering contex- words to make virtually the same
tual factors such as the individual’s point. Detrich (2008) described the
family environment, peer networks, function of evidence-based practice
and expectations in school or at the as ‘‘consumer protection.’’ Detrich,
workplace, In addition, the practice Keyworth, and States (2007) stressed
requires determining what is feasible that the identification of evidence-
given constraints such as resource based practices involves the review of
limitations and preferences of the research findings to guide providers
people involved. and consumers in their choice of
These views are consistent with interventions and the development
standard definitions of a practice of strategies to help them implement
across professions. Particularly when the interventions even in the presence
used in the phrase evidence-based of ‘‘social, political, fiscal, and hu-
practice, a practice refers to a fusion man resource’’ (p. 4) barriers that
of research findings, consumer pref- might exist. Similarly, Horner and
erences, and clinical assessment Kratochwill (2011) referred to a
(American Psychological Association practice as ‘‘any operationally de-
[APA], 2005; Sackett, Rosenberg, fined set of procedures that are used
Muir Gray, Haynes, & Richardson, by a specified target audience, under
1996). This meaning is encapsulated defined conditions/contexts, to achieve
in one helpful synopsis: valued outcomes for one or more
defined populations’’ (p. 2). In keeping
There is an abundance of definitions of with definitions in other professions,
evidence-based practice (EBP). Fortunately, these definitions characterize a practice
most of them say essentially the same thing. … as a package for implementing a
The goal of EBP is the integration of (a) procedure or procedures. Horner and
clinical expertise/expert opinion, (b) external Kratochwill’s definition is especially
scientific evidence, and (c) client/patient/care-
giver values to provide high-quality services. clear that an operationally defined
(American Speech-Language Hearing Associ- procedure is only one part of a practice.
ation, n.d.) Practices can vary in the range of
behaviors they target. For example,
Thus, evidence-based practice in psy- as emphasized by Carr et al. (2002),
chology is defined as ‘‘the integration practices developed within PBIS of-
of the best available research with ten aim to reduce a problem behavior
clinical expertise in the context of by improving the individual’s overall
patient characteristics, culture, and quality of life. In contrast, functional
preferences’’ (APA, 2005, p. 5). Sim- communication training is a practice
ilarly, evidence-based medicine is that focuses more specifically on
defined as an integration of clinical replacing a problem behavior with a
expertise and research evidence for communication skill that serves the
‘‘the conscientious, explicit and judi- same function as the problem behav-
cious use of current best evidence’’ ior (Carr & Durand, 1985). As
(Sackett et al., 1996, p. 71) to guide another example, EIBI is intended
decisions about service delivery. In to accelerate all aspects of develop-
these definitions a practice is not an ment in young children with ASD
isolated procedure. Rather, it is a (Lovaas, 1987), whereas the picture
procedure or (more commonly) a set exchange communication system is
of procedures that a skilled provider directed mainly toward helping chil-
can adapt to meet the needs of an dren with ASD begin to communi-
individual case in the context of a cate (Bondy & Frost, 2002).
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:21 12 Cust # MS TBA12-15EVIDENCE-BASED BEHAVIOR ANALYSIS 13
However, it is a misnomer to tion of self-help activities). However,
describe an individual procedure they did not identify a way to select
(e.g., modeling) as a practice, as has procedures that match the needs of
been done in some ABA reports (e.g., an individual child with ASD, the
NAC, 2009), unless the procedure by scope and sequence of skills to teach,
itself offers a solution to a problem the frequency and length of time to
and providers have guidance on how devote to each procedure, or methods
to use it in practice. If research shows to deploy the procedures expediently
that a procedure reliably changes in community settings such as schools.
behavior, it may become a compo- Therefore, although Odom et al.’s set
nent of a practice. It may also be of procedures is a valuable resource, it
worthy of dissemination in its own mainly consists of pieces (i.e., kernels)
right. Embry and Biglan (2008) de- that need to be assembled by provid-
scribed some procedures as ‘‘ker- ers and consumers. Thus, it consti-
nels’’—fundamental units that can tutes an important step toward for-
be incorporated into a wide range of mulating a practice but remains a
intervention or prevention packages. work in progress.
Examples include ‘‘time-out, written Going beyond procedures, packages
praise notes, self-monitoring, framing (e.g., intervention programs developed
relations among stimuli to affect the within PBIS) do include specific man-
value of a given stimulus, and phys- uals for assessment of the needs of
iological strategies such as nasal individual consumers (Sugai, Lewis-
breathing when upset’’ (p. 77). They Palmer, Horner, & Todd, 2005), de-
noted that kernels tend to be simpler velopment of an intervention plan for
than complete packages and thus may an individual (Dunlap et al., 2010) or a
be more readily disseminated in some group (Sugai et al., 2010), and evalu-
circumstances. However, they ac- ation of the fit between the interven-
knowledged that users will need to tion and the context in which it takes
combine kernels with other approach- place (Horner, Salentine, & Albin,
es and that kernels are not a substi- 2003). As discussed above, this kind
tute or replacement for packages. of integration of procedures into pack-
Even under the best of circum- ages is crucial for creating practices.
stances, community providers and Moreover, given the voluminous liter-
consumers would have great difficul- ature of SSED studies on the domain
ty determining how to use procedures targeted in PBIS (problem behavior),
in practice. For example, expanding one might hope that investigators have
on the NAC (2009) report, Odom, met Birnbrauer’s (1979) and John-
Hume, Boyd, and Stabel (2012) ston’s (1996) criteria for studying
identified 24 evidence-based teaching procedures thoroughly before packag-
procedures for children with ASD ing them. However, it is possible that
such as discrete-trial training, prompt the literature may still contain gaps. As
delay, and visual schedules. They also Johnston (1996) commented, many
laid out step-by-step instructions for SSED studies are aimed at responding
setting up and implementing each to an immediate need that arises in a
procedure, along with written tutori- service setting, rather than advancing a
als and instructional videos to show systematic program of research. Con-
how to use the procedures correctly. sequently, ‘‘the considerable size of the
To supplement this information, applied literature may suggest more
Odom et al. offered general guide- than it can actually deliver’’ (p. 39).
lines on the kinds of skills that can be Accordingly, a careful review of the
taught with each procedure (e.g., literature may be warranted to deter-
noting that visual schedules have mine whether the research is sufficient-
been found to be especially useful ly far along to guide the creation of a
for promoting independent comple- package. Another issue to consider is
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 13 Cust # MS TBA12-1514 TRISTRAM SMITH
that, because packages encompass a of our research focuses on interven-
multiplicity of procedures and strate- tions delivered by study personnel,
gies for their deployment, they can usually in tightly controlled environ-
take an almost infinite variety of forms ments such as laboratories, special-
(Baer, 2004). Thus, if existing packages ized ABA classrooms, or distraction-
(e.g., PBIS approaches) raise concerns free areas set up to provide one-to-
(Johnston et al., 2006), investigators one instruction (e.g., Kasari & Smith,
can propose alternatives. in press; Rehfeldt, 2011; Trahan,
Investigators may debate whether Kahng, Fisher, & Hausman, 2011).
to conduct SSED studies to fill in This discrepancy may reflect a dilem-
gaps in the literature on problem ma that behavior analysts have had
behavior, focus on PBIS, or develop trouble resolving: We recognize that
new intervention packages. Such de- conducting studies in practice settings
bates also may arise in connection may require sacrificing some scientif-
with interventions for other prob- ic rigor because the primary mission
lems. What is clear, however, is that of such settings is to deliver services
the key to resolving these debates is rather than conduct research (John-
to identify ways to promote progress ston, 1996), yet we regard the quality
from isolated procedures to integrat- of many studies in these settings as
ed packages. The establishment of unacceptable (Johnston et al., 2006).
evidence-based practices in ABA de- The root of this dilemma is that
pends on achieving this integration. findings from poorly designed studies
To summarize, a practice is a are difficult or impossible to inter-
solution to a problem presented by pret, no matter where they take place.
a consumer. Most problems require Thus, the best way to move forward
a combination of procedures, along is to seek opportunities to carry out
with strategies for implementing rigorous studies in practice settings.
them in a service setting (i.e., a Of course, this is easier said than
package). The process of implemen- done, but several areas of research
tation necessitates the integration of offer guidance on how to proceed.
evidence about the practice with the One such area is implementation
preferences of the individual who is science (Fixsen, Naoom, Blase, Fried-
being served and the skills of the man, & Wallace, 2005), which grew
service provider. Although SSED out of studies on ABA programs such
studies are invaluable in the identifi- as Achievement Place. Related areas
cation of individual procedures to of investigation, notably diffusion of
include in a practice, researchers still innovations (Rogers, 2003), socio-
must find ways to put procedures technical technology transfer (Backer,
together into a package for use in a David, & Soucy, 1995), and improve-
practice setting. The next section ment models (Langley et al., 2009),
describes methodologies for examin- arose from efforts to deploy other
ing whether a procedure is likely to technologies in community settings.
be useful in a practice setting and for Research in these areas has highlight-
packaging procedures. ed many factors that contribute to
success in working with community
Research Implications agencies. Particularly important ele-
ments include the establishment of an
The potential utility of procedures ongoing partnership with the agency,
in practice. Behavior analysts often identification of a champion within
emphasize the need to study the the agency for introducing a new
effects of ABA procedures in the technology, assembly of a team to
context of typical practice settings work together to implement the tech-
(e.g., Johnston, 1996). However, re- nology, securing support from admin-
views indicate that the large majority istrators to ensure that adequate
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 14 Cust # MS TBA12-15EVIDENCE-BASED BEHAVIOR ANALYSIS 15
resources are available, obtaining burg, Kelley, Roane, Kisamore, &
approbation from other influential Brown, 2009), or a need to augment
individuals in the agency, and starting the original intervention (e.g., Moore
with small pilot projects (Damschro- & Fisher, 2007). Still, a 100% rate
der et al., 2009). A commendable of improvement that usually occurs
example in ABA research is the right away belies the practical reality
Behavior-Analytic Consultation to that many individuals make slow or
Schools program, which has complet- minimal progress, even in exemplary,
ed a large number of high-quality intensive ABA intervention programs
functional analyses in public schools (cf. Hagopian & Wachtel, n.d.; Leaf,
(Mueller, Nkosi, & Hine, 2011). Other Taubman, McEachin, Leaf, & Tsuji,
examples are the studies of EIBI by 2011).
Howard, Sparkman, Cohen, Green, The tendency to publish only
and Stanislaw (2005) and Cohen, positive results may be due to the
Amerine-Dickens, and Smith (2006), priority given in ABA research to the
which were made possible when one establishment of experimental control
of the authors (Cohen) negotiated a that shows a functional relation
four-way agreement among state between an intervention and a behav-
agencies, local school districts, private ior. Beginning with Skinner (1956),
service providers, and parents on a behavior analysts have held that it is
system for publicly funding EIBI. more informative to present data that
Along with the need to test the epitomize experimental control than
generality of intervention effects to present data on a large number of
across service settings and providers, participants or on an average re-
it is also important to test the sponse to the intervention. As a
generality of effects across individu- corollary, null findings, which reflect
als who receive interventions and to a lack of experimental control, may
identify factors that predict success or be uninformative and hence unwor-
failure of an intervention for a given thy of publication.
individual. Despite the caveats noted However, behavior analysts are
in the introduction, meta-analyses interested not only in establishing
can help fulfill this objective by experimental control but also in
systematically organizing data ob- performing replication studies to
tained across many SSED studies. determine the generality of findings
(Because meta-analyses are also used (Johnston & Pennypacker (2009).
in group studies, they may have the Given that ABA interventions are
added advantage of presenting results not universally effective, this process
from SSED studies in a way that is must include documentation of both
understandable and convincing to successes and failures. Although, to
researchers outside ABA.) However, this writer’s knowledge, behavior
reliable information on generality of analysts have not described ways to
effects is difficult to obtain from the provide such documentation, we can
ABA literature because failures are consider approaches developed in
hardly ever reported. For example, other disciplines as a starting point.
data from all treatment articles in In group studies on intervention, the
Volumes 40 through 44 of JABA standard approach is to use an
indicate that every study participant ‘‘intention-to-treat’’ analysis (Schultz,
responded favorably to intervention, Altman, Moher, & CONSORT
ordinarily within a few sessions. To Group, 2010). In this kind of analysis,
be sure, the results were tempered in researchers report outcome data from
some cases by individual variations in all participants who entered the
response (e.g., Dunn, Sigmon, Thom- study, not just those who completed
as, Heil, & Higgins, 2008), incom- it. To put these data in context, they
plete generalization (e.g., Shillings- also describe the flow of participants
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 15 Cust # MS TBA12-1516 TRISTRAM SMITH through the study. This description display routine problem behavior includes the recruitment procedures such as talking out of turn, refusing (e.g., in person or by telephone or to complete tasks, or breaking rules. mail, from one agency or more), the Deliberately sampling such individu- inclusion and exclusion criteria for als may be an important step toward entry into the study, the number of ascertaining the utility of functional potential participants who were analysis across a wide spectrum of screened for eligibility but did not problem behavior. enroll, the reasons why they did not Overall, although many ABA in- enroll (e.g., declining to give consent), tervention procedures show consider- and the number of dropouts, along able promise, the tendency of SSED with the timing and reasons for studies on these procedures to take dropping out (Schultz et al., 2010). place in controlled settings, rely on The goal is to disclose the full breadth study personnel to deliver interven- of outcomes of intervention, includ- tions, and focus on successes may ing the possibility of not receiving the exaggerate the potential benefits of intervention at all, rather than pre- the procedures in practice. As a senting only successes. corrective, researchers would do well In other social sciences, researchers to take advantage of findings from who conduct studies with small sam- implementation science and related ples incorporate ‘‘purposeful sam- areas of research on how to carry out pling strategies’’ (Creswell, 2006; high-quality research in practice set- Miles & Huberman, 1994). For exam- tings. In addition, it may be advisable ple, researchers sometimes seek max- to use intention-to-treat analyses or imum variation in the target problem purposeful sampling strategies (or (e.g., problem behavior that ranges both). from mild to severe) or actively look Packaging intervention procedures. for disconfirming cases. Alternatively, Baer, Wolf, and Risley (1987) drew they may seek participants who are attention to a formidable technolog- considered typical or representative in ical challenge in developing interven- some way of a larger population. Such tion packages: the need to determine approaches could be valuable in which procedures in the package SSED studies. As an illustration, should be ‘‘followed faithfully, no although experimental functional matter where or when the program is analysis has been highly productive used,’’ and which procedures ‘‘users in ABA research, it is seldom used in should be allowed, and even encour- community practice (Dixon, Vogel, & aged, to modify … to fit their local Tarbox, 2012). Researchers have at- situations and contingencies’’ (p. 321). tempted to bridge the gap between In the years since Baer et al. high- research and practice by streamlining lighted this challenge, investigators functional analysis procedures (Iwata have made little progress in elucidat- & Dozier, 2008), showing that they ing principles that govern how to can be completed in real-world envi- strike an appropriate balance be- ronments (Mueller et al., 2011), and tween standardization and individu- testing their social validity (Lang- alization. The differing levels of thorne & McGill, 2011). However, standardization in ABA intervention another barrier may be that, because packages for individuals with ASD functional analysis emerged from (NAC, 2009) or other problems may work in inpatient or day-treatment reflect this uncertainty. programs with individuals who exhib- Nevertheless, investigators have it severe problem behavior (e.g., honed a technology for proposing a aggression or self-injury), it is not balance that can be tested: an inter- clear whether such analysis is neces- vention manual. The advent of man- sary or helpful with individuals who uals has transformed research on The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 16 Cust # MS TBA12-15
EVIDENCE-BASED BEHAVIOR ANALYSIS 17
behavioral, psychological, and edu- (Frankel, 2004). It may include one
cational treatments (McHugh & Bar- or more types of supporting materials
low, 2012). Manuals standardize in- such as handouts, worksheets, videos
tervention packages by giving step- for training interventionists or for
by-step instructions for each stage of illustrating concepts for consumers,
implementation, along with problems data sheets for monitoring progress,
that the user is likely to encounter and so on. A manual may undergo
and possible solutions to these prob- several revisions to take into account
lems (Hibbs et al., 1997). They also findings from initial testing of its
define the problem or population for acceptability (social validity) and
whom the intervention package is usability (extent to which providers
intended and the qualifications that implement procedures with fidelity
providers who implement the manual and recipients of the intervention
should possess (Hibbs et al., 1997). adhere to the protocol). As described
At the same time, they allow ‘‘con- previously, the manual may encom-
strained flexibility’’ (MacMahon, pass one intervention procedure or
2004) by delineating a limited set of many, provided that it offers a
acceptable variations. For example, solution to a problem presented in
the package may be divided into practice and strategies that commu-
modules with decision rules or as- nity providers can use to implement
sessment procedures for selecting the procedures effectively and effi-
which modules to implement and ciently. If the manual centers on only
under what circumstances (Weisz et a few procedures, it may be possible
al., 2012). The manual may describe to embed it within a research article;
different ways for providers to deliver otherwise, it is likely to require a
the package (e.g., procedures for separate publication (Chambless &
implementing the intervention one- Hollon, 1998).
to-one or in groups), incorporate the The availability of a manual is
consumer’s own interests into activi- widely viewed as a requirement for
ties, and collaborate with the con- classifying an intervention as evi-
sumer to set goals (cf. Kendall & dence based (Chambless & Hollon,
Chu, 2000; Kendall, Chu, Gifford, 1998; Society for Prevention Re-
Hayes, & Nauta, 1998; Tee & Ka- search, 2004). As stated by Cham-
zantsis, 2011). bless and Hollon, ‘‘research projects
Manuals may take a variety of for which a treatment manual was
forms. For example, a manual for not written and followed are of
outpatient therapy might lay out a limited utility in terms of assessment
sequence of 1-hr sessions, whereas a of treatment efficacy’’ (p. 11). This
manual for an educational program view is well founded. It would be
might consist of a curriculum in virtually impossible to generate rep-
which the learner has to master each licable findings about unstandardized
step before going to the next, along packages such as an ‘‘antecedent
with techniques for the teacher to use package’’ or a ‘‘behavioral package’’
in delivering instruction. For an ABA (NAC, 2009) to reduce problem
intervention package, it might be behavior. There are no criteria for
organized around functional rela- determining when to use one or the
tions (e.g., a set of procedures for other package, how many elements to
problem behavior that serves to include, which elements should al-
escape or avoid situations, another ways be included in the package and
set of procedures for behavior that which can vary across individuals, in
serves to gain access to tangible what sequence to introduce them,
items). The manual might present a what recourse is available if the initial
complete script for a session or package is insufficient, or how to
merely outline key points to cover monitor fidelity of implementation.
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 17 Cust # MS TBA12-1518 TRISTRAM SMITH Without such criteria, researchers of acceptance and commitment ther- and providers cannot determine apy (Christensen et al., 2004), the whether or not they have accurately community reinforcement approach replicated an unstructured package. plus vouchers more often than other In contrast, as previously noted, contingency management systems for interventions with manuals, such as substance abuse (Higgins et al., 2003), those developed within PBIS (e.g., and the comprehensive behavioral Dunlap et al., 2010), do include intervention for tics more broadly specific methods to address these than other packages that involve habit issues. Thus, manuals create the op- reversal (Piacentini et al., 2011). portunity to test a well-defined inter- The requirement for replicability vention package across consumers by independent providers (often ne- and providers, as must occur to amass cessitating the availability of a man- a body of research that would justify ual) sharply reduces the number of classifying an intervention as evidence ABA approaches that can be regard- based. Although future researchers ed as evidence based. Of the 11 may develop technologies other than interventions for ASD identified in manuals for devising replicable inter- the NAC (2009) report, only the three vention packages, manuals are cur- that have been standardized in man- rently the only such technology uals might be considered to be (McHugh & Barlow, 2012). practices, and even these may be In this writer’s judgment, the role incomplete. For example, a manual of manuals in establishing evidence- is available for writing stories to based practices may explain why the include in a story-based intervention efficacy of schoolwide PBIS interven- package (Gray, 2004), but not for tions has been evaluated much more how to use the stories to change the broadly (i.e., in much larger studies, behavior of an individual with ASD. with independent researchers) than Thus, there are many ABA proce- unstructured ABA packages for prob- dures and unstructured packages, but lem behavior (see Horner, Sugai, & relatively few practices. Anderson, 2010) or even ABA pack- Nevertheless, some possible draw- ages that have been described in more backs of manuals deserve mention. A depth but not presented in manuals potential weakness of practices that (e.g., functional communication train- involve intervention packages with ing; Tiger, Hanley, & Bruzek, 2008). manuals is that the step-by-step The same pattern holds for other instructions are liable to give the ABA interventions with manuals impression that unskilled providers compared to unstructured packages can implement the package (John- for the same problems. The efficacy of ston et al., 2006). In so doing, they the UCLA–Lovaas model of EIBI has may tempt agencies to employ such been replicated much more broadly providers, thus diminishing the effec- than any other comprehensive ABA tiveness of the package. It is certainly intervention for children with ASD true that, when demand has grown (Smith, 2011), pivotal response treat- for ABA interventions such as EIBI, ment more broadly than other ap- unskilled providers have proliferated proaches to incidental teaching (Koe- (Green, 1999). Thus, it may be gel, Koegel, Vernon, & Brookman- necessary to anticipate this unintend- Frazee, 2010), the picture exchange ed consequence by specifying mini- communication system more broadly mum qualifications for providers and than other forms of augmentative or disseminating information to con- alternative communication (Flippin, sumers about these qualifications Reszka, & Watson, 2010), integrative (e.g., Autism Special Interest Group behavioral marital therapy more of the Association for Behavior broadly than unstandardized forms Analysis, 2007; Shook, Johnston, & The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 18 Cust # MS TBA12-15
EVIDENCE-BASED BEHAVIOR ANALYSIS 19
Mellichamp, 2004). A manual is an to counsel parents on how to disci-
aid to clinical decision making, not a pline their typically developing chil-
substitute. With its balance of stan- dren. Parents are encouraged to use
dardization and flexibility, it struc- positive reinforcement frequently for
tures the package yet gives providers appropriate behavior and to discour-
latitude to tailor it to individual age acting-out behavior with proce-
cases. dures such as time-out rather than
Another concern is that manuals verbal reprimands or corporal pun-
may stifle expert providers (Johnson ishment. However, in keeping with
et al., 2006). Analogous concerns the previous discussion about the
have been voiced by professionals in difficulty of replicating unstandard-
other disciplines (see Addis & Kras- ized packages, little is known about
now, 2000). Research in these disci- how best to promote accurate imple-
plines provides scant evidence that mentation of such guidelines (Grim-
expert providers achieve better out- shaw et al., 2004).
comes if left to their own devices than In sum, interventions that are
if given a manual (e.g., Durlak & bundled into a package and detailed
DuPre, 2008; Woolf & Johnson, in a manual are not the be-all-and-
2005). However, the large and com- end-all of treatment research. Indi-
plex literature of SSED studies on vidual procedures in the form of
many ABA procedures raises the behavioral kernels and unstructured
possibility that there really could be packages in the form of guidelines
an advantage to letting expert pro- also have utility. Nevertheless, inter-
viders rely on the literature and their ventions with manuals are central to
experience rather than a manual. evidence-based practice because they
Indeed some writers have portrayed provide the integration necessary to
‘‘behavioral artists’’ (Foxx, 1996) address a problem thoroughly and
who have an uncanny knack for the standardization needed to repli-
creating and administering ABA in- cate findings across studies.
terventions. It may be worth evalu-
ating whether these providers are SSEDS OR GROUP DESIGNS
more effective without a manual. If
so, consistent with the technological Behavior analysts concede that
focus of ABA, the next step would be group designs have some valid uses,
to conduct research that seeks to particularly for addressing actuarial
encapsulate what these providers do questions such as whether one inter-
so that others can replicate it. Thus, vention tends to be more effective
this line of inquiry would lead toward than another (Johnston, 1988; Skin-
improving manuals rather than dis- ner, 1968). Nevertheless, we have
carding them. repeatedly asserted that SSED studies
An additional concern is that, are sufficient to establish an inter-
because standardized packages are vention as evidence based. To high-
more detailed than unstructured light the idiographic tradition in
packages, they may be more difficult ABA (i.e., the focus on evaluation
to disseminate in community settings. of interventions for each individual
Unstructured packages can be (and rather than for a group), Keenan and
frequently are) promulgated widely in Dillenburger (2011) described ABA
the form of practice guidelines or as evidence based because interven-
recommendations. For example, re- tions are evaluated carefully and
lying largely on ABA research, the continuously for each individual
American Academy of Pediatrics who receives them, and the standard
(Committee on Psychosocial Aspects intervention procedures are part of a
of Child and Family Health, 1998) general-purpose technology distilled
developed guidelines for pediatricians from more than a century of research
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:22 19 Cust # MS TBA12-1520 TRISTRAM SMITH in laboratory and field settings with monitor progress regularly, and to a variety of human and nonhuman adjust the plan as new information populations. Examples of such pro- becomes available. In the process, it cedures include differential reinforce- makes sense to start with tried-and- ment, prompting and prompt fading, true procedures such as shaping. shaping, and chaining. ‘‘But the dangers of drifting too far To support their position, Keenan toward the idiographic are that we and Dillenburger (2011) cited Reese, end up reinventing the wheel every Howard, and Reese’s (1978) general time a new client comes into the procedure for applied behavior anal- office or clinic’’ (Barlow, 1997, ysis, which includes strategies for p. 448). As shown in the preceding identifying target behaviors, conduct- section, reinventing the wheel is a ing a task analysis, selecting among formidable undertaking. Thus, inves- general-purpose ABA interventions, tigators have recommended ways to and testing outcomes. Comparable standardize even dynamic processes systems have been proposed by oth- such as shaping (Galbicka, 1994). ers. For example, Kameenui and Moreover, documenting that the Simmons (1990, p. 87) proposed ‘‘five wheel is working requires more than generic design-of-instruction fea- just showing an uptick on a graph of tures’’ for academic tasks. These an individual’s behavior. A behavior features involve the use of ABA may change but remain a problem for principles to assess what children the individual, or may be only a small need to learn, to select curriculum component of a much larger cluster materials, to plan lessons, to gener- of problems such as addiction or alize skills to new contexts, and to delinquency (Baer et al., 1987). Also, pinpoint successful and unsuccessful a procedure may be appropriate for aspects of the lessons. Reese et al.’s an individual yet can be administered and Kameenui and Simmons’s sys- in the context of a program that may tems present their frameworks for not be inappropriate. implementation of ABA technology Such complications abound in without reference to any specific ABA intervention, as illustrated by studies to support their deployment the following examples: for a particular problem or with a particular population of individuals. 1. Many ABA interventions have Keenan and Dillenburger (2011) been directed toward increasing the used the example of shaping to rate at which preschoolers answer illustrate why a procedure can be questions appropriately (Ingvarsson, considered evidence based without Tiger, Hanley, & Stephenson, 2007) the need for prior RCTs or other or the rate at which children with studies to document its effectiveness. disabilities initiate or respond to They note that, when implemented overtures from peers (Odom et al., for an individual learner, shaping is 1999). However, such gains may not not based on ‘‘fixed treatment reci- be enough to overcome the children’s pes’’ (p. 8) copied from previous initial problems in these areas. Fur- research. Rather, it is a dynamic thermore, to be liked and make process that is individualized for the friends, children are likely to need learner and that evolves continuously many other skills (e.g., skills for based on data obtained on the sustaining interactions around shared individual’s progress. These data interests and for resolving conflicts). constitute the evidence on the efficacy 2. Most ABA interventions for of shaping for that individual. teaching children with disabilities to Certainly, it is necessary to develop read focus on sight reading, but an individualized intervention plan literacy also entails other critical read- for each recipient of services, to ing skills such as decoding, phonemic The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:23 20 Cust # MS TBA12-15
EVIDENCE-BASED BEHAVIOR ANALYSIS 21
awareness, and comprehension (Brow- the exigencies of field settings seldom
der, Wakeman, Spooner, Ahlgrim- allow providers the luxury of spend-
Delzell, & Algozzine, 2006). ing so much time on data collection
3. Some ABA studies have shown and analysis (Johnston, 1993). Thus,
success in teaching children to handle some behavior analysts emphasize
guns safely, but, as investigators have the need for a sequence of studies
emphasized, it is at least as important on an intervention, in addition to
to help families store firearms secure- case-by-case evaluation. Johnston
ly (Miltenberger et al., 2005). (1993, 1996) recommended beginning
4. Most studies of ABA interven- the sequence with studies in relatively
tions for feeding difficulties docu- controlled settings and then moving
ment an increase in acceptance of to more typical field settings. The
novel foods. However, some popula- sequence consists entirely of SSED
tions that are prone to feeding studies instead of actuarial group
difficulties, such as children with studies.
ASD, are seldom malnourished or To illustrate this sequence, John-
underweight (Hyman et al., 2012). ston (1996) cited a series of studies
Further, investigators seldom show that led to the creation of an inter-
that a particular child with ASD in a vention for teaching women to per-
study on ABA feeding interventions form breast self-examinations. The
has health problems (but see McCart- first studies elucidated the character-
ney, Anderson, & English, 2005, for a istics of lumps in breast tissue. Later
notable exception), Thus, for many studies refined procedures for teach-
of these children, outcomes other ing women to identify these lumps
than acceptance of novel foods are in laboratory settings. Finally, studies
probably more important. Such out- tested the procedures under condi-
comes might include reduced stress at tions that increasingly resembled
mealtimes for children and caregivers real-world situations (Pennypacker
or a more varied overall diet that & Iwata, 1990). Johnston described
promotes long-term health. the teaching procedure as ‘‘the state
5. The goal of EIBI is to normalize of the art in breast self-examination’’
the functioning of children with ASD (p. 41). Pennypacker (2010) lamented
by helping them catch up to peers that the Cochrane Collaboration,
(Lovaas, 1987). However, despite the which conducts influential reviews
acquisition of many new skills, some of the medical literature to identify
children do not reach this goal evidence-based practices, concluded
(Smith, 2011). For these children, that breast self-examination failed to
providers and families need to go qualify as evidence based. Indeed,
beyond the data on acquisition of SSED studies on ABA procedures for
individual skills to determine whether breast self-examination provide im-
adjusting the intensity, content or pressive analyses of what to teach
method of instruction, or goals and how to teach it (see Pennypacker
would make the program a better fit & Iwata, 1990).
(Delmolino & Harris, 2011). However, the status of breast self-
examination as an evidence-based
In principle, providers could at- intervention hinges on a quintessen-
tempt to address these issues by tial actuarial question: Does it pre-
conducting exceptionally rigorous, vent deaths from breast cancer? The
thorough evaluations of an individu- Cochrane review examined RCTs
al’s progress, with SSED studies of that addressed this question and
all behaviors of concern and data on found that the answer was no,
social validity (the extent to which perhaps because learning a discrete
others see the intervention as accept- procedure for self-examination fails
able and useful). In reality, however, to address barriers to performing the
The Behavior Analyst bhan-36-01-02.3d 18/4/13 09:17:23 21 Cust # MS TBA12-15You can also read
