California CASA Youth Outcomes Project - Summary

August 2013

California CASA Youth Outcomes Project – Summary

Introduction
Those concerned with child welfare are all too familiar with the studies that show foster youth suffering
from disproportionately poor outcomes. Statistically high rates of homelessness, involvement with the
justice system, poor educational outcomes all plague the youth we serve. Many advocates in California
are desperate to ensure a positive future for each child we serve; and we need to do a better job with
our youth who find themselves in the foster care system.

Child welfare practice should be rigorously scrutinized to determine what works and what doesn’t. This
scrutiny can exist in many forms; perhaps the most compelling of which is when youth actually call out
what made the difference in their lives, which is perhaps the only true way of measuring effectiveness.
However many fields, including the field of child welfare are increasingly demanding more scientific
proof, often rom a randomized control‐group study that results in an evidence‐base proving up the fact
that interventions are effective.

This paper discusses the process that California CASA and Researchers from UC Berkeley have
undergone to try to design and implement such a study, the outcome, and the next steps proposed by
the UC Berkeley Research team. Dr. Jill Duerr Berrick presented her concluding thoughts in a brief
entitled “CASA Effectiveness Study, UC Berkeley Considerations,” that can be found here:
www.californiacasa.org/casa‐youth‐outcomes‐project

The CASA Model
In California, the CASA model consists of recruiting, training, and supervising a community volunteer
who is matched one‐on‐one with an abused and/or neglected child. That volunteer becomes a sworn
officer of the court – a CASA – and then serves at the discretion of the court to investigate the
circumstances of the child and make best interests recommendations to the court. Additionally, the
CASA volunteer spends considerable time with the child, helping the child with the foster care system
and advocating for their needs in in both the court and the community.

We believe that the CASA model in California provides children and youth with additional service
components that benefit children. Namely, as mentioned, CASA volunteers spend significant one‐on‐
one time with youth – mentoring them and helping them navigate life and the foster care experience.
Also, CASAs investigate and provide firsthand information and best interest recommendations to the
court. Finally, the CASA model is to use trained community volunteers, who do not have certain limits
that might come with a more bureaucratic structure.

                                                                                                       1
California CASA Association 2013

We know that studies have proven mentoring as an evidence‐based model, and the CASA model in
California includes mentoring the youth.1 However, what the research base lacks is a study definitively
measuring the effectiveness of CASA’s rich, more complex model of service. There is no study showing
that the full panoply of CASA service, not just the pieces of CASA service, is an evidence‐based practice.
In the current funding landscape, this has very real consequences for CASA programs, and thus the
provision of CASA volunteers to children and youth.

Existing CASA Research
Judges, children and child welfare system partners across the state clearly report that they trust CASA.2
Further, we know from thousands of success stories that CASA immeasurably improves the lives of
children and youth; however, in today’s philanthropic environment, it is useful to have a study bear out
that benefit.

Considerable effort has been spent researching the CASA model, in its various forms. However, the
research is not sufficient to dub CASA an “evidence‐based practice.” Therefore, in 2010, UC Berkeley
and California CASA Association began talking to Juvenile Court judges and local CASA programs in an
effort to explore the possibility of conducting a randomized study to measure CASA efficacy.

This effort has taught us much about the challenges of measuring the efficacy of a model like that
provided by CASA; and has also moved the effort into the next stage. Now, we have a good literature
review of existing CASA research, as well as ideas about how to accomplish a significant study in the
future, as suggested by the UC Berkeley Research Team, Jennifer Lawson and Jill Duerr Berrick.3 Also the
recent work has provided a direction to the CASA Network’s “stronger, more systematic, and uniform
data collection process” that will certainly provide evidence of CASA efficacy.

Exploration of a Randomized Control‐group Study
The Study Design

In 2010, California CASA and UC Berkeley, with the generous support of the Walter S. Johnson
Foundation, began exploring the feasibility of conducting a study to determine CASA program
effectiveness in selected counties throughout California. The general idea was to choose four local CASA
programs, based on their workability for such a study (i.e. program diversity and size and ability to have
enough referrals during the study to provide statistically significant data).

Since the one thing that everyone seemed to agree on was that judges appoint CASA volunteers to the
most difficult cases, there needed to be an experimental design that compared children referred to

1
  This is not true of CASA programs in all states. Models of child welfare services and legal representation of minors vary among
the states with 51% of CASA programs operating with a friend of the court model similar to California, 26% with a guardian ad
litem (GAL) model, and 19% using a friend of the court/GAL Attorney team model.
(National CASA Association Local Program Survey 2012).
2
 See Mensing, James F. (2008). Court Appointed Special Advocates and the Courts: An Assessment. Judicial Council of California
Administrative Office of the Courts. Available online as of 9/1/2013 at:
http://www.courts.ca.gov/documents/CASA_Assessment_Research_Summary_03‐16‐09_Final.pdf
3
 See Jennifer Lawson & Jill Duerr Berrick (2013) Establishing CASA as an Evidence‐Based Practice, Journal of Evidence‐Based
Social Work, 10:4, 321‐337, DOI: 10.1080/15433714.2012.663674 Available online as of 9/1/13 at:
http://www.tandfonline.com/doi/abs/10.1080/15433714.2012.663674#.UiZOVBDpx8E

                                                                                                                               2
California CASA Association 2013

CASA with similarly situated children (e.g. cases with similar levels of difficulty). It was not enough to
merely compare youth served by CASA with those that were not. Therefore, the UC Berkeley team had
to create an experimental design that account for this.

The design: one hundred children would be taken into the study on a first‐come, first serve basis. One
child would receive CASA services and one would not; and instead be placed on the waitlist. Once on
the waitlist, the child could not receive services of any kind until the data collection portion of the study
was complete, which would be at least one year. Thus, even if the ordinary practice in that local
program would be to provide “triage” services to a child, such as helping to arrange special education
services, this would be halted for the period of the study.

Then, researchers would pull set data from social worker reports and court case files to identify youth
outcomes along the three federal child welfare goals: Safety, Permanency, and Wellbeing. These data
would then be compared to the CWS/CMS, and comparisons would be made with federal CFSR (Child
and Family Service Review) outcomes and AB 636 (Child Welfare System Improvement and
Accountability Act of 2001) indicators. The comparison of these data would presumably indicate what
effect, if any, a CASA volunteer has on the identified youth outcomes at the end of one year of service.

Ethical Considerations

  Unfortunately, using randomized designs for CASA research is an aspiration that is not easily
  realized. In the real‐world functioning of dependency courts, child welfare services, and CASA
  advocacy, there tend to be ethical qualms about the random denial of services that would be
  required for a randomized design.
                 ‐ Jennifer Lawson & Dr. Jill Duerr Berrick, Establishing CASA as an Evidence‐Based
                                                             Practice

Researchers felt that since some CASA programs had an existing waitlist, there would not be any
“denial” of CASA services in those counties. While one child would be excluded for a period of time,
another child would have a CASA appointed to him or her. The waitlist, the argument goes, made it
such that children as a whole would not be denied a CASA.

However, some CASA program staff as well as some Judges did find this argument persuasive. While it
was true that fifty (50) children would have the benefit of a CASA, there were fifty (50) others that
where to excluded for the year of the study. Therefore, if seven year old “Billy” needed a CASA but he
fell into the control group, Billy would have to go without. The thought that another, perhaps “equally
deserving” child would enjoy the benefit of a CASA was seemingly cold comfort. Researchers tried to
mitigate this issue by allow each CASA program the ability to pull out up to five (5) children from the
control group if their circumstances were so dire to require it. This, too, was not enough to convince
some of the stakeholders.

Other Considerations Affecting Research

Another issue with the study was the ability to maintain, and thus test, a true CASA model of advocacy.
Because each CASA program is designed to serve the needs of the local children, families, and the court,
programs had slight variations of service. Some programs, for example provided some services to

                                                                                                           3
California CASA Association 2013

children on its waitlist, others did not. Some had special projects aimed at serving targeting
populations, like medically fragile youth, or 0‐5 year olds, others did not. Also there were issues with
selection bias, as it seems evidence strongly suggests that judges appoint CASA volunteers to the most
difficult cases. Further, the way CASA programs “subjectively and intuitively” match children to
volunteers presents further issues. By randomizing the assignment process, it seems that a study would
change some of the very things it is attempting to measure.

Study Currently Unfeasible

The UC Berkeley Research Team concluded that because of these issues, especially the ethical issues
that some judges and CASA program staff had with the control group design, the study is unfeasible.

          The research team feels strongly that rigorous and methodologically sound research would
          be of value to children and CASA stakeholders, but only if it adds substantially to the
          knowledge base regarding the effectiveness of CASA for improving outcomes.
          Unfortunately, there are substantial barriers to conducting this type of research in the
          context of real‐world CASA functioning, and these barriers appear insurmountable at this
          time.
                       ‐ Dr. Jill Duerr Berrick, CASA Effectiveness Study, UC Berkeley Considerations

The unfeasibility of a study of this type is not surprising. It is certainly not the first attempt to study the
complexities of the CASA model. Indeed, many know that there are significant limitations to requiring
an experimental method based study for social service programs.4 Further, some are beginning to
realize that some social service interventions are simply not designed to fit a control‐group type study.
Smyth and Shorr, in their working paper “A Lot to Lose: A Call to Rethink What Constitutes “Evidence” is
Finding Social Interventions that Work, ”point out that promising practices that work may fail to receive
the funding and support that they need just because they lack a “linear relationship between cause and
effect.”5 Indeed, “[w]e risk not being able to make reliable judgments about the effectiveness of those
programs for which experimental‐design evaluation is a poor fit.6

Collection of Youth Outcomes Data

      Although a randomized trial is not currently feasible, we believe that CASA agencies
      and the field more generally could benefit substantially from a much stronger,
      more systematic, and uniform data collection process that would result in outcome
      indicators – a first step in describing “what happens” for children involved with the
      CASA program.
                       ‐ Dr. Jill Duerr Berrick, CASA Effectiveness Study, UC Berkeley Considerations

4
 See Smyth and Schorr (2009) A Lot to Lose: A Call to Rethink What Constitutes “Evidence” is Finding Social Interventions that
Work. Working Paper Series: Malcolm Wiener Center for Social policy, Harvard, Kennedy School of Government. Available
online as of 9/1/13 at: http://www.hks.harvard.edu/ocpa/pdf/A%20Lot%20to%20Lose%20final.pdf
5
    Id.
6
    Id.

                                                                                                                                 4
California CASA Association 2013

Since the UC Berkeley team determined that a randomized study seems beyond the California CASA
Network’s reach at this time, UC Berkeley began to inform ways in which CASA programs could begin to
identify and collect data on youth outcome indicators.

In July of 2012, UC Berkeley researchers with California CASA brought together representatives from 16
local CASA programs, and the California Administrative Office of the Courts, Center for Families,
Children, and the Courts in an effort to identify youth outcome indicators that could be used to show
CASA volunteer efficacy. This convening resulted in local CASA programs, under the direction of the UC
Berkeley Researcher, identifying CASA volunteer efforts that could affect youth outcomes in the
following six domains:

       Domain 1: PHYSICAL, MENTAL, AND BEHAVIORAL HEALTH AND DEVELOPMENT
       Domain 2: WELL‐BEING
       Domain 3: EDUCATION
       Domain 4: INDEPENDENT LIVING
       Domain 5: PERMANENCY
       Domain 6: PLACEMENTS

As a result, this year, with funding from the Walter S. Johnson Foundation, the CASA database program
CASA Tracker will be modified to allow programs to systematically collect data in these domain areas,
and longitudinally track whether progress is made, and goals achieved. This software modification is
currently (as of July 2013) underway. Next, with the investment of Kaiser Permanente California
Community Benefit Programs, California CASA is preparing to work closely with six local CASA programs
to pilot the collection of the youth outcomes data. This will allow for the software interface to be
improved, but more importantly, get everyone on the same page, answering the same questions, in the
same way across programs. The goal is to have reliable data that show, for example: “70% of children
served by the CASA program received educational advocacy services from their volunteer, and of those
children that needed an IEP, 90% had one in place within three months.” Once we can do that, then we
can begin to analyze the aggregate data received from all programs that are participating.

Next steps will include: piloting the outcomes collection program, including the tool, and guidebook over
a year’s time, and then opening up CASA Tracker’s data collection capability to the entire CASA Network
with care to ensure fidelity of data collection.

Conclusion
While it is clear that CASA is highly valued and well respected, especially among judges and child welfare
professionals, research hasn’t brought the necessary rigor to deem CASA an evidence based practice.
So, while a randomized control group study is not currently feasible; CASA is making progress toward
collecting data to show the effect that CASA volunteer advocacy has on youth outcomes.

                                                                                                        5
California CASA Association 2013