Capital One/JA Finance Park - Simulation Model Evaluation
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Simulation Model Evaluation:
Capital One/JA Finance Park ®
TECHNICAL REPORT: MARCH 2017
Written for Capital One® and Junior Achievement USA®
By Princeton Research and Evaluation Partners LLC (PREP)
Evaluation made possible through the generous support of Capital One
ACKNOWLEDGEMENTS
JA of Central Upstate New York JA of New Jersey JA of South Florida
JA of Oregon and SW Washington JA of New York JA of Tampa Bay
JA of Central Virginia JA of Northern California JA of Washington
JA of Eastern Iowa JA of Oklahoma JA of Delaware
JA of Greater Washington JA of Kentuckiana
Princeton Research and Evaluation Partners LLC (PREP) is grateful to the 14 JA local areas who participated in
the JA Finance Park Usability Study and Simulation Evaluation. Their collaboration in data collection made this
study possible.
We are grateful as well to JA USA and JA of Greater Washington for granting us permission to use photos in
this report. In addition, we are particularly appreciative of the support that JA USA provided to us throughout all
stages of the work. Dr. Joe Thomas, Dr. Lavonne Henry, and Nancy Tim were invaluable in our efforts to secure
all the data needed and access all parties critical to evaluating the simulation experience. The findings presented
in this report, however, are an independent, third-party evaluation of JA Finance Park by Princeton Research and
Evaluation Partners.
The financial support of Capital One made this evaluation possible. We are honored to have been selected
to conduct this evaluation and appreciate the generous support of the donor in making possible an objective
assessment of program impact.
Prepared by PREP for Junior Achievement USA
2
ABOUT THE AUTHORS
Princeton Research and Evaluation Partners LLC (PREP) serves the K-16 sector, specializing in youth
development and education. PREP partners with clients to design research tailored to their unique program
environments and organizational needs. PREP’s collaborative approach brings data to life for a diverse array of
audiences. We empower clients to make pragmatic, cost-effective decisions to advance their goals and provide
objective analysis assessing mission achievement.
PREP’s expertise spans planning and evaluation, program/product development, organizational analysis, public
policy, and diversity.
Report authors, Catherine J. VonFange and Debra L. Dodson, PhD, each have more than two decades’
experience leading research and evaluation in the non-profit, for-profit, and academic sectors.
Catherine VonFange is Chief Consulting Partner with PREP. She leads efforts in market research, business
research services, and statistical analysis. Prior to joining PREP, Cathy served at Abt SRBI as a Sr. Vice Presi-
dent for the Financial Services and Non-profit sectors. Her focus has been in market assessment, new product
development, customer segmentation, benchmarking, competitive analysis, and customer value management.
Ms. VonFange’s numerous accomplishments include: 1) Designing a leading US marketing research practice
for clients that included Fortune 100 Consumer Packaged Goods, Financial Services, Health Care, Retail,
Media, Technology, and Communications companies; 2) Providing new product development consulting for a
national brand of meal replacement products from ideation through successful launch of five new brand ex-
tensions; and 3) Guiding the development and introduction of an innovative retail channel business model and
resource allocation process for a consumer electronics manufacturer through benchmarking and best practices
implementation.
Prior to joining Abt SRBI, Ms. VonFange was a Partner with Yankelovich Incorporated. In this capacity she was
responsible for designing, developing, and deploying the Multicultural MONITOR and all related research. Prior
to her work with the Multicultural team, Catherine was Vice President of Client Services with The Segmentation
Company where she was an integral part of delivering highly-actionable customer segmentation models and
targeting tools. Ms. VonFange is a graduate of Kansas State University.
Prior to establishing PREP, Debra Dodson was Director of Program Evaluation and Outcomes Measurement
for Girl Scouts of the USA (GSUSA). Dodson led the conceptualization, design, and implementation of the
innovative, National Program Evaluation System. That system shifted the 100-year old Girl Scouting movement
into a new era of data-driven program assessment and improvement. As Director, she conducted evaluations in
partnership with the 112 local units.
Prior to joining GSUSA, Dodson was a research faculty member at the Eagleton Institute of Politics, a unit of
Rutgers University. There she conducted policy and political research. Among her many publications is the
Impact of Women in Congress (Oxford University Press, 2006). Debra L. Dodson is a graduate of University
of Michigan (PhD) and Northwestern University (BA/MA). She is currently pursuing an MS in Integrated
Marketing Communications from Northwestern University.
Simulation Model Evaluation: Capital One/JA Finance Park 3
TABLE OF CONTENTS
Executive Summary ................................................................................................................................................7
Introduction ..............................................................................................................................................................8
Background ......................................................................................................................................................8
Purpose and Objectives ...................................................................................................................................9
Method ...........................................................................................................................................................10
Sample Design .....................................................................................................................................10
Data Collection ..................................................................................................................................... 11
Data Processing ...................................................................................................................................12
Data Analysis .......................................................................................................................................12
Top-Two Box Analysis ................................................................................................................12
Effect Size Analysis ....................................................................................................................12
Report Overview ............................................................................................................................................13
Detailed Findings ...................................................................................................................................................14
Pre-to-Post Survey Comparisons ..................................................................................................................14
Were there pre-to-post changes in overall student knowledge, confidence,
behavioral intentions, and attitudes? ..........................................................................................14
Financial Confidence ...........................................................................................................................15
Confidence by Simulation Mode..................................................................................................15
Confidence by Grade Level ........................................................................................................16
Financial Behavior ...............................................................................................................................18
Behavioral Change by Simulation Mode ....................................................................................18
Behavioral Change by Grade Level............................................................................................19
4
Educational Aspiration .........................................................................................................................21
Post-Simulation Overall Comparisons............................................................................................................21
How do overall student post-survey evaluations compare across
modes and by grade level? ........................................................................................................21
Overall Simulation Experience Evaluation ...........................................................................................22
Indicator of Overall Satisfaction by Simulation Mode ................................................................. 22
Perceived Change in Financial Confidence as a Result of Participation
in JA Finance Park Simulation .............................................................................................................23
Importance of JA Finance Park Simulation to My Future...................................................................... 24
Usefulness of JA Finance Park Simulation to My Life ..........................................................................25
Student Description of the JA Finance Park Simulation Experience ............................................................. 27
Impact of the Interval Between JA Finance Park
Classroom Work and Simulation ..........................................................................................................28
The Most Helpful Activity of the JA Finance Park Simulation ............................................................... 29
The Learning that Stuck with Students Most ......................................................................................30
Student Assessment of JA Finance Park Simulation’s Impact on Them ............................................ 32
The Most Valuable Aspect to Students .........................................................................................................33
What was the most valuable part of the JA Finance Park Simulation
from the student’s perspective? .................................................................................................33
Endnotes ................................................................................................................................................................36
Appendices ............................................................................................................................................................37
Simulation Model Evaluation: Capital One/JA Finance Park 5
‘‘ The most valuable part of
JA Finance Park is the whole idea
of being a person in a simulated
world and having to make the ‘‘
tough decisions on how to
budget and spend your money.
– 7th Grade Student, Site-based Simulation
EXECUTIVE SUMMARY
The JA Finance Park Model Simulation Evaluation n Students found JA Finance Park simulation fun,
is designed to assess site-based, mobile, and virtual unique, and meaningful – but also challenging.
simulations from the perspective of the ultimate
l Across modes, students ranked ‘fun,’ ‘unique,’
end-users – the students. The main purpose is to ‘challenging,’ and ‘meaningful’ as top at-
compare the effectiveness of the simulation modes. tributes that describe the JA Finance Park
simulation.
ll JA Finance Park simulations achieved
nA
significant gains in student pre-to-post l More site-based students mentioned
financial confidence and ratings. 'challenging' than those experiencing the other
modes.
l or all modes, confidence levels on “I know
F
how to handle financial risk” and “I can find n Regardless of simulation mode, the majority of
solutions to money problems” rose signifi- students gained the knowledge JA Finance Park
cantly following the simulation. simulation intends to impart.
l oth “I (will) use a budget to plan my spend-
B l Pre-to-post simulation comparisons showed
ing and savings” and “I (will) talk with adults substantial improvement in students’ ability to:
about my financial goals” ratings increased understand how to manage money, make the
significantly for all modes. connection between classroom and ‘real life,’
lA
bout 2/3 of students from site-based (66%), find new ways to solve financial problems, and
mobile (68%), and virtual (63%) modes easily understand finance.
reported increased financial confidence as a l When students apply their academic
result of JA Finance Park simulation. knowledge to “real life” in the JA Finance Park
nT
he majority of students believe most or nearly simulation, they (regardless of simulation
all of what they learned in JA Finance Park will mode) make significant pre-to-post gains on
be useful in their lives. financial literacy outcomes (e.g., financial
confidence, resolve to use budgeting in their
l ite-based (82%), mobile (85%), and virtual
S lives, etc.).
(80%) participants estimate that most or
nearly all of what they learned through the n The JA Finance Park Simulation Model
simulation will be useful in their lives. Evaluation suggests a combination of in-class
and experiential learning yields greater student
l ite-based (31%) and mobile (35%) students
S gains on financial literacy outcomes1 than does
say the simulation will be extremely import- even highly impactful classroom instruction
ant to their futures while only (19%) of virtual without a simulation experience.2
students think the same.
l alf of all site-based, 54% of mobile, and
H
46% of virtual students would like to take
more courses like JA Finance Park.
Simulation Model Evaluation: Capital One/JA Finance Park 7
INTRODUCTION
Background JA Finance Park®, the JA Capstone program for
secondary students, introduces middle and high
Junior Achievement (JA) is the world’s largest school students to key concepts in personal finance
non-profit organization dedicated to giving young and budgeting, building a foundation for making solid
people the knowledge and skills they need to own financial decisions throughout their lives. JA Finance
their own economic success, plan for their future, and Park combines 13 required classroom-based lessons
make smart academic and economic choices.3 Junior with 21 optional hands-on extension activities. These
Achievement’s purpose is to inspire and prepare culminate in a hands-on budgeting simulation.7
young people to succeed in a global economy.4
JA’s focus is three major core content areas: work
readiness, entrepreneurship, and financial literacy. The JA Finance Park simulation is offered
Junior Achievement’s program ignites the spark in in three distinct delivery modes:
young people to experience and realize the opportuni-
ties and realities of work and life in the 21st-century.5 nS
ite-based – a permanent facility, typically
near a Junior Achievement office;
The 109 Junior Achievement USA® offices, distribut-
ed across the country, recruit volunteers to provide nM
obile – a facility designed to be movable
in-school and after-school programs to students in from location to location; and
Grades K-12. Over 237,680 volunteers serve more
irtual –
nV an online, web-based experience
than 4.8 million students in 209,651 classrooms and
accessible at school or home.6
after-school locations in the USA.6
JA USA has developed and implemented an impres-
sive array of programs. JA’s volunteer-delivered, K-12
In each mode, students receive a simulated life pro-
programs foster work readiness, entrepreneurship,
file, with a family situation, job and salary, and debt
and financial literacy skills using experiential learn-
level. Students then learn about a comprehensive set
ing to inspire students to dream big and reach their
of budgeting areas, set budget targets for each area,
potential. Age-appropriate curricula are designed to
make spending choices, and discover how to spend
teach children how they can impact the world around
within their means.
them as individuals, workers, and consumers. These
programs begin at the elementary school level and
continue throughout middle and high school.
8
JA Finance Park Simulation Model The study objectives guided the development and
Evaluation: Purpose and Objectives administration of paper and web-based student
surveys, analysis, and reporting. Specifically, the
In Fall 2015, JA USA engaged Princeton Research JA Finance Park Simulation Model Evaluation was
and Evaluation Partners LLC (PREP) to conduct a designed by JA USA and PREP to address the
strategic research initiative to evaluate the current following research objectives:
JA Finance Park simulation component. The purpos-
es were to: ensure that current goals are met, inform
uu Assess pre-to-post survey changes in overall
ongoing improvement, suggest new developments/
student perceived knowledge, behavioral
enhancements, identify best practices, assess intentions, and attitudes by simulation mode
current results, and compare the effectiveness of and grade level for:
the three simulation delivery modes.
• Financial Confidence:
The initiative is divided into two phases: · manage money well,
· handle financial risk, and
n The JA Finance Park Usability Study, summa-
· find solutions to money problems.
rized in a separate document, focuses on the
• Current and Future Behaviors:
perceptions of key adult stakeholders (e.g., JA
local area presidents and staff, educators, and · use a budget,
volunteers) integral to reaching students with the · set spending limits and savings goals, and
simulation. · talk to adults about finances.
• Educational Aspirations.
n The JA Finance Park Simulation Model Evalu-
ation is reported in this document. It analyzes uu Compare pre- and post-survey experiential
pre- and post-survey data from students who results across the three simulation modes
completed a site-based, mobile, or virtual and by grade level for:
simulation in 2016. • Overall simulation experience ratings:
· likelihood to recommend to a friend,
· importance to future, and
· usefulness in life.
• Simulation impact on:
· having new ways to make financial decisions,
“The most valuable thing · better understanding money management,
· making finance easier to understand, and
about the JA Finance Park · connecting classwork and real life.
• Confidence overall in handling finances.
simulation was learning how uu Student perceptions of simulation value:
to manage money so you • Most important learning,
• Most memorable aspect, and
can be successful in life.” • Interest in further JA Finance Park
experiences.
– 10th Grade Student, Virtual Simulation uu Provide recommendations and conclusions:
• Identify gaps between modes.
• Suggest improvements in JA Finance Park.
Simulation Model Evaluation: Capital One/JA Finance Park 9
Method Table 2 details the number of surveys by simu-
lation mode. All pre-to-post-survey comparisons
Sample Design. JA USA selected 14 JA local areas use matched pairs. All questions asked only in the
from across the country to provide a good cross sec- post-survey use all post-survey participant responses.
tion of local areas representing the three simulation
modes. Table 1 (below) lists JA local areas participat- TABLE 2
ing in the evaluation by simulation mode. Student Participants by
JA Finance Park Simulation Mode
TABLE 1
Matched Pairs Total Student
Local Area Participants by
for Comparison Surveys
JA Finance Park Simulation Mode
Site-based Mobile Virtual Pre- Post-
Matched Pairs Surveys Surveys
JA of Delaware JA of Central Virginia JA of Eastern Iowa
Site-based 2817 4777 4046
JA of Washington JA of New Jersey JA of Oklahoma
Mobile 1960 2710 2916
JA of Kentuckiana JA of New York JA of Tampa Bay
JA of Greater JA of Oregon and JA of Northern Virtual 958 2697 1347
Washington SW Washington California
JA of Central Total 5,735 10,184 8,309
JA of South Florida Upstate NY
In addition to comparison of modes, PREP observed
The JA Finance Park Simulation Model Evaluation
responses by school grade. Once again, for pre- and
meets study objectives with a combination of quanti-
post-simulation comparisons, the matched sample is
tative data from three sources:
used. For all other questions asked only in the post-
n 0,184 pre-surveys conducted with students
1 survey, all student responses are included. Table 3
prior to exposure to a site-based, mobile, or lists the student sample by grade level.
virtual simulation experience;
TABLE 3
n 8,309 post-surveys conducted with students
immediately following completion of a site- Student Participants by
School Grade Level
based, mobile, or virtual simulation experience;
and Matched Pairs Total Student
for Comparison Surveys
n ,735 pre- and post-survey matched pairs
5
conducted with students who completed both Pre- Post-
Matched Pairs
surveys. (The Pre-Survey and Post-Survey Surveys Surveys
instruments are in Appendix A.) 7th Grade 1399 2174 2022
8th Grade 2150 3681 3053
“Most valuable was learning 9th Grade 855 1263 1179
10th Grade 411 599 653
how to budget, save, and
11th Grade 307 580 530
manage my money.” 12th Grade 613 1887 872
Total 5,735 10,184 8,309
– 12th Grade Student, Mobile Simulation
10In addition to simulation model and school grade, Students completed a 5-minute paper or an Internet
some data are viewed by length of time since last pre-survey after completion of classroom work and
classroom work. The length of time between class- before commencing JA Finance Park simulation ac-
room work and simulation experience was asked tivities. Then students completed a 10-minute paper
only in the pre-simulation survey. Thus, only matched or Internet post-survey at the conclusion of the JA
pairs can be used to assess differences in results by Finance Park simulation. All students were promised
the length of time between classroom and simulation individual confidentiality. All were reminded that they
experiences. Sample size by length of time since could opt out of the survey entirely or refuse to answer
classroom work is provided in Table 4. any individual questions.
TABLE 4 PREP provided data collection support by providing on-
Student Participants site presence for the three days of data collection at four
by Recency site-based, four mobile, and one virtual local area. PREP
provided all sites using paper surveys, the pre- and
Matched Pairs
for Comparison
Total post-surveys, instruction sheets, and pre-paid return
materials. Virtual sites received preliminary teacher
Matched Pairs Pre- % recruitment assistance (see Appendix B) and instructions
Surveys
for web-based survey completion. Some areas received
< 1 Month 3233 5399 53% teacher incentives to assist with completion rates. In-
structional materials and examples of preparatory, teach-
> 1 Month 724 1141 11% er e-mail, and reminders are provided in Appendix B.
No Classes 889 1847 18% It is important to note that the JA Finance Park Sim-
ulation Model Evaluation is based on comparisons
Don’t know / of ratings from students who experienced only one
Prefer not 889 1797 18%
to say simulation mode. Thus, the three sets of ratings are
independent and mutually exclusive. Comparison
Total 5,735 10,184 100% across the modes is intended to provide an assess-
ment of how well students who experience each
mode benefit from the simulation.
Total, matched, and within-mode sample sizes for the
JA Finance Park Simulation Model Evaluation are ro-
Site-based and mobile simulations are both
n
bust within and between simulation models such that
held at JA Finance Park facilities designed to
comparative results are provided with a margin of er-
support a day-long simulation conducted by
ror of +/- 0.98% at 95% confidence. Results have also
JA staff and volunteers. Course content, use
been compared using Effect Size measures through
of electronic tablets, and materials used by the
Cohen’s d. See Appendix C for detailed discussion of
site-based and mobile modes are much
the use of significance testing and ES measurement.
the same.
Data Collection. PREP implemented a comprehen- n Site-based facilities are permanent locations
sive mixed mode approach that included a strong with somewhat more elaborate experiential
local area recruitment and communications system to environments than the mobile sites.
support the completion of both paper and web-based
surveys. A total of 7,487 pre- and 6,962 post-surveys Mobile sites, by definition, are capable of being
n
were completed by students across the selected five moved from place to place during the school
site-based and four mobile sites. Paper surveys were year to expand student reach. Mobile sites
used in these nine sites. In addition, 2,697 pre- and may have experiential exhibits for students to
1,347 post-surveys were completed by students from use throughout the day, but are somewhat
the five virtual simulation mode sites. The pre- and limited by the parameters of rented space.
post- survey instruments are included in Appendix A.
Simulation Model Evaluation: Capital One/JA Finance Park 11Virtual mode is a school-based computer
n Top-two box scores. Throughout the detailed find-
simulation experience that is self-administered ings comparative percentages shown for student rat-
following completion of the in-class curriculum. ings of JA Finance Park are based on the net of the
The virtual simulation is self-paced and some- two highest categories of each rating scale. By using
what different in content than the site-based the top-two box percentages, the students with the
and mobile modes. Students using the virtual strongest reactions to the statements are compared.
mode do not interact with JA staff or outside For example, on Financial Confidence attributes, a
volunteers during the simulation. five-point confidence scale of “not at all confident, not
very confident, somewhat confident, very confident,
Data Processing. Each site-based and mobile and completely confident” was used. The percentage
location received materials from PREP that included of students rating “very confident” or “completely con-
survey instructions, paper pre- and post-surveys, fident” are then combined to indicate those with the
and pre-paid return materials. Once all data collec- strongest positive response to the attribute.
tion was completed, each site returned all surveys
to PREP for data entry. Surveys were checked in, For behavioral and simulation impact attributes, a
counted, and manually entered into the student data six-point agreement scale was used: strongly dis-
base. PREP built-in standard data entry safe guards agree, disagree, somewhat disagree, somewhat
to assure the entries were complete, used correct agree, agree, and strongly agree. The top-two box
scaling, and fell within any defined numeric bounds. percentages include those who responded “agree” or
In addition, each sites’ data was checked for “strongly agree.”
entry errors, incomplete surveys, and duplication.
All open-ended responses were entered verbatim. All top-two box ratings are compared using both
significant difference testing (+/- 5% with 95%
Teachers of classes that participated in the virtual confidence) and Cohen’s d Effect Size Measures for
JA Finance Park simulation received emailed instruc- pre-to-post analysis. They suggest where compara-
tions and sign-on information for both the pre- and tive differences are strongest.
post-surveys. Web-based surveys completed by vir-
tual participants were entered by the students directly Effect Size Measure. Effect size refers to a group of
into the online portal. A separate portal was used indices used to measure the relative magnitude of a
for pre- and post- surveys. All virtual surveys were treatment effect. These measures are not dependent
cleaned for duplicates. Data from both paper and upon sample size and are based on standards of
web-based surveys was combined for analysis. measure related to a particular discipline such as ed-
ucation. There are a number of Effect Size measure-
Data Analysis. The JA Finance Park Simulation ment indices and approaches that result in similar
Model Evaluation is designed as a pre- and post- categorization. For the JA Finance Park Simulation
treatment (simulation) comparison study. Confidence, Model Evaluation, PREP used Cohen’s d to provide
behavioral, attitudinal, and classification information a standardized measure of differences between
was gathered prior to student participation through a matched pre-and-post survey samples.
pre-survey. Then, at the conclusion of the simulation
experience (treatment), a second more comprehen- Cohen’s d is calculated by subtracting the mean of
sive post-survey was conducted. Due to scheduling, the control group from the mean of the treatment
transportation, and other circumstances, not all group and dividing the result by the pooled standard
students completed both surveys. As a result, only deviation of the two groups. The result is a descrip-
students completing both a pre- and a post-survey tive measure that provides a value for comparison
are included in the control and treatment analysis. All against the Cohen’s d scale. The Cohen’s d value
pre- and post-surveys are included in other evalua- scale provides a value of ‘small,’ ‘medium,’ or ‘large’
tive analysis. A brief description of the primary analyt- difference between the pre- and post-surveys. A
ical methods follows. more detailed discussion of Cohen’s d is provided in
Appendix C.
12Report Overview
The JA Finance Park Simulation Model Evaluation
Report is organized by study objectives and populated
by research questions. The Detailed Findings sec-
tion begins with a comparison of pre-to-post student
results for site-based, mobile, and virtual simulations
on financial confidence, behaviors, and educational
aspirations. Secondly, pre-and post-simulation expe-
riential results are explored by mode, school grade,
and classroom influences. Finally, student perceptions
of the importance and overall value of the simulation
experience are compared.
Simulation Model Evaluation: Capital One/JA Finance Park 13DETAILED FINDINGS
JA Finance Park Simulation
Pre-to-Post Survey Comparison
Were there pre-to-post changes in overall student uu Financial Confidence (Confidence Scale):
perceived knowledge, behavioral intentions, and
• I can manage money well when I am out
attitudes? One of the main goals of the JA Finance on my own,
Park Simulation Model Evaluation was to understand
• I know how to handle financial risk, and
whether there were substantial differences in student
impact and benefit between modes. To gauge differ- • I can find solutions to money problems.
ences, the level of confidence and attitudinal change uu Current and (Future) Behaviors
within each of the simulation modes was measured. (Agreement Scale):
Three categories of financial aptitude were selected
• I (will) use a budget to plan my spending
for measurement: Financial Confidence, Financial
and savings,
Behaviors, and Educational Aspiration. Each of these
• I (will) set limits for spending and goals
was tested prior to participation through a pre-
for savings, and
survey and again following the simulation experience
• I (will) talk to adults about my
through a post-survey using the indicators as follow.
financial goals.
uu Educational Aspiration.
• W
hat is the highest level of education
you hope to complete?
“It is really helpful knowing
what it’s like to have a family
and what your job is so you can
manage your money wisely.”
– 8th Grade Student, Mobile Simulation
14Financial Confidence
Differences in
Learning Environments
Exhibit A
As students gain knowledge and a working under-
standing of personal financial concepts, it is assumed
that their confidence and capability will increase.
The simulation is designed to build upon the knowl-
edge gained through the JA Finance Park classroom
course work and overlay the concepts on a real-life
situation to foster experiential learning. The simulation
provides an opportunity to experience adult financial
life through a persona and a set of budgeting,
purchasing, and spending activities.
Confidence by Simulation Mode. For all modes,
there were significant positive gains in student con-
fidence. Pre-to-post simulation confidence levels for
‘I know how to handle financial risk’ and ‘I can find Exhibit B
solutions to money problems’ increased significantly
after the simulation and showed small to medium
effect sizes. Exhibits A and B show that all modes are
delivering the desired experiential impact to increase
confidence at about the same level.
Prior to the simulation, students regardless of mode
reported high levels of confidence in response to ‘I
can manage my money well when I am out on my
own.’ They made no significant increase in ratings
post-simulation (Exhibit C). Only students using the
virtual simulation mode demonstrated some positive
(albeit small) effect on this attribute.
It is important to note that the site-based student Exhibit C
sample is composed primarily of 7th, 8th, and 9th grad-
ers while the mobile and virtual samples have greater
percentages of older students. (See Exhibit C-1 in
Appendix C.) As will be shown in the grade level
comparisons, younger students exhibit some of the
highest levels of pre- and post-survey confidence on
‘I can manage my money well when I am out on my
own.’ The higher level of confidence may be based
on the longer time horizon before younger
students will in fact be ‘out on their own’ or attrib-
utable to more classroom preparation. It may also
be that they underestimate the challenge posed by
managing money in real life. Regardless of cause,
the fact remains that they are the most confident.
Simulation Model Evaluation: Capital One/JA Finance Park 15Confidence by Grade Level. Just as was true across the three simulation modes, all grade levels
demonstrate significant increases in confidence on handling financial risk and finding solutions to money
problems. Younger grades tend to show the greatest increases in confidence from the simulation experience.
Pre-to-post-survey rating changes resulted in medium effect sizes for most grades on handling financial risk
(Exhibit D). The 7th, 8th, 9th, and 12th grades exhibited the greatest positive change.
Exhibit D
Large positive changes in pre-to-post confidence ratings on ‘I can find solutions to money problems’ are also
significant for all grades (Exhibit E). In spite of large rating increases, effect sizes are small across all grade
levels.
Exhibit E
16“After I completed the JA
simulation, I left more
confident because I learned
how to manage money.
This is extremely valuable to
me because understanding
these concepts will help me
in the real world.”
– 8th Grade Student, Mobile Simulation
Exhibit F
Exhibit F shows student confidence in their ability to manage money when they are on their own was high in the
pre-survey and experienced little change in the post-survey. Despite the fact that 9th graders experienced a slight
drop in confidence (53% to 51%) in their pre-to-post ratings, the difference is not statistically significant.
Simulation Model Evaluation: Capital One/JA Finance Park 17setting limits and goals might be a first step. Though
Financial Behavior not as strong as budgeting, students of all modes
indicate a significant increase in setting spending
limits and using savings goals (Exhibit I). Effect sizes
range from small to medium.
In addition to gauging pre-to-post perceived confi-
dence levels on specific financial factors, a set of Exhibit G
behavioral attributes was also tested to gauge the
likelihood that real behavioral change might also
result from the JA Finance Park simulation. These
indicate positive change in planned financial
behaviors from students in all three modes.
Behavioral Change by Mode. Pre-to-post simulation
behavioral changes are significant and greatest for
‘I (will) use a budget to plan my spending and sav-
ings’ (Exhibit G) and ‘I (will) talk with adults about my
financial goals’ (Exhibit H).
All modes show at least a medium effect size on
budgeting practices (Exhibit G). Mobile simulation
Exhibit H
participants’ planned behavioral changes reflect a
significantly higher rate of change than the other two
modes.
Talking to adults about financial goals was one of
the lowest rated attributes in the pre-survey (Exhibit
H). Students from both mobile and virtual modes
increased their willingness to talk with adults about
financial goals significantly more than site-based
students. Both the site-based and mobile modes
use adult volunteers to lead small groups during the
simulation experience, while the virtual mode does
not. However, as the results indicate, interaction with
adults during the simulation is not a necessary con-
dition for increasing student awareness of the need
to talk with adults about financial decisions. A com-
bined impact of simulation curriculum and maturity Exhibit I
level may account for the differences among modes.
Analysis later in the report, comparing grade level
differences, will reinforce this hypothesis.
The possibility that some students might find the term
“budget” too formal to describe the financial planning
behaviors they use, led JA USA and PREP to include
“I (will) set limits for spending and goals for savings.”
It was thought that students may not need a formal
budgeting process at this point in their lives, but
18Behavioral Change by Grade Level. JA Finance Park simulation is built upon a foundation of budget building,
financial plan implementation, and evaluating success.
Student results indicate that the budgeting message was strong with all grades significantly raising their
intentions to use a budget from pre-to-post survey (Exhibit J). Planning spending and savings by using a budget
garnered the greatest behavioral change for all grade levels (Exhibit K). Effect sizes are also large across the
board except for 7th grade where the effect size is medium.
Exhibit J
‘I will set limits for spending and goals for saving’ did not realize significant increases for 7th graders while all
higher grades rose significantly (Exhibit K). Effect sizes, typically, are in the medium range for high school
grades – with the exception of 12th grade where the effect size is small.
Exhibit K
Simulation Model Evaluation: Capital One/JA Finance Park 19“It [simulation] gave me a
sense of what life will be
like once I get out on my
own. It also taught me that
budgeting could make a big
difference and affect your
overall efficiency in the
real world. ”
– 12th Grade Student, Virtual Simulation
JA Finance Park simulation greatly increases student interest in talking with adults about financial goals
regardless of age (Exhibit L).
Exhibit L
Based on future intentions, students of all grade levels reported significantly higher levels of talking with adults
about financial goals after the simulation. As demonstrated by greater effect sizes, students in 9th grade and
above reported a greater interest in talking with adults about financial goals after completing the simulation.
20Educational Aspiration
Student educational aspirations were not significant- Educational Aspiration by Mode. Student educa-
ly changed by JA Finance Park simulation. In spite tional aspirations are very high across all modes.
of the fact that students receive randomly selected Exhibit M shows nine in ten students in every mode
personas to simulate a life-like environment, the aspire to complete a college degree or more. This
exposure to greater or lesser education as a factor in overwhelming choice to attend college prior to the
determining income level does not seem to change simulation exposure negates the opportunity for the
students’ personal educational aspirations. experience to raise aspirations. The fact that student
aspirations are equal between pre-and-post surveys
eliminates the need to further explore this success
factor.
Exhibit M
JA Finance Park Post – Simulation Overall Comparisons
How do overall post-survey student evaluations compare across modes and by grade level? The JA
Finance Park Simulation Evaluation was designed to compare pre-to-post survey findings between modes and
to compare overall evaluative results by mode and grade level. Thus measures of overall experiential outcomes
are included in this section for all students that completed the post-survey. Post-survey experiential results
across the three simulation modes and by grade level are provided to:
uu Evaluate the overall Simulation Experience: uu Measure the impact of JA Finance Park simulation on:
• Interest in more experiences like the • S howing new ways to make financial
JA Finance Park simulation, decisions,
• Importance of the JA Finance Park simulation • M
aking finance easy to understand,
to your future, and • Helping better understand how to
• Usefulness of the JA Finance Park simulation. manage money, and
• M
aking the connection between real life and
the classroom.
Simulation Model Evaluation: Capital One/JA Finance Park 21Overall Simulation
Experience Evaluation
Indicator of overall satisfaction. An indicator of
overall satisfaction is whether or not a student would
personally like to have more experiences similar to
JA Finance Park. This is a particularly rigorous
standard given students completed the in-class
curriculum prior to their simulation experience.
About half of the students from each mode want to
have more experiences like it (Exhibit N). The willing-
ness to have more experiences like JA Finance Park
simulation suggests the simulation creates substan-
tial segments of students eager to continue learning.
Exhibit N
22Perceived change in financial confidence as a result of participation in the JA Finance Park simulation.
Not only were student ratings of confidence and behavioral attributes compared through pre-and-post-survey
questions, but students were also asked to gauge how much their confidence in handling financial matters had
decreased, stayed the same, or increased as a result of participating in the JA Finance Park simulation.
Exhibit O
Exhibit O shows about two in three students from all three modes reported an increase in financial confidence
due to the JA Finance Park simulation experience. The student-reported perception of change in confidence
due to JA Finance Park simulation is a strong measure of how the overall experience impacts students – likely
stronger than the individual confidence items compared in the pre-to-post survey. When students completed
the pre-survey they may have underestimated just how much they had to learn in order to have a valid sense of
confidence in their financial abilities. This reflective question in the post-survey captures that reality.
Just as high percentages of students from all simulation modes reported increased confidence on financial
matters, so did students in every grade level (Exhibit P).
Exhibit P
Simulation Model Evaluation: Capital One/JA Finance Park 23“The most valuable part of
this experience is that it is
very real. It helped me
realize that there are lots
of things I have to prepare
for in the future.”
– 11th Grade Student, Site-based Simulation
Importance of JA Finance Park to the Future. Students were asked how important they thought what they
practiced in the JA Finance Park simulation would be to their future. Exhibit Q shows that on a scale of ‘not at all
important,’ ‘somewhat important,’ ‘important,’ ‘very important,’ and ‘extremely important,’ about 90% of site-based
and mobile and 70% of virtual simulation students thought the things they learned would be at least ‘important.’
About one-third of site-based and mobile students considered the practice to be ‘extremely important.’
Exhibit Q
24In parallel with the simulation modes, student results by grade level also reflect very high importance ratings for
students of all ages. Exhibit R shows over 80% of students consider what they practiced to be important, very
important, or extremely important – regardless of grade levels. Over 90% of 10th and 11th grade students found
what they practiced to be important, very important, or extremely important.
Exhibit R
Usefulness of JA Finance Park Simulation to Life. The vast majority of students believe most or nearly all of
what they learned during the simulation will be useful in their lives. Site-based (82%) and mobile (85%) partic-
ipants estimated that most or nearly all of what they learned through the simulation will be useful in their lives.
Despite the fact that virtual participants rated the future importance significantly below both site-based and
mobile, nearly 7 in 10 still said that they would use most or nearly all of the information from the simulation in
their lives (Exhibit S).
Exhibit S
Simulation Model Evaluation: Capital One/JA Finance Park 25Regardless of grade levels, overwhelming majorities rate what they learned in the simulation as very useful.
However, 10th and 11th graders were most likely to rate as useful nearly all or most of what they learned in the
JA Finance Park simulation. Intensity was highest as well for 10th and 11th graders. Forty percent of 10th grade
and 11th grade students believe that nearly all that they learned in the JA Finance Park simulation will be useful
in their lives (Exhibit T).
Exhibit T
26“It is a fun and interactive way to learn
how to budget and spend money wisely.”
– 9th Grade Student, Site-based Simulation
Student Description of the JA Finance Park Sim- In addition to being fun, unique, and meaning-
n
ulation Experience. Students were asked to select ful, site-based students also found the simu-
from a list of items all the descriptors that they found lation significantly more challenging than the
applicable to their JA Finance Park simulation experi- other modes. Site-based students were almost
ence. The items were taken from student reactions to at parity with mobile participants in saying the
previous site-based, mobile, and virtual simulations. simulation was life-changing.
They were also given the option of adding to the list,
Virtual students were the least enthusiastic
n
but very few mentioned anything else.
about the simulation. Significantly fewer said
Students found the JA Finance Park simulation to be it was “life-changing” or “challenging” than did
fun, unique, and meaningful – but also challenging those using other modes.
(Exhibit U):
Mobile simulation students were the most
n
enthusiastic about their experience overall,
describing it as fun, unique, and meaningful.
Exhibit U
Simulation Model Evaluation: Capital One/JA Finance Park 27Exhibit V shows all grade levels described the JA Finance Park simulation most often as fun, unique, challenging,
and meaningful. Tenth grade students mentioned ‘fun’ significantly more often than the other grades. Seventh
and 8th grade students were significantly less likely, and 10th and 11th graders significantly more likely, than all
other grades to say the simulation was unique. The 12th graders were significantly less likely to say the simulation
was challenging, with 7th and 8th graders finding it significantly more challenging than the other grades.
Exhibit V
Impact of the Interval Between Classroom Work and Simulation. No significant differences were found
between students who completed classroom work less than one month prior, more than one month prior, or
who had no classroom work prior to their simulation experience – except on their description of JA Finance Park
simulation as fun (Exhibit W).
Exhibit W
Students who completed any classroom work were significantly more likely than those with no classroom
experience to say the simulation was fun, unique, and challenging. Students who had classes more than a
month prior to their simulation were significantly more likely than those with no classes to say the simulation
was meaningful.
28“I would say that, it’s very
helpful when it comes to
understanding about financial
interest. I believe that making
your own budget was very
interesting and fun.”
– 10th Grade Student, Mobile Simulation
The Most Helpful Activity of the JA Finance Park Exhibit Y shows all grade levels agree that making a
Simulation. Regardless of mode, making a budget budget and spending decisions are the most helpful
and spending decisions were the two activities most JA Finance Park activities. Budgeting was most help-
often cited as helpful. Exhibit X shows site-based ful to all grade levels, with 9th through 12th graders
and mobile students found both budget making and mentioning the activity significantly more often than
making spending choices to be most helpful. Both 7th or 8th graders. Making spending decisions was
mobile and site-based students rated making spend- rated the most helpful activity most often by 7th and
ing choices significantly higher than virtual simulation 10th grade students.
students. Virtual students reported that making a
budget was the most helpful activity at a significantly
higher level.
Exhibit X
Simulation Model Evaluation: Capital One/JA Finance Park 29Exhibit Y
The Learning that Stuck With Students Most. What sticks with most students is appreciation of parents and
learning how to make a better future. Site-based and mobile students cite the same things as sticking with
them: realizing their parents do a lot for them, seeing how to apply what they learned to make their future better,
and having a budget is worth it. Virtual students differ from the other modes by less often citing appreciation that
their parents do a lot for them (Exhibit Z).
Exhibit Z
For younger students what is significantly most memorable about the simulation is how much their parents do
(Exhibit AA). The top three most memorable aspects of JA Finance Park simulation for the 7th, 8th, and 9th
grade levels are: how much their parents do for them, knowing how to make their future better, and understand-
ing the value of budgeting.
30Exhibit AA
Older students learned parents do a lot, gained appreciation for how to make their futures better, and
discovered the value of budgeting (Exhibit AB). It appears that as students approach graduation, the ‘take
away’ from JA Finance Park shifts to practical skills such as the value of budgeting and how it will be when
they are on their own.
Exhibit AB
Simulation Model Evaluation: Capital One/JA Finance Park 31Student Assessment of JA Finance Park Simulation’s Impact on Them. The simulation aims to impact
students on four dimensions: understanding how to manage money; connecting classroom learning to real life;
teaching new ways of making financial decisions; and making finance easy to understand. Regardless of simu-
lation mode, the majority of students achieved these outcomes. Six in ten students agree or strongly agree the
JA Finance Park simulation delivers on major impact areas for them.
However, Exhibit AC shows some differences among modes in the proportion of students benefitting from their
simulation experience. Mobile participants agreed at significantly higher rates than those using the other two
modes that the simulation impacted them on each of the four dimensions. Interestingly, site-based and virtual
students assessed the simulation’s impact in a similar fashion.
Exhibit AC
Differences in the perceptions of simulation benefits varied by grade level. The 9th, 10th, 11th, and 12th grade stu-
dents, more often than younger students, agreed or strongly agreed that the simulation benefited them. These
differences between older and younger students were statistically significant. As evidenced by their significantly
higher top-two (agree or strongly agree) ratings on all impact items, 9th, 10th, and 11th graders experienced the
greatest impact from the JA Finance Park simulation (Exhibit AD).
Exhibit AD
32Most Valuable
Differences
Differences Aspect
in in Realism of
to Learning
Students
Learning Environments
Environments “Overall just everything
What was the most valuable part of the JA
in the simulation helps
Finance Park simulation from the students’
perspective? At the conclusion of the post-survey,
to create a powerful
students were asked what the most valuable part of
the JA Finance Park simulation was to them. More
connection between life
than 9,200 comments to this open-ended question
were provided from students from all modes. Overall, now and life later.”
students mentioned ‘real-life’ scenarios and budget-
ing as most valuable (Exhibit AE). – 7 th Grade Student,
Site-based Simulation
Exhibit AE
Simulation Model Evaluation: Capital One/JA Finance Park 33Exhibit AF shows site-based simulation students valued the ‘real-life’ aspect of the experience in every activity.
Exhibit AF
Mobile simulation students emphasized budgeting, then “real-life” scenarios as being most valuable – reversing
the order of the top two benefits compared with site-based simulation participants (Exhibit AG).
Exhibit AG
34Virtual simulation students emphasize the importance of the simulation to understanding the ‘real world’ of
finance (Exhibit AH).
Exhibit AH
“It is hard to manage real-life
money, but if you make a
plan and budget, then you
are more successful.”
– 7th Grade Student,
Site-based Simulation
Simulation Model Evaluation: Capital One/JA Finance Park 35ENDNOTES
1 Princeton Research and Evaluation Partners LLC. Usability Study: Capital
One/JA Finance Park Technical Report| January 2017. Written for Capital
One and Junior Achievement USA
2
See KPMG Foundation Sponsored Curriculum Evaluation: JA Finance
Park® Final Report | January 2016. https://www.juniorachievement.org/
documents/20009/133368/JA%20Finance%20Park%20Evaluation%20
Report
3
J unior Achievement USA. (2015) Corporate Identity Manual. Colorado
Springs, CO: Author, pg. 1.
4 Junior Achievement USA, (2015) Corporate Identity Manual. Colorado
Springs, CO: Author, pg. 3.
5
Junior Achievement USA, (2015) Corporate Identity Manual. Colorado
Springs, CO: Author.
6
Junior Achievement USA, (2015) Corporate Identity Manual. Colorado
Springs, CO: Author.
7 See https://www.juniorachievement.org/web/ja-usa/ja-programs?p_p_
id=56_INSTANCE_abcd&p_p_lifecycle=0&p_p_state=maximized&p_p_
mode=view&p_p_col_id=ja-maincontent&p_p_col_count=1&_56_
IN- STANCE_abcd_groupId=14516&_56_INSTANCE_abcd_arti-
cleId=19406.
36APPENDICES
Appendix A
Student Pre-Survey
Student Post-Survey
Appendix B
Implementation of Student Evaluations in Local Areas
Appendix C
Technical Appendix
Simulation Model Evaluation: Capital One/JA Finance Park 37APPENDIX A
Exhibit A-1: Student Pre-Survey
Junior Achievement
Finance Park Simulation
Pre-Survey
About You: Tell us a little about you…
S1. What is the name of your school?
th th th th th th
S2. What grade are you in? (Circle One) 7 8 9 10 11 12 Other _____
S3. What is your birthdate? Month _____ Day _____
S4. What are the first three letters of your last name? _____ _____ _____
Financial Confidence: Next, please tell us how confident you are about money management.
Please fill in the circle that shows your level of confidence for each of the following items:
Don’t
Not at All Not Very Somewhat Very Completely Know/Prefer
Confident Confident Confident Confident Confident Not to Say
I can find solutions to money
problems. ¡ ¡ ¡ ¡ ¡ ¡
I can manage my money well
when I am out on my own. ¡ ¡ ¡ ¡ ¡ ¡
I know how to handle financial
risk. ¡ ¡ ¡ ¡ ¡ ¡
Future Behavior: Before you complete the simulation, we are interested in how you manage your
money.
Please fill in the circle that shows your level of agreement for each of the following items:
Strongly Somewhat Somewhat Strongly Prefer Not
Disagree Disagree Disagree Agree Agree Agree to Say
I talk to adults about my
financial goals. ¡ ¡ ¡ ¡ ¡ ¡ ¡
I use a budget to plan my
spending and savings. ¡ ¡ ¡ ¡ ¡ ¡ ¡
I set limits for spending and
goals for saving. ¡ ¡ ¡ ¡ ¡ ¡ ¡
Demographics: For Classification Purposes Only
Before today's simulation, when did you have your last JA Finance Park class at school? (Circle one)
A. Within the last week
B. 2-3 weeks ago
C. A month ago
D. Over a month ago
E. I have not had any classes at school
F. Don’t Know/Prefer Not to Say
What is the highest level of education you hope to complete? (Circle one)
A. Some High School
B. High School
C. Trade School or Vocational School
D. A College Degree or Higher
E. Don’t Know/Prefer Not to Say
THANK YOU for your answers! We will check in with you again at the end of the simulation.
-1-
38You can also read
