Measuring Therapeutic Riding Program Impact - Cathy Smith Hybels, Ph.D. PATH Intl Region One Conference April 2015 - PATH Intl.
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Measuring Therapeutic
Riding Program Impact
A Multicenter Approach and Feasibility Study
1
Cathy Smith Hybels, Ph.D.
PATH Intl Region One Conference April 2015
Topics:
TR Program Impact
Measuring Impact
Collaboration
Feasibility Study Results
Expert panel
2
We want to
TR Program Impact
in TR
3
Hybels
2015
From success defined as “how much
we do”…
TR Program Impact
# Participants
# New
Participants # Volunteers
# Volunteer Hours
# Lessons
$ value of volunteer time
Types of TR programs
4
Hybels
2015
…to also include “how well we perform”
and to answer “is anyone better off?”
TR Program Impact
Impact on the
How well we Community
perform Results-Based
How much Accountability
we do Is Anyone Evidence-Based
Better Practices
Off? Research on TR
Effectiveness
5
Hybels
2015
We want to advance accountability in
TR at each of these levels
TR Program Impact
Participant TR program Instructor
TR Center and the TR Industry
6
Hybels
2015
Changing direction requires
Measuring Impact
7
Hybels
2015
Tools for measuring impact
Measuring Impact
8
Hybels
2015
Session Report
Measuring Impact
Product of a Collaborative Effort
9
Hybels
2015
Session Report
Measuring Impact
Identifies TR Skill Areas
Horsemanship Skills Supporting Skills
Mount Expressive Communication
Dismount Attending to Task
Back up Balance
At halt--posture
At halt--reins
Coordination
At halt--halt-walk transitions Posture
At walk--posture Follows direction
At walk—hold reins Social interaction
At walk—steer with reins Safety Awareness
At walk--walk-halt transitions Regulates Behavior
At walk--half seat
At trot/jog--posture
At trot/jog--seated trot
At trot/jog--half seat
At trot/jog--posting
At canter/lope
Jump
Tack
Untack
10
Groom
Lead Hybels
Horse Safety 2015Session Report
Measuring Impact
Defines TR Skill Areas
Horsemanship Skills
Uses Natural Aids: Uses hands, legs, seat, voice, ears, and eyes
to effectively communicate with the horse
• Example: Uses voice to “whoa” or “walk on.”
Horse Safety: Recognizes potentially unsafe or hazardous
circumstances both mounted and unmounted and reacts
accordingly.
• Example: When riding in a group, recognizes when
approaching too close to another horse and directs horse to
an open space in the arena.
11
Hybels
2015Session Report
Measuring Impact
Tracks P Progress in Skill Areas
Horsemanship Skill Area Section
Level of Support Required Leader Involvement
1. Maximum physical assistance—hands on 80-100% of the time 1. Leader is totally in charge of the horse—rider does not have reins
2. Moderate physical assistance—hands on 50% of the time 2. Leader is totally in charge of the horse—rider is beginning to use
reins
3. Minimal physical assistance—hands on less than 25% of the 3. Leader or rider may control horse
time
4. Instructor supervision with verbal/visual cues 80-100% of the 4. Rider is in control of the horse—leader remains on-lead
time 5. Rider is in control of the horse—leader is off-lead at horse’s head
5. Instructor supervision with verbal/visual cues 50% of the time 6. Rider is in control of the horse at all times—no leader needed
6. Instructor supervision with verbal/visual cues less than 25% of
the time
7. Instructor supervision only—rider is independent
8. Instructor observation only—independent & proficient
Mounted Skills Baseline: Jan/Feb Mar/Apr May/June July/Aug Sept/Oct Nov/Dec
Uses Natural Aids
Halt
Back Up
Walk – overall
Walk – half seat
Trot/Jog –Sit
Trot/Jog – half seat
Trot/Jog – post 12
Canter/Lope
Jump
Hybels
Horse Safety
2015Session Report
Measuring Impact
Tracks P Session Goals
Session Goals & Progress Notes Section
Session Goals (March/April):
1.
2.
3.
Progress notes on goals for this session (Indicate progress and to what
degree the goal was met)
Recommended Future Goals:
1.
2.
3.
13
Please note anything that could be helpful to know about this rider:
Hybels
2015Session Report
Measuring Impact
As a Tool for Measuring Impact
Defines TR skill areas TR instructor use only
Tracks P skill progress over a year Instructor rates the progress
in several TR skill areas of their students in broad skill
Tracks P progress on individual areas—vulnerable to bias
goals Does not allow aggregation of
Designed by instructors for data to determine program
instructors effectiveness
Field tested Limited value outside of TR
because there is no research 14
Facilitates communication
to establish its credibility as
between TR instructors Hybels
an assessment tool 2015Goal Attainment Scaling (GAS) is a widely
Measuring Impact
implemented and systematic goal setting &
evaluation process that measures the extent to
which a participant’s goal(s) is achieved
Assess P
Baseline Define
Assess goal
Performance Identify Goal Expected
attainment
in a TR Skill Outcomes
Area
15
TR Instructor TR Instructor & others TR Instructor Independent Rater
Hybels
2015GAS Process
Measuring Impact
• At the beginning of each TR session for each
participant
Set Goals
• Providing opportunities to practice skills
Deliver
Instruction
• Independent rater attends lesson at the end of
the TR session to observe and assess performance
Measure
Performance in goal areas
16
Hybels
2015GAS Encourages:
Measuring Impact
Collaboration in Goal Setting
Participant
17
Hybels
2015GAS Encourages:
Measuring Impact
Support Team Collaboration
18
Hybels
2015GAS Requires:
SMART Goals
Measuring Impact
19
Hybels
2015Goal Scale Development
Start with Defining Baseline
Measuring Impact
Performance
Anticipated goal
attainment at Performance Level Descriptions
end of session
Baseline/Much
Less Than Participant performs a sitting trot down the long side
Expected
Less than
expected
Expected
Better than
expected
Much better
than expected
20
Baseline performance as observed by the TR
Hybels
instructor at the beginning of the session 2015Goal Scale Development
Measuring Impact
Define the Expected Performance
Goal
Performance Level Descriptions
Attainment
Baseline/MLTE Participant performs a sitting trot down the long side
Less than
expected
Expected Participant performs a posting trot down the long side
Better than
expected
Much better
than expected
Expected performance at the end of the TR session based 21
upon the TR instructor’s understanding of the p, and Hybels
drawing upon the instructor’s training and past experience 2015Goal Scale Development
Measuring Impact
Define the other possible levels
Goal
Performance Level Descriptions
Attainment
Baseline/ Participant performs sitting trot down the long side
MLTE
Less than Participant performs posting trot for less than half the long side
expected
Expected Participant performs posting trot down the long side
Better than Participant performs posting trot for half the entire ring
expected
Much Participant performs posting trot for the entire ring
better than
expected
Other possible outcomes then are established: less than 22
expected, better than expected level, and much better Hybels
than expected 2015Goal Scales In Use
This is what the Rater Sees
Please check the box next to the description below that best describes the
participant’s performance that you observed today:
Participant performs sitting trot down the long side
Participant performs posting trot for less than half the long side
Participant performs posting trot down the long side
Cathy Hybels 2015
Participant performs posting trot for half the entire ring
Participant performs posting trot for the entire ring
Please describe any circumstances present that you feel may have affected the
participant’s performance: 23
Hybels
2015Goal Attainment Scales
How Scores Are Assigned
Measuring Impact
Goal
Score Performance Level Descriptions
Attainment
-2 BL/Much Participant demonstrates sitting trot down the long
less than side
expected
-1 Less than Participant demonstrates posting trot for less than half
expected the long side
0 Expected Participant demonstrates posting trot down the long
side
1 Better than Participant demonstrates posting trot for half the
expected entire ring
2 Much better Participant demonstrates posting trot for the entire
than ring
expected
24
Hybels
How to interpret the rating The independent rater selects the level that best 2015
matches the performance they observedTR Goal Attainment Scale
Example #2
Measuring Impact
Goal
Rating Attainment Level Description
BL/MLTE
-2 Participant does not hold reins
Less than
Participant picks up the reins after a verbal cue, then
-1 expected
immediately puts them down
Expected Participant picks up the reins after a verbal cue and
0
holds them for 1 minute, then puts them down
Better than
Participant picks up the reins after a verbal cue and
1 expected
holds them for 2 minutes, then puts them down
Much better
than Participant picks up the reins after a verbal cue and 25
2
expected holds them for 3 minutes, then puts them down Hybels
2015GAS Allows Aggregation of
Measuring Impact
Scores to Help Answer “How
30 Well Did We Do?”
25
GAS Score Frequency
20
15
10
5
0
Baseline/Much Worse than Expected Better than Much better
worse than expected progress expected than expected
expected progress progress progress 26
progress
GAS Score Hybels
2015GAS Results Help to Answer
Measuring Impact
“Is Anyone Better Off?”
60
No. of Participants with a goal(s)
50
40
30 82%
20
10
14%
0 4%
Demonstrated No Progress No GAS score
Progress on goal(s) Demonstrated on obtained 27
goal(s) Hybels
2015GAS Results Communicate the Skill
Area Focus of TR Lessons
Measuring Impact
90
80
70
60
Number of goals
50
40
30
20
10
0
Mount Dismount Uses Natural Halt Back Up Walk--overall Walk--half Steering
28
Aids seat
Hybels
TR Skills As Defined on the Session Report 2015GAS Results Reveal the Skill Area
Focus of Different Instructors
Measuring Impact
25 25
Instructor A-Chart 1 Instructor B-Chart 1
20 20
15 15
10 10
5 5
0 0
29
Hybels
2015GAS Results Reveal Factors That May Have
Potentially Limited Goal Attainment
Measuring Impact
Support team issues
Lesson horse issues
P well-being
Other issues
30
Hybels
2015GAS as a Tool for Measuring Impact
Measuring Impact
Client-specific measures
Requires time , effort,
Sensitive measures training, and experience to
Low cost produce good GAS scales
Allows aggregation of scores Requires a sufficient number
Established credibility & validity of trained raters & scheduling
Enhances team collaboration can be an issue
Flexible--accommodates multiple Rater presence can
goals, weighted goals, group potentially be intrusive in
goals lessons
31
Implementation can be scaled Hybels
2015
Intuitive processSession Reports & GAS:
Measuring Impact
Complimentary and Synchronized Tools
Session
TRAIN Tool Key Attributes GAS
Report
Definitions of common TR terms
Systematically tracks instructors’ notes on p progress
Describe what skills are being taught in TR
Intended for sharing with p’s, families, care
providers, and volunteers
Provides independent assessments of p goal
attainment
Produces numerical performance measures in TR 32
skill areas Hybels
2015Multicenter Collaboration
Why Collaborate?
• To reveal patterns
that only emerge
when there enough
data points
• To help each other
learn & develop a
new way of thinking
• To promote
discussions of
accountability
33
Hybels
2015Multicenter Collaboration
Therapeutic Riding Assessment of Impact Network = 34
TRAIN
Hybels
20152014 TRAIN Member Centers
Multicenter Collaboration
35
Hybels
20152014 TRAIN Member Center Attributes
Multicenter Collaboration
36
All are premier
accredited centers
Hybels
2015Feasibility Study
2014 TRAIN Feasibility Study Purpose 37
1. Assess the likelihood of success of a collaborative approach
2. Identify issues that need to be addressed before employing the methods on a Hybels
2015
larger scale2014 Use of GAS at 4 TR Centers
Feasibility Study
10 instructors 5 Instructors
2 TR Sessions 4 TR Sessions
32 Goals 70 Goals
30 Raters 19 Raters
TRAIN
2 Instructors 2 Instructors
4 TR Sessions 3 TR Sessions
170 Goals 25 Goals
44 Raters 13 Raters
38
Hybels
201513
TR
Feasibility Study
sessions
Dropbox
19 TR
instructors
318
goal
attainment
scores
2 TRAIN 4 TR
Meetings
Centers
106 The horses
raters empowered…
39
Hybels
2015Various Strategies For Using GAS Were Utilized at
Feasibility Study
the 4 TR Centers and By Different Instructors
Approach Strategy Variant 1 Strategy Variant 2
Participant (p) inclusion Every p had a GAS goal Selected p’s had GAS goals
Number of GAS goals/p 2 goals per p 1 goal per p
GAS Goal variance Unique goal for each p Similar goals for all p’s
GAS Goal originality Original Recycled/revised
Instructor’s use of GAS One session Every session
GAS Goal periods 6 weeks 12 weeks
Expected p performance Rated on typical Rated on best performance
performance
40
Hybels
2015Different TRC Approaches to Implementing GAS
Feasibility Study
Wide range,
yet all Less experimentation
involved with goal writing
>50% of staff
TRAIN Member Center # Instructors Ave # Goals/Instructor # Raters Ave # Goals/Rater
High Hopes 10 3.2 30 1.06
High Horses 5 14 17 3.68
Southern Vermont 2 12.5 13 1.92
UpReach 2 85 44 3.86
Greater Experience as
experimentation Wider raters
with goal writing diffusion
41
Hybels
2015Number of 2014 GAS Ratings Attained per Instructor
Feasibility Study
TR Instructors
0 10 20 30 40 50 60 70 80 90 100 110 42
Number of Rated GAS Goals
Hybels
2015GAS Goal Horsemanship Skill Areas
(part 1)
Feasibility Study
90
80
70
60
Number of goals
50
40
30
20
10
0
Mount Dismount Uses Natural Halt Back Up Walk--overall Walk--half Steering
43
Aids seat
Hybels
TR Skills As Defined on the Session Report 2015GAS Goal Horsemanship Skill Areas
(part 2)
Feasibility Study
90
80
70
60
Number of goals
50
40
30
20
10
0
44
Hybels
TR Skills As Defined on the Session Report 2015Observed Progress In Goal Areas*
*Caveat: data obtained from a proof of concept study
Feasibility Study
Raters observed
improved performance
90
on 245 of 318 goals (77%)
80
Number of Goals
70
Baseline/No Progress
60
Expected
50
40
30
20
10
0
-2 -1 0 1 2
More 45
Scores assigned by independent raters
Hybels
201519%
Feasibility Study
On
of the goals,
raters reported that participants’
performances may have been
negatively affected by
extenuating circumstances 46
Hybels
2015Factors Perceived by Raters as Having
Potentially Limited Goal Attainment
Feasibility Study
Support team issues
(13 goals/4%)
Lesson horse behaviors
(27 goals/8%)
P well-being
(10 goals/3%)
Other issues
(27 goals/8%)
47
Hybels
2015On 2%
Feasibility Study
of GAS goals,
raters felt that extenuating
circumstances
may have had a positive 48
effect on goal attainment.
Hybels
2015Comparing Goal Area Focus at 2 TRCs
Feasibility Study
40 40
35 35
30 30
25 25
20 20
15 15
10 10
5 5
0 0
49
Hybels
2015Comparing the Goal Area Focus of 2 Instructors
Feasibility Study
Instructor A Instructor B
25 25
20 20
15 15
10 10
5 5
0 0
50
Hybels
2015Goal Attainment Score Distributions
Across TR Centers
Feasibility Study
Center 1 Center 2
-2 -1 0 1 2 More -2 -1 0 1 2 More
Center 3 Center 4
51
Hybels
2015
-2 -1 0 1 2 More -2 -1 0 1 2 MoreGAS Score Distributions for 2 Instructors
Feasibility Study
Instructor A Instructor B
52
-2 -1 0 1 More
2 -2 -1 0 1 2
More Hybels
2015TRAIN Member Centers Reported
Who Else Is Better Off?
Conversations Changed with…
Feasibility Study
• Families & Care Providers
• TR Instructors
• TR Volunteers
• TR Boards of Directors
53
Hybels
20152015 TRAIN Focus
• Introduce GAS-Lite
TRAIN 2015
• Gather qualitative data on how skills acquired in
TR programs transfer into other activities
• Continue to develop collaborative efforts
54
Hybels
2015GAS-Lite
Participant: # of GAS goals: Goals this session #of weeks in session:
Accompanied by: Type of lesson: Private Group Session start date: Click here to enter a date.
Instructor: Lesson day: time: Goal assessment date: Click here to enter a date.
Horse: TR Center: Choose an item. Rater:
Does the participant (or family or care provider) have a session goal? Yes No Don’t know
If yes, what is their goal?
TRAIN 2015
Baseline Performance Expected Outcome Goal Attainment Variance
Instructor: Please clarify whether the rater Rater: please check one Rater: if goal attainment differs from the
should assess best or typical performance box below expected outcome, describe how it differs below
Less than baseline
Same as baseline
Partially achieved
As expected
A little better
Much Better
Rater: Please describe any circumstances present that you feel may have affected the participant’s performance:
Horsemanship Skills (Please check the appropriate box(es) below to indicate the most relevant goal skill area(s))
Mount Backup Trot/jog-sit Jump Groom
Dismount Walk-overall Trot/jog-half seat Horse Safety Lead
Uses Natural Aides Walk-half-seat Trot/jog-post Tack Other:
Halt Walk-post Canter/lope Un-Tack Other:
Supporting Skills (Please check the appropriate box(es) below to indicate the most relevant goal skill area(s))
Attends to task Correct posture Confidence Social interaction Manages sensory input
Balance Expressive communication Joy Adaptability Regulates behavior
Coordination Problem solving Safety awareness Follows direction Other:
How do you anticipate this skill might carry over or transfer to the participant’s activities of daily living?
Support Team Instructions
Helmet: Note: All lessons may include a trail ride
Definitions:
Precautions: Special Mount/Dismount Instructions:
55
Special side walker verbal or physical prompts &/or assists: Special leader assists:
Hybels
TRAIN 2015 GAS Lite v1.2 2015TRAIN Expert Panel
• Lorna Young, Southern Vermont Therapeutic Riding
Expert Panel
Center
• Sue Miller, High Horses Therapeutic Riding Program
• Betsy Medinger, High Horses Therapeutic Riding
Program
• Kristen McGraw, UpReach Therapeutic Equestrian
Center
56
Hybels
2015Thank you!
For more information, contact:
Cathy Smith Hybels, Ph.D.
Therapeutic Riding Assessment of Impact Network (TRAIN)
chybels@sover.net
Previous project reports and TRAIN forms can be found on the High Horses
Therapeutic Riding Program website (www.highhorses.org) or by following the
links below:
Measuring Outcomes at High Horses: Pilot Study Results
Measuring Outcomes at High Horses: 2013 Feasibility Study Results
Getting WISE at High Horses and Measuring the Outcomes: 2013 Pilot Study
Results
Building Capacity for Evaluating Outcomes in Therapeutic Riding: A
Collaborative Approach
Building Capacity for Evaluating Outcomes in Therapeutic Riding Using GAS
and a Collaborative Approach: A Summary of the Lessons Learned
57
TRAIN GAS Rater Form v2.0
Hybels
TRAIN GAS Lite Form v1.2 2015You can also read
