Evaluation Proposal: Seminole Football Athletic Training Program - Cameron Fielding - C.Fielding
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Evaluation Proposal: Seminole Football Athletic Training Program Cameron Fielding 2019
Table of Contents
i. General Introduction ................................................................................................................................................... 2
ii. Executive Summary .................................................................................................................................................. 2
iii. Detailed Overview: Training Program and its Purpose ............................................................................................ 4
1. Systemic Analysis Approach ..................................................................................................................................... 5
2. Systemic Analysis Method and Tool ......................................................................................................................... 6
3. Training Program Components.................................................................................................................................. 6
Function 1. Improvement of Trainees’ Knowledge, Skills, and Attitudes (KSA) ........................................... 6
Training Inputs .................................................................................................................................. 7
Needs for Training .............................................................................................................. 7
Learners Characteristics ..................................................................................................... 7
Instruction Specific Context .............................................................................................. 8
Info on Performance Context ............................................................................................. 8
Training Inputs.................................................................................................................... 8
Instruction & Training Context ........................................................................................................ 9
Modality.............................................................................................................................. 9
Content................................................................................................................................ 9
Development Process.......................................................................................................... 9
Implementation of Training ............................................................................................................ 10
Training Outputs ............................................................................................................................. 10
Monitoring ...................................................................................................................................... 10
Function 2. Improvement of Trainees’ Job Performance .............................................................................. 11
Workplace Transfer Readiness ....................................................................................................... 11
Function 3. Improvement of Organization’s Performance ............................................................................ 12
Organization’s Maintenance of Training Transfer .......................................................................... 12
Discussion of Organization’s Overall Commitment and Support for Program ............................................. 12
Organization’s Mission and Commitment to Innovation ................................................................ 13
Allocated Resources for Training and Support of the Program ...................................................... 13
Capabilities for Transfer and Maintenance ..................................................................................... 13
Incentives for Employee Participation ............................................................................................ 14
4. Evaluation Logic Model .......................................................................................................................................... 14
5. Evaluation Methodology Overview ......................................................................................................................... 15
6. Evaluation Management Plan .................................................................................................................................. 16
Justification of Evaluation Methods................................................................................................ 16
Process Evaluation ............................................................................................................ 17
Outcome Evaluation ......................................................................................................... 17
7. Evaluation Procedures, Instruments and Measures.................................................................................................. 17
Process Evaluation Methods ...............................................................................................................17
Needs for Training and How the Training Process Relates to Those ..........................................18
Training Instructional Context ...............................................................................................18
Quality and Relevance of Training Materials and Modality .........................................18
Trainees’ Readiness .................................................................................................19
Experience and Expertise in Trainers.........................................................................20
Performance Context ...............................................................................................20
Support for Transfer of Training and Transfer Readiness ............................................20
Training Delivery and Implementation Process ..........................................................21
Outcome Evaluation Methods .............................................................................................................22
Improvement in Trainees’ KSA .............................................................................................22
Improvements in Trainees’ Job Performance ...........................................................................23
Improvements in Organizational Performance Due to the Training ...........................................23
8. Evaluation Communication and Reporting plan ...................................................................................................... 24
9. Conclusion ............................................................................................................................................................... 25
References ................................................................................................................................................................... 27
Appendix ..................................................................................................................................................................... 29
1
i. General Introduction:
The following is an analysis of the Florida State Seminoles ‘football education’ training
program conducted per the requirements of EME6357: Evaluation of Training Programs in HPT.
It seeks to outline the major features of the program, to place these features within the context of
a systemic model, and to offer a systematic analysis of the program within this context. On the
basis of the systemic analysis, we next offer summary of our proposed evaluation procedures
following another systematic model: The Evaluation Management Plan. Here, we outline the
high-level evaluation questions that will inform evaluation procedures, explore the specific
instruments and procedures that will constitute the evaluation, and offer justification for the
evaluation methodology
The paper will begin with an Executive Summary that presents a broad overview of the
program and essential takeaways. Following this is a detailed introducing the Seminole training
program along with foundational models for systemic analysis. We next deconstruct the program
in terms of the presented systemic models and offer a logic model for approaching evaluation.
Using this model as a basis, we proceed to outline our Evaluation Management Plan and
Communication Plan. In combination, these elements will provide the reader with an
understanding of the mechanics that compose the athletic training program while also offering a
clear path toward successful evaluation. We begin with an Executive Summary.
ii. Executive Summary:
• Program Title: Florida State Seminoles football athletic training program
• Program Sponsor: Florida State University
• Program’s Purpose: The purpose of the program is to generate revenue and prestige for
FSU by enabling the football team to win football matches. It attempts to improve the
athletic performance of varsity players using both internal metrics (measured against the
player’s past performance) and external metrics (using data from games played against
other teams).
• Duration of Training: The training lasts approximately 4 years, or the length of time
that the athlete is enrolled at FSU and performing to athletic / academic standards.
2
• How often it is offered: The training program is offered on an ongoing basis, and
changes training components / focus based on the athlete’s needs or a predetermined
schedule (requiring at various times conditioning, strength training, rehabilitation, etc.).
• Training Instructional Context: Training for the football program takes place at FSU,
and all trainees are also students at the university. Consequently, the context for athletic
training is somewhat complicated by the academic requirements of university and the
athletic program functions within the larger context of the university.
o Trainers and training staff and their qualifications: Primary trainers include
the head coach for the team who makes executive decisions for the program. The
chain of command passes down in a hierarchical structure to Assistant Head
Coaches, associate coaches, etc. Also on staff are specialized trainers for strength
training, nutrition, and rehabilitation.
o Trainees general profile: All trainees are undergraduates at FSU and in the
varsity football program. Most players receive training starting their freshman
year of college, however some are accepted into the program as sophomores. The
approximate age of all trainees is between 18-22.
o Training instructional content: The training includes elements such as strength
building, conditioning, rehabilitation, nutrition, strategy building, and team
coordination drills.
o Training delivery mode: The training is generally delivered by coaches who
have various specializations and focus on his / her area of expertise. FSU has
extensive and specialized facilities in which these trainings are conducted.
o Training process (how it is implemented): Coaches keep statistics for each
trainee using devices such as body composition monitoring in order to track
progress throughout the program and identify player weaknesses. Trainings are
generally conducted in groups, however individualized programming is used to
the extent that players have different functions within the team and must
specialize.
o Trainees’ Assessment procedures and Tools: The trainers use various
technologies and experts to effect trainee development. Such tools include GPS
monitoring devices to record the athlete’s movement through space, in addition to
3
devices that monitor body composition over time (BodPod). Likewise, trainees
are assessed by certified strength and conditioning coaches at progress intervals.
• Trainees’ Performance context: The trainees’ performance context is both in practice
sessions and in actual gameplay during football matches. Trainees are assessed, as
mentioned, both on the basis of their individual performance and their performance as a
group when playing football matches.
o Where / how they use the knowledge and skills gained as a result of this
training: Ultimately, players use the skills primarily to play and win football
games against other university teams. Their success generates revenue for the
university and this money is partly fed back into the training program.
iii. Detailed Overview: Training Program and its Purpose:
In elaboration of the information presented in our Executive Summary, the purpose of the
Florida State Seminoles’ football training program is to enable the football team to win football
matches against other teams. It attempts to improve the athletic and psychological performance
of players using both internal metrics (measured against the player’s past performance) and
external metrics (using data from games played against other teams). We have deconstructed the
program into its constituent parts, and also examined the methods by which the program tracks
and utilizes the aforementioned metrics.
In order to accomplish these goals, we have adopted a Systemic Analysis Approach to the
program. A system, for our purposes, is defined as “a network of interdependent components that
work together to try to accomplish the aim of the system” (Deming, 1994) We adopt this
systemic approach in order both to construct a robust understanding of the program as a system
and to deconstruct the functions of the system in a systematic manner. We will here introduce
this process by contextualizing the training program within a larger system in the model below.
4
1. Systemic Analysis Approach
The above figure illustrates our template for creating a broad-scale understanding of the
problem space. Following the outline, we see that the athletic training program functions as a
subsystem of a larger performance system. In this case, the Organizational Environment is that of
the FSU Athletics department. Increasing the level of granularity, we see a host of Structures
operating within this broader environment: Seminole Football, Strength & Conditioning, Sports
Medicine, Sports Nutrition, Accountancy, Equipment, and IT all function as semi-autonomous
but interrelated departments / subsystems. Structures gain identity and interaction through
various Processes that define their flow of goods, information and services. Workforces
(coaches, medics, IT workers, etc.) execute these Processes, and deliver the Training Service
with which we are concerned. Finally, Feedback from the Training Service creates feedback
loops for the Workforce to use in improving processes.
Understanding the training program as a subsystem within the broader system outlined
above allows us to deconstruct its processes in the context of the larger environment in which it
functions and to which it is beholden. With a robust understanding of context at our backs, we
turn now to the constituent parts of the training program, and to a systemic method for
understanding these parts in terms of our ‘network of interdependent components’. Below we
introduce a Systemic Analysis Method that illustrates our approach to the program’s moving
parts and primary functions.
52. Systemic Analysis Method and Tool
In order to deconstruct the training program, we have employed an iterative analysis.
This technique views the training program in terms of its primary functions and the interactive
components that are used to achieve these functions. It is diagrammed below.
We have used the model illustrated above to inform the breakdown of Training Program
Components that composes the next section. We address each of the items in order and as they
relate to the three primary functions of training: 1) improving individual performance, 2)
improving job performance, and 3) improving organizational performance (Darabi, 2017).
3. Training Program Components:
Having filtered the athletic training program through the iterative analysis model, we now
proceed to highlight our core findings accordingly. Referring to the model, we will begin with
the first function of training and follow the progression outward.
• Function 1. Improvement of trainees’ knowledge, skills, and attitudes (KSA)
At the KSA level, the Football Education program seeks to improve the physical and
mental abilities of trainees in order to improve their football performance. The program
emphasizes several areas of instruction in order to address the full range of competencies
demanded of trainees. We shall, for the purpose of this analysis, break these areas of
6emphasis into three general categories: physical training, mental training, and nutritional
training.
A broad range of physical training, with a primary emphasis on programming for
‘Strength & Conditioning’, develops trainees’ ability to execute football-specific movements
(Purinton, 2006). This program is supplemented with mental training that develops both the
attitudes and the mental skills of trainees. Attitudes are refined through the integration of
ongoing psychological reviews through a mental-health program (Pastor, 2018), and mental
skills are developed through performance reviews (“FSU all-access,” 2010). Finally,
nutritional training ensures that the athlete is capable of maintaining an optimized diet.
Trainees receive ongoing nutritional instruction, individual counseling and personalized meal
plans in order to optimize their other programming (“Seminole Nutrition,” 2019).
Following the progression of our iterative analysis, we proceed now to break down the
functional elements of the training program as they relate to Function 1:
o Training Inputs:
- Needs for training: In 1947, in response to an overwhelming number of WWII
veterans returning from combat and seeking higher education, an Act of
Legislature re-named and reconfigured the then Florida State College for Women
into the co-educational Florida State University (“FSU History,” 2019). The
football program was immediately re-established as a byproduct of this act, and
thus the need for training new athletes emerged.
- Learners’ Characteristics: Learners consist of young adults enrolled in FSU as
undergraduates. They must meet or exceed baseline athletic and academic
performance standards based on the position for which they are being drafted or
applying. Entry into the team may be secured either through high-school
recruiters whereby the player is scouted and offered placement based on his
varsity performance, or through a tryout process once the player is accepted to
FSU (Sonnone, 2019). New players generally enter the training program in their
freshman year of university, however some non-freshman players stand as
exceptions to this generality and are known as ‘Redshirts’ when they join the
team at another grade (Nee, 2018). Within the program, the trainees are broadly
7subdivided into 3 groups based on experience: Developmental, Travel Squad,
Special Teams (Purinton, 2006).
- Inst. Specific Context: Instruction occurs within the training facilities offered by
FSU, usually in designated football training facilities but with some overlap into
general athletic facilities where nutrition and sports medicine are concerned. All
of the trainees are full-time university students and so the training occurs around
the trainees’ academic commitment (Sonnone, 2019). The training program must
also adhere to regulations set by the NCAA for the specific dates and duration in
which training may take place (Purinton, 2006).
- Info. on Performance Context: The end performance context from the
standpoint of the training program lies in playing competitive football matches
against the teams of other universities. The overall success of the program may be
judged by the win-loss statistics year over year (SRCFB, n.d.). Generalizing
beyond these statistics, we may view performance with a focus on producing both
winning teams and noteworthy players. Winning teams win games, and
noteworthy players win fame. With either focus, success during a competitive
game serves as justification for the program.
- Training inputs: We may generalize key program inputs as they relate to the
previously-identified categories of physical training, mental training, and
nutritional training. Physical training utilizes an array of physical and
technological inputs to effect performance. These include weight lifting
equipment such as barbells, body composition monitoring (BodPod) and
statistical tracking (Catapult) devices, and practice-specific clothing. The mental
training component utilizes such tools as paper notetaking and projectors for
presentations and game reviews. The nutritional component involves both
materials for abstract presentations such as projectors and materials for
consumption such as fruit and Muscle Milk for protein shakes (Rubin, 2016).
Each of the aforementioned training components employs content-specific
monitoring software to measure and schedule movement over time, along with a
specialized staff to manage affairs and the trainees to be monitored.
8o Instruction & Training Context: Training occurs within the context of FSU’s broader
function as an academic institution. The relationship between the athletic education and
general education requirements complicates the procedures of the former because it is
subservient to the latter. In order to maintain eligibility for the football program trainees
must be enrolled in 15 credit hours and maintain a minimum GPA in their academic
coursework (Sonnone, 2019). The program occurs likewise within the broader context of
nationally-accredited collegiate sports, which means that the program must also conform
to regulations set by the NCAA.
- Modality: The majority of training occurs as practice. Players are assigned to
coaches and these coaches oversee the players as they perform. Both Strength &
Conditioning and Sports Nutrition employ this primary approach with an
emphasis on practical, real-time coaching. Classroom learning also factors into
both aspects of the training program with scheduled nutritional lectures and game
reviews. Sports Psychology operates using similar modalities, but also offers a
formal therapeutic mode which the other two lack.
- Content: The Strength & Conditioning program begins with a period of
hypertrophy and strength training until the Spring playing season begins. During
this period, trainees are directed toward increasing lean mass. This programming
is scaled back during play season, during which a maintenance program is
implemented in order to maintain the trainees’ prior gains in strength and
conditioning. Full training is resumed during a nine-week intensive Summer
training (Purinton, 2006). Interspersed throughout the core physical training
program are group classes and individual counseling from staff nutritionists
(“Seminole Nutrition,” 2019) who rely on the same metrics as the S&C staff to
formulate their own individualized programming (Rubin, 2016). Likewise, sports
psychology programming is offered on an ongoing basis with a designated sports
psychologist (Pastor, 2018). Additional mental skills and attitudes training is
delivered in coordination with outside company The Program which specializes in
building leadership skills in conjunction with physical education (Schoffel, 2018).
- Development process: Program development is conducted through a series of
increasingly-specific categorizations. Players are first generalized on the basis of
9experience into developmental groups and travel squads. Programming is then
narrowed on the basis of position within the team. It is finally narrowed further
using metrics derived from individual players. An outline of this process is as
follows. Strength & Conditioning begins with a post-bowl injury assessment for
returning players and physical assessment for new players. This is done in
coordination with the sports medicine department in order to establish training
baselines. Using the metrics gained from these assessments, coaches customize
routines for individual players based on their team function and experience.
Ongoing metrics are captured using the Catapult monitoring system, which takes
live readings of such factors as age, weight, and heartrate and delivers these to
coaches as ‘Player Load’. Coaches use these live metrics to modify ongoing and
day-to-day training. High-level revision of strategy is conducted by senior
coaches and advisors based on reviews of in-game performances, and this strategy
revision affects the general orientation of team training (ESPNU, 2010).
o Implementation of training: The training as described above is implemented by several
teams of coaches over the course of a year. Using a combination of industry-wide best
practices and data collected on individual trainees, and adhering to the restrictions set by
both NCAA and FSU regulations, trainers outline and implement a schedule in which
training may occur. Having ensured the optimal number of staff and ratio of trainers to
trainees is available to carry out the training schedule, these are organized into hierarches
under the general direction of the Head Coach. Decision making authority for
implementing and altering training is based on rank within this hierarchy.
o Training outputs: The training is expected to produce approximately 100 football
players per year (FOX Sports, n.d.). The program is further expected to produce an
appropriate number of practice sessions, drills, presentations and medical / nutritional /
therapeutic services to deliver training to the trainees. In more abstract terms, the training
program is expected to produce a positive statistical trend on the basis of games won
during the season.
o Monitoring: Initial trainee physicals are conducted using medical equipment through
FSU’s Sports Medicine department. Particularly important in these physicals are the
biodex isokinetic testing machine and BodPod which are used to test and track body
10composition and mobility over time (“FSU Sports Medicine,” 2018). The most
significant single technology used in the program is the Catapult system which tracks
over 100 trainee metrics in real time to provide coaches with ongoing performance data
and GPS monitoring. This data is used to personalize and alter ongoing training as
needed in order to maximize optimize trainee effort, reduce soft-tissue injuries, and refine
spatial movements (Davis, 2018). Retroactive review is also employed by coaches
watching game and practice footage in order to make player assessments, and this review
supplements other modes of data collection. For all aspect of the training program,
trainers use specialized tracking software to monitor the flow of goods and services to
trainees. For example, in addition to the metrics provided by S&C coaches and medics,
designated nutritionists also use software to track personalized data on trainees’
macronutrients in order to create personalized meal plans (Rubin, 2016).
• Function 2. Improvement of trainees’ job performance
At the Job Performance level, the training program focuses on improving the ability of
the team to win football matches. All of the aforementioned KSA development is directed
toward developing trainees’ ability to perform in a competitive arena against similarly-
trained teams from other universities. By winning games against these opposing teams, the
Seminole team justifies their KSA training and meets the expectations set by the workplace.
To help the team accomplish this end, FSU and the Athletics Department provide
considerable organizational support to these competitive matches.
o Workplace Transfer Readiness: Overall, FSU offers a robust network of support for the
trainees as they transfer their newly-acquired KSAs into the workplace. The most visible
support that FSU offers to football matches is in the form of the Doak Campbell Stadium,
a facility primarily dedicated to hosting visiting teams for competitive play. Despite the
only intermittent use of the stadium for this purpose, the university still works to maintain
and prepare the stadium, playing field, parking lots and related facilities during and
between matches. Some of this work is performed in coordination with external
organizations such as SODEXO, which administers foodservices for the stadium
(Buzalka, 2017). However much of the work is internally orchestrated by the university,
which must at all times employ a wide array of support workers to maintain the stadium’s
11physical structure, cleanliness, and usability. Likewise, the university develops and
enforces game-time regulations for its parking lots and transportation services in order to
serve the interests of the football matches (“Transportation,” n.d.), and coordinates with
local government to ensure transportation availability to and from games (TalGov, n.d.).
This extensive organizational support ensures that trainees are given the opportunity to
apply their newly-acquired KSAs in the manner that the sponsoring organization expects:
by exhibiting these KSAs during games.
• Function 3. Improvement of organization’s performance
At the Organizational Performance level, the training program results in more money,
more prestige, and thus a greater ability to recruit talented players. The football program is a
potentially lucrative operation for FSU. If the team wins more games, then the most
immediate reward for the university is monetary: more people will purchase tickets to watch
live football matches in addition to purchasing concessions during these matches. Playing
success also ensures that FSU ranks highly against other NCAA schools, thereby increasing
its fame and prestige in the collegiate circuit. The combination of money and prestige
increase the university’s ability to recruit talented high-school footballers to the program,
thus creating a virtuous cycle and perpetuating FSU’s ability to gain money and prestige
through its football team.
o Organization’s Maintenance of Training Transfer: In addition to training for football-
specific KSAs, trainees are provided with a variety of supporting instruction aimed at
transferring their training both to the workplace and beyond. The NOLES program offers
athletes training in positive habits, community development, professional development,
job placement, and leadership development (Seminoles, 1999). Thus, the organization
attempts to prepare trainees for transfer on an ongoing basis that extends beyond
immediate gameplay.
o Discussion of organization’s overall commitment and support for training program:
Given the aforementioned incentives for FSU to have a successful football team, the
organization’s commitment to ensuring its success is understandably strong. The
university has pledged both dollars and resources to support the training program.
12o Organization’s Mission and commitment to innovation: Innovation is a
consistent commitment for FSU, provided that the innovation continues to be
justified by performance results. In addition to funding tech innovations such as
the $60,000/year Cataput program (Davis, 2018), FSU also utilizes its function as
a research university to produce continuing innovations such as newly designed
bracing system (Boehm, 2019). Indeed, the organization’s self-stated mission is to
be “among the nation’s most entrepreneurial and innovative universities” (Florida
State University, n.d.). Given this existing commitment, it is likely that continuing
commitment to innovation will be forthcoming for the program.
o Allocated resources for training and support of the program: The
‘Unconquered Campaign’ represents both the most recent and the most significant
pledge of organizational support for the football education program. Running
through 2022, the program commits to $100-million in fundraising, $60-million
of which will be allocated to the construction of a new, all-inclusive Dunlap
Football Facility (Seminole Boosters, 2018). In addition to the program-specific
support provided by FSU, the university also offers ongoing indirect support to
program elements such as sports nutrition with a $173-million foodservices
contract with Sodexo (Buzalka, 2017), and sports medicine with ‘state-of-the-art’
medical technologies in the Don Fauls Athletic Training Facility (FSU Sports
Medicine, 2018). Such ancillary services are supported by FSU’s employment of
a wide array of support personnel (cooks, janitors, vendors, etc.) that contribute to
the overall infrastructure in which both training and competitive playing occur.
o Capabilities for transfer and maintenance: Despite the extensive organizational
supports, we must re-emphasize the structural conflict between FSU as a host to
the football program and FSU as an academic institution. Trainees are necessarily
unable to devote full attention to football due to academic requirements. There is
no way for FSU to circumvent this problem while still functioning within the
regulation of the NCAA (NCAA, 2019), and thus the program must suffer.
Overall, however, the financial and resource commitment provided by FSU
suggests that the program can be maintained and expanded in perpetuity and to
the extent that it produces successful teams.
13o Incentives for Employees participation: There are several inbuilt incentives for
trainee participation in the program. The most immediate and quantifiable of these
is the potential for scholarships through the university. FSU offers full athletic
scholarships for NCAA Division 1 athletes (Office of Financial Aid, 2019).
Successful players also gain subjective gratification from the relative fame and
praise that they receive as a result of winning games. Less immediate is the
potential for successful players to be scouted into the NFL and thus seed a long-
term and well-paid career as a professional football player.
4. Evaluation Logic Model
In order to organize the above analysis into an actionable format for creating an
evaluation, we will employ the Logic Model shown below. This model will serve as the
guideline for constructing an evaluation plan. In the step ‘1. Examine and Analyze’ portion of
the model, we have given a brief summary of the information detailed above. The model
contains our basic logic for approaching the problem of evaluation, and we will follow its
progression as we move to step ‘2. Evaluate’. The program will, however, require a multi-phase
evaluation and we may need further information after the award of the evaluation contract.
14Thus far, we have taken the essential first steps to determining the evaluation needs of the
FSU Seminoles football training program. The previous pages have detailed our analysis of the
program’s current purpose and structure. We have outlined a systemic approach to this problem
and our reasoning for adopting it. The understanding gained from this systemic analysis will
serve as the foundation for the formation and implementation of evaluation protocols tailored to
the actual needs of the program. We now proceed to outline our Evaluation Methodology &
Implementation plan for the Seminoles ‘football education’ training program.
5. Evaluation Methodology Overview:
In the proceeding text, we will outline the high-level evaluation questions that will inform
evaluation procedures, explore the specific instruments and procedures that will constitute the
evaluation, and offer justification for the evaluation methodology. The action steps outlined in
this paper derive directly from our Systemic Analysis of the football education program. The
present content is a continuation of the Evaluation Logic Model.
We will begin by outlining in broad strokes the Evaluation Management Plan (EMP): a
systematic model for evaluations which informs our overall approach. We will next provide a
detailed breakdown of the guiding questions derived from the EMP, the procedures to address
those questions, and the logic that led to the procedures. Finally, we will outline a
Communication Plan for stakeholders within the program. In combination, these elements will
provide the reader with a clear understanding of the path that we will take toward a successful
evaluation. We proceed with an outline of our model:
156. Evaluation Management Plan:
The EMP template shown above serves as our guide for developing an evaluation
methodology. It increases in granularity of focus from general categorizations to specific
instruments. Following the outline, we see the evaluation broken broadly into ‘Processes’ that
address program implementation and ‘Outcomes’ that address the effectiveness of the
implemented program. Each of these broad functions is subcategorized into Relevant Training
Features. These training features parallel those outlined in our Systemic Analysis (see Appendix
A), and they act as the basis for forming our high-level evaluation questions. With our high-level
evaluation questions established, we proceed to clarify the nature and source of information
required to answer these questions, and finally we determine the specific collection methods that
are most appropriate.
• Justification of Evaluation Methods: As outlined above, the methodology for this
evaluation follows a logical progression from categorizations to questions to instruments
of measurement. The evaluation concerns that we have identified for the football
education program arise from the findings of our prior analysis of program features.
Maintaining the systematic approach used in our analysis ensures that our data collection
methods for the Processes and Outcomes of the program are both relevant and
appropriate.
16o Process Evaluation: As mentioned above, we have narrowed our area of focus
based on thematic criteria. The first of two broad evaluation frameworks is
Process Evaluation. This examines the overall implementation of the program to
establish whether it is appropriately formed and executed. During this portion of
the evaluation we examine the analyzed inputs, training delivery, and
implementation process. By examining these processes, we may verify the
integrity of the training program. Each of our evaluation questions addresses a
core element of Process Evaluation, as outlined in the EMP’s Relevant Evaluation
Issues, and investigates the integrity of that program function.
o Outcome Evaluation: In compliment to the Process Evaluation outlined above,
we will also conduct an Outcome Evaluation in order to make a judgement about
the short and long-term outcomes of the training program. We will focus on each
of the three Training Functions outlined in our Systemic Analysis: KSAs, Job
Performance, and Organizational Performance. KSAs target shorter term
outcomes of the training, while Job Performance and Organizational Performance
target longer term outcomes. Addressing these elements will, in combination,
offer a robust picture of what the program actually accomplishes and fails to
accomplish.
7. Evaluation Procedures, Instruments and Measures:
Three types of data collection inform our approach: Qualitative, Quantitative, and Mixed.
Quantitative data collection is strictly numerical in nature. We utilize this methodology when
statistics are available and appropriate to address the issue at hand. Qualitative data is more
subjective, but essential in defining correct procedures, assessing attitudes, and the like. Often, a
Mixed approach to data collection is most appropriate in order to form a full picture of the
problem space. This section will address the specific Evaluation Questions and data collection
methods for both the Process Evaluation and the Outcome Evaluation.
• Process Evaluation Methods: Outlined below are the key Training Features of our
Process Evaluation, the Evaluation Questions that we will use to investigate them, and a
discussion of the methodology that will be used to answer those questions. We proceed to
discuss the specific Process Evaluation Issues:
17o Needs for training and how the training process relate to those: Generally
speaking, evaluation questions in this category are used to establish whether the
training process satisfies an actual need for the organization. In the case of the
football education program, although the training may be said to serve a de facto
purpose, our systemic analysis found that the need for training arose from the
1947 Act of Legislature that created FSU. Thus, our evaluation question “Does
the program being implemented align with the intended purpose?” may be
answered in the affirmative through a qualitative examination of historical
documents.
o Training instructional context: Here we examine the broad context in which the
training occurs in order to evaluate whether the training is appropriate.
▪ Quality and relevance of training set-up and modality: In order to
establish whether the training set-up indeed serves the intended function of
training, we pose the following evaluation questions. First, “To what
extent does the training modality align with the instructional objectives?”
and second, “How appropriate are the training processes given the
objectives of the program?” The first question examines the underlying
goals of the program in order to ensure that the program’s format is
appropriately aligned with these goals. The second question again
examines the underlying goals of the program, but focuses on the
processes within the training format. If there is a problem with the former,
then the latter will also be inappropriate. In order to establish the
appropriateness of both elements, we shall employ qualitative data
collection methods using the trainers as our information source. Further,
because of the complex and subjective nature of the information that we
wish to collect, we will use interviews as our collection instrument. The
semi-structured interview approach will allow us greater flexibility in
questioning and generate a better understanding of the problem space.
Please see Appendix B for examples of our interview questions on this
topic.
18▪ Quality and relevance of training materials: Moving from the overall
modality to address the actual training materials, we pose two interrelated
evaluation questions. First, we must ask “Are the training materials timely
and relevant?” and next “Are the selected materials up-to-date with
Athletic Science best practices?” The first question addresses the
appropriateness of the program-specific training materials while the
second question adopts a comparative view of the program at a high level.
Because the information needed to establish the answers for these
questions involves both performance measures and abstractions, we will
employ a Mixed method of data collection. To a large extent, the first
question can be answered through statistics generated by the Catapult
program. This program offers a built-in feedback loop for data collection,
and because the program has already been used to make alterations to the
training materials it presents a large pool of past performance data from
which to draw comparisons. The latter evaluation question, however,
requires a more qualitative approach. In order to answer this question, we
will rely on Subject Matter Experts (SMEs) in the field of Athletic
Sciences. These experts can view the program through an up-to-date lens
of best practices in their area of expertise. Here, again because of the
conceptual nature of the information needed, we will use a semi-structured
interview as our instrument. Sample questions from this instrument are in
Appendix B.
▪ Trainee’s readiness: In order to establish whether the trainees are
prepared for the instruction at the time of their entry, we begin by asking
“Are trainees put into the right training groups?” This is important to
establish because the training program contains multiple layers of training
segmentation based on athletic profiles and experience. If the individual
player is overworked or underworked due to incorrect placement then a
greater risk of injury or strength loss occurs. We adopt a mixed approach
to addressing this question. A statistical approach observes the PlayerLoad
variable for entry players and looks for discrepancies in this data between
19new and established players. Additionally, a questionnaire will be
distributed to coaches that asks such questions as “What are the primary
variables that determine which training group entry players go into?” in
order to collect qualitative considerations that strict numbers may
overlook. In addition to this question of trainee grouping, we will ask the
evaluation question “Are trainees adequately informed of program
expectations and resources?” The most straightforward source of this
information is the players themselves. An attitude questionnaire that
gathers qualitative data on a Likert scale will amply serve the purpose, and
examples from this questionnaire can be found in Appendix B.
▪ Experience and expertise in trainers: In order to establish the
experience of trainers, we have formed two interrelated evaluation
questions. In ascertaining how appropriately the trainers are being
deployed, we start by asking “Are the skillsets of trainers matched
appropriately to job function?” This question requires qualitative data to
answer and its answer can be got from employment records, which are
readily available. In this instance, we will simply index the required
qualifications for job positions against the resumes of the trainers actually
doing the job. To adopt a broader view of the problem space, we continue
the line of inquiry with the guiding question “Are trainers qualified to
industry standards?” Once again using qualitative measures, we turn to
lead coaches as our source of information. Because the information
needed is limited in scope a questionnaire will be sufficient. See Appendix
B for examples.
o Performance context: In this section, we examine the readiness of the organization
to help the trainees employ the training that they gain through the program.
▪ Support for transfer of training and transfer readiness: In order to
evaluate whether or not the organization is ready and willing to support
training transfer for its athletes, we will start by examining the actual
transfer on a statistical basis. Because the program operates on an ongoing
basis, with repeat trainings between competitive games, we are using what
20may appear to be an Outcome-oriented question for our Process
evaluation. Thus: “Does performance during practice transfer / replicate
performance in competitive playing?” We will use the cyclical nature of
the training & transfer to advantage and compare ongoing statistics in-
game vs. out-game. In addition to comparing the statistical data, we will
interview trainers about transfer / replication of strategic and tactical
training in the competitive field. The combination of the two methods will
give us a picture of both the raw athletic transfer and the higher-order
cognitive transfer. Any discrepancies will point to an ongoing blockage in
the movement from training to job performance. By examining the
individual statistics, we can identify faults at the organizational level. To
compliment this examination, we ask the more direct evaluation question
“To what extent does FSU facilitate the transfer of training to game
environment?” This question will be answered using mixed methods.
Financial data is available for the Athletics Department and will be
reviewed to determine exactly what budgetary allocations are made for
training transfer. Additionally, trainers will be interviewed via
questionnaire in order to address transfer issues that the financial data may
miss. See Appendix B for examples.
o Training delivery and implementation process: As in the previous section, the first
question that we will use to evaluate the implementation process appears to be
oriented toward Outcomes rather than Processes. However, the ongoing nature of the
training program makes it useful to treat this question (“To what extent does the
implementation of Catapult contribute to improved player KSAs?”) in the Process
section of our evaluation: the implementation is an ongoing process. Once again, the
abundance of player statistics aids us in our quest to answer this question. We will
compare the data pre and post-Catapult to measure exactly how much value the
software delivers to the training program. In addition to this examination of the
technological implementations of the program, we will examine the program more
generally with the evaluation question “Is the training program being implemented
according to its design?” The answer to this question relies on qualitative data derived
21from participant observation. Through our Systemic Analysis, we have generated an
understanding of how the program ought to be implemented. We will next go and
look to see if it is actually being implemented in this manner.
• Outcome Evaluation Methods: Here we present our evaluation strategy as it relates to
Outcomes of the training program. Refer to the section ‘Justification of Evaluation
Methods’ above for a full description. We proceed in order of scope from KSA to Job
Performance to Organizational Performance.
o Improvements in trainees KSA: Here we will focus specifically on the impact
that training had on trainees. Our highest-level guiding questions are “Did the
program produce the intended player outcomes?” and “To what extent did the
training program increase the entry KSAs of trainees?” Generating answers to
these questions involves a mixed approach to data collection from several sources.
Because this category involves three separate elements (Knowledge, Skills, and
Attitudes), we will address each separately:
▪ Knowledge: The evaluation of knowledge acquisition will be based on a
pretest measured against a posttest for the essential elements of the
program. As outlined in our Systemic Analysis, the program trains several
areas of knowledge: Nutrition, Physical Training, and Mental Skills. The
most appropriate way to address a change of knowledge in these
categories is by assessing the trainees’ entry knowledge at the time they
join the program or are scheduled to encounter the material within the
programming, and then following up on this assessment with posttests to
see the knowledge gained. In this case, we will create pretests & posttests
as needed for each of the areas of knowledge. See Appendix B for sample
questions.
▪ Skills: The evaluation of skill acquisition will involve a mixed collection
approach of statistical records acquired through Catapult, along with
observations of player behavior. Taking the metrics of players before and
after athletic training offers fertile ground for quantitative comparison. In
addition, conducting player observations allows us to evaluate more
22technical and cognitively intensive skills that the raw statistics can readily
accommodate. The combination of the two methods offers both a direct
and a comprehensive path to evaluating skill development in trainees.
▪ Attitudes: In order to evaluate the attitude development of trainees as a
result of the program, we must rely on qualitative measures. We form an
additional evaluation to direct these measures: “How satisfied are trainees
with the program as implemented?” The answer to this question must be
found at the source: present and past trainees. Because we are not
addressing complex cognitive skills, a simple questionnaire on a Likert
scale will suffice to gather the data. An example of this questionnaire can
be found in Appendix B.
o Improvements in trainees’ job performance: Here we address the extent to
which the trainings contributed to the trainee’s on-the-job performance (in this
case, playing competitive games). Here, based on our analysis, we have identified
the newest and largest variable within the program as tech expenditure. Thus, it is
timely to ask two interrelated questions about this variable. 1. “To what extent has
increased tech expenditure contributed to an increase in games won?” and 2. “To
what extent has tech expenditure contributed to decreased injury / out time among
players?” As the largest single expense of the training program, it is critical to
establish the actual contribution of the technological rollout to job performance.
In order to establish these contributions, we rely on the abundance of year-over-
year game and injury statistics. By comparing the data from years prior to
implementation with years after implementation, we can distinguish any upward
or downward trends in game performance. Likewise, we will compare medical
records along the same lines in order to determine the total ‘out-time’ (and thus,
reduced job performance) due to injury.
o Improvements in organizational performance due to the training: In order to
determine the organization-wide benefits that the training program has brought to
FSU, we will pose three guiding questions. The first two are as follows. “To what
extent has the training program contributed financially to FSU? What is the cost /
benefit ratio of the program?” In this case it is possible to use quantitative
23methods by combing FSU’s financial records to find answers. By looking at the
ongoing and startup expenses for the training program, and comparing it against
the revenue generated by competitive games, we will arrive at a cost-benefit ratio
from the standpoint of the organization. Our third question in this category (“To
what extent have national rankings for the Football Athletic Program impacted
player recruitment?”) requires a more mixed approach in order to generate
answers. In order to address this question, we will rely on three sources: Player
Entry Statistics, Players, and Recruiters. By once again employing a quantitative
approach with year-over-year player statistics, we can correlate national rankings
with how athletically skilled new players are. If the program receives better
players when it has a high national ranking than when it does not, we may
strongly infer that the success of the training program is an important contributor.
In addition to this quantitative measure, we will also seek feedback from players
and recruiters to gain additional perspective. A questionnaire will be sufficient to
determine the extent to which national rankings impacted players’ choice of
schools (ex: “What were the national rankings of your top three school choices?”)
and recruiters in their ability to recruit new talent (ex: “Do you find it easier to
recruit top players when the program increases ranking?”). The combination of
player statistics, player feedback, and recruiter feedback will provide a depth of
understanding that no individual component can.
8. Evaluation Communication and Reporting plan:
As we gather the above data, it is important to have a clear plan for communicating the
findings in a timely manner and to the appropriate individuals. In order to ensure this process, we
use the model presented below. As with the Evaluation Management Plan, it follows a logical
sequence from the general area of focus to the specific implementation.
24Information on our Process evaluation will be delivered to trainers and supervisors on an
ongoing basis in order to ensure their participation. By making them aware that the evaluation is
taking place and that the supervisors are endorsing it, the trainers are more likely both to comply
with our procedures and to perform well. This report will be in the form of a weekly summary,
and will be delivered ideally by superiors who have also received the report. An additional
summary along with a conference will be provided to trainers at the conclusion of the process
evaluation in order to ensure that clear next-action steps are in place.
Our communication on the training Outcomes will break into three separate categories
based on the three functions of training. The KSA report will be delivered, along with the final
Process report, to trainers and administrators as part of their summary and conference at the end
of the evaluation. Additionally, a Job Report will be delivered to the department heads and
administrators along with the financial accounting department. This information will take the
form of a paper summary along with a meeting to discuss at the end of evaluation. Finally, an
Organizational Report will be delivered to the executive committee of the Athletic Department,
along with the financial accounting department. This will take the form of an executive summary
in the case of the department heads, and a full financial report in the case of the accounting
department. These will be mail-delivered with an additional conference if needed.
9. Conclusion
In this paper, we have taken the essential first steps to determining the evaluation needs
of the FSU Seminoles football training program and creating an evaluation protocol that
25You can also read
