Institutional Effectiveness Report
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Institutional Effectiveness Report
Report for Academic Year: 2009-2010
Discipline: History
Collegium:
Discipline Description:
History is one of the central disciplines in a liberal education. As broad as human
experience, it provides a context for the understanding of literature, art,
philosophy, and the sciences. Akin to both the Humanities and Social Sciences,
history gives attention to the individual and to society as a whole, revealing the
vast range of human experiences, the extraordinary variety of human institutions,
and the inevitability of change. The study of history builds skills and knowledge
that are indispensable for any career: clarity in writing and speaking; effective
use of evidence and argument; the ability to perform independent research; and
an awareness of cultural differences and commonalities. It is thus excellent
preparation for a wide variety of fields—law, teaching, business, public service,
journalism, and even medicine.
Faculty & Staff:
Barnet Hartston, Associate Professor of History, 2009-10 Coordinator
Andrew Chittick, E. Leslie Peter Professor of East Asian Humanities
Kip Curtis, Assistant Professor of Environmental Studies
Lee Irby, Assistant Professor of History
Cathy Griggs, Associate Professor of Anthropology & American Studies
Carolyn Johnston, Professor of American Studies and History
Kathleen Keller, Assistant Professor of History
Greg Padgett, Associate Professor of History
Heather Vincent, Assistant Professor of Classics and Ancient Studies
Modifications Resulting from Last Assessment Cycle:
As a result of our last assessment cycle, we made the following modifications to
the discipline:
1.) In response to weak scores in historiography we instituted changes in the
course "Making History," which is offered by different faculty each year. All
"Making History" courses now require Anthony Brundage's "Going to the
Sources," and more specific discussions of historiography to be integrated into
class time.
Results: Our students’ scores in historiography rose slightly (see Discussion
section below for more detail). Although scores increased, students still did not
Institutional Effectiveness Report
Page 1 of 9meet the benchmark in this category. However, the “Making History” course is
taken by underclassmen, so we cannot expect the modifications to have an
impact on the last assessment cycle which evaluated senior comps.
2.) In response to relatively low scores in world history (as compared to American
and European history content scores), we broadened our range of offerings in
World History courses by adding "Modern African History" and "Religion in Global
History" beginning in Spring of 2010. Our goal was for these new electives to
provide more opportunities for our majors to gain exposure to World History
outside of our required surveys and Asian History offerings. World/Asian History
was the lowest-rated of all the content sub-sections for the 2008-2009 cohort.
Results: The data from the comprehensive exam suggests that these
modifications did not have a significant impact. Student scores in world history
stayed about the same. Our students did not meet the benchmark either year.
Again, as in the previous modification, it seems that the addition of new courses
will have a long-term impact. As the courses are electives, we also know they will
not impact every student. As they were offered in the spring 2010 (after
comprehensive exams were mostly complete), they did not impact the 2009-2010
assessment cycle, but will hopefully impact SLOs in world history in the future.
3.) We made efforts to solve logistical problems so that all PEL students were
evaluated by the oral proficiency rubric.
Results: We succeeded in collecting data of all PEL students who were
evaluated according to the oral proficiency rubric, thus improving the collection of
data.
4.) Finally, we decided to make significant alterations to our listed SLOs for the
history discipline; some of these changes were simply intended to bring about
greater clarity in assessment, but the changes also were intended to force us to
pay greater attention to several skill-based SLOs that have sometimes been
weaknesses in the past. First, we merged our SLO #2 and #3, since they both
deal with methodological/historiographical issues. The new single
historiography/methods SLO are still be measured by the historiography sub-
section of the Mastery of Content rubric. Second, we decided to add independent
writing and oral SLOs instead of integrating these measurements in our other
four outcomes. The writing SLO was separated from the SLO on Information
Literacy (Sources), an area where our assessment scores have been relatively
weak. Our goal in revision of SLOs is to allow us to pay closer attention to key
skill areas for graduates, and to simplify the assessment process by bringing our
SLOs into better alignment with the rubrics we are currently using. See revised
SLOs below.
Results: The modifications to our SLOs have simplified the assessment process.
Institutional Effectiveness Report
Page 1 of 9Outcomes:
Student Learning Outcomes
Students who graduate with a degree in History from Eckerd College will be able
to:
1. Demonstrate an understanding of key events, trends, and debates in
American, European, and World History. (Measured by the ETS Major Field
Exam and the first three content-related sub-sections of the Mastery of
Content and Historiography rubric).
2. Display a general awareness of historical methods and historiography,
and thorough knowledge of the historiography of at least one field.
. This includes the ability to think historically with regard to
issues such as causation, cultural diffusion, the role of the individual in
history, geographic and demographic influences in history, and gender and
minority issues in the past, citing examples from both the Western tradition
and the wider global experience. (Measured by the Historiography sub-
section of the Mastery of Content and Historiography rubric.)
3. Engage in a variety of types of scholarly writing such as book
and film reviews, annotated bibliographies, and historical essays.
(Measured by the Thesis, Analysis, and Mechanics sub-sections of the
Research and Writing Proficiency rubric.)
4. Show a basic level of information literacy. This includes the ability to locate,
use, and and properly cite bibliographical information from both print and
electronic sources. (Measured by the Sources sub-section of the
Research and Writing Proficiency rubric.)
5. Express themselves clearly, including demonstarting a basic competency in
the content, organization, delivery, and style of both formal and informal
oral presentations. (Measured by the Oral Proficiency Rubric.)
Other Program Outcomes
None.
Assessments:
Our assessment was based on both internal and external direct measures of
student outcomes. First, three rubrics were used to measure student
performance on the four main components of the Senior History Comprehensive
Exam (see appendix 4 for rubrics). Second, student performance on the ETS
Institutional Effectiveness Report
Page 1 of 9History Major Field Exam was evaluated in comparison to other colleges across
the country. We had originally intended to use the ACAT History exam as a
substitute for the soon-to-be-defunct ETS exam. However, we decided to use the
ETS for one more year before making the transition.
Internal Measures:
1.) Senior Comprehensive Exams -- The Senior Comprehensive Exam in history
consists of four parts. Although each part of the exam is unique in its
requirements, all four sections are useful in measuring each of the SLOs
listed above. That is, each of the four parts of the exam provides a different
measure of the same basic skills, but within four different contexts that are
common both for graduate students and professional historians.
Part I.) Written Comprehensive Exam
Seniors are required to submit four formal short research papers
based on a choice of topics within four categories: American
History, European/Ancient History, World History, and
Historiography. These essays must include evidence of
research using legitimate secondary and/or primary sources.
Part II.) Research Exemplar
Students must submit a substantial history research paper that they
wrote while at Eckerd. This 'Research Exemplar' should be an
example of their very best work.
Part III.) “Book of the Year” Exam
During a Winter Term seminar, seniors are assigned a book that
has made a substantial impact on the field of history. They then
take an in-class essay exam, which is formally submitted in their
comprehensive exam packet.
Part IV.) Oral Exam
After submitting the first three parts of the Comprehensive Exam,
seniors sit for a one hour oral exam before members of the history
faculty. This exam is designed to allow students to present,
elaborate on, and defend their work on the written portions of the
exam.
As a part of our regular internal assessment process, the history faculty
now uses three evaluative rubrics (appendix 4) to measure student learning
outcomes on
each individual comprehensive exam as a whole:
a.) Mastery of Content and Historiography
b.) Research and Writing Proficiency
c.) Oral Proficiency
Institutional Effectiveness Report
Page 1 of 9Each of these rubrics is divided into four sub-categories or sub-fields, and
student performance in a particular area is rated on a 5 point scale (1=Failing,
5=Excellent). Our basic benchmark for student performance is that
70% of students will score at least a 3.5 average (midway between adequate
and good) on each of the sub-categories as well as on each of the rubrics as
a whole.
External Measure:
2) ETS Major Field Exam – this is a standardized test that measures general
knowledge of European and American history. This exam is designed to test
how well our students are achieving student learning outcome #1 as
compared to students in other institutions. Our benchmark is that students
will on average finish above the 50th percentile nationally.
Results/Discussion:
Assessment was conducted in 2009-2010 on the student learning outcomes of
15 history seniors (one of who re-took the exam). Between 3 and 5 history
faculty evaluated each comprehensive exam based on availability of faculty.
The data discussed below can also be found in two attached appendices.
Appendix 1 provides a summative table of student scores on the comprehensive
exam and percentage of students reaching the benchmark in all 5 SLOS.
Appendix 3 provides more detailed student-by-student scores and data medians
and averages as well as the results of the ETS exam.
1. SLO #1 (Content):
Based on the comprehensive exam-based rubric evaluation, 62.5% of 15
students met the benchmark of a score of 3.5 in combined content scores
(compared to 53% in 2008-2009 and 72% in 2007-2008).
In comparison to the 2007-2008 and 2008-2009 comprehensive exam
assessments, changes in specific content area scores were mixed. Scores in
American history dropped from 4 median/4.06 average in 2007-2008 to 3.67
median/3.8 average in 2008-2009 to 3.45 median/3.51 average in 2009-2010. In
European history, scores fell from 2007-2008 when they rated 4 median/3.98
average to 3.67median/3.70 average in 2008-2009 and rose again to 3.9/3.76 in
2009-2010. In World/Asian history scores went from 4 median/4.12 average in
2007-2008 down to 3.5 median/3.58 average in 2008-2009 and remained about
the same at 3.55 median/3.43 average in 2009-2010. While overall a decline can
be noted between 2007-2008 and 2009-2010, the 2007-2008 data could be an
aberration or due to different raters conducting comprehensive exams (see inter-
rater reliability discussion below). Despite some changes in content scores over
the three years on which we have collected data, no significant change in scores
Institutional Effectiveness Report
Page 1 of 9emerges. Our students’ average and median scores have stayed in the 3.5 to
4.0 range.
In the ETS exam, looking at three years of data according to the median
score, overall test scores have roughly stayed the same. World history dropped
in 2008-2009, but improved again in 2009-2010. In 2009-2010 Our students met
the benchmark of 50th percentile for the overall test and in world history, but not
in American or European history. In 2007-2008 students met the benchmark in
all areas, however, only four students took the exam that year. We have
succeeded in getting more students to take the exam. What may appear to be a
drop in ETS scores is more likely the result of the gathering of more accurate
data. Unfortunately, while the sub scores for world history met the benchmark,
the sub scores for American and European History did not (despite the fact that
the actual test averages were almost equal). This is a trend reversal from last
year, when world history comparatively suffered. Furthermore, several students
scored in the 0 percentile, which indicates that some students may not be taking
the test seriously.
2. SLO #2 (Historiography)
Students’ historiography scores improved slightly from 2008-2009 to
2009-2010 when scores went from 3.33 median/3.49 average to 3.85
median/3.64 average although they did not reach the high scores of 2007-2008
of 4.24median/3.86 average. 56% of the 15 students met the benchmark of 3.5.
3. SLO #3 (Writing):
Looking at writing scores from comprehensive exam assessment, our
students writing scores declined from 4.13 median/4.07 average in 2007-2008 to
3.35 median/3.36 average in 2008-2009 and rose again in 2009-2010 to
3.69median/3.6 average. 56% of 15 students met the benchmark of 3.5.
4. SLO #4 (Sources):
In 2007-2008 students scores were 4.33 median/4.12 average. They
declined in 2008-2009 to 3.6 median/3.56 average and stayed about the same in
2009-2010 at 3.65 median/3.51 average. 56% of 15 students met the benchmark
of 3.5.
5. SLO #5 (Oral):
We only have two years of data on oral proficiency for our students.
Between 2008-2009 and 2009-2010 we show a fairly significant decline from
4.05 median/3.89 average to 3.6 median/3.7 average. 56% of 15 students met
the benchmark of 3.5.
6. General SLOs: Overall trends in data collected on comprehensive exams
indicate a drop in student performance from the 2007-2008 year to 2008-2009
and a slight rise in the 2009-2010 year. This trend can be observed in the
graphs in appendix 2. Oral proficiency appears to be largest drop, but that drop is
based on only two years of data. Overall, over the past three years our senior
Institutional Effectiveness Report
Page 1 of 9history majors have averaged scores between 3.5 and 4 (between average and
good). Nevertheless, students have failed to meet our benchmarks in two of
three categories in 2008-2009 and all three in 2009-2010.
Proposed Modifications:
1) Modifications to assessment:
a) Evaluating the role of a new standardized test as part of assessment
This year we will be using the ACAT as an outside standardized test because the
ETS test is no longer available. One of our goals will be to evaluate the
usefulness of the ACAT. Its major flaw is that it does not include general world
history questions (Latin America is the only subject field offered), which constitute
a significant part of our instructional content. While using the ACAT, we will
investigate other possible external standardized tests. The comprehensive
exams instructor will continue to impress upon students the importance of this
test in evaluating our program in order to avoid students scoring in the 0%. For
now the exam will not be part of the grade.
b) Norming of ratings
Despite the descriptions that accompany our rubrics, our scoring of students is
fairly inconsistent between raters. New raters have joined the comprehensive
exam evaluation since 2007-2008 and may account for some of the
inconsistency. A lack of inter-rater reliability could account for the drop in scores
from 2007-2008 to 2009-2010. In response, the discipline will conduct a norming
exercise at the beginning of each session of comprehensive exams (for a total of
2 norming sessions) relating to the written portion of the exam to attempt to bring
consistency to scoring.
2) Modifications to the major:
a) Considering modifications to senior comps/seminar.
We now have three years of data to consider based largely on
assessment of the comprehensive exams. Each year our students have failed to
meet some benchmarks. However, anecdotal and qualitative evidence suggests
that these scores do not accurately reflect the learning outcomes of our students.
Based on this observation, the history faculty are currently considering if
significant modification to the senior comprehensive exam process (now
conducted over a three-week winter term) is appropriate to obtain better data on
which to assess the strengths and weaknesses of our program. It has been
observed that our students might better demonstrate their abilities and produce
better work if given more time and attention in a semester-long course. A new
format would allow for more in-depth discussions with their professor and peers.
More time would allow students to more effectively demonstrate research,
Institutional Effectiveness Report
Page 1 of 9writing, and oral skills. However, such a new course cannot be implemented
immediately as a format and curriculum would need to be created and the course
approved by College Council. The history faculty will hold a series of meetings
over the 2010-2011 year to determine possible alternatives to the current senior
comps model and propose some modification to comps in the next assessment
cycle.
Discipline coordinator Kathleen Keller will investigate other history
programs at similar institutions, especially senior capstone courses,
comprehensive exams, theses or other ways of evaluating senior history majors.
This research will help the history faculty to decide if our senior comps program
is in accordance with the best practices in our field.
b) Investigation of historiographical methods instruction
Having decided in the last assessment cycle to make modifications to
improve learning of historiography/methods, the history faculty will continue to
monitor the impact of changes made to “Making History.” In addition, we will use
our meetings to discuss where historiography/methodology skills are being
taught in different courses within the major (not only in “Making History” and
senior comps). We will consider whether it is appropriate to introduce
historiography/methods instruction in a systematic manner throughout the major
beginning with the survey courses.
c) Investigation of oral proficiency instruction
Because of the drop in oral proficiency scores, the history faculty will
convene to discuss which courses already include an oral proficiency
component, the type of pedagogy used, and how oral proficiency is evaluated
throughout the major. As with historiography, the faculty will consider whether a
systematic and developmental approach to teaching oral communication would
be beneficial to student learning.
d) Improving the major
Projects a, b, and c, described above have the goal of improving the major
overall by confronting three main components of the history major (and history
assessment): seniors comprehensive exams, historiography, and oral
proficiency. As our data reflects no obvious strengths or weaknesses and little
significant change over time, the proposed meetings will help us identify ways we
can improve student learning outcomes within the major overall.
Signed: __Kathleen Keller___________,
2010-20011 History Discipline Coordinator
Approved by Educational Assessment Committee:
Institutional Effectiveness Report
Page 1 of 9Approved by Educational Program & Policy Committee:
Institutional Effectiveness Report
Page 1 of 9You can also read
