Fostering Quality of Reflection in First-year Honours Students in a Bachelor engineering program Technology, Liberal Arts & Science

Fostering Quality of Reflection in First-year Honours Students in a
Bachelor engineering program Technology, Liberal Arts & Science

Conference Key Areas: Competence development for engineering professions,
Engineering skills
Keywords: student reflection

ABSTRACT
This study focused on fostering the quality reflection displayed in semester self-
evaluation reports (SERs) of First-year Honours Students in a Bachelor engineering
program Technology, Liberal Arts and Science (ATLAS) of the University of Twente
in the Netherlands. The underlying problem that inspired this study was that the
quality of reflection was considered too low. In addition, there was unclarity in the
program on to what was expected of students with respect to reflection. Twenty-nine
participants, not previously exposed to academic training on reflection before,
received a Reflection Guide on how to write written reflections in their SERs. Two
online lectorials (interactive lectures) were provided to elaborate on the content of
the guide and to address any questions of the students. Quality of reflection in the
SERs was assessed using a standardized rubric and quality scores in the
intervention group were compared with scores of the student cohort of the previous
academic year (n = 33). Results showed that the intervention group reflected
significantly on a higher level than the comparison group.

1   INTRODUCTION

1.1. Background
ATLAS University College Twente is a bachelor of science program in Technology,
Liberal Arts & Science that aims at educating the ‘New Engineer’ (Goldberg &
Sommerville, 2014). The program has embraced the concept of self-directed
learning (Gibbons, 2002; Saks & Leijen, 2014), meaning that students attain learning
goals mostly in their own way. Semester goals (there are six semesters) define a
developmental framework that allows students to gradually build their own academic
profile as New Engineers. At the same time, the framework safeguards that all
students reach the intended learning outcomes of the program as a whole. At the
start of each semester, students write a plan in which they explain how they intend to
reach the semester goals of their current semester. At the end of each semester,
students write a self-evaluation report (SER) in which they reflect on their
development and evaluate whether they reached the goals of the semester. The
SER is input for a semester assessment meeting in which a group of assessors
decides to what extent the students’ self-evaluation can be justified. As part of this
assessment, students receive feedback on the quality of reflection and self-
evaluation displayed in their SERs. As a requirement to pass the semester, quality of
reflection and self-evaluation needs to be at a sufficient level.
        Reflecting and self-evaluating are considered important skills by the program,
both for the students’ learning, and for their role as New Engineers in society. New
Engineers typically work interdisciplinary, meaning that they need to be able to
quickly familiarize themselves with new fields of expertise to be a linking pin in
interdisciplinary teams. Actively monitoring what one knows and what still needs to
be learned through continuous reflection and self-evaluation is therefore of seminal
importance. Life-long learning characterizes the professional practice of New
Engineers and reflection is regarded an important skill in this respect (Rogers, 2001,
see also Yost et al., 2000).
        Skills in reflection do not come naturally, but can be trained (e.g. Gün, 2011;
Kori et al., 2014; Russel, 2005). However, reflection is considered challenging for
students and, for different reasons, support is needed (Abou Baker El-Dib, 2007;
Lee, 2005). For example, reflection requires creative thinking and seeing
alternatives, which might be challenging for students (e.g. Leijen et al. 2012).
Obviously, it is not enough to tell students to just “go and reflect” (Welch, 1999).
        Kori et al. (2014) reviewed several studies that aimed at supporting reflection
in technology-enhanced learning environments. Several support measures were
identified, ranging from videos, blogs and portfolios, to prompts and guiding
questions, and peer/teacher interactions. Their research showed that these
measures all have their benefits, but that this depends on the type of learning
environment and the specific reflection activity. The authors also state that measures
to evaluate the effect of the support measures in the studies they included in their
review lacked in terms of validity and reliability.

For the underlying study, we based our intervention on the work of Ash and
Clayton (2004) on articulated learning. On the one hand, their approach aligns with
what Poldner et al. (2011) considered to be common to reflection definitions; 1)
reflections are directed at something, 2) are part of a cyclical process, 3) can vary in
strength, 4) include an affective component and 5) include an intention to change
based on newly acquired insights. Also, there is evidence that their approach can
foster quality of reflection (Ash et al., 2005) as measured with rubrics for assessing
quality (or strength) of reflections. The rubric that was developed for this study was
further inspired by Kember et al. (2008). Other means to assess the quality of
reflection can be found in Dyment and O’ Connell (2011).
        On the other hand, their approach to support reflection is straightforward, it
makes use of scaffolding questions to structure student reflections and stimulates
students to think deeply about their object of reflection. Peer and teacher interaction,
and feedback cycles (Ash et al., 2005) might add to the training effect, but in this
case we decided not to add more to the intervention group’ workload than strictly
necessary because of the Covid-19 situation (the intervention was part of their
curricular work). However, to increase the impact of the intervention, examples of
reflections of different quality levels were provided, with an explanation of why,
according to specific criteria, the example would fall into a certain reflection quality
level category. This add-on is based on the well-established effect of learning from
examples (see e.g. Hoogerheide & Roelle, 2020).
         Up until the moment of the current study, the ATLAS program offered support
to first-year students in writing their SERs by providing two pages of written
instructions and one 1.5-hour lectorial (interactive lecture, similar to a workshop) on
the nature of reflection and self-evaluation. However, the effect of this support,
especially with respect to reflection, was not reflected in their SERs. Teacher-
assessors generally noted the poor quality of reflection displayed in the SERs, but
also appeared to use different criteria for assessing the SER reflections. Typically,
the assessment manuals for assessors contained no information about criteria to use
in providing feedback on the quality of reflection. Moreover, students did not seem to
be aware what was expected of them with respect to reflection. These observations
formed the motive for this study, the underlying problem being unclarity surrounding
what constitutes proper reflection, both on the students’ and the teachers’ side.
Without such clarity, the conditions needed to develop reflection skills are at least
suboptimal. Typically, quality of reflections in the SERs remained stable throughout
the program.
1.2 Research question
This study was part of a larger project that contained five studies. The research
questions of the overall project were 1) What would be an appropriate reflection
method for first-year ATLAS students that includes both support measures and
feedback criteria? 2) What is the effect of an intervention targeted at fostering quality
of reflection? and 3) What is the level of perceived usefulness and value of the

proposed reflection method on which the intervention was based for students and
teacher-assessors? In this paper, results of a study that targeted the second
question will be reported.
1.3 Intervention design
A qualitative pilot test made clear that students tend to focus their reflections on their
academic performance. In this, the distinction between reflection and self-evaluation
becomes blurry, which is corroborated by several student statements that point to
confusion about the distinction between them. Although most students feel that they
can meet expectations regarding reflection, the majority states to need additional
educational support. Clear instructions, templates and examples, feedback cycles
and interactive SER writing sessions were suggested.
      Due to the Covid-19 pandemic, education went online during the study and the
first-year students were under quite some strain already. Therefore, it was decided to
keep the intervention low-key. A Reflection Guide (based on the work of Ash &
Clayton, 2004) was provided with clear instructions, scaffolding questions that
served as a template to structure reflections and to stimulate students to think deeply
about a reflection object, quality criteria (rubric), and examples of low to high quality
reflections with fictitious examples that highlighted the criteria. These ingredients
mostly aligned with the needs identified in the pilot study, with the exception of
including peer/teacher interactions and feedback cycles, for reasons mentioned
before. The Reflection Guide was written to be self-contained as additional
instruction to write the SER, but the content of the guide was also discussed in two
non-mandatory online lectorials of approximately 1.5 hours. The first lectorial
focused in the students’ task of writing reflections in their SERs, the second one
repeated the first (on purpose; very few students showed up the first time), but also
elaborated on the quality criteria and the reflection examples. There was room for
questions in both lectorials.

2    METHODOLOGY
2.1 Participants
The intervention group consisted of 29 first-year ATLAS students in their first
semester (Class of 2023). There were 14 females, 58.6% of the whole class was of
native Dutch origin. For comparison, the first semester SERs of 33 students of the
Class of 2022 were used (also 14 females, 60.6% of the whole class was of native
Dutch origin1). Students in both cohorts were admitted on the basis of the same
admission criteria. Students from both cohorts that failed their semester were
excluded from analysis.

1 The author could not find evidence to support the idea that quality of reflection is affected by cultural
differences. ATLAS is an international study program, these data are included to show to what extent
the intervention and comparison group could be compared.

2.2 Materials
To assess the quality of reflection displayed in the SERs of the students, a rubric
was constructed based on the levels of reflection described in the Reflection Guide.
There are four levels of reflection and the rubric also contains intermediate levels,
see Table 1 below. The parameters that define the level of reflection are: 1)
understanding of the critical incident in term of the student’s learning (understanding
dimension); 2) implication of the learning for other situations in which it might apply
(transfer dimension); 3) the personal impact or - significance of the incident (personal
dimension) and 4) the extent to which the incident has had implications for future
goals or behaviours (future dimension). The highest level, which is not common
according to Kember et al. (2008), also includes evidence of a fundamental change
in perspective.
2.3 Procedure
SERs were retrieved from the program’s data base. The principal researcher first
identified the different reflections in the SERs and then assigned a level to each of
them on the basis of the rubric. An excerpt was considered a reflection when 1)
students apparently “looked inside”, and apparently thought about an experience or
2) when the excerpt was headed with “reflection” (this was common in the
intervention group because they were instructed to structure their SERs in separate
reflection and self-evaluation paragraphs). Students in the intervention group were
instructed to write three reflections, while students in the comparison group could
write as many as they liked. In this group, when students wrote more than three
reflections, the three highest-scoring reflections were taken as units of analysis.
2.4 Data analysis
Level 1 reflections were assigned a score of 1, 1+ reflections were assigned with a
score of 1.5, level 2 reflections were assigned a score of 2, etc. Mean reflection level
was calculated by averaging the scores of the three reflections with the highest
individual scores. Difference in mean quality of reflection was tested with a T-test for
independent samples. Possible effects of gender and nationality (Dutch/foreign)
were controlled for.

Table 1. scoring rubric quality of reflection
 Level 1: Non- reflection
 The intended reflection shows no evidence of the student attempting to reach
 understanding of a critical incident or significant learning experience. The student
 is basically evaluating his or learning on a general level, the learning is not related
 to any other situation in which it might apply. The reflection is not personal, it could
 have been written by any student. No implications for future goals or behaviours
 mentioned.
 Level 1+: the reflection scores higher on certain criteria, but not enough for the
 reflection to be fully classified on the next level.
 Level 2: Understanding
 A critical incident or learning experience is described to a certain extent, and the
 student is able to explain its meaning, but does not relate it to any other situation in
 which it might apply. Also, it does not become clear in what way the experience
 matters for the student personally, or how it affected future goals or behaviours.
 Level 2+: the reflection scores higher on certain criteria, but not enough for the
 reflection to be fully classified on the next level.
 Level 3: Reflection
 A critical incident is described clearly, and the student is able to explain its
 meaning and how it relates to other situations in which it might apply. It is also
 clear how and why the experience has mattered to the student on a personal level.
 Level 3+: the reflection scores higher on certain criteria, but not enough for the
 reflection to be fully classified on the next level.
 Level 4: Critical reflection
 All of the above (level 3), while it is also clear how the incident has affected the
 student’s future goals or behaviours. In addition, there is clear evidence of a
 change in perspective over a fundamental belief.
Note: criteria relate to four dimensions: understanding, transfer, personal and future

3. Results & Discussion
First, it must be noted that students in the comparison group delivered reflections
that were assessible with the criteria in the Reflection Guide. This indicates that they,
to a certain extent, had similar ideas about what reflection entails compared to the
intervention group. Therefore, it seems fair to compare the groups.
Average level of reflection in the intervention group was 2.0 (SD: .56, range 1.17 –
2.67). In the comparison group, this was 1.3 (SD: .30, range 1.0 – 2.17). A T-test for

Table 2. Percentages of reflections at
                   different levels in comparison and intervention group
                                   Reflection levels
                          SRs      1         1.5       2         2.5      3         3.5         4
 Comparison               T1       39.4      33.3      21.2      6.1      -         -           -
 group
 (n = 33)                 T2       54.5      36.4      6.1       3.0      -         -           -
                          T3       90.9      9.1       -         -        -         -           -

 Intervention             T1       -         13.8      24.1      20.7     34.5      6.9         -
 group
 (n = 29)                 T2       13.8      27.6      31.0      13.8     10.3      3.4         -
                          T3       34.5      31.0      17.2      13.8     3.4       -           -
Note: SRs: three strongest reflections found in the semester 1 Self-Evaluation Reports of the
participants (T1 – T3). Reflection levels: 1 = lowest, 4 = highest (see also Table 1).

independent samples showed this difference to be significant (t = -6.64, df = 60, p

and examples), educators can foster their students’ reflective abilities. Adding
feedback cycles and (peer-) interaction can further strengthen and reinforce quality
of reflection.
Two important shortcomings of this study will now be addressed. Firstly, the study
uses a between subject design to study the effect of the intervention. A pre-post
control group design would have been a more valid test, because now we must
assume that the intervention and comparison group were comparable in their initial
reflective ability. However, splitting the Class of 2023 in two to create a control and
comparison group was not deemed ethical because students might profit from the
intervention in terms of study progress (a SER rewrite is needed when the quality of
reflection is not up to par). Moreover, collecting base-rate information was deemed
inappropriate because of the serious workload the students had. On the other hand,
the comparison and intervention group were comparable in terms of gender
distribution, Dutch/foreign nationality distribution and ATLAS admission criteria. Of
course, the relevancy of these characteristics for reflective ability is not known (in
fact, results showed gender and nationality were unrelated to mean level of reflection
quality) and measures that could predict reflection ability were not taken, but overall
it is assumed the comparison was fair.
Secondly, the reflections in the SERs were identified and coded by the main
researcher. Although the coding rubric is deemed elaborate and insightful, a certain
expectation on his hand could have influenced the results. Involving a second coder
to assess the interrater reliability of the coding protocol would have solved this
problem.
Further research and development could focus on testing the effect of a more
elaborate intervention including feedback cycles and peer/teacher interaction.

REFERENCES

1. Ash, S.L., & Clayton, P.H. (2004). The articulated learning: An approach to
   reflection and assessment. Innovative Higher Education, 29, pp. 137-154.
2. Gibbons, M. (2002). The Self-directed learning Handbook: Challenging
   Adolescents Students to Excel. Wiley.
3. Goldberg, D. E., & Sommerville, M. (2014). A whole new engineer (1st ed.).
   Douglas, MI: ThreeJoy Associates Inc.
4. Gün, B. (2011). Quality Self-reflection through Reflection Training. ELT Journal,
   65.

5. Kember, D., McKay, J., Sinclair, K., & Yuet Wong, F.K. (2008) A four‐category
   scheme for coding and assessing the level of reflection in written work,
   Assessment & Evaluation in Higher Education, 33 (4), pp. 369-379.
6. Kori, K., Pedaste, M., Leijen, Ä., & Mäeots, M. (2014). Supporting reflection in
   technology-enhanced learning. Educational Research Review, 11, pp. 45–55.
   https://doi.org/10.1016/j.edurev.2013.11.003.
7. Rogers, R. R. (2001). Reflection in higher education: A concept analysis.
   Innovative Higher Education, 26(1), pp. 37–57.
   https://doi.org/10.1023/A:1010986404527.
8. Russell, T. (2005). Can reflective practice be taught? Reflective Practice, 6, pp.
   199-204. 10.1080/14623940500105833.
9. Saks, S. & Leijen, Ä. (2014). Distinguishing self-directed and self-regulated
   learning and measuring them in the E-learning Context. Procedia - Social and
   Behavioral Sciences, 112, pp. 190-198 10.1016/j.sbspro.2014.01.1155
10. Yost, D. S., Sentner, S. M., & Forlenza-Bailey, A. (2000). An examination of the
   construct of critical reflection: Implications for teacher education programming in
   the 21st century. Journal of Teacher Education, 51(1), 39.
   doi:10.1177/002248710005100105.