DSC Assessment Moderation Guidelines - August 2020 - DSC Teaching & Learning
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
DSC Assessment Moderation Guidelines August 2020
—
Contents
About these guidelines..........................................................................................................5
What is assessment moderation?...................................................................................6
Phases............................................................................................................................................7
Why is assessment moderation important?............................................................ 12
Who should be involved in moderation?................................................................... 13
Recommendations for effective practice ................................................................. 14
Guiding policy.......................................................................................................................... 16
RMIT policy............................................................................................................................... 16
Further information
info and support.................................................................................... 17
Selected references.............................................................................................................. 17
Appendix 1................................................................................................................................ 18
4 DSC Assessment Moderation Guidelines - August 2020
—
About these
guidelines
These guidelines have been
prepared to support and develop
effective assessment practice and
quality educational practice. They
explain what assessment moderation
is and why it is important. The
guidelines provide a holistic
moderation process complete with
examples, recommendations and
a checklist for adaptation to your
learning and teaching (L&T) context.
DSC Assessment Moderation Guidelines - August 2020 5
—
What is assessment
moderation?
(‘The process’)?
Assessment moderation is a cyclical process As shown in figure 1, There are five (5)
that assures systematic fairness. It includes phases of the assessment moderation cycle
internal and external processes whereby (Bloxham, Hughes, & Adie, 2015). These
multiple stakeholders engage in processes may overlap and more than one may be in
that support them to agree upon, assure and process at any one time.
monitor standards (Beutel, Adie, & Lloyd,
2016).
Figure 1: Phases of the assessment moderation cycle
6 DSC Assessment Moderation Guidelines - August 2020
—
Phases
PHASE 1:
Designing assessment for equity EFFECTIVE PRACTICE
(fairness and consistency)
Calibrating assessment tasks
In this phase the focus is on planning for the (Examinations in the School
quality of assessment tasks with reference of Property, Construction and
to course and program learning outcomes Project Management)
and to achieve fairness and consistency. The School of Property, Construction
Equity considerations may include whether and Project Management (PCPM) has
all students have equal opportunity developed rigorous moderation processes
to demonstrate learning outcomes or and tools to moderate examinations. In
any emergent bias or discrimination in collaboration with the responsible program
the assessment criteria, their possible manager, the course coordinator appoints
interpretation and application (Bloxham et one or more examination moderators
al., 2015). Relevant processes may include to check a proposed exam before it
informal or formal peer scrutiny, or review by is released. Using a pro forma (refer
a committee or professional accreditation PCPM Exam Moderation Checklist) the
moderator reviews content coverage,
body.
technical accuracy, the academic validity
of the examination questions and the
marking scheme, clarity of questions and
instructions and the quality of the layout.
The course coordinator actions agreed
amendments to create a new moderated
version that is then referred to the program
manager for approval. Only after associate
dean or program manager approval is the
exam finalised.
DSC Assessment Moderation Guidelines - August 2020 7
PHASE 2:
Calibrating understanding of EFFECTIVE PRACTICE
assessment requirements,
Calibrating assessment practice
standards and achievement. – feedback and critique in the
This phase is concerned with reaching
School of Art
agreement and shared understanding about Feedback and critique activities are
the assessment task(s), requirements, scaffolded across programs in the
performance criteria and standards. School of Art to foster the development
‘Calibrating’ individuals’ expectations in of students’ creative practice. Feedback
this way helps to assure the rigor and and critique ranges from simply reading
consistency of assessment tasks and each other’s creative work through to
processes. Calibration may involve sharing providing nuanced and detailed critical
samples of student work, independent analysis. In 2019, academics in the School
of Art began to calibrate these activities
marking, comparison and justification
by collaboratively developing four agreed
of judgements through a process of
domains of feedback and critique practice
‘consensus moderation’, or discussion held
when students and staff are engaged in
to reach agreement on a final mark. It is group tutorial sessions about student
important that consensus building involves creative practice:
professional/disciplinary judgement, genuine
debate and reference to student work (Beutel 1) Engagement: the personal attributes
et al., 2016; Bloxham et al., 2015). required to engage in ongoing dialogues
about creative practice
2) Process: the methods and processes
used to make creative works
3) Context: the disciplinary, social,
political and historical context of the
practice and the discourse surrounding
those contexts
4) Encounter: the experience of the
artefact/installation/performance/creative
outcome.
These domains were developed
collaboratively by teaching staff. The
descriptors and processes were calibrated
amongst the group for authenticity
within the disciplinary field. Each domain
was then defined to support shared
understandings and assessment practices
that foster learner agency in developing
creative work. The process was iterative
and is ongoing. In 2020, teaching staff
are working with these domains and
developing teaching resources that relate
to their areas of specialisation in art
(ceramics, drawing, gold & silversmithing,
painting, printmaking, sculpture, video)
and photography (art, documentary,
fashion, social practice, portrait). These
resources will be made available across
program teams and within the school more
broadly.8 DSC Assessment Moderation Guidelines - August 2020
PHASE 3:
Justifying marking judgements EFFECTIVE PRACTICE
(credibility and reliability).
Consensus moderation – School
of Media and Communication
In this phase the focus is on internal
processes for assuring the quality of In the Bachelor of Communication (Media)
assessors’ judgements. Relevant factors consensus moderation occurs at course
will be the extent to which assessors’ level. Each assessment piece worth
judgements are aligned to assessment more than 20% is moderated by pairs of
criteria, the credibility of evidence, markers who share samples at different
determination of agreed common standards grade levels and check these against
and the consistency of assessors’ the marking rubric. Major assessment
judgements. Practices may include random (worth 40% or more of the total mark) is
consensus moderated by the full teaching
checks, double marking, consensus
team. All fails and borderline grades are
moderation and checking of grade
cross marked.
distribution. Review of grade distribution may
be useful as a starting point for discussion of
standards, but assessment criteria, student
work and the alignment between these
and assessors’ decisions should also be
considered (Bloxham et al., 2015).
PHASE 4:
Externally validating processes
and judgements.
Building on the internal processes for
justifying assessors’ judgements, this
phase focuses on benchmarking those
standards through external comparisons.
Relevant external processes may include
peer review or professional accreditation.
Some disciplines and university groupings
have agreed processes for cross institutional
peer review to benchmark assessment
outcomes, e.g. https://www.iru.edu.au/iru-
work/calibration/. It is important that peer
review incorporates review of inputs, such
as course outlines, assessment tasks and
marking criteria, as well as marking itself.
It is also critical to train external reviewers,
undertake calibration processes and
allow time for discussions so that shared
understandings and standards can be
established (Krause et al., 2013).DSC Assessment Moderation Guidelines - August 2020 9
PHASE 5:
Monitoring and evaluation of EFFECTIVE PRACTICE
assessment efficacy.
BP309 Bachelor of Design
(Digital Media)
During this phase the focus is to evaluate
the overall quality and efficacy of the
In the Bachelor of Design (Digital Media),
assessment, including review of different all Digital Media courses at RMIT Vietnam
phases of the assessment moderation are moderated at the program level by
process. This reflective process may include course coordinators and lecturers. At the
looking at samples of student work, student end of semester, final moderation involves
evaluation data, examiners’ reports, grade review of final grades, with instances of
distribution and lecturers’ perceptions a Fail (NN) grade supported by written
about the adequacy of assessment task reasons for that grade. Grade distribution
design, information, criteria and standards, is also reviewed and discussed with any
including a marking guide. This may lead to revisions or adjustments undertaken
adaptation in future iterations of the course. through a further moderation process
This culminating phase assures quality by the RMIT Vietnam teaching team
and contributes to effective educational before final results being published. The
grade moderation offers opportunities
practice development as an ongoing cycle of
for reflection and development, as both
reflection and adaptation.
program managers gain better insight
into successes, challenges, similarities
Please refer to the checklist provided in
and differences at each campus for the
Appendix 1 for further guidance.
semester. Program managers conduct
regular meetings to discuss all updates
Courses offered at multiple and issues relating to the program. As
locations – equivalence and part of this peer review process, L&T
comparability discussions focus on course content and
approaches at each campus.
Where courses are delivered at multiple
locations, assessment moderation will
require consideration and planning for the
equivalence and comparability of those
offerings. Equivalence refers to maintenance
of quality standards across multiple
offerings in terms of course management,
learning outcomes, resources, assessment,
teaching, work-integrated learning (WIL)
and internationalisation. Comparability
is concerned with the extent to which
offerings have been contextualised and
customised to meet local needs, including
those of the student cohort, to achieve
equivalence. The design of curriculum,
learning activities, assessment and WIL may
all need contextualisation. In the process
of contextualisation, learning activities,
resources, assessment, delivery, teaching
practice and student learning support may
need to be customised to promote learning
for a specific student cohort.
See RMIT program and course review
process for further information.10 DSC Assessment Moderation Guidelines - August 2020
—
Why is assessment
moderation important?
Assessment moderation is critical for the of assessment criteria and standards,
ongoing development of effective educational and contribute to ongoing reflection and
practice. It assures academic quality by development of educational practice. Where
verifying the credibility of assessment (as there are multiple offerings of a course
evidence of student achievement) and across different locations, assessment
the credibility and reliability of assessors’ moderation processes will be required to
judgements (Bloxham, Hughes, & Adie, assure the equivalence and comparability of
2016). Assessment moderation can also those offerings, particularly for transnational
support equity, meet professional and system education.
requirements, build shared understandingsDSC Assessment Moderation Guidelines - August 2020 11
—
Who should be involved
in moderation?
(‘The people’)?
At RMIT, course coordinators have principal advisory committees and may also include
responsibility for assessment design and professional and industry bodies, relevant
quality, including moderation processes. community partners and contacts.
However, as the assessment moderation
process shows, this responsibility needs Genuine collaboration and reciprocity among
to involve multiple stakeholders in ongoing teaching teams and other stakeholders
discussions and genuine exchange. Such is especially important when designing
discussions need to happen when designing curriculum and assessment in transnational
assessment (alignment with professional teaching contexts. This means valuing and
standards, qualification level, learning drawing upon the specific and contextual
outcomes and activities), assessment criteria professional knowledge and expertise that
and marking guides, through calibration of all members of the teaching team bring;
assessment requirements, standards and using data to learn about the student cohort;
achievement, validation of judgements differentiating core curriculum content from
to ongoing evaluation and adaptation of customisable content; constructively aligning
assessment design and practice. As shown curriculum; being adaptive during delivery;
in figure 2, stakeholders will include teaching and evaluating performance data (Clarke,
teams (local and international) and program Johal, Sharp, & Quinn, 2015).
Figure 2: Key stakeholders in assessment moderation12 DSC Assessment Moderation Guidelines - August 2020
—
Recommendations for
effective practice
The following recommendations are derived al., 2016). This offers several benefits. It will
from scholarship on assessment moderation help all assessors to internalise requirements
practices and practical tips drawn from and standards and inform conversations with
student appeals. students about the assessment requirements
(Bloxham et al., 2015).
Assessment moderation should
involve ongoing dialogue Keep in mind the role of culture in shaping
communication, interaction, pedagogy
Start talking about assessment as early as (student and teacher roles, teaching
possible with your teaching team (Bloxham practices) and assessment, including
et al., 2015) and keep up the conversation underlying values and assumptions. Be
during the teaching period (Beutel et al., alert to stereotypes and ethnocentricity and
2016). If possible, collaborate on developing remember the importance of context for L&T,
assessment criteria, standards and evidence particularly for transnational offerings.
of performance, keeping in mind the need to
align assessment criteria, assessment tasks,
learning outcomes and pedagogy (Beutel etDSC Assessment Moderation Guidelines - August 2020 13
Consider developing marking Practical tips
guidelines and exemplars and aim to
moderate assignments such as oral ࢚ Be sure to align feedback assessment criteria
and performance descriptors so that students
presentations in real time. can relate comments back to these.
࢚ Make sure that your Part B course guide
accurately describes assessment and related
Involve students in understanding
administrative matters such as extensions,
assessment criteria
special consideration and penalties for late
By involving students in the use of assessment submission
criteria you can help them to understand those
criteria and develop evaluative judgement about
࢚ Group work assignments should clearly define
the group component (see clause 1.20 RMIT
their own performance. Consider asking students
Assessment Processes) and include a clearly
to review each other’s work using developed
defined process for when a group becomes
assessment criteria and/or a marking rubric to help
dysfunctional so that academic staff can
them to internalise standards.
intervene before assessment submission (see
clause 1.18 of RMIT Assessment Processes).14 DSC Assessment Moderation Guidelines - August 2020
—
Guiding policy
Assessment moderation is guided by external and RMIT policy. The Higher Education
Standards Framework Threshold Standards 2015 (Cth) (HESF) include several relevant
provisions. HESF standard 1.4.3 requires consistency between assessment and demonstration
of learning outcomes and validation of student attainment and grades awarded. HESF
standard 5.3 specifies requirements around monitoring, review and improvement of courses.
These include activities that could encompass assessment moderation processes, such
as comprehensive reviews of assessment methods and students’ achievement of learning
outcomes (at least every 7 years); interim monitoring of student progress (grade distribution);
and regular peer review of student success against comparable programs of study from
another provider (student progress, assessment methods, grading with cohort analysis) to
‘calibrate’ assessors’ grading.
—
RMIT policy
The RMIT assessment processes (clauses 3.1-3.30) set out practices for moderating
assessment during the teaching period, such as specification of a marking scheme, allocation
of a single marker (and a moderator) and double marking of assessment where there are
new markers in a course. To ensure consistency, cross marking and review of assessors’
marking may occur. Before results are finalised, consensus moderation will occur via Course
Assessment Committees (CACs) that check grade distribution and results across different
offerings of same course (clause 3.1). To assist this process the course coordinator reports
to the CAC on moderation processes, any issues arising with assessment, reasons for
adjustments made to marks, and any other matter relevant to approval of final results (clause
3.25.6). Adjustments need to be documented, with reasons included. All narrow fails (45-49)
must be double marked (clause 3.7).
Moderation by the CAC may lead to adjustment of student results to overcome disparities
in the difficulty of assessments and/or severity of marking (clause 3.23). Wherever possible,
such adjustments must be made before the marks for the assessment are communicated to
students (clause 3.23.1). Program assessment boards (PABs) consider CAC recommendations
on student or program cohort performance, how effective program assessment is and whether
it needs enhancement (clause 3.13).
RMIT program and course review process includes provisions for assuring equivalence and
comparability of courses offered across multiple locations. Section 24 of the Program and
Course Policy states requirements where RMIT programs are delivered by partners.DSC Assessment Moderation Guidelines - August 2020 15
—
Further information
info
and support
DSC Digital Learning and L&T Support: dsc.lt@rmit.edu.au
DSC Quality team: dsc.quality@rmit.edu.au
DSC academic services team : dscacademicservices@rmit.edu.au
—
Selected references
Beutel, D., Adie, L., & Lloyd, M. (2017). Assessment https://www.ed.ac.uk/institute-academic-
moderation in an Australian context: Processes, development/learning-teaching/staff/assessment/
practices, and challenges. Teaching in Higher moderation-guidance
Education, 22(1), 1-14.
https://www.iru.edu.au/iru-work/calibration/
Bloxham, S., Hughes, C., & Adie, L. (2016).
What’s the point of moderation? A discussion of
the purposes achieved through contemporary https://lo.unisa.edu.au/course/view.php?id=8539
moderation practices. Assessment & Evaluation in
Higher Education, 41(4), 638-653.
RMIT program and course review processes
Clarke, A., Johal, T., Sharp, K., & Quinn, S. (2016).
TEQSA guidance note on academic quality
Achieving equivalence: A transnational curriculum
design framework. International Journal for
Academic Development, 21(4), 364-376. TEQSA external referencing guidance note
Higher Education Standards Framework Threshold
Standards 2015 (Cth) (Standard 5.3) – Download
from https://www.legislation.gov.au/Details/
F2015L01639)
Krause, K., Scott, G., Aubin, K., Alexander, H.,
Angelo, T., Campbell, S., Carroll, M., Deane, E.,
Nulty, D., Pattison, P., Probert, B., Sachs, J.,
Solomonides, I., Vaughan, S. (2013). Assuring final
year subject and program achievement standards
through inter-university peer review and moderation.
https://www.uws.edu.au/__data/assets/pdf_
file/0005/576914/Handbook_2014_Web_2.pdf16
Appendix 1
DSC Assessment Moderation Guidelines - August 2020
DSC Assessment Moderation Checklist
This checklist is a guide to key steps in the assessment moderation process. Use or
adapt it with your teaching team to suit your context.
Phase 1: Designing assessment for equity (fairness and consistency)
� Have you reviewed all assessment tasks in your course?
� Have you checked all assessment tasks for alignment with relevant course and
program learning outcomes?
� Have you considered whether all students would have equal opportunity to
demonstrate the learning outcomes in all assessment tasks?
� Does the assessment cater for diversity in learning styles, English language
proficiency and cultural knowledge?
� Is there a variety of assessment types?
� Did you develop and/or review assessment criteria with your teaching team?
� Are your assessment criteria clear?
� Did you develop a marking guide to support your assessment criteria?
Phase 2: Calibrating understanding of assessment requirements, standards and
achievement
� Did you hold a consensus moderation meeting with all the markers in your course,
including those at different locations?
� Did you independently mark a sample of assessments at different standards of
performance, compare and agree results?
� Were those samples anonymised?
� Did you discuss reasons for different grades, including possible bias?
� Did you refer to student work in discussing variances in grading?
� Did you cross mark fails, work with borderline grades and any assignments graded
with a perfect score?
Phase 3: Justifying marking judgements (credibility and reliability)
� Did you do random checks of different markers’ grades and grade distributions?
� Did you review overall grade distribution for the course?
� Did you complete a report on moderation practices, any issues arising from
moderation and any adjustments to results, including the reasons for those?
� Did your teaching team contribute to the development of the moderation report?
� Did you contextualise communication and pedagogy in your course, particularly where
a course is offered in multiple locations, including outside Australia?
DSCAssessment_Moderation_Checklist.docx
Author: DSC ADG
Save Date: 29/07/2020
College of Design and Social Context Page 1 of 2DSC Assessment Moderation Guidelines - August 2020 17
DSC Assessment Moderation Checklist
Phase 4: Externally validating processes and judgements
� Did you benchmark assessment through comparison of assessment tasks, marking
criteria and results with a comparable course in another program at RMIT?
� Did you benchmark assessment through comparison of assessment tasks, marking
criteria and results with a comparable course at another university?
Phase 5: Monitoring and evaluation of assessment efficacy
� Did you refer to grade distribution in evaluating the efficacy of assessment in your
course?
� Did you ask your teaching team for their views about the efficacy of assessment, and
any related information such as a marking guide or marking rubric, in your course?
� Did you evaluate the efficacy of assessment with reference to students’ work?
� Did you refer to student feedback or evaluation in evaluating the efficacy of
assessment in your course?
� Did you identify any adjustments or improvements that need to be made to curriculum,
pedagogy, assessment criteria or assessment design in future offerings of the course?
DSCAssessment_Moderation_Checklist.docx
Author: DSC ADG
Save Date: 29/07/2020
College of Design and Social Context Page 2 of 2You can also read
