TRANSFORMINGEXAMS.COM - Robust online authentic e-Exams Mathew Hillier, Monash University - ATEM
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
TRANSFORMINGEXAMS.COM
A Scalable Examination Platform for BYOD Invigilated Assessment
Robust online
authentic e-Exams
Mathew Hillier,
Monash University
Andrew Fluck,
University of Tasmania
A key motivation - The gap
Real world of work Exams
World Economic Forum – How will digital change your working world. Exams at Monash Caufield in 2015 (mathew.hilier[at]monash.edu)
https://agenda.weforum.org/wp-content/uploads/rtr2m8vm1-628x330.jpg 70,000 student university.
We are faced with a growing disconnect between
the way high stakes testing is conducted using
pen on paper exams and students’ everyday
experiences of study, work and life.
2
What we are doing about it…
SAMR Theory - Puentedura, 2012
e-Exam project progress (now at phase 3)
Start > >>> >>> >>> >>> > Future >
Get Ready Phase 1 Phase 2 Phase 3 Phase 4 Phase 5
Paper Post-paper WhitelistedOpen but fully
Medium to
Institutional equivalent small to and logged logged
large scale.
approvals, small scale. medium. Internet Internet
research Basic doc Expanding the Adding the Network
ethics, Network
exams to app and media power of an mixed mode
hardware and BYOD exam.
begin! landscape. LMS. BYOD exam.
infrastructure
Crawling Walking Running Jumping Flying!
http://ta.vu/e-exam-roadmap
This session.
Moodle resistant to
Extension work: network outages!
An offline e-learning platform
see moleap.org
4
Pedagogical Affordances
Authentic assessments. It is a platform, not an app!
Exam doc Video
LMS quiz
Scratch SDK
Specialist applications
PDF
Sims
e-Exam trials
Recent e-Exams 2017 and 2018
Exams Typists Pen Weight Minutes
Monash • Business Statistics
Sum 1750 1309 ~ 4145 • Chinese online media (‘Robust’ online).
• Geography x 3 classes [Monash college 2016]
Mean 40 37 32% 106 • Globalisation x 4 classes [Monash college 2017]
• Language Translation (some NAATI)
Smallest 1 ~ 5% 15 • Introductory Chinese language (offline,
Spreadsheet used as a form)
Largest 166 ~ 50% 180
• Introduction to Chinese (‘Robust’ online).
Typists: UQ • French language translation
I would recommend the e-Exam system to others. CQU • Knowledge Management Principles (IT)
• Ethics and Social Issues (IT)
• IT management
UTAS • ICT in Education (post-paper exams)
• Environmental Chemistry
MqU • ICT in Education
UNSW • Air power (ADFA)
ECU • Teaching Introductory Computer Programming
• OHS for trades
UniSA • Science & Math for secondary teaching
Research Questions for Moodle Phase
Area Example research questions
Intention Were candidates in favour of the e-Exam system?
and attitude Would they recommend it to others or use it again?
Did they have any concerns about undertaking an e-exam?
If they typed their exam, was their attitude changed or any concerns
lessened following the event?
Ease of use Were the students able to use the system with relative ease?
Did they have any issues related to the e-exam processes?
Technical Did any technical issues or interruptions arise?
reliability If so, did such issues interfere with the exam or result in lost work?
Were students concerned about network outages?
Would the system allay any fears?
Would the system itself work when the network fails?
Network Trial Sem 1 2018
41 tertiary students.
Two units (subjects):
Intro to Chinese
(first year – 14 elected to type).
Chinese online media
(third year – all 27 typed).
Intro Online
Chinese Chinese Gender
language Media totals
Female 10 22 32
Male 4 5 9
Unit totals 14 27 41
Sem 2: eexam with 80 students
Caveat: Not random samples –
descriptive of these groups only.
Pre and post response trends (preliminary)
Strongly Disagree 11 22 33 44 55 Strongly Agree
Written instructions were easy to follow
It was easy to start my computer using the e-Exam USB stick
I can use the e-Exam system just as well as my own laptop system
N
It was easy to use the office suite (word processor/spread sheet)
It was easy to use software applications beyond the word processor
It was easy to save my response files into the correct place Pre 38
It was easy to answer multiple-choice questions in the e-Exam system
Overall, I feel the e-Exam System is easy to use Post 1 37
I feel the e-Exam System is reliable against technical failures
I feel the e-Exam System is secure against cheating Post 2 39
I now feel relaxed about using the e-Exam system for my exam
I would recommend the e-Exam System to others Post 3 40
My laptop is reliable for use in a computerised exam
My typing skills are fast enough for a computerised exam
Computerised exams make me more stressed than handwritten exams
I would like to use a computer for exams in the future
I am concerned about network outages impact ing my exam
I am reassured the e-Exam system was robust against network outages
The included software was useful [e.g DimSum]
Moodle worked well as an exam environment
Caveat: Not random samples – descriptive of these groups only.
e-Exam Process 1) Two weeks prior: practice session + pre-survey. 2) Exam day: In-class, graded, supervised assessment task + post survey.
e-Exam process: robust online Moodle (Phase 3)
Pre-exam: prepare learning materials Pre-session: Exam content
2. Configure, load and test: 4. Student laptop resides on a server.
Moodle quiz + Gateway USB setup &
practice.
5. Network setup
1. Teacher creates exam: 3. Deploy quiz to Moodle. and USBs to
venue
Network Moodle.
Moodle quiz, media, selects apps. Gateway USBs duplicated.
6. Exam venue: Linux Live USB
a. Students enter room SEB + e-tools:
b. Given USB (s.o.s. WIFI dongle) Libre Office, apps,
c. Start laptop from USB SDK, sims, PDFs,
& connect to Moodle with key large media.
d. Do exam in Moodle Local cache of Moodle
e. Finalise and shutdown
f. Return USB & dongle content. Response
g. Leave room backup to USB in case
of network outage.
8B In case of network Post-exam: assessment Recycle USB & dongle for
outage: Get backup 9. e-responses 10. next exam. Gateway
responses from USBs. available to Results USBs can be reused as-is
7. Responses
teacher. submission, (from step 5) or updated
autosave to 8 Responses
Auto and analytics & (step 2).
finalised to server manual reporting
server
marking.
(each 1 min) (USB resets *)
11. e-Feedback via network * USB auto reset still under
development.Details of e-Exam Room Set-up and Procedure
Room set up:
Paper 'e-Exam quick start guide' and post-exam survey.
A power socket was provided for each typist.
Hand-writers were given a paper copy of the exam questions and response booklets. These were
available to typists upon request.
Both typists and hand-writers sat in the same room.
1) Students enter the room and seated.
2) Typists were given an e-Exam USB stick containing the questions.
3) Students start their computer with a USB stick progressing to the e-Exam desktop. A desktop
background image provides a visual check that all have booted from the correct USB.
4) Students enter their student ID and name into the Wi-Fi connect screen.
5) Once connected the system opens the browser to the Moodle logon page.
6) They login with credentials. Then press start Quiz now button.
7) A password box appears. All students now wait.
8) Invigilator announces the start of the exam giving out the password. Students enter password and
begin.
9) Auto-save occurs every 1 minute with fall back to local USB storage.
10) At the end of the exam, the student presses the submit all and finish’ button. When done they
shut down the computer.
11) Students return the USB sticks.
12) Students complete the post-exam survey before leaving the room.Start up
Select LOTE
(if applicable)
Connect to WiFi
Eduroam networkStarting the exam at the same time - password
Log into
Moodle
server
Password given
out in class at the
exam start time.Third party software included.
This is an offline
dictionary tool
Moodle ‘Dim Sum’
questions in
Safe Exam
BrowserQuestions
Lecture test 1
A range of form based question
types were used in these e-exams.Questions
Listening test 1 Audio data files cached at the
start of the exam.
Students used headsets to listen.Questions
Constructed 2. Software
application used
enquiry to interrogate
and construct a
response.
1. Download file
3. Respond via formQuestions
1. Open software
Constructed
response
(file upload)
2. Use software
application to
construct a
response.
3. Respond by file uploadMulti layered backup - Network Outage
I
a) Whilst there is a
network connection
student data is saved to
the server each minute.
b) If there is a network outage, then
student response data is saved to the
USB drive in an encrypted file.Multi layered backup - Network Outage at Submission
I
• If at submission time there is a network outage,
then student response data is saved to the USB
drive.
• Ideally the student should try again or call
an exam supervisor for help.
• If the connection cannot be reinstated then
after the exam the response file is retrieved
from the USB and uploaded to Moodle.Successful Submission
Immediate feedback
(optional setting)
Confirmed submission
to server triggers USB
self clean up
ready for
next exam.Summary - Key Findings a) The e-Exam system was rated well by the typists: 4+ out of 5. b) Robust network features worked (at least one obvious wifi outage) Responses were auto saved to USB, retrieved following exam and re-joined the e-workflow in Moodle. Next steps i) Trials of using third party software tools via Moodle: For example: use a spread sheet in conjunction with Moodle question. For example: use constructed responses in third party software with file upload response via Moodle quiz. ii) Add/refine features. Better USB clean up triggering, add digital highlighter pen for Moodle.
Thank you For further information contact: Mathew.Hillier@monash.edu TransformingExams.com eExam Roadshow: Aug-Oct 2018, ADL, PER, SYD, BNE, CBR, HBA ADL Friday 31 Oct @ UniSA city west. Briefing session: online attendance option available. + Workshop: full. Other cities still available! http://TransformingExams.eventbrite.com.au eExam Symposium: 24 Nov 2018, Melbourne. Full day mini conf. Two International keynotes. http://eExamSymposium.eventbrite.com.au
You can also read
