Educational Technology - ERCIM NEWS
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
ercim-news.ercim.eu Number 120 January 2020
ERCIM NEWS
Special theme:
Educational
Technology
Joint
Editorial Information SPECIAL THEME
ERCIM News is the magazine of ERCIM. Published quarterly, it reports The special theme “Educational Technology” has been
on joint actions of the ERCIM partners, and aims to reflect the contribu- coordinated by Vassilis Katsouros (Athena Research and
tion made by ERCIM to the European Community in Information Innovation Centre) and Martin Hachet (Inria)
Technology and Applied Mathematics. Through short articles and news
items, it provides a forum for the exchange of information between the Introduction to the special theme
institutes and also with the wider scientific community. This issue has a 4 Educational Technology
circulation of about 6,000 printed copies and is also available online. by Vassilis Katsouros (Athena Research and Innovation
Centre) and Martin Hachet (Inria)
ERCIM News is published by ERCIM EEIG
BP 93, F-06902 Sophia Antipolis Cedex, France 6 Α Logic-Based Affective Tutoring System
+33 4 9238 5010, contact@ercim.eu by Achilles Dougalis and Dimitris Plexousakis (ICS-
Director: Philipp Hoschka, ISSN 0926-4981 FORTH)
Contributions 7 Learning Introductory Programming with Smart
Contributions should be submitted to the local editor of your country Learning Environment
by Boban Vesin (University of South-Eastern Norway),
Copyrightnotice Katerina Mangaroska and Michail Giannakos (NTNU)
All authors, as identified in each article, retain copyright of their work.
ERCIM News is licensed under a Creative Commons Attribution 4.0 9 Interoperable Education Infrastructures: A
International License (CC-BY). Middleware that Brings Together Adaptive, Social
and Virtual Learning Technologies
Advertising by Christopher Krauss and Manfred Hauswirth
For current advertising rates and conditions, see (Fraunhofer FOKUS)
https://ercim-news.ercim.eu/ or contact peter.kunz@ercim.eu
10 Fraunhofer IAIS IoT Programming Language
ERCIMNewsonlineedition:ercim-news.ercim.eu/ NEPO® in the Open Roberta® Lab
by Thorsten Leimbach (Fraunhofer IAIS), Daria Tomala
Nextissue: (Fraunhofer IAIS)
April 2020: The Climate Challenge
12 Modelling Student Learning and Forgetting for
Subscription Optimally Scheduling Skill Review
Subscribe to ERCIM News by sending an email to by Benoît Choffin (LRI), Fabrice Popineau (LRI) and
en-subscriptions@ercim.eu Yolaine Bourda (LRI)
EditorialBoard: 13 Ontology-based Learning Analytics in Medicine
Central editor: by Fabrice Jouanot (Univ. Grenoble Alpes), Olivier
Peter Kunz, ERCIM office (peter.kunz@ercim.eu) Palombi (Univ. Grenoble Alpes, CHU de Grenoble) and
Marie-Christine Rousset (Univ. Grenoble Alpes, IUF)
Local Editors:
Austria: Erwin Schoitsch (erwin.schoitsch@ait.ac.at) 15 Blockchain for Education: Lifelong Learning
Cyprus: Georgia Kapitsaki (gkapi@cs.ucy.ac.cy Passport
France: Christine Azevedo Coste (christine.azevedo@inria.fr) by Wolfgang Prinz, Sabine Kolvenbach and Rudolf
Germany: Alexander Nouak (alexander.nouak@iuk.fraunhofer.de) Ruland (Fraunhofer FIT)
Greece: Lida Harami (lida@ics.forth.gr),
Athanasios Kalogeras (kalogeras@isi.gr) 16 An Educational Platform for Logic-based Reasoning
Hungary: Andras Benczur (benczur@info.ilab.sztaki.hu) by Dimitrios Arampatzis (FORTH-ICS), Maria
Italy: Maurice ter Beek (maurice.terbeek@isti.cnr.it) Doulgeraki (FORTH-ICS), Michail Giannoulis (Univ. of
Luxembourg: Thomas Tamisier (thomas.tamisier@list.lu) Crete), Evropi Stefanidi (FORTH-ICS), and Theodore
Norway: Monica Divitini, (divitini@ntnu.no), Patkos (FORTH-ICS)
Are Magnus Bruaset (arem@simula.no)
Poland: Hung Son Nguyen (son@mimuw.edu.pl) 17 Authoring Game-Based Learning Activities that are
Portugal: José Borbinha (jlb@ist.utl.pt) Manageable by Teachers
Sweden: Maria Rudenschöld (maria.rudenschold@ri.se) by Pedro Cardoso (FEUP/ FBAUP and INESC TEC),
Switzerland: Harry Rudin (hrudin@smile.ch) Leonel Morgado (Universidade Aberta and INESC
The Netherlands: Annette Kik (Annette.Kik@cwi.nl) TEC), and António Coelho (FEUP and INESC TEC)
W3C: Marie-Claire Forgue (mcf@w3.org)
19 LudiMoodle: Adaptive Gamification to Improve
Coverillustrtation: source: Shutterstock. Learner Motivation
by Élise Lavoué (Université Jean Moulin Lyon 3)
ERCIM NEWS 120 January 2020
CONTENTS
20 Can Tangible Robots Support Children in Learning 36 When a Master of Sciences on EdTech becomes an
Handwriting? International Community
by Arzu Guneysu Ozgur, Barbara Bruno, Thibault by Margarida Romero, Saint-Clair Lefèvre (UCA,
Asselborn and Pierre Dillenbourg (EPFL) INSPE, LINE) and Thierry Viéville (Inria)
22 “You Tell, I Do, and We Swap until we Connect All 38 DE-TEL - A European initiative for Doctoral
the Gold Mines!” Education in Technology-Enhanced Learning
by Jauwairia Nasir, Utku Norman, Barbara Bruno and by Mikhail Fominykh and Ekaterina Prasolova-Førland
Pierre Dillenbourg (EPFL) (NTNU)
23 Gestures in Tangible User Interfaces 39 Ethical Teaching Analytics in a Context-Aware
by Dimitra Anastasiou and Eric Ras (Luxembourg Classroom: A Manifesto
Institute of Science and Technology) by Romain Laurent(Univ. Grenoble Alpes/LaRAC),
Dominique Vaufreydaz (Univ. Grenoble Alpes, CNRS,
25 Kniwwelino: A Microcontroller-based Platform Inria, Grenoble INP, LIG), and Philippe Dessus (Univ.
Introducing Children to Programing and Electronics Grenoble Alpes/LaRAC)
by Valérie Maquil and Christian Moll (Luxembourg
Institute of Science and Technology)
RESEARCH ANd INNOvATION
26 Virtual Simulation Environment for Medical
Training 41 KANDINSKY Patterns: A Swiss-Knife for the Study
by Thomas Luiz, Dieter Lerner, Dominik Schnier of Explainable AI
(Fraunhofer IESE) by Andreas Holzinger (Medical University Graz and
Alberta Machine Intelligence Institute Edmonton,
27 Mixed Reality and Wearables in Industrial Training Canada), Peter Kieseberg (University of Applied
by Ralf Klamma (RWTH Aachen University), István Sciences, St.Poelten) and Heimo Mülller (Medical
Koren (RWTH Aachen University) and Matthias Jarke University Graz)
(Fraunhofer FIT and RWTH Aachen University)
43 Building Reliable Storage Systems
29 Personalized Interactive Edutainment in Extended by Ilias Iliadis (IBM Research – Zurich Laboratory)
Reality (XR) Laboratories
by Aris S. Lalos (ISI, ATHENA R.C.), Chairi Kiourt 45 Browse, Visualise, Analyse EU Procurement Data
(ILSP, ATHENA R.C.), Dimitrios Kalles (Hellenic Open by Ahmet Soylu and Till C. Lech (SINTEF)
University) and Athanasios Kalogeras (ISI, ATHENA
R.C.) 46 More on 5G: Millimetre-Waves
by Gregor Dürrenberger (FSM) and Harry Rudin
30 Music in Education through Technology
by Maximos Kaliakatsos-Papakostas, Kosmas Kritsis,
Vassilis Katsouros (Institute for Language and Speech ANNOuNCEMENTS
Processing, Athena Research and Innovation Centre)
48 FMICS 2020: 25th International Conference on Formal
32 Dance Education and Digital Technologies Methods for Industrial Critical Systems
by Katerina El Raheb and Yannis Ioannidis (“Athena”
Research and Innovation Center and National and 48 ERCIM Membership
Kapodistrian University of Athens)
49 ERCIM “Alain Bensoussan” Fellowship Programme
33 Intelligent Classroom: Materialising the Vision of
Ambient Intelligence for Education 50 Dagstuhl Seminars and Perspectives Workshops
by Asterios Leonidis, Maria Korozi, Margherita Antona
and Constantine Stephanidis (FORTH-ICS) 50 Tim Berners-Lee Launches Global Action Plan to Pre-
vent "Digital Dystopia"
35 Innovative Monitoring of Learning Habits and
Motivation in Undergraduate Mathematics 50 HORIZON 2020 Project Management
Education
by Brigitta Szilágyi (Budapest University of Technology 51 W3Cx Introduction to Web Accessibility – New Online
and Economics), Szabolcs Berezvai (Budapest Course
University of Technology and Economics) and Daniel
Horvath (EduBase Online Ltd.) 51 HAL is Opening Up to Software
51 David Chaum and Guido van Rossum awarded Dijkstra
Fellowship
ERCIM NEWS 120 January 2020 3
Special Theme: Educational Technology
Introduction to the Special Theme
Educational Technology
by Vassilis Katsouros (Athena Research and Innovation Centre) and Martin Hachet (Inria)
This special theme addresses the state of the art in Krauss and Hauswirth (p. 9) facilitates content
educational technologies, “EdTech”, illustrating from different learning management systems and
the range of scientific fields and challenges faced enriches it with innovative technologies such as
by the research community when it comes to inte- gamification, social learning, virtual reality and
grating tools and systems that apply to real-life learning recommenders. Leimbach and Tomala (p.
learning situations. 10) propose an integrated programming environ-
ment combined with a graphical programming lan-
Education, either formal or informal, is a key guage that allows learners to code a wide range of
driver for the future of our societies; it fosters per- robotic systems, promoting STEM education to
sonal fulfilment and development, social inclusion young children.
and active citizenship, as well as generating inno-
vation and economic activities. Right now, digital Choffin, Popineau and Bourda (p. 12) suggest
tools are opening up promising new opportunities modelling student learning and forgetting for opti-
to take education to the next level. mally scheduling distributed practice of skills, and
Jouanot , Palombi and Rousset (p. 13) propose an
Digital tools in education go far beyond the intro- ontology-based method for making learning ana-
duction of computers to schools. The introduction lytics transparent and explainable using a query
of computers to children can even have counter- language that allows users to express specific
productive effects on their ability to acquire needs of data exploration and analysis. As certifi-
knowledge and skills. It is vital that we take a cates play an important role in education, and indi-
holistic approach to educational technologies, vidual learning records become essential for
where the learner and the teacher stay at the centre people’s professional careers, the blockchain for
of the loop. education platform (Prinz, Kolvenbach and
Ruland, p. 15) introduces a secure and intuitive
As a consequence, modern educational technolo- solution for issuing, sharing, and validating educa-
gies are emerging from multidisciplinary research, tional certificates.
building notably on advances in pedagogical and
learning theories, educational psychology, interac- Motivation is an essential lever for engaging stu-
tion technologies and artificial intelligence. This dents in learning processes. Hence, several
special issue illustrates the richness of the current authors have explored how motivation can be
research in EdTech, where human factors, software maintained and enhanced through gamification
and hardware technologies, and even organisa- mechanisms. This is notably the case for EduBAI
tional and ethical considerations all contribute (Arampatzis et al., p. 16) whose goal is to help
towards building tomorrow's education, either in build reasoning by leaning on a basketball game
formal, informal, or professional learning contexts. rationale. In the BEACONING project presented
by Cardoso, Morgado and Coelho (p. 17), gamifi-
Intelligent tutoring systems that adapt learning cation is used to make lesson plans more fluid.
content to the individual’s progress are becoming The LudiMoodle project, described by Lavoué (p.
more sophisticated with the ability to consider the 19), similarly evaluates the impact of gamification
learner’s emotions and learning style, as illustrated on motivation.
by Dougalis and Plexousakis (p. 6). Vesin,
Mangaroska and Giannakos (p. 7) present a per- Motivation and engagement can also be encour-
sonalised and adaptive tutoring system covering a aged by physically involving the learner in the
complex interplay of content, tasks, instructions, task. Tangible, hands-on activities can help build
social dynamics and learning analytics to teach knowledge, and enhance collaborative work and
introductory programming to university students. social interaction. Hence, compared with purely
A common learning middleware in the article by digital approaches, hybrid approaches that mix
4 ERCIM NEWS 120 January 2020
physical and digital components have great poten- p. 35) is a learning-management system (LMS)
tial. The CHILI Lab at EPFL explores approaches that provides numerous teaching and testing inter-
based on robots to support handwriting (Ozgur et faces. Romero, Lefèvre and Viéville (p. 36) present
al., p. 20) or to accompany a collaborative the SmartEdTech Master program. It is dedicated
problem-solving task (Nasir et al. p.22). Problem- to the teaching of, and with, educational tech-
solving is also the topic of the article by Anastasiou nology. Similarly, at a doctoral level, the DE-TEL
and Ras (p. 23) where tangible user interfaces are program (Fominykh and Prasolova-Førland, p. 38)
combined with 2D and 3D gestures. In is a European initiative dedicated to a new form of
Kniwwelino (Maquil and Moll, p. 25), the hands- technology-enhanced learning. For all these
on activities are supported by a combination of a research works and initiatives in EdTech, ethics
visual programming language and a physical should remain a central pillar of the upcoming gen-
microcontroller equipped with sensors and dis- eration of teaching tools and approaches. This is
plays. the focus of a manifesto presented by Laurent,
Vaufreydaz and Dessus (p. 39).
In the EPICSAVE project (Luiz, Lerner and
Schnier, p. 26), an immersive room-scaled multi- The articles in this special theme give a broad
user 3D virtual simulation environment for med- overview of the current state of research and appli-
ical training scenarios enabling realistic and sus- cations and insight into ongoing projects in the
tainable training experiences is presented. The educational technologies field.
WEKIT project (Klamma, Koren and Jarke, p. 27)
has developed an industrial training platform com- Digital enhanced learning is also addressed by the
bining sensor technologies with mixed and aug- recent DEL4ALL (Digital enhanced learning for
mented reality for on-the-job training. Virtual, aug- All) project [L1], funded by the European Union.
mented and mixed reality is also used in the project With a forward-looking perspective, it analyses
described by Lalos et al. (p. 29) to provide an edu- best-practice, success stories as well as challenges
cational platform for virtual scientific laboratories and opportunities offered by the increasing adop-
for STEM education. Kaliakatsos-Papakostas, tion of technologies, such as blockchain, artificial
Kritsis and Katsouros (p. 30) introduce iMusciCA, intelligence and others. DEL4All is expected to
a web-based workbench that integrates advanced provide a basis for future research directions and
core enabling technologies, including 3D design policy recommendations to enable the transition
and printing of musical instruments, body tracking from Horizon 2020 to Horizon Europe funded
sensors for gesture recognition, interactive pens research and innovation projects in the area of dig-
and tablets as well as sound generation and pro- ital enhanced learning.
cessing tools for STEA(rts)M education. The
WhoLoDancE project (El Raheb and Ioannidis p. Link:
32) has developed web-based tools for the [L1] https://www.del4all.eu/
analysis, segmentation, annotation and blending of
dance movements as well as interactive experi- Please contact:
ences that integrate augmented and mixed reality, Vassilis Katsouros
sonification of movement and visualisation of the Athena Research and Innovation Centre, Greece
human body and movement in different avatars vsk@athenarc.gr
and environments. Leonidis et al. (p. 33), present a
student-oriented and educator-friendly intelligent Martin Hachet
classroom that integrates several features, such as Inria, France
synchronous or event-based communication, iden- martin.hachet@inria.fr
tification of learners’ behaviour, etc. and attractive,
situational environments for learning.
In addition to these research activities and techno-
logical developments, various initiatives aim at
organising and teaching EdTech. For example, the
Edubase Online Platform (Szilágyi and Berezvai,
ERCIM NEWS 120 January 2020 5
Special Theme: Educational Technology
Α Logic-Based Affective Tutoring System
by Achilles Dougalis and Dimitris Plexousakis (ICS-FORTH)
In science fiction, artificial agents are portrayed as being capable of interacting with and helping
humans. This aid could take the form of holding intelligent conversations and even acting as
teachers and coaches. Some progress has been made in this direction in real life. Indeed, systems
utilising intelligent agents, such as duolingo™, have proven capable of acting as personal tutors.
These “intelligent tutoring systems” (ITS) emulate a human tutor by using AI techniques to adapt
instructions and teaching according to each individual learner’s background and progress but also
guide the learner through an exercise by providing hints and feedback.
These systems are results of a combina- any preferences of the user into they are mainly concerned about the
tion of multiple disciplines, such as account and none of them have been course’s presentation.
computer science, cognitive psychology, used for affective input. In order to
human-robot interaction and educational tackle the user’s preference problem, In order to address these limitations of
research. Some advantages of ITS are researchers have created the field of both fields, we have built AFFLOG, an
that they are location-independent, “adaptive learning systems”. These adaptive cognitive affective tutoring
easily accessible, and offer great flexi- systems use machine learning tech- system that uses answer set program-
bility, allowing students to learn at their niques in order to adapt the content of a ming and the event calculus action lan-
own pace and not to have to rely on rigid given course according to the prefer- guage [2] in order to represent the var-
classroom schedules. ences of their current user. However, ious components and actions of an ATS,
these systems do not offer feedback to and perform reasoning tasks such as
Researchers have found that a learning the user (emotional or cognitive) as planning for creating a course suitable
session can be improved if the teacher is
empathetic to the emotions of the
learner. Such improvements could take
the form of an ITS offering help when it
detects that the learner is confused, or by
offering the learner motivating remarks
when it detects boredom. ITS systems
that make use of emotions are known as
affective tutoring systems (ATS). ATS
combine tutoring strategies and emotion
sensing techniques into a single system.
Also, evaluations have shown that they
can contribute positively to the user’s Figure1:Acourseasadirectedgraph.Eachsubchapterisdepictedasanode,withoneor
learning experience [1]. moretutorialsrepresentedasvectorslinkingitwithothernodes.Achaptercanbetraversed
eitherbyonetutorial(e.g.tutorial8forchapter1)orbyaselectionofsmallertutorials.
A disadvantage of these systems is that
they are designed for teaching a specific
subject to specific users, making
reusability difficult or even impossible.
Moreover, the more diverse and compli-
cated a course is, the more difficult it is
to manage it. In other words, there is a
need for a universal, well-defined struc-
ture in order to facilitate the design of
ITS. Fortunately, similar problems in
different domains have been dealt with
successfully using a knowledge repre-
sentation and reasoning approach. Such
approaches use declarative logic and
logical formalisms known as action lan-
guages in order to formalise the problem
and use AI techniques, such as projec-
tion and planning, in order to solve it.
ITS that make use of this approach
already exist and are known as cognitive
tutoring systems. However, these sys-
tems are always modelled around a spe-
cific course or problem, they don’t take Figure2:EmotionalStrategiesaccordingtotheuser’scurrentemotion.
6 ERCIM NEWS 120 January 2020
to the current user, as well as offering rent user according to his/her learning ning” by replacing the wrong tutorials
the tools to create a new course from style. A “test” is the method the system with unused tutorials for that chapter.
scratch. The main contributions of this uses in order to understand whether the When all the chapters are completed and
work are: user understood a part of the course. all the tutorials are correct, the course has
• Course representation: We represent been completed successfully.
a course and its chapters in such a A “course” is the material that the tutor
way that it can be effectively used by uses to teach in one or more “sessions”. The work here represents a part of a
our system while also providing the It is essentially a collection of tutorials PhD in emotional adaptive tutoring sys-
author of the course with flexibility in and tests that are presented to the user. tems. Future work includes system
terms of which modalities to use and Each course consists of a number of evaluations, linking the system with the
also the design, structure and size of “chapters” and each chapter consists of a semantic web via the RDF language,
the course. number of “subchapters”. Subchapters and using machine learning methods to
• User representation: Which parts of a are traversed using tutorials and can be offer more personalised tutoring.
specific course the user has learned, described as nodes in a graph where the
his/her preferred learning styles, as tutorials act as the vectors (Figure 2). References:
well as his/her emotional state Once the tutorials that comprise a [1] B. Kort, R. Reilly, R. W. Picard:
regarding a course or a part of it. chapter have been presented to the user, “An affective model of interplay
• The integration of emotions and the the tutor presents him/her with a test to between emotions and learning:
user’s learning style in the tutor’s determine whether the user understood Reengineering educational
decision making process (Figure 2). the chapter. If the user passes the test, the pedagogy-building a learning
tutor assumes that the user understood companion, 2001.”
We use the term “tutorial” to describe the chapter and proceeds to the next [2] M. Sergot, R. Kowalski: “A logic-
the main building block for a course. A chapter. If not, the tutor assumes that the based calculus of events”, New
tutorial can be plain text, a picture, an user has not understood one or more of Generation Computing 4.1 (1985):
audio file or a video file. Important the tutorials of the particular chapter, but 67-95.
semantic properties of a tutorial written since it cannot differentiate between
in ASP include the subchapters that which tutorial was misunderstood and Please contact:
denote its beginning and end, as well as which not, all the tutorials of that chapter Achilles Dougalis
its modality, relative difficulty, its dura- are labelled as wrong. Consequently, the FORTH-ICS, Greece
tion, and how suitable it is for the cur- course must be modified using “plan- achil@ics.forth.gr
Learning Introductory Programming
with Smart Learning Environment
by Boban Vesin (University of South-Eastern Norway), Katerina Mangaroska and Michail Giannakos (NTNU)
Programming Tutoring System (ProTuS) is an adaptive learning system developed to support introductory
programming. ProTuS utilises a cross-platform architecture that aggregates and harmonises learner
analytics coming from different systems and quantifies learners’ performance through a set of indicators.
ProTuS has been successfully used within universities to support teaching and learning.
Programming tutoring system The current version of ProTuS includes are aligned with the curriculum pre-
ProTuS [L1] was developed in the interactive learning resources from var- sented in the introduction to Java
Learner-Computer Interaction lab [L2] ious content providers, developed at course. The objective behind the
at the Norwegian University of Science several universities from Norway, USA reading content is to allow students to
and Technology (NTNU), in coopera- and Canada. In addition, the system leverage on their background knowl-
tion with the University of South- offers adaptive features based on con- edge on procedural programming (nor-
Eastern Norway (USN). The system rep- tent recommendation, automatic assess- mally taught via Python) and progress
resents an effort to develop an adaptive ment and grading. with object-oriented programming
and interactive environment that pro- (OOP). To do so, the content provides a
vides personalised learning to students Interactive learning content comparison of the syntax and the basic
(Figure 1) [1]. ProTuS covers a complex The interactive learning content (i.e., concepts in Python (procedural) and
interplay of learning resources, tasks, resources) included in the system com- Java (OOP).
instructions, social dynamics, interac- prises four types of activity that support
tions, and learning analytics aimed at individual work. Students can practice Interactive examples. For each of the
helping students to learn introductory and learn programming through the fol- proposed topics, the students have the
programming [2]. The system has been lowing learning resources [2]: chance to explore different examples,
utilised at several universities for over a Explanations. ProTuS contains reading called Program Construction EXamples
decade [3]. content (i.e., tutorials) on 15 topics that (PCEX). Each PCEX content starts with
ERCIM NEWS 120 January 2020 7
Special Theme: Educational Technology
Figure1:
Personalisedlearning
inProTuS.
a worked-out example with explanation tems into a set of meaningful indicators. Links:
of how to write a code for a particular Thus, it acts as a learning eco-system [L1] https://protus.idi.ntnu.no/
problem. The explanations (that are that leverages on various learning ana- [L2] https://lci.idi.ntnu.no/
hidden until a student clicks on the lines lytics to enhance personalised and adap-
of interest) are available for almost all tive learning. To address barriers hin- References:
lines in the example and they focus on dering the usefulness and efficiency of [1] B. Vesin, K. Mangaroska, M.
why students need to write a code in a an adaptive learning system, we exam- Giannakos: “Learning in smart
certain way or why certain program- ined ProTuS usability and evaluated its environments: user-centered design
ming constructs are used in the code. cross-platform learning analytics capac- and analytics of an adaptive
ities to support personalised and adap- learning system,” Smart Learn.
Interactive challenges. To allow a tive learning [1]. The system has been Environ.
seamless connection between theory tested in different universities [2] K. Mangaroska, B. Vesin, and M.
and practice, as well as give students the throughout Norway and other European Giannakos, “Cross-platform
possibility to try out the given exam- countries, with the results indicating analytics: A step towards
ples, we present a challenge after each that it shows promise for supporting personalization and adaptation in
example. This allows students to solve introductory programming. education,” LAK
one or more challenges related to the [3] A. Klašnja-Milićević, B. Vesin, and
example that they previously viewed. In the future, the authors plan to further M. Ivanović, “Social tagging
develop the learning analytics compo- strategy for enhancing e-learning
Coding exercises. This is a type of smart nent of ProTuS and provide more experience,” C&E
content that requires students to write learner-centred visualisations and
code or complete a given code to cross-analytics (e.g., data coming from Please contact:
achieve a certain goal. Each coding assignments, project-work, course Boban Vesin
exercise has a problem description and grading, and third-party software) [2]. University of South-Eastern Norway,
a baseline code. When students submit Furthermore, the overall goal is to Norway
their code, it is tested against a set of investigate how analytics derived from boban.vesin@usn.no
predefined unit tests and the student various sources can help us to construct
receives automated feedback on efficient teaching strategies and support
whether the tests were passed or not. online and blended teaching and
learning of introductory programming.
Interactive examples and challenges
(Mastery Grids, PSEX) were developed This project is a product of a research
at the University of Pittsburgh, and collaboration between NTNU and USN,
coding exercises (PCRS) at the receiving funding from Norwegian
University of Toronto [2]. Agency for International Cooperation
and Quality Enhancement in Higher
Scaling up research efforts to Education (DIG-P44/2019), the
support introductory programming Norwegian Research Council
As a cross-platform architecture, (255129/H20) and NTNU’s Excellent
ProTuS aggregates and harmonises Education program (NTNU
learner-generated data from several sys- Toppundervisning).
8 ERCIM NEWS 120 January 2020
Interoperable Education Infrastructures:
A Middleware that Brings Together Adaptive,
Social and virtual Learning Technologies
by Christopher Krauss and Manfred Hauswirth (Fraunhofer FOKUS)
What should a course provider do if all course content, which is stored in Moodle, needs to be migrated
to a new learning management system? How could a provider easily use advanced technologies like
learning analytics, learning recommender systems or virtual learning to create a compelling learning
experience? How can a provider incorporate the content of another provider into an existing course? To
address such questions, we developed the Common Learning Middleware in a joint project with several
Fraunhofer institutes trying to solve these typical challenges facing educational institutions.
A wide range of technological compo- for content structures [L3], learning narios, which were realised on the basis
nents support or facilitate many suc- objects [L4], and quizzes [L5], and of individual solutions.
cessful approaches in the field of educa- standards for persisting activity data
tion, such as blended learning or flipped [L6]. Figure 1 shows the overall archi- Together with the institutes Fraunhofer
classroom learning. Learning analytics tecture of the Common Learning FIT and Fraunhofer IML, we have cre-
and educational recommender systems Middleware. In the underlying concep- ated a learning offer for the field of data
are based on statistical models and artifi- tual design, every element that can be science in which the learning content
cial intelligence; gamification and integrated into the learning context, be from the databases of several ILIAS
social- and peer-learning require appro- it a text, image, video, dashboard or vir- platforms is presented in a self-devel-
priate backend services; learning con- tual reality, is abstracted as a tool. The oped learning portal from an external
tent and media of these systems are typi- different servers act as tool providers service provider. The portal also offers
cally hosted on different servers; and and can publish and subscribe their programming exercises (as individual
virtual and augmented reality used in offers to the middleware. From there, Jupyter notebooks) after each major
educational systems require specialised the user interfaces, such as traditional learning unit. Course participants can
hardware or software. Many promising learning management systems (e.g., communicate via a tool provided by a
approaches are being developed as iso- Moodle or ILIAS) or advanced learning start-up´s innovative social learning
lated solutions, which individually are applications, can access the tools platform. In another case, together with
quite successful, but would only reach through standardised interfaces. The Fraunhofer IOSB and Fraunhofer FIT,
their full potential when used jointly. middleware verifies the roles and rights we combined various tools for the cyber
Together with the Fraunhofer Academy, of the requesting users via its own user security domain. It merges online
Fraunhofer FOKUS is coordinating a enrolment system for each access. In learning content from ILIAS and Open
research project that seeks to integrate addition, existing user management sys- edX learning platforms with interactive
isolated solutions with each other tems can be connected to the middle- learning applications developed by an
through a middleware component. ware to enable cross-system logins. external service provider into a single
This enables the most diverse learning offering. In addition, the learned theo-
The Common Learning Middleware scenarios, which can be very well tai- ries can be practically applied in the
[L1] is based on open standards and lored to the respective learning context, serious virtual reality game Lost Earth
specifications for educational technolo- without requiring programming skills 2307 [1], in which the learner has to
gies, including standardised interface of the content creator. To showcase our solve various security-relevant missions
definitions [L2], metadata specifications system, we give a few example sce- in a future scenario. The game is seam-
lessly loaded into the platform. The
middleware is also utilised for univer-
sity courses in computer science [2].
And in a parallel project, we even used
these interfaces to link six different edu-
cational institutes, which normally have
little contact, from the fields of crafts,
computer science and general school
education. This created synergies
between the institutes on the basis of
content and technology. On the one
hand, courses on bookkeeping and
accounting only had to be developed
once and could be utilised by all institu-
tions in their respective learning plat-
Figure1:SimplifiedarchitectureofalearninginfrastructurebasedontheCommonLearning forms. On the other hand, more
Middleware. advanced components, such as learning
ERCIM NEWS 120 January 2020 9
Special Theme: Educational Technology
analytics, gamification, learning paths system is the “forgetting effect”, which Links:
and a learning recommender system, provides information on whether the [L1] https://kwz.me/hKr
were realised as tools and offered for content was supposedly forgotten again [L2] https://kwz.me/hKY
the respective courses via the middle- based on the type of media, its scope [L3] https://www.imsglobal.org/cc/
ware. and complexity as well as the time [L4] https://www.imsglobal.org/metadata/
elapsed since the last learning. The [L5] https://www.imsglobal.org/question/
Our recommender system [3] is a spe- users of the platform then see general [L6] https://www.adlnet.gov/projects/xapi/
cial tool provider that plays a central and thematic recommendations in cate- [L7] https://akademie.fokus.fraunhofer.de/
role, which focuses on determining gories such as: “With these topics you
“learning needs”. These are numerical can prepare for the next lecture.”, “You References:
values that represent the relevance of haven't done so well in these exercises [1] A. Streicher, J. D. Smeddinck:
the corresponding content to each indi- yet.” or “You might have forgotten this “Personalized and adaptive serious
vidual course participant. The higher already.” games”, Entertainment Computing
the individual’s learning need for a and Serious Games. Springer,
topic, the more important it is that the The Common Learning Middleware 2016. 332-377.
learner should work on it. The learning makes it possible to combine the most [2] C. Krauss, et al.: “Teaching
platform loads the recommendation diverse educational technologies Advanced Web Technologies with
tool, which presents the content recom- without having to forego content pro- a Mobile Learning Companion
mendations for the most relevant topics tection and rights management. Content Application”, in: Proc. of mLearn
in order to make the learning process from different learning management 2017, ACM, , Larnaca, Cyprus.
more efficient and effective. The rec- systems, such as Moodle, ILIAS or [3] C. Krauss, A. Merceron, S.
ommender takes into account informa- open edX, can easily be merged via a Arbanowski: “Smart Learning
tion that concerns the learning content common interface definition and Object Recommendations based on
itself. This includes, for example, data enriched with innovative technologies, Time-Dependent Learning Need
on whether certain content is relevant such as learning analytics, gamification, Models”, in Proc. of EDM 2019,
for exams or whether a lecture is about social learning, virtual reality or M. Desmarais, et al. (eds.), pp. 599
to take place. At the same time, infor- learning recommender systems. Various - 602, July 2-5, 2019, Montréal,
mation collected through direct user institutions have already tested this Canada.
interaction is also included. For middleware and Fraunhofer is success-
example, whether the participant has fully using it in several of its courses, Please contact:
already edited all the underlying con- such as those of the Fraunhofer Christopher Krauss
tent, how the participant self-assesses in FOKUS-Akademie [L7]. In response to Fraunhofer FOKUS, Germany
the subject area, how he or she has per- high demand, the middleware is being christopher.krauss@fokus.fraunhofer.de
formed in exercises or how much of the opened up to external companies that
content has already been viewed with value the independence of certain solu- Manfred Hauswirth
the platform and for how long. A dis- tions. Fraunhofer FOKUS, Germany
tinctive feature of the recommender manfred.hauswirth@fokus.fraunhofer.de
Fraunhofer IAIS IoT Programming Language
NEPO® in the Open Roberta® Lab
by Thorsten Leimbach (Fraunhofer IAIS), Daria Tomala (Fraunhofer IAIS)
Technology now pervades all areas of our lives, including our home life, education and work. As
society becomes increasingly digitalised, digital skills such as “computational thinking” are becoming
more important – this applies to children in school, adults in the workforce and senior citizens alike.
At Fraunhofer Institute for Intelligent “Roberta” has reached over 450,000 ular, both initiated with the support of
Analysis and Information Systems IAIS, children in Germany. This is one of Google.org.
the business unit “Smart Coding and Europe’s largest STEM initiatives, its
Learning” is dedicated to the topic of expansion being due in part to the EU “Open Roberta Lab” is an integrated
conveying digital skills, such as coding, project “Roberta goes EU” [L4]. In programming environment (IDE) that
in a sustainable way, regardless of addition to the success of its hands-on is available online, free of charge [L2].
gender, age or prior knowledge. As part didactic concept, one key component of As a state-of-the-art and open source
of this, the educational program the project is the development and cloud programming technology (CPT),
“Roberta® – Learning with robots” [L1] implementation of new technology: the it enables the web-based programming
has been successfully accompanying open-source programming environment of hardware systems, such as robots
teachers as well as inspiring students in “Open Roberta Lab” and the visual pro- and microboards, with the program-
STEM subjects since 2002. To date gramming language NEPO® in partic- ming language NEPO on any computa-
10 ERCIM NEWS 120 January 2020Figure1:theOpenRobertaLab.OnthelefttheNEPOblocks,ontherightthecorrespondinggeneratedsourcecode.
tional device (e.g., smartphone, tablet, ming language made by Fraunhofer vidual NEPO coding blocks as well as
PC) and on standard operating systems IAIS. NEPO (New Easy Programming the ability to integrate short tutorials.
(e.g., Mac OS, Linux, Windows) in Online) reduces the complexity of text-
which conventional browsers run. This based programming languages, without NEPO allows anyone without previous
independence of computational devices being inferior in scope and perform- knowledge to code even sophisticated
and operating systems as well as the ance. Despite its block-based approach, robotic systems with various sensors
web-based approach itself (i.e., no NEPO is a full programming language and actuators in less than five minutes.
installation is needed), minimises the that allows the creation of programs So far, it has been successfully used by
technical obstacles faced by many containing any regular coding compo- a wide audience: from primary schools,
users, especially in educational and nents, such as commands, control struc- high-schools and universities, to coding
workplace environments, and supports tures, lists, variables and functions. hubs and computer clubs for the elderly
the adoption and potential purview of and even in industrial and workplace
the technology. The programming principle is easily training.
understood: The coding components in
The enthusiastic reception to Open the form of NEPO blocks are adjusted As mentioned above, an increasing
Roberta is reflected in the continuously via “drag and drop” to a pre-defined variety of robotic systems is program-
growing number of users worldwide starting point of the program and are mable with Open Roberta. What started
since its launch in 2014. In 2015 the themselves colour-coded to allow the as a platform to let users code one robot
Open Roberta Lab recorded 11,000 visi- recognition of semantics beforehand alone, has now advanced into a multi-
tors, while in 2017 the website was vis- and therefore to minimise errors. The system programming solution that lets
ited more than 107,000 times. By block-based structure enables syntactic users code a wide range of robotic sys-
October 2019 this number more than errors to be avoided altogether. tems intuitively. Starting with educa-
tripled to a total of 336,270 users from + Furthermore, the textual designation of tional robots and microcontrollers for
100 countries. the respective blocks makes it possible children aged 7+, humanoid robots,
to quickly understand the functionality physical computing platforms, smart
One key to the success of the platform is of the program created. The CPT frame- home devices or other Internet of
NEPO, the intuitive graphical program- work also provides online help to indi- Things (IoT) applications for the per-
sonal use, through to potential applica-
tions in the industrial sector.
Since NEPO blocks are specially
designed by Fraunhofer IAIS to fit any
chosen target system and can therefore
be adapted into a customised solution,
the areas of application are almost end-
less. On the website, the code genera-
tion – from NEPO Blocks to textual
programming languages, such as
Python and Java, C – runs completely
on Fraunhofer servers in Germany.
However, a copy of the open source
server itself can be ported to other sys-
Figure2:NEPOblockscanbeindividuallyadaptedtotherespectivehardwaresystem. tems, e.g., Raspberry Pi [L3].
ERCIM NEWS 120 January 2020 11Special Theme: Educational Technology
The functionalities of the Open Roberta integrated into classrooms – in the full Yet another One?", IEEE MTEL
Lab also include: availability in 17 lan- spectrum from primary schools to uni- workshop in conjunction with ISM
guages, a gallery to share programs versities. The project, based at 2014.
online, a virtual simulation and the Fraunhofer IAIS, is now aiming to [2] M. Ketterl, et al: “Open Roberta - a
ability to describe the programs as well extend its approach to one of the major Web Based Approach Visually
as to look into the textual source code research and impact fields of Fraunhofer Program Real Educational Robots”,
behind the NEPO blocks. This combi- IAIS, artificial intelligence (AI). LOM, ISSN: 1903-248X.
nation of features results in a low- [3] A. Bredenfeld, T. Leimbach, “The
threshold and sustainable introduction Links: Roberta® Initiative”, in: Proc. of
to coding. [L1] https://www.roberta-home.de/ SIMPAR 2010, Darmstadt
[L2] https://lab.open-roberta.org/ (Workshop Teaching robotics,
Open Roberta follows the well-proven [L3] https://kwz.me/hKB teaching with robotics), 2010, S. 1-2.
concept of “Roberta”, to make abstract [L4] https://kwz.me/hKb
constructs in coding tangible and easily Please contact:
understood by a wide audience. In com- References: Daria Tomala
bination with teacher trainings, teaching [1] B. Jost, et al.: “Graphical Fraunhofer IAIS, Germany
materials and a nationwide network, Programming Environments for annette.daria.tomala@iais.fraunhofer.de
Open Roberta is becoming increasingly Educational Robots: Open Roberta -
Modelling Student Learning and Forgetting
for Optimally Scheduling Skill Review
by Benoît Choffin (LRI), Fabrice Popineau (LRI) and Yolaine Bourda (LRI)
Current adaptive and personalised spacing algorithms can help improve students’ long-term memory
retention for simple pieces of knowledge, such as vocabulary in a foreign language. In real-world educational
settings, however, students often need to apply a set of underlying and abstract skills for a long period. At the
French Laboratoire de Recherche en Informatique (LRI), we developed a new student learning and forgetting
statistical model to build an adaptive and personalised skill practice scheduler for human learners.
Forgetting is a ubiquitous phenomenon These tools sequentially decide which example skill reviewing policy from an
of human cognition that not only pre- item (or question, exercise) to ask the adaptive spacing algorithm is given in
vents students from remembering what student at any time based on her past Figure 1.
they have learnt before but also hinders study and performance history. By
future learning, as the latter often builds focusing on weaker items, they show To address this issue, we chose to
on prior knowledge. Fortunately, cogni- substantial improvement of the learners’ follow a model-based approach: a stu-
tive scientists have uncovered simple yet retention at immediate and delayed tests dent learning and forgetting model
robust learning strategies that help compared to fixed review schedules. should help us to accurately infer the
improve long-term memory retention: Some popular flashcard learning tools, impact on memory decay of selecting
for instance, spaced repetition. Spacing such as Anki and Mnemosyne, use this any skill or combination of skills at a
one’s learning means to temporally dis- type of algorithm. given timestamp. Previous work from
tribute learning episodes instead of Lindsey et al. [2] had similarly chosen
learning in a single “massed” session, However, adaptive spacing schedulers to select the item whose recall proba-
i.e., cramming. Furthermore, carefully have historically focused on pure mem- bility was closest to a given threshold:
selecting when to schedule the subse- orisation of single items, such as foreign in other words, recommending the item
quent reviews of a given piece of knowl- language words or historical facts. Yet, that is on the verge of being forgotten.
edge has a significant impact on its in real-world educational settings, stu- Unfortunately, no student model from
future recall probability [1]. dents also need to acquire and apply a the scientific literature was able to infer
set of skills for a long period (e.g., in skill mastery state and dynamics when
On the other hand, concerns about the mathematics). In this case, single items items involve multiple skills.
“one-size-fits-all” human learning para- potentially involve several skills at the
digm have given rise to the development same time and are practiced by the stu- To bridge this gap, we developed a new
of adaptive learning technologies that dents to master these underlying skills. student learning and forgetting statis-
tailor instruction to suit the learner’s With this ongoing research project, we tical model called DAS3H [3]. DAS3H
needs. In particular, recent research aim at developing skill review sched- builds on the DASH model from
effort has been put into developing adap- ulers that will recommend the right item Lindsey et al. [2]. DASH represents
tive and personalised spacing schedulers at the right time to maximise the proba- past student practice history (item
for improving long-term memory reten- bility that the student will correctly attempts and binary correctness out-
tion of simple pieces of knowledge. apply these skills in future items. An comes) within a set of time windows to
12 ERCIM NEWS 120 January 2020code has been made public on GitHub:
Heuristics
Heuristics see [L1].
Da ta-driven policies
Data-driven policies
Sk
Skill
ill 1 Ot policies
her policies
Other
(1st acquisition)
acquisition) Besides its performance, our model
DAS3H has the advantage of being
suited to the adaptive skill practice and
Timet
Time review scheduling problem that we are
trying to address. For instance, it could
be used at any timestamp to predict the
ALICE
LICE Skill
Skill 2 impact of making the student review
(1st acquisition)
acquisition)
each skill and then to select the most
Figure1:Exampleskillreviewingschedulefromanadaptivespacingalgorithm.Solidline promising skill or combination of skills.
indicatesthefirstacquisitionofaskillandthedashedline,thesubsequentreviewsofthatskill. In future research, we intend to investi-
Aliceistryingtomastertwoskills(resp.inorangeandgreen)withanadaptivespacingtool. gate heuristics and data-driven policies
Shefirstreviewsskill1buteitherbecauseshealreadyforgotitorbecauseshedidnotmasterit for selecting items to maximise long-
inthefirstplace,shefailstorecallitandisrapidlyrecommendedtoreviewitagain.The term memory retention on a set of
situationisdifferentforskill2:herfirstattemptisawinandthealgorithmmakesherwait underlying skills.
longerbeforereviewingitagain.Diversemethodscanbeusedtopersonalizespacing:
heuristics,data-drivenreviewingpolicies,etc. Link:
[L1] https://kwz.me/hKt
predict future student performance and To evaluate our model’s potential use in References:
is inspired by two major cognitive an adaptive and personalised spacing [1] N. J. Cepeda et al.: “Spacing effects
models of human memory, ACT-R and algorithm for reviewing skills, it was in learning: A temporal ridgeline of
MCM. Most importantly, DASH has necessary to first compare its predictive optimal retention”, Psychological
been used in a real-world experiment power to that of other student models on science, 19(11), 2008
[2] to optimise reviewing schedules for real-world data. Thus, we tested [2] R. V. Lindsey et al.: “Improving
Spanish vocabulary learning in an DAS3H on three large educational students’ long-term knowledge
American middle school. Compared to datasets against four state-of-the-art retention through personalized
a fixed review scheduler, the person- student models from the educational review”, Psychological science,
alised DASH-based algorithm achieved data mining literature. We split the stu- 25(3), 2014
a 10% improvement at a cumulative dent population into five disjoint [3] B. Choffin et al.: “DAS3H:
exam at the end of semester. Our model groups, trained all models each time on Modeling Student Learning and
DAS3H extends DASH by incorpo- four groups and predicted unknown stu- Forgetting for Optimally Scheduling
rating item-skills relationships infor- dent binary outcomes on the fifth group. Distributed Practice of Skills”,
mation inside the model structure. The On every dataset, DAS3H showed sub- Educational Data Mining, 2019
goal was both to improve model per- stantially higher predictive performance
formance when prior information is at than its counterparts, suggesting that Please contact:
hand and to use the model estimates of incorporating prior information about Benoît Choffin
skill mastery state and memory item-skills relationships and the forget- Laboratoire de Recherche en
dynamics it encapsulates to prioritise ting effect improves over models that Informatique (LRI), France
item recommendation. consider one or the other. Our Python benoit.choffin@lri.fr
Ontology-based Learning Analytics in Medicine
by Fabrice Jouanot (Univ. Grenoble Alpes), Olivier Palombi (Univ. Grenoble Alpes, CHU de Grenoble) and
Marie-Christine Rousset (Univ. Grenoble Alpes, IUF)
Through SIDES 3.0, we are developing an ontology-based e-learning platform in medicine to make
learning analytics transparent and explainable to end-users (learners and teachers). In this project, the
educational content, the traces of students' activities and the correction of exams are linked and
related to items of an official reference program in a unified data model. As a result, an integrated
access to useful information is provided for student progress monitoring and equipped with a powerful
query language allowing users to express their specific needs relating to data exploration and analysis.
The goal of SIDES 3.0 is to empower We have followed the Semantic Web [1] ontology serving as a pivot high-level
medical students and teachers in data and Linked Data [2] standards for the vocabulary of the query interface with
analytics to enable them to take charge semi-automatic and modular construc- users, and of a huge dataset that is auto-
of their own progress monitoring and tion of OntoSIDES [3], a knowledge matically extracted from SIDES dumps
training. graph comprising a lightweight domain using mappings. Both the ontological
ERCIM NEWS 120 January 2020 13Special Theme: Educational Technology
Figure1:Parametrized
queriesforstudents.
statements and the factual statements to express directly complex queries in built upon the SIDES e-learning plat-
are uniformly described in RDF format SPARQL, we have designed a set of form, which has been used since 2013
[L1], which makes it possible to query parametrised queries that users can cus- by all medical schools in France for
OntoSIDES using the SPARQL query tomise through a user-friendly inter- online graduation and assessment.
language [L2]. face. Figure 1 shows the interface for
students who can choose to launch one Links:
It is important to note that no personal of the queries in the left column that [L1] http://www.w3.org/TR/rdf-concepts/
data concerning students are extracted will be parametrised by the student’s [L2] https://www.w3.org/TR/
from SIDES. Instances of students in identifier (sides:etud12402 in the
OntoSIDES are just represented by example). The chart in the right side of References:
identifiers of the form sides:etu12402 the figure shows the result returned to [1] D. Allemang, J. Hendler:
(as shown in Figure 1). the first query for that student, i.e., the “Semantic Web for the Working
average results obtained per specialty Ontologist: Modeling in RDF,
The current version of the OntoSIDES by the student (to all the questions RDFS and OWL”, Morgan
knowledge graph has scaled up to 6,5 he/she answered related to this spe- Kaufmann, 2011.
billion RDF triples describing training cialty), compared to the overall average [2] T. Heath, C. Bizer: “Linked Data :
and assessment activities performed by obtained per specialty by the whole Evolving the Web into a Global
more than 145,000 students over almost group of active students. DataSpace”, Morgan and Claypool,
six years on the SIDES French national 2011.
e-learning platform. In OntoSIDES, This methodology is not specific to [3] O. Palombi, et al.: “OntoSIDES:
exams and training tests are made up of medicine ; it can be applied to other dis- Ontology-based student progress
multiple choice questions and student ciplines for enriching or building monitoring on the national
activities are described at the granularity learning management systems. The evaluation system of French
of time-stamped clicks of students’ effort required depends mainly on the Medical Schools”, Artificial
selected answers to each question. existence of a reference standard for the Intelligence in Medicine, vol 96,
educational program in the target disci- 2019.
Through SPARQL queries, students can pline.
explore parts of the program that they Please contact:
haven’t yet studied, or launch a compar- SIDES 3.0 is an ongoing project funded Marie-Christine Rousset
ative analysis of their own progress in a by the French Programme Investissement Univ. Grenoble Alpes, France
specific part of the program. The same d’Avenir (PIA) through the ANR call Marie-Christine.Rousset@univ-
mechanisms let teachers analyse the DUNE (Développement d'Universités grenoble-alpes.fr
strengths and weaknesses of their class Numériques Expérimentales). It is con-
compared to other groups at the same ducted under the authority of UNESS
study level in other universities. (Université Numérique en Santé et en
Sport) and involves Université
Since we do not expect end-users to Grenoble Alpes, Inria, and Ecole
master the raw syntax of SPARQL and Normale Supérieure as partners. It is
14 ERCIM NEWS 120 January 2020You can also read
