Hardware prototype with component specification and usage description Final version - WEKIT
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Ref. Ares(2018)3470317 - 29/06/2018
Wearable Experience for
Knowledge Intensive Training
Hardware prototype with component
specification and usage description
Final version
Authors:
Roland Klemke (OUNL)
Daniele Di Mitri (OUNL)
Bibeg Hang Limbu (OUNL)
Puneet Sharma (UiT)
Wearable Experience for Knowledge Intensive Training
Project No 687669
Wearable Experience for
Knowledge Intensive Training
Revision History
Version Date Contributor(s) Modification
1 09 January Daniele Di Mitri Initial setup of the Document
2018
2 17 April 2018 Daniele Di Mitri First changes after the responsibility shift
& Bibeg Limbu from myndplay to OUNL
3 08 May 2018 Bibeg Limbu, Restructuring, general planning, section
Roland Klemke responsibilities
4 31 May 2018 Daniele di Mitri Overall system architecture
5 14 June 2018 Bibeg Limbu Key requirements and SPU
6 20 June 2018 Puneet Sharma WEKIT sensor PCB, images.
7 28 June 2018 Paul Lefrere, Profread, finalisation of Proofread version
Roland Klemke
8 29 June 2018 Mikhail Quality review, style and formatting
Fominykh
Disclaimer: All information included in this document is subject to
change without notice. The Members of the WEKIT Consortium
make no warranty of any kind with regard to this document,
including, but not limited to, the implied warranties of
merchantability and fitness for a particular purpose. The Members
of the WEKIT Consortium shall not be held liable for errors
contained herein or direct, indirect, special, incidental or
consequential damages in connection with the furnishing,
performance, or use of this material.
WEKIT consortium Dissemination: Public Page 2/22
Wearable Experience for
Knowledge Intensive Training
Hardware prototype with
component specification and usage
description
Final version
WP 3 | D3.5
Editors:
Roland Klemke (OUNL)
Authors:
Daniele di Mitri (OUNL), Bibeg Hang Limbu (OUNL), Puneet Sharma (UiT)
Reviewers:
Paul Lefrere (CCA), Mikhail Fominykh (EP)
Deliverable number D3.5
Dissemination level Public
Version 1.0
Status Final
Date 30.06.2018
Due date 30.06.2018
WEKIT consortium Dissemination: Public Page 3/22
Wearable Experience for
Knowledge Intensive Training
Table of Contents
REVISION HISTORY ............................................................................................................................... 2
EXECUTIVE SUMMARY ......................................................................................................................... 5
1. INTRODUCTION ............................................................................................................................ 6
2. SYSTEM ARCHITECTURE ................................................................................................................ 6
2.1. HOLOLENS................................................................................................................................ 8
2.2. SENSOR PROCESSING UNIT ........................................................................................................... 9
2.3. SENSOR COMPONENTS AND THEIR INTEGRATION .............................................................................. 10
3. FINAL HARDWARE COMPONENTS .............................................................................................. 12
4. KEY REQUIREMENTS ................................................................................................................... 13
5. PROTOTYPE AND USAGE DESCRIPTION ....................................................................................... 14
5.1. INSTALLATION AND CONFIGURATION............................................................................................. 14
5.1.1. Windows Stick PC with Windows 10 or Windows 10 PC ..................................................... 14
5.1.2. Recorder configuration ........................................................................................................ 17
5.2. USAGE .................................................................................................................................. 18
5.2.1. Starting and using applications ............................................................................................ 18
6. MAPPING OF HARDWARE SUPPORT FOR THE FRAMEWORK ....................................................... 20
7. CONCLUSIONS ............................................................................................................................ 21
WEKIT consortium Dissemination: Public Page 4/22
Wearable Experience for
Knowledge Intensive Training
Executive summary
This D3.5 report presents the final version of the experience capturing hardware
prototype’s design and architecture for materializing the WEKIT framework and design. It
builds upon deliverable 3.6 “Software prototype with sensor fusion API specification and
usage description”. It also closely follows the recommendation and findings of D3.4
“Requirement analysis and sensor specifications”. The deliverable presents the final
hardware setup which incorporates various devices and sensors into a single platform.
The deliverable also details various components such as the Hololens, sensor processing
unit and the sensor jacket. The final design accounting for wearability and ergonomics of
the sensor jacket is outlined in “D5.8 Wearable design solution”. This deliverable is
focused on providing an overview of the hardware eco-system.
WEKIT consortium Dissemination: Public Page 5/22
Wearable Experience for
Knowledge Intensive Training
1. Introduction
The primary purpose of this deliverable is to present the hardware prototype. It also
outlines how the hardware setup enables the WEKIT training methodology to be
manifested. See table 1, section 6 for a full summary of how specific training needs and
methods map to hardware decisions. In this version of the deliverable, we present the final
design for the hardware which includes three major components:
1. The Hololens
2. The Sensor Processing Unit
3. The Sensor Jacket
The deliverable outlines sensors that are incorporated into the hardware eco-system
along with their specifications. The selection of sensors has been guided by the
requirements stated in D3.4 and suggestion made on D3.2. Each time we added a new
sensor or tried a new mix of sensors, we gained experience of how to recognise and
address new interoperability challenges and new workflows associated with creating new
variants of the hardware prototype. This underpins what we report on in section 2.
The deliverable begins by providing a bird-eye view of the hardware ecosystem in section-
2: Final architecture and the communication between the components in section 2.1:
Communication between the components. It then proceeds by providing detailed
description on each of the components namely, section 2.2: Hololens, section 2.3: Sensor
processing unit and 2.3 Sensor jacket. The sections that will follow, will reflect on the
previous deliverables (D3.2 and D3.4). The deliverable will justify how the current eco-
system meets the key requirements put forward by previously mentioned deliverables
and the WEKIT framework. It will also provide a use case example which also outlines how
the hardware ecosystem can be used.
2. System architecture
The hardware prototype described in this deliverable is part of the overall system
architecture as specified in D2.3 (Final Architecture and Learning Experience Content
Model) and belongs to the final prototype as described in D2.5 (Final Prototype). This
deliverable concentrates on the hardware related aspects and their relation to other
aspects of the overall WEKIT solution.
The following figure illustrates the overall system architecture comprising the WEKIT.ONE
unified software installed on Hololens, the sensor jacket with installed sensor electronics,
and the sensor processing unit (SPU) as main aspects of the wearable solution (Fig. 2.1).
The figure also displays their connections to the backend components for authentication,
cloud-based data storage, and community access.
In the following, we will concentrate on the communication between these components
with a key focus on how sensor data is communicated through the various hardware and
software components.
WEKIT consortium Dissemination: Public Page 6/22
Wearable Experience for
Knowledge Intensive Training
Fig 2.1 Overview of the complete WEKIT.ONE architecture comprising hardware and
software components
As we added sensors, we found we could not rely on the specifications for linking sensors
by wire. Accordingly, the communication of data between sensor jacket and Stick PC and
Hololens is done over wifi. In contrast to D3.4’s suggestion to use wired USB connection, a
wifi connection was chosen to simplify the hardware setup, as first internal test runs
indicated unstable cable connections in the wearable design.
WEKIT consortium Dissemination: Public Page 7/22
Wearable Experience for
Knowledge Intensive Training
Wifi was preferred over Bluetooth-based communication as it offers larger bandwidth and
connectivity. Most modern stick PC offers the possibility to broadcast wifi hotspot
eliminating the need to have a third-party router.
The following swim lane diagram shows the general communication flow between
Hololens, SPU, sensors in the sensor jacket and feedback actuators in the same jacket (Fig.
2.2). A user interaction within the wekit.one software triggers Hololens to start
communication with the SPU, which itself manages all communication to the connected
sensors and feedback actuators. Sensor-based data collection and storage is initially done
on the SPU, from where data is communicated back to the Hololens, which in turn
manages the backend upload.
Fig. 2.2 Swimlane diagram of communication flow
2.1. Hololens
Microsoft Hololens is a head mounted wearable augmented reality device1. The Hololens is
processed by Custom Microsoft Holographic Processing Unit HPU 1.0 with 2 Gb of RAM.
More RAM and higher HPU power would have been useful for some of our prototype
configurations and we anticipate that the HPU 1.0 will be replaced by a more performant
unit. Our architecture should be compatible with upgraded versions of the HPU. The HPU
1.0 is an untethered stand-alone computing device running on specialised version of
Microsoft Windows 10. Hololens is capable of projecting virtual contents mapped to the
physical world space in an unobtrusive manner. It also hosts other sensors such as
microphone, audio, Infrared camera, which allow different types of user interactions (Fig.
2.1.1). This makes Hololens ideal for acting as a central user interaction component. The
1
https://www.microsoft.com/en-us/hololens
WEKIT consortium Dissemination: Public Page 8/22
Wearable Experience for
Knowledge Intensive Training
WEKIT prototype software is built on the Hololens platform. It runs both the recorder and
the player.
Figure 2.1.1 Microsoft Hololens
Hololens also offers wifi and Bluetooth connection allowing the Hololens to communicate
with other components wirelessly.
2.2. Sensor Processing Unit
A stick PC is a device which has independent CPU or processing chips and which does not
rely on another computer. In the prototype, the stick PC functions as a Sensor Processing
Unit (SPU). Among all the stick PCs in the market, Intel® Compute Stick is chosen for
WEKIT project since it is an Intel® x86-based Stick PC (see Figure 2.2.1). One of the latest
version of Intel® Compute Stick is with pre-installed Windows 10 Home x86 and a quad-
core Intel® AtomTM processor that makes many applications available to be used.
Figure 2.2.1 Intel® Stick PC [34]
For the connectivity of the Intel® Compute Stick, it uses the built-in antennas and Intel
wireless technology for connecting and accessing files in the cloud. The built-in Bluetooth
can be used for connecting wireless peripherals or transfer files from smartphone to the
stick PC´s on-board storage. Furthermore, there are USB ports to connect the devices such
as keyboard, mouse or Ethernet adapter [35].
For the power supply of the Intel® Compute Stick, a Micro-USB charging cable and power
brick are used [36].
WEKIT consortium Dissemination: Public Page 9/22
Wearable Experience for
Knowledge Intensive Training
2.3. Sensor components and their integration
The sensors are integrated into the WEKIT jacket using ESP32. ESP32 performs as
a complete standalone system or as a slave device to a host, reducing
communication stack overhead on the Hololens. ESP32 can interface with other
systems to provide Wi-Fi and Bluetooth functionality as well. ESP32 collects the
data from different wearable sensors such as: heart rate, galvanic skin response,
IMUs, temperature and humidity, and also receives messages from other
components to turn on or off the two vibration motors.
Figure 2.3.1 WEKIT Sensor Hardware PCB (printed circuit board), this enables the
connection of different sensors and ESP32 on a single board.
As shown in Figure 2.3.1, the PCB reduces the complexity of connections by clear labelling
of different components and different connectors also enable easier debugging. For
example, the heart rate sensors are connected on Pulse 0 and Pulse 1, the IMUs are
connected on I2C0 and I2C1, Galvanic Skin Response sensor is connected on GSR,
temperature and humidity sensors are connected on Temp0 and Temp1, finally, the two
vibration motors are connected on M0 and M1.
WEKIT consortium Dissemination: Public Page 10/22Wearable Experience for
Knowledge Intensive Training
Figure 2.3.2, WEKIT Sensor PCB with ESP32 mounted.
3D printed mount for
heart rate sensor
IMU
Figure 2.3.3 Heart-rate sensor placement and IMU location
A 3D printed mount for Hololens which enables the placement of a heart rate sensor on
the forehead of the user is shown on the figure above. It also shows the location of an IMU
on the back of the WEKIT garment (Fig. 2.33).
WEKIT consortium Dissemination: Public Page 11/22Wearable Experience for
Knowledge Intensive Training
For communication of sensor data from ESP32 to Hololens, SPU, and other components,
we use MQTT2 (Message Queuing Telemetry Transport), an ISO standard publish-
subscribe-based messaging protocol, which uses wifi (See section 5.1.1 for details on
MQTT installation). It is a machine-to-machine (M2M)/"Internet of Things" connectivity
protocol designed as a lightweight publish/subscribe messaging transport.
The different sensor components such as: SPU, ESP32 and the associated sensors, and
battery pack will be placed in a garment designed (in WP5).
3. Final hardware components
The following table provides the specifications of the final hardware components of the
WEKIT.one prototype (Table 3.1).
Table 3.1. Final hardware components of WEKIT.one
Processor Product Specifications
Microsoft Hololens 1 IMU
Four environment understanding cameras
1 depth camera
1 2MP photo / HD video camera
Mixed reality capture
4 microphones
1 ambient light sensor
ESP32 with display Has a display (can display messages from SPU/
stick PC)
Both analog and digital inputs / outputs
Wi-Fi and Bluetooth
2.3V to 3.6V operating voltage
Running at 240 MHz dual core
Vibration Motor* DC3V 0834 Mobile phone micro flat vibration
motor
1.5V-3.7V DC
12000-2500RPM Min
3.4mm height, 8mm width
Rated current: 70 mA maximum
Galvanic Skin Response Grove - GSR Sensor
Input Voltage: 5V/3.3V
2
http://mqtt.org/
WEKIT consortium Dissemination: Public Page 12/22Wearable Experience for
Knowledge Intensive Training
Sensitivity adjustable via a potentiometer
External measuring finger cots. Measures GSR at
fingers.
IMU Power supply :3-5V (internal low dropout
regulator)
Communication: Standard IIC/SPI
communication protocol
Gyro range: ± 250 500 1000 2000 ° / s
Acceleration range: ± 2 ± 4 ± 8 ± 16g
3.5mA operating current when all 9 motion
sensing axes and the Digital Motion Processor
are enabled
Environmental temperature and Supply Voltage: 3.5 - 5V
humidity sensor Low power consumption 30mW
Defined humidity: 0 ... 100% RH
Absolute humidity measurement accuracy +/-
2% RH (relative humidity 10-90%)
Sampling rate: ≤ 1 Hz
4. Key requirements
The hardware prototype is designed to meet the key technical requirements, which
include managing interference, limitations and environmental factors which may affect the
overall quality of the data, signal and or the user experience. In this section we justify the
selection of the hardware based on the key requirements identified in D3.4.
For the three use cases (mentioned in D6.1, D6.2, and D6.3), the first wave of the WEKIT
prototype was evaluated and the results are discussed (in D6.4, D6.5, and D6.6). Based on
the results, our experience of resolving connectivity and interoperability issues arising
with mixes of sensors, comments from the reviewers, and a meeting of technical partners
on 20 and 21 November 2017, a final set of requirements were formulated for the second
wave of the trials. With our current state of the hardware, we address these key
requirements in the following list.
RQ1 (Processing power): The processing power needed for data processing and
for communicating the sensor data to the stick PC i.e., Sensor Processing Unit
(SPU) is handled by the built-in processing unit of Hololens. However, the Hololens
processor is still limited and in future as more advanced AR glasses are built, we
can transmit the data directly to the AR glasses hence reducing or eliminating the
need for SPU.
RQ2 (Battery life): The WEKIT jacket allows battery for sensor processing to be
swapped.
WEKIT consortium Dissemination: Public Page 13/22Wearable Experience for
Knowledge Intensive Training
RQ3 (Reliable communication): Quick and reliable communication, implemented
in terms of built in wifi functionalities in the Sensor processing unit and Hololens.
RQ4 (Offline usage): SPU and Hololens can be used offline (with no live internet
connection) by storing the data locally. Backend upload can be delayed until
Internet connection is established again or recorder sessions can be downloaded
manually from Hololens and SPU at a later point in time.
RQ5 (Sensor voltage): All sensors approximately run on same voltage to the best of
the current availability.
RQ6 (Open platforms for hard- and software): Unity and Mixed reality toolkit and
other development kits used in WEKIT are all open source. The jacket also uses
individually picked sensors with minimal constraints compared to a ready made
solution. All sensors used comply to open hardware specifications according to the
Arduino open source hardware standard except for selected commercial products
integrated such as MYO.
RQ7 (Cost-efficient hardware): Hardware cost have been kept to a minimum with
future projections in mind. Technology tends to get cheaper as they mature, which
will allow the solution to be much more affordable in the future. Microsoft
Hololens being the most expensive hardware component of the architecture has
already announced a cheaper second version and competitors rising up.
RQ8 (modularity and extensibility): The WEKIT hardware architecture is modular
and extensible. New Sensors can be added or removed. With the SPU separating
specific sensor hardware from the main WEKIT application abstractions have been
implemented that allow for replacement of individual components without
affecting the overall solution.
5. Prototype and Usage Description
5.1. Installation and configuration
5.1.1. Windows Stick PC with Windows 10 or Windows 10 PC
The learning hub requires a Windows 10 compatible device. The following components
need to be installed:
1. Learning Hub
a. Download the Learning Hub Folder3
b. Unzip it and move to the Desktop
c. Run the HubDesktop.exe. You’ll see the following screen from Figure
5.1.1.1
3
https://goo.gl/Mtv7Bw
WEKIT consortium Dissemination: Public Page 14/22Wearable Experience for
Knowledge Intensive Training
Figure 5.1.1.1 Learning Hub configuration window
d. Change the %username% in the MQTTDataProvider path to the current
windows user. When the path is set, hit save and close the Learning Hub for
now. The MQTTDataProvider listens to the wekit/vest topic (localhost)
and sends data to the predefined topics. A list of those topics can be found
in the Drive4.
2. MQTT Broker
a. Installing MQTT Broker (Mosquitto5)
b. Install OpenSSL6 (required for Mosquitto)
c. Download PThreadVC2.DLL7 and move to Mosquitto install folder
d. Make sure the ESP32 is broadcasting on the topic: wekit/vest
4
https://docs.google.com/spreadsheets/d/1XkfKYgYf5mWZJcqHW0fgbDk5Vc1Nwjkstsd-
gpIvb8s/edit#gid=0
5
https://mosquitto.org/download/
6
http://slproweb.com/products/Win32OpenSSL.html
7
ftp://sources.redhat.com/pub/pthreads-win32/dll-latest/dll/x86/
WEKIT consortium Dissemination: Public Page 15/22Wearable Experience for
Knowledge Intensive Training
3. Wifi-Hotspot
a. Connect Windows 10 PC to local network (Wifi/Cable)
Figure 5.1.1.2 Accessing the Mobile Hotspot settings in the Windows pc
b. Name the Wifi-Hotspot WEKIT[1|2|3|...] and assign a simple password (Fig.
5.1.1.3)
Figure 5.1.1.3 Configuring the hotspot
c. Enable Wifi Hotspot
d. Test: Connect Hololens to that Wifi-Hotspot and check network access
WEKIT consortium Dissemination: Public Page 16/22Wearable Experience for
Knowledge Intensive Training
4. Recommended additional tools on Windows PC
a. TeamViewer8: allows remote control and management of the stick PC
without connecting screen/keyboard - very helpful specifically for stick
PCs, would even allow remote control for the sticks from remote sites
b. When installing TeamViewer make sure to install it as follows (Fig. 5.1.1.4)
Figure 5.1.1.4 TeamViewer installation setup
c. When TeamViewer is installed, go to Extras -> Options -> Security and set a
simple password. Press OK and remember the password + the ID as shown
on the main screen.
5.1.2. Recorder configuration
1. Unity:
a. The IP of the stick PC has to be manually inserted into the
WEKITLearningHubControl Class from the editor. Click on the
WEKITManagers objects in the scene and check the component. (Do not
change the port number as it is set by default. If needed to be changed
change in the learning hub as well.)
8
https://www.teamviewer.com/nl/download/windows/
WEKIT consortium Dissemination: Public Page 17/22Wearable Experience for
Knowledge Intensive Training
Figure 5.1.2.1. WEKITLearningHubControl configuration in unity
b. Build the application
2. Backend
a. The recorded data can be found online and downloaded.
o Root: http://wekitproject.appspot.com/
o Session overview: http://wekitproject.appspot.com/storage/sessions
o Download:
http://wekitproject.appspot.com/storage/fetch?filename=XYZ.zip
5.2. Usage
5.2.1. Starting and using applications
Hololens only:
1. Place all the 3d models into the Hololens.
a. Connect the Hololens to the windows pc with USB cable.
WEKIT consortium Dissemination: Public Page 18/22Wearable Experience for
Knowledge Intensive Training
b. Follow http://127.0.0.1:10080/ in the browser
c. Navigate to http://127.0.0.1:10080/FileExplorer.htm /CameraRoll
d. Browse and upload the models
2. Start WEKIT.One application on Hololens.
It is generally a good idea to save the ARLEM files manually too from Hololens to a PC
using the device portal to avoid possible data losses according to network connectivity
issues.
Hololens + learning hub:
1. Start WEKIT.One application on Hololens.
2. Select Recorder/Player
WEKIT consortium Dissemination: Public Page 19/22Wearable Experience for
Knowledge Intensive Training
6. Mapping of hardware support for the
framework
In this section, we present an overview of the WEKIT Transfer Mechanisms supported by
the selection of WEKIT hardware (Table 6.1). The Transfer Mechanisms are individual
learning design methods that exploit expert performance with the help of Augmented
reality and Wearable technology (WEKIT D1.5). Microsoft Hololens and built in sensors
can support range of transfer mechanisms such as remote symmetrical tele-assistance,
directed focus etc. The MYO armband and the IMU sensors supports transfer mechanisms
such as Interactive virtual/ tangible objects. For self-awareness of physical state, we will
need metrics such as heart rate variability and posture information which are integrated
into the WEKIT jacket. For haptic hints, the two vibration motors are placed on the arms of
the user in the jacket.
Table 6.1 Transfer Mechanisms supported by the WEKIT hardware
Transfer Description Sensors Key products
mechanism
Augmented Augmenting virtual path atop the smart/AR Hololens, MYO
Path physical world in a way which allows glasses, smart Gesture, IMU
the trainee to guide the his/her armband, Inertial in the WEKIT
motion with precision sensors jacket
Remote In instances where a remote expert is smart/AR glasses Hololens
symmetrical needed, it acts as an instant
tele- communication channel without
assistance having to divert from the workflow.
Interactive Interactable virtual objects to practice smart armband MYO Gesture
Virtual / with physical interactions relying on control
tangible the 3d models and animation armband
objects
Haptic Lightweight force feedback for Vibrotactile MYO,
feedback perception and manipulation of bracelets and Vibration
authentic objects by means of haptic rings motors
sensor, to provide feedback and
guidance
Annotations Allow a physical object to be Smart glasses, Infrared
annotated by the expert during task tracking of camera on
execution (similar to sticky notes but physical Hololens
with more modalities) environment
Directed Visual pointer for relevant objects Gaze direction / Hololens
focus outside the visual area of the trainee object
recognition, EEG
(attention/focus/
mental effort)
WEKIT consortium Dissemination: Public Page 20/22Wearable Experience for
Knowledge Intensive Training
Point of Provides expert point-of-view video Head mounted Hololens
View Video which may provide perspectives not HD, high frame mounted
available to a third person. rate camera camera
X-ray vision Visualizing objects and processes that Infrared camera Hololens with
are hidden behind physical surfaces tracking of
and otherwise invisible to the eye, to physical
enhance understanding. environment
Highlight Highlight physical objects in the Marker-based Hololens with
object of visual area indicating to the trainee and location- tracking of
interest that the expert marked it as an object based physical physical
of interest tracking environment.
3D Models 3d models and animations assist in Smart glasses Hololens with
and easy interpretation of Complex gesture
Animation models and phenomena which recognition
require high spatial processing ability
Object Virtually amplify the effect of the Smart Glasses Hololens
Enrichment process to enable trainees to
understand the consequences of
certain events or actions in the
process which may be too subtle to
notice
Contextual Provide information about the Dedicated Temperature
Information process that is frequently changing sensors for each sensors and
but is important for performance parameter, such heart rate
as temperature, sensors in the
heart rate WEKIT jacket
7. Conclusions
The goal of D3.5 is to present the final design, functionality and core architecture of the
hardware prototype so that users are able to fully experience the WEKIT learning
methodology and its implementation. The document outlines the hardware prototype and
also instructions for setting up, running, and administering the system. The hardware will
be used and tested in the second upcoming trials for functionality and effectiveness.
This document is part of a number of deliverables about the final WEKIT prototype and
should thus be seen in conjunction with D2.3 (Final Architecture and Learning Experience
Content Model), D2.5 (Final prototype), D3.4 (Requirement analysis and sensor
specifications), D3.6 (Software prototype with sensor fusion API specification and usage
description), D5.8 (Wearable design solution).
WEKIT consortium Dissemination: Public Page 21/22Wearable Experience for
Knowledge Intensive Training
Wearable Experience for Knowledge Intensive Training
Project No 687669
WEKIT consortium Dissemination: Public Page 22/22You can also read
