VIRTUAL REALITY IN HEALTH Current status of VR in health - TIC SALUT SOCIAL
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Report
VIRTUAL
REALITY
IN HEALTH
Current status of VR in health
F U N D A C I Ó
TIC SALUT SOCIAL
Index
Acknowledgements 4
1. PRESENTATION 7
2. VIRTUAL REALITY EVOLUTION 8
3. DEFINITION OF VR/AR/MR 12
3.1 What do we understand by VR? 12
3.2 What do we understand by AR and MR? 14
4. APPLICATIONS OF VR/AR/MR 18
4.1 Applications of VR by sectors 18
4.2 Applications in health 28
5. VR/AR/MR DEVICES 30
6. GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 34
6.1 Main regions of the world 34
6.2 Companies dedicated to health 46
6.3 Catalan projects identified by the FTSS observatory 50
7. KEY ASPECTS TO START A PROJECT IN HEALTH VR/AR/MR 52
7.1 Detection of needs and improvement of processes 52
7.2 Who are the receptors? 56
7.3 What type of devices is needed 58
7.4 Human, technical resources and budget 60
7.5 Safety and maintenance 66
7.6 Ethical and legal aspects 68
8. CONCLUSIONS 76
Bibliography 78
Images 82
2 3
Acknowledgements
We would like to thank the collaboration of all those
professionals who have contributed to preparing this
report. Especially, we would like to mention the SISCAT
professionals who assisted in research and innovation,
entrepreneurial ecosystem and Catalonian biotechnology
start-ups and the Foundation i2Cat for their technological
expertise in VR, as well as the contribution of Huawei of
an insight of future technological application.
4 5
PRESENTATION
The Social Health TIC Foundation is the body of the Ministry of Health that
works to promote the use of ICT and networking in the field of health. In
addition, it becomes an observatory for new tendencies, innovation and
monitoring of emerging initiatives and offers standardisation and product
authorisation services. ICT implantation in health is already unstoppable
and is seen as one of most transforming elements in the future of health.
In this framework, we cannot overlook that the virtual reality, augmented
reality, and mixed reality markets have significantly increased in the last
years. In this sense, last 4 September, the Social Health TIC Foundation
proposed the First Encounter of the Virtual and Augmented Reality Applied
to Health in Catalonia Interest Group, with the purpose of establishing the
bases for creating an interest group in technologies of this filed applied to
health with the participation of the majority of actors of the area involved.
More than thirty professionals from different centres participated and,
currently, many more are already forming part. In fact, the last report of the
Map of Tendencies highlighted that 27% of the centres used virtual reality,
and in 2019 more than fifty initiatives in the field were collected. Therefore,
these different initiatives must be connected, joining efforts, and promoting
innovation among the different member entities of the Catalonian health
ecosystem.
This report pretends being a starting point for the current status of virtual
reality in health. It covers the historical evolution of this technology, the
different definitions, and applications, as well as the most used devices. It
compares the different international initiatives and defines guidelines for
promoting virtual reality projects in health centres. This content, which is
fully available for you all, is the fruit of the work of the Social Health TIC
Foundation professionals and begins with the will of becoming a document
for consultation among the system professionals working in this field.
The information contained in this document reflects a study based on the Social Health
TIC Foundation knowledge with the collaboration of the health sector and ICT, but not
necessarily manifesting the global VR industry position.
6 7
2. V
IRTUAL REALITY Virtual Reality (VR) is a complex concept and currently known in the
technology sector, but historically it has not always been like that, because
there has been great difficulty in defining it and searching for its origin.
EVOLUTION A retrospective analysis of the main historic developments that have
contributed to the evolutions of this technology over the years up to now is
shown below:
Morton Heilig invented a machined called Sensorama (1),
patented in 1962 (2), that allowed showing 3D stereoscopic images 1957
1960
and offered a great immersion while it projected five small films. Morton Heilig also patented the first head-
In addition, the machine was multisensory, meaning, it allowed fitted screen (HMD), called Telesphere Mask
stimulating different senses in the user, using stereo sound, (3). The device provided stereoscopic 3D
aromas and air breezes (fig. A). images with wide vision and stereo sound.
Philco Corporation developed a visual device like a helmet,
called Headsight (4), that could be controlled with movements 1961
of the head by the user. This device was one of the first to use
1965
applied VR in education and training, specifically, for military Ivan Sutherland created The Ultimate Display (5) a device
personnel training. that allowed simulating reality showing a virtual world with 3D
sound and providing tactile feedback to the user.
Ivan Sutherland, with the help of his
1968
student Bob Sproull, created what could be
considered as the first virtual and augmented 1975 Myron Krueger created the first interactive VR platform called Videoplace (7).
It combined graphics generated by computers, projectors, cameras and video
reality helmet, called The Sword of Damocles screens and position sensors. The technology consisted in using large video
(6). However, it was very heavy, and the 3D screens to surround the user in VR in dark rooms. In addition, it allowed the users
images it produced were static. to be located in different rooms and interact with the silhouettes of other users of
the same virtual world
The Massachusetts Institute of Technology (MIT)
developed the Aspen Movie Map (8) that allowed 1978 1985 During the eighties, Jaron Lanier (9) popularised the term VR. In addition, Lanier
founded VPL Research in 1985, together with Thomas Zimmerman. This company
showing a virtual simulation of the city of Aspen created devices for medical uses, flight simulators, vehicle industrial design and
(Colorado) using photographs that had been taken experiences for military uses.
with a car, similar to Google Street View (fig. B).
Figure B.
Aspen (Colorado)
Figure A. Sensorama
The military engineer Thomas Furness developed an
ambitious simulator project which he decided to call 1986
Super Cockpit. The system had an entertainment cabin
that generated by computer 3D maps, advanced infra-red
and radar images that allowed the pilot to see and hear
in real time. The helmet monitoring system and sensors
also allowed controlling the aircraft using gestures, voice
instructions and eye movement.
8 VIRTUAL REALITY EVOLUTION 9
NASA created the Virtual Interface Environment Workstation (VIEW) (10),
a stereoscopic visualisation system worn on the head that had a screen that 1990
could be a computer-generated artificial environment or a real environment
transmitted by remote video cameras. DataGlove had a series of optic fibre
cables and sensors that detected any movement of the fingers of the wearer
and transmitted the information to a host computer. 1991 During the nineties, different publicly accessible
recreation machines and VR helmets appeared, for
playing video games, like, for example, the Virtuality
ones, created by Virtuality Group (11). In these machines
the users wore VR glasses to play with 3D stereoscopic
visual environments in real time.
Nintendo launched Virtual Boy (12), a video game console that
showed monochromatic graphics that were a 3D sensation 1995
thanks to a stereoscopic effect. The console did not sell very
1999
well and was removed from the market a few months later. The film Matrix, by the Wachowskis, premiered
around the world. Although films had appeared
that highlighted the virtual world or VR concept,
like Tron (1982), The Lawnmower Man (1992), or
Johnny Mnemonic (1995), its cultural impact on
society was with Matrix when it premiered.
Google improved its maps adding
the Google Street View function 2007
(fig. C), that allowed viewing with
360º images practically any street
in the world. Later, in 2010, Google
added a stereoscopic 3D mode to the
2012
Google Street View (13) In 2012 VR started to become more popular around the world with the
Oculus Rift, after its founder, Palmer Luckey, launched the first kit for
developers, in a Kickstarter project to obtain financing. He managed
to double his objective of 250,000 $ in just over two hours (14).
Facebook acquired the Palmer
Luckey project, and the Oculus 2014
Rift glasses for two thousand
million dollars (15). Facebook
has plans to expand the Oculus
2016 The first Oculus Rift development kit (fig. D) was finally released
on the market in 2016 and, although originally it was designed
technology to sectors like for use in video games, may other sectors, like education,
communication, education, as motor racing, publicity, tourism, or health, saw the opportunity
well as video games. of making use of this technology to offer new immersion
experiences for users.
Figure C. Google street View
The use of VR in smartphones started to become more
popular, above all thanks to the increase of mobile phone 2018
capacity, like graphic performance. In addition, the availability
of standalone devices (autonomous glasses that do not
need connecting to a PC or incorporate a smartphone), like
Oculus GO or Oculus Quest, in another factor motivating
the use of VR, together with low cost of this technology and
the possibility that more users decide to purchase their own
glasses.
Figure D. Oculus rift
10 VIRTUAL REALITY EVOLUTION 11
3. D EFINITION
OF VR/AR/MR
Immersion. This is the perception of being
3.1 W
hat do we understand by physically present in a non-physical world.
Immersion will be better with well prepared
Virtual Reality? and credible virtual environments. Sound
effects and melodies help to increase the
effect.
Virtual reality (VR) is the term used to describe a computer-generated three-
dimensional environment that can be explored and executed by a person(15).
This interaction with a three-dimensional environment may be through
voice instructions or through peripherals such as controls or gloves, with or
without motion detection.
Although the VR concept existed in the last century, currently it is related to Sensory feedback. Stimulation of the
the experience that, technology in helmets or glasses is used to completely senses requires sensory feedback that is
isolate the person from a real environment to obtain total immersion. obtained by means of integrated hardware
There are diverse elements which are important to have an optimal VR and software, also known as inputs.
experience. These are the most relevant ones:
Virtual world. A three-dimensional
environment that often, but not
necessarily, is generated by a means
(meaning, representation, visualisation,
etc.). Interaction is possible and objects
can be created as part of this interaction. Interactivity. Interaction elements are
crucial for VR experiences, having to
provide enough comfort to naturally
become related to the virtual environment.
12 DEFINITION OF VR/AR/MR 13
In AR, the digital elements shown are static
or animated, both in 2D or in 3D figures.
3.2 What do we understand Normally they are superposed on a figure
the application recognises, like a QR code, a
by Augmented and Mixed pattern drawn on paper or even a real object
Reality?
that has been previously scanned by the
application. It can also use GPS localisation
to generate digital elements in the real
environment without the need of depending
Augmented reality (AR) and mixed reality (MR) are technologies on recognition of patterns to position the
that together with virtual reality (VR) have evolved more over the last digital element. Pokemon GO is one of
years. Both AR and MR have common aspects with VR, like the need the most successful examples (fig. F), by Figure F. Pokemon GO
of hardware to allow generating digital elements (2D or 3D) or the Niantic (17).
possibility of interacting with these digital elements.
Regarding MR, this is defined as the result
AR is term used to define vision of real-world physical
of combining the physical world with the
environment through a technological device adding real
digital world. It can capture the position of
digital elements (16). This is usually used by smartphones,
a person in a physical environment, surface,
tablets, or even smartglasses like Google Glasses, and is
and limits (for example, spatial mapping,
controlled by touch pulsations or voice instructions.
and spatial understanding), illumination and
environmental sound, recognizing objects
and localization (fig. G).
Figure G.
MR application that allows manipulating
a lung in 3D as a hologram
MR not only shows digital elements in real environments, but in addition persons
can see and manipulate them. This is the case, for example, of the creation of an
anatomy model as a hologram1, from different angles (fig. H).
Figure E. AR app showing a lung
MI XE D R E ALI TY
The main difference with a VR device is that, while this latter
completely isolates the user from the real environment in a virtual Physical Digital
environment and manages to obtain the best possible immersion, AR reality reality
does not isolate the user from the real environment, but the user can
continue seeing the real environment but with superposed digital Augmented Virtual
elements (fig. E).
Figure H. Mixed reality spectrum
1. A hologram is a type of phenomena of the visual field or photograph by which light treatment means there are
diverse levels and that way, appears in three-dimensional format. However, in the context of this text, holograms
are 3D computer objects.
14 DEFINITION OF VR/AR/MR 15
There are MR experiences that are very close to what is known as AR, The following figure (19) shows schematically the main differences between VR, AR,
but these usually are a more evolved level (for example, they allow and MR that have been commented throughout this chapter (fig. I):
interacting with holograms). In fact, normally AR applications are
categorized as MR experiences. Also, there are MR experiences that
are closer to what is known as VR, because all the user sees on the
visor is virtual, but with main difference that the virtual environment REALITAT AUGMENTED MIXED
generates takes into account the surrounding real environment, like,
for example, a room and the objects in it.
REALITY REALITY REALITY
It is worth highlighting that one of the glasses devices that has most (VR) (AR) (MR)
popularised MR is the HoloLens by Microsoft, distributed in 2016
and updated in 2019 with the launching of the second version, the The virtual objects are The virtual environment
HoloLens 2 (18) (fig. I). Artificial
environment superposed on real is combined with the real
world environments world
MI X ED REALI T Y
Immerse
PHYSICAL devices
DIGITAL
reality reality
Holographic
devices Total immersion in the The real world is improved It allows interacting with
virtual environment with digital objects both the real world and
the virtual environment
Microsoft Acer
HoloLens Examples of mixed reality devices Windows
Figure I. MR holographic devices and immersion devices
Figure J. Differences between VR, AR, and MR
2. H
oloLens was the first wireless holographic computer. It was controlled by sight, finger gestures or voice. It
has a set of twin glasses where the holograms are projected, and they give information about depth, shape
and dimensions.
16 DEFINITION OF VR/AR/MR 17
4. A
PPLICATIONS OF
VR/AR/MR Also, it is worth highlighting that other entertainment areas that are offering this type of
experiences. Applications like Oculus Venues (21), in fact, have contributed to distributing
activities like watching a film or following a sports event in VR, adding a social component
thanks to the possibility of remotely participating in the development, in the virtual
environment, together with other users.
Virtual Reality (VR) can offer multiple applications that can be used in diverse The other industry, known as Location Based VR (LBVR), is gaining an important position
sector thanks to the capacity of recreating any environment and situation. in the VR entertainment world. LBVR is all types of environments where users participate
in a VR simulated experience, accompanied by specific structures capable of creating
The sector that generates most content and benefits using VR is multi-sensory effects like vibrations, aroma, and simulated objects that the users can
entertainment, specifically video games. Currently, also, other sectors like touch and manipulate. Thanks to the possibility of having these infrastructures, LBVR
tourism, motor racing, sports, companies, education and above all health experiences can provide the entertainment world in VR an added dimension that, in
services use this technology to offer new experiences and possibilities to a domestic environment, is even difficult to attain because of the high costs of installation
their users. and maintenance.
Some of the applications that VR allows in different sectors, as well as some
of the most outstanding that AR and MR devices allow.
Extending the entertainment concept to
areas like tourism and culture, is very common,
4.1 Applications of VR by sectors currently, using VR applications to visit three
dimensional recreations of museums and
remote tourist attractions. In addition to the
Entertainment most typical virtual visits, also participating
As previously commented, the most outstanding sector in VR is new experiences, like three-dimensional
entertainment, specifically that of video games. This industry is always recreations of famous paintings in VR (22),
generating greater benefits and many international technology that bring the previously mentioned LBVR
companies that are committed to it, for example, Microsoft, Samsung or concept to museums (fig. L) and exhibitions.
Facebook. Currently, regarding Playstation VR, the Sony VR glasses for the
domestic video console PlayStation 4 (PS4), has sold more than 5 million
units around the world (20) (fig. K). Likewise, more video games are being Figure L. VR experience simulating a museum
published with possibilities of being experimented in VR, not only for
domestic video consoles, but also for computers and smartphones.
Also, AR applications are being distributed in tourist and cultural environments as
substitutes or as complements for audio guides (23) that allow the users to visualize
additional information on smartphone screens, tablets, or specific glasses (like the
HoloLens by Microsoft) (24).
The popularity of devices like smartphones and tablets is also contributing to the
distribution of many augmented reality applications among the users.
Online shops for smartphones and tablets, like Play Store (Google) or App Store (Apple), offer
a wide range of AR apps. Some of the most successful are Pokemon GO that uses AR and
GPS localization to show digital elements in 3D of roads and emblematic locations, or others
like Snapchat or Instagram allowing to apply a multitude of filters and digital accessories
using AR (25).
Figure K.
PSVR Glasses
18 APPLICATIONS OF VR/AR/MR 19Education
VR can be used in teaching and training situations, allowing the accessible Industry
and easy presentation of complex data to students (fig. LL). In addition to Training can be conducted using VR in
discovering an innovative environment for new possibilities. For example, many industrial processes, from working
VR allows travelling in time and personally seeing historic events, not only on an assembly line (fig. M), making virtual
to obtain information and data, but also for other types of perceptions visits to the facilities, or driving industrial
that only VR immersion can offer. machinery in a mining excavation, among
others.
Figure M. VR Experience in industry
Also, occupational accidents can be simulated with this technology or simulating
risky operations, as well as emergency action simulations and escape routes.
In addition, VR simulators can also be used to optimize tasks in the industrial
sector, for example, offering interactive warehouse management systems.
On the other hand, the vehicle industry is also especially benefiting from this
technology, like for example by visualising virtual full-scale prototypes of cars,
or by establishing design revisions with the collaboration of different engineers.
For over a decade, the automation world has already
implemented AR applications, such as ADAS
Figure LL. AR Colouring. Disney Research Hub
(Advanced Autonomous Driving Systems) that thanks
to virtual projections applied to natural image provide
There are also a multitude of AR apps (aimed at teaching) for learning information attached to the driver, estimating the
(26), like for example, in the case of astronomy, identification of pantes vehicle trajectory, and guiding the driving (29) (fig. N).
and stars by means of focussing our mobile phone on them (Sky Map Both AR and MR are an integrated part of the Industry
app). Or mathematics where primary students can learn to add and 4.0 concepts, because they allow the workers access
subtract using real world scenarios (FETCH! Lunch Rush app) and like to digital information and they can superpose it on the
in the case of languages, where they can directly translate texts printed physical world (fig. O). Figure N. Continental. ADAS System for AR
on objects, focussing the object with Google Translate (27).
With MR, students can manipulate objects generating better
understanding of the same, and also interact with series of data,
complex formulas, and abstract concepts, that way facilitating their
understanding (28).
AR applications are also distributed in infant education environments,
where young users can participate in creative activities, like colouring
or drawing. These applications add a three-dimensional dimension that
results more attractive than traditional ones.
20
Figure O. Industry 4.0 APPLICATIONS OF VR/AR/MR 21Sports Other sectors
The great gamification offered by VR platforms
Many other sectors also take advantage of VR, AR, or MR to offer new experiences
also make them ideal for improving the sports
to their users. For example, in the property world, VR allows recreating properties
performance of the users from the commodity
before they are built and showing them to interested parties (32). On the other
of their homes and while they compete and
hand, AR can, for example, digitally represent furniture in the home of the user by
socialize with their family and friends.
simply focussing on it with the smartphone or tablet camera (33).
Now, this technology can facilitate practising
In the tourism sector, persons can experiment virtual routes (fig. R) offered by travel
a large variety of sports from the same place,
agencies when deciding for a destination, or once the user is touring. Thanks to
thanks to the fact that VR simulations transfer
AR apps, they can enrich their travel experience with extra digital information they
the users to real sports environments like
provide when focussing on emblematic places with smartphone cameras (34).
gymnasiums o open spaces (fig. P).
The fashion sector also uses VR software to check virtual fashion shops, preparing
Likewise, VR can also be combined with other
Figure P. VR connected to personal welfare 3D avatars to help designing on smartphone screens (35).
machines or devices to augment immersion. For
example, it can be combined with static bikes with Regarding the construction sector, any project can be elaborated in a virtual
a real environment simulation for mountain cycling. environment offering advantages when, for example, testing diverse factors
without neither the cost of structure construction, reducing the amount of
errors that could appear once the building is finished. On the other hand, with
Safety MR, in a construction the understanding and communication of complex spatial
conditions can be improved by combining the real word with the virtual world (36).
Safety forces can use VR for different training modes. For example, in military
cases, real combat situations or other hazardous scenario simulations can be Finally, the scientific sector must be highlighted a using this technology to express
recreated to learn how to efficiently react (30). This technology can also help complex ideas and concepts, like the case of molecular models or statistical
when learning how to pilot a military vehicle, like tank or fighter jet. results.
Likewise, VR simulations are safer and less costly
than traditional training methods and can avoid any
serious or slight risk for the professionals (fig. Q).
On the other hand, VR can also be used to handle
Figure R. VR use in simulated virtual tours
post-traumatic stress disorder in soldiers that have
any disorder on the battlefield (31). If the users
gradually submit to expositions that trigger their
condition, they can learn how to face the symptoms
in a safe environment.
AR also has great presence in this field due to
the quantity of possibilities it can offer for users
of advanced military goggles or helmets with
this technology. For example, for a jet fighter
pilot, a smart helmet can provide him with crucial
information of the surrounding environment while Figure Q. Training armed forces with VR
flying at more than 1,500 km/h.
22 APPLICATIONS OF VR/AR/MR 23In fact, the light field captures system allows the
creation of three-dimensional content thanks to the
possibility of knowing the angle of incident of each
4.1.1 The future of 3D, its applicability in VR and its ray going through the lenses. That way creating
impact in the health field knowledge about the position of the objects in the
3D world and generating viewpoints ideally from any
position.
Figure V. Holographic screens in a car
The success of VR is not limited to the use already mentioned in
this report, because current development in the 3d video field has On the other hand, the representation systems using the same technology, called holographic
the objective of transforming the daily use of certain technologies. or stereoscopic screens (fig. V), allow visualization of three-dimensional content without the
Concepts like volumetric3 video, holograms, and light fields4 have need of using VR or AR glasses. This is an ideal situation when the user needs perfect visibility,
achieved great popularity in the research world and soon will have as is the case, for example, of driving a vehicle.
a daily impact on the lives of users.
The benefits of the advances this vision offers will have a considerable impact in the health
The way we communicate is already one of the first initiatives affected sector. We refer to learning, for example, thanks to holographic transport, it will be possible
by this revolution. Linking volumetric video with real time transmission to remotely participate in face-to-face sessions of medical training in virtual environments.
systems (like video conferences), can extend to Holoportation (37) Multi-sensory simulations can be shared, in which an instructor, for example, can reproduce
(fig. S), meaning, holographic transport, a system that allows two a physical sensation, a weight or a movement and transmit it to the students.
persons to communicate in real time in a VR or AR environment. In
the first case, both communicate in a virtual environment, and, in the
second case, they are projected into the real world. Taking advantage of light field technologies, in medical
Figure S. Holoportation
Even so, volumetric video can also be an improvement without the environments, improvements can be made in the
background of VR experiences. At the moment, the user is represented surgical field like, for example, in endoscopies. In fact,
by an avatar in the most popular VR experiences (fig. T), or now, thanks to the endoscopic light field (38), the limited
without any sort of representation. Holograms can be projected as field of vision of a camera can be replaced by a system
a self-representation, so the user has the sensation of seeing his/her of images with less presence of occlusions and now
own body, with real movements acting at the same time. Thanks to with the capacity of seeing through cloudy fluids, with
this revolution, many of the mentioned uses can gain another human presence of blood or other poor visibility situations.
and social dimension.
Figure W. 3D reconstruction of a heart
Many activities needing high interaction or that involve certain risks
can be conducted remotely without losing the benefits of face-to-
Thanks to fusion techniques among different and diverse technologies, it will be possible
face communication. Professional education or training in the use
Figure T. User observing his/her hands to join the images created by an endoscope to information obtained by body scanning
represented in VR of heavy and hazardous machinery are examples of applications in
technologies, like ultrasounds, to create 3D reconstructions of human cavities without any
which natural human interactions with an added dimension of realism
type of invasive action. Any endoscopic surgery operation will now have a new dimension in
or simulated environments.
terms of visibility for the medical team.
Lastly, the systems based on light fields are positioned as the main
Including all these cases of using new networks or high speed and low latency transmissions
candidates for creating high quality 3D content and the three-
(like 5G), VR/AR/MR applications can be used not only for learning or for face-to-face
dimensional representation in holographic screens (without the need
interventions, but also to conduct activities that up till now required the presence of
of a VR/AR screen) (fig. U).
professionals and immediate actuations. Delicate situations like, for example, surgical
interventions, can be remotely conducted thanks to MR jointly with optical devices (associated
to the sense of touch) (fig. W).
This vision shows research development at medium and long term; however, virtual
applications are already strongly rooted in the health sector.
Figure U. VR use for surgical
training
3. Volumetric or holographic video is the technique that captures movement in a three-dimensional space.
4. This function completely characterizes the light flow through an obstructed space in a static scenario with
fixed illumination.
24 APPLICATIONS OF VR/AR/MR 254.1.2 AV/AR/MR related to other technologies like 5G and
Wi-Fi 6
The repercussion of 5G or other connectivity technologies
regarding AV/AR/MR in the health field and how they will
accompany us in our daily lives in the coming future has been
analysed with future projection thanks to the collaboration
between Huawei and the Social Health TIC Foundation.
The truth is that the evolution of AV/AR/MR devices over the last
five years has very significantly improved. In fact, these devices
have improved by promoting their immersion effect and the 3D
sound effects, as well as the fact of providing the gigantic IMAX
screen experience.
Beyond technical and design specifications, companies like
Huawei, among others, have accompanied the launching of VR
glasses with a platform of more than 30,000 content of high-
definition reproduction and with the expectancy of adding more
films, series and video games over the next years. Like what
happened with platforms like Netflix, Amazon Prime and Disney+,
to mention some of the actors in the streaming world, there is
the possibility that, over the coming years, the competition in VR
moves from hardware to the generation and emission of content.
For example, the apparition of exclusive retransmission of events
like football and concerts, applications in the education, culture,
and business worlds, etc.
On the other hand, an increase in the percentage of 4K/8K, 3D All that means that our current communication infrastructure, either 3/4G or Wi-Fi n/ac,
and interactive content is foreseeable something that will require will no longer be viable to cover this growth of VR content and quality. In addition, it is
greater bandwidth and minimum latency to guarantee good expected that VR will become one of the fundamental 5G applications, as well as the
user experience. We refer to a minimum of 50 Mbps and latency exponential growth of terminals with this technology. Given that the 5G radio electronic
below 20 ms per user in applications above all in the health resource is finite, also Wi-Fi infrastructures are expected to be updated to the new
sector. Wireless formats will predominate as the excellent means standard, or simply Wi-Fi 6, that shares many services with 5G, above all in bandwidth
because of its freedom of movement. and low latency terms.
26 APPLICATIONS OF VR/AR/MR 274.2 Applications in Health
The health branch is one of the greatest users of VR, allowing multiple From a surgical viewpoint, MR glasses are being used in the operating theatres during
possibilities, like, for example, the treatment of phobias (39), surgery interventions, for example, to visualize interactive panels allowing professionals to
simulations and training of abilities (40) or promoting the recuperation of have access to, consult and manipulate clinical information of the patient in real time,
cognitive capacities (fig. X). without deviating attention form the surgery (44) (fig. Z).
In virtual training environments, operating theatres can be represented, Likewise, both AR and MR can allow entering into the patient using 3D holograms
where health professionals improve their abilities and acquire new one generated form magnetic reactions (45) (fig. AA).
in a safe and controlled environment. Also, health emergency situations
In addition, this technology can be used as such by diverse assistance professionals,
can be recreated, scenarios that are difficult to recreate in a physical
from different places, being able to participate and collaborate in the same intervention.
environment (41). In addition, it can be used to learn anatomy exploring
This is possible thanks to MR glasses online and representing as holograms the
inside the human body, developing surgery simulations, or showing three-
professionals that are physically present in the operating theatre (46).
dimensional images of the body so medicine students can explore it (42).
On the other hand, we must highlight that in February 2020, the Hospital de la Santa
Also, VR can help to reduce anxiety and calm pain (43), for example, in
Creu i Sant Pau in Barcelona was the first in Spain to use a 3D hologram as a support
cases of “ghost member” effect due to amputation or in persons with
image for highly complex surgery. The hologram allowed recreating th anatomy of
fibromyalgia.
the patient being operated. It was visualized in the same operating theatre in a glass
In addition, many technology companies and hospitals are committed pyramid of about two meters height, showing th three-dimensional image to all the
to health projects using both AR and MR. For example, health sciences assistance professionals that were participating in the intervention (47).
students can learn anatomy an interact with 3D holograms representing
the different layers of the human body, (skin, muscles, blood vessels, etc.)
(fig. Y).
Figure AA. Representation of users in holographic avatars
Figure Y. Holographic representations of the human body in its differ-
ent layers Figure Z. Use of MR to see “through” a body
Figure X. VR for rehabilitation
28 APPLICATIONS OF VR/AR/MR 295. V R/AR/MR Graphic capacity (processor, RAM...):
Mainly, in autonomous devices or also known as
standalone.
Graphic capacity (processor,
RAM…): Mainly, in autonomous devices
DEVICES
or also known as standalone.
As an emerging technology, the number of companies dedicated to VR/
AR/MR is considerably increasing, making it more accessible through
innovative companies and start-ups that are entering the market.
Standalone
There is a large diversity of models of VR glasses available (fig. AB), form PC
low cost glasses for mobile phones up to high level devices that allow
offering a much greater quality immersion experience.
Console
Smartphone
Figure AC. Connections to other devices according to requirements of different VR glasses
Figure AB Diverse VR glasses Non-autonomous glasses: Need connecting to a PC, if connected by cable or wireless.
As in any technological device, the functions and characteristics are what
differentiates some glasses form others and what marks the price. The main Accessories: Those incorporated in the same product, like controllers and/or external sensors.
factors to be considered for any VR experience are shown below (fig. AC): Also, there are important differences between more simple controllers that only serve as pointers,
and other more advanced ones, with different buttons that detect movements.
Screen incorporation: Currently this differentiates VR Movement: The simplest VR glasses only detect rotational head movement, but the most
glasses for mobiles from the rest. The mobile phone is introduced sophisticated ones can also detect translational movement of the user (inclinations, walking
into glasses that do not include it, so the mobile phone projects or turning the body 360º, or even the same movement of the retina). Also, considering the
the images. space where the user moves and the precision is reflected in the virtual experience (if there
is a delay in the response or not, meaning, if the real movements are truly represented in the
virtual space). Some glasses offer up to 3.5 m x 3.5 m of free space, but others offer up to 10
Screen resolution: The higher the solution, the better quality of m x 10 m, also according to the quantity of sensors used and their precision. Likewise, the
the images projected on the visor screen. The texts will be easier to read newest models start to not use external sensors, because the same devices incorporate the
the projected images will have better definition, which provides the best sensors to detect the space where the users move, that way facilitating the installation and
immersion experience to the user. use of glasses.
Material: We can find from cardboard devices to other more resistant and
Frequency of refreshment (Hz): This number indicates the quantity of ergonomic ones, like those made from carbon fibre.
times per second that the screen image changes. According to the experts, the
recommendation is that VR glasses offer a minimum of 90 Hz, because lesser
refreshment frequencies can cause dizziness or discomfort in the users (48).
30 VR/AR/MR DEVICES 31Some of the most outstanding manufacturers of VR/AR/MR glasses (48) on the
market, the visors they offer and the most outstanding main characteristics:
Displacement Displacement
Company Device detection 5 Autonomous 6 Controllers 7 Price 8 Company Device detection 5 Autonomous 6 Controllers 7 Price 8
Oculus GO No Yes Yes (1 unit) 159 € (32 GB) Glass Enterprise No Yes No $ 1,195.00
(2018) 219 € (64 GB) Edition 2
(2019)
Oculus Rift Yes No Yes (2 units) Not available on
GOOGLE 78 i 79
(2016) - Uses external - Needs to be con- - Oculus Touch official website Google Cardboard No No No From 6€
sensors nected to a PC that - Needs to include a
meets the minimum smartphone
requirements Mirage Solo Yes Yes Yes (1 unit) € 249.95
FACEBOOK 74
Oculus Rift S (2019) Yes No Yes (2 units) 449 € (2018) - Does not need
-Improved version - Does not need - Needs to be - Oculus Touch external sensors
of Oculus Rift external sensors connected to a PC LENOVO 80 Lenovo Explorer Yes No Yes (2 units) Not available on
that meets the mini- (2017) - Does not need - Needs to be official website
mum requirements external sensors connected to a PC
Oculus Quest Yes Yes Yes (2 units) 449 € (64 GB) that meets the mini-
- Does not need - Oculus Touch 549 € (128 GB) mum requirements
external sensors HP Windows Yes No Yes (2 units) € 655.82
Vive Cosmos Yes No Yes (2 units) 829 € Mixed Reality - Does not need - Needs to be
(2019) - Does not need - Needs to be - VIVE Cosmos HP 81 Headset external sensors connected to a PC
external sensors connected to a PC Controller (2017) that meets the mini-
that meets the mini- mum requirements
mum requirements Huawei VR Glass Yes No Yes (3 units) 2,999 Yuan (about
Vive Pro HMD Yes No Yes 679 € (2019) - Does not need Needs to be - NOLO CV1 Air 400 €)
(2018) - Including two - Needs to be - Sold separately external sensors connected to a VR 3D Console
base stations for connected to a PC smartphone (only Controllers
HTC 75 one 5m x 5m space that meets the mini- with the Huawei
mum requirements HUAWEI 82 Ma-te 30, Mate 20
or P30 series, or a
Vive Yes No Yes (2 units) Out of stock com-puter mee-
- Including two - Needs to be - Controller ting the minimum
base stations for connected to a PC requirements
one 3.5m x 3.5m that meets the mini-
space mum requirements
Vive Focus Yes Yes Yes (1 unit) 716,48 € + 159,90
(2019) - Does not need € 12 XR-1 Dev Edition Yes No No € 9,995
external sensors - Needs to be
connected to a PC
MICROSOFT HoloLens 2 Yes Yes No 3.500 $ 14 that meets the mini-
MICROSOFT 76 (2019) - Does not need mum requirements
external sensors
VR-2 Pro Yes No No € 5,995
SONY Playstation VR Yes No Yes 299 € VARJO 83 - Needs to be
(2016) - Includes a 2.0 - Needs connec- - sold separately connected to a PC
SONY 77 camera ting to the PS4 or (PS Move) that meets the mini-
a PC mum requirements
VR-2 Yes No No € 4,995
- Needs to be
connected to a PC
that meets the mini-
mum requirements
Magic Leap 1 Yes Yes Yes (1 unit) $ 2,295
5. P ositional tracking indicates if the device, as well as detecting head rota-tion like all VR glasses, also allows, detecting if the
MAGIC LEAP 84
user moves, meaning, if he/she crouches, walks, inclines, etc.
6. Standalone indicates if the device does not need connecting to another devices (for example to a PC), to func-tion.
7. The controllers or commands allow detecting arm movement. The most sophisticated also detect finger move-ment.
8. Prices obtained from the official website of the manufacturer when preparing this report.
9. Extendible to 10 m x 10 m with four base stations.
10. This model obliges to acquire an additional service for two years for 159.90 €
11. Microsoft likewise offers, other pur-chase operations (https://www.microsoft.com/en-us/hololens/buy).
32 VR/AR/MR DEVICES 336. G
ENERAL VIEW AT
WORLDWIDE SCALE
OF VR/AR/MR
6.1 Main regions of the world ASIA
ÀSIA
EUROPA
EUROPE
Like what happens with the majority
of technologies, virtual reality (VR),
USA
EEUU
augmented reality (AR) and mixed
reality (MR) are fully globalized products.
All the parties involved in the research,
development, production, and creation of
content are located in different parts of the
world. The most outstanding regions on
a global scale in VR are Asia, Europe, and
the United States. A referential report in the
VR/AR field (34) allows obtaining a certain
perspective of how VR is developing on
European scale. Likewise, the report (49)
approaches a series of data that have been
agglutinated and complemented with the
same experience of the FTSS observatory
during 2018. In addition, specific reports
about the implantation status of this
technology in specific countries like
Australia (50) or New Zealand (51) has
allowed completing a global vision of the
VR/AR. An approximation to the realities
of each continent by means of articles and
the reports to which we have previously
referred.
AFRICA
ÀFRICA OCEANIA
34 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 35The United States of America have a clear leading position in the global VR/AR
market. The research and development of both hardware and software are centre
in the Silicon Valley region, where companies like Google, Apple or Facebook
are located. The large video game studios established in Los Angeles are the
main producers of content for VR/AR/MR that are exported on a global scale. Regarding Europe, the research capacity and tradition of universities stands out due to
Regarding financing, the United States have venture capital funds specialized development of highly specialized content. Research is usually centred on developing
in VR and AR that promote the creation of new companies and solutions. This complex technologies for industry, in the health field, etc. Generally speaking, in the VR/
contrasts with South America where there has been much less activity. This AR field in Europe, diverse members states stand out for concentrating this research
confirms that there are not any globally referential company, but there are some activity: France, United Kingdom, Germany, Sweden or Spain (specifically, Barcelona) are
very atomized technological centres. Generally, there are no governmental grants the main exponents of this activity. In all cases, European universities have integrated as
or large venture capital funds for VR/AR. work plans the generation of research consortiums, which collaborate in international
projects financed by European research funds.
ÀSIA
EUROPE
EUROPA
USA
EEUU
Manchester / Newcastle
Berlin / Munich
París
Catalunya
Silicon Valley
Los Angeles
Regarding Oceania, the activity is concentrated in the main
metropolis of Australia and New Zealand. This industry is deeply
linked to the West Coast of the United States and where audio-visual
content creation companies stand out. There is a strong local market
where ad hoc solutions are developed, fostered, and financed by
local institutions and by the government
Some corpuscles of VR/AR activity can be found in Africa, in countries
like Nigeria. However, there are no governmental policies or venture
capitals that allow correct development of companies. The lack
Johannesburg
of persons with abilities and knowledge to develop projects with
this technology is also endemic to the majority of the continent.
Exceptionally, the South African capital Johannesburg, has activity
around VR/AR with diverse companies and Hubs centred of producing
content
AFRICA
ÀFRICA OCEANIA
36 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 37If we consider the most outstanding countries of the European
continent, we can see which are the most outstanding institutions Germany has diverse VR and AR nodes well distributed around
and the environments of the different states according to the the country. Its main node is in Berlin. The capital, with its essence
report of 2017 (34). of disruptive creation, makes available to VR a complete creative
ecosystem that promotes innovative content and applications.
There is a manufacturing culture in south Germany represented by
Regarding the United Kingdom, its capital is considered the main node of the vehicle, aero spatial and biotechnological industries. These large
VR/AR activity network. Generally, the country has a favourable environment companies are, likewise, potential clients for the VR/AR industry, to
for start-ups and development of VR/AR projects. Access to private financing which they demand solutions to improve their industrial processes or
sources and the large quantity of governmental investments in innovation and to complement their own products.
technology favours sector dynamism and provides a good rhythm of company
On the other hand, Munich agglutinates prolific cinematographic and
creation and solutions, also in cities like Manchester or Newcastle.
audio-visual content industries.
London acts like a referential Hub on European scale and has strong connections
In addition, Germany has a very good rhythm creating start-ups.
with other Hubs like Beijing (Asia) and Silicon Valley (United States). With this
However, their growth is often unused by a very consolidated
latter they strengthen the ties, because they share the language and have long
entrepreneurial structure that does not allow these start-ups to enter
experience of collaborations.
the market occupied by large corporations.
The connections on European scales are very string with Berlin, the Netherlands
and France, from where they received technical knowledge. ÀSIA
Universities and research centres follow the same line as companies.
We see that the Munich Technical University is a reference in the
EUROPA application of VR for industrial design. Regarding the Brandenburg
region, the Berlin University or the clusters are centred on research
of new products in any field, although currently concentrating on VR
applications, mainly, in the neuroscience field.
In France, the cities of Paris and Laval are references on Manchester / Newcastle
European scale and form a strong node in the VR/AR Berlin / Munich
ecosystem. Developing applications is highlighted as París
the main activity of the French VR activity. Prestigious
3D design industry, universities and schools in this
Catalunya
subject have provoked that development of artistic and
Silicon Valley
visual content become the main motor of this industry.
Nearly two years ago theLos city Angeles
of Laval welcomed
a reference event in the VR world, called Laval Virtual,
Pel que fa a l’Estat Spain, the main activity is based on creativity and the abilities of its professionals.
that transforms this small western France city into the VR
Competitive prices promote a creative scenario in which start-ups find in Barcelona a favourable space for
capital once a year. This event, in addition, has caused
experimenting and developing new VR/AR solutions. On the other hand, obtaining grants and investors to
the creation of the Laval Virtual Centre, a stable space
develop the projects is difficult compared with neighbouring countries.
for the creation and research of this technology.
In Catalonia, different institutions specialised in VR, like EventLAB, Barcelona VR, VR Centre Barcelona and
All these nodes and structures are strengthened at
Wayra stand out. Spanish universities and companies compete with different events related with VR/AR/
government level by a series of policies that want to
MR, either because of the central subject of the event or because indirectly VR and AR have a space in the
favour high quality products produced in France both in
programming. Some of the most outstanding acts or events are the Mobile World Congress, 4YFN, VR Thec,
software and in hardware.
the Barcelona 360° VR and AR market, Expoelearning, Healthio and many others that consolidate the use of
these technologies in the country.
Johannesburg
38 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 39Approximation to the global VR/AR
research state VR/AR Research tendency
Regarding the VR/AR application in the health world, this is still
subject to many research questions, both in technology and A bibliographic research has been conducted with a a specific
in their applications. To objectify the way research centres are chain (“Virtual Reality” AND “Health”) to know the usage
globally operating and the behaviour of research in the health field tendency of VR in the health field, which has provided a certain
concerning VR/AR, the FTSS has conducted a series of revisions approximation, in publications, to the general use and research
of scientific literature that it pretends approaching below. conducted of VR in the health field. The research was conducted
in the general field using the metasearcher Google Scholar and
the PubMed and Scielo databases, with the following results of
evolution in publications over the last years (fig. AD):
2018 32 9
2017 23 3
2016 19 5
2015 174
0 50 100 150 200 250 300 350
Figure AD. Annual number of VR publications in the health field (2015-2018)
40 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 41Subjects on which the research concentrates
A research has been made on the following terms to establish which are the most relevant VR subjects: “Virtual Reality”
AND “Health”. On this occasion the revision was centre on the Pubmed database and the reduced time range of the
year 2018. Of 342 publications, 27% have been excluded for not adapting to the study subject. After a more exhaustive
analysis of these, we can highlight that 28% is of open access while the remaining 71% is only by private license. It is
worth highlighting that the percentage of private licenses is in consonance with the European average of scientific
publication in open access format, as indicated by the “la Caixa” Social Observatory (52).
As we can see in figure 45, the topic most considered in VR research in Mental health
the health field is that of mental health, followed by training of abilities, This topic includes spaces for handling phobias (53) and psychopathies (54) and
systematic revisions of rehabilitation, cognition research and seven for working social abilities (55), among others. This subject is large and allows
other topics that have been identified. The main subjects and some of exploring the VR capacity to build controlled realities in which the patients are
the articles representing them and that have been selected for revision exposed to different situations simulating real environments.
are shown below (fig. AE):
Training/Education
Regarding training of abilities, there are some studies that pretend validating
VR as an educational means. In this case, there are two well differentiated lines:
training of abilities by health professionals (56) and training of abilities in persons
SUBJECTS with a pathology (57).
Systematic revision
Mental health
Salut mental This cannot be considered as a research subject as such, but we must highlight
the research community interest in scientific evidence regarding VR. An example
Training
Entrenament
of this is the interest shown by Bremen University for the evidence of VR in the
Systematic revision
Revisió sistemàtica surgery field (58). Another example is the interest of researchers from Katowice
Rehabilitation
Rehabilitació University due to its application of virtual reality standards in balance training
programs (59), among others.
Cognition
Cognició
Gent gran
Adults % Rehabilitation
Rehabilitation has been one of the most treated subjects in that regarding
Canvi de
Change ofconductes
conducts simulation over the last years, and the VR/AR applications are not any exception.
Exercise
Exercici A Randomized Controlled Trial, conducted by the Korean Keimyung University,
demonstrates the effectiveness of 3D-ARS systems to improve balance in old age
Neurology
Neurologia
(60).
Ophthalmology
Oftalmologia
Emotions
Emocions Cognition
Lastly, we must highlight the cognition field, as a recurrently strong subject in
Development
Desenvolupament VR applications. Some studies refer to distraction as a tool for reducing pain and
anxiety in persons that submit to painful or invasive techniques, or even before
surgery (61).
Figure AE. Subjects detected in the VR publications (2015-2018)
42 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 43VR/AR publications by countries Approximation to Catalonian research
We have analysed a significant part of these articles and we have extracted Regarding Catalonia and following the same research chain (“Virtual
some conclusions to obtain an image from research conducted around the Reality” AND “Health”) in only one database, we see that over the last
world during 2018 that allows establishing a comparison to corroborate what three years (2017-2019) scientific production has mainly treated the same
international consultants declare and to whom we have referred throughout subjects observed as a tendency in the rest of the world. It is noteworthy
this report. that a member of a Catalonian university is the main researcher of the
The same revision of 2018 literature has been used to compile this information, publications mainly focussed on the VR applications in mental health.
aimed at focussing on the origin of the main researcher of each scientific
article or author of correspondence. We can say the Barcelona University is the
ASIA
ÀSIA most productive institution in VR, above
EUROPE
EUROPA
all in the mental health field. We also
find other institutions, like the Universitat
USA
EEUU Autònoma of Barcelona, or programmes/
entities like ICREA or EURECAT. It could be
5,88
1,47
surprising that Catalonia does not have a
2,94 2,94
1,47
more notable impact in research of VR/AR
1,47
5,88 2,94
1,47
2,94
in the health field. As we have seen in the
1,47 2,94
1,47 5,88
previous phases on the Report, Catalonia
7,35
2,94
1,47 has a knowledge transfer tradition
1,47
regarding the product. That means that
36,76
a good part of the knowledge generated
is used directly by companies to generate
solutions aimed the market and that are
not compiled as scientific publications all
the innovations being conducted around
Spain, or that, due to intellectual property
rights (patents, registries...), the option of
1,47 publishing is not the bets for the research
4,41
group conducting the project in question.
Argentina
Austria Australia
Brazil
France
Germany 1,47
Holland
Lithuania Canada
Sweden China
Switzerland United Kingdom
Taiwan
Thailand
Italy
ÁFRICA
ÀFRICA
Ireland
Korea
Netherlands
Poland
OCEANIA
Romania United States
Spain
44 GENERAL VIEW AT WORLDWIDE SCALE OF VR/AR/MR 45You can also read
