Effect of the Opponent's Appearance on Interpersonal Cognition that Affects User-to-User Relationship in Virtual Whole-Body Interaction - J-Stage
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
https://doi.org/10.20965/jrm.2021.p1029
Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
Paper:
Effect of the Opponent’s Appearance on Interpersonal
Cognition that Affects User-to-User Relationship
in Virtual Whole-Body Interaction
Sho Sakurai, Takumi Goto, Takuya Nojima, and Koichi Hirota
The University of Electro-Communications
1-5-1 Chofugaoka, Chofu, Tokyo 182-8585, Japan
E-mail: {sho, goto.takumi, hirota}@vogue.is.uec.ac.jp, tnojima@nojilab.org
[Received April 16, 2021; accepted August 23, 2021]
People infer the internal characteristics (attitude, in- Therefore, we infer the internal characteristics of a person
tent, thoughts, ability, relationship, etc.) of others (in- from perceived and perceivable information. The process
terpersonal cognition, IC) from the impressions they involved in this act or function is called interpersonal cog-
form from the personality or attributes of those oth- nition (henceforth referred to as IC).
ers (impression formation). Studies premised on in- The formed impressions about the personality or at-
terpersonal communication in a seated condition have tributes of the other person provide the basis for IC. The
confirmed that, regardless of whether the communi- main clues of impression formation consist of preexisting
cation is in the real world or in a media environment, knowledge (looks, attire, the views of others, status, etc.)
the appearance of the other person affects IC and the and the other’s behavior (utterances, acts, etc.) [1], among
outcome of the communication. People also develop which visual information in particular provides powerful
relationships based on impressions or images of the clues [2]. It is thought that one processes such information
other party. The psychological relationship manifests on others based on one’s implicit personality theory (IPT,
in physical relationships, that is, the relative positions the body of vague views or beliefs regarding the interrela-
of the body or the movement. In this study, we eval- tionships between outward features and personality char-
uate the effects of the appearance of the opponent’s acteristics) to make inferences about the other’s internal
avatar on the players’ IC in whole-body interaction characteristics and behavior [3]. IPT includes stereotypes
taking place in a virtual reality (VR) space. Moreover, (categorized or generalized fixed notions about the ob-
we examine the feasibility of constructing a method ject’s attributes) about sex, race, occupation, etc., as well
of changing the players’ relationship in interpersonal as the relationship between physical outward features and
interactions that accompany the control and interfer- internal characteristics, and those assumed among multi-
ence of the entire body, “whole-body interaction,” by ple internal characteristics, such as “gloomy people are
manipulating their appearances. In this study, we nervous.” IPT asserts that the external appearance of the
selected the party game Twister as a case model of other provides key information for impression formation,
whole-body interaction and developed a system that which is the basis of IC.
allows users to play Twister in VR space. Using this So far, many studies have examined the connection be-
system environment, we conducted an experiment to tween outward facial features, such as attractiveness [4,
evaluate the players’ IC based on the gender and real- 5], facial expressions [6, 7], similarity [8, 9], and gen-
ism of the opponent’s avatar. The results showed that der [10], and IC. Moreover, other studies, using telep-
differences in the appearance of the opponent’s avatar resence or virtual reality (VR), showed that perceptions
affected the IC of male players. We also indicated that of the other’s ability [11], decision making [12], or the
changes in IC observed in the experiment can affect ability to influence others [13] vary when the appearance
the players’ relationship, thus identifying issues that transmitted via a medium has been visually altered. Most
must be resolved in order to realize the method. of these studies were premised on situations of face-to-
face communication between sitting parties.
On the other hand, the impressions and images formed
Keywords: embodiment, interpersonal recognition,
from the other’s appearance also affect the relationship
avatar, human relationship, whole-body interaction
that is established between two or more people. Some
studies have investigated that the psychological distance
1. Introduction or power dynamics corresponds to physical relationships,
such as the body-to-body distance [14] or relative posi-
It is impossible for one to correctly understand all tions [15]. In the case of physical interference, i.e., sit-
the internal characteristics, such as personality, psychol- uations involving physical contact, the bodily motions
ogy, intent, thoughts, ability, or relationship, of the other. used to control the physical relationship and muscular ten-
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1029
© Fuji Technology Press Ltd. Creative Commons CC BY-ND: This is an Open Access article distributed under the terms of
the Creative Commons Attribution-NoDerivatives 4.0 International License (http://creativecommons.org/licenses/by-nd/4.0/).
Sakurai, S. et al.
sion/relaxation are perceived via contact. While physical munication, physical, non-verbal information, such as fa-
contact affects one’s behavior and decision making [16], cial expressions, voice, and posture, provides clues for in-
this effect and the sense of pleasure or displeasure caused ferring the emotions of the other person [6, 7, 24, 25]. In
by people coming into contact with each other vary de- particular, one study found that visual information on fa-
pending on the other’s apparent gender [17–23]. These cial expressions and behavior makes up 55% of the clues
findings suggest that the external appearances of interact- to emotional impressions [2]. There are also many stud-
ing parties may regulate not only the psychological but ies on the similarity of looks; for example, some studies
also the physical relationship between them in situations found that strong attractiveness or trust is felt when a per-
involving whole-body control and interference. son’s looks resemble one’s own as compared to when they
In this study, we define “whole-body interaction” as the do not resemble oneself [8, 9], while other studies found
“interpersonal interaction that involves the mutual control that the gender determined from a person’s looks affects
of and interference between whole bodies (of the interact- whether he or she is perceived to be attractive or trustwor-
ing parties).” The objective of this study was to evaluate thy [10].
the effects of the other party’s appearance on IC in whole- Meanwhile, physical information can be processed for
body interaction and to examine whether it is possible transmission in communication over media. Thus, stud-
to realize a method of changing the relationship between ies have been carried out on how IC or communication
parties engaged in whole-body interaction through visual is affected when the physical (i.e., body) information of
manipulation of their appearances. If such a method can communicating parties is visually processed for transmis-
be realized, it may be possible to develop a media technol- sion. For instance, a study that examined the effect of
ogy that can improve intimacy in interpersonal communi- facial expressions in a video chat environment found that
cation or allow psychological and physical adjustment of creativity improved drastically when the communicating
the relationship between parties engaged in whole-body parties’ expressions were altered by image processing to
interaction, such as the manipulation of power balance express smiles as compared to when they were altered to
in games and sports. Furthermore, realization of above show neutral or sad faces [13].
method will contribute to knowledge on the effect of ap- The appearance of an avatar, which gives the user a
pearances in interactions with non-human agents (human- substitute body in VR space, also affects IC. Zanbaka
agent interaction [HAI]), such as robots or personified et al. found that virtual characters capable of speech, such
agents. as a cat or human avatar, were perceived as “bolder” than
In the subsequent sections, we first review the litera- actual humans [11]. Banakou investigated the effect of
ture on the connection between external appearance and avatars on sociability in VR space and found that users
IC, relationships, and physical contact. We then describe exhibited a higher proactivity or intent to carry on the
Twister [a], which is used in this study as a case model conversation when the other user’s avatar encountered in
of whole-body interaction, and the configuration of a sys- VR space was more elaborate or physically attractive [26].
tem that allows users to play Twister in VR space (VR This study points out the similarities between the effects
Twister). Next, we perform an experiment using VR of the avatar’s physical attractiveness or gender on IC
Twister designed to evaluate the effect of the appearance or behavior and the effects of other person’s appearance
of the opponent’s avatar on the user. Finally, based on the found in the real world. Studies have also been con-
results of the experiment, we discuss how the appearance ducted on perceived impressions based on the external ap-
of the opponent’s avatar in VR Twister can affect the user pearance of manmade objects. Kuroda et al. found that
relationship, whether it is possible to realize this technol- the effect of product recommendation, that is, the user’s
ogy, and issues that must be addressed to achieve this. purchase intention, increases when the user’s impressions
of the personified sales agent’s appearance and behavior
match each other [12]. Moreover, Mori proposed the phe-
2. Related Studies nomenon of “uncanny valley” (the phenomenon in which
people feel greater affinity with an artificial object as its
2.1. Interpersonal Cognition and Appearance human likeness in appearance or movement increases, but
this affinity turns into aversion or fear when the likeness
There is a wide range of accumulated knowledge, reaches a certain point) [27]. Subsequently, Mathur and
mainly in the field of social psychology, on the connec- Reichling experimentally verified the “uncanny valley”
tion between IC and appearance. Jackson et al. carried out phenomenon by investigating the human likeness of and
a meta-analysis of physical attractiveness and intellectual people’s perceived affinity with robots and androids with
competence and found that people who have an attractive various appearances [28]. These studies [12, 27, 28] sug-
appearance are perceived as competent compared to those gest that the conversation partner via some media, when it
who are not physically attractive [4]. Dion et al. reported has a nonhuman appearance, will not necessarily invoke a
that appearance-based impressions tend to be related to neutral impression or IC.
certain personality traits. For instance, if a person’s ap- The above studies show that, in the real world as in the
pearance is attractive, his or her other traits and actions are media environment, the other party’s appearance is capa-
seen to be attractive, and they are treated with a positive ble of varying impression formation and IC, thus affecting
attitude by others [5]. It has also been found that, in com- the mutual behavior as well as the results of communica-
1030 Journal of Robotics and Mechatronics Vol.33 No.5, 2021
Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
tion. However, very little research has been conducted on differs in the real world and in a VR environment, or the
whole-body interactions. user’s intent to engage with the target differs depending
on whether the user and target share the same space.
2.2. Relationships and Appearance In addition, a theory has been proposed that structurally
Personal relationships are endowed with an emotional explains the correspondence between physical posture or
aspect, such as love-hate, affection-enmity, dominant- movements and psychological state (Laban Movement
submissive, advantageous-disadvantageous, acceptance- Analysis, LMA [33]). LMA identifies the correlation be-
denial, or cooperative-disruptive. We perceive these re- tween the geometric structure of posture (shape) and the
lationships at the cognitive level. Situations in which per- quality of movement (effort), on the one hand, and the
sonal relationships develop are generally thought to in- psychological state between the two polarities of fighting
volve communication and empathy. However, regardless and indulging, on the other hand. For example, the ef-
of whether there is an intent to communicate or not, there fort factor of “weight” (strength, speed, heaviness, etc., of
will always be physical interactions in situations in which body movement) corresponds to the psychology of active
the same space is shared with another party with a cor- or passive. Although LMA was originally advocated to
poral presence. In such cases, one unconsciously issues be used in dance choreography, efforts have been made
one’s honest attitude or intent using nonverbal informa- to apply it to behavioral analysis or psychological assess-
tion (honest signals), which are signals interpreted by the ment [34]. This suggests that a correlation may exist, also
other party to predict one’s behavior [29]. For instance, in whole-body interactions, between the psychology or re-
when two strangers take an elevator together, they usu- lationship based on one’s impressions of the other party
ally maintain physical distance from each other and avoid and one’s body posture or movement.
making eye contact. In this instance, they are using phys-
ical distance and line-of-sight to regulate the relationship
of “have no intent to develop an intimate relationship.” In 2.3. Physical Contact and Appearance
other words, the nonverbal, physical behavior functions It is known in research on social touch (touch as a so-
as a meta-message of the intent on the relationship. Such cial behavior) that physical contact has positive effects,
adjustments of relationships are usually carried out below such as inciting altruistic behavior or increasing approval
the level of consciousness [30, 31]. rate [16]. However, the effect of contact or the perception
In situations in which it is possible to control one’s en- of pleasure/displeasure of contact differs depending on
tire body, the interpersonal relationship manifests in var- one’s gender or that of the person with whom one comes
ious physical arrangements. For example, Hall classifies into contact. Men tend to avoid contact with the same sex
interpersonal distance (relative distance between people) but feel pleasure when making physical contact with the
into four types: intimate (45 cm or less), personal (45– other sex, regardless of whether they are close to or famil-
120 cm), social (120–360 cm), and public (360 cm or iar with the other party [17, 18]. However, women per-
more), and he asserts that people differentially employ ceive physical contact with someone of the opposite sex
these distances in their interpersonal relationships [14]. with whom they are close as being pleasant but feel that
In general, the more intimate the relationship, the shorter contact with the same sex is unpleasant [19]. Similar ten-
the interpersonal distance. In addition to the other party’s dencies were found in simulated contact based on remote
gender, his or her personality or information concerning haptic transmission [20], simulated contact without hap-
his/her identity, that is, one’s IC, determines whether to tic feedback [21], and contact with robots [22]. It has also
allow or reject invasion of one’s personal space (the space been reported that, in male-to-male interactions, positive
contained within the intimate and personal distances; an interpersonal effects of physical contact are produced if
invasion of this space gives rise to discomfort, anger, or the other party’s avatar assumes a female character [23].
anxiety [14]). This suggests that the other party’s appear- There are few studies that relate social touch to the
ance may affect one’s intent to accept or reject his or her other party’s appearance and discuss factors other than
approach. gender differences. The effect of physical contact in
The correspondence between the relationship that a hu- whole-body interactions has not been clearly established.
man user develops with an artificial object and their physi- However, it is natural to think that the gender of the other
cal arrangement has also been investigated. Sakamoto and party or affinity with their appearance will affect the per-
Takeuchi analyzed the correspondence between the state ception of pleasure/displeasure of contact or the intent to
of control of the relationship in the initial phase of interac- accept or reject it.
tion, expressed by a two-axis model consisting of “active
engagement with target (control)” and “active acceptance
of target engagement (acceptance),” and the user’s phys- 3. Constructing the Environment of VR
ical arrangement (distance, position, etc.) relative to the
target [15]. Komatsu and Abe found that the user’s in-
Whole-Body Interaction
tent to engage with a personified agent increases when
3.1. Search for a Model of Whole-Body Interaction
the latter is presented in the form of an actual robot as
opposed to when presented onscreen [32]. This can be in- To assess the effect of the appearance of the other party
terpreted in two ways; the effect of the target’s appearance on IC in whole-body interactions, we examined examples
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1031
Sakurai, S. et al.
of whole-body interactions that met the following four re- 3.2. Twister
quirements: Twister is a competitive exercise game with a win/lose
(i) A mode of interaction that does not lead to a sense of outcome, but it contains elements of entertainment. Mul-
rejection or resistance and in which any person can tiple players stand on a mat with several rows of circles,
participate. where each row has a different color, and the players take
turns placing their hand or foot on a circle of a certain
(ii) One in which physical contact is not the objective color, as determined by the outcome of a spinner. In a
itself but is required. two-player game, the player who falls first loses the game.
While there have been no studies on how the appear-
(iii) One whose outcome does not rely on physical capa- ance of the opponent affects IC or the relationship be-
bility. tween the players in a Twister game, it is not difficult to
(iv) One with virtual contact but that does not require imagine that the impressions or image of the opponent
physical haptic feedback. would affect the power dynamics or the game outcome.
For example, when an adult male is playing against an
(i) prevents the sense of rejection or resistance in the opponent such as a woman, child, or elderly person, etc.,
whole-body interaction from being misattributed as the who is apparently physically weaker than himself, it is
impressions or internal characteristics of the opponent. unlikely that he will try to beat that opponent in earnest.
(ii) intents to investigate the sense of pleasure/displeasure In such cases, he will most likely observe the opponent’s
or intent of acceptance or avoidance of proximity and posture, guess the opponent’s intent, thoughts, and phys-
physical contact stemming from the opponent’s appear- ical ability, and be attentive to avoid cornering the oppo-
ance and to examine subconscious effects of the accep- nent into an excessively difficult posture by controlling
tance or avoidance on the psychological and physical re- his own movements or strength. Conversely, if the oppo-
lationship between users. (iii) and (iv) were selected to nent’s appearance is manipulated so that it looks weaker
remove the effects of physical ability, such as physical than the player, it should be possible to make the player
strength, and tactile information, including haptic percep- consider for the opponent.
tion, thus allowing examination of the effects of the oppo- In addition, the game of Twister gives rise to many
nent’s appearance only. opportunities of making physical contact, since it re-
Whole-body interactions, such as hugging and sexual quires players to control their whole body within a lim-
acts, normally express intimacy and take place only be- ited movement range. Here, the intent to accept or avoid
tween those who are close to or trust each other. In these the opponent’s approach or contact or the sense of plea-
cases, acceptance of the act is based on the relationship sure/displeasure likely affects one’s selection of the par-
itself, and as such, they fail to meet requirements (i) and ticular circle to land one’s hand or foot and thus change
(ii). the physical distance or posture. Since this selection is
Martial arts, such as sumo or judo, and sports that re- accompanied by changes in posture, body motion, muscle
quire a partner, such as coordinated group gymnastics or tension, and other factors, it directly affects the difficulty
warm-up exercises, are also examples of whole-body in- of maintaining posture.
teractions. They can be performed in a lighthearted man- If the psychological and physical relationship is ad-
ner if not taken too seriously, while physical contact is a justed by impression formation or the psychological ap-
means rather than an end, so they satisfy requirements (i) proach or contact based on the opponent’s appearance,
and (ii). However, physical contact is essential in these as stated above, the power balance or game outcome be-
cases. Since the participants’ physical abilities directly tween players can perhaps be changed by manipulating
affect the outcome (i.e., winning or losing) of the whole- the players’ appearances.
body interaction in martial arts, they do not fully meet
requirements (iii) and (iv).
We selected Twister [a] as a whole-body interaction 3.3. Implementation of VR Twister
that satisfied all of the above requirements. In the game To evaluate the effect of the opponent’s appearance
of Twister, two or more participants take turns and move on interpersonal cognition in the whole-body interaction
their body parts according to the given instruction within a taking place in a game of Twister, we developed a VR
restricted area, and the party who falls first loses (the rules Twister. VR Twister is a system that allows two play-
will be described later in more detail). In Twister, physical ers who are remotely located to play a game of Twister
contact is neither a means nor an end but is incidental to in a VR environment by using avatars. The system con-
the game process. Furthermore, being characterized as a sisted of a motion capture system (OptiTrack Motive 2.2,
party game, physical ability is not a determining factor of NaturalPoint Inc.), a head mount display (HMD) (Vive
the outcome. The key factor that determines whether one Pro, HTC Corp., resolution: 2880 × 1600, field of view:
loses or wins is the ability to maintain one’s posture. We 110◦ , refresh rate: 90 Hz), and a PC (CPU Intel i9-9900K
felt that a game of Twister will be possible as long as the 3.60 GHz; GPU NVIDIA GeForce RTX 2070 Super
participant is capable of sensing his or her contact with GDDR6 8 GB; RAM DDR4 16 GB), each set of which
the floor, and thus we chose it as a model of whole-body was installed in two separate rooms (Fig. 1) and was op-
interaction. erated at 90 fps to match the HMD refresh rate, using a
1032 Journal of Robotics and Mechatronics Vol.33 No.5, 2021
Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
Fig. 1. System configuration of VR Twister.
Fig. 3. Environment of VR Twister. Left: view of the VR
Twister mat from above. Right: mannequin for instruction
of next move.
face so that the face of the opponent’s avatar would come
into view even when the player looked down to check the
color and position of the circle or assumed a posture in
which his hands and feet were touching the floor. A man-
nequin positioned beside the mat (Fig. 3 (right)) and au-
tomatic readout were used to instruct the hand or foot to
be moved and the color of the circle on which to place
that hand or foot. When the Unity program detected a
collision between any one of the circle and a hand or foot
of the avatars, the program proceeded to give the next in-
struction. To ensure that the avatar limbs made contact
with the floor, the avatar bases were positioned 10 cm be-
low the floor level.
This system provided no haptic feedback even when the
avatars came into contact with each other so that the play-
ers relied only on visual perception. Haptic information
Fig. 2. Components of the motion tracking system. was limited to that of the floor when the player actually
touched it with a hand or foot.
program written with Unity2019.4.25f1. The motion cap-
ture system in each room employed 12 motion capture
4. Experiment: Evaluating the Effect of the
cameras (OptiTrack Flex13×18 and Prime 13×18, Natu- Appearance of the Opponent’s Avatar on
ralPoint Inc.) to track the positions of the reflective mark- the Players’ Interpersonal Cognition in VR
ers attached to the motion capture suit and HMD worn Twister
by each player, thus acquiring the player’s motion data
(Fig. 2). The purpose of this experiment was to investigate the
The two motion capture systems were connected via a effect of the appearance of the avatar used by the opponent
network, and each player’s motion data transmitted via PC in VR Twister on the players’ IC, including the psychol-
was reflected in the avatar used by each player in real time. ogy of approach and contact. In addition, we investigated
We adjusted the position and rotation of the HMD tracked the effect of the technical specifications of the VR Twister
by the motion capture system to follow the movement of environment on the senses of the player’s body and move-
the avatar’s head. This design allowed the player to freely ment.
move the entire body of his or her avatar as if it were the
player’s own body in the VR space, while viewing avatars
of the player and the opponent from the first-person view. 4.1. Experiment Design
This system did not scale the avatars to fit the individual In this experiment, we set the task in which the par-
player. There were, therefore, in slight discrepancies be- ticipant and experimenter played a two-player game of
tween the body movement of the player and that of avatar VR Twister. We asked participant to evaluate his or her
when the two were of substantially different sizes. IC based on the appearance of the avatar used by the ex-
In the VR space created with Unity, a mat on which perimenter, including impressions and the psychology of
four rows of circles – each row colored red, blue, yellow, approach or contact.
and green – were arranged was placed at the floor cen- The experimenter’s avatar was given one of four con-
ter (Fig. 3 (left)), simulating the actual Twister mat [a]. ditions based on two attributes: gender (man/woman) and
The mat was 110 cm × 160 cm in size, and the circles had rendering (toon/real) (Table 1). According to previous
a diameter of 20 cm. The mat was given a mirror sur- studies [10, 17–23] and the remarks given in Section 3.2,
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1033
Sakurai, S. et al.
Table 1. Conditions of appearance of the opponent’s (ex-
perimenter’s) avatar.
5HQGHULQJ
7RRQ
5HDO
0DQ
*HQGHU
07&RQGLWLRQ +HLJKW FP 05&RQGLWLRQ +HLJKW FP
Fig. 4. Avatar used by participants (Ex MT condition).
:RPDQ
ticipants to evaluate their IC and impressions about the
overall game (hereafter simply referred to as interpersonal
cognition assessment) in a questionnaire. The question-
naire consisted of 16 questions: overall impressions of the
opponent (Q1), psychology of approach and contact (Q2–
:7&RQGLWLRQ +HLJKW FP :5&RQGLWLRQ +HLJKW FP
Q6), psychology related to adjustment of relationship (Q7
and Q8), consistency between appearance and behavior
(Q9), inference of the opponent’s IC (Q10–Q13), and
gender is expected to affect both IC and the psychology overall impression of the Twister game (Q14–Q16). We
of approach and contact. These effect of gender in out- also carried out a questionnaire survey on the assessment
ward appearance can be expected to bring about changes of VR Twister specifications (hereafter referred to as VR
in the positional relationship and power dynamics be- Twister assessment). Details about the questions in the
tween the players as well as the game outcome. Rendering questionnaire are described later.
was selected to evaluate the effect of the avatar’s realism,
based on the recognition that recent social VR services
(VRChat [b], Cluster [c], etc.) often employ renderings 4.2. Experiment Procedure
based on animation or cartoon characters. The participants were first invited to the authors’ lab,
The experimenter’s avatar was given one of the above where they were briefed about the experiment and its ob-
four conditions for a given trial. Here, the experimenter’s jective. They were told that the objective of the exper-
IC and the behavior should be also changed if the appear- iment was to “investigate the effect on players in VR
ance of the participant’s avatar changed. For the purposes Twister using avatars”; this made it more abstract so that
of this experiment, it was necessary to make the appear- advance knowledge of specific objectives would impart
ance of the participant’s avatars use the same in order to bias to the results. Only those who agreed to participate
avoid the change in the experimenter’s behavior, which were selected as participants in the experiment.
should affect the participant’s IC. On the other hand, in The participants were each asked to put on the motion
addition to the “uncanny valley” phenomenon [27, 28], it tracking suit and HMD before the experiment began, and
has been reported [35] that one or group of entity who adjustments were made so that the avatar used by the par-
has the same appearance as the user in a VR space makes ticipant and the HMD line-of-sight correctly followed the
the user feel discomfort and eerie. When the partici- whole-body motions of the participant. After adjustments
pant’s avatar has given a neutral appearance such as a were made, as the practice phase, the participant was told
mannequin or an appearance the same as the avatar of the to make 12 moves of the Twister game in a VR space
experimenter, these uncomfortable senses were generated that displayed the participant’s avatar and the Twister mat
and could be misattributed to the experimenter’s impres- (Fig. 3 (left)) to familiarize him with movements using the
sion. The participant’s avatar was therefore given the ap- avatar. Following the practice phase, we instructed the
pearance based on the MT condition (Ex MT condition) participant to play Twister in VR space against an oppo-
(Fig. 4) in all trials. nent for a certain number of turns until he was instructed
For avatars with the MT, FT, and Ex MT conditions, that the game was over. Specifically, one trial consisted
the default avatars of Vroid Studio (80.12.0) [d] were of 36 turns per player, although the participant was not
used. The avatars with MR and FR conditions were pro- notified of this number. In addition, we instructed that
duced using MakeHuman (1.1.0) [e] and Mixamo [f], re- he was allowed to place more than a single limb in the
spectively. Following each game task, we asked the par- same circle, place a hand or foot on a circle already oc-
1034 Journal of Robotics and Mechatronics Vol.33 No.5, 2021
Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
A total of eight participants, consisting of male under-
graduate and graduate students between the ages of 21 and
25, participated in the experiment based on the within-
subject design. The experiment was performed with the
approval of the ethics committee of The University of
Electro-Communications to which the authors belong (ap-
proval No.20061). The experiment was terminated when
the eight participants had completed the experiment fol-
lowing the declaration of a state of emergency due to
COVID-19.
4.3. Results
The questions and results of the IC assessment ques-
Fig. 5. View of VR Twister from above. tionnaire are shown in Fig. 6. The results for each avatar
condition were subjected to the Shapiro-Wilk test to test
for normality in each question, and it was found that
Q6 (W = 0.72, p < .01), Q7 (W = 0.74, p < .01), Q10
(W = 0.76, p < .05), Q12 (W = 0.46, p < .05), and Q13
cupied by the opponent, and take time out to correct his
(W = 0.78, p < .05) did not display normality.
posture if it became difficult to maintain posture and then
The results for the various avatar conditions for those
continue playing. These actions are not allowed by the
questions that satisfied normality were subjected to a
original Twister rules. However, we adopted these action
two-way factorial analysis of variance (repeated measure-
as special rules based on the recognition of the difficulty
ment). The results showed that the main effect of gen-
of maintaining an unstable posture for an extended time
der was statistically significant for Q8 (F(1, 7) = 7.14,
period while wearing the HMD.
p < .05, ηG2 = 0.152) and marginally significant for Q4
After giving the above instructions, the experimenter
moved to another room, where he put on the motion cap- (F(1, 7) = 4.36, p < .10, ηG2 = 0.238) and Q9 (F(1, 7) =
ture suit and HMD as the participant did, and checked 3.63, p < .10, ηG2 = 0.046). Meanwhile, the main ef-
the avatar’s movements. Then, the participant and experi- fect of rendering was significant for Q1 (F(1, 7) = 45.37,
menter took up their starting positions (the centers of the p < .001, ηG2 = 0.558), Q2 (F(1, 7) = 15.65, p < .01,
two long sides of the mat) in a standing posture and began ηG2 = 0.239), Q4 (F(1, 7) = 13.82, p < .01, ηG2 = 0.197),
the game task as soon as the system gave the instruction to Q5 (F(1, 7) = 5.92, p < .05, ηG2 = 0.093), Q15 (F(1, 7) =
begin. A view of the VR Twister game in the experiment 6.21, p < .05, ηG2 = 0.059), and Q16 (F(1, 7) = 11.41,
is shown in Fig. 5. The sequence of the color of the cir- p < .05, ηG2 = 0.063), while it was marginally signifi-
cle and the hand or foot to place was set to be in random cant for Q3 (F(1, 7) = 5.40, p < .10, ηG2 = 0.052) and
order for both the experimenter and participant. In this Q9 (F(1, 7) = 5.42, p < .10, ηG2 = 0.030).
experiment, a 23-year-old male university student served The assessment results for the avatar conditions for
as the experimenter. The participants were not given any those questions in which normality was not satisfied
information about the identity of the experimenter. were subjected to a two-way factorial analysis of vari-
After each game task, the participant removed the ance (repeated measurement) based on aligned rank trans-
HMD and filled in the questionnaire on IC mentioned ear- form (ART). This showed that the main effect of gender
lier (Fig. 6 in Section 4.3). A single trial consisted of a was significant for Q6 (F(1, 21) = 4.73, p < .05, ηP2 =
Twister game, filling the questionnaire, and a five-minute 0.184) and Q12 (F(1, 21) = 7.18, p < .05, ηP2 = 0.255)
break, and four trials were conducted for each participant. and the main effect of rendering was significant for Q6
The appearance condition of the experimenter’s avatar (F(1, 21) = 6.34, p < .05, η p2 = 0.232) and marginally
used against the participants was randomly selected. After
significant for Q13 (F(1, 21) = 3.15, p < .10, η p2 =
completing the set of four trials, the participant was asked
0.130).
to fill in the VR Twister assessment questionnaire (Fig. 8
The assessment values (mean ± standard error) for the
in Section 4.3). In addition, he was asked to write in free
various conditions for questions in which the main effects
writing his overall impressions about the experiment.
of the factors were significant or marginally significant are
Responses to the questionnaires were based on the vi-
shown in Fig. 7. Since the analysis of variance involves
sual analog scale (VAS). Thus, the participants were asked
two factors and two levels, the significance of the main
to give a rating for each question by drawing a vertical
effect refers to the difference of the assessment scores be-
line on a 100 mm number line, the left and right ends of
tween the two levels of that factor. No mutual effect be-
which corresponded to the descriptions “not at all” and
tween the two factors was found in any of the questions.
“very much so,” respectively. The ratings were converted
The questions and results of the system assessment
to scores ranging from 0 to 100 points by converting the
questionnaire are shown in Fig. 8.
length in millimeters from the left end of the number line
to points (i.e., 1 mm = 1 point).
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1035
Sakurai, S. et al.
Fig. 7. Results of questions for which the main effect of
factors was significant or marginally significant according to
the avatar’s appearance condition (mean±SE). Above: gen-
der. Below: rendering.
Fig. 8. Results of the system assessment questionnaire. In
the notched box plots, the box partition lines indicate the me-
dians, the left and right boxes end the first and third quartile
points, the horizontal lines on the right and left indicate the
maximum and minimum values, and the crosses (×) indicate
the mean values.
Fig. 6. Assessment results of interpersonal cognition assess-
ment questionnaire (mean±SE).
1036 Journal of Robotics and Mechatronics Vol.33 No.5, 2021
Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
5. Discussion given information about the latter. However, it was found
in the interviews conducted after the experiment that all
5.1. Evaluation of Interpersonal Cognition the participants knew that their opponent had been the ex-
No interaction was found between gender and render- perimenter. In other words, the participants felt pleasure
ing in responses to questions of the interpersonal cog- when it appeared that they were being approached by a
nition assessment questionnaire. This indicates that the woman, even though they understood that the opponent
gender and rendering of the avatar’s appearance in VR was actually a man. This finding agrees with that of pre-
Twister independently affect the IC of male players. In vious studies [17–23]. The assessment of disgust at be-
this section, we discuss the effects of gender and ren- ing approached can also be explained by the participants’
dering on IC and pleasure/displeasure due to approach aversion to contact a male avatar.
or contact among male players according to the question With regard to rendering, the difference between real
blocks (Section 4.1). and toon was significant for Q2 (desire to approach oppo-
nent), with toon > real, marginally significant for Q3 (re-
sistance to approach), with toon < real, significant for Q4
5.1.1. Overall Impression
(happiness due to proximity), with toon > real, and sig-
The main effect of gender was not significant for Q1 nificant for Q5 (fear of approach) and Q6 (disgust due to
(overall impression of the opponent). However, rendering proximity), with toon < real. Thus, rendering affected the
was significant, with toon > real; thus, it can be concluded assessment of all questions under this grouping. These
that only the avatar’s realism affected impression forma- results indicate that the avatar’s realism in a VR environ-
tion. ment affected the sense of pleasure or displeasure due to
Comments found in the free-writing question follow- approach and contact, just as gender did. Furthermore,
ing the experiment suggested a connection between im- realism was found to have a greater effect than gender
pressions based on the avatar’s expressions and the ap- on producing a sense of pleasure or displeasure based on
pearance conditions. One participant stated that “it was virtual contact. The participants’ comments described in
scary when the opponent’s avatar approached because his Section 5.1.1 suggest that the impressions of the oppo-
expression was fixed,” regardless of the condition. Mean- nent contributed to resistance against contact (Q3) and
while, two other participants wrote, with regard to the fear of approach (Q5). Since gender was not a factor in
“real” rendering, that “it looked a bit frightening” and impression formation, however, the process of psycholog-
“I wanted to keep a certain distance because the realistic ical change due to contact and approach may differ for
avatar’s eyes seemed empty, and its expressions were hard gender and rendering.
to read.” Although the avatar’s expressions were fixed un-
der all conditions in this experiment, the appearance and
expressions of the “real” avatar did not match each other 5.1.3. Psychology of Relational Adjustment
in terms of realism. This seems to have produced a feeling Gender difference was significant for Q8 (concern
of wrongness or displeasure that can be explained by the over the opponent’s posture), where it was greater for
“uncanny valley” phenomenon [27, 28], which was then “woman.” However, the main effect of rendering was in-
imprinted as impressions of the opponent. Since expres- significant for this question. As stated in Section 5.1.2, the
sions strongly affect IC [6, 7], it is very likely that com- participants were aware that the opponent was actually a
pletely different assessments of the avatar’s impressions man. However, the male players were concerned about
would have resulted in any of the conditions if the avatar’s the opponent’s posture when the opponent appeared as a
expressions had been animated or had reflected the partic- woman. This suggests that, as stated in Section 3.2, the
ipants’ expressions. avatar’s appearance can be manipulated to incite concern
Moreover, one participant stated that a “tall person felt for the opponent and thus regulate the power balance.
intimidating,” although he did not specify the avatar’s
condition. Since an opponent’s imposing presence can 5.1.4. Consistency Between Impressions of
curtail a player’s intent to attack or interfere in a game Appearance and Behavior
of Twister, the design of the avatar’s height is likely to
The main effects of gender and rendering were
allow adjustment of the balance of power.
both marginally significant for Q9 (consistency be-
tween impressions of appearance and behavior), where
5.1.2. Psychology of Contact and Approach man > woman and toon < real, respectively. This indi-
Regarding to gender, the difference of the assessment cates that the appearance and behavior of an avatar with
scores between “man” and “woman” was marginally conditions of gender and realism that were clearly differ-
significant for Q4 (happiness due to proximity), with ent from those of the experimenter were judged to be in-
woman > man, and significant for Q6 (disgust due to consistent.
proximity) with woman < man. These results show that The low assessment rating obtained for this question
the gender of the opponent’s avatar only affected the sense as compared to the overall impressions or the positive as-
of pleasure or displeasure due to physical distance. As sessment of pleasure against approach and contact is con-
stated in Section 4.2, the participants and experimenter trary to the adaption-gap theory. According to this theory,
were all men in this experiment, and the former were not the user feels that the value of an interaction is reduced
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1037
Sakurai, S. et al.
when the impressions and functions expected of an ob- is highly probable that one or both of them influences the
ject (agent) of that interaction differ from those gained assessment results in the present experiment.
through actual interaction [36]. In this experiment, the
overall assessment of the Twister game was positive when 5.1.6. Overall Impression of the Twister Game
the conditions of gender and rendering differed from those
There was no difference in either condition in Q14 (dif-
of the experimenter (this is discussed in detail in Sec-
ficulty). This reason is probably due to the special rules
tion 5.1.6). The assessment obtained from this question
adopted to give a higher priority to the avoidance of an
suggests that, rather than the inconsistency between the
unstable posture rather than the game outcome.
avatar’s behavior and appearance, it was the avatar’s ap-
pearance that determined the value of the whole-body in- On the other hand, the difference in rendering was sig-
nificant for both Q15 (enjoyment) and Q16 (desire to con-
teraction taking place in VR Twister.
tinue playing), where toon > real. Since the main effect of
It must be noted, however, that the participants were
gender difference was not observed, it can be concluded
aware that the opponent was the experimenter. It can-
that the entertainment quality of Twister in this experi-
not be determined from this experiment whether the as-
ment depended on the avatar’s realism. A greater enjoy-
sessments were based only on the avatar’s appearance,
ment when playing Twister or the desire to continue the
whether they were affected by the impressions of the
game can be interpreted as a greater acceptance of the ap-
actual experimenter, or whether the impressions of the
avatar and experimenter were confused. proach or contact between players or a stronger intent to
become involved. In fact, “toon” was evaluated as more
fun than “real” with regard to the psychology of approach
5.1.5. Inference of the Opponent’s Interpersonal and contact (Q2–Q6). However, it can also be interpreted
Cognition that the entertainment quality of Twister depended on the
pleasure or displeasure of contact and approach based on
Gender differences were significant for Q12 (percep- rendering.
tion of the opponent’s resistance to approach), where
man < woman. The rendering difference was marginally
significant for Q13 (perception of the opponent’s intent 5.2. Assessment of VR Twister
to interfere), where toon < real. These findings suggest In the system assessment questionnaire, it was found
that the opponent’s psychology was perceived differently that the need for haptic feedback (Q1) had a high median
when the appearance of the opponent’s avatar had been of 67.5, but the variance was very large. There were no di-
manipulated. The possible causes for these results are rect references to haptic perception in the post-experiment
that the experimenter’s psychological states or behaviors questionnaire essays. One participant, however, wrote
reflecting his psychology under different gender or ren- that “since the opponent’s information consisted only of
dering conditions, 1) were perceived by the participants visual ones unlike in real situations, my gaze often wan-
to be different or 2) had in fact been different. dered and the opponent’s presence tended to become di-
If we assume the cause to be 1), this means that the luted when I had to assume a difficult posture.”
avatar’s appearance and its psychology or the psychology On the other hand, another participant expressed his ap-
based on its behavior were interpreted in a similar man- proval stating that “only in VR is it possible for the players
ner by the participants. Stated differently, the IPTs were to have overlapping bodies.” The presence or absence of
similar. However, it is difficult to believe that the partic- haptic feedback is likely to affect the perception of sharing
ipants had all made the connection between gender and space with an opponent or the reality of Twister. However,
resistance to contact as well as the connection between the former comment does not explicitly state a clear need
realism and the opponent’s intent to interfere. or reason for haptic feedback, and it will be necessary to
If, on the other hand, we assume the cause to be 2), the gather more views to arrive at any conclusions.
Proteus effect is likely to be an important factor. The Pro- The assessments of sense of agency (Q2) and sense of
teus effect refers to the phenomenon in which the appear- ownership (Q3) both had medians of 67.5, with small vari-
ance of the player’s avatar affects the player’s psychology ances. Therefore, it appears that the participants were able
or movements [37, 38]. The manner in which the Proteus to establish a certain degree of senses of agency and own-
effect can influence IC or the sense of pleasure or displea- ership. These sensations are mainly produced by the inter-
sure caused by contact has not been sufficiently clarified. action between vision and somatosensory perception [39,
However, if the experimenter had experienced pleasure or 40]. Therefore, the assessment of these two items will
displeasure of contact as a man when playing as a male depend on factors such as the haptic feedback that ac-
avatar and as a woman when playing as a female avatar, companies virtual physical contact, the difference in the
it is quite possible that he displayed different behaviors outward features and body build between the player and
based on the sense of pleasure or displeasure. The fact avatar, and whether the avatars’ scales have been adjusted
that the participants’ perceptions of the experimenter’s be- to conform to individual players. The outliers are perhaps
havior or psychology interpreted from that behavior dif- due to tracking errors, which will be described later.
fered would then be understandable. The discomfort and unnaturalness of playing Twister
These remarks are only speculative. However, IPT and wearing an HMD (Q4) had a low median of 23 with a very
the Proteus effect can both affect the process of IC, and it small variance. This assessment result indicates that the
1038 Journal of Robotics and Mechatronics Vol.33 No.5, 2021Effect of Opponent’s Appearance on IC in VR Whole-Body Interaction
participants were not that uncomfortable playing Twister male player plays a VR Twister game against a female
with an HMD. The participant who achieved the high- player. We assume that the Proteus effect comes into play
est score stated that “it was difficult to see whenever I in VR Twister and affects IC. If this assumption is valid,
moved my foot since I hadn’t strapped on the HMD tight the present experiment carried out with female players
enough.” His sense of unnaturalness would have been re- should yield results similar to the present ones. Further-
duced if he had properly worn the HMD so that it would more, if both players are affected by the Proteus effect, the
not move about. IC of the players and their relationship will differ greatly
Meanwhile, VR sickness due to tracking disruptions depending on the particular combination of avatar appear-
(Q5) had a relatively high median of 54.5, although the ances.
variance was large. The written essays of two partic- Meanwhile, self-awareness (awareness of one’s own
ipants in the post-experiment questionnaire also stated characteristics) and metacognition (awareness of one’s
that “realism was lacking due to collapse of the model, own thought or cognitive processes), such as self-schema
which diluted the sense that I was engaged in a one-to- or perspective-taking, also come into play in IC. Self-
one body communication,” and “it was scary when the schema is the relatively enduring and stable cognitive im-
opponent’s face became shifted,” suggesting the need to age of oneself formed from one’s physical features, abil-
improve the tracking accuracy. The reason for tracking ity, character, etc. [41]. Perspective-taking is to put one-
disruption would be that the optical motion capture sys- self in someone else’s position and imagine how that per-
tem used for VR Twister was unable to track the reflec- son feels [42]. The image formed in such cases can be de-
tive markers used for motion tracking when they were oc- fined in terms of two aspects: sensory-visual and concep-
cluded by other body parts. Not only can tracking disrup- tual [43]. The other person’s characteristics can be judged
tion reduce the senses of agency and ownership, but the in relative terms by comparison with one’s own charac-
resulting VR sickness or discomfort may be misattributed teristics. For example, the player’s own behavior against
to the impression of the opponent. For this reason, we in- an opponent will vary considerably depending on whether
tend to investigate a method, such as the use of a magnetic the player perceives that he or she is weaker than the op-
motion capture system, that will make it possible to track ponent, even when the opponent feels that the player is
the whole body in a stable manner for any kind of body stronger, or whether the player judges that the opponent
posture. perceives himself/herself to be weaker. If, in addition, the
Proteus effect influences IC, it is also likely to affect self-
perception and metacognition.
5.3. Feasibility and Issues of the Proposed Method Based on the above considerations, it will be necessary
We have shown that, for male players in VR Twister, to examine how manipulation of the two players’ appear-
IC, which affects the relationship between players includ- ances in whole-body interaction will affect IPT, the Pro-
ing the psychology of approach or contact, can be affected teus effect, self-perception, and metacognition, and how
by the opponent’s avatar gender and realism (rendering). these in turn will affect IC, in order to obtain firm evi-
While the results of the experiment show that it is possi- dence that the proposed method is indeed realizable. In
ble to realize the proposed method, they also point to the these investigations, it is essential to maintain total se-
need for further investigation and verification. crecy about the opponent’s information. This is because,
The second participant whose comment was quoted in since the participants had become aware that the oppo-
Section 5.1.1, regarding his impressions on the expres- nent had been the experimenter, it is not possible to de-
sions of a “real” opponent, also wrote that “since I had termine whether the assessments obtained in the present
to concentrate on the actions to take during my turn when experiment were based on information on the avatar or
playing Twister, I didn’t have the leisure to think about the the experimenter himself and also because of the possibil-
distance between the opponent.” However, it is unusual in ity that the assessments were intentionally distorted. Fur-
face-to-face communication or when making partial body thermore, in the future, we intend to investigate whether
contact for a person to be unaware of the other person’s the proposed method can be realized according to both
body. For this reason, it will be necessary, if we are to qualitative and quantitative assessments. Specifically, we
realize the proposed method targeted at whole-body inter- plan, in view of previous studies [14, 15, 33, 34], to com-
actions, to investigate the effects of the outward features pare and investigate the effects of manipulating players’
of body parts that are visible even in unstable postures, appearances on behavior in the game of Twister with the
such as body build, musculature, skin texture, etc. original rules, such as posture, body arrangement, posture
Furthermore, we still do not know why the appear- sustaining time, and the game outcome. We also hope
ance conditions used in this experiment, particularly the to resolve the technical issues of VR Twister, as noted in
avatar’s realism, affect IC. Investigation of this topic will Section 5.2.
require an experimental design that incorporates a process Finally, it should be noted that in the present experi-
to verify the IPT of the avatar’s appearance. ment, we were unable to secure a sufficient number of
IPT is also related to the Proteus effect, as mentioned participants that would guarantee the testing power of the
in Section 5.1.5. This is because the Proteus effect is con- experimental design. Thus, the possibility remains that,
sidered to derive from the psychology of trying to behave with regard to some questions, interactions between con-
in a manner that fits one’s appearance. Suppose that a ditions or main effects that are significant, which were not
Journal of Robotics and Mechatronics Vol.33 No.5, 2021 1039You can also read
