Human operator cognitive availability aware Mixed-Initiative control
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Human operator cognitive availability aware Mixed-Initiative control
Giannis Petousakis1 , Manolis Chiou2 , Grigoris Nikolaou1 , and Rustam Stolkin2
Abstract— This paper presents a Cognitive Availability suggestions [8]. Hence, the agents lack the ability to override
Aware Mixed-Initiative Controller for remotely operated mobile each other’s actions. Additionally, the robot’s AI initiative
robots. The controller enables dynamic switching between is often limited within a specific LOA, e.g. in safe mode
different levels of autonomy (LOA), initiated by either the AI
or the human operator. The controller leverages a state-of-the- [7]. Research on MI systems that are able to switch LOA
art computer vision method and an off-the-shelf web camera dynamically is fairly limited. Moreover, some of the MI
arXiv:2108.11885v1 [cs.RO] 26 Aug 2021
to infer the cognitive availability of the operator and inform systems proposed are not experimentally evaluated e.g. [9].
the AI-initiated LOA switching. This constitutes a qualitative Our previous work [10] proposed a novel AI ”expert-
advancement over previous Mixed-Initiative (MI) controllers. guided” MI controller and identified some major challenges:
The controller is evaluated in a disaster response experiment,
in which human operators have to conduct an exploration the conflict for control between the operator and the robot’s
task with a remote robot. MI systems are shown to effectively AI and the design of context aware MI controllers. In this
assist the operators, as demonstrated by quantitative and paper we extend the MI controller proposed in [10] in
qualitative results in performance and workload. Additionally, order to tackle the above mentioned challenges by using
some insights into the experimental difficulties of evaluating information on the cognitive availability of the human oper-
complex MI controllers are presented.
ator and by improving the controller’s assumptions. We call
I. I NTRODUCTION our approach Cognitive Availability Aware Mixed-Initiative
In this paper we follow a Variable Autonomy (VA) ap- (CAA-MI) control. Cognitive availability indicates if the
proach in order to improve Human-Robot Systems (HRS) operator’s attention is available to focus on controlling the
that are being increasingly used in high risk, safety-critical robot.
applications such as disaster response. VA in this context In our work, cognitive availability inference is based on
refers to systems with autonomous capabilities of varying the operator’s head pose estimation provided by a state-of-
scales. the-art deep learning computer vision algorithm. A low cost,
The task of manually controlling the robot combined off-the-self webcamera mounted on the Operator Control
with various operator straining factors, increase the cogni- Unit (OCU) provides video streaming to the computer vision
tive workload of the operator [1]. Moving towards robotic algorithm in real time. The head pose estimation provides
systems that can actively assist the operators and reduce input to the fuzzy CAA-MI controller where the cognitive
their workload can lead in reduced errors and improved availability inference and the LOA switching decisions are
performance [1], [2]. The advantage of such HRS lies in made. Somewhat related to our paper is the work of Gateau
the complementing capabilities of humans and Artificial et al. [8] which uses cognitive availability to inform decisions
Intelligence (AI). In a VA system control can be traded on asking the operator for help. Gateau et al. [8] uses
between a human operator and a robot by switching between a specialized eye tracker and does not involve cognitive
different Levels of Autonomy [3], e.g. the robot can navigate availability informed LOA switching.
autonomously while the operator is multitasking. Levels This work is explicitly making use of operator’s cognitive
of Autonomy (LOAs) refer to the degree to which the availability in MI controller’s LOA switching decision pro-
robot, or any artificial agent, takes its own decisions and cess and is contributing by: a) extending the MI controller
acts autonomously [4]. This work addresses the problem of in [10] to explicitly include cognitive availability and active
dynamically changing LOA during task execution using a LOA information; b) including those parameters provides
form of VA called Mixed-Initiative control. Mixed-initiative a qualitative advancement that tackles assumptions of the
(MI) refers to a VA system where both the human operator original controller; c) providing proof of concept on using
and the robot’s AI have authority to initiate actions and to state-of-the-art computer vision methods for informing MI
change the LOA [5]. control on operator’s status.
In many VA systems found in literature the LOA is chosen II. E XPERT- GUIDED M IXED -I NITIATIVE CONTROL WITH
during the initialization of the system and cannot change COGNITIVE AVAILABILITY INFERENCE
on the fly [6], [7]. Other systems allow only the operator
to initiate actions, although these are based on system’s The MI control problem this paper addresses is allowing
dynamic LOA switching by either the operator or the robot’s
1 University of West Attica, Greece gspetou@gmail.com, AI towards improving the HRS performance. In this work we
nikolaou@uniwa.gr assume a HRS which has two LOAs: a) teleoperation, where
2 Extreme Robotics Lab (ERL) and National Center for Nuclear
Robotics (NCNR), University of Birmingham, UK {m.chiou, the human operator has manual control over the robot via a
r.stolkin}@bham.ac.uk joystick; b) autonomy, where the operator clicks on a desired
B. Cognitive availability via head pose estimation
In this work we consider the information on whether the
operator is attending or not at the Graphical User Interface
(GUI) to be directly relevant to his current cognitive avail-
ability. The CAA-MI controller presented here infers the
operator’s cognitive availability by monitoring the operator’s
head pose via a computer vision algorithm. By introducing
cognitive availability perception capabilities to the HRS
system, we make a qualitative change to the robot’s ability to
perceive information about the operator and his availability
to control the robot.
Fig. 1. The block diagram of the CAA-MI control system. By using a computer vision technique for inferring the
cognitive availability of the operator we avoid intrusive
methods such as biometric sensors (e.g. EEG) and the use
location on the map with the robot autonomously executing of cumbersome apparatus. Our aim is to keep the approach
a trajectory to that location. generalizable and versatile. The latest head pose estimation
Operator’s initiated LOA switches are based on their techniques are robust in real time use and do not require
judgment. Previous work [3], [11] showed that humans are specialized image capturing equipment [12], [13], contrary
able to determine when they need to change LOA in order to to most state-of-the-art gaze tracking implementations (e.g.
improve performance. AI initiated LOA switches are based [14]). We chose head pose estimation over gaze tracking
on a fuzzy inference engine. The controller’s two states preferring perception of more pronounced natural cues.
are: a) switch LOA; b) do not switch LOA. The controller Our focus is on using a robust CV algorithm to provide
uses four input variables: a) an online task effectiveness cognitive availability input to the controller rather than eval-
metric for navigation, the goal-directed motion error; b) uating the various CV algorithms. The Deepgaze algorithm
the cognitive availability information provided via the deep [12] is used in our system for head pose estimation. It makes
learning computer vision algorithm; c) the currently active use of state-of-the-art methods such as a pre-trained Convo-
LOA; d) the current speed of the robot. Please refer to Figure lutional Neural Network (CNN), is open source and actively
1 for the block diagram of the system. maintained. In our implementation, deepgaze measures the
operator’s head rotation on the vertical axis (yaw). The head
A. Goal-directed motion error rotation was filtered through an exponential moving average
(EMA) [15] in order to reduce noise in the measurements.
The CAA-MI controller uses an expert-guided approach
The parameters of the EMA were calculated by analyzing
to initiate LOA switches. It assumes the existence of an AI
data regarding the attention of the operator in the secondary
task expert that given a navigational goal can provide the
task in previous experiments. Lastly, baseline data on what
expected task performance. The comparison between the run-
values of rotation constitute attending or not at the GUI
time performance of the system with the expected expert
were gathered in a pilot experiment and in a standardized
performance yields an online task effectiveness metric. This
way. These were mapped into the fuzzy input for cognitive
metric expresses how effectively, the system, performs the
availability.
navigation task.
The controller’s task effectiveness metric is the goal- C. The fuzzy rule base
directed motion error (refereed to as “error” from now on) A fuzzy bang-bang controller is used with the Largest
and is the difference between the robot’s current motion (i.e. of Maxima defuzzification method. We have introduced the
speed in this case) and the motion of the robot required to active LOA and the Cognitive availability of the operator
achieve its goal according to an expert. To provide more as fuzzy inputs. A hierarchical fuzzy approach is used for
context we extract the expert motion performance from a the CAA-MI controller, meaning that the first in order rule
concurrently active navigation system which is given an activated has priory over the others. The cognitive availability
idealized (unmapped obstacle and noise-free) view of the of the operator and the active LOA take precedence in the
robot’s world. In essence, the expert provides an idealized fuzzy rule activated. The rules added to the original MI
model of possible robot behavior that can be seen as an upper controller make sure that when the operator is not attending
bound on system performance. the screen the robot will operate in autonomy, with the rest
Additionally, our controller encodes expert knowledge of the rule base architecture being similar to the previous MI
from human operators data learned using machine learning controller.
techniques from a previous experiment [3] on: a) what is
considered to be a large enough error to justify a LOA III. E XPERIMENTAL EVALUATION
switch; and b) the time window in which that error needs This experiment evaluates the CAA-MI controller and
to accumulate. For detailed information the reader is encour- investigates its potential advantages over the previously built
aged to read our previous work [10]. MI controller [10] (referred simply as MI controller for thethe MI controller. Counterbalancing was used in the order
of the two controllers for different participants. Each par-
ticipant underwent extensive standardized training ensuring
a minimum skill level and understanding of the system.
Participants were instructed to perform the primary task as
quickly and safely as possible. They were also instructed that
(a) (b) when presented with the secondary task, they should do it as
quickly and as accurately as possible. They were explicitly
Fig. 2. 2(a): The experimental apparatus: composed of a mouse and a told that they should give priority to the secondary task over
joystick, a laptop and a desktop computer, a screen showing the GUI; a
web-camera mounted on the screen; and a laptop presenting the secondary the primary task and should only perform the primary task
task. 2(b): A typical example of the secondary task. if the workload allowed. Lastly, after every trial, participants
completed a raw NASA-TLX workload questionnaire.
IV. R ESULTS
A. Statistical analysis
The CAA-MI controller displayed a trend, though not
statistically significant (Wilcoxon signed-rank tests and t-
tests), of improved performance over the MI controller in
terms of lower number of LOA switches, more secondary
task number of correct answers and less perceived workload
(NASA-TLX), as demonstrated in table I.
TABLE I
Fig. 3. The GUI: left, video feed from the camera, the LOA in use and TABLE SHOWING DESCRIPTIVE STATISTICS FOR ALL THE METRICS .
the status of the robot; right, the map showing the position of the robot, the
current goal (blue arrow), the AI planned path (green line), the obstacles’ metric descriptive statistics
laser reflections (red) and the walls (black). The letters denote the waypoints primary task MI: M = 242 sec, SD = 21.8
to be explored. completion time CAA-MI: M = 243.5 sec, SD = 24.4
MI: M = 7.6, SD = 3.1
secondary task no.
CAA-MI: M = 8.5, SD = 3.5
of correct answers
Baseline: M = 8.6, SD = 3.1
rest of the paper). The Gazebo simulated robot was equipped secondary task MI: M = 84.2, SD = 15.6
accuracy (% of CAA-MI: M = 87.2, SD = 18.9
with a laser range finder and a RGB camera. The software correct answers) Baseline: M = 82.9, SD = 14.2
was developed in Robot Operating System (ROS) and is NASA-TLX
MI: M = 34.9, SD = 17.4
described in more detail in [10]. The ROS code for the CAA- (Lower score means
CAA-MI: M = 30.4, SD = 14.97
MI controller is provided under MIT license in our repository less workload)
number of MI: M = 6.4, SD = 3.8
[16]. The robot was controlled via an Operator Control Unit LOA switches CAA-MI: M = 5.5, SD = 3.4
(OCU) (see Figure 2(a)) including a GUI (Figure 3). The number of AI MI: M = 2, SD = 1.6
experiment’s test arena was approximately 24m × 24m. The LOA switches CAA-MI: M = 1.75, SD = 1.5
software used for the secondary task was the OpenSesame
[17] and the images used as stimuli were previously validated
for mental rotation tasks in [18]. B. Discussion
The participants were tasked with a primary exploration Performance towards the primary task was on the same
task and a cognitively demanding secondary task. The pri- level for both controllers. This can be attributed to the
mary task was the exploration of the area along a set of varied exploration strategies of the operators as supported
waypoints in order to identify the number of cardboard by anecdotal evidence. Additionally, it is possible that the
boxes placed on those waypoints. In the secondary task performance degradation periods did not last enough to
the operators were presented with pairs of 3D objects. The amplify the advantages of the CAA-MI controller compared
operators were required to verbally state whether or not the to the MI one. By increasing secondary task duration and
two objects were identical or mirrored. complexity, clearer insight to the advantages of the CAA-
Artificially generated sensor noise was used to degrade the MI controller can be provided. Given the above evidence,
performance of the autonomous navigation. The secondary a trade-off must be found between having a meaningful
task was used to degrade the performance of the operator. exploration task (i.e. to be used for system evaluation) and
Each of these performance degrading situations occurred minimizing variance in operator strategies.
once concurrently (i.e. overlapping) but at random. During MI controller trials and while the secondary task
A total of 8 participants participated in a within-groups overlapped with the sensor noise, the robot would switch
design in which every participant performed two identical from autonomy to teleoperation. This negatively affected
trials, one with each controller: the CAA-MI controller and operators’ focus on the secondary task with some of themexpressing their frustration. In contrast, this was not ob- R EFERENCES
served in the case of the CAA-MI controller since the robot [1] J. L. Casper and R. R. Murphy, “Human-Robot Interactions During
could perceive that the operator was unavailable and hence the Robot-Assisted Urban Search and Rescue Response at the World
maintained, or switched to, autonomy. Supporting evidence Trade Center,” IEEE Transactions on Systems, Man, and Cybernetics,
Part B: Cybernetics, vol. 33, no. 3, pp. 367–385, 2003.
can be found in the NASA-TLX frustration scale which [2] H. A. Yanco, A. Norton, W. Ober, D. Shane, A. Skinner, and
while not statistically significant, showed lower frustration J. Vice, “Analysis of Human-robot Interaction at the DARPA Robotics
in CAA-MI (M = 27.5, SD = 21.9) compared to MI Challenge Trials,” Journal of Field Robotics, vol. 32, no. 3, pp. 420–
444, 2015.
(M = 33.25, SD = 18.3). Participants also performed better [3] M. Chiou, R. Stolkin, G. Bieksaite, N. Hawes, K. L. Shapiro, and T. S.
on the secondary task. In conjunction with the above, a trend Harrison, “Experimental analysis of a variable autonomy framework
for lower LOA switches has been observed in the case of for controlling a remotely operating mobile robot,” IEEE International
Conference on Intelligent Robots and Systems, pp. 3581–3588, 2016.
the CAA-MI controller. This is a possible indication of more [4] T. B. Sheridan and W. L. Verplank, “Human and computer control of
efficient LOA switching compared to the MI as the operators undersea teleoperators,” MIT Man-Machine Systems Laboratory, 1978.
relied more on the CAA-MI system’s capabilities. [5] S. Jiang and R. C. Arkin, “Mixed-Initiative Human-Robot Interaction:
Definition , Taxonomy , and Survey,” in IEEE International Confer-
Lastly, note that our previous work [3], [10] has already ence on Systems, Man, and Cybernetics (SMC), 2015, pp. 954–961.
demonstrated that dynamic LOA switching (e.g. MI control) [6] C. W. Nielsen, D. A. Few, and D. S. Athey, “Using mixed-initiative
is significantly better than pure teleoperation or/and pure human-robot interaction to bound performance in a search task,” in
IEEE International Conference on Intelligent Sensors, Sensor Net-
autonomy. Hence, it can be inferred that the CAA-MI works and Information Processing, 2008, pp. 195–200.
controller is also advantageous over pure teleoperation or [7] D. J. Bruemmer, D. A. Few, R. L. Boring, J. L. Marble, M. C.
pure autonomy. Walton, and C. W. Nielsen, “Shared Understanding for Collaborative
Control,” IEEE Transactions on Systems, Man, and Cybernetics - Part
V. C ONCLUSIONS AND FUTURE WORK A: Systems and Humans, vol. 35, no. 4, pp. 494–504, 2005.
[8] T. Gateau, C. P. C. Chanel, M.-h. Le, and F. Dehais, “Considering
This paper presented a Cognitive Availability Aware Human’s Non-Deterministic Behavior and his Availability State When
Mixed-Initiative (CAA-MI) controller and its experimental Designing a Collaborative Human-Robots System,” in IEEE Interna-
evaluation. The CAA-MI controller’s advantage lies with the tional Conference on Intelligent Robots and Systems (IROS), 2016, pp.
4391–4397.
use of operator’s cognitive availability status into the AI’s [9] D. J. Bruemmer, J. L. Marble, D. D. Dudenhoeffer, M. O. Anderson,
LOA switching decision process. and M. D. Mckay, “Mixed-initiative control for remote characterization
Research in MI control often requires integrating different of hazardous environments,” in 36th Annual Hawaii International
Conference on System Sciences, 2003, pp. 9 pp.–.
advanced algorithms in robotics and AI. Thus, the use of [10] M. Chiou, N. Hawes, and R. Stolkin, “Mixed-Initiative variable
a state-of-the-art deep learning computer vision algorithm, autonomy for remotely operated mobile robots,” arXiv preprint
demonstrated proof of concept that such algorithms can 1911.04848, 2019. [Online]. Available: http://arxiv.org/abs/1911.
04848
provide reliable and low cost advancements to MI control. [11] M. Chiou, G. Bieksaite, N. Hawes, and R. Stolkin, “Human-Initiative
Experimental results showed that the CAA-MI controller Variable Autonomy: An Experimental Analysis of the Interactions
performed at least as good as the MI controller in the primary Between a Human Operator and a Remotely Operated Mobile Robot
which also Possesses Autonomous Capabilities,” AAAI Fall Sympo-
exploration task. Also trends found in the results (e.g. sium Series: Shared Autonomy in Research and Practice, pp. 304–310,
NASA-TLX frustration scale, number of LOA switches) and 2016.
anecdotal evidence indicate a more efficient LOA switching [12] M. Patacchiola and A. Cangelosi, “Head pose estimation in the wild
using Convolutional Neural Networks and adaptive gradient methods,”
in certain situations compared to the MI controller. This is Pattern Recognition, vol. 71, pp. 132–143, 2017.
due to the qualitative advantage that the operator’s cognitive [13] N. Ruiz, E. Chong, and J. M. Rehg, “Fine-grained head pose esti-
status information gives to the CAA-MI compared to the MI mation without keypoints,” IEEE Computer Society Conference on
Computer Vision and Pattern Recognition Workshops, vol. 2018-June,
controller. pp. 2155–2164, 2018.
Concluding, we identify the current experimental [14] J. Lemley, A. Kar, A. Drimbarean, and P. Corcoran, “Convolutional
paradigms as a limitation in testing the full potential of neural network implementation for eye-gaze estimation on low-quality
consumer imaging systems,” IEEE Transactions on Consumer Elec-
more complex MI systems. This is despite the recent tronics, vol. 65, no. 2, pp. 179–187, 2019.
advances on experimental frameworks [3]. Evaluating [15] R. G. Brown, Smoothing, forecasting and prediction of discrete time
MI Human-Robot Systems in realistic scenarios involves series. Courier Corporation, 1963.
[16] G. Petousakis and M. Chiou, “Cognitive availability aware
difficult intrinsic confounding factors that are hard to predict mixed-initiative controller code,” https://github.com/uob-erl/fuzzy mi
or overcome. Hence, further research should devise a more controller.git, 2019.
complex experimental protocol along the lines proposed in [17] S. Mathôt, D. Schreij, and J. Theeuwes, “OpenSesame: An open-
source, graphical experiment builder for the social sciences,” Behavior
the discussion, i.e. finding a trade-off between realism and Research Methods, vol. 44, no. 2, pp. 314–324, 2012.
meaningful scientific inference. [18] G. Ganis and R. Kievit, “A New Set of Three-Dimensional Shapes
for Investigating Mental Rotation Processes : Validation Data and
ACKNOWLEDGMENT Stimulus Set,” Journal of Open Psychology Data, vol. 3, no. 1, 2015.
This work was supported by the following grants of
UKRI-EPSRC: EP/P017487/1 (Remote Sensing in Extreme
Environments); EP/R02572X/1 (National Centre for Nuclear
Robotics); EP/P01366X/1 (Robotics for Nuclear Environ-
ments). Stolkin was also sponsored by a Royal Society
Industry Fellowship.You can also read
