RF Sensing for Continuous Monitoring of Human Activities for Home Consumer Applications
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
RF Sensing for Continuous Monitoring of Human Activities
for Home Consumer Applications
Moeness G. Amina , Arun Ravisankarb , and Ronny G. Guendela
a
Center for Advanced Communication, Villanova University, Villanova, PA 19085, USA
b
Comcast Labs, Comcast, Philadelphia, PA 19103, USA
ABSTRACT
arXiv:2003.09699v1 [eess.SP] 21 Mar 2020
Radar for indoor monitoring is an emerging area of research and development, covering and supporting different
health and wellbeing applications of smart homes, assisted living, and medical diagnosis. We report on a
successful RF sensing system for home monitoring applications. The system recognizes Activities of Daily
Living(ADL) and detects unique motion characteristics, using data processing and training algorithms. We
also examine the challenges of continuously monitoring various human activities which can be categorized into
translation motions (active mode) and in-place motions (resting mode). We use the range-map, offered by a
range-Doppler radar, to obtain the transition time between these two categories, characterized by changing
and constant range values, respectively. This is achieved using the Radon transform that identifies straight
lines of different slopes in the range-map image. Over the in-place motion time intervals, where activities have
insignificant or negligible range swath, power threshold of the radar return micro-Doppler signatures,which is
employed to define the time-spans of individual activities with insignificant or negligible range swath. Finding
both the transition times and the time-spans of the different motions leads to improved classifications, as it
avoids decisions rendered over time windows covering mixed activities.
Keywords: Radar, indoor monitoring, smart homes, motion classification
1. INTRODUCTION
With our aging population, and the desire for independence, the need to monitor Activities of Daily Living
(ADL) is a growing problem in society.1, 2 As life expectancy increases, so will the demand for elders to age in
place as opposed to relocating to assisted living environments. Aging in place describes empowering seniors with
the ability to live in ones own home, safely and independently. Since decline in health is inevitable, aging-in-
place requires increasing amounts of support to maintain an independent existence. The combination of sensing
technologies and data analytics are powerful tools in providing support to those caring for seniors who are living
independently. Sensors provide insights that characterize the potential for living a quantified life. To compress
morbidity, healthcare professionals will benefit from the insights provided from continuous monitoring of both
classified medical data, such as vitals, and lifestyle monitoring of activities of daily living. Activities of daily living
are the routine matters of our day, most notably including eating, sleeping, movement, hygiene and toileting.
Fitness and cognitive activities are also strong indicators of state of health.
Apart from the health and wellness applications, RF-based sensors promise non-intrusive, safe and efficient
home automation/security applications.3–11 Falls are considered as an abnormal activity that should be accu-
rately detected and classified with high sensitivity and specificity.12–14 Gesture recognition/control has great
potential in enhancing Man Machine Interface (MMI). The ability of radar-based sensors to precisely detect
motion as well as describe unique characteristics of a motion activity provides high fidelity sensor data to the
home security/automation system. The penetration of RF signals through walls gives added advantage. Radar
offers fine controls for sensing configurations (selectively choosing what to track; ignoring targets that are of no
interest). It works in all lighting and weather conditions.15–18 Human daily activities can be further categorized
into translation and in-place motions. Whereas the former mainly describes crawling and gait articulations, via
normal or abnormal walking, or running actions, the latter is primarily associated with motions that do not
exhibit considerable changes in range. In-place motions include sitting, standing, kneeling, and bending: each
is performed without any stride. The accumulated range under each in-place is typically less than 1 meter. Fall
can be considered a translation or in-place motion, depending on its type. Whereas progressive fall exhibits a
large range span, drop like fall is confined to a relatively small range extent. Examples of translation motions
are given in Fig. 2 which depicts two extremes of fast and slow walking speeds. Fig. 1 shows cases of in-place
motions of jumping, arm gestures, and sitting. Fig. 3 shows two types of falls pertaining to two different fall
categories.
The contribution of this paper is two-fold. We report the working of a successful RF sensor for home
applications, which is jointly developed by Villanova University with support from Comcast Labs, the advanced
technology arm of the company where ideas, technologies and user experiences are tested, evaluated and refined.
It is important to point out that from a Comcast Labs point of view, this is a technological test, not a product, and
if any productization of this test would only occur, it would only be with the full, opt-in consent of the individual
being monitored. That said, it is also our view that a radar-based approach for in-home monitoring plausibly
provides peace of mind for both the person being monitored, and the caregiver, for the following reasons: 1) The
intended customer can age in place at home for a longer period of time, without the use of cameras to monitor
activity and 2) Caregivers (often adult children) receive peace-of-mind, also without having to rely on cameras
to look in on their charge(s). We also introduce a simple method based on the Radon transform, applied to the
range-map, for separating translation from in-place motions. The range-map depicts the range vs slow-time of
a moving target. Accordingly, horizontal lines correspond to in-place motions, whereas translation motions are
manifested by lines with non-zero slopes. We also remark that, in the range-map, fast translation motions are
associated with steep lines. For example, running has higher slopes than walking, and walking has a higher slope
than crawling. Further, elderly gait typically leave behind smaller signature slopes compared to those of young
adults. Translation motion stemming from using a wheelchair is comparable with an average walking speed,
producing a similar range slope. Positive slopes describe range translation away from radar, and negative slopes
are a result of walking towards the radar. It is also noted that the range-map does not exhibit abrupt, sudden
step change behavior, like that of square waveforms. In essence, it consists of piece-wise straight lines with finite
slopes.
Figure 1: In-place motions, jumping (left) arm gesture and sitting (middle), kneeling (right)
Figure 2: Translation motions, running (left), walking (middle), slow walking (right)
Figure 3: Fall can be either considered a translation motion (top) or an in-place (bottom) motion
2. ACTIVITY CLASSIFICATION SYSTEM
With more than a decade spent by many researchers worldwide to develop algorithms and device motion and
fall classification techniques, it has become vital - now more so than ever before - to transition these efforts into
a working system. In 2018, Comcast Labs and Villanova University embarked on Phase-I of a project which
aimed at identifying key and effective algorithms that could be applied to radar for indoor monitoring, without
the use of cameras. The results were presented at the 2018-SPIE.19 The experiments discussed in that paper
included a variety of motions. The data was collected using a 24 GHz radar that was used to test a data-driven
feature-learning algorithm, namely, two-dimensional principal component analysis (2D-PCA),20–25 followed by a
nearest neighbor classifier. In essence, a complete motion classification application was built using both hardware
and software components. Fig. 4 shows a potential application of a radar-based sensor installed on the wall and
covering an area of the home. In this case, the radar monitors the human inside the home and provides motion
location and posture, and immediately displays this information on a wide screen. The system is also able to tell
if the person exhibits no movement/light movement/heavy movement. This information could prove important
for health providers and is useful in assessing deviations from average daily routines.
Figure 4: Radar application showing presence and postures.
Fig. 6 shows the image of the radar chip that was used in the project, which is now advancing to Phase-II.
The radar antenna is connected to a micro-controller that does basic processing and gives out range and Doppler
Figure 5: Steps involved in the activity classification Figure 6: Radar used for Activity classification appli-
project. cation.
data. This information is then processed by 2D-PCA for feature extraction. The classifier would then identify the
activity that is taking place in the radar arena. Activities include walking, sitting, standing, bending, gesture,
and falling.
Fig. 5 shows the steps involved in the project. In this system, the algorithm would detect motion and send
a message to an application that would display as a dashboard. The spectrograms are generated from the radar
returns. Feature extraction is followed by kNN classifiers to determine the activity. Activity detected by the
classifier is then sent to a python application which would be used to visualize the data. We also had gesture
control to turn on/off home appliances. A notification would be sent to smart assistants, like Alexa, when the
radar detects a fall. Fig. 7 shows this application, and depicts current and past activities, as well as gesture
control of smart home appliances.
3. RADON TRANSFORM FOR RANGE-MAP PROCESSING
The Radon transform is considered an effective tool in detecting dominant color schemes in images, especially in
medical image processing. The Radon transform is an integral transform method, while its inverse is typically
used for reconstructing images from medical CT scans.26 In the underlying problem, the goal is to apply the
Radon transform to the range-map of the radar signal returns to detect pertinent line structures. In employing
the Radon transform for human motion recognition problems, we recognize that horizontal lines in the range-
map correspond to in-place motions with no noticeable range extent, whereas lines with non-zero inclinations
represent continuous changes in range gates stemming from motion translations, such as constant speed walking,
crawling, or running. It is noted that acceleration or deceleration give rise to curvy signatures in the range-
map where Hough transform can be applied.27 In-place motions include (a) sitting, (b) kneeling down from a
standing position, (c) standing up from a kneeling or sitting position, and (d) bending while picking an object
from the ground. Successful separations between translation and in-place classes of motions helps in identifying
the ”actions” vs ”resting” modes at home. Information in the range-map is sufficient to label the human activity
as one of the two aforementioned classes. In this respect, any follow-on classification of human motions using
spectrograms for microDoppler signature estimation can proceed without confusing one class with another. For
example, classifiers applied to time intervals of in-place motions will not have walking as a possible outcome.
Under the proposed framework, we process the range-map and microDoppler representations sequentially with
the former preceding.
Figure 7: Fall detection notification sent to Alexa (Top Left), Application Detects Sitting (Top Right). Gesture
control tuning on smart light (Bottom)
3.1 Application of the Radon Transform
The range-map describes the target distance to the radar vs. slow-time. We consider the range-map as an image,
x(m, n) with each sample converted to a decibel, absolute value. The radar data collected produces an image of
size M ×N = 256×12, 000, where M and N represent the number of range bins (rows) and slow-times (columns),
respectively. An initial step is to remove signal power dependency on range by column-wise normalization of the
range-map. The strongest pixel xmax (n) of each column n is computed and used to divide all pixel values in
that column. This step allows equal treatments of all ranges, despite their distances from the radar. Fig. 8a and
8b show the range-map before and after normalization. In this case, the subject begins with translation motion,
through walking, then follows with two consecutive in-place motions, namely, sitting and standing. It is noted
that, in this example, the signal-to-noise ratio is considerably high such that a simple thresholding can be used
to ”clean” the image and highlight the target range signature. The normalized, resized, filtered, and thresholded
range-map image is shown in Fig. 8d. Referring to the range-map, the following equations describe the above
pre-processing steps. For normalization,
1
RMnorm (n) = RM (n) , n = 1, 2, ..., N (1)
xmax (m)
It is noted that the original image size of 256 × 12, 000 is excessively large, and thus can be reduced by down-
sampling. We perform uniform sub-sampling over slow-time, and produce a smaller image size, referred to
as RMds , of dimension Mnew × Nnew = 128 × 384. To improve the end result of the Radon transform, the
down-sampled image RMds is filtered with a (3 × 3) smoothing kernel, given by,
1 1 1
1
H= 1 1 1 (2)
9
1 1 1The filtered image is expressed as,
k
X k
X
RMf iltered (m, n) = H(i, j) · RMds (m + i, n + j) , m = 1, 2, ..., Mnew ; n = 1, 2, ..., Nnew (3)
i=−k j=−k
The final pre-processing step is applying a threshold. Values below a certain level in the above image equation
are set to zero, i.e.,
(
RMf iltered (m, n), if RMf iltered (m, n)≥ 0.75
RMth (m, n) = (4)
0, otherwise
After performing normalization, resizing, filtering, and thresholding, we apply the Radon transform to the image
in Fig. 8d.
(a) Original RM (b) Normalized RM
(c) Resized RM (d) Filtered and thresholded RM
Figure 8: Range-map pre-processing steps.
The Radon transform converts a Cartesian coordinate system (x, y) to an angle and a distance from a
center point representation (θ, x0 ). It integrates all pixel values over y 0 . In other words, each pixel is projectedonto the new subspace abscissa x0 . The projection point of every pixel on the x0 axis can be rewritten as
x0 = x · cosθ + y · sinθ. The Radon transform is graphically illustrated in Fig. 9. The computing algorithm
of Matlab divides each pixel into four subpixels. These subpixels are projected onto the subspace abscissa x0
separately. Matlab separates the abscissa x0 into bins. The subpixel which crosses a bin is counted with the full
value. On the other side, if a subpixel hits the border of two bins, it will be split between these bins evenly.28–30
The integration over the subspace ordinate axis y 0 is given by,
Z∞
0
Rθ (x ) = f (x, y)dy 0 (5)
−∞
with,
x0
cos θ sin θ x
= · (6)
y0 − sin θ cos θ y
The inverse transform (rotation) of axes is given by,
0
x cos θ − sin θ x
= · (7)
y sin θ cos θ y0
Accordingly, Eq. 5 can be written as,
Z∞
0
Rθ (x ) = f (x0 cos θ − y 0 sin θ, x0 sin θ + y 0 cos θ)dy 0 (8)
−∞
Figure 9: Radon transform operations
We apply the Radon transform to the above example, specifically to the range-map of Fig. 8d. The best
results were achieved by setting ∆θ = 1◦ . In examining the Radon transform in Fig. 10, it is clear that there is
a peak at θ = 90◦ angle, which refers to an in-place motion. The other peak off θ = 90◦ refers to a translation
motion. It is noted that the peak location defines both the angle θ and a distance from the origin x0 . According(a) Radon transform - 3D representation (b) Radon transform - 2D representation
Figure 10: The Radon transform
to this information, a linear function is computed by,
y = m(k) · x + n(k) (9a)
π
m(k) = tan( − θ(k)) = cot(θ(k)) (9b)
2
x0
n(k) = (9c)
sin(θ(k))
The index k = 1, .., K − 1 is the number of all detected motions. The slope m(k) as well as the intersection point
with the abscissa axis n(k) of each linear function y is computed by Eq. 9b and Eq. 9c, respectively. With all
linear functions in the range-map determined, the intersection point between the two consecutive lines, describing
translation and in-place motions, can be simply obtained by solving the two linear equations,
m(k) −1 x −n(k)
· = , k = 1, .., K − 1 (10)
m(k + 1) −1 y −n(k + 1)
Fig. 11a shows the intersection point of the two lines, plotted over the RMth of Fig. 8d as well as over the
spectrogram, shown in Fig.11b. The intersection time is very close to the true value when motion transitioned.
3.2 Power Burst Curve (PBC)
From the previous section, the separation between in-place and translation motions can be achieved by using
the Radon transform applied to the range-map. To determine whether there is one or a sequence (multiple) of
in-place motions, we need to examine the microDoppler signatures over the in-place motion interval. Referring
to the spectrogram S(n, k) in Fig. 11b, the subject walks towards the radar in a translation motion, followed
by sitting and standing activities, both of which represent in-place motions. In order to separate the two
consecutive in-place motions, where range information is no longer a factor, we deal with the radar scattering
as a deterministic signal, rather that a random process,31 and measure the rise and fall of the signal energy in
S(n, k) over slow-time which is known as the power burst curve (PBC).32, 33
The Doppler frequencies at and around the zero Doppler axis are not of interest, since their strong power gain
biases the PBC. In this work, the chosen frequency bands for power computation are between KP 1 = 20Hz and(a) RM with intersection point (b) MD with intersection time (red line)
Figure 11: Walking towards the radar followed by sitting and then standing.
KP 2 = 270Hz for positive Doppler frequencies and between KN 2 = −20Hz and KN 1 = −270Hz for negative
Doppler frequencies. We compute the PBC for the combined frequency bands by,
K
X P2 K
X N2
2 2
S(n) = |S(n, k1 )| + |S(n, k2 )| , n = 1, 2, ..., N (11)
k1 =KP 1 k2 =KN 1
In the above equation, the index n = 1, 2, ..., N represents the columns of the spectrogram. The PBC is drawn
in Fig. 12b.
The computations of Eq. 11 results in a fluctuating power curve stemming from intricate microDoppler
signatures of human motions. Such fluctuation may exhibit small values close to zero over a single motion time
extent. These values could mistakenly define wrong event boundaries.
(a) MD with Freq.-Bands (b) Unfiltered and filtered PBC
Figure 12: Illustration of PBC.
To mitigate the above problem, we apply a moving average filter. The window size is set to w = 5. Thefiltered PBC, Sf (n, k) is computed by,
1
Sf (n) = (S(n) + S(n − 1) + ... + S(w − n)) , n = 1, 2, ..., N (12)
w
The filtered PBC is plotted in Fig. 12b, and is used to determine the start and end of an activity. The threshold
has been found empirically as 3% over the minima (Sfmin + 0.03 · (Sfmax − Sfmin )). Based on the this threshold
value, the in-place activities are separated by,
(
Sf (n), if Sf (n) ≥ threshold
S̄f (n) = (13)
end, otherwise
The application of PBC to the underlying example identified, with good approximation, the extent of each
of the two in-place motions considered. This is shown by the corresponding arrows in the spectrogram and the
PBC in Fig. 12. The combination of Radon transform and PBC is slated to avoid making classification decisions
over time-windows which have inter-motion and intra-motion activities. The former refers to a window extending
over two different classes, i.e., consecutive translation and in-place motions, whereas the latter represents the
case of a window covering portions of two motions of the same class, i.e., consecutive in-place motions. Mixed
motions within the same window confuses the classifier and may render decisions of neither motion at hand. The
only way to remedy such confusion is to train the classifier with mixed activities which increases the complexity
of the problem. Examples of inter- and intra-motion windows are shown in Fig. 11b by rectangulars.
4. CONCLUSION
In this paper, we presented a camera-less technique based on the Radon transform for detecting the transition
time between translation and in-place motions, and vice versa. This technique uses the range-map that depicts
range vs. slow-time of the moving target. The horizontal lines in the range-map correspond to in-place motions,
whereas other lines describe a time-dependent range. The power burst curve was used over the interval of in-place
motions to define the time boundaries to define the time boundaries of what could possibly be multiple motions,
like sitting then standing. Finding the transition times as well as event time-span allows us to avoid rendering
wrong motion classification decisions due to mixing two different motions within the same time window under
testing. We provided an example that demonstrates the effectiveness of the proposed approach, which is deemed
useful in dealing with continuous-time monitoring of human activities. Again, it is important to point out that
from a Comcast Labs point of view, this is a technological test, not a product, and if any productization of this
test would only occur, it would only be with the full, opt-in consent of the individual being monitored.
REFERENCES
[1] M. G. Amin, Y. D. Zhang, F. Ahmad, and K. C. Ho, “Radar signal processing for elderly fall detection:
The future for in-home monitoring,” IEEE Signal Processing Magazine 33(2), 71–80 (2016).
[2] M. G. Amin, Ed., [Radar for Indoor Monitoring Detection, Classification, and Assessment ], CRC Press
(2017).
[3] S. Z. Gurbuz, C. Clemente, A. Balleri, and J. Soraghan, “Micro-Doppler-based in-home aided and unaided
walking recognition with multiple radar and sonar systems,” IET Radar, Sonar Navigation 11(1), 107–115
(2017).
[4] A.-K. Seifert, M. G. Amin, and A. M. Zoubir, “New analysis of radar micro-Doppler gait signatures for
rehabilitation and assisted living,” in [2017 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) ], 4004–4008 (March 2017).
[5] M. S. Seyfiolu, S. Z. Grbz, A. M.zbayolu, and M. Yksel, “Deep learning of micro-Doppler features for aided
and unaided gait recognition,” in [2017 IEEE Radar Conference (RadarConf )], 1125–1130 (May 2017).
[6] A.-K. Seifert, A. M. Zoubir, and M. G. Amin, “Detection of gait asymmetry using indoor Doppler radar,”
in [Proceedings of the IEEE Radar Conference (RadarConf ), Boston, MA ], (April 2019).[7] L. Pallotta, C. Clemente, A. De Maio, J. J. Soraghan, and A. Farina, “Pseudo-zernike moments based radar
micro-Doppler classification,” in [2014 IEEE Radar Conference ], 0850–0854 (May 2014).
[8] Y. Kim and H. Ling, “Human activity classification based on micro-Doppler signatures using a support
vector machine,” IEEE Transactions on Geoscience and Remote Sensing 47, 1328–1337 (May 2009).
[9] R. M. Narayanan and M. Zenaldin, “Radar micro-Doppler signatures of various human activities,” IET
Radar, Sonar Navigation 9(9), 1205–1215 (2015).
[10] S. S. Ram, C. Christianson, Y. Kim, and H. Ling, “Simulation and analysis of Human Micro-Doppler in
through-wall environments,” IEEE Transactions on Geoscience and Remote Sensing 48, 2015–2023 (April
2010).
[11] B. G. Mobasseri and M. G. Amin, “A time-frequency classifier for human gait recognition,” Proc.
SPIE 73062 (2009).
[12] J. Hong, S. Tomii, and T. Ohtsuki, “Cooperative fall detection using Doppler radar and array sensor,” in
2013 IEEE 24th International Symposium on Personal Indoor and Mobile Radio Communications , 3492–
3496 (2013).
[13] B. Y. Su, K. C. Ho, M. J. Rantz, and M. Skubic, “Doppler radar fall activity detection using the wavelet
transform,” IEEE Transactions on Biomedical Engineering 62(3), 865–875 (2015).
[14] Q. Wu, Y. D. Zhang, W. Tao, and M. G. Amin, “Radar-based fall detection based on Doppler time-frequency
signatures for assisted living,” Sonar Navigation IET Radar 9(2), 173–183 (2015).
[15] M.G. Amin, [Through-the-Wall Radar Imaging ], CRC Press (2016).
[16] J.D. Taylor, [Introduction to Ultra-Wideband Radar Systems ], Taylor & Francis (1994).
[17] Skolnik, M., [Introduction to Radar Systems ].
[18] V. C. Chen, [The micro-Doppler effect in radar], Artech House (2011).
[19] B. Erol, M., A. Ravisankar, and M. G. Amin, “Realization of radar-based fall detection using spectrograms,”
10658 (2018).
[20] B. Jokanovic and M. G. Amin, “Suitability of data representation domains in expressing human motion
radar signals,” IEEE Geoscience and Remote Sensing Letters 14, 2370–2374 (Dec 2017).
[21] A.-K. Seifert, L. Schfer, M.G. Amin, and A.M. Zoubir, “Subspace classification of human gait using radar
Micro-Doppler signatures,” (08 2018).
[22] J. Li, S. L. Phung, F. H. C. Tivive, and A. Bouzerdoum, “Automatic classification of human motions using
Doppler radar,” in [The 2012 International Joint Conference on Neural Networks (IJCNN)], 1–6 (June
2012).
[23] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “MPCA: Multilinear Principal Component Analysis
of Tensor Objects,” IEEE Transactions on Neural Networks 19, 18–39 (Jan 2008).
[24] B. Jokanovic, M. G. Amin, F. Ahmad, and B. Boashash, “Radar fall detection using principal component
analysis,” Proc. SPIE 9829 (2016).
[25] J. Ye, R. Janardan, and Q. Li, “GPCA: An efficient dimension reduction scheme for image compression and
retrieval,” in [Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining], KDD ’04, 354–363, ACM, New York, NY, USA (2004).
[26] K. L. Wininger, “Basis of CT: the Radon transform,” Radiologic technology 84(4), 413–418 (2013).
[27] L. Cirillo, A. Zoubir, and M. G. Amin, “Parameter estimation for locally linear fm signals using a time-
frequency hough transform,” IEEE Transactions on Signal Processing 56, 4162–4175 (Sep. 2008).
[28] S. Helgason, [The Radon transform ], Progress in mathematics, Birkhäuser (1980).
[29] J. Xu, J. Yu, Y. Peng, and X. Xia, “Radon-Fourier transform for radar target detection, i: Generalized
Doppler filter bank,” IEEE Transactions on Aerospace and Electronic Systems 47, 1186–1202 (April 2011).
[30] MathWorks, “Radon transform.” Website: https://www.mathworks.com/help/images/
radon-transform.html. (Accessed: 8 March 2019).
[31] M. G. Amin, “Minimum variance time-frequency distribution kernels for signals in additive noise,” IEEE
Transactions on Signal Processing 44, 2352–2356 (Sep. 1996).
[32] B. Erol, M. G. Amin, and B. Boashash, “Range-Doppler radar sensor fusion for fall detection,” in [2017
IEEE Radar Conference (RadarConf ) ], 0819–0824 (May 2017).[33] L. R. Rivera, E. Ulmer, Y. D. Zhang, W. Tao, and M. G. Amin, “Radar-based fall detection exploiting
time-frequency features,” in [2014 IEEE China Summit International Conference on Signal and Information
Processing (ChinaSIP) ], 713–717 (July 2014).You can also read
