Technical Report At Home Workout Assistant - OpenCV
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Technical Report At Home Workout Assistant Group: Daniel Rodrigues Perazzo Gustavo Camargo Rocha Lima Natália Souza Soares Victor Gouveia de Menezes Lyra
Technical Report
At Home Workout Assistant
Table of Contents
1 Problem 2
1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Implemented Solution 3
2.1 Human Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Motion Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Results 5
3.1 Pose Estimation Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Motion Sensing Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2.1 Side Lateral Raise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2.2 Squat movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4 Limitations 8
5 Future Works 8
References 9
OpenCV Spatial AI Competition Page 1 of 9
Version 1.0- September 21, 2020
Technical Report
At Home Workout Assistant
1 Problem
1.1 Problem Statement
Due to the closure of gyms and public spaces this year, doing physical activities while maintaining
social distancing became a challenge. However, exercising alone can be difficult and even dangerous
due to possible muscle injuries. Even virtual classes with physical instructors can be difficult due to the
non-physicality of the video media. We will attack this problem by guiding a non-experienced user during
exercise routines.
1.2 Problem Description
According to a report made by market research group Mintel at the end of 2019 [9], ”digital fitness” in
the UK will grow from £4.9bn to £5.3bn by 2023. Digital fitness is loosely described as enterprises that
offer fitness exercises using digital means and from their homes.
Jump to today, and it is almost completing a year since the start of the COVID-19 outbreak. Besides
the countless lives lost and broken by the disease, psychologists predict that the mental toil due to the
anxiety and depression caused by the pandemic, the lockdowns, and social distancing measures enforced
due to it will define a generation. However, a New York Times Editorial [11] highlighted a surprising
remedy for these mental illnesses: physical exercises.
However, gyms are not safe places for exercise due to the agglomeration and the sharing of equipment.
In the same vein, even generally, lonesome activities, such as cycling or running, are being studied [12]
due to the possibility of tiny aerosol particles spreading the coronavirus. Nevertheless, many countries’
lockdown measures closed gyms and public spaces, leaving only the home as a place for exercise.
With the reemergence of COVID-19 on the US and Europe [10], with countries like Britain and France
reintroducing lockdown orders, it seems that the current situation will stay for a while.
According to Google trends [5], ”home workout” and ”home gym” had a massive rise in March, during
the eve of the pandemic, as shown in Figure 1. This phenomenon indicates that the primary solution for
this lack of places to workout was to do exercises at home. Furthermore, studies are forecasting [3] that
this new fitness culture will continue after the pandemic.
Figure 1: Searches over time for ”home workout” and ”home gym”. Image provided by Google Trends.
However, being home exercising can be dangerous for novice users due to the risk of muscle lesions
and potential injuries. Even video workouts can be challenging for trainers and trainees due to the
non-physicality of our video media. With this in mind, we want to help people who want to keep exercising
but do not have the skills or body knowledge to judge whether they are doing the right movements. By
this, we propose a movement correcting and orienting system for workout routines using the OAK-D
Spatial AI Camera.
OpenCV Spatial AI Competition Page 2 of 9
Version 1.0- September 21, 2020
Technical Report
At Home Workout Assistant
2 Implemented Solution
We developed a Motion Analysis for Home Gym System (MAHGS) to assist users during at-home
workouts. To build our solution, for which we had also designed a logo illustrated in Figure 2, we needed
to estimate the person’s 3D human pose and analyze the skeleton’s movements, returning the appropriate
feedback to help the user to perform an exercise correctly.
Figure 2: Logo designed for our solution.
Our system is composed of two main modules: 3D human pose estimation and motion analysis.
The first one was developed using the neural inference capabilities of the OAK-D [8] and the DepthAI
library [7]. The second module was implemented using a technique called Ikapp [4]. Figure 3 shows a
diagram of our proposed pipeline. The human pose estimation module will run on a PC/Raspberry Pi 4
connected with the OAK-D since it does not have WiFi nor Bluetooth yet. The movement analysis module
will be running on a smartphone. We use a TCP/IP protocol with the ZeroMQ framework to perform the
communication between these devices.
Figure 3: Our solution’s pipeline. Stages 1 to 3 constitute the 3D human pose estimation module. The
last stage is executed by the motion analysis module.
OpenCV Spatial AI Competition Page 3 of 9
Version 1.0- September 21, 2020
Technical Report
At Home Workout Assistant
2.1 Human Pose Estimation
Human pose estimation is a challenging problem, but, with the advent of deep learning, many different
solutions started appearing, the most famous of which being OpenPose [2]. However, most of these
techniques require high computing power and struggle to run in more portable systems such as a
cellphone or even a Raspberry Pi.
Nevertheless, due to the OAK-D’s high computing power, we can afford to have a portable embedded
system that estimates a high-quality human skeleton from an image. We used the OpenVINO [6] human-
pose-estimation model, available on the DepthAI library, to estimate the 2D human pose. But even
the OAK-D struggled to run this model, managing to process around 4-5 frames per second (FPS) for
inference and decoding, which is unviable for our application. So, we reduced the size of the model’s
input from 256x456 to 128x224 so the application could run with better FPS without losing too much in
precision.
Because the neural network output is not directly the joints, we still need to decode the result. This
decoding process can affect the FPS in less capable devices such as a raspberry. Given that, our solution
achieves a different frame rate depending on which platform the decoding is running.
Since the OAK-D also estimates depth, we can also obtain the individual joints’ depth values resulting
in a 3D skeleton. Being 3D provides more information for the motion analysis that can result in better
human pose evaluation. The joint data is sent via TCP/IP to a running the app that makes this analysis.
To make this connection, we used some of the code available in [1].
2.2 Motion Analysis
We utilized Ikapp’s motion analysis framework to perform the motion analysis, implemented on Unity
due to its ease of use and portability across devices. Since Ikapp was developed to run on the Microsoft
Kinect and Astra 3D Camera, we need to convert from the Openpose skeleton to the internal one adopted
by Ikapp.
With the human skeleton, we can compute the angles between different bones on different motion
planes and, with them, compare with those angles of a specific human pose. Because our system only
compares static skeleton poses, it needs to reach a series of different static poses sequentially to analyze
movements. To decide what specific posture we will compare with, we implemented a state machine to
change, which poses our system is trying to detect. We also aid the user by providing feedback on the
exercise routine by analyzing the user’s movements.
OpenCV Spatial AI Competition Page 4 of 9
Version 1.0- September 21, 2020
Technical Report
At Home Workout Assistant
3 Results
3.1 Pose Estimation Performance Evaluation
Our initial idea was to connect the OAK-D with a Raspberry Pi 3 to send the packages with the
estimated 3D pose to the mobile phone. However, the framerate achieved with a Raspberry Pi 3b was
not fast enough to be used during workouts. Therefore, we decided to test a Raspberry Pi 4. With this
integration, we captured the human pose estimation. We sent it to a smartphone to be used in our motion
analysis module with approximately 7 FPS for inference and decoding, which was slightly better than the
3 FPS for inference and decoding obtained with the Raspberry Pi 3. With this integration, our solution
works fine with less dynamic and controlled movements.
Before testing the Raspberry Pi 4, we have also used our solution with a laptop communicating the
two modules, i.e., the human pose estimation and the motion analysis. With this setup, we were able to
achieve 15 FPS for inference and decoding. This integration is less portable than the previous ones, but
it is more suitable for more dynamic movements. Table 1 compiles the comparison between all three
tested configurations.
Table 1: Comparison between three tested configurations for the human pose estimation and motion
analysis integration.
Intermediary Device FPS
Raspberry Pi 3 ≈3
Raspberry Pi 4 ≈7
Laptop ≈ 15
3.2 Motion Sensing Evaluation
We provide visual feedback for the user to see whether he has done the exercise correctly. To see
our system in action, we implemented two exercise routines, which we discuss below. We strongly
recommend to the reader to see the accompanying video demo at this link
Figure 4: Interface for our solution, we can see the Instruction that the user must to do, some specific
instruction regarding the exercise and a counter for the number of repetitions.
OpenCV Spatial AI Competition Page 5 of 9
Version 1.0- September 21, 2020
Technical Report
At Home Workout Assistant
3.2.1 Side Lateral Raise
The first movement that we have implemented was the side lateral raise. Our system managed to
identify whether the user is in the right position or not. It also managed to count the number of movements
the user has done by analyzing changes between relaxation and arms raised, as shown in Figure 5. If
only one of the arms is raised, it asks the user to lift the other, as shown in Figure 6. You can see the
execution of this movement in the accompanying video.
Figure 5: Our system Identified that user is on the Side Lateral Raise position. Now it is asking for the
user to Relax (e.g., lower his arms).
Figure 6: The system identified that the user has only one arm raised, and asked for the user to lift the
lower arm.
3.2.2 Squat movement
We have also made a demo for squats, which consists of the user lowering himself till the angle
between his thighs and calf becomes ≈ 90◦ . Our solution identifies if the user is keeping the exercise
in the selected range of acceptable angles for this movement. It only counts complete movements, i.e.,
whenever the user has been into the designated pose and raised himself again. Figures 7 and 8 show
the user squatting and after finishing the movement, respectively. First, the user is being prompted to go
OpenCV Spatial AI Competition Page 6 of 9
Version 1.0- September 21, 2020Technical Report
At Home Workout Assistant
to a relaxed state. While in the second, the repetition counter increments because he finished one squat,
and he is prompted to make another squat.
Figure 7: On the left, we can see the Openpose rendering of the skeleton. On the center, we can see
the Ikapp software determining that the user is squatting and telling the user to relax. On the right, we
can see another angle of the user doing the squat.
Figure 8: On the left, we can see the Openpose rendering of the skeleton, on the center we see the
Ikapp software instructing the user to do squat, on the right we can see another angle of the movement.
OpenCV Spatial AI Competition Page 7 of 9
Version 1.0- September 21, 2020Technical Report
At Home Workout Assistant
4 Limitations
During our research, we encountered some limitations to our technique. We noticed that the pose
estimation network that we used had some stability problems and smaller accuracy regarding illumination
and color changes due to the smaller image size. Sometimes, the OAK-D camera’s perspective deforms
the user’s skeleton, and, due to this deformation, the movement analysis can have difficulty detecting some
movements, especially leg movements. Also, since the camera can only see through one perspective,
the depth values extracted from 2 or more joints can become superposed if the body is in the lateral
position, leading to a wrong depth value for one of the joints or making it disappear.
Our movement analysis also breaks down whenever more than one person is present on the camera
and if the user is not in full view of the camera. Another issue is complex movements related to rotation.
Since the OpenPose skeleton model from OpenVino doesn’t have joints such as thumbs, which enable
the orientation of hands, we can only track simple mechanical movements such as squat and dumbbell
exercises. The model also has problems tracking a person horizontally or upside down, not enabling
us to track movements like pushups or handstands. The low fps rate also restricts us from moderated
exercises since we can’t track fast movements like jumping jacks, jump rope, and burpees.
5 Future Works
Our method managed to perform movement analysis from a video of the user.
For future works, our goals are:
• Improve the FPS count for the human pose estimation module.
• Provide Bluetooth connection for the cell-phone and the OAK-D, maybe with the soon-to-be-launched
OAK-D module accompanied with Bluetooth.
• Create a better front-end interface for the movement analysis module.
• improve the accuracy and stability of the movement analysis module.
• Use some signal processing filters, for example, a Kalman filter, to provide more stability for our
system.
• Add more types of feedback, like sounds and movement illustration.
• Implement a recording feature for professionals to do an exercise and compare it with the users’
movement.
Also, part of this Work will be presented next month in the Embedded Systems School at the X
Brazilian Symposium on Computing Systems Engineering as the tutorial ”Introducing the OAK-D: A
smart camera with neural inference and depth perception in real-time.” We plan to introduce the OAK-D’s
capabilities and functionalities concerning pose estimation and movement correction. Finally, some of the
underlying technology, the Ikapp motion analysis framework, will become the main product of a startup.
OpenCV Spatial AI Competition Page 8 of 9
Version 1.0- September 21, 2020Technical Report
At Home Workout Assistant
References
[1] Unity-python. https://github.com/off99555/Unity3D-Python-Communication, 2020. 4
[2] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. Openpose: realtime multi-person 2d pose
estimation using part affinity fields. arXiv preprint arXiv:1812.08008, 2018. 4
[3] Lockdowns have changed the fitness industry forever — here’s what to expect of post-pandemic
workouts. shorturl.at/bvEIS, 2020. Last accessed 30 October 2020. 2
[4] A. E. F. Da Gama, T. de Menezes Chaves, P. Fallavollita, L. S. Figueiredo, and V. Teichrieb.
Rehabilitation motion recognition based on the international biomechanical standards. Expert
Systems with Applications, 116:396–409, 2019. 3
[5] Google trends - home workout. shorturl.at/quvU4. Last accessed 30 October 2020. 2
[6] Intel. Distribution of openvino™ toolkit for linux with fpga support–openvino toolkit. Source:¡
https://docs. openvinotoolkit. org/2019 R1/ docs install guides installing openvino linux fpga. html,
2018. 4
[7] Luxonis. Depthai. https://luxonis.com/depthai, 2020. 3
[8] Introducing oak: Spatial ai powered by opencv. https://opencv.org/
introducing-oak-spatial-ai-powered-by-opencv/, 2020. 3
[9] The rise of digital fitness: can the new wave of high-intensity home workouts replace the gym?
shorturl.at/lvCKY, 2019. Last accessed 30 October 2020. 2
[10] Global coronavirus report: Us adds nearly 500,000 cases in a week; europe faces more lockdowns.
shorturl.at/qGJL4, 2020. Last accessed 30 October 2020. 2
[11] A possible remedy for pandemic stress: Exercise. shorturl.at/dHLT3, 2020. Last accessed 30
October 2020. 2
[12] Are running or cycling actually risks for spreading covid-19? shorturl.at/qHIS0, 2020. Last
accessed 30 October 2020. 2
OpenCV Spatial AI Competition Page 9 of 9
Version 1.0- September 21, 2020You can also read
