Run and profile Arm NN on the Raspberry Pi - Tutorial Version 21.08 Non-Confidential
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Run and profile Arm NN on the Raspberry Pi Version 21.08 Tutorial Non-Confidential Issue 01 Copyright © 2021 Arm Limited (or its affiliates). 102640_2108_01_en All rights reserved.
Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Run and profile Arm NN on the Raspberry Pi
Tutorial
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Release information
Document history
Issue Date Confidentiality Change
2108-01 18 August 2021 Non-Confidential Initial release
Proprietary Notice
This document is protected by copyright and other related rights and the practice or
implementation of the information contained in this document may be protected by one or more
patents or pending patent applications. No part of this document may be reproduced in any form
by any means without the express prior written permission of Arm. No license, express or implied,
by estoppel or otherwise to any intellectual property rights is granted by this document unless
specifically stated.
Your access to the information in this document is conditional upon your acceptance that you
will not use or permit others to use the information for the purposes of determining whether
implementations infringe any third party patents.
THIS DOCUMENT IS PROVIDED “AS IS”. ARM PROVIDES NO REPRESENTATIONS AND NO
WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, INCLUDING, WITHOUT LIMITATION,
THE IMPLIED WARRANTIES OF MERCHANTABILITY, SATISFACTORY QUALITY, NON-
INFRINGEMENT OR FITNESS FOR A PARTICULAR PURPOSE WITH RESPECT TO THE
DOCUMENT. For the avoidance of doubt, Arm makes no representation with respect to, and has
undertaken no analysis to identify or understand the scope and content of, third party patents,
copyrights, trade secrets, or other rights.
This document may include technical inaccuracies or typographical errors.
TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL ARM BE LIABLE FOR
ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL,
INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND
REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS
DOCUMENT, EVEN IF ARM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
This document consists solely of commercial items. You shall be responsible for ensuring that
any use, duplication or disclosure of this document complies fully with any relevant export laws
and regulations to assure that this document or any portion thereof is not exported, directly
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 2 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
or indirectly, in violation of such export laws. Use of the word “partner” in reference to Arm’s
customers is not intended to create or refer to any partnership relationship with any other
company. Arm may make changes to this document at any time and without notice.
If any of the provisions contained in these terms conflict with any of the provisions of any click
through or signed written agreement covering this document with Arm, then the click through or
signed written agreement prevails over and supersedes the conflicting provisions of these terms.
This document may be translated into other languages for convenience, and you agree that if there
is any conflict between the English version of this document and any translation, the terms of the
English version of the Agreement shall prevail.
The Arm corporate logo and words marked with ® or ™ are registered trademarks or trademarks
of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. Other brands
and names mentioned in this document may be the trademarks of their respective owners. Please
follow Arm’s trademark usage guidelines at https://www.arm.com/company/policies/trademarks.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Arm Limited. Company 02557590 registered in England.
110 Fulbourn Road, Cambridge, England CB1 9NJ.
(LES-PRE-20349)
Confidentiality Status
This document is Non-Confidential. The right to use, copy and disclose this document may be
subject to license restrictions in accordance with the terms of the agreement entered into by Arm
and the party that Arm delivered this document to.
Unrestricted Access is an Arm internal classification.
Product Status
The information in this document is Final, that is for a developed product.
Web address
developer.arm.com
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 3 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Inclusive language commitment
Arm values inclusive communities. Arm recognizes that we and our industry have used language
that can be offensive. Arm strives to lead the industry and create change.
We believe that this document contains no offensive language. To report offensive language in this
document, email terms@arm.com.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 4 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Contents
Contents
1 Introduction....................................................................................................................................................... 6
1.1 Conventions......................................................................................................................................................6
1.2 Additional reading........................................................................................................................................... 7
1.3 Feedback........................................................................................................................................................... 7
1.4 Other information........................................................................................................................................... 8
2 Overview............................................................................................................................................................ 9
2.1 Before you begin............................................................................................................................................ 9
3 Overview of running Ubuntu Linux on the Raspberry Pi....................................................................10
3.1 Run Ubuntu Linux on the Raspberry Pi..................................................................................................10
4 Overview of building the Arm NN software stack............................................................................... 11
4.1 Build the Arm NN software stack............................................................................................................11
5 MNIST Draw...................................................................................................................................................12
5.1 Set up MNIST Draw.................................................................................................................................... 12
5.2 Machine learning model............................................................................................................................. 13
5.3 MNIST demo application............................................................................................................................14
6 Streamline........................................................................................................................................................ 15
6.1 Run the MNIST inference.......................................................................................................................... 15
6.2 Use Streamline to connect and profile the application....................................................................... 16
6.3 Automate the launch and capture........................................................................................................... 17
7 Related information...................................................................................................................................... 19
8 Next steps....................................................................................................................................................... 20
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 5 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Introduction
1 Introduction
1.1 Conventions
The following subsections describe conventions used in Arm documents.
Glossary
The Arm Glossary is a list of terms used in Arm documentation, together with definitions for
those terms. The Arm Glossary does not contain terms that are industry standard unless the Arm
meaning differs from the generally accepted meaning.
See the Arm® Glossary for more information: developer.arm.com/glossary.
Typographic conventions
Arm documentation uses typographical conventions to convey specific meaning.
Convention Use
italic Introduces special terminology, denotes cross-references, and citations.
bold Highlights interface elements, such as menu names. Denotes signal names. Also used for terms in
descriptive lists, where appropriate.
monospace Denotes text that you can enter at the keyboard, such as commands, file and program names, and source
code.
monospace italic Denotes arguments to monospace text where the argument is to be replaced by a specific value.
monospace bold Denotes language keywords when used outside example code.
monospace underline Denotes a permitted abbreviation for a command or option. You can enter the underlined text instead of
the full command or option name.
Encloses replaceable terms for assembler syntax where they appear in code or code fragments. For ex-
ample:
MRC p15, 0, , , ,
SMALL CAPITALS Used in body text for a few terms that have specific technical meanings, that are defined in the
Arm Glossary. For example, IMPLEMENTATION DEFINED, IMPLEMENTATION SPECIFIC, UNKNOWN, and
UNPREDICTABLE.
This represents a recommendation which, if not followed, might lead to system failure or damage.
This represents a requirement for the system that, if not followed, might result in system failure or
damage.
This represents a requirement for the system that, if not followed, will result in system failure or damage.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 6 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Introduction
Convention Use
This represents an important piece of information that needs your attention.
This represents a useful tip that might make it easier, better or faster to perform a task.
This is a reminder of something important that relates to the information you are reading.
1.2 Additional reading
This document contains information that is specific to this product. See the following documents
for other relevant information:
Table 1-2: Arm publications
Document Name Document ID Licensee only
None - -
1.3 Feedback
Arm welcomes feedback on this product and its documentation.
Feedback on this product
If you have any comments or suggestions about this product, contact your supplier and give:
• The product name.
• The product revision or version.
• An explanation with as much information as you can provide. Include symptoms and diagnostic
procedures if appropriate.
Feedback on content
If you have comments on content then send an e-mail to errata@arm.com. Give:
• The title Run and profile Arm NN on the Raspberry Pi Tutorial.
• The number 102640_2108_01_en.
• If applicable, the page number(s) to which your comments refer.
• A concise explanation of your comments.
Arm also welcomes general suggestions for additions and improvements.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 7 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Introduction
Arm tests the PDF only in Adobe Acrobat and Acrobat Reader, and cannot
guarantee the quality of the represented document when used with any other PDF
reader.
1.4 Other information
See the Arm website for other relevant information.
• Arm® Developer.
• Arm® Documentation.
• Technical Support
• Arm® Glossary.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 8 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Overview
2 Overview
Arm NN is an inference engine for CPUs, GPUs, and NPUs. Arm NN bridges the gap between
existing neural network frameworks and the underlying hardware IP. It enables efficient translation
of existing neural network frameworks, such as TensorFlow Lite (TFLite). This guide explains how
to create Linux applications which load TFLite trained neural network models, run them on a
Raspberry Pi, and profile application performance with Arm Streamline. The Raspberry Pi 3 Model
B features a Cortex-A53, while the Raspberry Pi 4 Model B features a Cortex-A72. We can install
Ubuntu Linux on these boards and use them for development. This guide shows you how to set
up a Raspberry Pi with Ubuntu Linux for running applications using Arm NN. This guide also shows
you how to use Arm Streamline for Machine Learning (ML) application profiling. The following is the
process:
1. Set up Linux on the Raspberry Pi.
2. Build the Arm NN software stack.
3. Build and run the example ML applications.
4. Use Streamline for performance analysis.
2.1 Before you begin
In addition to your Raspberry Pi and appropriate power supply, you must also be able to read a
microSD card on your computer. You can also optionally use an HDMI monitor, HDMI cable, and
a USB keyboard during this guide. If you want to avoid the extra clutter, you can also access the
Raspberry Pi using SSH.
Duration: You need about 1.5-2 hours to complete the instructions in this guide. This time duration
excludes the library install time.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 9 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Overview of running Ubuntu Linux on the Raspberry Pi
3 Overview of running Ubuntu Linux on the
Raspberry Pi
Installing Ubuntu Linux on the Raspberry Pi is simple and involves downloading the image and
flashing it to the SD card of the Raspberry Pi.
3.1 Run Ubuntu Linux on the Raspberry Pi
The steps in this section cover installing Ubuntu Linux on the Raspberry Pi.
About this task
The sudo apt upgrade -y command in step two can take approximately 20
minutes after a fresh installation.
Procedure
1. Use the How to install Ubuntu Server on your Raspberry Pi tutorial to install a 64-bit Ubuntu
Linux distribution on you Raspberry Pi. This Ubuntu tutorial is the best place to find instructions
for your operating system and Raspberry Pi version. Using this guide, install a 64-bit Ubuntu
Linux distribution onto your Raspberry Pi. Arm has tested Ubuntu Server 20.04 on a Raspberry
Pi 3 Model B.
2. Enter the following commands to ensure that you have the necessary libraries installed:
$ sudo apt update
$ sudo apt upgrade -y
$ sudo apt-get install --no-install-recommends make build-essential libssl-dev
zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libn\
curses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblz\
ma-dev libhdf5-103 libhdf5-dev python3-pip
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 10 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Overview of building the Arm NN software stack
4 Overview of building the Arm NN
software stack
Building the Arm NN software stack can take a few hours. Build the software stack with enough
time in advance.
4.1 Build the Arm NN software stack
The steps in this section cover building the Arm NN software stack.
Procedure
1. Clone the Tool-Solutions repo, which contains all the example code and the Arm NN build
script.
$ git clone https://github.com/ARM-software/Tool-Solutions.git
2. Enter the following commands to build the Arm NN stack.
$ cd Tool-Solutions/ml-tool-examples/build-armnn/
$ ./build-armnn.sh
This build script downloads and compiles all the software that Arm NN requires. This download
and compilation takes a few hours to complete. Arm recommends you do it in advance, or build
on another machine and copy the build over. For example, you can build on a graviton1, if you
have access to such a machine.
3. Enter the following commands to set your LD_LIBRARY_PATH to the appropriate armnn install
path:
$ export LD_LIBRARY_PATH="/home/ubuntu/armnn-devenv/armnn/build"
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 11 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
MNIST Draw
5 MNIST Draw
MNIST Draw is a fun, single-page website that enables you to hand-draw and classify digits
between zero and nine using machine learning. This guide uses a machine learning model trained
against the MNIST dataset for classification.
The project is a modified version of mnist-draw, which uses the Arm NN SDK to perform
inferences on an Arm Cortex-A CPU. The application runs on the Raspberry Pi and can be accessed
over a network using a browser.
5.1 Set up MNIST Draw
The steps in this section cover setting up MNIST Draw.
Procedure
1. Navigate to the MNIST Draw code example, using the following commands:
#Go to the repository
$ cd $HOME/Tool-Solutions/ml-tool-examples/mnist-draw
#Build the armnn-draw application
$make -C armnn-draw
$ pip3 install -r requirements.txt
#Set LD_LIBRARY_PATH for Arm NN (if not already done)
#This is also helpful to put in $HOME/.bashrc for future use
$ export LD_LIBRARY_PATH=$HOME/armnn-devenv/armnn/build
#Start python server
$ python3 -m http.server --cgi 8000
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 12 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
MNIST Draw
2. Open a browser on any machine which can access the Raspberry Pi, and go to http://ip-
address:8000
The following image is an example of the interface of the website:
Figure 5-1: Writing a number in MNIST draw
3. Using the mouse, draw a digit between zero and nine on the empty canvas and hit the predict
button to process the drawing. MNIST Draw indicates any errors during processing with
a warning icon and prints them to the console. Common errors include not compiling the
application in armnn-draw/ and not using python3.
Results
MNIST Draw displays results as a bar graph, in which each classification label receives a score
between 0.0 and 1.0 from the machine learning model. Clear the canvas with the Clear button to
draw and process other digits.
5.2 Machine learning model
For more information about how the web application translates the digit drawn into an image file
processed by Arm NN, refer to the Python script in the cgi-bin/mnist.py directory path.
A convolutional neural network (CNN) is defined within the model/ directory, and is used by
the program in armnn-draw/ which incorporates the Arm NN SDK. This model is configured for
MNIST data inputs. The default model is optimized_mnist.tflite.
Using this model, let us try something that includes more than one image. In this example
application, we use a larger set of MNIST images, and show inference on the CPU.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 13 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
MNIST Draw
5.3 MNIST demo application
The MNIST demo application uses a TensorFlow Lite neural network that is trained for MNIST. The
demo application also uses Arm NN for inference on Arm Cortex-A or Mali.
There are two example applications, one with the simple neural network and one with a better
neural network. Look at the C++ files for each version. mnist_tf_convol.cpp is the better
neural network and mnist_tf_simple.cpp is the simple single-layer network. Both applications
read a TensorFlow Lite model. Models are stored in the model/ directory in tflite binary
format. The MNIST data is stored in the data/ directory in a simple format that is designed for
storing vectors. This directory contains the MNIST test data and labels.
Build the applications using make. Use the following commands:
$ cd $HOME/Tools-Solutions/ml-tool-examples/mnist-demo
$ make
The make command builds both applications.
The purpose of these examples is to demonstrate how to use Arm NN to load and execute
TensorFlow Lite models in a C++ application.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 14 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Streamline
6 Streamline
Streamline is a performance analyzer for software running on Arm processors. You can download
and install the software from Arm Developer. There are thirty-day trials available for people
without a license.
Before profiling the application with Streamline, let's look at the code in the mnist-demo
application. Open either mnist_tf_convol.cpp or mnist_tf_simple.cpp.
After Import the TensorFlow Lite graph, the code is common regardless of the
framework that you started with.
Here are the sections of code to look at:
• Load and parse the MNIST data
The helper function in mnist_loader.hpp scans the file in the dataset and returns a
MnistImage struct with two fields: the label and an array of pixel values.
• Import the TensorFlow Lite graph
You can import a TensorFlow Lite graph from a .tflite file. Importing a graph consists of
binding the input and output points of the model graph. You can find these points by visualizing
the model in TensorBoard.
• Optimize for a specific compute device
Arm NN supports optimization for both CPU and GPU devices, however the Raspberry Pi only
supports CPU acceleration for mnist-demo.
It is easy to specify the device when creating the execution runtime context in the code.
• Run the graph
Running the inference on the chosen compute device is performed through the
EnqueueWorkload() function of the context object.
The result of the inference can be read directly from the output array and compared to the
MnistImage label that we read from the data file.
6.1 Run the MNIST inference
The steps in this section cover building the Arm NN software stack.
Procedure
Enter the following commands to run the application:
# Optimisation modes: 0 for CpuRef, 1 for CpuAcc, 2 for GpuAcc
# Input size: 1 to 2000 (number of images to predict)
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 15 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Streamline
$ ./mnist_tf_convol 1 10
Next steps
Try different program configurations and compare the execution times. For example, enter the
following commands:
$ time ./mnist_tf_convol 0 10 # 10 images on unoptimised CPU device
$ time ./mnist_tf_convol 1 100 # 100 images on optimised CPU device
6.2 Use Streamline to connect and profile the application
The steps in this section cover profiling with Streamline.
Procedure
1. Start gatord on the Raspberry Pi. After building all the software, the gatord daemon is available
in the home directory. Run the gatord daemon using sudo to receive access to all hardware
counters. Use the --app option to specify a command during a capture. The following example
code runs the gatord daemon:
$ sudo $HOME/gatord --app ./mnist_tf_convol 1 100
2. Connect Streamline to the target board
a. Run Streamline on another device and under Pick Target Device Type choose the TCP
(Advanced) option.
b. Enter the IP address of the Raspberry Pi under Enter the Target Details. If the Raspberry Pi
already shows up as an existing target, choose it from there.
Figure 6-1: Streamline view
3. Click Start a capture to start a capture in Streamline
4. Ensure that the application we want to profile is running correctly. The application binary is
compiled as covered in the previous steps. If the application we want to profile is running
correctly, you see output that resembles the following example output:
Optimisation mode: CpuAcc
#1 | Predicted: 7 Actual: 7
…
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 16 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Streamline
#100 | Predicted: 9 Actual: 9
Prediction accuracy: 100%
5. Observe the results with Streamline
• Review the CPU workload, the correlation between the phases of execution, the functions
called, and the correlation between the functions and the kernels.
• Ensure you click the Heat Map along the bottom and change it to Compute Library to see
the ML information.
6.3 Automate the launch and capture
The steps in this section cover how to automate the launch and capture in Streamline.
Procedure
1. Run the following command on the target device:
$ sudo ~/gatord -a
2. Automate the launch and capture. Use the Streamline Capture & Analysis Options dialog to
specify a working directory and a command to run.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 17 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Streamline
3. Click the Stop on Exit checkbox to stop the capture when the run is over.
The following screenshot shows how to set up an automated capture:
Figure 6-2: Streamline Configure Application dialog
The following screenshot is an example capture obtained from Streamline:
Figure 6-3: Streamline example capture data
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 18 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Related information
7 Related information
Here are some resources:
• Arm Community - ask development questions and find articles and blogs on specific topics
from Arm experts.
• Arm NN SDK
• MNIST
• MNIST Draw
• Raspberry Pi
• Streamline download
• Streamline Performance Analyzer
• Streamline 30-day trial
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 19 of 20Run and profile Arm NN on the Raspberry Pi Tutorial Document ID: 102640_2108_01_en
Version 21.08
Next steps
8 Next steps
This guide has described how to set up Linux on the Raspberry Pi, build the Arm NN software
stack, run ML examples, and use Streamline for performance analysis. You can use what you have
learned in this guide to begin writing your own applications using Arm NN.
To explore related areas, look at our how to guides on other ML frameworks. To ask questions and
get help, visit the Machine Learning Forum on Arm Community.
Copyright © 2021 Arm Limited (or its affiliates). All rights reserved.
Non-Confidential
Page 20 of 20You can also read
