STM32 Artificial Intelligence Solutions - Raphael Apfeldorfer - Feb 2021 MDG/MCD/AI Solutions - STMicroelectronics
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
STM32 Artificial Intelligence Solutions Raphael Apfeldorfer - Feb 2021 MDG/MCD/AI Solutions
Artificial Intelligence at the Edge
Moving part of intelligence closer to the data acquisition
Better user Optimized Cloud
experience usage
Privacy by design
Realtime, no latency
(GDPR compliant)
Sustainable on
Reliable
energy
Add new functions and
services with Embedded AI
Contact us at edge.ai@st.com 2
Computer Vision for STM32
Give vision to your STM32 product for
new features and add-on services
And more
User 2
Box
Box
Optimized NN Customer
files application
Food Person presence Face Multiple object
classification detection recognition detection STM32.Vision
STM32.AI lib
lib
FP-AI- FP-AI- FP-AI-
Q2 2021*
VISION1 v1.0 VISION1 v2.0 FACEREC v1.0
*available for alpha customers run-time
3
Condition monitoring for STM32
Monitor STM32 equipment health for Get started using dedicated
improved uptime and lower maintenance cost And industrial boards
Optimized NN Customer
files application
Vibration monitoring for In-field Condition monitoring with
retrofit of existing systems current for build-in systems STM32.AI lib ML models
FP-AI-
Q1 2021*
NANOEDG1 v1.0
*available for alpha customers run-time
4
AI tools for STM32
Simple, fast, optimized
The key steps behind Neural Networks
Neural Network (NN) Model Creation Operating Mode
Process & analyze new
Capture data Train NN Model data using trained NN
2 4
1 3 5
Clean, label data Convert NN into
Build NN topology optimized code for MCU 6
ST toolbox for Neural Networks
Process & analyze new
Capture data data using trained NN
Wide board
ecosystem
Clean, label data Convert NN into
Build NN topology optimized code for MCU 7
Easily implement Neural Networks on STM32
Train Neural Network using Convert NN into optimized Run on optimized runtime
any major AI frameworks code
run-time
• Select most appropriate MCU • Validate code directly on target
• Review computation and • Get accuracy and inference time
and more… memory consumption per layer • Optimize memory usage
8
STM32Cube.AI main features
STM32Cube.AI is available both as graphical and command line interface
• Quickly assess model footprint requirements
Dimension • Select and configure MCU in STM32CubeMX
• Review model layers in STM32Cube.AI
• Generate C-code for pre-trained model
• Support quantized models to reduce RAM,
Optimize flash and latency with minimal loss of accuracy
• Use light run-time libraries
• Optimize for performance
• Optimize memory allocation
• Fine control of weight mapping
Fine Tune • Split between internal and external memory
• Update model without full FW update
And quickly iterate thanks to on-target validation 9
STM32Cube.AI, an STM32CubeMX expansion
Power Consumption
MCU Selector
Calculator
Code Generation Pinout Configuration
Middleware Parameters Clock Tree Initialization
Peripherals Configuration
10Collecting data
& architecting a NN topology
Services provided by Partners ST tools to support
Capture Data ST BLE Sensor mobile phone application
Collect and label data from the SensorTile.
ST BLE
Sensor
Selected partners
Neural Networks engineering services support.
Data scientists and Neural network architects.
Clean, label data
Build NN topology 11Example form factor hardware
to capture and process data
SensorTile
Capture Data Inference on STM32L476
STM32L4
Motion MEMS Motion MEMS
www.st.com/SensorTile
12
www.st.com/SensorTile-eduFast go to market module
to capture data with more accuracy
SensorTile.Box
Capture Data Inference on STM32L4R9
Microsoft IoT
services ready
More advanced, high accuracy and low power sensors
• First Inertial module with Machine Learning capabilities.
• Motion (accelerometer and gyroscope, magnetometer) and slow motion (inclinometer)
• Altitude (pressure), environment (pressure, temperature, humidity, compass) and sound
(sound and ultrasound analog microphone)
• Microsoft IoT services ready to make available on a web dashboard the result of the
embedded processing
13
www.st.com/SensorTileBoxDistributed AI: sensor + STM32
Optimize performance and power consumption
Smart Sensor Smart STM32
with Machine Learning Core Second level of AI processing
FSM MLC
up to 16 up to 8 Raw Data Deep Learning
Event Decision Neural Networks
Inertial Sensor FSM and MLC Machine Learning
New LSM6DSOX Re-configuration
• Best ultra-low-power sensing at high performance • More advanced and complex NNs
• 550µA (gyroscope and accelerometer) • Decisions on multiple sensors
➔ 200µA less than closest competitor • NN input can be sensor data and/or
• 20~40µA (Accelerometer only for HAR) sensor Machine Learning decisions
• Efficient Finite State Machines: 2µA • Multiple Neural Networks support
• Configurable Machine Learning Core: 4~8µA • Actuation & communication
14Form factor hardware
AI IoT node for more connectivity
B-L475E-IOT01A
Sub-1GHz Dynamic NFC Tag
+ Wi-Fi
Capture Data Inference on STM32L4
More debug capabilities
• Integrated ST-Link/V2.1
• PMOD extension connector
• Arduino Uno extension connectors
15
www.st.com/IoTnodeWireless Industrial node
to capture data at industrial grade
STWIN
Capture Data Inference on STM32L4R9
Industrial-grade sensors
• Industrial scale 9-DoF motion sensors including accelerometer,
gyrometer and an ultra wide-bandwidth vibrometer with ultra low noise
• Very high frequency audio and ultrasound microphone
• High precision temperature and environmental monitoring
• Micro SD card for standalone data logging
• BLE5.0 connectivity and WiFi expansion board
• USART
www.st.com/stwin 16STM32H7 discovery boards with camera
STM32H747I-DISCO
with B-CAMS-OMV
Capture Data Inference on STM32H747
Computer Vision on microcontroller
• STM32H747 high-performance and DSP with DP-FPU, Arm Cortex-M7
at 480 MHz + Cortex-M4 MCU with 2MB internal Flash, 1MB internal
RAM, Chrom-ART Accelerator
• External memory 2x64MB Quad-SPI NOR Flash and 32MB SDRAM
• 4” capacitive touch LCD display module with MIPI® DSI interface
• Camera module adapter board and camera module based on OV5640
5MPx 8b color rolling shutter
• ST-MEMS digital microphones
• Ethernet RJ45 and Wi-Fi / cellular expansion boards
17OpenMV integration
Fast machine vision prototyping
Configure Machine Vision in
real-time over USB in Python
Run and validate optimized
Neural Network
OpenMV CAM
Running MicroPython over STM32
18
https://github.com/openmv/openmvMaking AI Accessible Now
Leader in Arm® Cortex®-M 32-bit General Purpose MCU
STM32MP1
MPU 4158 CoreMark
650 MHz Cortex -A7
209 MHz Cortex -M4
STM32F2 STM32F4 STM32H7 STM32F7
High Perf 398 CoreMark 608 CoreMark 3224 CoreMark 1082 CoreMark
240 MHz Cortex -M4 216 MHz
MCUs 120 MHz 180 MHz
480 MHz Cortex -M7
STM32F0 STM32G0 STM32F1 STM32F3 STM32G4
Mainstream 106 CoreMark 142 CoreMark 177 CoreMark 245 CoreMark 550 CoreMark
MCUs 48 MHz 64 MHz 72 MHz 72 MHz 170 MHz
STM32L0 STM32L1 STM32L5 STM32U5 STM32L4 STM32L4+
Ultra-low Power 75 CoreMark 93 CoreMark 424 CoreMark 651 CoreMark 273 CoreMark 409 CoreMark
MCUs 32 MHz 32 MHz 110 MHz 160 MHz 80 MHz 120 MHz
Compatible with Machine Learning
Partner ecosystems
Wireless STM32WL
161 CoreMark
STM32WB
216 CoreMark Compatible with Deep Learning
MCUs 48 MHz 64 MHz STM32Cube.AI ecosystem
Arm® Cortex® core -M0 -M0+ -M3 -M33 -M4 -M7 dual -A7& -M4
More than 40,000 customers Over 4 Billion STM32 shipped since 2007 19Function Packs
Simple, fast, optimizedAI Solutions on STM32
A full development ecosystem to create your AI application
AI extension for STM32CubeMX Person presence
to map pre-trained Neural detection
Networks FP-AI-VISION1 Food classification
STM32 Community with People activity
dedicated Neural Networks topic recognition
and AI expert partners FP-AI-SENSING1 Audio scene classification
Trainings, hands on, MOOCs and Condition-based
partners videos monitoring
FP-AI-NANOEDG1
21FP-AI-SENSING1
Audio scene classification (ASC)
3 classes
Audio Data capture Labelling controlled Data stored on the device Indoor, Outdoor, In vehicle
by smartphone application SD card for future learning labelling
NN & example
dataset provided
Embedded audio Inferences running Inference result
pre-processing on the microcontroller displayed on mobile app
22
► Demo availableFP-AI-SENSING1
Human activity motion recognition (HAR)
5 classes example
Motion Data Capture Labelling controlled Data stored on the device Stationary, walking, running,
by smartphone application SD card for future learning biking, driving labelling
NN & example
dataset provided
Embedded motion Inferences running Inference result
pre-processing on the microcontroller displayed on mobile app
23
► Demo availableFP-AI-VISION1
Image classification
Enjoy the food classification demo Full end-to-end optimized software example
• Default demo based on 18 classes • from camera acquisition to image pre-processing before feeding
(224x224 RGB pictures) the NN
• Several camera image output size possible • Multiple memory mapping possibilities to optimize and test
impact on performances
• Retrain this NN with your own dataset
• Quantize your trained network to optimized inference time and
memory usage
Pizza
99%
NN & example
STM32H7 dataset provided 150ms
Embedded image Inferences running
pre-processing (SW) on on the microcontroller
the STM32H747
24
► Demo availableFP-AI-VISION1
Person presence detection
One-class image classification demo Full end-to-end optimized software example
• Models from tensorflow.org (L4R and H7) • from camera acquisition to image pre-processing before feeding
and MobileNet v2 (H7 only) the NN
• QVGA 320x240 color image on the LCD • Multiple models fitting STM32L4R to STM32H7 depending on
• Can adapt camera flipping depending on required performance and cost
which side camera is placed • Visual wake word for Smart home or cities security cameras
• Reduce false alarms due to object movement detection
Unused 9.4 %
Person 34.8 %
No Person 87.1 %
Inferences running
B-CAMS-OMV Resize on the microcontroller
camera bundle
DCMI Crop Greyscale
34.8 %
QVGA 320x240
Embedded image acquisition
STM32L4R display
and pre-processing (SW)
Chrom-GRC™
accelerations
Chrom-ART™
Pixel Format conversion Graphics optimized MMU
(GFXMMU)
3.5 fps
274 ms
Display frame buffer
25
► H7 demo available on FP-AI-VISION1User_2
FP-AI-FACEREC1
Embed face recognition in your IoT project
STM32H7 User-personalized services / features
Adjust automatically per user
• Device preferred settings or ergonomics per user
• Customized device behavior / action for registered user
• Customize alerts
• Prevent child injury with underage appliance lock
• Create user-specific automations
▪ On-device face enrollment of multiple users
Features ▪ Real-time face recognition, display enrolled image
▪ Displays match accuracy and inference speed
Available on-demand as a software library 26
Contact edge.ai@st.com to get accessFP-AI-NANOEDG1
Condition monitoring on STWIN
Get straight to proof-of-concept with full anomaly detection system
without deep Data Science knowledge
Collect dataset from Generate free Integrate and Install on Incremental learning Monitor anomalies
industrial-grade ML library deploy premise on-target on-target
vibration sensor
Download the dedicated SW package
27AI solutions for STM32MP1
Running AI on ST
MicroprocessorsSTM32MP1 microprocessor
Augmented intelligence
Cortex-M4
Realtime OS
STM32CubeMX
STM32MP1
= +
X-LINUX-AI
support for
Dual Cortex-A7 &
OpenSTLinux Distribution
STM32CubeMx
• STM32Cube.AI to convert pre-trained NNs for the Cortex-M4 core
• TensorFlow Lite STM32MP1 support up streamed for native NN inferences support on the dual Cortex-A side
29X-LINUX-AI Package
for STM32MP1 AI Applications
Application examples in C/C++ and Python
AI NN Apps and models
• Image classification: 1000 objects classified
CV Framework Pillow
• Multiple object detection: 90 classes
AI Framework Includes code for camera acquisition and image pre-processing
OS distribution Linux® Python
Hardware Cortex®-A7
AI, CV frameworks
& application
STM32MP1
examples provided
Inferences running on the Displayed on STM32MP157-DK2,
USB camera or
microprocessor in 80ms STM32MP157-EV1 and Avenger96 board
built-in camera
for image classification
module
30
► 2x demos availableST co-development and partnerships
Leverage the power of Edge AI
ST AI
Expert
AI co-development partnerships
team Contact us at edge.ai@st.com
Multiple object
detection with
thermal imager
Meet our expert AIS partners
Visit st.com/stm32cubeai
Predictive
maintenance of
reflow oven
https://www.st.com/content/st_com/en/campaigns/artificial-intelligence-at-the-edge.html
31For more information
www.st.com/STM32CubeAI
Contact us at edge.ai@st.com
Capture Run on
Train NN
Data STM32
Label
Data
32Releasing your creativity
/STM32
@ST_World
community.st.com
www.st.com/STM32CubeAI
33Thank you © STMicroelectronics - All rights reserved. ST logo is a trademark or a registered trademark of STMicroelectronics International NV or its affiliates in the EU and/or other countries. For additional information about ST trademarks, please refer to www.st.com/trademarks. All other product or service names are the property of their respective owners.
You can also read
