TinyML Comes to Embedded World 2023 Leave a comment

//php echo do_shortcode(‘[responsivevoice_button voice=”US English Male” buttontext=”Listen to Post”]’) ?>

NUREMBERG, GERMANY—TinyML companies, from microcontroller (MCU) manufacturers to tools companies to model optimizers, put on a range of demonstrations at Embedded World 2023. While machine learning (ML) is being applied to camera, audio and other sensor data, the majority of demos are still focused on computer vision.

Many demos were under the magic 1 mW figure, generally the threshold for “always-on” applications.

STMicroelectronics

EE Times was granted exclusive access to a demo of STMicro’s forthcoming STM32N6 MCU. This part will feature a dedicated artificial intelligence (AI) accelerator on-chip that has been developed in-house at STMicro. The demo showed the -N6 up and running, with a custom version of YOLO running at an impressive 314 fps; this is between one and two orders of magnitude faster than the same network running on the STM32H7, STMicro’s most powerful existing MCU, which uses an Arm Cortex-M7.

Also on the STMicro booth were another couple of fun demos, including a washing machine that could tell how much laundry was in the machine in order to optimize the amount of water added. This system is sensorless; it is based on AI analysis of the current required to drive the motor, and predicted the weight of the 800g laundry load to within 30g. A robot vacuum cleaner equipped with a time-of-flight sensor also used AI to tell what type of floor surface it was cleaning, to allow it to select the appropriate cleaning method.

Renesas

Next stop was the Renesas booth, to see the Arm Cortex-M85 up and running in a not-yet-announced product (due to launch in June). This is the first time EE Times has seen AI running on a Cortex-M85 core, which was announced by Arm a year ago.

The M85 is a larger core than the Cortex-M55, but both are equipped with Helium—Arm’s vector extensions for the Cortex-M series—ideal for accelerating ML applications. Renesas’ figures had the M85 running inference 5.3× faster than a Renesas M7-based design, though the M85 was also running faster (480 MHz compared with 280).

Renesas’ demo had Plumerai’s person-detection model up and running in 77 ms per inference.

Renesas field application engineer Stefan Ungerechts also gave EE Times an overview of the DRP-AI (dynamically reconfigurable processor for AI), Renesas’ IP for AI acceleration. A demo of the RZ/V2L device, equipped with a 0.5 TOPS @ FP16 (576 MACs) DRP-AI engine, was running tinyYOLOv2 in 27 ms at 500 mW (1 TOPS/W). This level of power efficiency means no heat sink is required, Ungerechts said.

The DRP-AI is, in fact, a two-part accelerator; the dynamically reconfigurable processor handles acceleration of non-linear functions, then there is a MAC array alongside it. Non-linear functions in this case might be image preprocessing functions or model pooling layers of a neural network. While the DRP is reconfigurable hardware, it is not an FPGA, Ungerechts said. The combination is optimized for feed-forward networks like convolutional neural networks commonly found in computer vision, and Renesas’ software stack allows either the whole AI workload to be passed to the DRP-AI or use of a combination of the DRP-AI and the CPU.

Also available with a DRP-AI engine are the RZ/V2MA and RZ/V2M, which offer 0.7 TOPS @ FP16 (they run faster than the -V2L at 630 MHz compared to 400, and have higher memory bandwidth).

A next-generation version of the DRP-AI that supports INT8 for greater throughput, and is scaled up to 4K MACs, will be available next year, Ungerechts said.

Squint

Squint, an AI company launched earlier this year, is taking on the challenge of explainable AI.

Squint CEO Kenneth Wenger told EE Times that the company wants to increase trust in AI decision making for applications like autonomous vehicles (AVs), healthcare and fintech. The company takes pre-production models and tests them for weaknesses—identifying in what situations they are more likely to make a mistake.

This information can be used to set up a mitigating factors, which might include human-in-the-loop—perhaps flagging a medical image to a doctor—or trigger a second, more specialized model that has been specifically trained for that situation. Squint’s techniques can also be used to tackle “data drift”—for maintaining models over longer periods of time.

Embedl

Swedish AI company Embedl is working on retraining models to optimize them for specific hardware targets. The company has a Python SDK that fits into the training pipeline. Techniques include replacing operators with alternatives that may run more efficiently on the particular target hardware, as well as quantization-aware retraining. The company’s customers so far have included automotive OEMs and tier 1s, but they are expanding to Internet of Things (IoT) applications.

Embedl has also been a part of the VEDL-IoT project, an EU-funded project in collaboration with Bielefeld University that aims to develop an IoT platform, which distributes AI across a heterogeneous cluster.

Their demo showed managing AI workloads across different hardware: an Nvidia AGX Xavier GPU in a 5G basestation and an NXP i.MX8 application processor in a car. With sufficient 5G bandwidth available, “difficult” layers of the neural network could be computed remotely in the basestation, and the rest in the car, for optimum latency. Reduce the 5G bandwidth available, and more or all of the workload goes to the i.MX8. Embedl had optimized the same model for both hardware types.

Silicon Labs

Silicon Labs had several xG24 dev kits running AI applications. One had a simple Sparkfun camera with the xG24 running people counting, and calculating the direction and speed of movement.

A separate wake word demo ran in 50 ms on the xG24’s accelerator, and a third board was running a gesture recognition algorithm.

BrainChip

BrainChip had demos running on a number of partner booths, including Arm and Edge Impulse. Edge Impulse’s demo showed the company’s FOMO (faster objects, more objects) object detection network running on a BrainChip Akida AKD1000 in under 1 mW.

Kinara

Kinara’s first-gen chip, the ARA-1, was demonstrated on the AMD booth, where a Xilinx Kria K26 module sat alongside four kinara chips in an edge box designed to process multiple streams of video simultaneously in real time. Kinara CEO Ravi Annavajjhala told EE Times that while four cameras were demonstrated, the system can handle up to eight H.264 1080p streams.

Aspinity

Aspinity gave EE Times a preview of the development environment it has been building for its AML100 chip, an analog processor for ultra-low power, always-on AI in acoustics, bio signals and industrial anomaly detection applications. Aspinity’s software programmable hardware allows the entire signal chain to remain in the analog domain, with the development environment designed to offer a high level of abstraction, including reference models and tools.

Aspinity has joined the Renesas Preferred Partners Program, with Aspinity’s glassbreak sensing application board demonstrated with the Renesas Quick-Connect IoT rapid prototyping platform on the Renesas booth at the show.

Useful Sensors

Useful Sensors CEO Pete Warden and software engineer Nat Jefferies were on the OKdo booth showing off their object detection sensor module, the size of a postage stamp, which is manufactured by OKdo.

Useful Sensors has recently open-sourced its TensorFlow Lite version of the OpenAI model Whisper (for automatic speech recognition). Warden told EE Times that the company is also working on a QR code sensor and another hardware module for speech processing.

Menta

EE Times got an introduction to French eFPGA IP company Menta. CTO Yoan Dupret explained that the company works in two markets—with the majority of customers in the defense sector and a minority in edge processing, including devices and edge servers.

While the eFPGA IP is often used for cryptography acceleration and 5G networking, where algorithms may need to be changed or updated after deployment, there is a growing market for AI acceleration at the edge, he said.

The ratio of FPGA LUTs to DSP blocks in Menta’s soft IP is flexible, and custom blocks can also be implemented.

The IP is scalable from 100 LUTs (perhaps in an IoT sensor module) to as much as 200k LUTs. While not a typical customer, Japanese research group Raas has made a reticle-sized implementation for a data center AI application. More typical applications range from AI for cloud detection and climate change monitoring in satellites to more straightforward people-counting applications in smart retail.

Innatera

Innatera has been working on its analog/mixed-signal spiking neural network accelerator. The company’s demo showed its first-gen chip running audio scene classification in under 1 mW active power for always-on applications. 1 s of audio could be classified in about 1 ms

A forthcoming second-gen device with twice the number of neurons, visible on the company’s booth, will offer improved latency and power consumption thanks to improved calibration techniques and further work done on analog circuit design, according to Innatera neuromorphic engineer George Vathakkatil Joseph. A third-gen device, due to tape out in the coming months, will offer a more complete system-on-chip–type solution, he said.

Hailo

At the Hailo booth were a variety of demos of edge AI servers from the company’s partners using the Hailo-8 accelerator, including an industrial machine vision system from Danish manufacturer Qtechnologies designed for sorting potatoes at high speed. The system was simultaneously sorting potatoes by size, shape, quality and variety, and the demo showed segmentation, tracking and sorting.

Arm

Following the launch of the Cortex-M85 AI-capable MCU core last year, Paul Williamson, senior VP and GM of the IoT line of business at Arm, told EE Times that Arm will continue to invest in its Ethos product line of dedicated AI accelerator cores. While the Ethos line is “very active” and “a concentration of our continued investment,” Williamson said, Arm believes that in the world of MCUs, it will be important to have a stable, software targetable framework that complements more machine learning capability in the MCU.

Announced at the show was enhanced integration for Arm virtual hardware (AVH) into the latest version of the Keil MCU development kit. AVH allows developers to quickly perform “does it fit” checks for their algorithms on specific Arm cores with specific memory sizes, helping them decide whether or not dedicated accelerators like Ethos are required.

Arm is also working closely with third-party accelerator IP providers for applications that require more acceleration than the Ethos line can offer, including BrainChip (on Arm’s booth, a demo showed an Arm M85 working with BrainChip Akida IP).

Eurotech

Italian edge computer builder Eurotech had fun demos, including a robot hand that could mimic movements of booth visitors via a camera system.

The company’s DynaCOR 40-36 edge server system, which won the Best in Show Award in the AI category, is designed for collecting data that will be used to train AVs, as well as in-vehicle training and inference. This edge server features an Intel Xeon CPU and has four PCIe slots—the version on the booth had four Nvidia A30s. It is liquid cooled with the CPUs sandwiched between cold plates—fanless operation means it is silent.

The company also showed off its industrial edge gateway and software framework offerings on the Arm booth, which help businesses extract value from their data.

Eurotech’s CTO Marco Carrer told EE Times the company has recently added industrial PCs to its offering via the acquisition of German firm InoNet.

GreenWaves

GreenWaves invited EE Times to try the audio demos on its booth, including active noise cancellation and spatial audio. The spatial audio demo, in collaboration with Idun Audio, allows the user to hear different parts of the music, or different speakers in a meeting, from different directions.

An IMU in the headphones allows the user to turn their head and still hear the relevant audio from the same direction. A demonstration that appeared to position the user in the center of an orchestra was particularly effective. Idun Audio’s spatial audio AI runs on GreenWaves’ GAP9 processor in under 2.7 mW.

GreenWaves’ GAP9 processor was awarded first prize in the hardware category of the Embedded Awards.