Edge AI Human Activity Recognition on ESP32-S3

A model on a laptop is not a product

Human activity recognition from inertial sensors is a solved-looking problem right up until you have to run it somewhere real. On a microcontroller there is no Python, no floating-point luxury, a few hundred kilobytes of usable memory, and a recurrent layer that the on-device runtime flatly refuses to execute.

This project takes the problem the whole distance: sixteen activities classified from dual-IMU multivariate time series — accelerometer and gyroscope — trained, optimised, converted, quantised and finally deployed as C++ firmware that classifies live sensor data on an ESP32-S3.

Getting the split right, first

Time-series activity data punishes careless splitting. Slice windows at random and neighbouring, near-identical windows land on both sides of the train/test boundary — the model scores brilliantly and has learned nothing transferable.

So the windowing and splitting scheme came before the modelling: a leakage-free, segment-based design where windows are cut within contiguous recording segments and whole segments — never individual windows — are assigned to a split. Every accuracy figure below rests on that decision.

Segment-based splitting — whole segments move together, so no window is ever seen twice

Three architectures, searched not guessed

• Benchmarked CNN, CNN-LSTM and CNN-GRU against the same splits — convolution alone versus convolution plus recurrence, on identical ground.
• Hyperparameters chosen by Optuna search rather than intuition, so the comparison between architectures is a comparison of architectures and not of how much time each one got tuned.
• Trained on GPU with CUDA and automatic mixed precision, reaching up to 85.6% accuracy.

The recurrent layer that wouldn't convert

Going from PyTorch to TFLite Micro means passing through ONNX and TensorFlow, and recurrent layers do not survive that trip cleanly: they arrive as dynamic WHILE loop operators, which the microcontroller runtime does not implement. The usual response is to drop the recurrent models and ship the CNN.

Instead I wrote custom unrolled-RNN export wrappers that expand the recurrence into a fixed sequence of operations at export time, removing the unsupported WHILE ops entirely. The unrolled graphs match the original models to within 1e-4 output deviation — close enough that the conversion is not a source of error worth reasoning about.

Every model was then quantised to INT8 for TFLite Micro.

Accuracy is one axis of three

On a microcontroller the most accurate model is frequently the wrong one. Selection used a composite score across accuracy, latency and model size — the deployment candidate won with 5× fewer parameters than the largest contender.

Firmware, not a demo script

The device side is C++ firmware on an ESP32-S3 running all six quantised models against live IMU data — not a single frozen artefact, but a switchable set you can compare on the bench.

• A touchscreen model-picker UI — swap the active model on the device, no reflash.
• Bit-exact StandardScaler preprocessing reimplemented in C++, so the features the firmware computes are the features the model was trained on. This is where silent accuracy loss usually hides.
• PSRAM tensor arenas, because six models will not share the ESP32-S3's internal RAM.
• On-device latency and memory benchmarking, so the numbers used to pick a model are measured on the target rather than estimated from the host.
• A custom 16 MB dual-slot OTA partition table and wireless firmware updates — the device can be re-flashed in place and rolled back if a build misbehaves.

On the wrist

The firmware runs on a round touchscreen ESP32-S3 in a watch body. Pick a model from the grid, and it classifies live IMU data with the prediction, confidence and the measured inference time on screen.

Source

Training pipeline, export wrappers and firmware on GitHub.