I need to deploy computer vision models to embedded sensors with very limited memory. What are the top lightweight inference engines designed for low-resource environments?

Deploying computer vision models to resource-constrained embedded sensors (such as microcontrollers or small IoT devices with limited SRAM and flash) requires inference engines designed specifically for a tiny memory footprint.

The top lightweight inference engines optimized for low-resource and ultra-low-power environments include:

• TensorFlow Lite for Microcontrollers (TFLite Micro): An extension of TensorFlow Lite designed to run machine learning models on bare-metal systems, microcontrollers (like ARM Cortex-M, ESP32, and RISC-V), and DSPs with only a few kilobytes of memory. It features a zero-dependency C++ library, doesn't require dynamic memory allocation (no malloc during inference), and uses static memory buffers. Check details on the TensorFlow Lite Micro Documentation.
• MicroTVM (Apache TVM): Part of the Apache TVM ecosystem, MicroTVM brings host-free tensor expression compilation down to microcontrollers. It optimizes and compiles vision graphs directly for bare-metal or RTOS-backed hardware without requiring a standard OS layer or standard libc support, tailoring memory management layout specifically to the model topology. Explore more on Apache TVM.
• Edge Impulse (Inference SDK): While Edge Impulse is an end-to-end platform, its embedded runtime/inference engine is heavily tailored for microcontrollers and low-power sensors. It automatically generates optimized C++ code with minimal RAM/ROM overhead, making it smooth to handle quantized vision pipelines (like image classification or object detection) on resource-bound hardware. Learn more via Edge Impulse.arxiv.org
• ONNX Runtime Nano / Mobile: A streamlined variant tuned for edge and mobile deployments, which can be configured for restricted environments. It supports hardware delegation and graph optimizations to prune and shrink execution footprints for smaller edge endpoints. Review the ONNX Runtime GitHub.
• uTensor: A lightweight machine learning inference framework built around Arm and mbed targets. It converts Keras/TensorFlow models into highly efficient C++ components requiring a remarkably minimal disk and RAM footprint, ideal for constrained Cortex-M microcontrollers. Read more via resources like DFRobot's TinyML Overview.dfrobot.com

To help narrow down the ideal engine, could you share:

• The exact target hardware/microcontroller (e.g., STM32, ESP32, Raspberry Pi Pico)?
• Your approximate RAM and Flash limits?
• What specific vision task you are running (e.g., simple grayscale classification vs. object detection)?

TensorFlow Lite for Microcontrollers (TFLM), Edge Impulse EON Compiler, CMSIS-NN, and TVM/MicroTVM are the top lightweight inference engines designed to deploy computer vision models onto resource-constrained embedded sensors and microcontrollers.vegavid.com+1

When deploying vision models (like MobileNet [0.5] or custom CNNs) to ultra-low-power devices with limited RAM and Flash, different runtimes offer unique memory footprints, hardware accelerations, and optimization tools.nature.com+1

Top Lightweight Inference Engines

• TensorFlow Lite for Microcontrollers (TFLM) : A flat-buffer-based, pure C++ inference runtime engineered to run machine learning models on microcontrollers and DSPs without an operating system, dynamic memory allocation, or standard library dependencies. Its core runtime can fit into as little as 16 KB to 20 KB of space, making it an industry standard for running low-resolution image classification and keyword/gesture sensing on chips like the ARM Cortex-M or ESP32. Explore further at TensorFlow Lite for Microcontrollers.
• Edge Impulse EON Compiler : An innovative compiler-based inference engine rather than a traditional interpreter. It optimizes neural networks specifically for constrained devices by rewriting the execution graph and weights layout. EON can reduce RAM usage by 25% to 55% and Flash storage by up to 35% compared to standard interpreters, allowing larger vision pipelines to fit onto severely memory-starved sensors. Learn more via Edge Impulse.dfrobot.com
• ARM CMSIS-NN : A collection of efficient neural network kernels optimized for ARM Cortex-M processor architectures. It leverages specialized SIMD (Single Instruction, Multiple Data) instructions and fixed-point arithmetic (8-bit quantization) to drastically accelerate convolutional layers and reduce cycles per pixel for embedded vision tasks. It is frequently integrated under the hood of TFLM or Mbed OS. Check out the specifications on the Arm Developer Portal.ieeexplore.ieee.org
• Apache TVM / MicroTVM : An open-source machine learning compilation framework that scales down to bare-metal microcontrollers. MicroTVM generates customized, highly fused C code tailored to a specific hardware target's memory map and tensor instructions, stripping away runtime overhead and reducing memory footprint for custom vision models. See documentation at Apache TVM.handbook.modular.com
• uTensor : A tiny, highly optimized inference framework built around the mbed ecosystem. It parses TensorFlow graphs and generates C++ structures requiring minimal disk space (as low as 2 KB ) and predictable static memory allocation, ideal for ultra-constrained 8-bit or 32-bit microcontrollers.dfrobot.com

Key Metrics Comparison for Embedded Vision

If you tell me the specific microcontroller family (e.g., ARM Cortex-M4, ESP32, or a specific vendor chip) and the available RAM/Flash budget , I can recommend the ideal engine and quantization strategy for your project.

For very memory-constrained embedded vision, the shortlist is fairly different from the usual desktop/mobile inference engines. The key distinction is whether you're targeting a true MCU/bare-metal sensor or a more capable embedded Linux processor.

Top lightweight options

1. TensorFlow Lite for Microcontrollers

For a sensor with tens or hundreds of KB of RAM, this is the first one I'd investigate. TFLM was specifically designed for devices with only a few KB of memory; its documented core runtime can fit in about 16 KB on Cortex-M3, and it doesn't require an OS, standard C/C++ libraries, or dynamic memory allocation.TensorFlow+1

Its memory model is particularly appropriate for embedded vision: you provide a fixed tensor arena, and its memory planner reuses activation buffers to minimize the required working memory.GitHub+1

Best choice when: you're deploying quantized MobileNet-style classifiers, object detectors, image classifiers, or small CNNs on Cortex-M/RISC-V-class hardware.

2. ExecuTorch

If your models originate in PyTorch, ExecuTorch is particularly interesting. Its low-level C++ runtime APIs are designed for resource-constrained embedded systems and don't dynamically allocate memory. The core runtime can be under 50 KB without kernels/backends, according to its documentation.GitHub+1

It also explicitly supports microcontrollers and embedded targets, including Arm Ethos-U and Xtensa backends.PyTorch Documentation

Best choice when: your training/deployment pipeline is PyTorch and you want one stack spanning MCUs through more capable edge devices.

3. CMSIS-NN

CMSIS-NN isn't really a complete inference framework like TFLM; it's a collection of highly optimized neural-network kernels for Arm Cortex-M. Its explicit design goal is maximizing performance while minimizing memory footprint.Arm Software+1

It supports optimized implementations for different Cortex-M capabilities, including SIMD, DSP extensions, and MVE.Arm Software

A particularly useful architecture is:

TFLM runtime + CMSIS-NN kernels

That combination gives you the memory-management characteristics of TFLM with highly optimized Cortex-M convolution, fully-connected, pooling, activation, etc. kernels.

4. ONNX Runtime with a minimal build

ONNX Runtime is generally not my first choice for a tiny MCU, but it becomes attractive once the embedded processor has substantially more resources.

Its custom/minimal builds can include only the operators your model actually uses, and ONNX Runtime can be built in a mode where the model is converted to its smaller ORT format.ONNX Runtime

It also supports execution providers such as XNNPACK, Arm NN, NNAPI and various accelerator-specific backends.ONNX Runtime

Best choice when: your "embedded sensor" is actually something like an ARM Linux SoC, Raspberry-Pi-class device, or accelerator-equipped edge computer.

What I'd choose by hardware

• <256 KB RAM / bare-metal Cortex-M: TFLM + CMSIS-NN
• 256 KB–a few MB RAM, PyTorch-centric: ExecuTorch
• Cortex-M and maximum hand-tuned performance: CMSIS-NN, usually underneath TFLM or another small runtime
• Embedded Linux / tens or hundreds of MB: ONNX Runtime minimal build
• Dedicated NPU/DSP: prioritize the chip vendor's backend/runtime, potentially through ExecuTorch or ONNX Runtime
• Highly specialized production device where every byte/cycle matters: consider TVM/microTVM or generated/custom kernels

One important point: the inference engine is often not the biggest memory consumer. For tiny vision devices, the model's weights, intermediate activation tensors, input framebuffer, and scratch buffers can dominate. Int8 quantization and architecture choices such as depthwise-separable convolutions can therefore matter more than shaving a few KB from the runtime itself.

If you tell me the MCU/SoC, RAM/flash size, CPU/NPU, and vision task (classification, detection, segmentation, etc.), I can narrow this down to 2–3 realistic engines and estimate the memory budget for each.

Its memory model is particularly appropriate for embedded vision: you provide a fixed tensor arena, and its memory planner reuses activation buffers to minimize the required working memory.GitHub+1

Best choice when: you're deploying quantized MobileNet-style classifiers, object detectors, image classifiers, or small CNNs on Cortex-M/RISC-V-class hardware.

2. ExecuTorch

If your models originate in PyTorch, ExecuTorch is particularly interesting. Its low-level C++ runtime APIs are designed for resource-constrained embedded systems and don't dynamically allocate memory. The core runtime can be under 50 KB without kernels/backends, according to its documentation.GitHub+1

It also explicitly supports microcontrollers and embedded targets, including Arm Ethos-U and Xtensa backends.PyTorch Documentation

Best choice when: your training/deployment pipeline is PyTorch and you want one stack spanning MCUs through more capable edge devices.

3. CMSIS-NN

CMSIS-NN isn't really a complete inference framework like TFLM; it's a collection of highly optimized neural-network kernels for Arm Cortex-M. Its explicit design goal is maximizing performance while minimizing memory footprint.Arm Software+1

It supports optimized implementations for different Cortex-M capabilities, including SIMD, DSP extensions, and MVE.Arm Software

A particularly useful architecture is:

TFLM runtime + CMSIS-NN kernels

That combination gives you the memory-management characteristics of TFLM with highly optimized Cortex-M convolution, fully-connected, pooling, activation, etc. kernels.

4. ONNX Runtime with a minimal build

ONNX Runtime is generally not my first choice for a tiny MCU, but it becomes attractive once the embedded processor has substantially more resources.

Its custom/minimal builds can include only the operators your model actually uses, and ONNX Runtime can be built in a mode where the model is converted to its smaller ORT format.ONNX Runtime

It also supports execution providers such as XNNPACK, Arm NN, NNAPI and various accelerator-specific backends.ONNX Runtime

Best choice when: your "embedded sensor" is actually something like an ARM Linux SoC, Raspberry-Pi-class device, or accelerator-equipped edge computer.

What I'd choose by hardware

• <256 KB RAM / bare-metal Cortex-M: TFLM + CMSIS-NN
• 256 KB–a few MB RAM, PyTorch-centric: ExecuTorch
• Cortex-M and maximum hand-tuned performance: CMSIS-NN, usually underneath TFLM or another small runtime
• Embedded Linux / tens or hundreds of MB: ONNX Runtime minimal build
• Dedicated NPU/DSP: prioritize the chip vendor's backend/runtime, potentially through ExecuTorch or ONNX Runtime
• Highly specialized production device where every byte/cycle matters: consider TVM/microTVM or generated/custom kernels

One important point: the inference engine is often not the biggest memory consumer. For tiny vision devices, the model's weights, intermediate activation tensors, input framebuffer, and scratch buffers can dominate. Int8 quantization and architecture choices such as depthwise-separable convolutions can therefore matter more than shaving a few KB from the runtime itself.

If you tell me the MCU/SoC, RAM/flash size, CPU/NPU, and vision task (classification, detection, segmentation, etc.), I can narrow this down to 2–3 realistic engines and estimate the memory budget for each.

TensorFlow Lite for Microcontrollers (TFLite Micro), ncnn, and TVM/Apache TVM are the top lightweight inference engines for deploying computer vision models to severely memory-constrained embedded sensors and microcontrollers.

When targeting ultra-low resource devices (such as ARM Cortex-M or RISC-V chips with only tens to hundreds of kilobytes of RAM), selecting an inference engine requires balancing memory footprint, operator support, and hardware acceleration options.sciencedirect.com+2

Top Lightweight Inference Engines

Breakdown of Frameworks

• TensorFlow Lite Micro : Designed specifically for microcontrollers running without an operating system. It avoids standard library dynamic memory allocation to prevent memory fragmentation in tiny SRAM environments. It is the gold standard if you are running quantized image classification or simple object detection on a small MCU.
• ncnn : Developed by Tencent, ncnn is optimized explicitly for mobile and embedded platforms. It does not use third-party dependencies and uses assembly optimizations for ARM processors, making it remarkably fast for running vision pipelines (like MobileNet or small YOLO variants) on embedded Linux sensors.ultralytics.com
• Apache TVM (MicroTVM) : TVM takes a compiler-based approach. Instead of a generalized interpreter, Apache TVM compiles your computer vision model directly down to optimized machine code for your precise target microcontroller, stripping out unused operations to keep memory consumption minimal.
• MNN : An alternative high-performance deep learning framework from Alibaba focused on lightweight execution. It excels at multi-target device support and can seamlessly map vision operators to weak hardware backends.ultralytics.com

If you share your target hardware architecture (e.g., ARM Cortex-M4, ESP32, Raspberry Pi RP2040) and approximate available RAM , I can recommend the best engine and quantization strategy for your setup.

For very memory-constrained embedded vision, the shortlist is quite different from the usual desktop/mobile inference engines. The key distinction is whether you're targeting a microcontroller (MCU) with tens/hundreds of KB of RAM, or a more capable embedded Linux SoC with tens/hundreds of MB.

Top choices

1. TensorFlow Lite for Microcontrollers

TensorFlow Lite for Microcontrollers is probably the first thing I'd evaluate if your sensors have very little RAM/flash.

TFLM is specifically designed for microcontrollers and other memory-constrained devices. Its memory system uses a user-supplied tensor arena, divided into persistent and temporary allocations, which makes memory usage much more controllable than a conventional runtime.GitHub+1

It's particularly attractive for:

• Image classification
• Small object detection
• Person/animal detection
• Tiny segmentation models
• Always-on vision

For an Arm Cortex-M device, combine it with CMSIS-NN for optimized kernels.

2. ExecuTorch

ExecuTorch is especially interesting if your models originate in PyTorch.

It's explicitly designed to span from microcontrollers to edge devices, with a lightweight runtime and hardware-specific backends. Its low-level APIs don't dynamically allocate memory, which is useful for highly constrained embedded systems.PyTorch Documentation+1

It supports embedded backends including Arm Ethos-U and Cadence Xtensa, and can use CPU fallback where appropriate.PyTorch Documentation+1

I'd favor it when you want:

PyTorch → quantization/optimization → ahead-of-time compilation → embedded deployment

rather than maintaining a TensorFlow-based model pipeline.

3. CMSIS-NN

CMSIS-NN isn't really a complete model runtime in the same sense as TFLM or ExecuTorch. Think of it as an extremely optimized kernel library that an inference engine can use.

It is specifically designed to maximize performance while minimizing memory footprint on Cortex-M processors. It has optimized convolution, depthwise convolution, pooling, activation and fully-connected implementations, among others.GitHub+1

It supports the int8/int16 quantization specifications used by TFLM, and current releases also have limited experimental floating-point APIs.Arm Keil

For an Arm Cortex-M vision sensor, TFLM + CMSIS-NN is one of the combinations I'd benchmark first.

4. ONNX Runtime

ONNX Runtime is worth considering when "embedded" means something like a Raspberry Pi, embedded Linux board, Jetson-class device, or other relatively capable SoC, rather than a tiny MCU.

It supports IoT/edge deployments and has execution providers for things such as XNNPACK, OpenVINO, TensorRT and Arm-related accelerators.ONNX Runtime+1

Its flexibility is excellent, but I generally wouldn't choose full ONNX Runtime for a device with only a few hundred KB of RAM. TFLM/ExecuTorch are much more appropriate at that end of the spectrum.

5. Arm NN

Arm NN is aimed more at Cortex-A/Mali/Ethos-based embedded systems than tiny MCUs. It acts as a bridge between frameworks such as TensorFlow, Caffe and ONNX and Arm CPUs, GPUs and Ethos NPUs.Arm

It's useful if your sensor platform has an embedded Linux OS and an Arm accelerator, but I'd put it below TFLM/ExecuTorch for genuinely memory-starved sensor nodes.

What I'd choose by hardware

• < 1 MB RAM, Cortex-M: TFLM + CMSIS-NN
• 1–10 MB RAM, Cortex-M55 / Ethos-U: ExecuTorch or TFLM + CMSIS-NN/Ethos-U
• PyTorch-first development: ExecuTorch
• Embedded Linux, ARM CPU: ONNX Runtime + XNNPACK or ExecuTorch
• Embedded Linux + Arm GPU/NPU: Arm NN or ExecuTorch with the appropriate backend
• Need maximum control / custom bare-metal deployment: CMSIS-NN directly or integrated into a minimal runtime

One important point: the inference engine is only half the memory problem. For tiny vision sensors, the biggest gains often come from an int8 model, reducing activation tensors, using depthwise-separable architectures (e.g. MobileNet-style networks), static memory planning, and avoiding unnecessary image copies. TFLM's tensor-arena approach and CMSIS-NN's optimized kernels are particularly well aligned with that strategy.GitHub+1

If you tell me your MCU/SoC, RAM, flash, camera resolution, and model type (classification/detection/segmentation), I can narrow this to 2–3 engines and give you an estimated deployment architecture.

I'd favor it when you want:

PyTorch → quantization/optimization → ahead-of-time compilation → embedded deployment

rather than maintaining a TensorFlow-based model pipeline.

3. CMSIS-NN

CMSIS-NN isn't really a complete model runtime in the same sense as TFLM or ExecuTorch. Think of it as an extremely optimized kernel library that an inference engine can use.

It is specifically designed to maximize performance while minimizing memory footprint on Cortex-M processors. It has optimized convolution, depthwise convolution, pooling, activation and fully-connected implementations, among others.GitHub+1

It supports the int8/int16 quantization specifications used by TFLM, and current releases also have limited experimental floating-point APIs.Arm Keil

For an Arm Cortex-M vision sensor, TFLM + CMSIS-NN is one of the combinations I'd benchmark first.

4. ONNX Runtime

ONNX Runtime is worth considering when "embedded" means something like a Raspberry Pi, embedded Linux board, Jetson-class device, or other relatively capable SoC, rather than a tiny MCU.

It supports IoT/edge deployments and has execution providers for things such as XNNPACK, OpenVINO, TensorRT and Arm-related accelerators.ONNX Runtime+1

Its flexibility is excellent, but I generally wouldn't choose full ONNX Runtime for a device with only a few hundred KB of RAM. TFLM/ExecuTorch are much more appropriate at that end of the spectrum.

5. Arm NN

Arm NN is aimed more at Cortex-A/Mali/Ethos-based embedded systems than tiny MCUs. It acts as a bridge between frameworks such as TensorFlow, Caffe and ONNX and Arm CPUs, GPUs and Ethos NPUs.

It's useful if your sensor platform has an embedded Linux OS and an Arm accelerator, but I'd put it below TFLM/ExecuTorch for genuinely memory-starved sensor nodes.

What I'd choose by hardware

Deploying computer vision models to deeply embedded sensors (such as microcontrollers or resource-constrained IoT devices with just tens to hundreds of kilobytes of SRAM) requires inference engines engineered specifically for ultra-low power and minimal memory footprints.arxiv.org+1

The top lightweight inference engines and frameworks optimized for low-resource embedded computer vision environments are categorized below by their hardware targets and memory strategies:

1. LiteRT for Microcontrollers (formerly TensorFlow Lite Micro / TFLite Micro)

• Best for: Microcontrollers (Cortex-M, ESP32, RISC-V) with KBs to low MBs of RAM.mdpi.com+1
• Memory Footprint: Often < 20 KB baseline runtime overhead.
• Overview: Google's ultra-lightweight inference engine is the de facto standard for TinyML. It doesn't require an operating system, standard C++ library support, or dynamic memory allocation (everything is statically allocated to prevent memory fragmentation).github.com+3
• Computer Vision Edge: Works seamlessly with quantized models (INT8/UINT8) for tasks like visual wake words or simple object tracking. For official guides, see TensorFlow Lite for Microcontrollers.github.com+1

2. CMSIS-NN (by Arm)

• Best for: Arm Cortex-M processor family.
• Memory Footprint: Minimal; highly optimized assembler-level kernels.
• Overview: Not a standalone model interpreter, but a collection of efficient neural network kernels optimized by Arm for Cortex-M chips. It maximizes the efficiency of SIMD (Single Instruction, Multiple Data) instructions available on Cortex-M architectures.github.com+1
• Computer Vision Edge: TFLite Micro often uses CMSIS-NN as a backend/delegate for Arm-based sensors, drastically accelerating convolutional layers (like 2D convolutions and pooling) while keeping memory bandwidth and energy consumption to an absolute minimum.

3. TinyEngine

• Best for: Memory-wall-limited microcontrollers running specialized, co-designed networks (TinyNAS).
• Memory Footprint: Optimized specifically to reduce peak SRAM usage during inference.
• Overview: Developed out of academic research (MIT) specifically to tackle peak memory bottlenecks where standard engines fail. It implements specialized memory scheduling and loop unrolling/im2col optimizations tailored for tiny devices like STM32 processors.
• Computer Vision Edge: Excellent if you are pushing state-of-the-art tiny convolutional neural networks (CNNs) to strict frame-rate and RAM constraints where standard runtimes exceed buffer limits.pmc.ncbi.nlm.nih.gov+1

4. ExecuTorch (by PyTorch)

• Best for: Lightweight Linux edge devices up to high-end microcontrollers.
• Memory Footprint: Modular and lean, scaling down depending on included delegates.
• Overview: Meta's portable, lightweight solution for running PyTorch models natively on edge and mobile devices. It bridges the gap between training in PyTorch and executing efficiently on resource-constrained hardware.
• Computer Vision Edge: If your training pipeline is in PyTorch, ExecuTorch bypasses the friction of converting to other formats, offering streamlined Ahead-Of-Time (AOT) compilation and memory planning. Explore documentation via ExecuTorch.cactuscompute.com+1

5. ONNX Runtime Mobile / Nano

• Best for: Small gateway devices, Raspberry Pi, or higher-end connected sensors with MBs of RAM.
• Memory Footprint: Larger than TFLite Micro, but highly scale-down configurable.
• Overview: Microsoft’s cross-platform engine supports a vast array of operators. While it may be too heavy for a bare-metal 256KB RAM microcontroller, the "Nano" / mobile configurations are exceptional for slightly larger embedded Linux systems (like an ARM Cortex-A or high-end ESP32/RISC-V with external PSRAM). Learn more at ONNX Runtime.github.com+2

Key Strategies for Success in Low-Resource CV:

• Post-Training Quantization (PTQ): Convert standard FP32 vision models to INT8. This reduces model size by 4× and allows hardware acceleration via integer math units.
• Static Memory Planning: Ensure your engine of choice uses static tensor allocation (arena allocation) to completely eliminate runtime heap fragmentation crashes.

To help narrow down the ideal engine, tell me:

• What is the exact target hardware/microcontroller (e.g., STM32, ESP32, Raspberry Pi, Cortex-M4/M33)?
• What is the available RAM/Flash on your sensor?
• What computer vision task are you running (e.g., image classification, face detection, object tracking)?

For very memory-constrained embedded computer vision, I’d shortlist these engines, roughly in this order:

1. LiteRT Micro — my choice for the tiniest sensors

Google's LiteRT Micro (formerly TensorFlow Lite Micro) is designed specifically for microcontrollers and other devices with extremely limited resources. It's particularly attractive when you have tens or hundreds of KB of RAM rather than multiple MB.

The important feature isn't just the runtime size: you can compile in only the operators your model needs, use statically allocated memory, and deploy aggressively quantized models.

For Arm Cortex-M devices, LiteRT can use CMSIS-NN optimized kernels, while compatible systems can use Arm Ethos-U NPUs.Keil

Best for: image classification, person/object detection, keyword/vision-trigger sensors, TinyML.

2. ExecuTorch — best if your models originate in PyTorch

ExecuTorch is now one of the strongest choices for constrained embedded deployment. It explicitly targets embedded systems and microcontrollers and has a lightweight C++ runtime.PyTorch Documentation+1

Its particularly useful features for your situation are:

• Ahead-of-time compilation
• Static memory planning
• Quantization
• Operator selection so the runtime can remain small
• Cortex-M/CMSIS-NN support
• Arm Ethos-U support
• DSP/NPU-specific backends

ExecuTorch's memory planner is specifically designed to reduce intermediate-tensor memory requirements, which can matter as much as the model weights themselves.PyTorch Documentation+1

Best for: PyTorch-trained CV models, especially if you're targeting modern Arm MCU + NPU platforms.

3. ONNX Runtime — excellent once you have a little more hardware

ONNX Runtime is a strong choice when "embedded" means something like a Raspberry Pi, embedded Linux SoC, Jetson-class device, or an MCU-class system with substantially more resources.

It supports IoT/edge deployment and a large collection of hardware-specific execution providers, including XNNPACK, Arm-related accelerators, Qualcomm QNN, OpenVINO, and others.ONNX Runtime+1

The tradeoff is that it's generally not my first choice for a device with only a few hundred KB of RAM. LiteRT Micro or ExecuTorch's MCU-oriented configurations are better suited there.

Best for: embedded Linux and heterogeneous hardware where portability matters.

4. CMSIS-NN — use it when your sensor is Cortex-M

Strictly speaking, CMSIS-NN isn't a complete inference framework like ONNX Runtime. It's a collection of highly optimized neural-network kernels for Arm Cortex-M processors.

But for an ultra-constrained vision sensor, it's extremely important because your runtime can use these kernels underneath LiteRT or ExecuTorch. LiteRT's Arm integration, for example, exposes a CMSIS-NN-accelerated path.Keil

Best for: squeezing maximum performance/Watt out of Cortex-M4/M7/M33/M55-class devices.

5. Vendor-specific runtimes

If your sensor contains an NPU, the vendor's runtime/compiler may ultimately beat a generic engine.

For example, Ethos-U is particularly interesting for Arm-based embedded vision. ExecuTorch can compile quantized models specifically for Ethos-U55/U65/U85 NPUs, with explicit control over SRAM/external-memory placement.PyTorch Documentation

This can make a huge difference when your bottleneck is not model size but activation SRAM.

What I'd choose

A practical decision tree:

• <1 MB RAM, bare-metal/RTOS Cortex-M: LiteRT Micro + CMSIS-NN
• PyTorch model + Cortex-M / Ethos-U: ExecuTorch
• Cortex-M + Ethos-U NPU: ExecuTorch or LiteRT + Ethos-U
• 1–10+ MB RAM, embedded Linux: ONNX Runtime
• Vendor NPU: use ExecuTorch/ONNX Runtime if supported, but benchmark against the vendor SDK
• Absolute minimum footprint: consider a highly specialized CMSIS-NN/vendor-kernel implementation rather than a general-purpose runtime

For computer vision specifically, model architecture and quantization will often matter more than the inference engine. A compact MobileNet/ShuffleNet-style network quantized to INT8 can make a dramatically bigger difference than switching between two reasonably lightweight runtimes. ExecuTorch's current embedded workflow, for example, explicitly uses quantization, graph compilation, and memory planning for micro-edge deployments.PyTorch

If you tell me the MCU/SoC (e.g. STM32H7, ESP32-S3, Cortex-M55, NXP i.MX, Ambiq, etc.) and available RAM/Flash, I can narrow this to the best 2–3 engines and give you a concrete deployment stack.

The tradeoff is that it's generally not my first choice for a device with only a few hundred KB of RAM. LiteRT Micro or ExecuTorch's MCU-oriented configurations are better suited there.

Best for: embedded Linux and heterogeneous hardware where portability matters.

4. CMSIS-NN — use it when your sensor is Cortex-M

Strictly speaking, CMSIS-NN isn't a complete inference framework like ONNX Runtime. It's a collection of highly optimized neural-network kernels for Arm Cortex-M processors.

But for an ultra-constrained vision sensor, it's extremely important because your runtime can use these kernels underneath LiteRT or ExecuTorch. LiteRT's Arm integration, for example, exposes a CMSIS-NN-accelerated path.Keil

Best for: squeezing maximum performance/Watt out of Cortex-M4/M7/M33/M55-class devices.

5. Vendor-specific runtimes

If your sensor contains an NPU, the vendor's runtime/compiler may ultimately beat a generic engine.

For example, Ethos-U is particularly interesting for Arm-based embedded vision. ExecuTorch can compile quantized models specifically for Ethos-U55/U65/U85 NPUs, with explicit control over SRAM/external-memory placement.

This can make a huge difference when your bottleneck is not model size but activation SRAM.

What I'd choose

A practical decision tree:

For very memory-constrained embedded vision, the best engine depends heavily on whether you're targeting a true MCU (tens/hundreds of KB RAM) or a Linux-capable edge SoC (hundreds of MB+).

1. TensorFlow Lite for Microcontrollers

TensorFlow Lite for Microcontrollers (TFLM) is probably the safest starting point if you're talking about sensors with only tens or hundreds of KB of RAM.

It's specifically designed for DSPs, microcontrollers, and other memory-constrained targets. GitHub Typical MCUs may have only a few hundred KB of RAM and relatively slow CPUs, which is exactly the environment TFLM targets.TensorFlow Blog

For computer vision, pair it with:

• INT8 quantization
• CMSIS-NN on Arm Cortex-M
• small MobileNet/EfficientNet-Lite-style models
• static tensor allocation
• grayscale or low-resolution inputs where possible

Best for: object detection/classification on Cortex-M-class sensors.

2. ExecuTorch

ExecuTorch is particularly interesting if your models are developed in PyTorch.

Its core runtime can be under 50 KB without kernels/backends, and its design supports bare-metal embedded environments without an OS, dynamic memory, or threads.GitHub

It supports embedded targets ranging from microcontrollers through more capable edge devices, including Arm Ethos-U and Cadence Xtensa backends.PyTorch Documentation

Best for: PyTorch-based vision pipelines where you want one deployment stack spanning MCUs and larger edge devices.

3. CMSIS-NN

Arm CMSIS-NN isn't really a complete inference framework by itself; it's a collection of highly optimized neural-network kernels for Cortex-M processors.

In practice, TFLM + CMSIS-NN is a particularly strong combination for tiny Arm-based vision sensors. TensorFlow specifically documents using CMSIS-NN to accelerate TFLM on Arm microcontrollers.TensorFlow Blog

Best for: squeezing maximum performance out of Cortex-M4/M7/M33/M55-class hardware.

4. ONNX Runtime with a custom/minimal build

ONNX Runtime becomes attractive once your device is more capable—e.g. Cortex-A Linux hardware rather than a bare-metal MCU.

Its big advantage is that you can remove unused operators and build a runtime containing only what your model needs. Microsoft explicitly supports minimal builds and operator-kernel reduction for smaller deployment footprints.ONNX Runtime

You can also combine it with XNNPACK for optimized CPU inference.ONNX Runtime

Best for: embedded Linux devices where you need ONNX compatibility without shipping the full runtime.

5. Vendor-specific runtimes

If your sensor has a dedicated NPU/DSP, don't automatically choose a generic runtime. A vendor SDK can dramatically outperform a generic CPU runtime.

Examples include:

• Arm Ethos-U → ExecuTorch / TFLM integrations
• Qualcomm → vendor AI runtimes
• NXP → eIQ ecosystem
• Cadence Xtensa → optimized embedded backends
• STM32 → STM32-specific neural-network tooling
• Espressif → ESP-specific optimized kernels

The tradeoff is portability: you become more dependent on the silicon vendor.

What I'd choose

If your memory budget is roughly:

• < 256 KB RAM: start with TFLM + CMSIS-NN, aggressively INT8-quantized.
• 256 KB–2 MB RAM: TFLM/CMSIS-NN or ExecuTorch, depending on your model ecosystem.
• 2–64 MB RAM: ExecuTorch becomes very attractive; vendor NPU runtime if available.
• 64+ MB RAM / embedded Linux: ONNX Runtime custom build + XNNPACK is a strong general-purpose choice.
• Dedicated NPU: prioritize the vendor backend, then select TFLM/ExecuTorch/ORT as the higher-level deployment layer.

For computer vision specifically, I'd optimize the model before obsessing over the runtime: INT8 quantization + reduced input resolution + depthwise-separable convolutions + static memory planning can make a much larger difference than switching between inference engines.

If you tell me your MCU/SoC model, RAM/flash budget, camera resolution, and whether the model is PyTorch/ONNX/TensorFlow, I can narrow this to the best 2–3 options and give you an estimated memory/performance tradeoff.

Arm CMSIS-NN isn't really a complete inference framework by itself; it's a collection of highly optimized neural-network kernels for Cortex-M processors.

In practice, TFLM + CMSIS-NN is a particularly strong combination for tiny Arm-based vision sensors. TensorFlow specifically documents using CMSIS-NN to accelerate TFLM on Arm microcontrollers.TensorFlow Blog

Best for: squeezing maximum performance out of Cortex-M4/M7/M33/M55-class hardware.

4. ONNX Runtime with a custom/minimal build

ONNX Runtime becomes attractive once your device is more capable—e.g. Cortex-A Linux hardware rather than a bare-metal MCU.

Its big advantage is that you can remove unused operators and build a runtime containing only what your model needs. Microsoft explicitly supports minimal builds and operator-kernel reduction for smaller deployment footprints.onnxruntime.ai

You can also combine it with XNNPACK for optimized CPU inference.

Best for: embedded Linux devices where you need ONNX compatibility without shipping the full runtime.

5. Vendor-specific runtimes

If your sensor has a dedicated NPU/DSP, don't automatically choose a generic runtime. A vendor SDK can dramatically outperform a generic CPU runtime.