Mobile lightweight on-device AI frameworks

Building a lightweight, on-device AI agent for mobile requires splitting your choice into two layers: the Inference Engine (which runs the small language model locally on iOS/Android hardware) and the Agent Orchestration/Logic Layer (which handles state, tool/function calling, and memory).

Because traditional heavy agent frameworks (like LangChain or CrewAI) are designed for server-side Python, building strictly on-device requires leaner, low-overhead tools.

Phase 1: Choosing the On-Device Inference Runtime

The runtime executes your quantized model (e.g., a 1B to 3B parameter model like Llama-3-8B-Instruct quantized, or Phi-3/Gemini Nano equivalents) using CPU/GPU/NPU acceleration.medium.com+1

1. llama.cpp / GGML (Best for Open-Source Flexibility)
   • What it is: A pure C/C++ inference engine optimized for running GGUF-quantized models locally.
   • Why it fits: Extremely lightweight, massive model compatibility, and runs exceptionally well on mobile CPUs and metal/opencl.
   • Best for: Complete freedom over what open-source models you deploy and absolute cross-platform parity.picovoice.ai+1
2. ExecuTorch (Best for Meta/PyTorch Ecosystem & NPU Acceleration)
   • What it is: Meta’s on-device AI framework built to carry exported PyTorch models natively to mobile and edge backends.
   • Why it fits: Features incredible AOT (Ahead-of-Time) memory planning and first-class hardware delegation (like Qualcomm QNN for Snapdragon NPUs).
   • Best for: Production-grade performance where tight NPU hardware acceleration is mandatory.picovoice.ai+1
3. Google MediaPipe LLM Inference API (Best for Android/Quick Start)
   • What it is: Google's on-device solution packaged cleanly for mobile developers.
   • Why it fits: Offers direct, high-level APIs for running lightweight tasks (like Gemma models) on Android/iOS with minimal boilerplate.
   • Best for: Fast prototyping if you plan to stick closely to supported Google/Gemma models.meetprajapati.com
4. Core ML (Best for Native iOS)
   • What it is: Apple's native framework optimized directly for Apple Neural Engine (ANE) hardware.
   • Why it fits: Unmatched power efficiency and speed on iPhones and iPads.medium.com

Phase 2: Choosing the Agent & Function-Calling Framework

Once your runtime is set, your agent needs to parse intents, execute local device tools (like reading local SQLite databases, triggering notifications, or fetching local device states), and manage state.

1. Custom State Machine / Lightweight State Graph (Recommended for Mobile)
   • The Approach: Because mobile apps have strict memory limits and battery constraints, complex reflection loops can cause jank. Writing a lightweight state machine (using Kotlin Coroutines on Android or Swift Async/Await on iOS) to handle a deterministic "Observe → Think → Act" loop is usually safer and faster than a heavy framework.
2. Guided Generation / Structured Outputs via Grammars
   • The Tool: Use json-schema or grammar constraints directly at the inference layer (supported natively by llama.cpp via grammars). This forces the tiny on-device model to output valid JSON for tool calls every single time, drastically reducing agent failures caused by model hallucinations.

Summary Recommendation Matrix

• If you want a proven, flexible open-source stack , pair llama.cpp (via cross-compiled mobile wrappers) with a custom state-machine loop implementing strict JSON-schema grammar constraints for tool use.
• If you are building deeply into Android/Snapdragon NPUs or PyTorch workflows , go with ExecuTorch.
• If you want the fastest path on iOS , build directly on Core ML combined with native Swift async execution loops.alephzerolabs.com+1

To help narrow down the ideal stack, tell me:

• Are you targeting iOS only, Android only, or cross-platform (e.g., Flutter, React Native)?
• Do you have a specific model size in mind (e.g., sub-1B, 3B parameters)?

Best frameworks for lightweight on-device AI agents

My default recommendation is Google’s LiteRT-LM for a cross-platform, native mobile app, or ExecuTorch if you want a PyTorch-centered stack with more control over model deployment. If you’re building with React Native, there are also integrations that make local inference much easier.

The important distinction is that an on-device AI agent needs more than an LLM inference engine. It needs to run a model locally, decide which tools to call, manage state and context, and execute actions safely — all within a phone’s memory, battery, and latency constraints.

Google’s LiteRT-LM now explicitly supports cross-platform LLM execution, hardware acceleration, multimodal inputs, and tool use, making it a strong starting point for this kind of app.developers.google.com+1

My shortlist

• 

1. LiteRT-LM

(Best default for native cross-platform)

Android · iOS · C++ · Web

A purpose-built LLM runtime with GPU acceleration, supported multimodal models, and function calling. Particularly compelling if you want a local agent that invokes app functions without sending user data to a cloud model.developers.google.com+1

Official documentation

---

• 

2. ExecuTorch

Android · iOS · embedded

My pick if you want a flexible production stack and are comfortable with model export and native integration. It supports LLM deployment, quantization, and hardware-specific backends. It's especially attractive when your ML pipeline already uses PyTorch.docs.pytorch.org+1

Official documentation

---

• 

3. llama.cpp

Android · iOS · desktop

Best when model choice and portability matter most. Its GGUF ecosystem offers broad model flexibility and quantized models. The trade-off is more native integration work, especially if you need polished mobile APIs, robust tool calling, and lifecycle management.edgeaistack.ai+1

GitHub repository

---

• 

4. React Native LiteRT-LM integration

React Native · iOS · Android

Worth considering if your app UI is already JavaScript or TypeScript. The community react-native-litert-lm package exposes local model execution, streaming, and tool-call events without requiring you to build every native bridge yourself. Verify package maturity and platform support before committing.litert.dev

Project and documentation

What I would actually build

For a lightweight agent, I'd use a small local model plus a deterministic tool-execution loop, rather than a heavyweight multi-agent orchestration framework.

Mobile app

Native Swift/Kotlin or React Native

Agent controller

State · context · permissions · bounded loop

Local LLM

LiteRT-LM or ExecuTorch

Tool registry

Typed app functions and local data

The agent requests a tool call; the controller validates and executes it, then returns the result to the model.

Keep the first version small

• Model: Start by testing a quantized model in the roughly 0.5B–2B parameter range. Smaller isn't always better; benchmark task accuracy as well as latency.
• Agent loop: Limit the number of tool calls and total execution time. Avoid letting the model execute arbitrary code.
• Memory: Store conversation state and structured user preferences locally. Add embeddings and retrieval only if needed.
• Permissions: Require confirmation for consequential actions, and validate every tool argument.
• Performance: Measure peak RAM, time to first token, battery use, and cold-start latency on actual target phones.

One important caveat: model size on disk is not the same as runtime memory. For example, Google's published LiteRT-LM measurements show that even a roughly 2.6 GB model can have substantially different memory use across devices and execution backends.developers.google.com+2

Which stack fits your app?

The best choice depends on a few architectural decisions.

1. Which platforms are you targeting?

iOS only

Android only

iOS and Android

---

2. How are you building the app UI?

Native Swift / Kotlin

React Native

Flutter

Not decided yet

---

3. What's the main priority?

Small footprint and fast responses

Maximum choice of models

Hardware acceleration and deployment control

---

4. What will the agent do?

Chat, summarize, or extract information

Call tools and perform app actions

Understand images, audio, or camera input

Run proactive or background workflows

[Recommend my stack]

My starting point would be LiteRT-LM with a small quantized model and a hand-written tool loop. If you're using React Native, investigate its LiteRT-LM integration first; if you need a broader model ecosystem and deeper control over export and backends, evaluate ExecuTorch alongside it.developers.google.com+2

For a mobile app that needs an AI agent running entirely on-device, I’d prioritize frameworks that support quantized small language models, hardware acceleration, streaming responses, and tool calling. The best choice depends on whether you’re building for iOS, Android, or both, and whether your agent needs to execute actions or simply generate text.

As of October 2026, these are the frameworks I’d evaluate first. One important distinction: an inference runtime runs the model locally, while an agent layer manages conversation state, tool execution, memory, and multi-step workflows. You may need both.

Recommended frameworks

• 

1. LiteRT-LM

(Best starting point for native mobile)

Google's current runtime for on-device LLMs, with streaming generation, conversation management, multimodal capabilities, and agentic tool calling. Supports Android and iOS, with additional web and desktop options.

Best for: A production mobile assistant that needs to call local app functions while keeping data on-device.

Trade-off: You need models compatible with its runtime and should benchmark performance on your target devices.

Official documentation

developers.google.comdevelopers.google.cndevelopers.google.comdevelopers.google.cn

---

• 

2. llama.cpp

(Best for lightweight GGUF models)

A portable C/C++ inference engine designed to run quantized models efficiently on CPUs and supported GPUs. It has a mature ecosystem and gives you direct control over model loading and inference.

Best for: Offline-first apps, compact local models, custom inference pipelines, and developers who want to control the runtime themselves.

Trade-off: You will generally build more of the agent orchestration and mobile integration yourself.

GitHub repository

---

• 

3. ExecuTorch

A PyTorch-focused deployment framework for mobile and edge inference, with support for hardware-specific backends and supported LLM architectures.

Best for: Teams already using PyTorch, custom models, and applications that need to combine language models with other ML workloads.

Trade-off: Model export, operator coverage, and backend compatibility can require more engineering than a ready-to-run model runtime.

Official documentation

---

• 

4. MLC LLM

A cross-platform LLM deployment framework that compiles models for supported hardware and runtimes, including mobile GPU paths.

Best for: Teams that want to optimize open-weight LLMs for iOS, Android, and potentially browser-based deployment.

Trade-off: Compilation, model packaging, and target-specific tuning add complexity.

Official project site

---

• 

5. ONNX Runtime Mobile

A versatile mobile inference runtime for ONNX models, with platform-specific acceleration options.

Best for: Apps that combine an LLM with embeddings, classifiers, speech models, vision models, or other conventional ML components.

Trade-off: It is particularly attractive for general ML deployment; for a complete local LLM agent, check the current generation APIs and supported model path before committing.

Official mobile documentation

---

6. Core ML

Apple's native framework for running supported ML models on iPhone and iPad hardware, including available Neural Engine acceleration.

Best for: An iOS-only app, particularly one that combines a language model with native vision, audio, or other Apple-optimized ML features.

Trade-off: It is not a shared runtime for Android, and LLM conversion and generation workflows need to be checked for your chosen model.

Official documentation

Google's current documentation specifically directs LLM developers toward LiteRT-LM rather than its older MediaPipe LLM Inference API, which is now in maintenance-only mode. That makes LiteRT-LM a stronger starting point for a new Google AI Edge-based project.developers.google.com+2

Quick comparison

For further technical detail, see the official guides for ExecuTorch, MLC LLM, and ONNX Runtime Mobile.docs.pytorch.org+2

How I’d build the agent

A practical starting architecture would look like this:

Mobile app UI

Swift / Kotlin / React Native / Flutter

Agent controller

Conversation state · tool permissions · action loop

Local inference runtime

LiteRT-LM or llama.cpp · quantized model

Local tools and data

App APIs · SQLite · search index · device features

To keep the app lightweight, start with a small instruction-tuned model, use 4-bit quantization where quality permits, limit the context window, and test memory usage and battery consumption on actual devices. Let the agent invoke only explicitly permitted app functions, and validate every tool call in application code.

Three questions to narrow the recommendation

1. Which platforms are you targeting?

iOS only

Android only

Both iOS and Android

---

2. What must the agent do?

Offline chat and text generation

Call tools, execute app actions, and follow multi-step plans

Search local documents or a personal knowledge base

Understand images, audio, or voice commands

---

3. What is your app technology?

Native Swift / Kotlin

React Native

Flutter

Other / undecided

---

Optional: How much memory can the model use?

Strict limit; older or lower-end phones

Typical modern smartphone

Newer flagship devices; larger models acceptable

[Get a tailored recommendation]

Developing a local, on-device AI agent for mobile requires balancing strict constraints: memory overhead (RAM), thermal throttling, battery drain, and Neural Processing Unit (NPU) utilization.

The top-recommended frameworks for lightweight, cross-platform or native on-device deployment are organized below by their architectural strengths.

1. llama.cpp / GGML (The Industry Standard for GGUF Models)

• Best For: Cross-platform (iOS & Android) raw performance, custom tool-use/function calling, and running quantized open-weights models (like Llama 3/3.1, Qwen 2.5/3.5, or Gemma derivatives).docs.liquid.ai+1
• How it works: Written in pure C/C++ with zero dependencies. It compiles directly into mobile apps via native bindings or Swift/JNI wrappers.docs.liquid.ai+1
• Mobile Advantages:
  • Exceptional hardware acceleration via Apple Metal on iOS and OpenCL/Vulkan/NNAPI on Android.
  • Supports GGUF quantization (4-bit, 5-bit, 8-bit), allowing a 3B–7B parameter model to fit comfortably within mobile RAM limits (using only 1.5GB–3.5GB of memory).
  • Native parsing support for structured outputs and tool/function calling.docs.liquid.ai
• Integration Tip: Use pre-compiled wrappers like Stanford SpeziLLM for Swift/iOS or build via the native C-API for cross-platform wrappers.github.com+1

2. Google MediaPipe / LiteRT (Formerly TensorFlow Lite)

• Best For: Android-first or deeply integrated Google ecosystem apps, ultra-lightweight task-specific models, and cross-platform pipelines.developers.google.com
• How it works: Google’s MediaPipe Framework and LiteRT (the evolution of TFLite) are optimized for mobile NPUs and edge hardware.linkedin.com+1
• Mobile Advantages:
  • Deep hardware-level optimization, targeting Qualcomm Hexagon NPUs, Apple Neural Engine, and mobile GPUs seamlessly.
  • Ideal if you plan to use smaller, specialized agentic models like Google’s FunctionGemma (purpose-built for fast, on-device function calling and tool use).
  • Highly predictable memory footprint and minimal binary size impact compared to full LLM runtimes.linkedin.com

3. MLX (Apple Silicon Native)

• Best For: Native iOS/iPadOS/macOS apps where you only target Apple hardware.
• How it works: Apple’s machine learning framework designed specifically for unified memory architecture on Apple silicon.
• Mobile Advantages:
  • Allows operations to share memory seamlessly between the CPU and GPU with virtually zero copy overhead.
  • Extremely fast token generation speeds on modern iPhones and iPads equipped with high-end A-series and M-series chips.
  • Perfect if your app ecosystem leans heavily into Swift and Apple's native ML stack.

4. Native Platform APIs (Apple Foundation Models & AICore)

• Best For: Maximum system integration with zero binary bloat.
• How it works: Leveraging platform-native intelligence layers—such as Apple’s Foundation Models framework on iOS or Android's system-level AI components.linkedin.com
• Mobile Advantages:
  • The OS manages memory swapping, power states, and NPU prioritization globally.
  • Zero impact on your app's download size since you aren't bundling model weights or heavy runtime binaries.
  • Trade-off: Limited to newer device tiers that support the respective OS-level on-device models.

Key Rules of Thumb for Mobile Agent Architecture:

• Model Size: Stick to 1B to 4B parameters max quantized to 4-bit (GGUF Q4_K_M or similar). Anything above 7B will aggressively trigger out-of-memory (OOM) killers or thermal throttling on standard smartphones.
• Agent Loops: Keep the reasoning loop short. Multi-step recursive reflection agents will drain a mobile battery in under an hour; design your agent to lean on deterministic client-side code for repetitive logic, using the local LLM strictly for intent parsing, entity extraction, and lightweight decision-making.

To help narrow down the best choice, tell me:

• Are you targeting iOS only, Android only , or Cross-Platform (e.g., React Native, Flutter)?
• What specific tasks does your AI agent need to perform locally (e.g., text generation, local RAG, strict function calling)?