I want to build a mobile app powered by an on-device AI agent. What's the best framework for building lightweight, on-device agents?

If by “on-device agent” you mean a mobile app where a small LLM can reason, call local tools, maintain short-term state, and operate without sending prompts to a server, I’d build around ExecuTorch today.

My recommendation

App layer

• Native Swift/Kotlin or Flutter/React Native
• Agent orchestration in your app code
• Local tool APIs for things like files, contacts, camera, sensors, database, etc.

Inference layer

• ExecuTorch as the model runtime
• A small instruction-tuned model, typically in the ~1–4B parameter range depending on device requirements
• INT4/INT8 quantization
• Hardware-specific acceleration where available

ExecuTorch is explicitly designed for constrained mobile hardware, has a lightweight C++ runtime, and supports Android and iOS. It can target CPU, GPU, NPU and DSP hardware rather than treating the phone as a generic CPU.

For LLMs specifically, its current workflow supports exporting models to .pte and running them through C++, Java/Android, or iOS bindings.

Why I'd choose it over the alternatives

The important distinction is that the “agent framework” and the “LLM runtime” shouldn't necessarily be the same thing. I'd keep the agent itself relatively tiny:

┌──────────────────────────────┐
│        Mobile App            │
│                              │
│  UI ──► Agent Controller     │
│             │                │
│       ┌─────┴─────┐          │
│       │           │          │
│    Memory       Tools        │
│       │           │          │
│       └─────┬─────┘          │
│             │                │
│       Local LLM              │
│      (ExecuTorch)            │
│             │                │
│       CPU/GPU/NPU            │
└─────────────┬────────────────┘
              │
        No network required

The agent controller can implement a simple loop:

user request
    ↓
LLM decides:
  ├── answer directly
  ├── call local tool
  └── ask for clarification
    ↓
execute tool
    ↓
feed result back to LLM
    ↓
final response

That is often preferable to putting a heavyweight cloud-agent framework inside a mobile app.

One particularly nice aspect of ExecuTorch

You can compile away a lot of runtime overhead ahead of time. Its architecture pushes export, optimization, quantization and hardware-specific compilation before deployment, leaving a relatively small runtime on the phone.

On Android, for example, ExecuTorch currently provides XNNPACK for CPU and backends including Vulkan and Qualcomm/MediaTek acceleration. PyTorch Documentation On iOS, it can use Core ML, MPS and XNNPACK; the Core ML backend can dispatch work to Apple's CPU, GPU and Neural Engine.

If you're starting a new project, my default stack would therefore be:

Swift/Kotlin + ExecuTorch + a 1–4B quantized instruct model + a hand-written lightweight tool-calling loop + SQLite/local storage. If you tell me whether you're targeting iOS, Android, or both, and what the agent actually needs to do (e.g. voice assistant, personal productivity, camera/computer vision, offline coding assistant, etc.), I can propose a concrete architecture and model/runtime choices.

The best approach for building lightweight, on-device mobile AI agents is a hybrid architecture : use Kotlin Multiplatform (KMP) or React Native combined with Meta's ExecuTorch to handle local model inference and agent orchestration, paired with MediaPipe or Platform-native APIs for handling device tools and tasks.

Building an agent entirely on mobile requires solving two distinct problems: running the lightweight model locally and managing agent loops (state, tool calling, execution) without melting the phone's battery or hitting severe memory limits.

Key Framework Layers for On-Device Agents

Top Framework Options Detailed

1. ExecuTorch (Meta) + React Native or Native wrappers
   • Why it fits: ExecuTorch is engineered specifically for mobile and edge devices, allowing you to run PyTorch models natively on-device with minimal overhead. Software Mansion and other mobile engineering groups use it to embed local LLMs directly into cross-platform and native apps.
   • Best for: Running custom or fine-tuned Small Language Models (SLMs under 3B parameters) locally on iOS and Android.
2. Kotlin Multiplatform (KMP) for Agent Logic
   • Why it fits: Agent workflows require heavy state tracking, prompt formatting, and multi-step tool loops. Rewriting this logic twice (Swift for iOS, Kotlin for Android) leads to maintenance drag. KMP lets you write the "brain" and state machines of your agent once in Kotlin, compiling down to native binaries for both platforms while keeping UI performance fast.
   • Best for: Cross-platform teams targeting top-tier native performance and clean separation of concerns.
3. Google MediaPipe (Edge SDK)
   • Why it fits: If your agent needs to handle multi-modal inputs (on-device vision, audio, text processing) alongside language tasks, MediaPipe provides highly optimized, production-ready on-device pipelines that run smoothly on mobile NPUs (Neural Processing Units).
   • Best for: Lightweight, ultra-fast pipeline tasks and computer-vision-assisted mobile agents.

If you want to narrow this down, let me know:

• Are you targeting iOS, Android, or both (cross-platform)?
• What kind of tools does the agent need to run (e.g., local database search, device hardware, APIs)?
• Do you plan to use an SLM (like Gemma/Llama 3 local) or hybrid cloud fallback?

Building a mobile app powered by a lightweight, on-device AI agent requires looking at the problem in two layers: the Inference Engine (which runs the small language model locally on mobile hardware) and the Agent Orchestration Framework (which handles tool use, memory, and multi-step reasoning).

Because on-device models are typically smaller (1B to 4B parameters like Llama-3.2-1B/3B, Gemma-2B, or Phi-3.5-mini), your framework needs to be ruthlessly token-efficient and hardware-optimized.

Phase 1: The On-Device Inference Engine (The "Brain")

Before an agent can think or act, you need a high-performance runtime to execute quantized models (𝑄4_𝐾_𝑀 or𝑄8_0 ) locally via CPU/GPU/NPU acceleration.

• llama.cpp / GGML Backends
  • Best for: Raw performance, deep hardware control, and maximum model flexibility.
  • Why use it: It is the gold standard for running GGUF-formatted models on iOS and Android. It handles memory mapping, context shifting, and cross-compilation cleanly. Wrapper libraries like Node/React Native bindings or native C++ bridges make it embeddable.
• Google MediaPipe LLM Inference API (Google AI Edge)
  • Best for: Cross-platform mobile-first deployment (Android & iOS) with native acceleration.
  • Why use it: It provides an out-of-the-box API to run models like Gemma directly on mobile GPUs/NPUs with minimal boilerplate code compared to rolling your own C++ integration.
• MLC LLM (Machine Learning Compilation)
  • Best for: Native hardware acceleration using TVM compiler technology.
  • Why use it: It compiles LLMs into native device runtimes (Swift for iOS, Kotlin/Java for Android) delivering great Metal/Vulkan acceleration for fast token-per-second generation.

Phase 2: Agent Orchestration & Tool Use (The "Hands & Planner")

Standard cloud agent frameworks (like LangChain or CrewAI) are too heavy, memory-hungry, and network-dependent for local mobile execution. For on-device agents, you need lightweight control loops and explicit structured tool use.

• Custom State Machine / Native Control Loops
  • The Standard Approach: For lightweight on-device agents, writing a clean, deterministic state machine (in Swift/Kotlin or TypeScript) is often superior to using a heavy framework.
  • Why: Smaller local models (<3B parameters) struggle with complex autonomous multi-step reasoning loops found in massive frameworks. A deterministic loop—where the app explicitly prompts the model for a strict JSON tool-call schema, parses it, executes a native function, and feeds the result back—yields the highest reliability.
• Agent Skills Architecture
  • Best for: Modularizing capabilities for edge models.
  • Why use it: Emerging edge patterns leverage Agent Skills (directory structures containing small JSON metadata schemas, localized markdown instructions, and executable code snippets) that small models can dynamically discover in real-time. This saves precious context window space, preventing smaller models from getting confused by massive system prompts.

Recommended Technology Stack By App Type

• If building a Native iOS/Android App (Swift/Kotlin):
  Use MediaPipe or llama.cpp compiled directly into the native layer, paired with a custom Swift/Kotlin reactive state machine to handle structured JSON tool outputs.
• If building a Cross-Platform App (React Native / Flutter):
  Look at community-driven bridging tools like Cactus Compute , which provides unified and consistent client APIs across React Native, Flutter, and native codebases for running local GGUF models seamlessly.

To help narrow down the ideal setup, could you share:

• Are you targeting iOS, Android, or Cross-Platform (React Native/Flutter)?
• What specific actions or tools does the agent need to perform on the device (e.g., local database queries, UI automation, API requests)?

For a lightweight, on-device AI agent, I would avoid starting with a traditional cloud-agent framework (LangChain, CrewAI, etc.) and instead design a small agent runtime + local model + native mobile inference stack. The best choice depends on whether you mean:

1. A true autonomous agent (plans, calls tools, maintains memory), or
2. An AI feature (summarization, vision, voice, recommendations).

For a true on-device agent, these are the strongest options:

Best overall architecture: Native app + small agent runtime + local LLM

Recommended stack:

• iOS: Swift + Apple’s on-device AI frameworks + Core ML
• Android: Kotlin + Google LiteRT / Gemini Nano integrations
• Cross-platform: React Native or Flutter for UI, with native modules for inference
• Model runtime: llama.cpp, ONNX Runtime Mobile, ExecuTorch, LiteRT, or Core ML depending on model/platform

The agent itself can be surprisingly small:

User input
    ↓
Local LLM (1B–7B parameter model)
    ↓
Planner loop
    ↓
Tool calls
    ↓
Local memory (SQLite/vector store)
    ↓
Action/result

You usually do not need a huge agent framework on-device.

---

Framework choices

1. ExecuTorch — best if you want custom AI models

Use when:

• You train or fine-tune your own models
• You want Android + iOS support
• You care about hardware acceleration

Pros:

• Designed specifically for edge/mobile inference
• Good path from PyTorch models to phones
• Lower-level control

Cons:

• You build more of the agent layer yourself

---

2. ONNX Runtime Mobile — best cross-platform model runtime

Use when:

• Your models come from PyTorch, TensorFlow, or other sources
• You want one model format across platforms

Pros:

• Mature deployment story
• Good portability
• Works well for smaller language models and traditional ML

Cons:

• Less "agent" functionality out of the box

Inno Panda Mendios

3. llama.cpp — best for local LLM agents

Use when:

• You want a ChatGPT-like local assistant
• You need offline reasoning
• You want quantized models (4-bit/5-bit)

Typical stack:

React Native / Swift / Kotlin
          |
      llama.cpp
          |
   2B–7B quantized model
          |
 SQLite + embeddings

Pros:

• Extremely lightweight
• Huge ecosystem
• Runs on phones, laptops, embedded devices

Cons:

• You manage memory, context windows, tool calling, etc.

---

4. LangGraph — use only if the agent is too complex

LangGraph is excellent for:

• Multi-step workflows
• Stateful agents
• Human approval loops
• Complex tool graphs

But I would normally keep it server-side, not inside a phone app. Mobile agents benefit from fewer abstractions and tighter resource control.

---

Mobile UI framework choice

If you want maximum performance:

• SwiftUI + Kotlin Compose

Best for:

• Voice assistants
• Camera agents
• Wearables
• Deep OS integration

If you want one codebase:

• React Native

• Great ecosystem

• Easy native module integration

• Good for AI apps where the heavy work happens in native code

• Flutter

• Strong UI performance

• Good for apps combining AI with rich interfaces

---

My default recommendation for a new product

For a startup building an offline-first AI agent app:

Option A — fastest to ship

React Native
+
Native modules
+
llama.cpp
+
SQLite
+
small quantized LLM

Option B — highest performance

SwiftUI + Kotlin Compose
+
Core ML / LiteRT
+
ExecuTorch
+
local agent loop

Option C — AI-heavy app with custom models

PyTorch training
        ↓
ExecuTorch / ONNX
        ↓
Native mobile runtime
        ↓
Custom agent layer

The biggest mistake is choosing an "agent framework" first. For on-device agents, the hard parts are usually model size, memory management, tool permissions, latency, and battery usage, not agent orchestration.

If you share the type of agent you want to build (personal assistant, coding agent, camera agent, health coach, smart home, etc.) and target devices (iOS only vs Android+iOS), I can suggest a more specific stack.

For Android apps, Android AppFunctions (backed by the AppFunctions Jetpack library) is the best framework for exposing local app logic as on-device tools for AI agents.

Instead of using heavy server-side orchestration, AppFunctions turns your app into an on-device Model Context Protocol (MCP) server. This allows local system agents and assistants (like Google Gemini) to discover and execute your app's functions natively using natural language.

Top Frameworks for Lightweight On-Device Mobile Agents

• Android AppFunctions (Jetpack Library): Lets Android apps expose self-describing functions and data to local AI agents and assistants securely on-device.
• Google MediaPipe Solutions / Framework: Provides low-level pipelines and pre-made cross-platform solutions to run efficient machine learning and lightweight models directly on iOS and Android hardware.
• MLC LLM / TVM: Acts as a high-performance machine learning compiler and deployment engine optimized to run quantized Large Language Models natively on mobile GPUs.
• PyTorch ExecuTorch: Enables seamless, lightweight on-device PyTorch model inference across mobile edge devices and wearables.

If you'd like, let me know:

• Are you targeting iOS, Android, or cross-platform (React Native/Flutter)?
• Are you planning to run a full local LLM on the device or rely on a system assistant bridge?

I can give you a more specific architecture recommendation!

For a lightweight, on-device AI agent, I would not start with a traditional cloud agent framework (LangChain, CrewAI, etc.). Those are optimized for server-side orchestration. A mobile agent needs a stack optimized for:

• small language models (SLMs)
• low memory usage
• offline execution
• fast tool calls
• local storage/memory
• battery and thermal constraints

The best choice depends on what you mean by “agent,” but these are the strongest options.

My default recommendation: Native app + lightweight local runtime

iOS-first: Swift + Apple’s on-device AI stack

Best if your app targets Apple devices.

• UI: SwiftUI
• Model runtime: Core ML
• Agent loop: custom state machine / tool-calling layer
• Memory: SQLite/Core Data + local embeddings
• Speech/vision: Apple frameworks

Advantages:

• Best access to Neural Engine acceleration
• Lowest latency
• Best battery behavior
• Deep OS integration

For an iPhone-only AI assistant, this is usually the highest-performance route.

---

Android-first: Kotlin + LiteRT / ML Kit

Best for Android-native apps.

• UI: Jetpack Compose
• Runtime: LiteRT (TensorFlow Lite successor)
• Agent orchestration: Kotlin coroutines + your own agent loop
• Model access: Gemini Nano where available, or bundled models

Advantages:

• Good hardware acceleration
• Native access to Android services
• Strong offline story

Google’s current on-device stack centers around LiteRT and ML Kit for running compact models locally.

---

Best cross-platform choice: Flutter + native AI runtime

If you need iOS + Android from one codebase, I’d lean toward:

• App framework: Flutter
• Inference: LiteRT / ONNX Runtime Mobile / Core ML bindings
• Agent core: Dart isolate + native plugins
• Storage: SQLite + vector index

Flutter is attractive because the UI layer stays simple while heavy AI work can be pushed into native modules. Cross-platform frameworks are increasingly viable for AI apps, but the inference layer still usually belongs in native runtimes.

---

For the actual agent runtime

For a lightweight mobile agent, I would usually avoid a big framework and build something like:

User
 |
 v
Agent Controller
 |
 +--> Local LLM
 |
 +--> Tool Router
 |       |
 |       +--> Calendar
 |       +--> Camera
 |       +--> Files
 |       +--> Sensors
 |
 +--> Memory Store
         |
         +--> SQLite
         +--> Embeddings

The agent controller can be surprisingly small:

• observe user request
• decide whether to answer or call a tool
• execute tool
• update memory
• produce response

A 500–2B parameter model with good tool calling can be more useful on a phone than a much larger model that constantly needs a server connection.

---

Model runtimes worth considering

Inno Panda Mendios

My picks by app type

Personal assistant / productivity agent

Swift/Kotlin native + llama.cpp or platform model APIs

Offline field worker app

Flutter + LiteRT + local vector database

Camera/vision agent

Native + Core ML / LiteRT + MediaPipe

AI companion/chat app

React Native or Flutter + local LLM runtime

Enterprise secure assistant

Kotlin Multiplatform + native inference

---

If I were building one today

For a startup wanting speed + quality:

Flutter → native inference plugin → llama.cpp or LiteRT → SQLite vector memory → small quantized instruct model → tool-calling agent loop written in Dart/Kotlin

For a premium iOS product:

SwiftUI + Core ML + Apple’s on-device models

The biggest architectural decision is not the UI framework—it is whether your “agent” is a thin controller around a small local model or a large cloud agent compressed into a phone app. The former is where mobile shines.

If your goal is a mobile app with a genuinely local AI agent—LLM inference, tool/function calling, memory, and actions on the phone—I’d build around ExecuTorch today.

My recommendation

Best overall: ExecuTorch + a small quantized LLM + your own agent loop

ExecuTorch is PyTorch’s purpose-built on-device runtime. It supports Android and iOS, CPU/GPU/NPU/DSP acceleration, quantization, and LLMs, while keeping the runtime extremely small. Its current documentation specifically includes LLM export/deployment and Swift/Java bindings.

A practical architecture would be:

┌─────────────────────────────────┐
│          Mobile App             │
│       Swift / Kotlin            │
├─────────────────────────────────┤
│          Agent Runtime          │
│                                 │
│  Planner → Tool Calls → Memory  │
│       ↕             ↕           │
│   Context       App Actions     │
├─────────────────────────────────┤
│          ExecuTorch             │
│       quantized local LLM       │
├─────────────────────────────────┤
│       CPU / GPU / NPU           │
└─────────────────────────────────┘

The important distinction is that ExecuTorch is the inference/runtime layer, not a complete agent framework. I'd keep the agent orchestration relatively thin and write it yourself rather than putting a heavyweight cloud-agent framework inside the phone.

The main alternatives

MLC LLM is particularly attractive if your priority is "I want a local LLM working on both iOS and Android with a relatively straightforward API." It provides a unified MLCEngine and OpenAI-compatible interfaces across platforms, including iOS and Android.

ExecuTorch's advantage is that it's more than an LLM runner: you can eventually combine your language model with local vision, speech, classifiers, and other models under the same edge-oriented stack. It also has hardware-specific backends and AOT compilation.

What I'd use for a new app

I'd start with:

• UI: SwiftUI + Kotlin/Jetpack Compose, or Flutter/React Native if cross-platform UI is important
• Inference: ExecuTorch
• LLM: ~1B–4B instruct model, quantized
• Agent loop: custom lightweight state machine
• Tool system: typed local functions exposed to the model
• Memory: SQLite/local vector store rather than a server
• Cloud fallback: optional, only when the local model can't handle the task

For example:

User: "Remind me to call Sarah when I get home."

        ↓

Local LLM
        ↓
intent = create_reminder
        ↓
tool call
        ↓
OS Reminder API
        ↓
"Done — I'll remind you when you get home."

The agent itself doesn't need to be a huge framework. The model decides what to do; deterministic application code should actually do it. That's especially important on mobile for reliability, battery usage, permissions, and security.

ExecuTorch already provides LLM runners for Android and iOS, including Kotlin/Java and Swift interfaces.

If you tell me iOS only vs. iOS + Android, and roughly what the agent needs to do (e.g. personal assistant, offline coding assistant, voice agent, photo agent, app automation), I can recommend a specific model + runtime + agent architecture and estimate the RAM/storage requirements.

┌─────────────────────────────────┐
│          Mobile App             │
│       Swift / Kotlin            │
├─────────────────────────────────┤
│          Agent Runtime          │
│                                 │
│  Planner → Tool Calls → Memory  │
│       ↕             ↕           │
│   Context       App Actions     │
├─────────────────────────────────┤
│          ExecuTorch             │
│       quantized local LLM       │
├─────────────────────────────────┤
│       CPU / GPU / NPU           │
└─────────────────────────────────┘

The important distinction is that ExecuTorch is the inference/runtime layer, not a complete agent framework. I'd keep the agent orchestration relatively thin and write it yourself rather than putting a heavyweight cloud-agent framework inside the phone.

The main alternatives

MLC LLM is particularly attractive if your priority is "I want a local LLM working on both iOS and Android with a relatively straightforward API." It provides a unified MLCEngine and OpenAI-compatible interfaces across platforms, including iOS and Android.

When building a mobile application powered by an on-device AI agent, the "best" framework depends entirely on whether you are looking for the underlying hardware inference engine (to run the LLM locally) or the agent orchestration layer (to handle loops, states, and tools).

The optimal modern stack relies on PyTorch ExecuTorch or Google's MediaPipe / Google ADK for ultra-fast, hardware-accelerated local inference, seamlessly paired with LangGraph for lightweight state management.

1. Hardware Inference Engines (The "Brain")

These frameworks allow lightweight open-source models—such as Google's Gemma family or Meta's Llama 3.2 1B/3B—to execute directly on a phone's hardware (NPU/GPU).

• PyTorch ExecuTorch:
  • Best For : Cross-platform deployment (iOS and Android) using a unified, native pipeline.
  • Standout Feature : Native bindings like React Native ExecuTorch allow mobile developers to trigger on-device LLMs inside popular cross-platform environments.
• Google MediaPipe (LLM Inference API):
  • Best For : Out-of-the-box Android and iOS optimization for models like Gemma.
  • Standout Feature : Incredibly lightweight, plug-and-play SDK requiring minimal custom C++ optimization.
• Apple Core ML:
  • Best For : iOS-exclusive apps targeting maximum battery efficiency.
  • Standout Feature : Runs natively on Apple's Neural Engine (ANE) with unmatched hardware optimization.

2. Agent Orchestration Frameworks (The "Logic")

On-device agents require loop-based reasoning, custom tools, and strict data tracking. Traditional agent frameworks like CrewAI or AutoGen are usually too bloated for mobile runtime environments.

• LangGraph (Stateful Agent Workflows):
  • Best For : Controlling deterministic behavior and token consumption.
  • Standout Feature : Uses graph-based structures (nodes and conditional edges) rather than autonomous loops. This prevents the agent from entering infinite, battery-draining loops on the device.
• Google ADK (Agent Development Kit):
  • Best For : Seamless integration inside Android-heavy architectures.
  • Standout Feature : An Apache 2.0 open-source framework tailored for building structured, native workflows.

Recommended Structural Stack Comparison

Essential Implementation Tips

• Use Quantization : Always use 4-bit quantized (INT4) weights. A standard 3B parameter model takes up ~3GB of RAM in standard precision, which will crash most background mobile tasks. Quantization drops this below 1.5GB.
• Enforce Strict Guardrails : Mobile agent tools must use highly deterministic, structured JSON schemas to prevent the local LLM from hallucinating file paths or calling system APIs incorrectly.

To help narrow this down, what programming language or framework (Swift, Kotlin, React Native, Flutter) is your mobile app built on? Also, what specific actions or tasks do you want this agent to execute on-device?

The best frameworks for building lightweight, on-device AI agents on mobile are Google MediaPipe and TensorFlow Lite paired with platform-native toolkits like MediaPipe LLM Inference for running local models (like Gemma or Llama Nano) completely offline.

Top Frameworks for On-Device Mobile Agents

Key Ingredients for Mobile Agent Architecture

• The Engine: Use MediaPipe or ExecuTorch to load small, quantized open-source models (such as 1B-3B parameter variants of Gemma or Llama) that fit within mobile RAM limits.
• Function Calling: Implement local tool-use logic so your agent can interact with device APIs, local SQLite databases, or hardware sensors without an active internet connection.
• Local RAG: Use lightweight embedding models locally to index user data on the device securely.

If you tell me which mobile platform (iOS, Android, or cross-platform like Flutter/React Native) you are targeting and what tasks the agent needs to perform , I can recommend a precise technical stack and model size.

If your goal is a mobile app with a genuinely lightweight, autonomous on-device agent, my current first choice would be ExecuTorch.

It’s not an “agent framework” in the same sense as LangChain or OpenAI Agents SDK. Instead, it gives you the small, optimized model runtime underneath your agent, while you implement the agent loop yourself. That distinction is important for mobile.

My recommended stack

Mobile app

• Swift/Kotlin or React Native/Flutter
• Native bridge to the inference runtime

On-device model

• ExecuTorch
• A small instruct model, typically 1–4B parameters depending on target devices
• 4-bit/8-bit quantization

Agent layer

User
  ↓
Agent controller
  ├── short-term state
  ├── tool selection
  ├── planning / reasoning
  └── safety / permissions
        ↓
   Local LLM
        ↓
   Tools
   ├── filesystem
   ├── contacts
   ├── calendar
   ├── camera
   ├── location
   └── your app's APIs

The agent controller itself can be ordinary Swift/Kotlin code. You don't necessarily want another heavyweight framework sitting between your app and the model.

Why ExecuTorch?

It is specifically designed for constrained edge devices and supports Android and iOS, with CPU, GPU, NPU and DSP acceleration.

For Android, it provides Java/Kotlin integration through an AAR, and supports XNNPACK, Vulkan, Qualcomm and MediaTek backends. PyTorch Documentation On iOS, it provides Swift/Objective-C integration and Core ML/MPS/XNNPACK backends.

It also has explicit LLM deployment tooling, including quantization and model export, rather than being merely a generic tensor runtime.

And importantly for your use case, the runtime is deliberately designed to push work into ahead-of-time compilation so the device-side runtime stays lightweight.

What I would not do

I wouldn't start with:

LangChain/LangGraph + a local LLM

They're excellent for server-side agents, but on mobile I'd rather have:

             YOUR AGENT
                 │
       ┌─────────┴─────────┐
       │                   │
  Agent state          Tool router
       │                   │
       └─────────┬─────────┘
                 │
          ExecuTorch LLM
                 │
        ┌────────┴────────┐
        │                 │
      NPU/GPU           CPU

This gives you much more control over RAM, battery, latency, model size and privacy.

One important caveat

If you're building an agent rather than simply a chatbot, the model runtime is only half the problem.

The interesting engineering is the agent loop:

while not finished:
    observation = get_state()

    action = model.generate(
        system_prompt,
        conversation,
        observation,
        available_tools
    )

    if action.tool_call:
        result = execute_tool(action)
        conversation.append(result)
    else:
        return action.response

I'd make tool calls strongly typed, permissioned, and deterministic wherever possible. Let the LLM decide what to do, but don't let it directly manipulate arbitrary native APIs.

My shortlist

If you're targeting both iOS and Android, I'd prototype with ExecuTorch. If you're targeting iOS only, I'd seriously consider Core ML directly. If you tell me (1) iOS, Android, or both, (2) your target phone age, and (3) what the agent actually needs to do, I can recommend the specific model + runtime + agent architecture I'd use.