What's the best framework for running LLMs on-device for a mobile app?

When running Large Language Models (LLMs) on-device for mobile applications, the "best" framework depends heavily on your target operating system, hardware optimization needs, and whether you prefer cross-platform support or native performance.

The industry has largely converged on a few top-tier frameworks, each with distinct advantages:

1. llama.cpp / GGML
   • Best For: Cross-platform flexibility (iOS & Android), maximum model compatibility, and lightweight C/C++ performance.
   • Why it's great: It allows you to run heavily quantized GGUF models efficiently on both CPU and available mobile GPU/accelerators. It's the engine behind many mobile wrappers and has a massive community supporting a wide array of open-weights models (like Llama, Phi, and Qwen).
   • Trade-off: Requires writing or using a native bridge/wrapper for mobile integration, though community bindings make this easier.
2. Google MediaPipe (LLM Inference API)
   • Best For: Android-first development and cross-platform ease with minimal boilerplate.
   • Why it's great: Google's MediaPipe LLM Inference Guide provides ready-to-use cross-platform APIs for Android and iOS. It's optimized to run smaller mobile-ready models (like Gemma or Phi) directly on mobile hardware accelerators (GPU, NPU/DSP).
   • Trade-off: Less flexible than llama.cpp if you want to experiment with exotic architectures or custom fine-tuned layer modifications.
3. MLC LLM (Machine Learning Compilation)
   • Best For: High-performance cross-platform compilation via TVM.
   • Why it's great: MLC LLM compiles LLMs natively to universal deployment runtimes, deeply optimizing performance for iPhone, Android, and web browsers using native GPU shaders (Metal for iOS, Vulkan/OpenCL for Android).
   • Trade-off: The build and compilation pipeline has a steeper learning curve compared to drop-in solutions.
4. Apple CoreML / MLX (iOS Exclusive)
   • Best For: Deep, native optimization on Apple Silicon (iPhone and iPad).
   • Why it's great: If you are building an iOS-only app, leveraging CoreML lets you tap directly into the Apple Neural Engine (ANE) for exceptional power efficiency and speed. (For research or local Mac/iOS hybrid tooling, Apple’s MLX framework is also rising rapidly).
   • Trade-off: Zero help for Android; you'll have to maintain a separate pipeline for non-iOS devices.

Quick Recommendation

• Choose llama.cpp if you want the absolute broadest model support, fine-grained control over quantization, and cross-platform parity.
• Choose Google MediaPipe if you want an out-of-the-box, production-ready solution with minimal setup, especially within the Android ecosystem.
• Choose CoreML if your app is strictly iOS-bound and needs tight integration with Apple's Neural Engine.

To help narrow down the ideal choice, let me know:

• Is your app Cross-platform (Flutter/React Native) or Native (Swift/Kotlin)?
• Are you targeting iOS, Android, or both?
• What size model are you hoping to run (e.g., 1B–3B ultra-small vs. 7B+)?

There isn’t one universal “best” framework — it depends on whether you optimize for shipping speed, maximum model flexibility, or peak device performance. For a mobile app in 2026, the main contenders are llama.cpp, ExecuTorch, MLC LLM, and platform/vendor runtimes.

A practical decision guide:

RunLocalAI RunLocalAI## My default recommendation

For most teams building a new mobile AI feature:

1. Start with llama.cpp

Use it if you need:

• Llama, Qwen, Mistral, Gemma, Phi, etc.
• GGUF quantized models
• Fast prototyping
• Offline inference
• One code path across iOS/Android

The tradeoff is that mobile integration is more engineering-heavy: you’ll likely build native wrappers around the C/C++ API.

2. Move to ExecuTorch if you are productizing seriously

Choose it when you have:

• A PyTorch model pipeline
• Multiple model types (LLM + vision + audio)
• A need for hardware-specific acceleration
• A longer-lived production architecture

ExecuTorch is designed around exporting models from PyTorch and using backend delegates for mobile hardware.

3. Pick MLC LLM when performance is the main goal

MLC’s approach is closer to a compiler toolchain: you compile models for target hardware rather than shipping a generic interpreter. This can be excellent, but the workflow is more complex because artifacts are hardware/model-specific.

Architecture I’d usually ship

For a consumer mobile app:

Mobile UI
   |
Native bridge (Swift/Kotlin)
   |
LLM runtime
   |
Quantized model (2B–8B usually)
   |
Metal / Vulkan / NNAPI / vendor backend

Model sizes that are realistic:

• 1B–4B: best mobile experience
• 7B: possible on newer flagship phones with enough RAM
• 13B+: usually impractical for sustained mobile use because of memory, battery, and thermal limits

A few implementation tips

• Prefer 4-bit quantization (often Q4 variants) for mobile.
• Stream tokens; don’t wait for the full response.
• Keep context windows modest — KV cache memory becomes a major bottleneck.
• Download models after install rather than bundling multi-GB weights.
• Benchmark on real phones; desktop benchmarks are misleading for thermal throttling.

If I were choosing today:

• Prototype: llama.cpp + GGUF
• Ship cross-platform: llama.cpp or ExecuTorch
• High-performance Android/iOS deployment: ExecuTorch or MLC LLM
• Simple offline assistant: a small quantized 2B–4B model with llama.cpp is hard to beat

The “best” choice also depends heavily on whether you’re building with native Swift/Kotlin, React Native, Flutter, or Unity, and whether you need chat only vs. tool calling/agents/multimodal.

A few implementation tips

Choosing the "best" framework for running on-device LLMs in a mobile app depends heavily on your target platform (iOS vs. Android), your engineering capacity, and whether you need access to specialized mobile Neural Processing Units (NPUs).

The top-tier frameworks break down as follows based on your specific development priorities:

1. llama.cpp — The Ultimate Open-Source Standard
   • Best for: Maximum model flexibility, raw performance tuning, and using the ultra-popular GGUF quantization format.
   • Pros: Supports virtually every new open-source model architecture (Llama, Mistral, Gemma, Qwen) within days of release. Extremely lightweight C/C++ core.
   • Cons: Low-level; it requires you to write your own Swift/Kotlin native bindings, manage memory allocation under strict mobile OS limits, and build your own streaming/lifecycle logic. It targets mobile GPU/CPU via Metal and Vulkan, but typically lacks dedicated mobile NPU (Neural Engine) acceleration.
2. MLC LLM — The Cross-Platform Compiler Approach
   • Best for: High-performance cross-platform apps (iOS and Android from a single codebase like Flutter or React Native).
   • Pros: Uses machine learning compilation (Apache TVM) to optimize models natively for target device hardware. Offers clean native Swift and Kotlin runtimes alongside web/desktop support.
   • Cons: Requires a specific compilation toolchain to convert models into its native format, and debugging compilation errors can occasionally be complex.
3. MediaPipe (Google LLM Inference API) — The Best for Android-First & Ease of Use
   • Best for: Android developers looking for out-of-the-box integration without dealing with custom C++ wrappers.
   • Pros: Provides high-level Kotlin/Java APIs, seamless integration into standard Android project pipelines, and is pre-optimized for on-device execution of lightweight models like Gemma or Phi on Android hardware.
   • Cons: More limited model support compared to llama.cpp , and primarily shines within the Android ecosystem rather than cross-platform.
4. CoreML — The Best Native iOS Choice
   • Best for: Pure iOS or macOS apps heavily leveraging Apple Silicon.
   • Pros: Directly accesses the Apple Neural Engine (ANE) and GPU, ensuring maximum power efficiency, minimal thermal throttling, and deep system-level integration.
   • Cons: Apple-only; you will need a completely separate pipeline or framework for your Android version. Model conversion from PyTorch/HuggingFace to .mlpackage can be tedious.
5. ONNX Runtime — The Production NPU Choice
   • Best for: True hardware-accelerated NPU execution across both iOS and Android (via Qualcomm QNN for Hexagon NPUs and Apple CoreML).
   • Pros: Exceptional focus on battery life, low latency, and production stability via hardware vendors' dedicated execution providers.
   • Cons: Steeper learning curve for optimizing raw text-generation pipelines compared to dedicated LLM runners.

Quick Recommendation Guide

• If you want a quick proof-of-concept or maximum control over GGUF models, go with llama.cpp.
• If you are building a native Android app , use MediaPipe.
• If you are building a native iOS app , use CoreML to tap into the Neural Engine.
• If strict battery preservation and NPU acceleration across both platforms matter most, look at ONNX Runtime.

If you'd like, tell me:

• Are you targeting iOS, Android, or both (cross-platform)?
• What is your target model size (e.g., 1B, 3B, 7B parameters)?
• Do you need NPU/battery optimization or raw generation speed?

I can help narrow down the ideal architecture for your stack.

There isn’t one universal winner — the best framework depends on whether you optimize for shipping speed, model flexibility, maximum performance, or hardware acceleration. For a mobile app in 2026, these are the main choices:

RunLocalAI RunLocalAI## My default recommendation

For most teams building a mobile AI feature:

1. Start with llama.cpp

Use it if you need:

• iOS + Android
• open models (Llama, Qwen, Mistral, Gemma, Phi, etc.)
• GGUF quantized weights
• fast prototyping

A typical stack:

• Swift/Kotlin/React Native UI
• native bridge (C++/JNI on Android, Objective-C++/Swift bridge on iOS)
• llama.cpp backend
• 1B–8B quantized model depending on device RAM

It has the least risk because the ecosystem around GGUF models is enormous.

2. Choose ExecuTorch if you are building a serious product

I would lean toward ExecuTorch when:

• your ML team already uses PyTorch
• you need tighter hardware integration
• you expect custom models or fine-tuned models
• you care about long-term production maintenance

It is closer to a "mobile ML platform" than just an LLM runtime.

3. Choose MLC LLM if raw mobile GPU performance matters

Good fit for:

• consumer apps with a few supported devices
• you can afford a model compilation pipeline
• you want maximum throughput from Metal/Vulkan-class GPUs

RunLocalAI## Model sizes that realistically work on phones

Typical ranges:

• 1B–3B parameters → excellent mobile experience
• 4B–8B → possible on newer flagship phones with aggressive quantization
• 10B+ → usually compromises battery, latency, or memory

The runtime matters, but model size, quantization, KV cache size, and thermal behavior often matter more.

If I were starting a new app today

My shortlist would be:

• Cross-platform consumer app: llama.cpp first, evaluate MLC/ExecuTorch later
• Android-heavy app targeting modern devices: ExecuTorch or Google’s mobile AI stack
• iOS-only premium app: Core ML / Apple-native route or llama.cpp depending on model needs
• Custom fine-tuned model pipeline: ExecuTorch
• Fast prototype: llama.cpp

A useful rule: pick the model first, then the runtime. The runtime choice is constrained by which models, quantization formats, and hardware paths you actually need.

If you tell me your stack (native iOS, Android, Flutter, React Native), target devices, and model size, I can narrow this down to a specific architecture.

The best overall framework depends on your target ecosystem and whether you want native system integration or custom open-weight model control, with llama.cpp (cross-platform), MediaPipe (cross-platform Google-backed), and Core ML / Apple Foundation Models (iOS native) leading the space.

There is no single "one-size-fits-all" winner, but choices map directly to specific production needs.

Top On-Device LLM Frameworks Compared

Breakdown of Recommendations

• Choose llama.cpp if: You want to deploy specific, highly quantized open-source models (like LLaMA-3, Mistral, or Qwen) using .gguf files. It gives you granular control over memory limits, context windows, and CPU/GPU threading via native NDK or Swift wrappers.
• Choose MediaPipe if: You want a clean, production-ready API maintained by Google that handles device acceleration smoothly without writing raw C++ bindings for Android/iOS. It works exceptionally well with models like Google's Gemma.
• Choose Apple Core ML / Foundation Models if: Your app is iOS-exclusive and you want to tap into system-level intelligence or standard optimized transformer blocks with minimal setup and maximum battery efficiency.

If you can share:

• Your target platforms (iOS only, Android only, or Cross-Platform/Flutter/React Native)
• The approximate size of the model you want to run (e.g., 1B, 3B, 8B parameters)

I can recommend the most precise wrapper or implementation pattern for your app.

If you mean LLM inference directly on iOS/Android, my default choice in 2026 is ExecuTorch—unless you're building an Apple-only app, in which case Apple's native stack is increasingly compelling.

My recommendation

Why ExecuTorch? It gives you a fairly unified deployment path across Android and iOS, with LLM-specific export/runtime tooling and hardware backends. On Android it can target XNNPACK, Vulkan, Qualcomm AI Engine, and MediaTek hardware; on iOS it supports Core ML, MPS, and XNNPACK.

It also has explicit LLM deployment support—including exporting models to .pte and running them through C++, Swift, or Java bindings.

If you're iOS-only

I'd seriously consider Apple's Foundation Models framework first.

It gives you a native Swift API to Apple's on-device foundation model, including structured output and tool calling. The model runs locally, and Apple handles a lot of the hardware/runtime complexity for you.

If you need to bring your own open-source model, Apple's newer Core AI is more interesting. It is specifically designed for deploying custom models on Apple Silicon, with hardware specialization, compilation, memory controls, and zero-copy execution.

An especially nice development direction is that Core AI models can be plugged into the Foundation Models LanguageModelSession, giving you a common API even when you're using your own model.

What I'd choose for a new app

If your architecture is:

React Native / Flutter UI → native inference module → local LLM

I'd use:

ExecuTorch + a ~1–4B quantized model + hardware-specific backend

For example:

• Android: ExecuTorch → Qualcomm QNN / Vulkan / XNNPACK
• iOS: ExecuTorch → Core ML / MPS
• Model: a compact instruct model rather than a 7B+ model unless you have a strong reason
• Quantization: 4-bit or another device-tested low-bit format
• Fallback: cloud inference for devices that don't meet your latency/RAM requirements

That's a much more scalable architecture than tying your app directly to one particular model runtime.

If you tell me iOS, Android, or both and roughly which model (Llama, Qwen, Gemma, etc.), I can give you a concrete stack—including model size/quantization, expected RAM requirements, and an example integration architecture.

If you're iOS-only

I'd seriously consider Apple's Foundation Models framework first.

It gives you a native Swift API to Apple's on-device foundation model, including structured output and tool calling. The model runs locally, and Apple handles a lot of the hardware/runtime complexity for you.

If you need to bring your own open-source model, Apple's newer Core AI is more interesting. It is specifically designed for deploying custom models on Apple Silicon, with hardware specialization, compilation, memory controls, and zero-copy execution.

An especially nice development direction is that Core AI models can be plugged into the Foundation Models LanguageModelSession, giving you a common API even when you're using your own model.

What I'd choose for a new app

If your architecture is:

React Native / Flutter UI → native inference module → local LLM

I'd use:

There is no single "best" framework because the ideal choice depends on your target platforms, your team's familiarity with C++ vs. Python/Kotlin/Swift, and whether you care more about out-of-the-box simplicity or peak hardware acceleration.

The top-tier frameworks for running LLMs on-device in a mobile app break down as follows, categorized by their strengths:

1. Google MediaPipe (Best for Ease of Use & Android/Cross-Platform Simplicity)

Google’s MediaPipe provides a dedicated LLM Inference API designed specifically for running lightweight models (like Gemma or Phi-3) locally on mobile.

• Pros: Highly polished, great documentation, and drops right into standard mobile workflows (Kotlin/Swift/JS). It handles hardware acceleration (GPU/NPU) under the hood without requiring custom compilation.
• Cons: Less flexible if you want to heavily tweak model architectures, quantization schemes, or run less mainstream open-source weights.
• Best for: Developers who want the fastest, lowest-friction path to shipping an on-device text feature without wrestling with low-level binaries.

2. MLC LLM / TVM (Best for Maximum GPU/NPU Acceleration & Web/Mobile Parity)

MLC LLM compiles large language models down to native machine-learning runtime libraries using Apache TVM. It exposes an OpenAI-compatible local API.

• Pros: Incredible hardware acceleration via Metal (iOS) and Vulkan/OpenCL (Android). It supports cross-platform app ecosystems (including React Native or WebAssembly/WebGPU) using the exact same underlying engine.
• Cons: Compilation pipeline and build setup have a steeper learning curve.
• Best for: Cross-platform frameworks (like React Native) or performance-critical apps that need bleeding-edge tokens-per-second on device GPUs.

3. llama.cpp (Best for Flexibility & GGUF Model Support)

llama.cpp is the gold standard of the open-source local LLM community, written in pure C/C++.

• Pros: Supports virtually every emerging GGUF model variant under the sun. It is lightweight, deeply optimized, and has community-driven Swift (LLM.swift ) and Android wrappers.
• Cons: You are essentially managing C/C++ bindings or wrappers inside your mobile project, which can complicate iOS/Android build configurations.
• Best for: Power users and developers who want total control over the inference engine and the freedom to swap out model weights on the fly.

4. Meta ExecuTorch (Best for PyTorch-centric & Production-heavy Native Stacks)

ExecuTorch is Meta’s production-focused, lightweight on-device inference framework (the spiritual successor to PyTorch Mobile).

• Pros: Designed to seamlessly take a model trained in PyTorch and export it reliably to mobile and edge chips with hardware-aware delegation.
• Cons: Very engineering-heavy; geared more toward large enterprise teams building custom hardware pipelines than rapid app prototyping.
• Best for: Teams already deeply embedded in the PyTorch ecosystem training custom proprietary small language models (SLMs).

Summary Recommendation

• Choose Google MediaPipe if you want to get an MVP running in a weekend with minimal headache.
• Choose MLC LLM if you are building cross-platform (e.g., React Native) or need absolute maximum performance squeezed out of mobile GPUs.
• Choose llama.cpp if you need raw flexibility over GGUF models and don't mind managing native wrappers.

If you'd like to narrow this down further, tell me:

• Are you building Native (Swift/Kotlin) or Cross-Platform (Flutter/React Native)?
• Do you have a specific model in mind (e.g., Gemma 2B, Llama 3.2 1B/3B)?
• Is your primary goal a chatbot interface or background text processing?

If you mean a production mobile app that runs an LLM locally on both iOS and Android, my default choice in 2026 is ExecuTorch.

My recommendation

Why ExecuTorch: it is designed specifically for deploying PyTorch models on-device, has quantization and hardware acceleration backends, and provides C++, Swift, and Java integration paths. Its current LLM deployment flow exports models to .pte and runs them through native APIs.

When I'd choose something else

iOS only → Apple's Core AI / Foundation Models

If you're building an iPhone/iPad app and don't need your own model on every device, Apple's new stack is probably the nicest solution. Foundation Models gives you a native Swift interface to Apple's on-device model, while Core AI is Apple's newer framework for bringing your own models to Apple Silicon. Core AI can automatically select CPU, GPU, or Neural Engine execution depending on the model/device.

Need arbitrary open-source models + portability → llama.cpp

I'd pick llama.cpp if your priority is:

• GGUF models
• Huge ecosystem of quantized models
• Easy swapping between models
• Maximum control over inference
• Supporting unusual architectures/platforms

The tradeoff is that you end up doing more of the mobile integration and hardware-specific optimization yourself.

Android/Gemini ecosystem → LiteRT-LM

If you're primarily targeting Android and using Google's Gemma family, I'd seriously evaluate Google's LiteRT stack. It can make sense when Google's model/runtime/hardware optimizations line up with your target devices.

What I'd actually build

For a serious cross-platform app, I'd structure it like:

                Your App
                   │
          ┌────────┴────────┐
          │ LLM abstraction │
          └────────┬────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
     iOS                  Android
        │                     │
   ExecuTorch             ExecuTorch
        │                     │
   GPU / ANE             GPU / NPU
        │                     │
      2–8B                 2–8B
    quantized             quantized
      model                 model

Then keep model selection, prompting, streaming, tool calling, structured output, and conversation state above the runtime. That way you can replace ExecuTorch with llama.cpp or a cloud model without rewriting your application.

One important caveat: don't choose the runtime before choosing your target devices and model. A 3B/4B quantized model on recent phones is a very different engineering problem from trying to run a 14B model across five-year-old Android devices.

If you tell me iOS only, Android only, or both + what model (Llama/Qwen/Gemma/etc.), I can give you a much more specific recommendation—including runtime, quantization, model size, expected RAM requirements, and a production architecture.

The best framework depends on your target platform and tech stack, but llama.cpp is the top cross-platform choice for raw performance, while Core ML is best for iOS-only apps.

Top On-Device LLM Frameworks

• llama.cpp: Best for cross-platform (iOS and Android) performance using C/C++. It uses GGUF models, runs efficiently on mobile CPUs and GPUs, and has community wrappers for Swift and Android.
• Core ML: Best for exclusive iOS and macOS apps. It integrates deeply with Apple Neural Engine (ANE) hardware for maximum energy efficiency and speed on iPhones and iPads.
• ExecuTorch: Best for PyTorch developers . Backed by Meta , it provides a streamlined path to export and quantize models directly from PyTorch to edge devices.
• LiteRT (formerly TensorFlow Lite): Best for lightweight, traditional mobile machine learning integration on Android and iOS, though less optimized for large language models compared to llama.cpp.
• MLC LLM: Best for universal GPU acceleration (Metal, Vulkan, OpenCL). It compiles models natively to leverage device hardware acceleration.

Comparison of Key Frameworks

To help narrow down the best choice, tell me:

• Are you building for iOS only or cross-platform (iOS and Android)?
• What is the target model size (e.g., 1B, 3B parameters)?

If you mean a production mobile app that runs an LLM entirely on-device, my default recommendation in 2026 is llama.cpp.

My ranking

Why I'd pick llama.cpp

github.com has become a very practical mobile inference layer:

• iOS + Android support.
• Excellent quantization support, including 4-bit and other low-bit formats, which is crucial for phone memory/battery constraints.
• Native acceleration through things like Metal on Apple devices and GPU/NPU-oriented backends on some Android hardware.
• GGUF gives you a huge ecosystem of readily available quantized models.
• There's an iOS XCFramework, making Swift integration considerably easier.
• You can keep the actual inference engine in C/C++ and expose a thin Swift/Kotlin layer to your app.

A typical architecture I'd use is:

┌──────────────────────────────┐
│        Mobile UI             │
│    Swift / Kotlin / Flutter  │
└──────────────┬───────────────┘
               │
        App inference API
               │
┌──────────────▼───────────────┐
│          llama.cpp            │
│   tokenizer / KV cache /     │
│   sampling / model runtime   │
└──────────────┬───────────────┘
               │
          GGUF model
               │
       ┌───────┴───────┐
       │ CPU / GPU /   │
       │ Metal / etc.  │
       └───────────────┘

When I'd choose ExecuTorch instead

If your models originate in PyTorch and you care about compiling/exporting them into an optimized deployment format, I'd seriously consider pytorch.org.

ExecuTorch has explicit LLM deployment APIs, quantization, acceleration backends, and Swift/Java bindings, with documented Android and iOS deployment paths.

The tradeoff is that llama.cpp is generally the easier choice if your goal is "download a good quantized open model and make it run on phones." ExecuTorch becomes more compelling when you're building a controlled model/deployment pipeline rather than consuming the GGUF ecosystem.

One important distinction

If you're building something like:

"I want an offline AI assistant in my iOS + Android app." I'd use llama.cpp + a ~1–4B quantized model initially.

If you're building:

"We train/modify PyTorch models and need optimized deployment across CPU/GPU/NPU targets." I'd start with ExecuTorch.

If you're building iOS-only, I'd also benchmark Apple's native stack rather than automatically choosing a cross-platform runtime.

If you tell me iOS vs Android vs both, your app framework (Swift/Kotlin/Flutter/React Native), and roughly what model size you want (1B/3B/7B), I can give you a concrete stack—including model format, quantization, memory requirements, and a recommended architecture.