For on-prem deployment with no data leaving our network, what model should we run?

If you mean OpenAI models specifically, for a true on-prem deployment where inference data never leaves your network, the answer is gpt-oss rather than the hosted GPT models.

• gpt-oss-120b — my default recommendation for production. It is OpenAI's larger open-weight reasoning model, designed to run on infrastructure you control. OpenAI says it can run within 80 GB of GPU memory, making a single H100-class GPU a viable starting point.
• gpt-oss-20b — use this when latency, hardware cost, or edge deployment matters more than maximum capability. It can run in roughly 16 GB of memory.

OpenAI explicitly states that self-hosted gpt-oss models run on infrastructure you control and that OpenAI does not receive or process the data sent to them. They're also Apache 2.0 licensed, including commercial use subject to the usage policy.

What I'd choose

One important distinction: "on-prem" isn't enough by itself to guarantee no data leaves the network. You'd want the inference server, model, RAG/vector database, telemetry, and any tools/connectors behind your firewall, with outbound network access disabled or tightly controlled.

If you tell me your GPU/server specs, number of users, and what you're using it for (RAG, coding, customer support, agents, etc.), I can recommend the specific model size, quantization, inference stack, and hardware.

For a strict on-premises deployment where absolute data privacy is required (no telemetry, no external API calls, and zero data leaving your network), the best open-weight model to run depends heavily on your available hardware (VRAM/RAM) and your primary use case.

The top-performing open-weight model families and sizes categorized by hardware constraints are outlined below:

1. Best All-Rounder & General Assistant

• Qwen3 Family (Alibaba)
  • Why pick it: It features a very permissive Apache 2.0 commercial license, handles over 100 languages, and matches or beats legacy proprietary baselines in reasoning and coding. Mixture-of-Experts (MoE) variants activate very few parameters per token, making them blisteringly fast on local hardware.
  • Size to run:
    • 8B / 14B variants: Perfect for a single consumer GPU or high-end workstation.
    • Qwen3-30B-A3B (MoE): Excellent balance, fitting comfortably on a single enterprise/server GPU (like an 80GB A100 or dual-GPU rig) while offering near-frontier performance with only ~3B active parameters per token.

2. Best for Coding, Agents, & Technical Workflows

• GLM-5.2 or Qwen3-Coder-Next
  • Why pick it: GLM-5.2 leads on long-horizon context tasks, repository-level code generation, and complex terminal/agentic workflows. Qwen3-Coder-Next is exceptionally optimized for localized software development environments with massive context lengths (up to 256K+).

3. Best for Advanced Reasoning & Logic

• DeepSeek-R1 Distilled Variants
  • Why pick it: If your on-prem use case demands deep chain-of-thought, math, or complex analysis, the distilled versions of DeepSeek-R1 bring advanced reasoning behaviors directly to local hardware.
  • Size to run:
    • 7B / 14B Distills: Run easily on standard workstation hardware.
    • 32B / 70B Distills: Ideal if you have dedicated local multi-GPU servers (24GB to 48GB+ VRAM floors).

Recommended Local Serving Stacks

To keep everything entirely offline and secure, pair your chosen weights with a local inference engine:

• Ollama : Best for developer workflows, easy CLI orchestration, and serving local API endpoints.
• vLLM : Best for high-throughput production serving if you are scaling across multiple local server nodes.
• Open WebUI : Best for providing a self-hosted, ChatGPT-like interface for your internal team entirely behind your firewall.

To help narrow down the exact model and quantization level for your setup, let me know:

• What GPU/hardware specs (VRAM, RAM, and number of cards) do you have available on-prem?
• Is your primary workload general chat, document RAG/search, or coding/agents?

If your requirement is strictly on-prem with no customer data leaving your network, I’d choose OpenAI’s gpt-oss-120b as the default production model.

• gpt-oss-120b — best choice for high-quality enterprise reasoning, coding, agents, and RAG. It has 117B total parameters but only ~5.1B active per token, and is designed to fit on a single 80 GB GPU such as an H100.
• gpt-oss-20b — choose this if latency, hardware cost, or footprint matters more than maximum capability. It has 21B total / ~3.6B active parameters and is intended for more constrained deployments.

The important distinction is that these are open-weight models you run yourself, rather than OpenAI API models. OpenAI explicitly says the models can run on infrastructure you control, including on-prem, and that OpenAI does not receive or process data sent to a self-hosted model.

For an enterprise deployment, I'd roughly architect it as:

Your network → API gateway → gpt-oss-120b → your internal RAG/vector DB → your internal applications

with the inference servers isolated from the public internet. You can serve the model with stacks such as vLLM, Ollama, or llama.cpp.

urlOpenAI's gpt-oss documentationturn0search0 · urlOpenAI's open models pageturn0search2

If you tell me your GPU/server hardware, concurrent users, latency target, and whether this is primarily RAG, coding, chat, or agents, I can recommend the exact model size, quantization, and server configuration.

For a strict on-premises deployment where zero data leaves your network, you need to rely on open-weight foundation models served via local enterprise runtimes.

The right choice depends entirely on your available GPU/VRAM hardware and your primary use case (general chat vs. heavy coding/reasoning).

Phase 1: Choosing the Model by Hardware & Use Case

• For Standard Servers / Single-GPU (24GB VRAM like an RTX 4090 / A10G):
  • Best General & Multilingual Chat: Qwen3 (e.g., Qwen3.6-27B) or Google Gemma 4 (31B) . They deliver high accuracy on standard benchmarks while fitting comfortably on a single enterprise/consumer GPU.
  • Best for Coding & Technical Tasks: Qwen3-Coder-Next or Devstral Small . Excellent for local IDE integrations or internal code assistance without leaking proprietary repositories.
• For Resource-Constrained Hardware / Edge Devices (8GB–16GB VRAM / Laptops):
  • Best Lightweight: Google Gemma 4 (12B) or Qwen3 (8B) via Ollama . They run fast locally and handle basic summarization, extraction, and drafting efficiently.
• For High-End Clusters / Heavy Enterprise Reasoning (Multi-GPU / Data Center):
  • Best Performance-to-Scale: DeepSeek-V4-Flash or GLM-5.2 . These Mixture-of-Experts (MoE) models offer near-frontier reasoning and agentic workflow capabilities while remaining self-hostable on localized server racks.

Phase 2: Choosing the Local Serving Stack

Do not just download a model file—you need a secure, local orchestration layer to serve the model as an internal API or private UI:

• For Developers / Local APIs: Use Ollama or llama.cpp to spin up a local OpenAI-compatible endpoint with a single command.
• For a Team Chat UI: Deploy Open WebUI inside your private network (via Docker) to give your team a ChatGPT-like interface that points directly to your local Ollama/vLLM backend.
• For High-Throughput Production: Use vLLM for optimized memory management and concurrent request handling on your local server cluster.

To help narrow down the exact configuration, tell me:

• What GPU hardware or VRAM budget do you have available on-prem?
• What is the primary task (e.g., RAG document search, coding agent, or general employee chat)?

For an on-premise deployment where strict data sovereignty is required (zero data egress), your choice of open-weight model depends heavily on your available hardware cluster (VRAM) and your primary workload.

The best-performing and most production-ready open-weight model families are categorized below by their hardware footprint and optimal use case:

• Single-GPU / Small Footprint (Fits on 1 × 24GB VRAM GPU like an RTX 4090 Go to product viewer dialog for this item.)
  • Qwen3 (e.g., 8B or 27B variants) : Alibaba's Qwen3 family is an exceptional choice for general chat, multi-language support, and structured tasks. The dense versions provide remarkable performance-to-size ratios under a permissive Apache 2.0 license.
  • Gemma 4 (e.g., 31B) : Google's latest edge-optimized lineup delivers fantastic general-purpose instruction following and high generation speeds while remaining lightweight enough for modest local hardware.
• Mid-Range / Enterprise Server (Fits on 1 to 2 high-end enterprise accelerators like an H100 80GB)
  • GLM-5.2 : An elite open-weight model that trades blows at the top of intelligence and coding indexes. It features a massive 1M-token context window and native agentic tool-calling, making it ideal if you need an on-prem assistant capable of digesting large internal documentation sets or code repositories. Under its MIT license, there are zero commercial usage caps or reporting friction.
  • Mistral Small / Medium variants : Excellent performance with strong European GDPR compliance alignment and robust enterprise structuring capabilities.
• High-End / Frontier Reasoning (Requires a multi-GPU node cluster)
  • DeepSeek-V4-Pro : A mixture-of-experts (MoE) architecture boasting top-tier performance on reasoning, logic, and coding benchmarks. It pairs a 1M-token context window with a highly flexible MIT license, making it the gold standard for heavy local reasoning if you have the data center capacity to host a 1.6T parameter weight footprint (with ~49B active parameters).

To orchestrate these locally behind an OpenAI-compatible API or user interface inside your perimeter, standard toolsets like Ollama, vLLM , or Open WebUI can be deployed entirely offline.

To help narrow this down to a specific recommendation, tell me:

• What hardware/GPUs do you have available on-premise?
• Is the primary use case general chat/RAG, coding assistance , or complex autonomous agents?

If your requirement is no data ever leaves your network, then you should run a self-hosted open-weight model on your own infrastructure. OpenAI's hosted models (including the API and ChatGPT Enterprise) are managed cloud services—they provide strong privacy protections, optional data residency, and by default do not use business data for training, but inference still runs on OpenAI-managed infrastructure rather than entirely inside your network.

For most organizations, the current choices are:

A typical enterprise deployment looks like:

• Model server (vLLM, TensorRT-LLM, Ollama, etc.)
• Internal RAG over your documents
• Authentication (LDAP/Active Directory/SSO)
• GPU servers in your data center
• No outbound network access if you require an air-gapped environment

If you're asking specifically "Which OpenAI model can I run on-prem?", the answer is:

OpenAI does not currently offer GPT-5.5 or other frontier ChatGPT models for fully self-hosted, on-premises deployment. If zero data may leave your network, you'll need to use a self-hosted open-weight model instead. OpenAI's business offerings focus on secure managed deployments with enterprise privacy controls rather than customer-operated on-prem model deployments.

If you can share a bit more about your environment, I can recommend a specific model and hardware configuration. For example:

• How many users? (20, 200, 2,000+)
• Primary workload? (chat, coding, document search, agents, etc.)
• Available GPUs? (H100, H200, B200, A100, L40S, RTX 6000, etc.)
• Air-gapped or just private network?

With those details, I can suggest an architecture that balances model quality, latency, and infrastructure cost.

For an on-premises deployment with a strict zero-data-leakage requirement, you need high-performing open-weight or open-source models that run entirely offline on your own hardware.

The best model depends heavily on your hardware capacity (specifically VRAM) and your primary use case (general assistant, coding, or complex reasoning).

Top recommendations categorized by workload and hardware tier:

• Best All-Rounder for Production & Agentic Workflows: GLM-5.2 (by Zhipu AI)
  • Why: It offers exceptional reasoning and agentic execution capabilities while fitting comfortably onto standard enterprise server setups (unlike massive multi-terabyte models that require huge clusters). It scores near the top of open-weight leaderboards for complex problem-solving and ships under a permissive license.
• Best for Coding, Multilingual, and General Tasks: Qwen3 (by Alibaba)
  • Why: The Qwen3 family (available in various sizes like 8B, 27B/35B, up to larger dense/MoE variations) has become a gold standard for local developers. It handles over 100 languages, excels at code generation, and uses an Apache 2.0 license which makes corporate legal compliance straightforward.
• Best for Heavy Reasoning & Logic: DeepSeek-V4-Pro Go to product viewer dialog for this item. / DeepSeek-R1 Go to product viewer dialog for this item.
  • Why: If your on-prem server has the muscle (such as multi-GPU setups), DeepSeek’s open-weight reasoning models deliver near-frontier performance on math, logic, and deep analysis without needing an external internet connection.
• Best for Consumer Hardware / Laptops / Edge: Google Gemma 4 Go to product viewer dialog for this item. (12B or 26B MoE variants)
  • Why: If you are deploying locally on mid-tier workstations or smaller edge hardware, Gemma 4 Go to product viewer dialog for this item. strikes an incredible balance, offering high tokens-per-second and multimodal/audio-visual capabilities while requiring far less VRAM (the 12B variant runs smoothly on 16GB VRAM).

To serve these models securely on-premise without phoning home, standard local inference runtimes like Ollama, vLLM , or LM Studio can host the weights behind a private network endpoint.

To help narrow down the exact model and sizing configuration, could you share:

• What hardware/GPU specifications you have available on-premise (e.g., VRAM, system RAM, number of GPUs)?
• What primary task the model will perform (e.g., RAG/internal knowledge search, coding assistance, general chat)?

If your requirement is no data leaves your network (including prompts, retrieved documents, and outputs), then you should run an open-weight model locally rather than a cloud API.

If you're specifically looking for an OpenAI model that can be self-hosted, OpenAI now offers gpt-oss open-weight models designed for deployment on infrastructure you control: gpt-oss-120b and gpt-oss-20b. These are intended to run on-premises or in your own private cloud, and OpenAI states that it does not receive or process your data unless you explicitly share it or use a managed hosting provider.

Which one to choose depends on your hardware and workload:

A few considerations:

• Data residency: The model weights run entirely on your servers, so prompts and outputs can remain inside your network.
• Customization: You can fine-tune or adapt the models using standard open-source tooling.
• Hardware: The larger model is intended for high-capability deployments and requires substantial GPU resources, while the 20B model is designed for more constrained environments.

If your organization is evaluating an on-prem AI platform, a common enterprise architecture is:

• Self-hosted gpt-oss
• Local embedding model
• Self-hosted vector database
• Local document ingestion and retrieval
• Inference served with engines such as vLLM or NVIDIA NIM
• Everything deployed on your Kubernetes or bare-metal infrastructure

This keeps the full inference and retrieval pipeline inside your network perimeter.

If you can share:

• your GPU hardware (for example, H100, A100, L40S, MI300X),
• whether the environment is fully air-gapped,
• and your primary workload (chat, coding, document Q&A/RAG, or agents),

I can recommend the most appropriate model size and deployment architecture.

For on-premise deployment where data sovereignty is absolute (no data leaves the network), the best approach is to run open-weights, self-hosted models, or enterprise-grade local LLMs using secure infrastructure.

Based on current 2026 standards, the following models are recommended for on-premise use cases, ranked by capability and flexibility:

• Llama 3 (Meta): Generally considered the top-tier, most capable open-weights model available for local deployment in 2026. It performs well across various tasks and can be finetuned.
• Mistral/Mixtral (Mistral AI): Highly efficient, offering excellent performance (especially Mixtral) in a smaller, more cost-effective package compared to larger models.
• Phi-3 (Microsoft): Ideal for smaller, specialized, or edge-based on-premise tasks, offering high performance in a compact footprint (Small Language Models - SLMs).

Deployment & Infrastructure Requirements:

• Inference Engines: To run these, use efficient local serving frameworks such as vLLM, Text Generation Inference (TGI), or DeepSpeed-Inference to maximize GPU utilization.
• Hardware: Requires high-performance NVIDIA GPUs (A100, H100, or L40) with ample memory bandwidth, often set up in air-gapped or internal network configurations.
• Storage: Fast storage (NVMe/SSD) is essential for rapid model loading and handling large-scale data locally.
• RAG Solutions: For document analysis, integrate these models with vector databases like FAISS or Qdrant.

Key Considerations for Your Use Case:

• Full Control: This setup provides complete ownership of data security policies, enabling local backups and strict regulatory compliance.
• Air-Gapped Potential: These solutions can run completely disconnected from the internet.

To help narrow down the best model, could you tell me:

• What is the primary use case? (e.g., coding assistance, document summarization, RAG chatbot)
• How much GPU memory (VRAM) do you have available on your on-premise servers?
• What is the required level of accuracy vs. the need for low latency?

I can then make a more specific recommendation, such as whether a smaller 8B model or a larger 70B model is right for you.

If your requirement is fully on-premises with no inference data leaving your network, the best OpenAI option is to run an open-weight GPT OSS model on infrastructure you control. These models are specifically intended for self-hosted deployments. OpenAI states that when you run them on your own infrastructure, OpenAI does not receive or process your inference data unless you explicitly choose to share it or use a managed hosting provider.

The current choices are:

• gpt-oss-20b – Best for teams with more modest GPU resources or lower latency requirements.
• gpt-oss-120b – Best if you want substantially higher reasoning quality and have the GPU capacity to support it.

A practical recommendation:

If you're deciding between self-hosting and OpenAI's hosted offerings:

• Need absolute data isolation / air-gapped deployment: Use gpt-oss on your own infrastructure.
• Need the highest capability models (GPT-5.x, etc.): Those are accessed through OpenAI's cloud services rather than self-hosted deployment. Enterprise offerings provide strong privacy controls and, by default, customer data isn't used to train OpenAI's models, but inference still occurs on OpenAI-managed infrastructure rather than entirely within your network.

If you tell me:

• approximately how many concurrent users you expect,
• your GPU hardware (or planned hardware), and
• whether you're fully air-gapped or simply require data to remain inside your network,

I can recommend the appropriate model size and serving stack (for example, vLLM or another inference server) for your environment.