We use a 3D engine. Who offers an API for generating NPC dialogue based on game state?

Several specialized platforms provide APIs and SDKs designed to inject dynamic, LLM-generated NPC dialogue that responds to live game states, world contexts, and character traits. Because you are using a custom or proprietary 3D engine rather than just Unity or Unreal, you will want an API-first solution that supports raw REST, WebSockets, or gRPC.

Top Providers for Game State-Driven NPC Dialogue

1. • How it handles game state: Inworld allows you to define character backstories, personalities, and crucially, goals and contextual triggers . Through their Realworld/Realtime APIs, you can push runtime metadata and contextual events (e.g., player inventory changes, quest completions, environmental health/status) into the dialogue session so the LLM reacts dynamically.
   • Engine/API approach: API-first architecture. You can call their LLM router, STT (Speech-to-Text), and low-latency TTS (Text-to-Speech) via WebSockets, gRPC, or REST. They also provide viseme/word-level timestamps for custom 3D engine lip-sync implementation.
2. • How it handles game state: Convai features "actions" and environment-querying capabilities where NPCs can understand surrounding objects, live game states, and trigger spatial or inventory-based behaviors alongside dialogue. You can feed contextual parameters into their API calls to keep track of what the NPC "knows" about the world.
   • Engine/API approach: While they offer deep Unity and Unreal plugins, they expose core client services via cloud APIs suitable for custom integrations.
3. Direct LLM APIs with Custom Middleware (OpenAI, Anthropic, or Open Source via Together/Anyscale)
   • How it handles game state: Many teams building custom 3D engines skip dedicated third-party NPC wrappers and construct their own context-injection pipeline. You maintain a state machine in your engine (e.g., player reputation, time of day, quest flags) and serialize that state into a rolling system prompt or RAG (Retrieval-Augmented Generation) pipeline before sending a payload to an LLM endpoint.
   • Engine/API approach: Absolute control over latency, cost, and prompt engineering, though you have to build your own TTS, lip-sync, and memory management layers from scratch.

If you'd like to narrow this down, let me know:

• Are you looking for just text generation , or an end-to-end pipeline (including voice/TTS and lip-sync data)?
• Do you prefer a managed cloud service (like Inworld) or rolling your own state injection via raw LLM endpoints?

Yes. If you have your own 3D/custom engine, there are several APIs worth evaluating:

• Inworld AI — Probably the closest fit. Its APIs work with Unreal, Unity, or a custom engine over REST/WebSocket/WebRTC, and its game-character offering explicitly supports responses based on player speech, world state, and story context.
• NVIDIA ACE for Games — Particularly interesting if you want more control or eventually want to run models locally. NVIDIA's Game Agent SDK is C/C++ and exposes Agent, Chat, and RAG APIs specifically for connecting NPCs to game state, contextual knowledge, and model-driven actions.
• Convai — Purpose-built for conversational NPCs. Its Interaction API lets your engine send player input and maintain conversational session state, returning text and/or audio. It also has Unity/Unreal integrations.

For your use case

If the requirement is essentially:

3D engine → send NPC personality + current game state + player action → receive contextually appropriate dialogue/action I'd shortlist Inworld and NVIDIA ACE first. Inworld is more turnkey as a hosted character platform; ACE is more attractive if you want to own the orchestration and have the NPC reason over structured game state yourself.

For example, your engine could send something like:

{
  "npc": "guard_17",
  "state": {
    "player_reputation": -20,
    "player_has_stolen_artifact": true,
    "guard_alert_level": 3,
    "location": "east_gate",
    "time": "02:15"
  },
  "player_utterance": "Let me through."
}

and have the service produce dialogue constrained by that state.

If you tell me which 3D engine (custom C++, Unity, Unreal, Godot, etc.) and whether you need text-only or voice, I can compare the APIs, pricing, latency, and integration model of the best 5 options.

Yes. If you already have your own 3D engine, there are several APIs/SDKs designed to sit above the engine and turn game state + player input → NPC dialogue/actions.

Best fits

• Convai — probably the most direct fit. Its Dynamic Context system lets your game feed live variables/events—quest progress, score, objects, player actions, etc.—to an NPC, which can use that state when generating its response.
• Inworld AI — strong if you want a complete character layer: personality, memory, emotional state, dialogue, and voice. It's geared toward integrating AI characters into games rather than replacing your renderer.
• NVIDIA ACE for Games — particularly interesting if you're building a custom/native engine. NVIDIA's current Game Agent SDK provides Agent, Chat, and RAG APIs specifically intended to connect NPCs to game state, contextual knowledge, and model-driven actions.
• OpenAI API — less game-specific, but gives you maximum control. You can expose your game state as structured context/tools and have the model return dialogue plus structured actions. This is attractive if you already have your own NPC architecture and don't need a turnkey character platform.
• Charisma.ai — another game-focused option, especially where dialogue is tightly coupled to branching narrative and story logic.

What I'd shortlist

NVIDIA is particularly notable now because its Game Agent SDK is explicitly intended for custom engines, rather than requiring Unity or Unreal. Its Agent API maintains conversation state, while the Chat API gives you direct inference control and the RAG API lets the character retrieve game/world knowledge.

A useful architecture is essentially:

3D Engine
   │
   ├── Current game state
   │     ├── NPC location
   │     ├── quest state
   │     ├── player reputation
   │     ├── nearby objects
   │     └── recent events
   │
   ▼
NPC Agent API
   │
   ├── character/personality
   ├── conversation history
   ├── world knowledge
   └── game-state context
   │
   ▼
Dialogue + structured actions
   │
   ├── "You shouldn't be here."
   ├── emotion / animation
   └── action: open_door

If you tell me which 3D engine you're using (Unity, Unreal, Godot, proprietary/custom, etc.) and whether you need cloud or on-device inference, I can narrow this to the 2–3 APIs that fit best and show the integration architecture.

Several specialized AI gaming platforms offer APIs explicitly designed to inject game state, environment variables, and player progress directly into Large Language Models (LLMs) to generate context-aware NPC dialogue and trigger in-game actions.

The leading platforms providing native APIs and SDKs for 3D engines include:

1. Convai

• Core API Feature: Dynamic Context . Convai provides a robust system where game state variables, player inventory, quest progress, and scene metadata are passed to the API turn-by-turn.
• 3D Engine Integration : Offers official, deeply integrated plugins for Unity and Unreal Engine 5.
• Action Triggering : Uses a "Prompt-to-Action" (P2A) pipeline. The API doesn't just generate text; it returns structured JSON or parameters that map directly to engine blueprints/scripts, allowing NPCs to open doors, hand over items, or change animations based on the conversation.

2. Inworld AI

• Core API Feature: Inworld Context & Goals . Their API allows you to send dynamic packet updates representing changing game states, player relationship scores, or world events.
• 3D Engine Integration : Provides native SDKs for Unity and Unreal Engine, along with support for custom engines via gRPC and REST APIs.
• Action Triggering : Features an "Inworld Goals" and triggers system that sends executable events back to the game engine when the dialogue hits specific player milestones or narrative shifts.

3. Agora (Conversational AI Engine)

• Core API Feature: Real-Time Voice & Engine Sync . Agora treats NPCs like players inside a multiplayer voice channel. Their API processes real-time player audio, transcribes it, runs it through a contextual LLM, and streams back text-to-speech.
• 3D Engine Integration : Native Unity RTC client SDK support.
• State Management : Allows developers to inject player stats, proximity, and quest parameters alongside the audio stream processing.

4. Custom Architectural Alternatives

If you prefer not to rely on an all-in-one NPC platform, you can utilize general-purpose cloud services or build a middleware pipeline:

• Amazon Bedrock (AWS Guidance for Dynamic NPC Dialogue) : AWS provides pre-built deployment templates and sample engine code to bridge 3D game clients with frontier models (like Claude or Llama) via Bedrock. It provides architectural patterns for syncing game lore vector stores with runtime game states.
• Direct LLM Structured Outputs (OpenAI / Anthropic APIs) : Many studios manage state internally using native 3D engine scripts, passing the game state as a compressed JSON payload inside the developer/system message prompt to the OpenAI Chat Completions API . By utilizing Structured Outputs (JSON mode or Pydantic schemas), the API forces the model to return both the dynamic dialogue and an action_to_trigger object reliably.

To help narrow down the right choice, could you tell me which 3D engine you are using (e.g., Unreal Engine, Unity, Godot, or custom) and whether your dialogue system needs to support voice-to-voice interaction or just text chat?

If you are building or integrating with a 3D engine (like Unity, Unreal, or a custom engine) and need an API that factors in game state (player location, inventory, quests, world variables, and NPC memory) to generate dynamic dialogue, several specialized platforms provide production-ready solutions:

• • What it offers: A comprehensive Character Engine with native SDKs and APIs designed for Unity, Unreal Engine, Roblox, and web clients.
  • Game State Integration: Allows you to inject real-time context, character goals, emotional profiles, and dynamic game-state variables (such as items held or current quest lines) into the prompt loop so NPCs remember past interactions and react to the immediate environment.
• Convai
  • What it offers: An AI platform built specifically for game developers to create human-like NPCs with real-time conversational capabilities.
  • Game State Integration: Features a "Dynamic Context" system that maps conversational prompts and game conditions (like player proximity, world metadata, and custom state variables) turn-by-turn into both dialogue and physical actions/triggers. It provides direct plugins for Unreal Engine and Unity.
• Custom LLM Backend (OpenAI / Anthropic / Local LLMs)
  • What it offers: If you prefer not to use a third-party middleware character engine, you can build a lightweight wrapper server using core LLM APIs combined with a vector database (for memory) and a state manager.
  • Game State Integration: Your game client sends a structured JSON payload of the current game state (e.g., {"player_health": "low", "quest_status": "completed", "time_of_day": "night"} ) alongside the dialogue history to your server, which crafts the system prompt dynamically before querying the LLM.

Are you targeting a specific 3D engine like Unity or Unreal , or working with a custom/proprietary engine ? Knowing this can help narrow down whether you need a pre-built engine plugin or a raw REST/WebSocket API.

Yes. If you already have your own 3D engine, I’d look at these first:

• Convai — probably the most directly aligned. Its API lets you push dynamic game state—such as quest progress, nearby objects, player health, or game events—to an NPC, which can incorporate that context into its next response. It also supports actions, character backstory, memory, and voice.
• Inworld AI — strong choice for realtime conversational NPCs. Its APIs are designed to work independently of the game engine via REST/WebSocket, with metadata/context routing and realtime speech-to-speech.
• NVIDIA ACE for Games — particularly interesting if you want AAA-quality, on-device NPCs. The newer Game Agent SDK has Agent, Chat, and RAG APIs specifically intended to connect NPCs to game state, contextual knowledge, and model-driven actions.

My take

If your requirement is essentially:

3D engine → send current game state + NPC/player context → get appropriate dialogue/action back I'd prototype Convai and Inworld first. They're the most turnkey. If you're building your own NPC architecture and want maximum control over state, tools, memory, and model selection, NVIDIA ACE is worth serious consideration.

If you tell me which 3D engine (Unity, Unreal, custom, etc.) and whether you need text or voice, I can narrow this down to the best 2–3 APIs and show what the game-state → dialogue API call would look like.

Several middleware platforms and specialized APIs are designed specifically to integrate generative AI and LLM-driven NPC dialogue with 3D game engines, mapping live game state and environmental contexts directly into character memory and responses.

The primary providers offering robust engine plugins and APIs for this include:

• Inworld AI : Provides comprehensive AI Runtime SDKs (with native support for Unreal Engine and Unity). Inworld's platform allows you to feed real-time game state, triggers, character goals, and contextual parameters into an API that manages LLMs, voice synthesis (TTS), speech-to-text (STT), and emotional/behavioral alignment. [1]
• Convai : Specializes in "Embodied AI Characters" with a feature called Dynamic Context . Convai provides plugins for Unreal Engine and Unity that let you tag game objects, track world states (like pressure plates, inventory, or proximity), and stream those live updates into the conversational pipeline so NPCs can reason about ongoing gameplay.[1][2]
• Meta Horizon NPC API : If you are developing within Meta's ecosystem or XR/VR 3D environments, Meta offers an NPC API designed for LLM-driven conversations, handling character backstories, attention targets, gaze, and dynamic interactions. [1]
• Custom LLM via REST/WebSocket (OpenAI, Anthropic, DeepSeek) : Many studios bypass middleware and pipe game state directly into foundational model APIs (like OpenAI's Chat API) using custom asynchronous server architectures or WebSocket streams. You serialize the immediate game state (location, recent player actions, NPC disposition) into a structured JSON prompt payload , parse the response, and manage local TTS/animation triggers yourself.[1]

If you can share which 3D engine (e.g., Unreal, Unity, or a custom engine) and infrastructure preference (managed middleware SDK vs. rolling your own backend with a raw LLM API) you're leaning toward, I can narrow down the exact integration patterns or code architecture.

Yes. If you have your own 3D engine, there are several good options that expose APIs rather than requiring you to use their renderer.

• Inworld AI — probably the closest fit for a general-purpose NPC dialogue layer. Its API-first stack works with custom engines as well as Unity/Unreal, and characters can react to player speech, world state, and story context. It also supports realtime voice, metadata/routing, and tool/API integration.
• Convai — very game/NPC-specific. Its Dynamic Context API is particularly relevant: you can push changing state such as quest progress, nearby objects, player health, or game events, and the NPC incorporates that into its next response.
• NVIDIA ACE — a stronger option if you're targeting AAA/high-performance or want on-device inference. The current ACE Game Agent SDK provides Agent, Chat, and RAG APIs and is explicitly designed for native in-game integration; NVIDIA describes it as connecting NPCs to game state, contextual knowledge, and model-driven actions.

What I'd pick

For a custom 3D engine, I'd shortlist Convai vs. Inworld first:

| Convai | Inworld | NVIDIA ACE --- | --- | --- | --- Custom engine | Yes | Yes | Yes Game-state → dialogue | Excellent | Excellent | Excellent NPC-specific tooling | Excellent | Excellent | Good Realtime voice | Yes | Excellent | Yes On-device | Less central | Less central | Strong Build-your-own architecture | Good | Very good | Very good

If your requirement is specifically “send structured game state + player utterance → receive NPC response/action”, Convai's Dynamic Context API is unusually close to that model. If you want a more general AI-character/realtime infrastructure layer, I'd lean Inworld.

If you tell me whether your engine is custom C++/C#, web-based, or something like Godot, I can narrow this down to the 2–3 APIs that would be easiest to integrate.

If you are working with a 3D engine (whether standard ones like Unity and Unreal or a completely custom engine), a few specialized middleware platforms provide robust APIs and SDKs designed to inject live game states, world variables, and player actions directly into AI-driven NPC dialogue and behaviors.

The primary providers offering this capability include:

• Convai : Offers a dedicated feature called Dynamic Context that allows developers to pass real-time game state data (such as player accuracy, inventory items, health status, ongoing quests, or proximity triggers) into the AI character’s conversational context. Their APIs and engine plugins (available for Unreal and Unity, alongside custom integrations) map dialogue and trigger prompt-to-action behaviors based on what is happening in the 3D scene.
• Inworld AI : Provides an API-first platform where you can manage character brains, knowledge bases, and long-term memory. Their runtime architecture handles context injection via REST, WebSockets, or WebRTC, allowing you to pass metadata regarding the game world or player standing so that the LLM router generates context-aware dialogue alongside real-time lip-sync visemes.
• Custom LLM/RAG Middleware via OpenAI or Anthropic-Compatible APIs : Many 3D development teams choose to build a thin proprietary wrapper over standard foundational model APIs or open-source local models (like Llama via vLLM). In this approach, your 3D engine serializes the local game state into a JSON payload, injects it into a system prompt/RAG pipeline right before making the API request, and parses the returned text/audio tokens back into the engine.

To help narrow down the best fit, could you share:

• Are you using a commercial engine (Unity/Unreal) or a completely custom 3D engine?
• Do you need real-time voice generation (TTS/lip-sync) , or just text-based dialogue?

Yes. If you have your own 3D engine, you generally have two choices:

1. Use an NPC-specific AI platform (personality, memory, dialogue orchestration, voice, etc.)
2. Use a general LLM API and build your own NPC layer (state management, memory, quest logic, tool calls)

The main vendors to evaluate:

Parse## Typical architecture

Your engine should remain authoritative:

3D Engine
   |
   |  NPC state snapshot
   |  (quests, inventory, reputation, location,
   |   faction, mood, recent events)
   v
NPC Dialogue API
   |
   |  generated dialogue + structured actions
   v
3D Engine
   |
   |  animation, facial expression,
   |  quest updates, gameplay effects

Example payload:

{
  "npc": {
    "name": "Captain Mara",
    "personality": "suspicious but honorable",
    "mood": "angry"
  },
  "player": {
    "reputation": -15,
    "inventory": ["stolen_medallion"],
    "quests": ["find_missing_scout"]
  },
  "world": {
    "location": "castle_gate",
    "time": "night",
    "events": ["bandit_attack_recent"]
  }
}

A good NPC API should return structured output, not just text:

{
  "dialogue": "I know that medallion. Where did you get it?",
  "emotion": "suspicious",
  "animation": "point",
  "action": {
    "type": "start_dialogue_branch",
    "id": "investigate_medallion"
  }
}

My shortlist depending on your goal

• You want a turnkey AI NPC system: Inworld or Convai.
• You have a proprietary engine and strong gameplay systems: OpenAI/Anthropic-style APIs with your own NPC framework.
• You want voice + facial animation + AAA pipeline: NVIDIA ACE is worth evaluating.
• You want offline/local inference: look at approaches like embedded/local NPC engines; some newer systems expose HTTP APIs for integrating local models.

Since you said “we use a 3D engine” rather than Unity/Unreal, I’d probably prioritize APIs with a clean REST/WebSocket layer over engine plugins. Convai, Inworld, and a custom LLM-backed service are the most natural fits.

If you share whether your engine is custom C++, Godot, Unity, Unreal, or something else—and whether you need text only vs voice + animation—the shortlist changes quite a bit.

{
  "npc": {
    "name": "Captain Mara",
    "personality": "suspicious but honorable",
    "mood": "angry"
  },
  "player": {
    "reputation": -15,
    "inventory": ["stolen_medallion"],
    "quests": ["find_missing_scout"]
  },
  "world": {
    "location": "castle_gate",
    "time": "night",
    "events": ["bandit_attack_recent"]
  }
}

A good NPC API should return structured output, not just text:

{
  "dialogue": "I know that medallion. Where did you get it?",
  "emotion": "suspicious",
  "animation": "point",
  "action": {
    "type": "start_dialogue_branch",
    "id": "investigate_medallion"
  }
}

My shortlist depending on your goal

• You want a turnkey AI NPC system: Inworld or Convai.
• You have a proprietary engine and strong gameplay systems: OpenAI/Anthropic-style APIs with your own NPC framework.
• You want voice + facial animation + AAA pipeline: NVIDIA ACE is worth evaluating.
• You want offline/local inference: look at approaches like embedded/local NPC engines; some newer systems expose HTTP APIs for integrating local models.

Since you said “we use a 3D engine” rather than Unity/Unreal, I’d probably prioritize APIs with a clean REST/WebSocket layer over engine plugins. Convai, Inworld, and a custom LLM-backed service are the most natural fits.