I need a Unity plugin that generates texture maps from text descriptions at runtime.

Generating texture maps (like Albedo, Normal, Roughness, and Metallic maps) from text descriptions at runtime inside a Unity game requires an external or embedded AI inference pipeline. Because local diffusion models (like Stable Diffusion) are heavy and cloud APIs have latency, doing this live in a player build requires a specific architectural approach rather than a standard drag-and-drop editor asset.

The primary strategies and tools available to achieve runtime text-to-texture generation in Unity include:

1. Cloud-Based API Integration (Fastest & Most Reliable for Builds)

Instead of running the AI model on the user's local machine, your Unity C# script sends a REST/Websocket call to a cloud generation API, receives the image bytes back, and maps them to a dynamic Texture2D.

• How it works:
  1. Use Unity's UnityWebRequest or HttpClient to send a prompt to an image-generation API (e.g., OpenAI's DALL-E 3, Stability AI / Stable Diffusion API, or Replicate).
  2. Request multiple maps or use a secondary fast pass/shader to derive normal and roughness maps from the generated albedo map.
  3. Load the byte array into a runtime-created Texture2D using ImageConversion.LoadImage().
  4. Assign the textures dynamically to a Material instance on your GameObject.
• Pros: Works on low-end hardware, mobile-friendly (if connected to the internet), high-quality output.
• Cons: Requires an active internet connection, API costs per generation, and a 2–6 second latency.

2. Local AI Execution via Unity Sentis (On-Device Runtime AI)

If you need this to run entirely offline inside the client build without hitting a web API, Unity Sentis is Unity's cross-platform neural network inference engine that runs directly on the player’s device GPU/CPU.reddit.com

• How it works:
  1. Export a lightweight neural network (such as a smaller style-transfer or distilled latent generation ONNX model) that can run inside Sentis.
  2. Execute the model inference directly inside Unity via C# during gameplay based on the text string or latent tokens.
• Pros: Fully offline, zero per-request cloud cost, runs directly in-engine.
• Cons: True text-to-image/texture diffusion models are extremely heavy for mobile/client memory footprints; you are constrained to smaller, highly optimized ONNX models or super-resolution/stylization passes rather than full photorealistic 1024x1024 PBR generation.

3. Hybrid Local Python Bridge (For Desktop/Editor or Localhost Server)

For desktop-only standalone games, VR setups, or tool development, developers often run a local background instance of Stable Diffusion (via AUTOMATIC1111 or ComfyUI API) locally on localhost:7860.

• How it works:
  • Unity communicates via WebSockets/REST to the local Python script running Stable Diffusion on the user's PC.
  • The local server outputs the texture maps and returns them to Unity.youtube.com
• Pros: Uses state-of-the-art open-source models locally (SDXL, ControlNet for normal/roughness maps).
• Cons: Restricted to platforms powerful enough to run a local AI model alongside the game (heavy VRAM requirements).

Suggested Implementation Steps (Cloud API approach)

If you decide to go the Cloud API route (which is the most practical for a smooth runtime experience), your C# implementation structure will look similar to this conceptual layout:

To help narrow down the best architecture, tell me:

• Is this for a client-side standalone build (PC/Mobile) or a connected online game?
• Do you need true PBR maps (Normal, Metallic, Roughness) generated alongside the Albedo map, or just a single color texture?

Generating AI textures from text descriptions at runtime (in a standalone player build rather than just the Unity Editor) comes with specific technical constraints. Most off-the-shelf AI texture/material plugins (like Unity’s built-in Unity AI Material Generator or Meshy's Meshy Unity Plugin ) operate strictly inside the Editor as developer productivity tools.meshy.ai+2

Running text-to-texture generation live in-game requires connecting to a Cloud API (such as Replicate, OpenAI's DALL-E, Stability AI, or Fal.ai) via UnityWebRequest , or running a lightweight local model via Unity Barracuda.discussions.unity.com

How to Implement Runtime Text-to-Texture Generation

Since a rigid, plug-and-play asset store package for pure runtime text-to-texture generation doesn't natively exist off-line (due to model sizes), the standard architecture involves a lightweight wrapper script communicating with a cloud generation endpoint.

Recommended Architecture

1. The Prompt: Player enters text into an TMP_InputField.
2. The Request: Unity sends a POST request via UnityWebRequest to an AI image generation API (e.g., Stability AI / Stable Diffusion or Replicate).
3. The Response: The API returns raw image bytes (PNG/JPG).
4. The Runtime Conversion: Unity reads the bytes into a Texture2D using ImageConversion.LoadImage() , and dynamically assigns it to a Material or MeshRenderer.

Step-by-Step Runtime Implementation Example

Here is a clean, production-ready script template to handle runtime text-to-texture generation using a REST API approach:

using System;
using System.Collections;
using UnityEngine;
using UnityEngine.Networking;
using TMPro;

public class RuntimeAITextureGenerator : MonoBehaviour
{
    [Header("UI References")]
    [SerializeField] private TMP_InputField promptInputField;
    [SerializeField] private Renderer targetRenderer;

    [Header("API Settings")]
    [SerializeField] private string apiEndpoint = "https://stability.ai";
    [SerializeField] private string apiKey = "YOUR_API_KEY_HERE";

    public void GenerateTextureFromPrompt()
    {
        string prompt = promptInputField.text;
        if (!string.IsNullOrEmpty(prompt))
        {
            StartCoroutine(RequestAITexture(prompt));
        }
    }

    private IEnumerator RequestAITexture(string promptText)
    {
        // Construct JSON payload for the API request
        string jsonPayload = $"{{\"text_prompts\":[{{\"text\":\"{promptText}\"}}],\"cfg_scale\":7,\"steps\":30,\"width\":512,\"height\":512}}";
        
        using (UnityWebRequest request = new UnityWebRequest(apiEndpoint, "POST"))
        {
            byte[] bodyRaw = System.Text.Encoding.UTF8.GetBytes(jsonPayload);
            request.uploadHandler = new UploadHandlerRaw(bodyRaw);
            request.downloadHandler = new DownloadHandlerBuffer();
            
            request.SetRequestHeader("Content-Type", "application/json");
            request.SetRequestHeader("Accept", "application/json");
            request.SetRequestHeader("Authorization", $"Bearer {apiKey}");

            yield return request.SendWebRequest();

            if (request.result != UnityWebRequest.Result.Success)
            {
                Debug.LogError($"Texture Generation Failed: {request.error}");
            }
            else
            {
                // Parse response or handle direct image bytes depending on the chosen API provider
                // Assuming raw binary data or base64 decoding goes here:
                Texture2D dynamicTex = new Texture2D(2, 2);
                if (ImageConversion.LoadImage(dynamicTex, request.downloadHandler.data))
                {
                    ApplyTextureToMaterial(dynamicTex);
                }
            }
        }
    }

    private void ApplyTextureToMaterial(Texture2D tex)
    {
        if (targetRenderer != null)
        {
            targetRenderer.material.mainTexture = tex;
        }
    }
}

Use code with caution.

To help narrow down the ideal infrastructure for your game, tell me:

• Do you need this to run completely offline (locally on-device) using local inference, or is an online Cloud API acceptable?
• What is your target platform (PC, Mobile, or WebGL)?

Generating texture maps from text descriptions at runtime (during gameplay in a built standalone application, rather than just inside the Unity Editor) requires hitting a cloud-based AI REST API or communicating with a local/hosted Stable Diffusion backend (like an Automatic1111 WebUI instance).

Most out-of-the-box Unity tools (like Unity's Muse Texture Generator or the Meshy Unity Plugin ) are strictly editor-time tools meant to speed up developer workflows. For a true runtime solution, you will need an API wrapper approach.discussions.unity.com+1

How to Implement Runtime Text-to-Texture Generation

Since heavy diffusion models cannot run natively inside a standard client-side Unity build without heavy external piping, the architecture usually looks like this:

1. The Request: Unity sends a UnityWebRequest containing the text prompt (and desired maps like normal, roughness, or albedo) to an AI image generation API (such as OpenAI's DALL-E 3, Stability AI API, or a self-hosted Stable Diffusion server).
2. The Response: The server returns raw image bytes (PNG/JPG).
3. Runtime Decoding: Unity uses ImageConversion.LoadImageIntoTexture to parse the bytes into a Texture2D at runtime.
4. Application: The dynamic texture is assigned to a runtime material (Material.SetTexture ) applied to your game object.

Recommended Approach & Tools

• Stability AI REST API / Stable Diffusion WebUI API:
  If you want full control over generation parameters (and generation of multi-channel PBR maps like normal or displacement maps), connecting Unity to a Stable Diffusion API is the gold standard.
  • You can write a lightweight custom C# wrapper or look for open-source client implementations on GitHub tailored for Stable Diffusion / Automatic1111 API integration.
• OpenAI DALL-E 3 API:
  Easiest to integrate for pure albedo/diffuse color maps. You send a JSON POST request with your prompt, receive an image URL or base64 string, and load it straight into a dynamic material via a short async C# method.
• Asset Store Alternatives:
  Packages like Dynamic Text To Texture [URP] or the HDRP variant exist on the Unity Asset Store , but double-check their documentation—most handle procedural text/UI-to-texture generation locally rather than AI text-to-image synthesis.assetstore.unity.com+2

To help point you to the right architecture or code snippet, tell me:

• Are you looking to connect to a cloud API (like OpenAI or Stability AI) or a local/self-hosted AI server?
• Do you need full PBR map sets (Albedo, Normal, Roughness) or just a single diffuse/color texture?