We need to monitor token cost and latency across several models — what should we use?

To monitor token costs, latency, and performance across multiple models, you need an LLM Observability or AI Gateway platform.

The ideal tool depends on whether you want a lightweight proxy gateway (which also handles routing/fallbacks) or a deep tracing suite (for complex agent workflows and granular debugging).

Top Recommendations

1. • Langfuse — Best overall for open-source and deep tracing. It provides granular span-based tracing, tracks token consumption and latency out of the box, and can be self-hosted or used via cloud. It is exceptional if you need to inspect nested tool calls or multi-step agent actions without runaway pricing.
2. • Portkey — Best if you want an AI Gateway combined with monitoring. It sits in front of your LLM calls, giving you real-time token analytics, cost tracking per model/app, and latency breakdowns while simultaneously handling production routing, load balancing, and fallbacks.
3. • Braintrust — Best for CI/CD eval-gated deployments and generous free tiers. It tracks spend and spans effectively, though it shines brightest if your team relies heavily on structured evaluation workbenches and regression testing.
4. • Helicone — Best for a lightning-fast, drop-in proxy setup. If you just need a single header change in your API calls to instantly view cost, latency, and user analytics on a clean dashboard, it's one of the leanest options available.
5. • Arize Phoenix — Best open-source option geared toward semantic search, RAG evaluation, and heavy ML observability/drift detection.

Quick Selection Guide

• Use Portkey or Helicone if you want an API gateway approach where monitoring is tied to request routing and proxy management.
• Use Langfuse if you want a code-level SDK approach for deep tracing, cost attribution, and debugging complex multi-model pipelines.

If you want to narrow this down, tell me:

• Are you looking for a self-hosted/open-source tool or a fully managed cloud service?
• Do you also need the tool to handle API routing and fallbacks , or strictly monitoring and analytics?

For monitoring token cost + latency across multiple LLM providers/models, I’d use an OpenTelemetry-based observability stack rather than building provider-specific logging from scratch.

Recommended setup

App → OpenTelemetry → observability backend

Capture, at minimum:

• Model/provider: model name, provider, API endpoint
• Tokens: input, output, cached, and reasoning tokens where available
• Cost: calculated from the provider/model pricing table
• Latency: total duration, TTFT (time to first token), and ideally tokens/sec
• Request metadata: endpoint, status, finish reason, retries
• Trace context: tie LLM calls to the user request, agent step, retrieval, and tool calls

OpenTelemetry's current GenAI conventions specifically define token-usage and latency metrics, including token type, provider, requested/response model, request duration, and time-to-first-token.

For the backend, Langfuse + OpenTelemetry is a practical combination if you want an LLM-focused UI. Langfuse's current SDK is OTEL-native and adds LLM-specific token/cost tracking and scoring on top of OpenTelemetry.

Dashboard I'd build

Have dimensions for:

The important thing is not to aggregate only average latency or total tokens. Histograms and p50/p95/p99 are much more useful for comparing models, and OpenTelemetry's GenAI metrics are designed around this approach.

If you're starting from scratch

I'd choose:

OpenTelemetry → Langfuse → Grafana/your existing metrics stack

rather than instrumenting each model separately. That gives you a common schema across OpenAI, Anthropic, Gemini, Bedrock, etc.; OpenTelemetry explicitly defines provider/model fields for multiple major providers.

If you tell me your stack (Python/Node, direct APIs vs. LiteLLM, Kubernetes/serverless, and which models), I can give you a concrete architecture and instrumentation example.

To monitor token cost, prompt mixes, and latency across multiple models, you should use a dedicated LLM observability or gateway platform rather than building custom loggers . The best choice depends on whether you prefer an open-source/self-hosted stack or a managed cloud gateway.

Top LLM Monitoring Platforms

Key Metrics You Should Track

To get actionable insights out of your multi-model architecture, configure your chosen platform to capture:

1. Token Mix Breakdown: Do not just track total token consumption. Monitor the balance of input tokens, output tokens , and cached tokens to identify bloated prompts or inefficient context windows.
2. Latency Percentiles (p50, p95, p99): Track Time to First Token (TTFT) for streaming applications alongside total request completion time.
3. Granular Cost Tagging: Tag requests by feature, user, or team so you can trace expensive background agent loops or rogue retry loops directly to their source.

If you can share how you currently route your model requests (e.g., direct API calls, via LangChain/LlamaIndex, or through a proxy) and whether you prefer an open-source self-hosted or fully managed cloud tool, I can recommend the exact integration path.

For monitoring token cost and latency across multiple models, you’re looking for an LLM observability layer rather than just normal APM. The key metrics to capture are:

• Input/output tokens (and cached tokens if applicable)
• Cost per request, user, feature, and model
• Latency: time-to-first-token (TTFT), total latency, p50/p95/p99
• Model/provider breakdowns
• Prompt/version changes
• Failures, retries, and fallback routing

OpenLegionA practical shortlist:

LangChain PremaiMy usual recommendations:

If you have several providers (OpenAI + Anthropic + Gemini + self-hosted models)

Use:

• Langfuse for traces/cost dashboards
• OpenTelemetry instrumentation where possible
• Export into your existing observability stack if needed

This gives you portability instead of tying your metrics to one vendor.

If your main question is “why did our bill explode?”

Use:

• Helicone or Langfuse
• Add tags like: - app=chat
• customer_id
• feature=search
• model=gpt-x
• environment=prod

Then you can answer:

• “Which feature consumed 80% of tokens?”
• “Which model is slowest?”
• “Are retries causing cost spikes?”

OpenLegion### If you already run enterprise monitoring

If your team lives in Datadog, New Relic, Grafana, etc., it may be better to keep LLM telemetry there rather than adding another dashboard.

For a new production AI system, a solid default stack would be:

Application
   |
   +-- LLM calls
   |
   +-- OpenTelemetry / SDK instrumentation
            |
            +-- Langfuse (cost + traces + prompts)
            |
            +-- Existing APM (alerts + infra)

A few things I would avoid:

• Only tracking provider invoices (too coarse)
• Only tracking total tokens (doesn’t explain user/feature cost)
• Logging raw prompts/responses everywhere without redaction controls

If you tell me your stack (OpenAI/Anthropic/etc., language, Kubernetes/serverless, and whether you need self-hosting), I can narrow this to a specific architecture.

Application
   |
   +-- LLM calls
   |
   +-- OpenTelemetry / SDK instrumentation
            |
            +-- Langfuse (cost + traces + prompts)
            |
            +-- Existing APM (alerts + infra)

A few things I would avoid:

• Only tracking provider invoices (too coarse)
• Only tracking total tokens (doesn’t explain user/feature cost)
• Logging raw prompts/responses everywhere without redaction controls

If you tell me your stack (OpenAI/Anthropic/etc., language, Kubernetes/serverless, and whether you need self-hosting), I can narrow this to a specific architecture.

If you’re comparing several LLM providers/models, I’d use an LLM observability layer rather than relying on each provider’s dashboard.

My pick: Langfuse

langfuse.com is a strong default because it can track input/output tokens, cost, latency, model, traces, and usage over time across providers. It also supports custom model pricing, which is useful when you mix OpenAI, Anthropic, Google, or self-hosted models.

For each request, I'd capture:

• Model/provider
• Input tokens
• Output tokens
• Cached tokens
• Cost in USD
• TTFT (time to first token)
• Total latency
• Tokens/sec
• Request success/error
• Prompt/use-case/version
• Environment (prod/staging)

Then build dashboards comparing, for example:

Important: use provider-reported token counts whenever possible. Inferring cost by tokenizing text can be wrong for reasoning models and cached-token accounting.

If you only use OpenAI

You may not need another platform initially. OpenAI's Usage Dashboard/API gives you usage and cost data, while API responses expose token counts. For latency analysis, OpenAI recommends looking at P50/P75/P95, TTFT, request time, and token velocity rather than averages alone.

Practical architecture

I'd do:

Your app → OpenTelemetry/Langfuse → dashboards/alerts

and tag every LLM span with:

provider + model + use_case + environment + input_tokens + output_tokens + cost + TTFT + duration

If you're already using Datadog, Grafana, or another OTel backend, you can also keep your existing observability stack and emit standardized GenAI telemetry rather than introducing a second monitoring system.

Bottom line: for a multi-model production setup, I'd start with Langfuse + OpenTelemetry, and use provider-native billing dashboards as the source of truth for invoice reconciliation.

You should use an LLM observability and AI gateway platform like Langfuse, Helicone, Portkey , or LangSmith to track token spend and latency across multiple models.

Top Monitoring Tools

• Langfuse : Open-source LLM engineering platform offering robust cost tracking, latency metrics, and detailed tracing with deep framework integrations.
• Helicone : Lightweight proxy and observability tool designed specifically for fast integration, real-time cost tracking, and latency monitoring via custom headers.
• Portkey : Production-ready LLM gateway and operations suite that supports fallback rules, load balancing, semantic caching, and granular cost tracking by user or model.
• LangSmith : Comprehensive platform by LangChain that features gateway spend limits, evaluation tools, and granular latency/token telemetry.

Key Metrics to Track

If you can share your primary tech stack (e.g., LangChain, LlamaIndex, or raw API calls) and hosting environment (cloud vs. self-hosted), I can recommend the easiest integration path.

For monitoring token cost and latency across multiple models/providers, you usually want an LLM observability layer rather than only traditional APM. The key signals to capture are:

• Input/output tokens (and reasoning/cache tokens if applicable)
• Cost per request, model, feature, user, and team
• Time to first token (TTFT)
• End-to-end latency (P50/P95/P99)
• Error rates, retries, fallbacks
• Model/version and prompt version
• Trace context across multi-step agent workflows

A few good choices:

A common production architecture:

Your app
   |
   v
LLM gateway (optional)
   |
   +--> OpenAI
   +--> Anthropic
   +--> Gemini
   +--> self-hosted models
          |
          v
OpenTelemetry traces
          |
          v
Observability backend
(Langfuse / Phoenix / Datadog / Grafana)

My practical recommendations:

• Small team / startup: Langfuse + OpenTelemetry.
• Need drop-in visibility across many providers: Helicone or an LLM gateway such as LiteLLM + dashboards.
• Enterprise already standardized on monitoring: Datadog or Grafana stack.
• Self-hosting/privacy requirements: Langfuse or Phoenix.

At minimum, create dashboards by:

• model
• application feature
• customer/team
• prompt version
• environment
• cost/day
• P95 latency
• tokens/request

Also add alerts such as:

• “Daily spend projected > budget”
• “P95 latency increased 30%”
• “Token usage per request doubled”
• “Model fallback rate exceeded threshold”

If you tell me your stack (OpenAI/Anthropic/self-hosted, LangChain or not, cloud provider, and scale), I can suggest a more specific setup.

To monitor token costs and latency across multiple LLM providers effectively, you need tools that fall into two main categories: AI Gateways (for inline routing, rate limiting, and instant proxy-based logging) and LLM Observability Platforms (for deep tracing and debugging).

Here are the top options depending on your setup:

1. Dedicated AI Gateways (Best for Proxy & Spend Control)

• LiteLLM: An open-source proxy/gateway that translates OpenAI-compatible formats to 100+ providers. It maps token pricing out-of-the-box, tracks spend per API key, user, or team, and handles load balancing and fallbacks.
• Helicone: A lightweight, one-line proxy integration that logs per-request token consumption, exact dollar costs, and latency metrics via a clean dashboard within minutes.

2. LLM Observability & Tracing (Best for Deep Analytics & Debugging)

• Langfuse: An open-source LLM engineering platform that records cost and latency directly inside granular multi-step application traces, letting you see spend down to a specific user or agent execution step.
• Galileo: An enterprise-grade observability suite that excels at real-time production traffic evaluation, tracking quality metrics alongside cost and latency.

3. Cloud & Infrastructure Cost Management

• CloudZero / Vantage: If you prefer pulling token spend straight from provider billing APIs and correlating it with your wider cloud infrastructure costs for unit-level business metrics.

To help narrow down the ideal pick, tell me:

• Are you looking for an inline proxy/gateway (to also handle routing/fallbacks) or a passive observability SDK?
• What is your stack/infrastructure preference (self-hosted open-source vs. fully managed cloud)?

If you’re comparing several LLMs/providers and care specifically about token cost + latency, I’d use OpenTelemetry (OTel) GenAI instrumentation as the common telemetry layer, then send it to whichever observability backend you already use.

OpenTelemetry’s current GenAI conventions cover exactly the metrics you want: token usage, request duration, time-to-first-token, and time-per-output-token. They also standardize provider/model attributes, making cross-model comparisons much easier.

What I’d track

For every model request, record:

• Model/provider — e.g. model A vs. model B
• Input tokens
• Output tokens
• Billable tokens, when the provider exposes them
• Cost per request — calculate from the model's current pricing
• Total latency
• TTFT — time to first token
• Tokens/sec — especially useful for streaming
• p50 / p95 / p99 latency
• Error/retry rate
• Prompt/version and route so you can explain regressions

OTel explicitly defines token-usage and latency metrics, and its guidance recommends using billed token counts when available.

Recommended architecture

Your application
      │
      ├── Model A
      ├── Model B
      ├── Model C
      │
      ▼
OpenTelemetry GenAI instrumentation
      │
      ├── traces
      ├── token metrics
      └── latency metrics
      │
      ▼
Your observability backend
      │
      ├── Cost/model dashboard
      ├── p50/p95/p99 latency
      ├── TTFT
      └── Alerts

The big advantage is provider neutrality: OTel's conventions already identify providers including OpenAI, Anthropic, Google, AWS Bedrock, Cohere, Mistral, Groq, and others.

If you want something you can stand up quickly rather than assembling the OTel stack yourself, I can also compare Langfuse vs. Helicone vs. Braintrust vs. Datadog vs. a pure OpenTelemetry stack specifically for multi-model cost/latency monitoring.

To monitor token cost and latency across multiple AI models, you need an LLM observability platform or an AI gateway . The best choice depends on whether you prefer an open-source self-hosted stack, a drop-in API proxy, or an enterprise-grade observability suite.

Top tools mapped to different use cases include:

• Langfuse (Langfuse ): An open-source LLM engineering platform that excels at tracking cost, latency, and detailed traces down to individual steps in agent runs. It can be self-hosted or used via cloud, making it great for granular analytics.
• Helicone (Helicone ): A lightweight, open-source proxy that requires just a one-line configuration change to your API base URL. It gives you instant dashboards for latency, token consumption, and cost tracking with minimal setup.
• LiteLLM (LiteLLM ): An open-source AI gateway that standardizes requests across 100+ model providers. It handles fallback logic, load balancing, and tracks real-time spend per API key, user, or team.
• Arize Phoenix / Arize AX (Arize AI ): Ideal for enterprise teams looking to deeply evaluate production traffic, track performance drift, and combine robust tracing with automated evaluations and cost analysis.
• LangSmith (LangChain ): The go-to choice if your stack is already built heavily around LangChain or LangGraph, offering native debugging, latency profiling, and token tracking.

To help narrow down the best fit, tell me:

• Are you looking for an open-source/self-hosted tool or a fully managed cloud service?
• Do you need an API proxy gateway that actively routes traffic, or purely a passive observability SDK?
• What frameworks or model providers (OpenAI, Anthropic, Bedrock, custom, etc.) make up your stack?