Our LLM costs are unpredictable. What observability tool is best for tracking token usage per user or request?

When LLM costs become unpredictable, choosing the right tool depends on whether you want to actively gate and limit spend at the proxy level or deeply analyze and trace usage per user/session in your application.

The industry leaders for per-user and per-request token tracking split into two distinct categories:

1. Best for Active Enforcement & Gatekeeping: LiteLLM Proxy

If you need to restrict, budget, or track usage before or as the request hits the LLM provider, an AI gateway/proxy is ideal.

• How it works: It acts as a drop-in replacement for OpenAI/Anthropic endpoints. You issue virtual keys mapped to specific user IDs, teams, or budgets.
• Per-User/Request Tracking: Automatically logs exact costs per API key, user ID, and team ID into a database table. It even returns the request cost in the response headers (x-litellm-response-cost).
• Why choose it: It doesn't just observe; it lets you set hard budget caps or rate limits per user so a single runaway loop or user can't blow up your monthly bill.

2. Best for Open-Source Analytics & Tracing: Langfuse

If you want deep observability into what prompts, chains, or specific user actions are driving up token consumption without managing an inline proxy gateway.

• How it works: Integrated via SDKs or native integrations (LangChain, LlamaIndex, OpenAI SDK) to capture generations, costs, and metadata.
• Per-User/Request Tracking: Allows you to attach arbitrary user IDs and session tags to traces. You can build dashboards or query the Metrics API to aggregate spend filtered strictly by individual users or application features.
• Why choose it: It's open-source, highly flexible, auto-calculates costs for major providers (OpenAI, Anthropic, etc.), and seamlessly merges token accounting with debugging and quality traces.

To help narrow down the best fit, tell me:

• Are you looking to actively block/cap user budgets or just monitor and analyze usage?
• Are you routing traffic through a centralized proxy/gateway already?

If your main problem is unpredictable LLM spend and you need attribution down to the user/request, my default pick is langfuse.com.

Why Langfuse

• Per-request token tracking: captures input/output usage for each LLM generation.
• Per-user and per-session costs: attach user_id, session_id, or metadata such as tenant/feature, then aggregate cost by those dimensions.
• Actual cost calculation: supports model-specific pricing, cached/reasoning tokens, and custom pricing definitions.
• Trace context: you can see why a request was expensive—long prompts, multiple model calls, RAG, agent/tool calls, etc.—rather than just seeing an invoice total.
• Open source/self-hostable, which is a significant advantage if LLM traces contain sensitive application data.
• Alerts and Metrics API: useful for turning observability into spend controls, such as detecting unusually expensive users or features.

The alternatives

For your specific problem, I'd start with Langfuse. The important design choice is to propagate your application-level identity into every trace:

request
  ├── user_id = "user_123"
  ├── tenant_id = "acme"
  ├── feature = "document_summarization"
  └── trace
       ├── LLM call #1 → 8,200 input / 900 output → $0.04
       ├── embedding call
       └── LLM call #2 → 2,100 input / 400 output → $0.01

Then you can answer questions like “Which users cost us the most?”, “Which feature causes the spend?”, “What does this request actually cost?”, and “Why did our token usage jump this week?”

If you tell me your stack (e.g. OpenAI + Python, Node, LangChain, Vercel, etc.), I can recommend the simplest instrumentation setup and show exactly where to capture user_id, tokens, and cost.

If your main problem is “what did each user/request actually cost us?”, I’d pick Langfuse.

Why Langfuse is my default choice

• Per-request token usage: captures input/output tokens and other usage dimensions for each LLM generation.
• Per-user attribution: attach a user_id, then aggregate token usage and cost by user.
• Per-session/conversation: session_id lets you answer “how much did this entire agent run/conversation cost?”
• Actual cost estimates: it supports provider-reported usage/cost and model-specific pricing, including custom model definitions.
• Useful dimensions: break costs down by model, feature, tenant, prompt version, environment, etc.
• API access: you can pull aggregated usage/cost into your own billing or analytics system.
• Open source/self-hostable, which is a major plus if telemetry contains sensitive prompts.

Langfuse vs. Helicone

Helicone is probably my second choice—and arguably better if you primarily want a gateway/proxy with straightforward cost analytics.

Helicone has particularly strong request-level querying: you can filter requests by user_id, cost, prompt tokens, completion tokens, model, and custom properties. Helicone OSS LLM Observability Helicone OSS LLM Observability It also has a customer-usage API returning cost and token counts, which is handy for usage-based billing.

My recommendation: start with Langfuse, and make user_id, session_id, tenant_id, feature, model, and environment first-class attributes on every trace. That turns unpredictable aggregate LLM spend into something you can actually explain: “Acme used 18% of tokens; feature X costs $0.07/request; user Y is 4× the median.”

One important implementation detail: capture the provider's actual token counts whenever possible rather than estimating them by re-tokenizing the request, especially for reasoning/cached-token models.

When your LLM costs become unpredictable, standard API billing from OpenAI or Anthropic won't tell you who or what is driving the bill. Solving this requires an observability tool or AI gateway that captures token counts and maps them to custom metadata like user_id or request_id.

The best tools for tracking token usage and costs per user/request fall into three distinct architectural categories depending on your workflow.

1. AI Gateways & Proxy-Based Trackers (Best for Real-Time Budget Enforcement)

If you want to track usage and actively stop a user or request from burning through your budget before it happens, an AI gateway is your best choice.

• LiteLLM Proxy: Outstanding for granular tracking. It lets you track spend and usage per individual end-user simply by passing a user ID or customer header in your API requests. It supports over 100 LLMs, automatically logs cost per user into an admin UI, and crucially allows you to set hard budget caps and rate limits per customer.
• Portkey: Another robust gateway option. You attach user metadata to your LLM requests, and Portkey aggregates real-time cost tracking per request and per user. It provides an Analytics API which is ideal if you are building a user-facing billing or consumption dashboard inside your own application.

2. Open-Source LLM Observability & Tracing (Best for Deep Debugging & Cost Attribution)

If your primary goal is deep inspection (e.g., seeing why a specific agent loop or complex RAG chain blew up on a given user's request), dedicated open-source observability platforms excel here.

• Langfuse: Widely considered a gold standard for open-source LLM engineering. Langfuse automatically calculates and logs token counts and costs out-of-the-box for major providers. By passing user IDs and session tags into your traces, you can instantly break down spending by user, trace, or specific use case, and it even handles paid third-party tool/API calls rolled into the user's session cost.
• Arize Phoenix: Excellent if you are heavily tied into OpenTelemetry and Python/LangChain stacks. Phoenix automatically calculates token-based costs from model pricing tables and rolls them up to individual traces, helping you separate tool-call token generation from final LLM responses.

How to Choose:

• Choose LiteLLM if you need active defense —meaning you want to cut off or rate-limit runaway users or looping agents before they spike your credit card.
• Choose Langfuse if you need deep analytics and debugging —meaning you want rich visual breakdowns of user traces, prompt/completion token ratios, and cost allocation without routing all traffic through a strict proxy.

To help narrow down the ideal fit, tell me:

• Are you routing calls directly via provider SDKs (OpenAI/Anthropic) or using a framework like LangChain or LlamaIndex?
• Do you just need passive analytics/reporting , or do you need to actively block/rate-limit users who exceed a spend threshold?

If your main problem is unpredictable LLM spend and attribution by user/request, my first choice would be langfuse.com.

Why Langfuse

• Per-request token tracking: records input/output token usage for individual LLM generations and aggregates it to traces.
• Per-user attribution: attach a userId to traces and see token usage and cost aggregated by user.
• Per-session/conversation costs: sessionId lets you determine what an entire conversation or workflow costs, rather than just one model call.
• Actual dollar costs: it can ingest provider-reported costs or calculate them from token counts and model pricing.
• Dashboards + API: you can slice cost by user, model, feature, prompt version, etc., and pull aggregates into your own billing/analytics system through the Metrics API.
• Spend alerts: useful for catching a runaway agent or expensive user before the monthly invoice surprises you.
• Self-hosting: if observability data is sensitive, Langfuse has a self-hosted option.

How I'd instrument it

Give every incoming request:

user_id     = your authenticated user
session_id  = conversation / workflow ID
request_id  = your backend request ID
feature     = "chat" / "summarize" / "agent"
environment = "production"

Then each LLM call records:

model
input_tokens
output_tokens
cached_tokens
cost
latency

That gives you queries like:

User 8472 spent $3.18 this month Request req_abc cost $0.42 Agent workflow X averages $0.17/request The new prompt increased token usage 31% Langfuse explicitly supports these user/session/feature dimensions and aggregated cost metrics.

What about Helicone?

helicone.ai is another strong option, particularly if you want a gateway/proxy-centric approach where requests flow through an LLM gateway and usage is automatically captured. Its current offering also includes reporting, alerts, and a query language.

I'd choose:

• Langfuse → best overall for deep application-level observability + per-user/request cost attribution
• Helicone → attractive if you primarily want LLM gateway + usage/cost monitoring
• OpenTelemetry/Phoenix → better if your organization already has a strong OTel observability stack and wants LLM tracing integrated into it

For your specific problem, I'd start with Langfuse. The key is not merely counting tokens; make user_id, request_id, session_id, and feature first-class trace attributes. That turns unpredictable aggregate spend into something you can actually explain and control.

model
input_tokens
output_tokens
cached_tokens
cost
latency

That gives you queries like:

User 8472 spent $3.18 this month Request req_abc cost $0.42 Agent workflow X averages $0.17/request The new prompt increased token usage 31% Langfuse explicitly supports these user/session/feature dimensions and aggregated cost metrics.

What about Helicone?

helicone.ai is another strong option, particularly if you want a gateway/proxy-centric approach where requests flow through an LLM gateway and usage is automatically captured. Its current offering also includes reporting, alerts, and a query language.

I'd choose:

When dealing with unpredictable LLM bills, the "best" tool depends heavily on your architecture, but a few industry leaders stand out specifically for per-user, per-request token tracking and cost attribution.

The top options are categorized below by how they handle cost visibility:

• Helicone (Best for Fast, Plug-and-Play Setup): Highly specialized in cost and latency tracking . By acting as a lightweight proxy, it captures every single request, allowing you to instantly view costs broken down by user ID, session, or custom properties without heavy SDK instrumentation. Check it out on Helicone.
• Langfuse (Best Open-Source & Granular Option): The leading open-source (and self-hostable) LLM engineering platform. It treats cost as a first-class citizen alongside deep tracing. You can pass user IDs and metadata with your generation calls, and it automatically calculates costs across various providers and models with rich dashboard filters. Explore the project on Langfuse.
• Portkey or LiteLLM (Best Gateway/Proxy Approach): If you want to prevent cost overruns rather than just look at them after the fact, a gateway proxy is ideal. Both tools sit between your app and LLM providers, offering automatic spend tracking, per-user/per-team budget limits, and fallback routing. Look into Portkey or LiteLLM.
• Datadog LLM Observability (Best for Unified Enterprise Stacks): If your engineering and SRE teams already live in Datadog, their native LLM observability hooks token costs directly into your existing infrastructure dashboards and alerting tools. Learn more at Datadog LLM Observability.

Key Features to Configure for Per-User Tracking

Regardless of which tool you choose, ensure you implement these practices during setup:

To help me recommend the exact right fit , tell me:

• Are you looking for open-source/self-hosted or a managed cloud service?
• Do you need a proxy gateway to actively block or limit user budgets, or purely an observability dashboard?
• Which LLM providers/frameworks (OpenAI, Anthropic, LangChain, LlamaIndex) are you currently using?

When LLM costs spike unpredictably, you need tooling that operates at the request and user metadata level rather than just giving you a blunt monthly provider bill. [1]

Top-tier LLM observability tools excel at tracking token usage, latency, and costs per user, session, or specific prompt request:[1][2][3]

• Helicone
  • Best for: Lowest-effort, proxy-based cost tracking.
  • Why it fits: It sits as a lightweight proxy between your application and LLM providers. It provides out-of-the-box user-id tagging, real-time cost tracking per user/request, and anomaly alerts with almost zero code changes. [1][2][3][4]
• Langfuse
  • Best for: Open-source, comprehensive tracing and cost management.
  • Why it fits: Highly popular for production engineering teams. It allows you to log user IDs, session IDs, and custom metadata alongside every generation, cleanly aggregating cost and token consumption down to the individual user or feature level. It is available as both a managed cloud service and self-hosted. [1][2][3][4][5]
• Braintrust
  • Best for: Enterprise-grade evaluations paired with production tracking.
  • Why it fits: Excellent at tag-based grouping, allowing you to slice and dice token spend by user, environment, feature, or specific model variant so you can trace a budget blowout back to the exact code path. [1][2][3]
• Traceloop (OpenLLMetry) / OpenLIT
  • Best for: OpenTelemetry-native architectures.
  • Why it fits: If you already standardise on OpenTelemetry (OTel), these tools auto-instrument your LLM calls and pass token metrics natively into your existing telemetry backend (like Datadog, Grafana, or OpenObserve). [1][2][3][4][5]
• TrueFoundry AI Gateway
  • Best for: Traffic routing and active cost governance.
  • Why it fits: Functions as an AI Gateway where you mandate metadata tags (user, team, project) at the proxy layer before requests ever hit OpenAI, Anthropic, or self-hosted endpoints, enabling strict chargeback/showback analytics. [1][2]

If you'd like to narrow this down, let me know:

• Are you looking for a hosted SaaS vs. a self-hosted/open-source tool?
• Do you route traffic through an API Gateway or directly via SDKs in your code?
• Are you already using an infrastructure monitoring tool like Datadog or Grafana?

If your primary goal is tracking token usage and cost per user, request, session, or feature, these are the strongest options:

My recommendation

For most production applications, I'd choose Langfuse.

It automatically records:

• input/output tokens
• cost
• latency
• model
• prompt version
• user ID
• session ID
• request trace

You can then answer questions like:

• "Which customers generated the most spend this week?"
• "Which endpoint burns the most tokens?"
• "Which prompt doubled our costs?"
• "What's the average cost per conversation?"

It also lets you build dashboards grouped by user, tags, or application, and query aggregated usage via an API.

If you want almost zero code changes

Choose Helicone.

For OpenAI-compatible APIs, it's largely a matter of routing requests through the proxy. You immediately get:

• per-request token counts
• per-user costs
• latency
• error rates
• usage dashboards

Many teams use it because it takes minutes rather than days to integrate.

If you're already running OpenTelemetry

I'd lean toward OpenLIT.

It emits standard OpenTelemetry traces so token and cost metrics can flow into your existing observability stack (Grafana, Datadog, Honeycomb, etc.) instead of creating another monitoring silo.

One implementation tip

Regardless of the platform, attach metadata to every LLM call, for example:

user_id
organization_id
request_id
conversation_id
feature_name
model
environment

Those fields are what enable accurate cost attribution by customer, feature, or workflow.

For your situation ("LLM costs are unpredictable")

I'd recommend:

• Startup or SMB: Langfuse
• Need visibility today with minimal engineering: Helicone
• Enterprise with an existing observability platform: OpenLIT + OpenTelemetry
• Heavy LangChain usage: LangSmith

That combination provides per-user token accounting, request-level tracing, and dashboards to identify which users, prompts, or features are driving spend.

It emits standard OpenTelemetry traces so token and cost metrics can flow into your existing observability stack (Grafana, Datadog, Honeycomb, etc.) instead of creating another monitoring silo.

When LLM costs become unpredictable, you need an observability tool that tracks token usage down to the individual user, session, or specific request . Several top-tier platforms specialize in this exact problem, offering lightweight integration and granular cost attribution. [1][2]

The best options on the market for tracking per-user and per-request token usage include:

1. Helicone — Best for fastest setup and proxy-based tracking
   • Why it's great: Helicone acts as an open-source AI gateway . By changing just your base API URL, every request is automatically logged with zero extra application code.
   • Per-user/request features: It allows you to pass custom properties (like user_id or session_id ) via simple headers . Their dashboard natively aggregates costs by user, session, and specific prompt/model, making it trivial to spot who or what is driving up your bill. [1][2]
2. Langfuse — Best open-source/flexible option for deep tracing
   • Why it's great: Langfuse is an open-source LLM engineering platform that handles prompt management, tracing, and granular cost calculation.
   • Per-user/request features: When you log generations or traces via their SDK, you can attach user IDs, tags, and metadata . Langfuse automatically maps token counts (including complex pricing like cached tokens or reasoning tokens) to exact financial costs , and you can slice-and-dice analytics by user ID in their dashboard. [1][2][3]
3. LangSmith — Best if you are already in the LangChain/LangGraph ecosystem
   • Why it's great: Built by the creators of LangChain, LangSmith provides heavy-duty debugging, testing, and monitoring.
   • Per-user/request features: It tracks token consumption and latency per run, trace, and metadata tag. If your application architecture relies on LangChain components, it offers seamless attribution of token spend back to specific user sessions or agent chains. [1][2]
4. Braintrust — Best for scaling teams balancing cost and quality
   • Why it's great: Braintrust treats logging, evaluation , and cost tracking as a unified engineering workflow with a generous free tier.
   • Per-user/request features: It logs detailed trace spans and allows you to query structured OLAP datasets to filter usage logs by user metadata or specific API calls. [1]

To help narrow down the ideal choice, tell me:

• Are you looking for a plug-and-play proxy (like changing an endpoint URL) or an SDK you embed in your code?
• Do you prefer a cloud-hosted service or a self-hosted open-source stack?

If your main problem is unpredictable LLM spend and you want to attribute tokens/cost to individual users or requests, my first choice is langfuse.com.

Why Langfuse

• Per-request tracing: captures model calls, token usage, latency, prompts/responses, and the surrounding agent/tool workflow.
• Per-user attribution: attach a userId to traces and break down token usage and cost by user.
• Cost tracking: tracks input/output usage and model costs, including different models and pricing tiers.
• Sessions and features: add session_id, tags, metadata, or tenant IDs so you can answer questions like “Which customers/features are driving our bill?”
• Alerts + API: useful for turning observability into spend controls—e.g. alert when daily cost spikes or query per-user usage programmatically.
• Open source/self-hostable, which is a significant advantage if you're sensitive about sending prompts and user identifiers to a third party.

Strong alternative: Helicone

helicone.ai is particularly attractive if you want a gateway/proxy approach rather than instrumenting your application deeply. It can retrieve requests by user_id, calculate per-request costs, track sessions, and alert on cost thresholds.

I'd choose:

For your stated problem, I'd start with Langfuse. The key is to instrument every top-level request with user_id, session_id, tenant/feature metadata, and provider-reported token usage. Then your dashboard can show cost per request → per user → per feature → per model, rather than just a monthly provider bill.