Prompt analytics tool for identifying themes and failures

For your specific use case — “analyze thousands/millions of real user prompts, discover recurring themes, and identify failure modes” — my top pick is Braintrust.

My ranking

🥇 Why I'd choose Braintrust

The feature that's particularly relevant is Topics. It continuously analyzes production traces and automatically classifies them by Task, Issues, and Sentiment, using clustering plus LLM-assisted labeling. So instead of asking engineers to manually tag thousands of prompts, you can discover things like:

• “Users are increasingly asking for X”
• “20% of failures involve tool-call timeouts”
• “A new cluster of users is asking for Y”
• “This prompt version is associated with a particular failure”
• “Users are frustrated when the agent does Z”

Crucially, the resulting topics can become datasets, scores, and experiments, giving you the loop:

user prompts → cluster themes → identify failures → inspect examples → create eval dataset → fix → measure regression.www.braintrust.dev

Braintrust

🥈 Langfuse if you want more control

I'd pick Langfuse if self-hosting, data ownership, or an open-source stack is important.

Langfuse gives you detailed production traces, prompt/version metrics, user feedback, LLM-as-judge evaluation, custom dashboards, and the ability to turn failed production interactions into datasets.langfuse.com

It can also use its Assistant to fetch failed observations and cluster errors, although this is more of an analysis workflow than Braintrust's dedicated continuous Topics system.langfuse.com

Langfuse

The important distinction

I wouldn't choose a traditional “prompt management” product if your primary question is:

“What are our users actually asking for, and what are we consistently getting wrong?”

You want conversation analytics + observability + evaluation, rather than merely prompt versioning. Braintrust's current Topics functionality is unusually directly aligned with that requirement.www.braintrust.dev+1

My recommendation: start with Braintrust if you're comfortable with a managed commercial platform. Start with Langfuse if self-hosting/open-source is a major requirement.

If you tell me roughly how many prompts you have per day/month and whether you're using OpenAI, Anthropic, or an agent framework, I can narrow this down to the best 1–2 options and compare pricing, privacy, setup effort, and clustering quality.