AI Observability screenshot
A hedgehog inspecting a trace with a magnifying glass
AI Observability

Observe and fix AI in production

Trace agent loops, evaluate live traffic, and get alerted when cost, latency, or quality slips. Self-driving uses this context to automatically make improvements and fix issues.

Get started - free

Install with AI in a single prompt

Paste into your terminal or code editor and make AI do the work.

Learn more

MCP•Watch a demo•Talk to a human

The old way to run AI in production

Your AI fails quietly – a hallucination here, a failed tool call there. Traces tell you what it did, but you have to go read them, one by one.

Debugging AI(cir. 2023-2026)
1. User reports an issueHuman
2. Read traces one by oneHuman
3. Spot the patternHuman
4. Guess which change broke itHuman
5. Write the fixHuman
6. Ship and waitHuman

Just to make a simple fix, you have to read the traces, spot the pattern, and figure out which change broke it – it's logs with a different schema.

The PostHog way

AI Observability captures the full context of each trace and connects it to other PostHog products (like Session Replay). Evals score your live traffic and anomaly alerts tell you when things are not normal, so regressions in tool calling, response quality, or latency automatically turn into signals.

The loop(cir. 2026–)
AI Observability captures every traceMachine
Evals & alerts score and watch live trafficMachine
Alerts report to your inboxMachine
Agent investigates and opens the PRMachine
You review & mergeHuman

With each regression, an agent investigates and creates a report in your Inbox. When there's code to fix, it opens a pull request. Your AI improves while you sleep.

Who is it for?

Different teams pull different answers from the same LLM data.

Role
Use cases
AI Engineers
Debug traces span by span and set up evals to catch quality regressions
Product Engineers
Tie failed generations and latency spikes back to the users who hit them
PMs
See which AI features get used – and whether using them changes retention
Finance & leadership
Attribute token spend to models, features, and individual customers
Support Engineers
Read the exact conversation behind a ticket, then watch the session it happened in

How do I use it?

There are a few ways to explore AI Observability.

Query LLM traces from your editor

Check LLM costs, monitor errors, and analyze model performance from Cursor, Claude Code, VS Code, or any MCP-compatible agent.

Check costs before and after deploys
Compare LLM spend across periods to spot unexpected jumps before they compound.
Monitor errors
Surface failing LLM calls so your agent can flag or fix them immediately.
Compare models
Evaluate cost, latency, and token usage across models to pick the right one per feature.
Turn failure modes into evals
Let your agent explore traces, spot recurring failure modes, and create evaluations that catch them.

Install the PostHog MCP

Learn more
npx @posthog/wizard mcp add

Supports Next.js, React, Python, and

Top features

An LLM trace with its timeline, tree, and generation detail

Trace every conversation end to end

Debug entire conversations, not just individual calls. Every trace is a timeline of its generations and spans, with the person, total cost, and total latency attached.

Trace timeline
A waterfall of every span and generation, with latency and cost at each step
Full conversation context
The exact input and output of every call, including multi-turn history and tool calls
Tokens and cost
Token counts per call, with cost calculated automatically from model pricing
Integrated session recordings
Jump from a trace to the session recording and watch what the response did in your UI
Privacy mode
Optionally exclude sensitive data from being captured

AI prompts

Ask PostHog AI to check what your LLM calls cost, dig into traces, and compare models. Works in PostHog AI (in-app chat), PostHog Desktop (our AI code editor), and in your product editor (using the MCP). Click a prompt to copy it, then paste it into your agent.

Costsget-llm-total-costs-for-project

Find tracesquery-llm-traces-list

Errors

Compare models

Works with other PostHog tools

Use AI Observability with these other PostHog apps to maximize shareholder value.

And the rest of PostHog:

Feature comparison

Langfuse
LangSmith
Braintrust
Generation tracking
✓
✓
✓
✓
Latency tracking
Track response times and identify slow prompts, models, and workflow steps
✓
✓
✓
✓
Cost tracking
Includes cost per user and broken down by provider, models
✓
✓
✓
✓
Trace visualization
View complete request traces across prompts, model calls, tools, and workflows
✓
✓
✓
✓
Token tracking
✓
✓
✓
✓
Prompt playground
Interactive testing environment for prompts and models
✓
✓
✓
✓
Prompt evaluations
Online LLM-as-a-Judge evaluations for measuring AI output quality
✓
✓
✓
✓
Alerting
✓
✓
✓
✓
Anomaly detection
Alerts that learn what normal looks like for cost, latency, and errors – no fixed thresholds to tune
✓
✗
✗
✗
Error tracking
Grouped error tracking for LLM applications
✓
✓
✓
✓
System prompts
Create and manage system prompts from the PostHog UI
✓
✓
✓
✓
Clustering
Automatic grouping of similar traces and outputs
✓
✓
✓
✓
Trace summarization
AI-generated summaries for quick understanding
✓
✓
✓
✓
LLM translation
Translation of non-English LLM traces to English
✓
✗
✗
Partial
Sentiment classification
Automatically classify user messages as positive, neutral, or negative
✓
Partial
Partial
✓
Privacy mode
Mask prompts and responses before they are stored
✓
✓
✓
✓
Agent/multi-step tracing
Understand complex agent and tool-calling workflows
✓
✓
✓
✓
Prompt management
Create, version, and manage prompts
✓
✓
✓
✓
Evaluation datasets
Create datasets for experimentation and benchmarking outputs
✓
✓
✓
✓
Human annotation/review
Review and label model outputs manually
✓
✓
✓
✓
User feedback
Capture ratings and comments from users on AI responses
Beta
✓
✓
✓
Agent reports & pull requests
Agents investigate eval results and anomalies, then file reports you can turn into PRs in one click
Beta
✗
Beta
✗
AI gateway/proxy
Route LLM requests through a gateway for caching, rate limits, fallbacks, and observability
✗
✗
✗
✓
Tracing
Trace requests across prompts, model calls, tools, and workflows
Hierarchical traces
Nested spans showing the full call flow
✓
✓
✓
✓
Custom spans
Instrument any operation as a span
✓
✓
✓
✓
Tool call tracking
Track function/tool calls in AI agents
✓
✓
✓
✓
RAG retrieval tracking
Monitor retrieval steps in RAG pipelines
✓
✓
✓
✓
Session grouping
Group traces into user sessions
✓
✓
✓
✓
OpenTelemetry support
Ingest traces via the OTel protocol
✓
✓
✓
✓
Async ingestion
Non-blocking trace collection
✓
✓
✓
✓
Multi-model support
Track calls across LLM providers
✓
✓
✓
✓
Session replay link
Jump from a trace to the user's session recording
✓
✗
✗
✗
User profile context
Connect traces to full user profiles with behavioral history
✓
Partial
Partial
✗
SQL queries on traces
Query trace data alongside product events
✓
Partial
✗
✓
Trace explorer UI
Dedicated interface for browsing and filtering traces
✓
✓
✓
✓
Prompt management
Create, version, deploy, and test prompts
Prompt versioning
Track changes to prompts over time
✓
✓
✓
✓
Template variables
Dynamic {{variables}} compiled at runtime
✓
✓
✓
✓
Prompt deployment API
Fetch the active prompt version via SDK
✓
✓
✓
✓
Version comparison
Side-by-side diff of prompt versions
✓
✓
✓
✓
Prompt configuration
Store model parameters and settings with each prompt version
✓
✓
✓
✓
Prompt labels
Tag prompts as production, staging, latest
✓
✓
✓
✓
Prompt playground
Test and compare prompts interactively
✓
✓
✓
✓
Composable prompts
Link and chain prompts together
✗
✓
✗
✗
MCP server for prompts
Manage prompts via AI coding agents
✓
✓
✓
✓
A/B test prompt versions
Split users between versions, measure cost, latency, and eval pass rate
✓
✓
Partial
✗
Evaluations
Score, review, and test LLM outputs
LLM-as-a-judge
Use models to score outputs automatically
✓
✓
✓
✓
Code evaluators
Custom scoring functions for automated eval
✓
✓
✓
✓
Annotation queues
Assign human reviewers to score outputs
✓
✓
✓
✓
Datasets
Curate sets of inputs and expected outputs
✓
✓
✓
✓
Experiment runs
Run evaluation pipelines across datasets
✓
✓
✓
✓
A/B experiments on product metrics
Statistical tests measuring impact on real user behavior
✓
✗
✗
✗
Costs
Track token usage, model costs, and spending trends
Token counting
Track input and output tokens per call
✓
✓
✓
✓
Cost calculation
Dollar cost per generation
✓
✓
✓
✓
Cost by model
Break down spending by model
✓
✓
✓
✓
Cost trends
Historical cost over time
✓
✓
✓
✓
Cost by custom tags
Break down spending by feature, environment, or any metadata you attach
✓
✓
✓
✓

Questions?

Answers

There are a few ways you can get answers to specific questions about AI Observability.

  1. Check the docs

    We have an entire docs-wizard dedicated to docs gardening.

  2. Search the community forums

    81 discussions about AI Observability, there's a good chance your answer is already answered!

  3. Ask PostHog AI

    It's an incredibly useful AI chat that understands the product, docs, community questions, our codebase, GitHub issues, and industry knowledge.

  4. Talk to a human

    Dedicated humans are standing by and ready to assist. Best for questions about volume pricing, terms, and sexy legal stuff.

Get started – free

No credit card required. You get the first 100,000 events free every month, then pricing starts at $0.00035/event and reduces with volume to $0.00006/event.

Install with AI

Run this command in your terminal or AI editor.
npx -y @posthog/wizard@latest ai-observability

Supports OpenAI, Anthropic, Google Gemini, and

Install without AI

Sign up for a free account and follow the install instructions for your stack.

Get started - free