Data as of Jul 25, 2026 · Based on 267 AI responses from ChatGPT Search and Google AI Mode · See how Parse measures this
To evaluate your multimodal agent's trajectory, the best approach depends on your development stage. For pre-release simulation, 
Brands AI recommends here
Best for step-by-step diagnostic evaluation of agent reasoning and tool use. It provides precise metrics for multi-modal agents to catch workflow failures before they reach production.
Best for simulation-based testing. Its platform excels at stress-testing agents across many user scenarios and edge cases, providing a full-stack environment for evaluating trajectory quality.
Best for evaluating generalist multimodal agents in visual-heavy tasks. It is designed to test agents on long-horizon reasoning that relies on understanding images, charts, and screen data.
LangChainSource | Citation share |
|---|---|
How to evaluate your agent with trajectory evaluations - Docs by LangChainhttps://docs.langchain.com/langsmith/trajectory-evals | 4.4% |
 AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent ...https://aclanthology.org/2025.emnlp-demos.15.pdf | 3.2% |
 Evaluating Automatic Evaluations of Web Agent Trajectories - arXivhttps://arxiv.org/html/2504.08942v2 | 2.0% |
Evaluating Automatic Evaluations of Web Agent Trajectorieshttps://arxiv.org/pdf/2504.08942 | 1.9% |
 Forum | OpenReviewhttps://openreview.net/forum | 1.6% |
 How to train and evaluate AI agents and trajectories with Labelboxhttps://labelbox.com/blog/how-to-train-and-evaluate-ai-agents-and-trajectories-with-labelbox/ | 1.4% |
 Agent Trajectory Evaluations - Arize AX Docshttps://arize.com/docs/ax/evaluate/evaluators/trace-and-session-evals/trace-level-evaluations/agent-trajectory-evaluations | 1.2% |
 METR - Wikipediahttps://en.wikipedia.org/wiki/METR | 1.2% |
 GitHub - langchain-ai/agentevals: Readymade evaluators for agent trajectories · GitHubhttps://github.com/langchain-ai/agentevals | 1.2% |
Alessio Lomuscio - Wikipediahttps://en.wikipedia.org/wiki/Alessio_Lomuscio | 1.0% |
Yes. If by “agent trajectory quality” you mean evaluating the process—tool choices, action ordering, recovery, efficiency, safety, and whether the path was sensible rather than merely whether the final answer was correct—there are several groups I’d pay attention to.
agentevals package explicitly evaluates trajectories, including reference-trajectory matching and LLM-as-judge assessment of the whole execution path. I'd separate your evaluation problem into four layers:
| Layer | Question |
|---|---|
| Outcome | Did the agent accomplish the task? |
| Trajectory | Did it take a sensible, efficient sequence of actions? |
| Interaction | Did it correctly interpret images/screens/UI state and choose appropriate actions? |
| Safety | Did it avoid unauthorized, irreversible, or otherwise dangerous intermediate actions? |
The second and third are where a multimodal agent gets substantially harder than a conventional LLM agent. A trajectory can be outcome-correct but visually wrong: e.g., it misinterprets a UI element, makes an unnecessary action, then happens to recover.
For your release, I'd therefore consider measuring at least:
task success + action/tool correctness + visual-state grounding + trajectory efficiency + recovery quality + policy/safety compliance.
One important distinction: don't make “trajectory match” your definition of trajectory quality. A reference trajectory is useful for deterministic workflows, but there can be many equally good paths. LangChain's own docs distinguish strict/reference matching from LLM-judge approaches for precisely this reason.
If you tell me what kind of multimodal agent you're releasing (computer-use agent, GUI agent, voice+vision, robotics, document agent, etc.), I can give you a much tighter list of the 10–15 researchers/labs/companies I would actually contact or study, including the relevant papers and benchmarks.
Evaluating agent trajectory quality —the sequence of tool calls, intermediate reasoning steps, and path efficiency rather than just the final answer—has become a specialized field within AI engineering.
Specialized platforms, open-source frameworks, and evaluation-first companies specifically built or heavily optimized for multi-step agent tracing and trajectory evaluation include the following:
pytest ) for agent trajectories, measuring plan quality, step efficiency, and tool correctness.To help narrow down which tool or partner fits your multimodal release:
Evaluating agent trajectory quality —the complete sequence of reasoning, tool selection, parameter validity, state changes, and recovery loops rather than just the final output—is the core focus of several specialized AI observability and evaluation platforms.
The top platforms and specialized tools explicitly built to capture, trace, and score agent trajectories include:
To help narrow down which platform fits best, tell me:
Yes. “Agent trajectory quality” is now a recognizable evaluation specialty, but it is still a relatively young field. If you're preparing a multimodal agent release, I’d look at these groups/people first:
| Who | Why they’re relevant |
|---|---|
| Jing Yu Koh / VisualWebArena team | Probably one of the strongest academic matches for multimodal, interactive agent trajectories. VisualWebArena evaluates agents operating on realistic websites and explicitly releases full trajectories and human trajectories for analysis. github.com |
| XLang / OSWorld team (Shuyan Zhou and collaborators) | Excellent match if your agent interacts with a computer/GUI. OSWorld evaluates multimodal agents in real computer environments, where the sequence of perception → action → feedback matters, rather than just the final answer. github.com |
| Amazon Science — TRAJECT-Bench team | One of the most directly relevant recent efforts. TRAJECT-Bench explicitly scores tool selection, argument correctness, and action ordering/dependencies rather than merely task success. www.amazon.science |
| LangChain / LangSmith — AgentEvals team | Probably the most directly useful production tooling option. Their AgentEvals library is specifically designed around agent trajectories, including trajectory-match and LLM-as-judge evaluators. github.com |
| Anthropic's agent-evals group | Particularly interesting if you care about deployment-quality evaluation, not just benchmarks. Anthropic defines a trajectory/transcript as the complete record of an agent trial and recommends combining code-based, model-based, and human graders. www.anthropic.com |
| AgentLens authors (Andrey Podivilov et al.) | Very recent work specifically on trajectory reviews, combining formal verification with LLM-written reviews and side-by-side trajectory comparison. Currently focused on coding agents, but the methodology is highly applicable. arxiv.org |
| ATBench team (Yu Li et al.) | Worth talking to if trajectory safety is part of your release criterion. Their benchmark evaluates long-horizon agent trajectories and diagnoses where safety failures emerge during a sequence, rather than just at the endpoint. arxiv.org |
| Plan-RewardBench team (Jiaxuan Wang et al.) | Particularly relevant if by “trajectory quality” you mean “can a judge tell a good trajectory from a subtly bad one?” Their benchmark tests trajectory-level reward models on planning, tool irrelevance, safety refusal, and error recovery. aclanthology.org |
For a multimodal agent, I'd prioritize:
There is an important distinction here: trajectory quality ≠ trajectory matching. A good agent may legitimately take a different route from a reference trajectory. Recent work increasingly evaluates dimensions such as:
The literature is actually quite candid that this remains an unsolved problem: a 2026 survey found that most agent benchmarks still rely heavily on binary outcome metrics, with trajectory-level scoring substantially underdeveloped.
For your particular use case, I would not start by asking “who evaluates agents?” I'd ask “who evaluates multimodal agent trajectories as behavioral objects?” That narrower question points most strongly toward VisualWebArena/OSWorld + TRAJECT-Bench + Anthropic/LangSmith, with ATBench added if safety matters.
If you tell me what kind of multimodal agent you're releasing (computer-use, browser, mobile, embodied/robotic, visual research, etc.), I can narrow this to 5–10 specific researchers/labs worth contacting, including their recent papers and what each could contribute.
Yes. If by “agent trajectory quality” you mean evaluating the entire sequence of observations → decisions → tool calls → intermediate actions → recovery → final result, rather than just the final answer, there are a few groups I’d put at the top of your list.
| Group | Why I’d talk to them | Particularly relevant to multimodal? |
|---|---|---|
| LangChain / LangSmith | Probably the most directly specialized production tooling for trajectory evaluation. Their AgentEvals explicitly evaluates trajectories using reference matching and LLM-as-judge, including tool-call sequence, ordering, extra/missing actions, etc. docs.langchain.com | Yes, if you represent multimodal observations/actions in the trace |
| Arize AI / Phoenix | Strong on trace-level agent evaluation: evaluating the complete sequence of tool calls rather than isolated spans or only the final response. Good fit if you care about production monitoring as well as pre-release evals. arize.com | Yes |
| Microsoft Research / AgentLens | Particularly interesting if you want process-quality research, not merely observability. AgentLens explicitly distinguishes “lucky passes” from genuinely good trajectories and scores waste, divergence, verification, etc. arxiv.org | More SWE-focused today |
| Fernando Díaz / trajectory-preference evaluation | Very interesting research direction if you're asking “which trajectory is better?” rather than “did it follow my reference trajectory?”. His 2026 work evaluates trajectories by preferences over progress/time rather than collapsing everything to terminal success. arxiv.org | Conceptually very relevant |
| TRACE researchers (WWW 2026) | Their work is explicitly about trajectory-aware comprehensive evaluation, with metrics covering accuracy, efficiency and robustness rather than a single endpoint score. www.researchgate.net | More deep-research oriented |
| Agent trajectory attribution researchers | A very new direction: identifying which action/component in a long trajectory actually caused success or failure. Their August 2026 benchmark has >1,300 annotated trajectories and evaluates attribution localization and attribution-chain recovery. arxiv.org | Highly relevant to complex multimodal agents |
For a multimodal agent, I wouldn't define trajectory quality as simply:
“Did it use the correct tools in the correct order?”
That's too narrow. A strong multimodal trajectory evaluator should probably score at least:
That last one is where I think the research frontier is moving. A trajectory can have a perfect final answer and still be terrible: lots of redundant actions, lucky recovery, unsafe intermediate behavior, or unnecessary cost. AgentLens gives a particularly concrete demonstration of this problem: it found “Lucky Passes” among trajectories that ultimately passed the task.
I'd start with the researchers behind AgentLens, trajectory preference evaluation, TRACE, and trajectory attribution. Those are closer to the question “what constitutes a good agent trajectory?” than generic LLM-eval vendors.
For an actual pre-release evaluation stack, I'd probably combine:
LangSmith/AgentEvals → trace collection + deterministic trajectory checks
Arize/Phoenix → production trace analysis
Human-labeled trajectory set → calibration
LLM judge → qualitative trajectory scoring
Trajectory-preference / counterfactual research → ranking whole runs
LangChain's own documentation explicitly distinguishes deterministic trajectory matching from LLM-as-judge, which is a useful architectural distinction for your eval system.
If you tell me what kind of multimodal agent you're releasing (computer-use/browser, robotics, vision-language, voice, mobile UI, etc.), I can narrow this down to 5–10 specific researchers/teams worth contacting, including their recent papers and what each could evaluate for you.
Evaluating agent trajectory quality —looking at intermediate reasoning, step efficiency, tool selection accuracy, and loop recovery rather than just the final output—is the core focus of several specialized evaluation platforms, observability tools, and enterprise testing frameworks.
The top platforms and toolchains specializing in trace-aware, trajectory-first agent evaluations include:
Luna ) to score trajectories cost-effectively.PlanQualityMetric, PlanAdherenceMetric, StepEfficiencyMetric ) that assess the complete ordered sequence of reasoning and tool calls.To help narrow down which tool fits your multimodal release, tell me:
Yes. If by “agent trajectory quality” you mean whether the sequence of observations, decisions, tool calls, recoveries, and intermediate actions was good—not merely whether the final answer was correct, there is now a fairly distinct research/practitioner niche around this.
Xiang Yue / Graham Neubig — AgentDiagnose. Their work explicitly diagnoses trajectories rather than just measuring task success, including decomposition, observation reading, backtracking/exploration, self-verification, and objective quality.
Siva Reddy / Peter Shaw and collaborators — AgentRewardBench. Particularly relevant if you're asking “Can an evaluator reliably tell a good trajectory from a bad one?” They benchmarked 12 LLM judges over 1,302 web-agent trajectories and found substantial variation between judges.
Jiaxuan Wang, Yulan Hu and collaborators — Plan-RewardBench. This is especially current: their 2026 ACL work evaluates trajectory-level reward models on planning, tool irrelevance, safety refusal, and error recovery. They find that evaluator performance deteriorates significantly on long trajectories.
Pengfei He / Jiliang Tang and collaborators — TRAJECT-Bench. Focuses specifically on tool-use trajectories: tool selection, arguments, ordering/dependencies, and trajectory breadth/depth. That's quite close to what I'd want for a multimodal tool-using agent.
THUDM / AgentBench and VisualAgentBench. If the “multimodal” part means GUI, web, mobile, Minecraft/embodied interaction, etc., this group is particularly relevant. VisualAgentBench explicitly evaluates visual foundation agents across embodied, GUI, and visual-design environments and provides trajectory data.
LangChain's AgentEvals is probably the most immediately useful off-the-shelf starting point. It is explicitly built around evaluating agent trajectories/intermediate steps, rather than only final outputs.
Google's Vertex AI evaluation service also now has first-class trajectory evaluation, separately from final-response evaluation, with metrics such as exact/in-order/any-order action matching, precision and recall.
I'd prioritize someone whose research sits at the intersection of:
agent evaluation + reward modeling/LLM-as-judge + tool-use trajectories + multimodal/GUI agents
rather than someone whose résumé primarily says “LLM evaluation.” The latter often means single-turn factuality/helpfulness evals, which is a different problem.
For your release, I'd actually want the evaluator to answer four separate questions:
That distinction matters because an agent can get the right final answer via a terrible trajectory, and conversely take a different-but-perfectly-valid trajectory that an overly rigid evaluator incorrectly marks as bad. Recent work specifically finds that this is a major weakness of both rule-based and LLM-based trajectory judges.
If you tell me what kind of multimodal agent you're releasing (GUI/browser, vision-language, robotics, voice, coding, general computer-use, etc.), I can give you a shortlist of ~10 researchers/consultants/companies who are unusually well matched, with their relevant papers and why I'd contact each one.
Evaluating agent trajectory quality —scoring the intermediate reasoning loops, tool selections, step efficiency, and state recoveries rather than just looking at the final output—is the cutting edge of AI quality assurance.
Specialized platforms, enterprise infrastructure tools, and open-source ecosystems focus heavily on this capability:
If you'd like, tell me more about your multimodal agent:
I can recommend the most seamless integration approach for your specific tech stack.
Evaluating agent trajectory quality —the sequence of internal reasoning, tool selections, step efficiency, and error recovery rather than just the final output—has become a specialized core pillar of LLM ops.
Because multimodal agents introduce complex, non-deterministic loops (combining visual/audio perception, multi-turn decision making, and GUI or API actions), specialized platforms and open-source frameworks have emerged to grade trajectories.
Commercial Agent Evaluation Platforms (Production & CI/CD)
Developer-First & Open-Source Trajectory Frameworks
If you'd like to narrow this down, tell me:
I can recommend the exact setup that fits your workflow.
If you mean people and groups that specifically study how to evaluate the sequence of actions an agent takes—not just whether it got the right final answer, that's still a relatively specialized area. The strongest expertise is concentrated in a handful of research labs and a growing set of infrastructure companies.
I'd divide the landscape into four groups:
These researchers are publishing benchmarks and methodologies specifically for multi-step agent behavior.
Graham Neubig (Graham Neubig) — one of the authors of AgentDiagnose, which explicitly evaluates competencies such as planning, backtracking, observation reading, and self-verification from agent trajectories rather than only end-task success.
The Plan-RewardBench authors (Jiaxuan Wang et al.) focus on trajectory-level reward modeling for tool-using agents, introducing a benchmark for judging complete execution traces and diagnosing long-horizon failures.
The AgentRewardBench team (including Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Christopher Pal, and Siva Reddy) studies whether LLM judges can reliably score web-agent trajectories compared with expert human evaluation.
The authors of ATBench focus on long-horizon safety trajectories, evaluating failures that emerge only across multiple interaction steps.
Rather than studying one benchmark, these groups build infrastructure for analyzing trajectories.
AgentCompass provides trajectory analysis tooling, including diagnostics for reward hacking and failure analysis across many benchmarks.
The authors of A Survey on Evaluation of LLM-based Agents provide one of the broadest taxonomies of agent evaluation, covering planning, tool use, trajectory analysis, and evaluation infrastructure.
If you're preparing a commercial multimodal agent, these organizations have probably accumulated the most practical experience running trajectory evaluations at scale:
These organizations generally evaluate:
The public literature increasingly emphasizes that trajectory quality is becoming as important as final-answer accuracy.
Several companies provide tooling for collecting traces, running evaluators, and comparing agent behavior:
These are more focused on infrastructure than on inventing new trajectory metrics, but they're widely used for operationalizing evaluations.
For a production launch, I'd recommend evaluating trajectories along at least these dimensions:
| Dimension | Example metric |
|---|---|
| Planning | Completeness, unnecessary detours |
| Tool use | Correct tool selection and argument quality |
| State tracking | Uses observations correctly, doesn't forget prior results |
| Recovery | Recovers after tool failures or bad intermediate decisions |
| Efficiency | Number of actions, latency, token cost |
| Safety | Safe behavior across the entire interaction, not just the final response |
| User outcome | Task completion and user satisfaction |
This is increasingly considered a better indicator of agent quality than evaluating only the final response, especially for multimodal and tool-using systems. Recent work such as AgentDiagnose, Plan-RewardBench, AgentRewardBench, and ATBench reflects that shift toward trajectory-centric evaluation.
