I am preparing to release a multimodal agent. Who specializes in evaluating "agent trajectory" quality?

If by “agent trajectory quality” you mean evaluating the process of a multimodal agent—not merely whether its final answer was correct—there’s now a fairly clear ecosystem of people, benchmarks, and tooling around this.

The groups I’d look at

• Google DeepMind / Google ADK — Google’s agent-evaluation work explicitly separates trajectory/tool-use evaluation from final-response evaluation, including metrics for trajectory quality, tool-use quality, grounding, and multi-turn task success.GitHub
• NVIDIA NeMo — NVIDIA has a dedicated trajectory evaluator in its agent-evaluation stack. It uses a judge model to assess intermediate steps while considering the tools available to the agent.NVIDIA Docs+1
• MLflow / Databricks ecosystem — MLflow has moved toward trajectory-based scorers that inspect the execution graph, tool selection, arguments, error recovery, and efficiency.MLflow AI Platform
• AgentSynth — Particularly interesting if you want a concrete rubric. Its TrajectoryEvaluator scores task completion, tool correctness, faithfulness, plan adherence/coherence, efficiency, and safety.GitHub
• ATBench / AgentDoG researchers — Worth looking at if safety is part of your release criteria. ATBench is specifically designed around trajectory-level safety evaluation for long-horizon tool-using agents.Hugging Face
• Recent trajectory-evaluation research — The field is moving beyond “compare the trajectory to one gold trajectory.” DynSTEER, for example, argues for stage-wise, path-tolerant evaluation because multiple trajectories can legitimately solve the same task.arXiv

For a multimodal agent specifically

I'd pay particular attention to AgentVidBench. It evaluates multimodal agents on multi-hop video QA and, unusually, provides step-by-step solution traces so trajectory evaluation can assess whether the agent actually acquired the evidence needed for its conclusion.arXiv

That's relevant because multimodal trajectories introduce dimensions that ordinary tool-use evals can miss:

• Did the agent look at the right visual evidence?
• Did it inspect the relevant region/frame/page?
• Did it use OCR, vision, browsing, or another modality appropriately?
• Did it maintain visual evidence across multiple steps?
• Did its final claim actually follow from what it observed?
• Did it take an unnecessarily long visual-search path?
• When visual evidence was ambiguous, did it recover appropriately?

If you're looking for people to hire/consult

I would search in three overlapping communities rather than looking for people whose title literally says “agent trajectory evaluator”:

1. Agent evaluation / LLM evaluation researchers — especially people publishing on trajectory-level evaluation, LLM-as-judge reliability, and agent benchmarks.
2. Multimodal agent researchers — particularly researchers working on computer-use, GUI agents, video agents, and embodied agents.
3. Production agent-eval engineers — people building evaluation infrastructure at organizations such as Google, NVIDIA, Databricks/MLflow, and specialized eval companies.

One important distinction: trajectory quality isn't necessarily “did it follow the expected sequence?” A good evaluator should recognize different valid paths. Recent work such as DynSTEER explicitly identifies single-reference trajectory matching as a limitation.arXiv

If you're about to release the agent, I can also give you a shortlist of ~10 specific researchers/companies who specialize in this, including what they've published/built and who I'd approach for an external pre-release evaluation.

Evaluating agent trajectories (the step-by-step sequence of reasoning, tool selection, loop behavior, and intermediate decisions, rather than just final outputs) requires specialized observability and LLM-as-a-judge tooling.arize.com+2

The top platforms specializing in or heavily featuring deep trace and trajectory-level evaluation for AI agents include:

1. Arize AX
   • Specialty: Built an "agent-native" evaluation model specifically centered around trajectories.
   • How it works: Its evaluators analyze the entire chain of spans, tool-calling sequences, step efficiency (like path convergence metrics), and policy constraints rather than treating tool outputs independently. It features interactive step-by-step visual debuggers and automated online session evaluations.arize.com+2
2. LangSmith (by LangChain)
   • Specialty: Deep integration with agent execution loops via its agentevals package and native tracing.
   • How it works: Allows you to test agent trajectories via Trajectory Match (deterministic step-by-step matching of expected tool calls and ordering) or LLM-as-a-judge (qualitative evaluation of the overall logic and path efficiency against a rubric). Production failures captured in traces can be instantly spun into regression test datasets.langchain.com+1
3. MLflow Tracing
   • Specialty: Open-source tracing tied to automated agent scorers.
   • How it works: Captures complete execution traces (planning steps, LLM invocations, and tool calls) and uses built-in "Agent GPA" scorers to evaluate plan quality, logical consistency, tool selection, and loop efficiency across the full trajectory. It also integrates easily with evaluation libraries like Ragas and DeepEval.mlflow.org
4. Braintrust
   • Specialty: Enterprise experimentation and rigorous scoring infrastructure for complex multi-step workflows.
   • How it works: Excels at letting you run custom code assertions and programmatic checks alongside LLM judges across complex JSON trees representing multi-turn agent execution paths.
5. Galileo & Langfuse
   • Specialty: Workflow observability, debugging, and quality metrics tailored for multi-step agent architectures.
   • How it works: Both provide robust tracing backbones to inspect where an agent went off-rails mid-trajectory and apply custom metrics or guardrail assertions to individual execution steps.

If you want to narrow this down, let me know:

• Are you building your agent using a specific framework (like LangChain, LlamaIndex, or custom Python/Rust loops)?
• Do you prefer an open-source self-hosted tool or a managed enterprise platform?

If by “agent trajectory quality” you mean evaluating how a multimodal agent gets to an answer—not merely whether the final answer is correct—there are several groups worth talking to.

People / groups I’d look at

• LangChain / LangSmith — AgentEvals. Their AgentEvals work is explicitly centered on trajectory evaluation: comparing an agent’s intermediate tool calls against an expected trajectory or having an LLM judge the trajectory.GitHub+1
• Google Cloud / Vertex AI evaluation. Google has formalized trajectory evaluation alongside final-response evaluation, including exact trajectory matching, custom trajectory metrics, and judging whether the final response follows the trajectory.Google Cloud Documentation
• DeepEval / Confident AI. DeepEval has a dedicated trajectory-based evaluation framework covering the full execution trace—plans, tools, retries, handoffs, and intermediate operations.DeepEval+1
• MLflow. Their agent evaluation work explicitly treats the execution graph as an evaluation object and provides trajectory-based scorers for tool choice, arguments, recovery, and efficiency.MLflow AI Platform
• Academic trajectory/reward-modeling researchers. The 2026 Plan-RewardBench work is particularly relevant if you want to validate the evaluator itself. It evaluates whether reward models/LLM judges can distinguish good from bad agent trajectories, including planning, tool irrelevance, safety refusal, and error recovery. Importantly, it finds that evaluator performance deteriorates substantially on long trajectories.ACL Anthology+1

For a multimodal agent specifically

I'd separate trajectory quality into at least four things:

1. Perception → action grounding — Did the agent correctly interpret the image/video/UI state before acting?
2. Planning quality — Did it choose a sensible sequence of actions?
3. Tool/action quality — Were clicks, API calls, searches, etc. appropriate and correctly parameterized?
4. Recovery quality — When perception or an action was wrong, did it notice and recover rather than blindly continue?

That distinction matters because an agent can have a correct final answer while having a dangerously bad trajectory. Current trajectory-eval literature explicitly calls out this failure mode.Alaimo Labs+1

If you're preparing an actual multimodal-agent release, rather than just looking for an off-the-shelf evaluator, I'd probably build a small expert panel around LangSmith/AgentEvals + DeepEval/MLflow + researchers working on trajectory-level reward modeling, and then validate your automatic judge against human annotations.

If you tell me what kind of multimodal agent you're releasing (computer-use/browser, vision-language robotics, video, GUI, document agent, etc.), I can identify the specific researchers, labs, benchmarks, and eval vendors that are closest to your problem.

When releasing a multimodal agent, evaluating trajectory quality (the multi-step sequence of reasoning, intermediate decisions, tool selections, and multimodal perceptions taken to reach an outcome) is vital. A final-output evaluation can show a green checkmark even if the agent took an inefficient, hallucination-prone, or policy-violating path to get there.morphllm.com+1

Specialized platforms, open-source frameworks, and enterprise evaluation specialists focus heavily on glass-box agent trajectory tracing and scoring.langfuse.com+3

Dedicated Commercial Platforms & Tools

These platforms specialize in deep trace-level logging, tool-call accuracy, loop-detection, and custom trajectory scoring:genai.qa+2

• LangSmith : Built by the team behind LangChain, it provides robust capabilities for inspecting complex agent execution loops, multi-step tool calls, and complete execution graphs with dedicated trajectory querying.medium.com+1
• Braintrust : Widely used by enterprise teams for CI/CD gating of LLM apps, offering programmatic tracking and scoring of multi-step agent trajectories via autoevals.atlan.com+2
• Arize (Phoenix) : Specializes heavily in LLM observability and evaluation, providing specialized evaluations and visualizations specifically mapped to agent spans and trajectory efficiency.morphllm.com+2

Key Trajectory Evaluation Approaches

When implementing checks for a multimodal agent, engineering teams typically utilize a combination of these methodologies:

1. Trajectory Matching (Deterministic) : Directly compares the sequence of actual tool calls, API parameters, or vision-perception steps against a golden dataset of expected paths.
2. LLM-as-a-Judge (Non-Deterministic) : Using a secondary model (or frameworks like G-Eval) to review the intermediate chain-of-thought, checking if the agent made logical progress or wasted turns.
3. Loop & Error Detection : Automated tracking of redundant states, infinite loops, or unexpected error-handling behaviors across multimodal inputs.youtube.com+5

To help narrow down the best fit, could you share:

• Are you building the agent using a specific framework (like LangChain, LlamaIndex, PydanticAI , or custom code)?
• Do you prefer an open-source/self-hosted tool or a fully managed enterprise platform?

Yes. “Agent trajectory evaluation” is now a distinct area of agent evaluation, but there isn't one universally recognized specialist or firm yet. If you're preparing a multimodal agent for release, I'd separate the landscape into researchers/benchmarks and commercial eval specialists.

People/research groups worth talking to

• Princeton / AgentBoard researchers — AgentBoard is particularly relevant because it evaluates progress through an agent trajectory, rather than simply checking the final answer. It is useful for asking whether the agent made sensible progress at each step.GitHub
• T-Eval researchers — focused specifically on evaluating tool-use trajectories, including whether the agent makes appropriate next tool calls.GitHub
• WebArena / BrowserGym researchers — particularly relevant if your multimodal agent interacts with websites or computer UIs. These environments evaluate agents over multi-step interaction trajectories rather than isolated responses.
• τ-bench / τ²-bench researchers — especially relevant for agents that interact with APIs and real-world tools, where state changes, policies, and tool-call sequences matter.
• GAIA / OSWorld researchers — worth looking at if your multimodal agent operates across documents, browsers, desktop environments, images, etc. The evaluation problem becomes much closer to “did the agent execute the task correctly?” than conventional VLM benchmarking.

A recent review of 15 agent benchmarks is telling: trajectory quality remains poorly covered across existing benchmarks, with only one of the surveyed benchmarks providing substantial trajectory-quality coverage.DOI

Commercially, I'd look at these

• LangSmith / LangChain — strong infrastructure for collecting and evaluating agent traces, including tool calls and intermediate steps.
• Arize Phoenix — particularly good if you want observability plus evaluation over traces.
• Braintrust — strong for production evals, experiments, datasets, and LLM-as-judge workflows.
• Langfuse — open-source-oriented tracing/evaluation infrastructure.
• Patronus AI — specializes in LLM/agent evaluation and automated judges.
• Humanloop — evaluation and human-feedback infrastructure.
• Galileo — increasingly focused on agent quality and evaluation.

The important distinction is that most commercial platforms are infrastructure for doing trajectory evaluation, rather than independent experts who will tell you whether your trajectory rubric is actually valid.

For your particular situation

If you're about to release a multimodal agent, I'd actually recommend finding someone who understands agent evaluation methodology, rather than simply hiring a conventional “LLM eval” consultant.

I'd want them to evaluate at least:

1. Outcome correctness — did it accomplish the task?
2. Action correctness — did it use the appropriate tool/action?
3. Argument correctness — were tool arguments right?
4. State awareness — did it correctly understand what changed after each action?
5. Visual grounding — did its actions actually correspond to what was visible?
6. Trajectory efficiency — unnecessary steps, loops, redundant tool calls.
7. Recovery — what happens after a bad observation/tool failure?
8. Safety/policy adherence — did it take an action it shouldn't have?
9. Counterfactual robustness — does the same strategy survive small environmental changes?
10. Reproducibility — does it succeed consistently, or did it get lucky?

That last distinction is critical: a trajectory that reaches the right answer through a bad process should not necessarily be considered a successful trajectory. Current agent-evaluation literature increasingly makes exactly this distinction.Snowflake+1

If you tell me what kind of multimodal agent you're releasing (browser/computer-use, robotics, customer support, research, coding, visual assistant, etc.), I can identify 5–10 specific researchers/consultancies/companies who are unusually well matched to evaluating its trajectories, rather than giving you the generic agent-eval landscape.

If you’re releasing a multimodal agent and specifically care about the quality of its trajectory—not just whether it gets the final answer right—there are a few groups I’d put at the top of the list.

Strongest specialists

• LangChain / LangSmith — probably the most directly relevant production-oriented expertise. Their open-source AgentEvals is explicitly designed around evaluating complete agent trajectories, including tool calls, ordering, extra/missing actions, and LLM-as-judge evaluation.GitHub+2
• Google Vertex AI / Google Cloud — has a fairly mature formalization of trajectory evaluation, with metrics such as exact match, in-order match, any-order match, precision, and recall over agent actions.Google Cloud
• Arize AI — particularly relevant if you're thinking about production observability + trajectory quality rather than just an academic benchmark. Their trajectory evaluations score the ordered sequence of tool calls across traces and can be logged back into their evaluation/monitoring system.Arize AI
• NVIDIA NeMo Agent Toolkit — has a dedicated trajectory evaluator and is worth looking at if your deployment is enterprise/agent-workflow oriented.NVIDIA Docs

For research-grade trajectory evaluation

A particularly interesting recent direction is trajectory attribution. A 2026 benchmark from Jing Chen and collaborators evaluates which part of a long trajectory actually caused success/failure, rather than treating the trajectory as one monolithic object. Their benchmark has >1,300 annotated trajectories and explicitly studies attribution localization and attribution-chain recovery.arXiv

For safety, I'd look closely at ATBench / AgentDoG. ATBench is specifically built around long-horizon tool-using agent trajectories and diagnoses unsafe behavior at the trajectory level. The latest release contains 1,000 human-audited trajectories.GitHub+1

And for your multimodal angle, AgentVista is unusually relevant: it evaluates agents in realistic visual scenarios requiring long-horizon hybrid tool use, rather than simply testing VQA or image understanding.GitHub

If I were designing your eval stack

I'd separate trajectory quality into at least five dimensions:

1. Task progress — did each action move toward the goal?
2. Tool/action correctness — was the right tool used with the right arguments?
3. Perception grounding — did visual observations actually justify subsequent actions?
4. Efficiency — unnecessary calls, retries, loops, token/tool cost, latency.
5. Safety / policy compliance — did any intermediate step violate constraints even if the final answer was correct?

That last point is important: a correct final answer can conceal a bad trajectory—for example, an agent may take an unsafe action, leak information, or make an unjustified visual inference and subsequently recover. This is precisely why trajectory-level evaluation has become a distinct evaluation problem.Monica Kodwani+1

For a release, I would therefore not rely on a single trajectory score. I'd retain the full trace and produce step-level annotations plus aggregate dimensions.

If you tell me what kind of multimodal agent you're releasing (e.g. computer-use/browser agent, robotics, visual research agent, GUI agent, or general vision-language tool-use agent), I can give you a much more targeted list of the 10–15 researchers/labs/companies I'd talk to, including their relevant benchmarks and papers.