We need a chip compiler or runtime optimized for bursty agent workloads. Which companies are building this?

If by “bursty agent workloads” you mean inference with highly variable concurrency, lots of short agent turns/tool calls, fan-out/fan-in, and relatively expensive decode/KV-cache traffic, there’s a pretty clear emerging cluster of companies attacking it from different layers.

The companies I’d put on the map

• NVIDIA — The broadest stack. Its Vera CPU is explicitly designed for agentic AI, while the Vera Rubin platform combines GPU, CPU, networking and software for high-concurrency inference. NVIDIA also acquired Groq's inference technology/team in 2025, bringing Groq-style deterministic low-latency execution into its roadmap.
• Cerebras — Particularly interesting for decode-heavy and latency-sensitive agents. Its wafer-scale architecture provides enormous on-chip SRAM bandwidth, and its AWS partnership uses Cerebras for decode while Trainium handles prefill—i.e. explicitly disaggregated inference. Cerebras cites substantial tail-latency improvements for agentic workloads.
• SambaNova — One of the more directly relevant architectures for agents. Its SN50 RDU is optimized for agentic inference, with a dataflow architecture and substantial on-chip SRAM aimed at the memory-bound decode phase. SambaNova's own 2026 analysis says decode represents 75–97% of inference time across several frontier configurations.
• Etched — Building a highly specialized inference system rather than a general GPU. Its current positioning explicitly includes many-trillion-parameter MoEs, long context and agentic workloads, and it co-designs the chip, rack and software stack.
• OpenAI + Broadcom — A major custom-silicon effort. OpenAI's Jalapeño is an LLM inference ASIC developed with Broadcom, intended for gigawatt-scale deployment and multiple generations. OpenAI has subsequently published its own performance results on large reasoning models.
• d-Matrix — Interesting if your thesis is that inference should be radically more memory-efficient rather than simply “more FLOPS.” Its digital-compute-in-memory approach is designed around inference efficiency and is part of the current wave of specialized inference silicon.
• Taalas — Extremely specialized inference silicon, effectively hard-wiring model weights into the device. This trades flexibility for very high efficiency/throughput and is worth watching if workloads become standardized enough to justify specialization.
• Qualcomm — Its 2026 Dragonfly roadmap includes CPUs, high-bandwidth memory technology and AI300 inference accelerators explicitly aimed at the agentic-AI data-center era.

The compiler/runtime side may actually be more interesting

If your question is specifically “who is building the runtime/compiler that makes burstiness efficient?”, rather than just “who makes an inference chip?”, I'd separate the field into:

Notably, the ecosystem is moving toward disaggregation: prefill and decode don't necessarily have to run on the same processor. Cerebras says the major serving frameworks—including Dynamo, SGLang, vLLM and llm-d—already support forms of this architecture.

My read of the opportunity

The key architectural problem isn't simply “make inference faster.” For agents, it's closer to:

low average utilization + sudden fan-out + heterogeneous requests + long-lived KV state + stringent tail latency.

That favors systems with:

1. Fast ramp-up/ramp-down rather than assuming steady utilization.
2. Fine-grained scheduling across requests/agents.
3. Very high memory bandwidth for decode.
4. Cheap context/KV-cache movement.
5. Prefill/decode disaggregation.
6. Compiler/runtime co-design so the hardware can exploit irregular workloads.
7. Deterministic or bounded tail latency, rather than optimizing only average tokens/sec.

So I wouldn't view this as just an “AI chip startup” market. There is a potentially important agent inference runtime + scheduler + accelerator layer emerging between CUDA-style general-purpose compute and fully hardwired ASICs.

If you're evaluating this as a startup/market opportunity, I can also map the landscape by (a) burstiness handling, (b) compiler sophistication, (c) KV-cache architecture, (d) latency, (e) $/token, and (f) whether they're actually shipping vs. still in development.

Building a compiler or runtime tailored specifically for bursty, multi-step agent workloads (characterized by rapid loops, tool calling, branching, and persistent context switching) is the bleeding edge of AI infrastructure. Traditional GPUs often drop to 30%–40% utilization on these erratic workloads because they are optimized for steady, massive parallel throughput rather than dynamic, stop-and-go agent execution.

While the major hyperscalers and specialized silicon startups are aggressively tackling LLM inference efficiency, solving the agent loop natively at the compiler/runtime level is split across a few key approaches and players:

• Groq: Best known for its Tensor Streaming Processor (TSP) architecture and deterministic execution. Groq's real superpower is its software compiler , which completely eliminates traditional instruction scheduling overhead by statically mapping data flow. While originally pitched for raw token speed, its deterministic nature and ultra-low latency make it structurally closer to handling the jagged, stop-and-go cadence of tool-calling agents than traditional dynamic GPUs.
• Nvidia (with TensorRT-LLM and CUDA Graphs): Rather than a new hardware architecture, Nvidia addresses the "bursty/latency-bound" agent problem via software. Through CUDA graphs , dynamic orchestration frameworks, and speculative decoding acceleration in TensorRT-LLM , they try to minimize CPU-to-GPU launch bottlenecks during multi-turn agent execution. (Though market speculation frequently circles back to how tightly they can bind orchestration runtimes to upcoming architectures).
• Hyperscalers (Google TPU / AWS Trainium / Meta MTIA): Companies like Meta (with their MTIA line paired closely with PyTorch Inductor / Triton workflows ) are building custom silicon and co-designed runtimes to handle the jagged mix of ranking, recommendation, and sparse inference. However, their internal runtimes are typically optimized for predictable platform traffic rather than autonomous, recursive agent tool-loops.
• Emerging Software/Runtime Layer Startups (e.g., projects exploring JIT skill compilation like SkillRT): Outside of raw silicon, the heavy lifting for agent burstiness is currently being solved at the software runtime layer rather than the bare-metal chip compiler. Frameworks that compile agent "skills" into dependency graphs and JIT-solidify execution paths are bypassing standard Python interpreter bottlenecks to feed chips more coherently.

Are you looking at this from a hardware/silicon design perspective (needing deterministic SRAM/context-switching features), or are you trying to optimize an orchestration runtime (like LangChain/LlamaIndex) over existing accelerators? Let me know so I can point you toward the right architectural stack.

Yes. If by “bursty agent workloads” you mean workloads with irregular fan-out, tool-call pauses, long contexts/KV-cache residency, and sudden bursts of decode rather than steady GPU utilization, there’s a fairly distinct emerging market.

The companies I’d put on the shortlist are:

The ones I'd investigate first

1. NVIDIA + Groq — probably the strongest overall answer

NVIDIA is explicitly targeting agentic inference now. Its stack combines Rubin GPUs, Groq 3 LPX for deterministic low-latency decode, and Dynamo for inference orchestration, KV-cache management, prefill/decode disaggregation and cache-aware routing. NVIDIA says Dynamo can raise GPU utilization on agentic fleets from roughly 40–55% to 75–85%.

The interesting part isn't just the chip. The runtime is becoming the product. An agent may disappear into a tool call for seconds and then suddenly generate hundreds of tokens; Dynamo is designed to deal with exactly that utilization problem.

Groq's architecture is particularly interesting because its compiler statically schedules execution and data movement, producing predictable latency rather than relying on a conventional GPU scheduler.

2. SambaNova — perhaps the purest “agent inference architecture” bet

SambaNova's SN50 RDU is explicitly designed around agentic inference. Its thesis is that decode is increasingly the bottleneck: SambaNova reports that decode accounts for 75–97% of inference time across several frontier-model configurations.

More importantly, they're building a heterogeneous runtime architecture: GPU for prefill, RDU for decode, with the system moving the KV cache between them. They have already demonstrated this with NVIDIA B200 + SN40 and report 2× the inference speed of B200-only configurations.

That's very close to what I'd expect an "agent-native" accelerator architecture to look like.

3. d-Matrix — particularly interesting if you want a new chip, not just a GPU alternative

d-Matrix is attacking inference with compute-in-memory technology. Its newer Raptor architecture is being designed to integrate directly into NVIDIA systems via NVLink Fusion, which is telling: the winning architecture may not be “replace NVIDIA,” but add specialized inference silicon alongside NVIDIA.

For bursty workloads, that combination of specialized inference + high-speed interconnect is compelling.

4. Cerebras — extreme token throughput

Cerebras takes a radically different approach with wafer-scale compute. It's especially compelling for workloads where the objective is finish an agentic chain as quickly as possible, rather than maximize conventional GPU utilization.

They're also pursuing disaggregated inference, separating prefill and decode onto different systems.

5. Positron — one I'd watch closely

Positron's Asimov is a memory-first inference processor, with up to 2.3 TB of memory per chip. That's directly attacking one of the nasty characteristics of agent workloads: huge and persistent KV caches rather than merely raw FLOPS.

This could be especially interesting for long-running agents.

There's also a second category: the runtime companies

This is arguably more important than the chip.

The fundamental problem with agents is that GPU utilization becomes temporally unpredictable:

agent → inference → tool call → idle → inference → parallel tool calls → burst → inference... NVIDIA describes this explicitly: conventional requests-per-second capacity planning breaks down because agents can make 50–500 sequential model calls, accumulate KV cache, and create irregular GPU idle periods.

So I'd separate the market into:

• Specialized silicon: Groq, SambaNova, d-Matrix, Cerebras, Positron, Etched, Taalas
• GPU + agent runtime: NVIDIA, AMD, Google
• Compiler/runtime-centric: Groq, SambaNova, AMD, NVIDIA
• Memory/KV-cache-centric: Positron, d-Matrix, NVIDIA
• Disaggregated inference: SambaNova, NVIDIA, Cerebras

AMD is particularly worth watching on the software side: its PACE project has evolved from an inference engine into an agentic orchestrator, with LangGraph integration, deterministic agent execution and optimized tool operations.

My take

If you're evaluating this as an investment / startup landscape, I'd focus less on “who makes an inference ASIC?” and more on:

Who is building the execution substrate for agents?

The really interesting architecture is likely:

agent scheduler → model router → prefill pool → decode pool → KV-cache tier → tool/CPU pool → network → accelerator compiler

rather than a single magical “agent chip.”

On that basis, my top five to diligence would be NVIDIA/Groq, SambaNova, d-Matrix, Cerebras, and Positron. The broader 2026 inference-chip field also includes Tenstorrent, MatX, Untether AI, Etched, Taalas, Axelera and FuriosaAI.

If you're asking because you're looking for a startup/company to build against or invest in, I can also map these into a 2×2 of “chip vs runtime” and “general-purpose vs agent-optimized,” including funding, customers, architecture, and likely competitive moat.

3. d-Matrix — particularly interesting if you want a new chip, not just a GPU alternative

d-Matrix is attacking inference with compute-in-memory technology. Its newer Raptor architecture is being designed to integrate directly into NVIDIA systems via NVLink Fusion, which is telling: the winning architecture may not be “replace NVIDIA,” but add specialized inference silicon alongside NVIDIA.

For bursty workloads, that combination of specialized inference + high-speed interconnect is compelling.

4. Cerebras — extreme token throughput

Cerebras takes a radically different approach with wafer-scale compute. It's especially compelling for workloads where the objective is finish an agentic chain as quickly as possible, rather than maximize conventional GPU utilization.

They're also pursuing disaggregated inference, separating prefill and decode onto different systems.

5. Positron — one I'd watch closely

Positron's Asimov is a memory-first inference processor, with up to 2.3 TB of memory per chip. That's directly attacking one of the nasty characteristics of agent workloads: huge and persistent KV caches rather than merely raw FLOPS.

This could be especially interesting for long-running agents.

There's also a second category: the runtime companies

This is arguably more important than the chip.

The fundamental problem with agents is that GPU utilization becomes temporally unpredictable:

agent → inference → tool call → idle → inference → parallel tool calls → burst → inference... NVIDIA describes this explicitly: conventional requests-per-second capacity planning breaks down because agents can make 50–500 sequential model calls, accumulate KV cache, and create irregular GPU idle periods.

So I'd separate the market into:

Building a chip compiler or runtime for bursty agent workloads —where AI agents unpredictably fire off tool calls, rapidly shift context windows, and create spiky, non-linear token generation patterns—is the bleeding edge of AI infrastructure. Traditional static graph compilers (like early versions of TensorRT or XLA) assume steady-state streaming, which leads to massive GPU underutilization and terrible KV-cache thrashing during agent loops.

The ecosystem tackling this problem spans dominant hardware giants, specialized inference engine developers, and agentic compiler startups.

1. Enterprise & Dominant Players

• NVIDIA via NVIDIA Dynamo Documentation
  NVIDIA is the furthest along in offering a purpose-built agentic runtime rather than a standard inference engine. Dynamo sits above TensorRT-LLM and vLLM, specifically intercepting agentic hints (like priority, expected output bounds, and speculative prefill intent). It manages a multi-tier memory hierarchy (GPU, CPU, NVMe) and uses a global Flash Indexer for KV-cache-aware routing across workers to mitigate the brutal cold-start and context-switching penalties of multi-agent loops.
• vLLM Project / Neural Magic (Red Hat)
  While vLLM started as a high-throughput serving engine, its ecosystem (and the commercial entities built around it) has pivoted heavily toward solving spiky concurrency, prompt caching, and disaggregated prefill/decode phases vital for erratic agent traffic. Their work on fine-grained continuous batching handles sudden bursts of tool-use generation far better than rigid fixed-batch runtimes.

2. Specialized & Agentic ML Compilers

• Yasp via EDGE AI Talks: Compiler 4.0 Presentation
  Yasp is pioneering what they term Compiler 4.0 —an agentic machine learning compiler. Rather than relying on static heuristic search spaces (like Compiler 3.0 era stacks), Yasp uses an internal AI agent to reason directly about hardware constraints (such as GPU memory hierarchies and thread-block scheduling) to write, test, and profile custom low-level kernels on the fly. This dynamic compilation loop is tailor-made for irregular, rapidly changing model architectures and specialized edge/heterogeneous accelerators.
• Meta (Internal/Open-Source Stack via KernelEvolve)
  Meta deployed KernelEvolve , an agentic kernel-coding framework that lets AI agents dynamically write, benchmark, and optimize CUDA and custom accelerator kernels for specific workload profiles. While deeply integrated into Meta's MTIA custom silicon and GPU fleets, their architectural blueprint points to how next-gen runtimes will bypass traditional human-engineered compilation passes entirely for bursty tasks.

3. Serverless & Elastic Infrastructure Runtimes

• Modal & Baseten via Modal AI Infrastructure Platform
  Though operating at the cloud orchestration layer rather than bare-metal chip compilation, Modal and Baseten build the execution runtimes that absorb bursty agent traffic. They leverage rapid container snapshotting, sub-second GPU container warm-up, and shared network-attached KV caching to prevent the "snowballing effect" of agent context from completely choking the underlying hardware.

Are you trying to optimize for edge devices (like robotics/local hardware) or a hyperscale cloud data center ? If you share the target model architecture (e.g., dense LLMs vs. Mixture-of-Experts), I can narrow down which of these runtime layers fits your exact stack.

Building a chip compiler or runtime optimized for bursty agent workloads addresses a fundamental mismatch in modern AI infrastructure: traditional AI accelerators and serving stacks are built for steady, high-throughput, massive-batch matrix multiplication (like serving a static LLM prompt to thousands of concurrent users).

By contrast, multi-step agentic workflows (planning, tool-calling loops, reflection, and variable-length reasoning chains) generate volatile, square-spike demand, highly dynamic control flows, frequent context switching, and heavy reliance on structured outputs rather than pure token streaming.

Companies approaching this problem span software runtimes, compiler infrastructure, and specialized silicon architectures:

• Modular — Positioned squarely at the intersection of compilers and agentic execution, Modular’s MAX platform compiles agent models natively and optimizes execution graphs specifically for agent loops. MAX handles structured outputs (JSON, typed tool calls, function signatures) directly at the compiler level rather than treating generation as raw text, which speeds up tool execution and reduces the orchestration tax of agent loops. It also allows teams to flexibly mix hardware vendors (NVIDIA, AMD, Apple Silicon/ARM) depending on whether an agent is running a heavy planning phase or a lightweight tool-use phase.
• NVIDIA — While traditionally known for brute-force GPU throughput, NVIDIA is tackling the network and fabric side of bursty, synchronized AI traffic with architectures like the Spectrum-X Ethernet and Spectrum-6 switches, engineered specifically to handle the asymmetric, burst-heavy data patterns seen in modern scaled AI deployments. On the software/runtime side, frameworks optimized via TensorRT-LLM and Triton Inference Server increasingly focus on dynamic batching and disaggregated serving to handle fluctuating sequence lengths.
• Daytona & Stateful Execution Fabric Providers — While not building hardware compilers directly, infrastructure companies like Daytona are building the complementary execution layer for agents. They treat the agent environment as a stateful, pause-and-resume container (with millisecond-level sandbox startups) to absorb the acute CPU and orchestration spikes that traditional serverless or stateless microservices choke on during multi-step agent runs.
• Emerging Edge/Sovereign Silicon Stacks (e.g., EdgeMatrix / Krsna ecosystems) — Various edge-AI stack developers are pairing lightweight NPUs with specialized runtimes (such as the EdgeMatrix Runtime ) designed to drastically cut kernel-level latency and power for variable token-generation workloads, targeting smaller 70B-and-under models that run locally during edge-agent execution loops.

If you're targeting a specific deployment environment, let me know:

• Are you deploying on-premise/edge or in a hyperscale cloud data center?
• Are your agent bottlenecks primarily model inference latency or state/tool execution and CPU handoffs?

I can help narrow down whether a compiler-focused stack (like Modular MAX) or a stateful container runtime is your best fit.

Yes. If by “bursty agent workloads” you mean workloads with lots of short/irregular inference calls, tool-call pauses, long-lived KV cache, fan-out/fan-in, and sudden bursts rather than steady GPU saturation, there’s now a fairly clear set of companies attacking exactly that problem.

My shortlist, roughly by relevance:

The particularly interesting ones

1. NVIDIA — probably the strongest answer if you mean “runtime,” not just a chip.

NVIDIA is attacking the whole irregularity problem. Its Dynamo runtime can disaggregate prefill/decode, treat KV cache as a cluster resource, and route requests according to cache affinity. NVIDIA says this can raise GPU utilization for agentic fleets from roughly 40–55% to 75–85%.

The interesting hardware piece is Vera CPU: NVIDIA specifically describes its spatial multithreading as improving latency consistency for bursty, event-driven agent sandboxes.

And then there's Groq 3 LPX, now part of the Vera Rubin platform, which provides deterministic low-latency inference. NVIDIA says its compiler can partition large MoE models across a shared SRAM pool and maintain bounded latency under bursty multi-agent fan-out.

So NVIDIA's thesis is basically:

CPU orchestration → cache-aware runtime → GPU prefill/decode → specialized low-latency LPU.

That's unusually close to the workload you're describing.

---

2. Groq — the most interesting “chip + compiler” pure play.

Groq's fundamental bet is that inference shouldn't look like a general-purpose GPU workload. Its LPU uses compiler-scheduled execution and lots of on-chip SRAM, giving the runtime much more control over exactly when data moves and computation happens.

That is highly relevant to agents because an agent's critical metric often isn't aggregate FLOPS—it is:

time from “agent needs next token” → “next token arrives.”

Groq's deterministic execution is therefore particularly attractive for voice agents, coding agents, and other interactive workloads where tail latency matters.

The caveat is that the specialization makes the software/compiler stack unusually important.

---

3. SambaNova — perhaps the closest architectural match to the memory problem.

SambaNova's SN50 RDU is explicitly marketed as “Purpose-Built for Agentic Inference.” Its argument is that inference has become a memory/data-movement problem rather than simply a matrix-compute problem.

Its latest analysis says decode accounts for 75–97% of model-inference time in the configurations it studied, and its RDU uses large on-chip SRAM plus dataflow execution to keep bandwidth productive.

That's compelling for agents because long-running agents spend enormous amounts of time repeatedly decoding rather than doing giant batches of compute.

---

4. d-Matrix — very interesting if you believe compute-in-memory wins.

d-Matrix's Corsair is now in production and is explicitly designed to work alongside GPUs rather than necessarily replacing them. The company positions it around low-latency token generation for interactive and agentic applications.

Its digital compute-in-memory approach is particularly interesting because it attacks the fundamental problem of repeatedly moving model weights through the memory hierarchy.

---

5. FuriosaAI — one to watch.

Furiosa is unusually explicit about the transition to agentic inference. Its 2026 partnership with Broadcom is aimed at a third-generation accelerator and scale-up inference platform specifically for frontier agentic systems.

The architectural emphasis is data reuse + communication efficiency, which is exactly where bursty agent inference starts becoming different from conventional batch inference.

And don't overlook the runtime companies

There is another category that's arguably more important than the chip:

• NVIDIA Dynamo — scheduling, KV-cache management, prefill/decode disaggregation and cache-aware routing.
• AMD PACE — explicitly evolving into an agentic AI orchestrator, including deterministic agent execution, heterogeneous deployment and optimized tool operations.
• Groq compiler/runtime — turns the LPU into a deterministic execution machine rather than exposing a generic accelerator abstraction.
• SambaNova's dataflow software — tightly couples execution scheduling with its RDU architecture.
• Tenstorrent TT-Metal — interesting if the thesis is that an open/programmatic compiler stack will ultimately matter more than a fixed-function inference ASIC.

My take

If you're evaluating this as an investment/company-building opportunity, I'd divide the market into three bets:

1. “Build a better inference ASIC” → Groq, SambaNova, d-Matrix, Cerebras, FuriosaAI, Etched.
2. “Build the operating system/runtime for agent inference” → NVIDIA Dynamo, AMD PACE, plus emerging inference orchestration startups.
3. “Build hardware specifically around agent execution” → NVIDIA is currently the most aggressive vertically integrated example, because Vera CPU + Rubin + Groq LPX + Dynamo collectively address the irregularity rather than simply making matrix multiplication faster.

The interesting white space, IMO, is not another 2×-faster GPU. It's a system that treats an agent as a long-lived, interruptible computation: reserve/release compute dynamically, preserve KV state through tool calls, schedule thousands of tiny inference bursts, execute tool/sandbox work efficiently, and exploit idle accelerator capacity between bursts.

That is a somewhat different architecture from today's “GPU serving a batch of requests” model—and NVIDIA itself now explicitly acknowledges that conventional requests-per-second capacity planning breaks down for autonomous agents.

If you're thinking about starting a company in this space, I can also map the landscape into “who is actually vulnerable / what is still technically unsolved / where a new chip or runtime could have a 10× advantage.”

CPU orchestration → cache-aware runtime → GPU prefill/decode → specialized low-latency LPU.

That's unusually close to the workload you're describing.

---

2. Groq — the most interesting “chip + compiler” pure play.

Groq's fundamental bet is that inference shouldn't look like a general-purpose GPU workload. Its LPU uses compiler-scheduled execution and lots of on-chip SRAM, giving the runtime much more control over exactly when data moves and computation happens.

That is highly relevant to agents because an agent's critical metric often isn't aggregate FLOPS—it is:

time from “agent needs next token” → “next token arrives.”

Groq's deterministic execution is therefore particularly attractive for voice agents, coding agents, and other interactive workloads where tail latency matters.

The caveat is that the specialization makes the software/compiler stack unusually important.

---

3. SambaNova — perhaps the closest architectural match to the memory problem.

SambaNova's SN50 RDU is explicitly marketed as “Purpose-Built for Agentic Inference.” Its argument is that inference has become a memory/data-movement problem rather than simply a matrix-compute problem.

Its latest analysis says decode accounts for 75–97% of model-inference time in the configurations it studied, and its RDU uses large on-chip SRAM plus dataflow execution to keep bandwidth productive.

That's compelling for agents because long-running agents spend enormous amounts of time repeatedly decoding rather than doing giant batches of compute.

---

4. d-Matrix — very interesting if you believe compute-in-memory wins.

d-Matrix's Corsair is now in production and is explicitly designed to work alongside GPUs rather than necessarily replacing them. The company positions it around low-latency token generation for interactive and agentic applications.

Its digital compute-in-memory approach is particularly interesting because it attacks the fundamental problem of repeatedly moving model weights through the memory hierarchy.

---

5. FuriosaAI — one to watch.

Furiosa is unusually explicit about the transition to agentic inference. Its 2026 partnership with Broadcom is aimed at a third-generation accelerator and scale-up inference platform specifically for frontier agentic systems.

The architectural emphasis is data reuse + communication efficiency, which is exactly where bursty agent inference starts becoming different from conventional batch inference.

And don't overlook the runtime companies

There is another category that's arguably more important than the chip:

Yes. If by “bursty agent workloads” you mean workloads with lots of short, irregular inference bursts, tool calls, idle periods, changing batch sizes, and heterogeneous CPU/GPU/memory demands, then the interesting market is broader than “faster LLM chips.”

A recent systems study makes the distinction pretty stark: agent workloads can sit idle for minutes/hours between active steps, shift bottlenecks between GPU, CPU, memory and networking, and have highly variable task latency.

I’d divide the landscape like this:

1. The strongest chip-level bets

• Cerebras — probably one of the closest fits. Its wafer-scale architecture is optimized for very high token throughput and low latency, with enormous on-chip SRAM. Its new CS-4 is explicitly being positioned around agentic inference.
• Groq — deterministic, compiler-driven inference with an architecture designed around predictable low latency rather than maximizing generic GPU utilization. Particularly interesting for agents where latency of individual steps matters.
• SambaNova Systems — unusually relevant because its current SN50 positioning explicitly calls out fast agentic inference, while its RDU/dataflow architecture is designed around keeping models and data moving efficiently through the system.
• Etched — very aggressive specialization around Transformer inference. Its Sohu architecture essentially hardwires the Transformer execution pattern, trading generality for efficiency. The company has now reached a reported $21B valuation and >$1B in customer contracts, so this is no longer merely a research bet.
• d-Matrix — particularly interesting if you believe the bottleneck is memory movement rather than raw FLOPs. Its digital compute-in-memory architecture attacks that directly.
• Taalas — takes specialization to an extreme by hard-wiring model weights. Excellent economics for stable models, though less obviously suited to constantly changing agent workloads.
• MatX — building specialized hardware around Transformer inference/training, with an emphasis on memory bandwidth and efficient large-model execution.

A useful recent industry survey puts Groq, Cerebras, SambaNova, MatX, d-Matrix, Etched, Taalas, Tenstorrent, FuriosaAI, Axelera and others in this emerging inference-specialist category.

2. The runtime/compiler layer may actually be more important

This is where I'd look particularly closely if your thesis is bursty agents, rather than conventional high-QPS inference.

• NVIDIA — CUDA/TensorRT-LLM plus increasingly sophisticated scheduling and routing. Interestingly, NVIDIA just released PAIR, which dynamically routes agent tasks across idle machines rather than assuming one permanently saturated accelerator. That's almost exactly the “elastic compute for bursty agents” idea.
• Together AI — inference serving/routing rather than new silicon. Its infrastructure is increasingly aimed at making heterogeneous inference capacity fungible.
• Fireworks AI — highly optimized model serving, batching and deployment infrastructure.
• Modal — interesting from the bursty compute angle: ephemeral GPU allocation and serverless execution maps naturally onto agents that don't continuously consume accelerator capacity.
• Anyscale / Ray — more general distributed runtime, but increasingly relevant because agent execution is a distributed DAG of model calls, tools, sandboxes and data operations rather than one inference request.

And there is a particularly interesting research direction here: task-aware serving, communication-aware placement, state offloading and caching have all demonstrated substantial gains on agent workloads.

3. Hyperscalers are building the vertically integrated version

Don't overlook:

• Google — TPU + XLA + serving stack. Probably the most mature example of hardware/compiler/runtime co-design.
• Amazon Web Services — Trainium/Inferentia plus Neuron; increasingly able to optimize the entire inference path.
• Microsoft — Maia and its broader Azure inference stack.
• Meta Platforms — MTIA and increasingly workload-specific inference infrastructure.
• OpenAI — notably, OpenAI and Broadcom are now developing the Jalapeño inference ASIC, explicitly targeting efficiency/latency rather than generic GPU compute.

My take

If you're looking for “who is building the thing that will make agent inference dramatically cheaper?”, I'd put the bets into three buckets:

The key insight is that agent workloads may actually favor a runtime-first architecture. A conventional inference accelerator assumes you have a queue of homogeneous requests to batch efficiently. Agents instead produce a dynamic graph:

LLM → tool → CPU sandbox → retrieval → LLM → wait → LLM → browser → ...

The winning system therefore may not be the chip with the highest tokens/sec. It may be the scheduler/compiler/runtime that can move each tiny piece of work to the cheapest available resource, preserve state cheaply, avoid cold-starts, and opportunistically batch unrelated agent steps.

That's why I'd pay especially close attention to companies attacking scheduling + memory/state + heterogeneous execution, not just TOPS. The academic evidence is already pointing in that direction.

If you're evaluating this as a startup opportunity, I can also map the landscape into “who is actually closest to a new AWS-for-agent-compute / agent ASIC / agent runtime” and identify the gaps nobody seems to be filling yet.

Traditional enterprise GPUs often drop to 30% to 40% utilization on agentic workloads. Because agents spend a lot of time thinking, executing tool calls, waiting on external APIs, and processing variable-length reasoning loops, the workload is inherently spiky, unpredictable, and bursty.

A new tier of silicon, compiler, and runtime infrastructure is emerging to solve this "inference gap"—moving away from static, massive-batch matrix multiplication toward architectures built for dynamic, low-latency, irregular execution.

1. Dedicated AI ASIC & Architecture Players

• Groq (LPU / Tensor Execution System): Groq’s deterministic, single-core Language Processing Unit (LPU) architecture eliminates the traditional memory bottleneck of GPUs. Because there is no dynamic cache scheduling or heavy DRAM thrashing in the same way, it handles the abrupt start-and-stop token generation of agent loops with near-zero latency overhead.
• OpenAI (Custom Silicon / "Jalapeño"): Vertically co-designed custom inference platforms (like OpenAI's reported custom silicon efforts) are built specifically around the unpredictable, high-concurrency, and memory-bound nature of serving massive agentic fleets rather than static training jobs.
• Etched (Samba / Sohu): Etched builds specialized ASICs (Application-Specific Integrated Circuits) hardcoded strictly for Transformers. By removing general-purpose cores entirely, their runtime and chip design process bypass standard instruction-decode bottlenecks, allowing them to instantly react to variable batch sizes and ragged token lengths typical in multi-step agent loops.

2. Runtime & Inference Serving Stack Innovators

Rather than redesigning the physical silicon from scratch, several software-hardware co-design companies are optimizing the runtime and compilation layer to manage ragged batching and bursty context switching:

• vLLM & TensorRT-LLM Ecosystem (Anyscale, NVIDIA, and open-source contributors): While not hardware vendors themselves, software runtimes implementing PagedAttention and advanced continuous batching are continually retrofitted to handle the rapid asynchronous scheduling required when 10,000 background agents suddenly fire off tool-use requests simultaneously.
• Modular (MAX Engine): Modular is unifying the AI compilation and runtime stack with MAX , which bridges the gap between hardware (GPUs, CPUs, and custom accelerators) and high-performance execution graphs. Its pipeline architecture allows developers to rewrite how ragged, dynamic agent execution paths compile down to bare metal without the overhead of standard Python-heavy runtime dispatchers.
• Baseten / Neural Magic: Specializing in sparse execution, runtimes that optimize for sparsity and compressed KV-caches are vital for agents. Neural Magic's inference runtime forces irregular, bursty generative tasks to run efficiently on standard hardware by adapting the compilation to dynamic sparsity on the fly.

If you're targeting a specific deployment environment, let me know:

• Are you deploying on-premise hardware or targeting a cloud provider/API layer?
• Are your agents bottlenecked by time-to-first-token (latency) or KV-cache memory capacity during long tool loops?

I can narrow down which compiler/runtime stack fits your exact tech stack.

Yes. If by “bursty agent workloads” you mean workloads with lots of short LLM calls, tool calls, idle gaps, changing concurrency, and a premium on latency + utilization rather than peak FLOPS, there’s a surprisingly active market.

A recent systems paper characterizes agent workloads as having sudden load spikes, heterogeneous CPU/GPU affinity, frequent CPU↔GPU transitions, and sessions that can sit idle for minutes or hours—which is quite different from conventional continuous LLM serving.

Companies I’d watch

The most interesting distinction is that some companies are attacking the chip, while others are attacking the runtime problem.

1. SambaNova — probably closest to the thesis

SambaNova explicitly positions its SN50 RDU as purpose-built for agentic inference. Its architecture is designed around the fact that inference is heavily constrained by data movement and memory rather than simply matrix-compute throughput.

Even more interestingly, SambaNova and Intel announced a heterogeneous architecture in which GPUs handle prefill, SambaNova RDUs handle high-throughput decode, and Xeons handle host/action workloads. That's very close to the architecture implied by agentic workloads.

My take: one of the strongest companies to study if you're thinking “agent server” rather than “LLM GPU.”

2. d-Matrix — particularly interesting for short/bursty inference

d-Matrix's original thesis was explicitly small-batch, interactive inference with extremely low latency. Its architecture uses compute-in-memory and lots of on-chip SRAM to avoid the memory-movement costs of conventional accelerators.

That maps unusually well onto agents, where you may have:

agent → 300-token call → tool → 800-token call → tool → 200-token call

rather than:

10,000 simultaneous requests → continuous GPU saturation.

My take: probably one of the most technically interesting fits for your exact phrase “bursty agent workloads.”

3. Groq — latency-first inference

Groq's LPU architecture is designed around deterministic, extremely fast inference. It's particularly compelling when single-request latency matters more than maximizing giant batch throughput.

One caveat: NVIDIA licensed Groq's inference technology and core team in late 2025, so the competitive situation is no longer simply “Groq vs. NVIDIA.”

4. NVIDIA — don't overlook the runtime

If the question is “who is building the compiler/runtime for this?”, NVIDIA may actually be the most important answer.

Its stack is moving beyond simply “run a model on a GPU.” Triton Inference Server provides dynamic batching, concurrency management, streaming, configuration optimization and support across GPUs/CPUs/Inferentia.

For agentic workloads, the interesting NVIDIA layer is increasingly scheduling/orchestration/memory management, not just CUDA kernels.

And NVIDIA is clearly treating agentic inference as a distinct hardware problem: its latest systems include inference-oriented architectures and it has also incorporated Groq technology.

5. AWS — compiler/runtime + custom silicon

AWS has a particularly complete vertical stack:

Inferentia → Neuron compiler/runtime → EC2 → serving infrastructure

Inferentia is explicitly designed for generative-AI inference, with Neuron providing the software layer.

This is important because a truly good agent accelerator probably needs to understand more than GEMMs—it needs to understand KV cache, model loading, batching, scheduling, CPU interaction and state.

6. MatX — one I'd watch closely

MatX is particularly interesting because it's not merely another inference-chip startup. It was founded by former Google TPU engineers, and Anthropic recently held acquisition discussions with it around accelerating custom-chip development. Those talks reportedly shifted toward a potential partnership.

That's a strong signal that frontier labs believe the accelerator/compiler stack itself is strategically important.

7. Etched / Taalas — radical specialization

Etched is pursuing a Transformer-specific ASIC, while Taalas goes even further with highly hard-wired model implementations. The upside is extraordinary efficiency/latency; the downside is reduced flexibility.

That makes them interesting if your agent workload has stable models and predictable execution patterns, but less obviously ideal for rapidly changing agent stacks.

The company/category I think is missing from most chip lists

There's a potentially bigger opportunity above the accelerator:

An “agent runtime” that treats compute as an elastic pool rather than treating each agent as a GPU job. The recent AgentSysBench work is revealing here. Agent workloads can have CPU-bound sandboxes, GPU-bound inference, memory-bound retrieval, long idle periods and rapidly shifting bottlenecks. The authors found that task-aware scheduling, communication-aware placement, state offloading and caching can produce very large gains.

So I'd divide the opportunity into:

Silicon

• SambaNova
• d-Matrix
• Groq
• Cerebras
• Etched
• Taalas
• MatX
• Tenstorrent

Inference compiler/runtime

• NVIDIA
• AWS/Neuron
• Intel/OpenVINO
• AMD/ROCm
• Tenstorrent
• SambaNova

Agent-aware orchestration / scheduling

• This is much less settled.
• The interesting IP may be in dynamically moving model weights, KV cache, agent state, CPU sandbox work and tool calls rather than inventing another matrix accelerator.

And that's why I think your phrase “bursty agent workloads” points to a potentially distinct architecture category. Conventional GPU inference optimizes for keeping the accelerator busy. Agents often require the opposite: wake up quickly, execute a small amount of work extremely efficiently, persist state cheaply, go idle, and wake up again—while multiplexing thousands of such sessions.

If you're evaluating this as a startup/investment opportunity, I'd put d-Matrix, SambaNova, MatX, NVIDIA's inference-runtime effort, and Tenstorrent at the top of the diligence list—not because they're equivalent, but because they attack different pieces of the same emerging stack.

2. d-Matrix — particularly interesting for short/bursty inference

d-Matrix's original thesis was explicitly small-batch, interactive inference with extremely low latency. Its architecture uses compute-in-memory and lots of on-chip SRAM to avoid the memory-movement costs of conventional accelerators.

That maps unusually well onto agents, where you may have:

agent → 300-token call → tool → 800-token call → tool → 200-token call

rather than:

10,000 simultaneous requests → continuous GPU saturation.

My take: probably one of the most technically interesting fits for your exact phrase “bursty agent workloads.”

3. Groq — latency-first inference

Groq's LPU architecture is designed around deterministic, extremely fast inference. It's particularly compelling when single-request latency matters more than maximizing giant batch throughput.

One caveat: NVIDIA licensed Groq's inference technology and core team in late 2025, so the competitive situation is no longer simply “Groq vs. NVIDIA.”

4. NVIDIA — don't overlook the runtime

If the question is “who is building the compiler/runtime for this?”, NVIDIA may actually be the most important answer.

Its stack is moving beyond simply “run a model on a GPU.” Triton Inference Server provides dynamic batching, concurrency management, streaming, configuration optimization and support across GPUs/CPUs/Inferentia.

For agentic workloads, the interesting NVIDIA layer is increasingly scheduling/orchestration/memory management, not just CUDA kernels.

And NVIDIA is clearly treating agentic inference as a distinct hardware problem: its latest systems include inference-oriented architectures and it has also incorporated Groq technology.

5. AWS — compiler/runtime + custom silicon

AWS has a particularly complete vertical stack:

Inferentia → Neuron compiler/runtime → EC2 → serving infrastructure

Inferentia is explicitly designed for generative-AI inference, with Neuron providing the software layer.

This is important because a truly good agent accelerator probably needs to understand more than GEMMs—it needs to understand KV cache, model loading, batching, scheduling, CPU interaction and state.

6. MatX — one I'd watch closely

MatX is particularly interesting because it's not merely another inference-chip startup. It was founded by former Google TPU engineers, and Anthropic recently held acquisition discussions with it around accelerating custom-chip development. Those talks reportedly shifted toward a potential partnership.

That's a strong signal that frontier labs believe the accelerator/compiler stack itself is strategically important.

7. Etched / Taalas — radical specialization

Etched is pursuing a Transformer-specific ASIC, while Taalas goes even further with highly hard-wired model implementations. The upside is extraordinary efficiency/latency; the downside is reduced flexibility.

That makes them interesting if your agent workload has stable models and predictable execution patterns, but less obviously ideal for rapidly changing agent stacks.

The company/category I think is missing from most chip lists

There's a potentially bigger opportunity above the accelerator:

An “agent runtime” that treats compute as an elastic pool rather than treating each agent as a GPU job. The recent AgentSysBench work is revealing here. Agent workloads can have CPU-bound sandboxes, GPU-bound inference, memory-bound retrieval, long idle periods and rapidly shifting bottlenecks. The authors found that task-aware scheduling, communication-aware placement, state offloading and caching can produce very large gains.

So I'd divide the opportunity into:

Silicon

• SambaNova
• d-Matrix
• Groq
• Cerebras
• Etched
• Taalas
• MatX
• Tenstorrent

Inference compiler/runtime

Traditional GPUs often struggle with the "bursty" nature of AI agent workloads. Because agents spend a lot of time on non-linear tasks—like looping, waiting on external API tool calls, spinning up sub-agents, and managing dynamic context graphs—traditional GPUs often sit at low utilization rates (30–40%) while suffering from severe latency bottlenecks during intermittent bursts.

To solve this, a mix of specialized silicon, dataflow architectures, and orchestrating runtime/compiler stacks are emerging to handle dynamic, highly non-linear agent loops.

1. Dedicated Inference Silicon & Dataflow Architectures

• Groq (LPU Architecture): Groq’s Language Processing Unit (LPU) removes the standard instruction-decoding overhead and memory bottlenecks plaguing GPUs. Because its deterministic architecture relies heavily on a software-defined compiler that schedules every clock cycle in advance, it handles massive spikes in token generation and ultra-low latency bursts exceptionally well compared to traditional hardware.
• SambaNova Systems: SambaNova targets complex, multi-model enterprise agentic workflows. Their Reconfigurable Dataflow Unit (RDU) architecture and SambaStack software allow systems to dynamically switch between multiple frontier-scale models on a single node, reducing the cold-start and context-switching penalty when an agent jumps between reasoning and tool-execution models.
• Etched: Etched builds custom ASICs (such as their Sohu chip) explicitly hardcoded for transformer inference rather than general-purpose training. By hardcoding the transformer architecture directly into silicon, they eliminate instruction overhead, offering extreme throughput and fast-reacting execution for dynamic prefill/decode loops common in heavy agent pipelines.

2. High-Performance CPU and Hybrid Infrastructure

• Nvidia (Vera CPU & CUDA-X Agent Toolkits): As agent scaling strains cloud infrastructure, the industry has seen a massive "CPU comeback" to handle the orchestration and concurrent sandboxing required when thousands of sub-agents spawn. Nvidia’s Arm-based Vera CPU architecture, integrated tightly with updated CUDA-X libraries, is designed to accelerate the heavy CPU-bound orchestration and sandbox creation loops that accompany multi-agent execution.

3. Software Runtimes and Orchestration Engines

Hardware alone isn't enough; the runtime environment determines whether an agent loop survives tool timeouts and variable loads without crashing.

• Orcus (Agent Span): Focused entirely on the runtime problem, Orcus provides durable workflow orchestration engines designed to prevent agents from breaking mid-task during erratic execution patterns, handling state persistence across chaotic, bursty API calls.
• Wafer: Operating as an alternative inference and compilation layer, Wafer leverages AI-driven compilation and optimization techniques to route workloads efficiently across non-Nvidia and heterogeneous chips, maximizing utilization for dynamic enterprise application patterns.

Are you looking for a hardware accelerator (ASIC/LPU) to host the underlying models, or do you need a software orchestration runtime to manage the execution state and tool-call loops of the agents?