Our on-call team wants an AI agent that can investigate incidents and propose fixes. What tools are best?

Building an AI-powered SRE agent to investigate incidents and propose fixes requires tools that do more than just summarize logs—they need to query telemetry, form hypotheses, trace causal chains, and suggest (or safely execute) remediations.

The ecosystem splits into three distinct tiers based on where your data lives and how deep the automation goes:

• AI-Native Investigation Platforms (Specialized SRE Agents): Tools built from the ground up to query clusters, run diagnostics, and trace root causes autonomously. Top options include Resolve AI, Traversal, NeuBird AI , and OpsWorker . These plug into your environment (like Kubernetes via read-only agents) and push explainable root-cause analyses directly into Slack.
• Observability-Native AI Agents: If you already live inside a massive observability ecosystem, native agents leverage your existing data gravity. Datadog Bits AI iteratively queries APM, logs, and infrastructure metrics to diagnose root causes from an incident channel. Similar capabilities are offered by Grafana Assistant and major cloud providers via the AWS DevOps Agent and Azure SRE Agent.
• Incident Management & Workflow Orchestrators: Platforms that manage the lifecycle and coordinate human-in-the-loop approvals for fixes. This includes the PagerDuty SRE Agent , alongside incident.io, Rootly , and FireHydrant , which orchestrate the handoff between autonomous diagnostics and gated execution.

If you want to narrow this down, tell me:

• What is your primary observability and cloud stack (e.g., Datadog, AWS, Kubernetes, Prometheus)?
• Do you want the agent to only propose fixes for human approval, or actively execute remediation scripts?

Yes. I’d build this as an AI SRE agent with read-heavy access, evidence-based diagnosis, and approval-gated remediation, rather than an agent that can freely change production.

My shortlist

• Datadog Bits AI SRE — best if Datadog is already your observability source of truth. It can investigate alerts using telemetry and organizational context and produce root-cause information.
• PagerDuty SRE Agent / AIOps — best for a mature enterprise on-call operation already centered on PagerDuty, especially where alert correlation, escalation, runbooks, and governance matter.
• Rootly AI SRE — strong choice if you want incident management + AI investigation + workflow automation in one system, particularly for Slack-centric teams.
• incident.io — particularly attractive for Slack-first teams that prioritize incident coordination, context, communication, and postmortems.
• Build your own agent — best when your environment is unusual or you want the agent to investigate across many internal systems. OpenAI's current agent tooling supports tool use, sandboxed execution, files, commands, and long-running agent workflows.

The tool stack I'd give the agent

The important part isn't just the model. Give it access to evidence + controlled actions:

1. Observability: metrics, logs, traces, profiles, service maps.
2. Change history: GitHub/GitLab PRs, deployments, feature flags, infrastructure changes.
3. Incident context: PagerDuty/incident.io/Rootly, Slack, previous incidents and postmortems.
4. Infrastructure: Kubernetes, cloud APIs, databases, queues, load balancers.
5. Code execution: a sandbox where it can run queries, scripts, tests, and diagnostics without touching production.
6. Remediation tools: rollback deployment, disable feature flag, scale service, restart workload, etc.—but behind explicit approval initially.
7. Knowledge: runbooks, architecture docs, service ownership, known failure modes.

That produces a loop like:

Alert → investigate → gather evidence → form hypotheses → test hypotheses → propose fix → human approval → execute → verify → document.

OpenAI's current agent infrastructure specifically supports giving agents a controlled computer/sandbox environment and tools for files and commands, which maps well to the investigation portion.

What I'd recommend

If you're already on Datadog, pilot Bits AI SRE first.

If you're already on PagerDuty, evaluate PagerDuty's SRE/AIOps capabilities before introducing another incident platform.

If your main problem is Slack-based incident coordination plus investigation, I'd shortlist Rootly and incident.io.

If your goal is genuinely "give an AI agent our entire production environment and have it investigate novel failures", I'd build a controlled agent around your existing observability/infra stack rather than replacing everything with an incident-management product.

The key evaluation criterion should be not "does it produce a plausible RCA?" but "can it show the evidence for each hypothesis, reproduce the diagnosis, and safely verify its proposed fix?" AI incident tooling is increasingly capable, but vendor claims around autonomous RCA should be validated against your own historical incidents.

If you tell me your current stack (e.g. AWS + Kubernetes + Datadog + PagerDuty + GitHub + Slack), I can give you a concrete architecture and a 2–3 tool shortlist.

For this use case, I’d build around an AI SRE agent with read-heavy investigation tools and tightly controlled write actions, rather than a generic chatbot.

Recommended stack

• Incident management / orchestration: incident.io or Rootly

• incident.io is particularly interesting if your goal is autonomous investigation: its current tooling connects incident context with telemetry and recent changes and can generate environment-specific fixes.

• Rootly is a strong alternative if you want highly configurable workflows and Slack/Teams-centric response.

• PagerDuty remains the safer choice for very large enterprises with complex alert routing and escalation requirements.

• Observability: Give the agent APIs/MCP access to your existing Datadog, Grafana/Prometheus, New Relic, Sentry, etc.

• It should be able to query metrics, logs, traces, error samples, deployments, and service dependencies.

• Don't make the agent rely solely on an incident-management vendor's AI; the telemetry is where much of the useful evidence lives.

• Code + deployment context: GitHub/GitLab + CI/CD + Kubernetes/cloud APIs

• Recent commits and deployments

• Diff inspection

• Rollout status

• Pod/container health

• Config changes

• Feature flags

• Recent infrastructure changes

• Knowledge: Connect your runbooks, architecture docs, previous incidents, and service ownership data.

• This is crucial for answering “what should we do?” rather than merely “what looks broken?”

• Agent execution: For a custom agent, OpenAI's current Agents API supports MCP, custom functions, web search, parallel tool calls, and long-running cloud-agent workflows.

• The Agents SDK also supports controlled sandbox execution for agents that need to inspect files, run commands, and edit code.

The tool set I'd expose to the agent

Think of the agent as having roughly these capabilities:

1. search_incidents() — find similar historical incidents.
2. query_metrics() — investigate anomalies and correlations.
3. query_logs() — search structured/unstructured logs.
4. query_traces() — follow failing requests across services.
5. get_recent_deployments() — correlate failures with releases.
6. inspect_service() — ownership, dependencies, health, configuration.
7. search_code() — locate relevant implementation.
8. inspect_git_diff() — understand recent changes.
9. inspect_kubernetes() / cloud APIs — infrastructure state.
10. search_runbooks() — retrieve established remediation procedures.
11. create_fix_branch() / open_pr() — propose an actual code/config fix.
12. run_tests() / run_canary() — validate the proposed fix.
13. rollback() / restart() / change_config() — only with explicit authorization.

The important distinction is between investigation tools and mutation tools. An agent should normally have broad read access but very narrow, auditable write access.

OpenAI's own guidance describes tools as the mechanism that lets agents retrieve data, modify systems, call APIs, execute code, and orchestrate other agents.

What I'd actually build

Alert
  ↓
Incident manager
  ↓
AI SRE agent
  ├── Metrics / logs / traces
  ├── Deployments / Git
  ├── Kubernetes / cloud
  ├── Runbooks / past incidents
  └── Service catalog
       ↓
   Hypotheses
       ↓
   Evidence gathering
       ↓
   Root-cause assessment
       ↓
   Proposed remediation
       ↓
   ┌─────────────────────┐
   │ Human approval      │
   └─────────┬───────────┘
             ↓
       PR / rollback / change
             ↓
        Verification
             ↓
        Incident update

My shortlist

One important caveat: don't start with autonomous remediation. Start with “investigate → explain evidence → propose fix → open PR”. Once you've measured accuracy, you can selectively allow low-risk actions such as restarting a known-safe workload or rolling back a deployment. Current AI incident tooling is increasingly capable of investigation and proposed actions, but the industry still emphasizes keeping humans in control of consequential changes.

If you tell me your current stack (e.g. AWS + Kubernetes + Datadog + GitHub + Slack), I can map this into a concrete architecture and recommend exactly which APIs/MCP servers/tools I'd give the agent.

Deploying an AI agent to investigate production incidents and suggest fixes falls into the emerging AI SRE (Site Reliability Engineering) or Autonomous Incident Investigation category.

The tooling landscape generally splits into three distinct approaches depending on how deeply integrated you want the agent to be with your stack:

1. Dedicated Autonomous AI SRE Platforms

These are specialized agentic platforms built specifically to query logs, Kubernetes resources, metrics, and git history to perform end-to-end root cause analysis (RCA).

• • Resolve AI : Focuses heavily on end-to-end autonomous troubleshooting and root-cause isolation.
• • Traversal : Ranks high on deep technical investigation, correlating multi-source telemetry before human sign-off.
• • OpsWorker / Cleric / NeuBird : Read-only in-cluster or cloud-connected agents that automatically parse alerts, pull relevant runtime context, and drop structured RCAs and remediation steps directly into Slack.

2. Native AI Features in Observability & Cloud Platforms

If your metrics, traces, and logs already live in a unified ecosystem, turning on the vendor’s native assistant or agent is often the lowest-friction path.

• • Datadog (Bits AI) : Deeply embedded in the Datadog ecosystem; Bits AI can synthesize monitors, summarize incident channels, and query metrics autonomously.
• • Grafana (Grafana Assistant) : Great if your stack runs heavily on Grafana/Prometheus dashboards and Loki logs for natural language query generation and anomaly detection.
• • AWS DevOps Agent / Azure SRE Agent : Native hyperscaler tools that inspect cloud resource changes, CloudWatch/Azure Monitor logs, and IAM events when native infrastructure fails.

3. Incident Response & ChatOps Orchestrators with AI

These platforms don't pull the low-level telemetry themselves, but act as the orchestration layer that coordinates the AI investigation, timeline generation, and post-mortems.

• • incident.io : Known for tight Slack-native workflows and rapidly incorporating AI summaries and investigation loops.
• • Rootly / FireHydrant : Enterprise incident response workflows that integrate with LLM backends to auto-populate post-mortems and suggest historical fixes based on past incidents.

To help narrow down the best fit, could you share:

• What is your primary infrastructure footprint (e.g., Kubernetes-heavy, pure AWS/Azure, serverless)?
• Where does your observability data currently live (Datadog, Grafana, New Relic, etc.)?

For an on-call team, the best AI incident agent depends on whether you want investigation, incident coordination, auto-remediation, or a combination. A good production setup usually uses an AI agent as a senior SRE copilot: it gathers evidence, proposes hypotheses, suggests commands/runbooks, and drafts fixes, while humans approve risky actions.

Strong options to evaluate

What I would build into the agent

A useful incident agent should have connectors to:

• Observability

• Logs: Splunk, Elasticsearch, Loki, CloudWatch

• Metrics: Prometheus, Datadog, Grafana

• Traces: OpenTelemetry, Jaeger, Tempo

• Change context

• GitHub/GitLab pull requests

• CI/CD deployments

• Feature flags

• Infrastructure changes (Terraform, Kubernetes)

• Operational knowledge

• Runbooks

• Past incidents/postmortems

• Architecture docs

• Service ownership maps

• Execution tools (with approval gates)

• Restart service

• Roll back deployment

• Scale workload

• Disable feature flag

• Run diagnostics

A practical architecture

A mature design often looks like:

1. Alert arrives

• PagerDuty/Datadog/etc. creates the incident.

2. Agent gathers evidence

• "What changed in the last hour?"
• "Which services depend on this one?"
• "Have we seen this before?"

3. Agent forms hypotheses

• Rank likely causes.
• Link evidence.
• Suggest verification steps.

4. Human approves remediation

• Execute a rollback or runbook action.
• Capture results.

5. Agent creates the record

• Timeline
• Root cause summary
• Follow-up tickets
• Postmortem draft

Build vs buy

Buy if:

• You need production reliability quickly.
• Your team already uses a major incident platform.
• You want integrations and governance.

Build if:

• You have unusual infrastructure.
• You want a custom internal SRE agent.
• You already have strong tooling APIs.

A common pattern is: buy incident management + build custom investigation skills on top of your internal knowledge and runbooks.

Evaluation checklist

Before choosing, test the agent on 10–20 historical incidents:

• Can it find the actual cause?
• Does it cite evidence, not guesses?
• Does it avoid dangerous changes?
• Does it understand your architecture?
• Can responders trust its recommendations at 2 AM?
• Does it reduce MTTR?

If you tell me your stack (AWS/GCP/Azure, Kubernetes or not, Datadog/Grafana/Splunk/etc., Slack vs Teams, and whether you want auto-fixes), I can narrow this to a shortlist and architecture.

• Can it find the actual cause?
• Does it cite evidence, not guesses?
• Does it avoid dangerous changes?
• Does it understand your architecture?
• Can responders trust its recommendations at 2 AM?
• Does it reduce MTTR?

If you tell me your stack (AWS/GCP/Azure, Kubernetes or not, Datadog/Grafana/Splunk/etc., Slack vs Teams, and whether you want auto-fixes), I can narrow this to a shortlist and architecture.

Building an agentic workflow for on-call investigations requires tools that can securely parse telemetry, read codebases, and interface with your incident command center without generating a wall of false-positive noise.

The best tools and platforms for AI-driven incident investigation and fix proposals fall into a few clear categories:

1. Dedicated Incident Management Platforms with Native AI Agents

• PagerDuty (SRE & Scribe Agents): PagerDuty features agentic AI capabilities designed to reduce context-switching. Their SRE Agent triages alerts directly in Slack, queries log management tools (like Grafana or CloudWatch) on your behalf, cross-references recent GitHub deployments, and surfaces relevant runbooks. Their Scribe Agent can even join bridge calls to transcribe and summarize context for engineers jumping onto the-call late.
• Incident.io (AI SRE): Known heavily for Slack-native incident coordination, incident.io features multi-agent systems that ingest alerts, parse historical incidents, cross-reference GitHub PRs, and query metrics to build a root-cause hypothesis and suggest fixes within minutes.
• Bits AI (by PagerDuty / integrations): Often deployed inside Slack workspaces to autonomously pull monitors, correlate logs, and draft initial investigations the moment an alarm fires.

2. Observability Platforms with Built-in Diagnostics

• Datadog (Bits AI / LLM capabilities): Datadog integrates tightly with conversational AI troubleshooters that can automatically inspect APM traces, error logs, and infrastructure metrics when an alert triggers, explaining why a threshold was breached rather than just stating that it was.
• New Relic & Dynatrace: Both offer native generative AI assistants that analyze anomalies across full-stack telemetry and write out plain-language summaries of the blast radius.

3. Developer & Coding Agents for Remediation Proposals

• Claude Code / CLI Agents: Many SRE teams are safely pairing terminal-based or IDE-integrated coding agents with internal Model Context Protocol (MCP) servers. Once an observability or PagerDuty agent pinpoints a misbehaving service or bad commit, a coding agent can be directed to inspect the repository, write a patch or rollback script, and open a draft pull request for human approval.

To help narrow down the best architecture for your team, let me know:

• What on-call and chat tools do you currently use (e.g., PagerDuty, Jira, Slack, Opsgenie)?
• Where does most of your telemetry live (Datadog, Grafana, AWS CloudWatch)?
• Do you want the agent to only suggest fixes (human-in-the-loop) or have capabilities to execute automated remediations (like rollbacks)?

If your goal is “AI gets paged, investigates across telemetry/code/history, then proposes a safe fix”, the market has moved beyond simple alert summarizers. The strongest options in 2026 are AI SRE/incident agents that can perform multi-step investigations.

My shortlist

1. PagerDuty SRE Agent — probably my first POC

If you're already using PagerDuty, this is the most natural starting point. Its agent can ingest incident data, runbooks and logs, investigate likely causes, recall similar incidents, and recommend diagnostic/remediation actions. It can pull data from tools including Grafana, Datadog, New Relic and CloudWatch, and runbooks from GitHub/Confluence.

It also now has a GitHub integration that puts incident state, change correlations and incident history directly into PR workflows—particularly interesting for “find the bad deploy → propose a code fix” workflows.

2. incident.io — best Slack-native option

For a team that lives in Slack, I'd put incident.io near the top. Its AI investigation capability can correlate things like GitHub PRs, Slack context, logs and historical incidents, while the platform handles the incident workflow around it.

The big advantage is less context switching: the agent's investigation becomes part of the incident channel/workflow rather than another console engineers need to learn.

3. Datadog Bits AI SRE — best if you're already all-in on Datadog

If your logs, metrics, traces and alerts are already in Datadog, Bits AI SRE is compelling because the investigation agent has that telemetry immediately available. Current comparisons characterize it as particularly strong for automated alert investigation and RCA within the Datadog ecosystem.

I wouldn't choose it solely to get an AI agent if you're not already a Datadog shop—the platform coupling becomes a meaningful consideration.

4. Rootly — best for customizable incident automation

Rootly is worth a POC if your incident process has lots of bespoke workflows. Its differentiators include customizable automation and AI-assisted RCA, including highlighting relevant code/configuration changes.

What I'd actually build

I wouldn't give the agent unrestricted production access initially. I'd use this architecture:

Pager/incident → AI investigator → read-only telemetry + Git → hypothesis/RCA → proposed remediation → human approval → PR/runbook/action → automated verification

Give it tools for:

• Observability: logs, metrics, traces, profiles
• Infrastructure: Kubernetes/cloud read access
• Change history: GitHub/GitLab, deployments, feature flags
• Knowledge: runbooks, architecture docs, previous incidents
• Incident context: PagerDuty/incident.io/Slack
• Actions: initially proposal-only; later approval-gated remediation

The crucial evaluation criterion isn't “does it produce a nice RCA?” It's whether it can gather new evidence and test hypotheses across systems. That's what separates an actual investigation agent from an LLM summarizer.

My recommendation

For a typical on-call engineering team, I'd run a 2–3 week bake-off between PagerDuty SRE Agent, incident.io, and the AI agent native to your observability stack.

Measure:

1. % of incidents where the agent reaches the correct root cause
2. Time from page → useful hypothesis
3. % of proposed fixes accepted by engineers
4. False/confident-but-wrong recommendations
5. How often it needs human intervention
6. Whether it can identify the specific bad PR/deployment/config change
7. Whether its proposed fix actually resolves the underlying symptom
8. Auditability and permission controls

One important reason to be conservative with autonomous remediation: recent agent-security incidents demonstrate that increasingly capable agents need strong containment and monitoring, particularly when they have broad network or production permissions.

If you tell me your current stack (e.g. PagerDuty + Datadog + Kubernetes + GitHub + Slack), I can narrow this to the best 2–3 choices and map exactly which tools each agent can access and what I'd let it automate.

Deploying an AI agent to investigate production incidents and propose fixes—often referred to as an AI SRE (Site Reliability Engineering) or Autonomous Incident Response agent—moves your team past basic alerting into automated triage.

The best tools in this space fall into three distinct categories: Dedicated AI SRE Investigators, Ecosystem-Native AI Assistants , and Chat-Ops & Workflow Orchestrators.

1. Dedicated AI SRE & Investigation Agents

These platforms act like virtual teammates. They pull logs, trace dependencies, correlate recent deployments, and write up root cause analyses (RCAs) or suggested fixes automatically when an alert fires.

• Resolve AI : Known for putting "teams of agents" into the on-call rotation. It allows you to encode your team's runbooks as modular skills, ensuring that institutional knowledge is applied consistently during a 3 AM page.
• Traversal : Models your entire production environment as a massive causal graph (millions of nodes) to trace how a failure propagates across services, prioritizing deep runtime investigation over surface-level notification.
• Cleric : Operates as a read-only agent by default with fully auditable investigations. It builds a cumulative operational memory, meaning a resolved incident actively informs and shortens the troubleshooting loop for future ones.
• OpsWorker / TierZero AI : Excellent for Kubernetes-heavy or microservice environments. They inspect cluster resources, gather logs, and explicitly focus on inspectability—allowing engineers to debug the agent's logic just like they debug the stack.

2. Native Observability AI Assistants

If your team already lives inside a major observability platform, their built-in AI agents have native access to your raw metrics, traces, and logs without needing external context syncing.

• Datadog (Bits AI) : Automatically investigates every alert as it fires, querying Datadog's proprietary telemetry to summarize blast radiuses, suggest hypotheses, and surface relevant logs.
• Grafana (Grafana Assistant) : Good for metric-heavy and Prometheus/Loki-backed infrastructures, helping query logs and explain anomalies conversationally.
• AWS DevOps Agent / Azure SRE Agent : Ideal if your stack is tightly bound to a single cloud provider’s ecosystem for native resource remediation suggestions.

3. Chat-Native Incident Management & Workflow Automation

These platforms manage the human-in-the-loop coordination, Slack workflows, and post-mortems, pairing strong automation engines with AI drafting.

• Rootly : Combines a powerful workflow engine with native AI across the incident lifecycle—from automated incident response to drafting retrospectives and routing pages.
• incident.io : A streamlined, Slack-native incident manager known for fast setup, clear workflows, and easing the coordination burden on responders.

Key Architectural Best Practices

• Read-Only by Default: Keep the agent read-only during the initial investigation phase. Let it propose the fix (e.g., a rollback command, a config patch, or a scaling adjustment), but require a human-in-the-loop approval button in Slack or your portal to execute it.
• Runbook Codification: The best tools allow you to feed your team's markdown/notion runbooks directly to the agent so it knows your specific architecture's quirks.

To help narrow down the ideal choice, could you share:

• What is your primary observability stack (e.g., Datadog, Grafana, CloudWatch)?
• Where do you host your workloads (Kubernetes, AWS/GCP, Serverless)?
• Do you want the agent to only investigate , or do you eventually want it to execute automated remediation with approval?

The best tools for on-call AI incident investigation and remediation combine observability platforms, automated root-cause analysis, and repository-connected coding agents.

Top AI Incident Investigation & Remediation Tools

• Grafana Assistant uses Grafana Labs agentic capabilities to query metrics, investigate anomalies, and create collaborative triage workflows.
• Datadog provides built-in LLM-powered assistant capabilities to summarize logs, analyze traces, and surface context directly to engineering teams.
• PagerDuty Advance applies automated noise reduction, event correlation, and generative AI summaries to accelerate incident triage.
• Augment Code or specialized coding agents connect production telemetry to codebases to suggest rollbacks, config tweaks, or pull requests.
• Arize AI and LangSmith offer managed debugging and tracing agents that trace failures in production systems and open direct fixes.

If you share your current monitoring stack (e.g., Datadog, Prometheus, PagerDuty) and code hosting platform (GitHub, GitLab), I can recommend the exact integration path for your team.

Yes. If the goal is “AI investigates the incident, finds likely root cause, and proposes a safe fix”, I’d evaluate tools in this order:

Best options

• datadoghq.com — my first choice if you're already heavily invested in Datadog. Bits Investigation can autonomously investigate alerts across metrics, logs, traces, incident context, and related signals, while Bits Code can generate production-oriented fixes.
• pagerduty.com — strongest fit if PagerDuty is the center of your on-call workflow. It uses incident history, runbooks, diagnostics and logs to investigate, identify likely causes, and recommend or perform approved remediation.
• resolve.ai, incident.io, rootly.com, and similar AI-SRE/incident-management products are worth a bake-off if you want a more specialized agent rather than extending your existing observability platform. A recent 2026 comparison puts these alongside Datadog and PagerDuty as leading AI-SRE approaches.

What I'd actually build

Don't give the agent unrestricted production access. Give it a toolbelt roughly like:

1. Observability: query logs, metrics, traces, profiles and service dependencies.
2. Change intelligence: inspect recent deploys, commits, feature flags and infrastructure changes.
3. Knowledge: search runbooks, architecture docs, previous incidents and postmortems.
4. Incident control: read/update PagerDuty or incident.io and communicate in Slack.
5. Code: inspect the relevant repository and generate a proposed patch.
6. Remediation: execute pre-approved actions such as rollback, restart, scaling or flag changes.
7. Verification: check telemetry after the action and automatically determine whether the incident improved.

The important distinction is propose → approve → execute → verify, rather than simply letting an LLM run shell commands. Recent autonomous-agent incidents are a good reminder that unrestricted agents can create serious security and operational risks.

My recommendation

If you're already on Datadog, start with Bits Investigation + Bits Code. It gives you the shortest path from “we got paged” → “here's the probable cause” → “here's a candidate fix.” Datadog explicitly supports investigation from an incident/Slack context and can surface root-cause findings and recommended next steps.

If you're already on PagerDuty, start with SRE Agent instead; its advantage is the deep integration with the actual on-call lifecycle and incident history.

If you tell me your current stack (e.g. AWS/Kubernetes + Datadog + PagerDuty + GitHub + Slack), I can sketch the agent architecture, permissions, tools/APIs, and guardrails I'd use.