DevOps agent framework for self-healing systems

Building autonomous DevOps agents for self-healing infrastructure

Yes. You can build autonomous agents that detect CI/CD pipeline failures, investigate logs and telemetry, identify likely root causes, execute approved remediation steps, and verify that services have recovered.

The most effective approach is not to give an LLM unrestricted access to your infrastructure. Instead, combine an agent framework for reasoning and diagnosis with deterministic automation, infrastructure controllers, and tightly scoped permissions.

There are two complementary technologies to consider:

• Agent frameworks such as LangGraph, which coordinate investigation, tool use, decision-making, and recovery workflows.
• Self-healing automation platforms such as Kubernetes Operators and cloud-native runbook systems, which continuously enforce desired state and execute predictable repairs.

A particularly relevant recent example is LangChain's published autonomous SRE agent for Kubernetes, which combines proactive health checks, specialized diagnostic agents, and human approval for infrastructure changes.www.langchain.com

1. Frameworks and platforms worth evaluating

LangGraph

(Best for custom AI workflows)

Use it to build stateful agents with branching workflows, retries, checkpoints, specialized diagnostic agents, and human approval steps. Suitable for investigating CI failures, correlating logs with deployments, and selecting a remediation from a controlled toolset.

Docs:LangGraph documentation

Kubernetes Operator Framework

(Best for continuous self-healing)

Operators use control loops to compare actual infrastructure state with desired state and reconcile differences. They are ideal for restarting unhealthy workloads, correcting configuration drift, managing application upgrades, and restoring known-good configurations. They do not require an LLM for routine repairs.kubernetes.io

Docs:Kubernetes Operator pattern

AWS Systems Manager Automation

(Best for AWS-native remediation)

Provides predefined and custom runbooks for maintenance, deployment failures, configuration drift, and infrastructure recovery. An AI agent can recommend a runbook and provide parameters while the automation service executes the actual operations under IAM permissions.docs.aws.amazon.com+1

Docs:Systems Manager Automation

Microsoft Agent Framework

A candidate for multi-agent systems in which separate agents analyze build logs, review configuration changes, investigate observability data, and propose fixes. Evaluate it if you prefer Microsoft's agent ecosystem and .NET or Python integration.

Project:Microsoft Agent Framework on GitHub

Conductor

Useful for durable, long-running remediation workflows with retries, timeouts, and explicit execution steps. It is a workflow orchestrator rather than an autonomous reasoning framework, so pair it with an LLM agent when dynamic diagnosis is needed.

Project:Conductor on GitHub

My recommendation: start with LangGraph for intelligent diagnosis plus deterministic runbooks for execution. Add Kubernetes Operators for continuously correcting known infrastructure conditions. This separates uncertain reasoning from high-impact infrastructure changes.

2. Reference architecture for a self-healing DevOps agent

A practical design separates detection, reasoning, execution, and verification.

1. Detect a failure

CI/CD events · Prometheus alerts · CloudWatch · Kubernetes events · Logs

2. Diagnose with an agent

Collect evidence, correlate events, classify the failure, estimate confidence

3. Apply policy and risk checks

Check permissions, allowed actions, blast radius, approval requirements

Low-risk fix

Run an approved remediation automatically

High-risk fix

Request human approval before execution

4. Verify and learn

Re-run tests, check health metrics, confirm recovery, record the outcome

For production use, persist the investigation state, make actions idempotent, and record every proposed and executed change. LangGraph supports checkpointing and workflows that can pause for human review and resume later.github.com+2

3. What should the agents actually do?

Start with a small set of specialized capabilities rather than a large collection of agents that can all modify infrastructure.

These can be logical roles within one orchestrated workflow; they do not all need separate LLMs or separate services.

For example, GitHub Actions exposes APIs to inspect workflow logs and rerun failed jobs, making it possible to integrate a pipeline investigator without giving it shell access to your production environment.docs.github.com+1

4. An implementation plan

1. Instrument the systems

   Connect your CI/CD platform, observability stack, deployment system, and infrastructure inventory. Normalize failures into a common event format with timestamps, environment, service, deployment ID, and error details.

2. Build a read-only diagnostic agent

   Give the agent tools such as get_pipeline_logs, query_metrics, get_pod_events, inspect_recent_deployments, and get_terraform_plan. Ask it to return structured findings with evidence and a proposed fix.

3. Create an allowlisted remediation catalog

   Implement deterministic functions or runbooks for known failure types. Examples include retrying a transient job, rolling back a failed release, restarting a specific workload, or cleaning up a known temporary resource. Validate all parameters outside the LLM.

4. Add policy-based autonomy

   Automatically execute only known, reversible, low-impact actions. Require approval for production rollbacks with significant impact, infrastructure deletion, IAM changes, database migrations, and network policy modifications.

5. Verify recovery

   After a fix, check deployment status, error rates, latency, service health, and relevant tests. If the system remains unhealthy, stop further automated attempts and escalate rather than looping indefinitely.

6. Evaluate and expand

   Replay historical incidents, measure diagnosis accuracy and remediation success, and test failure scenarios in staging. Expand the action catalog as the system demonstrates reliable results.

5. Example: a LangGraph-based remediation workflow

Here is a simplified Python design. The model diagnoses the failure, but deterministic code controls whether a remediation is permitted.

from typing import Literal
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END

class Incident(TypedDict):
    failure: str
    evidence: str
    diagnosis: str
    action: str
    status: str

ALLOWED_ACTIONS = {
    "retry_transient_job",
    "restart_unhealthy_workload",
    "request_human_approval",
}

def investigate(state: Incident):
    # Replace with read-only calls to your CI/CD,
    # logs, metrics, and infrastructure APIs.
    evidence = collect_failure_evidence(state["failure"])

    diagnosis = diagnose_with_llm(
        failure=state["failure"],
        evidence=evidence,
    )

    return {
        "evidence": evidence,
        "diagnosis": diagnosis["summary"],
        "action": diagnosis["proposed_action"],
    }

def enforce_policy(state: Incident):
    action = state["action"]

    if action not in ALLOWED_ACTIONS:
        return {"status": "blocked"}

    if action == "request_human_approval":
        return {"status": "approval_required"}

    # Apply additional checks for environment,
    # resource scope, confidence, and retry limits.
    return {"status": "approved_for_execution"}

def route(state: Incident) -> Literal["execute", "stop"]:
    if state["status"] == "approved_for_execution":
        return "execute"
    return "stop"

def execute(state: Incident):
    # Dispatch only to a preimplemented, allowlisted
    # remediation function with validated parameters.
    result = execute_remediation(
        action=state["action"],
        evidence=state["evidence"],
    )
    return {"status": result["status"]}

graph = StateGraph(Incident)
graph.add_node("investigate", investigate)
graph.add_node("policy", enforce_policy)
graph.add_node("execute", execute)

graph.add_edge(START, "investigate")
graph.add_edge("investigate", "policy")
graph.add_conditional_edges(
    "policy", route,
    {"execute": "execute", "stop": END},
)
graph.add_edge("execute", END)

app = graph.compile()

The functions collect_failure_evidence, diagnose_with_llm, and execute_remediation are placeholders you would implement against your actual systems. This skeleton demonstrates orchestration, not a production-ready security boundary. In production, enforce authorization in the execution service itself, use durable checkpoints, and implement an actual approval-and-resume flow for actions requiring review.github.com+2

6. How to choose the right approach

For AWS, Systems Manager Automation supports custom runbooks, IAM-controlled execution, concurrency limits, and integration with event-driven automation. That makes it useful even if you decide not to introduce an LLM agent initially.docs.aws.amazon.com+1

7. Critical safeguards

• Least privilege: Give diagnostic tools read-only access and remediation services narrowly scoped write permissions.
• Bounded autonomy: Define allowed actions by environment, severity, confidence, and blast radius.
• Idempotency: Repeated execution should not create additional damage.
• Loop prevention: Set retry budgets, timeouts, and a maximum number of remediation attempts per incident.
• Auditability: Log the evidence, recommendation, policy decision, approval, execution result, and recovery checks.
• Prompt-injection resistance: Treat build logs, commit messages, issue descriptions, and tool output as untrusted data—not instructions to the agent.
• Safe failure: If diagnosis is uncertain or verification fails, stop and escalate.

The guiding principle is to make the agent autonomous about investigation and selecting from approved actions, while keeping infrastructure mutation constrained by code and policy.

Let's narrow this down to your stack

I can turn this into a concrete architecture with a working starter implementation. Which environment are you primarily using?

1. CI/CD and infrastructure

Kubernetes + GitHub Actions

GitLab CI + Terraform

Azure DevOps

Jenkins

Mixed or other

2. Cloud environment

AWS

Azure

Google Cloud

On-premises or multi-cloud

3. Desired autonomy

Diagnose and recommend fixes only

Automatically apply low-risk fixes

Automatic in staging; approval for production changes

Fully autonomous within strict policies

4. Preferred language

Python

TypeScript

Go

C# / .NET

[Design my implementation]