How do I add durable execution and retries to a flaky AI agent workflow?

Adding durable execution and retries to a flaky AI agent workflow requires shifting from in-memory state management to an orchestration engine or checkpointer pattern . Because LLM calls time out, external APIs rate-limit, and tools fail mid-execution, you need a system that records state transitions and replays or retries cleanly.temporal.io+1

Here are the primary ways to build durability and retries into your agent stack, broken down by architectural approach:

1. Use a Dedicated Durable Orchestration Engine (e.g., Temporal)
   • How it works: Engines like Temporal treat your agent's steps as durable workflows and individual tool calls/LLM requests as isolated activities. State and event histories are persisted externally. If a worker crashes or an API returns a 500 error, the platform automatically retries the specific activity using a backoff policy and resumes execution from the exact point of failure.
   • Implementation path: You can wrap your agent steps using native integrations, such as the Temporal LangGraph Plugin Documentation , which runs graph nodes as retryable activities.docs.temporal.io+3
2. Pair Native Agent Framework Checkpointers with External Storage (e.g., LangGraph + Postgres)
   • How it works: Frameworks like LangGraph support state persistence through checkpointers . By swapping an in-memory checkpointer for a persistent database (PostgreSQL, Redis , or SQLite ), every super-step of the agent's reasoning loop is saved.
   • Implementation path: If a node fails, you catch the exception, log the state, and reload the graph from the last committed checkpoint in the database to resume or route to a fallback prompt.temporal.io
3. Implement Application-Level Retries with Exponential Backoff and Jitter
   • How it works: LLM providers (OpenAI, Anthropic ) and vector databases frequently throw rate limits (429) or temporary gateway errors (502, 504). Wrapping individual LLM calls or tool executions in a robust retry decorator (using libraries like tenacity in Python) isolates failures locally.
   • Configuration tip: Always apply exponential backoff combined with random jitter to prevent "thundering herd" issues when hitting recovering downstream APIs.
4. Design Idempotent Tools and Safe Side-Effects
   • How it works: Retries only work safely if repeating an action doesn't duplicate side effects (e.g., sending the same email three times or double-charging a credit card).
   • Best practice: Design your agent tools to accept idempotency keys, or separate your agent graph into "reasoning/planning" phases (which are pure and re-playable) and "action/mutation" phases (which require explicit transactional guarantees or external state checks before firing).

If you want, tell me:

• What framework or stack are you currently using to build your agent (e.g., LangGraph, CrewAI, custom Python loops)?
• Are you worried more about LLM rate limits/timeouts or infrastructure/server crashes?

I can dive into community discussions on the r/LocalLLM or LangChain Community subreddits regarding how other developers handle gotchas like prompt-drift during workflow replays, or provide a tailored code snippet for your setup.

You can add durable execution and retries to a flaky AI agent workflow by integrating a dedicated orchestration framework like Temporal, Inngest , or Pydantic Logfire/AI that automatically checkpoints state, replays events, and wraps LLM and tool calls in resilient retry policies.reddit.com+3

AI agent loops fail due to transient network drops, rate limits, flaky APIs, and unpredictable model outputs. Moving from basic scripts to production-grade durability requires structural design changes.inngest.com+1

Core Strategies for Durable AI Workflows

• Use a Durable Workflow Engine : Offload state management to platforms like Temporal or Inngest. These frameworks record every step (LLM generation, tool execution) in an append-only event log so that if a node crashes mid-flight, execution resumes from the last successful checkpoint instead of restarting from scratch.reddit.com+1
• Isolate Tool Calls as Activities : Treat every external API call, database write, or deterministic calculation as an isolated "activity" or step with its own custom retry and exponential backoff configuration.pydantic.dev+1
• Implement Hard Guardrails and Budgets : Never rely solely on the LLM to decide when to stop. Enforce strict orchestrator-level limits on maximum iterations, execution time, and total token expenditure to prevent infinite loops.linkedin.com
• Leverage Suspend/Resume for Human-in-the-Loop : Use native workflow pause features when the agent requires user validation or external data. Durable engines can sleep indefinitely without wasting compute resources until a webhook or human response arrives.

Comparison of Popular Durable Execution Options

Tool / Framework	Best For	Key Mechanism
Temporal	Complex, long-running, multi-service agent graphs	Code-as-workflow with deterministic event replay and robust workers
Inngest	Serverless and event-driven architectures	Step functions using standard queues and HTTP triggers
Pydantic AI	Pydantic-native Python agents

Step-by-Step Implementation Approach

1. Define State Boundaries : Map your agent workflow into distinct, discrete steps (e.g., Parse Prompt → Call LLM → Execute Tool → Evaluate Progress).
2. Wrap External Calls : Apply explicit retry policies (e.g., max 3 attempts with exponential backoff) specifically targeting HTTP 429 rate limits or 5xx server errors from LLM providers.reddit.com
3. Persist Context : Ensure conversation history and intermediate tool outputs are saved externally to a persistent database or state store before moving to the next action.reddit.com
4. Add Progress Detectors : Track whether sequential tool iterations are returning identical outputs; if the agent gets stuck in a loop, programmatically terminate or escalate the task.linkedin.com

If you share which language or framework (e.g., Python with LangChain/Pydantic AI, or TypeScript) and infrastructure (serverless vs. dedicated workers) you are using, I can provide a code example for setting up retries.

A flaky AI agent workflow usually fails for the same reasons distributed systems fail: processes die, APIs time out, tools return transient errors, and partial side effects happen. Durable execution adds a workflow layer that persists progress, retries failed steps, and resumes from checkpoints instead of restarting the whole agent.Pydantic+1

A good architecture looks like this:

                Durable Workflow
              (persistent state machine)
                       |
        +--------------+--------------+
        |              |              |
     LLM call       Tool call     Human approval
    (Activity)     (Activity)       (Signal)
        |              |              |
     retries        retries        waits safely

1. Separate orchestration from side effects

Do not put your entire agent loop in one process:

while True:
    response = llm(messages)
    if response.tool_call:
        result = tool(response.tool_call)
        messages.append(result)
    else:
        return response

If that process dies after step 5, you lose state.

Instead:

• Workflow code: decides what happens next; should be deterministic.

• Activities/steps: do external work:

• LLM calls

• database writes

• API calls

• tool execution

Durable workflow engines such as Temporal use this split: workflows replay saved history, while activities perform non-deterministic I/O.Pydantic

2. Checkpoint after every meaningful step

Persist things like:

{
  "workflow_id": "research-job-123",
  "messages": [...],
  "completed_tools": [
    "search_web",
    "fetch_document"
  ],
  "current_step": "summarize"
}

Avoid only saving the final result. The expensive part of an agent is often the path it took.

Good checkpoints:

• after an LLM response
• after a tool result
• before irreversible actions
• before waiting for humans

3. Put retries around the right boundary

A common mistake:

retry entire agent
    └── call LLM
    └── call database
    └── call email API

This can duplicate work.

Prefer:

workflow
 ├── LLM activity       retry 5 times
 ├── Search activity    retry 3 times
 ├── Database activity  retry with idempotency
 └── Email activity     carefully controlled

Durable systems typically let you configure retry policies per activity, including maximum attempts and backoff.GitHub

Example policy:

RetryPolicy(
    initial_interval=2 seconds,
    backoff_coefficient=2,
    maximum_interval=60 seconds,
    maximum_attempts=5
)

4. Make every tool call idempotent

Retries mean something may run twice.

Bad:

send_email("invoice paid")

If retried, users may receive duplicates.

Better:

send_email(
    message_id="workflow-123-invoice-email",
    body="invoice paid"
)

The email service checks:

Have I already processed message_id?
    yes -> return previous result
    no  -> send

Idempotency keys are one of the main techniques for making retries safe.AWS Documentation

5. Handle different failure classes differently

Not every error deserves a retry.

Retry

• timeout
• connection reset
• HTTP 429 rate limit
• temporary provider outage

Do not retry

• invalid prompt
• permission denied
• malformed tool arguments
• policy rejection

Example:

try:
    result = call_llm()
except RateLimitError:
    retry_later()
except InvalidRequestError:
    fail_workflow()

For LLM APIs specifically, rate limits often need special handling (for example, respecting retry-after headers), while invalid requests usually should not consume retry budget.Temporal

6. Add heartbeats for long-running tools

Some agent tools take minutes:

• browser automation
• document processing
• code execution
• batch jobs

Have them report progress:

activity started
  |
  ├── downloaded files
  ├── processed 500/1000 records
  └── completed

If the worker dies, the workflow knows whether to retry or resume.

7. Put limits around agent loops

Agents can retry themselves accidentally:

Agent retry
    -> tool retry
        -> SDK retry
            -> workflow retry

Set budgets:

max_agent_steps: 50
max_llm_attempts: 5
max_tool_attempts: 3
max_runtime: 30 minutes
max_cost: $5

Retries should be observable and bounded.

8. Add human-in-the-loop as a durable wait

Instead of:

while not approved:
    sleep(60)

use a durable signal/event:

Workflow:
    generate report
    wait for approval
    continue

The workflow can sleep for days without holding compute.

Practical stack options

Common approaches:

• Temporal Technologies — workflow engine with replay, activities, retries, and long-running execution patterns.Pydantic
• Durable Task frameworks — similar patterns for checkpointing and retries in cloud environments.Microsoft Learn
• Agent frameworks with durable execution integrations — for example, OpenAI Agents SDK integrations with Temporal.GitHub

A production-grade agent usually ends up looking like:

User request
    |
Durable workflow
    |
    +-- Plan step (LLM)
    |
    +-- Tool A (retry + idempotency)
    |
    +-- Tool B (retry + timeout)
    |
    +-- Human approval signal
    |
    +-- Final response

The key design shift is: your agent is no longer a running program; it is a recoverable process whose state and progress are stored outside the worker.

Adding durable execution and robust retries to an AI agent workflow transforms it from a fragile demo into a production-ready system . Because AI agents are stateful, long-running, and prone to external hiccups (rate limits, flaky APIs, or transient LLM provider outages), standard try/catch blocks and fire-and-forget queues fall short.inngest.com+1

To achieve durability, you need state checkpointing, automatic state reconstruction , and isolated step-level retries.docs.temporal.io+1

1. Choose Your Architecture Approach

You can implement durable execution via dedicated general-purpose engines, agent-native frameworks, or serverless developer primitives.

• Dedicated Durable Orchestrators: Engines like Temporal AI Documentation or Inngest treat your entire agent loop as code, automatically persisting every step, variable, and tool call to an event history database. If a server dies mid-execution, another worker picks up precisely where it left off.medium.com
• Agent-Native Frameworks with Checkpointing: Frameworks like LangGraph use built-in memory savers (like Postgres or Redis checkpointers) to save state after every node transition, allowing you to replay or fork execution history.
• Cloud & Serverless Workflows: Platforms like Cloudflare Workflows or AWS Durable Functions allow you to write step-based orchestration directly into serverless functions with native state persistence.

2. Implement Step-by-Step Design Patterns

1. Separate Reasoning from Side Effects (Activities vs. Workflows)
   Never execute non-deterministic LLM calls or external API side-effects (like sending an email or writing to a database) directly inside a raw retry wrapper without recording state. Divide your agent into:
   • The Orchestrator (The Brain): Decides what to do next.
   • Activities / Tools (The Hands): Isolated functions that perform the actual web requests, DB writes, or LLM completions. These should be individually retryable.
2. Configure Intelligent, Budget-Aware Retries
   LLM providers and external APIs fail differently. Standard linear retries will drain your token budget or trigger stricter rate limits. Configure your retries with:
   • Exponential Backoff with Jitter: Increase wait time exponentially (𝑡=b a s e×2 a t t e m p t+j i t t e r ) to prevent thundering herds on rate-limited APIs.
   • Exception Filtering: Retry transient errors (429 Rate Limit, 500 Internal Server Error, timeout exceptions) but fail fast on deterministic errors (400 Bad Request, invalid JSON tool arguments, authentication failure).
   • Max Retry Caps: Limit LLM generation retries to 3-5 attempts before falling back to a smaller model or routing to a human-in-the-loop review queue.
3. Persist State Checkpoints Incrementally
   If an agent takes 10 sequential steps and fails on step 9, you don't want to re-run steps 1 through 8 (and re-pay for those input tokens). Ensure your framework checkpoints the conversation history and tool outputs immediately after every discrete node execution.daily.dev
4. Handle Human-in-the-Loop (HITL) as a Paused State If your agent requires user approval before executing a high-risk tool (e.g., executing code or making a financial transaction), use a durable primitive. The workflow should pause indefinitely without consuming active compute resources, waiting safely for an external webhook or UI interaction to wake it back up.

If you want to tailor this further, tell me:

• What framework or stack are you currently using to build the agent (e.g., LangGraph, raw Python/Asyncio, Temporal, LlamaIndex)?
• Are your failures mostly related to LLM rate limits / timeouts or downstream tool/API failures?

I can provide a concrete code snippet or architecture blueprint for your exact setup.

Adding durable execution and robust retries to a flaky AI agent workflow transforms your system from a fragile script that breaks on a single rate limit (429) or API timeout into a resilient engine that survives infrastructure crashes and transient third-party failures.temporal.io+2

Two primary architectural patterns achieve this: using a native framework checkpointer (like LangGraph with PostgreSQL) or pairing your agent logic with an enterprise durable execution engine (like Temporal).docs.temporal.io+3

1. Choose Your Orchestration Approach

• Framework-Native Checkpointing (e.g., LangGraph + Postgres): Best if your state and agent control loops are self-contained. It saves state transitions to a database after every node execution. If a step fails, you can resume from the last known good state.temporal.io+1
• Durable Execution Engine (e.g., Temporal Integration): Best for production systems requiring guaranteed execution across distributed workers, complex timeouts, and advanced exponential backoff policies for flaky external APIs (OpenAI, Anthropic, or custom tools).medium.com+3

2. Implementation Steps

1. Isolate LLM Calls and Tool Executions into Activities/Nodes
   Separate your pure cognitive/routing logic from side-effect-heavy or flaky operations (network requests, API calls to LLMs, database writes). In a setup like the Temporal LangGraph Integration , flaky API calls are wrapped as activities so they can fail and retry independently without corrupting the broader workflow memory.docs.temporal.io+2
2. Configure Exponential Backoff Retry Policies
   LLM providers frequently throw rate limits (429) and gateway timeouts (504). Configure explicit retry policies with jitter and backoff multipliers rather than standard linear loops:
   • Initial interval: 1 second
   • Backoff coefficient: 2.0 (doubling the wait time each retry)
   • Maximum interval: 60 seconds
   • Maximum attempts: 5 to 10 for transient network/rate-limit errors; fail fast on 400-level client validation errors.medium.com+1

If you want to dive deeper, let me know:

• Which AI orchestration framework (LangGraph, CrewAI, LlamaIndex, or custom Python code) are you currently using?
• What kind of flakiness or errors are hitting you the hardest (rate limits, timeouts, or state corruption mid-run)?

I can provide a tailored code snippet or architectural layout for your specific stack.

The key is to separate orchestration from AI calls. Don’t make your agent process itself responsible for remembering where it was after a crash.

A robust architecture looks like:

             ┌──────────────────────┐
             │ Durable orchestrator │
             │ workflow/state machine│
             └──────────┬───────────┘
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
     fetch context   call LLM     execute tool
          │             │             │
       retryable      retryable     retryable
        activity      activity       activity
          │             │             │
          └─────────────┼─────────────┘
                        ▼
                  persist result
                        │
                        ▼
                 next agent step

A system such as Temporal is designed specifically for this: workflows can resume after crashes, network failures, or infrastructure outages rather than disappearing.Temporal

1. Make each meaningful step durable

Instead of:

result = agent.run()

model the agent as a sequence of durable steps:

state = {
    "goal": goal,
    "messages": [],
    "plan": None,
    "results": {},
}

state["context"] = await fetch_context()
state["plan"] = await generate_plan(state)
state["research"] = await research(state)
state["answer"] = await generate_answer(state)

return state["answer"]

The important part is that each external/expensive operation is independently recoverable.

If research() fails, you don't want to regenerate the plan and call the LLM again unnecessarily.

2. Put unreliable operations behind retries

Use different retry policies for different failure classes:

retry_policy = {
    "initial_interval": 1,
    "backoff_coefficient": 2,
    "maximum_interval": 30,
    "maximum_attempts": 5,
}

Conceptually:

attempt 1 → wait 1s
attempt 2 → wait 2s
attempt 3 → wait 4s
attempt 4 → wait 8s
attempt 5 → fail permanently

Use retries for things like:

• transient HTTP 5xx errors
• rate limits
• temporary provider outages
• network timeouts
• temporary database failures

Don't blindly retry:

• invalid API requests
• malformed tool arguments
• authentication failures
• deterministic application bugs
• an LLM response that violates a business invariant

Those should generally fail fast or go through a corrective path.

3. Make retries idempotent

This is the part people often miss.

Suppose the agent does:

charge_credit_card()

and the request times out.

You don't know whether the charge happened.

A retry could charge the customer twice.

Instead, give the operation an idempotency key:

charge_credit_card(
    customer_id=customer_id,
    amount=amount,
    idempotency_key=f"{workflow_id}:charge"
)

Your external service/database should treat repeated requests with the same key as the same operation.

The same principle applies to:

• sending emails
• creating tickets
• submitting jobs
• writing records
• calling external APIs
• executing agent tools

4. Persist agent state, not just the final answer

A durable agent should have checkpoints resembling:

{
  "workflow_id": "abc123",
  "status": "researching",
  "goal": "...",
  "plan": ["...", "..."],
  "completed_steps": [
    "retrieve_customer",
    "search_documents"
  ],
  "current_step": "call_llm",
  "attempt": 2
}

That gives you recovery semantics like:

workflow starts
      ↓
retrieve customer ✓
      ↓
search documents ✓
      ↓
LLM call ✗
      ↓
worker crashes
      ↓
worker restarts
      ↓
resume at LLM call

rather than:

worker crashes
      ↓
start entire agent again
      ↓
repeat everything

5. Treat the LLM as an unreliable dependency

Don't let the model implicitly control your entire execution lifecycle.

I'd structure an agent roughly like:

while not finished:

    observation = load_state()

    decision = llm_decide(
        goal=observation.goal,
        history=observation.history,
        available_tools=observation.tools,
    )

    validate(decision)

    if decision.type == "tool":
        result = execute_tool(
            decision.tool,
            decision.arguments,
            idempotency_key=make_key(observation, decision),
        )
        persist_observation(result)

    elif decision.type == "final":
        validate_final_answer(decision)
        persist_final_answer(decision)
        return decision.answer

The orchestrator owns control flow. The LLM proposes the next action.

That distinction makes retries, limits, observability, and recovery much easier.

6. Put bounds around agent retries

You don't want:

LLM → tool failure → LLM → tool failure → LLM → ...

Use explicit budgets:

MAX_STEPS = 20
MAX_TOOL_RETRIES = 3
MAX_LLM_RETRIES = 3
MAX_RUNTIME = timedelta(minutes=10)

And distinguish:

retry same operation
        vs.
ask the agent to reconsider
        vs.
terminate workflow

For example, a 429 from an API might warrant an infrastructure retry, while "tool argument 'customer_id' is invalid" should probably send a structured error back to the agent so it can correct its plan.

7. Add deterministic workflow logic around nondeterministic AI

A useful rule is:

Durable workflow code should be deterministic; nondeterministic/external work should happen in activities/tasks. So instead of putting arbitrary LLM calls, network requests, and current-time lookups directly into the durable workflow, isolate them behind activity boundaries.

That is one of the reasons workflow engines like Temporal work well for AI agents: the orchestration layer can reliably replay/recover while the actual AI/tool calls happen as externally managed activities.Temporal

A practical production recipe

If I were hardening a flaky agent today, I'd implement these in roughly this order:

1. Give every run a durable workflow ID.
2. Break the agent into discrete activities such as retrieval, LLM calls, tool calls, and persistence.
3. Checkpoint after every meaningful state transition.
4. Add exponential backoff + jitter to transient failures.
5. Make side-effecting tools idempotent.
6. Classify errors into retryable, agent-correctable, and fatal.
7. Put hard limits on steps, retries, tokens, and wall-clock time.
8. Record every attempt with workflow ID, step ID, tool, model, latency, error, and attempt number.
9. Add human escalation for workflows that repeatedly fail.
10. Test recovery explicitly by killing workers halfway through workflows.

For a greenfield implementation, I'd seriously consider Temporal for the durable execution layer; its core promise is crash-resumable execution, including for AI-agent workloads.Temporal

If you're already using LangGraph, Python/asyncio, Node, or a queue like Celery, the implementation details differ substantially. I can show a concrete durable/retryable architecture for whichever stack you're using.

Yes. The key is to treat your agent as a durable state machine, not as one long function call.

A robust architecture looks like:

Request
  │
  ▼
Durable workflow
  ├── Load state
  ├── Plan
  ├── Call tool/API ── retryable
  ├── Run LLM step ── retryable
  ├── Validate result
  ├── Persist checkpoint
  └── Continue / compensate / fail

Platforms such as Temporal are designed specifically to resume workflows after crashes, network failures, and infrastructure outages rather than restarting from scratch.Temporal

1. Separate orchestration from side effects

Make your workflow deterministic and put unreliable operations into activities/tasks:

def agent_workflow(request):
    state = load_state(request)

    plan = plan_task(request, state)

    for step in plan:
        result = execute_task(step)   # retryable activity
        state = update_state(state, result)
        save_checkpoint(state)

    return state

Conceptually:

• Workflow: decides what should happen next.
• Activity/task: performs network calls, LLM calls, database writes, tool calls, etc.
• Checkpoint/state: records enough information that execution can resume safely.

Don't put a raw while True retry loop around the entire agent. You want to retry the failed operation, not repeat everything that succeeded before it.

2. Give every operation an idempotency key

Retries create the biggest danger when an operation succeeds but your process crashes before receiving the response.

For example:

attempt 1:
  charge_card()
  <-- payment succeeds
  <-- process crashes before response arrives

attempt 2:
  charge_card()
  <-- potentially charges twice

Instead:

charge_card(
    amount=100,
    idempotency_key=f"{workflow_id}:charge"
)

Your downstream service should treat repeated requests with the same key as the same operation.

Do this especially for:

• payments
• emails/messages
• ticket creation
• database mutations
• external API calls
• tool calls that change the world

3. Use different retry policies for different failures

Don't blindly retry every exception.

A useful policy is:

For example:

retry_policy = {
    "initial_delay": 1,
    "backoff": 2,
    "max_delay": 60,
    "max_attempts": 5,
}

Add jitter so thousands of agents don't retry simultaneously.

4. Checkpoint after meaningful transitions

Suppose your agent does:

1. Research
2. Analyze
3. Generate answer
4. Send email

You don't want an outage during step 4 to cause steps 1–3 to run again.

Persist something like:

{
  "workflow_id": "abc123",
  "completed": [
    "research",
    "analysis",
    "generation"
  ],
  "answer": "...",
  "next_step": "send_email"
}

Then recovery becomes:

crash
  ↓
resume workflow
  ↓
see "send_email" is next
  ↓
retry send_email

This is the central benefit of durable execution: the workflow can resume from its persisted progress rather than relying on in-memory state.Temporal

5. Put a budget around agent retries

AI agents have another failure mode: semantic loops.

For example:

call tool
→ bad result
→ retry
→ bad result
→ retry
→ retry
→ ...

Use multiple limits:

MAX_TOOL_ATTEMPTS = 3
MAX_AGENT_STEPS = 20
MAX_RUNTIME = 10 * 60
MAX_COST = 2.00

Then distinguish:

technical retry
    ↓
same operation again

semantic failure
    ↓
re-plan / ask another model / escalate

budget exhausted
    ↓
persist failure + stop

6. Make LLM steps reproducible enough to debug

Store the important inputs/outputs around each model invocation:

{
  "step": "research",
  "model": "…",
  "prompt_version": "research-v7",
  "input": "...",
  "output": "...",
  "attempt": 2,
  "timestamp": "...",
  "tool_calls": ["search", "database"],
  "status": "success"
}

You generally don't need to persist every transient token-level detail, but you should be able to answer:

"What did the agent believe, what did it attempt, and why did it move to the next state?"

7. Consider a workflow engine once this gets serious

If your workflow needs to survive process/container/VM crashes and resume days later, don't reinvent durable execution with a database plus a bunch of polling workers.

A system such as Temporal provides durable workflow execution and lets activities be retried independently. Its documentation explicitly positions this model for AI-agent workflows as well as traditional business processes.Temporal

A good production architecture is therefore:

                  ┌───────────────┐
User request ───► │ Durable       │
                  │ workflow      │
                  └───────┬───────┘
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
         LLM task      Tool task    DB task
             │            │            │
             └──── retry ─┴── retry ───┘
                          │
                          ▼
                    Persisted state
                          │
                     crash/restart
                          │
                          ▼
                    Resume workflow

The practical rule

Retry operations, checkpoint state, make side effects idempotent, and put a bounded state machine around the LLM.

If you're building this in Python/TypeScript, I can also show a concrete implementation using Temporal, LangGraph, or a lightweight Postgres-based approach.

To add durable execution and retries to a flaky AI agent workflow, adopt a durable workflow orchestration engine (such as Temporal, Inngest , or cloud native durable functions) to checkpoint state and manage retries automatically .reddit.com+3

AI agents fail often due to rate limits, flaky third-party tool APIs, server timeouts, and unpredictable LLM outputs. Traditional try/catch blocks and ad-hoc loops fail to scale or recover state when infrastructure crashes.inngest.com+4

Step-by-Step Implementation Guide

1. Separate Deterministic Orchestration from Non-Deterministic AI Calls
   • Keep your workflow definition deterministic (pure code logic for routing, state, and loops).
   • Isolate LLM generations and tool executions into distinct, isolated "activities" or "steps" that can fail, timeout, and be retried independently.
2. Integrate a Durable Execution Engine
   • Choose an orchestration layer designed for state persistence:
     • Temporal for self-hosted or managed robust code-as-workflow infrastructure.
     • Inngest for serverless and event-driven architectures.
     • AWS Lambda Durable Functions if operating inside the AWS ecosystem.aws.amazon.com+4

Comparison of Durable Execution Options

If you can share which programming language or AI framework (e.g., LangChain, OpenAI Agents SDK, custom scripts) you are using, I can provide a concrete code snippet demonstrating a retry and checkpoint loop.