How do I add durable execution and retries to a flaky AI agent workflow?

Adding durable execution and retries to a flaky AI agent workflow requires separating the probabilistic reasoning (the LLM deciding what to do next) from the deterministic orchestration (managing state, crashing, and retrying network I/O).

Trying to solve infrastructure failures with prompt engineering or basic try/catch blocks inevitably fails. Instead, production architectures rely on a clear Brain vs. Muscle split: the agent framework handles reasoning, while a durable execution engine or explicit checkpointer manages state and network boundaries.

Step 1: Separate Non-Deterministic I/O into Isolated Units

The golden rule of durable execution is that side effects and external API calls cannot live directly inside volatile orchestration scripts . Wrap every individual LLM call, vector search, MCP (Model Context Protocol) call, and tool execution into an isolated function (an Activity or Task).

• Why: If an LLM call or a third-party API returns a rate-limit error (HTTP 429) or times out, only that specific unit of work retries. The parent workflow remembers that previous steps already completed successfully and does not re-run them (saving tokens and avoiding duplicate side effects like double-charging a credit card).

Step 2: Choose Your Durability Layer

Depending on your stack and complexity tolerance, implement one of these primary architectural patterns:

1. Dedicated Orchestration Engines (e.g., Temporal):
   • Best for complex, long-running agent workflows that need to survive server crashes or pause for hours/days waiting for human approval.
   • You write the agent's ReAct loop as a deterministic Workflow , while LLM generation and tool execution are registered as Activities equipped with custom exponential backoff retry policies. Check out the Temporal Documentation for PydanticAI or Temporal's Durable AI Guide to see this in action.
2. Lightweight Task Orchestrators (e.g., Prefect):
   • Best if you want code-as-you-go ergonomics without heavy infrastructure overhead.
   • You decorate your agent execution functions with @task(retries=3) or use the Prefect Durability Integration to automatically cache and retry failed steps.
3. Graph-Native Checkpointing (e.g., LangGraph Checkpointers):
   • Best if you are already heavily invested in graph-based agent topologies.
   • Configure a persistent checkpointer (PostgresSaver or RedisSaver ) set to sync mode so that the agent's state graph writes a checkpoint to your database after every single node transition.

Step 3: Enforce Strict Tool Idempotency

Retrying a tool call blindly can break things if the tool isn't safe to run twice (e.g., sending an email or executing a database write).

• Pass a deterministic idempotency key (derived from your workflow_id and unique step_id ) into every external mutation tool.
• Design your downstream tool APIs to recognize and deduplicate requests matching that key.

To help tailor this architecture to your specific stack, tell me:

• What agent framework are you currently using (e.g., LangGraph, CrewAI, custom loops)?
• Are your tools read-only (like web searches) or do they perform external mutations (like database updates or sending messages)?

Durable execution and retries solve two different problems:

• Retries: “This step failed; try the step again.”
• Durable execution: “The process died; resume the workflow from the last safe point without repeating completed work.”

For AI agents, you usually need both because workflows combine unreliable external calls (LLMs, APIs, tools) with expensive intermediate state.

1. Break the agent into durable steps

Avoid a single function like:

def run_agent(task):
    plan = llm.plan(task)
    docs = search(plan)
    answer = llm.write(docs)
    send(answer)

If it fails during send(), you may repeat the planning, search, and generation.

Instead, model it as steps:

Receive task
    ↓
Plan
    ↓
Gather information
    ↓
Call tools
    ↓
Generate response
    ↓
Deliver result

Each boundary becomes a checkpoint. A failure after “Gather information” can resume from there.

---

2. Persist workflow state

Store the agent state outside process memory:

AgentState = {
    "task_id": "...",
    "messages": [...],
    "plan": "...",
    "tool_results": [...],
    "status": "running"
}

Persist after meaningful transitions:

save_checkpoint(state)

result = call_llm()

state["draft"] = result
save_checkpoint(state)

Good checkpoint contents:

• conversation history
• tool outputs
• intermediate plans
• completed step IDs
• retry counts
• human approval status

Avoid storing only the final answer; you need enough information to resume.

---

3. Add retries at the step level

Do not retry the entire agent blindly.

Bad:

Agent failed → restart everything

Better:

Tool call failed → retry tool call
LLM timeout → retry LLM step
Payment failed → run compensation flow

Example retry policy:

retry_policy = {
    "max_attempts": 4,
    "initial_delay": 1,
    "backoff": 2,
    "max_delay": 60
}

This produces:

Attempt 1: immediately
Attempt 2: wait 1s
Attempt 3: wait 2s
Attempt 4: wait 4s

Use retries only for transient failures:

Retry:

• network timeout
• HTTP 429 rate limits
• HTTP 5xx errors
• temporary database outage

Do not retry:

• invalid input
• authentication failure
• bad tool arguments
• malformed application logic

Frameworks such as LangGraph support per-node retry policies, timeouts, and recovery handlers for this pattern.

---

4. Make every side effect idempotent

Retries can execute the same operation more than once.

Dangerous:

charge_credit_card()

If the request succeeds but the response is lost, retrying may charge twice.

Use idempotency keys:

charge_credit_card(
    payment_id="order_123_attempt_1"
)

Other examples:

---

5. Separate deterministic orchestration from nondeterministic work

Agent workflows contain things that change:

• LLM outputs
• current time
• random sampling
• external API responses

Those belong in retryable tasks/activities.

The workflow controller should mostly decide:

If step completed:
    continue

If failed:
    retry or compensate

not:

Generate random plan
Call API
Look at current time

Durable workflow engines such as Temporal use this model: workflows are replayed from history while external work runs as retryable activities.

---

6. Add timeouts

Retries without timeouts can leave stuck agents running forever.

Example:

call_llm(
    timeout=30,
    retry_policy={
        "max_attempts": 3
    }
)

Think in layers:

Workflow timeout
    ↓
Step timeout
    ↓
Network timeout
    ↓
Retry policy

---

7. Add recovery paths, not just retries

Some failures need a different route:

Generate answer
      |
      v
Validation failed
      |
      +--> Retry generation
      |
      +--> Ask human
      |
      +--> Use fallback model

Examples:

• LLM cannot parse tool output → ask model to repair
• Missing customer information → pause for user input
• Tool permanently unavailable → use alternate provider

LangGraph describes this as distinguishing transient failures, LLM-recoverable failures, user-fixable failures, and unexpected failures.

---

8. A production architecture often looks like this

                Durable Workflow Engine
                         |
        +----------------+----------------+
        |                |                |
     Planner          Tools           Reviewer
        |                |                |
     Retry             Retry            Retry
        |                |                |
     Checkpoint       Checkpoint       Checkpoint

                         |
                  Persistent State Store

Common implementation choices:

• LangGraph checkpointing + retries for agent-centric workflows.
• Temporal workflows + activities when you need strong durability across worker crashes, long-running jobs, and complex business processes.
• A combination: agent framework for reasoning, durable workflow engine for execution guarantees.

A good rule of thumb: make every expensive or failure-prone action a resumable step, make every step idempotent, and retry only the smallest unit that failed.

                Durable Workflow Engine
                         |
        +----------------+----------------+
        |                |                |
     Planner          Tools           Reviewer
        |                |                |
     Retry             Retry            Retry
        |                |                |
     Checkpoint       Checkpoint       Checkpoint

                         |
                  Persistent State Store

Common implementation choices:

charge_credit_card(
    payment_id="order_123_attempt_1"
)

Other examples:

---

5. Separate deterministic orchestration from nondeterministic work

Agent workflows contain things that change:

• LLM outputs
• current time
• random sampling
• external API responses

Those belong in retryable tasks/activities.

The workflow controller should mostly decide:

Tool call failed → retry tool call
LLM timeout → retry LLM step
Payment failed → run compensation flow

Example retry policy:

retry_policy = {
    "max_attempts": 4,
    "initial_delay": 1,
    "backoff": 2,
    "max_delay": 60
}

This produces:

Attempt 1: immediately
Attempt 2: wait 1s
Attempt 3: wait 2s
Attempt 4: wait 4s

Use retries only for transient failures:

Retry:

• network timeout
• HTTP 429 rate limits
• HTTP 5xx errors
• temporary database outage

Do not retry:

• invalid input
• authentication failure
• bad tool arguments
• malformed application logic

Frameworks such as LangGraph support per-node retry policies, timeouts, and recovery handlers for this pattern.

---

4. Make every side effect idempotent

Retries can execute the same operation more than once.

Dangerous:

The key is to separate orchestration from failure-prone work and make every meaningful step resumable.

A good production architecture looks like:

                 ┌───────────────┐
                 │ Durable       │
                 │ workflow      │
                 └───────┬───────┘
                         │
             ┌───────────▼───────────┐
             │ Agent loop / state     │
             │ persisted after steps │
             └───────┬───────────────┘
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       LLM call    Tool/API   DB/write
       activity    activity    activity
          │          │          │
       retries    retries    idempotency

1. Persist state between agent steps

Don't keep the entire agent run in process memory. Persist:

• conversation / messages
• current step
• tool calls and results
• intermediate outputs
• retry/attempt metadata
• workflow/run ID

Then a worker crash becomes "resume from the last checkpoint", rather than "start the agent over."

For example, LangGraph's checkpointer provides this model: successful work is persisted and a resumed execution doesn't redo completed work. Its sync, async, and exit durability modes let you trade persistence guarantees against overhead.

2. Retry at the smallest useful boundary

Don't blindly retry the entire agent.

Instead:

agent workflow
   ├── call LLM       ← retry
   ├── search API     ← retry
   ├── execute tool   ← retry
   ├── validate       ← usually don't retry
   └── write result   ← carefully retry/idempotency

For transient failures, use bounded exponential backoff + jitter:

RetryPolicy(
    max_attempts=4,
    initial_interval=1.0,
    backoff_factor=2.0,
    max_interval=30.0,
    jitter=True,
)

LangGraph exposes essentially this policy directly, including exception-specific retry rules.

A particularly important distinction is:

429 / rate limit       → retry
502 / 503 / network    → retry
timeout                → retry
invalid API argument   → don't retry
bad model output       → usually recover differently
permission denied      → don't retry

3. Put side effects behind idempotent operations

This is the part people often miss.

Suppose the agent does:

charge_credit_card()

The process crashes after the payment succeeds but before the workflow records success.

A retry could charge the customer twice.

Give external operations an idempotency key:

payment_id = f"{workflow_id}:payment"

charge_card(
    amount=100,
    idempotency_key=payment_id,
)

The same principle applies to:

• sending emails
• creating tickets
• submitting orders
• writing database records
• calling APIs that mutate state

Durable execution means work may be re-executed, so side effects need to tolerate that.

4. Add timeouts as well as retries

A retry without a timeout can leave an agent stuck indefinitely.

Use:

attempt
  │
  ├── succeeds → continue
  │
  └── timeout/error
        │
        ├── retryable → exponential backoff → attempt again
        │
        └── permanent → recovery/fallback

For example:

call_llm(
    timeout=30,
    retries=3,
)

LangGraph explicitly composes per-node timeouts with retry policies; a timed-out attempt can be retried with its own fresh timeout.

5. Separate "infrastructure retry" from "agent recovery"

These are different mechanisms.

Infrastructure retry:

"The API returned 503. Try the exact same operation again."

Agent recovery:

"The search tool returned an error. Tell the model what happened and let it choose another strategy."

For the second case, feed the error back into the agent:

LLM
 ↓
tool call
 ↓
tool fails
 ↓
record error
 ↓
LLM sees error
 ↓
choose alternate tool / fix arguments / abandon task

LangGraph specifically recommends this pattern for errors that the LLM itself can recover from.

6. For long-running agents, use a durable workflow engine

If your agent might run for minutes, hours, or days—or needs human approval—I'd strongly consider a workflow engine rather than building persistence/recovery yourself.

Temporal is a strong option. Its model is roughly:

Workflow = deterministic orchestration
Activity = failure-prone external work

LLM calls and tool calls become Activities, while the agent loop remains in the durable Workflow. Temporal automatically retries Activities according to retry policies and replays completed work rather than executing it again after a worker restart.

The OpenAI Agents SDK also has a Temporal integration specifically for durable, long-running agents.

A practical recipe

If you're starting with an existing flaky agent, I'd implement these in this order:

1. Give every run a durable ID.
2. Persist state after every meaningful agent step.
3. Move LLM/tool/API calls into retryable units.
4. Add 20–60s timeouts to external calls.
5. Retry only transient errors.
6. Use exponential backoff + jitter + maximum attempts.
7. Make every side effect idempotent.
8. Add explicit fallback/recovery paths for non-transient agent errors.
9. Record attempt count, error type, latency, and workflow ID for observability.
10. Test by deliberately killing workers halfway through runs.

If you're already using LangGraph, its built-in checkpointer + RetryPolicy + timeouts are probably the shortest path. If you're building a long-running production agent where crash recovery is a first-class requirement, Temporal is worth evaluating. LangGraph and Temporal can also be combined rather than treated as mutually exclusive choices.

Adding durable execution and retries to an AI agent workflow transforms it from a fragile script into a resilient, production-ready system. AI agents are notoriously flaky due to network timeouts, rate limits, API outages, and non-deterministic LLM responses.

1. Choose a Durable Execution Framework

Writing custom state-machines and retry loops in standard application code quickly becomes an unmaintainable mess. Instead, leverage established orchestration engines that handle state persistence, timeouts, and retries natively.

• Temporal : The industry standard for durable execution. It treats your workflow as code, automatically persisting state after every step (step₁ → step₂) so that if a server crashes mid-agent-thought, it resumes exactly where it left off.
• Inngest : A developer-friendly, serverless-native workflow engine that handles retries, step-by-step execution, and event-driven architectures without requiring a dedicated state cluster.
• LangGraph (with Persistence) : If you are building within the LangChain ecosystem, LangGraph allows you to save agent state to a database (like Postgres or Redis) after every node execution, enabling time-travel and resumption upon failure.
• Prefect or Dagster : Excellent if your AI agent workflow leans heavily into data pipelines, RAG ingestion, or batch processing steps.

2. Implement Granular, Step-Level Retries

Do not wrap your entire agent loop in a single massive try/catch block. Break your agent workflow into distinct, atomic steps and apply targeted retry policies to each:

• LLM Calls (High Flakiness) : Configure exponential backoff with jitter to handle rate limits (HTTP 429) and transient server errors (HTTP 500, 503).
• Tool Execution / External APIs : Wrap web searches, database lookups, or API calls in independent retry wrappers with strict timeouts (T ≤ 30 seconds) so a hanging API doesn't freeze the whole agent.
• Idempotency : Ensure that every step (especially tool calls like writing to a database or sending an email) is idempotent . If a step fails halfway and retries, running it twice should not result in duplicate side effects.

3. Design for Non-Deterministic Failures (Self-Correction)

Retrying a broken API call is easy, but retrying when an LLM outputs malformed JSON or hallucinates a tool argument requires agent-specific logic:

• Structured Output Validation : Use libraries like Pydantic or Instructor to validate LLM outputs against a rigid schema. If validation fails, catch the validation error, feed the exact schema mismatch error back to the LLM as a prompt correction, and retry the generation step.
• Circuit Breakers : If an external tool or LLM provider fails N consecutive times (e.g., N = 5), trip a circuit breaker to halt the workflow and alert a human or fallback to a cheaper/more stable model rather than burning through API rate limits.

4. Persist External Memory and State

Durable execution relies on externalizing state away from the volatile process memory of the agent runner:

• State Checkpointing : Save the full agent conversation history, intermediate tool outputs, and variable states to an ACID-compliant database after each turn.
• Human-in-the-Loop Interruption : When an agent hits a low-confidence threshold or encounters an unrecoverable tool error, use durable pause/resume primitives to wait for human intervention without losing context.

If you want, tell me:

• What stack/language you are using (Python, TypeScript, etc.)
• Where you are hosting your agent (AWS, serverless, Kubernetes)

I can recommend the exact framework or code pattern that fits your architecture.

The key is to treat the agent as a durable state machine, not as one big function with a retry around it.

A robust architecture looks like:

                 ┌──────────────┐
                 │ Agent state  │
                 │ + checkpoint │
                 └──────┬───────┘
                        │
                  ┌─────▼─────┐
                  │ Plan/LLM  │◄── retry
                  └─────┬─────┘
                        │
                  ┌─────▼─────┐
                  │ Tool call │◄── timeout + retry
                  └─────┬─────┘
                        │
                    checkpoint
                        │
                  ┌─────▼─────┐
                  │ Next step │
                  └───────────┘

1. Break the agent into durable steps

Don't do:

def run_agent():
    response = llm(...)
    result = tool_a(response)
    response = llm(result)
    result = tool_b(response)
    return response

Instead, make each meaningful operation a separately recoverable step:

state = {
    "messages": ...,
    "tool_results": ...,
    "status": "running",
}

llm_result = durable_step("reason", call_llm, state)
state = checkpoint(state, llm_result)

tool_result = durable_step("tool", run_tool, state)
state = checkpoint(state, tool_result)

...

If the process dies during tool_b, you want to resume from the last durable boundary—not pay for llm() and tool_a() again.

Frameworks such as LangChain's LangGraph implement this with checkpoints; successful work from a failed super-step can be preserved so it doesn't have to be recomputed.

2. Retry only failures that are actually transient

A good default policy is:

Use exponential backoff + jitter, e.g.:

attempt 1 → immediately
attempt 2 → ~1s
attempt 3 → ~2s
attempt 4 → ~4s

with randomness added so thousands of agents don't retry simultaneously.

LangGraph's current retry policy, for example, supports max attempts, exponential backoff, maximum interval, jitter, and custom exception filtering.

3. Put timeouts around every external operation

Retries without timeouts can make a bad dependency hang your entire workflow.

Use separate limits for:

LLM call       → 30–120s
normal API     → 5–30s
database       → 5–15s
long-running tool → explicit heartbeat/progress timeout

Then combine:

retry(
    timeout(call_llm, 60),
    max_attempts=4,
    backoff="exponential",
    jitter=True,
)

The important detail is that each retry gets its own timeout.

4. Make tool calls idempotent

This is the biggest gotcha.

Suppose the agent says:

charge_customer($100)

The request reaches Stripe, succeeds, and then your worker crashes before recording the result.

On retry, blindly calling the tool again could charge the customer twice.

Give externally visible operations an idempotency key:

run_id = "agent-run-8472"
step_id = "charge-customer"
idempotency_key = f"{run_id}:{step_id}"

charge_customer(
    amount=100,
    idempotency_key=idempotency_key,
)

For non-idempotent operations, use one of:

• idempotency keys
• transactional/outbox patterns
• a durable tool-result store
• explicit compensation/rollback
• human approval before the irreversible operation

Retries solve transient failure; idempotency solves duplicate execution. You need both.

5. Persist state outside the worker process

Don't rely on:

agent_memory = {}

or an in-memory queue.

Persist at least:

run_id
current_state
completed_steps
tool results
attempt counts
timestamps
errors
workflow status

Then a worker can disappear and another worker can pick up the same run.

If you're using LangGraph, a checkpointer provides this state persistence and associates execution with a thread_id.

For a framework-agnostic durable execution layer, Temporal is another strong option: its workflow history lets an agent resume after worker crashes, with completed LLM/tool activities replayed from durable history rather than executed again.

6. Separate workflow logic from side effects

A useful mental model is:

Workflow
  ├── decide what should happen
  ├── record state
  ├── schedule activity
  └── react to result

Activity
  ├── call LLM
  ├── call API
  ├── query DB
  └── send email

The workflow should be deterministic and durable; side effects belong in activities/tasks that have explicit retry and timeout policies.

Temporal's current AI-agent architecture uses essentially this split: the workflow orchestrates the agent loop while LLM and tool calls are durable activities.

7. Have an explicit terminal failure path

Don't retry forever.

                    transient?
                   /           \
                 yes            no
                  │              │
               retry          fail/handle
                  │
              attempts
              exhausted?
              /       \
            no         yes
            │           │
          retry     fallback
                        │
                  human / dead-letter

After, say, 3–5 attempts:

• switch to a fallback model/tool,
• ask the user for clarification,
• request human intervention,
• compensate a partial transaction,
• or put the run in a dead-letter/recovery queue.

LangGraph, for example, supports a node-level error handler that runs after retries are exhausted.

A practical production recipe

If you're building this today, I'd use:

                    Durable workflow
                           │
                 ┌─────────▼─────────┐
                 │ checkpointed state │
                 └─────────┬─────────┘
                           │
                    ┌──────▼──────┐
                    │ LLM activity│
                    │ retry 3–4x  │
                    │ timeout     │
                    └──────┬──────┘
                           │
                    checkpoint
                           │
                    ┌──────▼──────┐
                    │ Tool activity│
                    │ retry +      │
                    │ idempotency  │
                    └──────┬──────┘
                           │
                    checkpoint
                           │
                       next step

If you're already using LangGraph, its checkpointer + RetryPolicy + timeout + error-handler primitives give you most of this directly.

If you need very long-running workflows, strong guarantees around external side effects, or infrastructure-level durability, I'd lean toward Temporal and put your agent loop inside a Temporal workflow. Google's current Gemini/Temporal example uses exactly this pattern for durable agent execution.

The most important rule is: never put a single retry around the entire agent. Retry and checkpoint at the smallest meaningful unit of work, and make every side effect safe to repeat.

Adding durable execution and automatic retries to an AI agent workflow transforms it from a fragile prototype into a production-ready system. AI workflows are uniquely flaky due to rate limits, context-window timeouts, non-deterministic model outputs, and third-party API outages.

Here is how to implement durable execution and smart retries for your AI agent:

• Adopt a Orchestration Framework : Instead of writing custom retry loops and state storage in raw Python/Node.js, use a dedicated durable execution engine. These frameworks automatically persist the state of your agent's execution step-by-step.
  • Temporal : The industry standard for general-purpose durable execution. It treats your agent workflows as code, automatically retrying failed activities and persisting state to a database. You can explore the Temporal Documentation to get started.
  • Inngest : Great for serverless and event-driven architectures, offering simple step functions, reliable retries, and background jobs. Check out Inngest for easy integration.
  • LangGraph : Specifically designed for stateful, multi-actor AI applications, offering built-in persistence (checkpoints) to pause, resume, or replay agent steps. Visit the LangGraph Documentation to learn more.
  • Prefect / Dagster : Excellent if your AI agent workflow doubles as a data pipeline or sequential LLM chain.
• Isolate LLM Calls into Discrete "Activities" or "Steps" : Never wrap your entire agent loop in a single retry block. Break the workflow down into atomic operations:
  • Fetching data / RAG retrieval
  • The actual LLM completion call
  • Tool execution (e.g., executing a SQL query or calling an external API)
  • State evaluation / reflection
• Implement Semantic and Exponential Backoff Retries : Standard linear retries will hammer an already-overloaded LLM provider (like OpenAI or Anthropic hitting 429 Rate Limit errors).
  • Use exponential backoff with jitter (randomized delay) so concurrent agent threads don't synchronized-retry and re-trigger rate limits.
  • Catch specific errors (e.g., 429, 503, or JSON parsing failures) to trigger retries, while failing fast on unrecoverable errors (e.g., 400 Bad Request or invalid API keys).
• Enforce Output Validation with Self-Correction Retries : AI agents frequently output malformed JSON or fail tool-calling schemas. Wrap your LLM parsing steps in a validation layer (like Pydantic). If validation fails, feed the validation error back to the agent as a prompt correction within the same retry cycle rather than just throwing a hard exception.
• Persist State Passports (Checkpoints) : Ensure that when an agent calls an external tool and succeeds, that intermediate result is saved immediately. If the workflow crashes right before the next LLM call, the system should resume from the last successful checkpoint instead of re-running expensive or side-effect-heavy tool calls (like sending an email or charging a payment).

If you'd like to dive deeper, let me know:

• What framework or language are you currently building your agent in?
• Are you running into specific issues like rate limits (429), bad JSON outputs , or server timeouts?

I can provide a code snippet or architectural pattern tailored to your stack.

The key is to separate agent reasoning from durable orchestration. Don’t make the whole agent process one retryable function; make each meaningful LLM/tool step a checkpointed unit of work.

A good production architecture looks like:

             Durable Workflow
                    │
        ┌───────────┴───────────┐
        │                       │
   LLM activity            Tool activity
        │                       │
   retry/backoff             retry/backoff
        │                       │
   checkpoint result       checkpoint result
        └───────────┬───────────┘
                    │
             next agent step

Systems such as Temporal explicitly support this pattern: the workflow survives process crashes, while individual LLM/tool calls are durable activities that can retry independently.

1. Checkpoint after every meaningful step

Instead of:

def run_agent(task):
    plan = llm(task)
    data = search(plan)
    answer = llm(data)
    send_email(answer)

use something conceptually like:

def workflow(task):
    plan = durable_step("plan", lambda: call_llm(task))
    data = durable_step("search", lambda: search(plan))
    answer = durable_step("answer", lambda: call_llm(data))
    durable_step("send_email", lambda: send_email(answer))

    return answer

If answer fails, you want the workflow to resume from answer, not call the planner and search again.

That distinction is the heart of durable execution. AWS's durable execution SDK, for example, checkpoints after operations and can resume from the last completed operation after an environment failure.

2. Retry only transient failures

Don't blindly retry everything.

A useful policy is:

Use exponential backoff + jitter, e.g.:

1s → 2s → 4s → 8s → 16s

with a maximum delay and attempt limit. The OpenAI Agents SDK currently exposes configurable retry policies, including maximum retries, backoff, jitter, and provider-specific retry advice.

3. Make tools idempotent

This is arguably more important than retries.

Suppose your agent does:

charge_customer($100)

The request succeeds, but your worker crashes before recording the result.

A retry could charge the customer twice.

Instead give the operation an idempotency key:

charge_customer(
    amount=100,
    idempotency_key=f"{workflow_id}:charge"
)

The downstream service should treat repeated requests with that key as the same operation.

For side effects where you can't get true idempotency, use a durable state machine:

PENDING
   ↓
EXECUTING
   ↓
COMPLETED

and reconcile ambiguous EXECUTING states rather than blindly repeating the action.

4. Keep workflow code deterministic

A durable workflow may be replayed to reconstruct its state. Therefore, avoid doing arbitrary nondeterministic work directly inside the workflow:

# Bad inside workflow
random.random()
datetime.now()
requests.get(...)
llm(...)

Instead:

# Workflow
result = execute_activity(call_llm, prompt)

The external call happens in the activity; its result becomes part of the durable execution history.

This is one reason the Temporal pattern maps nicely to agents: the workflow contains orchestration, while LLM calls and tools are activities. Google's current Temporal agent example uses exactly this architecture.

5. Put budgets around the agent

Retries can turn a flaky workflow into an expensive infinite loop.

Track at least:

max workflow duration
max agent turns
max LLM calls
max tool calls
max retry attempts / step
max token budget
max dollar budget

For example:

if state.llm_calls >= 30:
    raise AgentBudgetExceeded()

if state.elapsed > timedelta(minutes=20):
    raise WorkflowTimeout()

Also distinguish retry attempts from agent attempts. A transient HTTP failure shouldn't consume an entire reasoning turn.

6. Treat external side effects as a separate reliability boundary

A robust agent often looks like:

LLM: decide what to do
       ↓
Tool: validate intended action
       ↓
Durable checkpoint
       ↓
External side effect
       ↓
Durable checkpoint
       ↓
LLM: continue

For dangerous or irreversible operations, add an approval gate:

agent proposes action
       ↓
validation
       ↓
human approval
       ↓
execute
       ↓
record result

Durable runtimes are particularly useful here because the workflow can sit paused for hours or days without losing its state. The OpenAI Agents SDK documents integrations with Temporal, Dapr, Restate, and DBOS specifically for long-running and human-in-the-loop agent workflows.

7. Make the execution observable

Give every workflow a stable:

workflow_id
run_id
step_id
attempt
agent_version
model
prompt/version
tool/version

Then emit structured events:

{
  "workflow_id": "wf_123",
  "step": "search_customer",
  "attempt": 2,
  "status": "retrying",
  "error": "timeout"
}

You want to be able to answer:

"Why did this agent take 14 minutes and call the model 23 times?"

rather than merely seeing "workflow failed."

The OpenAI Agents SDK has built-in tracing for agent runs, LLM generations, tool calls, handoffs, guardrails, and custom events.

A practical stack

If you're building this today, I'd consider:

• Temporal — strongest general-purpose choice when you need sophisticated durable workflows, retries, timers, signals, and long-running execution.
• AWS Durable Execution SDK — attractive if your infrastructure is already heavily AWS/Lambda-oriented.
• OpenAI Agents SDK + a durable runtime — useful if you're already using the Agents SDK; its current documentation explicitly supports durable integrations.

The important architectural rule is:

Persist the result of every meaningful unit of work, retry the unit—not the entire agent, and make every externally visible side effect idempotent.

That turns a flaky agent from "start over when something breaks" into "resume from the last known-good state."

Adding durable execution and retries to an AI agent workflow transforms it from a fragile script that fails completely on a network timeout or crash into a resilient system that can self-heal, resume from checkpoints, and wait safely for human input.

An effective transition requires tackling the problem across four distinct structural layers:

1. Implement Error-Returning Tools (Instead of Crashing)

By default, when an agent tool (like an API or database query) fails, it throws an exception that bubbles up and kills the agent loop.

• The Fix: Wrap your tool execution blocks in try/except statements and return the failure message as structured data/text back to the model.
• Why it matters: This lets the agent "see" that the tool failed, reflect on the error (e.g., "Ah, I passed the wrong date format, I'll try again" ), and dynamically self-correct without crashing the pipeline.

2. Make Your Tools Idempotent

Retrying a failed step is dangerous if the first attempt partially succeeded. If an agent tries to charge a credit card, send an email, or write a database record, a timeout might mean the action actually happened on the server even though your code threw an error.

• The Fix: Use idempotency keys (like unique transaction or request UUIDs) for any external side-effecting API. Maintain a lightweight deduplication table for critical operations like messaging or billing.
• Why it matters: Retrying a safe, idempotent tool guarantees that running it twice has the exact same net effect as running it once.

3. Separate Decisions from Execution (State Checkpointing)

Re-running an entire multi-step agent sequence from scratch because a single LLM call timed out wastes time and incurs massive token costs.

• The Fix: Store conversation history, intermediate reasoning steps, and tool outputs to an external persistent store (such as Postgres or Redis) after every node or distinct decision boundary.
• How to implement:
  • For native agent graphs, frameworks like LangGraph use built-in checkpointers to save thread states.
  • For lightweight custom setups, save state snapshots manually or use agent-friendly orchestration extensions.

4. Adopt a Production-Grade Durable Execution Runtime

If your agents run for longer than 30 seconds, execute side-effects, or require human approval gates that span hours or days, custom code for retries will eventually break. At this stage, you should rely on a dedicated orchestration layer:

• Temporal: Offers robust orchestration where agent graphs run as workflows and distinct LLM/tool calls execute as activities with native exponential backoff and timeouts. Temporal features a dedicated LangGraph Plugin to run graph checkpoints natively.
• Inngest: Provides step memoization, durable sleeps, and automatic retries without forcing a full deterministic replay model.
• Cloud Runtimes / Managed Services: Options like Cloudflare Workflows or Microsoft's Durable Task for AI Agents offer built-in suspension, retry, and resume primitives on serverless infrastructure.

If you'd like to narrow down the best path, tell me:

• What framework are you currently using to build your agent (e.g., LangGraph, CrewAI, plain Python, OpenAI SDK)?
• Are these long-running tasks (minutes/hours) or quick execution loops?

I can provide a tailored code architecture or snippet for your specific stack.

To fix a flaky AI agent workflow, you must implement durable execution and transient error retries so your system saves its state after every step and automatically recovers from crashes without repeating expensive LLM calls. A standard stateless agent loop fails completely if a network glitch, token rate limit, or server pod eviction occurs mid-run, wasting tokens and duplicating tool operations.

1. Leverage a Durable Execution Framework

Do not write custom try/except while-loops with manual thread sleeps to manage your agent's state. Use a dedicated durable execution runtime that intercepts your code steps, persists their results, and replays them automatically if the infrastructure crashes.

• Temporal : The gold standard for deterministic workflow orchestration. You isolate non-deterministic LLM calls and tool executions into Activities . Temporal caches the results of completed activities; if the worker restarts, it fast-forwards through completed steps and resumes exactly at the failure point.
• Inngest or Restate : Lightweight alternatives focused on step memoization. They persist the output of each function step (step.run ). When a failure occurs, the engine skips already executed blocks and injects the stored results.
• DBOS + Pydantic AI : A Python-native open-source pairing. Wrapping a Pydantic AI agent in a DBOS agent automatically checkpoints your conversation loops, tool invocations, and LLM states to a database.

2. Implement Layered Retry Logic

Isolate your retries into two distinct layers: transient LLM/network failures and structural application logic failures.

• Transient Network & API Layer : Wrap your base LLM client calls using libraries like tenacity in Python. Target transient HTTP status codes (429 Rate Limits, 500/503 Server Errors) and connection timeouts. Configure an exponential backoff with jitter so your agent does not overload the model provider.
• Workflow Engine Layer : Let your durable execution backend handle persistent system failures. If a worker goes down or a critical tool fails repeatedly, the framework keeps the workflow state alive for hours or days until the environment recovers.

3. Checkpoint the Message History and Agent State

If you are building custom durability, you must establish checkpointing boundaries.

• State Boundaries : Save the complete system state and conversation history database immediately after every completed LLM response and every tool execution.
• Recovery Mechanism : When a workflow initializes, check for an existing Session ID or Run ID . If found, hydrate the agent's memory with the last saved state checkpoint instead of starting a blank conversation.

4. Enforce Idempotency and Safe Tool Design

Retrying a failed agent step can cause unintended double-actions (e.g., charging a card twice or sending duplicate emails).

• Idempotency Keys : Generate a unique, deterministic ID for every external action your agent takes (e.g., hash(session_id + step_number) ). Pass this key to downstream APIs so they ignore duplicate requests if a step is retried.
• Error-Returning Tools : Wrap all tool actions in internal try/except blocks. Instead of letting a tool crash your entire application, catch the error and return it to the LLM as structured text data (e.g., {"status": "error", "message": "Database timeout"} ) so the agent can reason through how to correct its path.

5. Protect with Circuit Breakers and Guardrails

Prevent infinite loops where an agent repeatedly retries a broken step and wastes your API token budget.

• Consecutive Failures : Implement a circuit breaker that halts execution and pages a developer if a specific tool or API fails 3 to 5 times consecutively with the exact same error.
• Max Token / Loop Limits : Hardcode an absolute threshold for the maximum number of turns or tokens allowed per agent run to ensure an agent cannot run infinitely in the background.

To tailor this setup to your stack, could you share which programming language you are using, your current agent framework (like LangChain, CrewAI, Pydantic AI, or custom), and what specific tools or APIs are causing the flakiness?

The key idea is to separate reasoning from execution. Your LLM can decide what to do next, but a durable workflow engine should own when, whether, and how often each step executes.

For a flaky AI agent, the architecture typically looks like this:

User request
      │
      ▼
AI agent (planning/reasoning)
      │
      ▼
Workflow engine
      │
 ┌────┴────┐
 │         │
LLM step  Tool/API step
 │         │
 ▼         ▼
Checkpoint after every durable step

This lets the workflow survive crashes, process restarts, network failures, or timeouts without restarting from the beginning.

1. Checkpoint after every meaningful step

Instead of treating the agent as one long function, break it into small durable steps:

• Retrieve context
• Generate plan
• Call search API
• Analyze results
• Update CRM
• Send email

Persist the output of each completed step.

If the process dies after step 4, the workflow resumes at step 5 instead of rerunning everything. Durable workflow systems accomplish this by replaying workflow state while reusing previously committed results.

---

2. Retry only transient failures

Not every error should be retried.

Good retry candidates:

• HTTP 429
• HTTP 503
• temporary DNS failure
• timeout
• connection reset

Poor retry candidates:

• authentication failure
• invalid request
• malformed prompt
• missing resource

A common retry policy is:

attempt 1
wait 1s

attempt 2
wait 2s

attempt 3
wait 4s

attempt 4
wait 8s

give up

Use exponential backoff with random jitter to avoid many workers retrying simultaneously. Durable workflow systems often persist the retry schedule so waiting doesn't consume compute resources.

---

3. Make side effects idempotent

Retries mean the same operation may run more than once.

Bad:

charge_credit_card()

Better:

charge_credit_card(
    amount=50,
    idempotency_key=workflow_id + "-payment"
)

If the retry happens after the payment already succeeded, the payment provider returns the original result instead of charging twice.

The same idea applies to:

• emails
• Slack messages
• CRM updates
• database writes

Use:

• idempotency keys
• upserts
• unique constraints
• conditional writes

rather than assuming exactly-once execution.

---

4. Keep orchestration deterministic

Workflow code should avoid non-deterministic behavior during replay.

Avoid inside the orchestration logic:

• random numbers
• current timestamps
• mutable globals
• direct network requests

Instead:

• wrap external calls in activities/tasks
• record results
• replay recorded outputs

This ensures replay reconstructs the same state instead of making different decisions.

---

5. Separate LLM reasoning from execution

A useful pattern is:

LLM:
"I should update Salesforce."

↓

Workflow:
Execute UpdateSalesforce activity

↓

Activity:
Retry if network fails

↓

Workflow:
Checkpoint success

↓

LLM:
Continue planning

The LLM never owns retry logic for external systems; the workflow engine does.

---

6. Distinguish replay from retry

These concepts are easy to confuse:

• Replay reconstructs workflow state from durable history after a crash. Completed steps are not re-executed.
• Retry reruns a failed activity according to its retry policy.

Keeping them separate avoids duplicated side effects while still recovering from failures.

---

7. Add observability

For every step, record:

• workflow ID
• step name
• attempt number
• duration
• input reference
• output reference
• error type
• retry count

This makes it much easier to answer questions like:

• Which tool is flaky?
• Which prompt fails most often?
• Are retries succeeding?
• Where are workflows spending time?

---

Example

workflow:
    retrieve_documents()

    summary = retry(
        llm.summarize,
        max_attempts=3,
        backoff="exponential"
    )

    retry(
        crm.update_customer,
        idempotency_key=workflow_id,
        max_attempts=5
    )

    retry(
        send_email,
        idempotency_key=workflow_id + ":email"
    )

If the worker crashes after updating the CRM but before sending the email:

• document retrieval is replayed from history
• summarization is replayed from history
• CRM update is not repeated
• workflow resumes at send_email

---

Recommended stack

If you're building production AI agents, common choices include:

• Temporal for long-running, durable workflows with rich retry policies and replay.
• Inngest for event-driven workflows with durable execution.
• AWS Durable Execution SDK for checkpointing, replay, retries, and idempotency within AWS Lambda environments.

The general pattern is the same regardless of the platform: treat the LLM as the planner, keep workflow state durable, make external operations idempotent, and let the workflow engine—not the model—manage retries and recovery.