When an LLM struggles with internal APIs, the issue is rarely just "the model isn't smart enough." It's usually a combination of poor tool definition (ambiguous schemas), lack of self-correction loops , and context bloat (giving the model too many or poorly documented APIs at once).

Instead of picking a bloated autonomous agent framework, the industry standard for robust API integration has shifted toward structured orchestration frameworks with strict validation, or protocol-based layers.

1. The Leading Frameworks for Robust Tool Use

• LangGraph (by LangChain)
  • Why it’s robust: Unlike basic LangChain chains that fail silently or loop endlessly, LangGraph models agents as state machines with explicit cyclic graphs. You define precise control flows, meaning you can enforce a validation step after an API call, route back to the LLM if parameters are malformed, and implement strict human-in-the-loop approval gates before write-operations.
• Model Context Protocol (MCP) (by Anthropic / Open Source)
  • Why it’s robust: MCP is an open standard that decouples the LLM client from the data/API source. Instead of hardcoding messy API schemas into your prompt every time, MCP standardizes how tools, prompts, and resources are exposed to models via secure local or remote servers. It dramatically cuts down schema hallucinations because the tool definitions are isolated, version-controlled, and modular.
• Semantic Kernel (by Microsoft)
  • Why it’s robust: Built heavily with enterprise in mind, it treats APIs as native "plugins" and handles automatic function calling with strict typing. It natively integrates robust logging, telemetry, and security filtering, making it easier to trace why an API call failed in a production environment.
• OpenAI Agents SDK / Low-Level Structured Outputs
  • Why it’s robust: Sometimes the best framework is minimal primitives. Forcing Strict Mode on structured tool/function calling (supported by OpenAI, Anthropic, and Gemini) guarantees that the model's JSON output matches your Pydantic schemas 100% of the time at the API level, eliminating structural malformations before they ever hit your internal API.

2. Architectural Best Practices to Fix Unreliable APIs

Regardless of which framework you choose, your APIs will keep failing if you don't enforce these engineering patterns:

1. Use Pydantic for Strict Schema Validation
   Never let an LLM generate loose JSON arguments. Define every tool parameter using strict Pydantic models with explicit descriptions, field constraints (ge, le, regex ), and clear Enum values for parameters with limited options.
2. Implement a "Two-Tier" Tool Registry
   Don't dump 50 internal API endpoints into the LLM's system prompt. Use a retriever or hierarchical tool approach where the agent first calls a get_api_documentation or has access to only a high-level router tool, reducing the cognitive load on the model.
3. Add a Self-Correction/Reflection Loop
   When your internal API returns a 4xx or 5xx error (e.g., "Missing field X" or "Invalid ID format"), do not crash the agent . Pass the raw error message back to the LLM as the tool output with a prompt injection like: "The API returned this error. Correct your payload and try again." Models are remarkably good at self-correcting their arguments when shown the error response.
4. Isolate Read vs. Write Permissions
   Give your agent robust read-only tools first. For any mutation API (POST, PUT, DELETE), enforce a deterministic validation layer or a human confirmation step.

To help narrow down the right choice, tell me:

• Are you using open-source models (like Llama/DeepSeek) or proprietary models (like Claude/OpenAI)?
• Are the internal APIs mostly read-only or do they perform state-changing writes?

I can give you a tailored architectural pattern or code snippet to lock down your tool reliability.

Yes. The key insight is that “robust API tools for an LLM” is less about picking an agent framework and more about designing a good tool contract + deterministic execution layer.

If I were building this today, I’d use an architecture like:

LLM → small, strongly typed tool interface → deterministic tool adapter → internal API rather than exposing your internal APIs directly to the model.

The framework I’d recommend

For an OpenAI-based agent, my default choice would be OpenAI Agents SDK + strongly typed schemas (Zod/Pydantic) + your own tool-adapter layer. The Agents SDK supports schema validation, structured tool outputs, tool-choice controls, and tracing.

If you need interoperability across different model vendors or want your internal tools to be usable by many agent runtimes, MCP is the better tool protocol. MCP supports explicit input and output schemas and annotations such as read-only/idempotent/destructive behavior.

But I would not simply turn every internal REST/RPC endpoint into an MCP tool.

The most important design principle

Your tool should represent an agent-level operation, not an API endpoint.

Bad:

update_customer(
    customer_id,
    field,
    value,
    ...
)

Better:

change_customer_email(
    customer_id,
    new_email
)

Even better when the workflow is known:

update_customer_contact_info(
    customer_id,
    email,
    phone
)

The model shouldn't have to understand your internal API's quirks, sequencing requirements, IDs, pagination, authentication, retries, etc. Code should handle those things.

OpenAI's current tool guidance explicitly recommends this approach: make invalid states difficult to represent, avoid asking the model for arguments your application already knows, and combine functions that are always used together.

A robust tool should have 5 layers

I'd structure every internal tool roughly like this:

                 ┌────────────────────┐
                 │       Agent        │
                 └─────────┬──────────┘
                           │
                    typed tool call
                           │
                 ┌─────────▼──────────┐
                 │   Tool Contract    │
                 │ JSON Schema/Zod    │
                 └─────────┬──────────┘
                           │
                  validation / auth
                           │
                 ┌─────────▼──────────┐
                 │   Tool Adapter     │
                 │ deterministic code │
                 └─────────┬──────────┘
                           │
                 retries / timeout /
                 idempotency / mapping
                           │
                 ┌─────────▼──────────┐
                 │   Internal API     │
                 └────────────────────┘

The LLM should essentially only control the first arrow.

1. Strict input schemas

Use structured/strict tool calling wherever possible. OpenAI's Structured Outputs can make tool arguments conform to the supplied JSON Schema when strict: true is enabled.

For example:

const CreateRefund = z.object({
  order_id: z.string(),
  reason: z.enum([
    "duplicate_charge",
    "customer_request",
    "fraud",
    "other"
  ]),
  amount_cents: z.number().int().positive()
});

Don't give the model:

amount: string

and hope it figures out whether that's dollars, cents, or "all".

Make the schema encode the business rules.

2. Make invalid states impossible

This is probably the biggest improvement you can make.

Instead of:

{
  "status": "pending | approved | rejected | whatever"
}

use an enum.

Instead of:

{
  "start_date": "...",
  "end_date": "..."
}

where end_date < start_date is possible, validate that in the adapter.

Instead of:

{
  "customer_id": "..."
}

when the customer is already established in the conversation, don't ask the model to supply it.

Your runtime can inject it.

This dramatically reduces the amount of reasoning the model has to do.

3. Give tools semantic outputs too

A common mistake is:

Tool → raw internal API response → LLM

Don't do that.

Instead:

Internal API response
        ↓
adapter
        ↓
small semantic result
        ↓
LLM

For example:

{
  "refund_id": "rf_123",
  "status": "submitted",
  "amount_cents": 4999,
  "estimated_completion": "2026-09-12"
}

rather than dumping 3 MB of your internal RefundResponse object into the context.

If you're using MCP, output schemas are explicitly supported for this purpose, and clients can validate structured results.

4. Treat execution reliability as software engineering

The LLM shouldn't be responsible for:

• retries
• exponential backoff
• authentication
• pagination
• rate limits
• idempotency
• circuit breaking
• timeout handling
• API-version differences
• translating internal errors
• request correlation
• authorization

Your tool runtime should handle all of those.

For example:

async function refundCustomer(args, ctx) {
  const validated = RefundSchema.parse(args);

  return retryWithBackoff(
    () =>
      payments.refund({
        orderId: validated.order_id,
        amountCents: validated.amount_cents,
        idempotencyKey: ctx.toolCallId
      }),
    {
      retries: 3,
      retryOn: ["429", "502", "503", "504"]
    }
  );
}

The model sees:

refund_customer(...)

It does not need to understand the machinery underneath.

5. Separate read tools from mutation tools

This is especially important for reliability.

I'd make the distinction explicit:

get_order
search_orders
get_customer

versus:

cancel_order
refund_order
change_shipping_address

And make mutations much harder to accidentally invoke.

For example:

get_refund_preview
       ↓
confirm_refund
       ↓
refund_order

can be preferable to giving the model a single powerful:

modify_order(...)

MCP even has annotations for things such as readOnlyHint, destructiveHint, and idempotentHint, although those are hints rather than security guarantees.

The surprisingly important part: tool count

Don't give an agent 150 internal APIs.

Even if the model can technically choose among them, you've created a giant classification problem.

OpenAI's current guidance suggests keeping the initially available function set small—roughly fewer than 20 is a useful starting point—and deferring larger tool surfaces when necessary.

I'd organize your tools hierarchically:

Agent
 │
 ├── Customer tools
 │     ├── get_customer
 │     ├── update_customer
 │     └── search_customers
 │
 ├── Order tools
 │     ├── get_order
 │     ├── search_orders
 │     └── cancel_order
 │
 └── Billing tools
       ├── get_invoice
       ├── create_refund
       └── ...

Then dynamically expose the relevant subset.

And I'd add a tool test suite

This is where many agent implementations fall down.

Don't evaluate:

"Does the agent seem good?" Evaluate:

"Given these 500 representative requests, did it select the correct tool, generate valid arguments, and produce the expected side effect?" For every tool, test at least:

1. Correct tool selection
2. Correct arguments
3. Missing information
4. Ambiguous information
5. Invalid arguments
6. API failure
7. Permission failure
8. Duplicate invocation
9. Timeout
10. Adversarial/unexpected input

And measure separately:

tool_selection_accuracy
argument_accuracy
execution_success_rate
unnecessary_tool_call_rate
mutation_error_rate
latency
cost

This gives you a much more useful signal than end-to-end "agent success."

---

My recommended stack

If you're starting from scratch, I'd use:

                 OpenAI model
                      │
              Agents SDK / Responses
                      │
              ┌───────▼────────┐
              │ Tool registry  │
              │ 5–20 tools     │
              └───────┬────────┘
                      │
              Zod / Pydantic
              strict schemas
                      │
              ┌───────▼────────┐
              │ Tool adapters  │
              │ deterministic  │
              └───────┬────────┘
                      │
          retries / auth / idempotency
                      │
              ┌───────▼────────┐
              │ Internal APIs  │
              └────────────────┘

Use MCP on the boundary if you need cross-agent/vendor interoperability; use the Agents SDK/runtime for orchestration and execution. You don't have to choose one or the other.

The important architectural decision is:

Don't make your internal APIs into LLM tools. Make an LLM-native tool layer on top of your internal APIs. That layer is where you encode your business semantics, validation, safety, retries, idempotency, and compact results.

If you tell me what your internal APIs look like (REST/gRPC/GraphQL, number of endpoints, and a couple examples of tools the agent currently gets wrong), I can sketch a concrete tool architecture and schema pattern for your system.

If you need interoperability across different model vendors or want your internal tools to be usable by many agent runtimes, MCP is the better tool protocol. MCP supports explicit input and output schemas and annotations such as read-only/idempotent/destructive behavior.

But I would not simply turn every internal REST/RPC endpoint into an MCP tool.

The most important design principle

Your tool should represent an agent-level operation, not an API endpoint.

Bad:

update_customer(
    customer_id,
    field,
    value,
    ...
)

Better:

change_customer_email(
    customer_id,
    new_email
)

Even better when the workflow is known:

update_customer_contact_info(
    customer_id,
    email,
    phone
)

The model shouldn't have to understand your internal API's quirks, sequencing requirements, IDs, pagination, authentication, retries, etc. Code should handle those things.

OpenAI's current tool guidance explicitly recommends this approach: make invalid states difficult to represent, avoid asking the model for arguments your application already knows, and combine functions that are always used together.

A robust tool should have 5 layers

I'd structure every internal tool roughly like this:

                 ┌────────────────────┐
                 │       Agent        │
                 └─────────┬──────────┘
                           │
                    typed tool call
                           │
                 ┌─────────▼──────────┐
                 │   Tool Contract    │
                 │ JSON Schema/Zod    │
                 └─────────┬──────────┘
                           │
                  validation / auth
                           │
                 ┌─────────▼──────────┐
                 │   Tool Adapter     │
                 │ deterministic code │
                 └─────────┬──────────┘
                           │
                 retries / timeout /
                 idempotency / mapping
                           │
                 ┌─────────▼──────────┐
                 │   Internal API     │
                 └────────────────────┘

The LLM should essentially only control the first arrow.

1. Strict input schemas

Use structured/strict tool calling wherever possible. OpenAI's Structured Outputs can make tool arguments conform to the supplied JSON Schema when strict: true is enabled.

For example:

const CreateRefund = z.object({
  order_id: z.string(),
  reason: z.enum([
    "duplicate_charge",
    "customer_request",
    "fraud",
    "other"
  ]),
  amount_cents: z.number().int().positive()
});

Don't give the model:

amount: string

and hope it figures out whether that's dollars, cents, or "all".

Make the schema encode the business rules.

2. Make invalid states impossible

This is probably the biggest improvement you can make.

Instead of:

{
  "status": "pending | approved | rejected | whatever"
}

use an enum.

Instead of:

{
  "start_date": "...",
  "end_date": "..."
}

where end_date < start_date is possible, validate that in the adapter.

Instead of:

{
  "customer_id": "..."
}

when the customer is already established in the conversation, don't ask the model to supply it.

Your runtime can inject it.

This dramatically reduces the amount of reasoning the model has to do.

3. Give tools semantic outputs too

A common mistake is:

Tool → raw internal API response → LLM

Don't do that.

Instead:

Internal API response
        ↓
adapter
        ↓
small semantic result
        ↓
LLM

For example:

{
  "refund_id": "rf_123",
  "status": "submitted",
  "amount_cents": 4999,
  "estimated_completion": "2026-09-12"
}

rather than dumping 3 MB of your internal RefundResponse object into the context.

When an LLM is unreliable at using internal APIs, the bottleneck is rarely the model's intelligence—it’s usually schema ambiguity, lack of runtime validation, poor error feedback loops, or bloated context.

Depending on whether your problem is getting the LLM to format the parameters correctly or managing the infrastructure of the tools themselves, the ecosystem offers a few standout frameworks tailored for robustness.

1. For Bulletproof Parameter Validation & Retries: Instructor or PydanticAI

If your agent fails because it hallucinates wrong types, misses required fields, or sends malformed JSON payloads to your internal APIs, you don't need a heavy agent framework; you need strict schema enforcement.

• Instructor: Built on top of Pydantic, Instructor lets you define your tool inputs as strict data classes. If the LLM outputs bad data or misses an argument, Instructor catches the validation error, feeds the exact error message back to the LLM in a self-correction loop, and retries automatically. It turns flaky API calls into deterministic, typed functions.
• PydanticAI: Created by the maintainers of Pydantic, this is a lightweight, highly production-focused agent/tool framework. It treats tool definitions as first-class, heavily typed citizens with built-in dependency injection, making it seamless to connect validated inputs directly to internal service calls.

2. For Standardizing and Securing Tool Definitions: Model Context Protocol (MCP)

If your internal APIs are messy, scattered, or hard for the model to reason about because the documentation/definitions change or conflict, look at Anthropic's open standard, the Model Context Protocol (MCP).

• Instead of hardcoding custom tool definitions inside every agent framework you test, MCP allows you to expose your internal APIs and data sources as standardized, modular "servers."
• It cleanly decouples your tool infrastructure from your agent logic, preventing context window clutter and drastically improving how reliably an LLM discovers and invokes local or remote capabilities.

3. For Complex, Multi-Step Orchestration & State: LangGraph (or LlamaIndex)

If your agent fails because it gets lost mid-workflow, forgets previous API outputs, or doesn't know how to handle multi-step dependency chains (e.g., Call API A → parse ID → conditionally call API B or C ), basic function calling isn't enough; you need graph-based control flow.

• LangGraph: Rather than a black-box agent loop, LangGraph forces you to define your tool-use logic as an explicit state machine (nodes and edges). You explicitly program the transition rules and error-handling branches. If an internal API fails, you can route the execution directly to a "recovery" or "re-prompt" node instead of letting the agent hallucinate a dead end.
• LlamaIndex: Excellent if your API tool use relies heavily on retrieving internal documentation, schemas, or enterprise data contexts right before executing the tool call.

Recommendation on How to Fix It

1. If the model sends bad arguments: Wrap your tools using Instructor or PydanticAI to force self-correcting validation loops on the arguments.
2. If the model gets confused by too many APIs: Encapsulate and isolate them using MCP so the model only sees clean, relevant tool definitions.
3. If the workflow logic falls apart: Move away from autonomous "infinite loop" agents and map out your execution steps explicitly in LangGraph.

To help narrow down the best fix, could you share:

• Are the failures mostly bad parameter formatting/types , or is the agent calling the wrong APIs in the wrong order?
• Are you writing this stack in Python or TypeScript?

Yes. The biggest shift I’d make is: don’t expose your internal APIs directly to the LLM. Build a tool layer specifically designed for probabilistic callers.

If I were designing this today, I’d use MCP as the transport/interface standard, but put a fairly opinionated reliability layer in front of your APIs. MCP gives you standardized tool discovery, JSON-schema inputs/outputs, and tool metadata, but it does not by itself make tool calling reliable.

The architecture I recommend

                    ┌──────────────────┐
                    │       LLM        │
                    └────────┬─────────┘
                             │
                       tool call
                             │
                    ┌────────▼─────────┐
                    │   Tool Gateway   │
                    │                  │
                    │ • strict schemas │
                    │ • auth           │
                    │ • validation     │
                    │ • idempotency    │
                    │ • retries        │
                    │ • timeouts       │
                    │ • error mapping  │
                    │ • observability  │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │ Domain Tools     │
                    │                  │
                    │ get_customer     │
                    │ search_orders    │
                    │ cancel_order     │
                    │ create_invoice   │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │ Internal APIs    │
                    └──────────────────┘

The crucial idea is that a tool is not an API endpoint.

An API might look like:

POST /customers/{id}/orders/search

The LLM-facing abstraction should instead be something like:

search_customer_orders(
    customer_id,
    status?,
    date_from?,
    date_to?
)

That gives the model a much smaller and more semantic action space.

1. Make tools extremely opinionated

A common failure mode is exposing something like:

{
  "name": "call_api",
  "arguments": {
    "method": "POST",
    "url": "...",
    "body": {}
  }
}

That's essentially asking the model to be an API programmer.

Instead:

{
  "name": "create_refund",
  "description": "Create a refund for an existing paid order. Use this only after confirming the order ID and refund amount.",
  "inputSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The exact order ID to refund."
      },
      "amount_cents": {
        "type": "integer",
        "minimum": 1,
        "description": "Amount to refund, in cents."
      },
      "reason": {
        "type": "string",
        "enum": ["customer_request", "duplicate_charge", "fraud"]
      }
    },
    "required": ["order_id", "amount_cents", "reason"]
  }
}

Notice the constraints are doing part of the reasoning for the model.

MCP explicitly supports JSON Schema for tool inputs and outputs, and recommends additionalProperties: false for parameterless tools; output schemas can likewise be used to validate returned data.

2. Give the LLM domain operations, not CRUD

This is probably the highest-leverage change.

Bad:

get_customer
update_customer
get_orders
update_order
delete_order

Better:

find_customer
get_order_status
change_shipping_address
cancel_order
issue_refund
reschedule_delivery

Even better when workflows are complicated:

prepare_order_cancellation
confirm_order_cancellation

The tool should encode your business semantics and invariants.

Your application code should be responsible for knowing how to accomplish the operation. The LLM should only decide which operation is appropriate.

---

3. Make invalid calls impossible—or nearly impossible

Don't rely on the model to follow instructions such as:

"Make sure amount is positive and order ID exists." Enforce that in code.

Have three layers of validation:

LLM output
   ↓
Schema validation
   ↓
Business validation
   ↓
Authorization / policy validation
   ↓
API call

For example:

def create_refund(order_id, amount_cents, reason):
    validate_schema(...)
    order = orders.get(order_id)

    if order.status != "paid":
        raise ToolError(
            code="ORDER_NOT_REFUNDABLE",
            message="Order must be paid before it can be refunded."
        )

    if amount_cents > order.remaining_refundable_cents:
        raise ToolError(
            code="AMOUNT_EXCEEDS_REMAINING_REFUND",
            message="Refund amount exceeds remaining refundable amount.",
            details={
                "remaining_refundable_cents":
                    order.remaining_refundable_cents
            }
        )

    authorize(...)
    return refunds.create(...)

The model doesn't need to understand every invariant.

The tool should.

---

4. Design errors specifically for recovery

This is another huge one.

Don't return:

{
  "error": "400 Bad Request"
}

Return something the agent can reason about:

{
  "success": false,
  "error": {
    "code": "ORDER_NOT_REFUNDABLE",
    "message": "Order ORD-123 is already fully refunded.",
    "retryable": false,
    "suggested_action": "Do not retry. Tell the user the order has already been refunded."
  }
}

I'd standardize errors across every tool:

type ToolError = {
  code: string
  message: string
  retryable: boolean
  retry_after_ms?: number
  suggested_action?: string
  details?: Record<string, unknown>
}

This turns:

API failed into:

I know what happened and what I'm allowed to do next.

---

5. Separate read tools from mutation tools

For example:

READ
├── find_customer
├── get_order
├── search_orders
└── get_refund_status

WRITE
├── cancel_order
├── create_refund
└── change_address

This makes it easier to:

• apply different authorization
• require confirmation
• retry safely
• audit mutations
• give the model clearer affordances

MCP itself has annotations such as readOnlyHint, destructiveHint, and idempotentHint for communicating these characteristics to clients. They're hints rather than guarantees, so your gateway should enforce the actual behavior.

---

6. Make mutations idempotent

This is essential for agents.

Suppose the model calls:

create_refund(order=123, amount=$50)

The API succeeds.

Then the network times out.

The agent doesn't know whether it succeeded.

Without idempotency, retrying can produce:

$50 refund
$50 refund

Instead:

idempotency_key = hash(
    tool_name +
    user/session +
    semantic_operation_id
)

Then:

create_refund(...)
       │
       ▼
idempotency layer
       │
   ┌───┴────┐
   │        │
new       existing
   │        │
execute   return original result

This makes retries safe.

---

7. Don't expose huge tool catalogs

If your agent has 150 internal APIs available simultaneously, reliability will often deteriorate.

Instead, use tool namespaces / progressive discovery:

customer tools
order tools
billing tools
shipping tools

Then expose only the relevant subset.

MCP supports tool discovery through tools/list, and its specification specifically discusses deterministic tool ordering and caching of tool lists.

I'd go one step further and have an internal tool router:

User request
     ↓
"billing"
     ↓
billing tools
     ↓
create_refund

rather than dumping the entire enterprise API surface into every context.

---

8. Keep tool responses small and structured

Don't give the model this:

{
  "customer": {
    "...": "500 fields..."
  }
}

Give it:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "status": "active",
  "open_orders": 2
}

And if it needs more:

get_customer
get_customer_orders
get_customer_payment_methods

rather than one enormous get_customer_everything tool.

Structured outputs are particularly useful because clients can validate them before returning them to the model.

---

The framework I'd actually build

I'd call this an LLM Tool Gateway:

                 ┌──────────────────────┐
                 │      MCP Server      │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │   Tool Definitions   │
                 │                      │
                 │ schemas              │
                 │ descriptions         │
                 │ examples             │
                 │ permissions          │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │   Tool Executor      │
                 │                      │
                 │ validate             │
                 │ authorize            │
                 │ dedupe               │
                 │ timeout              │
                 │ retry                │
                 │ circuit-break        │
                 │ normalize errors     │
                 │ validate response    │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │ Internal API / RPC   │
                 └──────────────────────┘

And I'd make every tool conform to roughly this contract:

interface LLMTool<I, O> {
  name: string
  description: string

  inputSchema: JSONSchema
  outputSchema: JSONSchema

  readonly: boolean
  destructive: boolean
  idempotent: boolean

  authorize(ctx: Context, input: I): Promise<void>

  validate(input: I): Promise<void>

  execute(input: I, ctx: Context): Promise<O>

  classifyError(error: unknown): ToolError
}

MCP then becomes the standard wire protocol, rather than the thing responsible for your application's reliability. That's an important distinction: the MCP specification itself explicitly leaves the implementation's interaction patterns open, while requiring servers to validate inputs, implement access controls, rate-limit calls, and sanitize outputs.

The testing framework matters just as much

I'd also stop measuring:

"Did the model call the API correctly?" and start measuring:

"Can an agent reliably complete this task using these tools?" Build a test set like:

100 customer-service tasks
        ↓
agent
        ↓
tool calls
        ↓
expected business outcome

Measure:

• correct tool selection
• correct arguments
• unnecessary calls
• invalid calls
• recovery from errors
• duplicate mutations
• authorization violations
• final business outcome
• number of tool calls
• latency/cost

Then intentionally inject failures:

API timeout
API 500
stale data
missing entity
permission denied
rate limit
duplicate request
partial response
conflicting records

A robust tool should let the agent recover without requiring a heroic prompt.

---

My recommendation in one sentence

Use MCP for the protocol, JSON Schema/Pydantic/Zod for contracts, and build a domain-specific tool gateway that handles validation, authorization, idempotency, retries, error normalization, and observability.

The most important principle is:

Move complexity out of the LLM and into the tool implementation. If your current agent is unreliable, I would not start by making the prompt more elaborate. I'd first redesign 5–10 of your highest-value APIs into proper LLM-native tools and benchmark them. That usually tells you very quickly whether the problem is the model—or the tool interface.

idempotency_key = hash(
    tool_name +
    user/session +
    semantic_operation_id
)

Then:

create_refund(...)
       │
       ▼
idempotency layer
       │
   ┌───┴────┐
   │        │
new       existing
   │        │
execute   return original result

This makes retries safe.

---

7. Don't expose huge tool catalogs

If your agent has 150 internal APIs available simultaneously, reliability will often deteriorate.

Instead, use tool namespaces / progressive discovery:

customer tools
order tools
billing tools
shipping tools

Then expose only the relevant subset.

MCP supports tool discovery through tools/list, and its specification specifically discusses deterministic tool ordering and caching of tool lists.

I'd go one step further and have an internal tool router:

User request
     ↓
"billing"
     ↓
billing tools
     ↓
create_refund

rather than dumping the entire enterprise API surface into every context.

---

8. Keep tool responses small and structured

Don't give the model this:

{
  "customer": {
    "...": "500 fields..."
  }
}

Give it:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "status": "active",
  "open_orders": 2
}

And if it needs more:

get_customer
get_customer_orders
get_customer_payment_methods

rather than one enormous get_customer_everything tool.

Structured outputs are particularly useful because clients can validate them before returning them to the model.

---

The framework I'd actually build

I'd call this an LLM Tool Gateway:

                 ┌──────────────────────┐
                 │      MCP Server      │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │   Tool Definitions   │
                 │                      │
                 │ schemas              │
                 │ descriptions         │
                 │ examples             │
                 │ permissions          │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │   Tool Executor      │
                 │                      │
                 │ validate             │
                 │ authorize            │
                 │ dedupe               │
                 │ timeout              │
                 │ retry                │
                 │ circuit-break        │
                 │ normalize errors     │
                 │ validate response    │
                 └──────────┬───────────┘
                            │
                 ┌──────────▼───────────┐
                 │ Internal API / RPC   │
                 └──────────────────────┘

And I'd make every tool conform to roughly this contract:

interface LLMTool<I, O> {
  name: string
  description: string

  inputSchema: JSONSchema
  outputSchema: JSONSchema

  readonly: boolean
  destructive: boolean
  idempotent: boolean

  authorize(ctx: Context, input: I): Promise<void>

  validate(input: I): Promise<void>

  execute(input: I, ctx: Context): Promise<O>

  classifyError(error: unknown): ToolError
}

MCP then becomes the standard wire protocol, rather than the thing responsible for your application's reliability. That's an important distinction: the MCP specification itself explicitly leaves the implementation's interaction patterns open, while requiring servers to validate inputs, implement access controls, rate-limit calls, and sanitize outputs.

The testing framework matters just as much

I'd also stop measuring:

"Did the model call the API correctly?" and start measuring:

"Can an agent reliably complete this task using these tools?" Build a test set like:

100 customer-service tasks
        ↓
agent
        ↓
tool calls
        ↓
expected business outcome

Measure:

The API succeeds.

When an LLM struggles with internal APIs, the issue is rarely just the "framework"—it’s usually a combination of poor schema definition, lack of strict data validation, ambiguous parameter descriptions, and no guardrails for error recovery.

Instead of jumping straight to a heavy multi-agent framework, the most robust approach focuses on typed tool definitions and schema enforcement . Top frameworks and architectural patterns address this reliability gap effectively:

• PydanticAI : A production-grade Python framework built by the team behind Pydantic. It treats tools as strictly typed functions backed by Pydantic validation. If the LLM passes a malformed argument, Pydantic catches it, feeds the validation error back to the model in a self-correction loop, and forces correct behavior. It is model-agnostic and natively supports structured outputs.
• Model Context Protocol (MCP) : Created by Anthropic and widely adopted across the ecosystem, MCP standardizes how applications provide context and tools to LLMs. Moving your internal APIs to a secure, localized MCP server ensures that tool definitions, schemas, and error responses are presented to the agent in a clean, predictable contract rather than ad-hoc JSON blobs.
• LangGraph : If your APIs require multi-step workflows (e.g., calling API A, parsing the output, checking a database, then conditionally calling API B), LangGraph lets you orchestrate this as a deterministic state machine. It eliminates "agent wandering" by hardcoding the transitions between tool calls while letting the LLM handle the variable generation inside specific nodes.
• Semantic Kernel / Microsoft Agent Framework : If your stack is enterprise .NET or Azure-centric, Microsoft's framework handles plugin architecture natively. It maps native code functions directly into semantic functions with heavy enterprise security, strict typing, and robust audit logging.

How to Fix Unreliable API Use Immediately (Regardless of Framework)

To help narrow down the best fix, tell me:

• What language/stack is your backend written in (Python, TypeScript, C#, etc.)?
• What specific failure modes are you seeing (e.g., hallucinating parameters, calling the wrong API, or failing on malformed JSON)?
• Are these single-step API calls or complex multi-turn workflows?

Yes. The key shift is: don’t make the LLM responsible for API correctness. Make the tool layer responsible for correctness, and let the LLM propose an invocation.

The framework I’d use is essentially a typed, validated, state-aware tool adapter layer between the agent and your internal APIs.

The architecture

                    ┌──────────────────┐
                    │       LLM        │
                    │                  │
                    │ "I need to..."   │
                    └────────┬─────────┘
                             │
                    typed tool call
                             ▼
              ┌──────────────────────────┐
              │       Tool Layer         │
              │                          │
              │  1. Schema validation    │
              │  2. Auth / permissions   │
              │  3. Semantic validation  │
              │  4. Normalization        │
              │  5. Execution policy     │
              │  6. Retry / timeout      │
              │  7. Idempotency          │
              │  8. Error translation    │
              └────────────┬─────────────┘
                           │
                           ▼
                 ┌──────────────────┐
                 │ Internal API(s)  │
                 └──────────────────┘

The important idea is that your agent tools should not simply be thin wrappers around REST endpoints.

Instead:

Design tools around what the agent needs to accomplish, not around what your backend happens to expose. For example, don't give the model:

POST /customers/{id}/orders
PATCH /orders/{id}
GET /customers/{id}
GET /inventory

Give it higher-level capabilities such as:

find_customer
get_order
search_inventory
create_order
cancel_order

And make each tool extremely difficult to misuse.

1. Use strict schemas

This is the first major reliability improvement.

Every tool should have a narrow JSON schema:

{
  "name": "get_order",
  "description": "Retrieve a specific order by its ID.",
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The exact order ID."
      }
    },
    "required": ["order_id"],
    "additionalProperties": false
  },
  "strict": true
}

With Structured Outputs/function calling, the model's generated arguments can be constrained to the supplied schema rather than merely asked to follow it.

But this only solves syntactic correctness.

The model can still give you a perfectly valid:

{"order_id": "123456"}

when order 123456 doesn't exist.

So I would use three validation layers:

LLM schema validation
        ↓
application/semantic validation
        ↓
backend validation

Never skip the middle layer.

---

2. Make tools small and composable

A common failure mode is a giant tool like:

manage_customer(
    action,
    customer_id,
    order_id,
    fields,
    options,
    ...
)

That's terrible for an LLM.

Instead:

get_customer
search_customers
update_customer
get_order
cancel_order
refund_order

Each tool should have:

• one clear purpose
• few arguments
• explicit types
• explicit enums
• no mysterious optional parameters
• descriptions explaining when to use it
• descriptions explaining when not to use it

For example:

{
  "name": "cancel_order",
  "description":
    "Cancel an order that has not shipped. Use this only when the user explicitly "
    "requests cancellation. Do not use this tool for refunds or returns.",
  ...
}

That "do not use this for..." language is surprisingly valuable.

---

3. Separate reads from writes

I'd make a hard architectural distinction:

READ TOOLS
    ↓
mostly safe
    ↓
GET / search / lookup

WRITE TOOLS
    ↓
high scrutiny
    ↓
create / update / delete / cancel / send

For writes, your tool executor should enforce additional policy.

For example:

def cancel_order(args, context):

    validate_schema(args)

    order = get_order(args.order_id)

    if order.status in ["shipped", "delivered"]:
        return ToolError(
            code="ORDER_NOT_CANCELLABLE",
            message="Order cannot be cancelled after shipment.",
            retryable=False
        )

    require_permission(context, "orders.cancel")

    return cancel(order.id)

Notice what's happening:

The LLM does not decide whether cancellation is allowed.

It proposes:

cancel_order(order_id=123)

Your application decides whether that operation is actually legal.

That's the fundamental pattern.

---

4. Give tools typed, machine-readable errors

This is probably the biggest thing missing from unreliable agent integrations.

Don't return:

400 Bad Request

or:

Something went wrong.

Return something like:

{
  "ok": false,
  "error": {
    "code": "INVALID_STATUS_TRANSITION",
    "message": "Order 123 is already shipped and cannot be cancelled.",
    "retryable": false,
    "suggested_action": "Use create_return_request instead."
  }
}

Now the LLM can recover.

Even better, distinguish:

INVALID_ARGUMENT
NOT_FOUND
PERMISSION_DENIED
CONFLICT
RATE_LIMITED
TEMPORARY_FAILURE
PRECONDITION_FAILED
INVALID_STATE

The agent shouldn't have to infer error semantics from English prose.

---

5. Make writes idempotent

This is essential for agents.

Imagine:

LLM → create_payment

The API times out.

The model doesn't know whether the payment happened.

It retries.

Now you've charged the customer twice.

Your tool layer should therefore assign an idempotency key:

idempotency_key =
    conversation_id + tool_call_id

Then:

create_payment(key=abc)
       ↓
payment succeeds
       ↓
network timeout
       ↓
agent retries
       ↓
create_payment(key=abc)
       ↓
same result returned

Never rely on the LLM to understand this.

---

6. Don't expose your raw API surface

This is where I think many teams go wrong.

If you have 200 internal endpoints, don't turn them into 200 LLM tools.

Create an agent-facing API:

Internal systems

CRM ─────┐
ERP ─────┤
Orders ──┤
Billing ─┤──► Agent Tool Layer ──► LLM
Search ──┘

The tool layer should handle:

• endpoint selection
• authentication
• parameter transformation
• pagination
• retries
• rate limits
• response normalization
• permissions
• business rules

The LLM should see a much simpler world.

This also gives you freedom to change your internal APIs without changing your agent.

---

7. Normalize responses aggressively

Don't dump your backend's gigantic response into the context.

Instead of:

{
  "customer": {
    "...": "200 fields"
  }
}

return exactly what the model needs:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "status": "active",
  "orders": [
    {
      "id": "ord_456",
      "status": "shipped",
      "total": 129.50
    }
  ]
}

Think of the tool response as a context interface, not an API response.

---

8. Put a state machine around dangerous workflows

For anything consequential, don't let the LLM freestyle a multi-step process.

For example, refunds:

                    ┌──────────────┐
                    │ refund_start │
                    └──────┬───────┘
                           ↓
                    validate_order
                           ↓
                    validate_amount
                           ↓
                    check_permissions
                           ↓
                    ┌──────────────┐
                    │ confirmation │
                    └──────┬───────┘
                           ↓
                    execute_refund

The LLM can navigate the workflow, but your application owns the state machine.

This is dramatically more reliable than putting:

"When refunding, first check X, then check Y, then..." into a prompt.

---

9. Build a tool contract

I'd give every internal tool a contract roughly like:

interface Tool<I, O> {
  name: string

  description: string

  inputSchema: JSONSchema

  execute(
    input: I,
    context: ToolContext
  ): Promise<ToolResult<O>>
}

And:

type ToolResult<T> =
  | {
      ok: true
      data: T
    }
  | {
      ok: false
      error: {
        code: string
        message: string
        retryable: boolean
        suggestedAction?: string
      }
    }

Then build all your tools on top of the same runtime:

Tool
 ├── schema validation
 ├── authorization
 ├── timeout
 ├── retry policy
 ├── idempotency
 ├── telemetry
 ├── circuit breaker
 └── error normalization

That becomes your LLM Tool Runtime.

---

10. Treat tool calls as untrusted input

Even if the LLM is your own model, its output should be treated like input from an external client.

So:

LLM
 ↓
parse
 ↓
schema validation
 ↓
authorization
 ↓
business validation
 ↓
execution

Never:

LLM
 ↓
internal API

This also gives you a clean security boundary.

---

11. Instrument everything

For every tool call, log something like:

trace_id
conversation_id
agent_id
tool_name
input
validation_result
authorization_result
backend_request
backend_status
latency
retry_count
output
error_code

Then you can answer questions such as:

"Why did the agent fail 8% of the time?" Instead of merely seeing:

"The agent is unreliable." You'll discover things like:

37% = wrong tool selected
22% = missing parameter
18% = invalid state
13% = backend timeout
 7% = authorization
 3% = actual model error

That changes the engineering problem completely.

---

12. Evaluate tools independently from the agent

This is another important distinction.

Have an eval suite like:

Tool correctness
────────────────────────
1000 valid inputs
1000 invalid inputs
500 boundary cases
500 permission cases
500 state conflicts
500 transient failures

Then separately:

Agent evals
────────────────────────
Which tool did it select?
Did it provide the right arguments?
Did it recover from errors?
Did it stop when it should?
Did it ask for clarification?

You want to know whether the problem is:

model
  vs.
tool schema
  vs.
tool implementation
  vs.
workflow

Otherwise teams tend to keep changing the prompt when the actual problem is an API contract.

---

The framework I'd actually build

If I were starting this from scratch, I'd make five layers:

                    ┌──────────────────────┐
                    │        Agent         │
                    └──────────┬───────────┘
                               │
                         Tool definitions
                               │
                    ┌──────────▼───────────┐
                    │     Tool Router      │
                    │                      │
                    │ selection / policy   │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │    Tool Runtime      │
                    │                      │
                    │ schema               │
                    │ auth                 │
                    │ validation           │
                    │ retries              │
                    │ idempotency           │
                    │ observability         │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │   Domain Adapters    │
                    │                      │
                    │ orders               │
                    │ CRM                  │
                    │ billing              │
                    │ inventory            │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │    Internal APIs     │
                    └──────────────────────┘

And I'd enforce one principle above all:

The LLM proposes actions; deterministic software validates and executes them. Structured Outputs/function calling can make the shape of the proposal highly reliable, but they don't guarantee that the proposed values are semantically correct.

So if your current architecture is essentially:

LLM → internal API

I'd change it to:

LLM
 ↓
strict typed tool
 ↓
deterministic validator
 ↓
policy/authorization
 ↓
domain adapter
 ↓
internal API
 ↓
normalized typed result
 ↓
LLM

That is the framework I'd bet on for production reliability.

If you tell me what your internal APIs look like (REST/OpenAPI, GraphQL, gRPC, Python functions, etc.) and what stack you're using (TypeScript/Python/Java/etc.), I can sketch the concrete tool-runtime architecture and show what a production-quality tool definition + executor would look like.

into a prompt.

---

9. Build a tool contract

I'd give every internal tool a contract roughly like:

When an LLM struggles with internal APIs, the issue is rarely just the model's intelligence—it is usually a failure in schema definition, type enforcement , or state/error handling . An LLM needs deterministic guardrails rather than free-form conversation loops to interact reliably with backend systems. [1]

The top frameworks for building robust API tool use depend on your stack and how much control you need over the execution layer:

• Pydantic AI — Best for Python-centric, strongly-typed reliability. It leverages Pydantic for strict data validation on both the inputs the LLM generates and the outputs it receives back from your APIs. If an LLM tries to pass a malformed payload, it catches the error and feeds it back to the model to self-correct before hitting your actual infrastructure. [1]
• LangGraph — Best for multi-step, stateful, and complex API sequences. Instead of treating an agent as a blind black box, LangGraph forces you to map out workflows as explicit state graphs. You can insert human-in-the-loop validation steps, retry loops, and deterministic checkpoints between API calls. [1][2]
• Model Context Protocol (MCP) — Best for standardizing and decoupling tool definitions. Developed as an open standard by Anthropic and widely integrated across the ecosystem, MCP lets you expose your internal APIs as isolated servers. This gives any compatible agent a clean, secure, and well-documented boundary for discovering and calling your tools. [1]
• CrewAI — Best if you want structured role-based task delegation paired with built-in tracing and observability for multi-agent API execution. It abstracts the trial-and-error loop into structured tasks with strict parameters. [1]

If you want to narrow down the best fit, tell me:

• What programming language/stack is your backend built on?
• Are the internal APIs read-only or performing state-changing mutations (writes/deletes)?
• Which LLM provider are you currently using?

Yes. The key is to stop thinking of an internal API wrapper as a normal API and start treating it as an LLM-facing interface.

My recommendation is a typed, contract-first “agent tool layer” in front of your internal APIs:

                  ┌──────────────────┐
User ────────────►│      LLM         │
                  └────────┬─────────┘
                           │ tool call
                           ▼
                  ┌──────────────────┐
                  │  Agent Tool API  │  ← designed for the model
                  │  schemas + rules │
                  └────────┬─────────┘
                           │ validated/normalized
                           ▼
                  ┌──────────────────┐
                  │ Internal APIs    │  ← designed for engineers
                  └──────────────────┘

This separation is probably the single biggest improvement you can make.

The framework I'd use

I would build around JSON Schema + strict tool calling + a thin tool-execution layer, optionally exposed through MCP if you need portability across agents/models.

The important part isn't actually MCP. MCP standardizes how tools are exposed; it doesn't magically make a badly designed tool reliable.

I'd use five layers:

1. Give the model a small number of semantic tools

Don't expose your internal REST endpoints one-for-one.

Bad:

GET /customers/{id}
GET /customers/{id}/orders
GET /customers/{id}/subscriptions
POST /orders
PATCH /orders/{id}
...

That forces the model to reconstruct your API architecture.

Instead:

find_customer
get_customer_context
search_orders
update_order

Even better, make tools correspond to things the agent is trying to accomplish, not HTTP verbs.

Anthropic's tool-design guidance similarly recommends meaningful namespacing, consolidating related operations, and returning only information useful for the agent's next decision.

---

2. Make the schema do as much work as possible

Don't rely on prompting the model to produce valid arguments.

For example:

{
  "name": "update_order",
  "description": "Update an existing customer order. Use this only after identifying the order. Do not use it to create a new order.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The canonical order ID returned by search_orders."
      },
      "status": {
        "type": "string",
        "enum": ["pending", "approved", "cancelled"]
      },
      "reason": {
        "type": "string",
        "description": "Why the order is being changed."
      }
    },
    "required": ["order_id", "status", "reason"],
    "additionalProperties": false
  }
}

Then validate again on your server.

Modern tool-calling APIs support strict schema-constrained arguments; for example, OpenAI's function calling supports strict: true, and Anthropic provides strict tool use that constrains calls to the supplied JSON Schema.

The model should never be able to send:

{
  "order_id": "maybe-123",
  "status": "do whatever",
  "foo": "..."
}

to your production API.

---

3. Put business invariants in the tool, not the prompt

This is where many agent systems go wrong.

Don't write:

You should only cancel an order if it hasn't shipped. Make the tool enforce it:

def cancel_order(order_id: str, reason: str):
    order = orders.get(order_id)

    if order.status == "shipped":
        raise ToolError(
            code="ORDER_ALREADY_SHIPPED",
            message="This order cannot be cancelled because it has shipped."
        )

    return orders.cancel(order_id, reason)

The model then gets a machine-readable failure:

{
  "ok": false,
  "error": {
    "code": "ORDER_ALREADY_SHIPPED",
    "message": "This order cannot be cancelled because it has shipped.",
    "recoverable": false
  }
}

That's much more robust than hoping the model remembers a paragraph in a system prompt.

Think of your tool as:

an API contract + validation + policy enforcement + useful model-facing output rather than merely:

POST /orders/cancel

---

4. Make tool errors recoverable

This is surprisingly important.

A normal API might return:

400 Bad Request

An agent needs something more like:

{
  "ok": false,
  "error": {
    "code": "AMBIGUOUS_CUSTOMER",
    "message": "Multiple customers match 'John Smith'.",
    "next_action": "ask_user",
    "candidates": [
      {"id": "cus_123", "name": "John Smith", "company": "Acme"},
      {"id": "cus_456", "name": "John Smith", "company": "Globex"}
    ]
  }
}

Now the model has a clear recovery path.

I generally classify errors as:

INVALID_ARGUMENT       → fix arguments
MISSING_INFORMATION    → ask user
NOT_FOUND              → search / clarify
AMBIGUOUS              → ask user
CONFLICT               → reconsider / retry
PERMISSION_DENIED      → stop
RATE_LIMITED           → retry
TRANSIENT_FAILURE      → retry
BUSINESS_RULE          → explain / stop

This turns your agent from:

call → fail → hallucinate into:

call → structured failure → recover appropriately. The underlying tool-use model is explicitly a contract: the model generates a structured request, your application executes it, and the result is fed back into the model.

---

5. Return agent-shaped responses

Don't dump your internal API response into the context.

Suppose your database returns 200 fields.

The agent probably needs:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "account_status": "active",
  "open_orders": 2,
  "has_overdue_balance": true
}

rather than 40 KB of CRM metadata.

This matters for both reliability and cost. Anthropic specifically recommends returning high-signal information and stable semantic identifiers rather than opaque internal references.

---

The part I'd add that most teams miss: tool evaluations

If your complaint is “the agent isn't reliable”, don't primarily fix it with more prompting.

Build a test suite.

For every tool, create perhaps 50–500 representative scenarios:

✓ correctly identifies tool
✓ correctly fills required parameters
✓ asks for missing information
✓ refuses invalid operation
✓ handles ambiguous entity
✓ handles API failure
✓ handles permission failure
✓ uses result of previous tool
✓ doesn't invent IDs
✓ doesn't repeat a failed call indefinitely

Then measure:

tool_selection_accuracy
argument_accuracy
successful_execution_rate
recovery_rate
hallucinated_identifier_rate
unnecessary_tool_call_rate
average_calls_per_task

And run these evaluations whenever you change:

• tool descriptions
• schemas
• models
• system prompts
• API behavior
• tool response formats

Tool descriptions themselves materially affect model performance, so they should be treated almost like executable interface documentation rather than prose nobody tests.

---

A useful design pattern

I'd make every tool conform to something roughly like:

class AgentTool:

    name: str
    description: str
    input_schema: JSONSchema

    def validate(self, args):
        ...

    def authorize(self, context, args):
        ...

    def execute(self, args):
        ...

    def format_result(self, result):
        ...

    def format_error(self, error):
        ...

And then put your internal API behind it:

LLM
 │
 ▼
Tool selection
 │
 ▼
JSON Schema validation
 │
 ▼
Authorization / policy
 │
 ▼
Argument normalization
 │
 ▼
Internal API
 │
 ▼
Result normalization
 │
 ▼
Agent-shaped response
 │
 ▼
LLM

That gives you a very clean boundary.

Where MCP fits

If you have multiple agents or want your tools usable by different model providers, I'd expose this tool layer through MCP.

For example:

                    ┌── Claude
                    │
Agent Tool Layer ───┼── OpenAI
                    │
                    ├── internal agent
                    │
                    └── other clients

MCP gives you a standardized transport/interface for exposing the tools. But I'd still keep your domain-specific tool layer behind MCP rather than making your internal REST API itself the MCP interface.

---

My recommended stack

If I were building this today:

Tool definition

• JSON Schema
• strict structured tool calling
• Pydantic/Zod/etc. as the source of truth

Execution

• typed Python/TypeScript functions
• validation
• authorization
• retries/timeouts/idempotency
• structured errors

Agent protocol

• MCP when tools need to be shared across agents/clients
• native tool calling underneath when you control the agent

Reliability

• deterministic business rules in code
• tool-use evaluations
• tracing
• per-tool success/error metrics
• replayable production traces

Model-facing design

• fewer semantic tools
• excellent descriptions
• explicit parameter semantics
• examples for tricky tools
• compact, high-signal outputs

Anthropic's newer tool-use work is particularly aligned with this philosophy: tool examples help with usage patterns that schemas can't express, while tool search addresses large tool libraries and programmatic calling reduces context pollution.

The key principle

Don't try to make the LLM better at using your API. Make the API easier for an LLM to use.

If you tell me what your internal APIs look like (e.g. REST/GraphQL, 20 vs. 2,000 endpoints, Python/TypeScript, and what kind of agent you're using), I can sketch a concrete production architecture and tool schema pattern for your setup.

---

2. Make the schema do as much work as possible

Don't rely on prompting the model to produce valid arguments.

For example:

{
  "name": "update_order",
  "description": "Update an existing customer order. Use this only after identifying the order. Do not use it to create a new order.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The canonical order ID returned by search_orders."
      },
      "status": {
        "type": "string",
        "enum": ["pending", "approved", "cancelled"]
      },
      "reason": {
        "type": "string",
        "description": "Why the order is being changed."
      }
    },
    "required": ["order_id", "status", "reason"],
    "additionalProperties": false
  }
}

Then validate again on your server.

Modern tool-calling APIs support strict schema-constrained arguments; for example, OpenAI's function calling supports strict: true, and Anthropic provides strict tool use that constrains calls to the supplied JSON Schema.

The model should never be able to send:

{
  "order_id": "maybe-123",
  "status": "do whatever",
  "foo": "..."
}

to your production API.

---

3. Put business invariants in the tool, not the prompt

This is where many agent systems go wrong.

Don't write:

You should only cancel an order if it hasn't shipped. Make the tool enforce it:

def cancel_order(order_id: str, reason: str):
    order = orders.get(order_id)

    if order.status == "shipped":
        raise ToolError(
            code="ORDER_ALREADY_SHIPPED",
            message="This order cannot be cancelled because it has shipped."
        )

    return orders.cancel(order_id, reason)

The model then gets a machine-readable failure:

{
  "ok": false,
  "error": {
    "code": "ORDER_ALREADY_SHIPPED",
    "message": "This order cannot be cancelled because it has shipped.",
    "recoverable": false
  }
}

That's much more robust than hoping the model remembers a paragraph in a system prompt.

Think of your tool as:

an API contract + validation + policy enforcement + useful model-facing output rather than merely:

POST /orders/cancel

---

4. Make tool errors recoverable

This is surprisingly important.

A normal API might return:

400 Bad Request

An agent needs something more like:

{
  "ok": false,
  "error": {
    "code": "AMBIGUOUS_CUSTOMER",
    "message": "Multiple customers match 'John Smith'.",
    "next_action": "ask_user",
    "candidates": [
      {"id": "cus_123", "name": "John Smith", "company": "Acme"},
      {"id": "cus_456", "name": "John Smith", "company": "Globex"}
    ]
  }
}

Now the model has a clear recovery path.

I generally classify errors as:

INVALID_ARGUMENT       → fix arguments
MISSING_INFORMATION    → ask user
NOT_FOUND              → search / clarify
AMBIGUOUS              → ask user
CONFLICT               → reconsider / retry
PERMISSION_DENIED      → stop
RATE_LIMITED           → retry
TRANSIENT_FAILURE      → retry
BUSINESS_RULE          → explain / stop

This turns your agent from:

call → fail → hallucinate into:

call → structured failure → recover appropriately. The underlying tool-use model is explicitly a contract: the model generates a structured request, your application executes it, and the result is fed back into the model.

---

5. Return agent-shaped responses

Don't dump your internal API response into the context.

Suppose your database returns 200 fields.

The agent probably needs:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "account_status": "active",
  "open_orders": 2,
  "has_overdue_balance": true
}

rather than 40 KB of CRM metadata.

This matters for both reliability and cost. Anthropic specifically recommends returning high-signal information and stable semantic identifiers rather than opaque internal references.

---

The part I'd add that most teams miss: tool evaluations

If your complaint is “the agent isn't reliable”, don't primarily fix it with more prompting.

Build a test suite.

For every tool, create perhaps 50–500 representative scenarios:

✓ correctly identifies tool
✓ correctly fills required parameters
✓ asks for missing information
✓ refuses invalid operation
✓ handles ambiguous entity
✓ handles API failure
✓ handles permission failure
✓ uses result of previous tool
✓ doesn't invent IDs
✓ doesn't repeat a failed call indefinitely

Then measure:

tool_selection_accuracy
argument_accuracy
successful_execution_rate
recovery_rate
hallucinated_identifier_rate
unnecessary_tool_call_rate
average_calls_per_task

And run these evaluations whenever you change:

• tool descriptions
• schemas
• models
• system prompts
• API behavior
• tool response formats

Tool descriptions themselves materially affect model performance, so they should be treated almost like executable interface documentation rather than prose nobody tests.

---

A useful design pattern

I'd make every tool conform to something roughly like:

When an agent struggles with internal APIs, the bottleneck is rarely the LLM's raw intelligence—it is almost always type safety, schema ambiguity, and lack of strict validation/retry loops between the model and the code execution layer.

Instead of treating tool-calling as a loose prompt-and-pray mechanism, the most robust frameworks enforce strict data contracts using validation libraries like Pydantic or native TypeScript types.

The top frameworks for creating robust, reliable API tools depend heavily on your tech stack and how much control you want over the validation layer:

• Pydantic AI (Python): Best for type-safe, production-grade Python agents. Built directly around Pydantic, it forces your API inputs and outputs into rigid, validated schemas. If the LLM hallucinates a bad parameter type, Pydantic catches it, feeds the validation error back to the model, and forces a self-correction loop before it ever hits your internal API.
• LangGraph (Python): Best if your APIs require multi-step, stateful workflows (e.g., Step A fetches a user ID, which must be validated before Step B can call the update API). It gives you deterministic control over the graph state, meaning you can programmatically inject human-in-the-loop approvals or hardcoded fallback states if an API call fails.
• Mastra (TypeScript/Node.js): Best full-stack framework if your internal services live in a TypeScript ecosystem. It provides stellar first-class abstractions for defining tools, handling state, and logging agent execution steps with native JS/TS type safety.
• Vercel AI SDK (TypeScript/JavaScript): Best lightweight toolkit if you are building inside modern web apps or serverless functions and need strongly-typed tool definitions (zod schemas) without adopting a heavy agentic monolith.
• OpenAI Agents SDK / Claude Agent SDK : Best if you are deeply anchored to a specific model provider. They handle the native tool-loop orchestration efficiently, though you will still need to wrap your internal APIs in strict JSON schemas (using Zod or Pydantic) to prevent malformed payloads.

Pro-Tips for Making Internal APIs Reliable

Regardless of the framework you choose, architectural patterns matter more than the framework itself:

1. Never expose raw internal endpoints as tools. Create dedicated "Agent Wrappers" that simplify payloads, strip unnecessary fields, and use ultra-descriptive parameter names and docstrings.
2. Use Zod or Pydantic for validation. If the framework doesn't automatically feed validation errors back to the LLM to retry, implement a middleware catch block that says: "Your input failed validation with error X, please fix it."
3. Limit the toolset per turn. Don't give the agent access to 50 internal APIs at once. Use a router or hierarchical agent pattern to restrict the active tool pool to 3–5 relevant endpoints per context window.

To help narrow down the best fit, tell me:

• What language/stack is your backend written in (Python, TypeScript, Go)?
• Are these read-only data-fetching APIs, or do they write/mutate critical data?

Yes. The key insight is: don’t make the LLM responsible for API correctness. Make the LLM responsible for choosing what it wants to accomplish, while a deterministic tool layer handles validation, authorization, retries, idempotency, and API semantics.

For most teams, I’d use this architecture:

User
  ↓
LLM / Agent
  ↓
Small, task-oriented tool definitions
  ↓
Tool runtime / gateway
  ├─ Schema validation
  ├─ Auth + permissions
  ├─ Argument normalization
  ├─ Preconditions
  ├─ Retry / timeout / backoff
  ├─ Idempotency
  ├─ API-specific error handling
  ├─ Response validation
  └─ Observability / tracing
  ↓
Internal APIs

My recommendation

1. Use MCP as the tool interface

If you're building an internal tool ecosystem today, Model Context Protocol (MCP) is probably the best standard interface to build around.

MCP explicitly models tools with names, descriptions, input schemas, optional output schemas, and execution metadata. The current July 2026 specification also adds things like stateless operation, improved authorization, caching, and routing.

But I would not expose your raw internal APIs directly through MCP.

Instead:

                   MCP
                    │
             ┌──────▼──────┐
             │ Tool Gateway │
             └──────┬──────┘
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
   Orders API   CRM API      Billing API

MCP should be your LLM-facing contract, not your internal API contract.

---

2. Design tools around business operations, not HTTP endpoints

This is probably the biggest improvement you can make.

Bad

GET /customers/{id}
POST /orders
PATCH /orders/{id}
GET /orders?customer_id=...
POST /payments

An LLM has to figure out API choreography, required fields, identifiers, ordering, etc.

Better

get_customer
search_orders
create_order
cancel_order
refund_order
update_shipping_address

Even better for complex workflows

prepare_refund
execute_refund

or:

create_order_draft
confirm_order

The tool should represent a safe business capability.

For example:

{
  "name": "refund_order",
  "description": "Refund an order that is eligible for refund. Use this when the customer explicitly requests a refund.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The internal order ID."
      },
      "reason": {
        "type": "string",
        "enum": ["customer_request", "duplicate_charge", "damaged_item"]
      }
    },
    "required": ["order_id", "reason"],
    "additionalProperties": false
  }
}

The LLM chooses refund_order.

Your server determines whether the order actually can be refunded.

That's a crucial distinction.

---

3. Make schemas aggressively strict

If you're using OpenAI function calling, use Structured Outputs / strict: true where supported. It constrains generated tool arguments to your supplied JSON Schema rather than merely asking the model nicely to produce valid JSON.

But don't confuse:

"The arguments match the schema" with:

"The arguments are correct." For example:

{
  "customer_id": "123",
  "amount": 500
}

may be perfectly valid according to the schema while being completely wrong for the actual customer.

So have two layers of validation:

LLM
 ↓
JSON Schema
 ↓
Business validation
 ↓
Authorization
 ↓
API

---

4. Put business logic inside the tool

This is where many "unreliable agents" go wrong.

Don't do:

tool("refund_order", order_id, amount)
    → POST /refund

Do:

refund_order(order_id, reason):
    order = orders.get(order_id)

    if not order:
        return OrderNotFound(...)

    if order.status not in REFUNDABLE_STATES:
        return NotRefundable(...)

    if not authorization.can_refund(order):
        return PermissionDenied(...)

    return payments.refund(
        ...,
        idempotency_key=...
    )

The model shouldn't have to remember your business rules.

If a rule matters for correctness, enforce it in code.

---

5. Give tools typed, machine-readable errors

This is surprisingly important.

Don't return:

Something went wrong. Please try again.

Return something like:

{
  "ok": false,
  "error": {
    "code": "ORDER_NOT_REFUNDABLE",
    "message": "Order 123 cannot be refunded because it was already refunded.",
    "retryable": false
  }
}

Or:

{
  "ok": false,
  "error": {
    "code": "PAYMENT_SERVICE_UNAVAILABLE",
    "message": "Payment service temporarily unavailable.",
    "retryable": true,
    "retry_after_seconds": 5
  }
}

Now the agent can reason over the result instead of trying to interpret arbitrary HTTP/API errors.

MCP itself supports structured tool results and output schemas, which makes this pattern particularly natural.

---

6. Make writes idempotent

For internal APIs, this is essential.

Imagine:

LLM → create_payment
             ↓
        API succeeds
             ↓
     network timeout
             ↓
LLM thinks it failed
             ↓
     create_payment AGAIN

You just charged someone twice.

Your tool layer should generate/pass an idempotency key and make retries safe:

create_payment
    idempotency_key = agent_run_id + tool_call_id

Then you can safely retry transient failures.

I'd make this a property of the tool runtime, rather than something the model needs to understand.

---

7. Don't expose dozens of microscopic tools

Tool selection itself becomes unreliable as the tool count grows.

Instead of:

get_customer
get_customer_address
get_customer_orders
get_customer_subscriptions
get_customer_payment_methods
get_customer_status
...

consider a small number of well-designed capabilities:

lookup_customer
lookup_customer_orders
manage_subscription
manage_order

The exact granularity depends on your domain, but a good rule is:

One tool = one meaningful action the model can explain to a human. Not:

One tool = one endpoint in your API gateway.

---

8. Add an evaluation harness

This is the part I'd prioritize if your existing agent is already unreliable.

Create a corpus of real tasks:

"Refund my most recent order."

"Change the shipping address on order 8271."

"Cancel my subscription but don't refund the current month."

"Find the invoice for Acme's March payment."

...

For each, define expected behavior:

task: refund_recent_order

expected_tools:
  - lookup_orders
  - refund_order

must_not_call:
  - cancel_subscription

expected_properties:
  refund_reason: customer_request

Then measure:

• correct tool selected
• correct arguments
• correct tool ordering
• invalid calls rejected
• authorization respected
• retries safe
• final answer matches tool result
• no unnecessary API calls
• latency
• token cost

This lets you distinguish:

LLM failure
     vs
tool-definition failure
     vs
API failure
     vs
orchestration failure

That's enormously valuable.

---

What framework should you actually choose?

I'd think about the stack in three layers:

If you're primarily using OpenAI, the Responses API + function calling/Agents SDK + MCP-compatible tools is a very reasonable stack. OpenAI's current function-calling stack supports tool calling and Structured Outputs, and the Agents platform includes tracing.

If you have complex deterministic workflows—long-running processes, branching, human approvals, resumability—I would consider LangGraph or a similar explicit state-machine/workflow layer rather than letting the LLM freely decide the entire sequence.

---

The architecture I'd build

For an internal enterprise agent, I'd probably implement:

                    ┌─────────────────┐
                    │       LLM       │
                    └────────┬────────┘
                             │
                     MCP / function calls
                             │
                    ┌────────▼────────┐
                    │   Tool Gateway  │
                    │                 │
                    │ Schema validate │
                    │ Normalize args  │
                    │ AuthZ           │
                    │ Preconditions   │
                    │ Idempotency     │
                    │ Retry/backoff   │
                    │ Rate limits     │
                    │ Audit logging   │
                    │ Output validate │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
          Orders API      CRM API       Billing API

And I'd make every tool contract look roughly like this:

Tool
 ├── name
 ├── description
 ├── input schema
 ├── output schema
 ├── authorization policy
 ├── preconditions
 ├── side-effect level
 ├── idempotency policy
 ├── timeout
 ├── retry policy
 └── error taxonomy

The important conceptual shift is:

The LLM should never be your API client. It should be the planner/user interface for a deterministic API-client layer.

That tends to produce dramatically more reliable systems than trying to improve the agent prompt until it "learns" how your internal APIs work.

If you tell me what your current stack is (e.g. OpenAI + Python/FastAPI, TypeScript, LangChain/LangGraph, existing REST/gRPC APIs), I can recommend a concrete architecture and tool contract pattern for it.

The tool should represent a safe business capability.

For example:

{
  "name": "refund_order",
  "description": "Refund an order that is eligible for refund. Use this when the customer explicitly requests a refund.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The internal order ID."
      },
      "reason": {
        "type": "string",
        "enum": ["customer_request", "duplicate_charge", "damaged_item"]
      }
    },
    "required": ["order_id", "reason"],
    "additionalProperties": false
  }
}

The LLM chooses refund_order.

Your server determines whether the order actually can be refunded.

That's a crucial distinction.

---

3. Make schemas aggressively strict

If you're using OpenAI function calling, use Structured Outputs / strict: true where supported. It constrains generated tool arguments to your supplied JSON Schema rather than merely asking the model nicely to produce valid JSON.

But don't confuse:

"The arguments match the schema" with:

"The arguments are correct." For example:

{
  "customer_id": "123",
  "amount": 500
}

may be perfectly valid according to the schema while being completely wrong for the actual customer.

So have two layers of validation:

LLM
 ↓
JSON Schema
 ↓
Business validation
 ↓
Authorization
 ↓
API

---

4. Put business logic inside the tool

This is where many "unreliable agents" go wrong.

Don't do:

tool("refund_order", order_id, amount)
    → POST /refund

Do:

refund_order(order_id, reason):
    order = orders.get(order_id)

    if not order:
        return OrderNotFound(...)

    if order.status not in REFUNDABLE_STATES:
        return NotRefundable(...)

    if not authorization.can_refund(order):
        return PermissionDenied(...)

    return payments.refund(
        ...,
        idempotency_key=...
    )

The model shouldn't have to remember your business rules.

If a rule matters for correctness, enforce it in code.

---

5. Give tools typed, machine-readable errors

This is surprisingly important.

Don't return:

Something went wrong. Please try again.

Return something like:

{
  "ok": false,
  "error": {
    "code": "ORDER_NOT_REFUNDABLE",
    "message": "Order 123 cannot be refunded because it was already refunded.",
    "retryable": false
  }
}

Or:

{
  "ok": false,
  "error": {
    "code": "PAYMENT_SERVICE_UNAVAILABLE",
    "message": "Payment service temporarily unavailable.",
    "retryable": true,
    "retry_after_seconds": 5
  }
}

Now the agent can reason over the result instead of trying to interpret arbitrary HTTP/API errors.

MCP itself supports structured tool results and output schemas, which makes this pattern particularly natural.

---

6. Make writes idempotent

For internal APIs, this is essential.

Imagine:

LLM → create_payment
             ↓
        API succeeds
             ↓
     network timeout
             ↓
LLM thinks it failed
             ↓
     create_payment AGAIN

You just charged someone twice.

Your tool layer should generate/pass an idempotency key and make retries safe:

create_payment
    idempotency_key = agent_run_id + tool_call_id

Then you can safely retry transient failures.

I'd make this a property of the tool runtime, rather than something the model needs to understand.

---

7. Don't expose dozens of microscopic tools

Tool selection itself becomes unreliable as the tool count grows.

Instead of:

get_customer
get_customer_address
get_customer_orders
get_customer_subscriptions
get_customer_payment_methods
get_customer_status
...

consider a small number of well-designed capabilities:

lookup_customer
lookup_customer_orders
manage_subscription
manage_order

The exact granularity depends on your domain, but a good rule is:

One tool = one meaningful action the model can explain to a human. Not:

One tool = one endpoint in your API gateway.

---

8. Add an evaluation harness

This is the part I'd prioritize if your existing agent is already unreliable.

Create a corpus of real tasks:

"Refund my most recent order."

"Change the shipping address on order 8271."

"Cancel my subscription but don't refund the current month."

"Find the invoice for Acme's March payment."

...

For each, define expected behavior:

task: refund_recent_order

expected_tools:
  - lookup_orders
  - refund_order

must_not_call:
  - cancel_subscription

expected_properties:
  refund_reason: customer_request

Then measure:

• correct tool selected
• correct arguments
• correct tool ordering
• invalid calls rejected
• authorization respected
• retries safe
• final answer matches tool result
• no unnecessary API calls
• latency
• token cost

This lets you distinguish:

LLM failure
     vs
tool-definition failure
     vs
API failure
     vs
orchestration failure

That's enormously valuable.

---

What framework should you actually choose?

I'd think about the stack in three layers:

If you're primarily using OpenAI, the Responses API + function calling/Agents SDK + MCP-compatible tools is a very reasonable stack. OpenAI's current function-calling stack supports tool calling and Structured Outputs, and the Agents platform includes tracing.