Designing Production Agentic AI Systems: Architecture Patterns, Guardrails, and Evaluation

A production agentic AI system is a controlled loop, not a clever prompt: a planner decomposes the goal, tools are called through typed contracts, memory carries state, and a verifier checks every output before it can act. Reliability comes from that scaffolding and from rollout gates that grant autonomy incrementally, not from the model.

#Why the Chat Interface Rarely Becomes a System

What this means for you: if the visible artefact of your AI programme is a chat window, you are funding a demonstration surface rather than an operating capability — and the distance between the two is where most of the budget goes.

Most enterprise AI programmes start with a chat interface. It makes the technology visible to a steering committee inside a fortnight. It also defers every hard part: data integration, grounding, guardrails, and the ability to act against a system of record.

The demo holds up until a business process needs the same answer twice.

Agentic systems are built around goals rather than prompts. The system works out what it needs to know, calls tools to obtain it, checks what came back, and then either acts or escalates.

Identify, call, check, act, escalate. Each of those verbs is an engineering surface with its own failure modes, and production quality is decided by how deliberately each one is designed.

This article is about what happens inside a single agent. Three neighbouring decisions are covered elsewhere: where the agent sits in the wider estate is set out in the four-layer autonomous enterprise architecture; whether a given use case deserves an agent at all is scored in the enterprise agentic AI readiness framework; and how several agents coordinate is covered in LLM orchestration patterns.

#What Makes a System Agentic

What this means for you: four traits separate an agent from a prompt chain, and each one carries a build obligation your team will be asked to evidence before go-live.

Traits three and four are the ones a risk committee will interrogate. They are also the ones most commonly implemented as prompt instructions, which is the same as implementing them nowhere.

#The Agent Loop, Component by Component

What this means for you: production systems are won or lost across four components — planner, tool contracts, memory, and verifier — and each one has a named failure mode you can budget against.

#Reference Architecture: the Agentic Reasoning Loop

[User Intent / Business Goal]
          ↓
┌─────────────────────────────────┐
│       SUPERVISOR AGENT          │◄────┐
│ (Planning & Task Orchestration)  │     │
└────────────────┬────────────────┘     │
                 │                      │
       ┌─────────┴─────────┐            │
       ▼                   ▼            │ [Feedback / Correction]
┌──────────────┐   ┌──────────────┐     │
│  RETRIEVER   │   │   ANALYSER   │     │
│ (Data Access)│   │ (Logic / ML) │     │
└──────┬───────┘   └───────┬──────┘     │
       │                   │            │
       └─────────┬─────────┘            │
                 ▼                      │
┌─────────────────────────────────┐     │
│       VALIDATION AGENT          │─────┘
│ (Hallucination & Policy Check)  │
└────────────────┬────────────────┘
                 ▼
     [Final Action / Response]

In practice the specialists cluster into three types: retrieval specialists that fetch and ground context, analysis specialists that run SQL, computation, or machine learning, and action specialists that execute side effects against operational systems. The distinction matters because each type needs a different approval posture.

#The Planner: Bounded, or It Loops

The planner decomposes the objective into steps, selects tools, and integrates feedback. Its characteristic failures are looping on a failed step, scope drift into a confidently solved adjacent problem, and over-decomposition that burns budget on trivial sub-tasks.

Three mitigations do most of the work. A hard step budget per run. A retry cap per sub-task — typically two or three — after which the planner must change strategy or escalate. And a plan diff check: if a revised plan no longer references the original objective's entities, the run is flagged rather than continued.

Escalation is a designed outcome rather than a failure state. A run that ends with "I could not complete this because X, and here is what I found" is a success for the system even when it is a miss for the task.

#Tool Contracts: the Highest-Leverage Artefact

Tools are the agent's hands. A production tool contract specifies typed inputs and outputs, preconditions, side-effect class (read-only, reversible write, irreversible write), timeout and retry semantics, and a structured error taxonomy.

Untyped tools are the fastest route to an agent that does real damage at speed.

The error taxonomy lets the planner distinguish "retry this" from "replan around this" from "stop and escalate". Irreversible-write tools should additionally demand a human approval token, or run only in the gated rollout stages described below.

The single highest-leverage design decision in most builds is refusing to expose any tool whose failure modes have not been enumerated in writing.

#Memory: Three Horizons, Two Failure Modes

Staleness is the second failure mode, and it is handled with time-to-live metadata on cached retrievals. Each tool contract should state how long its outputs may be trusted.

A customer balance fetched an hour ago is not a fact. It is a dated observation.

Where that governed knowledge comes from is a platform question rather than an agent question. Semantic AI engineering covers how the glossary, entity resolution, and knowledge graph are built and maintained.

#The Verifier: an Independent Check, Not a Second Opinion

The verifier sits between the loop and the outside world, and it runs three classes of check.

On rejection, the verifier returns structured feedback to the planner, and correction cycles are capped. A response that fails verification three times goes to a human with the full trace attached.

A verifier that argues with a planner indefinitely is just a more expensive loop.

#A Worked Example: Tool Contract and Guardrail Config

What this means for you: these two artefacts are what your architecture review should ask for by name, because together they make autonomy an auditable configuration rather than a property of the code.

The following is a generic and deliberately simplified illustration of the pair that every tool integration should ship with — a contract the planner can reason over, and a guardrail config the runtime enforces outside the model.

{
  "tool": "execute_sql_readonly",
  "description": "Run a read-only SQL query against the governed analytics warehouse.",
  "side_effect_class": "read_only",
  "input_schema": {
    "query": {"type": "string", "must_not_contain": ["INSERT", "UPDATE", "DELETE", "DROP", "GRANT"]},
    "max_rows": {"type": "integer", "maximum": 10000},
    "workspace": {"type": "string", "enum": ["finance_curated", "sales_curated"]}
  },
  "output_schema": {
    "rows": "array",
    "row_count": "integer",
    "source_tables": "array",
    "executed_at": "timestamp"
  },
  "errors": {
    "retryable": ["TIMEOUT", "WAREHOUSE_BUSY"],
    "replan": ["TABLE_NOT_FOUND", "COLUMN_NOT_FOUND"],
    "escalate": ["PERMISSION_DENIED", "POLICY_BLOCKED"]
  },
  "timeout_seconds": 60,
  "result_ttl_minutes": 30
}

{
  "guardrails": {
    "per_run_budget": {"max_steps": 12, "max_tool_calls": 25, "max_retries_per_step": 2},
    "data_policies": {
      "pii_columns": "mask_before_model_context",
      "row_level_security": "inherit_from_requesting_user"
    },
    "action_policies": {
      "read_only": "allow",
      "reversible_write": "allow_with_audit_log",
      "irreversible_write": "require_human_approval"
    },
    "escalation": {
      "on_verifier_rejections": 3,
      "on_budget_exhausted": true,
      "route_to": "data-ops-oncall",
      "attach": ["full_trace", "plan_history", "tool_io_log"]
    }
  }
}

Two details carry most of the value. The error taxonomy tells the planner what recovery is legitimate, which closes off the most common class of retry loop.

Autonomy becomes a configuration decision rather than a code change — which is exactly what makes it governable.

That is what the mapping from side_effect_class to action_policies buys you. Expressing those policies as versioned, reviewable artefacts is the same discipline described in governance as code.

#Evaluation Harness Design

What this means for you: you cannot unit-test an agent into reliability, but you can measure it into something you are willing to sign for. A harness is the evidence base for every autonomy decision that follows.

Golden tasks. A versioned suite of end-to-end tasks with known-good outcomes: the input objective, the expected terminal state — an answer, a record change, or an escalation — and acceptance criteria written as executable checks wherever possible.

Golden suites should over-sample the ugly cases: ambiguous requests, missing data, tools that fail mid-run. Those are where agents diverge from demos. Start with a few dozen tasks drawn from real tickets.

LLM-judge scoring, with caveats. For outputs where correctness is not mechanically checkable, a model scoring against an explicit rubric is useful — but it is an instrument that needs calibration, not an oracle.

Known judge biases include position bias in pairwise comparisons, a preference for longer and more confident answers, and leniency toward outputs from models similar to the judge. Mitigate by scoring against written rubrics rather than open-ended preference, randomising comparison order, and periodically auditing a sample of verdicts against human review.

If judge-to-human agreement drifts, fix the rubric before trusting the trend line.

Regression gates. Harness results only matter if they can block a change. Any modification to prompts, tools, models, or retrieval configuration runs the golden suite in CI, and a drop in task success, grounding, or policy compliance beyond an agreed threshold blocks the release.

Track cost and latency per task alongside quality. A change that lifts accuracy while tripling token spend is a decision, and it should be a visible one.

#Production Rollout: Four Gates from Retrieval to Autonomy

What this means for you: autonomy is earned in stages, and each stage has an entry criterion the harness can measure. This table is the one to take into a steering committee.

The pattern worth noticing: humans do not leave the system. They move from being in the loop on every action to being on the loop, setting policy, reviewing escalations, and auditing samples.

#The Numbers a Board Will Ask For

What this means for you: four metrics answer most of the questions an executive committee raises about an agentic system, and all four should exist before the first production run rather than after the first incident.

• Task success rate against the golden suite — whether the system works at all.
• Grounding rate — whether it is inventing, measured rather than assumed.
• Escalation rate — how much human capacity the system actually consumes, which is the number that decides whether the business case holds.
• Cost and p95 latency per completed task — whether it is affordable at volume.

An agent that is correct and unaffordable does not stay in production for long, which is why the last of those four belongs in the service level objectives rather than in a monthly review.

These belong in the same operational reporting as the rest of the platform. The instrumentation discipline is the same one described in continuous data observability, and the pipelines feeding the agent's retrieval layer need the same treatment, which is where data engineering work usually starts.

If you are building this and want a second pair of eyes on the guardrail design, our agentic AI delivery practice reviews exactly these artefacts.

#Frequently Asked Questions

#What is the difference between an agent and a prompt chain?

A prompt chain is a fixed sequence of model calls; if a step fails or the input deviates, the chain produces a wrong answer or breaks. An agent runs a loop: it plans, calls tools under contracts, checks results, and revises, with explicit budgets and escalation paths. The distinguishing feature is not intelligence but designed failure handling.

#How do you stop an agent from looping forever?

With hard budgets enforced outside the model: a maximum step count per run, a retry cap per sub-task, and a cap on verifier-rejection cycles. Each converts a potential infinite loop into a structured escalation carrying the full trace. Loop prevention belongs in the runtime and guardrail configuration rather than in prompt instructions, because a model cannot be relied on to police its own termination.

#Are LLM judges reliable enough to evaluate agent quality?

They are useful and insufficient. Model judges scale evaluation beyond what humans can review, but they carry known biases: toward longer answers, toward certain positions in pairwise tests, and toward outputs resembling their own style. Treat the judge as an instrument. Score against explicit rubrics, audit samples of its verdicts against human review, and avoid letting a judge-only metric gate irreversible actions.

#When should an agent be allowed to act without human approval?

When three conditions hold: the action is classified read-only or reversible in its tool contract; the agent has cleared the earlier rollout gates on live workload rather than test data; and audit sampling is in place to catch drift after approval is removed. Irreversible actions such as payments, deletions, and external communications should keep a human approval step until the business owner signs the change.

#What does an agentic AI system cost to run in production?

Cost is dominated by tokens consumed per completed task, which is driven by plan length and retry behaviour rather than by the headline model price. Instrument cost per task and p95 latency from the first pilot, alongside accuracy and grounding. Budget caps belong in the guardrail configuration, because an unbounded planner is a spend risk before it is a quality risk.

---

Unolabs is a Data and AI first engineering consultancy, headquartered in the United Kingdom with engineering operations in Pune and active engagements across the UK, Australia, and Hong Kong. We help enterprises build the architectural foundation for autonomous AI execution — governed data platforms, semantic intelligence, and agentic systems that enterprises can stand behind.

If you are deciding how much autonomy to grant an agent and what evidence should justify it, book a discovery call and we will review your tool contracts and guardrail config with you.