A production-ready agentic system is shaped by decisions made before the first agent is built!

Agentic systems are easy to demonstrate and difficult to operate. A prompt, an orchestrator, a few specialised agents and a vector store can produce an appealing prototype in an afternoon.

However, production introduces a different set of conditions: variable traffic, incomplete data, ambiguous requests, latency expectations and a cost model that must survive procurement.

Under these conditions, agents may start to retrieve the wrong facts, resolve the same entity differently, or incur a token bill that invalidates the business case.

These failures are often attributed to the model. More often, they are architectural consequences of decisions made before the first agent even ran.

Software engineering calls the corrective principle shift-left. Address quality concerns during design when the system can still accommodate them, rather than discovering them in production, when the cost of change is highest. Agentic systems demand an especially strong version of this principle. Their critical properties are not features to add after deployment; they emerge from how the system is decomposed, evaluated, supplied with context, and grounded in enterprise data. Across enterprise AI engagements, four design commitments repeatedly determined whether a prototype could become a dependable product.

These are:
• Accuracy: behaviour is evaluated continually rather than inferred from a convincing demo.
• Reusability: capabilities can be adapted to new use cases without entirely rebuilding them.
• Performance: latency and cost constrain the architecture from the outset.
• Coherence: independent agents operate over the same grounded representation of reality.

Figure 1: The four design commitments surrounding an enterprise agentic system

All these commitments rest on AI readiness of data.
An agent cannot reason reliably over data that is poorly organised, ambiguously defined, or missing the metadata needed to interpret it. These considerations need to be addressed during design, rather than after the system begins to scale or struggles in production.

The sections below examine each commitment through its mechanism and operational consequence.

Accuracy

Accuracy begins with a discipline that agentic systems make easy to neglect: do not trust a system just because it performs well on a small, hand-picked set of questions.
For instance, a SQL agent we built for a client, who is an automobile manufacturer, translated natural-language questions into SQL queries convincingly in isolation. Broader testing with the large quantity of data ingested from automobile sensors, exposed a more serious behaviour: the agent generated syntactically valid SQL against tables that did not even exist. The queries looked plausible, but their answers had no grounding in the underlying schema. We added a schema validator and a correction loop that rewrites a failing query before execution. More importantly, we built the evaluation framework that made this failure visible. 

That framework needs, at the very least, two complementary forms of test:

Component evaluations: 
• Assess a single agent against a golden dataset of inputs and known-correct outputs.
• They provide the localisation benefits of unit tests: when behaviour degrades, the affected component is apparent.

End-to-end evaluations
• Assess the complete, often multi-turn flow. They must assert the path that an agentic system takes to get to an answer, as well as the final answer.
• A question about mileage of a vehicle should reach the API agent to call the mileage-API; a policy question should reach the policy agent to retrieve the vehicle insurance policy terms and reason over it.
• An orchestrator that obtains a correct answer through an accidental route has not demonstrated reliable behaviour.

Figure 2: Two complementary evaluation tiers: component evaluation for an individual agent and end-to-end evaluation for the entire system

Two practices make these evaluations durable.

• Use an LLM judge only where the output is inherently language-shaped and cannot be assessed programmatically. Deterministic behaviour deserves deterministic assertions, which are cheap, fast and easier to diagnose.
• Build an evaluation store early: a replayable set of representative cases large enough to expose regressions. Once that store exists, a model swap, prompt change, or routing change becomes a measured experiment rather than a leap of faith. Accuracy needs to be re-evaluated after every material change to the system.

Reusability

Enterprises rarely need a single-purpose agent.
They need a capability, be it search, triage, query, or resolution, that can serve the next use case without a substantial rewrite. In a client engagement that I was part of, we built an AI-search framework for a vehicular fleet-management application. This app helped users with live tracking of vehicles, historical data-driven insights on vehicle health and operational efficiency of the fleet.
The agentic system consisted of a SQL agent, API agent and a retrieval tool, among other use-case specific agents. Later, we were asked to support a sales assistant for small commercial vehicles and a resolution assistant for the finance team, banking on the same AI-search framework. That extension was possible because the original system was assembled from independently replaceable components, not wired around one application.

Reusability begins within the agent itself. Its prompt, model, and harness should each be engineered as modular components, because a new use case commonly requires changes to all three. The prompt should be a set of composable layers: role, domain expertise, persona, guardrails, and task instructions, rather than one indivisible block of text. The model should be plug-and-play, selected according to the capability, latency, and cost requirements of the task. The harness should separate pre-processing and post-processing so that input normalisation, retrieval, validation, and output formatting can be re-engineered without entangling the rest of the agent.

For instance, a new sales assistant may need different domain-prompt layers, a less expensive model for a bounded task, and different input and output processing from the AI-search assistant for fleet management. With a prompt library, a model interface, and a modular harness, these changes become deliberate substitutions rather than requiring reconstruction.

Figure 3: Reuse depends on making the agent's sources of variation explicit, so that new use cases require controlled adaptation rather than wholesale reconstruction.

At system scale, this means defining components by their responsibilities and the contracts between them, rather than building a single large prompt-driven application, a design approach I discuss in  my earlier piece on the mental model for agentic system design . This makes it possible to add or replace a retrieval path, validator, domain agent, or model without redesigning the entire flow. It also keeps complexity proportional to the problem. Anthropic recommends simple, composable patterns over heavy frameworks ( Building Effective Agents , 2024), while OpenAI recommends starting with a single agent and adding further agents only when a specific bottleneck justifies them ( A Practical Guide to Building Agents , 2025). The architecture should therefore remain modular enough to accommodate additional agents or capabilities without requiring a rebuild.

Performance

Latency and cost are often considered close to deployment, when they can expose architectural decisions that are difficult to change. We encountered this in a voice assistant, where even modest delay degrades the interaction. The first design placed every capability and tool description in one orchestrator. It was conceptually simple, but every call carried the full prompt and tool surface. Splitting responsibilities by capability reduced the prompt context each agent had to process. Additionally, we performed a usage data analysis from production to identify FAQs, for which we cached the SQL queries needed. In runtime, these were retrieved using a semantic matching between user questions and FAQs. These 2 strategies helped us bring down the latency by 10x.

The cost of agentic execution also depends on how work is allocated across the system. Anthropic reports that agents use roughly four times the tokens of a chat interaction, while multi-agent systems use about fifteen times as many ( How we built our multi-agent research system , 2025). On an early seven-agent design, a projection using a frontier model at prevailing prices reached several crores of Indian rupees annually before optimisation. That estimate changed the architecture before implementation, not because the model was unsuitable, but because the work had not been allocated according to its difficulty.

Two mechanisms address this. A mixture of models assigns a capable large model to orchestration and reserves smaller models for narrow, well-bounded tasks such as validation, formatting, and API invocation; escalation is justified only when evaluation shows that the smaller model is insufficient. Prompt caching reduces the repeated cost of stable prompt fragments, and is available across Anthropic, OpenAI, and Google. These mechanisms depend on the same component boundaries that enable reuse. A modular prompt can be cached; a bounded task can be assigned to an appropriately priced model.

Figure 4: Model selection and prompt caching reduce the cost of execution by matching model capability to task complexity and reusing stable context.

A multi-agent design is justified only when its additional coordination cost is lower than the bottleneck it removes. Latency and spend should be modelled during design, when they can still influence the architecture, rather than treated as afterthoughts in production.

Coherence

Coherence is a failure mode we have observed that is difficult to detect at the component level. A system may be reusable, accurate at the component level, and performant, yet fail because its agents construct incompatible interpretations of the same request. We use coherence to mean that independent agents resolve shared entities consistently and act on a common representation of the world.In the fleet-management AI assistant system the we built, two agents received the same vehicle registration number and resolved it independently. One associated it with one vehicle record; the other selected another. Neither response was obviously hallucinatory. Each agent had made a locally plausible choice. The system is now left with reasoning over incompatible facts, a defect that would not be exposed by testing either agent alone.

The cause is architectural. A conventional orchestrator is often asked to perform two distinct functions: understand the request by extracting and resolving its entities, and coordinate the work needed to answer it. These functions have different failure modes. In entity-heavy domains, like fleets, financial accounts, or parts catalogues, language understanding should be established once, before work is distributed, rather than reconstructed by every downstream agent.

Concretely, entity extraction and resolution become a separate upstream step. It produces a canonical state: a grounded set of entity references that the orchestrator and all downstream agents consume. If a registration number resolves to one vehicle at the head of the pipeline, downstream agents receive that resolution rather than independently attempting it again. This is a practical form of context engineering: curate the context that each agent receives instead of giving every agent all available information ( Effective Context Engineering for AI Agents , 2025). Coherence is not improved by tuning agents one at a time. It is created by establishing a shared boundary for interpretation.

Figure 5: Coherence is established by resolving shared entities once, before distributed work begins

Data readiness

The four commitments above assume that the system has reliable material to reason over. Enterprise tables, APIs, and documents are often inconsistently organised, lack useful metadata, or use conflicting terminology. A well-designed agentic system cannot compensate indefinitely for these data issues. Data readiness is therefore an architectural concern, not preliminary housekeeping; I examine it in more detail in  The AI-Ready Enterprise: Preparing Data for the Age of AI .

Conclusion

Moving from an agentic demonstration to a dependable product requires architectural decisions around reusable boundaries, measurable correctness, performance constraints, and a canonical state for coherent reasoning. These decisions are only as strong as the data foundation beneath them.

Deferring these concerns transfers their cost to production, where dependencies are tighter, failures are harder to localise, and redesign is more expensive. Addressing them during design gives the system a better chance of meeting its operational requirements as it scales.