Architecting AI Powered SaaS: Lessons from Building a Production Multi Agent System

There is a moment, somewhere between the first working prototype and the first paying customer, when every team building with large language models runs into the same wall. The demo that felt magical on Tuesday starts behaving strangely on Thursday. A workflow that handled ten test cases perfectly falls apart on the eleventh. A single well crafted prompt that looked like the whole product quietly reveals itself to be a fragile hinge holding up an entire business.

What follows is a field report drawn from time spent building production systems on top of language models. It is not a survey of frameworks or a celebration of what is possible in theory. It is a set of architectural lessons earned the hard way, written for engineers and technical founders who are past the prototype stage and are now trying to ship something real.

Why Just Calling the API Stops Working

The first version of almost every AI product looks the same. A user types something into a box, the application forwards that text to a model endpoint, the response comes back, and the interface renders it. For a surprisingly wide class of problems, this is enough to build something that feels genuinely useful. It is also enough to fool the team into thinking the hard part is over.

The hard part has not even started.

The trouble is that a single model call is stateless, unstructured, and non deterministic. Every one of those properties is fine in isolation and catastrophic in combination once the product has to do real work. Real work means remembering what happened yesterday, coordinating multiple steps, touching a database, calling an external service, and producing results that downstream code can actually rely on. The moment any of those requirements enters the picture, a bare API call stops being an architecture and starts being a liability.

The symptoms appear gradually. First, the output format drifts. The model returns JSON most of the time, but every few hundred requests it decides to wrap the JSON in a polite sentence. Then the model starts hallucinating fields that were never in the schema. Then a user reports that their data was modified in a way no one can explain, and someone on the team discovers that an agent confidently invented a primary key and wrote it into the database. By the time the second or third incident like this happens, the team understands that the model is not the product. The scaffolding around the model is the product, and that scaffolding needs to be designed with the same rigor as any other critical system.

This is the point at which architectural discipline starts paying for itself.

The Functional Core and the Imperative Shell

The single most useful pattern for structuring AI systems is an old one, borrowed from a talk Gary Bernhardt gave more than a decade ago. The idea is to split the system into two layers with very different characters. The functional core is pure. It contains the domain logic, the validation rules, the state transitions, the business invariants. It is made of functions that take inputs and return outputs and have no side effects. The imperative shell is where all the messy things live. It talks to the database, calls the model, sends emails, writes to queues, reads from Redis, and generally deals with the outside world.

In a traditional web application this split is useful. In an AI application it is essential.

The reason is that language model calls are the most unpredictable thing in the stack. They are slow, expensive, occasionally wrong, and structurally impossible to unit test in the normal sense. If they are scattered through the business logic, every function becomes contaminated by that unpredictability. Debugging becomes archaeology. Testing becomes a matter of prayer. But if model calls are treated as effects that belong exclusively in the shell, and if the core consumes the results as plain data, almost all of the properties that make software engineering tractable come back.

In practice this looks like a rule. Model calls happen at the edges. The shell makes the call, receives a response, validates it against a schema, and passes a clean typed object into the core. The core does not know that a model was involved. It sees a parsed result, applies rules, computes the next state, and returns instructions for the shell to carry out. The shell then performs the database writes, the external calls, the notifications. This discipline sounds heavy until you have maintained a codebase that did not follow it. After that, it starts to feel like the only sane way to build.

Contracts, Schemas, and the Art of Distrust

Once the model lives at the boundary, the next question is how to cross that boundary safely. The answer is contracts, and specifically contracts that are enforced at runtime rather than merely hoped for in comments.

Every prompt sent to a model should be paired with a schema. The schema describes the exact shape of the response expected, including field names, types, enumerated values, and constraints like string length or numeric range. When the response comes back, it is parsed against that schema before anything else happens. If the parse fails, the shell retries with a corrective prompt that includes the validation error. If the retry fails again, the request is escalated or routed to a fallback path. The core never sees an unvalidated response. Ever.

This sounds like bureaucratic overhead until you measure what it prevents. In a well instrumented system, a small but non trivial fraction of model responses will have some kind of structural problem that would have caused a downstream bug if it had been trusted. Most are minor. A few are not. Without the schema layer, each of those would be a production incident waiting to happen. With it, they become metrics on a dashboard and retry attempts in a log.

There is a related discipline that matters even more when the model is writing to a database. Call it the branded identifier rule. It saves you from a class of bugs that is almost impossible to catch any other way. The rule is that every identifier in the system carries a type that distinguishes it from every other kind of identifier. A user id is not merely a string. It is a UserId, which is a string with a type level brand that the compiler checks. A project id is a ProjectId. An invoice id is an InvoiceId. The model cannot produce a branded id directly because it produces plain strings. The shell is the only place where strings get turned into branded ids, and only after the id has been verified to exist in the database.

The effect of this rule is that it becomes structurally impossible for an agent to invent a primary key and have it accepted by the core. If the agent hallucinates an id, the lookup in the shell fails and the operation is rejected before any damage is done. If the agent returns a real id that belongs to the wrong kind of entity, the type system catches it. This is the kind of protection that sounds paranoid on day one and looks prescient on day ninety.

Short Term Memory, Long Term Memory, and the Boundary Between Them

An AI product that cannot remember anything feels like talking to someone with severe amnesia. An AI product that remembers everything forever feels like talking to someone who stalks you. The art of memory design is finding the line between those two failures, and in a production system that line almost always runs between Redis and Postgres.

Short term memory is conversational. It holds the current turn, the last several turns, the working context of whatever the user is doing right now. It needs to be fast to read, fast to write, cheap to discard. Redis is almost perfect for this. Conversation state typically lives under a key that includes the session id and expires automatically after some window of inactivity. The shape is a simple list of messages with roles and timestamps, plus a small bag of scratch variables that the agent can use as a notepad during a task. When the session ends, the memory evaporates, and that is exactly what the user expects.

Long term memory is different. It is the stuff that should survive across sessions, across weeks, across devices. A customer's preferences. A project's history. The decisions that were made and the reasons they were made. This belongs in Postgres, in proper tables, with proper foreign keys and proper audit trails. The model never writes to these tables directly. The shell writes to them, after validating that the write is consistent with the rules in the core.

The interesting design question is what moves from short term to long term, and when. The naive answer is to summarize every session and store the summary. This works poorly in practice because summaries compound errors. A small misreading in one session becomes a slightly larger misreading in the next, and after a month the long term memory is a museum of misunderstandings. The better answer is to treat long term memory as an explicit consequence of explicit actions. When a user confirms a decision, write it. When a task is completed, record it. When a preference is stated clearly and acted upon, store it. Let the model propose what should be remembered, and let the user or the system confirm what actually gets written. Memory then becomes a deliberate act rather than a silent accumulation.

For retrieval, vector search has its place, but it is less useful than people claim. Most of the time, a well designed relational query over properly tagged records outperforms a vector similarity search, because the questions users actually ask are more structured than they appear. Vectors shine when the query is genuinely fuzzy and the corpus is genuinely large. For everything else, boring database queries win.

When to Use Multiple Agents and When One Good Prompt Wins

The phrase multi agent system has become a kind of shibboleth, signaling seriousness about AI architecture. Some of that signal is deserved. Much of it is not. Teams that reach for multiple agents early tend to regret it, and teams that keep things as simple as possible for as long as possible tend to build something more reliable.

The honest rule of thumb is this. If a task can be accomplished by a single well structured prompt calling a small set of tools, do that. If the task genuinely involves distinct phases with distinct reasoning styles and distinct success criteria, consider splitting it into multiple agents. The test is whether each agent's responsibility can be described in one sentence without using the word and. If it cannot, the split is wrong and should collapse back into one.

When multiple agents are actually warranted, the architecture that works best is a coordinator pattern. A single top level agent receives the user request and decides which specialized agents to invoke. Each specialized agent has a narrow remit, a small tool set, and a clear output contract. The coordinator is the only agent that talks to the user. The specialists talk only to the coordinator. This creates a tree rather than a graph, and trees are vastly easier to reason about than graphs when something goes wrong.

The failure mode to avoid is the peer to peer swarm where every agent can talk to every other agent. It looks elegant in diagrams and behaves like a nightmare in production. Messages get duplicated, loops appear, costs explode, and debugging becomes an exercise in reconstructing a conversation between three or four different model instances from log files. Unless there is a very specific reason to build that shape, do not build it.

One more point on orchestration. The coordinator should be allowed to fail gracefully. If a specialist agent returns something malformed, the coordinator should notice, decide whether to retry, and if retries fail, fall back to a simpler path or escalate to a human. Agents that assume their subordinates always succeed are the agents that produce the worst production incidents. Build for the case where every call might fail and the system still has to give the user something useful.

Cost, Latency, and the Observability Gap

A production AI system has three budgets, and they are all tighter than you expect.

The first is the cost budget. Model calls are not free, and a workflow that casually chains five or six calls to a large model can turn a cheap feature into an expensive one very quickly. The discipline here is to ask, for every call, whether a smaller model would do. A routing step that decides which agent to invoke almost never needs a frontier model. A formatting step that turns structured data into a user facing sentence almost never needs a frontier model. Save the big models for the parts of the workflow where reasoning actually matters, and use small fast models for everything else. The cost savings are often an order of magnitude, and the latency savings come along for free.

The second budget is the latency budget. Users will tolerate a few seconds of thinking if it is visibly productive. They will not tolerate thirty seconds of silence. This means that long running workflows need to stream their progress. Not just the final tokens, but the intermediate steps. When the coordinator decides to invoke a specialist, the user should see that. When a database lookup is in flight, the user should see that. Streaming is not a visual effect. It is a trust mechanism. It tells the user that something is happening, and it buys the system the time it needs to actually think.

The third budget is the observability budget, which is to say the time you will spend debugging when something goes wrong. This one is insidious because you do not notice you are overspending until you are already overspent. The fix is to build observability in from the first day. Every model call should be logged with its prompt, its response, its latency, its cost, and the context in which it happened. Every agent invocation should carry a trace id that links it to the user request that spawned it. Every database write that resulted from an agent decision should carry a pointer back to the conversation that caused it. When something breaks at two in the morning, there will not be time to add instrumentation. There will only be time to read what is already there.

The single most useful observability tool to build is a replay viewer. Given a trace id, it reconstructs the entire sequence of events that made up a request. User input, every prompt sent to every model, every response received, every schema validation, every database operation, every retry. Seeing the whole sequence at once is how you understand what the system actually did, as opposed to what you thought it did. Build this early. You will use it more than you expect.

Voice as a Stress Test for AI Architecture

Voice is worth thinking about because it takes every principle discussed so far and pushes it to the breaking point. Two brutal constraints make it the ultimate test case. The first is latency, because humans on phone calls tolerate silence for about one second before they start feeling uncomfortable and about three seconds before they hang up. The second is irreversibility, because a voice conversation cannot be edited after the fact the way a chat message can. Whatever the agent says, it said, and the human heard it.

The obvious first approach to voice is the one most teams try. Speech to text, then a single large model call to decide what to say, then text to speech. It works in demos and fails under real load, because the total latency runs four to six seconds per turn and the conversations feel broken. Users start talking over the agent, the agent gets confused, and the whole thing spirals.

The approach that actually works is built around a different principle. Instead of one slow decision, many fast ones. A small cheap model handles turn detection and intent classification in parallel with the transcription. A second small model handles short acknowledgments and filler phrases that can be spoken immediately, buying time. A larger model handles the actual reasoning about what to do next, and its output is streamed directly into the speech synthesis layer so that the agent starts talking as soon as the first tokens arrive. Perceived latency drops from six seconds to under one, not because any single component got faster, but because the pipeline stopped waiting for itself to finish before producing audio.

This is, in a sense, the whole article compressed into one example. The model is not the product. The orchestration around the model is the product. The decisions about where to put latency, where to accept cost, where to enforce contracts, where to keep memory, and where to let the system fail gracefully are what separate a demo from a service.

The Stack That Actually Works

People ask which stack works best, and the honest answer is that the specific technologies matter less than the properties you should demand from whatever you choose.

You need a language and runtime with a strong type system, because the branded identifier discipline and the schema validation discipline both depend on having types you can trust. TypeScript on Node is a reasonable choice. Other options are fine as long as they give you the same guarantees.

You need a relational database that the team understands deeply, because long term memory lives there and because the audit trails that let you debug production incidents live there. Postgres is the obvious choice and rarely the wrong one.

You need a fast in memory store for short term state and for rate limiting and for job coordination. Redis is almost universal here for good reason.

You need a query layer that makes it easy to express complex reads without losing sight of what the database is actually doing. The important property is not the specific library. It is that you can see what the queries look like and reason about their performance.

You need a job runner, because anything that takes more than a few seconds belongs in the background, and anything that might need to retry belongs behind a queue. The specific choice matters less than having one and using it consistently.

And you need a deployment story that lets you roll back quickly, because the first time a model provider changes the behavior of an endpoint you thought was stable, you will want to be able to revert prompts and redeploy within minutes rather than hours.

None of this is exotic. All of it is boring in the best sense of the word. The exciting part of the system is the product. The infrastructure should be as dull and dependable as you can make it.

The Mindset Shift That Makes Everything Else Possible

If there is one message worth leaving with, it is not about any specific framework or technique. It is a shift in mindset. The shift is to stop thinking of the language model as a solution and start thinking of it as a component, with the same status as a database or a message queue or a third party API. A powerful and unusual component, certainly. But a component, not a replacement for engineering.

Everything in this article follows from that shift. The functional core and imperative shell pattern follows from it, because you only bother to isolate components that you do not fully trust. The schema and contract discipline follows from it, because you only bother to validate the output of components whose output might be wrong. The memory architecture follows from it, because you only bother to separate short term and long term state when you understand that the component holding the short term state is not the same thing as the component holding the truth. The observability investment follows from it, because you only bother to instrument components whose behavior you need to inspect after the fact.

The teams that struggle are almost always the ones that treated the model as magic and built everything else as an afterthought. The teams that succeed are the ones that treated the model as a powerful but fallible tool and built the rest of the system to contain its failures and amplify its strengths.

There is a common worry in this field that the rapid pace of model improvement will obsolete any architecture you design. The worry is understandable and mostly wrong. Models will get better. What they are good at will expand. But the need for structured contracts, durable memory, safe database access, observable workflows, and graceful failure handling will not go away. Those are properties of the systems we build, not properties of the models inside them. A system designed around those principles will get better automatically as the models it contains get better, because the scaffolding is not what is holding it back. A system designed without them will not, because no amount of model improvement can fix an architecture that was never built to be fixed.

Build the scaffolding. Treat the model with respect and suspicion in equal measure. Put contracts in writing and enforce them in code. Know where memory lives and why. Keep agents few and the coordinator simple. Log everything. Stream progress. And remember that the goal is not to build the most impressive demo. The goal is to build a system that still works on the Tuesday six months from now when a user does something none of the team anticipated and the product has to respond gracefully anyway.

That is what production means. That is what the work actually is. And if you do it well, the result is something genuinely new in the world, which is the part that makes all of the rest of it worthwhile.