Building Autonomous AI Agents with Next.js, LangChain, and Vector Databases

Chat completion is a request-response problem. An agent is a control-flow problem. That difference is what makes the second one hard to ship, and it is why an agent that demos well on a Tuesday can be unusable by the time it is in front of real users.

This is the architecture I use for agents built on Next.js, LangChain and a vector store, and the specific failure modes each layer exists to contain.

The five layers, and what each one is responsible for

Separating these is not architectural ceremony. Each boundary is where a specific class of failure gets caught before it reaches the user.

Retrieval is where most agents actually fail

The instinct is to blame the model when an agent answers badly. In practice the model is usually reasoning correctly over bad context. Three retrieval decisions matter more than the choice of model:

Chunking has to follow document structure

Fixed-size chunking – 512 tokens with a 50-token overlap – is the default in every tutorial and it is wrong for most real corpora. It splits tables down the middle and separates a heading from the paragraph that gives it meaning. Chunk on structural boundaries instead: sections, list items, table rows. Then attach the parent headings to each chunk as a prefix, so a fragment retrieved in isolation still carries the context that makes it interpretable.

Pure vector search loses exact matches

Embeddings are good at “documents about this topic” and bad at “the document containing error code E4021”. Product codes, version numbers, function names and identifiers are precisely what technical users search for, and they are precisely what semantic similarity blurs. Run BM25 keyword search alongside vector search and fuse the results – reciprocal rank fusion is a reasonable default and needs no training.

Retrieve wide, then rerank narrow

Pull 40 to 50 candidates, rerank them with a cross-encoder, and pass the top handful to the model. Retrieving the top 5 directly is cheaper and consistently worse, because the embedding that decides similarity has never seen the query and the document together.

// Hybrid retrieval: semantic recall, keyword precision, reranked for relevance.
const [semantic, keyword] = await Promise.all([
  index.query({ vector: await embed(query), topK: 40 }),
  bm25.search(query, { limit: 40 }),
]);

const fused  = reciprocalRankFusion(semantic, keyword);
const ranked = await reranker.rank(query, fused.slice(0, 40));

return ranked.slice(0, 6);

Tools are an API contract, not a prompt instruction

A tool description written in prose is a suggestion. A tool defined by a schema is a contract you can enforce. Define every tool with a validator – Zod, JSON Schema, whatever your stack already uses – and validate the model’s arguments before execution, not after.

Two rules that prevent most of the damage:

• Separate read tools from write tools. Read tools can be retried freely. Write tools need idempotency keys, because “the request timed out” and “the request succeeded and the response was lost” look identical from the caller’s side.
• Return errors to the model as data, not as exceptions. A tool that returns { ok: false, reason: "no customer with that id" } lets the agent correct itself. A thrown exception ends the run and produces a support ticket.

Orchestration: the loop has to be able to stop

An agent loop needs three termination conditions, and all three need to exist before the first user sees it: a maximum step count, a wall-clock deadline, and a token budget for the whole run. Without them, one ambiguous question can produce a conversation that runs until something else times out.

Modelling the loop as an explicit state machine rather than a while loop with an accumulating message array pays for itself the first time you need to answer “what did it actually do?”. Each transition is a record: which tool, which arguments, which result, how many tokens. That record is your debugging story, your cost attribution and your audit trail.

Streaming is a correctness feature, not a nicety

Next.js server components and streaming responses matter here for a reason that is not aesthetic. An agent run takes seconds, sometimes tens of seconds. A page that shows nothing until it finishes gets reloaded by the user, which starts a second run against the same budget while the first is still executing. Streaming intermediate state – which tool is being called, what has been retrieved – is what stops that.

Send cancellation signals through to the orchestrator too. When a user navigates away, the run should stop, not continue burning tokens on an answer nobody will read. The same reasoning applies to any long request; I have written about the wider version of it in Full-Stack Web Performance: From Next.js SSR to Database Query Budgets.

Guardrails and cost

Token cost is not a billing concern, it is a design constraint that changes the architecture. Some things worth deciding early rather than after the first invoice:

• Per-conversation and per-tenant ceilings, enforced in the orchestrator, not in a dashboard alert after the fact.
• Model routing by task. Classification, routing and extraction rarely need your most capable model. Synthesis usually does.
• Cache retrieval, not just completions. Embedding the same query repeatedly is pure waste, and identical queries are more common than they look.
• Trace every run with the retrieved chunk ids attached. When an answer is wrong, the first question is always which documents it saw – and without the trace, that question is unanswerable.

What this looks like in a repository

The retrieval layer described here – hybrid search with reciprocal rank fusion and reranking – is published as a working starter rather than described in the abstract. The provider-integration side of the same architecture, including streaming, structured output validation and fallback routing across providers, is covered in Integrating OpenAI & Claude APIs into Enterprise Web Applications.

Questions people actually ask

Do I need an agent, or would retrieval-augmented generation be enough?

If the task is “answer questions from these documents”, plain RAG is enough and far cheaper to operate. You need an agent when the task requires taking actions with consequences, or when the number of steps is not known in advance. Building an agent for a task that RAG solves adds latency, cost and failure modes for nothing.

Which vector database should I use?

For most applications this matters much less than chunking and reranking. Pinecone and Qdrant are both reasonable defaults, and pgvector is often the right answer if you already run PostgreSQL – one less system to operate. Choose on operational fit; the retrieval quality difference between them is smaller than the difference a reranker makes.

How do you stop an agent from looping forever?

Three limits enforced in the orchestrator: a maximum step count, a wall-clock deadline, and a token budget for the run. Prompt instructions telling the model to stop are not a control mechanism – they are a request, and under an ambiguous input they are ignored.

Why does the agent give confident but wrong answers?

Almost always a retrieval problem rather than a model problem. Check what was actually retrieved for the failing query before changing the prompt. The usual culprits are chunks split across structural boundaries, and pure vector search missing an exact identifier that keyword search would have found immediately.

Can this run on serverless?

Partly. Retrieval and single-step calls fit serverless well. Long multi-step runs collide with execution time limits, so the durable pattern is to start the run, persist state after each step, and let the client resume – which is the same shape as the webhook and queue architecture described in Designing Resilient Multi-Platform Data Pipelines.