Short answer
Integrating an LLM API into an enterprise application is a reliability problem, not a prompt-engineering problem. The provider will be slow, will rate-limit you, will occasionally return output that does not match the shape you asked for, and will change model behaviour between versions. Everything below exists to make those events boring rather than incidents.
Calling OpenAI or Claude from a web application takes about four lines. Running that call inside a system that people depend on takes considerably more, and almost none of the additional work is about prompts.
These are the patterns that make the difference between a feature that survives its first bad week and one that gets switched off.
1. Treat structured output as untrusted input
The most common production failure is not a refusal or a hallucination. It is valid-looking output that does not match the shape your code expects – a missing field, a string where a number belongs, a category that is not in your enum.
Both providers support constrained output, and you should use it. But schema enforcement at the API layer is not a substitute for validating on receipt:
const Result = z.object({
category: z.enum(['billing', 'technical', 'account']),
confidence: z.number().min(0).max(1),
summary: z.string().max(280),
});
const parsed = Result.safeParse(JSON.parse(raw));
if (!parsed.success) {
// One retry with the validation error fed back, then fall through
// to a deterministic default. Never surface a parse error to the user.
return retryWithFeedback(raw, parsed.error) ?? FALLBACK;
}The rule that matters: every LLM-derived value crosses a trust boundary. Validate it exactly as you would validate a form submission from a browser.
2. Stream, and make streaming cancellable
Time-to-first-token is the number users experience; total completion time is the number they tolerate. Streaming converts a six-second wait into a response that begins in under a second, and that changes abandonment behaviour more than any latency optimisation on your own infrastructure.
Two details that get missed:
- Propagate cancellation. When the client disconnects, abort the upstream request. Otherwise you keep paying for tokens nobody will see, and under load those orphaned requests consume the rate limit that live requests need.
- Buffer before parsing. If the response is structured, do not attempt to parse partial JSON. Stream it to the user as text if it is prose; accumulate and validate once if it is data.
3. Fallback routing, and what it can and cannot fix
Running against more than one provider is worth the abstraction cost – but only if you are honest about what it buys you. It protects against provider-level outages and rate-limit exhaustion. It does not protect against a bad prompt, and it does not give you identical behaviour across providers.
| Failure | Correct response | Wrong response |
|---|---|---|
| 429 rate limited | Exponential backoff with jitter, then route to secondary | Immediate retry – amplifies the overload |
| 5xx from provider | Retry twice, then route to secondary | Fail the user request on the first error |
| Timeout | Cancel upstream, then route to secondary | Retry without cancelling – now paying twice |
| Schema validation failed | One retry with the error fed back, then deterministic fallback | Route to another provider – the prompt is the problem |
| Content filtered / refused | Surface a specific message to the user | Retry until it complies |
Wrap providers behind a thin internal interface – complete(), stream(), embed() – rather than adopting a heavyweight abstraction layer. You want to be able to read the adapter and know exactly which HTTP request it makes.
4. Pin models and version your prompts
Pin an explicit model version rather than a floating alias. A model that silently updates underneath a prompt tuned against it is an outage you cannot reproduce, because the thing that changed is not in your repository.
Version prompts alongside code and store the prompt version with every logged completion. When output quality changes, the first question is always “what changed?” – and if prompts live in a database edited through an admin screen, that question has no answer.
5. Cost control belongs in the code path
Per-request cost varies by orders of magnitude depending on context length, and context length grows quietly as conversations continue and retrieved documents get attached. Practical controls:
- Route by task. Classification, extraction and routing rarely justify your most capable model; synthesis and reasoning usually do. Routing on task type is the single largest cost lever available.
- Use prompt caching for the stable prefix – system instructions, schemas, few-shot examples. Both providers support it and it is close to free to adopt.
- Cap context explicitly. Truncate conversation history on a token budget, not on a message count.
- Meter per tenant in the request path. A dashboard alert tells you about overspend after it has happened.
6. Observability: log the inputs, not just the outputs
Log the model and version, the prompt version, token counts in and out, latency, retry count, which provider served the request, and – if retrieval was involved – the identifiers of the documents that were attached. Without the inputs, a bad output is not debuggable; you can only guess and re-roll.
Redact before logging. Prompts frequently contain customer data, and an LLM request log is a data store subject to the same retention and access rules as any other.
Where this fits
These patterns are the integration layer underneath the agent architecture described in Building Autonomous AI Agents with Next.js, LangChain, and Vector Databases. If you are adding a single AI feature to an existing product rather than building an agent, this article is the whole job.
Questions people actually ask
- Should I use OpenAI or Claude?
- Test both against your own evaluation set rather than against published benchmarks – task fit varies more than aggregate scores suggest. The more useful decision is to build the adapter so that switching is a configuration change, then let measured results on your workload decide.
- How do I stop the model returning malformed JSON?
- Use the provider’s structured-output or tool-calling mode rather than asking for JSON in the prompt, then validate the result against a schema on receipt anyway. On a validation failure, retry once with the validation error included in the message, and fall back to a deterministic default rather than surfacing a parse error.
- Where should the API key live?
- Server-side only, in a secrets manager, never in an environment variable that reaches the client bundle. All model calls go through your own backend endpoint – which is also the only place you can enforce rate limits, cost ceilings and per-tenant metering.
- How do I handle rate limits at scale?
- Queue rather than retry. Exponential backoff with jitter on the individual request, a bounded concurrency limit across the application, and a durable queue for work that does not need to be synchronous. Retrying immediately on a 429 makes the overload worse for everyone including you.
- Do I need a framework like LangChain for this?
- For a single integration, no – a thin adapter over the provider SDK is easier to reason about and easier to debug. Frameworks earn their cost once you have multi-step orchestration, tool routing and retrieval to coordinate, which is the agent case rather than the integration case.