Building Production-Ready AI Applications with Node.js | Engineering Blog | Neeraj Prajapati

If you've ever built an AI-powered feature and shipped it, you already know the dirty secret: getting a chatbot or an AI endpoint working in a demo is easy. Getting it to survive real users, real traffic spikes, real API failures, and a real billing dashboard that doesn't make your CFO cry — that's a completely different problem.

I've spent a good chunk of the last year building AI-integrated features into Node.js applications — everything from simple "summarize this document" endpoints to full conversational agents with memory and tool-calling. Along the way I made almost every mistake you can make: no retry logic on flaky model responses, no token budgeting (hello, surprise $400 API bill), no streaming (users staring at a blank screen for 12 seconds), and no proper error boundaries when the model returned malformed JSON in production at 2 AM.

This post is everything I wish someone had told me before I started. We'll go from "it works on my machine" to "it survives Black Friday traffic" — architecture, streaming, error handling, rate limiting, caching, observability, security, and cost control. Grab a coffee, this is a long one.

Why AI Applications Break Differently

Before we write a single line of code, it's worth understanding why AI applications need a different mental model than a typical CRUD API.

A normal REST endpoint is deterministic and fast. You send a request, the database responds in milliseconds, you send back JSON. AI applications flip almost every one of those assumptions:

Latency is unpredictable. A model call might take 300ms or 15 seconds depending on prompt length, load, and output length.
Responses aren't guaranteed to be valid. Ask a model for JSON and sometimes you'll get JSON wrapped in a paragraph of explanation, or a trailing comma that breaks JSON.parse.
Cost scales with usage in a very direct, very visible way. Every token in and out has a price tag. A single careless "include full conversation history" bug can 10x your bill overnight.
Providers rate-limit you, and they will 429 you at the worst possible time — usually right when you're trending on Hacker News.
Failures are partial, not total. A model can "succeed" technically (HTTP 200) while producing a completely wrong or nonsensical answer. Your code needs to handle semantic failure, not just HTTP failure.

Once you internalize this, the rest of the architecture decisions basically write themselves: you need retries, you need streaming, you need queuing, you need caching, and you need to treat the model output as untrusted input — the same way you'd treat anything coming from a user.

Choosing Your Architecture

For most Node.js AI applications, I've settled into a pattern that looks like this:

code

Client (browser/mobile)
      │
      ▼
API Gateway / Express or Fastify server
      │
      ├── Rate Limiter (per-user)
      ├── Auth Middleware
      ├── Request Validator
      │
      ▼
Job Queue (BullMQ + Redis)  ──► Worker Process ──► AI Provider (OpenAI/Anthropic/etc.)
      │                              │
      ▼                              ▼
  Response cache (Redis)        Structured logging + tracing

The key architectural decision is this: don't call the AI provider directly inside your request handler for anything that isn't trivially fast or where a user is actively waiting on a stream. For "fire and forget" or long-running generation tasks (report generation, batch summarization, document processing), push the work into a queue and let a worker handle it. This alone will save you from a huge category of production incidents — timeouts, dropped connections, and duplicate charges from client retries.

For real-time chat-style interactions, you'll want a direct streaming connection (we'll cover that below), but even there, the request should pass through rate limiting and validation first.

Project Setup

Let's set up a lean but production-shaped Node.js project. I'm using Fastify here because it's noticeably faster than Express for high-throughput APIs, but everything below applies to Express with minor syntax differences.

bash

mkdir ai-production-app && cd ai-production-app
npm init -y
npm install fastify @fastify/rate-limit @fastify/cors
npm install ioredis bullmq
npm install zod
npm install pino pino-pretty
npm install openai
npm install dotenv

A sane folder structure keeps this maintainable as it grows:

code

src/
  config/
    env.js
  lib/
    aiClient.js
    cache.js
    logger.js
    queue.js
  routes/
    chat.js
    generate.js
  middleware/
    rateLimiter.js
    validateRequest.js
  workers/
    generationWorker.js
  index.js
.env

Environment config, validated at startup rather than discovered at runtime when something crashes:

// src/config/env.js
import 'dotenv/config';
import { z } from 'zod';

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'production', 'test']).default('development'),
  PORT: z.coerce.number().default(3000),
  OPENAI_API_KEY: z.string().min(1, 'OPENAI_API_KEY is required'),
  REDIS_URL: z.string().url(),
  MAX_TOKENS_PER_REQUEST: z.coerce.number().default(2000),
  RATE_LIMIT_PER_MINUTE: z.coerce.number().default(20),
});

const parsed = envSchema.safeParse(process.env);

if (!parsed.success) {
  console.error('❌ Invalid environment variables:', parsed.error.flatten().fieldErrors);
  process.exit(1);
}

export const env = parsed.data;

This one small habit — validating env vars with Zod at boot instead of letting undefined leak through your app — has saved me from more 3 AM pages than almost anything else on this list. Fail fast, fail loud, fail at startup.

Talking to the Model Provider (Properly)

Here's the naive version everyone writes first:

// ❌ Don't do this in production
const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [{ role: 'user', content: userInput }],
});

This works fine until: the provider has a hiccup, the network blips, or you hit a rate limit. Now your user gets a 500 error and your app looks broken. Instead, wrap every provider call with retries, timeouts, and circuit-breaking logic.

// src/lib/aiClient.js
import OpenAI from 'openai';
import { env } from '../config/env.js';
import { logger } from './logger.js';

const openai = new OpenAI({ apiKey: env.OPENAI_API_KEY });

const RETRYABLE_STATUS_CODES = new Set([429, 500, 502, 503, 504]);

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

export async function callModel({ messages, model = 'gpt-4o', maxTokens = env.MAX_TOKENS_PER_REQUEST, retries = 3 }) {
  let lastError;

  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      const controller = new AbortController();
      const timeout = setTimeout(() => controller.abort(), 30_000);

      const response = await openai.chat.completions.create(
        {
          model,
          messages,
          max_tokens: maxTokens,
          temperature: 0.7,
        },
        { signal: controller.signal }
      );

      clearTimeout(timeout);
      return response;
    } catch (error) {
      lastError = error;
      const status = error?.status;

      logger.warn({ attempt, status, message: error.message }, 'Model call failed');

      // Don't retry on client errors that won't fix themselves (bad request, auth)
      if (!RETRYABLE_STATUS_CODES.has(status)) {
        throw error;
      }

      // Exponential backoff with jitter, capped
      const backoff = Math.min(1000 * 2 ** attempt, 8000);
      const jitter = Math.random() * 300;
      await sleep(backoff + jitter);
    }
  }

  throw lastError;
}

A few things worth calling out here, because each one is a lesson learned the hard way:

AbortController with a timeout. Without it, a hung request from the provider will hold your Node.js event loop's resources indefinitely, and under load that becomes a resource leak that slowly chokes your server.
Only retry retryable errors. Retrying a 400 (bad request) or 401 (invalid key) three times just wastes time and money — it's not going to fix itself.
Exponential backoff with jitter. If ten of your server's requests all hit a rate limit at once, and they all retry after exactly 1 second, you've just created a thundering herd that hits the rate limit again. Jitter spreads that out.

Streaming Responses to the Client

Nothing kills the feel of an AI feature faster than a spinner sitting on screen for 8 seconds before any text appears. Streaming token-by-token responses makes even a slow model feel fast, because the user sees progress immediately.

Here's a streaming endpoint using Server-Sent Events (SSE), which I generally prefer over WebSockets for this use case — it's simpler, works over plain HTTP, and browsers handle reconnection for you.

// src/routes/chat.js
export async function chatRoute(fastify) {
  fastify.post('/api/chat', async (request, reply) => {
    const { message, conversationId } = request.body;

    reply.raw.setHeader('Content-Type', 'text/event-stream');
    reply.raw.setHeader('Cache-Control', 'no-cache');
    reply.raw.setHeader('Connection', 'keep-alive');
    reply.raw.flushHeaders();

    try {
      const stream = await openai.chat.completions.create({
        model: 'gpt-4o',
        messages: [{ role: 'user', content: message }],
        stream: true,
      });

      for await (const chunk of stream) {
        const token = chunk.choices[0]?.delta?.content;
        if (token) {
          reply.raw.write(`data: ${JSON.stringify({ token })}\n\n`);
        }
      }

      reply.raw.write('data: [DONE]\n\n');
      reply.raw.end();
    } catch (error) {
      reply.raw.write(`data: ${JSON.stringify({ error: 'Stream failed, please retry.' })}\n\n`);
      reply.raw.end();
    }

    // Clean up if the client disconnects mid-stream
    request.raw.on('close', () => {
      reply.raw.end();
    });
  });
}

A subtle but important detail: listen for client disconnects. If a user closes the tab mid-generation, you want to stop consuming (and paying for) tokens from the provider rather than generating a full response into the void. Most provider SDKs let you call .abort() on the stream when the client connection closes — wire that up, it's an easy win for cost control.

Handling Errors Like an Adult

AI responses fail in three distinct ways, and each needs different handling:

Transport failures — network errors, timeouts, provider downtime. Handled by the retry logic above.
Malformed output — you asked for JSON, you got prose with JSON embedded in it.
Semantic failures — the call succeeded, the JSON parsed fine, but the content is wrong, off-topic, or the model refused the request.

For malformed structured output, never trust JSON.parse blindly. Extract and validate:

import { z } from 'zod';

const responseSchema = z.object({
  summary: z.string(),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
  keyPoints: z.array(z.string()).max(5),
});

function extractJson(rawText) {
  // Models sometimes wrap JSON in markdown fences or add explanation text
  const match = rawText.match(/\{[\s\S]*\}/);
  if (!match) throw new Error('No JSON object found in model response');
  return JSON.parse(match[0]);
}

function parseModelResponse(rawText) {
  const extracted = extractJson(rawText);
  const result = responseSchema.safeParse(extracted);

  if (!result.success) {
    logger.error({ errors: result.error.flatten(), rawText }, 'Model returned invalid schema');
    throw new Error('Model response failed validation');
  }

  return result.data;
}

And then have a real fallback strategy — not just a try/catch that returns a 500. Depending on the feature, that might mean: retry once with a stricter prompt ("respond with ONLY valid JSON, no other text"), fall back to a smaller/faster model, or gracefully degrade to a canned response like "I couldn't process that, please rephrase."

async function getSummaryWithFallback(text) {
  try {
    const raw = await callModel({ messages: buildSummaryPrompt(text) });
    return parseModelResponse(raw.choices[0].message.content);
  } catch (firstError) {
    logger.warn('First attempt failed, retrying with stricter instructions');
    try {
      const raw = await callModel({ messages: buildStrictSummaryPrompt(text) });
      return parseModelResponse(raw.choices[0].message.content);
    } catch (secondError) {
      logger.error({ secondError }, 'Both summary attempts failed');
      return { summary: 'Summary unavailable right now.', sentiment: 'neutral', keyPoints: [] };
    }
  }
}

The user gets something usable instead of a spinning wheel or a scary error page. That's the difference between an app that feels solid and one that feels like a science project.

Rate Limiting and Queuing

Two separate concerns get lumped together here, and it's worth separating them clearly:

Rate limiting protects you from abuse and controls cost — limiting how often a given user can hit your AI endpoints.
Queuing protects your provider account from being rate-limited by them, and smooths out bursty traffic.

For per-user rate limiting, Fastify's plugin makes this straightforward with Redis as the store (so it works across multiple server instances):

// src/index.js
import rateLimit from '@fastify/rate-limit';
import Redis from 'ioredis';
import { env } from './config/env.js';

const redis = new Redis(env.REDIS_URL);

await fastify.register(rateLimit, {
  max: env.RATE_LIMIT_PER_MINUTE,
  timeWindow: '1 minute',
  redis,
  keyGenerator: (request) => request.user?.id || request.ip,
});

For heavier generation work — batch document processing, report generation, anything that doesn't need an instant response — use a real job queue. BullMQ (built on Redis) is the standard choice in the Node.js ecosystem:

// src/lib/queue.js
import { Queue } from 'bullmq';
import { env } from '../config/env.js';

export const generationQueue = new Queue('ai-generation', {
  connection: { url: env.REDIS_URL },
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
    removeOnComplete: { age: 3600 }, // clean up after 1 hour
    removeOnFail: { age: 86400 },    // keep failures for a day to debug
  },
});

// src/workers/generationWorker.js
import { Worker } from 'bullmq';
import { callModel } from '../lib/aiClient.js';
import { env } from '../config/env.js';

const worker = new Worker(
  'ai-generation',
  async (job) => {
    const { prompt, userId } = job.data;
    const result = await callModel({ messages: [{ role: 'user', content: prompt }] });
    // Store result somewhere the user/client can retrieve it (DB, cache, websocket push)
    return result.choices[0].message.content;
  },
  {
    connection: { url: env.REDIS_URL },
    concurrency: 5, // tune this based on your provider's rate limits
  }
);

worker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed:`, err.message);
});

The concurrency setting is your main lever for staying under provider rate limits — set it deliberately based on your actual API tier limits, not just whatever number felt right at 11 PM.

Caching to Save Money and Latency

This is the single highest-leverage optimization for both cost and speed, and it's the one people skip most often because "AI responses are all different anyway." In practice, a surprising number of AI requests are semantically identical or close enough — repeated FAQs, common document types, popular queries.

A basic exact-match cache with Redis:

// src/lib/cache.js
import Redis from 'ioredis';
import crypto from 'crypto';
import { env } from '../config/env.js';

const redis = new Redis(env.REDIS_URL);

function hashPrompt(messages, model) {
  const payload = JSON.stringify({ messages, model });
  return crypto.createHash('sha256').update(payload).digest('hex');
}

export async function getCachedOrCall(messages, model, callFn, ttlSeconds = 3600) {
  const key = `ai:response:${hashPrompt(messages, model)}`;

  const cached = await redis.get(key);
  if (cached) {
    return { ...JSON.parse(cached), fromCache: true };
  }

  const result = await callFn();
  await redis.setex(key, ttlSeconds, JSON.stringify(result));
  return { ...result, fromCache: false };
}

For fuzzier matching (semantically similar but not identical prompts), you'd reach for a vector database (Pinecone, pgvector, Qdrant) to cache based on embedding similarity rather than exact string match — worth it once you're at scale and see the same kinds of questions repeatedly, just phrased differently.

A word of caution: never cache responses that include user-specific or time-sensitive data unless your cache key accounts for that. I've seen a bug where "what's my account balance" got cached and served to a different user because the cache key didn't include the user ID. Always scope your cache keys carefully.

Prompt and Context Management

As conversations grow, so does your token count — and your bill. A common mistake is sending the entire conversation history on every single request without ever trimming it.

function trimConversation(messages, maxTokens = 3000) {
  // Rough estimate: ~4 characters per token
  const estimateTokens = (text) => Math.ceil(text.length / 4);

  let totalTokens = 0;
  const trimmed = [];

  // Always keep the system message
  const systemMessage = messages.find((m) => m.role === 'system');
  if (systemMessage) totalTokens += estimateTokens(systemMessage.content);

  // Walk backwards from the most recent message, keep what fits
  for (let i = messages.length - 1; i >= 0; i--) {
    if (messages[i].role === 'system') continue;
    const tokens = estimateTokens(messages[i].content);
    if (totalTokens + tokens > maxTokens) break;
    trimmed.unshift(messages[i]);
    totalTokens += tokens;
  }

  return systemMessage ? [systemMessage, ...trimmed] : trimmed;
}

For longer-running conversations, a better pattern than raw trimming is summarization — periodically compress older turns of the conversation into a short summary message, so the model retains context without paying full token price for every historical message. Many production chat apps trigger this once a conversation crosses a threshold (say, 10 turns), replacing the oldest turns with a single "Previous conversation summary: ..." system-style message.

Tool Calling / Function Calling Safely

If your AI app uses tool/function calling (letting the model trigger real actions — searching a database, sending an email, calling an internal API), treat every tool call the model requests as untrusted input that needs validation, exactly like you'd validate a form submission from a browser.

const availableTools = {
  searchOrders: async ({ userId, query }) => { /* ... */ },
  sendEmail: async ({ to, subject, body }) => { /* ... */ },
};

const toolSchemas = {
  searchOrders: z.object({ userId: z.string().uuid(), query: z.string().max(200) }),
  sendEmail: z.object({ to: z.string().email(), subject: z.string().max(200), body: z.string().max(5000) }),
};

async function executeToolCall(toolCall, context) {
  const { name, arguments: argsJson } = toolCall.function;

  const tool = availableTools[name];
  const schema = toolSchemas[name];

  if (!tool || !schema) {
    throw new Error(`Unknown tool requested: ${name}`);
  }

  const parsedArgs = schema.parse(JSON.parse(argsJson));

  // Critical: never let the model supply userId or auth-sensitive fields directly.
  // Inject them from your actual authenticated session context instead.
  const safeArgs = { ...parsedArgs, userId: context.authenticatedUserId };

  return tool(safeArgs);
}

That last comment is the important one. If a malicious or simply confused prompt causes the model to hallucinate a userId field pointing at someone else's account, and your code blindly trusts it, you have an authorization bypass — one that came from an AI model rather than a hacker, but is just as real a security hole. Always inject sensitive identity fields from your own trusted session/auth context, never from model output.

Observability: Logs, Metrics, and Tracing

When something goes wrong in a normal API, you check the stack trace. When something goes wrong in an AI feature, you need to know: what prompt was sent, what model responded, how many tokens were used, how long it took, and whether it was served from cache. Structured logging with Pino makes this queryable later:

// src/lib/logger.js
import pino from 'pino';
import { env } from '../config/env.js';

export const logger = pino({
  level: env.NODE_ENV === 'production' ? 'info' : 'debug',
  transport: env.NODE_ENV !== 'production' ? { target: 'pino-pretty' } : undefined,
});

export function logAiCall({ model, promptTokens, completionTokens, durationMs, fromCache, userId }) {
  logger.info(
    {
      event: 'ai_call',
      model,
      promptTokens,
      completionTokens,
      totalTokens: promptTokens + completionTokens,
      durationMs,
      fromCache,
      userId,
      estimatedCostUsd: estimateCost(model, promptTokens, completionTokens),
    },
    'AI call completed'
  );
}

Feed these structured logs into whatever metrics platform you already use (Grafana, Datadog, CloudWatch) and build a couple of dashboards early:

Token usage and estimated cost, over time and per user
P50/P95/P99 latency for model calls
Error rate by error type (timeout vs. rate limit vs. validation failure)
Cache hit rate

That last one — cache hit rate — is one of the most satisfying metrics to watch climb after you ship caching. It's a direct line to your monthly bill going down.

Security Considerations

AI features open up attack surfaces that don't exist in typical web apps. A few worth taking seriously:

Prompt injection. If your app processes user-supplied or third-party content (a webpage, an uploaded PDF, a customer email) and feeds it to the model alongside your instructions, an attacker can embed hidden instructions in that content trying to hijack the model's behavior. Mitigate this by clearly separating system instructions from untrusted content (using structured message roles, not just string concatenation), and by never letting model output alone trigger destructive actions without a validation layer.

Data leakage. Be deliberate about what you send to third-party model providers. Strip PII where you can, and understand your provider's data retention policy before you send anything sensitive through the API.

Output sanitization. If you're rendering model output directly into a webpage, treat it like any other user-generated content — sanitize before rendering to avoid XSS, especially if the model is capable of outputting HTML or Markdown that gets rendered client-side.

API key hygiene. This one's basic but still gets missed: never expose your provider API key to the client. All model calls should be proxied through your server. It sounds obvious until someone on the team ships a "quick prototype" that calls OpenAI directly from a React app, and now your key is sitting in the browser bundle.

Cost Control

Beyond caching (covered above), a few more concrete levers:

Set hard per-user or per-organization spend caps, enforced server-side, not just monitored after the fact.
Use the smallest model that does the job. Not every task needs your most expensive model — a classification or extraction task often works fine on a cheaper, faster model, reserving the expensive one for tasks that genuinely need deeper reasoning.
Cap max_tokens deliberately per use case, rather than leaving it unset or generously high everywhere.
Track cost per feature, not just per app. It's common to discover that one feature (say, a "regenerate 5 variations" button) is responsible for a wildly disproportionate share of your bill.

async function checkSpendLimit(userId) {
  const key = `spend:${userId}:${new Date().toISOString().slice(0, 7)}`; // monthly key
  const currentSpend = parseFloat((await redis.get(key)) || '0');

  if (currentSpend >= MONTHLY_SPEND_LIMIT_USD) {
    throw new Error('Monthly usage limit reached');
  }
}

async function recordSpend(userId, costUsd) {
  const key = `spend:${userId}:${new Date().toISOString().slice(0, 7)}`;
  await redis.incrbyfloat(key, costUsd);
  await redis.expire(key, 60 * 60 * 24 * 35); // auto-cleanup after ~35 days
}

Testing AI Features

You can't unit test "did the model give a good answer" the way you test a pure function, but you absolutely can and should test everything around the model call:

Test your validation and parsing logic with a fixture set of realistic (and intentionally malformed) model outputs.
Test your retry/backoff logic by mocking the provider client to throw specific errors and asserting the right retry behavior.
Test your fallback paths by forcing failures and confirming the app degrades gracefully instead of crashing.
Snapshot test your prompts. Prompts are code — track changes to them in version control and review diffs the same way you'd review a logic change, because a small prompt tweak can meaningfully change output quality.

For the model output itself, an evaluation harness (running a fixed set of test prompts against the model periodically and scoring outputs, either with rules or a second model as judge) is a worthwhile investment once your AI feature matters to the business — it catches quality regressions when you change prompts or switch model versions.

Deployment Checklist

Before shipping an AI feature to production, I run through this list every time:

If you can check every box on this list, you're in genuinely good shape — better than most AI features I've seen shipped, honestly.

Final Thoughts

The gap between an AI demo and a production AI application isn't the model — it's everything you build around the model. Node.js is a genuinely great fit for this work: its event-driven, non-blocking nature handles the I/O-heavy, latency-variable nature of AI provider calls well, and the ecosystem (BullMQ, Redis clients, Fastify, Zod) gives you everything you need to build the reliability layer without reinventing it from scratch.

If there's one takeaway to hold onto, it's this: treat the model like an unreliable third-party service, because that's exactly what it is. Once that mental shift happens, retries, timeouts, caching, queuing, and validation stop feeling like extra work and start feeling like the obvious, necessary foundation they are.

Build accordingly, and your AI features will hold up long after the demo applause fades.

*If you found this useful, I write about Node.js architecture, performance, and building real-world applications.