Cost and Latency Optimization for LLM Apps

LLM API calls are expensive compared to most backend operations, and they are slow by default. In a production application where AI features sit on the critical path, unoptimised LLM calls translate directly into slow pages, frustrated users, and high infrastructure bills.

This post is a practical guide to the levers available for reducing both cost and latency, with concrete patterns for Node.js and Next.js.

Understand Where the Cost Comes From

LLM APIs charge based on the number of tokens processed — both input tokens (the prompt and context you send) and output tokens (the text the model generates). Output tokens are typically priced higher than input tokens. Every character in your system prompt, every turn of conversation history, and every retrieved document chunk is a cost multiplier applied to every request.

Latency, meanwhile, is dominated by two components:

• Time to first token (TTFT): How long until the model starts generating. Affected by network latency, queuing, and prompt length.
• Token generation speed: How fast the model produces subsequent tokens. Shorter outputs finish faster.

Optimisation strategies target one or both of these levers.

Strategy 1: Right-Size Your Model

Using a large, high-capability model for every task is the single easiest way to overspend. The Anthropic family covers a spectrum:

• Claude Haiku 4.5 (claude-haiku-4-5): Fast and inexpensive. Appropriate for classification, simple extraction, guardrail checks, and any task where output quality requirements are modest.
• Claude Sonnet 4.6 (claude-sonnet-4-6): A balanced mid-tier. Good for most generation tasks, summarisation, and moderate reasoning.
• Claude Opus 4.8 (claude-opus-4-8): The most capable default model. Reserve this for complex reasoning, nuanced generation, or tasks where quality directly drives business value.

ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Fast, cheap: classify intent
async function classifyIntent(message: string): Promise<string> {
  const res = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 20,
    system: "Classify the user message as: question, complaint, or other. Reply with one word.",
    messages: [{ role: "user", content: message }],
  });
  return res.content[0].type === "text" ? res.content[0].text.trim() : "other";
}

// Expensive, capable: generate a detailed answer
async function generateAnswer(question: string, context: string): Promise<string> {
  const res = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages: [{ role: "user", content: `Context: ${context}\n\nQuestion: ${question}` }],
  });
  return res.content[0].type === "text" ? res.content[0].text : "";
}

Routing requests to the cheapest model that produces acceptable quality is one of the highest-leverage optimisations available.

Strategy 2: Control Output Length

Output tokens are costly and slow. Set max_tokens to the minimum value needed for the task, and be explicit in the prompt about the expected response length:

• "Respond in 2-3 sentences."
• "Return only a JSON object with no surrounding prose."
• "List exactly five items."

A model left to its own devices will often produce longer responses than necessary. Explicit constraints reduce both cost and TTFT for the output phase.

Strategy 3: Cache Repeated Prompts

Many applications send the same system prompt with every request. Caching eliminates redundant processing for the static parts of a prompt.

Anthropic supports prompt caching via cache-control markers. Long system prompts, retrieved document chunks, and few-shot examples are ideal candidates. Cached tokens are processed at a fraction of the cost of uncached tokens.

ts
const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: LONG_SYSTEM_PROMPT,
      // Signal to the API to cache up to this point
      cache_control: { type: "ephemeral" },
    },
  ],
  messages: [{ role: "user", content: userMessage }],
});

Ephemeral cache entries persist for a short window (typically a few minutes). For longer-lived static content, structure your prompts so the cacheable prefix is as large as possible. This pairs well with the application-level caching patterns in Caching Strategies: CDN, ISR, Redis.

Strategy 4: Stream Responses

Streaming does not reduce token cost, but it dramatically improves perceived latency. The user sees content appearing within a second rather than waiting for the full generation. For chat UIs and document generation features, streaming is almost always the right choice.

See Streaming LLM Responses in a Next.js App for the complete implementation.

Strategy 5: Cache at the Application Layer

For deterministic or semi-deterministic prompts, cache the complete LLM response in Redis or a similar store. A FAQ chatbot answering the same ten questions repeatedly should not hit the LLM API on every request.

ts
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
const CACHE_TTL = 60 * 60; // 1 hour

async function cachedAnswer(question: string): Promise<string> {
  const cacheKey = `llm:${Buffer.from(question).toString("base64")}`;
  const cached = await redis.get(cacheKey);
  if (cached) return cached;

  const answer = await generateAnswer(question, "");
  await redis.set(cacheKey, answer, { EX: CACHE_TTL });
  return answer;
}

Use a semantic cache for more sophisticated use cases: embed the incoming question, find the nearest cached question via vector search, and return the cached answer if similarity is above a threshold. This handles paraphrased variants of the same question.

Strategy 6: Trim the Context Window

Every token in the input costs money. Audit what you are sending:

• Conversation history: Use a sliding window or summarisation rather than the full history. Covered in Building an AI Chatbot with Memory.
• RAG chunks: Retrieve only the top-3 or top-5 most relevant chunks rather than a large corpus. Fewer, more targeted chunks often produce better answers anyway.
• System prompt: Compress verbose instructions. Every sentence you remove from a system prompt that runs thousands of times per day has a measurable impact on monthly cost.

Strategy 7: Async and Background Processing

Not every LLM call needs to be synchronous. Email summaries, content moderation, SEO descriptions, and similar tasks can run in background jobs, decoupling them from user-facing latency. See Background Jobs and Queues in Node.js for the queue patterns that apply here.

Measuring What You Optimise

Track per-request token usage using the usage field in API responses:

ts
const response = await client.messages.create({ ... });
console.log(
  `Input: ${response.usage.input_tokens}, Output: ${response.usage.output_tokens}, Cache hit: ${response.usage.cache_read_input_tokens ?? 0}`
);

Log these numbers alongside request duration, model used, and feature name. Without this instrumentation you are guessing at where the cost is coming from. Monitoring and Observability for Web Apps covers the logging infrastructure.

You can find examples of these optimisation patterns applied in real features on the /projects page, and discuss LLM performance trade-offs during system design rounds at /interview-prep.

Conclusion

Cost and latency optimisation for LLM apps is not one big intervention — it is a stack of incremental improvements. Start with model selection (use the cheapest model that is good enough), add output length constraints, enable prompt caching for repeated static content, and stream responses to the user. Then add application-level caching for repeated queries and background processing for non-urgent tasks. Each layer compounds, and the cumulative effect on your API bill and response times is significant.