← All posts

How to Design Scalable Backend Systems Well

A backend rarely fails because a team forgot how to add servers. It fails because an early shortcut became a permanent dependency: one database query now serves every screen, a slow third-party API sits on the request path, or background work is still running inside a web request. To design scalable backend systems, start by making the system understandable under normal load, then make its behavior predictable when load, data volume, or failures increase.

Scale is not only a traffic problem. A product can have modest daily usage and still struggle with large files, bursty imports, expensive AI requests, a growing event history, or a few customers whose usage patterns differ sharply from the average. The right architecture depends on those realities. Measure what the product needs, choose the smallest design that meets it, and leave clear room to change it.

Start With the Work the System Must Do

Before choosing databases, queues, or cloud services, write down the operations that matter. A product API may need to create accounts, accept uploads, return a dashboard, process documents, send notifications, or synchronize data with another service. These are not all the same kind of workload, and treating them as one produces unnecessary complexity.

For each operation, define four things:

  • Expected request volume
  • Acceptable response time
  • Data size
  • Failure cost

A dashboard that can load in a few seconds has different requirements from a payment confirmation or a permissions change. An AI-generated report may reasonably take longer if the interface clearly shows progress. A request that changes financial, security, or customer data needs stronger guarantees than a read-only analytics query.

This exercise exposes useful trade-offs early. Strong consistency can be worth the cost for a record of ownership, while eventual consistency is often fine for a search index or activity feed. Synchronous processing is easier to reason about, but it is a poor fit for work that can take minutes or depends on a provider outside your control.

Design Scalable Backend Systems Around Clear Boundaries

The most durable systems have boundaries that make change local. That does not mean beginning with dozens of independent services. For a new product, a well-structured modular application is often faster to ship, cheaper to operate, and easier to debug than a distributed system.

Keep domain logic separate from HTTP handlers, database access, and vendor-specific code. A route should validate input, call an application service, and return a response. It should not contain the only copy of business rules, build complex SQL inline, and call three external providers before it finishes. Those patterns are quick at first and expensive later.

A useful boundary also exists between the request path and background work. If a customer uploads a file, the API can validate it, store a reference, create a job, and respond. A worker can then extract content, generate previews, call an AI model, or notify the customer when processing is complete. The user gets an honest status instead of a browser tab waiting on a fragile chain of work.

Use Queues for Control

A queue absorbs bursts and lets workers process tasks at a rate the system can safely sustain. It also introduces new responsibilities. Jobs can run more than once, arrive late, fail repeatedly, or complete after a user has changed their mind.

Make job handlers idempotent: running the same job twice should not charge twice, send duplicate messages, or corrupt a record. Store a durable job state, use stable idempotency keys where appropriate, and define retry rules based on the failure type. Retrying a temporary network failure makes sense. Retrying invalid input twenty times does not.

Use a dead-letter path or an equivalent failure state for work that cannot complete automatically. Someone needs a way to inspect the failure, correct the cause, and safely retry or cancel it. Without that visibility, a queue only postpones problems.

Don't Lose Events Between the Database and the Queue

Workers and events create a quiet failure mode known as the dual-write problem. The request handler saves an order to the database, then publishes an `order.created` message to the queue. If the process crashes between those two steps, or the broker is briefly unavailable, the database says the order exists but nothing downstream ever hears about it. If you reverse the order and publish first, you can announce an order that was never saved.

The transactional outbox pattern fixes this. Nothing is published directly from the request. Instead:

  1. In the same database transaction that changes business data, insert a row into an `outbox` table with the event type, payload, and a unique ID.
  2. A separate relay process reads unpublished outbox rows, sends them to the queue, and marks them as published.
  3. Consumers deduplicate by event ID, because the relay may deliver the same event more than once after a crash.

Either the business change and its event are both committed, or neither is. Delivery becomes at-least-once rather than "usually once," which is why idempotent handlers matter so much.

A few practical notes:

  • Polling the outbox table every second or two is enough for most products. Change data capture tools (for example, Debezium reading the database log) are worth it only when latency or volume require them.
  • Index the outbox on the published flag and creation time, and delete or archive published rows on a schedule so the table doesn't grow forever.
  • Monitor the age of the oldest unpublished row. It is the most direct signal that event delivery has stalled.

Treat the Database as a Product Constraint

Most backend performance problems eventually reach the database. The answer is rarely to add a cache before understanding the query. Start with indexes that match actual access patterns, inspect slow queries, and avoid loading more rows or columns than a screen needs.

Data modeling matters as much as query tuning. Store the source of truth in a form that preserves integrity and supports the transactions the product requires. Add derived tables, materialized views, denormalized records, or search indexes when measurements show that reads need them. These can make a system much faster, but they also create synchronization work and more failure modes.

Pagination deserves attention early. Offset pagination is simple but becomes less reliable and more expensive with large, changing datasets. Cursor-based pagination is usually a better choice for timelines, logs, and ordered records, provided the ordering is stable and the cursor is based on indexed fields.

Plan for data lifecycle, not just data creation. Decide what gets retained, archived, deleted, or aggregated. Privacy-conscious products should collect only what they need, limit access to sensitive fields, and make deletion a real system behavior rather than a support promise. Backups matter, but recovery must be tested regularly — restore them, don't just create them.

Add Caching When You Can Explain Its Invalidation

A cache is useful once you know which reads are expensive, how often they repeat, and how stale the data is allowed to be. Without those answers, a cache mostly hides slow queries until the moment it misses.

Start with the least risky layers:

  • HTTP caching and CDNs for public or rarely changing responses, using correct `Cache-Control` headers and ETags.
  • Cache-aside for hot reads. The application checks the cache, falls back to the database on a miss, and stores the result with a TTL.
  • Precomputed results for expensive aggregates such as dashboard totals, refreshed by a job instead of on every request.

For each cached value, decide in advance:

  • Staleness budget. Can this be 30 seconds old? Five minutes? Never?
  • Invalidation trigger. Does a TTL alone suffice, or must a write explicitly clear the key?
  • Key design. Include the tenant or user ID in keys for per-customer data. A shared key leaking one customer's data to another is a security incident, not a performance bug.
  • Stampede protection. When a popular key expires, hundreds of requests can hit the database at once. Use request coalescing, a short lock, or refresh keys slightly before they expire.
  • Behavior when the cache is down. The system should get slower, not fail. Test that path.

Never make the cache the only copy of data you can't regenerate.

Make External Dependencies Optional to the Request Path

Your uptime is partly determined by services you do not operate. Payment providers, email platforms, mapping APIs, AI models, analytics tools, and identity services can all slow down or fail. A scalable design assumes this will happen.

  • Set timeouts deliberately.
  • Use bounded retries with backoff.
  • Apply circuit breakers or temporary feature degradation when an upstream dependency is failing.
  • Decide what the customer should see. If a nonessential enrichment service is unavailable, the core record may still be created. If a required verification fails, tell the user clearly that the action could not be completed and preserve enough context to retry safely.

Do not make every internal component wait on every other component. Event-driven patterns can help when a completed action should trigger independent follow-up work, such as audit logging, notification delivery, or analytics processing. They are less helpful when used to hide a simple request-response relationship behind an elaborate event graph.

Treat Capacity as a Measurement Practice

Autoscaling can add instances. It cannot fix a database lock, an unbounded query, a memory leak, or a worker pool overwhelmed by expensive jobs. Capacity planning begins with observability.

Track these signals and correlate them with deploys and product events:

  • Request rate and error rate
  • Latency percentiles (p50, p95, p99)
  • Queue depth and job duration
  • Database connection usage and slow queries
  • Resource saturation (CPU, memory, disk, I/O)

Averages hide the experience of the slowest customers, so look at percentiles and outliers.

Load testing is useful when it resembles real behavior. Test a burst of uploads, a surge of dashboard reads after an email campaign, or many concurrent jobs competing for the same limited provider quota. Test degraded states too: a slow database replica, unavailable cache, delayed queue, or failed dependency. The goal is not to survive an unrealistic number, but to find where the system stops behaving predictably.

Set explicit limits on:

  • Upload and request body sizes
  • Concurrent jobs
  • Expensive report ranges
  • Per-customer access to costly operations

Rate limits are not only a security control. They protect shared capacity and prevent one faulty integration from turning into a full outage.

Keep Deployments Boring and Reversible

A backend that scales technically but cannot be changed safely will slow the product down. Build a release process that supports small, reversible changes. Use versioned API contracts where clients cannot update immediately.

Deploy database changes in stages:

  1. Add a compatible column or table.
  2. Ship code that works with both shapes.
  3. Backfill carefully.
  4. Remove old paths later.

Feature flags can reduce rollout risk, but they need ownership. Old flags create confusing branches and make incidents harder to diagnose. Record why a flag exists, who owns it, and when it should be removed.

Security and privacy belong in this operating model. Use least-privilege access, protect secrets, separate environments, log meaningful security events, and avoid placing personal data in logs by default. These choices reduce incident scope while making systems easier to reason about.

Scalability Checklist

Use this as a starting review for an existing backend or a design document:

  • Every critical operation has a defined volume, latency target, data size, and failure cost.
  • Long-running or provider-dependent work runs in background jobs, not in the request path.
  • Job handlers are idempotent and have retry rules based on error type plus a dead-letter state.
  • Database changes and the events they produce are committed together (transactional outbox or equivalent).
  • Slow queries are monitored, and indexes match real access patterns.
  • Large lists use cursor-based pagination on stable, indexed fields.
  • Every cache has a staleness budget, invalidation rule, and tenant-safe keys.
  • Every external call has a timeout, bounded retries, and a defined degraded behavior.
  • Explicit limits exist for upload size, request size, concurrency, and per-customer expensive operations.
  • Dashboards show latency percentiles, error rate, queue depth, and database connection usage.
  • Schema changes deploy in backward-compatible stages.
  • Backups are restored in regular tests, not just created.

Where to Start

The best next step is usually not a rewrite. Pick the most expensive request, the most common failure, or the data path that worries you most. Instrument it, define the limit it needs to meet, and improve that boundary first.

This is how uAgency approaches architecture work for client products: clear operational boundaries, practical privacy defaults, and a system the product team can still understand after launch.

Want a second pair of eyes on your backend? Book an architecture review with uAgency — we start with your most expensive request path and the failures that cost you the most.

← All posts Our apps