Scalable software development: the architecture and code practices that matter

Scalable software keeps working as users, data and the team grow, without a rewrite. In practice that comes from a few habits: a data model built for your main queries, stateless application servers, caching for read-heavy data, queues for slow work, and monitoring that shows the bottleneck before users feel it. These basics matter more than the choice of architecture style.

Be clear about what needs to scale

Scaling means different things, and the fixes differ. Work out which of these you face:

• More traffic: more requests per second than the servers can handle.
• More data: queries that were fast with thousands of rows slow down with millions.
• Sharper spikes: steady most of the time, then overwhelmed by a sale, a deadline or a morning rush.
• More engineers: the code runs fine, but it is too tangled for a bigger team to change safely.

Measure before you change anything. A profiler or a slow-query log usually points to a handful of endpoints or queries that cause most of the pain.

Start with the data model

The database is the hardest part to scale later, so it deserves the most care early on.

• Design tables around how the data is read, not only how it is written. Know your most frequent queries.
• Add indexes for the filters and sort orders those queries use, and check query plans for full table scans.
• Paginate every list. Never return an unbounded result set to a client.
• Watch for N+1 queries, where code runs one query per item in a loop. They look fine in testing and fall over with real data.
• Keep transactions short. Long transactions hold locks and block other users.
• Move heavy reporting to a read replica or a separate store, so reports don't slow down the product.

Keep application servers stateless

If any server can handle any request, you scale by adding servers behind a load balancer. That only works when no user state lives in a server's memory or on its local disk. Keep sessions in a shared store, files in object storage and configuration in the environment. Autoscaling then becomes a setting rather than a project.

Choose the scaling signal with care. For a global healthcare provider's telehealth video platform, we scaled on the number of concurrent video rooms rather than on generic CPU load. The rebuilt platform supports 50k+ concurrent sessions with 99.99% service uptime.

Cache data that is read often and changes rarely

A cache stores the result of expensive work so the next request can skip it. Used well, it is one of the cheapest performance wins there is. Used carelessly, it shows users stale or wrong data.

• Cache at the right layer: a CDN for static files and public pages, an in-memory store such as Redis for computed data, and HTTP cache headers for API responses that allow it.
• Give every cached item an expiry time, and decide in advance how stale each type of data is allowed to be.
• Clear the cache on write for data that must be current, such as prices or stock levels.
• Guard against a stampede, where many requests try to rebuild the same expired item at once.
• Measure the hit rate. A cache that rarely hits only adds complexity.

Move slow work to queues

Anything that doesn't have to finish before the user gets a response belongs in a background job: sending email, generating PDFs, calling slow third-party APIs, resizing images. A queue absorbs spikes, because work waits instead of failing. It also lets you scale workers separately from web servers.

Queues come with their own rules:

• Make jobs idempotent, so running one twice does no harm. Many queues deliver each job at least once.
• Retry with backoff, and send jobs that keep failing to a dead-letter queue for a person to inspect.
• Don't update the database and publish an event as two separate steps. If one succeeds and the other fails, your systems disagree.

That last rule caused ledger drift for a core banking client. Services wrote to the database and published events separately, so when the message broker faltered, downstream fraud, analytics and notification systems missed updates. We fixed it with a transactional outbox and change data capture. Each event is written in the same transaction as the change and streamed out from the database log, and consumers ignore duplicates. The result was 100% data consistency.

Make the system observable

You can't scale what you can't see. Google's site reliability engineering book names four golden signals to monitor for any user-facing system: latency, traffic, errors and saturation. Start there, then add:

• Structured logs with a request ID that follows each request across services.
• Tracing, to see where the time goes inside a slow request.
• Service level objectives for the journeys that matter, such as sign-in or checkout, with alerts when you are burning through the error budget rather than on every spike.
• Dashboards the team actually looks at, reviewed after every incident.

Test for scale, not just correctness

• Unit and integration tests keep refactoring safe as the codebase grows.
• Contract tests catch breaking changes between services, or with partner APIs.
• Load tests with realistic data volumes show where the system bends before your users find out. Run them before big launches and seasonal peaks.
• Failure tests matter too. Stop a server, slow a dependency or fill a disk, and check that the system degrades instead of falling over.

Design for failure

At scale, something is always failing somewhere. Set a timeout on every network call. Use circuit breakers so one slow dependency can't tie up every worker. Decide in advance what the product does when a non-essential service is down. A product page that loads without recommendations is better than one that doesn't load at all.

Keep the architecture as simple as the problem allows

You don't need microservices to scale. A well-structured monolith with clear modules, a tuned database, caching and queues can carry a lot of traffic, and it is far easier to run. Split a service out when one part needs to scale or deploy on its own. Our microservices vs monolith decision guide covers when that point arrives.

Where to start

Find your three slowest endpoints and your three most expensive queries, and fix those first. Add the four golden signals to your monitoring if they aren't there yet. Then load test your busiest journey at a few times today's peak. If you want a second pair of eyes, our secure backend systems and site reliability engineering teams can review the results with you.