Scaling services: when more instances do not help
Why a service can be slow with idle CPU, how extra instances overload the database and what to check before autoscaling.
Peak traffic and expensive requests
Checkout slows during a promotion, yet application CPU stays low. Adding servers looks reasonable until every new process opens another database connection pool. The database becomes busier and checkout gets slower. Before scaling the fleet, find where a request spends time and what limits useful completions.
We split quick reads, writes, expensive computations and background jobs. Each class gets its own latency budget and concurrency limit. One aggregate average lets cheap reads hide slow reports, and the load test can look much healthier than the workload we actually need to run.
In an illustrative stable workload, 200 requests per second and 0.2 seconds of average time in the system imply roughly 40 concurrent requests. If average time grows to two seconds at the same rate, concurrency rises to about 400. This relates averages in a stable system; it does not calculate p99 or a required connection-pool size.
Where requests wait
A request can spend 30 ms computing and 900 ms waiting for a database connection. Application CPU stays available. Extra instances create more contenders without speeding up the database. Measure pool waits, queues, locks and external calls alongside total latency.
Increase load gradually and find where throughput stops growing while unfinished work accumulates. An unlimited queue postpones rejection but adds waiting, memory consumption and work for clients who have already left.
We set the operating limit below the point where response time starts falling apart. Headroom covers ordinary bursts and losing part of our capacity. A rule like "keep CPU below 70%" cannot tell us that boundary: it depends on the workload and where requests wait.
The connection budget
A pool allowing 20 connections per process can produce 80 connections across four processes and 400 across twenty. If the database supports fewer, application autoscaling hits a shared limit. Budget connections across applications, background jobs and administration.
Check local state too. A file on one instance does not appear on another; an in-memory session can disappear during routing; a scheduled task can run once per process. Each requires an explicit ownership or storage decision, not merely a load balancer.
We do not automatically enlarge the pool with request volume. More connections can mean more competing database work and worse response time. We find a range that completes useful work without excessive pressure. Different operation classes may need separate budgets to protect the interactive path.
What to do at capacity
Limit concurrency, queue length and waiting time. Rejecting some work promptly can preserve useful throughput instead of letting all requests expire. Priorities can protect checkout while a heavy export temporarily waits or fails.
Retries without a shared budget can amplify an outage. Three call layers making up to three attempts each can turn one request into 27 calls to the final dependency. We choose which layer owns the retry, cap attempts and use backoff with jitter so clients do not retry together. Timing out only tells us that we did not receive an answer.
For repeated writes, use an operation identifier and durable duplicate handling. An in-memory check disappears when the process fails. Recording the result and returning it again must remain consistent with the actual data change.
Amazon’s load-shedding article distinguishes offered traffic from goodput: useful requests completed successfully. For checkout, the test is whether accepted orders keep completing under overload, not whether every request enters a queue.
Time needed to scale up
An instance needs startup, warmup and traffic admission. A shorter burst can overwhelm the service before reactive capacity arrives. Predictable events may need capacity in advance; unpredictable ones need limits and defined overload behavior.
CPU is not an adequate signal for every workload. Queue workers need arrival rate, completion rate and oldest-job age. A thousand quick jobs and a thousand minute-long jobs require different responses. Stop adding workers when the downstream dependency becomes the limit.
We measure the entire reaction delay: metric collection, controller decision, process startup, warmup and readiness for useful traffic. A quick container start does not establish quick usable capacity. A service with expensive warmup may need a permanent reserve despite having a reactive controller.
Test the limit under load
Use representative data sizes, operation distribution and client retry behavior. Remove an instance or slow a dependency during the test. Capacity that exists only when every resource is available does not provide headroom for that failure.
After the change, we compare successful throughput, tail latency, cost and recovery. We record the test conditions and the limit we found. The next step might be a query fix or a pool setting rather than another instance. Useful work completed by the service is the outcome we care about.
The generator must preserve the intended arrival pattern as responses slow. If each virtual client waits for completion before sending again, longer latency lowers offered traffic. A beautiful report may then exist because the test stopped producing the real peak. We choose the generation model deliberately.
- Replay a representative mix of checkout, reads and reports. Include large accounts and cold starts; record arrivals as well as successful completions.
- Measure execution, pool wait and dependency time separately. First limit expensive reports while keeping instance count and incoming traffic unchanged.
- If checkout improves, give interactive and background work separate budgets. Cap total database connections across the fleet before enabling more application instances.
- Exercise a slow dependency and remove one instance. Check that rejection remains bounded, retries do not grow without limit and accepted work still finishes.
- Compare throughput, tail latency, recovery time and cost against the baseline. Keep the change only if it improves useful work under the recorded conditions.
Keep reading
How to define MVP scope without building too much
How to build a first release that tests the idea: scope, manual work and essential quality in a job-management service.
From a business problem to an implementation
From a request for a customer portal to workable scope, a real integration and a measurable business result.