Level 05 · Cruise
Real traffic, a bill that quadrupled, and a 3am you would rather not repeat.
Scaling problems feel like infrastructure problems and almost never are. At this level the constraint is usually one query, one loop, or one third party, and the reason it is hard to find is that nobody is measuring. The instruments come first; the fixes are easy once you can see.
You are here if
- It is slow, and you are guessing at which part.
- The bill went up faster than the traffic did.
- You found out about the outage from a user.
- You have added caching and it did not help.
Measure first, and mean it
The instinct is to optimise what you suspect. The suspicion is wrong often enough that acting on it costs more than measuring. Time the endpoints, find the slowest one under real conditions, and look at what it does. In web applications the answer is a database query far more often than it is the language, the framework or the host.
The specific thing to look for is a query in a loop: one query for the list, then one per item, thirty times. It is the single most common cause of a page that is fine with ten rows and unusable with a thousand, and it is the easiest thing in the world for a model to write, because each individual line looks correct.
The four fixes that cover most of it
Index what you filter and sort by. An unindexed lookup on a growing table degrades quietly and then suddenly. Fetch related data in one query instead of many. Paginate everything that can grow, including the endpoints nobody looks at, because those are the ones that fall over first.
And move slow work out of the request. Anything that talks to a third party, sends mail, or processes a file belongs in a background job, so a slow provider becomes a delayed job rather than a timed-out page. Pick a queue you can actually inspect: a job system whose failures are invisible is worse than none.
Cost has a shape
Bills rise for structural reasons, and they are worth naming: paying for idle capacity you do not need, paying per request for work you repeat, paying for data leaving the platform, and paying a model provider per token for calls you make more often than you think.
Two habits keep it honest. Put a spend alert on every provider that can bill you by usage, before you need it. And when a bill jumps, find the line item before changing anything, because the intuitive culprit and the actual one are usually different.
Instruments that get read
Three things, and no more at this level. Logs with a request identifier, so one user’s report can be traced through the system. Timing on the handful of operations you care about. And one alert per failure you would actually get out of bed for.
The failure mode is alerting on everything, which trains you to ignore all of it. Alert on symptoms users feel (the site is down, errors are up, the queue is not draining), not on every metric that can move.
Run it yourself
The checklist we would run.
There is nothing held back in it. If you work through this list and nothing on it is open, you have left this level, and you did not need us to do it.
- 01
The slowest endpoint is measured, not guessed
Under real data volume. Ten rows tells you nothing.
- 02
No query inside a loop
Search your data access for it specifically. It is the most common one by a distance.
- 03
Indexes on what you filter and sort by
Then confirm the query actually uses them, rather than assuming.
- 04
Every list is paginated
Including the internal ones.
- 05
Slow work happens in the background
Third parties, mail, file processing. Nothing that waits on someone else’s server holds a request open.
- 06
A spend alert on every usage-billed provider
Set before you need it, at a number that would surprise you.
- 07
Logs carry a request id
So one complaint can be followed end to end.
- 08
One alert you would get out of bed for
If everything alerts, nothing does.
What done looks like
- You can name the slowest thing and say why.
- You hear about problems from a graph before a customer.
- You can explain each line of the bill.
Where it usually goes wrong
Caching before measuring
It hides the problem, then adds a stale-data bug on top of the one you did not fix.
Scaling the machine instead of fixing the query
It buys a month and multiplies the bill. The query is still there.
A queue you cannot inspect
If you cannot see failed jobs and retry them, work will disappear silently, which is worse than it failing loudly.
If you want company
Where it actually bends: the queries, the N+1, the queue, the cache, the cold start, the runaway spend. Then the boring instruments (logs, traces, alerts that mean something) so the next surprise arrives as a graph and not a message from a customer.
Most engagements at this level are short: find the bend, fix it, leave the instruments behind so the next one is visible.
It starts with a conversation and innovate@isyncso.com, subject Hangar. Tell us the level you landed on and what is in front of you. If the answer is on this page, we will say so.