Latency budget (p99 spike)
Each customer tier comes with a p99 latency target — the slowest 1% of their requests must still come back within X milliseconds. It's deliberately about the tail, not the average: a service can look perfectly healthy on average while quietly failing the requests that hurt most. Gold and Platinum tenants watch this number like a hawk.
Cross roughly twice the budget and you've got a latency spike. The customer notices, their p99 dashboard goes red, and high-tier tenants start eyeing the exit. Adding capacity makes it disappear for a single tick and then it's right back — because the fix is the underlying cause (a noisy neighbour, a saturated uplink, oversubscribed CPU), not more headroom papered over the symptom.
Detailed explanation
CustomerLatencySpike fires when observed p99 reaches >= 2x the tier budget; recovery clears at 1.2x, so there's hysteresis to stop it flapping. p99 is the canonical SRE metric on purpose — averages hide tail-latency contention, and the contract is written against the tail. The incident is per-customer, so you see exactly which tenant is over budget rather than a fleet-wide aggregate.
Common drivers: noisy-neighbour IOPS contention on shared SSDs, uplink saturation under DDoS, oversubscribed CPU during the diurnal peak. The open-incidents probe surfaces live spikes in #incidents. Fix the cause, not the symptom — adding a host hides it for one tick, then you're back here. Spread workload (anti-affinity), relieve the contended resource, or relieve the oversubscribed uplink.