UptimeUptime Wiki

Thermal runaway (cooling cascade)

Every server you run throws off heat, and your cooling has a ceiling just like your power does. If the heat your fleet produces outruns the cooling you've installed, the room temperature climbs and servers start to overheat and fail one by one. Cooling is the quiet constraint — it doesn't trip in an instant like a breaker, it creeps up on you.

Here's the vicious part: when a few servers fail from heat, the survivors pick up their load and run even hotter, which makes THEM fail too. That's the runaway — each failure makes the next one more likely. A bad afternoon turns into a fleet-wide cascade unless your cooling had headroom to begin with. Size cooling for the fleet you're growing into, not the one you have today.

Detailed explanation

Facility-wide cooling cap

Cooling is a facility-wide ceiling (5kW Garage scaling to ~380kW Hyperscale, roughly 70% of the site's max rack-power). Sustained heat output over that budget for THERMAL_GRACE_MINUTES (30) flips the facility into a CoolingTick every 5 sim-minutes that fails 1-3 hosts per hour. Unlike a breaker trip, there's a grace window — the system gives you time to react before the cascade starts.

Cascade dynamics and cap

Once running, failures compound: heat from surviving hosts absorbing the dropped load pushes the ambient further over budget, accelerating the next CoolingTick failure. The cascade is bounded at 50% of the fleet (THERMAL_FLEET_FAIL_CAP) so it can't take everything, but losing half your hosts is still catastrophic. The open-incidents probe surfaces live thermal failures; the fix is cooling headroom bought ahead of the heat, not mid-cascade.

On this page