Sub-10ms failover: how we keep GPUs running through grid events
When the grid drops, our batteries take over in under 10 milliseconds. Here's the switchgear, the controls, and the load-shedding logic that makes it work without interrupting your training job.
A GPU running a training job is, from a power perspective, a 700W resistive load with very tight tolerance for interruption. Drop power for 50ms and the job crashes. Drop for 200ms and you risk corrupting the checkpoint. Drop for 1 second and you've lost hours of compute.
Our job is to make sure that when the grid fails, the GPUs don't notice.
The detection problem
The first challenge is detecting the grid event fast enough. Grid frequency in the US nominally runs at 60 Hz, but actual frequency varies by ±0.05 Hz normally and ±0.5 Hz during disturbances. We sample the grid signal at 1 kHz — every millisecond — using dedicated PMUs at each site.
Our detection threshold: any sustained deviation outside ±0.1 Hz for more than 50ms triggers the failover sequence. That's tight enough to catch real events but loose enough to not false-trigger on normal frequency noise.
The switchover problem
Once we've detected the event, we need to physically disconnect from the grid and connect the batteries. Mechanical relays can't do this in under 10ms — they're too slow. We use solid-state switchgear: IGBT-based, no moving parts, sub-cycle switching.
Total detection + switchover: under 8ms at every site we've deployed. Well within the 10ms target.
The load shedding problem
When the batteries take over, they're carrying the full compute load. If the load exceeds the batteries' instantaneous discharge capability (which can happen if a GPU job suddenly demands more power), we need to shed load gracefully — slow down non-critical workloads before we lose power entirely.
AURA handles this by monitoring every GPU's instantaneous power draw and pre-emptively throttling jobs that are close to the limit. Job-level throttling, not cluster-level. Your training job slows down by 5% for the duration of the grid event, then resumes normal speed once grid power is restored.
Email engineering@smarttec.dev. We'll send you the actual trace from a recent event — voltage, frequency, GPU clock speeds, all of it.