Self-hosted GPU inference platform — 36B MoE, voice AI and a 70-GPU farm

Budgeted the entire stack against a single 25.7 GB VRAM ceiling: the 36B MoE quantized to IQ4_XS with a q4 KV cache at 64K context, whisper-small ASR and the embedding model pinned to the same card, TTS alongside — measured at 25.3/25.7 GB with every layer accounted for. When the fleet started crashing, treated the log as a forensic trail: kernel oopses pointed to GPU reset storms, which led to the discovery that a modded VBIOS was running the card at a 3136 MHz state the silicon could not hold. Capped the clock at 2500 MHz, migrated to ROCm 7.14, added ZFS snapshots, and rebuilt the stack so a single process serves LLM, embeddings, ASR and TTS concurrently. The farm grew to 10 nodes and 70 GPUs at ~10 kW with a +35 °C thermal delta.

One 24 GB card now serves a 36B MoE model, embeddings, ASR and TTS simultaneously — at up to 2,426 tok/s prefill and 87–125 tok/s decode across a 10-language voice loop. The 70-GPU farm runs stable 24/7 since the firmware fix: the answer was a clock state in the VBIOS, not the software stack.