The Philosophical Developer — Chapter 52: Voxtype, Two GPUs, and the Device-Index Trap

Dictation got fast this week. Not “faster” — fast. A clip that took my desktop twenty-four seconds to transcribe now finishes before the audio has finished playing. The model sits in VRAM, the GPU is bored, and I stopped noticing the transcription pipeline entirely. Which is exactly what dictation should be: a pipe you do not think about, because the moment you think about it it has already broken your flow.

Getting there was a lesson about dual-GPU machines that I will not forget.

The setup

Ryoku ships dictation as a first-class citizen. Voxtype, on whisper.cpp, running the large-v3-turbo model, bound to a key. You hold the hotkey, speak, and it types at your cursor. Out of the box it worked — and ran on the CPU at roughly one times realtime. A fourteen-second sentence meant a fourteen-second wait. Acceptable, but you feel it, and you feel it on every single utterance.

This box has two NVIDIA cards: an RTX 5090 and an RTX 4070 Ti. Two separate jobs. The 5090 is the workhorse — it serves the local LLM, the quadlets, the inference that actually does the thinking. The 4070 Ti drives the desktop. The obvious move: take the transcription load off the CPU and give it to the card that is not busy doing something else. The 4070 Ti.

The trap

Voxtype’s GPU path is Vulkan, not CUDA. So I enabled the Vulkan backend, restarted the daemon, and watched it… keep being slow. Backend says GPU. The model is not on the GPU. Classic.

First mistake: I read a device list from a tool that numbers the GPUs one way and trusted it. Device index 3. Clean, in range, obviously the discrete card I wanted. It was not. Whisper’s kernel lists exactly three Vulkan devices and puts CPU at index three. I had told it to transcribe on the CPU. It did — politely, silently, no faster than before.

Second mistake: device index 1. That loaded the model properly. The GPU lit up. Also the wrong GPU — it was the 5090, the card I was explicitly trying to keep free for the LLM. Because the kernel’s Vulkan enumeration numbers the cards differently from everything else on the system. It lists the 4070 Ti as device 0 and the 5090 as device 1. The opposite of nvidia-smi, which calls the 5090 device 0.

The right device was 0 all along.

The verify

The whole thing hinged on reading the engine’s own report instead of the GUI’s. The daemon’s journal says exactly what it bound:

whisper_backend_init_gpu: device 0: Vulkan0 (type: 1)
whisper_model_load: Vulkan0 total size = 1623.92 MB

And nvidia-smi confirmed it: 2.9 GB resident on the 4070 Ti, the 5090 sitting at its idle baseline, untouched. Two tools, two opposite answers about which card is first. The engine tells you the truth; the summary tool tells you what it wants.

The honest bit

Three restarts. Two wrong cards — one of them literally the CPU. All because two pieces of software number the same two GPUs in opposite orders. That is the whole failure, and it is the kind that feels stupid until you realize it is exactly why you verify with the thing that does the work.

The outcome is clean, though. The transcription model lives on the 4070 Ti, warm and ready, and the 5090 stays free for the inference that actually matters. Dictation is real-time. The pipeline is invisible.

On a dual-GPU box, “the first GPU” is whoever’s enumeration table you happen to be reading. The fix is the same every time: ask the engine.