The Philosophical Developer — Chapter 62: The Model That Pretended to Understand Greek

I ran a full-duplex voice model on a 12 GB card for a week — downloaded thirteen projects, audited every one before running it, and let an agent talk to me through my headphones. The model was charming. It had opinions about Halloween costumes, it never lost the floor, and when I switched to Greek it did the most dangerous thing a voice system can do: it pretended to understand. “Oh wow, that sounds like a wonderful holiday. What’s it all about?” It heard the rhythm of my language and filled the gap with confidence. This chapter is the whole experiment — which duplex options earned their VRAM, which I dropped, and the architecture the numbers forced on me.

The question

The companion platform (chapters 53 onward) needs a voice layer that listens continuously, takes turns like a human, and hands content to a local LLM backend. Push-to-talk dictation (voxtype + whisper) already works; what I wanted was the thing the research said was possible on consumer hardware in 2026: full-duplex models that speak and listen at the same time, plus an event bus so perception can run always-on beside them. The plan: try every viable option on the 4070 Ti alone, one heavyweight at a time, audit before run, measure everything.

What actually runs on 12 GB

PersonaPlex-7B (NVIDIA’s Moshi derivative) has an official 16 GB floor. The community path — a bitsandbytes NF4 quant of the temporal transformer only, keeping embeddings and depformer in bf16 — lands at ~9.8 GB measured, with whisper still resident beside it. Getting there took six defects: the fork’s --quantize path re-quantized an already-quantized checkpoint and crashed; an editable install and a WORKDIR shadow each defeated PYTHONPATH; the server made an unconditional HuggingFace counter-fetch that dies under HF_HUB_OFFLINE=1; two undeclared dependencies; and torch.compile wanting a C compiler the runtime image did not have. All fixed, all in a quadlet, all behind the same control plane as the rest of the lab.

MiniCPM-o 4.5 in Q4_K_M GGUF (llama.cpp-omni fork) ran true full-duplex audio+vision at 9.7 GB, peak 11.7 GB with whisper co-resident — 95% of the card, no room for a third GPU service. It produced 10,660 listen/speak decisions on the shipped fixtures and correct bilingual scene commentary: it watched a video of a floor counter and said “到 20楼了，还有四层。” The “sees the table” datapoint is real. As an always-on service it is too fat; as a batch captioner leg on the bus it is viable.

BayLing-Duplex measured 19.1 GB with no quant path in its modeling code. Dropped without a fight.

The honest negative: the turn controller nobody needed

The B3 candidate was SoulX-Duplug, a 0.6B streaming state-predictor that classifies idle/nonidle/speak every 160 ms and was supposed to replace pipecat’s Smart Turn. I patched its cascade ASR onto CPU (three hardcoded device="cuda" lines) so it could coexist with dictation — 4.0 GB VRAM, it runs fine. Then the A/B on 25 Full-Duplex-Bench pause samples (36 annotated pauses):

SoulX treats every ASR segment as a turn candidate — and a pause is exactly where its bilingual ASR segments. The small dedicated model lost to the boring ONNX classifier already in the stack. B3 keeps Smart Turn. The lesson generalises: a turn controller is only as good as its pause model, and “streaming state” is not the same as “understands this speaker is thinking.”

The benchmark and the by-ear test

Full-Duplex-Bench v1.0, 50 samples, judged locally by strata (the judge’s OpenAI call pointed at my own 5090 — no cloud anywhere in the loop): pause_handling TOR 0.84; user_interruption rating 2.84/5 with TOR 1.0. Read carefully: it never loses the floor and always responds to interruptions, but the responses are related, not focused. The by-ear session agreed — five minutes of holiday smalltalk with a voice called James, unprompted follow-ups, long monologues, and my speech spikes visible in the mic meter while James rarely yielded. A good conversationalist, a mediocre listener.

And the Greek. The model heard me switch languages, guessed “calendar?” from a Greek word, asked about my favourite Greek holiday, then papered over with generic praise and invented content (“it’s not for vegetarians, right?”). A frontend that pretends to understand is worse than one that admits it cannot hear. That single session settled the architecture: PersonaPlex supplies the form — turn-taking, barge-in, prosody, the 0x02 inner-monologue stream as a delegation trigger — and the Hermes/strata backend supplies content and language. The dictation stack already beats the duplex model at Greek (whisper large-v3-turbo configured ["el","en"]); the backend is where comprehension lives.

The bus and the eyes

NATS with JetStream became the spine: Wyoming-style subjects (voice.vad.started, voice.transcript.final, perception.face.expression), three streams, durable replay. Two pitfalls cost an hour: stream subjects need > multi-level wildcards (voice.* matches one level only, so voice.vad.started publishes into the void), and JetStream needs a couple of seconds after container start before it answers. On the bus: MediaPipe face landmarks at 89 fps on CPU (52 blendshapes, expression events), an AST audio-tagging leg (RTF 0.31, “Speech” 0.88), emotion2vec (RTF 0.052), ECAPA+Silero diarization correct on two-voice audio. All CPU, all far under the 2-core target, all emitting typed events that replay later. The always-on perception layer is not a research problem anymore; it is plumbing.

What I would tell another builder

• Quantise per scenario, not per model: NF4 for the duplex transformer where VRAM is the wall, Q4_K_M GGUF where the C++ stack gives you KV control, fp32-plus-CPU-offload where the model is small but the pipeline is not.
• Read the server’s send path before trusting any protocol note. The “raw Opus, not Ogg” claim I inherited described one repo’s client; the server I ran Ogg-muxes both directions. My harness decoded 485-sample chunks into a crash because pw-record does not care about frame sizes and sphn does.
• A single-session asyncio lock that hands you a handshake before taking your connection is a silent-deafness machine. The bridge looked alive; the server was politely queueing forever. Restart clears it; the fix is a client that notices its own dying tasks (one add_done_callback found the real bug after three wrong guesses).
• Declare direct dependencies and let the resolver work. Half my “version hell” was hand-pinned crash-trial combos; the other half was reading the installed library’s actual surface (dir()) and finding a trivial rename.
• Model services stay a human decision. Every heavyweight here runs as a quadlet with Conflicts= and an arbiter that moves the daily driver off the card — starting an experiment never surprises the machine.

The honest bit: none of the duplex models is good enough to be the brain. PersonaPlex at 2.84/5 on interruptions and MiniCPM-o at 95% card occupancy are form, not content. The value of this week is knowing exactly which form is worth its VRAM — and that the dangerous failure mode of a small full-duplex model is not silence but confidence.

Sources

1. https://github.com/NVIDIA/personaplex — NVIDIA’s full-duplex voice model (Moshi derivative), code MIT, weights NVIDIA Open Model License.
2. https://huggingface.co/brianmatzelle/personaplex-7b-v1-bnb-4bit — the community NF4 quant that made 12 GB possible, and its own patched moshi package.
3. https://github.com/Soul-AILab/SoulX-Duplug — the 0.6B streaming turn controller that lost the A/B.
4. https://github.com/OpenBMB/MiniCPM-o — omni model; https://github.com/tc-mb/llama.cpp-omni — the C++ full-duplex path.
5. https://github.com/DanielLin94144/Full-Duplex-Bench — the benchmark harness the papers use; v1.0/v1.5 run fully local here.
6. https://github.com/nats-io/nats-server — the event bus; JetStream for durable replay.
7. https://github.com/pipecat-ai/pipecat — Smart Turn V3, the boring classifier that won.