Docs — fusion-runtime

Overview

Most teams build voice agents by renting every stage from an API — Deepgram for STT, OpenAI/Groq for the LLM, ElevenLabs/Cartesia for TTS. That works, but it means per-minute bills, a network hop (and a third party's queue) on every turn, and audio leaving your infrastructure. fusion-runtime runs all three stages in one process with open-weight models, so none of that is required in the default path.

Where this fits. Narrow-domain agents (reservations, order status, IVR replacement), privacy/data-residency-sensitive teams, and local-first developers. Open weights are behind the best closed APIs today — this trades some ceiling for control, cost and privacy. See the trade-offs on the landing page.

Quickstart

Requires Python 3.11+.

git clone https://github.com/<org>/fusion-runtime.git
cd fusion-runtime
pip install -e ".[talk]"

frun models pull          # ~0.9 GB: Whisper tiny, Qwen2.5 0.5B, Kokoro, Silero VAD
frun doctor                # checks libraries, GPU, models and audio, and says how to fix problems

Start the server:

frun up

Then open http://localhost:8000 and click Talk. That page is served by the runtime itself — no build step, nothing else to install. Talk over the reply to interrupt it.

Or talk from a second terminal instead:

frun talk

On macOS, allow microphone access for your browser (or, for frun talk, your terminal app) under System Settings → Privacy & Security → Microphone. With headphones, frun talk --no-aec turns echo cancellation off.

Browser client

The runtime serves the browser client it uses itself, so putting a voice agent on your own page takes two script tags and no build step:

<script src="https://your-server/fusion-runtime.js"></script>
<button id="talk"></button>
<script>FusionRuntime.attach({ button: "#talk" });</script>

With no url, it connects back to the server the script came from. Everything else is optional:

const session = FusionRuntime.attach({ button: "#talk", token: sessionToken });

session.on("transcript", msg => { if (msg.is_final) show("You: " + msg.text); });
session.on("response",   msg => { if (msg.is_final) show(msg.text); });
session.on("trace",      msg => console.log(msg.summary));   // TTFA, tokens/s, per-stage times
session.on("error",      e   => show(e.message));

FusionRuntime.connect(options) returns the same session without binding a button, for a page with its own controls (session.start(), session.stop(), session.interrupt()).

The client uses the browser's own echo canceller, resamples the microphone in an AudioWorklet so a busy page can't stutter the audio, and schedules replies slightly ahead of real time so network jitter doesn't leave gaps. Interruptions are decided on the server, which hears clean audio: when it says the caller interrupted, queued audio is dropped immediately — including the tail of a sentence already synthesized.

Two things before you deploy it. Browsers only hand over a microphone on https:// (or localhost), so the page and the WebSocket both need TLS — https:// and wss://. And a page never holds an API key: your backend mints it a short-lived token, which is what token above is for — see Authentication.

Authentication

Generate a key and put it where the server runs:

frun key new
FUSION_ACCEPTED_KEYS=web:frun_kR7m...        # .env, or the environment

Without keys the server answers on localhost only, and frun up --host 0.0.0.0 refuses to start. There is no flag to switch that off: a runtime reachable from elsewhere with no authentication is someone else's GPU, on your bill.

Two kinds of client, because only one of them can keep a secret:

Client	Presents
frun talk, your backend, curl	The key, as `Authorization: Bearer <key>`
A browser page	A session token its backend minted — never a key

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"
# {"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}

A token works once and expires in about a minute, so a leaked URL, screenshot or log line is worthless by the time anyone reads it. frun token prints a console URL with one in it for your own machine.

Several keys

One variable, comma separated — not one variable per key, which would simply overwrite itself. The label in front is optional and grants nothing; it exists so frun keys list and your logs can say which key is busy or failing without the key itself ever appearing.

FUSION_ACCEPTED_KEYS=web:frun_kR7m...,mobile:frun_9xQ2...,partner:frun_Lm4v...

With more than one key configured, the per-key session cap changes on its own: instead of "the whole server", each key gets all but one slot, so no single key can lock the others out. Set FUSION_MAX_SESSIONS_PER_KEY for a different split.

Past a handful, put them in a file — one per line, # comments allowed — which also makes them reloadable:

FUSION_ACCEPTED_KEYS_FILE=/etc/fusion/keys

The two variables are not a pair. FUSION_ACCEPTED_KEYS is a list a server accepts. FUSION_API_KEY is the single key a client presents to a server somewhere else — frun talk, or your backend. On one development machine both exist and hold the same string; in a deployment they live on different machines. Starting a server with only the client one set names the mistake rather than running unprotected.

Rotating and revoking

The key list takes several, so rotation is: add the new key, move clients across, drop the old one. Point FUSION_ACCEPTED_KEYS_FILE at a file and kill -HUP re-reads it without a restart — which matters on a GPU, where restarting reloads the models. Removing a key is complete: its tokens are dropped and its conversations are closed.

Limits

Authentication says who may use the server; these say how much of it one caller may take.

A ceiling, not a capacity claim. Conversations are fully isolated, but the in-process model decodes one reply at a time, so callers beyond the first queue. Point the LLM at vLLM or llama-server (see Configuration) and they run in parallel; speech-to-text is the next bottleneck either way.

FUSION_MAX_SESSIONS	Conversations at once (4)
FUSION_MAX_SESSIONS_PER_KEY	Per key. One key gets the server; when keys are shared, all but one slot
FUSION_MAX_MESSAGE_BYTES	One WebSocket message (1 MB)
FUSION_MAX_TURN_AUDIO_S	Speech without a pause (60)
FUSION_MAX_SESSION_S	One conversation (900)
FUSION_IDLE_TIMEOUT_S	A socket that went quiet (60)
FUSION_CONNECTIONS_PER_MINUTE	New sockets, and failed keys, per address (30)
FUSION_TOKENS_PER_MINUTE	Tokens one key may mint (600)

Origins and proxies

FUSION_ALLOWED_ORIGINS lists the websites whose pages may open a socket. Unset means only pages this server itself serves: browsers do not stop one site connecting to another, so the server checks. Clients that aren't browsers send no Origin and are unaffected.

Behind a proxy that terminates TLS — RunPod's, nginx, Cloudflare — name it with FUSION_TRUSTED_PROXY=10.0.0.0/8. Until you do, its X-Forwarded-* headers are ignored, because anyone who can reach the port can write those headers. The forwarded address is used for counting only, never to decide who may in.

The frun CLI

Command	What it does
frun up	Starts the server on 127.0.0.1:8000. Checks models are installed and the port is free first.
frun up --config production --host 0.0.0.0 --port 8080	Production models, reachable from other machines.
frun up --log-format json	One JSON log line per event, for deployments and log collectors.
frun talk	Talks to the server with your mic and speakers; a one-line latency summary per turn.
frun talk --verbose	Also prints each turn's full timeline.
frun talk --url ws://host:8080/v1/voice/ws	Talks to a server elsewhere.
frun talk --key <key>	...to a server with authentication on. Also: FUSION_API_KEY.
frun key new [name]	Generates a key. Shown once — nothing stores it.
frun keys list	The configured keys: names and fingerprints, never the keys.
frun token	Mints a session token and prints a console URL to open.
frun models list	Shows every model, its size, whether it's installed, and which profile uses it.
frun models pull	Downloads what the development profile needs.
frun models pull --config production	Downloads what the production profile needs.
frun models pull --llm	Only one stage; also --whisper, --kokoro, --vad.
frun models pull qwen2.5-7b-q4	A specific model by ID.
frun doctor	Checks Python, libraries (incl. torch/torchaudio matching and Silero VAD actually loading), GPU support, models, port and audio. Exits 1 if something is broken.
frun version	Installed version.

fusion-runtime works as an alias for frun. If the command isn't on your PATH, use python3 -m fusion_runtime.cli.

Models

Every download is pinned to an exact Hugging Face commit and checked by file size.

ID	Stage	Size	License	Used by
whisper-tiny.en	Speech-to-text	78 MB	MIT	development, production
qwen2.5-0.5b-q4	LLM (GGUF)	491 MB	Apache-2.0	development
qwen2.5-7b-q4	LLM (GGUF)	4.7 GB	Apache-2.0	production
kokoro-v1.0	Text-to-speech (ONNX)	328 MB	Apache-2.0	development, production
silero-vad	Voice activity detection	2 MB	MIT	development, production

Where models are stored, first match wins: FUSION_MODEL_DIR if set, a models/ folder next to the source (source checkouts), or ~/.cache/fusion-runtime/models. Silero VAD is the exception — it's cached by PyTorch in ~/.cache/torch/hub.

Using your own models

Those five are the defaults, not the limit. A model reference can be a catalog ID, a path on disk, a Hugging Face repo as hf:owner/repo, or the URL of an OpenAI-compatible server. Nothing is detected from a model's name — the runtime is chosen from the file format, and the model's own metadata does the rest.

Stage	What works	What doesn't, yet
LLM	Any GGUF. The architecture, context length and chat template are read out of the file, so Llama, Mistral, Gemma, Phi and Qwen all load with no new code. Also any OpenAI-compatible endpoint — vLLM, llama-server, Ollama, a hosted API.	A GGUF with no chat template stored in it: pass one with the `chat_template` option. Tool calling isn't wired up.
Speech-to-text	Any Whisper converted to CTranslate2 — every size from tiny to large-v3, distil-whisper, and fine-tunes for other languages. Which languages a model speaks is read from its own vocabulary.	Non-Whisper speech models. They need a new runtime, not a setting.
Text-to-speech	Kokoro, with any of its voices.	Other ONNX voice models. Each family needs a small class describing how its text becomes tensors.

frun models pull hf:Systran/faster-whisper-small

Then name it in the agent file. Download first — resolving a model never touches the network:

agent = Agent(
    stt=STT("hf:Systran/faster-whisper-small"),
    llm=LLM("hf:bartowski/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf"),
    tts=TTS("kokoro-v1.0", voice="af_heart"),
)

A repo holding several quantizations needs the file named, as above. Safetensors models aren't run in process — serve one with vLLM and give the agent its URL.

How it works

microphone audio │ ▼ Silero VAD ──► faster-whisper ──► turn detection ──► llama.cpp ──► Kokoro ──► audio out (speech only) (rolling window) (punctuation + (streams (speaks each silence) tokens) sentence) │ └──► barge-in watcher: if you talk over the reply, generation and playback stop

Everything runs in one Python process today. The local client removes the bot's own voice from the microphone (echo cancellation), so the server hears clean audio and can decide when you're actually interrupting it versus hearing its own playback.

Configuration

Choose a profile with frun up --config (or FUSION_CONFIG if you start the server another way):

Profile	STT	LLM	TTS	For
development (default)	Whisper tiny, CPU int8	Qwen2.5 0.5B, CPU	Kokoro	Laptops, 8 GB RAM
production	Whisper tiny, CUDA	Qwen2.5 7B, all GPU layers	Kokoro	NVIDIA GPU
hybrid	Whisper tiny	gpt-4o-mini over HTTP	Kokoro	Speech local, LLM elsewhere

frun up --config production

LLM from any OpenAI-compatible endpoint

Speech stays local; the LLM comes from vLLM, llama-server, Ollama or a hosted API:

frun up --llm-url http://localhost:8080/v1 --llm-model my-model

export GROQ_API_KEY=...
frun up --llm-url https://api.groq.com/openai/v1 --llm-model <model> --llm-api-key-env GROQ_API_KEY

API keys are never accepted in config — only the name of the environment variable holding one. frun doctor checks that the key is set, the endpoint answers, and it serves the model.

Turn detection: when the agent answers

By default the agent answers after 500 ms of silence, and stops speaking once the caller has talked over it for 300 ms:

frun up --turn-wait-ms 800          # more patient with pauses mid-sentence
frun up --interrupt-after-ms 500    # ignore coughs and "mm-hm" in noisy places

Speaking again right after a pause continues the same turn: if the caller resumes within 1.5s of their turn ending, both parts reach the LLM as one message. A turn-detector model can plug in to shorten or stretch the wait based on how likely the caller is done — silence always confirms the end of a turn, and a detector that's slow or fails just leaves the default wait.

frun up --turn-detector my_package.turns:MyDetector

Languages

Set what callers speak with STTConfig(language="hi") (omit it to detect per turn; English-only Whisper models like tiny.en refuse other languages at startup). Kokoro speaks its voice's language, or set TTSConfig(language=...). Reply text is split into sentences for speech correctly across scripts. The catalog currently ships English models only.

Server API

Seven endpoints, and it's clearest by who calls them.

A browser

GET /	The console the runtime serves. Open it and talk
GET /fusion-runtime.js	The client your own page embeds. Sent to every origin; it holds no secrets
WS /v1/voice/ws	Where the conversation happens

The socket takes raw 16 kHz mono PCM as binary messages, and two JSON control messages: {"type":"playback","playing":true|false} while the agent is audible, and {"type":"interrupt"} from a client that stops itself. It sends 24 kHz PCM back as binary, and these as JSON:

config	First message: both sample rates, the session id, and the next session token
transcript	What the caller said. `is_final` marks the end of a turn
response	One per token; `is_final` carries the whole reply
interrupted	The caller talked over the agent — drop queued audio
turn_resumed	They kept talking after a pause, so both halves are answered as one turn
echo_discarded	The "caller" was the agent's own voice coming back
turn.trace	Every turn's timings: TTFA, per-stage, tokens/sec
error	code, message, stage, retryable, fix — never a stack trace

Your backend

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"

{"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}

One call per visitor, on page load. The token works once and expires in about a minute, and a connected page is handed its next one over the socket — so reconnecting costs no round trip. This is the only endpoint a page's backend needs.

A script, or a test

curl -X POST https://your-server/v1/voice/chat -H "Authorization: Bearer $FUSION_API_KEY" \
  -d '{"audio_base64":"..."}'

{"audio_base64":"...", "transcript":"...", "response_text":"...", "latency_ms":612,
 "turns":[{"user":"...","agent":"...","outcome":"completed","metrics":{...}}]}

The whole conversation in one request, no socket. Useful for regression tests, for comparing models on the same audio, and for anything that isn't a live microphone.

Monitoring

GET /health	Status, version, whether models are loaded, active sessions. Open, because load balancers can't hold a key
GET /metrics	Prometheus: latency histograms, turns, errors, event-loop lag, memory. Needs the key

Everything except /health, / and /fusion-runtime.js needs a key — or, on the WebSocket, a session token. See Authentication.

On an NVIDIA GPU

torch on Linux already carries CUDA — its PyPI wheels bundle the NVIDIA libraries — so a normal install is the GPU one. Two packages need help, and both fail quietly if they don't get it.

pip install -e ".[cuda]"
CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86" pip install llama-cpp-python

llama-cpp-python is published as source only and builds without CUDA unless told otherwise; the prebuilt CUDA wheels people link to stopped at 0.2.66. Set CMAKE_CUDA_ARCHITECTURES to your card — 86 for a 3090 or A10, 89 for an L4 or 4090, 80 for an A100 — or it compiles kernels for every architecture and takes about an hour.

Text-to-speech silently on the CPU. onnxruntime-gpu from 1.30 is built for CUDA 13 while torch pins 12.8. Installed together the GPU provider can't load and onnxruntime falls back to the CPU without stopping — synthesis then takes seconds instead of milliseconds. The [cuda] extra caps below 1.30; newer CUDA 12 builds are on Microsoft's own feed:

pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

Voice detection failing to load. If the machine has its own CUDA toolkit (most GPU images do) it may come ahead of torch's bundled libraries: Whisper loads the older libcudart first, torch then imports against it and dies on a missing symbol — which takes the voice detector with it, so turns end on a timer and interruptions stop working. Put torch's own libraries first:

export LD_LIBRARY_PATH="$(python -c "import site,glob,os; p=site.getsitepackages()[0]; print(':'.join(sorted(glob.glob(os.path.join(p,'nvidia','*','lib')))))"):$LD_LIBRARY_PATH"

Check both before trusting a measurement: the log must show vad model.loaded and no Failed to create CUDAExecutionProvider. Either one missing means you are measuring a CPU.

Deploying

Any machine with an NVIDIA GPU runs this; there is nothing provider-specific in the runtime. The docs are written against RunPod because a pod has no request timeout and its proxy gives you https:// and wss:// without handling a certificate — which browsers require before granting a microphone.

With Docker

docker run --env-file .env -p 8000:8000 -v /models:/models fusion-runtime

--env-file reads the file on the host and sets real environment variables in the container. Never COPY .env into an image: that writes your keys into a layer anyone who pulls it can read, even if a later layer deletes the file.

Without Docker

[Service]
EnvironmentFile=/etc/fusion/runtime.env
ExecStart=/opt/fusion/bin/frun up /etc/fusion/agent.py --host 0.0.0.0

It is a Python package, so a GPU VM with a systemd unit works as well as a container. Keep that environment file root:root and mode 0600, written at boot from Secrets Manager, SSM Parameter Store or equivalent — not baked into an image.

On a cloud that doesn't terminate TLS for you, put a load balancer or reverse proxy in front with a certificate, and set FUSION_TRUSTED_PROXY to its address range — until you do, its forwarded headers are ignored and token minting refuses, because a token in a URL over plain HTTP is readable by every hop.

Models belong on a volume rather than in the image: a rebuild then doesn't re-download several gigabytes, and two machines can share one copy. Point FUSION_MODEL_DIR at it.

Observability

Every stage emits structured events with a timestamp, session id, turn id, stage, model and duration, so a deployed agent never runs blind.

20:47:49.441  95b134d2 t1   vad      speech_start       audio_offset_ms=192 probability=0.88
20:47:52.162  95b134d2 t1   turn     end_detected       reason=silence detector=silence threshold_ms=500 wait_ms=569
20:47:52.334  95b134d2 t1   llm      first_token        171ms runtime=llama_cpp model=qwen2.5-0.5b-instruct-q4_k_m.gguf
20:47:52.670  95b134d2 t1   tts      first_chunk        326ms audio_ms=2525
20:47:52.671  95b134d2 t1   audio    first_sent         response_ms=509 ttfa_ms=778

Logs: frun up --log-format pretty|json, --log-level debug|info|warning|error. Errors include a stable code, whether retrying can help, a suggested fix, and the stack trace (server logs only).

Private by default: logs record text lengths, not what people said. --log-content includes transcripts and replies. API keys are always redacted.

Per turn: time to first audio, end-of-turn wait, transcription delay, STT time, LLM time-to-first-token and tokens/s, TTS time-to-first-audio and real-time factor, playback start, interruptions and how fast generation stopped.

Development

pip install -e ".[dev,talk]"     # or: uv sync --extra dev --extra talk
pytest tests/

Tested on Python 3.11, 3.12 and 3.13.

Package layout: cli/ (frun commands), catalog/ (model catalog, installs, downloads), config.py (settings, profiles, model directory), server.py (FastAPI + WebSocket), contract/ (the interface every model runtime implements), resolver.py (model reference → runtime), registry.py (runtime names and plugins → classes), runtimes/ (one adapter per engine: llama_cpp, ctranslate2, onnx, openai_http), engine/ (orchestrator, scheduler, streaming, barge-in), web/ (the console and the browser client), vad/, telemetry/, testing/ (conformance kit and fakes), audio/ (echo cancellation, duplex).

License

Apache-2.0. Embed fusion-runtime in a commercial product, rebrand it, and ship it closed. What Apache-2.0 asks in return is that you keep the copyright notice and the NOTICE file in what you distribute, and that you don't use the project's name in a way that implies it endorses your product. There is no CLA and no copyright assignment for contributors.

The models pulled by default are permissive too — Whisper MIT, Silero VAD MIT, Qwen2.5 Apache-2.0, Kokoro Apache-2.0 — so the whole default path is clear for commercial use, weights included. A model you point it at yourself carries its own licence, and some open-weight models restrict commercial use or rebranding. Check that one before you ship it.

Documentation

Overview

Quickstart

Browser client

Authentication

Several keys

Rotating and revoking

Limits

Origins and proxies

The frun CLI

Models

Using your own models

How it works

Configuration

LLM from any OpenAI-compatible endpoint

Turn detection: when the agent answers

Languages

Server API

A browser

Your backend

A script, or a test

Monitoring

On an NVIDIA GPU

Deploying

With Docker

Without Docker

Observability

Development

License