Models, speech and visual input

Agents v0.10 · Early access · Updated 23 October 2026

Inference services are separate processes. Start with text, confirm the data destination, then enable speech or snapshots.

Language model

Set private .env values and restart:

MODEL_BASE_URL=http://127.0.0.1:1234/v1
MODEL_NAME=your-loaded-model
MODEL_API_KEY=
MODEL_VISION=false
MODEL_STREAM=true

The endpoint accepts POST /chat/completions with model, messages, temperature, max_tokens and stream. SSE replies use choices[0].delta.content with a finish reason or [DONE]. Buffered replies use choices[0].message.content. Set MODEL_STREAM=false for buffered-only services. Incomplete streams report failure rather than persisting a finished answer. Evidence and flow modes bypass ordinary model reasoning.

Remote inference

Use HTTPS and ALLOW_REMOTE_MODEL=true for remote models. Keys remain server-side. The destination receives conversation context; review retention, access and availability independently. Configuration is not a successful provider probe or a quality guarantee.

Speech endpoints

SPEECH_BASE_URL=http://127.0.0.1:8000/v1
SPEECH_API_KEY=
STT_MODEL=your-transcription-model
TTS_MODEL=your-speech-model
TTS_VOICE=your-supported-voice
ALLOW_REMOTE_SPEECH=false

The service implements POST /audio/transcriptions for recorded-turn transcription and POST /audio/speech for reply generation as specified in the package README. Either can be configured independently. When a server function is unconfigured, supported browser recognition/synthesis can be used. Browser recognition may use an external service and is not guaranteed offline. Configured server failures are shown rather than silently changing destination.

Turn-taking and latency

Recognition uploads a recorded turn; synthesis buffers a completed reply. Speech starts after text completion. This is not continuous duplex audio or acoustic barge-in. Interrupt stops playback and cancels in-flight model work, with generation revisions rejecting stale output. Measure time to first audible speech separately from first text on the intended microphone, browser, network and hardware.

Optional local speech

Read the package docs/LOCAL-SPEECH.md before npm run speech:setup. Setup downloads models and creates a separate Python environment; the base app has no such dependency. Review model licence suitability for your use/distribution. A previous warm CPU observation took 18.7 seconds for 7.4 seconds of audio. It is hardware-specific and is not a latency promise.

Visual input

Enable MODEL_VISION=true only for a model supporting JPEG data URLs in image_url content. The UI permits preview and one explicit snapshot for the next message. It does not continuously stream video or render an avatar. Pixels go to the configured model; the store keeps a hasImage marker instead of pixels. Evidence and workflow modes disable this visual path.