Commit 79e81b0bd for llama.cpp
commit 79e81b0bd4098f33b52756e30ef90bfe599cce7e
Author: Aleksander Grygier <aleksander.grygier@gmail.com>
Date: Fri Nov 20 16:34:52 2026 +0100
ui: voice messages and dictation with live waveforms and transcripts (#32531)
* ui: add a voice engine for capture, peaks and transcription
Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.
- VoiceCapture: one microphone start gives live levels (RMS on a dB
curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
an inline AudioWorklet (ScriptProcessor fallback), and on stop a
16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
queueing audio until session.created, a final commit, and a typed
RealtimeUnsupportedError when the model cannot stream or the socket
is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
HTTP transcription of the clip, and expose levels and live text.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add voice waveform and transcript components to Llama UI
Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.
VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.
The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add stories and tests for the voice components
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.
The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: store voice messages with their transcript and send it to models that cannot hear
A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.
The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.
A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.
Copying a message now includes its voice transcript.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add Voice settings and resolve what the microphone does per model
The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.
Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.
- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
any model with audio input; on a single-model server it only says
whether that model can transcribe. It notes that dictating with
another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.
The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: record voice messages and dictation from the chat form
The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.
Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.
- Dictation writes the live transcript into the message at the caret
as it streams. DictationSpan replaces only the dictated stretch, moves
it when the user types before it, keeps what is typed after it, and
hands text the user edits over to them. When nothing streams, the row
shows Transcribing... after stop and the text goes in once it is back.
Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
not into the message. On stop the clip is attached as a voice file
with its peaks and length, and its transcript is followed by the voice
transcripts store (unless Transcribe voice messages is off).
Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.
The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: draw audio attachments as playable waveforms with their transcript
A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.
- The audio plays from an object URL made from the stored base64 (or the
uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
audio file, gets them computed from its audio once per session, in
memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
and shows Transcribing... while the voice transcripts store waits for
it, Transcription unavailable when it failed, and nothing when none
was asked for.
- The full-screen preview keeps its player and shows the transcript.
Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: drop the old audio recorder and proxy realtime transcription in dev
The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate with a model that is already loaded when the chat model cannot hear
The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.
Assisted-by: Claude (Opus 5.5 Medium)
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* ui: fill the recording waveform, and open voice message transcripts
- The live waveform fills the row from the start: a recording shorter
than the row is stretched over all its bars, a longer one scrolls with
the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
while it comes in and in the sent message, with the toggle beside it
to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
folded instead, opening on hover, keyboard focus or a tap.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f948ed976e40f693ce5d3acf1168284b05)
* ui: record voice without ducking other audio, from the first word
- Voice input asks for the raw microphone signal: no echo cancellation,
noise suppression or auto gain. Echo cancellation made the system treat
a recording as a call, which lowers or interrupts other playback, and
auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
loaded once, suspended in between and, where the browser allows it,
without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
Starting... instead of running the timer. Development builds log how
long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
built-in microphone when the default is a Bluetooth headset, which
would switch to its low-quality call mode), System default, or a
listed microphone.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7210d21617d734fdfde677e84af7273153)
* ui: show voice text live in the message box, and follow transcripts as spoken
- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
while you record: the realtime client commits the audio piece by piece,
at the pauses the levels show, one piece at a time, and the final
commit takes only what is left.
- A voice message's live text shows as the empty message box's
placeholder. Clicking into the box makes it text to edit there, and
what is said next follows it. On stop the text leaves the box and goes
with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
full foreground colour, which they keep, the rest muted. A clip's
transcript follows its playback (estimated over the parts with speech,
as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce773017f902f8a5690ab9ecca368cecb6427422)
* ui: let the capture test start with silence
A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee5337b83fef685a5cbaff176925f1647ce9d)
* ui: show live voice text sooner, and keep silence from becoming "yeah"
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
cut at a 0.35 s pause after speech, or at the next quiet moment once a
piece reaches 3 s, so the text follows the speaker within a few
seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
only if speech was heard in it. LFM3-Omni transcribes a commit of pure
silence as "Yeah." (or "はい。"), which ended up at the end of the text.
A recording whose only piece is that rest still commits it.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cbab0fba9d8e84b9491890a406e3c6c90f3)
* ui: transcribe a recording cut at pauses once more as a whole
Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.
Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd9632e6c88000a1d23c2575bd1e6fcdd9ec1)
* ui: give live text over HTTP for models without a realtime session
A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when the browser processes the microphone despite the raw request
The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone beside the stop and send buttons
While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show the voice components' stories with Llama UI
VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their
stories move to the library's Storybook, under Llama UI/Voice, with the demo
audio they play. Llama Chat's Storybook shows them there too, and its own
voice stories take the demo audio from the library. VoiceWaveformLive stays
with Llama Chat, as its demo records through the chat form's microphone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a noise suppression setting for voice input
Recordings stay raw by default. Settings > Voice can now switch the
browser's noise suppression on; echo cancellation and automatic gain stay
off either way, as they interrupt other sound and fade the first words in.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep live voice text coming when the room is never quiet
Pieces were cut only at a quiet moment, so with music or talk in the room
none was ever cut and no text showed until the recording ended. A piece half
as long again as its longest length is now cut where it is.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add echo cancellation and automatic gain settings for voice input
Settings > Voice now has a switch for each kind of microphone processing the
browser offers. All three stay off by default, so recordings are raw; each
one says what switching it on costs.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: put the microphone after the stop button, filled like it
While a reply streams, the microphone is the last button of the row and
looks like the stop and skip buttons beside it, instead of a bare icon
before them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say when dictated text is being refined, and send it refined
After a recording with live text, the whole clip is transcribed once more
so pauses do not end sentences. The form now says so while that pass is
out, and a send asked for meanwhile waits for it, so the message goes out
with the refined text rather than the pieces.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the send button in place and put the microphone beside it
The send button no longer gives way to the microphone on an empty
message. The microphone stands before it as a quieter, tertiary button,
with the same tooltip, warning and recording states as before.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let a recording be sent while a reply streams, once it has words
While a reply streams, the recording row now shows its send button as
soon as the first transcribed words are in the message, so the
recording no longer has to be stopped first. The microphone also stays
beside the send button once a voice message is attached.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: raise quiet speech for transcription, and say when it is too quiet
A quiet microphone gave the model near silence to transcribe. A new
Voice setting, Boost quiet speech (on by default), raises speech to a
level speech models hear well, up to eight times louder: the audio
streamed for live text as it goes, and the whole recording in one pass
when it stops, so its first words get the same gain as the rest. Only
speech is measured, never the silence around it, loud speech is left
as it is, and a soft limiter keeps raised peaks from clipping. When
speech stays quiet even at the highest gain, the recording row says so.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine free of app code
The engine in lib/voice imported the microphone messages from the app's
utils and the WAV type from its enums, the only two things keeping it
from moving into a package of its own. The microphone checks and
messages move into the engine (voice/support.ts), next to the capture
that needs them; the uploaded-file helpers stay in the app's utils.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: start voice sessions from one voice service
How a recording reaches the server (routes, API key, whether the
realtime socket can be used) and how the Voice settings become capture
and transcribe options were built inside the chat form's voice hook,
and copied into every other feature that records. VoiceService builds
them in one place: a feature asks it for its session options and adds
only its own callbacks.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tell speech from the room in one place
The pause detector, the speech gain and (in voice mode) the listener for
speech over a reply each followed a noise floor with their own numbers.
A SpeechDetector now does it for all of them, in dB, with one set of
settings in VOICE_SPEECH.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let any text field take dictation
Writing dictated text into a field (at the caret, around the user's
typing, replaced by the whole recording's text when it comes) moves out
of the chat form's hook into a plain createDictation(). The hook keeps
what is the chat form's own: voice messages, toasts and the hint.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the recording row into Llama UI
VoiceRecordingBar draws the row shown while recording (cancel, waveform
and time, send, and a caption or warning under it) from data alone, so
voice mode or a notes page can show the same row. The chat form passes
its labels and its microphone button as the stop button.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the voice engine's Svelte recorder behind its own entry
VoiceRecorder, the one part of lib/voice written with runes, moves to
lib/voice/svelte, so the rest of the engine is plain TypeScript any app
can use without Svelte.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move voice input into its own package, Llama Voice
The voice engine (capture, the speech signal, transcription over the
realtime socket or HTTP) and dictation into a text field move from the
app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a
notes app, can use them. The Svelte recorder is its ./svelte entry.
Llama Chat compiles the sources, as it does Llama UI's, and keeps its
server and settings wiring. The package has its own type check, tests
and build, and a CI job for them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show which model does what in voice input
The model picker and the Models manager mark each model that hears:
Dictation on the one that transcribes dictation, Speech to text on
transcription models such as Qwen3-ASR, and Voice on chat models that
take voice messages. A loaded model kept only for dictation sits apart
from the chat models, a model can be made the dictation model from its
row, and Settings > Voice groups the transcription models apart from
the chat models with audio input. A voice store now resolves what the
microphone does, for the chat form and the model lists alike.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: warn of a quiet microphone inside the recording row
The too-quiet line under the waveform made the form grow and jump. The
warning now stays inside the row: the waveform turns the warning colour,
an icon sits by the time, and the advice opens above the row in a
tooltip for a few seconds, then on hover or focus. Screen readers still
hear it when it appears.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the microphone while the pointer is on its button
The first seconds of a recording could come out quiet while the
microphone settled, and the too-quiet warning showed for them. With the
pointer or the focus on the microphone button, the microphone now opens
ahead of the press (only when the page may already use it, so nothing
asks), and the recording takes it at once; it is let go shortly after
the pointer leaves without a press. The warning also waits until the
speech has stayed too quiet for two seconds.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tidy voice input after review
Fix a microphone left open when a recording is cancelled while it starts
or the pointer leaves while it opens, a capture that is still letting go
silencing the next one, and live text stopping after a realtime error.
Share the dictation model choice, the transcript lookup, base64 decoding,
PCM joining and stopping a stream instead of repeating them, and export
only Llama Voice's public API.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin Llama Voice's dev dependencies to Llama UI's versions
Assisted-by: Claude (Opus 5.5 Medium)
* ui: let demos speak into a synthetic microphone
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the microphone for voice input, off the audio files entry
The attach menu's Audio Files entry used a microphone, which now sits
beside the form's own microphone button. The microphone stands for
recording and dictation only; a model's audio modality is headphones, and
the audio files entry is a file with headphones.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: set voice input on the slimmed design tokens
Voice input's components use the type roles, the one corner scale and
shadcn's utility names for the neutral colours, as the rest of Llama UI
and Llama Chat do since the design tokens were slimmed down.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name voice states, platform strings, patterns and tunings
Addresses the review on the voice PR:
- Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and
RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException
names, AudioContext states, transcription form fields, socket protocols,
device kinds, worklet messages, error messages, regexes and tuning values
are named constants.
- Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and
VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType
join its UI enums; the voice components' sizes, labels, timings, selectors
and patterns move to constants/voice.constants.ts. The voice unit tests
move from src to tests/unit.
- App: ChatFormVoiceState is an enum, the recording bar's status comes from a
map, and the voice code uses the enums above, BooleanString, HEADERS and
new voice constants in place of inline strings, regexes and numbers.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep all of Llama UI's tests under tests
The Model UI formatting spec moves to tests/unit and the button spec to
tests/client, next to the voice specs; the test projects now read only
tests/unit and tests/client.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: show a transcript's words without the model's own format
Qwen3-ASR names the language before its text (language French<asr_text>...) and
VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they
are. The voice engine now takes both off, over HTTP and over the realtime socket, also
while segments still stream, so the message box and the clip show the words alone.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: dictate when a speech-to-text model is picked for the conversation
A speech-to-text model has audio input, so on Automatic the microphone recorded a voice
message for it, which such a model has nothing to answer with. It now dictates, as the
model's badge already says it is for.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: say why a live-only model cannot transcribe behind an API key
A browser websocket cannot send the API key, so the realtime socket is left out on a
server that needs one, and a model that only transcribes over it (Voxtral Realtime)
answers the HTTP route with a 501. The clip and the dictation toast now say so instead of
only that transcription is unavailable.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pick the dictation model with a microphone on its row
The model that transcribes dictation was marked with a filled Dictation badge. It is now
marked by a red microphone among the row's actions in the model picker, beside the heart,
and every other model that hears shows the same microphone on hover: a click makes it the
transcription model, and a click on the chosen one hands the choice back to Automatic. The
badge keeps saying what kind of hearing model a row is, and the Models manager marks the
dictation model with a small microphone beside its name.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
diff --git a/.github/workflows/ui-self-hosted.yml b/.github/workflows/ui-self-hosted.yml
index 346f1cfb4..4967de07c 100644
--- a/.github/workflows/ui-self-hosted.yml
+++ b/.github/workflows/ui-self-hosted.yml
@@ -19,6 +19,7 @@ on:
'.github/workflows/ui-build-self-hosted.yml',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
+ 'ui/lib/voice/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -28,6 +29,7 @@ on:
'.github/workflows/ui-build-self-hosted.yml',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
+ 'ui/lib/voice/**.*',
'tools/server/tests/**.*'
]
diff --git a/.github/workflows/ui.yml b/.github/workflows/ui.yml
index 10d7ed0ed..65e5c36ff 100644
--- a/.github/workflows/ui.yml
+++ b/.github/workflows/ui.yml
@@ -150,6 +150,37 @@ jobs:
run: npm run prepack
working-directory: ui/lib/ui
+ voice-checks:
+ name: Voice checks
+ runs-on: ubuntu-24.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v6
+ with:
+ ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v6
+ with:
+ node-version: "24"
+ package-manager-cache: false
+
+ - name: Install dependencies
+ run: npm ci
+ working-directory: ui/lib/voice
+
+ - name: Run type checking
+ run: npm run check
+ working-directory: ui/lib/voice
+
+ - name: Run tests
+ run: npm test
+ working-directory: ui/lib/voice
+
+ - name: Build the package
+ run: npm run prepack
+ working-directory: ui/lib/voice
+
e2e-tests:
name: E2E Tests
needs: ui-build
diff --git a/ui/README.md b/ui/README.md
index b70228d0d..0f210a40c 100644
--- a/ui/README.md
+++ b/ui/README.md
@@ -5,7 +5,8 @@ The UI of llama.cpp: the packages in `lib` and the apps built on them.
```text
ui/
├── lib/
-│ └── ui/ @ggml-org/llama-ui: design tokens, fonts, surfaces and components
+│ ├── ui/ @ggml-org/llama-ui: design tokens, fonts, surfaces and components
+│ └── voice/ @ggml-org/llama-voice: microphone capture, transcription and dictation
└── apps/
└── chat/ Llama Chat, the chat app that llama-server serves
```
@@ -13,12 +14,15 @@ ui/
- [`lib/ui`](lib/ui/README.md) draws the parts and handles interaction, with no business logic. It is built with
`@sveltejs/package` and published to npm as `@ggml-org/llama-ui`. Apps in this repo skip the package: they compile
its sources as part of their own build, so they hold no copy of a component, a token or a font.
+- [`lib/voice`](lib/voice/README.md) records the microphone and turns speech into text through llama-server, with no
+ UI: Llama UI draws the voice parts. It is plain TypeScript built with `tsc`, plus a Svelte entry
+ (`@ggml-org/llama-voice/svelte`). Apps in this repo compile its sources too. It is not published to npm yet.
- [`apps/chat`](apps/chat/README.md) talks to llama-server, stores conversations and runs the agentic loop. CMake
builds it into llama-server (see `scripts/ui-assets.cmake`), and it is published to npm as `@ggml-org/llama-chat`
(with `llama-chat` as its short name), so `npx llama-chat` runs it against any llama-server.
Each folder is its own npm package with its own `package-lock.json`: run `npm ci` in the one you work on. Building,
-checking and testing `apps/chat` needs no install in `lib/ui`.
+checking and testing `apps/chat` needs no install in `lib/ui` or `lib/voice`.
## Releases
diff --git a/ui/apps/chat/.storybook/main.ts b/ui/apps/chat/.storybook/main.ts
index 3e8c1a39d..e2b8dad14 100644
--- a/ui/apps/chat/.storybook/main.ts
+++ b/ui/apps/chat/.storybook/main.ts
@@ -16,7 +16,7 @@ const config: StorybookConfig = {
'../tests/stories/**/*.stories.@(js|ts|svelte)',
{
directory: '../../../lib/ui/tests/stories',
- files: '{foundations,components,app,models,mockups}/**/*.stories.svelte'
+ files: '{foundations,components,app,models,mockups,voice}/**/*.stories.svelte'
}
],
addons: [
diff --git a/ui/apps/chat/DESIGN.md b/ui/apps/chat/DESIGN.md
index f3a1eaf98..efbc65d93 100644
--- a/ui/apps/chat/DESIGN.md
+++ b/ui/apps/chat/DESIGN.md
@@ -1154,6 +1154,12 @@ screens. Status lamps, avatars and the round icon buttons in the chat form are f
(`--sidebar-reveal-x`), taking no taps while the page is aside. The page keeps room under its
last row (`max-md:pb-safe-24`) so the key never covers it. One per page, only for its main
action; from `md` up the action stays a button in the header.
+- **Voice clips:** a recorded clip is a round primary play key, its waveform and the time in
+ `caption`: bars in the muted foreground at 40%, the played part in the primary colour, so
+ the clip stays neutral in both modes. The live waveform while recording is as tall as the
+ input's line (24px), the newest level on the right, and holds still with reduced motion. A
+ voice message's transcript opens under the clip on hover, keyboard focus or a tap, and
+ stays in the DOM for screen readers while closed.
- **Copy feedback:** a copy action turns its icon into a `fg-success` check for 1.5s
(`COPY_FEEDBACK_RESET_MS`), only when the copy succeeded, then back. The icon pops in with a
150ms fade and scale, skipped on first render and with reduced motion.
diff --git a/ui/apps/chat/README.md b/ui/apps/chat/README.md
index 5fa37286a..abee5d43b 100644
--- a/ui/apps/chat/README.md
+++ b/ui/apps/chat/README.md
@@ -89,6 +89,7 @@ The app imports the library as `@ggml-org/llama-ui/ui/<name>` and compiles its s
| Routes | `src/routes` | `/`, `/chat/[id]`, `/models`, `/mcp-servers`, `/settings`, `/search` |
| App components | `src/lib/components/app` | Chat, navigation, models, MCP, settings and dialogs, grouped by area |
| UI components | `ui/lib/ui` | shadcn-svelte on bits-ui, from Llama UI |
+| Voice | `ui/lib/voice` | Microphone capture, transcription and dictation, from Llama Voice (`@ggml-org/llama-voice`) |
| Hooks | `src/lib/hooks` | View logic between components and stores: scroll, drag and drop, pickers, keyboard |
| Stores | `src/lib/stores` | Application state as Svelte 5 runes: chat, conversations, the agentic loop, tools, models, settings |
| Services | `src/lib/services` | Stateless I/O: the llama-server API, IndexedDB, MCP, the Hugging Face Hub, the tool sandbox |
diff --git a/ui/apps/chat/demo/README.md b/ui/apps/chat/demo/README.md
index 8769da6ef..e92962687 100644
--- a/ui/apps/chat/demo/README.md
+++ b/ui/apps/chat/demo/README.md
@@ -115,7 +115,7 @@ Records every `scripts/docs/*.demo.ts` from the storage target and uploads `vide
## Writing a demo
-Scripts get `test` and `expect` from `helpers.ts` without importing them. `demo.click`, `demo.hover` and `demo.type` glide a visible cursor to the target before acting, with timing varied like a person's (seeded per demo, so re-takes match); `demo.scroll` turns the mouse wheel over a target; `demo.zoom` pinches in on a control or a point to show a detail too small to read in the frame, holds, and pinches back out (with the fingertips drawn on a touch device); `demo.caption` shows a caption at the top of the frame. Use `test.use({ settings, startPath })` to seed other settings or start on another route.
+Scripts get `test` and `expect` from `helpers.ts` without importing them. `demo.click`, `demo.hover` and `demo.type` glide a visible cursor to the target before acting, with timing varied like a person's (seeded per demo, so re-takes match); `demo.scroll` turns the mouse wheel over a target; `demo.zoom` pinches in on a control or a point to show a detail too small to read in the frame, holds, and pinches back out (with the fingertips drawn on a touch device); `demo.caption` shows a caption at the top of the frame. For voice input, `demo.microphone()` gives the page a microphone that hears nothing but `demo.speak(text)`, which reads the text in a synthetic voice (macOS `say`) and shows it as a caption, since the video has no sound. Use `test.use({ settings, startPath })` to seed other settings or start on another route.
## Conversations
diff --git a/ui/apps/chat/demo/helpers.ts b/ui/apps/chat/demo/helpers.ts
index 1c0ad7734..97e7bdb11 100644
--- a/ui/apps/chat/demo/helpers.ts
+++ b/ui/apps/chat/demo/helpers.ts
@@ -3,7 +3,15 @@ import { generateTrace } from './fixtures/traces.ts';
import { ScreencastRecorder } from './recorder.ts';
import { type CDPSession, type Locator, type Page, test as base } from '@playwright/test';
import { execFileSync } from 'node:child_process';
-import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import {
+ appendFileSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ writeFileSync
+} from 'node:fs';
+import { tmpdir } from 'node:os';
import { basename, join, resolve } from 'node:path';
const VIDEOS_DIR = resolve(import.meta.dirname, 'videos');
@@ -112,6 +120,68 @@ function seedSettings(config: Record<string, unknown>): void {
}
}
+/**
+ * Swaps the page's microphone for one that hears only what `window.demoSpeak` plays into it,
+ * silence otherwise. Chromium's fake capture device crashes the page, so voice input is
+ * recorded with this instead. Runs in the page.
+ */
+function installFakeMicrophone(): void {
+ const context = new AudioContext();
+ const output = context.createMediaStreamDestination();
+
+ navigator.mediaDevices.getUserMedia = async (constraints) => {
+ if (!constraints?.audio || constraints.video) {
+ throw new DOMException('demo: only the microphone is faked', 'NotSupportedError');
+ }
+
+ await context.resume();
+
+ // each opening gets its own tracks, so a recording that stops them leaves the source on
+ return new MediaStream(output.stream.getAudioTracks().map((track) => track.clone()));
+ };
+
+ Object.assign(window, {
+ demoSpeak: async (wavBase64: string) => {
+ const bytes = Uint8Array.from(atob(wavBase64), (c) => c.charCodeAt(0));
+ const source = context.createBufferSource();
+
+ source.buffer = await context.decodeAudioData(bytes.buffer);
+ source.connect(output);
+ await context.resume();
+ await new Promise((done) => {
+ source.onended = done;
+ source.start();
+ });
+ }
+ });
+}
+
+/** `text` read by a synthetic voice, as a base64 WAV. Uses macOS `say`. */
+function synthesizeSpeech(text: string, voice: string): string {
+ const dir = mkdtempSync(join(tmpdir(), 'demo-speech-'));
+ const file = join(dir, 'speech.wav');
+
+ try {
+ execFileSync('say', [
+ '-v',
+ voice,
+ '-o',
+ file,
+ '--file-format=WAVE',
+ '--data-format=LEI16@24000',
+ text
+ ]);
+
+ return readFileSync(file).toString('base64');
+ } catch (error) {
+ throw new Error(`demo: speak() needs macOS \`say\` with the voice ${voice}`, {
+ cause: error
+ });
+ } finally {
+ rmSync(dir, { force: true, recursive: true });
+ }
+}
+
/** Seeded randomness, so a demo moves the same way on every recording. */
function createRandom(seed: string): () => number {
let state = [...seed].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 2654435761), 1);
@@ -143,19 +213,9 @@ export class Demo {
/** Shows a caption at the top of the frame for `ms`, clear of the chat form. */
async caption(text: string, ms = 2200): Promise<void> {
- await this.page.evaluate((t) => {
- const el = document.getElementById('demo-caption');
-
- if (!el) return;
-
- el.textContent = t;
- el.classList.add('visible');
- }, text);
+ await this.showCaption(text);
await this.page.waitForTimeout(ms);
- await this.page.evaluate(() =>
- document.getElementById('demo-caption')?.classList.remove('visible')
- );
- await this.page.waitForTimeout(250);
+ await this.hideCaption();
}
async click(target: Locator, ms = 400): Promise<void> {
@@ -182,6 +242,15 @@ export class Demo {
await this.pause(ms);
}
+ /**
+ * Gives the page a microphone that hears only what `speak` says, and allows the page to
+ * use it. Call it before the microphone is first used.
+ */
+ async microphone(): Promise<void> {
+ await this.page.context().grantPermissions(['microphone']);
+ await this.page.evaluate(installFakeMicrophone);
+ }
+
/** Glides the cursor to a point near the center of `target`. */
async moveTo(target: Locator): Promise<void> {
await target.scrollIntoViewIfNeeded();
@@ -240,6 +309,23 @@ export class Demo {
await this.pause(ms);
}
+ /**
+ * Says `text` into the microphone from `microphone` in a synthetic voice (macOS `say`), and
+ * returns once it is said. The words show as a caption, since the video has no sound.
+ */
+ async speak(text: string, voice = 'Samantha', ms = 400): Promise<void> {
+ const wav = synthesizeSpeech(text, voice);
+
+ await this.showCaption(`“${text}”`);
+ await this.page.evaluate(
+ (base64) =>
+ (window as unknown as { demoSpeak(wav: string): Promise<void> }).demoSpeak(base64),
+ wav
+ );
+ await this.hideCaption();
+ await this.pause(ms);
+ }
+
/** Types quickly with uneven key timing, a little slower after spaces and punctuation. */
async type(target: Locator, text: string, ms = 500): Promise<void> {
await this.click(target, 200);
@@ -305,6 +391,13 @@ export class Demo {
return { x: box.x + box.width / 2, y: box.y + box.height / 2 };
}
+ private async hideCaption(): Promise<void> {
+ await this.page.evaluate(() =>
+ document.getElementById('demo-caption')?.classList.remove('visible')
+ );
+ await this.page.waitForTimeout(250);
+ }
+
/** A pinch that reaches the screen's edge can end without its last touchend reaching the page. */
private async liftFingers(): Promise<void> {
await this.page.evaluate(() =>
@@ -340,6 +433,17 @@ export class Demo {
return this.cdp;
}
+
+ private async showCaption(text: string): Promise<void> {
+ await this.page.evaluate((t) => {
+ const el = document.getElementById('demo-caption');
+
+ if (!el) return;
+
+ el.textContent = t;
+ el.classList.add('visible');
+ }, text);
+ }
}
interface DemoOptions {
diff --git a/ui/apps/chat/scripts/llama-ui.js b/ui/apps/chat/scripts/llama-ui.js
index c4eb26af7..f8963593a 100644
--- a/ui/apps/chat/scripts/llama-ui.js
+++ b/ui/apps/chat/scripts/llama-ui.js
@@ -26,6 +26,7 @@ const LLAMA_UI_EXPORTS = [
['@ggml-org/llama-ui/enums', `${source}/enums/index.ts`],
['@ggml-org/llama-ui/code', `${source}/utils/index.ts`],
['@ggml-org/llama-ui/utils', `${source}/components/ui/utils.ts`],
+ ['@ggml-org/llama-ui/voice', `${source}/components/voice/index.ts`],
['@ggml-org/llama-ui/styles/*', `${source}/styles/*`],
['@ggml-org/llama-ui/theme.css', `${source}/styles/theme.css`],
['@ggml-org/llama-ui/tokens.css', `${source}/styles/tokens.css`],
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItem.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItem.svelte
index d7284aa61..5a0987154 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItem.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItem.svelte
@@ -1,9 +1,11 @@
<script lang="ts">
+ import { VoiceWaveformSize } from '@ggml-org/llama-ui/voice';
import {
ChatAttachmentsListItemMcpPrompt,
ChatAttachmentsListItemMcpResource,
ChatAttachmentsListItemThumbnailFile,
- ChatAttachmentsListItemThumbnailImage
+ ChatAttachmentsListItemThumbnailImage,
+ ChatAttachmentsListItemVoice
} from '#lib/components/app/index.ts';
import { AttachmentType } from '#lib/enums/index.ts';
import type {
@@ -12,7 +14,7 @@
DatabaseMessageExtraMcpResource,
MCPResourceAttachment
} from '#lib/types/index.ts';
- import { isMcpPrompt, isMcpResource, isPdfFile } from '#lib/utils/index.ts';
+ import { isAudioFile, isMcpPrompt, isMcpResource, isPdfFile } from '#lib/utils/index.ts';
interface Props {
class?: string;
@@ -93,6 +95,15 @@
class="flex-shrink-0 {className} {scrollClasses}"
onclick={() => onMcpResourcePreview?.(mcpResource)}
/>
+{:else if isAudioFile(item.attachment, item.uploadedFile)}
+ <!-- a clip plays where it is: the waveform takes the presses a chip would -->
+ <ChatAttachmentsListItemVoice
+ class="{layoutClasses} {className} {scrollClasses}"
+ {item}
+ onRemove={onFileRemove}
+ {readonly}
+ size={readonly ? VoiceWaveformSize.MD : VoiceWaveformSize.SM}
+ />
{:else if item.isImage && item.preview}
<ChatAttachmentsListItemThumbnailImage
class="{layoutClasses} cursor-pointer {className} {scrollClasses}"
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItemVoice.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItemVoice.svelte
new file mode 100644
index 000000000..9da8a28db
--- /dev/null
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItemVoice.svelte
@@ -0,0 +1,126 @@
+<!--
+ An audio attachment drawn as a clip: a playable waveform, and for a voice message its transcript under it, open while
+ it comes in and after, unless the Collapse transcripts setting folds it (then it opens on hover, keyboard focus or a
+ tap). `sm` is the chip in the chat form, `md` the clip in a sent message. A clip
+ stored without peaks (an older message, an uploaded file) gets them computed from its audio.
+-->
+<script lang="ts">
+ import {
+ spokenLength,
+ VoiceTranscriptReveal,
+ VoiceWaveform,
+ VoiceWaveformSize
+ } from '@ggml-org/llama-ui/voice';
+ import { X } from '@lucide/svelte';
+ import { ActionIcon } from '#lib/components/app/index.ts';
+ import { SETTINGS_KEYS, VOICE_MESSAGE_LABEL } from '#lib/constants/index.ts';
+ import { settingsStore, voiceTranscriptsStore } from '#lib/stores/index.ts';
+ import type { ChatAttachmentDisplayItem } from '#lib/types/index.ts';
+ import { audioUrl, lazyPeaks, voiceAttachmentSource } from '#lib/utils/voice-attachment.ts';
+
+ interface Props {
+ class?: string;
+ item: ChatAttachmentDisplayItem;
+ onRemove?: (id: string) => void;
+ readonly?: boolean;
+ size?: VoiceWaveformSize;
+ }
+
+ let {
+ class: className = '',
+ item,
+ onRemove,
+ readonly = false,
+ size = VoiceWaveformSize.MD
+ }: Props = $props();
+
+ let source = $derived(voiceAttachmentSource(item));
+ let computedPeaks = $state<number[] | null>(null);
+ let src = $state<string | undefined>();
+
+ let peaks = $derived(source.peaks ?? computedPeaks ?? []);
+ let shown = $derived(voiceTranscriptsStore.shown(source.name, source.transcript));
+ let transcript = $derived(shown.transcript);
+ let status = $derived(shown.status);
+ let errorReason = $derived(shown.errorReason);
+ let label = $derived(source.voice ? VOICE_MESSAGE_LABEL : source.name);
+ // the transcript lights up word by word as the clip plays, and stays lit
+ let playbackMs = $state<number | null>(null);
+ let spoken = $derived(
+ transcript && playbackMs !== null && source.durationMs
+ ? spokenLength(transcript, peaks, source.durationMs, playbackMs)
+ : null
+ );
+ let collapsed = $derived(settingsStore.config[SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS] === true);
+
+ // an object URL lives as long as the clip is on screen
+ $effect(() => {
+ const audio = source.audio;
+
+ if (!audio) {
+ src = undefined;
+
+ return;
+ }
+
+ const playable = audioUrl(audio, source.mimeType);
+
+ src = playable?.url;
+
+ return () => {
+ if (playable?.revoke) URL.revokeObjectURL(playable.url);
+ };
+ });
+
+ $effect(() => {
+ const { audio, name, peaks: stored } = source;
+
+ computedPeaks = null;
+
+ if (stored || !audio) return;
+
+ let cancelled = false;
+
+ lazyPeaks(name, audio).then((value) => {
+ if (!cancelled) computedPeaks = value;
+ });
+
+ return () => {
+ cancelled = true;
+ };
+ });
+</script>
+
+<div
+ class={[
+ 'group relative min-w-0',
+ size === VoiceWaveformSize.SM
+ ? 'w-60 max-w-[calc(100vw-4rem)] rounded-lg border border-border bg-muted py-2 pr-9 pl-2'
+ : 'w-80 max-w-full rounded-2xl matte-sheen bg-(--user-bubble-background) px-3 py-2.5 backdrop-blur-md dark:bg-primary/15',
+ className
+ ]}
+ data-slot="chat-attachment-voice"
+ data-voice={source.voice || undefined}
+>
+ {#key collapsed}
+ <VoiceTranscriptReveal defaultOpen={!collapsed} {errorReason} {spoken} {status} {transcript}>
+ <VoiceWaveform bind:playbackMs durationMs={source.durationMs} {label} {peaks} {size} {src} />
+ </VoiceTranscriptReveal>
+ {/key}
+
+ {#if !source.voice}
+ <p class="mt-1 truncate px-1 text-caption text-muted-foreground">{source.name}</p>
+ {/if}
+
+ {#if !readonly && onRemove}
+ <div class="absolute top-1/2 right-1 -translate-y-1/2">
+ <ActionIcon
+ ariaLabel={source.voice ? 'Remove voice message' : `Remove ${source.name}`}
+ icon={X}
+ onclick={() => onRemove?.(item.id)}
+ stopPropagationOnClick
+ tooltip="Remove"
+ />
+ </div>
+ {/if}
+</div>
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsPreview/ChatAttachmentsPreviewCurrentItem/ChatAttachmentsPreviewCurrentItemAudio.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsPreview/ChatAttachmentsPreviewCurrentItem/ChatAttachmentsPreviewCurrentItemAudio.svelte
index f1d6a46cc..2d2ee228b 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsPreview/ChatAttachmentsPreviewCurrentItem/ChatAttachmentsPreviewCurrentItemAudio.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatAttachments/ChatAttachmentsPreview/ChatAttachmentsPreviewCurrentItem/ChatAttachmentsPreviewCurrentItemAudio.svelte
@@ -1,12 +1,23 @@
<script lang="ts">
+ import { VoiceTranscriptStatus } from '@ggml-org/llama-ui/voice';
import { Music } from '@lucide/svelte';
+ import { VOICE_MESSAGE_LABEL } from '#lib/constants/index.ts';
+ import { voiceTranscriptsStore } from '#lib/stores/index.ts';
+ import type { ChatAttachmentDisplayItem } from '#lib/types/index.ts';
+ import { voiceAttachmentSource } from '#lib/utils/voice-attachment.ts';
interface Props {
- currentItem: { name?: string } | null;
+ currentItem: ChatAttachmentDisplayItem | null;
audioSrc: string | null;
}
let { audioSrc, currentItem }: Props = $props();
+
+ let source = $derived(currentItem ? voiceAttachmentSource(currentItem) : null);
+ // a voice message's transcript, from the message or still on its way
+ let shown = $derived(source ? voiceTranscriptsStore.shown(source.name, source.transcript) : null);
+ let transcript = $derived(shown?.transcript);
+ let transcribing = $derived(shown?.status === VoiceTranscriptStatus.TRANSCRIBING);
</script>
<div class="flex flex-1 items-center justify-center p-8">
@@ -21,6 +32,19 @@
<p class="mb-4 text-white/70">Audio preview not available</p>
{/if}
- <p class="text-body-sm text-white/50">{currentItem?.name || 'Audio'}</p>
+ <p class="text-body-sm text-white/50">
+ {source?.voice ? VOICE_MESSAGE_LABEL : currentItem?.name || 'Audio'}
+ </p>
+
+ {#if transcript}
+ <p
+ class="mt-6 max-h-[40vh] overflow-y-auto text-left text-body-sm whitespace-pre-wrap text-white/80"
+ data-slot="voice-preview-transcript"
+ >
+ <span class="sr-only">Transcript: </span>{transcript}
+ </p>
+ {:else if transcribing}
+ <p class="mt-6 text-caption text-white/50">Transcribing...</p>
+ {/if}
</div>
</div>
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatForm.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatForm.svelte
index e99c1eac3..727ceccd7 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatForm.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatForm.svelte
@@ -1,10 +1,13 @@
<script lang="ts">
import ContextGaugePopup from './ChatFormContextGauge/ContextGaugePopup.svelte';
+ import { microphoneUnavailableReason, warmUpVoiceCapture } from '@ggml-org/llama-voice';
+ import { VoiceRecorder } from '@ggml-org/llama-voice/svelte';
import { Maximize, Minimize } from '@lucide/svelte';
import {
ActionIcon,
ChatAttachmentsList,
ChatFormActions,
+ ChatFormActionsRecording,
ChatFormCurrentWorkingDirectory,
ChatFormInput,
ChatFormInputFileInputInvisible,
@@ -18,7 +21,8 @@
ICON_CLASS_DEFAULT,
INITIAL_FILE_SIZE,
PROMPT_CONTENT_SEPARATOR,
- SETTING_CONFIG_DEFAULT
+ SETTING_CONFIG_DEFAULT,
+ VOICE_EDITOR_SELECTOR
} from '#lib/constants/index.ts';
import {
ContentPartType,
@@ -30,6 +34,10 @@
} from '#lib/enums/index.ts';
import { useChatFormExpansion } from '#lib/hooks/use-chat-form-expansion.svelte.ts';
import { useChatFormPickers } from '#lib/hooks/use-chat-form-pickers.svelte.ts';
+ import {
+ useChatFormVoice,
+ type VoiceRecorderLike
+ } from '#lib/hooks/use-chat-form-voice.svelte.ts';
import {
chatStore,
conversationsStore,
@@ -39,7 +47,8 @@
serverStore,
settingsStore,
toolsStore,
- uiStore
+ uiStore,
+ voiceTranscriptsStore
} from '#lib/stores/index.ts';
import type {
FileMentionEntry,
@@ -48,14 +57,6 @@
MCPResourceInfo,
PromptMessage
} from '#lib/types/index.ts';
- import {
- AudioRecorder,
- convertToWav,
- createAudioFile,
- describeMicrophoneError,
- isAudioRecordingSupported,
- microphoneUnavailableReason
- } from '#lib/utils/browser-only.ts';
import {
buildMentionInsertion,
findCommandToken,
@@ -65,7 +66,7 @@
parseClipboardContent,
uuid
} from '#lib/utils/index.ts';
- import { onMount, untrack } from 'svelte';
+ import { onMount, tick, untrack } from 'svelte';
import type { ClassValue } from 'svelte/elements';
import { toast } from 'svelte-sonner';
@@ -96,6 +97,9 @@
onUploadedFileRemove?: (fileId: string) => void;
onUploadedFilesChange?: (files: ChatUploadedFile[]) => void;
onValueChange?: (value: string) => void;
+
+ /** Records voice input; tests and stories pass a fake, the app a VoiceRecorder. */
+ voiceRecorder?: VoiceRecorderLike;
}
let {
@@ -117,21 +121,20 @@
showModelSelector = true,
singleRow = false,
uploadedFiles = $bindable([]),
- value = $bindable('')
+ value = $bindable(''),
+ voiceRecorder
}: Props = $props();
// Component References
- let audioRecorder: AudioRecorder | undefined;
let chatFormActionsRef: ChatFormActions | undefined = $state(undefined);
let fileInputRef: ChatFormInputFileInputInvisible | undefined = $state(undefined);
let pickersRef: { handleKeydown: (event: KeyboardEvent) => boolean } | undefined =
$state(undefined);
let inputRef: ChatFormInput | undefined = $state(undefined);
- // Audio Recording State
- let isRecording = $state(false);
- let recordingSupported = $state(false);
+ // Voice input
+ const recorder: VoiceRecorderLike = untrack(() => voiceRecorder) ?? new VoiceRecorder();
/** Why this page can't record at all (no HTTPS, no recorder), which no tap fixes. */
let micUnavailableReason = $state<string | null>(null);
/** Why the last start failed (denied, no microphone, busy); the next tap tries again. */
@@ -189,6 +192,25 @@
}
});
+ const voice = useChatFormVoice({
+ addVoiceFile: (file) => {
+ uploadedFiles = [...uploadedFiles, file];
+ onUploadedFilesChange?.(uploadedFiles);
+ },
+ getCaretOffset: () => inputRef?.getCaretOffset() ?? value.length,
+ getValue: () => value,
+ onMicError: (message) => (micError = message),
+ recorder,
+ // on a phone placing the caret would open the keyboard
+ setCaretOffset: (offset) => {
+ if (!deviceStore.isMobile) inputRef?.setCaretOffset(offset);
+ },
+ setValue: (next) => {
+ value = next;
+ onValueChange?.(next);
+ }
+ });
+
async function handleWorkingDirectoryChange(newDir: string | null) {
// Committing a directory consumes the `/cwd` token; the chip's
// clear-X path has no token to consume.
@@ -234,6 +256,14 @@
(attachments && attachments.length > 0) || (uploadedFiles && uploadedFiles.length > 0)
);
let canSubmit = $derived(value.trim().length > 0 || hasAttachments);
+ // while a voice message records, what is being said shows in the empty message box; once
+ // it stops, the text goes with the clip, as its transcript
+ let voicePlaceholder = $derived(
+ voice.isActive && voice.liveCaption && !value.trim() ? voice.liveCaption : null
+ );
+
+ // while a reply streams, sending what is recorded waits for its first words in the form
+ let hasVoiceText = $derived(!!value.trim() || !!voice.liveCaption);
const EXPANDED_SHEET_FILL = 'max-md:bg-(--chat-form-sheet-fill)';
@@ -263,6 +293,7 @@
singleRow &&
!deviceStore.isMobile &&
!isExpanded &&
+ !voice.isActive &&
!hasAttachments &&
!mcpStore.resources.hasAttachments &&
!value.includes('\n')
@@ -327,9 +358,7 @@
);
onMount(() => {
- recordingSupported = isAudioRecordingSupported();
micUnavailableReason = microphoneUnavailableReason();
- audioRecorder = new AudioRecorder();
});
/** Expands for the message as it stands, e.g. one put back after a send failed. */
@@ -375,6 +404,10 @@
onAttachmentRemove?.(index);
}
} else {
+ const removed = uploadedFiles.find((file) => file.id === fileId);
+
+ if (removed?.voice) voiceTranscriptsStore.forget(removed.name);
+
onUploadedFileRemove?.(fileId);
}
}
@@ -399,9 +432,7 @@
event.preventDefault();
- if (!canSubmit || disabled || hasLoadingAttachments) return;
-
- onSubmit?.();
+ void submit();
}
}
@@ -586,7 +617,7 @@
inputRef?.replaceText(built.newValue, built.caretOffset);
}
- async function handleMicClick() {
+ function handleMicClick() {
// a phone shows no tooltip on a tap, so the reason comes as a toast
if (micUnavailableReason) {
toast.error(micUnavailableReason);
@@ -594,36 +625,61 @@
return;
}
- if (!audioRecorder || !recordingSupported) {
- console.warn('Audio recording not supported');
+ void voice.start();
+ }
- return;
+ // the audio set-up takes a moment, so it is made while the message is being written;
+ // it asks for no microphone, so the browser shows no recording indicator
+ function handleFocusIn(event: FocusEvent) {
+ if (voice.showButton && !voice.isActive) warmUpVoiceCapture();
+
+ // the voice message's live text shown as the placeholder turns into text to edit
+ if (voicePlaceholder && (event.target as Element | null)?.closest(VOICE_EDITOR_SELECTOR)) {
+ voice.adoptLiveText();
}
+ }
- if (isRecording) {
- isRecording = false;
- try {
- const audioBlob = await audioRecorder.stopRecording();
- const wavBlob = await convertToWav(audioBlob);
- const audioFile = createAudioFile(wavBlob);
+ // a send asked for while the dictated text is still being refined waits for it
+ let isSendWaiting = $state(false);
+
+ /** Sends the message; dictated text that is still being refined is sent refined. */
+ async function submit() {
+ if (!canSubmit || disabled || hasLoadingAttachments || isSendWaiting) return;
+
+ if (voice.isRefining) {
+ isSendWaiting = true;
- onFilesAdd?.([audioFile]);
- } catch (error) {
- console.error('Failed to stop recording:', error);
- }
- } else {
try {
- await audioRecorder.startRecording();
- isRecording = true;
- micError = null;
- } catch (error) {
- micError = describeMicrophoneError(error);
- toast.error(micError);
+ await voice.settled();
+ await tick();
+ } finally {
+ isSendWaiting = false;
}
+
+ if (!canSubmit || disabled || hasLoadingAttachments) return;
}
+
+ onSubmit?.();
+ }
+
+ /** Stops the recording and sends the message once its text or clip is in the form. */
+ async function handleStopAndSend() {
+ await voice.stop();
+ await tick();
+ await submit();
+ }
+
+ // Escape drops a recording, wherever the focus is
+ function handleWindowKeydown(event: KeyboardEvent) {
+ if (event.key !== KeyboardKey.ESCAPE || !voice.isActive || event.defaultPrevented) return;
+
+ event.preventDefault();
+ voice.cancel();
}
</script>
+<svelte:window onkeydown={handleWindowKeydown} />
+
<ChatFormInputFileInputInvisible bind:this={fileInputRef} onFileSelect={handleFileSelect} />
<form
@@ -636,12 +692,10 @@
'max-h-[calc(100dvh-var(--chat-tabs-offset,0px)-var(--chat-form-padding-top)-2rem)] max-md:max-h-dvh',
className
]}
+ onfocusin={handleFocusIn}
onsubmit={(event) => {
event.preventDefault();
-
- if (!canSubmit || disabled || hasLoadingAttachments) return;
-
- onSubmit?.();
+ void submit();
}}
>
<!-- the pickers' own triggers fill this box, so all of them share the anchor -->
@@ -753,13 +807,14 @@
{disabled}
expanded={isExpanded}
onInput={() => {
+ voice.handleUserInput();
expansion.handleInput();
pickers.handleInput();
onValueChange?.(value);
}}
onKeydown={handleKeydown}
onPaste={handlePaste}
- {placeholder}
+ placeholder={voicePlaceholder ?? placeholder}
/>
{#if mcpStore.resources.hasAttachments}
@@ -774,29 +829,55 @@
<!-- its own box, so the phone sheet can hold it by the bottom edge while it moves -->
<div bind:this={actionsEl} class={isSingleRow ? 'contents' : 'relative mt-auto'}>
- <ChatFormActions
- bind:this={chatFormActionsRef}
- canSend={canSubmit}
- class={isSingleRow ? undefined : 'px-3'}
- {disabled}
- inline={isSingleRow}
- {isLoading}
- isReasoning={chatStore.isReasoning}
- {isRecording}
- micWarning={micUnavailableReason ?? micError}
- onFileUpload={handleFileUpload}
- onMicClick={handleMicClick}
- {onStop}
- onSystemPromptClick={() =>
- onSystemPromptClick?.({ files: uploadedFiles, message: value })}
- onWorkingDirectoryClick={isWorkingDirectoryInAddMenu
- ? pickers.handleWorkingDirectoryOpen
- : undefined}
- {showAddButton}
- {showModelSelector}
- {uploadedFiles}
- workingDirectory={cwd}
- />
+ {#if voice.isActive}
+ <ChatFormActionsRecording
+ class="px-3"
+ elapsedMs={voice.elapsedMs}
+ levels={voice.levels}
+ liveCaption={voicePlaceholder ? '' : voice.liveCaption}
+ mode={voice.mode}
+ onCancel={voice.cancel}
+ onSend={isLoading && !hasVoiceText ? undefined : () => void handleStopAndSend()}
+ onStop={() => void voice.stop()}
+ quiet={voice.quiet}
+ state={voice.state}
+ />
+ {/if}
+
+ <!-- kept while recording, so its selector and menus keep their state -->
+ <div class={voice.isActive ? 'hidden' : 'contents'}>
+ <ChatFormActions
+ bind:this={chatFormActionsRef}
+ canSend={canSubmit}
+ class={isSingleRow ? undefined : 'px-3'}
+ {disabled}
+ inline={isSingleRow}
+ {isLoading}
+ isReasoning={chatStore.isReasoning}
+ {isSendWaiting}
+ isVoiceRefining={voice.isRefining}
+ micWarning={micUnavailableReason ?? micError}
+ onFileUpload={handleFileUpload}
+ onMicClick={handleMicClick}
+ {onStop}
+ onSystemPromptClick={() =>
+ onSystemPromptClick?.({ files: uploadedFiles, message: value })}
+ onWorkingDirectoryClick={isWorkingDirectoryInAddMenu
+ ? pickers.handleWorkingDirectoryOpen
+ : undefined}
+ {showAddButton}
+ {showModelSelector}
+ voice={{
+ action: voice.label,
+ hint: voice.hint,
+ prepare: () => void voice.prepare(),
+ show: voice.showButton,
+ unavailableReason: voice.unavailableReason,
+ unprepare: voice.unprepare
+ }}
+ workingDirectory={cwd}
+ />
+ </div>
</div>
</div>
</div>
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte
index 8664af6b2..33d52171d 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte
@@ -2,34 +2,95 @@
import { Button } from '@ggml-org/llama-ui/ui/button';
import * as Tooltip from '@ggml-org/llama-ui/ui/tooltip';
import { Mic, MicOff, Square } from '@lucide/svelte';
- import { ICON_CLASS_DEFAULT } from '#lib/constants/index.ts';
+ import {
+ ICON_CLASS_DEFAULT,
+ VOICE_MODE_MESSAGE,
+ VOICE_PRESS_CLICK_MS
+ } from '#lib/constants/index.ts';
+ import { MouseButton, PointerType } from '#lib/enums/index.ts';
interface Props {
+ /** What a press does: send a voice message or dictate. */
+ action?: string;
class?: string;
disabled?: boolean;
- hasAudioModality?: boolean;
+ /** Said after the action in the tooltip, e.g. that another model may be loaded. */
+ hint?: string | null;
isLoading?: boolean;
isRecording?: boolean;
/** Why the microphone can't record (no HTTPS, access denied...), shown on the button. */
micWarning?: string | null;
+ /** Why no model can take the recording; disables the button. */
+ unavailableReason?: string | null;
+ /** A quieter (tertiary) button, beside the main one: send or stop. */
+ secondary?: boolean;
+ /** With `secondary`: filled like the stop button it stands beside, not bare. */
+ filled?: boolean;
onMicClick?: () => void;
+ /** A mouse or the focus came to the button: a press may follow, the microphone can open. */
+ onPrepare?: () => void;
+ /** It left again without a press. */
+ onPrepareEnd?: () => void;
}
let {
+ action = VOICE_MODE_MESSAGE.voiceMessage,
class: className = '',
disabled = false,
- hasAudioModality = false,
+ filled = false,
+ hint = null,
isLoading = false,
isRecording = false,
micWarning = null,
- onMicClick
+ onMicClick,
+ onPrepare,
+ onPrepareEnd,
+ secondary = false,
+ unavailableReason = null
}: Props = $props();
- // a model without audio input disables the button; a microphone that can't record keeps it
+ // no model to take the recording disables the button; a microphone that can't record keeps it
// tappable, so a tap can say why
- const showWarning = $derived(hasAudioModality && !isRecording && !!micWarning);
+ const showWarning = $derived(!unavailableReason && !isRecording && !!micWarning);
+ // a mouse starts recording on press, not on release, so the first words are not lost;
+ // a touch or a key goes through the click, which is when the browser allows it
+ let pressedAt = -Infinity;
+
+ function handlePointerDown(event: PointerEvent) {
+ if (isRecording || disabled || isLoading || unavailableReason) return;
+
+ if (event.pointerType !== PointerType.MOUSE || event.button !== MouseButton.PRIMARY) return;
+
+ pressedAt = performance.now();
+ onMicClick?.();
+ }
+
+ // a touch presses at once, so only a mouse, a pen or the keyboard come ahead of the press
+ function handlePointerEnter(event: PointerEvent) {
+ if (event.pointerType !== PointerType.TOUCH && !isRecording) onPrepare?.();
+ }
+
+ function handlePointerLeave(event: PointerEvent) {
+ if (event.pointerType !== PointerType.TOUCH) onPrepareEnd?.();
+ }
+
+ function handleClick() {
+ // the click that ends the press that already started it; the recording bar may have
+ // taken the button's place meanwhile, so this one can also never come
+ if (performance.now() - pressedAt < VOICE_PRESS_CLICK_MS) {
+ pressedAt = -Infinity;
+
+ return;
+ }
+
+ onMicClick?.();
+ }
+
const tooltip = $derived(
- !hasAudioModality ? 'Current model does not support audio' : showWarning ? micWarning : null
+ isRecording
+ ? null
+ : (unavailableReason ??
+ (showWarning ? micWarning : [action, hint].filter(Boolean).join('. ')))
);
</script>
@@ -37,28 +98,41 @@
<Tooltip.Root>
<Tooltip.Trigger>
<Button
- class="size-8 max-md:size-9 rounded-full p-0 {isRecording
- ? 'matte-orb orb-danger animate-pulse text-white! hover:bg-transparent'
+ class="group size-8 max-md:size-9 rounded-full p-0 {isRecording
+ ? 'matte-orb orb-danger text-white! hover:bg-transparent'
: showWarning
? 'bg-surface-warning text-fg-warning hover:bg-surface-warning'
: ''}"
- disabled={disabled || isLoading || !hasAudioModality}
- onclick={onMicClick}
+ disabled={disabled || isLoading || !!unavailableReason}
+ onblur={() => onPrepareEnd?.()}
+ onclick={handleClick}
+ onfocus={() => !isRecording && onPrepare?.()}
+ onpointerdown={handlePointerDown}
+ onpointerenter={handlePointerEnter}
+ onpointerleave={handlePointerLeave}
type="button"
- variant={isRecording || showWarning ? 'ghost' : 'default'}
+ variant={isRecording || showWarning
+ ? 'ghost'
+ : secondary
+ ? filled
+ ? 'secondary'
+ : 'tertiary'
+ : 'default'}
>
<span class="sr-only"
>{isRecording
? 'Stop recording'
: showWarning
? `Recording unavailable: ${micWarning}`
- : 'Start recording'}</span
+ : action}</span
>
{#if isRecording}
- <Square class="{ICON_CLASS_DEFAULT} animate-pulse fill-white" />
+ <Square class="{ICON_CLASS_DEFAULT} fill-white" />
{:else if showWarning}
<MicOff class={ICON_CLASS_DEFAULT} />
+ {:else if secondary}
+ <Mic class="{ICON_CLASS_DEFAULT} stroke-muted-foreground group-hover:stroke-foreground" />
{:else}
<Mic class={ICON_CLASS_DEFAULT} />
{/if}
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionSubmit.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionSubmit.svelte
index bf2ae23df..fa26d6bbc 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionSubmit.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionSubmit.svelte
@@ -1,16 +1,24 @@
<script lang="ts">
import { Button } from '@ggml-org/llama-ui/ui/button';
import * as Tooltip from '@ggml-org/llama-ui/ui/tooltip';
- import { ArrowUp } from '@lucide/svelte';
+ import { ArrowUp, LoaderCircle } from '@lucide/svelte';
interface Props {
canSend?: boolean;
disabled?: boolean;
+ /** The send is on its way: it waits for something, e.g. dictated text being refined. */
+ loading?: boolean;
showErrorState?: boolean;
tooltipLabel?: string;
}
- let { canSend = false, disabled = false, showErrorState = false, tooltipLabel }: Props = $props();
+ let {
+ canSend = false,
+ disabled = false,
+ loading = false,
+ showErrorState = false,
+ tooltipLabel
+ }: Props = $props();
let isDisabled = $derived(!canSend || disabled);
</script>
@@ -28,7 +36,11 @@
>
<span class="sr-only">Send</span>
- <ArrowUp class="size-icon-md" />
+ {#if loading}
+ <LoaderCircle class="size-icon-md animate-spin" />
+ {:else}
+ <ArrowUp class="size-icon-md" />
+ {/if}
</Button>
{/snippet}
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActions.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActions.svelte
index d71a8da5d..c895ef0c3 100644
--- a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActions.svelte
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActions.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { Button } from '@ggml-org/llama-ui/ui/button';
- import { SkipForward, Square } from '@lucide/svelte';
+ import { LoaderCircle, SkipForward, Square } from '@lucide/svelte';
import {
ChatFormActionRecord,
ChatFormActionsAdd,
@@ -8,14 +8,17 @@
ChatFormContextGauge,
ModelsSelector
} from '#lib/components/app/index.ts';
- import { ICON_CLASS_DEFAULT, SET_WORKING_DIRECTORY_LABEL } from '#lib/constants/index.ts';
+ import {
+ ICON_CLASS_DEFAULT,
+ SET_WORKING_DIRECTORY_LABEL,
+ VOICE_MODE_MESSAGE
+ } from '#lib/constants/index.ts';
import { setChatFormActionsContext } from '#lib/contexts/index.ts';
- import { FileTypeCategory } from '#lib/enums/index.ts';
import { useChatFormModel } from '#lib/hooks/use-chat-form-model.svelte.ts';
import { useShowContextGauge } from '#lib/hooks/use-context-gauge.svelte.ts';
import { ChatService } from '#lib/services/index.ts';
import { conversationsStore, deviceStore, settingsStore, toolsStore } from '#lib/stores/index.ts';
- import { abbreviateWorkingDir, getFileTypeCategory } from '#lib/utils/index.ts';
+ import { abbreviateWorkingDir } from '#lib/utils/index.ts';
interface Props {
canSend?: boolean;
@@ -25,12 +28,25 @@
inline?: boolean;
isLoading?: boolean;
isReasoning?: boolean;
- isRecording?: boolean;
+ /** A send was asked for and waits for the dictated text to be refined. */
+ isSendWaiting?: boolean;
+ /** The dictated text in the message is still being refined. */
+ isVoiceRefining?: boolean;
/** Why the microphone can't record, shown on its button. */
micWarning?: string | null;
+ /** Voice input: whether the microphone shows, what it does, and why it can't. */
+ voice?: {
+ action: string;
+ hint: string | null;
+ show: boolean;
+ unavailableReason: string | null;
+ /** The pointer or the focus came to the microphone button, a press may follow */
+ prepare?: () => void;
+ /** It left without a press */
+ unprepare?: () => void;
+ };
showAddButton?: boolean;
showModelSelector?: boolean;
- uploadedFiles?: ChatUploadedFile[];
/** The chat's working directory, which the + menu names. */
workingDirectory?: string | null;
onFileUpload?: () => void;
@@ -47,7 +63,8 @@
inline = false,
isLoading = false,
isReasoning = false,
- isRecording = false,
+ isSendWaiting = false,
+ isVoiceRefining = false,
micWarning = null,
onFileUpload,
onMicClick,
@@ -56,7 +73,7 @@
onWorkingDirectoryClick,
showAddButton = true,
showModelSelector = true,
- uploadedFiles = [],
+ voice,
workingDirectory = null
}: Props = $props();
@@ -77,12 +94,17 @@
return '';
});
- let hasAudioAttachments = $derived(
- uploadedFiles.some((file) => getFileTypeCategory(file.type) === FileTypeCategory.AUDIO)
- );
- let shouldShowRecordButton = $derived(
- hasAudioModality && !canSend && !hasAudioAttachments && currentConfig.autoMicOnEmpty
- );
+ // the microphone sits beside the send or stop button: on an empty message when the setting
+ // asks for it (disabled, saying why, when no model can take a recording), and while a reply
+ // streams or the message has text or a recording, so more can be said without clearing it
+ let showStop = $derived(isLoading && !canSend);
+ let showRecord = $derived.by(() => {
+ if (!voice?.show) return false;
+
+ if (showStop || canSend) return !voice.unavailableReason;
+
+ return currentConfig.autoMicOnEmpty;
+ });
let selectorModelRef: ModelsSelector | undefined = $state(undefined);
@@ -159,6 +181,18 @@
{/if}
</div>
+ {#if isVoiceRefining}
+ <span
+ class="flex items-center gap-1.5 text-caption whitespace-nowrap text-muted-foreground"
+ data-slot="voice-refining"
+ role="status"
+ >
+ <LoaderCircle class="size-icon-sm animate-spin" />
+
+ {isSendWaiting ? VOICE_MODE_MESSAGE.sendAfterRefining : VOICE_MODE_MESSAGE.refining}
+ </span>
+ {/if}
+
{#if isReasoning && !canSend}
<Button
class="group size-8 rounded-full p-0 max-md:size-9"
@@ -176,7 +210,26 @@
</Button>
{/if}
- {#if isLoading && !canSend}
+ {#snippet record(filled: boolean)}
+ <ChatFormActionRecord
+ action={voice?.action}
+ {disabled}
+ {filled}
+ hint={voice?.hint}
+ {micWarning}
+ {onMicClick}
+ onPrepare={voice?.prepare}
+ onPrepareEnd={voice?.unprepare}
+ secondary
+ unavailableReason={voice?.unavailableReason}
+ />
+ {/snippet}
+
+ {#if showRecord && !showStop}
+ {@render record(false)}
+ {/if}
+
+ {#if showStop}
<Button
class="group size-8 rounded-full p-0 max-md:size-9 hover:bg-destructive/10!"
onclick={onStop}
@@ -189,21 +242,18 @@
class="size-icon-md fill-muted-foreground stroke-muted-foreground group-hover:fill-destructive group-hover:stroke-destructive hover:fill-destructive hover:stroke-destructive"
/>
</Button>
- {:else if shouldShowRecordButton}
- <ChatFormActionRecord
- {disabled}
- {hasAudioModality}
- {isLoading}
- {isRecording}
- {micWarning}
- {onMicClick}
- />
{:else}
<ChatFormActionSubmit
canSend={canSend && (showModelSelector ? hasModelSelected && isSelectedModelInCache : true)}
{disabled}
+ loading={isSendWaiting}
showErrorState={showModelSelector && hasModelSelected && !isSelectedModelInCache}
tooltipLabel={submitTooltip}
/>
{/if}
+
+ <!-- while a reply streams, the microphone is the last button, filled like stop -->
+ {#if showRecord && showStop}
+ {@render record(true)}
+ {/if}
</div>
diff --git a/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionsRecording.svelte b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionsRecording.svelte
new file mode 100644
index 000000000..5e3d2bbba
--- /dev/null
+++ b/ui/apps/chat/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionsRecording.svelte
@@ -0,0 +1,63 @@
+<!--
+ The chat form's actions row while it records: Llama UI's VoiceRecordingBar with the form's labels and its
+ microphone button as the stop button. A voice message's text, when the model streams it, runs under the waveform;
+ dictated text goes into the message instead. After a dictation that streamed nothing, it waits for the text in
+ place of the waveform. Speech that stays too quiet even boosted is said inside the row, with the advice in a tooltip above it, so the form does not jump.
+-->
+<script lang="ts">
+ import { VoiceRecordingBar } from '@ggml-org/llama-ui/voice';
+ import { ChatFormActionRecord } from '#lib/components/app/index.ts';
+ import { VOICE_MODE_MESSAGE, VOICE_RECORDING_BAR_STATUS } from '#lib/constants/index.ts';
+ import { type ChatFormVoiceState, VoiceMode } from '#lib/enums/index.ts';
+
+ interface Props {
+ class?: string;
+ elapsedMs: number;
+ levels: number[];
+ /** Text streaming while a voice message is recorded. */
+ liveCaption?: string;
+ mode: VoiceMode;
+ /** The speech is too quiet even at the highest gain: the row says so, without changing its height. */
+ quiet?: boolean;
+ state: ChatFormVoiceState;
+ onCancel: () => void;
+ onStop: () => void;
+ /** Stops and sends; left out, there is no send button. */
+ onSend?: () => void;
+ }
+
+ let {
+ class: className = '',
+ elapsedMs,
+ levels,
+ liveCaption = '',
+ mode,
+ onCancel,
+ onSend,
+ onStop,
+ quiet = false,
+ state
+ }: Props = $props();
+
+ const LABELS = {
+ starting: VOICE_MODE_MESSAGE.starting,
+ waiting: VOICE_MODE_MESSAGE.transcribing
+ };
+</script>
+
+<VoiceRecordingBar
+ caption={liveCaption}
+ class={className}
+ {elapsedMs}
+ label={mode === VoiceMode.DICTATION ? VOICE_MODE_MESSAGE.dictating : VOICE_MODE_MESSAGE.recording}
+ labels={LABELS}
+ {levels}
+ {onCancel}
+ {onSend}
+ status={VOICE_RECORDING_BAR_STATUS[state]}
+ warning={quiet ? VOICE_MODE_MESSAGE.tooQuiet : null}
+>
+ {#snippet stop({ disabled })}
+ <ChatFormActionRecord class="shrink-0" {disabled} isRecording onMicClick={onStop} />
+ {/snippet}
+</VoiceRecordingBar>
diff --git a/ui/apps/chat/src/lib/components/app/chat/index.ts b/ui/apps/chat/src/lib/components/app/chat/index.ts
index fce3962c4..f7626c73a 100644
--- a/ui/apps/chat/src/lib/components/app/chat/index.ts
+++ b/ui/apps/chat/src/lib/components/app/chat/index.ts
@@ -79,6 +79,14 @@ export { default as ChatAttachmentsListItemMcpResource } from './ChatAttachments
*/
export { default as ChatAttachmentsListItemThumbnailFile } from './ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItemThumbnailFile.svelte';
+/**
+ * An audio attachment drawn as a clip: a playable waveform (peaks computed from
+ * the audio when it was stored without them) and, for a voice message, its
+ * transcript under it on hover, focus or a tap, with the transcribing and failed
+ * states. `sm` in the chat form, removable; `md` in a sent message.
+ */
+export { default as ChatAttachmentsListItemVoice } from './ChatAttachments/ChatAttachmentsList/ChatAttachmentsListItem/ChatAttachmentsListItemVoice.svelte';
+
/**
* Thumbnail for image attachments with lazy loading and error fallback.
* Displays image preview with configurable dimensions. Falls back to placeholder
@@ -170,12 +178,20 @@ export { default as ChatForm } from './ChatForm/ChatForm.svelte';
export { default as ChatFormActionsAdd } from './ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionsAdd.svelte';
/**
- * Audio recording button with real-time recording indicator. Records audio
- * and converts to WAV format for upload. Only visible when the active model
- * supports audio modality and setting for automatic audio input is enabled. Shows recording duration while active.
+ * Microphone button: starts a voice message or a dictation (its label says which),
+ * or, disabled, says why no model can take a recording; while recording it is the
+ * stop button. Shown on an empty message when voice input is available and the
+ * setting for the microphone on empty input is on.
*/
export { default as ChatFormActionRecord } from './ChatForm/ChatFormActions/ChatFormActionRecord.svelte';
+/**
+ * The actions row while the chat form records: cancel, the live waveform and the
+ * elapsed time, and stop, with a voice message's streaming text under the waveform
+ * and a waiting state while dictated text is transcribed after recording.
+ */
+export { default as ChatFormActionsRecording } from './ChatForm/ChatFormActions/ChatFormActionsRecording.svelte';
+
/**
* Container for chat form action buttons. Arranges file attachment, audio record,
* and submit/stop buttons in a horizontal layout. Handles conditional visibility
diff --git a/ui/apps/chat/src/lib/components/app/models/ModelRowActions.svelte b/ui/apps/chat/src/lib/components/app/models/ModelRowActions.svelte
index 33be4c624..071b71000 100644
--- a/ui/apps/chat/src/lib/components/app/models/ModelRowActions.svelte
+++ b/ui/apps/chat/src/lib/components/app/models/ModelRowActions.svelte
@@ -1,7 +1,8 @@
<script lang="ts">
- import { Heart, HeartOff, Info } from '@lucide/svelte';
+ import { Heart, HeartOff, Info, Mic } from '@lucide/svelte';
import { ActionIcon } from '#lib/components/app/index.ts';
- import { modelsStore, uiStore } from '#lib/stores/index.ts';
+ import { VOICE_DICTATION_MODEL_LABEL } from '#lib/constants/index.ts';
+ import { modelsStore, serverStore, uiStore, voiceStore } from '#lib/stores/index.ts';
import type { ModelOption } from '#lib/types/models.js';
interface Props {
@@ -13,7 +14,19 @@
let { isFav, option, revealOnHover = true }: Props = $props();
- // the favorite heart stays visible, so the reveal applies to the other icons
+ // the favorite heart and the dictation model's microphone stay visible, so the reveal
+ // applies to the other icons
+ // any router model that hears can be the one that transcribes dictation
+ let canDictate = $derived(serverStore.isRouterMode && voiceStore.roleOf(option.model) !== null);
+ let dictates = $derived(voiceStore.isDictationModel(option.model));
+ let dictationLabel = $derived.by(() => {
+ if (!dictates) return VOICE_DICTATION_MODEL_LABEL.use;
+
+ return voiceStore.dictationIsAutomatic
+ ? VOICE_DICTATION_MODEL_LABEL.automatic
+ : VOICE_DICTATION_MODEL_LABEL.chosen;
+ });
+
const revealClass =
'pointer-events-none opacity-0 group-hover:pointer-events-auto group-hover:opacity-100 [@media(pointer:coarse)]:pointer-events-auto [@media(pointer:coarse)]:opacity-100';
</script>
@@ -35,6 +48,26 @@
/>
</span>
+ {#if canDictate}
+ <!-- like the heart: lit on the model that dictates, among the hover actions on the others -->
+ <span
+ class={!dictates && revealOnHover ? revealClass : ''}
+ data-dictates={dictates || undefined}
+ data-slot="model-dictation-action"
+ >
+ <ActionIcon
+ class={dictates
+ ? 'size-icon-lg text-destructive hover:text-destructive'
+ : 'size-icon-lg hover:text-foreground'}
+ icon={Mic}
+ iconSize="size-icon-md"
+ onclick={() => voiceStore.toggleDictation(option.model)}
+ tooltip={dictationLabel}
+ tooltipAsTitle
+ />
+ </span>
+ {/if}
+
{#if isFav}
<span class="flex size-5 items-center justify-center">
<span class="flex group-hover:hidden [@media(pointer:coarse)]:hidden">
diff --git a/ui/apps/chat/src/lib/components/app/models/ModelVoiceRole.svelte b/ui/apps/chat/src/lib/components/app/models/ModelVoiceRole.svelte
new file mode 100644
index 000000000..cabc88139
--- /dev/null
+++ b/ui/apps/chat/src/lib/components/app/models/ModelVoiceRole.svelte
@@ -0,0 +1,60 @@
+<!--
+ What kind of model with audio input a model is, as a small badge beside its name in the model picker and the Models
+ manager: a speech-to-text model that is for dictation rather than chat, or a chat model that hears and gets voice
+ messages. A model that cannot hear shows nothing. The tooltip says what the kind means. Which model transcribes
+ dictation is shown by the microphone among the row's actions instead (ModelRowActions).
+-->
+<script lang="ts">
+ import * as Tooltip from '@ggml-org/llama-ui/ui/tooltip';
+ import { AudioLines, Captions } from '@lucide/svelte';
+ import { VOICE_MODEL_ROLE_DESCRIPTION, VOICE_MODEL_ROLE_LABEL } from '#lib/constants/index.ts';
+ import { VoiceModelRole } from '#lib/enums/index.ts';
+ import { voiceStore } from '#lib/stores/index.ts';
+
+ interface Props {
+ model: string;
+ class?: string;
+ }
+
+ let { class: className = '', model }: Props = $props();
+
+ const ICONS = {
+ [VoiceModelRole.TRANSCRIPTION]: Captions,
+ [VoiceModelRole.VOICE]: AudioLines
+ };
+
+ let role = $derived(voiceStore.roleOf(model));
+ let description = $derived(
+ role === VoiceModelRole.TRANSCRIPTION
+ ? VOICE_MODEL_ROLE_DESCRIPTION.transcription
+ : VOICE_MODEL_ROLE_DESCRIPTION.voice
+ );
+</script>
+
+{#if role}
+ {@const Icon = ICONS[role]}
+
+ <Tooltip.Root>
+ <Tooltip.Trigger>
+ {#snippet child({ props })}
+ <span
+ {...props}
+ class={[
+ 'inline-flex h-4 shrink-0 items-center gap-0.5 rounded-md border border-border/60 px-1 text-caption-sm leading-none font-medium whitespace-nowrap text-muted-foreground',
+ className
+ ]}
+ data-role={role}
+ data-slot="model-voice-role"
+ >
+ <Icon aria-hidden="true" class="size-2.5" />
+
+ {VOICE_MODEL_ROLE_LABEL[role]}
+ </span>
+ {/snippet}
+ </Tooltip.Trigger>
+
+ <Tooltip.Content class="max-w-64">
+ <p>{description}</p>
+ </Tooltip.Content>
+ </Tooltip.Root>
+{/if}
diff --git a/ui/apps/chat/src/lib/components/app/models/ModelsManager/ModelsManagerModelRow.svelte b/ui/apps/chat/src/lib/components/app/models/ModelsManager/ModelsManagerModelRow.svelte
index 2c25dddd1..9f8913451 100644
--- a/ui/apps/chat/src/lib/components/app/models/ModelsManager/ModelsManagerModelRow.svelte
+++ b/ui/apps/chat/src/lib/components/app/models/ModelsManager/ModelsManagerModelRow.svelte
@@ -4,15 +4,20 @@
import ModelContext from '../ModelContext.svelte';
import ModelDraftSubline from '../ModelDraftSubline.svelte';
import ModelId from '../ModelId.svelte';
+ import ModelVoiceRole from '../ModelVoiceRole.svelte';
import ModelsManagerStatusCell from './ModelsManagerStatusCell.svelte';
import { modelRowActions } from './row-actions';
import { configuredContext, downloadProgressFor, isRowControlClick } from './utils';
import { ModelDownloadProgressBar } from '@ggml-org/llama-ui/models';
- import { Heart, MoreHorizontal } from '@lucide/svelte';
+ import { Heart, Mic, MoreHorizontal } from '@lucide/svelte';
import { ContextMenuActions, DropdownMenuActions } from '#lib/components/app/index.ts';
- import { MODEL_ROW_GRID_CLASS, MODEL_ROW_TRAILING_CELL_CLASS } from '#lib/constants/index.ts';
+ import {
+ MODEL_ROW_GRID_CLASS,
+ MODEL_ROW_TRAILING_CELL_CLASS,
+ VOICE_DICTATION_MODEL_LABEL
+ } from '#lib/constants/index.ts';
import { ModelRowDownloadState } from '#lib/enums/index.ts';
- import { deviceStore, modelsStore } from '#lib/stores/index.ts';
+ import { deviceStore, modelsStore, voiceStore } from '#lib/stores/index.ts';
import type { ModelOption, ModelSidecarBadge } from '#lib/types/models.js';
interface Props {
@@ -38,6 +43,7 @@
}: Props = $props();
let favorite = $derived(isFavorite(option));
+ let dictates = $derived(voiceStore.isDictationModel(option.model));
let isHidden = $derived(modelsStore.isHidden(option.id));
// live while the download runs, frozen while it is paused
let downloadProgress = $derived(downloadProgressFor(option.model));
@@ -101,9 +107,18 @@
title={option.model}
/>
+ <ModelVoiceRole model={option.model} />
+
<!-- a phone has no width for the modality icons, the id needs it more -->
<ModelCapabilities hideModalities={deviceStore.isMobile} {option} />
+ {#if dictates}
+ <!-- decorative like the heart: the text carries the state -->
+ <Mic aria-hidden="true" class="size-icon-sm shrink-0 text-destructive" />
+
+ <span class="sr-only">{VOICE_DICTATION_MODEL_LABEL.current}</span>
+ {/if}
+
{#if favorite}
<!-- the heart is decorative, the button's text carries the state -->
<Heart aria-hidden="true" class="size-icon-sm shrink-0 text-rose-500" />
diff --git a/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorList.svelte b/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorList.svelte
index 2dfe70949..d3e23c0a0 100644
--- a/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorList.svelte
+++ b/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorList.svelte
@@ -16,7 +16,7 @@
SETTINGS_KEYS
} from '#lib/constants/index.ts';
import { ModelDownloadConfirmAction } from '#lib/enums/index.ts';
- import { modelsStore, settingsStore } from '#lib/stores/index.ts';
+ import { modelsStore, settingsStore, voiceStore } from '#lib/stores/index.ts';
import {
type GroupedModelOptions,
groupModelFamilies,
@@ -53,6 +53,12 @@
showOrgName = true
}: Props = $props();
let render = $derived(renderOption ?? defaultOption);
+ // a loaded model that only transcribes dictation is listed apart from the chat models;
+ // the conversation's own model stays with them, even when it also dictates
+ const isDictationOnly = (item: ModelItem) =>
+ item.option.model === voiceStore.dictationModel && item.option.model !== currentModel;
+ let loadedForDictation = $derived(loaded.filter(isDictationOnly));
+ let loadedForChat = $derived(loaded.filter((item) => !isDictationOnly(item)));
// a large catalog mounts a window at a time: the sentinel at the end of the list
// grows the window when it scrolls into view
let visibleCount = $state(MODEL_ROW_WINDOW);
@@ -180,9 +186,25 @@
<Power class="size-icon-sm shrink-0" />
{/snippet}
- {#each loaded as item (item.option.id)}
+ {#each loadedForChat as item (item.option.id)}
{@render render(item, !showOrgName)}
{/each}
+
+ {#if loadedForDictation.length > 0}
+ <!-- the model that transcribes dictation, apart from the ones to chat with -->
+ {#if loadedForChat.length > 0}
+ <p
+ class="m-0 px-2 pt-2 pb-1 text-label-sm text-muted-foreground/70 select-none"
+ data-slot="models-loaded-dictation-heading"
+ >
+ For dictation
+ </p>
+ {/if}
+
+ {#each loadedForDictation as item (item.option.id)}
+ {@render render(item, !showOrgName)}
+ {/each}
+ {/if}
</ModelsSection>
{/if}
diff --git a/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorOption.svelte b/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorOption.svelte
index 09ddcf83c..94ec7ada8 100644
--- a/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorOption.svelte
+++ b/ui/apps/chat/src/lib/components/app/models/ModelsSelector/ModelsSelectorOption.svelte
@@ -5,7 +5,8 @@
ModelCapabilities,
ModelId,
ModelLoadControl,
- ModelRowActions
+ ModelRowActions,
+ ModelVoiceRole
} from '#lib/components/app/index.ts';
import { SETTINGS_KEYS } from '#lib/constants/index.ts';
import { ServerModelStatus } from '#lib/enums/index.ts';
@@ -89,6 +90,8 @@
title={option.model}
/>
+ <ModelVoiceRole model={option.model} />
+
{#if showCapabilities}
<ModelCapabilities {option} />
{/if}
diff --git a/ui/apps/chat/src/lib/components/app/models/index.ts b/ui/apps/chat/src/lib/components/app/models/index.ts
index 19f043b6b..d379f3d19 100644
--- a/ui/apps/chat/src/lib/components/app/models/index.ts
+++ b/ui/apps/chat/src/lib/components/app/models/index.ts
@@ -53,6 +53,8 @@ export { default as ModelLoadControl } from './ModelLoadControl.svelte';
export { default as ModelOrgAvatar } from './ModelOrgAvatar.svelte';
export { default as ModelRowActions } from './ModelRowActions.svelte';
export { default as ModelsSection } from './ModelsSection.svelte';
+/** **ModelVoiceRole** - a model's part in voice input (dictation, speech to text, voice), as a badge. */
+export { default as ModelVoiceRole } from './ModelVoiceRole.svelte';
/** Capability and modality icon row, shared by every model-id surface. */
export { ModelCapabilityIcons } from '@ggml-org/llama-ui/models';
diff --git a/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChat.svelte b/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChat.svelte
index f18c79c6d..d54d56a20 100644
--- a/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChat.svelte
+++ b/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChat.svelte
@@ -7,6 +7,7 @@
SettingsChatImportExportTab,
SettingsChatMobileHeader,
SettingsChatToolsTab,
+ SettingsChatVoiceTab,
SettingsFooter,
SettingsResetButton
} from '#lib/components/app/settings/index.ts';
@@ -185,6 +186,12 @@
<SettingsChatToolsTab />
{:else if currentSection.slug === SETTINGS_SECTION_SLUGS.IMPORT_EXPORT}
<SettingsChatImportExportTab />
+ {:else if currentSection.slug === SETTINGS_SECTION_SLUGS.VOICE && currentSection.fields}
+ <SettingsChatVoiceTab
+ fields={currentSection.fields}
+ {localConfig}
+ onConfigChange={handleConfigChange}
+ />
{:else if currentSection.fields}
<div class="space-y-6">
<SettingsChatFields
diff --git a/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChatVoiceTab.svelte b/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChatVoiceTab.svelte
new file mode 100644
index 000000000..c8a6eac2c
--- /dev/null
+++ b/ui/apps/chat/src/lib/components/app/settings/SettingsChat/SettingsChatVoiceTab.svelte
@@ -0,0 +1,226 @@
+<!--
+ Voice section of the settings: the registry's fields, with the microphone and the transcription model between them.
+ Those list this device's microphones and the server's models with audio input, which a static field cannot, so they
+ are drawn here.
+-->
+<script lang="ts">
+ import { Label } from '@ggml-org/llama-ui/ui/label';
+ import * as Select from '@ggml-org/llama-ui/ui/select';
+ import {
+ listMicrophones,
+ type Microphone,
+ VOICE_MICROPHONE_AUTO,
+ VOICE_MICROPHONE_DEFAULT
+ } from '@ggml-org/llama-voice';
+ import { SettingsChatFields } from '#lib/components/app/settings/index.ts';
+ import {
+ SETTING_CONFIG_INFO,
+ SETTINGS_KEYS,
+ VOICE_GGUF_EXTENSION_PATTERN,
+ VOICE_SETTINGS_LABEL,
+ VOICE_TRANSCRIPTION_MODEL_AUTO
+ } from '#lib/constants/index.ts';
+ import { modelsStore, serverStore } from '#lib/stores/index.ts';
+ import { isTranscriptionOnlyModel } from '#lib/utils/voice-mode.ts';
+ import { onMount } from 'svelte';
+
+ interface Props {
+ fields: SettingsFieldConfig[];
+ localConfig: SettingsConfigType;
+ onConfigChange: (key: string, value: string | boolean) => void;
+ }
+
+ let { fields, localConfig, onConfigChange }: Props = $props();
+
+ const KEY = SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL;
+
+ // the picker sits right after the Voice input field
+ let splitAt = $derived(fields.findIndex((f) => f.key === SETTINGS_KEYS.VOICE_INPUT_MODE) + 1);
+ let isRouter = $derived(serverStore.isRouterMode);
+ let current = $derived(String(localConfig[KEY] ?? VOICE_TRANSCRIPTION_MODEL_AUTO));
+
+ let audioModels = $derived.by(() => {
+ void modelsStore.props.cacheVersion;
+
+ return modelsStore
+ .getVisibleModels()
+ .filter((option) => modelsStore.props.modelSupportsAudio(option.model));
+ });
+
+ let options = $derived.by(() => {
+ const list = audioModels.map((option) => ({ label: option.name, value: option.model }));
+
+ // a model picked earlier that the server no longer lists stays visible as the choice
+ if (current && !list.some((option) => option.value === current)) {
+ list.push({ label: current, value: current });
+ }
+
+ return [
+ { label: VOICE_SETTINGS_LABEL.automatic, value: VOICE_TRANSCRIPTION_MODEL_AUTO },
+ ...list
+ ];
+ });
+
+ // speech-to-text models first: they are made for dictation, the chat models that hear can do it too
+ let optionGroups = $derived.by(() => {
+ const models = options.filter((option) => option.value !== VOICE_TRANSCRIPTION_MODEL_AUTO);
+
+ return [
+ {
+ heading: VOICE_SETTINGS_LABEL.speechToText,
+ items: models.filter((option) => isTranscriptionOnlyModel(option.value))
+ },
+ {
+ heading: VOICE_SETTINGS_LABEL.chatModels,
+ items: models.filter((option) => !isTranscriptionOnlyModel(option.value))
+ }
+ ].filter((group) => group.items.length > 0);
+ });
+
+ let selectedLabel = $derived(
+ options.find((option) => option.value === current)?.label ?? VOICE_SETTINGS_LABEL.automatic
+ );
+
+ const MIC_KEY = SETTINGS_KEYS.VOICE_MICROPHONE;
+
+ let microphones = $state<Microphone[]>([]);
+ let micCurrent = $derived(String(localConfig[MIC_KEY] ?? VOICE_MICROPHONE_AUTO));
+ // the browser names its microphones only once the page may use one
+ let micNamesHidden = $derived(microphones.some((mic) => !mic.label));
+ let micOptions = $derived.by(() => {
+ const list = microphones.map((mic, index) => ({
+ label: mic.label || VOICE_SETTINGS_LABEL.microphone(index + 1),
+ value: mic.deviceId
+ }));
+ const fixed = [VOICE_MICROPHONE_AUTO, VOICE_MICROPHONE_DEFAULT];
+
+ // a microphone picked earlier that is not plugged in now stays visible as the choice
+ if (!fixed.includes(micCurrent) && !list.some((option) => option.value === micCurrent)) {
+ list.push({ label: VOICE_SETTINGS_LABEL.disconnectedMicrophone, value: micCurrent });
+ }
+
+ return [
+ { label: VOICE_SETTINGS_LABEL.automatic, value: VOICE_MICROPHONE_AUTO },
+ { label: VOICE_SETTINGS_LABEL.systemDefault, value: VOICE_MICROPHONE_DEFAULT },
+ ...list.filter((option) => option.value)
+ ];
+ });
+ let micLabel = $derived(
+ micOptions.find((option) => option.value === micCurrent)?.label ??
+ VOICE_SETTINGS_LABEL.automatic
+ );
+
+ onMount(() => {
+ const devices = navigator.mediaDevices;
+ const refresh = () => {
+ listMicrophones().then(
+ (list) => (microphones = list),
+ () => (microphones = [])
+ );
+ };
+
+ refresh();
+ devices?.addEventListener('devicechange', refresh);
+
+ return () => devices?.removeEventListener('devicechange', refresh);
+ });
+
+ let singleModelId = $derived(modelsStore.activeModelId);
+ // the file's name, not the GGUF path a single-model server reports
+ let singleModel = $derived(
+ singleModelId
+ ? (singleModelId.split('/').pop() ?? singleModelId).replace(VOICE_GGUF_EXTENSION_PATTERN, '')
+ : null
+ );
+ let singleModelHasAudio = $derived(
+ singleModelId ? modelsStore.props.modelSupportsAudio(singleModelId) : false
+ );
+</script>
+
+<div class="space-y-6">
+ <SettingsChatFields fields={fields.slice(0, splitAt)} {localConfig} {onConfigChange} />
+
+ <div class="space-y-2" data-slot="voice-microphone">
+ <Label class="text-label" for={MIC_KEY}>Microphone</Label>
+
+ <Select.Root
+ onValueChange={(value) => onConfigChange(MIC_KEY, value)}
+ type="single"
+ value={micCurrent}
+ >
+ <Select.Trigger class="w-full" id={MIC_KEY}>{micLabel}</Select.Trigger>
+
+ <Select.Content>
+ {#each micOptions as option (option.value)}
+ <Select.Item label={option.label} value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+
+ <p class="mt-1 text-caption text-muted-foreground">{SETTING_CONFIG_INFO[MIC_KEY]}</p>
+
+ {#if micNamesHidden}
+ <p class="text-caption text-muted-foreground">
+ The browser names your microphones once you have allowed one, e.g. after your first
+ recording.
+ </p>
+ {/if}
+ </div>
+
+ <div class="space-y-2" data-slot="voice-transcription-model">
+ <Label class="text-label" for={KEY}>Transcription model</Label>
+
+ {#if isRouter}
+ <Select.Root
+ onValueChange={(value) => onConfigChange(KEY, value)}
+ type="single"
+ value={current}
+ >
+ <Select.Trigger class="w-full" id={KEY}>{selectedLabel}</Select.Trigger>
+
+ <Select.Content>
+ <Select.Item
+ label={VOICE_SETTINGS_LABEL.automatic}
+ value={VOICE_TRANSCRIPTION_MODEL_AUTO}
+ >
+ {VOICE_SETTINGS_LABEL.automatic}
+ </Select.Item>
+
+ {#each optionGroups as group (group.heading)}
+ <Select.Group>
+ <Select.GroupHeading>{group.heading}</Select.GroupHeading>
+
+ {#each group.items as option (option.value)}
+ <Select.Item label={option.label} value={option.value}>{option.label}</Select.Item>
+ {/each}
+ </Select.Group>
+ {/each}
+ </Select.Content>
+ </Select.Root>
+
+ <p class="mt-1 text-caption text-muted-foreground">
+ {SETTING_CONFIG_INFO[KEY]} Only models with audio input are listed.
+ </p>
+
+ <p class="text-caption text-muted-foreground">
+ A model you pick here is loaded on the server when you dictate, alongside the chat model, if
+ it isn't loaded already. A server that keeps one model at a time (<code>--models-max 1</code
+ >) unloads the chat model to do it.
+ </p>
+ {:else}
+ <p class="text-body-sm text-muted-foreground">
+ {#if singleModel}
+ This server runs one model, <span class="font-medium text-foreground">{singleModel}</span
+ >.
+ {singleModelHasAudio
+ ? 'It has audio input, so it transcribes your voice.'
+ : 'It has no audio input, so voice input is not available on this server.'}
+ {:else}
+ This server runs one model, which transcribes your voice when it has audio input.
+ {/if}
+ </p>
+ {/if}
+ </div>
+
+ <SettingsChatFields fields={fields.slice(splitAt)} {localConfig} {onConfigChange} />
+</div>
diff --git a/ui/apps/chat/src/lib/components/app/settings/index.ts b/ui/apps/chat/src/lib/components/app/settings/index.ts
index 2d2aca5af..2d44c8892 100644
--- a/ui/apps/chat/src/lib/components/app/settings/index.ts
+++ b/ui/apps/chat/src/lib/components/app/settings/index.ts
@@ -52,6 +52,15 @@ export { default as SettingsResetButton } from './SettingsResetButton.svelte';
*/
export { default as SettingsChatImportExportTab } from './SettingsChat/SettingsChatImportExportTab.svelte';
+/**
+ * **SettingsChatVoiceTab** - Voice section of the settings
+ *
+ * The registry's voice fields, with the transcription model picker between them:
+ * in router mode it lists the server's models with audio input, on a single-model
+ * server it says whether that model can transcribe.
+ */
+export { default as SettingsChatVoiceTab } from './SettingsChat/SettingsChatVoiceTab.svelte';
+
/**
* Section wrapper for import/export sections. Displays a title, description,
* icon button, and optional summary of recent actions.
diff --git a/ui/apps/chat/src/lib/constants/icons.constants.ts b/ui/apps/chat/src/lib/constants/icons.constants.ts
index ff2918ee2..c096e9a69 100644
--- a/ui/apps/chat/src/lib/constants/icons.constants.ts
+++ b/ui/apps/chat/src/lib/constants/icons.constants.ts
@@ -8,12 +8,13 @@ import LoadFailedIcon from '@lucide/svelte/icons/circle-alert';
import DiscoverIcon from '@lucide/svelte/icons/compass';
import UnloadIcon from '@lucide/svelte/icons/eject';
import FileIcon from '@lucide/svelte/icons/file';
+// not a microphone: that one is for recording and dictation
+import AudioIcon from '@lucide/svelte/icons/file-headphone';
import FileTextIcon from '@lucide/svelte/icons/file-text';
import CompactCommandIcon from '@lucide/svelte/icons/fold-vertical';
import CwdCommandIcon from '@lucide/svelte/icons/folder-open';
import ImageIcon from '@lucide/svelte/icons/image';
import LoadingIcon from '@lucide/svelte/icons/loader-2';
-import AudioIcon from '@lucide/svelte/icons/mic';
import ModelIcon from '@lucide/svelte/icons/package';
import PauseDownloadIcon from '@lucide/svelte/icons/pause';
import ResumeDownloadIcon from '@lucide/svelte/icons/play';
diff --git a/ui/apps/chat/src/lib/constants/index.ts b/ui/apps/chat/src/lib/constants/index.ts
index e014d457d..6f6ac5a10 100644
--- a/ui/apps/chat/src/lib/constants/index.ts
+++ b/ui/apps/chat/src/lib/constants/index.ts
@@ -47,7 +47,6 @@ export * from './mcp-form.constants';
export * from './mcp-resource.constants';
export * from './mention-badge.constants';
export * from './message-export.constants';
-export * from './microphone.constants';
export * from './path-display.constants';
export * from './model-id.constants';
export * from './model-loading.constants';
@@ -75,3 +74,4 @@ export * from './read-media';
export * from './get-datetime';
export * from './browser-info';
export * from './search-chats';
+export * from './voice.constants';
diff --git a/ui/apps/chat/src/lib/constants/microphone.constants.ts b/ui/apps/chat/src/lib/constants/microphone.constants.ts
deleted file mode 100644
index 170299f9b..000000000
--- a/ui/apps/chat/src/lib/constants/microphone.constants.ts
+++ /dev/null
@@ -1,10 +0,0 @@
-/** Why the chat form's microphone can't record, as its button and toast tell the user. */
-export const MICROPHONE_UNAVAILABLE_MESSAGE = {
- busy: 'The microphone is in use by another app.',
- denied: "Microphone access was denied. Allow it in the browser's site settings to record.",
- failed: "The microphone couldn't start.",
- insecure:
- 'The microphone only works over HTTPS or on localhost. Open this page over HTTPS to record.',
- notFound: 'No microphone was found.',
- unsupported: "This browser can't record audio."
-} as const;
diff --git a/ui/apps/chat/src/lib/constants/settings-keys.constants.ts b/ui/apps/chat/src/lib/constants/settings-keys.constants.ts
index 03e9a39d5..5bb1bb4ec 100644
--- a/ui/apps/chat/src/lib/constants/settings-keys.constants.ts
+++ b/ui/apps/chat/src/lib/constants/settings-keys.constants.ts
@@ -87,6 +87,17 @@ export const SETTINGS_KEYS = {
TYP_P: 'typ_p',
// General
USE_HUGGING_FACE_HUB: 'useHuggingFaceHub',
+ // Voice
+ VOICE_AUTO_GAIN: 'voiceAutoGain',
+ VOICE_BOOST: 'voiceBoost',
+ VOICE_COLLAPSE_TRANSCRIPTS: 'voiceCollapseTranscripts',
+ VOICE_ECHO_CANCELLATION: 'voiceEchoCancellation',
+ VOICE_INPUT_MODE: 'voiceInputMode',
+ VOICE_MICROPHONE: 'voiceMicrophone',
+ VOICE_NOISE_SUPPRESSION: 'voiceNoiseSuppression',
+ VOICE_TRANSCRIBE_MESSAGES: 'voiceTranscribeMessages',
+ VOICE_TRANSCRIPTION_LANGUAGE: 'voiceTranscriptionLanguage',
+ VOICE_TRANSCRIPTION_MODEL: 'voiceTranscriptionModel',
XTC_PROBABILITY: 'xtc_probability',
XTC_THRESHOLD: 'xtc_threshold'
} as const;
diff --git a/ui/apps/chat/src/lib/constants/settings.constants.ts b/ui/apps/chat/src/lib/constants/settings.constants.ts
index 4f1f77237..4df67abef 100644
--- a/ui/apps/chat/src/lib/constants/settings.constants.ts
+++ b/ui/apps/chat/src/lib/constants/settings.constants.ts
@@ -4,19 +4,24 @@ import { COMPACTION } from './compaction.constants';
import { DEFAULT_MCP_CONFIG } from './mcp.constants';
import { SETTINGS_KEYS } from './settings-keys.constants';
import { TITLE_GENERATION } from './title-generation.constants';
+import { VOICE_TRANSCRIPTION_MODEL_AUTO } from './voice.constants';
import { FILE_GLOB_SEARCH_PICKERS } from './working-directory.constants';
+import { VOICE_MICROPHONE_AUTO } from '@ggml-org/llama-voice';
+import AudioLines from '@lucide/svelte/icons/audio-lines';
import Code from '@lucide/svelte/icons/code';
import Database from '@lucide/svelte/icons/database';
import File from '@lucide/svelte/icons/file';
import Funnel from '@lucide/svelte/icons/funnel';
import GlassWater from '@lucide/svelte/icons/glass-water';
import ListRestart from '@lucide/svelte/icons/list-restart';
+import Mic from '@lucide/svelte/icons/mic';
import Monitor from '@lucide/svelte/icons/monitor';
import Moon from '@lucide/svelte/icons/moon';
import PencilRuler from '@lucide/svelte/icons/pencil-ruler';
import SlidersVertical from '@lucide/svelte/icons/sliders-vertical';
import Sun from '@lucide/svelte/icons/sun';
-import { SurfaceTheme, SyncableParameterType } from '#lib/enums/index.ts';
+import TextCursorInput from '@lucide/svelte/icons/text-cursor-input';
+import { SurfaceTheme, SyncableParameterType, VoiceInputMode } from '#lib/enums/index.ts';
import { SettingsFieldType } from '#lib/enums/settings.enums.ts';
import { ColorMode } from '#lib/enums/ui.enums.ts';
import type {
@@ -35,7 +40,8 @@ export const SETTINGS_SECTIONS = {
GENERAL: { slug: 'general', title: 'General' },
IMPORT_EXPORT: { slug: 'import-export', title: 'Import/Export' },
SAMPLING_PENALTIES: { slug: 'sampling-penalties', title: 'Sampling & Penalties' },
- TOOLS: { slug: 'tools', title: 'Tools' }
+ TOOLS: { slug: 'tools', title: 'Tools' },
+ VOICE: { slug: 'voice', title: 'Voice' }
} as const;
export const SETTINGS_SECTION_SLUGS = {
@@ -45,7 +51,8 @@ export const SETTINGS_SECTION_SLUGS = {
GENERAL: SETTINGS_SECTIONS.GENERAL.slug,
IMPORT_EXPORT: SETTINGS_SECTIONS.IMPORT_EXPORT.slug,
SAMPLING_PENALTIES: SETTINGS_SECTIONS.SAMPLING_PENALTIES.slug,
- TOOLS: SETTINGS_SECTIONS.TOOLS.slug
+ TOOLS: SETTINGS_SECTIONS.TOOLS.slug,
+ VOICE: SETTINGS_SECTIONS.VOICE.slug
} as const;
export const SETTINGS_SECTION_TITLES = {
@@ -55,7 +62,8 @@ export const SETTINGS_SECTION_TITLES = {
GENERAL: SETTINGS_SECTIONS.GENERAL.title,
IMPORT_EXPORT: SETTINGS_SECTIONS.IMPORT_EXPORT.title,
SAMPLING_PENALTIES: SETTINGS_SECTIONS.SAMPLING_PENALTIES.title,
- TOOLS: SETTINGS_SECTIONS.TOOLS.title
+ TOOLS: SETTINGS_SECTIONS.TOOLS.title,
+ VOICE: SETTINGS_SECTIONS.VOICE.title
} as const;
export const SETTINGS_REGISTRY: SettingsSectionEntry[] = [
@@ -132,7 +140,7 @@ export const SETTINGS_REGISTRY: SettingsSectionEntry[] = [
},
{
defaultValue: true,
- help: 'Automatically show microphone button instead of send button when textarea is empty for models with audio modality support.',
+ help: 'Show the microphone beside the send button while the message is empty, whenever voice input is available (see Voice).',
key: SETTINGS_KEYS.AUTO_MIC_ON_EMPTY,
label: 'Show microphone on empty input',
type: SettingsFieldType.CHECKBOX
@@ -405,6 +413,91 @@ export const SETTINGS_REGISTRY: SettingsSectionEntry[] = [
slug: SETTINGS_SECTION_SLUGS.DISPLAY,
title: SETTINGS_SECTION_TITLES.DISPLAY
},
+ // Voice: the transcription model picker lists the server's models (SettingsChatVoiceTab)
+ {
+ icon: Mic,
+ settings: [
+ {
+ defaultValue: VoiceInputMode.AUTO,
+ help: 'Automatic sends your voice to a model that understands audio, as a voice message with its transcript, and turns it into text in the message input for any other model. Always dictate turns it into text for every model, so only the text reaches the chat.',
+ key: SETTINGS_KEYS.VOICE_INPUT_MODE,
+ label: 'Voice input',
+ options: [
+ { icon: AudioLines, label: 'Automatic', value: VoiceInputMode.AUTO },
+ { icon: TextCursorInput, label: 'Always dictate', value: VoiceInputMode.DICTATION }
+ ],
+ type: SettingsFieldType.SELECT
+ },
+ {
+ defaultValue: VOICE_TRANSCRIPTION_MODEL_AUTO,
+ help: 'The model that turns speech into text when the chat model has no audio input. Automatic uses a model with audio input that is already loaded, so dictating loads nothing.',
+ key: SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL,
+ label: 'Transcription model',
+ standaloneField: false,
+ type: SettingsFieldType.SELECT
+ },
+ {
+ defaultValue: VOICE_MICROPHONE_AUTO,
+ help: "The microphone you record from. Automatic takes the system's default one, unless that is a Bluetooth headset's: recording from it switches the headset to its call mode, which plays music in low quality, so the built-in microphone is used instead.",
+ key: SETTINGS_KEYS.VOICE_MICROPHONE,
+ label: 'Microphone',
+ standaloneField: false,
+ type: SettingsFieldType.SELECT
+ },
+ {
+ defaultValue: true,
+ help: 'Write down what was said in each voice message, to read under its waveform. The chat model writes it, and a model without audio input gets it in place of the audio.',
+ key: SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES,
+ label: 'Transcribe voice messages',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: false,
+ help: 'Show the transcript of a voice message folded under its waveform, to open on hover or with a tap, instead of open.',
+ key: SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS,
+ label: 'Collapse transcripts',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: true,
+ help: 'Raise quiet speech to a level the model hears well, up to eight times louder, so a quiet microphone or a voice from afar is still understood. Only speech is measured, never the silence around it, and loud speech is left as it is.',
+ key: SETTINGS_KEYS.VOICE_BOOST,
+ label: 'Boost quiet speech',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: false,
+ help: 'Let the browser filter background noise out of what the microphone records. Off, the model gets the raw signal, which speech models take well.',
+ key: SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION,
+ label: 'Noise suppression',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: false,
+ help: 'Let the browser remove what your speakers play from the recording. It makes the system treat a recording as a call, which can lower or pause music and other sound while you record.',
+ key: SETTINGS_KEYS.VOICE_ECHO_CANCELLATION,
+ label: 'Echo cancellation',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: false,
+ help: 'Let the browser even out how loud you sound. It fades the microphone in, so the first words of a recording can come out quiet.',
+ key: SETTINGS_KEYS.VOICE_AUTO_GAIN,
+ label: 'Automatic gain',
+ type: SettingsFieldType.CHECKBOX
+ },
+ {
+ defaultValue: '',
+ help: 'The language you speak, as a two-letter code such as en or de, passed to the transcription. Leave it empty to let the model tell.',
+ key: SETTINGS_KEYS.VOICE_TRANSCRIPTION_LANGUAGE,
+ label: 'Transcription language',
+ placeholder: 'Automatic',
+ type: SettingsFieldType.INPUT
+ }
+ ],
+ slug: SETTINGS_SECTION_SLUGS.VOICE,
+ title: SETTINGS_SECTION_TITLES.VOICE
+ },
// MCP Servers (non-UI config object)
{
icon: PencilRuler,
diff --git a/ui/apps/chat/src/lib/constants/voice.constants.ts b/ui/apps/chat/src/lib/constants/voice.constants.ts
new file mode 100644
index 000000000..d897a1c73
--- /dev/null
+++ b/ui/apps/chat/src/lib/constants/voice.constants.ts
@@ -0,0 +1,130 @@
+import { STORAGE_APP_NAME } from './storage.constants';
+import { VoiceRecordingBarStatus } from '@ggml-org/llama-ui/voice';
+import { ChatFormVoiceState } from '#lib/enums/voice.enums.ts';
+
+/** What the chat form's microphone says it will do, or why it can't, on its button. */
+export const VOICE_MODE_MESSAGE = {
+ dictate: 'Dictate',
+ dictating: 'Dictating',
+ realtimeOnlyBehindKey:
+ 'This model only transcribes live, over a connection a browser cannot open on a server with an API key.',
+ recording: 'Recording a voice message',
+ refining: 'Refining transcript...',
+ sendAfterRefining: 'Sending once the transcript is refined',
+ separateModelHint:
+ 'The transcription model is loaded on the server alongside the chat model, which a server that keeps one model at a time unloads.',
+ starting: 'Starting...',
+ tooQuiet:
+ 'Your microphone sounds quiet. Move closer, or turn its input level up in your system settings.',
+ transcribing: 'Transcribing...',
+ transcriptionFailed: 'Transcription failed',
+ unavailable:
+ 'No model with audio input is available. In router mode, load one or pick a transcription model in Settings > Voice.',
+ voiceMessage: 'Send a voice message'
+} as const;
+
+/** The status of a server that has no such route for the model, e.g. a live-only model asked over HTTP. */
+export const VOICE_NOT_SUPPORTED_STATUS = 501;
+
+/** Where the chat form remembers that it showed the hint about a separate transcription model. */
+export const VOICE_SEPARATE_MODEL_HINT_LOCALSTORAGE_KEY = `${STORAGE_APP_NAME}.voiceSeparateModelHintSeen`;
+
+/** Name prefix of a voice message's WAV file; the rest makes it unique per recording. */
+export const VOICE_CLIP_NAME_PREFIX = 'voice-';
+
+/** File extension of a voice message. */
+export const VOICE_CLIP_EXTENSION = '.wav';
+
+/** Characters of a random id in a voice message's file name, after its time. */
+export const VOICE_CLIP_ID_LENGTH = 8;
+
+/** How often the recording's elapsed time is updated. */
+export const VOICE_ELAPSED_TICK_MS = 200;
+
+/**
+ * A mouse starts recording on press, so the click that ends that press comes within this
+ * long and is not another tap.
+ */
+export const VOICE_PRESS_CLICK_MS = 1000;
+
+/** How long a transcript that came back keeps looking for the message it belongs to. */
+export const VOICE_TRANSCRIPT_APPLY_RETRY_DELAYS_MS = [500, 1500, 4000] as const;
+
+/** A string that is already a URL (data, blob, http...), which an <audio> plays as it is. */
+export const VOICE_URL_PATTERN = /^[a-z][a-z0-9+.-]*:/i;
+
+/** A URL whose audio has to be fetched before its peaks can be drawn. */
+export const VOICE_FETCHABLE_URL_PATTERN = /^(blob|https?):/i;
+
+/** Number of characters at a stored clip's end that, with its name and length, tell clips apart. */
+export const VOICE_PEAKS_CACHE_TAIL_LENGTH = 32;
+
+/** The engine's server routes are relative (`./v1/...`); the app's base takes them from the root. */
+export const VOICE_RELATIVE_PATH_PATTERN = /^\./;
+
+/** A model file's extension, left out where a single model's name is shown. */
+export const VOICE_GGUF_EXTENSION_PATTERN = /\.gguf$/i;
+
+/** Element of the message box's editor: a tap there edits the live text of a voice message. */
+export const VOICE_EDITOR_SELECTOR = '.cm-editor';
+
+/** What Settings > Voice says beside its pickers. */
+export const VOICE_SETTINGS_LABEL = {
+ automatic: 'Automatic',
+ chatModels: 'Chat models with audio input',
+ disconnectedMicrophone: 'A microphone that is not connected',
+ microphone: (index: number) => `Microphone ${index}`,
+ speechToText: 'Speech to text',
+ systemDefault: 'System default'
+} as const;
+
+/** What a voice message attachment is called, in place of its file name. */
+export const VOICE_MESSAGE_LABEL = 'Voice message';
+
+/**
+ * The Transcription model setting's Automatic choice: the chat model when it hears, else a
+ * model with audio input that is already loaded, so dictating loads nothing.
+ */
+export const VOICE_TRANSCRIPTION_MODEL_AUTO = '';
+
+/**
+ * Names of speech-to-text models, which write down what is said and are not meant to chat
+ * (Qwen3-ASR, Whisper, Parakeet...). The server tells them apart by their chat template and
+ * does not say so, so the app goes by the name.
+ */
+export const VOICE_TRANSCRIPTION_ONLY_PATTERN =
+ /(^|[^a-z0-9])(asr|whisper|parakeet|canary|moonshine|sensevoice|stt)([^a-z0-9]|$)/i;
+
+/** How the model picker and the Models manager name a kind of model with audio input. */
+export const VOICE_MODEL_ROLE_LABEL = {
+ transcription: 'Speech to text',
+ voice: 'Voice'
+} as const;
+
+/**
+ * What the microphone on a model's row says: on the model that transcribes dictation for
+ * chat models that cannot hear, whether it was chosen or picked automatically (a loaded
+ * model with audio input), and on any other model that hears, that it can be the one.
+ */
+export const VOICE_DICTATION_MODEL_LABEL = {
+ automatic: 'Transcribes dictation, picked automatically. Click to keep it',
+ chosen: 'Transcribes dictation. Click to pick automatically',
+ current: 'transcribes dictation',
+ use: 'Use for dictation'
+} as const;
+
+/** What each kind means, in the badge's tooltip. */
+export const VOICE_MODEL_ROLE_DESCRIPTION = {
+ transcription:
+ 'A speech-to-text model: it writes down what is said, for dictation, and is not meant to chat.',
+ voice:
+ 'Hears audio: as the chat model it gets your voice messages, and it can also transcribe dictation.'
+} as const;
+
+/** What the recording bar shows for each state of the chat form's recording. */
+export const VOICE_RECORDING_BAR_STATUS: Record<ChatFormVoiceState, VoiceRecordingBarStatus> = {
+ [ChatFormVoiceState.IDLE]: VoiceRecordingBarStatus.RECORDING,
+ [ChatFormVoiceState.RECORDING]: VoiceRecordingBarStatus.RECORDING,
+ [ChatFormVoiceState.STARTING]: VoiceRecordingBarStatus.STARTING,
+ [ChatFormVoiceState.TRANSCRIBING]: VoiceRecordingBarStatus.WAITING
+};
diff --git a/ui/apps/chat/src/lib/enums/attachment.enums.ts b/ui/apps/chat/src/lib/enums/attachment.enums.ts
index 5f096b974..035fdac9c 100644
--- a/ui/apps/chat/src/lib/enums/attachment.enums.ts
+++ b/ui/apps/chat/src/lib/enums/attachment.enums.ts
@@ -50,5 +50,6 @@ export enum AttachmentLabel {
FILE = 'File',
MCP_PROMPT = 'MCP Prompt',
MCP_RESOURCE = 'MCP Resource',
- PDF_FILE = 'PDF File'
+ PDF_FILE = 'PDF File',
+ VOICE_TRANSCRIPT = 'Voice message transcript'
}
diff --git a/ui/apps/chat/src/lib/enums/index.ts b/ui/apps/chat/src/lib/enums/index.ts
index 29fd94373..c92a906d0 100644
--- a/ui/apps/chat/src/lib/enums/index.ts
+++ b/ui/apps/chat/src/lib/enums/index.ts
@@ -100,6 +100,8 @@ export {
HtmlInputType,
McpPromptVariant,
MenuKind,
+ MouseButton,
+ PointerType,
ScrollCarouselVariant,
SidebarAction,
SidebarSectionKind,
@@ -121,3 +123,5 @@ export {
} from './tools.enums';
export { SplashOrientation } from './splash.enums';
+
+export { ChatFormVoiceState, VoiceInputMode, VoiceMode, VoiceModelRole } from './voice.enums';
diff --git a/ui/apps/chat/src/lib/enums/ui.enums.ts b/ui/apps/chat/src/lib/enums/ui.enums.ts
index 2ea3e5f31..c46059434 100644
--- a/ui/apps/chat/src/lib/enums/ui.enums.ts
+++ b/ui/apps/chat/src/lib/enums/ui.enums.ts
@@ -1,4 +1,4 @@
-export { ColorMode, TooltipSide } from '@ggml-org/llama-ui/enums';
+export { ColorMode, MouseButton, PointerType, TooltipSide } from '@ggml-org/llama-ui/enums';
/** The light surface theme: Paper (white panels on a faint grey page) or Matte (grey glass on white). */
export enum SurfaceTheme {
diff --git a/ui/apps/chat/src/lib/enums/voice.enums.ts b/ui/apps/chat/src/lib/enums/voice.enums.ts
new file mode 100644
index 000000000..9d0dd3e96
--- /dev/null
+++ b/ui/apps/chat/src/lib/enums/voice.enums.ts
@@ -0,0 +1,36 @@
+/** Where the chat form's recording is: off, waiting for the microphone, recording, or waiting for dictated text. */
+export enum ChatFormVoiceState {
+ IDLE = 'idle',
+ RECORDING = 'recording',
+ STARTING = 'starting',
+ TRANSCRIBING = 'transcribing'
+}
+
+/** The Voice input setting: what the microphone does with a model that understands audio. */
+export enum VoiceInputMode {
+ /** Send the voice to a model with audio input, dictate otherwise. */
+ AUTO = 'auto',
+ /** Always turn speech into text; only the text reaches the chat. */
+ DICTATION = 'dictation'
+}
+
+/** What a recording becomes for the conversation's model, as the chat form resolves it. */
+export enum VoiceMode {
+ /** Speech becomes text in the input; no audio is attached. */
+ DICTATION = 'dictation',
+ /** No model can take audio, so the microphone has nothing to do. */
+ UNAVAILABLE = 'unavailable',
+ /** The clip is attached and sent as audio, with a transcript to read. */
+ VOICE_MESSAGE = 'voice-message'
+}
+
+/**
+ * What kind of model with audio input a model is, as the model picker and the Models manager
+ * badge it. Which of them transcribes dictation is told apart from it, by a microphone.
+ */
+export enum VoiceModelRole {
+ /** A speech-to-text model: for dictation, not for chat */
+ TRANSCRIPTION = 'transcription',
+ /** A chat model with audio input: it gets voice messages */
+ VOICE = 'voice'
+}
diff --git a/ui/apps/chat/src/lib/hooks/use-chat-form-voice.svelte.ts b/ui/apps/chat/src/lib/hooks/use-chat-form-voice.svelte.ts
new file mode 100644
index 000000000..da024261b
--- /dev/null
+++ b/ui/apps/chat/src/lib/hooks/use-chat-form-voice.svelte.ts
@@ -0,0 +1,411 @@
+/**
+ * Voice input for the chat form.
+ *
+ * Resolves what the microphone does for the conversation's model (a voice message,
+ * dictation, or nothing, see resolveVoiceMode) and runs a recording: live levels and
+ * elapsed time, and on stop either a voice message attachment, whose transcript follows,
+ * or the dictated text. Writing the text into the message is createDictation()'s; this
+ * adds what is the chat form's own around it.
+ * A voice message's live text shows as the empty message box's placeholder; focusing
+ * the box makes it editable text there until the recording stops, when it goes with
+ * the clip (edited, as its transcript).
+ */
+
+import {
+ createDictation,
+ describeMicrophoneError,
+ MicrophoneWarmup,
+ VoiceCaptureError,
+ type VoiceRecorderStatus,
+ type VoiceSessionOptions,
+ type VoiceSessionResult
+} from '@ggml-org/llama-voice';
+import {
+ SETTINGS_KEYS,
+ VOICE_CLIP_EXTENSION,
+ VOICE_CLIP_ID_LENGTH,
+ VOICE_CLIP_NAME_PREFIX,
+ VOICE_ELAPSED_TICK_MS,
+ VOICE_MODE_MESSAGE,
+ VOICE_SEPARATE_MODEL_HINT_LOCALSTORAGE_KEY
+} from '#lib/constants/index.ts';
+import { BooleanString, ChatFormVoiceState, MimeTypeAudio, VoiceMode } from '#lib/enums/index.ts';
+import { VoiceService } from '#lib/services/voice.service.ts';
+import {
+ serverStore,
+ settingsStore,
+ voiceStore,
+ voiceTranscriptsStore
+} from '#lib/stores/index.ts';
+import type { ChatUploadedFile } from '#lib/types/index.ts';
+import { uuid } from '#lib/utils/index.ts';
+import { voiceModeLabel } from '#lib/utils/voice-mode.ts';
+import { onDestroy } from 'svelte';
+import { toast } from 'svelte-sonner';
+
+/** What the form needs of a VoiceRecorder; tests and stories pass a fake. */
+export interface VoiceRecorderLike {
+ readonly levels: number[];
+ readonly live: boolean;
+ readonly liveText: string;
+ /** The speech stays quiet even at the highest gain; a recorder that cannot tell leaves it out */
+ readonly quiet?: boolean;
+ readonly status: VoiceRecorderStatus;
+ start(options?: VoiceSessionOptions): Promise<void>;
+ stop(): Promise<VoiceSessionResult>;
+ cancel(): void;
+}
+
+interface UseChatFormVoiceOptions {
+ recorder: VoiceRecorderLike;
+ getValue: () => string;
+ /** Writes the message as dictation changes it; not a user edit. */
+ setValue: (value: string) => void;
+ getCaretOffset: () => number;
+ /** Puts the caret after the dictated text once it is final. */
+ setCaretOffset: (offset: number) => void;
+ addVoiceFile: (file: ChatUploadedFile) => void;
+ /** Why the last start failed, for the microphone button; null once one worked. */
+ onMicError: (message: string | null) => void;
+}
+
+function readHintSeen(): boolean {
+ try {
+ return localStorage.getItem(VOICE_SEPARATE_MODEL_HINT_LOCALSTORAGE_KEY) === BooleanString.TRUE;
+ } catch {
+ return false;
+ }
+}
+
+function writeHintSeen(): void {
+ try {
+ localStorage.setItem(VOICE_SEPARATE_MODEL_HINT_LOCALSTORAGE_KEY, BooleanString.TRUE);
+ } catch {
+ // private mode: the hint shows again next time
+ }
+}
+
+function readAsDataUrl(blob: Blob): Promise<string> {
+ return new Promise((resolve, reject) => {
+ const reader = new FileReader();
+
+ reader.onload = () => resolve(reader.result as string);
+ reader.onerror = () => reject(reader.error);
+ reader.readAsDataURL(blob);
+ });
+}
+
+export function useChatFormVoice(options: UseChatFormVoiceOptions) {
+ const config = $derived(settingsStore.config);
+ const isRouter = $derived(serverStore.isRouterMode);
+ // what the microphone does for the conversation's model, shared with the model lists
+ const resolution = $derived(voiceStore.resolution);
+ // a router with a model that hears keeps the button, so it can say where to pick it
+ const anyAudioModel = $derived(voiceStore.anyAudioModel);
+
+ let hintSeen = $state(readHintSeen());
+ let state = $state<ChatFormVoiceState>(ChatFormVoiceState.IDLE);
+ let elapsedMs = $state(0);
+ // the mode of the recording under way, fixed when it starts
+ let recordingMode = $state<VoiceMode>(VoiceMode.UNAVAILABLE);
+ let ticker: ReturnType<typeof setInterval> | undefined;
+ // bumped when the dictation changes, so the getters that read it follow
+ let dictationVersion = $state(0);
+ // bumped by cancel, so a recording still starting or stopping is dropped
+ let generation = 0;
+ // the pointer or the focus is on the microphone button, see prepare
+ let preparing = false;
+
+ const warmup = new MicrophoneWarmup();
+ const dictation = createDictation({
+ getCaretOffset: options.getCaretOffset,
+ getValue: options.getValue,
+ onChange: () => dictationVersion++,
+ onError: (error) =>
+ toast.error(
+ VoiceService.failureReason(error) ??
+ (error instanceof Error ? error.message : VOICE_MODE_MESSAGE.transcriptionFailed)
+ ),
+ setCaretOffset: options.setCaretOffset,
+ setValue: options.setValue
+ });
+ const recorder = options.recorder;
+ const unavailableReason = $derived(
+ resolution.mode === VoiceMode.UNAVAILABLE ? resolution.reason : null
+ );
+ const hint = $derived(
+ resolution.mode === VoiceMode.DICTATION && resolution.separateModel && !hintSeen
+ ? VOICE_MODE_MESSAGE.separateModelHint
+ : null
+ );
+
+ function startTicker() {
+ const startedAt = Date.now();
+
+ elapsedMs = 0;
+ ticker = setInterval(() => (elapsedMs = Date.now() - startedAt), VOICE_ELAPSED_TICK_MS);
+ }
+
+ function stopTicker() {
+ clearInterval(ticker);
+ ticker = undefined;
+ }
+
+ /**
+ * A voice message's live text moves from the placeholder into the message, as text the
+ * user can edit, and what is said next follows it there.
+ */
+ function adoptLiveText() {
+ if (recordingMode !== VoiceMode.VOICE_MESSAGE || dictation.active) return;
+
+ if (state !== ChatFormVoiceState.STARTING && state !== ChatFormVoiceState.RECORDING) return;
+
+ dictation.begin({ at: options.getValue().length, keepsEdits: true });
+ dictation.write(recorder?.liveText ?? '');
+ options.setCaretOffset(dictation.end);
+ }
+
+ /** The session options of a recording started now: the settings, the server and the mode. */
+ function wiringFor(mode: VoiceMode, model: string | null) {
+ const transcribe =
+ mode === VoiceMode.DICTATION || config[SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES] !== false;
+
+ return VoiceService.sessionOptions(config, { isRouter, model, transcribe });
+ }
+
+ /**
+ * Opens the microphone while the pointer or the focus is on its button, if the page may
+ * already use it: by the click it has settled, so the first words are not quiet, and the
+ * recording starts at once. Let go shortly after the pointer leaves (unprepare).
+ */
+ async function prepare() {
+ if (!recorder || state !== ChatFormVoiceState.IDLE || resolution.mode === VoiceMode.UNAVAILABLE)
+ return;
+
+ preparing = true;
+
+ const wiring = await wiringFor(resolution.mode, resolution.transcriptionModel);
+
+ // the pointer may have left meanwhile, and a press asks for its own microphone
+ if (preparing && state === ChatFormVoiceState.IDLE) warmup.open(wiring.capture);
+ }
+
+ function unprepare() {
+ preparing = false;
+ warmup.release();
+ }
+
+ async function start() {
+ if (!recorder || state !== ChatFormVoiceState.IDLE || resolution.mode === VoiceMode.UNAVAILABLE)
+ return;
+
+ const mode = resolution.mode;
+ const model = resolution.transcriptionModel;
+
+ if (hint) {
+ hintSeen = true;
+ writeHintSeen();
+ }
+
+ recordingMode = mode;
+ state = ChatFormVoiceState.STARTING;
+
+ if (mode === VoiceMode.DICTATION) dictation.begin();
+
+ const run = ++generation;
+
+ try {
+ const wiring = await wiringFor(mode, model);
+ // the microphone the hover opened, when it is open for the same options
+ const stream = await warmup.take(wiring.capture);
+
+ // cancelled while the microphone opened
+ if (run !== generation) {
+ for (const track of stream?.getTracks() ?? []) track.stop();
+
+ return;
+ }
+
+ await recorder.start({
+ ...wiring,
+ capture: { ...wiring.capture, ...(stream ? { stream } : {}) },
+ // a voice message's text reaches the message only once the user made it editable
+ onText: (text) => dictation.write(text)
+ });
+ } catch (error) {
+ if (run !== generation) return;
+
+ state = ChatFormVoiceState.IDLE;
+ dictation.cancel();
+
+ const message =
+ error instanceof VoiceCaptureError ? error.message : describeMicrophoneError(error);
+
+ options.onMicError(message);
+ toast.error(message);
+
+ return;
+ }
+
+ // cancelled meanwhile: the recorder dropped its own session
+ if (run !== generation) return;
+
+ options.onMicError(null);
+ state = ChatFormVoiceState.RECORDING;
+ startTicker();
+ }
+
+ async function stop() {
+ if (!recorder || state !== ChatFormVoiceState.RECORDING) return;
+
+ const run = generation;
+ const mode = recordingMode;
+
+ stopTicker();
+
+ let result: VoiceSessionResult;
+
+ try {
+ result = await recorder.stop();
+ } catch (error) {
+ console.error('Failed to stop the recording:', error);
+ dictation.cancel();
+ state = ChatFormVoiceState.IDLE;
+
+ return;
+ }
+
+ if (run !== generation) return;
+
+ if (mode === VoiceMode.VOICE_MESSAGE) {
+ // the clip carries the text from here: what the user edited is its transcript
+ const taken = dictation.take();
+
+ state = ChatFormVoiceState.IDLE;
+ await attachVoiceMessage(result, taken?.userEdited ? taken.text : null);
+
+ return;
+ }
+
+ // with live text the message already shows it and can be edited, while the whole
+ // recording is transcribed once more to refine it; otherwise wait for the text
+ state =
+ result.transcript && !result.liveText
+ ? ChatFormVoiceState.TRANSCRIBING
+ : ChatFormVoiceState.IDLE;
+ await dictation.finish(result);
+
+ if (run === generation) state = ChatFormVoiceState.IDLE;
+ }
+
+ async function attachVoiceMessage(
+ { clip, transcript }: VoiceSessionResult,
+ edited: string | null = null
+ ) {
+ const name = `${VOICE_CLIP_NAME_PREFIX}${Date.now()}-${uuid().slice(0, VOICE_CLIP_ID_LENGTH)}${VOICE_CLIP_EXTENSION}`;
+ const file = new File([clip.blob], name, { type: MimeTypeAudio.WAV });
+
+ if (transcript) voiceTranscriptsStore.track(name, transcript);
+
+ if (edited !== null) voiceTranscriptsStore.setTranscript(name, edited);
+
+ options.addVoiceFile({
+ file,
+ id: uuid(),
+ name,
+ preview: await readAsDataUrl(clip.blob),
+ size: file.size,
+ type: MimeTypeAudio.WAV,
+ voice: { durationMs: clip.durationMs, peaks: clip.peaks }
+ });
+ }
+
+ /** Drops the recording and any text it dictated. */
+ function cancel() {
+ generation++;
+ stopTicker();
+ recorder?.cancel();
+ dictation.cancel();
+ state = ChatFormVoiceState.IDLE;
+ }
+
+ /** Follows an edit the user typed while dictation writes into the message. */
+ function handleUserInput() {
+ dictation.edited();
+ }
+
+ onDestroy(() => {
+ preparing = false;
+ warmup.drop();
+
+ if (state !== ChatFormVoiceState.IDLE) cancel();
+ });
+
+ return {
+ adoptLiveText,
+ cancel,
+ get elapsedMs() {
+ return elapsedMs;
+ },
+ handleUserInput,
+ get hint() {
+ return hint;
+ },
+ /** A recording (or its dictated text) is under way: the form shows the recording bar. */
+ get isActive() {
+ return state !== ChatFormVoiceState.IDLE;
+ },
+ /** The dictated text shown is still being refined by a pass over the whole recording. */
+ get isRefining() {
+ void dictationVersion;
+
+ return dictation.refining;
+ },
+ /** What the microphone button says it will do. */
+ get label() {
+ return voiceModeLabel(resolution);
+ },
+ get levels() {
+ return recorder?.levels ?? [];
+ },
+ /** Text streaming while a voice message is recorded, shown under the waveform. */
+ get liveCaption() {
+ void dictationVersion;
+
+ // once made editable, the text is in the message instead
+ return recordingMode === VoiceMode.VOICE_MESSAGE && !dictation.active
+ ? (recorder?.liveText ?? '')
+ : '';
+ },
+ get mode() {
+ return state === ChatFormVoiceState.IDLE ? resolution.mode : recordingMode;
+ },
+ /** The pointer or the focus is on the microphone button: see prepare(). */
+ prepare,
+ /** The recording's speech is too quiet even at the highest gain: the form says so. */
+ get quiet() {
+ return state === ChatFormVoiceState.RECORDING && (recorder?.quiet ?? false);
+ },
+ get resolution() {
+ return resolution;
+ },
+ /** Resolves once the text in the message is final, at once when nothing is refining it. */
+ settled(): Promise<void> {
+ return dictation.settled();
+ },
+ /** Whether the form shows a microphone button at all. */
+ get showButton() {
+ return resolution.mode !== VoiceMode.UNAVAILABLE || anyAudioModel;
+ },
+ start,
+ get state() {
+ return state;
+ },
+ stop,
+ get unavailableReason() {
+ return unavailableReason;
+ },
+ /** The pointer or the focus left the microphone button without a press. */
+ unprepare
+ };
+}
diff --git a/ui/apps/chat/src/lib/services/chat.service.ts b/ui/apps/chat/src/lib/services/chat.service.ts
index 848bd06ef..38c7eba14 100644
--- a/ui/apps/chat/src/lib/services/chat.service.ts
+++ b/ui/apps/chat/src/lib/services/chat.service.ts
@@ -50,6 +50,12 @@ import { getAuthHeaders, getJsonHeaders } from '#lib/utils/api-headers.ts';
import { formatAttachmentText } from '#lib/utils/formatters.ts';
import { streamIdentity } from '#lib/utils/stream-identity.ts';
+/** How stored messages are shaped for the model a request goes to. */
+export interface ApiMessageConversionOptions {
+ /** False when the model has no audio input: voice messages go as their transcript. */
+ audioInput?: boolean;
+}
+
interface ResumableStreamState {
bytesReceived: number;
updatedAt: number;
@@ -133,13 +139,24 @@ export class ChatService {
}
}
+ /**
+ * How messages are shaped for `model`: whether it takes audio, known only once its
+ * modalities are. Unknown modalities keep the audio, as before voice messages.
+ */
+ static conversionOptionsFor(model?: string | null): ApiMessageConversionOptions {
+ const modalities = modelsStore.props.getModelModalities(model ?? '');
+
+ return modalities ? { audioInput: modalities.audio } : {};
+ }
+
/**
* Converts a database message with attachments to API chat message format.
* Processes various attachment types (images, text files, PDFs) and formats them
* as content parts suitable for the chat completion API.
*/
static async convertDbMessageToApiChatMessageData(
- message: DatabaseMessage & { extra?: DatabaseMessageExtra[] }
+ message: DatabaseMessage & { extra?: DatabaseMessageExtra[] },
+ options: ApiMessageConversionOptions = {}
): Promise<ApiChatMessageData> {
// Handle tool result messages (role: 'tool')
if (message.role === MessageRole.TOOL && message.toolCallId) {
@@ -231,6 +248,20 @@ export class ChatService {
);
for (const audio of audioFiles) {
+ // a model that cannot hear reads what was said instead; one that can gets the audio alone
+ if (options.audioInput === false && audio.transcript) {
+ contentParts.push({
+ text: formatAttachmentText(
+ AttachmentLabel.VOICE_TRANSCRIPT,
+ audio.name,
+ audio.transcript
+ ),
+ type: ContentPartType.TEXT
+ });
+
+ continue;
+ }
+
contentParts.push({
input_audio: {
data: audio.base64Data,
@@ -816,14 +847,18 @@ export class ChatService {
* messages. Shared by sendMessage, preEncode and the agentic flow.
*/
static async normalizeMessagesForApi(
- messages: ApiChatMessageData[] | (DatabaseMessage & { extra?: DatabaseMessageExtra[] })[]
+ messages: ApiChatMessageData[] | (DatabaseMessage & { extra?: DatabaseMessageExtra[] })[],
+ model?: string | null
): Promise<ApiChatMessageData[]> {
+ const options = ChatService.conversionOptionsFor(model);
+
return (
await Promise.all(
messages.map((msg) => {
if ('id' in msg && 'convId' in msg && 'timestamp' in msg) {
return ChatService.convertDbMessageToApiChatMessageData(
- msg as DatabaseMessage & { extra?: DatabaseMessageExtra[] }
+ msg as DatabaseMessage & { extra?: DatabaseMessageExtra[] },
+ options
);
}
@@ -855,8 +890,10 @@ export class ChatService {
signal?: AbortSignal,
tools?: OpenAIToolDefinition[]
): Promise<ChatMessageTimings | null> {
- const normalizedMessages: ApiChatMessageData[] =
- await ChatService.normalizeMessagesForApi(messages);
+ const normalizedMessages: ApiChatMessageData[] = await ChatService.normalizeMessagesForApi(
+ messages,
+ model
+ );
const requestBody: Record<string, unknown> = {
messages: normalizedMessages.map((msg: ApiChatMessageData) => {
const mapped: Record<string, unknown> = {
@@ -1106,8 +1143,10 @@ export class ChatService {
xtc_probability,
xtc_threshold
} = options;
- const normalizedMessages: ApiChatMessageData[] =
- await ChatService.normalizeMessagesForApi(messages);
+ const normalizedMessages: ApiChatMessageData[] = await ChatService.normalizeMessagesForApi(
+ messages,
+ options.model
+ );
// Filter out image attachments if the model doesn't support vision
if (options.model && !modelsStore.props.modelSupportsVision(options.model)) {
diff --git a/ui/apps/chat/src/lib/services/database.service.ts b/ui/apps/chat/src/lib/services/database.service.ts
index de8ed4701..e77b97a96 100644
--- a/ui/apps/chat/src/lib/services/database.service.ts
+++ b/ui/apps/chat/src/lib/services/database.service.ts
@@ -572,6 +572,16 @@ export class DatabaseService {
return result;
}
+ /**
+ * Gets the messages written at or after `timestamp`, in any conversation.
+ *
+ * @param timestamp - Epoch ms
+ * @returns Messages, oldest first
+ */
+ static async getMessagesSince(timestamp: number): Promise<DatabaseMessage[]> {
+ return await db[IDXDB_TABLES.messages].where('timestamp').aboveOrEqual(timestamp).toArray();
+ }
+
/**
* Imports multiple conversations and their messages.
* Skips conversations that already exist.
diff --git a/ui/apps/chat/src/lib/services/index.ts b/ui/apps/chat/src/lib/services/index.ts
index c9b93d19a..243d77a83 100644
--- a/ui/apps/chat/src/lib/services/index.ts
+++ b/ui/apps/chat/src/lib/services/index.ts
@@ -371,3 +371,15 @@ export { MigrationService } from './migration.service';
* @see settingsStore in stores/settings/index.svelte.ts - reactive state + business logic
*/
export { SettingsService } from './settings.service';
+
+/**
+ * **VoiceService** - Wiring between the voice engine and this app
+ *
+ * Builds what the engine (@ggml-org/llama-voice) is handed from outside: how it reaches the active
+ * server (routes, API key, whether realtime can be used) and the capture and transcribe
+ * options the Voice settings ask for. Every feature that records starts its session with
+ * these, so they all reach the server the same way and follow the same settings.
+ *
+ * @see voice.service.ts
+ */
+export { VoiceService } from './voice.service';
diff --git a/ui/apps/chat/src/lib/services/voice.service.ts b/ui/apps/chat/src/lib/services/voice.service.ts
new file mode 100644
index 000000000..c794b1b64
--- /dev/null
+++ b/ui/apps/chat/src/lib/services/voice.service.ts
@@ -0,0 +1,138 @@
+import {
+ resolveMicrophone,
+ VOICE_MICROPHONE_AUTO,
+ type VoiceCaptureOptions,
+ type VoiceServerContext,
+ type VoiceSessionOptions,
+ VoiceTranscriptionError
+} from '@ggml-org/llama-voice';
+import {
+ HEADERS,
+ SETTINGS_KEYS,
+ VOICE_MODE_MESSAGE,
+ VOICE_NOT_SUPPORTED_STATUS,
+ VOICE_RELATIVE_PATH_PATTERN
+} from '#lib/constants/index.ts';
+import type { SettingsConfigType } from '#lib/types/index.ts';
+import { getAuthHeaders } from '#lib/utils/api-headers.ts';
+import { withBase } from '#lib/utils/base-path.ts';
+
+/** What a feature asks of the voice wiring, besides the settings. */
+export interface VoiceSessionRequest {
+ /** The model that transcribes, from resolveVoiceMode(); null leaves it to the server */
+ model: string | null;
+ /** Router mode: the model goes in the request, as the router routes on it */
+ isRouter: boolean;
+ /** False records without transcribing, e.g. a voice message when transcripts are off */
+ transcribe?: boolean;
+ /**
+ * The microphone processing to ask for, in place of what the Voice settings switch on,
+ * e.g. echo cancellation for a spoken conversation that plays replies aloud
+ */
+ constraints?: MediaTrackConstraints;
+}
+
+/**
+ * The session options voice features share: where to capture from, and how to transcribe.
+ * The capture is always plain options, so a feature can also open a VoiceCapture of its own
+ * with them.
+ */
+export type VoiceSessionWiring = Pick<VoiceSessionOptions, 'realtime' | 'transcribe'> & {
+ capture: VoiceCaptureOptions;
+};
+
+/**
+ * **VoiceService** - How the voice engine (@ggml-org/llama-voice) reaches this app's server and settings
+ *
+ * The engine imports no app code: it is handed a VoiceServerContext and plain capture
+ * options. This builds both from the active server and the Voice settings, once for every
+ * feature that records (the chat form's dictation and voice messages, voice mode), so they
+ * reach the server the same way and follow the same settings.
+ */
+export class VoiceService {
+ /**
+ * Why a transcription failed, where the app can say more than the server did, else null.
+ * A model that only transcribes live (Voxtral Realtime) has no HTTP route, and the
+ * realtime socket is left out for a server with an API key (see serverContext), so
+ * there it cannot transcribe at all.
+ */
+ static failureReason(error: unknown): string | null {
+ const liveOnly =
+ error instanceof VoiceTranscriptionError && error.status === VOICE_NOT_SUPPORTED_STATUS;
+
+ return liveOnly && !VoiceService.serverContext().realtime
+ ? VOICE_MODE_MESSAGE.realtimeOnlyBehindKey
+ : null;
+ }
+
+ /** The transcription language from the Voice settings, undefined to let the model tell. */
+ static language(config: SettingsConfigType): string | undefined {
+ return String(config[SETTINGS_KEYS.VOICE_TRANSCRIPTION_LANGUAGE] ?? '').trim() || undefined;
+ }
+
+ /**
+ * The microphone processing the Voice settings switch on, laid over the engine's raw
+ * signal; undefined when none is.
+ */
+ static processing(config: SettingsConfigType): MediaTrackConstraints | undefined {
+ const processing: MediaTrackConstraints = {
+ ...(config[SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION] === true ? { noiseSuppression: true } : {}),
+ ...(config[SETTINGS_KEYS.VOICE_ECHO_CANCELLATION] === true ? { echoCancellation: true } : {}),
+ ...(config[SETTINGS_KEYS.VOICE_AUTO_GAIN] === true ? { autoGainControl: true } : {})
+ };
+
+ return Object.keys(processing).length > 0 ? processing : undefined;
+ }
+
+ /**
+ * The server's routes and auth for the engine, and whether its realtime socket can be
+ * used: a browser websocket can not send the API key, so a server that needs one gets
+ * HTTP only.
+ */
+ static serverContext(): { context: VoiceServerContext; realtime: boolean } {
+ const hasAuth = Object.keys(getAuthHeaders()).some(
+ (key) => key.toLowerCase() === HEADERS.AUTHORIZATION.toLowerCase()
+ );
+
+ return {
+ context: {
+ getHeaders: () => getAuthHeaders(),
+ resolveUrl: (path) => withBase(path.replace(VOICE_RELATIVE_PATH_PATTERN, ''))
+ },
+ realtime: !hasAuth
+ };
+ }
+
+ /**
+ * The capture, realtime and transcribe options of a voice session, from the Voice
+ * settings (microphone, processing, boost, language) and the active server. Async:
+ * picking the microphone may list the devices.
+ */
+ static async sessionOptions(
+ config: SettingsConfigType,
+ request: VoiceSessionRequest
+ ): Promise<VoiceSessionWiring> {
+ const { context, realtime } = VoiceService.serverContext();
+ const deviceId = await resolveMicrophone(
+ String(config[SETTINGS_KEYS.VOICE_MICROPHONE] ?? VOICE_MICROPHONE_AUTO)
+ );
+ const constraints = request.constraints ?? VoiceService.processing(config);
+
+ return {
+ capture: {
+ boost: config[SETTINGS_KEYS.VOICE_BOOST] !== false,
+ ...(deviceId ? { deviceId } : {}),
+ ...(constraints ? { constraints } : {})
+ },
+ realtime,
+ transcribe:
+ request.transcribe === false
+ ? undefined
+ : {
+ context,
+ language: VoiceService.language(config),
+ model: request.isRouter && request.model ? request.model : undefined
+ }
+ };
+ }
+}
diff --git a/ui/apps/chat/src/lib/stores/agentic/index.svelte.ts b/ui/apps/chat/src/lib/stores/agentic/index.svelte.ts
index b607a1373..2eb93ad3a 100644
--- a/ui/apps/chat/src/lib/stores/agentic/index.svelte.ts
+++ b/ui/apps/chat/src/lib/stores/agentic/index.svelte.ts
@@ -347,8 +347,10 @@ class AgenticStore {
console.log(`[AgenticStore] Starting agentic flow with ${tools.length} tools`);
- const normalizedMessages: ApiChatMessageData[] =
- await ChatService.normalizeMessagesForApi(messages);
+ const normalizedMessages: ApiChatMessageData[] = await ChatService.normalizeMessagesForApi(
+ messages,
+ options.model
+ );
this.updateSession(conversationId, {
currentTurn: 0,
diff --git a/ui/apps/chat/src/lib/stores/chat/index.svelte.ts b/ui/apps/chat/src/lib/stores/chat/index.svelte.ts
index ef8a11100..20c95be39 100644
--- a/ui/apps/chat/src/lib/stores/chat/index.svelte.ts
+++ b/ui/apps/chat/src/lib/stores/chat/index.svelte.ts
@@ -1480,7 +1480,7 @@ class ChatStore implements ChatStreamHost, ChatFlowsHost {
const allMessages = await conversationsStore.getConversationMessages(convId);
const branchPath = [...filterByLeafNodeId(allMessages, leafId, false)];
const request = await agenticStore.requestFor(
- await ChatService.normalizeMessagesForApi(sliceAtLastCompaction(branchPath)),
+ await ChatService.normalizeMessagesForApi(sliceAtLastCompaction(branchPath), model),
this.toolPolicy()
);
@@ -1646,7 +1646,7 @@ class ChatStore implements ChatStreamHost, ChatFlowsHost {
// message shaping, cut before the preserved steps: the server
// reuses the prefix it cached and only the instruction is new.
const request = await agenticStore.requestFor(
- await ChatService.normalizeMessagesForApi(source.slice(0, cut)),
+ await ChatService.normalizeMessagesForApi(source.slice(0, cut), effectiveModel),
this.toolPolicy()
);
const configValue = settingsStore.config;
diff --git a/ui/apps/chat/src/lib/stores/index.ts b/ui/apps/chat/src/lib/stores/index.ts
index e98a48d06..3a2c5914c 100644
--- a/ui/apps/chat/src/lib/stores/index.ts
+++ b/ui/apps/chat/src/lib/stores/index.ts
@@ -61,6 +61,10 @@ export { permissionsStore } from './permissions.svelte';
// TOOLS
export { toolsStore } from './tools.svelte';
+// VOICE (which model does what, and transcripts of voice messages on their way)
+export { voiceStore } from './voice.svelte';
+export { voiceTranscriptsStore } from './voice-transcripts.svelte';
+
// ENVIRONMENT / META
export { versionStore } from './version.svelte';
diff --git a/ui/apps/chat/src/lib/stores/voice-transcripts.svelte.ts b/ui/apps/chat/src/lib/stores/voice-transcripts.svelte.ts
new file mode 100644
index 000000000..c3bf40373
--- /dev/null
+++ b/ui/apps/chat/src/lib/stores/voice-transcripts.svelte.ts
@@ -0,0 +1,176 @@
+/**
+ * voiceTranscriptsStore - Transcripts of voice messages that are on their way
+ *
+ * A voice message is attached as soon as the recording stops, while its transcript
+ * can take a few seconds more. This keeps each pending transcript by the clip's name
+ * (unique per recording), so the chip in the form and the sent message show its
+ * state, a send picks it up once it is there, and a message sent before it arrived
+ * gets it written into its stored attachment afterwards. Kept in memory only.
+ */
+
+import { VoiceTranscriptStatus } from '@ggml-org/llama-ui/voice';
+import { VOICE_TRANSCRIPT_APPLY_RETRY_DELAYS_MS } from '#lib/constants/index.ts';
+import { AttachmentType } from '#lib/enums/index.ts';
+import { DatabaseService } from '#lib/services/database.service.ts';
+import { VoiceService } from '#lib/services/voice.service.ts';
+// direct imports between stores, not via the barrel, to avoid circular deps
+import { conversationsStore } from '#lib/stores/conversations/index.svelte.ts';
+import type {
+ DatabaseMessage,
+ DatabaseMessageExtra,
+ DatabaseMessageExtraAudioFile
+} from '#lib/types/index.ts';
+import { SvelteMap } from 'svelte/reactivity';
+
+export type VoiceTranscriptState =
+ | { status: VoiceTranscriptStatus.TRANSCRIBING }
+ | { status: VoiceTranscriptStatus.DONE; transcript: string }
+ | {
+ status: VoiceTranscriptStatus.ERROR;
+ message: string;
+ /** Why, in words for the user, when the app can tell (VoiceService.failureReason) */
+ reason: string | null;
+ };
+
+function isPendingClip(extra: DatabaseMessageExtra, name: string): boolean {
+ return (
+ extra.type === AttachmentType.AUDIO &&
+ extra.name === name &&
+ extra.voice === true &&
+ extra.transcript === undefined
+ );
+}
+
+/** The extras with the clip's transcript filled in, or null when none of them is that clip. */
+export function withTranscript(
+ extras: DatabaseMessageExtra[] | undefined,
+ name: string,
+ transcript: string
+): DatabaseMessageExtra[] | null {
+ if (!extras?.some((extra) => isPendingClip(extra, name))) return null;
+
+ return extras.map((extra) =>
+ isPendingClip(extra, name)
+ ? ({ ...(extra as DatabaseMessageExtraAudioFile), transcript } as DatabaseMessageExtra)
+ : extra
+ );
+}
+
+class VoiceTranscriptsStore {
+ // clips whose transcript the user wrote while recording: the model's does not replace it
+ private edited = new Set<string>();
+ private entries = new SvelteMap<string, VoiceTranscriptState>();
+ private recordedAt = new Map<string, number>();
+
+ /** Drops a clip removed from the form before it was sent, or whose message has its transcript. */
+ forget(name: string): void {
+ this.entries.delete(name);
+ this.recordedAt.delete(name);
+ this.edited.delete(name);
+ }
+
+ /** The state of a clip's transcript, undefined when none was asked for in this session. */
+ get(name: string): VoiceTranscriptState | undefined {
+ return this.entries.get(name);
+ }
+
+ /** Sets a clip's transcript to the text the user edited while recording; the model's is ignored. */
+ setTranscript(name: string, transcript: string): void {
+ this.edited.add(name);
+ this.recordedAt.set(name, this.recordedAt.get(name) ?? Date.now());
+ this.entries.set(name, { status: VoiceTranscriptStatus.DONE, transcript });
+ }
+
+ /** A clip's transcript as it is shown: the one stored with it, else the one on its way. */
+ shown(
+ name: string,
+ stored?: string
+ ): { status: VoiceTranscriptStatus; transcript?: string; errorReason?: string | null } {
+ const transcript = stored ?? this.transcriptOf(name);
+
+ if (transcript !== undefined) return { status: VoiceTranscriptStatus.DONE, transcript };
+
+ const entry = this.entries.get(name);
+
+ return {
+ errorReason: entry?.status === VoiceTranscriptStatus.ERROR ? entry.reason : null,
+ status: entry?.status ?? VoiceTranscriptStatus.IDLE
+ };
+ }
+
+ /** Follows a clip's transcript, and writes it into the message it went out with. */
+ track(name: string, transcript: Promise<string>): void {
+ this.entries.set(name, { status: VoiceTranscriptStatus.TRANSCRIBING });
+ this.recordedAt.set(name, Date.now());
+
+ transcript.then(
+ (text) => {
+ if (!this.entries.has(name) || this.edited.has(name)) return;
+
+ this.entries.set(name, { status: VoiceTranscriptStatus.DONE, transcript: text });
+ void this.applyToMessage(name, text);
+ },
+ (error: unknown) => {
+ if (!this.entries.has(name) || this.edited.has(name)) return;
+
+ console.warn('Voice message transcription failed:', error);
+ this.entries.set(name, {
+ message: error instanceof Error ? error.message : String(error),
+ reason: VoiceService.failureReason(error),
+ status: VoiceTranscriptStatus.ERROR
+ });
+ }
+ );
+ }
+
+ /** The transcript once it is there. */
+ transcriptOf(name: string): string | undefined {
+ const entry = this.entries.get(name);
+
+ return entry?.status === VoiceTranscriptStatus.DONE ? entry.transcript : undefined;
+ }
+
+ private async applyToMessage(name: string, transcript: string): Promise<void> {
+ for (const delay of [0, ...VOICE_TRANSCRIPT_APPLY_RETRY_DELAYS_MS]) {
+ if (delay) await new Promise((resolve) => setTimeout(resolve, delay));
+
+ // in its message now, which shows it from there
+ if (await this.writeTranscript(name, transcript)) return this.forget(name);
+ }
+ // not sent (still in the form, or dropped): a send reads it from here
+ }
+
+ private async writeTranscript(name: string, transcript: string): Promise<boolean> {
+ const active = (conversationsStore.activeMessages as DatabaseMessage[]).find((message) =>
+ message.extra?.some((extra) => isPendingClip(extra, name))
+ );
+
+ if (active) {
+ // plain objects: IndexedDB cannot clone the state proxies
+ const extra = withTranscript($state.snapshot(active.extra), name, transcript)!;
+
+ await DatabaseService.updateMessage(active.id, { extra });
+ conversationsStore.updateMessageAtIndex(conversationsStore.findMessageIndex(active.id), {
+ extra
+ });
+
+ return true;
+ }
+
+ // the chat was left before the transcript came back: the message is newer than the clip
+ const since = this.recordedAt.get(name) ?? Date.now();
+ const stored = (await DatabaseService.getMessagesSince(since)).find(
+ (message: DatabaseMessage) => message.extra?.some((extra) => isPendingClip(extra, name))
+ );
+
+ if (!stored) return false;
+
+ await DatabaseService.updateMessage(stored.id, {
+ extra: withTranscript(stored.extra, name, transcript)!
+ });
+
+ return true;
+ }
+}
+
+export const voiceTranscriptsStore = new VoiceTranscriptsStore();
diff --git a/ui/apps/chat/src/lib/stores/voice.svelte.ts b/ui/apps/chat/src/lib/stores/voice.svelte.ts
new file mode 100644
index 000000000..be4d5fc15
--- /dev/null
+++ b/ui/apps/chat/src/lib/stores/voice.svelte.ts
@@ -0,0 +1,117 @@
+/**
+ * voiceStore - Which model does what in voice input
+ *
+ * One place for what the microphone does with the conversation's model (a voice message,
+ * dictation, or nothing, see resolveVoiceMode) and for the model that transcribes
+ * dictation, so the chat form, the model picker and the Models manager agree on it: where
+ * models are listed, a badge says what kind of hearing model each is (roleOf) and a
+ * microphone which one transcribes dictation (isDictationModel), and picks it.
+ */
+
+import { SETTINGS_KEYS, VOICE_TRANSCRIPTION_MODEL_AUTO } from '#lib/constants/index.ts';
+import { VoiceInputMode, VoiceModelRole } from '#lib/enums/index.ts';
+import { serverStore } from '#lib/stores/llama-server.svelte.ts';
+// direct imports between stores, not via the barrel, to avoid circular deps
+import { modelsStore } from '#lib/stores/models/index.svelte.ts';
+import { settingsStore } from '#lib/stores/settings/index.svelte.ts';
+import { dictationModelFor, resolveVoiceMode, voiceModelRole } from '#lib/utils/voice-mode.ts';
+
+class VoiceStore {
+ /** A router with a model that hears: the microphone stays, so it can say where to pick one. */
+ readonly anyAudioModel = $derived.by(() => {
+ void modelsStore.props.cacheVersion;
+
+ return (
+ serverStore.isRouterMode && modelsStore.models.some((option) => this.hasAudio(option.model))
+ );
+ });
+ private readonly config = $derived(settingsStore.config);
+
+ private readonly transcriptionSetting = $derived(
+ String(this.config[SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL] ?? VOICE_TRANSCRIPTION_MODEL_AUTO)
+ );
+
+ /** The dictation model is picked automatically, not chosen in Settings > Voice. */
+ readonly dictationIsAutomatic = $derived(
+ this.transcriptionSetting.trim() === VOICE_TRANSCRIPTION_MODEL_AUTO
+ );
+
+ /** Router mode: what Automatic transcribes with when the chat model can't hear, a loaded model with audio input. */
+ readonly loadedAudioModel = $derived.by(() => {
+ void modelsStore.props.cacheVersion;
+
+ if (!serverStore.isRouterMode) return null;
+
+ return (
+ modelsStore.models.find(
+ (option) => this.hasAudio(option.model) && modelsStore.isModelLoaded(option.model)
+ )?.model ?? null
+ );
+ });
+
+ /** The model that transcribes dictation for chat models that can't hear, null when none can. */
+ readonly dictationModel = $derived.by(() => {
+ const singleModel = serverStore.isRouterMode ? null : modelsStore.activeModelId;
+
+ return dictationModelFor({
+ isRouter: serverStore.isRouterMode,
+ loadedAudioModel: this.loadedAudioModel,
+ singleModel,
+ singleModelHasAudio: this.hasAudio(singleModel),
+ transcriptionModel: this.transcriptionSetting,
+ transcriptionModelHasAudio: this.hasAudio(this.transcriptionSetting || null)
+ });
+ });
+
+ /** What the microphone does for the conversation's model. */
+ readonly resolution = $derived.by(() => {
+ const chatModel = modelsStore.activeModelId;
+
+ return resolveVoiceMode({
+ chatModel,
+ chatModelHasAudio: this.hasAudio(chatModel),
+ isRouter: serverStore.isRouterMode,
+ loadedAudioModel: this.loadedAudioModel,
+ setting: String(this.config[SETTINGS_KEYS.VOICE_INPUT_MODE] ?? VoiceInputMode.AUTO),
+ transcriptionModel: this.transcriptionSetting,
+ transcriptionModelHasAudio: this.hasAudio(this.transcriptionSetting || null)
+ });
+ });
+
+ /** Whether a model has audio input, as far as its props say. */
+ hasAudio(model: string | null): boolean {
+ void modelsStore.props.cacheVersion;
+
+ return model ? modelsStore.props.modelSupportsAudio(model) : false;
+ }
+
+ /** Whether a model is the one that transcribes dictation for chat models that can't hear. */
+ isDictationModel(model: string): boolean {
+ return this.dictationModel === model;
+ }
+
+ /** What kind of model with audio input a model is, null when it cannot hear. */
+ roleOf(model: string): VoiceModelRole | null {
+ return voiceModelRole(model, this.hasAudio(model));
+ }
+
+ /**
+ * What a click on a model's microphone does: makes it the model that transcribes
+ * dictation, or, on the one already chosen, hands the choice back to Automatic.
+ */
+ toggleDictation(model: string): void {
+ const chosen = !this.dictationIsAutomatic && this.transcriptionSetting.trim() === model;
+
+ settingsStore.updateConfig(
+ SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL,
+ chosen ? VOICE_TRANSCRIPTION_MODEL_AUTO : model
+ );
+ }
+
+ /** Makes a model the one that transcribes dictation (Settings > Voice > Transcription model). */
+ useForDictation(model: string): void {
+ settingsStore.updateConfig(SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL, model);
+ }
+}
+
+export const voiceStore = new VoiceStore();
diff --git a/ui/apps/chat/src/lib/types/chat.d.ts b/ui/apps/chat/src/lib/types/chat.d.ts
index eca0dbec8..a72a8dfff 100644
--- a/ui/apps/chat/src/lib/types/chat.d.ts
+++ b/ui/apps/chat/src/lib/types/chat.d.ts
@@ -48,6 +48,14 @@ export interface ChatUploadedFile {
};
isLoading?: boolean;
loadError?: string;
+ /** A voice message recorded in the chat form. */
+ voice?: ChatUploadedVoice;
+}
+
+/** What a voice message recorded in the chat form carries besides its WAV file. */
+export interface ChatUploadedVoice {
+ durationMs: number;
+ peaks: number[];
}
export interface ChatAttachmentDisplayItem {
diff --git a/ui/apps/chat/src/lib/types/database.d.ts b/ui/apps/chat/src/lib/types/database.d.ts
index 50f54dc0b..e47d60fff 100644
--- a/ui/apps/chat/src/lib/types/database.d.ts
+++ b/ui/apps/chat/src/lib/types/database.d.ts
@@ -38,6 +38,13 @@ export interface DatabaseMessageExtraAudioFile {
size?: number;
base64Data: string;
mimeType: string;
+ /** Recorded in the app as a voice message, as opposed to an uploaded audio file. */
+ voice?: true;
+ /** What was said, for reading; a model without audio input gets it in place of the audio. */
+ transcript?: string;
+ /** Normalised 0..1 waveform of the clip; computed from the audio when left out. */
+ peaks?: number[];
+ durationMs?: number;
}
export interface DatabaseMessageExtraVideoFile {
diff --git a/ui/apps/chat/src/lib/types/index.ts b/ui/apps/chat/src/lib/types/index.ts
index 68cd6e92f..bfed7569c 100644
--- a/ui/apps/chat/src/lib/types/index.ts
+++ b/ui/apps/chat/src/lib/types/index.ts
@@ -57,6 +57,7 @@ export type {
export type {
AttachmentMenuItem,
ChatUploadedFile,
+ ChatUploadedVoice,
ChatAttachmentDisplayItem,
ChatMessageSiblingInfo,
ChatMessageActions,
diff --git a/ui/apps/chat/src/lib/utils/audio-recording.ts b/ui/apps/chat/src/lib/utils/audio-recording.ts
index d733fc4ef..24570ceba 100644
--- a/ui/apps/chat/src/lib/utils/audio-recording.ts
+++ b/ui/apps/chat/src/lib/utils/audio-recording.ts
@@ -1,158 +1,5 @@
-import { MICROPHONE_UNAVAILABLE_MESSAGE } from '#lib/constants/index.ts';
import { MimeTypeAudio } from '#lib/enums/index.ts';
-/**
- * AudioRecorder - Browser-based audio recording with MediaRecorder API
- *
- * This class provides a complete audio recording solution using the browser's MediaRecorder API.
- * It handles microphone access, recording state management, and audio format optimization.
- *
- * **Features:**
- * - Automatic microphone permission handling
- * - Audio enhancement (echo cancellation, noise suppression, auto gain)
- * - Multiple format support with fallback (WAV, WebM, MP4, AAC)
- * - Real-time recording state tracking
- * - Proper cleanup and resource management
- */
-export class AudioRecorder {
- private audioChunks: Blob[] = [];
- private mediaRecorder: MediaRecorder | null = null;
- private recordingState: boolean = false;
- private stream: MediaStream | null = null;
-
- cancelRecording(): void {
- const recorder = this.mediaRecorder;
- const stream = this.stream;
-
- this.mediaRecorder = null;
- this.audioChunks = [];
- this.stream = null;
- this.recordingState = false;
-
- if (recorder && recorder.state !== 'inactive') {
- // Drop the original handlers so the pending stop event does not touch the instance
- recorder.onstop = null;
- recorder.onerror = null;
- recorder.stop();
- }
-
- if (stream) {
- for (const track of stream.getTracks()) {
- track.stop();
- }
- }
- }
-
- isRecording(): boolean {
- return this.recordingState;
- }
-
- async startRecording(): Promise<void> {
- try {
- this.stream = await navigator.mediaDevices.getUserMedia({
- audio: {
- autoGainControl: true,
- echoCancellation: true,
- noiseSuppression: true
- }
- });
-
- this.initializeRecorder(this.stream);
-
- this.audioChunks = [];
- // Start recording with a small timeslice to ensure we get data
- this.mediaRecorder!.start(100);
- this.recordingState = true;
- } catch (error) {
- console.error('Failed to start recording:', error);
-
- // the browser's own error, which says why: see describeMicrophoneError
- throw error;
- }
- }
-
- async stopRecording(): Promise<Blob> {
- return new Promise((resolve, reject) => {
- const recorder = this.mediaRecorder;
- const chunks = this.audioChunks;
- const stream = this.stream;
-
- if (!recorder || recorder.state === 'inactive') {
- reject(new Error('No active recording to stop'));
-
- return;
- }
-
- // Detach instance state right away so a new startRecording can take over without race
- this.mediaRecorder = null;
- this.audioChunks = [];
- this.stream = null;
- this.recordingState = false;
-
- recorder.onstop = () => {
- const audioBlob = new Blob(chunks, {
- type: recorder.mimeType || MimeTypeAudio.WAV
- });
-
- if (stream) {
- for (const track of stream.getTracks()) {
- track.stop();
- }
- }
-
- resolve(audioBlob);
- };
-
- recorder.onerror = (event) => {
- console.error('Recording error:', event);
-
- if (stream) {
- for (const track of stream.getTracks()) {
- track.stop();
- }
- }
-
- reject(new Error('Recording failed'));
- };
-
- recorder.stop();
- });
- }
-
- private initializeRecorder(stream: MediaStream): void {
- const options: MediaRecorderOptions = {};
-
- if (MediaRecorder.isTypeSupported(MimeTypeAudio.WAV)) {
- options.mimeType = MimeTypeAudio.WAV;
- } else if (MediaRecorder.isTypeSupported(MimeTypeAudio.WEBM_OPUS)) {
- options.mimeType = MimeTypeAudio.WEBM_OPUS;
- } else if (MediaRecorder.isTypeSupported(MimeTypeAudio.WEBM)) {
- options.mimeType = MimeTypeAudio.WEBM;
- } else if (MediaRecorder.isTypeSupported(MimeTypeAudio.MP4)) {
- options.mimeType = MimeTypeAudio.MP4;
- } else {
- console.warn('No preferred audio format supported, using default');
- }
-
- this.mediaRecorder = new MediaRecorder(stream, options);
-
- this.mediaRecorder.ondataavailable = (event) => {
- if (event.data.size > 0) {
- this.audioChunks.push(event.data);
- }
- };
-
- this.mediaRecorder.onstop = () => {
- this.recordingState = false;
- };
-
- this.mediaRecorder.onerror = (event) => {
- console.error('MediaRecorder error:', event);
- this.recordingState = false;
- };
- }
-}
-
export async function convertToWav(audioBlob: Blob): Promise<Blob> {
try {
if (audioBlob.type.includes('wav')) {
@@ -249,51 +96,3 @@ export function createAudioFile(audioBlob: Blob, filename?: string): File {
type: audioBlob.type
});
}
-
-/**
- * Check if audio recording is supported in the current browser
- * @returns True if MediaRecorder and getUserMedia are available
- */
-export function isAudioRecordingSupported(): boolean {
- return !!(
- typeof navigator !== 'undefined' &&
- navigator.mediaDevices &&
- typeof navigator.mediaDevices.getUserMedia === 'function' &&
- typeof window !== 'undefined' &&
- window.MediaRecorder
- );
-}
-
-/**
- * Why this page can't record at all, or null when it can: a reason no tap will fix. Browsers
- * only expose the microphone to a secure context, HTTPS or localhost, so a llama-server
- * opened over plain HTTP on the local network or over Tailscale can't record.
- */
-export function microphoneUnavailableReason(): string | null {
- if (typeof window === 'undefined') return null;
-
- if (!window.isSecureContext) return MICROPHONE_UNAVAILABLE_MESSAGE.insecure;
-
- if (!isAudioRecordingSupported()) return MICROPHONE_UNAVAILABLE_MESSAGE.unsupported;
-
- return null;
-}
-
-/** Why the microphone would not start, from the error the browser raised. */
-export function describeMicrophoneError(error: unknown): string {
- switch (error instanceof DOMException ? error.name : '') {
- case 'NotAllowedError':
- case 'SecurityError':
- return MICROPHONE_UNAVAILABLE_MESSAGE.denied;
- case 'NotFoundError':
- case 'OverconstrainedError':
- return MICROPHONE_UNAVAILABLE_MESSAGE.notFound;
- case 'AbortError':
- case 'NotReadableError':
- return MICROPHONE_UNAVAILABLE_MESSAGE.busy;
- default:
- return error instanceof Error && error.message
- ? `${MICROPHONE_UNAVAILABLE_MESSAGE.failed} ${error.message}`
- : MICROPHONE_UNAVAILABLE_MESSAGE.failed;
- }
-}
diff --git a/ui/apps/chat/src/lib/utils/browser-only.ts b/ui/apps/chat/src/lib/utils/browser-only.ts
index 64eedc140..7a1468e2a 100644
--- a/ui/apps/chat/src/lib/utils/browser-only.ts
+++ b/ui/apps/chat/src/lib/utils/browser-only.ts
@@ -6,15 +6,8 @@
* only in client-side code or components that are not server-rendered.
*/
-// Audio utilities (MediaRecorder API)
-export {
- AudioRecorder,
- convertToWav,
- createAudioFile,
- describeMicrophoneError,
- isAudioRecordingSupported,
- microphoneUnavailableReason
-} from './audio-recording';
+// Audio file utilities (Web Audio decoding); microphone support lives in @ggml-org/llama-voice
+export { convertToWav, createAudioFile } from './audio-recording';
// PDF processing utilities (pdfjs-dist with DOMMatrix)
export {
diff --git a/ui/apps/chat/src/lib/utils/clipboard.ts b/ui/apps/chat/src/lib/utils/clipboard.ts
index a5c75e873..c2b05985e 100644
--- a/ui/apps/chat/src/lib/utils/clipboard.ts
+++ b/ui/apps/chat/src/lib/utils/clipboard.ts
@@ -110,6 +110,16 @@ export function formatMessageForClipboard(
extras?: DatabaseMessageExtra[],
asPlainText: boolean = false
): string {
+ // what was said in a voice message reads as the message's text
+ const transcripts =
+ extras?.flatMap((extra) =>
+ extra.type === AttachmentType.AUDIO && extra.transcript ? [extra.transcript] : []
+ ) ?? [];
+
+ if (transcripts.length > 0) {
+ content = [...transcripts, content].filter((part) => part.trim()).join('\n\n');
+ }
+
// Filter text-like attachments (TEXT, LEGACY_CONTEXT, MCP_PROMPT, and MCP_RESOURCE types)
const textAttachments =
extras?.filter(
diff --git a/ui/apps/chat/src/lib/utils/convert-files-to-extra.ts b/ui/apps/chat/src/lib/utils/convert-files-to-extra.ts
index 43d29f43c..4e50b089c 100644
--- a/ui/apps/chat/src/lib/utils/convert-files-to-extra.ts
+++ b/ui/apps/chat/src/lib/utils/convert-files-to-extra.ts
@@ -6,6 +6,7 @@ import { SETTINGS_KEYS } from '#lib/constants/index.ts';
import { AttachmentType, FileTypeCategory, SpecialFileType } from '#lib/enums/index.ts';
import { modelsStore } from '#lib/stores/models/index.svelte.ts';
import { settingsStore } from '#lib/stores/settings/index.svelte.ts';
+import { voiceTranscriptsStore } from '#lib/stores/voice-transcripts.svelte.ts';
import type {
ChatUploadedFile,
DatabaseMessageExtra,
@@ -83,13 +84,21 @@ export async function parseFilesToMessageExtras(
// Process audio files (MP3 and WAV)
try {
const base64Data = await readFileAsBase64(file.file);
+ // a voice message keeps its waveform, and its transcript once it is there
+ const transcript = file.voice ? voiceTranscriptsStore.transcriptOf(file.name) : undefined;
extras.push({
base64Data: base64Data,
mimeType: file.type,
name: file.name,
size: file.size,
- type: AttachmentType.AUDIO
+ type: AttachmentType.AUDIO,
+ ...(file.voice && {
+ durationMs: file.voice.durationMs,
+ peaks: file.voice.peaks,
+ voice: true as const,
+ ...(transcript !== undefined && { transcript })
+ })
});
} catch (error) {
console.error(`Failed to process audio file ${file.name}:`, error);
diff --git a/ui/apps/chat/src/lib/utils/index.ts b/ui/apps/chat/src/lib/utils/index.ts
index 1f1d46844..c6fdbb0d7 100644
--- a/ui/apps/chat/src/lib/utils/index.ts
+++ b/ui/apps/chat/src/lib/utils/index.ts
@@ -387,3 +387,15 @@ export { groupModelFamilies, modelFamilyKey, type ModelFamilyGroup } from './mod
// Svelte actions
export { nearViewport } from './near-viewport';
+
+// Voice: what a recording becomes for the conversation's model
+export {
+ type DictationModelInput,
+ dictationModelFor,
+ isTranscriptionOnlyModel,
+ resolveVoiceMode,
+ voiceModeLabel,
+ type VoiceModeInput,
+ voiceModelRole,
+ type VoiceModeResolution
+} from './voice-mode';
diff --git a/ui/apps/chat/src/lib/utils/voice-attachment.ts b/ui/apps/chat/src/lib/utils/voice-attachment.ts
new file mode 100644
index 000000000..896381ac6
--- /dev/null
+++ b/ui/apps/chat/src/lib/utils/voice-attachment.ts
@@ -0,0 +1,109 @@
+import { base64ToBytes, computePeaksFromAudio } from '@ggml-org/llama-voice';
+import {
+ VOICE_FETCHABLE_URL_PATTERN,
+ VOICE_PEAKS_CACHE_TAIL_LENGTH,
+ VOICE_URL_PATTERN
+} from '#lib/constants/index.ts';
+import { AttachmentType, MimeTypeAudio } from '#lib/enums/index.ts';
+import type { ChatAttachmentDisplayItem, DatabaseMessageExtraAudioFile } from '#lib/types/index.ts';
+
+// peaks of clips stored without them, kept for the session only
+const peaksCache = new Map<string, Promise<number[]>>();
+
+/** What a list item needs to draw an audio attachment as a clip. */
+export interface VoiceAttachmentSource {
+ /** Base64 or a URL of the audio, or the file itself. */
+ audio: string | Blob | null;
+ durationMs?: number;
+ mimeType: string;
+ name: string;
+ peaks?: number[];
+ /** Recorded in the app, as opposed to an uploaded file. */
+ voice: boolean;
+ transcript?: string;
+}
+
+/** The clip behind an audio item of the form or of a message. */
+export function voiceAttachmentSource(item: ChatAttachmentDisplayItem): VoiceAttachmentSource {
+ const uploaded = item.uploadedFile;
+
+ if (uploaded) {
+ return {
+ audio: uploaded.preview ?? (uploaded.file.size > 0 ? uploaded.file : null),
+ durationMs: uploaded.voice?.durationMs,
+ mimeType: uploaded.type,
+ name: uploaded.name,
+ peaks: uploaded.voice?.peaks,
+ voice: !!uploaded.voice
+ };
+ }
+
+ const attachment =
+ item.attachment?.type === AttachmentType.AUDIO
+ ? (item.attachment as DatabaseMessageExtraAudioFile)
+ : null;
+
+ return {
+ audio: attachment?.base64Data || null,
+ durationMs: attachment?.durationMs,
+ mimeType: attachment?.mimeType ?? '',
+ name: item.name,
+ peaks: attachment?.peaks,
+ transcript: attachment?.transcript,
+ voice: attachment?.voice === true
+ };
+}
+
+/** A URL an <audio> can play: URLs as they are, base64 and files as object URLs; null for bad data. */
+export function audioUrl(
+ audio: string | Blob,
+ mimeType: string
+): { url: string; revoke: boolean } | null {
+ // a data, blob or http URL plays as it is
+ if (typeof audio === 'string' && VOICE_URL_PATTERN.test(audio)) {
+ return { revoke: false, url: audio };
+ }
+
+ if (typeof audio !== 'string') return { revoke: true, url: URL.createObjectURL(audio) };
+
+ let bytes: Uint8Array<ArrayBuffer>;
+
+ try {
+ bytes = base64ToBytes(audio);
+ } catch {
+ // not base64: nothing to play
+ return null;
+ }
+
+ return {
+ revoke: true,
+ url: URL.createObjectURL(new Blob([bytes], { type: mimeType || MimeTypeAudio.WAV }))
+ };
+}
+
+/** Peaks of a clip stored without them, computed once per session. */
+export function lazyPeaks(name: string, audio: string | Blob): Promise<number[]> {
+ const key =
+ typeof audio === 'string'
+ ? `${name}:${audio.length}:${audio.slice(-VOICE_PEAKS_CACHE_TAIL_LENGTH)}`
+ : `${name}:${audio.size}`;
+
+ let peaks = peaksCache.get(key);
+
+ if (!peaks) {
+ const isUrl = typeof audio === 'string' && VOICE_FETCHABLE_URL_PATTERN.test(audio);
+ const bytes = isUrl ? fetch(audio as string).then((r) => r.blob()) : Promise.resolve(audio);
+
+ peaks = bytes
+ .then((source) => computePeaksFromAudio(source))
+ .catch((error) => {
+ console.warn(`Could not draw the waveform of ${name}:`, error);
+ peaksCache.delete(key);
+
+ return [];
+ });
+ peaksCache.set(key, peaks);
+ }
+
+ return peaks;
+}
diff --git a/ui/apps/chat/src/lib/utils/voice-mode.ts b/ui/apps/chat/src/lib/utils/voice-mode.ts
new file mode 100644
index 000000000..16250806c
--- /dev/null
+++ b/ui/apps/chat/src/lib/utils/voice-mode.ts
@@ -0,0 +1,128 @@
+import {
+ VOICE_MODE_MESSAGE,
+ VOICE_TRANSCRIPTION_MODEL_AUTO,
+ VOICE_TRANSCRIPTION_ONLY_PATTERN
+} from '#lib/constants/voice.constants.ts';
+import { VoiceInputMode, VoiceMode, VoiceModelRole } from '#lib/enums/index.ts';
+
+export interface VoiceModeInput extends Omit<
+ DictationModelInput,
+ 'singleModel' | 'singleModelHasAudio'
+> {
+ /** The Voice input setting. */
+ setting: VoiceInputMode | string;
+ /** The conversation's effective model, null when none is picked. */
+ chatModel: string | null;
+ chatModelHasAudio: boolean;
+}
+
+export type VoiceModeResolution =
+ | {
+ mode: VoiceMode.VOICE_MESSAGE;
+ /** The chat model: it writes the transcript too, so a router swaps no model. */
+ transcriptionModel: string | null;
+ separateModel: false;
+ }
+ | {
+ mode: VoiceMode.DICTATION;
+ transcriptionModel: string | null;
+ /** A router model other than the chat model, which transcribing may load. */
+ separateModel: boolean;
+ }
+ | { mode: VoiceMode.UNAVAILABLE; reason: string };
+
+/**
+ * What a recording becomes for the conversation's model. Automatic sends the voice to a
+ * chat model with audio input and dictates otherwise, also with a speech-to-text model
+ * picked for the conversation: it writes down what is said and has nothing to answer a
+ * voice message with. Dictation transcribes with the chat model when it can hear, else
+ * (router mode) with the chosen transcription model or, on Automatic, a model with audio
+ * input that is already loaded.
+ */
+export function resolveVoiceMode(input: VoiceModeInput): VoiceModeResolution {
+ const { chatModel, chatModelHasAudio, isRouter, setting } = input;
+ const sendsVoice =
+ setting !== VoiceInputMode.DICTATION && !(chatModel && isTranscriptionOnlyModel(chatModel));
+
+ if (sendsVoice && chatModelHasAudio) {
+ return { mode: VoiceMode.VOICE_MESSAGE, separateModel: false, transcriptionModel: chatModel };
+ }
+
+ if (chatModelHasAudio) {
+ return { mode: VoiceMode.DICTATION, separateModel: false, transcriptionModel: chatModel };
+ }
+
+ // a single-model server has no other model to transcribe with
+ if (!isRouter) return { mode: VoiceMode.UNAVAILABLE, reason: VOICE_MODE_MESSAGE.unavailable };
+
+ const transcriptionModel = dictationModelFor(input);
+
+ if (transcriptionModel) {
+ return {
+ mode: VoiceMode.DICTATION,
+ // on Automatic it is already loaded, so transcribing swaps no model in or out
+ separateModel:
+ input.transcriptionModel.trim() !== VOICE_TRANSCRIPTION_MODEL_AUTO &&
+ transcriptionModel !== chatModel,
+ transcriptionModel
+ };
+ }
+
+ return { mode: VoiceMode.UNAVAILABLE, reason: VOICE_MODE_MESSAGE.unavailable };
+}
+
+/** What the microphone button says it will do, or why it can't. */
+export function voiceModeLabel(resolution: VoiceModeResolution): string {
+ switch (resolution.mode) {
+ case VoiceMode.VOICE_MESSAGE:
+ return VOICE_MODE_MESSAGE.voiceMessage;
+ case VoiceMode.DICTATION:
+ return VOICE_MODE_MESSAGE.dictate;
+ default:
+ return resolution.reason;
+ }
+}
+
+/** Whether a model is a speech-to-text one, by its name: for dictation, not for chat. */
+export function isTranscriptionOnlyModel(modelId: string): boolean {
+ return VOICE_TRANSCRIPTION_ONLY_PATTERN.test(modelId);
+}
+
+export interface DictationModelInput {
+ isRouter: boolean;
+ /** The Transcription model setting; VOICE_TRANSCRIPTION_MODEL_AUTO picks one. */
+ transcriptionModel: string;
+ transcriptionModelHasAudio: boolean;
+ /** Router mode: a loaded model with audio input, which Automatic falls back to. */
+ loadedAudioModel?: string | null;
+ /** Single-model mode: the server's model, and whether it hears. */
+ singleModel?: string | null;
+ singleModelHasAudio?: boolean;
+}
+
+/**
+ * The model that transcribes dictation for chat models that cannot hear, as
+ * resolveVoiceMode() picks it, or null when none can: the chosen transcription model, or on
+ * Automatic a loaded model with audio input; a single-model server's model when it hears.
+ */
+export function dictationModelFor(input: DictationModelInput): string | null {
+ if (!input.isRouter) return input.singleModelHasAudio ? (input.singleModel ?? null) : null;
+
+ const chosen = input.transcriptionModel.trim();
+
+ if (chosen !== VOICE_TRANSCRIPTION_MODEL_AUTO) {
+ return input.transcriptionModelHasAudio ? chosen : null;
+ }
+
+ return input.loadedAudioModel ?? null;
+}
+
+/**
+ * What kind of model with audio input a model is: a speech-to-text model, a chat model
+ * that hears, or none (it cannot hear).
+ */
+export function voiceModelRole(modelId: string, hasAudio: boolean): VoiceModelRole | null {
+ if (!hasAudio) return null;
+
+ return isTranscriptionOnlyModel(modelId) ? VoiceModelRole.TRANSCRIPTION : VoiceModelRole.VOICE;
+}
diff --git a/ui/apps/chat/tests/client/chat-form-voice.svelte.test.ts b/ui/apps/chat/tests/client/chat-form-voice.svelte.test.ts
new file mode 100644
index 000000000..a03f8248c
--- /dev/null
+++ b/ui/apps/chat/tests/client/chat-form-voice.svelte.test.ts
@@ -0,0 +1,573 @@
+// Guards voice input in the chat form, with a fake recorder in place of the microphone:
+// what the microphone offers per model, the recording row, dictated text written into the
+// message around what the user types, and a voice message whose transcript comes later.
+
+import { modelServerProps, seedStores } from '../stories/chat/form/fixtures';
+import { deferred, FakeVoiceRecorder } from '../stories/voice/fake-voice-recorder.svelte';
+import ChatFormVoiceTestWrapper from './components/ChatFormVoiceTestWrapper.svelte';
+import { EditorView } from '@codemirror/view';
+import { VoiceTranscriptStatus } from '@ggml-org/llama-ui/voice';
+import { VoiceRecorderStatus } from '@ggml-org/llama-voice';
+import { SETTINGS_KEYS, VOICE_MODE_MESSAGE } from '#lib/constants/index.ts';
+import { ServerRole, VoiceInputMode } from '#lib/enums/index.ts';
+import { voiceTranscriptsStore } from '#lib/stores/index.ts';
+import { parseFilesToMessageExtras } from '#lib/utils/browser-only.ts';
+import { tick } from 'svelte';
+import { afterEach, describe, expect, it, vi } from 'vitest';
+import { userEvent } from 'vitest/browser';
+import { render } from 'vitest-browser-svelte';
+
+let restore: (() => void) | undefined;
+
+afterEach(() => {
+ restore?.();
+ restore = undefined;
+});
+
+function seed(
+ audio: boolean,
+ inputMode: VoiceInputMode = VoiceInputMode.AUTO,
+ config: Record<string, unknown> = {}
+) {
+ restore = seedStores({
+ config: {
+ ...config,
+ autoMicOnEmpty: true,
+ [SETTINGS_KEYS.VOICE_INPUT_MODE]: inputMode,
+ [SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES]: true
+ },
+ serverProps: modelServerProps({ modalities: { audio, video: false, vision: false } }),
+ serverRole: ServerRole.MODEL
+ });
+}
+
+async function renderForm({ isLoading = false } = {}) {
+ const recorder = new FakeVoiceRecorder();
+ const state = $state({ files: [] as ChatUploadedFile[], sent: 0, value: '' });
+ const screen = render(ChatFormVoiceTestWrapper, { isLoading, state, voiceRecorder: recorder });
+
+ await tick();
+
+ const content = screen.container.querySelector<HTMLElement>('.cm-content');
+ const editor = content && EditorView.findFromDOM(content);
+
+ if (!editor) throw new Error('editor not rendered');
+
+ // an edit as typing makes it: reported to the form like a keystroke
+ const type = (at: number, text: string) =>
+ editor.dispatch({ changes: { from: at, insert: text }, userEvent: 'input.type' });
+ const recordingRow = () => screen.container.querySelector('[data-slot="voice-recording-bar"]');
+
+ return { editor, recorder, recordingRow, screen, state, type };
+}
+
+// the tooltip's trigger wraps the button and takes its name, so the button is the inner one
+describe('the microphone', () => {
+ it('sends a voice message to a model with audio input', async () => {
+ seed(true);
+
+ const { screen } = await renderForm();
+
+ await expect
+ .element(screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last())
+ .toBeEnabled();
+ });
+
+ it('stands before the send button on an empty message, a quieter button than it', async () => {
+ seed(true);
+
+ const { screen } = await renderForm();
+ // the send button keeps its place, waiting for a message
+ const send = screen.getByRole('button', { exact: true, name: 'Send' });
+
+ await expect.element(send).toBeDisabled();
+
+ const mic = screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last();
+
+ await expect.element(mic).toBeEnabled();
+ expect(mic.element().getAttribute('data-variant')).toBe('tertiary');
+ expect(
+ mic.element().compareDocumentPosition(send.element()) & Node.DOCUMENT_POSITION_FOLLOWING
+ ).toBeTruthy();
+ });
+
+ it('dictates when the user always dictates', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { screen } = await renderForm();
+
+ await expect
+ .element(screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last())
+ .toBeEnabled();
+ });
+
+ it('stays beside the send button once the message has text, and dictates more', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state, type } = await renderForm();
+
+ type(0, 'Hello');
+ await expect.poll(() => state.value).toBe('Hello');
+ await expect.element(screen.getByRole('button', { name: 'Send' })).toBeEnabled();
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('there');
+ await expect.poll(() => state.value).toContain('there');
+ expect(state.value).toContain('Hello');
+ });
+
+ it('stays after the stop button while a reply streams, looking like it', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen } = await renderForm({ isLoading: true });
+
+ await expect.element(screen.getByRole('button', { exact: true, name: 'Stop' })).toBeEnabled();
+
+ const mic = screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last();
+
+ await expect.element(mic).toBeEnabled();
+
+ // the last button of the row, filled like the stop button before it
+ const stop = screen.getByRole('button', { exact: true, name: 'Stop' }).element();
+ const micButton = mic.element();
+
+ expect(stop.compareDocumentPosition(micButton) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
+ expect(micButton.getAttribute('data-variant')).toBe(stop.getAttribute('data-variant'));
+
+ await mic.click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ });
+
+ it('leaves the send button on a single-model server without audio input', async () => {
+ seed(false);
+
+ const { screen } = await renderForm();
+
+ await expect.element(screen.getByRole('button', { name: 'Send' })).toBeInTheDocument();
+ expect(screen.container.innerHTML).not.toContain(VOICE_MODE_MESSAGE.voiceMessage);
+ });
+
+ it('does not stand beside the send button without voice input', async () => {
+ seed(false);
+
+ const { screen, state, type } = await renderForm();
+
+ type(0, 'Hello');
+ await expect.poll(() => state.value).toBe('Hello');
+ await expect.element(screen.getByRole('button', { name: 'Send' })).toBeEnabled();
+ expect(screen.container.innerHTML).not.toContain(VOICE_MODE_MESSAGE.voiceMessage);
+ expect(screen.container.innerHTML).not.toContain(VOICE_MODE_MESSAGE.dictate);
+ });
+});
+
+describe('a pointer on the microphone', () => {
+ /** Lets the page use the microphone without asking, with a live stream as the device. */
+ function grantMicrophone() {
+ const context = new AudioContext();
+ const stream = context.createMediaStreamDestination().stream;
+ const query = vi
+ .spyOn(navigator.permissions, 'query')
+ .mockResolvedValue({ state: 'granted' } as PermissionStatus);
+ const getUserMedia = vi.spyOn(navigator.mediaDevices, 'getUserMedia').mockResolvedValue(stream);
+ const previous = restore;
+
+ restore = () => {
+ previous?.();
+ query.mockRestore();
+ getUserMedia.mockRestore();
+ void context.close();
+ };
+
+ return { getUserMedia, stream };
+ }
+
+ it('opens the microphone ahead of the press, and the recording takes it', async () => {
+ seed(true);
+
+ const { getUserMedia, stream } = grantMicrophone();
+ const { recorder, screen } = await renderForm();
+ const mic = screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last();
+
+ await mic.hover();
+ await expect.poll(() => getUserMedia.mock.calls.length).toBe(1);
+
+ await mic.click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect((recorder.options?.capture as { stream?: MediaStream }).stream).toBe(stream);
+ // handed over, not asked for again
+ expect(getUserMedia).toHaveBeenCalledOnce();
+ });
+
+ it('lets the microphone go when the pointer leaves without a press', async () => {
+ seed(true);
+
+ const { getUserMedia, stream } = grantMicrophone();
+ const { screen } = await renderForm();
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().hover();
+ await expect.poll(() => getUserMedia.mock.calls.length).toBe(1);
+
+ await userEvent.unhover(
+ screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last()
+ );
+ await expect.poll(() => stream.getAudioTracks()[0].readyState, { timeout: 4000 }).toBe('ended');
+ });
+
+ it('lets the microphone go when the recording is cancelled while it opens', async () => {
+ seed(true);
+
+ const { getUserMedia, stream } = grantMicrophone();
+ const opening = deferred<MediaStream>();
+
+ getUserMedia.mockReturnValueOnce(opening.promise);
+
+ const { recorder, screen } = await renderForm();
+ const mic = screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last();
+
+ await mic.hover();
+ await expect.poll(() => getUserMedia.mock.calls.length).toBe(1);
+ await mic.click();
+ await userEvent.keyboard('{Escape}');
+
+ opening.resolve(stream);
+
+ await expect.poll(() => stream.getAudioTracks()[0].readyState).toBe('ended');
+ expect(recorder.status).toBe(VoiceRecorderStatus.IDLE);
+ });
+});
+
+describe('while a reply streams', () => {
+ it('offers to send what is dictated as soon as its first words are in the message', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state } = await renderForm({ isLoading: true });
+
+ recorder.transcript = Promise.resolve('And one more thing');
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+
+ // nothing said yet: nothing to send
+ expect(screen.container.innerHTML).not.toContain('Stop and send');
+
+ recorder.say('And one more thing');
+ await expect.poll(() => state.value).toBe('And one more thing');
+
+ await screen.getByRole('button', { name: 'Stop and send' }).click();
+ await expect.poll(() => state.sent).toBe(1);
+ });
+
+ it('offers to send a voice message once its live text shows', async () => {
+ seed(true);
+
+ const { recorder, screen } = await renderForm({ isLoading: true });
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(screen.container.innerHTML).not.toContain('Stop and send');
+
+ recorder.say('What about tomorrow');
+ await expect.element(screen.getByRole('button', { name: 'Stop and send' })).toBeInTheDocument();
+ });
+
+ it('keeps the microphone beside the send button once a voice message is attached', async () => {
+ seed(true);
+
+ const { recorder, screen, state } = await renderForm({ isLoading: true });
+
+ recorder.transcript = Promise.resolve('What about tomorrow');
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('What about tomorrow');
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+ await expect.poll(() => state.files.length).toBe(1);
+
+ await expect.element(screen.getByRole('button', { exact: true, name: 'Send' })).toBeEnabled();
+ await expect
+ .element(screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last())
+ .toBeEnabled();
+ });
+});
+
+describe('recording', () => {
+ it('asks for the raw microphone, and for the processing that is switched on', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const raw = await renderForm();
+
+ await raw.screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => raw.recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(raw.recorder.options?.capture).not.toHaveProperty('constraints');
+ raw.screen.unmount();
+ restore?.();
+
+ seed(true, VoiceInputMode.DICTATION, {
+ [SETTINGS_KEYS.VOICE_ECHO_CANCELLATION]: true,
+ [SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION]: true
+ });
+
+ const filtered = await renderForm();
+
+ await filtered.screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => filtered.recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(filtered.recorder.options?.capture).toEqual({
+ boost: true,
+ constraints: { echoCancellation: true, noiseSuppression: true }
+ });
+ });
+
+ it('boosts quiet speech unless the setting is off', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const boosted = await renderForm();
+
+ await boosted.screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => boosted.recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(boosted.recorder.options?.capture).toMatchObject({ boost: true });
+ boosted.screen.unmount();
+ restore?.();
+
+ seed(true, VoiceInputMode.DICTATION, { [SETTINGS_KEYS.VOICE_BOOST]: false });
+
+ const plain = await renderForm();
+
+ await plain.screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => plain.recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(plain.recorder.options?.capture).toMatchObject({ boost: false });
+ });
+
+ it('says when the microphone is too quiet without moving the form, and stops once it is not', async () => {
+ seed(true);
+
+ const { recorder, screen } = await renderForm();
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ expect(screen.container.innerHTML).not.toContain(VOICE_MODE_MESSAGE.tooQuiet);
+
+ recorder.setQuiet(true);
+ // inside the recording row, so the form keeps its height: read out, and shown above it
+ await expect
+ .element(screen.getByRole('status').filter({ hasText: VOICE_MODE_MESSAGE.tooQuiet }))
+ .toBeInTheDocument();
+ await expect
+ .poll(() => document.querySelector('[data-slot="tooltip-content"]')?.textContent ?? '')
+ .toContain(VOICE_MODE_MESSAGE.tooQuiet);
+
+ recorder.setQuiet(false);
+ await expect
+ .poll(() => screen.container.querySelector('[data-slot="voice-recording-bar-warning"]'))
+ .toBeNull();
+ });
+
+ it('shows the live waveform with cancel and stop, and Escape drops it', async () => {
+ seed(true);
+
+ const { recorder, recordingRow, screen } = await renderForm();
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(recordingRow).toBeTruthy();
+
+ recorder.setLevels(Array.from({ length: 48 }, (_, i) => (i % 5) / 5));
+ await tick();
+
+ expect(recordingRow()?.querySelector('[data-slot="voice-waveform-live"]')).toBeTruthy();
+ await expect
+ .element(screen.getByRole('button', { name: 'Cancel recording' }))
+ .toBeInTheDocument();
+ await expect
+ .element(screen.getByRole('button', { name: 'Stop recording' }).last())
+ .toBeEnabled();
+
+ await userEvent.keyboard('{Escape}');
+ await expect.poll(recordingRow).toBeNull();
+ expect(recorder.cancelled).toBe(1);
+ });
+});
+
+describe('dictation', () => {
+ it('writes the live text into the message, keeping what is typed around it', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state, type } = await renderForm();
+ const transcript = deferred<string>();
+
+ recorder.transcript = transcript.promise;
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+
+ recorder.say('one');
+ await expect.poll(() => state.value).toBe('one');
+
+ // typed before and after the dictated text while it streams
+ type(0, 'Note: ');
+ await expect.poll(() => state.value).toBe('Note: one');
+ type(state.value.length, ' (end)');
+ recorder.say('one two');
+ await expect.poll(() => state.value).toBe('Note: one two (end)');
+
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+ transcript.resolve('One, two.');
+
+ await expect.poll(() => state.value).toBe('Note: One, two. (end)');
+ expect(state.files).toHaveLength(0);
+ });
+
+ it('waits for the text after recording when none streamed, then inserts it', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state } = await renderForm();
+ const transcript = deferred<string>();
+
+ recorder.transcript = transcript.promise;
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+
+ await expect.element(screen.getByText(VOICE_MODE_MESSAGE.transcribing)).toBeInTheDocument();
+
+ transcript.resolve('Hello there');
+ await expect.poll(() => state.value).toBe('Hello there');
+ expect(state.files).toHaveLength(0);
+ });
+
+ it('says the live text is being refined, and a send waits for the refined text', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state } = await renderForm();
+ const transcript = deferred<string>();
+
+ recorder.transcript = transcript.promise;
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('I went. To the store.');
+ await expect.poll(() => state.value).toBe('I went. To the store.');
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+
+ await expect.element(screen.getByText(VOICE_MODE_MESSAGE.refining)).toBeInTheDocument();
+
+ await screen.getByRole('button', { name: 'Send' }).click();
+ await expect
+ .element(screen.getByText(VOICE_MODE_MESSAGE.sendAfterRefining))
+ .toBeInTheDocument();
+ expect(state.sent).toBe(0);
+
+ transcript.resolve('I went to the store.');
+ await expect.poll(() => state.sent).toBe(1);
+ expect(state.value).toBe('I went to the store.');
+ expect(screen.container.querySelector('[data-slot="voice-refining"]')).toBeNull();
+ });
+
+ it('stops and sends the dictated text in one go', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state } = await renderForm();
+
+ recorder.transcript = Promise.resolve('Hello there');
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('Hello there');
+
+ await screen.getByRole('button', { name: 'Stop and send' }).click();
+ await expect.poll(() => state.sent).toBe(1);
+ expect(state.value).toBe('Hello there');
+ });
+
+ it('takes the dictated text back out on cancel', async () => {
+ seed(true, VoiceInputMode.DICTATION);
+
+ const { recorder, screen, state } = await renderForm();
+
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.dictate }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('never mind');
+ await expect.poll(() => state.value).toBe('never mind');
+
+ await screen.getByRole('button', { name: 'Cancel recording' }).click();
+ await expect.poll(() => state.value).toBe('');
+ });
+});
+
+describe('a voice message', () => {
+ it('turns its live text into text to edit on focus, which goes with the clip on stop', async () => {
+ seed(true);
+
+ const { editor, recorder, screen, state } = await renderForm();
+
+ recorder.transcript = Promise.resolve('model transcript');
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+ recorder.say('Hello wrold');
+
+ editor.focus();
+ await expect.poll(() => state.value).toBe('Hello wrold');
+
+ // fixed by hand, and what is said next follows it
+ editor.dispatch({ changes: { from: 7, insert: 'or', to: 9 }, userEvent: 'input.type' });
+ await expect.poll(() => state.value).toBe('Hello world');
+ recorder.say('Hello wrold, how are you');
+ await expect.poll(() => state.value).toBe('Hello world, how are you');
+
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+ await expect.poll(() => state.files.length).toBe(1);
+ expect(state.value).toBe('');
+ expect(voiceTranscriptsStore.transcriptOf(state.files[0].name)).toBe(
+ 'Hello world, how are you'
+ );
+ });
+
+ it('is attached as a voice clip that gets its transcript later', async () => {
+ seed(true);
+
+ const { recorder, screen, state } = await renderForm();
+ const transcript = deferred<string>();
+
+ recorder.transcript = transcript.promise;
+ await screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last().click();
+ await expect.poll(() => recorder.status).toBe(VoiceRecorderStatus.RECORDING);
+
+ // streamed text shows in the empty message box as its placeholder, not in the message
+ const placeholder = () => screen.container.querySelector('.cm-placeholder')?.textContent;
+
+ recorder.say('What is the weather on Mars');
+ await expect.poll(placeholder).toBe('What is the weather on Mars');
+ expect(state.value).toBe('');
+
+ await screen.getByRole('button', { name: 'Stop recording' }).last().click();
+ await expect.poll(() => state.files.length).toBe(1);
+ // the text now goes with the clip
+ await expect.poll(placeholder).not.toBe('What is the weather on Mars');
+
+ const [file] = state.files;
+
+ expect(file.voice).toMatchObject({ durationMs: 2400 });
+ expect(file.voice?.peaks).toHaveLength(64);
+ expect(state.value).toBe('');
+ expect(voiceTranscriptsStore.get(file.name)?.status).toBe(VoiceTranscriptStatus.TRANSCRIBING);
+
+ // drawn as a voice chip, its transcript on the way
+ const reveal = () =>
+ screen.container.querySelector('[data-slot="chat-attachment-voice"] [data-status]');
+
+ await expect
+ .poll(() => reveal()?.getAttribute('data-status'))
+ .toBe(VoiceTranscriptStatus.TRANSCRIBING);
+
+ transcript.resolve('What is the weather on Mars?');
+ await expect
+ .poll(() => voiceTranscriptsStore.transcriptOf(file.name))
+ .toBe('What is the weather on Mars?');
+
+ await expect.poll(() => reveal()?.getAttribute('data-status')).toBe(VoiceTranscriptStatus.DONE);
+
+ const { extras } = await parseFilesToMessageExtras($state.snapshot(state.files));
+
+ expect(extras[0]).toMatchObject({
+ durationMs: 2400,
+ transcript: 'What is the weather on Mars?',
+ voice: true
+ });
+
+ voiceTranscriptsStore.forget(file.name);
+ });
+});
diff --git a/ui/apps/chat/tests/client/components/ChatFormActionRecordWrapper.svelte b/ui/apps/chat/tests/client/components/ChatFormActionRecordWrapper.svelte
index cdc4a46ed..d836cd6dd 100644
--- a/ui/apps/chat/tests/client/components/ChatFormActionRecordWrapper.svelte
+++ b/ui/apps/chat/tests/client/components/ChatFormActionRecordWrapper.svelte
@@ -3,12 +3,16 @@
import ChatFormActionRecord from '#lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte';
let {
- hasAudioModality = true,
micWarning = null,
- onMicClick
- }: { hasAudioModality?: boolean; micWarning?: string | null; onMicClick?: () => void } = $props();
+ onMicClick,
+ unavailableReason = null
+ }: {
+ micWarning?: string | null;
+ onMicClick?: () => void;
+ unavailableReason?: string | null;
+ } = $props();
</script>
<Tooltip.Provider>
- <ChatFormActionRecord {hasAudioModality} {micWarning} {onMicClick} />
+ <ChatFormActionRecord {micWarning} {onMicClick} {unavailableReason} />
</Tooltip.Provider>
diff --git a/ui/apps/chat/tests/client/components/ChatFormVoiceTestWrapper.svelte b/ui/apps/chat/tests/client/components/ChatFormVoiceTestWrapper.svelte
new file mode 100644
index 000000000..34dfe4ce7
--- /dev/null
+++ b/ui/apps/chat/tests/client/components/ChatFormVoiceTestWrapper.svelte
@@ -0,0 +1,25 @@
+<script lang="ts">
+ import * as Tooltip from '@ggml-org/llama-ui/ui/tooltip';
+ import ChatForm from '#lib/components/app/chat/ChatForm/ChatForm.svelte';
+ import type { VoiceRecorderLike } from '#lib/hooks/use-chat-form-voice.svelte.ts';
+
+ interface Props {
+ /** A reply is streaming: the form offers stop in place of send. */
+ isLoading?: boolean;
+ /** The form's message and files, kept in step with it for the test to read, and how often it sent. */
+ state: { files: ChatUploadedFile[]; sent?: number; value: string };
+ voiceRecorder: VoiceRecorderLike;
+ }
+
+ let { isLoading = false, state, voiceRecorder }: Props = $props();
+</script>
+
+<Tooltip.Provider>
+ <ChatForm
+ bind:uploadedFiles={state.files}
+ bind:value={state.value}
+ {isLoading}
+ onSubmit={() => (state.sent = (state.sent ?? 0) + 1)}
+ {voiceRecorder}
+ />
+</Tooltip.Provider>
diff --git a/ui/apps/chat/tests/client/components/ModelsSelectorListWrapper.svelte b/ui/apps/chat/tests/client/components/ModelsSelectorListWrapper.svelte
new file mode 100644
index 000000000..c6435db54
--- /dev/null
+++ b/ui/apps/chat/tests/client/components/ModelsSelectorListWrapper.svelte
@@ -0,0 +1,18 @@
+<script lang="ts">
+ import * as Tooltip from '@ggml-org/llama-ui/ui/tooltip';
+ import ModelsSelectorList from '#lib/components/app/models/ModelsSelector/ModelsSelectorList.svelte';
+ import type { GroupedModelOptions, ModelItem } from '#lib/utils/index.ts';
+
+ interface Props {
+ currentModel: string | null;
+ groups: GroupedModelOptions;
+ loaded?: ModelItem[];
+ }
+
+ let { currentModel, groups, loaded = [] }: Props = $props();
+</script>
+
+<!-- The rows mount tooltips, which need the provider +layout.svelte supplies. -->
+<Tooltip.Provider>
+ <ModelsSelectorList activeId={null} {currentModel} {groups} {loaded} onSelect={() => {}} />
+</Tooltip.Provider>
diff --git a/ui/apps/chat/tests/client/microphone-unavailable.svelte.test.ts b/ui/apps/chat/tests/client/microphone-unavailable.svelte.test.ts
index 361506341..92495dc3d 100644
--- a/ui/apps/chat/tests/client/microphone-unavailable.svelte.test.ts
+++ b/ui/apps/chat/tests/client/microphone-unavailable.svelte.test.ts
@@ -4,8 +4,12 @@
// while staying tappable, so a tap on a phone, which shows no tooltip, can explain it.
import ChatFormActionRecordWrapper from './components/ChatFormActionRecordWrapper.svelte';
-import { MICROPHONE_UNAVAILABLE_MESSAGE } from '#lib/constants/index.ts';
-import { describeMicrophoneError, microphoneUnavailableReason } from '#lib/utils/browser-only.ts';
+import {
+ describeMicrophoneError,
+ microphoneUnavailableReason,
+ VOICE_MICROPHONE_MESSAGE
+} from '@ggml-org/llama-voice';
+import { VOICE_MODE_MESSAGE } from '#lib/constants/index.ts';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { render } from 'vitest-browser-svelte';
@@ -17,14 +21,14 @@ describe('microphoneUnavailableReason', () => {
it('names HTTPS when the page is not a secure context', () => {
vi.stubGlobal('isSecureContext', false);
- expect(microphoneUnavailableReason()).toBe(MICROPHONE_UNAVAILABLE_MESSAGE.insecure);
+ expect(microphoneUnavailableReason()).toBe(VOICE_MICROPHONE_MESSAGE.insecure);
});
- it('names the browser when it has no recorder', () => {
+ it('names the browser when it cannot capture audio', () => {
vi.stubGlobal('isSecureContext', true);
- vi.stubGlobal('MediaRecorder', undefined);
+ vi.stubGlobal('AudioContext', undefined);
- expect(microphoneUnavailableReason()).toBe(MICROPHONE_UNAVAILABLE_MESSAGE.unsupported);
+ expect(microphoneUnavailableReason()).toBe(VOICE_MICROPHONE_MESSAGE.unsupported);
});
it('has nothing to say when the page can record', () => {
@@ -36,17 +40,17 @@ describe('microphoneUnavailableReason', () => {
describe('describeMicrophoneError', () => {
it.each([
- ['NotAllowedError', MICROPHONE_UNAVAILABLE_MESSAGE.denied],
- ['SecurityError', MICROPHONE_UNAVAILABLE_MESSAGE.denied],
- ['NotFoundError', MICROPHONE_UNAVAILABLE_MESSAGE.notFound],
- ['NotReadableError', MICROPHONE_UNAVAILABLE_MESSAGE.busy]
+ ['NotAllowedError', VOICE_MICROPHONE_MESSAGE.denied],
+ ['SecurityError', VOICE_MICROPHONE_MESSAGE.denied],
+ ['NotFoundError', VOICE_MICROPHONE_MESSAGE.notFound],
+ ['NotReadableError', VOICE_MICROPHONE_MESSAGE.busy]
])('names a %s', (name, message) => {
expect(describeMicrophoneError(new DOMException('', name))).toBe(message);
});
it("keeps an unknown error's own message", () => {
expect(describeMicrophoneError(new Error('boom'))).toBe(
- `${MICROPHONE_UNAVAILABLE_MESSAGE.failed} boom`
+ `${VOICE_MICROPHONE_MESSAGE.failed} boom`
);
});
});
@@ -56,7 +60,7 @@ describe('the record button', () => {
it('shows the reason and stays tappable when the microphone cannot record', async () => {
const onMicClick = vi.fn();
const screen = render(ChatFormActionRecordWrapper, {
- micWarning: MICROPHONE_UNAVAILABLE_MESSAGE.insecure,
+ micWarning: VOICE_MICROPHONE_MESSAGE.insecure,
onMicClick
});
const button = screen.getByRole('button', { name: /^Recording unavailable: / }).last();
@@ -71,18 +75,18 @@ describe('the record button', () => {
const screen = render(ChatFormActionRecordWrapper, {});
await expect
- .element(screen.getByRole('button', { name: 'Start recording' }).last())
+ .element(screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last())
.toBeEnabled();
});
- it('stays disabled for a model without audio, whatever the microphone', async () => {
+ it('stays disabled when no model can take audio, whatever the microphone', async () => {
const screen = render(ChatFormActionRecordWrapper, {
- hasAudioModality: false,
- micWarning: MICROPHONE_UNAVAILABLE_MESSAGE.insecure
+ micWarning: VOICE_MICROPHONE_MESSAGE.insecure,
+ unavailableReason: VOICE_MODE_MESSAGE.unavailable
});
await expect
- .element(screen.getByRole('button', { name: 'Start recording' }).last())
+ .element(screen.getByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage }).last())
.toBeDisabled();
});
});
diff --git a/ui/apps/chat/tests/client/models-voice-role.svelte.test.ts b/ui/apps/chat/tests/client/models-voice-role.svelte.test.ts
new file mode 100644
index 000000000..687bc0875
--- /dev/null
+++ b/ui/apps/chat/tests/client/models-voice-role.svelte.test.ts
@@ -0,0 +1,180 @@
+// Guards how the model picker shows each model's part in voice input: a badge for a
+// speech-to-text model and for a chat model that hears, nothing for one that can't, and a
+// microphone among the row's actions that is lit on the model that transcribes dictation
+// and picks it on any other model that hears; and that a loaded model kept only for
+// dictation sits apart from the chat models, under its own heading.
+
+import ModelsSelectorListWrapper from './components/ModelsSelectorListWrapper.svelte';
+import { SETTINGS_KEYS, VOICE_TRANSCRIPTION_MODEL_AUTO } from '#lib/constants/index.ts';
+import { ServerModelStatus, ServerRole } from '#lib/enums/index.ts';
+import { serverStore } from '#lib/stores/llama-server.svelte.ts';
+import { modelsStore } from '#lib/stores/models/index.svelte.ts';
+import { settingsStore } from '#lib/stores/settings/index.svelte.ts';
+import type { ApiModelDataEntry, ModelOption } from '#lib/types/index.ts';
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { render } from 'vitest-browser-svelte';
+
+function option(model: string, audio: boolean): ModelOption {
+ return {
+ capabilities: [],
+ id: model,
+ modalities: { audio, video: false, vision: false },
+ model,
+ name: model
+ };
+}
+
+function loadedEntry(id: string): ApiModelDataEntry {
+ return {
+ created: 0,
+ id,
+ in_cache: true,
+ object: 'model',
+ owned_by: 'llamacpp',
+ path: `/models/${id}`,
+ status: { value: ServerModelStatus.LOADED }
+ };
+}
+
+const CHAT = option('org/chat-text:Q4_K_M', false);
+const OMNI = option('org/omni-2B:Q8_0', true);
+const ASR = option('org/Qwen3-ASR-0.6B:Q8_0', true);
+
+let previousConfig: typeof settingsStore.config;
+
+beforeEach(() => {
+ previousConfig = settingsStore.config;
+ modelsStore.models = [CHAT, OMNI, ASR];
+ modelsStore.routerModels = [loadedEntry(CHAT.model), loadedEntry(ASR.model)];
+ serverStore.role = ServerRole.ROUTER;
+});
+
+afterEach(() => {
+ settingsStore.config = previousConfig;
+});
+
+function useTranscriptionModel(model: string) {
+ settingsStore.config = {
+ ...settingsStore.config,
+ [SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL]: model
+ };
+}
+
+function renderList(currentModel: string) {
+ const items = [CHAT, OMNI, ASR].map((entry) => ({ option: entry }));
+
+ return render(ModelsSelectorListWrapper, {
+ currentModel,
+ groups: { available: [{ items, orgName: null }] },
+ loaded: [{ option: CHAT }, { option: ASR }]
+ });
+}
+
+/** The role badge of each listed row, by model, as the rows show them. */
+function roles(container: HTMLElement): Record<string, string | null> {
+ const out: Record<string, string | null> = {};
+
+ for (const row of container.querySelectorAll<HTMLElement>('[role="option"]')) {
+ const model = [CHAT, OMNI, ASR].find((entry) => row.querySelector(`[title="${entry.model}"]`));
+
+ if (model)
+ out[model.model] =
+ row.querySelector<HTMLElement>('[data-slot="model-voice-role"]')?.dataset.role ?? null;
+ }
+
+ return out;
+}
+
+/** The microphone action of a model's row, null when the row has none. */
+function microphone(container: HTMLElement, model: string): HTMLElement | null {
+ for (const row of container.querySelectorAll<HTMLElement>('[role="option"]')) {
+ if (row.querySelector(`[title="${model}"]`)) {
+ return row.querySelector<HTMLElement>('[data-slot="model-dictation-action"]');
+ }
+ }
+
+ return null;
+}
+
+/** The models whose microphone is lit, as the ones that transcribe dictation. */
+function dictating(container: HTMLElement): string[] {
+ return [CHAT, OMNI, ASR]
+ .map((entry) => entry.model)
+ .filter((model) => microphone(container, model)?.dataset.dictates !== undefined);
+}
+
+describe('voice roles in the model picker', () => {
+ it('badges a speech-to-text model and a chat model that hears', () => {
+ useTranscriptionModel(OMNI.model);
+
+ const screen = renderList(CHAT.model);
+
+ expect(roles(screen.container)).toEqual({
+ [ASR.model]: 'transcription',
+ [CHAT.model]: null,
+ [OMNI.model]: 'voice'
+ });
+ });
+
+ it('lights the microphone of the dictation model, and offers it on the others that hear', () => {
+ useTranscriptionModel(OMNI.model);
+
+ const screen = renderList(CHAT.model);
+
+ expect(dictating(screen.container)).toEqual([OMNI.model]);
+ expect(microphone(screen.container, ASR.model)).not.toBeNull();
+ // a model that cannot hear has no microphone
+ expect(microphone(screen.container, CHAT.model)).toBeNull();
+ });
+
+ it('picks the dictation model with a click, and hands it back to Automatic with another', async () => {
+ useTranscriptionModel(OMNI.model);
+
+ const screen = renderList(CHAT.model);
+ const click = (model: string) =>
+ microphone(screen.container, model)!.querySelector('button')!.click();
+
+ click(ASR.model);
+ await expect.poll(() => dictating(screen.container)).toEqual([ASR.model]);
+ expect(settingsStore.config[SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL]).toBe(ASR.model);
+
+ click(ASR.model);
+ await expect
+ .poll(() => settingsStore.config[SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL])
+ .toBe(VOICE_TRANSCRIPTION_MODEL_AUTO);
+ });
+
+ it('lists a loaded model kept for dictation apart from the chat models', () => {
+ useTranscriptionModel(ASR.model);
+
+ const screen = renderList(CHAT.model);
+ const heading = screen.container.querySelector('[data-slot="models-loaded-dictation-heading"]');
+
+ expect(heading?.textContent?.trim()).toBe('For dictation');
+ // the dictation model comes after the heading, the chat model before it
+ const rows = [...screen.container.querySelectorAll('[role="option"]')].slice(0, 2);
+
+ expect(rows[0].textContent).toContain('chat-text');
+ expect(rows[1].textContent).toContain('Qwen3-ASR');
+ expect(heading?.compareDocumentPosition(rows[1]) ?? 0).toBe(Node.DOCUMENT_POSITION_FOLLOWING);
+ });
+
+ it('keeps the conversation model with the chat models, even when it dictates', () => {
+ useTranscriptionModel(ASR.model);
+
+ const screen = renderList(ASR.model);
+
+ expect(
+ screen.container.querySelector('[data-slot="models-loaded-dictation-heading"]')
+ ).toBeNull();
+ });
+
+ it('on Automatic, gives dictation to the loaded model that hears', () => {
+ useTranscriptionModel(VOICE_TRANSCRIPTION_MODEL_AUTO);
+
+ const screen = renderList(CHAT.model);
+
+ expect(dictating(screen.container)).toEqual([ASR.model]);
+ expect(roles(screen.container)[OMNI.model]).toBe('voice');
+ });
+});
diff --git a/ui/apps/chat/tests/client/voice-capture.svelte.test.ts b/ui/apps/chat/tests/client/voice-capture.svelte.test.ts
new file mode 100644
index 000000000..3ce997a92
--- /dev/null
+++ b/ui/apps/chat/tests/client/voice-capture.svelte.test.ts
@@ -0,0 +1,184 @@
+// Guards VoiceCapture in a real browser, fed by a synthesized microphone (an oscillator
+// into a MediaStream, since the fake capture device flags are not available here): the
+// levels move, the PCM worklet loads from its Blob URL and streams chunks, stop() returns
+// a WAV clip with a plausible duration and peaks, and every path stops the tracks.
+
+import {
+ VOICE_MICROPHONE_MESSAGE,
+ VoiceCapture,
+ VoiceCaptureError,
+ VoiceCaptureState
+} from '@ggml-org/llama-voice';
+import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest';
+import { userEvent } from 'vitest/browser';
+
+const sources: AudioContext[] = [];
+
+/** A 440 Hz tone as a microphone stream. */
+function toneStream(): MediaStream {
+ const context = new AudioContext();
+ const oscillator = context.createOscillator();
+ const destination = context.createMediaStreamDestination();
+
+ oscillator.frequency.value = 440;
+ oscillator.connect(destination);
+ oscillator.start();
+ sources.push(context);
+
+ return destination.stream;
+}
+
+// audio contexts only run after a user gesture, as the mic button gives in the app
+beforeAll(async () => {
+ const button = document.createElement('button');
+
+ button.textContent = 'Record';
+ document.body.append(button);
+ await userEvent.click(button);
+ button.remove();
+});
+
+const wait = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
+
+/** Waits for `done`, polling: a loaded test run can start the audio graph late. */
+async function waitFor(done: () => boolean, timeoutMs = 5000) {
+ const deadline = Date.now() + timeoutMs;
+
+ while (!done()) {
+ if (Date.now() > deadline) throw new Error('timed out waiting for audio');
+
+ await wait(20);
+ }
+}
+
+afterEach(async () => {
+ vi.restoreAllMocks();
+ vi.unstubAllGlobals();
+
+ for (const context of sources.splice(0)) await context.close();
+});
+
+describe('VoiceCapture', () => {
+ it('streams levels and PCM chunks, and stops with a WAV clip', async () => {
+ const warn = vi.spyOn(console, 'warn');
+ const stream = toneStream();
+ const capture = new VoiceCapture({ levelsIntervalMs: 20, levelsLength: 16, stream });
+ const levels: (readonly number[])[] = [];
+ const chunks: Int16Array[] = [];
+
+ capture.onLevels((value) => levels.push(value));
+ capture.onPcm((chunk) => chunks.push(chunk));
+
+ await capture.start();
+ expect(capture.state).toBe(VoiceCaptureState.RECORDING);
+
+ // at least 600 ms of tone recorded, however long the audio graph took to start
+ await waitFor(() => chunks.length >= 6 && levels.length > 5);
+
+ const clip = await capture.stop();
+
+ // the worklet loaded from its Blob URL under COOP/COEP, as llama-server serves the UI, no fallback
+ expect(crossOriginIsolated).toBe(true);
+ expect(warn).not.toHaveBeenCalledWith(
+ expect.stringContaining('AudioWorklet unavailable'),
+ expect.anything()
+ );
+
+ expect(levels.length).toBeGreaterThan(5);
+ expect(levels.at(-1)).toHaveLength(16);
+ expect(Math.max(...levels.at(-1)!)).toBeGreaterThan(0.5);
+
+ // 100 ms chunks at 16 kHz, the last one may be shorter
+ expect(chunks.length).toBeGreaterThan(3);
+ expect(chunks[0]).toHaveLength(1600);
+
+ expect(clip.sampleRate).toBe(16_000);
+ expect(clip.durationMs).toBeGreaterThan(500);
+ expect(clip.durationMs).toBeLessThan(5000);
+ expect(clip.peaks).toHaveLength(64);
+ expect(Math.max(...clip.peaks)).toBe(1);
+ // a steady tone: once it reaches the capture, no span of the clip is silent. The capture
+ // starts with its first audio block, which can come before the tone stream's first sound
+ const toneFrom = clip.peaks.findIndex((peak) => peak > 0.5);
+
+ expect(toneFrom).toBeGreaterThanOrEqual(0);
+ expect(clip.peaks.slice(toneFrom).every((peak) => peak > 0.5)).toBe(true);
+ expect(clip.peaks.length - toneFrom).toBeGreaterThan(16);
+
+ const header = new DataView(await clip.blob.slice(0, 44).arrayBuffer());
+ const pcmBytes = chunks.reduce((sum, chunk) => sum + chunk.byteLength, 0);
+
+ expect(clip.blob.type).toBe('audio/wav');
+ expect(header.getUint16(22, true)).toBe(1);
+ expect(header.getUint32(24, true)).toBe(16_000);
+ expect(header.getUint16(34, true)).toBe(16);
+ expect(header.getUint32(40, true)).toBe(pcmBytes);
+
+ // the clip decodes as audio
+ const decoded = await new OfflineAudioContext(1, 1, 16_000).decodeAudioData(
+ await clip.blob.arrayBuffer()
+ );
+
+ expect(decoded.duration * 1000).toBeCloseTo(clip.durationMs, 0);
+ expect(stream.getTracks().every((track) => track.readyState === 'ended')).toBe(true);
+ expect(capture.state).toBe(VoiceCaptureState.STOPPED);
+ });
+
+ it('cancel stops the tracks and drops the recording', async () => {
+ const stream = toneStream();
+ const capture = new VoiceCapture({ stream });
+
+ await capture.start();
+ await wait(200);
+ capture.cancel();
+
+ expect(stream.getTracks().every((track) => track.readyState === 'ended')).toBe(true);
+ expect(capture.state).toBe(VoiceCaptureState.IDLE);
+ await expect(capture.stop()).rejects.toThrow();
+ });
+
+ it('cancel during start releases the microphone once it arrives', async () => {
+ const stream = toneStream();
+
+ let grant!: (value: MediaStream) => void;
+
+ vi.spyOn(navigator.mediaDevices, 'getUserMedia').mockImplementation(
+ () => new Promise((resolve) => (grant = resolve))
+ );
+
+ const capture = new VoiceCapture();
+ const started = capture.start().catch((e) => e);
+
+ capture.cancel();
+ grant(stream);
+
+ expect(await started).toBeInstanceOf(DOMException);
+ expect(stream.getTracks().every((track) => track.readyState === 'ended')).toBe(true);
+ });
+
+ it('a page hide tears the capture down', async () => {
+ const stream = toneStream();
+ const capture = new VoiceCapture({ stream });
+
+ await capture.start();
+ dispatchEvent(new PageTransitionEvent('pagehide'));
+
+ expect(capture.state).toBe(VoiceCaptureState.IDLE);
+ expect(stream.getTracks().every((track) => track.readyState === 'ended')).toBe(true);
+ });
+
+ it('asks for the raw microphone signal and names a refusal', async () => {
+ const getUserMedia = vi
+ .spyOn(navigator.mediaDevices, 'getUserMedia')
+ .mockRejectedValue(new DOMException('denied', 'NotAllowedError'));
+ const capture = new VoiceCapture();
+ const error = await capture.start().catch((e) => e);
+
+ expect(getUserMedia).toHaveBeenCalledWith({
+ audio: { autoGainControl: false, echoCancellation: false, noiseSuppression: false }
+ });
+ expect(error).toBeInstanceOf(VoiceCaptureError);
+ expect(error.message).toBe(VOICE_MICROPHONE_MESSAGE.denied);
+ expect(capture.state).toBe(VoiceCaptureState.IDLE);
+ });
+});
diff --git a/ui/apps/chat/tests/client/voice-service.svelte.test.ts b/ui/apps/chat/tests/client/voice-service.svelte.test.ts
new file mode 100644
index 000000000..c36a842c7
--- /dev/null
+++ b/ui/apps/chat/tests/client/voice-service.svelte.test.ts
@@ -0,0 +1,136 @@
+// Guards the wiring every voice feature starts its session with: the Voice settings become
+// capture options (boost, processing, a caller's own constraints), the language and the
+// transcription model go where a router needs them, and an API key keeps the realtime
+// socket off, since a browser websocket can not send it, which the app says of a model that
+// only transcribes over that socket.
+
+import { seedStores } from '../stories/chat/form/fixtures';
+import { VOICE_MICROPHONE_DEFAULT, VoiceTranscriptionError } from '@ggml-org/llama-voice';
+import { SETTINGS_KEYS, VOICE_MODE_MESSAGE } from '#lib/constants/index.ts';
+import { SETTING_CONFIG_DEFAULT } from '#lib/constants/settings.constants.ts';
+import { VoiceService } from '#lib/services/voice.service.ts';
+import type { SettingsConfigType } from '#lib/types/index.ts';
+import { afterEach, describe, expect, it } from 'vitest';
+
+let restore: (() => void) | undefined;
+
+afterEach(() => {
+ restore?.();
+ restore = undefined;
+});
+
+/** The default settings with `changes`, on the system microphone so no device is listed. */
+function settings(changes: Record<string, unknown> = {}): SettingsConfigType {
+ return {
+ ...SETTING_CONFIG_DEFAULT,
+ [SETTINGS_KEYS.VOICE_MICROPHONE]: VOICE_MICROPHONE_DEFAULT,
+ ...changes
+ } as SettingsConfigType;
+}
+
+describe('VoiceService.sessionOptions', () => {
+ it('boosts the raw signal by default, and transcribes with the model a router routes on', async () => {
+ const options = await VoiceService.sessionOptions(settings(), {
+ isRouter: true,
+ model: 'org/hearing-model'
+ });
+
+ expect(options.capture).toEqual({ boost: true });
+ expect(options.transcribe).toMatchObject({ language: undefined, model: 'org/hearing-model' });
+ });
+
+ it('leaves the model out for a single-model server', async () => {
+ const options = await VoiceService.sessionOptions(settings(), {
+ isRouter: false,
+ model: 'org/hearing-model'
+ });
+
+ expect(options.transcribe?.model).toBeUndefined();
+ });
+
+ it('follows the processing, boost and language settings', async () => {
+ const options = await VoiceService.sessionOptions(
+ settings({
+ [SETTINGS_KEYS.VOICE_AUTO_GAIN]: true,
+ [SETTINGS_KEYS.VOICE_BOOST]: false,
+ [SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION]: true,
+ [SETTINGS_KEYS.VOICE_TRANSCRIPTION_LANGUAGE]: ' de '
+ }),
+ { isRouter: false, model: null }
+ );
+
+ expect(options.capture).toEqual({
+ boost: false,
+ constraints: { autoGainControl: true, noiseSuppression: true }
+ });
+ expect(options.transcribe?.language).toBe('de');
+ });
+
+ it("takes a caller's own constraints over the processing settings", async () => {
+ const options = await VoiceService.sessionOptions(
+ settings({ [SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION]: true }),
+ { constraints: { echoCancellation: true }, isRouter: false, model: null }
+ );
+
+ expect(options.capture).toMatchObject({ constraints: { echoCancellation: true } });
+ });
+
+ it('records without transcribing when asked to', async () => {
+ const options = await VoiceService.sessionOptions(settings(), {
+ isRouter: false,
+ model: null,
+ transcribe: false
+ });
+
+ expect(options.transcribe).toBeUndefined();
+ });
+});
+
+describe('VoiceService.serverContext', () => {
+ it('uses the realtime socket when no API key is set', () => {
+ restore = seedStores({ config: { apiKey: '' } });
+
+ const { context, realtime } = VoiceService.serverContext();
+
+ expect(realtime).toBe(true);
+ expect(context.getHeaders?.()).toEqual({});
+ expect(context.resolveUrl('./v1/realtime')).toMatch(/\/v1\/realtime$/);
+ });
+
+ it('keeps to HTTP, with the key, when the server needs one', () => {
+ restore = seedStores({ config: { apiKey: 'secret' } });
+
+ const { context, realtime } = VoiceService.serverContext();
+
+ expect(realtime).toBe(false);
+ expect(context.getHeaders?.()).toEqual({ Authorization: 'Bearer secret' });
+ });
+});
+
+describe('VoiceService.failureReason', () => {
+ const notSupported = new VoiceTranscriptionError(
+ 'this server does not support chat/completions',
+ 501
+ );
+
+ it('says a live-only model cannot transcribe on a server with an API key', () => {
+ restore = seedStores({ config: { apiKey: 'secret' } });
+
+ expect(VoiceService.failureReason(notSupported)).toBe(VOICE_MODE_MESSAGE.realtimeOnlyBehindKey);
+ });
+
+ it("leaves other failures to the server's own words", () => {
+ restore = seedStores({ config: { apiKey: 'secret' } });
+
+ expect(
+ VoiceService.failureReason(new VoiceTranscriptionError('Loading model', 503))
+ ).toBeNull();
+ expect(VoiceService.failureReason(new Error('Failed to fetch'))).toBeNull();
+
+ restore();
+ restore = seedStores({ config: { apiKey: '' } });
+
+ // without a key the realtime socket was tried first: the model simply cannot hear
+ expect(VoiceService.failureReason(notSupported)).toBeNull();
+ });
+});
diff --git a/ui/apps/chat/tests/client/voice-waveform.svelte.test.ts b/ui/apps/chat/tests/client/voice-waveform.svelte.test.ts
new file mode 100644
index 000000000..202907756
--- /dev/null
+++ b/ui/apps/chat/tests/client/voice-waveform.svelte.test.ts
@@ -0,0 +1,39 @@
+// Guards VoiceWaveform's playback with real clicks, which a browser needs before it plays audio (the stories' clicks
+// are scripted, so their play functions cover the rest): play and pause toggle, and one clip plays at a time.
+
+import { demoClip } from '../stories/voice/fixtures';
+import { VoiceWaveform } from '@ggml-org/llama-ui/voice';
+import { describe, expect, it } from 'vitest';
+import { page, userEvent } from 'vitest/browser';
+import { render } from 'vitest-browser-svelte';
+
+describe('VoiceWaveform playback', () => {
+ it('toggles between play and pause', async () => {
+ const clip = demoClip(3, 1);
+
+ render(VoiceWaveform, { durationMs: clip.durationMs, peaks: clip.peaks, src: clip.src });
+
+ await userEvent.click(page.getByRole('button', { name: 'Play voice message' }));
+ await expect.element(page.getByRole('button', { name: 'Pause voice message' })).toBeVisible();
+
+ await userEvent.click(page.getByRole('button', { name: 'Pause voice message' }));
+ await expect.element(page.getByRole('button', { name: 'Play voice message' })).toBeVisible();
+ });
+
+ it('pauses one clip when another starts', async () => {
+ const first = demoClip(3, 2);
+ const second = demoClip(3, 3);
+
+ render(VoiceWaveform, { label: 'First clip', peaks: first.peaks, src: first.src });
+ render(VoiceWaveform, { label: 'Second clip', peaks: second.peaks, src: second.src });
+
+ await userEvent.click(page.getByRole('button', { name: 'Play first clip' }));
+ await expect.element(page.getByRole('button', { name: 'Pause first clip' })).toBeVisible();
+
+ await userEvent.click(page.getByRole('button', { name: 'Play second clip' }));
+ await expect.element(page.getByRole('button', { name: 'Pause second clip' })).toBeVisible();
+ await expect.element(page.getByRole('button', { name: 'Play first clip' })).toBeVisible();
+
+ await userEvent.click(page.getByRole('button', { name: 'Pause second clip' }));
+ });
+});
diff --git a/ui/apps/chat/tests/stories/chat/form/ChatFormActionRecord.stories.svelte b/ui/apps/chat/tests/stories/chat/form/ChatFormActionRecord.stories.svelte
index 476e9397c..66143acfa 100644
--- a/ui/apps/chat/tests/stories/chat/form/ChatFormActionRecord.stories.svelte
+++ b/ui/apps/chat/tests/stories/chat/form/ChatFormActionRecord.stories.svelte
@@ -1,8 +1,10 @@
<script lang="ts" module>
import { defineMeta } from '@storybook/addon-svelte-csf';
import ChatFormActionRecord from '#lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte';
+ import { VOICE_MODE_MESSAGE } from '#lib/constants/index.ts';
import StateGallery from '$llama-ui-stories/helpers/StateGallery.svelte';
import StateGalleryItem from '$llama-ui-stories/helpers/StateGalleryItem.svelte';
+ import { expect, screen } from 'storybook/test';
const { Story } = defineMeta({
component: ChatFormActionRecord,
@@ -14,35 +16,59 @@
</script>
<Story asChild name="States">
- <StateGallery columns={5}>
- <StateGalleryItem label="Idle">
- <ChatFormActionRecord hasAudioModality />
+ <StateGallery columns={6}>
+ <StateGalleryItem label="Voice message">
+ <ChatFormActionRecord action={VOICE_MODE_MESSAGE.voiceMessage} />
</StateGalleryItem>
- <StateGalleryItem label="Recording">
- <ChatFormActionRecord hasAudioModality isRecording />
+ <StateGalleryItem label="Dictate">
+ <ChatFormActionRecord action={VOICE_MODE_MESSAGE.dictate} />
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Recording" note="the stop button">
+ <ChatFormActionRecord isRecording />
</StateGalleryItem>
<StateGalleryItem label="Loading">
- <ChatFormActionRecord hasAudioModality isLoading />
+ <ChatFormActionRecord isLoading />
</StateGalleryItem>
<StateGalleryItem label="Disabled">
- <ChatFormActionRecord disabled hasAudioModality />
+ <ChatFormActionRecord disabled />
</StateGalleryItem>
- <StateGalleryItem label="No audio" note="model lacks audio">
- <ChatFormActionRecord hasAudioModality={false} />
+ <StateGalleryItem label="Unavailable" note="no model with audio">
+ <ChatFormActionRecord unavailableReason={VOICE_MODE_MESSAGE.unavailable} />
</StateGalleryItem>
</StateGallery>
</Story>
<!-- separate: play hovers the button to open its tooltip -->
<Story
- args={{ hasAudioModality: false }}
- name="NoAudioModalityTooltip"
+ args={{ unavailableReason: VOICE_MODE_MESSAGE.unavailable }}
+ name="UnavailableTooltip"
play={async ({ canvas, userEvent }) => {
+ const button = canvas.getAllByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage });
+
+ await expect(button.at(-1)).toBeDisabled();
// the tooltip trigger wraps the disabled button, hover the trigger
- await userEvent.hover(canvas.getAllByRole('button', { name: 'Start recording' })[0]);
+ await userEvent.hover(button[0]);
+ await expect(
+ (await screen.findAllByText(VOICE_MODE_MESSAGE.unavailable)).length
+ ).toBeGreaterThan(0);
+ }}
+/>
+
+<Story
+ args={{
+ action: VOICE_MODE_MESSAGE.dictate,
+ hint: VOICE_MODE_MESSAGE.separateModelHint
+ }}
+ name="DictateWithAnotherModelTooltip"
+ play={async ({ canvas, userEvent }) => {
+ await userEvent.hover(canvas.getAllByRole('button', { name: VOICE_MODE_MESSAGE.dictate })[0]);
+ await expect((await screen.findAllByText(/alongside the chat model/)).length).toBeGreaterThan(
+ 0
+ );
}}
/>
diff --git a/ui/apps/chat/tests/stories/chat/form/ChatFormVoice.stories.svelte b/ui/apps/chat/tests/stories/chat/form/ChatFormVoice.stories.svelte
new file mode 100644
index 000000000..adf4ce014
--- /dev/null
+++ b/ui/apps/chat/tests/stories/chat/form/ChatFormVoice.stories.svelte
@@ -0,0 +1,118 @@
+<script lang="ts" module>
+ import { demoClip } from '../../voice/fixtures';
+ import ChatFormVoiceStoryHost from './ChatFormVoiceStoryHost.svelte';
+ import {
+ FORM_CLASS,
+ modelServerProps,
+ resetPointerEvents,
+ seedStores,
+ ServerRole
+ } from './fixtures';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import { SETTINGS_KEYS, VOICE_MODE_MESSAGE } from '#lib/constants/index.ts';
+ import { VoiceInputMode } from '#lib/enums/index.ts';
+ import { expect, waitFor } from 'storybook/test';
+
+ const { Story } = defineMeta({
+ beforeEach: resetPointerEvents,
+ component: ChatFormVoiceStoryHost,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama Chat/Chat/ChatForm/Voice'
+ });
+
+ const SCRIPT = 'What is the difference between a KV cache and a prompt cache';
+
+ function audioServer(inputMode: VoiceInputMode) {
+ return seedStores({
+ config: {
+ autoMicOnEmpty: true,
+ [SETTINGS_KEYS.VOICE_INPUT_MODE]: inputMode,
+ [SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES]: true
+ },
+ serverProps: modelServerProps({ modalities: { audio: true, video: false, vision: true } }),
+ serverRole: ServerRole.MODEL
+ });
+ }
+
+ function voiceChip(): ChatUploadedFile {
+ const clip = demoClip(6, 2);
+
+ return {
+ file: new File([], 'voice-demo.wav', { type: 'audio/wav' }),
+ id: 'voice-demo',
+ name: 'voice-demo.wav',
+ preview: clip.src,
+ size: 96000,
+ type: 'audio/wav',
+ voice: { durationMs: clip.durationMs, peaks: clip.peaks }
+ };
+ }
+</script>
+
+<!-- a model with audio input: the recording becomes a voice message, its live text under the waveform -->
+<Story
+ args={{ class: FORM_CLASS, script: SCRIPT }}
+ beforeEach={() => audioServer(VoiceInputMode.AUTO)}
+ name="RecordingVoiceMessage"
+ play={async ({ canvas, canvasElement, userEvent }) => {
+ const mic = canvas.getAllByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage });
+
+ await userEvent.click(mic[mic.length - 1]);
+
+ await expect(await canvas.findByText(VOICE_MODE_MESSAGE.recording)).toBeInTheDocument();
+ await expect(canvas.getByRole('button', { name: 'Cancel recording' })).toBeInTheDocument();
+ // the live text shows as the empty message box's placeholder, the message stays empty
+ await waitFor(
+ () => expect(canvasElement.querySelector('.cm-placeholder')).toHaveTextContent(/What is the/),
+ { timeout: 3000 }
+ );
+ // a placeholder shows only while the message itself is empty
+ }}
+/>
+
+<!-- dictation: the text appears in the message as it is spoken -->
+<Story
+ args={{ class: FORM_CLASS, script: SCRIPT }}
+ beforeEach={() => audioServer(VoiceInputMode.DICTATION)}
+ name="RecordingDictation"
+ play={async ({ canvas, userEvent }) => {
+ const mic = canvas.getAllByRole('button', { name: VOICE_MODE_MESSAGE.dictate });
+
+ await userEvent.click(mic[mic.length - 1]);
+
+ await expect(await canvas.findByText(VOICE_MODE_MESSAGE.dictating)).toBeInTheDocument();
+ await waitFor(() => expect(canvas.getByRole('textbox')).toHaveTextContent('What is the'), {
+ timeout: 3000
+ });
+ }}
+/>
+
+<!-- stop records the voice message as a chip in the form, its transcript on the way -->
+<Story
+ args={{ class: FORM_CLASS, script: SCRIPT }}
+ beforeEach={() => audioServer(VoiceInputMode.AUTO)}
+ name="VoiceMessageRecorded"
+ play={async ({ canvas, userEvent }) => {
+ const mic = canvas.getAllByRole('button', { name: VOICE_MODE_MESSAGE.voiceMessage });
+
+ await userEvent.click(mic[mic.length - 1]);
+ await canvas.findByText(VOICE_MODE_MESSAGE.recording);
+ await new Promise((resolve) => setTimeout(resolve, 1200));
+
+ const stop = canvas.getAllByRole('button', { name: 'Stop recording' });
+
+ await userEvent.click(stop[stop.length - 1]);
+
+ await waitFor(() =>
+ expect(canvas.getAllByRole('slider', { name: /Voice message/ }).length).toBeGreaterThan(0)
+ );
+ }}
+/>
+
+<Story
+ args={{ class: FORM_CLASS, uploadedFiles: [voiceChip()] }}
+ beforeEach={() => audioServer(VoiceInputMode.AUTO)}
+ name="WithVoiceMessage"
+/>
diff --git a/ui/apps/chat/tests/stories/chat/form/ChatFormVoiceStoryHost.svelte b/ui/apps/chat/tests/stories/chat/form/ChatFormVoiceStoryHost.svelte
new file mode 100644
index 000000000..0c0e7a2d6
--- /dev/null
+++ b/ui/apps/chat/tests/stories/chat/form/ChatFormVoiceStoryHost.svelte
@@ -0,0 +1,33 @@
+<!--
+ The chat form with a fake microphone: a press on the microphone records levels like someone speaking and, with a
+ script, streams it as a realtime model would. Voice messages it records appear as chips.
+-->
+<script lang="ts">
+ import { DemoVoiceRecorder } from '../../voice/fake-voice-recorder.svelte';
+ import { demoLevel } from '../../voice/fixtures';
+ import ChatForm from '#lib/components/app/chat/ChatForm/ChatForm.svelte';
+ import { untrack } from 'svelte';
+
+ interface Props {
+ class?: string;
+ /** What the speaker says, streamed word by word; empty streams nothing. */
+ script?: string;
+ /** The message the form starts with. */
+ value?: string;
+ uploadedFiles?: ChatUploadedFile[];
+ }
+
+ let {
+ class: className = '',
+ script = '',
+ uploadedFiles = $bindable([]),
+ value = $bindable('')
+ }: Props = $props();
+
+ const recorder = new DemoVoiceRecorder(
+ untrack(() => script),
+ (t) => demoLevel(t)
+ );
+</script>
+
+<ChatForm bind:uploadedFiles bind:value class={className} voiceRecorder={recorder} />
diff --git a/ui/apps/chat/tests/stories/chat/messages/ChatMessageVoice.stories.svelte b/ui/apps/chat/tests/stories/chat/messages/ChatMessageVoice.stories.svelte
new file mode 100644
index 000000000..285a7ccbb
--- /dev/null
+++ b/ui/apps/chat/tests/stories/chat/messages/ChatMessageVoice.stories.svelte
@@ -0,0 +1,162 @@
+<script lang="ts" module>
+ import { demoStoredClip } from '../../voice/fixtures';
+ import { CONV_ID, noopChatActions, seedChatStory, STORY_CLASS } from './fixtures';
+ import { VoiceTranscriptStatus } from '@ggml-org/llama-ui/voice';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import ChatMessage from '#lib/components/app/chat/ChatMessages/ChatMessage/ChatMessage.svelte';
+ import { SETTINGS_KEYS } from '#lib/constants/index.ts';
+ import { AttachmentType, MessageRole } from '#lib/enums/index.ts';
+ import { voiceTranscriptsStore } from '#lib/stores/index.ts';
+ import type { DatabaseMessageExtraAudioFile, SettingsConfigValue } from '#lib/types/index.ts';
+ import { expect, waitFor } from 'storybook/test';
+
+ const { Story } = defineMeta({
+ component: ChatMessage,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama Chat/Chat/ChatMessages/ChatMessageVoice'
+ });
+
+ const TRANSCRIPT =
+ 'Can you explain how speculative decoding speeds up generation, and when it does not help?';
+
+ // filled in before each story: the clip is made in the browser
+ let message = $state<DatabaseMessage>({
+ children: [],
+ content: '',
+ convId: CONV_ID,
+ extra: [],
+ id: 'user-voice',
+ parent: null,
+ role: MessageRole.USER,
+ timestamp: Date.now(),
+ type: 'text'
+ });
+
+ async function seedVoice(
+ clip: Partial<DatabaseMessageExtraAudioFile>,
+ content = '',
+ settings: Record<string, SettingsConfigValue> = {}
+ ) {
+ const stored = await demoStoredClip(7, 3);
+ const name = clip.name ?? 'voice-story.wav';
+
+ message.content = content;
+ message.extra = [
+ {
+ base64Data: stored.base64,
+ durationMs: stored.durationMs,
+ mimeType: 'audio/wav',
+ name,
+ peaks: stored.peaks,
+ size: stored.size,
+ type: AttachmentType.AUDIO,
+ voice: true,
+ ...clip
+ } as DatabaseMessageExtraAudioFile
+ ];
+
+ const restore = seedChatStory({ settings });
+
+ return () => {
+ restore();
+ voiceTranscriptsStore.forget(name);
+ };
+ }
+</script>
+
+<!-- the transcript is open under the clip, and the toggle beside it folds it -->
+<Story
+ asChild
+ beforeEach={() => seedVoice({ transcript: TRANSCRIPT })}
+ name="VoiceMessage"
+ play={async ({ canvas, canvasElement }) => {
+ const reveal = await waitFor(() => {
+ const element = canvasElement.querySelector('[data-slot="voice-transcript-reveal"]');
+
+ if (!element) throw new Error('no voice message');
+
+ return element as HTMLElement;
+ });
+
+ await expect(reveal).toHaveAttribute('data-revealed');
+ // drawn word by word, see VoiceSpokenText
+ await expect(reveal).toHaveTextContent(TRANSCRIPT);
+ await expect(canvas.getByRole('button', { name: 'Hide transcript' })).toBeInTheDocument();
+ }}
+>
+ <ChatMessage chatActions={noopChatActions} class={STORY_CLASS} {message} />
+</Story>
+
+<!-- with Collapse transcripts on: the transcript opens under the clip on hover, keyboard focus or a tap -->
+<Story
+ asChild
+ beforeEach={() =>
+ seedVoice({ transcript: TRANSCRIPT }, '', { [SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS]: true })}
+ name="VoiceMessageCollapsed"
+ play={async ({ canvasElement, userEvent }) => {
+ const reveal = await waitFor(() => {
+ const element = canvasElement.querySelector('[data-slot="voice-transcript-reveal"]');
+
+ if (!element) throw new Error('no voice message');
+
+ return element as HTMLElement;
+ });
+
+ await expect(reveal).not.toHaveAttribute('data-revealed');
+ await userEvent.hover(reveal);
+ await waitFor(() => expect(reveal).toHaveAttribute('data-revealed'));
+ // drawn word by word, see VoiceSpokenText
+ await expect(reveal).toHaveTextContent(TRANSCRIPT);
+ }}
+>
+ <ChatMessage chatActions={noopChatActions} class={STORY_CLASS} {message} />
+</Story>
+
+<Story
+ asChild
+ beforeEach={() => seedVoice({ transcript: TRANSCRIPT }, 'Answer in two sentences.')}
+ name="VoiceMessageWithText"
+>
+ <ChatMessage chatActions={noopChatActions} class={STORY_CLASS} {message} />
+</Story>
+
+<!-- sent before its transcript came back -->
+<Story
+ asChild
+ beforeEach={async () => {
+ const restore = await seedVoice({ name: 'voice-pending.wav' });
+
+ voiceTranscriptsStore.track('voice-pending.wav', new Promise<string>(() => {}));
+
+ return restore;
+ }}
+ name="VoiceMessageTranscribing"
+ play={async ({ canvasElement }) => {
+ await waitFor(() =>
+ expect(canvasElement.querySelector('[data-slot="voice-transcript-reveal"]')).toHaveAttribute(
+ 'data-status',
+ VoiceTranscriptStatus.TRANSCRIBING
+ )
+ );
+ }}
+>
+ <ChatMessage chatActions={noopChatActions} class={STORY_CLASS} {message} />
+</Story>
+
+<!-- stored before voice messages kept a waveform: the peaks come from the audio -->
+<Story
+ asChild
+ beforeEach={() => seedVoice({ peaks: undefined, transcript: TRANSCRIPT })}
+ name="VoiceMessageWithoutPeaks"
+ play={async ({ canvasElement }) => {
+ await waitFor(() =>
+ expect(
+ canvasElement.querySelectorAll('[data-slot="voice-waveform"] [aria-hidden] span').length
+ ).toBeGreaterThan(0)
+ );
+ }}
+>
+ <ChatMessage chatActions={noopChatActions} class={STORY_CLASS} {message} />
+</Story>
diff --git a/ui/apps/chat/tests/stories/settings/SettingsSections.stories.svelte b/ui/apps/chat/tests/stories/settings/SettingsSections.stories.svelte
index 24b2c5c13..de2d6b8f9 100644
--- a/ui/apps/chat/tests/stories/settings/SettingsSections.stories.svelte
+++ b/ui/apps/chat/tests/stories/settings/SettingsSections.stories.svelte
@@ -17,6 +17,8 @@
<Story args={{ initialSection: SETTINGS_SECTIONS.DISPLAY.slug, open: true }} name="Display" />
+<Story args={{ initialSection: SETTINGS_SECTIONS.VOICE.slug, open: true }} name="Voice" />
+
<Story args={{ initialSection: SETTINGS_SECTIONS.TOOLS.slug, open: true }} name="Tools" />
<Story args={{ initialSection: SETTINGS_SECTIONS.AGENTIC.slug, open: true }} name="Agentic" />
diff --git a/ui/apps/chat/tests/stories/voice/VoiceLiveDemo.svelte b/ui/apps/chat/tests/stories/voice/VoiceLiveDemo.svelte
new file mode 100644
index 000000000..23c93f339
--- /dev/null
+++ b/ui/apps/chat/tests/stories/voice/VoiceLiveDemo.svelte
@@ -0,0 +1,54 @@
+<!--
+ The live waveform in a chat form row, fed by a fake microphone meter: a level every 50 ms while `active`, and the
+ recording time beside it.
+-->
+<script lang="ts">
+ import { demoLevel } from './fixtures';
+ import { formatClipTime, VoiceWaveformLive } from '@ggml-org/llama-ui/voice';
+ import ChatFormActionRecord from '#lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionRecord.svelte';
+ import { onMount } from 'svelte';
+
+ interface Props {
+ active?: boolean;
+ class?: string;
+ /** Levels to start from, e.g. a paused recording's. */
+ initialSeconds?: number;
+ seed?: number;
+ }
+
+ let { active = true, class: className = '', initialSeconds = 0, seed = 1 }: Props = $props();
+
+ const TICK_MS = 50;
+ const KEEP = 160;
+
+ let elapsedMs = $state(0);
+ let levels = $state<number[]>([]);
+
+ onMount(() => {
+ elapsedMs = initialSeconds * 1000;
+ levels = Array.from({ length: Math.min(Math.round(initialSeconds * 20), KEEP) }, (_, index) =>
+ demoLevel(index / 20, seed)
+ );
+
+ const timer = setInterval(() => {
+ if (!active) return;
+
+ elapsedMs += TICK_MS;
+ levels = [...levels.slice(-(KEEP - 1)), demoLevel(elapsedMs / 1000, seed)];
+ }, TICK_MS);
+
+ return () => clearInterval(timer);
+ });
+</script>
+
+<div
+ class="chat-form-surface flex items-center gap-2 rounded-(--chat-form-single-row-radius) px-3 py-2.5 {className}"
+>
+ <VoiceWaveformLive {active} class="flex-1 px-2" label="Recording" {levels} />
+
+ <span class="w-9 shrink-0 text-right text-caption text-muted-foreground tabular-nums">
+ {formatClipTime(elapsedMs)}
+ </span>
+
+ <ChatFormActionRecord isRecording={active} />
+</div>
diff --git a/ui/apps/chat/tests/stories/voice/VoiceWaveformLive.stories.svelte b/ui/apps/chat/tests/stories/voice/VoiceWaveformLive.stories.svelte
new file mode 100644
index 000000000..78715741a
--- /dev/null
+++ b/ui/apps/chat/tests/stories/voice/VoiceWaveformLive.stories.svelte
@@ -0,0 +1,82 @@
+<script lang="ts" module>
+ import { demoLevel } from './fixtures';
+ import VoiceLiveDemo from './VoiceLiveDemo.svelte';
+ import { VoiceWaveformLive } from '@ggml-org/llama-ui/voice';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import StateGallery from '$llama-ui-stories/helpers/StateGallery.svelte';
+ import StateGalleryItem from '$llama-ui-stories/helpers/StateGalleryItem.svelte';
+ import { expect, waitFor } from 'storybook/test';
+
+ // the waveform while recording, in the chat form's input row; here a fake microphone meter feeds it
+ const { Story } = defineMeta({
+ component: VoiceWaveformLive,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama Chat/Voice/VoiceWaveformLive'
+ });
+
+ const frozen = Array.from({ length: 48 }, (_, index) => demoLevel(index / 20, 4));
+
+ function barHeights(root: HTMLElement): string {
+ return [...root.querySelectorAll<HTMLElement>('[data-slot="voice-waveform-live"] span[style]')]
+ .map((bar) => bar.style.height)
+ .join(',');
+ }
+</script>
+
+<Story
+ asChild
+ name="Recording"
+ play={async ({ canvasElement }) => {
+ const live = canvasElement.querySelector('[data-slot="voice-waveform-live"]')!;
+
+ await expect(live.querySelector('.sr-only')).toHaveTextContent('Recording');
+ await expect(live.querySelector('[aria-hidden="true"]')).toBeInTheDocument();
+
+ const before = barHeights(canvasElement);
+
+ await waitFor(() => expect(barHeights(canvasElement)).not.toBe(before));
+ }}
+>
+ <StateGallery columns={1} width="40rem">
+ <StateGalleryItem label="Recording" note="a level every 50 ms, newest on the right">
+ <VoiceLiveDemo />
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Paused" note="active off: the bars turn muted and stop">
+ <VoiceLiveDemo active={false} initialSeconds={6} seed={2} />
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<Story asChild name="Levels">
+ <StateGallery columns={1} width="32rem">
+ <StateGalleryItem label="Filling up" note="fewer levels than bars: resting dots on the left">
+ <VoiceWaveformLive levels={frozen.slice(0, 12)} />
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Full" note="the last levels that fit">
+ <VoiceWaveformLive levels={frozen} />
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Silence">
+ <VoiceWaveformLive levels={[]} />
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Inactive">
+ <VoiceWaveformLive active={false} levels={frozen} />
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<Story
+ asChild
+ globals={{ viewport: { isRotated: false, value: 'mobile2' } }}
+ name="Narrow"
+ parameters={{ layout: 'fullscreen' }}
+>
+ <div class="p-4">
+ <VoiceLiveDemo />
+ </div>
+</Story>
diff --git a/ui/apps/chat/tests/stories/voice/fake-voice-recorder.svelte.ts b/ui/apps/chat/tests/stories/voice/fake-voice-recorder.svelte.ts
new file mode 100644
index 000000000..abbb5470c
--- /dev/null
+++ b/ui/apps/chat/tests/stories/voice/fake-voice-recorder.svelte.ts
@@ -0,0 +1,154 @@
+import type { VoiceSessionOptions, VoiceSessionResult } from '@ggml-org/llama-voice';
+import { VoiceRecorderStatus } from '@ggml-org/llama-voice';
+import type { VoiceRecorderLike } from '#lib/hooks/use-chat-form-voice.svelte.ts';
+
+/** A transcript the test resolves when it wants. */
+export function deferred<T>() {
+ let resolve!: (value: T) => void;
+ let reject!: (error: unknown) => void;
+
+ const promise = new Promise<T>((res, rej) => {
+ resolve = res;
+ reject = rej;
+ });
+
+ return { promise, reject, resolve };
+}
+
+/** A silent 16-bit mono WAV of `ms`, a valid clip to attach and play. */
+export function silentWav(ms = 500, sampleRate = 8000): Blob {
+ const samples = Math.round((ms / 1000) * sampleRate);
+ const buffer = new ArrayBuffer(44 + samples * 2);
+ const view = new DataView(buffer);
+ const write = (offset: number, text: string) => {
+ for (let i = 0; i < text.length; i++) view.setUint8(offset + i, text.charCodeAt(i));
+ };
+
+ write(0, 'RIFF');
+ view.setUint32(4, 36 + samples * 2, true);
+ write(8, 'WAVE');
+ write(12, 'fmt ');
+ view.setUint32(16, 16, true);
+ view.setUint16(20, 1, true);
+ view.setUint16(22, 1, true);
+ view.setUint32(24, sampleRate, true);
+ view.setUint32(28, sampleRate * 2, true);
+ view.setUint16(32, 2, true);
+ view.setUint16(34, 16, true);
+ write(36, 'data');
+ view.setUint32(40, samples * 2, true);
+
+ return new Blob([buffer], { type: 'audio/wav' });
+}
+
+/**
+ * A VoiceRecorder without a microphone: the test drives its levels and live text and
+ * decides the transcript, and reads the options the form started it with.
+ */
+export class FakeVoiceRecorder implements VoiceRecorderLike {
+ cancelled = 0;
+ levels = $state.raw<number[]>(new Array(48).fill(0));
+ live = $state(false);
+ liveText = $state('');
+ options: VoiceSessionOptions | undefined;
+
+ quiet = $state(false);
+ startError: Error | null = null;
+ status = $state<VoiceRecorderStatus>(VoiceRecorderStatus.IDLE);
+ transcript: Promise<string> | null = null;
+
+ cancel(): void {
+ this.cancelled++;
+ this.status = VoiceRecorderStatus.IDLE;
+ }
+
+ /** Live text as a realtime socket would stream it. */
+ say(text: string): void {
+ this.live = true;
+ this.liveText = text;
+ this.options?.onText?.(text);
+ }
+
+ setLevels(levels: number[]): void {
+ this.levels = levels;
+ this.options?.onLevels?.(levels);
+ }
+
+ /** The speech turning too quiet, or loud enough again, as the capture would tell. */
+ setQuiet(quiet: boolean): void {
+ this.quiet = quiet;
+ this.options?.onQuiet?.(quiet);
+ }
+
+ async start(options: VoiceSessionOptions = {}): Promise<void> {
+ if (this.startError) throw this.startError;
+
+ this.options = options;
+ this.liveText = '';
+ this.status = VoiceRecorderStatus.RECORDING;
+ }
+
+ async stop(): Promise<VoiceSessionResult> {
+ this.status = VoiceRecorderStatus.IDLE;
+
+ return {
+ clip: {
+ blob: silentWav(),
+ durationMs: 2400,
+ peaks: Array.from({ length: 64 }, (_, i) => Math.abs(Math.sin(i / 4))),
+ sampleRate: 8000
+ },
+ liveText: this.liveText.trim() || null,
+ transcript: this.options?.transcribe ? this.transcript : null
+ };
+ }
+}
+
+/**
+ * A fake recorder for stories: once started it moves its levels like someone speaking
+ * and, when given a script, streams it word by word as a realtime model would.
+ */
+export class DemoVoiceRecorder extends FakeVoiceRecorder {
+ private timer: ReturnType<typeof setInterval> | undefined;
+
+ override cancel(): void {
+ clearInterval(this.timer);
+ super.cancel();
+ }
+
+ constructor(
+ private script = '',
+ private level: (t: number) => number = () => Math.random()
+ ) {
+ super();
+ }
+
+ override async start(options: VoiceSessionOptions = {}): Promise<void> {
+ await super.start(options);
+
+ const words = this.script.split(' ').filter(Boolean);
+ const startedAt = Date.now();
+
+ let history = new Array(48).fill(0);
+
+ this.timer = setInterval(() => {
+ const t = (Date.now() - startedAt) / 1000;
+ const spoken = Math.min(Math.floor(t * 2.5), words.length);
+
+ history = [...history.slice(1), this.level(t)];
+ this.setLevels(history);
+
+ if (spoken > 0 && words.slice(0, spoken).join(' ') !== this.liveText) {
+ this.say(words.slice(0, spoken).join(' '));
+ }
+ }, 50);
+ }
+
+ override async stop(): Promise<VoiceSessionResult> {
+ clearInterval(this.timer);
+
+ if (!this.transcript && this.script) this.transcript = Promise.resolve(this.script);
+
+ return super.stop();
+ }
+}
diff --git a/ui/apps/chat/tests/stories/voice/fixtures.ts b/ui/apps/chat/tests/stories/voice/fixtures.ts
new file mode 100644
index 000000000..47788fa85
--- /dev/null
+++ b/ui/apps/chat/tests/stories/voice/fixtures.ts
@@ -0,0 +1,2 @@
+// the demo audio lives with the voice components' stories in Llama UI
+export * from '../../../../../lib/ui/tests/stories/voice/fixtures';
diff --git a/ui/apps/chat/tests/unit/voice-message-send.test.ts b/ui/apps/chat/tests/unit/voice-message-send.test.ts
new file mode 100644
index 000000000..36858e504
--- /dev/null
+++ b/ui/apps/chat/tests/unit/voice-message-send.test.ts
@@ -0,0 +1,119 @@
+// Guards how a voice message goes to a model: the audio alone to a model with audio input,
+// its transcript as text to one without, and the copied message carrying the transcript.
+
+import { AttachmentType, ContentPartType, MessageRole, MessageType } from '#lib/enums/index.ts';
+import { ChatService } from '#lib/services/chat.service.ts';
+import { modelsStore } from '#lib/stores/models/index.svelte.ts';
+import { withTranscript } from '#lib/stores/voice-transcripts.svelte.ts';
+import type { DatabaseMessage, DatabaseMessageExtra } from '#lib/types/index.ts';
+import { formatMessageForClipboard } from '#lib/utils/clipboard.ts';
+import { afterEach, describe, expect, it, vi } from 'vitest';
+
+const voiceClip: DatabaseMessageExtra = {
+ base64Data: 'UklGRg==',
+ durationMs: 2400,
+ mimeType: 'audio/wav',
+ name: 'voice-1.wav',
+ peaks: [0.2, 1, 0.4],
+ transcript: 'What is the weather like on Mars?',
+ type: AttachmentType.AUDIO,
+ voice: true
+};
+
+function userMessage(extra: DatabaseMessageExtra[], content = ''): DatabaseMessage {
+ return {
+ children: [],
+ content,
+ convId: 'c1',
+ extra,
+ id: 'm1',
+ parent: null,
+ role: MessageRole.USER,
+ timestamp: 0,
+ type: MessageType.TEXT
+ } as DatabaseMessage;
+}
+
+afterEach(() => {
+ vi.restoreAllMocks();
+});
+
+describe('sending a voice message', () => {
+ it('sends the audio alone to a model with audio input', async () => {
+ const result = await ChatService.convertDbMessageToApiChatMessageData(
+ userMessage([voiceClip]),
+ { audioInput: true }
+ );
+ const parts = result.content as { type: string; text?: string }[];
+
+ expect(parts).toHaveLength(1);
+ expect(parts[0].type).toBe(ContentPartType.INPUT_AUDIO);
+ expect(JSON.stringify(parts)).not.toContain('Mars');
+ });
+
+ it('sends the transcript as text to a model without audio input', async () => {
+ const result = await ChatService.convertDbMessageToApiChatMessageData(
+ userMessage([voiceClip], 'Answer briefly.'),
+ { audioInput: false }
+ );
+ const parts = result.content as { type: string; text?: string }[];
+
+ expect(parts.map((part) => part.type)).toEqual([ContentPartType.TEXT, ContentPartType.TEXT]);
+ expect(parts[0].text).toContain('Voice message transcript');
+ expect(parts[0].text).toContain('What is the weather like on Mars?');
+ expect(parts[1].text).toBe('Answer briefly.');
+ });
+
+ it('keeps the audio when there is no transcript to send instead', async () => {
+ const { transcript: _transcript, ...withoutTranscript } = voiceClip as {
+ transcript?: string;
+ } & DatabaseMessageExtra;
+ const result = await ChatService.convertDbMessageToApiChatMessageData(
+ userMessage([withoutTranscript as DatabaseMessageExtra]),
+ { audioInput: false }
+ );
+
+ expect((result.content as { type: string }[])[0].type).toBe(ContentPartType.INPUT_AUDIO);
+ });
+
+ it('keeps the audio while the model modalities are unknown', () => {
+ vi.spyOn(modelsStore.props, 'getModelModalities').mockReturnValue(null);
+
+ expect(ChatService.conversionOptionsFor('some-model')).toEqual({});
+ });
+
+ it('reads the audio input from the model modalities', () => {
+ vi.spyOn(modelsStore.props, 'getModelModalities').mockReturnValue({
+ audio: false,
+ video: false,
+ vision: true
+ });
+
+ expect(ChatService.conversionOptionsFor('text-model')).toEqual({ audioInput: false });
+ });
+});
+
+describe('a transcript that arrives after the send', () => {
+ it('fills in the clip it belongs to only', () => {
+ const pending = { ...voiceClip, transcript: undefined } as DatabaseMessageExtra;
+ const other = { ...pending, name: 'voice-2.wav' } as DatabaseMessageExtra;
+ const next = withTranscript([pending, other], 'voice-1.wav', 'Hello');
+
+ expect(next?.[0]).toMatchObject({ name: 'voice-1.wav', transcript: 'Hello' });
+ expect(next?.[1]).toBe(other);
+ });
+
+ it('leaves a message without that clip alone', () => {
+ expect(withTranscript([voiceClip], 'voice-1.wav', 'Hello')).toBeNull();
+ expect(withTranscript(undefined, 'voice-1.wav', 'Hello')).toBeNull();
+ });
+});
+
+describe('copying a voice message', () => {
+ it('copies the transcript as the text', () => {
+ expect(formatMessageForClipboard('', [voiceClip])).toBe('What is the weather like on Mars?');
+ expect(formatMessageForClipboard('Briefly.', [voiceClip], true)).toBe(
+ 'What is the weather like on Mars?\n\nBriefly.'
+ );
+ });
+});
diff --git a/ui/apps/chat/tests/unit/voice-microphone.test.ts b/ui/apps/chat/tests/unit/voice-microphone.test.ts
new file mode 100644
index 000000000..8662c449c
--- /dev/null
+++ b/ui/apps/chat/tests/unit/voice-microphone.test.ts
@@ -0,0 +1,71 @@
+// Guards which microphone a recording uses: the system default, unless that is a
+// Bluetooth headset and a built-in microphone exists, or the one picked in Settings.
+
+import {
+ pickMicrophone,
+ VOICE_MICROPHONE_AUTO,
+ VOICE_MICROPHONE_DEFAULT
+} from '@ggml-org/llama-voice';
+import { SETTING_CONFIG_DEFAULT, SETTINGS_KEYS } from '#lib/constants/index.ts';
+import { describe, expect, it } from 'vitest';
+
+const input = (deviceId: string, label: string) => ({
+ deviceId,
+ kind: 'audioinput' as const,
+ label
+});
+const output = (deviceId: string, label: string) => ({
+ deviceId,
+ kind: 'audiooutput' as const,
+ label
+});
+const BUILT_IN = input('built-in', 'MacBook Pro Microphone (Built-in)');
+const AIRPODS = input('airpods', 'AirPods Pro (Bluetooth)');
+const USB = input('usb', 'Yeti Stereo Microphone');
+
+describe('pickMicrophone', () => {
+ it('defaults to Automatic', () => {
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_MICROPHONE]).toBe(VOICE_MICROPHONE_AUTO);
+ });
+
+ it('keeps the system default when it is not a Bluetooth headset', () => {
+ const devices = [input('default', 'Default - Yeti Stereo Microphone'), BUILT_IN, USB, AIRPODS];
+
+ expect(pickMicrophone(devices, VOICE_MICROPHONE_AUTO)).toBeUndefined();
+ });
+
+ it('takes the built-in microphone over a Bluetooth headset that is the default', () => {
+ const devices = [
+ input('default', 'Default - AirPods Pro (Bluetooth)'),
+ AIRPODS,
+ BUILT_IN,
+ output('airpods-out', 'AirPods Pro (Bluetooth)')
+ ];
+
+ expect(pickMicrophone(devices, VOICE_MICROPHONE_AUTO)).toBe('built-in');
+ });
+
+ it('reads the first input as the default where the browser lists no default entry', () => {
+ expect(pickMicrophone([AIRPODS, BUILT_IN], VOICE_MICROPHONE_AUTO)).toBe('built-in');
+ expect(pickMicrophone([USB, AIRPODS], VOICE_MICROPHONE_AUTO)).toBeUndefined();
+ });
+
+ it('keeps a Bluetooth headset when there is no built-in microphone', () => {
+ const devices = [input('default', 'Default - AirPods Pro (Bluetooth)'), AIRPODS, USB];
+
+ expect(pickMicrophone(devices, VOICE_MICROPHONE_AUTO)).toBeUndefined();
+ });
+
+ it('uses the system default when asked to', () => {
+ expect(pickMicrophone([AIRPODS, BUILT_IN], VOICE_MICROPHONE_DEFAULT)).toBeUndefined();
+ });
+
+ it('uses a picked microphone while it is connected', () => {
+ expect(pickMicrophone([BUILT_IN, USB], 'usb')).toBe('usb');
+ expect(pickMicrophone([BUILT_IN], 'usb')).toBeUndefined();
+ });
+
+ it('knows nothing before the browser names the microphones', () => {
+ expect(pickMicrophone([input('', ''), input('', '')], VOICE_MICROPHONE_AUTO)).toBeUndefined();
+ });
+});
diff --git a/ui/apps/chat/tests/unit/voice-mode.test.ts b/ui/apps/chat/tests/unit/voice-mode.test.ts
new file mode 100644
index 000000000..8b0e0b6be
--- /dev/null
+++ b/ui/apps/chat/tests/unit/voice-mode.test.ts
@@ -0,0 +1,240 @@
+// Guards what the microphone does for a conversation's model: a voice message for a model
+// that hears, dictation otherwise, and why it can't when no model can transcribe; and each
+// model's part in voice input, as the model lists show it.
+
+import {
+ SETTING_CONFIG_DEFAULT,
+ SETTINGS_CHAT_SECTIONS,
+ SETTINGS_KEYS,
+ SETTINGS_SECTION_SLUGS,
+ VOICE_MODE_MESSAGE,
+ VOICE_TRANSCRIPTION_MODEL_AUTO
+} from '#lib/constants/index.ts';
+import { SettingsFieldType, VoiceInputMode, VoiceMode, VoiceModelRole } from '#lib/enums/index.ts';
+import {
+ dictationModelFor,
+ isTranscriptionOnlyModel,
+ resolveVoiceMode,
+ type VoiceModeInput,
+ voiceModeLabel,
+ voiceModelRole
+} from '#lib/utils/voice-mode.ts';
+import { describe, expect, it } from 'vitest';
+
+const base: VoiceModeInput = {
+ chatModel: 'gemma-3n',
+ chatModelHasAudio: true,
+ isRouter: true,
+ setting: VoiceInputMode.AUTO,
+ transcriptionModel: '',
+ transcriptionModelHasAudio: false
+};
+
+describe('resolveVoiceMode', () => {
+ it('sends a voice message to a chat model with audio, transcribed by that model', () => {
+ expect(resolveVoiceMode(base)).toEqual({
+ mode: VoiceMode.VOICE_MESSAGE,
+ separateModel: false,
+ transcriptionModel: 'gemma-3n'
+ });
+ });
+
+ it('dictates with a speech-to-text model picked for the conversation', () => {
+ expect(resolveVoiceMode({ ...base, chatModel: 'org/VibeVoice-ASR:Q8_0' })).toEqual({
+ mode: VoiceMode.DICTATION,
+ separateModel: false,
+ transcriptionModel: 'org/VibeVoice-ASR:Q8_0'
+ });
+ });
+
+ it('dictates with the chat model when the user always dictates', () => {
+ expect(resolveVoiceMode({ ...base, setting: VoiceInputMode.DICTATION })).toEqual({
+ mode: VoiceMode.DICTATION,
+ separateModel: false,
+ transcriptionModel: 'gemma-3n'
+ });
+ });
+
+ it('prefers the chat model over a chosen transcription model when it can hear', () => {
+ const result = resolveVoiceMode({
+ ...base,
+ setting: VoiceInputMode.DICTATION,
+ transcriptionModel: 'voxtral',
+ transcriptionModelHasAudio: true
+ });
+
+ expect(result).toMatchObject({ separateModel: false, transcriptionModel: 'gemma-3n' });
+ });
+
+ it('dictates with the chosen model for a chat model without audio, in router mode', () => {
+ expect(
+ resolveVoiceMode({
+ ...base,
+ chatModel: 'qwen3',
+ chatModelHasAudio: false,
+ transcriptionModel: 'voxtral',
+ transcriptionModelHasAudio: true
+ })
+ ).toEqual({ mode: VoiceMode.DICTATION, separateModel: true, transcriptionModel: 'voxtral' });
+ });
+
+ it('is unavailable when no model can transcribe', () => {
+ const noAudio = { ...base, chatModel: 'qwen3', chatModelHasAudio: false };
+
+ expect(resolveVoiceMode(noAudio)).toEqual({
+ mode: VoiceMode.UNAVAILABLE,
+ reason: VOICE_MODE_MESSAGE.unavailable
+ });
+ // a chosen model without audio input does not count
+ expect(
+ resolveVoiceMode({
+ ...noAudio,
+ transcriptionModel: 'llama',
+ transcriptionModelHasAudio: false
+ }).mode
+ ).toBe(VoiceMode.UNAVAILABLE);
+ });
+
+ it('falls back to a loaded model with audio input on Automatic, loading nothing', () => {
+ const noAudio = { ...base, chatModel: 'qwen3', chatModelHasAudio: false };
+
+ expect(resolveVoiceMode({ ...noAudio, loadedAudioModel: 'qwen3-asr' })).toEqual({
+ mode: VoiceMode.DICTATION,
+ separateModel: false,
+ transcriptionModel: 'qwen3-asr'
+ });
+ // a chosen model wins over the loaded one
+ expect(
+ resolveVoiceMode({
+ ...noAudio,
+ loadedAudioModel: 'qwen3-asr',
+ transcriptionModel: 'voxtral',
+ transcriptionModelHasAudio: true
+ })
+ ).toEqual({ mode: VoiceMode.DICTATION, separateModel: true, transcriptionModel: 'voxtral' });
+ // so does the chat model, when it hears
+ expect(resolveVoiceMode({ ...base, loadedAudioModel: 'qwen3-asr' }).mode).toBe(
+ VoiceMode.VOICE_MESSAGE
+ );
+ // and nothing is loaded behind the user's back
+ expect(resolveVoiceMode({ ...noAudio, loadedAudioModel: null }).mode).toBe(
+ VoiceMode.UNAVAILABLE
+ );
+ expect(
+ resolveVoiceMode({ ...noAudio, isRouter: false, loadedAudioModel: 'qwen3-asr' }).mode
+ ).toBe(VoiceMode.UNAVAILABLE);
+ });
+
+ it('ignores the transcription model on a single-model server', () => {
+ expect(
+ resolveVoiceMode({
+ ...base,
+ chatModelHasAudio: false,
+ isRouter: false,
+ transcriptionModel: 'voxtral',
+ transcriptionModelHasAudio: true
+ }).mode
+ ).toBe(VoiceMode.UNAVAILABLE);
+ });
+
+ it('labels the microphone with what it will do', () => {
+ expect(voiceModeLabel(resolveVoiceMode(base))).toBe('Send a voice message');
+ expect(voiceModeLabel(resolveVoiceMode({ ...base, setting: VoiceInputMode.DICTATION }))).toBe(
+ 'Dictate'
+ );
+ expect(voiceModeLabel(resolveVoiceMode({ ...base, chatModelHasAudio: false }))).toBe(
+ VOICE_MODE_MESSAGE.unavailable
+ );
+ });
+});
+
+describe('voice settings', () => {
+ const section = SETTINGS_CHAT_SECTIONS.find((s) => s.slug === SETTINGS_SECTION_SLUGS.VOICE);
+
+ it('default to automatic voice input and transcription model', () => {
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_INPUT_MODE]).toBe(VoiceInputMode.AUTO);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES]).toBe(true);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS]).toBe(false);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_BOOST]).toBe(true);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION]).toBe(false);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_ECHO_CANCELLATION]).toBe(false);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_AUTO_GAIN]).toBe(false);
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_TRANSCRIPTION_MODEL]).toBe(
+ VOICE_TRANSCRIPTION_MODEL_AUTO
+ );
+ expect(SETTING_CONFIG_DEFAULT[SETTINGS_KEYS.VOICE_TRANSCRIPTION_LANGUAGE]).toBe('');
+ });
+
+ it('have a section of their own, the model picker drawn by it', () => {
+ const keys = section?.fields?.map((field) => field.key);
+
+ expect(keys).toEqual([
+ SETTINGS_KEYS.VOICE_INPUT_MODE,
+ SETTINGS_KEYS.VOICE_TRANSCRIBE_MESSAGES,
+ SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS,
+ SETTINGS_KEYS.VOICE_BOOST,
+ SETTINGS_KEYS.VOICE_NOISE_SUPPRESSION,
+ SETTINGS_KEYS.VOICE_ECHO_CANCELLATION,
+ SETTINGS_KEYS.VOICE_AUTO_GAIN,
+ SETTINGS_KEYS.VOICE_TRANSCRIPTION_LANGUAGE
+ ]);
+ expect(section?.fields?.[0].type).toBe(SettingsFieldType.SELECT);
+ });
+});
+
+describe('isTranscriptionOnlyModel', () => {
+ it('knows speech-to-text models by name', () => {
+ expect(isTranscriptionOnlyModel('ggml-org/Qwen3-ASR-0.6B-GGUF:Q8_0')).toBe(true);
+ expect(isTranscriptionOnlyModel('ggerganov/whisper-large-v3')).toBe(true);
+ expect(isTranscriptionOnlyModel('nvidia/parakeet-tdt-0.6b')).toBe(true);
+ });
+
+ it('leaves chat models with audio input alone, even with the letters inside a word', () => {
+ expect(isTranscriptionOnlyModel('ggml-org/LFM3-Omni-2B-GGUF:Q8_0')).toBe(false);
+ expect(isTranscriptionOnlyModel('google/gemma-3n-E4B')).toBe(false);
+ expect(isTranscriptionOnlyModel('org/Lasr-chat')).toBe(false);
+ });
+});
+
+describe('dictationModelFor', () => {
+ const router = { isRouter: true, transcriptionModelHasAudio: true };
+
+ it('is the chosen transcription model when it hears', () => {
+ expect(dictationModelFor({ ...router, transcriptionModel: 'asr' })).toBe('asr');
+ expect(
+ dictationModelFor({ ...router, transcriptionModel: 'asr', transcriptionModelHasAudio: false })
+ ).toBeNull();
+ });
+
+ it('is a loaded model with audio input on Automatic, none when nothing hears', () => {
+ expect(
+ dictationModelFor({
+ ...router,
+ loadedAudioModel: 'omni',
+ transcriptionModel: VOICE_TRANSCRIPTION_MODEL_AUTO
+ })
+ ).toBe('omni');
+ expect(
+ dictationModelFor({ ...router, transcriptionModel: VOICE_TRANSCRIPTION_MODEL_AUTO })
+ ).toBeNull();
+ });
+
+ it("is a single-model server's model when it hears", () => {
+ const single = { isRouter: false, transcriptionModel: '', transcriptionModelHasAudio: false };
+
+ expect(dictationModelFor({ ...single, singleModel: 'omni', singleModelHasAudio: true })).toBe(
+ 'omni'
+ );
+ expect(
+ dictationModelFor({ ...single, singleModel: 'text', singleModelHasAudio: false })
+ ).toBeNull();
+ });
+});
+
+describe('voiceModelRole', () => {
+ it('tells speech-to-text models from chat models that hear, and leaves the rest out', () => {
+ expect(voiceModelRole('org/Qwen3-ASR', true)).toBe(VoiceModelRole.TRANSCRIPTION);
+ expect(voiceModelRole('omni', true)).toBe(VoiceModelRole.VOICE);
+ expect(voiceModelRole('text-only', false)).toBeNull();
+ });
+});
diff --git a/ui/apps/chat/tsconfig.json b/ui/apps/chat/tsconfig.json
index 574b1ab6a..74af0cce0 100644
--- a/ui/apps/chat/tsconfig.json
+++ b/ui/apps/chat/tsconfig.json
@@ -9,12 +9,15 @@
"@ggml-org/llama-ui/enums": ["../../lib/ui/src/lib/enums/index.ts"],
"@ggml-org/llama-ui/code": ["../../lib/ui/src/lib/utils/index.ts"],
"@ggml-org/llama-ui/utils": ["../../lib/ui/src/lib/components/ui/utils.ts"],
+ "@ggml-org/llama-ui/voice": ["../../lib/ui/src/lib/components/voice/index.ts"],
"@ggml-org/llama-ui/styles/*": ["../../lib/ui/src/lib/styles/*"],
"@ggml-org/llama-ui/theme.css": ["../../lib/ui/src/lib/styles/theme.css"],
"@ggml-org/llama-ui/tokens.css": ["../../lib/ui/src/lib/styles/tokens.css"],
"@ggml-org/llama-ui/fonts.css": ["../../lib/ui/src/lib/styles/fonts.css"],
"@ggml-org/llama-ui/surfaces.css": ["../../lib/ui/src/lib/styles/surfaces.css"],
"@ggml-org/llama-ui": ["../../lib/ui/src/lib/index.ts"],
+ "@ggml-org/llama-voice": ["../../lib/voice/src/index.ts"],
+ "@ggml-org/llama-voice/svelte": ["../../lib/voice/src/svelte/index.ts"],
"$llama-ui-stories/*": ["../../lib/ui/tests/stories/*"],
"@internationalized/date": ["./node_modules/@internationalized/date"],
"@internationalized/date/*": ["./node_modules/@internationalized/date/*"],
diff --git a/ui/apps/chat/vite.config.ts b/ui/apps/chat/vite.config.ts
index e88c853da..fb9c666a9 100644
--- a/ui/apps/chat/vite.config.ts
+++ b/ui/apps/chat/vite.config.ts
@@ -16,6 +16,7 @@ import { fileURLToPath } from 'url';
import { defineConfig, loadEnv, searchForWorkspaceRoot } from 'vite';
const __dirname = dirname(fileURLToPath(import.meta.url));
+const LLAMA_VOICE_SOURCE = resolve(__dirname, '../../lib/voice/src');
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const browserBaseConfig: any = {
enabled: true,
@@ -76,14 +77,25 @@ export default defineConfig(({ mode }) => {
],
resolve: {
- alias: {
- 'katex-fonts': resolve('node_modules/katex/dist/fonts')
- }
+ alias: [
+ // Llama Voice has no build step in the repo: the app compiles it from source. Exact
+ // matches, so the package's root does not swallow its ./svelte entry
+ {
+ find: /^@ggml-org\/llama-voice\/svelte$/,
+ replacement: resolve(LLAMA_VOICE_SOURCE, 'svelte/index.ts')
+ },
+ { find: /^@ggml-org\/llama-voice$/, replacement: resolve(LLAMA_VOICE_SOURCE, 'index.ts') },
+ { find: 'katex-fonts', replacement: resolve('node_modules/katex/dist/fonts') }
+ ]
},
server: {
fs: {
- allow: [searchForWorkspaceRoot(process.cwd()), resolve(__dirname, 'tests')]
+ allow: [
+ searchForWorkspaceRoot(process.cwd()),
+ resolve(__dirname, 'tests'),
+ LLAMA_VOICE_SOURCE
+ ]
},
headers: {
'Cross-Origin-Embedder-Policy': 'require-corp',
@@ -95,7 +107,8 @@ export default defineConfig(({ mode }) => {
'/props': SERVER_ORIGIN,
'/slots': SERVER_ORIGIN,
'/tools': SERVER_ORIGIN,
- '/v1': SERVER_ORIGIN
+ // ws: realtime transcription (/v1/realtime) is a WebSocket
+ '/v1': { target: SERVER_ORIGIN, ws: true }
}
},
diff --git a/ui/lib/ui/README.md b/ui/lib/ui/README.md
index 9fd1c350c..60abf7592 100644
--- a/ui/lib/ui/README.md
+++ b/ui/lib/ui/README.md
@@ -44,6 +44,10 @@ beside the apps built on it in `ui/apps`.
`ModelQuantOptions`, `ModelQuantChip`, `ModelQuantRow`, `ModelDraftSidecars`, `ModelDownloadProgressBar`, plus the formatters and
`minMemoryTierGb` and `summarizeSizes`, at `@ggml-org/llama-ui/models`. They take model data in and report what the user did out;
fetching and downloading stay in the app.
+- **Voice UI** (also `@ggml-org/llama-ui/voice`): `VoiceWaveformLive` (the waveform while recording), `VoiceWaveform` (a
+ recorded clip: play, seek, time) and `VoiceTranscriptReveal` (a clip with its transcript, opened on hover, keyboard
+ focus or a tap), plus `formatClipTime` and the waveform math. They take levels, peaks, an audio URL and a transcript
+ in; recording and transcribing stay in the app.
Next: llama.app's model page.
diff --git a/ui/lib/ui/package.json b/ui/lib/ui/package.json
index bee001ae5..1def1aae4 100644
--- a/ui/lib/ui/package.json
+++ b/ui/lib/ui/package.json
@@ -54,6 +54,10 @@
"types": "./dist/components/models/index.d.ts",
"svelte": "./dist/components/models/index.js"
},
+ "./voice": {
+ "types": "./dist/components/voice/index.d.ts",
+ "svelte": "./dist/components/voice/index.js"
+ },
"./utils": {
"types": "./dist/components/ui/utils.d.ts",
"svelte": "./dist/components/ui/utils.js"
diff --git a/ui/lib/ui/src/lib/components/voice/VoiceRecordingBar.svelte b/ui/lib/ui/src/lib/components/voice/VoiceRecordingBar.svelte
new file mode 100644
index 000000000..dc58005db
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/VoiceRecordingBar.svelte
@@ -0,0 +1,185 @@
+<!--
+ The row shown while a recording is under way: cancel, the live waveform and the time, the app's stop button, and
+ an optional send, which stops and sends in one go. While the recording is `waiting` (its text is still on the way)
+ a spinner takes the waveform's place. A `warning` (e.g. the speech is too quiet) stays inside the row, so nothing
+ around it moves: the waveform turns the warning colour, an icon sits by the time, and the text opens above the row
+ in a tooltip for a few seconds, then again on hover or focus. Under the row: the `caption`, text that streams in
+ while someone speaks.
+ It takes everything in and reports cancel and send; recording, and what stopping does, stay in the app, which
+ passes its stop button as the `stop` snippet. The default labels are English; an app passes its own through `labels`.
+-->
+<script lang="ts" module>
+ export interface VoiceRecordingBarLabels {
+ /** The cancel button, for screen readers */
+ cancel: string;
+ /** The send button, for screen readers */
+ send: string;
+ /** In place of the time until the first audio arrives */
+ starting: string;
+ /** In place of the waveform while the text is on its way */
+ waiting: string;
+ }
+</script>
+
+<script lang="ts">
+ import { ArrowUp, LoaderCircle, TriangleAlert, X } from '@lucide/svelte';
+ import type { Snippet } from 'svelte';
+ import { ICON_CLASS_DEFAULT } from '../../constants/css-classes.constants';
+ import {
+ VOICE_RECORDING_BAR_LABELS,
+ VOICE_WARNING_SHOWN_MS
+ } from '../../constants/voice.constants';
+ import { VoiceRecordingBarStatus } from '../../enums/voice.enums';
+ import { formatClipTime } from '../../voice/waveform';
+ import { Button } from '../ui/button';
+ import * as Tooltip from '../ui/tooltip';
+ import { cn } from '../ui/utils';
+ import VoiceSpokenText from './VoiceSpokenText.svelte';
+ import VoiceWaveformLive from './VoiceWaveformLive.svelte';
+
+ interface Props {
+ /** Input levels, 0..1, newest last. */
+ levels: number[];
+ /** Time recorded so far. */
+ elapsedMs: number;
+ status: VoiceRecordingBarStatus;
+ /** The stop button; `disabled` while nothing is being recorded yet. */
+ stop: Snippet<[{ disabled: boolean }]>;
+ onCancel: () => void;
+ /** Stops and sends; left out, there is no send button. */
+ onSend?: () => void;
+ /** What is being recorded, read by screen readers in place of the waveform, e.g. `Dictating`. */
+ label?: string;
+ /** Text streaming in while someone speaks. */
+ caption?: string;
+ /** A problem with the recording, said under the row while it lasts. */
+ warning?: string | null;
+ labels?: Partial<VoiceRecordingBarLabels>;
+ class?: string;
+ }
+
+ let {
+ caption = '',
+ class: className,
+ elapsedMs,
+ label,
+ labels = {},
+ levels,
+ onCancel,
+ onSend,
+ status,
+ stop,
+ warning = null
+ }: Props = $props();
+
+ let text = $derived({ ...VOICE_RECORDING_BAR_LABELS, ...labels });
+ let recording = $derived(status === VoiceRecordingBarStatus.RECORDING);
+ let warningOpen = $state(false);
+
+ // a warning that appears opens by itself, as nobody hovers an icon they have not seen yet
+ $effect(() => {
+ if (!warning) {
+ warningOpen = false;
+
+ return;
+ }
+
+ warningOpen = true;
+
+ const timer = setTimeout(() => (warningOpen = false), VOICE_WARNING_SHOWN_MS);
+
+ return () => clearTimeout(timer);
+ });
+</script>
+
+<div class={cn('flex w-full min-w-0 flex-col gap-1', className)} data-slot="voice-recording-bar">
+ <div class="flex w-full min-w-0 items-center gap-2">
+ <Button
+ class="size-8 shrink-0 rounded-full p-0 text-muted-foreground max-md:size-9"
+ onclick={onCancel}
+ type="button"
+ variant="ghost"
+ >
+ <span class="sr-only">{text.cancel}</span>
+
+ <X class={ICON_CLASS_DEFAULT} />
+ </Button>
+
+ {#if status === VoiceRecordingBarStatus.WAITING}
+ <div
+ class="flex min-w-0 flex-1 items-center gap-2 px-1 text-caption text-muted-foreground"
+ role="status"
+ >
+ <LoaderCircle class="size-icon-sm animate-spin motion-reduce:animate-none" />
+ {text.waiting}
+ </div>
+ {:else}
+ <VoiceWaveformLive
+ active={recording}
+ class={cn('flex-1 px-1', warning && 'text-fg-warning')}
+ {label}
+ {levels}
+ />
+
+ {#if warning}
+ <Tooltip.Root bind:open={warningOpen}>
+ <Tooltip.Trigger>
+ {#snippet child({ props })}
+ <button
+ {...props}
+ aria-label={warning}
+ class="flex size-5 shrink-0 items-center justify-center rounded-full text-fg-warning"
+ data-slot="voice-recording-bar-warning"
+ type="button"
+ >
+ <TriangleAlert aria-hidden="true" class="size-icon-sm" />
+ </button>
+ {/snippet}
+ </Tooltip.Trigger>
+
+ <Tooltip.Content class="max-w-64" side="top">
+ <p>{warning}</p>
+ </Tooltip.Content>
+ </Tooltip.Root>
+
+ <!-- read out when it appears; the tooltip is for the eye -->
+ <span class="sr-only" role="status">{warning}</span>
+ {/if}
+
+ <!-- the timer starts with the first audio: until then, what is said is not recorded yet -->
+ {#if status === VoiceRecordingBarStatus.STARTING}
+ <span class="shrink-0 text-caption text-muted-foreground" role="status">{text.starting}</span>
+ {:else}
+ <span class="w-10 shrink-0 text-right text-caption text-muted-foreground tabular-nums">
+ {formatClipTime(elapsedMs)}
+ </span>
+ {/if}
+ {/if}
+
+ {@render stop({ disabled: !recording })}
+
+ {#if onSend}
+ <Button
+ class="size-8 shrink-0 rounded-full p-0 max-md:size-9"
+ disabled={!recording}
+ onclick={onSend}
+ type="button"
+ >
+ <span class="sr-only">{text.send}</span>
+
+ <ArrowUp class="size-icon-md" />
+ </Button>
+ {/if}
+ </div>
+
+
+ {#if caption}
+ <p
+ aria-live="polite"
+ class="line-clamp-2 px-10 text-body-sm break-words text-muted-foreground max-md:px-11"
+ data-slot="voice-recording-bar-caption"
+ >
+ <VoiceSpokenText live text={caption} />
+ </p>
+ {/if}
+</div>
diff --git a/ui/lib/ui/src/lib/components/voice/VoiceSpokenText.svelte b/ui/lib/ui/src/lib/components/voice/VoiceSpokenText.svelte
new file mode 100644
index 000000000..5298bda18
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/VoiceSpokenText.svelte
@@ -0,0 +1,53 @@
+<!--
+ Text that is being spoken: what has been said so far in the full foreground colour, which it keeps, and the rest
+ muted. Used for a transcript followed during playback (`spoken`, e.g. from spokenLength()), for a reply read aloud,
+ and for text that streams in while someone speaks (`live`: each new word brightens as it arrives).
+ Without `spoken` or `live` all of it reads as spoken. The words are plain text for screen readers.
+-->
+<script lang="ts">
+ import { cn } from '../ui/utils';
+ import { spokenWords } from '../../voice/spoken';
+
+ interface Props {
+ text: string;
+ /** Characters spoken so far; left out, all of them. */
+ spoken?: number | null;
+ /** Text arriving as it is said: new words come in muted and brighten. */
+ live?: boolean;
+ class?: string;
+ }
+
+ let { class: className, live = false, spoken = null, text }: Props = $props();
+
+ let words = $derived(spokenWords(text));
+ let reached = $derived(spoken ?? text.length);
+</script>
+
+<span class={cn('whitespace-pre-wrap', className)} data-slot="voice-spoken-text"
+ >{#each words as { start, word }, index (live ? index : start)}<span
+ class={[
+ 'transition-colors duration-300 ease-out motion-reduce:transition-none',
+ start < reached ? 'text-foreground' : 'text-muted-foreground',
+ live && 'spoken-in'
+ ]}
+ data-spoken={start < reached || undefined}>{word}</span
+ >{/each}</span
+>
+
+<style>
+ @keyframes spoken-in {
+ from {
+ color: var(--fg-muted, currentColor);
+ }
+ }
+
+ .spoken-in {
+ animation: spoken-in 400ms ease-out;
+ }
+
+ @media (prefers-reduced-motion: reduce) {
+ .spoken-in {
+ animation: none;
+ }
+ }
+</style>
diff --git a/ui/lib/ui/src/lib/components/voice/VoiceTranscriptReveal.svelte b/ui/lib/ui/src/lib/components/voice/VoiceTranscriptReveal.svelte
new file mode 100644
index 000000000..8f867e1f4
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/VoiceTranscriptReveal.svelte
@@ -0,0 +1,171 @@
+<!--
+ A voice clip with its transcript under it, open from the start with `defaultOpen` (a toggle key beside the clip folds
+ it). Otherwise only the clip (the `children`, usually a VoiceWaveform) shows at first; the transcript opens below on
+ hover and on keyboard focus inside, or with a tap where there is no hover (the toggle key, or a tap on the clip
+ outside its controls). While it is transcribing, the open transcript shows that it is on its way.
+ The transcript stays in the DOM while closed, so screen readers always reach it.
+-->
+<script lang="ts">
+ import { ChevronDown } from '@lucide/svelte';
+ import type { Snippet } from 'svelte';
+ import { Button } from '../ui/button';
+ import { Skeleton } from '../ui/skeleton';
+ import VoiceSpokenText from './VoiceSpokenText.svelte';
+ import { cn } from '../ui/utils';
+ import {
+ VOICE_INTERACTIVE_SELECTOR,
+ VOICE_NO_HOVER_QUERY,
+ VOICE_TRANSCRIPT_LABELS
+ } from '../../constants/voice.constants';
+ import { PointerType } from '../../enums/ui.enums';
+ import { VoiceTranscriptStatus } from '../../enums/voice.enums';
+
+ interface Props {
+ /** The clip, usually a VoiceWaveform. */
+ children: Snippet;
+ transcript?: string | null;
+ /** Left out: `done` with a transcript, `idle` without. */
+ status?: VoiceTranscriptStatus;
+ /** Why there is no transcript, in words for the user, said after the `error` status' own label. */
+ errorReason?: string | null;
+ /** Start with the transcript open; read once, the toggle key changes it after that. */
+ defaultOpen?: boolean;
+ /** Characters of the transcript spoken so far, to follow playback; left out, all of it reads as spoken. */
+ spoken?: number | null;
+ class?: string;
+ }
+
+ let {
+ children,
+ class: className,
+ defaultOpen = false,
+ errorReason = null,
+ spoken = null,
+ status,
+ transcript = null
+ }: Props = $props();
+
+ const id = $props.id();
+
+ let root = $state<HTMLDivElement>();
+ const initiallyOpen = () => defaultOpen;
+ let open = $state(initiallyOpen());
+ let noHover = $state(false);
+ let hovered = $state(false);
+ let keyboardFocus = $state(false);
+
+ let shown = $derived<VoiceTranscriptStatus>(status ?? (transcript ? VoiceTranscriptStatus.DONE : VoiceTranscriptStatus.IDLE));
+ let hasContent = $derived(shown !== VoiceTranscriptStatus.IDLE);
+ let showToggle = $derived(hasContent && (noHover || open));
+ let revealed = $derived(hasContent && (open || hovered || keyboardFocus));
+
+ // plain matchMedia: in the app's type check svelte/reactivity resolves to the library's own copy of Svelte
+ $effect(() => {
+ const query = window.matchMedia(VOICE_NO_HOVER_QUERY);
+ const update = () => (noHover = query.matches);
+
+ update();
+ query.addEventListener('change', update);
+
+ return () => query.removeEventListener('change', update);
+ });
+
+ function handlePointerEnter(event: PointerEvent) {
+ if (event.pointerType === PointerType.MOUSE) hovered = true;
+ }
+
+ function handlePointerLeave(event: PointerEvent) {
+ if (event.pointerType === PointerType.MOUSE) hovered = false;
+ }
+
+ // a tap where there is no hover toggles; a mouse opens it by hovering instead
+ function handlePointerUp(event: PointerEvent) {
+ if (!hasContent || event.pointerType === PointerType.MOUSE) return;
+ if ((event.target as Element | null)?.closest(VOICE_INTERACTIVE_SELECTOR)) return;
+
+ open = !open;
+ }
+
+ // only keyboard focus opens it: after a click the mouse already hovers, and a tap toggles
+ function handleFocusIn(event: FocusEvent) {
+ keyboardFocus = (event.target as Element | null)?.matches(':focus-visible') ?? false;
+ }
+
+ function handleFocusOut(event: FocusEvent) {
+ if (!root?.contains(event.relatedTarget as Node | null)) keyboardFocus = false;
+ }
+</script>
+
+<!-- the tap is a shortcut: the toggle key does the same for keyboards and screen readers -->
+<!-- svelte-ignore a11y_no_static_element_interactions -->
+<div
+ bind:this={root}
+ class={cn('flex min-w-0 flex-col', className)}
+ data-revealed={revealed || undefined}
+ data-slot="voice-transcript-reveal"
+ data-status={shown}
+ onfocusin={handleFocusIn}
+ onfocusout={handleFocusOut}
+ onpointerenter={handlePointerEnter}
+ onpointerleave={handlePointerLeave}
+ onpointerup={handlePointerUp}
+>
+ <div class="flex min-w-0 items-center gap-1">
+ <div class="min-w-0 flex-1">
+ {@render children()}
+ </div>
+
+ {#if showToggle}
+ <Button
+ class="size-8 shrink-0 rounded-full p-0 text-muted-foreground"
+ aria-controls={id}
+ aria-expanded={open}
+ aria-label={open ? VOICE_TRANSCRIPT_LABELS.hide : VOICE_TRANSCRIPT_LABELS.show}
+ onclick={() => (open = !open)}
+ variant="ghost"
+ >
+ <ChevronDown
+ class={cn(
+ 'size-icon-md transition-transform duration-200 ease-out motion-reduce:transition-none',
+ open && 'rotate-180'
+ )}
+ />
+ </Button>
+ {/if}
+ </div>
+
+ {#if hasContent}
+ <div
+ {id}
+ class={[
+ 'grid transition-[grid-template-rows,opacity] duration-200 ease-out motion-reduce:transition-none',
+ revealed ? 'grid-rows-[1fr] opacity-100' : 'grid-rows-[0fr] opacity-0'
+ ]}
+ aria-busy={shown === VoiceTranscriptStatus.TRANSCRIBING || undefined}
+ data-slot="voice-transcript"
+ >
+ <div class="min-h-0 overflow-hidden">
+ <div class="pt-2">
+ {#if shown === VoiceTranscriptStatus.TRANSCRIBING}
+ <div class="flex flex-col gap-1.5">
+ <span class="text-caption text-muted-foreground">{VOICE_TRANSCRIPT_LABELS.transcribing}</span>
+ <Skeleton
+ class="h-2.5 w-3/4 rounded-full bg-muted-foreground/20 motion-reduce:animate-none"
+ />
+ </div>
+ {:else if shown === VoiceTranscriptStatus.ERROR}
+ <p class="text-caption break-words text-muted-foreground">
+ {VOICE_TRANSCRIPT_LABELS.error}{errorReason ? `: ${errorReason}` : ''}
+ </p>
+ {:else if transcript}
+ <p class="text-body-sm break-words whitespace-pre-wrap text-muted-foreground">
+ <span class="sr-only">{VOICE_TRANSCRIPT_LABELS.prefix}</span><VoiceSpokenText {spoken} text={transcript} />
+ </p>
+ {:else}
+ <p class="text-caption text-muted-foreground">{VOICE_TRANSCRIPT_LABELS.empty}</p>
+ {/if}
+ </div>
+ </div>
+ </div>
+ {/if}
+</div>
diff --git a/ui/lib/ui/src/lib/components/voice/VoiceWaveform.svelte b/ui/lib/ui/src/lib/components/voice/VoiceWaveform.svelte
new file mode 100644
index 000000000..f27cea028
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/VoiceWaveform.svelte
@@ -0,0 +1,303 @@
+<!--
+ A recorded voice clip: a play button, its waveform with the played part filled in, and the time. `sm` is the chip in
+ the chat form, `md` the clip in a message bubble; the surface around it is the caller's.
+ The waveform is a slider: click or drag to seek, arrow keys move 5 s. Playback uses its own <audio>, and starting
+ one clip pauses any other. Without `src` it shows the waveform and the length only.
+-->
+<script lang="ts" module>
+ // one clip plays at a time: starting one pauses the one before
+ let playingAudio: HTMLAudioElement | null = null;
+</script>
+
+<script lang="ts">
+ import { Pause, Play } from '@lucide/svelte';
+ import { onDestroy } from 'svelte';
+ import { Button } from '../ui/button';
+ import { cn } from '../ui/utils';
+ import {
+ VOICE_BAR_GAP_PX,
+ VOICE_SEEK_STEP_MS,
+ VOICE_WAVEFORM_DEFAULT_LABEL,
+ VOICE_WAVEFORM_SIZES
+ } from '../../constants/voice.constants';
+ import { KeyboardKey, MouseButton } from '../../enums/ui.enums';
+ import { VoiceWaveformSize } from '../../enums/voice.enums';
+ import {
+ barCapacity,
+ barHeightPercent,
+ formatClipPosition,
+ formatClipTime,
+ pointerFraction,
+ progressFraction,
+ resamplePeaks,
+ stepPosition
+ } from '../../voice/waveform';
+
+ interface Props {
+ /** The clip's peaks, 0..1, resampled to the bars that fit. */
+ peaks: number[];
+ /** The audio to play: an object URL or a data URL. */
+ src?: string;
+ /** The clip's length; read from the audio when left out. */
+ durationMs?: number;
+ size?: VoiceWaveformSize;
+ /** What the clip is, for the play button and the slider. */
+ label?: string;
+ /**
+ * Bound out: where playback is, to follow a transcript with. Null until the clip first plays, the clip's end
+ * once it played through.
+ */
+ playbackMs?: number | null;
+ class?: string;
+ }
+
+ let {
+ class: className,
+ durationMs,
+ label = VOICE_WAVEFORM_DEFAULT_LABEL,
+ peaks,
+ playbackMs = $bindable(null),
+ size = VoiceWaveformSize.MD,
+ src
+ }: Props = $props();
+
+ let audio = $state<HTMLAudioElement>();
+ let track = $state<HTMLDivElement>();
+ let paused = $state(true);
+ let currentTime = $state(0);
+ let mediaDuration = $state(0);
+ let failed = $state(false);
+ let dragging = $state(false);
+ let width = $state(0);
+
+ let sized = $derived(VOICE_WAVEFORM_SIZES[size]);
+ let playable = $derived(!!src && !failed);
+ let totalMs = $derived(
+ durationMs ?? (Number.isFinite(mediaDuration) ? Math.max(mediaDuration, 0) * 1000 : 0)
+ );
+ let elapsedMs = $derived(Math.min(currentTime * 1000, totalMs || Infinity));
+ let progress = $derived(progressFraction(elapsedMs, totalMs));
+ let bars = $derived(resamplePeaks(peaks, barCapacity(width, sized.barWidthPx, VOICE_BAR_GAP_PX)));
+ // the length until playback starts, then where it is
+ let shownMs = $derived(!paused || elapsedMs > 0 ? elapsedMs : totalMs);
+
+ // playback as it goes, read every frame while playing: the media element's own updates come a few times a second
+ let started = $state(false);
+ let playedThrough = $state(false);
+
+ $effect(() => {
+ if (!started) {
+ playbackMs = null;
+
+ return;
+ }
+
+ if (playedThrough) {
+ playbackMs = totalMs;
+
+ return;
+ }
+
+ playbackMs = elapsedMs;
+
+ if (paused || !audio) return;
+
+ const media = audio;
+ let frame = requestAnimationFrame(function tick() {
+ playbackMs = Math.min(media.currentTime * 1000, totalMs || Infinity);
+ frame = requestAnimationFrame(tick);
+ });
+
+ return () => cancelAnimationFrame(frame);
+ });
+
+ $effect(() => {
+ if (!src) paused = true;
+
+ failed = false;
+ currentTime = 0;
+ });
+
+ onDestroy(() => {
+ if (playingAudio === audio) playingAudio = null;
+ });
+
+ function toggle() {
+ if (!audio || !playable) return;
+
+ if (audio.paused) audio.play().catch(() => {});
+ else audio.pause();
+ }
+
+ function seekTo(ms: number) {
+ if (!audio || !playable) return;
+
+ audio.currentTime = ms / 1000;
+ currentTime = ms / 1000;
+ }
+
+ function seekToPointer(event: PointerEvent) {
+ if (!track) return;
+
+ const rect = track.getBoundingClientRect();
+
+ seekTo(pointerFraction(event.clientX, rect.left, rect.width) * totalMs);
+ }
+
+ function handlePointerDown(event: PointerEvent) {
+ if (!playable || event.button !== MouseButton.PRIMARY) return;
+
+ try {
+ track?.setPointerCapture(event.pointerId);
+ } catch {
+ // a scripted pointer has no capture to take; the press still seeks
+ }
+
+ dragging = true;
+ seekToPointer(event);
+ }
+
+ function handlePointerMove(event: PointerEvent) {
+ if (dragging) seekToPointer(event);
+ }
+
+ function handlePointerEnd(event: PointerEvent) {
+ if (!dragging) return;
+
+ dragging = false;
+
+ if (track?.hasPointerCapture(event.pointerId)) track.releasePointerCapture(event.pointerId);
+ }
+
+ function handleKeydown(event: KeyboardEvent) {
+ if (!playable) return;
+
+ const target =
+ event.key === KeyboardKey.ARROW_RIGHT || event.key === KeyboardKey.ARROW_UP
+ ? stepPosition(elapsedMs, VOICE_SEEK_STEP_MS, totalMs)
+ : event.key === KeyboardKey.ARROW_LEFT || event.key === KeyboardKey.ARROW_DOWN
+ ? stepPosition(elapsedMs, -VOICE_SEEK_STEP_MS, totalMs)
+ : event.key === KeyboardKey.HOME
+ ? 0
+ : event.key === KeyboardKey.END
+ ? totalMs
+ : null;
+
+ if (target === null) return;
+
+ event.preventDefault();
+ seekTo(target);
+ }
+
+ function handlePlay() {
+ if (playingAudio && playingAudio !== audio) playingAudio.pause();
+
+ started = true;
+ playedThrough = false;
+
+ playingAudio = audio ?? null;
+ }
+
+ function handlePause() {
+ if (playingAudio === audio) playingAudio = null;
+ }
+
+ function handleEnded() {
+ playedThrough = true;
+
+ if (audio) audio.currentTime = 0;
+
+ currentTime = 0;
+ }
+</script>
+
+{#snippet barRow(className: string)}
+ <div style:gap="{VOICE_BAR_GAP_PX}px" class={cn('flex h-full items-center', className)} aria-hidden="true">
+ {#each bars as peak, index (index)}
+ <span
+ style:width="{sized.barWidthPx}px"
+ style:height="{barHeightPercent(peak)}%"
+ class="shrink-0 rounded-full bg-current"
+ ></span>
+ {/each}
+ </div>
+{/snippet}
+
+<div
+ class={cn('flex w-full min-w-0 items-center', sized.gap, className)}
+ data-size={size}
+ data-slot="voice-waveform"
+>
+ <Button
+ class={cn('shrink-0 rounded-full p-0', sized.button)}
+ aria-label={paused ? `Play ${label.toLowerCase()}` : `Pause ${label.toLowerCase()}`}
+ disabled={!playable}
+ onclick={toggle}
+ >
+ {#if paused}
+ <Play
+ class={cn('translate-x-px fill-current', sized.icon)}
+ />
+ {:else}
+ <Pause class={cn('fill-current', sized.icon)} />
+ {/if}
+ </Button>
+
+ <div
+ bind:this={track}
+ class={cn(
+ 'relative min-w-0 flex-1 touch-none rounded-md outline-none select-none focus-visible:ring-3 focus-visible:ring-ring/50',
+ sized.track,
+ playable ? 'cursor-pointer' : 'cursor-default'
+ )}
+ aria-disabled={!playable || undefined}
+ aria-label="{label} position"
+ aria-valuemax={Math.round(totalMs / 1000)}
+ aria-valuemin={0}
+ aria-valuenow={Math.round(elapsedMs / 1000)}
+ aria-valuetext={formatClipPosition(elapsedMs, totalMs)}
+ bind:clientWidth={width}
+ onkeydown={handleKeydown}
+ onlostpointercapture={() => (dragging = false)}
+ onpointercancel={handlePointerEnd}
+ onpointerdown={handlePointerDown}
+ onpointermove={handlePointerMove}
+ onpointerup={handlePointerEnd}
+ role="slider"
+ tabindex={playable ? 0 : -1}
+ >
+ {@render barRow('text-muted-foreground/40')}
+
+ <div
+ style:clip-path="inset(0 {100 - progress * 100}% 0 0)"
+ class="absolute inset-0 text-primary"
+ >
+ {@render barRow('')}
+ </div>
+ </div>
+
+ <span
+ class={cn(
+ 'shrink-0 text-right text-caption text-muted-foreground tabular-nums',
+ sized.time
+ )}
+ aria-hidden="true"
+ >
+ {formatClipTime(shownMs)}
+ </span>
+
+ {#if src}
+ <audio
+ bind:this={audio}
+ bind:currentTime
+ bind:duration={mediaDuration}
+ bind:paused
+ onended={handleEnded}
+ onerror={() => (failed = true)}
+ onpause={handlePause}
+ onplay={handlePlay}
+ preload="metadata"
+ {src}
+ ></audio>
+ {/if}
+</div>
diff --git a/ui/lib/ui/src/lib/components/voice/VoiceWaveformLive.svelte b/ui/lib/ui/src/lib/components/voice/VoiceWaveformLive.svelte
new file mode 100644
index 000000000..3b9e70ec8
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/VoiceWaveformLive.svelte
@@ -0,0 +1,58 @@
+<!--
+ The waveform while recording, sized to sit in the chat form's input row and always filling its width: a short
+ recording is stretched over all the bars, a longer one scrolls with the newest level on the right. The app passes
+ the levels; this only draws them.
+ Decorative: hidden from screen readers, with an optional visually hidden `label` (e.g. `Recording`) in its place.
+ With reduced motion the bars stay still.
+-->
+<script lang="ts">
+ import { prefersReducedMotion } from 'svelte/motion';
+ import { cn } from '../ui/utils';
+ import { VOICE_BAR_GAP_PX, VOICE_LIVE_BAR_WIDTH_PX } from '../../constants/voice.constants';
+ import { barCapacity, barHeightPercent, liveBars } from '../../voice/waveform';
+
+ interface Props {
+ /** Input levels, 0..1, newest last. */
+ levels: number[];
+ /** Off when paused or not listening: the bars turn muted. */
+ active?: boolean;
+ /** Read by screen readers in place of the bars, e.g. `Recording`. */
+ label?: string;
+ class?: string;
+ }
+
+ let { active = true, class: className, label, levels }: Props = $props();
+
+ let width = $state(0);
+
+ let bars = $derived(
+ liveBars(prefersReducedMotion.current ? [] : levels, barCapacity(width, VOICE_LIVE_BAR_WIDTH_PX, VOICE_BAR_GAP_PX))
+ );
+</script>
+
+<div
+ class={cn('relative h-6 w-full min-w-0', active ? 'text-foreground' : 'text-muted-foreground', className)}
+ data-slot="voice-waveform-live"
+>
+ {#if label}
+ <span class="sr-only">{label}</span>
+ {/if}
+
+ <div
+ style:gap="{VOICE_BAR_GAP_PX}px"
+ class="flex h-full items-center justify-end overflow-hidden"
+ aria-hidden="true"
+ bind:clientWidth={width}
+ >
+ {#each bars as level, index (index)}
+ <span
+ style:width="{VOICE_LIVE_BAR_WIDTH_PX}px"
+ style:height="{barHeightPercent(level)}%"
+ class={[
+ 'shrink-0 rounded-full bg-current transition-[height] duration-100 ease-linear motion-reduce:transition-none',
+ level === null && 'opacity-40'
+ ]}
+ ></span>
+ {/each}
+ </div>
+</div>
diff --git a/ui/lib/ui/src/lib/components/voice/index.ts b/ui/lib/ui/src/lib/components/voice/index.ts
new file mode 100644
index 000000000..d65c88c20
--- /dev/null
+++ b/ui/lib/ui/src/lib/components/voice/index.ts
@@ -0,0 +1,14 @@
+/**
+ * Voice UI: how a voice clip is shown while it is recorded, once it is recorded, and with its transcript. Every
+ * component takes its data in (levels, peaks, an audio URL, a transcript); recording, analysing and transcribing stay
+ * in each app.
+ */
+export {
+ default as VoiceRecordingBar,
+ type VoiceRecordingBarLabels,
+} from "./VoiceRecordingBar.svelte";
+export { default as VoiceSpokenText } from "./VoiceSpokenText.svelte";
+export { default as VoiceTranscriptReveal } from "./VoiceTranscriptReveal.svelte";
+export { default as VoiceWaveform } from "./VoiceWaveform.svelte";
+export { default as VoiceWaveformLive } from "./VoiceWaveformLive.svelte";
+export * from "../../voice/index.js";
diff --git a/ui/lib/ui/src/lib/constants/index.ts b/ui/lib/ui/src/lib/constants/index.ts
index a9cdefa47..f46c2baa4 100644
--- a/ui/lib/ui/src/lib/constants/index.ts
+++ b/ui/lib/ui/src/lib/constants/index.ts
@@ -1,3 +1,4 @@
export * from './css-classes.constants';
export * from './code-block.constants';
export * from './ui.constants';
+export * from './voice.constants';
diff --git a/ui/lib/ui/src/lib/constants/voice.constants.ts b/ui/lib/ui/src/lib/constants/voice.constants.ts
new file mode 100644
index 000000000..d5552c040
--- /dev/null
+++ b/ui/lib/ui/src/lib/constants/voice.constants.ts
@@ -0,0 +1,76 @@
+import { VoiceWaveformSize } from '../enums/voice.enums.js';
+
+/** A peak under this share of the clip's loudest one counts as a pause, not speech. */
+export const VOICE_SPEECH_PEAK_FLOOR = 0.15;
+
+/** The smallest bar, as a fraction of the waveform's height, so silence still draws a dotted line. */
+export const MIN_BAR_HEIGHT = 0.12;
+
+/** Whitespace, where a word ends. */
+export const VOICE_WHITESPACE_PATTERN = /\s/;
+
+/** A word with the whitespace around it, or whitespace alone. */
+export const VOICE_WORD_PATTERN = /\s*\S+\s*|\s+/g;
+
+/** Space between two waveform bars, in px. */
+export const VOICE_BAR_GAP_PX = 2;
+
+/** Width of a live waveform bar, in px. */
+export const VOICE_LIVE_BAR_WIDTH_PX = 3;
+
+/** How far the arrow keys move a clip's position. */
+export const VOICE_SEEK_STEP_MS = 5000;
+
+/** How long a new recording warning stays open above the row before it waits for hover or focus. */
+export const VOICE_WARNING_SHOWN_MS = 6000;
+
+/** What counts as a control inside a transcript reveal: a tap on one does not toggle the transcript. */
+export const VOICE_INTERACTIVE_SELECTOR =
+ 'button, a, input, textarea, select, [role="slider"], [tabindex]';
+
+/** Media query of a device without hover, where a tap opens the transcript. */
+export const VOICE_NO_HOVER_QUERY = '(hover: none)';
+
+/** Bar width and classes of a VoiceWaveform at each size. */
+export const VOICE_WAVEFORM_SIZES = {
+ [VoiceWaveformSize.SM]: {
+ barWidthPx: 2,
+ button: 'size-7',
+ gap: 'gap-2',
+ icon: 'size-icon-sm',
+ time: 'min-w-8',
+ track: 'h-5'
+ },
+ [VoiceWaveformSize.MD]: {
+ barWidthPx: 3,
+ button: 'size-8',
+ gap: 'gap-3',
+ icon: 'size-icon-md',
+ time: 'min-w-9',
+ track: 'h-7'
+ }
+} as const satisfies Record<
+ VoiceWaveformSize,
+ { barWidthPx: number; button: string; gap: string; icon: string; time: string; track: string }
+>;
+
+/** What a VoiceWaveform is called when the app names it nothing else. */
+export const VOICE_WAVEFORM_DEFAULT_LABEL = 'Voice message';
+
+/** The recording bar's default labels, in English; an app passes its own. */
+export const VOICE_RECORDING_BAR_LABELS = {
+ cancel: 'Cancel recording',
+ send: 'Stop and send',
+ starting: 'Starting...',
+ waiting: 'Transcribing...'
+} as const;
+
+/** What a transcript reveal says. */
+export const VOICE_TRANSCRIPT_LABELS = {
+ empty: 'No speech detected',
+ error: 'Transcription unavailable',
+ hide: 'Hide transcript',
+ prefix: 'Transcript: ',
+ show: 'Show transcript',
+ transcribing: 'Transcribing...'
+} as const;
diff --git a/ui/lib/ui/src/lib/enums/index.ts b/ui/lib/ui/src/lib/enums/index.ts
index f0d4cc1de..5801ef0db 100644
--- a/ui/lib/ui/src/lib/enums/index.ts
+++ b/ui/lib/ui/src/lib/enums/index.ts
@@ -1,3 +1,4 @@
// The enums the components share with the apps built on them.
export { BooleanString } from './boolean-string.enums.js';
-export { ColorMode, TooltipSide } from './ui.enums.js';
+export { ColorMode, KeyboardKey, MouseButton, PointerType, TooltipSide } from './ui.enums.js';
+export { VoiceRecordingBarStatus, VoiceTranscriptStatus, VoiceWaveformSize } from './voice.enums.js';
diff --git a/ui/lib/ui/src/lib/enums/ui.enums.ts b/ui/lib/ui/src/lib/enums/ui.enums.ts
index f4b1188cc..cf626e753 100644
--- a/ui/lib/ui/src/lib/enums/ui.enums.ts
+++ b/ui/lib/ui/src/lib/enums/ui.enums.ts
@@ -11,3 +11,25 @@ export enum TooltipSide {
RIGHT = 'right',
TOP = 'top'
}
+
+// From ui/apps/chat's src/lib/enums/keyboard.enums.ts: the keys the components handle.
+export enum KeyboardKey {
+ ARROW_DOWN = 'ArrowDown',
+ ARROW_LEFT = 'ArrowLeft',
+ ARROW_RIGHT = 'ArrowRight',
+ ARROW_UP = 'ArrowUp',
+ END = 'End',
+ HOME = 'Home'
+}
+
+/** Mouse buttons a pointer event reports (MouseEvent.button). */
+export enum MouseButton {
+ PRIMARY = 0
+}
+
+/** What a pointer event comes from (PointerEvent.pointerType). */
+export enum PointerType {
+ MOUSE = 'mouse',
+ PEN = 'pen',
+ TOUCH = 'touch'
+}
diff --git a/ui/lib/ui/src/lib/enums/voice.enums.ts b/ui/lib/ui/src/lib/enums/voice.enums.ts
new file mode 100644
index 000000000..8abfada4a
--- /dev/null
+++ b/ui/lib/ui/src/lib/enums/voice.enums.ts
@@ -0,0 +1,20 @@
+/** Where a recording is, as the recording bar shows it: waiting for the microphone, recording, or waiting for its text. */
+export enum VoiceRecordingBarStatus {
+ RECORDING = 'recording',
+ STARTING = 'starting',
+ WAITING = 'waiting'
+}
+
+/** Where a voice clip's transcript is: none asked for, on its way, there, or failed. */
+export enum VoiceTranscriptStatus {
+ DONE = 'done',
+ ERROR = 'error',
+ IDLE = 'idle',
+ TRANSCRIBING = 'transcribing'
+}
+
+/** `SM` for a chip in the chat form, `MD` for a message bubble. */
+export enum VoiceWaveformSize {
+ MD = 'md',
+ SM = 'sm'
+}
diff --git a/ui/lib/ui/src/lib/index.ts b/ui/lib/ui/src/lib/index.ts
index 542e7d84a..68cf9e8ce 100644
--- a/ui/lib/ui/src/lib/index.ts
+++ b/ui/lib/ui/src/lib/index.ts
@@ -39,3 +39,4 @@ export * as ToggleGroup from './components/ui/toggle-group/index.js';
export * as Tooltip from './components/ui/tooltip/index.js';
export * from './components/mockups/index.js';
export * from './components/models/index.js';
+export * from './components/voice/index.js';
diff --git a/ui/lib/ui/src/lib/voice/index.ts b/ui/lib/ui/src/lib/voice/index.ts
new file mode 100644
index 000000000..c9a1e0962
--- /dev/null
+++ b/ui/lib/ui/src/lib/voice/index.ts
@@ -0,0 +1,4 @@
+export * from './spoken.js';
+export * from './waveform.js';
+export { MIN_BAR_HEIGHT } from '../constants/voice.constants.js';
+export { VoiceRecordingBarStatus, VoiceTranscriptStatus, VoiceWaveformSize } from '../enums/voice.enums.js';
diff --git a/ui/lib/ui/src/lib/voice/spoken.ts b/ui/lib/ui/src/lib/voice/spoken.ts
new file mode 100644
index 000000000..a13492d2e
--- /dev/null
+++ b/ui/lib/ui/src/lib/voice/spoken.ts
@@ -0,0 +1,72 @@
+import {
+ VOICE_SPEECH_PEAK_FLOOR,
+ VOICE_WHITESPACE_PATTERN,
+ VOICE_WORD_PATTERN
+} from '../constants/voice.constants.js';
+
+/**
+ * How many characters of `text` have been spoken `elapsedMs` into a clip, estimated
+ * from its waveform: the text is spread over the parts with speech, so pauses move
+ * nothing on, and a word counts as spoken from its start. A transcript without word
+ * timings can only be followed this way; it is close, not exact.
+ */
+export function spokenLength(
+ text: string,
+ peaks: readonly number[],
+ durationMs: number,
+ elapsedMs: number
+): number {
+ if (!(durationMs > 0) || elapsedMs >= durationMs) return text.length;
+ if (!(elapsedMs > 0)) return 0;
+
+ const fraction = speechFraction(peaks, elapsedMs / durationMs);
+
+ return wordEnd(text, Math.floor(fraction * text.length));
+}
+
+/** The share of the clip's speech that lies before `position` (0..1 of its length). */
+export function speechFraction(peaks: readonly number[], position: number): number {
+ const loudest = Math.max(0, ...peaks.filter(Number.isFinite));
+
+ if (peaks.length === 0 || loudest <= 0) return clamp(position);
+
+ const weights = peaks.map((peak) =>
+ Number.isFinite(peak) && peak > loudest * VOICE_SPEECH_PEAK_FLOOR ? 1 : 0
+ );
+ const total = weights.reduce<number>((sum, weight) => sum + weight, 0);
+
+ if (total === 0) return clamp(position);
+
+ const at = clamp(position) * weights.length;
+ const whole = Math.floor(at);
+ let before = 0;
+
+ for (let i = 0; i < whole; i++) before += weights[i];
+
+ if (whole < weights.length) before += weights[whole] * (at - whole);
+
+ return before / total;
+}
+
+/** The end of the word `index` falls in, so a word is lit as a whole. */
+function wordEnd(text: string, index: number): number {
+ if (index <= 0) return 0;
+ if (index >= text.length) return text.length;
+
+ const next = text.slice(index).search(VOICE_WHITESPACE_PATTERN);
+
+ return next === -1 ? text.length : index + next;
+}
+
+function clamp(value: number): number {
+ return Math.min(Math.max(value, 0), 1);
+}
+
+/** A text cut into words, each with the whitespace after it and where it starts. */
+export function spokenWords(text: string): { start: number; word: string }[] {
+ // matchAll works on a copy of the pattern, so the shared one keeps no lastIndex
+ return [...text.matchAll(VOICE_WORD_PATTERN)].map((match) => ({
+ start: match.index,
+ word: match[0]
+ }));
+}
diff --git a/ui/lib/ui/src/lib/voice/waveform.ts b/ui/lib/ui/src/lib/voice/waveform.ts
new file mode 100644
index 000000000..df59b8fb6
--- /dev/null
+++ b/ui/lib/ui/src/lib/voice/waveform.ts
@@ -0,0 +1,106 @@
+import { MIN_BAR_HEIGHT } from '../constants/voice.constants.js';
+
+function clampUnit(value: number): number {
+ if (!Number.isFinite(value)) return 0;
+
+ return Math.min(Math.max(value, 0), 1);
+}
+
+/** A clip time as m:ss (`0:07`, `1:05`, `72:30`), rounded to the nearest second. */
+export function formatClipTime(ms: number): string {
+ const seconds = Number.isFinite(ms) && ms > 0 ? Math.round(ms / 1000) : 0;
+
+ return `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, '0')}`;
+}
+
+/** What a screen reader hears for a clip's position, e.g. `0:12 of 1:05`. */
+export function formatClipPosition(elapsedMs: number, totalMs: number): string {
+ return `${formatClipTime(elapsedMs)} of ${formatClipTime(totalMs)}`;
+}
+
+/** How many bars of `barWidth` with `gap` between them fit in `width` (all in px). */
+export function barCapacity(width: number, barWidth: number, gap: number): number {
+ if (!(width > 0) || !(barWidth > 0)) return 0;
+
+ return Math.max(Math.floor((width + gap) / (barWidth + gap)), 0);
+}
+
+/**
+ * Peaks (0..1) resampled to `count` bars: fewer bars keep the loudest peak of each bucket, so short sounds do not
+ * vanish; more bars interpolate between neighbours.
+ */
+export function resamplePeaks(peaks: readonly number[], count: number): number[] {
+ if (count <= 0) return [];
+ if (peaks.length === 0) return Array.from({ length: count }, () => 0);
+ if (peaks.length === count) return peaks.map(clampUnit);
+
+ if (count < peaks.length) {
+ const bars: number[] = [];
+
+ for (let i = 0; i < count; i++) {
+ const start = Math.floor((i * peaks.length) / count);
+ const end = Math.max(Math.floor(((i + 1) * peaks.length) / count), start + 1);
+ let peak = 0;
+
+ for (let j = start; j < end; j++) peak = Math.max(peak, clampUnit(peaks[j]));
+
+ bars.push(peak);
+ }
+
+ return bars;
+ }
+
+ if (peaks.length === 1) return Array.from({ length: count }, () => clampUnit(peaks[0]));
+
+ const bars: number[] = [];
+
+ for (let i = 0; i < count; i++) {
+ const position = (i * (peaks.length - 1)) / (count - 1);
+ const left = Math.floor(position);
+ const right = Math.min(left + 1, peaks.length - 1);
+ const t = position - left;
+
+ bars.push(clampUnit(peaks[left]) * (1 - t) + clampUnit(peaks[right]) * t);
+ }
+
+ return bars;
+}
+
+/**
+ * `count` bars of the live levels, newest last, always filling the width: a recording with fewer levels than
+ * bars is stretched over all of them, a longer one shows its last `count` levels. Before the first level every
+ * bar is `null` (drawn as the resting dot).
+ */
+export function liveBars(levels: readonly number[], count: number): (number | null)[] {
+ if (count <= 0) return [];
+ if (levels.length === 0) return Array.from({ length: count }, () => null);
+ if (levels.length < count) return resamplePeaks(levels, count);
+
+ return levels.slice(-count).map(clampUnit);
+}
+
+/** A bar's height in percent of the waveform's, never under `MIN_BAR_HEIGHT`. */
+export function barHeightPercent(level: number | null): number {
+ return Math.max(clampUnit(level ?? 0), MIN_BAR_HEIGHT) * 100;
+}
+
+/** How far through the clip playback is, 0..1. */
+export function progressFraction(elapsedMs: number, totalMs: number): number {
+ if (!(totalMs > 0)) return 0;
+
+ return clampUnit(elapsedMs / totalMs);
+}
+
+/** Where a pointer at `clientX` falls along a track starting at `left`, 0..1. */
+export function pointerFraction(clientX: number, left: number, width: number): number {
+ if (!(width > 0)) return 0;
+
+ return clampUnit((clientX - left) / width);
+}
+
+/** A position moved by `deltaMs`, kept inside the clip. */
+export function stepPosition(elapsedMs: number, deltaMs: number, totalMs: number): number {
+ if (!(totalMs > 0)) return 0;
+
+ return Math.min(Math.max(elapsedMs + deltaMs, 0), totalMs);
+}
diff --git a/ui/lib/ui/src/lib/components/ui/button/button.svelte.spec.ts b/ui/lib/ui/tests/client/button.svelte.spec.ts
similarity index 95%
rename from ui/lib/ui/src/lib/components/ui/button/button.svelte.spec.ts
rename to ui/lib/ui/tests/client/button.svelte.spec.ts
index 629d434b0..3bffd6abf 100644
--- a/ui/lib/ui/src/lib/components/ui/button/button.svelte.spec.ts
+++ b/ui/lib/ui/tests/client/button.svelte.spec.ts
@@ -1,7 +1,7 @@
import { createRawSnippet } from 'svelte';
import { describe, expect, it, vi } from 'vite-plus/test';
import { render } from 'vitest-browser-svelte';
-import Button from './button.svelte';
+import Button from '../../src/lib/components/ui/button/button.svelte';
function textSnippet(text: string) {
return createRawSnippet(() => ({
diff --git a/ui/lib/ui/tests/stories/voice/VoiceRecordingBar.stories.svelte b/ui/lib/ui/tests/stories/voice/VoiceRecordingBar.stories.svelte
new file mode 100644
index 000000000..a7a380364
--- /dev/null
+++ b/ui/lib/ui/tests/stories/voice/VoiceRecordingBar.stories.svelte
@@ -0,0 +1,128 @@
+<script lang="ts" module>
+ import StateGallery from '../helpers/StateGallery.svelte';
+ import StateGalleryItem from '../helpers/StateGalleryItem.svelte';
+ import { VoiceRecordingBar, VoiceRecordingBarStatus } from '@ggml-org/llama-ui/voice';
+ import { Square } from '@lucide/svelte';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import { expect, fn } from 'storybook/test';
+
+ // the row while recording: cancel, the waveform and time, the app's stop button, send; text and warnings under it
+ const { Story } = defineMeta({
+ component: VoiceRecordingBar,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama UI/Voice/VoiceRecordingBar'
+ });
+
+ /** Syllable-like levels, newest last. */
+ const levels = Array.from({ length: 48 }, (_, index) =>
+ Math.max(0.08, Math.abs(Math.sin(index / 2.2)) * (0.4 + 0.3 * Math.sin(index / 7)))
+ );
+
+ const ROW_CLASS = 'w-[28rem] max-w-full rounded-2xl border border-border bg-muted p-2';
+</script>
+
+{#snippet stopButton({ disabled }: { disabled: boolean })}
+ <button
+ class="flex size-8 shrink-0 items-center justify-center rounded-full bg-destructive text-white disabled:opacity-50"
+ {disabled}
+ type="button"
+ >
+ <span class="sr-only">Stop recording</span>
+
+ <Square class="size-3.5 fill-current" />
+ </button>
+{/snippet}
+
+<Story asChild name="States">
+ <StateGallery columns={1} width="30rem">
+ <StateGalleryItem label="Starting" note="no audio yet: stop and send wait">
+ <div class={ROW_CLASS}>
+ <VoiceRecordingBar
+ elapsedMs={0}
+ levels={[]}
+ onCancel={fn()}
+ onSend={fn()}
+ status={VoiceRecordingBarStatus.STARTING}
+ stop={stopButton}
+ />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Recording" note="the time runs from the first audio">
+ <div class={ROW_CLASS}>
+ <VoiceRecordingBar
+ elapsedMs={12_400}
+ label="Recording"
+ {levels}
+ onCancel={fn()}
+ onSend={fn()}
+ status={VoiceRecordingBarStatus.RECORDING}
+ stop={stopButton}
+ />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Caption" note="text streaming in while speaking">
+ <div class={ROW_CLASS}>
+ <VoiceRecordingBar
+ caption="Could you summarise the notes from this morning and list what is still open"
+ elapsedMs={6_100}
+ {levels}
+ onCancel={fn()}
+ onSend={fn()}
+ status={VoiceRecordingBarStatus.RECORDING}
+ stop={stopButton}
+ />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Warning" note="e.g. speech too quiet even boosted">
+ <div class={ROW_CLASS}>
+ <VoiceRecordingBar
+ elapsedMs={3_000}
+ levels={levels.map((level) => level / 6)}
+ onCancel={fn()}
+ status={VoiceRecordingBarStatus.RECORDING}
+ stop={stopButton}
+ warning="Your microphone sounds quiet. Move closer, or turn its input level up."
+ />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Waiting" note="the text is on its way">
+ <div class={ROW_CLASS}>
+ <VoiceRecordingBar
+ elapsedMs={9_000}
+ {levels}
+ onCancel={fn()}
+ status={VoiceRecordingBarStatus.WAITING}
+ stop={stopButton}
+ />
+ </div>
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<!-- separate: play checks that stop and send wait for the first audio, and that the labels can be the app's -->
+<Story
+ name="Starting, with labels"
+ args={{
+ elapsedMs: 0,
+ labels: { cancel: 'Discard', send: 'Send now', starting: 'Waking the microphone...' },
+ levels: [],
+ onCancel: fn(),
+ onSend: fn(),
+ status: VoiceRecordingBarStatus.STARTING,
+ stop: stopButton
+ }}
+ play={async ({ args, canvas, userEvent }) => {
+ await expect(canvas.getByText('Waking the microphone...')).toBeVisible();
+ await expect(canvas.getByRole('button', { name: 'Send now' })).toBeDisabled();
+ await expect(canvas.getByRole('button', { name: 'Stop recording' })).toBeDisabled();
+
+ await userEvent.click(canvas.getByRole('button', { name: 'Discard' }));
+ await expect(args.onCancel).toHaveBeenCalledOnce();
+ }}
+/>
diff --git a/ui/lib/ui/tests/stories/voice/VoiceTranscriptReveal.stories.svelte b/ui/lib/ui/tests/stories/voice/VoiceTranscriptReveal.stories.svelte
new file mode 100644
index 000000000..83a7210af
--- /dev/null
+++ b/ui/lib/ui/tests/stories/voice/VoiceTranscriptReveal.stories.svelte
@@ -0,0 +1,153 @@
+<script lang="ts" module>
+ import StateGallery from '../helpers/StateGallery.svelte';
+ import StateGalleryItem from '../helpers/StateGalleryItem.svelte';
+ import { demoClip } from './fixtures';
+ import {
+ VoiceTranscriptReveal,
+ VoiceTranscriptStatus,
+ VoiceWaveform
+ } from '@ggml-org/llama-ui/voice';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import { expect, waitFor } from 'storybook/test';
+
+ // a voice message with its transcript: hidden at first, opened under the clip on hover, keyboard focus or a tap
+ const { Story } = defineMeta({
+ component: VoiceTranscriptReveal,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama UI/Voice/VoiceTranscriptReveal'
+ });
+
+ const clip = demoClip(6, 5);
+
+ const TRANSCRIPT = 'Can you summarise the three main points from the meeting notes I attached?';
+ const LONG_TRANSCRIPT =
+ "Okay, so here's what I'm thinking for the trip. We fly into Lisbon on Friday evening, spend the weekend there, and then on Monday take the train up to Porto. I'd like to stay somewhere near the river, nothing fancy, and I want at least one day with no plans at all. Can you put together a rough itinerary with a couple of options for each day, and flag anything we should book ahead, like restaurants or the tram tours?";
+
+ const BUBBLE_CLASS =
+ 'w-80 max-w-full rounded-2xl matte-sheen bg-(--user-bubble-background) px-2.5 py-2';
+
+ function region(root: HTMLElement): HTMLElement {
+ return root.querySelector<HTMLElement>('[data-slot="voice-transcript"]')!;
+ }
+
+ function reveal(root: HTMLElement): HTMLElement {
+ return root.querySelector<HTMLElement>('[data-slot="voice-transcript-reveal"]')!;
+ }
+</script>
+
+{#snippet message(
+ status?: VoiceTranscriptStatus,
+ transcript: string | null = TRANSCRIPT,
+ errorReason: string | null = null
+)}
+ <div class={BUBBLE_CLASS}>
+ <VoiceTranscriptReveal {errorReason} {status} {transcript}>
+ <VoiceWaveform durationMs={clip.durationMs} peaks={clip.peaks} src={clip.src} />
+ </VoiceTranscriptReveal>
+ </div>
+{/snippet}
+
+<Story asChild name="States">
+ <StateGallery minItemWidth="20rem">
+ <StateGalleryItem label="Done" note="hover or tab in to read it">
+ {@render message()}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Transcribing">
+ {@render message(VoiceTranscriptStatus.TRANSCRIBING, null)}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Error">
+ {@render message(VoiceTranscriptStatus.ERROR, null)}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Error with its reason" note="when the app can say why">
+ {@render message(
+ VoiceTranscriptStatus.ERROR,
+ null,
+ 'This model only transcribes live, over a connection a browser cannot open on a server with an API key.'
+ )}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="None" note="no transcript asked for: nothing opens">
+ {@render message(VoiceTranscriptStatus.IDLE, null)}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Long text">
+ {@render message(VoiceTranscriptStatus.DONE, LONG_TRANSCRIPT)}
+ </StateGalleryItem>
+
+ <StateGalleryItem label="No speech" note="done, but empty">
+ {@render message(VoiceTranscriptStatus.DONE, '')}
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<!-- separate: play hovers the clip, then tabs into it, and checks the transcript opens each time -->
+<Story
+ asChild
+ name="HoverAndFocus"
+ play={async ({ canvas, canvasElement, userEvent }) => {
+ const root = reveal(canvasElement);
+
+ // closed, but in the DOM for screen readers
+ // drawn word by word, see VoiceSpokenText
+ await expect(region(canvasElement)).toHaveTextContent(TRANSCRIPT);
+ await expect(root).not.toHaveAttribute('data-revealed');
+ await expect(getComputedStyle(region(canvasElement)).opacity).toBe('0');
+
+ await userEvent.hover(root);
+ await waitFor(() => expect(root).toHaveAttribute('data-revealed'));
+ await waitFor(() => expect(getComputedStyle(region(canvasElement)).opacity).toBe('1'));
+
+ await userEvent.unhover(root);
+ await waitFor(() => expect(root).not.toHaveAttribute('data-revealed'));
+
+ await userEvent.tab();
+ await expect(canvas.getByRole('button', { name: 'Play voice message' })).toHaveFocus();
+ await waitFor(() => expect(root).toHaveAttribute('data-revealed'));
+
+ await userEvent.tab();
+ await expect(canvas.getByRole('slider')).toHaveFocus();
+ await expect(root).toHaveAttribute('data-revealed');
+
+ await userEvent.tab();
+ await waitFor(() => expect(root).not.toHaveAttribute('data-revealed'));
+ }}
+>
+ <div class="flex flex-col gap-4">
+ {@render message()}
+
+ <button class="text-caption text-muted-foreground" type="button">After the message</button>
+ </div>
+</Story>
+
+<!-- separate: play taps the clip with a finger, then closes it with the toggle key -->
+<Story
+ asChild
+ name="Touch"
+ play={async ({ canvas, canvasElement, userEvent }) => {
+ const root = reveal(canvasElement);
+ const time = canvas.getByText('0:06');
+
+ await userEvent.pointer({ keys: '[TouchA]', target: time });
+ await waitFor(() => expect(root).toHaveAttribute('data-revealed'));
+
+ const hide = canvas.getByRole('button', { name: 'Hide transcript' });
+
+ await expect(hide).toHaveAttribute('aria-expanded', 'true');
+ await userEvent.pointer({ keys: '[TouchA]', target: hide });
+ await waitFor(() => expect(root).not.toHaveAttribute('data-revealed'));
+ }}
+>
+ <StateGallery columns={1} width="24rem">
+ <StateGalleryItem
+ label="Touch"
+ note="a tap outside the controls toggles it; with no hover a toggle key shows"
+ >
+ {@render message()}
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
diff --git a/ui/lib/ui/tests/stories/voice/VoiceWaveform.stories.svelte b/ui/lib/ui/tests/stories/voice/VoiceWaveform.stories.svelte
new file mode 100644
index 000000000..706270e48
--- /dev/null
+++ b/ui/lib/ui/tests/stories/voice/VoiceWaveform.stories.svelte
@@ -0,0 +1,150 @@
+<script lang="ts" module>
+ import StateGallery from '../helpers/StateGallery.svelte';
+ import StateGalleryItem from '../helpers/StateGalleryItem.svelte';
+ import { demoClip } from './fixtures';
+ import { VoiceWaveform, VoiceWaveformSize } from '@ggml-org/llama-ui/voice';
+ import { defineMeta } from '@storybook/addon-svelte-csf';
+ import { expect } from 'storybook/test';
+
+ // a recorded voice clip: play, seek along the waveform, the time beside it; the surface is the caller's
+ const { Story } = defineMeta({
+ component: VoiceWaveform,
+ parameters: {
+ layout: 'centered'
+ },
+ title: 'Llama UI/Voice/VoiceWaveform'
+ });
+
+ const short = demoClip(4.2, 1);
+ const long = demoClip(135, 2);
+ const other = demoClip(7, 3);
+
+ const CHIP_CLASS = 'w-64 max-w-full rounded-lg border border-border bg-muted py-1.5 pr-3 pl-1.5';
+ const BUBBLE_CLASS =
+ 'w-80 max-w-full rounded-2xl matte-sheen bg-(--user-bubble-background) px-2.5 py-2';
+</script>
+
+<Story asChild name="Sizes">
+ <StateGallery minItemWidth="18rem">
+ <StateGalleryItem label="sm" note="a chip in the chat form">
+ <div class={CHIP_CLASS}>
+ <VoiceWaveform
+ durationMs={short.durationMs}
+ peaks={short.peaks}
+ size={VoiceWaveformSize.SM}
+ src={short.src}
+ />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="md" note="a message bubble">
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform durationMs={short.durationMs} peaks={short.peaks} src={short.src} />
+ </div>
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<Story asChild name="States">
+ <StateGallery columns={1} width="28rem">
+ <StateGalleryItem label="Short clip" note="0:04">
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform durationMs={short.durationMs} peaks={short.peaks} src={short.src} />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="Long clip" note="2:15">
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform durationMs={long.durationMs} peaks={long.peaks} src={long.src} />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="No audio" note="no src: the waveform and length only">
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform durationMs={short.durationMs} peaks={short.peaks} />
+ </div>
+ </StateGalleryItem>
+
+ <StateGalleryItem label="No peaks yet" note="a flat line">
+ <div class={CHIP_CLASS}>
+ <VoiceWaveform durationMs={short.durationMs} peaks={[]} size={VoiceWaveformSize.SM} src={short.src} />
+ </div>
+ </StateGalleryItem>
+ </StateGallery>
+</Story>
+
+<!-- separate: play seeks with the keys and a press on the bars; pressing play needs a real click (tests/client/voice-waveform.svelte.test.ts) -->
+<Story
+ asChild
+ name="Playback"
+ play={async ({ canvas, userEvent }) => {
+ await expect(canvas.getByRole('button', { name: 'Play first clip' })).toBeEnabled();
+ await expect(canvas.getByRole('button', { name: 'Play second clip' })).toBeEnabled();
+
+ const slider = canvas.getByRole('slider', { name: 'Long clip position' });
+
+ slider.focus();
+ await userEvent.keyboard('{End}');
+ await expect(slider).toHaveAttribute('aria-valuenow', '135');
+ await userEvent.keyboard('{ArrowLeft}');
+ await expect(slider).toHaveAttribute('aria-valuenow', '130');
+ await expect(slider).toHaveAttribute('aria-valuetext', '2:10 of 2:15');
+ await userEvent.keyboard('{Home}');
+ await userEvent.keyboard('{ArrowRight}');
+ await expect(slider).toHaveAttribute('aria-valuenow', '5');
+
+ // a press halfway along the bars seeks to the middle
+ const rect = slider.getBoundingClientRect();
+
+ await userEvent.pointer({
+ coords: { clientX: rect.left + rect.width / 2, clientY: rect.top + rect.height / 2 },
+ keys: '[MouseLeft]',
+ target: slider
+ });
+ await expect(Number(slider.getAttribute('aria-valuenow'))).toBeCloseTo(67.5, -1);
+ }}
+>
+ <div class="flex w-80 flex-col gap-3">
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform
+ durationMs={short.durationMs}
+ label="First clip"
+ peaks={short.peaks}
+ src={short.src}
+ />
+ </div>
+
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform
+ durationMs={other.durationMs}
+ label="Second clip"
+ peaks={other.peaks}
+ src={other.src}
+ />
+ </div>
+
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform
+ durationMs={long.durationMs}
+ label="Long clip"
+ peaks={long.peaks}
+ src={long.src}
+ />
+ </div>
+ </div>
+</Story>
+
+<!-- separate: play checks that a clip without audio offers nothing to press -->
+<Story
+ asChild
+ name="NoSource"
+ play={async ({ canvas }) => {
+ await expect(canvas.getByRole('button', { name: 'Play voice message' })).toBeDisabled();
+ await expect(canvas.getByRole('slider')).toHaveAttribute('aria-disabled', 'true');
+ await expect(canvas.getByText('0:04')).toBeInTheDocument();
+ }}
+>
+ <div class={BUBBLE_CLASS}>
+ <VoiceWaveform durationMs={short.durationMs} peaks={short.peaks} />
+ </div>
+</Story>
diff --git a/ui/lib/ui/tests/stories/voice/fixtures.ts b/ui/lib/ui/tests/stories/voice/fixtures.ts
new file mode 100644
index 000000000..a18014c69
--- /dev/null
+++ b/ui/lib/ui/tests/stories/voice/fixtures.ts
@@ -0,0 +1,128 @@
+/**
+ * Demo audio for the voice stories, made in the browser so no binary file sits in the repo: a WAV of a hummed tone
+ * that rises and falls like syllables, with its peaks.
+ */
+export interface DemoClip {
+ src: string;
+ durationMs: number;
+ peaks: number[];
+}
+
+const SAMPLE_RATE = 8000;
+const PEAK_COUNT = 64;
+
+/** Syllable-like loudness at time `t` (s): bursts of about 0.25 s with short gaps between words. */
+function envelope(t: number, seed: number): number {
+ const syllable = Math.max(Math.sin(t * Math.PI * 4 + seed), 0);
+ const word = Math.sin(t * Math.PI * 0.9 + seed * 2) > -0.6 ? 1 : 0.08;
+ const swell = 0.6 + 0.4 * Math.sin(t * 1.3 + seed);
+
+ return syllable * word * swell;
+}
+
+function wavBlob(samples: Float32Array): Blob {
+ const buffer = new ArrayBuffer(44 + samples.length * 2);
+ const view = new DataView(buffer);
+ const write = (offset: number, text: string) => {
+ for (let i = 0; i < text.length; i++) view.setUint8(offset + i, text.charCodeAt(i));
+ };
+
+ write(0, 'RIFF');
+ view.setUint32(4, 36 + samples.length * 2, true);
+ write(8, 'WAVE');
+ write(12, 'fmt ');
+ view.setUint32(16, 16, true);
+ view.setUint16(20, 1, true);
+ view.setUint16(22, 1, true);
+ view.setUint32(24, SAMPLE_RATE, true);
+ view.setUint32(28, SAMPLE_RATE * 2, true);
+ view.setUint16(32, 2, true);
+ view.setUint16(34, 16, true);
+ write(36, 'data');
+ view.setUint32(40, samples.length * 2, true);
+
+ for (let i = 0; i < samples.length; i++) {
+ view.setInt16(44 + i * 2, Math.max(-1, Math.min(1, samples[i])) * 0x7fff, true);
+ }
+
+ return new Blob([buffer], { type: 'audio/wav' });
+}
+
+/** The loudest sample of each of `count` buckets, scaled so the loudest bucket is 1. */
+export function peaksOf(samples: Float32Array, count = PEAK_COUNT): number[] {
+ const size = Math.max(Math.floor(samples.length / count), 1);
+ const peaks = Array.from({ length: count }, (_, bucket) => {
+ let peak = 0;
+
+ for (let i = bucket * size; i < Math.min((bucket + 1) * size, samples.length); i++) {
+ peak = Math.max(peak, Math.abs(samples[i]));
+ }
+
+ return peak;
+ });
+ const loudest = Math.max(...peaks, 1e-6);
+
+ return peaks.map((peak) => peak / loudest);
+}
+
+function demoSamples(seconds: number, seed: number): Float32Array {
+ const samples = new Float32Array(Math.round(seconds * SAMPLE_RATE));
+
+ for (let i = 0; i < samples.length; i++) {
+ const t = i / SAMPLE_RATE;
+ const pitch = 180 + 40 * Math.sin(t * 2 + seed);
+
+ samples[i] = 0.5 * envelope(t, seed) * Math.sin(2 * Math.PI * pitch * t);
+ }
+
+ return samples;
+}
+
+/** The demo clip as a stored attachment holds it: base64 WAV, with its peaks and length. */
+export async function demoStoredClip(
+ seconds: number,
+ seed = 1
+): Promise<{ base64: string; durationMs: number; peaks: number[]; size: number }> {
+ const samples = demoSamples(seconds, seed);
+ const blob = wavBlob(samples);
+ const bytes = new Uint8Array(await blob.arrayBuffer());
+
+ let binary = '';
+
+ for (let i = 0; i < bytes.length; i += 0x8000) {
+ binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));
+ }
+
+ return {
+ base64: btoa(binary),
+ durationMs: seconds * 1000,
+ peaks: peaksOf(samples),
+ size: blob.size
+ };
+}
+
+const cache = new Map<string, DemoClip>();
+
+/** A clip of `seconds`, the same for the same arguments. */
+export function demoClip(seconds: number, seed = 1): DemoClip {
+ const key = `${seconds}:${seed}`;
+ const cached = cache.get(key);
+
+ if (cached) return cached;
+
+ const samples = demoSamples(seconds, seed);
+ const clip = {
+ durationMs: seconds * 1000,
+ peaks: peaksOf(samples),
+ src: URL.createObjectURL(wavBlob(samples))
+ };
+
+ cache.set(key, clip);
+
+ return clip;
+}
+
+/** One live input level (0..1) at time `t` (s), as a microphone meter would report it while someone speaks. */
+export function demoLevel(t: number, seed = 1): number {
+ return Math.min(envelope(t, seed) * (0.75 + 0.25 * Math.random()), 1);
+}
diff --git a/ui/lib/ui/src/lib/models/format.spec.ts b/ui/lib/ui/tests/unit/models/format.spec.ts
similarity index 97%
rename from ui/lib/ui/src/lib/models/format.spec.ts
rename to ui/lib/ui/tests/unit/models/format.spec.ts
index 0446a55eb..a91626979 100644
--- a/ui/lib/ui/src/lib/models/format.spec.ts
+++ b/ui/lib/ui/tests/unit/models/format.spec.ts
@@ -5,7 +5,7 @@ import {
getAvatarUrl,
minMemoryTierGb,
summarizeSizes
-} from './format.js';
+} from '../../../src/lib/models/format.js';
import { describe, expect, it } from 'vite-plus/test';
describe('Model UI formatting', () => {
diff --git a/ui/lib/ui/tests/unit/voice/spoken.spec.ts b/ui/lib/ui/tests/unit/voice/spoken.spec.ts
new file mode 100644
index 000000000..12322cca1
--- /dev/null
+++ b/ui/lib/ui/tests/unit/voice/spoken.spec.ts
@@ -0,0 +1,37 @@
+import { speechFraction, spokenLength, spokenWords } from '../../../src/lib/voice/spoken.js';
+import { describe, expect, it } from 'vite-plus/test';
+
+const TEXT = 'one two three four';
+
+describe('spoken text', () => {
+ it('lights nothing before playback and everything at its end', () => {
+ expect(spokenLength(TEXT, [1, 1, 1, 1], 4000, 0)).toBe(0);
+ expect(spokenLength(TEXT, [1, 1, 1, 1], 4000, 4000)).toBe(TEXT.length);
+ expect(spokenLength(TEXT, [1, 1, 1, 1], 0, 100)).toBe(TEXT.length);
+ });
+
+ it('lights a word from its start, as a whole', () => {
+ // a quarter of even speech reaches into "two"
+ const length = spokenLength(TEXT, [1, 1, 1, 1], 4000, 1000);
+
+ expect(TEXT.slice(0, length)).toBe('one two');
+ });
+
+ it('spreads the text over the speech, not over the pauses', () => {
+ // the first half is silence: nothing is spoken yet halfway through
+ expect(speechFraction([0, 0, 1, 1], 0.5)).toBe(0);
+ expect(speechFraction([0, 0, 1, 1], 0.75)).toBe(0.5);
+ expect(speechFraction([1, 0, 0, 1], 0.5)).toBe(0.5);
+ // no speech to go by: the clip's own progress
+ expect(speechFraction([], 0.3)).toBeCloseTo(0.3);
+ expect(speechFraction([0, 0], 0.3)).toBeCloseTo(0.3);
+ });
+
+ it('cuts a text into words with their spaces and starts', () => {
+ expect(spokenWords(' hi there ')).toEqual([
+ { start: 0, word: ' hi ' },
+ { start: 5, word: 'there ' }
+ ]);
+ expect(spokenWords('')).toEqual([]);
+ });
+});
diff --git a/ui/lib/ui/tests/unit/voice/waveform.spec.ts b/ui/lib/ui/tests/unit/voice/waveform.spec.ts
new file mode 100644
index 000000000..05d9c1d01
--- /dev/null
+++ b/ui/lib/ui/tests/unit/voice/waveform.spec.ts
@@ -0,0 +1,104 @@
+import { MIN_BAR_HEIGHT } from '../../../src/lib/constants/voice.constants.js';
+import {
+ barCapacity,
+ barHeightPercent,
+ formatClipPosition,
+ formatClipTime,
+ liveBars,
+ pointerFraction,
+ progressFraction,
+ resamplePeaks,
+ stepPosition
+} from '../../../src/lib/voice/waveform.js';
+import { describe, expect, it } from 'vite-plus/test';
+
+describe('voice clip times', () => {
+ it('formats m:ss, rounded to the second', () => {
+ expect(formatClipTime(0)).toBe('0:00');
+ expect(formatClipTime(7_000)).toBe('0:07');
+ expect(formatClipTime(600)).toBe('0:01');
+ expect(formatClipTime(65_400)).toBe('1:05');
+ expect(formatClipTime(4_350_000)).toBe('72:30');
+ });
+
+ it('treats negative and non-finite times as zero', () => {
+ expect(formatClipTime(-5)).toBe('0:00');
+ expect(formatClipTime(Number.NaN)).toBe('0:00');
+ expect(formatClipTime(Number.POSITIVE_INFINITY)).toBe('0:00');
+ });
+
+ it('says the position for screen readers', () => {
+ expect(formatClipPosition(12_000, 65_000)).toBe('0:12 of 1:05');
+ });
+});
+
+describe('waveform bars', () => {
+ it('counts the bars that fit a width', () => {
+ expect(barCapacity(100, 3, 2)).toBe(20);
+ expect(barCapacity(98, 3, 2)).toBe(20);
+ expect(barCapacity(97, 3, 2)).toBe(19);
+ expect(barCapacity(0, 3, 2)).toBe(0);
+ expect(barCapacity(Number.NaN, 3, 2)).toBe(0);
+ });
+
+ it('keeps peaks as they are when the count matches, clamped to 0..1', () => {
+ expect(resamplePeaks([0.2, 1.4, -1, Number.NaN], 4)).toEqual([0.2, 1, 0, 0]);
+ });
+
+ it('keeps the loudest peak of each bucket when there are fewer bars', () => {
+ expect(resamplePeaks([0.1, 0.9, 0.2, 0.3, 0.8, 0.1], 3)).toEqual([0.9, 0.3, 0.8]);
+ expect(resamplePeaks([0.1, 0.2, 0.7, 0.4, 0.5], 2)).toEqual([0.2, 0.7]);
+ });
+
+ it('interpolates when there are more bars', () => {
+ expect(resamplePeaks([0, 1], 5)).toEqual([0, 0.25, 0.5, 0.75, 1]);
+ expect(resamplePeaks([0.4], 3)).toEqual([0.4, 0.4, 0.4]);
+ });
+
+ it('draws a flat line without peaks', () => {
+ expect(resamplePeaks([], 3)).toEqual([0, 0, 0]);
+ expect(resamplePeaks([0.5], 0)).toEqual([]);
+ });
+
+ it('fills the live width with the newest levels last', () => {
+ // fewer levels than bars: stretched over the whole width
+ expect(liveBars([0.1, 0.2, 0.3], 5).map((bar) => Number(bar?.toFixed(2)))).toEqual([
+ 0.1, 0.15, 0.2, 0.25, 0.3
+ ]);
+ expect(liveBars([0.4], 3)).toEqual([0.4, 0.4, 0.4]);
+ // more levels than bars: the newest ones scroll in from the right
+ expect(liveBars([0.1, 0.2, 0.3, 2], 2)).toEqual([0.3, 1]);
+ // nothing heard yet: the resting dots
+ expect(liveBars([], 3)).toEqual([null, null, null]);
+ expect(liveBars([0.5], 0)).toEqual([]);
+ });
+
+ it('never draws a bar under the resting height', () => {
+ expect(barHeightPercent(null)).toBe(MIN_BAR_HEIGHT * 100);
+ expect(barHeightPercent(0)).toBe(MIN_BAR_HEIGHT * 100);
+ expect(barHeightPercent(0.5)).toBe(50);
+ expect(barHeightPercent(3)).toBe(100);
+ });
+});
+
+describe('seeking', () => {
+ it('gives the played fraction', () => {
+ expect(progressFraction(15_000, 60_000)).toBe(0.25);
+ expect(progressFraction(90_000, 60_000)).toBe(1);
+ expect(progressFraction(1_000, 0)).toBe(0);
+ });
+
+ it('maps a pointer to a fraction of the track', () => {
+ expect(pointerFraction(150, 100, 200)).toBe(0.25);
+ expect(pointerFraction(50, 100, 200)).toBe(0);
+ expect(pointerFraction(400, 100, 200)).toBe(1);
+ expect(pointerFraction(150, 100, 0)).toBe(0);
+ });
+
+ it('steps the position inside the clip', () => {
+ expect(stepPosition(10_000, 5_000, 60_000)).toBe(15_000);
+ expect(stepPosition(3_000, -5_000, 60_000)).toBe(0);
+ expect(stepPosition(58_000, 5_000, 60_000)).toBe(60_000);
+ expect(stepPosition(1_000, 5_000, 0)).toBe(0);
+ });
+});
diff --git a/ui/lib/ui/vite.config.ts b/ui/lib/ui/vite.config.ts
index a9a6a40a1..5cb03ff21 100644
--- a/ui/lib/ui/vite.config.ts
+++ b/ui/lib/ui/vite.config.ts
@@ -50,6 +50,7 @@ export default defineConfig({
"@ggml-org/llama-ui/ui/*": "src/lib/components/ui/*",
"@ggml-org/llama-ui/app/*": "src/lib/components/app/*",
"@ggml-org/llama-ui/models": "src/lib/components/models/index.ts",
+ "@ggml-org/llama-ui/voice": "src/lib/components/voice/index.ts",
"@ggml-org/llama-ui/constants": "src/lib/constants/index.ts",
"@ggml-org/llama-ui/enums": "src/lib/enums/index.ts",
"@ggml-org/llama-ui/code": "src/lib/utils/index.ts",
@@ -91,8 +92,7 @@ export default defineConfig({
},
],
},
- include: ["src/**/*.svelte.{test,spec}.{js,ts}"],
- exclude: ["src/lib/server/**"],
+ include: ["tests/client/**/*.svelte.{test,spec}.{js,ts}"],
},
},
{
@@ -100,8 +100,7 @@ export default defineConfig({
test: {
name: "server",
environment: "node",
- include: ["src/**/*.{test,spec}.{js,ts}"],
- exclude: ["src/**/*.svelte.{test,spec}.{js,ts}"],
+ include: ["tests/unit/**/*.{test,spec}.{js,ts}"],
},
},
{
diff --git a/ui/lib/voice/.gitignore b/ui/lib/voice/.gitignore
new file mode 100644
index 000000000..16339ab2e
--- /dev/null
+++ b/ui/lib/voice/.gitignore
@@ -0,0 +1,6 @@
+node_modules
+/dist
+
+# OS
+.DS_Store
+Thumbs.db
diff --git a/ui/lib/voice/.npmrc b/ui/lib/voice/.npmrc
new file mode 100644
index 000000000..b6f27f135
--- /dev/null
+++ b/ui/lib/voice/.npmrc
@@ -0,0 +1 @@
+engine-strict=true
diff --git a/ui/lib/voice/LICENSE b/ui/lib/voice/LICENSE
new file mode 100644
index 000000000..50c59c51c
--- /dev/null
+++ b/ui/lib/voice/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) ggml
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/ui/lib/voice/README.md b/ui/lib/voice/README.md
new file mode 100644
index 000000000..7688982d9
--- /dev/null
+++ b/ui/lib/voice/README.md
@@ -0,0 +1,74 @@
+# @ggml-org/llama-voice
+
+Llama Voice, the voice input of Llama Chat, as a package: it records the microphone, shows its level as it goes, and
+turns speech into text through llama-server, live while the user speaks and once more over the whole recording when
+they stop. It also writes dictated text into a text field around what the user types. It has no UI and no
+dependencies; Llama UI (`@ggml-org/llama-ui/voice`) draws the voice parts. It lives in llama.cpp at `ui/lib/voice`.
+
+## Use
+
+```ts
+import { createDictation, startVoiceSession } from "@ggml-org/llama-voice";
+
+const server = {
+ resolveUrl: (path: string) => new URL(path, "http://localhost:8080/").toString(),
+};
+
+// dictate into a textarea: live text at the caret, then the whole recording's text in its place
+const dictation = createDictation({
+ getValue: () => textarea.value,
+ setValue: (value) => (textarea.value = value),
+ getCaretOffset: () => textarea.selectionStart,
+ setCaretOffset: (offset) => textarea.setSelectionRange(offset, offset),
+});
+
+dictation.begin();
+
+const session = await startVoiceSession({
+ capture: { boost: true },
+ realtime: true,
+ transcribe: { context: server },
+ onText: dictation.write,
+});
+
+textarea.oninput = dictation.edited;
+
+// later
+await dictation.finish(await session.stop());
+```
+
+In a Svelte app, `VoiceRecorder` from `@ggml-org/llama-voice/svelte` holds the same session as runes: `levels`,
+`liveText`, `quiet` and `status` for a component to show.
+
+## What is in it
+
+- **Capture:** the microphone (`VoiceCapture`, `listMicrophones`, `pickMicrophone`), live levels, 16 kHz PCM chunks,
+ and a WAV clip with its peaks when it stops.
+- **Signal:** a `SpeechDetector` that tells speech from the room, the pause detector built on it, and a gain that
+ raises quiet speech for transcription and says when it stays too quiet.
+- **Transcription:** llama-server's realtime socket (`/v1/realtime`), with `/v1/audio/transcriptions` piece by piece
+ when the socket is not there, and `transcribeClip` for a finished recording.
+- **Dictation:** `createDictation` and the `DictationSpan` under it.
+- **Support:** whether the browser can record, and why not, in words for the user (`VOICE_MICROPHONE_MESSAGE`, in
+ English).
+
+## What the app keeps
+
+Llama Chat (`ui/apps/chat`) is the first app on it. It keeps:
+
+- **The server and settings:** `VoiceService` turns its Voice settings and active server into the session's options.
+- **Which model transcribes:** `resolveVoiceMode`, from the models the server has.
+- **Its features:** the chat form's dictation and voice messages, and voice mode.
+
+## Develop
+
+```sh
+npm ci
+npm run check # type check
+npm test # unit tests
+npm run prepack # build dist/ and lint the package
+```
+
+Llama Chat does not need a build of this package: it compiles `src/` directly, through the `@ggml-org/llama-voice`
+aliases in its `vite.config.ts` and paths in its `tsconfig.json`. Its client tests cover what needs a browser
+(recording, the recorder in a component).
diff --git a/ui/lib/voice/package-lock.json b/ui/lib/voice/package-lock.json
new file mode 100644
index 000000000..274f6fd3b
--- /dev/null
+++ b/ui/lib/voice/package-lock.json
@@ -0,0 +1,1544 @@
+{
+ "name": "@ggml-org/llama-voice",
+ "version": "0.1.0",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "@ggml-org/llama-voice",
+ "version": "0.1.0",
+ "license": "MIT",
+ "devDependencies": {
+ "@types/node": "24.19.2",
+ "publint": "0.3.25",
+ "svelte": "5.57.3",
+ "typescript": "6.0.3",
+ "vitest": "5.0.5"
+ },
+ "engines": {
+ "node": ">=20"
+ },
+ "peerDependencies": {
+ "svelte": "^5.0.0"
+ },
+ "peerDependenciesMeta": {
+ "svelte": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/@emnapi/core": {
+ "version": "2.0.0-beta.2",
+ "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-2.0.0-beta.2.tgz",
+ "integrity": "sha512-MzCVoWtcg4Pm2B3je5mMvYMvj3MmBKw+fU7vbsJc9/iT8ip6/eYIHo/Dsm/mr+wsHB6SEyJDuxB09Yy6r/8+4A==",
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "@emnapi/wasi-threads": "2.2.0",
+ "tslib": "^2.4.0"
+ }
+ },
+ "node_modules/@emnapi/runtime": {
+ "version": "2.0.0-beta.2",
+ "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-2.0.0-beta.2.tgz",
+ "integrity": "sha512-JcrSPzPvZuNZvJqPTF9Fw25LbOJ8avzFVlaf2grmH4Cku0oGP47tRgqPZadcfj3YoZMHkFVnCYFbQM8I+i5QEw==",
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
+ "node_modules/@emnapi/wasi-threads": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-2.2.0.tgz",
+ "integrity": "sha512-JP7OYH38Q7+pyTAe14raZHLQIK7BDKbdwji+Xi3pcd/6QkJYQH3SX4wu6EplGrGJWRepoNzedfm0G3B9Z4e9Rw==",
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
+ "node_modules/@jridgewell/gen-mapping": {
+ "version": "0.3.13",
+ "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz",
+ "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/sourcemap-codec": "^1.5.0",
+ "@jridgewell/trace-mapping": "^0.3.24"
+ }
+ },
+ "node_modules/@jridgewell/remapping": {
+ "version": "2.3.5",
+ "resolved": "https://registry.npmjs.org/@jridgewell/remapping/-/remapping-2.3.5.tgz",
+ "integrity": "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/gen-mapping": "^0.3.5",
+ "@jridgewell/trace-mapping": "^0.3.24"
+ }
+ },
+ "node_modules/@jridgewell/resolve-uri": {
+ "version": "3.1.3",
+ "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.3.tgz",
+ "integrity": "sha512-lhTtjtoEArWBjismtfS761S9Hlyn+nIZqOZVYoZ1Xn82PMNMVnk+jj/NhC4fAXGTs7ydONcJiR5vyQBTMDu27g==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=6.0.0"
+ }
+ },
+ "node_modules/@jridgewell/sourcemap-codec": {
+ "version": "1.6.0",
+ "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.6.0.tgz",
+ "integrity": "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/@jridgewell/trace-mapping": {
+ "version": "0.3.31",
+ "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz",
+ "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/resolve-uri": "^3.1.0",
+ "@jridgewell/sourcemap-codec": "^1.4.14"
+ }
+ },
+ "node_modules/@napi-rs/wasm-runtime": {
+ "version": "1.2.6",
+ "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.2.6.tgz",
+ "integrity": "sha512-mRyDmEw9/fCDXCGfL1sbgIw/GUK0PVxRW4YwgPN61F3LQ+fSyoBFarJVzgGyhF6kRzEQZ0FejDqm4WLXSeqQmw==",
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "@tybys/wasm-util": "^0.10.3"
+ },
+ "engines": {
+ "node": "^20.19.0 || ^22.13.0 || >=23.5.0"
+ },
+ "funding": {
+ "type": "github",
+ "url": "https://github.com/sponsors/Brooooooklyn"
+ },
+ "peerDependencies": {
+ "@emnapi/core": "^1.7.1 || 2.0.0-beta.2",
+ "@emnapi/runtime": "^1.7.1 || 2.0.0-beta.2"
+ }
+ },
+ "node_modules/@oxc-project/types": {
+ "version": "0.156.0",
+ "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.156.0.tgz",
+ "integrity": "sha512-1Duyg48TdJNdyhEB8Xn1L36bT9Pv6GUtlKjHxOys8xkaF/rQq3XJE+bAurSK/AgIz5mGFsb77cn9Pb20+WCScw==",
+ "dev": true,
+ "license": "MIT",
+ "peer": true,
+ "funding": {
+ "url": "https://github.com/sponsors/oxc-project"
+ }
+ },
+ "node_modules/@publint/pack": {
+ "version": "0.1.7",
+ "resolved": "https://registry.npmjs.org/@publint/pack/-/pack-0.1.7.tgz",
+ "integrity": "sha512-4EDEmvxWtgsCnnVeBvtFIFZtUhPPt1+bA9JrSwU4Sa//6oKtzCSlGGXYJr44OD9aGISymbieJ4mCKHUygUDU+g==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "tinyexec": "^1.3.0"
+ },
+ "engines": {
+ "node": ">=18"
+ },
+ "funding": {
+ "url": "https://bjornlu.com/sponsor"
+ }
+ },
+ "node_modules/@rolldown/binding-android-arm-eabi": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm-eabi/-/binding-android-arm-eabi-1.2.19.tgz",
+ "integrity": "sha512-e+Ebe0WT+9tbXmNKPIiC8YrWAHYwug3m6i4p3rZGGulfZri1jtI4HslT+OZnFf/OIHowWF71DTlHwXBP2SiA7A==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-android-arm64": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.19.tgz",
+ "integrity": "sha512-hOombrTJ+szt2ciEzS7oCA78lMPBgF5UMQWCJIyNfnrI7GLl8HYoFqCnpL2gcPOaXMqryGCU5ExdQKyDDcC+yA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-darwin-arm64": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.19.tgz",
+ "integrity": "sha512-XoMA2pwCKGTQzUjPgtRwkedH4WmXhdIPKT6l6oOxd1PUbNodGI6DcBekgN7EC//+x3pfBYX/qL1gNmZAqivw5g==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-darwin-x64": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.19.tgz",
+ "integrity": "sha512-zo8EsSSwMZCY/e+fRnGaX02YGgvJD0yvhNi/9hpGPA2MJ9ZF8dPCw75UGfpI6OpQ/myR/dIYmogRvqiZD0TrXg==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-freebsd-x64": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.19.tgz",
+ "integrity": "sha512-O8ISAOfI1nj7AuBPd30FSF47z7f7lzlk5poQVGgiikvQuGEBFYvW8ki+UEnT+YJPnNGjQzn304klt4O9FUE7mA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-arm-gnueabihf": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.19.tgz",
+ "integrity": "sha512-/lYCba5NO0yq4S0jGMH8QTBtkvQxoXjoQWzF5BcWp9MjB9jxVRD1GQDNofkVjcOj30DWyhLIN/4Oe60JKzaONw==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-arm64-gnu": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.19.tgz",
+ "integrity": "sha512-pm/apFMBnSbFb3O6nG5pHO1iqHhZrjGDFEjSm+ot9J17XXPvH9L90y5h+jgfHCE+Qdlsvy8qKEcU4VKxgjD+vA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-arm64-musl": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.19.tgz",
+ "integrity": "sha512-K1SmQp3FFichn8oqrNIwvFoZ5wcvjC7goBzpaMNvLf9FZc+QB/QvLmnpJZgAAGHiD980tP8FZZnXk2K94rvlxg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "libc": [
+ "musl"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-ppc64-gnu": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.19.tgz",
+ "integrity": "sha512-I0xwD57jP/3A7+5UPZin2w8Y7XHEmEM5zqpYZOcQ45MlNW6tUJUziymV3yRr7rL4vwyfEcRsRksC4a7Kc08p5Q==",
+ "cpu": [
+ "ppc64"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-s390x-gnu": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.19.tgz",
+ "integrity": "sha512-BrtEHT24SNXj9HiXqy+s/kXiSiQGt6EKCPUlRirAoIioVxKGnbXBUEkbZASAei7iZJB0/Ynm+0JP00Ipx7AnSw==",
+ "cpu": [
+ "s390x"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-x64-gnu": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.19.tgz",
+ "integrity": "sha512-rm75/9DkHKeU2WnA8w4X94WGYgLlDCFCfkQC+4bguVJmC/jvDRv2B+2wLOYhyE0fVuv7//Ckk39OoOufyCu76Q==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-linux-x64-musl": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.19.tgz",
+ "integrity": "sha512-irh/4+iYFM4GAzpluDErgjz89bj6NDtqqMAG/6O1LG38Ed3NpyebzCpcVNkFW8uhy+/3/BocUvnXDR33CXWe9A==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "libc": [
+ "musl"
+ ],
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-openharmony-arm64": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.19.tgz",
+ "integrity": "sha512-+KWTWNoPt+CfSq94CiceWu3yPE+QzkaG+phPyz9G60tmZLwfQkcA4AeksafzvJAw4pvg1mcd5a1TkUqVv0s4JA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "openharmony"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-wasm32-wasi": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.2.19.tgz",
+ "integrity": "sha512-7EH+skOsl5qpWnK0vzamgJFhOv1cwxQpsxm8tI883K9fur5c27p8q87fzEO6BbBPabA2/+s/zR63/pLc137iOg==",
+ "cpu": [
+ "wasm32"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "@emnapi/core": "2.0.0-beta.2",
+ "@emnapi/runtime": "2.0.0-beta.2",
+ "@napi-rs/wasm-runtime": "1.2.6"
+ },
+ "engines": {
+ "node": "^20.19.0 || ^22.13.0 || >=23.5.0"
+ }
+ },
+ "node_modules/@rolldown/binding-win32-arm64-msvc": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.19.tgz",
+ "integrity": "sha512-3Q1Yg1Q7tEvBWoZf5rFvG/GKZQPgwfru+p4wBajDfJFBs0SHCpL++dfTjW9gyAKd1T7ZpIYvdaUax7Pa/IhFVQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/binding-win32-x64-msvc": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.19.tgz",
+ "integrity": "sha512-y/sqRRc6d7hECzOFc5n3LSeqctmIp9mMjY1Kq4s14uLPd/LZ4z7i1zrW6fMtp2mPeuQIC4e6ULMcwQzMomqQag==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ }
+ },
+ "node_modules/@rolldown/pluginutils": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.1.1.tgz",
+ "integrity": "sha512-glFIJDvLNDtJhT9qrMFWLPbvS+skxgt4L0K8Ei2YtnhPysAJnsI7nyxSJxKj4ADeKX0OLjC4ntRkwpLOoS3rwA==",
+ "dev": true,
+ "license": "MIT",
+ "peer": true
+ },
+ "node_modules/@sveltejs/acorn-typescript": {
+ "version": "1.0.13",
+ "resolved": "https://registry.npmjs.org/@sveltejs/acorn-typescript/-/acorn-typescript-1.0.13.tgz",
+ "integrity": "sha512-wgKggnhZVL9Bfx1OaKKTrYY9BFRk6C8UAkQNUcIv1+llzYrIqy+RZm5HPKzn0NpEBvTVhTqB4kQyllZywsRBRQ==",
+ "dev": true,
+ "license": "MIT",
+ "peerDependencies": {
+ "acorn": "^8.9.0"
+ }
+ },
+ "node_modules/@tybys/wasm-util": {
+ "version": "0.10.5",
+ "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.5.tgz",
+ "integrity": "sha512-dOGOPxlaNV+G046mZzN+9C0dAcQZeSKcXiVaM0TYy/LM4ttTmgSBDxYfovcYI8Em3+eAbiAYpvWLOv8+ZVRJKA==",
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
+ "node_modules/@types/chai": {
+ "version": "5.2.3",
+ "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz",
+ "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/deep-eql": "*",
+ "assertion-error": "^2.0.1"
+ }
+ },
+ "node_modules/@types/deep-eql": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz",
+ "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/@types/estree": {
+ "version": "1.0.10",
+ "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.10.tgz",
+ "integrity": "sha512-lTyOMnOcmRBzdCv6/4zUZCrEhV0xrqQiUZBZtVcbmLHBXKLnN8q89v/wKRF/Q0HbwVsqbIG3o5HDoFl9sduM+g==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/@types/node": {
+ "version": "24.19.2",
+ "resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.2.tgz",
+ "integrity": "sha512-93d49wCWJkCyN87MxLpNN6KXPCwFw7bVUg79mGUV1OS40JGWwVCr8n61DXCSqz7CV6/VfPkb7xec7uP2M+QHWQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "undici-types": ">=7.24.0 <7.24.7"
+ }
+ },
+ "node_modules/@vitest/mocker": {
+ "version": "5.0.5",
+ "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-5.0.5.tgz",
+ "integrity": "sha512-xLL3lkDegJZw+MGoGaCCsFayDZthGgpAtr3Vziyfc4p7xH89K7KRPcRBBh/kEYm0Aqkt5DtTmkYpBtRV/FrHZg==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/trace-mapping": "0.3.31",
+ "@vitest/spy": "5.0.5",
+ "estree-walker": "^3.0.3",
+ "magic-string": "^1.4.3"
+ },
+ "funding": {
+ "url": "https://opencollective.com/vitest"
+ },
+ "peerDependencies": {
+ "msw": "^2.4.9 || ^3.0.0",
+ "vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
+ },
+ "peerDependenciesMeta": {
+ "msw": {
+ "optional": true
+ },
+ "vite": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/@vitest/mocker/node_modules/magic-string": {
+ "version": "1.4.5",
+ "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-1.4.5.tgz",
+ "integrity": "sha512-oLLC2+YvQUfAa+vxoufE/yc1icb87M9iXIRqAoR/2EhE/8cb3SF/+Gi0c2By1B+5Vz2e2vQtUvHiu1+63Znh+w==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/sourcemap-codec": "^1.6.0"
+ }
+ },
+ "node_modules/@vitest/spy": {
+ "version": "5.0.5",
+ "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-5.0.5.tgz",
+ "integrity": "sha512-lwJRExuH417257Q5wv9QYugN9IMJkoxcrKrYxGPQkBzl7wlNpTUxD/5yNX4ESKVUIShKPgkGchYPFMWOOk0wzw==",
+ "dev": true,
+ "license": "MIT",
+ "funding": {
+ "url": "https://opencollective.com/vitest"
+ }
+ },
+ "node_modules/acorn": {
+ "version": "8.19.0",
+ "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.19.0.tgz",
+ "integrity": "sha512-oJlA3XiRm7Cyk6qFD2Jn8ak9B7jSy0qy00ADO3+8dpT0LSjFihQYv4C02LFCSYJV3Q37xYwwRy5m+IpIiUzqWw==",
+ "dev": true,
+ "license": "MIT",
+ "bin": {
+ "acorn": "bin/acorn"
+ },
+ "engines": {
+ "node": ">=0.4.0"
+ }
+ },
+ "node_modules/aria-query": {
+ "version": "5.3.1",
+ "resolved": "https://registry.npmjs.org/aria-query/-/aria-query-5.3.1.tgz",
+ "integrity": "sha512-Z/ZeOgVl7bcSYZ/u/rh0fOpvEpq//LZmdbkXyc7syVzjPAhfOa9ebsdTSjEBDU4vs5nC98Kfduj1uFo0qyET3g==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/assertion-error": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz",
+ "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/axobject-query": {
+ "version": "4.1.0",
+ "resolved": "https://registry.npmjs.org/axobject-query/-/axobject-query-4.1.0.tgz",
+ "integrity": "sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/chai": {
+ "version": "6.3.0",
+ "resolved": "https://registry.npmjs.org/chai/-/chai-6.3.0.tgz",
+ "integrity": "sha512-XWAtwJ6OHO+tj0EKCs0Y2UamnyOxseZWltU4x2U2wh8g4AigdjwvtUjvLP2tqkA/avxHEtzxNaqGq/YGNwckKg==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/clsx": {
+ "version": "2.1.1",
+ "resolved": "https://registry.npmjs.org/clsx/-/clsx-2.1.1.tgz",
+ "integrity": "sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/detect-libc": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "peer": true,
+ "engines": {
+ "node": ">=8"
+ }
+ },
+ "node_modules/devalue": {
+ "version": "5.9.4",
+ "resolved": "https://registry.npmjs.org/devalue/-/devalue-5.9.4.tgz",
+ "integrity": "sha512-sPAT4pztbu6586/hrhOnMKS17IJrvg12mXiSPSS3W5qDeN2RGgvZ0diZCm31dBbnevfVmujNO3IM2wrS4Y2Rhg==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/es-module-lexer": {
+ "version": "2.3.2",
+ "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.2.tgz",
+ "integrity": "sha512-poHGpORABojJJucnV9KbOavETW8lBVnphkW77ER5/BQ5Fz7oXSoCNek7IH3vR5nRjdsEz926ibFYX8KtLQmdyw==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/esm-env": {
+ "version": "1.2.2",
+ "resolved": "https://registry.npmjs.org/esm-env/-/esm-env-1.2.2.tgz",
+ "integrity": "sha512-Epxrv+Nr/CaL4ZcFGPJIYLWFom+YeV1DqMLHJoEd9SYRxNbaFruBwfEX/kkHUJf55j2+TUbmDcmuilbP1TmXHA==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/esrap": {
+ "version": "2.4.2",
+ "resolved": "https://registry.npmjs.org/esrap/-/esrap-2.4.2.tgz",
+ "integrity": "sha512-WpVZsWnSheHBy+fGrO/0duwnbe6ODZH1QyBbjupreSCJGojaBWRncrq+HVeIJ8eHQjEwSJg9hC4yPJyNVOrpqg==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/sourcemap-codec": "^1.4.15"
+ },
+ "peerDependencies": {
+ "@typescript-eslint/types": "^8.2.0"
+ },
+ "peerDependenciesMeta": {
+ "@typescript-eslint/types": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/estree-walker": {
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz",
+ "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/estree": "^1.0.0"
+ }
+ },
+ "node_modules/expect-type": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz",
+ "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=12.0.0"
+ }
+ },
+ "node_modules/fdir": {
+ "version": "6.5.0",
+ "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz",
+ "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=12.0.0"
+ },
+ "peerDependencies": {
+ "picomatch": "^3 || ^4"
+ },
+ "peerDependenciesMeta": {
+ "picomatch": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/fsevents": {
+ "version": "2.3.3",
+ "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
+ "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
+ "dev": true,
+ "hasInstallScript": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "peer": true,
+ "engines": {
+ "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+ }
+ },
+ "node_modules/is-reference": {
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/is-reference/-/is-reference-3.0.3.tgz",
+ "integrity": "sha512-ixkJoqQvAP88E6wLydLGGqCJsrFUnqoH6HnaczB8XmDH1oaWU+xxdptvikTgaEhtZ53Ky6YXiBuUI2WXLMCwjw==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/estree": "^1.0.6"
+ }
+ },
+ "node_modules/lightningcss": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz",
+ "integrity": "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==",
+ "dev": true,
+ "license": "MPL-2.0",
+ "peer": true,
+ "dependencies": {
+ "detect-libc": "^2.0.3"
+ },
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ },
+ "optionalDependencies": {
+ "lightningcss-android-arm64": "1.33.0",
+ "lightningcss-darwin-arm64": "1.33.0",
+ "lightningcss-darwin-x64": "1.33.0",
+ "lightningcss-freebsd-x64": "1.33.0",
+ "lightningcss-linux-arm-gnueabihf": "1.33.0",
+ "lightningcss-linux-arm64-gnu": "1.33.0",
+ "lightningcss-linux-arm64-musl": "1.33.0",
+ "lightningcss-linux-x64-gnu": "1.33.0",
+ "lightningcss-linux-x64-musl": "1.33.0",
+ "lightningcss-win32-arm64-msvc": "1.33.0",
+ "lightningcss-win32-x64-msvc": "1.33.0"
+ }
+ },
+ "node_modules/lightningcss-android-arm64": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz",
+ "integrity": "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-darwin-arm64": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz",
+ "integrity": "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-darwin-x64": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz",
+ "integrity": "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-freebsd-x64": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz",
+ "integrity": "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-linux-arm-gnueabihf": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz",
+ "integrity": "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-linux-arm64-gnu": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz",
+ "integrity": "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-linux-arm64-musl": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz",
+ "integrity": "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "libc": [
+ "musl"
+ ],
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-linux-x64-gnu": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz",
+ "integrity": "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "libc": [
+ "glibc"
+ ],
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-linux-x64-musl": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz",
+ "integrity": "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "libc": [
+ "musl"
+ ],
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-win32-arm64-msvc": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz",
+ "integrity": "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/lightningcss-win32-x64-msvc": {
+ "version": "1.33.0",
+ "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz",
+ "integrity": "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MPL-2.0",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "peer": true,
+ "engines": {
+ "node": ">= 12.0.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/parcel"
+ }
+ },
+ "node_modules/locate-character": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/locate-character/-/locate-character-3.0.0.tgz",
+ "integrity": "sha512-SW13ws7BjaeJ6p7Q6CO2nchbYEc3X3J6WrmTTDto7yMPqVSZTUyY5Tjbid+Ab8gLnATtygYtiDIJGQRRn2ZOiA==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/magic-string": {
+ "version": "0.30.21",
+ "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
+ "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/sourcemap-codec": "^1.5.5"
+ }
+ },
+ "node_modules/mri": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/mri/-/mri-1.2.0.tgz",
+ "integrity": "sha512-tzzskb3bG8LvYGFF/mDTpq3jpI6Q9wc3LEmBaghu+DdCssd1FakN7Bc0hVNmEyGq1bq3RgfkCb3cmQLpNPOroA==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/nanoid": {
+ "version": "3.3.20",
+ "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.20.tgz",
+ "integrity": "sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==",
+ "dev": true,
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/ai"
+ }
+ ],
+ "license": "MIT",
+ "peer": true,
+ "bin": {
+ "nanoid": "bin/nanoid.cjs"
+ },
+ "engines": {
+ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
+ }
+ },
+ "node_modules/obug": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/obug/-/obug-3.0.0.tgz",
+ "integrity": "sha512-5vvB5+W7ePv+p3uqxi+RcW1XAzLW0/hxt3/4X4Lc4qHudzOhmBiBwOY6DRob4WnanAEGvNcLjF+KNOufrUoEQw==",
+ "dev": true,
+ "funding": [
+ "https://github.com/sponsors/sxzz",
+ "https://opencollective.com/debug"
+ ],
+ "license": "MIT",
+ "engines": {
+ "node": ">=12.20.0"
+ }
+ },
+ "node_modules/package-manager-detector": {
+ "version": "1.9.0",
+ "resolved": "https://registry.npmjs.org/package-manager-detector/-/package-manager-detector-1.9.0.tgz",
+ "integrity": "sha512-zAMfbta9nPA8AKkA4WdmNAT8R8s6Dk0/xwPsqNnRidMFcLo1YkLuPDTgs2f64XdzS8vbLNnoJ+uUIkgbz7oRHA==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/picocolors": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
+ "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==",
+ "dev": true,
+ "license": "ISC"
+ },
+ "node_modules/picomatch": {
+ "version": "4.0.7",
+ "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz",
+ "integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=12"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/jonschlinkert"
+ }
+ },
+ "node_modules/postcss": {
+ "version": "8.5.33",
+ "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.33.tgz",
+ "integrity": "sha512-RVHEU9/Pp1Fnjr/MpAJsZoADukCKhfIi15OunEYiGktIkYLkPVvC1sioWolGDS7XByMTx9m/IGbro0Js13zhnQ==",
+ "dev": true,
+ "funding": [
+ {
+ "type": "opencollective",
+ "url": "https://opencollective.com/postcss/"
+ },
+ {
+ "type": "tidelift",
+ "url": "https://tidelift.com/funding/github/npm/postcss"
+ },
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/ai"
+ }
+ ],
+ "license": "MIT",
+ "peer": true,
+ "dependencies": {
+ "nanoid": "^3.3.20",
+ "picocolors": "^1.1.1",
+ "source-map-js": "^1.2.2"
+ },
+ "engines": {
+ "node": "^10 || ^12 || >=14"
+ }
+ },
+ "node_modules/publint": {
+ "version": "0.3.25",
+ "resolved": "https://registry.npmjs.org/publint/-/publint-0.3.25.tgz",
+ "integrity": "sha512-WMMbn/d7JqRlfXQL/JUI4n5skC8cJ5WVr8/wT5mmiYWkUdENkZr1I5rgs2+dQPVRgoJd+JRYtB46MTsuDylw9g==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@publint/pack": "^0.1.7",
+ "package-manager-detector": "^1.8.0",
+ "picocolors": "^1.1.1",
+ "sade": "^1.8.1"
+ },
+ "bin": {
+ "publint": "src/cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ },
+ "funding": {
+ "url": "https://bjornlu.com/sponsor"
+ }
+ },
+ "node_modules/rolldown": {
+ "version": "1.2.19",
+ "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.19.tgz",
+ "integrity": "sha512-QIiV1J9d0pzN4jeeTH4sjPy0AZ6cAy2cG0ExeoE1+U8Rvj+hR3gYkokx0YbeUNGlZqiQJJ2utcETe0oKXwzsuA==",
+ "dev": true,
+ "license": "MIT",
+ "peer": true,
+ "dependencies": {
+ "@oxc-project/types": "=0.156.0",
+ "@rolldown/pluginutils": "^1.0.0"
+ },
+ "bin": {
+ "rolldown": "bin/cli.mjs"
+ },
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ },
+ "optionalDependencies": {
+ "@rolldown/binding-android-arm-eabi": "1.2.19",
+ "@rolldown/binding-android-arm64": "1.2.19",
+ "@rolldown/binding-darwin-arm64": "1.2.19",
+ "@rolldown/binding-darwin-x64": "1.2.19",
+ "@rolldown/binding-freebsd-x64": "1.2.19",
+ "@rolldown/binding-linux-arm-gnueabihf": "1.2.19",
+ "@rolldown/binding-linux-arm64-gnu": "1.2.19",
+ "@rolldown/binding-linux-arm64-musl": "1.2.19",
+ "@rolldown/binding-linux-ppc64-gnu": "1.2.19",
+ "@rolldown/binding-linux-s390x-gnu": "1.2.19",
+ "@rolldown/binding-linux-x64-gnu": "1.2.19",
+ "@rolldown/binding-linux-x64-musl": "1.2.19",
+ "@rolldown/binding-openharmony-arm64": "1.2.19",
+ "@rolldown/binding-wasm32-wasi": "1.2.19",
+ "@rolldown/binding-win32-arm64-msvc": "1.2.19",
+ "@rolldown/binding-win32-x64-msvc": "1.2.19"
+ }
+ },
+ "node_modules/sade": {
+ "version": "1.8.1",
+ "resolved": "https://registry.npmjs.org/sade/-/sade-1.8.1.tgz",
+ "integrity": "sha512-xal3CZX1Xlo/k4ApwCFrHVACi9fBqJ7V+mwhBsuf/1IOKbBy098Fex+Wa/5QMubw09pSZ/u8EY8PWgevJsXp1A==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "mri": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/source-map-js": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.4.tgz",
+ "integrity": "sha512-J7ZjMvHd9O1fcT/5V09QWaaSl2BbWx3F6O08spde82Namf9UB0GxRVIPiNbVsdKkoizHiSpTodo3vp2bAy91MA==",
+ "dev": true,
+ "license": "BSD-3-Clause",
+ "peer": true,
+ "engines": {
+ "node": ">=6.0.0"
+ }
+ },
+ "node_modules/std-env": {
+ "version": "4.3.0",
+ "resolved": "https://registry.npmjs.org/std-env/-/std-env-4.3.0.tgz",
+ "integrity": "sha512-OtU/EgQ1kIm5KwqQpBC6ZEMXrZRui11w8zgfTWp8cdO9B8OaPsbA8bTHO2P+HNo1VlUTGMVBwPhydu6poeXiag==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/svelte": {
+ "version": "5.57.3",
+ "resolved": "https://registry.npmjs.org/svelte/-/svelte-5.57.3.tgz",
+ "integrity": "sha512-5mAftM3SoDV9X0zdF386F9p2pv8iKOwi/KfBS1XnBAkVCKFzVOLHUoa8iGWAt4buoicLmyWVRpuuD5wplXOQsg==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/remapping": "^2.3.5",
+ "@jridgewell/sourcemap-codec": "^1.6.0",
+ "@sveltejs/acorn-typescript": "^1.0.13",
+ "@types/estree": "^1.0.9",
+ "acorn": "^8.18.0",
+ "aria-query": "5.3.1",
+ "axobject-query": "^4.1.0",
+ "clsx": "^2.1.1",
+ "devalue": "^5.9.2",
+ "esm-env": "^1.2.1",
+ "esrap": "^2.3.6",
+ "is-reference": "^3.0.3",
+ "locate-character": "^3.0.0",
+ "magic-string": "^0.30.11",
+ "zimmerframe": "^1.1.2"
+ },
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/tinybench": {
+ "version": "6.2.4",
+ "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-6.2.4.tgz",
+ "integrity": "sha512-SQ52jEncXxbMCzJuY2A/jGIbbAu6NYsBWW7grz2pNE+EnzQdoGaoBfpk4qC/opE7qiUcGmGgaGIz5EoHdS1O9w==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=20.0.0"
+ }
+ },
+ "node_modules/tinyexec": {
+ "version": "1.3.1",
+ "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.3.1.tgz",
+ "integrity": "sha512-GCvB3aoys96IuDFBMcTB46JOR6mdMtAToqwiW8JlWhsoh1mhHi/xn9ss/Dg7N555GiJyEt2qzoG/NHCwM6h1EA==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/tinyglobby": {
+ "version": "0.2.17",
+ "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz",
+ "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "fdir": "^6.5.0",
+ "picomatch": "^4.0.4"
+ },
+ "engines": {
+ "node": ">=12.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/SuperchupuDev"
+ }
+ },
+ "node_modules/tslib": {
+ "version": "2.9.1",
+ "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.9.1.tgz",
+ "integrity": "sha512-a4Ux9VKuIMHsOS7ENbEq6hK2ZGYIT27Rtl0E4eD9KqwxhKiQeMZE/9xb3IdYwWaSOyIBQLExhQgXDTYoiCxnWA==",
+ "dev": true,
+ "license": "0BSD",
+ "optional": true,
+ "peer": true
+ },
+ "node_modules/typescript": {
+ "version": "6.0.3",
+ "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
+ "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "bin": {
+ "tsc": "bin/tsc",
+ "tsserver": "bin/tsserver"
+ },
+ "engines": {
+ "node": ">=14.17"
+ }
+ },
+ "node_modules/undici-types": {
+ "version": "7.24.6",
+ "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz",
+ "integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/vite": {
+ "version": "8.3.8",
+ "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.8.tgz",
+ "integrity": "sha512-LzOn3h9rG/0vhLdXrV4yJ4QFA9eEoqCl+Yx+kGJd7aB9w6nfNgVHKSYwLPZUHrz4ZGa+t+kXjV9itvcHDLQouA==",
+ "dev": true,
+ "license": "MIT",
+ "peer": true,
+ "dependencies": {
+ "lightningcss": "^1.33.0",
+ "picomatch": "^4.0.7",
+ "postcss": "^8.5.32",
+ "rolldown": "~1.2.18",
+ "tinyglobby": "^0.2.17"
+ },
+ "bin": {
+ "vite": "bin/vite.js"
+ },
+ "engines": {
+ "node": "^20.19.0 || >=22.12.0"
+ },
+ "funding": {
+ "url": "https://github.com/vitejs/vite?sponsor=1"
+ },
+ "optionalDependencies": {
+ "fsevents": "~2.3.3"
+ },
+ "peerDependencies": {
+ "@types/node": "^20.19.0 || >=22.12.0",
+ "@vitejs/devtools": "^0.7.1 || ^0.8.0",
+ "esbuild": "^0.27.0 || ^0.28.0",
+ "jiti": ">=1.21.0",
+ "less": "^4.0.0",
+ "sass": "^1.70.0",
+ "sass-embedded": "^1.70.0",
+ "stylus": ">=0.54.8",
+ "sugarss": "^5.0.0",
+ "terser": "^5.16.0",
+ "tsx": "^4.8.1",
+ "yaml": "^2.4.2"
+ },
+ "peerDependenciesMeta": {
+ "@types/node": {
+ "optional": true
+ },
+ "@vitejs/devtools": {
+ "optional": true
+ },
+ "esbuild": {
+ "optional": true
+ },
+ "jiti": {
+ "optional": true
+ },
+ "less": {
+ "optional": true
+ },
+ "sass": {
+ "optional": true
+ },
+ "sass-embedded": {
+ "optional": true
+ },
+ "stylus": {
+ "optional": true
+ },
+ "sugarss": {
+ "optional": true
+ },
+ "terser": {
+ "optional": true
+ },
+ "tsx": {
+ "optional": true
+ },
+ "yaml": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/vitest": {
+ "version": "5.0.5",
+ "resolved": "https://registry.npmjs.org/vitest/-/vitest-5.0.5.tgz",
+ "integrity": "sha512-cPODOMF8YnkiiYtDdHo6n4wWmBrOBBbOIYH2d33M4l3CdNn72h+qMmO6uXffUN6A5AZFXNND/d+krS9mgkXRfg==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/chai": "^5.2.2",
+ "@vitest/mocker": "5.0.5",
+ "chai": "^6.3.0",
+ "es-module-lexer": "^2.3.2",
+ "expect-type": "^1.4.0",
+ "magic-string": "^1.4.3",
+ "obug": "^3.0.0",
+ "picomatch": "^4.0.7",
+ "std-env": "^4.3.0",
+ "tinybench": "^6.2.2",
+ "tinyexec": "^1.3.1",
+ "tinyglobby": "^0.2.17",
+ "why-is-node-running": "3.2.1"
+ },
+ "bin": {
+ "vitest": "vitest.mjs"
+ },
+ "engines": {
+ "node": "^22.12.0 || ^24.0.0 || >=26.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/vitest"
+ },
+ "peerDependencies": {
+ "@edge-runtime/vm": "*",
+ "@opentelemetry/api": "^1.9.0",
+ "@types/node": "^22.0.0 || >=24.0.0",
+ "@vitest/browser-playwright": "5.0.5",
+ "@vitest/browser-preview": "5.0.5",
+ "@vitest/browser-webdriverio": "^5.0.0-beta.5 || >=5.0.0",
+ "@vitest/coverage-istanbul": "5.0.5",
+ "@vitest/coverage-v8": "5.0.5",
+ "@vitest/ui": "5.0.5",
+ "happy-dom": "*",
+ "jsdom": "*",
+ "vite": "^6.4.0 || ^7.0.0 || ^8.0.0"
+ },
+ "peerDependenciesMeta": {
+ "@edge-runtime/vm": {
+ "optional": true
+ },
+ "@opentelemetry/api": {
+ "optional": true
+ },
+ "@types/node": {
+ "optional": true
+ },
+ "@vitest/browser-playwright": {
+ "optional": true
+ },
+ "@vitest/browser-preview": {
+ "optional": true
+ },
+ "@vitest/browser-webdriverio": {
+ "optional": true
+ },
+ "@vitest/coverage-istanbul": {
+ "optional": true
+ },
+ "@vitest/coverage-v8": {
+ "optional": true
+ },
+ "@vitest/ui": {
+ "optional": true
+ },
+ "happy-dom": {
+ "optional": true
+ },
+ "jsdom": {
+ "optional": true
+ },
+ "vite": {
+ "optional": false
+ }
+ }
+ },
+ "node_modules/vitest/node_modules/magic-string": {
+ "version": "1.4.5",
+ "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-1.4.5.tgz",
+ "integrity": "sha512-oLLC2+YvQUfAa+vxoufE/yc1icb87M9iXIRqAoR/2EhE/8cb3SF/+Gi0c2By1B+5Vz2e2vQtUvHiu1+63Znh+w==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@jridgewell/sourcemap-codec": "^1.6.0"
+ }
+ },
+ "node_modules/why-is-node-running": {
+ "version": "3.2.1",
+ "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-3.2.1.tgz",
+ "integrity": "sha512-Tb2FUhB4vUsGQlfSquQLYkApkuPAFQXGFzxWKHHumVz2dK+X1RUm/HnID4+TfIGYJ1kTcwOaCk/buYCEJr6YjQ==",
+ "dev": true,
+ "license": "MIT",
+ "bin": {
+ "why-is-node-running": "cli.js"
+ },
+ "engines": {
+ "node": ">=20.11"
+ }
+ },
+ "node_modules/zimmerframe": {
+ "version": "1.1.5",
+ "resolved": "https://registry.npmjs.org/zimmerframe/-/zimmerframe-1.1.5.tgz",
+ "integrity": "sha512-msJxIvYDYcoNL+PJsu+7qmpDWsYmAxTY+2TNYXXF0hzBzBk0BMecOqDOG/EckUoKCuKwObfbugIl8QpqHDXeFA==",
+ "dev": true,
+ "license": "MIT"
+ }
+ }
+}
diff --git a/ui/lib/voice/package.json b/ui/lib/voice/package.json
new file mode 100644
index 000000000..8f9438c86
--- /dev/null
+++ b/ui/lib/voice/package.json
@@ -0,0 +1,53 @@
+{
+ "name": "@ggml-org/llama-voice",
+ "version": "0.1.0",
+ "description": "Voice input for llama.cpp frontend apps: capture, transcription and dictation.",
+ "homepage": "https://github.com/ggml-org/llama.cpp/tree/master/ui/lib/voice",
+ "license": "MIT",
+ "repository": {
+ "type": "git",
+ "url": "git+https://github.com/ggml-org/llama.cpp.git",
+ "directory": "ui/lib/voice"
+ },
+ "files": [
+ "dist"
+ ],
+ "type": "module",
+ "sideEffects": false,
+ "types": "./dist/index.d.ts",
+ "exports": {
+ ".": {
+ "types": "./dist/index.d.ts",
+ "default": "./dist/index.js"
+ },
+ "./svelte": {
+ "types": "./dist/svelte/index.d.ts",
+ "svelte": "./dist/svelte/index.js",
+ "default": "./dist/svelte/index.js"
+ }
+ },
+ "scripts": {
+ "build": "tsc -p tsconfig.build.json",
+ "check": "tsc --noEmit",
+ "prepack": "npm run build && publint",
+ "test": "vitest --run"
+ },
+ "peerDependencies": {
+ "svelte": "^5.0.0"
+ },
+ "peerDependenciesMeta": {
+ "svelte": {
+ "optional": true
+ }
+ },
+ "devDependencies": {
+ "@types/node": "24.19.2",
+ "publint": "0.3.25",
+ "svelte": "5.57.3",
+ "typescript": "6.0.3",
+ "vitest": "5.0.5"
+ },
+ "engines": {
+ "node": ">=20"
+ }
+}
diff --git a/ui/lib/voice/src/audio-context.ts b/ui/lib/voice/src/audio-context.ts
new file mode 100644
index 000000000..daca8baee
--- /dev/null
+++ b/ui/lib/voice/src/audio-context.ts
@@ -0,0 +1,87 @@
+import { VOICE_AUDIO_CONTEXT_STATE, VOICE_AUDIO_SINK_NONE } from './constants.js';
+import { loadPcmTapModule } from './pcm-tap.js';
+
+/**
+ * The page's one AudioContext for voice capture. Making a context and loading the
+ * capture worklet take a noticeable moment, so this is done once (ideally ahead of
+ * the first recording, see warmUpVoiceCapture) and the context is suspended between
+ * recordings instead of closed.
+ *
+ * Where the browser allows it the context has no output device (`sinkId: none`):
+ * capture never opens the speakers, so it cannot click or change their format while
+ * something else plays.
+ */
+
+let shared: AudioContext | null = null;
+let modulesReady: Promise<void> | null = null;
+// captures using the shared context: it idles once the last one lets go, not before, as a
+// capture still letting go may overlap the next one's start
+let holders = 0;
+
+function createContext(): AudioContext {
+ try {
+ // Chrome renders without an output device; other browsers reject the option
+ return new AudioContext({
+ sinkId: VOICE_AUDIO_SINK_NONE
+ } as AudioContextOptions);
+ } catch {
+ return new AudioContext();
+ }
+}
+
+function sharedContext(): { context: AudioContext; ready: Promise<void> } {
+ if (!shared || shared.state === VOICE_AUDIO_CONTEXT_STATE.CLOSED) {
+ shared = createContext();
+ holders = 0;
+ modulesReady = loadPcmTapModule(shared).catch((error) => {
+ // the tap falls back to a ScriptProcessor
+ console.warn('AudioWorklet unavailable, using ScriptProcessor:', error);
+ });
+ }
+
+ return { context: shared, ready: modulesReady ?? Promise.resolve() };
+}
+
+/**
+ * The shared context for a capture, made on first use, with the PCM tap's worklet loading.
+ * The capture hands it back with idleCaptureContext.
+ */
+export function captureContext(): {
+ context: AudioContext;
+ ready: Promise<void>;
+} {
+ const handle = sharedContext();
+
+ holders++;
+
+ return handle;
+}
+
+/**
+ * Makes the context and loads the worklet ahead of a recording, e.g. when the chat
+ * form gets focus. It asks for no microphone, so the browser shows no recording
+ * indicator; outside a user gesture the context waits suspended.
+ */
+export function warmUpVoiceCapture(): void {
+ if (typeof AudioContext === 'undefined') return;
+
+ try {
+ void sharedContext().ready;
+ } catch {
+ // no audio here: the recording reports it when it starts
+ }
+}
+
+/** Hands back a capture's context: it idles between recordings, so it renders nothing. */
+export function idleCaptureContext(context: AudioContext): void {
+ if (context !== shared) {
+ if (context.state !== VOICE_AUDIO_CONTEXT_STATE.CLOSED) context.close().catch(() => {});
+
+ return;
+ }
+
+ holders = Math.max(0, holders - 1);
+
+ if (holders === 0 && context.state === VOICE_AUDIO_CONTEXT_STATE.RUNNING)
+ context.suspend().catch(() => {});
+}
diff --git a/ui/lib/voice/src/capture.ts b/ui/lib/voice/src/capture.ts
new file mode 100644
index 000000000..a6b11f63e
--- /dev/null
+++ b/ui/lib/voice/src/capture.ts
@@ -0,0 +1,468 @@
+import { captureContext, idleCaptureContext } from './audio-context.js';
+import {
+ VOICE_ANALYSER_FFT_SIZE,
+ VOICE_AUDIO_CONTEXT_STATE,
+ VOICE_CHUNK_MS,
+ VOICE_DOM_ERROR,
+ VOICE_ERROR_MESSAGE,
+ VOICE_FIRST_AUDIO_TIMEOUT_MS,
+ VOICE_GAIN,
+ VOICE_LEVELS_INTERVAL_MS,
+ VOICE_LEVELS_LENGTH,
+ VOICE_PEAKS_COUNT,
+ VOICE_SAMPLE_RATE
+} from './constants.js';
+import { VoiceCaptureState } from './enums.js';
+import { VoiceCaptureError } from './errors.js';
+import { normalizeSpeech, SpeechGain } from './gain.js';
+import { LevelHistory, rms, rmsToLevel } from './levels.js';
+import { microphoneRequest, stopStream } from './microphone.js';
+import { concatPcm16, createResampler, floatToPcm16, type Resampler } from './pcm.js';
+import { createPcmTap, type PcmTap } from './pcm-tap.js';
+import { computePeaks } from './peaks.js';
+import { describeMicrophoneError } from './support.js';
+import type {
+ VoiceCaptureOptions,
+ VoiceClip,
+ VoiceLevelsListener,
+ VoicePcmListener,
+ VoiceQuietListener
+} from './types.js';
+import { encodeWav } from './wav.js';
+
+/** Graph and device handles of a running capture, released together. */
+interface CaptureGraph {
+ analyser: AnalyserNode;
+ context: AudioContext;
+ mute: GainNode;
+ source: MediaStreamAudioSourceNode;
+ stream: MediaStream;
+ tap: PcmTap;
+}
+
+/**
+ * Microphone capture for voice input. One `start()` gives, at once: a live level
+ * history for the waveform, PCM16 mono chunks for streaming, and the recording,
+ * returned by `stop()` as a WAV clip with its peaks. With `boost`, quiet speech is
+ * raised (gain.ts): the chunks as they stream, the clip in one pass when it stops.
+ * Whether the speech is too quiet even so is told either way (onQuiet). `start()` resolves once audio
+ * flows, so what is said after it is in the clip. Every path (stop, cancel, a failed
+ * start, the page going away) releases the tracks and the nodes; the page's shared
+ * AudioContext is suspended for the next recording (see audio-context.ts).
+ */
+export class VoiceCapture {
+ /** Time between two level values. */
+ readonly levelsIntervalMs: number;
+
+ readonly sampleRate: number;
+ private chunks: Int16Array[] = [];
+ private readonly chunkSamples: number;
+ private currentState: VoiceCaptureState = VoiceCaptureState.IDLE;
+ // resolves the wait of start() for the first audio
+ private firstAudio: (() => void) | null = null;
+ private frame: number | null = null;
+ private gain: SpeechGain | null = null;
+
+ // bumped by every start, stop and cancel, so a start still awaiting the microphone can tell it was dropped
+ private generation = 0;
+ private graph: CaptureGraph | null = null;
+ private handlePageHide = () => this.cancel();
+ private readonly history: LevelHistory;
+ private levelListeners = new Set<VoiceLevelsListener>();
+ private readonly options: VoiceCaptureOptions;
+ private partial: number[] = [];
+ private pcmListeners = new Set<VoicePcmListener>();
+ private readonly peaksCount: number;
+ private quiet = false;
+ // recorded time from which the speech has stayed too quiet, null while it is not
+ private quietSinceMs: number | null = null;
+ private quietListeners = new Set<VoiceQuietListener>();
+ private recordedSamples = 0;
+
+ private resampler: Resampler | null = null;
+
+ /** Length of the audio captured so far. */
+ get durationMs(): number {
+ return ((this.recordedSamples + this.partial.length) / this.sampleRate) * 1000;
+ }
+
+ /** The level history now, 0..1, newest last. */
+ get levels(): number[] {
+ return this.history.snapshot();
+ }
+
+ get state(): VoiceCaptureState {
+ return this.currentState;
+ }
+
+ /** Whether the speech heard so far stays quiet even at the highest gain. */
+ get tooQuiet(): boolean {
+ return this.quiet;
+ }
+
+ /** Drops the capture, including one still asking for the microphone. */
+ cancel(): void {
+ this.generation++;
+ this.firstAudio?.();
+ this.teardown();
+ this.chunks = [];
+ this.partial = [];
+ this.recordedSamples = 0;
+ this.currentState = VoiceCaptureState.IDLE;
+ }
+
+ constructor(options: VoiceCaptureOptions = {}) {
+ this.options = options;
+ this.sampleRate = options.sampleRate ?? VOICE_SAMPLE_RATE;
+ this.chunkSamples = Math.max(
+ 1,
+ Math.round((this.sampleRate * (options.chunkMs ?? VOICE_CHUNK_MS)) / 1000)
+ );
+ this.history = new LevelHistory(options.levelsLength ?? VOICE_LEVELS_LENGTH);
+ this.levelsIntervalMs = options.levelsIntervalMs ?? VOICE_LEVELS_INTERVAL_MS;
+ this.peaksCount = options.peaksCount ?? VOICE_PEAKS_COUNT;
+ }
+
+ /** Listens to the level history; returns the unsubscribe function. */
+ onLevels(listener: VoiceLevelsListener): () => void {
+ this.levelListeners.add(listener);
+
+ return () => this.levelListeners.delete(listener);
+ }
+
+ /** Listens to the PCM16 chunks; returns the unsubscribe function. */
+ onPcm(listener: VoicePcmListener): () => void {
+ this.pcmListeners.add(listener);
+
+ return () => this.pcmListeners.delete(listener);
+ }
+
+ /** Listens to whether the speech is too quiet, told when it changes; returns the unsubscribe function. */
+ onQuiet(listener: VoiceQuietListener): () => void {
+ this.quietListeners.add(listener);
+
+ return () => this.quietListeners.delete(listener);
+ }
+
+ /**
+ * Asks for the microphone and starts capturing.
+ * @throws VoiceCaptureError with a message for the user when the microphone does not start
+ */
+ async start(): Promise<void> {
+ if (
+ this.currentState === VoiceCaptureState.STARTING ||
+ this.currentState === VoiceCaptureState.RECORDING
+ ) {
+ throw new Error(VOICE_ERROR_MESSAGE.captureAlreadyStarted);
+ }
+
+ const generation = ++this.generation;
+ const dropped = () => generation !== this.generation;
+
+ this.currentState = VoiceCaptureState.STARTING;
+ this.chunks = [];
+ this.partial = [];
+ this.recordedSamples = 0;
+ this.history.reset();
+ this.gain = new SpeechGain(VOICE_GAIN.MAX, this.sampleRate);
+ this.quietSinceMs = null;
+ this.setQuiet(false);
+
+ let stream: MediaStream | null = this.options.stream ?? null;
+ let context: AudioContext | null = null;
+ let started = false;
+
+ const timing = startTiming();
+
+ try {
+ const shared = captureContext();
+
+ context = shared.context;
+
+ // resumed now, while the click still allows it: the microphone may take a while
+ if (context.state === VOICE_AUDIO_CONTEXT_STATE.SUSPENDED) context.resume().catch(() => {});
+
+ const request = microphoneRequest(this.options);
+
+ stream ??= await navigator.mediaDevices.getUserMedia(request);
+ timing.mark('microphone');
+ timing.note(checkProcessing(stream, request.audio as MediaTrackConstraints));
+
+ if (dropped())
+ throw new DOMException(VOICE_ERROR_MESSAGE.captureCancelled, VOICE_DOM_ERROR.ABORT);
+
+ await shared.ready;
+ timing.mark('audio context');
+
+ const source = context.createMediaStreamSource(stream);
+ const analyser = context.createAnalyser();
+ const mute = context.createGain();
+
+ analyser.fftSize = VOICE_ANALYSER_FFT_SIZE;
+ mute.gain.value = 0;
+ this.resampler = createResampler(context.sampleRate, this.sampleRate);
+
+ const tap = await createPcmTap(context, (block) => this.handleBlock(block));
+
+ if (dropped()) {
+ tap.disconnect();
+
+ throw new DOMException(VOICE_ERROR_MESSAGE.captureCancelled, VOICE_DOM_ERROR.ABORT);
+ }
+
+ // the tap only runs while it is pulled by the destination, muted so nothing is heard
+ source.connect(analyser);
+ source.connect(tap.node);
+ tap.node.connect(mute);
+ mute.connect(context.destination);
+
+ this.graph = { analyser, context, mute, source, stream, tap };
+ started = true;
+ this.currentState = VoiceCaptureState.RECORDING;
+ this.watchLevels(analyser);
+ addEventListener('pagehide', this.handlePageHide);
+
+ await this.waitForFirstAudio();
+ timing.mark('first audio');
+ timing.report();
+
+ if (dropped())
+ throw new DOMException(VOICE_ERROR_MESSAGE.captureCancelled, VOICE_DOM_ERROR.ABORT);
+ } catch (error) {
+ if (!started) releaseDevice(stream, context);
+
+ if (!dropped()) {
+ this.currentState = VoiceCaptureState.IDLE;
+ }
+
+ if (error instanceof DOMException && error.name === VOICE_DOM_ERROR.ABORT && dropped())
+ throw error;
+
+ throw new VoiceCaptureError(describeMicrophoneError(error), {
+ cause: error
+ });
+ }
+ }
+
+ /** Ends the capture and returns the recording. */
+ async stop(): Promise<VoiceClip> {
+ const graph = this.graph;
+
+ if (!graph || this.currentState !== VoiceCaptureState.RECORDING) {
+ throw new Error(VOICE_ERROR_MESSAGE.captureNotStarted);
+ }
+
+ this.generation++;
+ await graph.tap.flush();
+
+ if (this.graph !== graph) throw new Error(VOICE_ERROR_MESSAGE.captureCancelled);
+
+ this.flushPartial();
+ this.teardown();
+
+ const recorded = concatPcm16(this.chunks);
+ // the whole recording at once: its first words get the gain the later ones set
+ const samples = this.options.boost
+ ? normalizeSpeech(recorded, this.sampleRate).samples
+ : recorded;
+
+ this.chunks = [];
+ this.currentState = VoiceCaptureState.STOPPED;
+
+ return {
+ blob: encodeWav(samples, this.sampleRate),
+ durationMs: (samples.length / this.sampleRate) * 1000,
+ peaks: computePeaks(samples, this.peaksCount),
+ sampleRate: this.sampleRate
+ };
+ }
+
+ private flushPartial(): void {
+ if (this.partial.length === 0) return;
+
+ const samples = Float32Array.from(this.partial);
+ // the recording keeps the raw signal: stop() raises it as a whole
+ const chunk = floatToPcm16(samples);
+ const boost = this.options.boost === true;
+ const streamed = this.gain?.process(samples, boost) ?? samples;
+
+ this.partial = [];
+ this.chunks.push(chunk);
+ this.recordedSamples += chunk.length;
+ this.followQuiet(this.gain?.tooQuiet ?? false);
+
+ const sent = boost && streamed !== samples ? floatToPcm16(streamed) : chunk;
+
+ for (const listener of this.pcmListeners) listener(sent);
+ }
+
+ private handleBlock(block: Float32Array): void {
+ if (!this.resampler || this.currentState !== VoiceCaptureState.RECORDING) return;
+
+ this.firstAudio?.();
+
+ const resampled = this.resampler.process(block);
+
+ for (let i = 0; i < resampled.length; i++) {
+ this.partial.push(resampled[i]);
+
+ if (this.partial.length === this.chunkSamples) this.flushPartial();
+ }
+ }
+
+ /**
+ * Tells that the speech is too quiet only once it has stayed so for a while: the first
+ * words can be quiet while the microphone settles, and a warning that comes and goes
+ * helps nobody. It clears at once.
+ */
+ private followQuiet(quiet: boolean): void {
+ if (!quiet) {
+ this.quietSinceMs = null;
+ this.setQuiet(false);
+
+ return;
+ }
+
+ const recordedMs = (this.recordedSamples / this.sampleRate) * 1000;
+
+ this.quietSinceMs ??= recordedMs;
+
+ if (recordedMs - this.quietSinceMs >= VOICE_GAIN.QUIET_HOLD_MS) this.setQuiet(true);
+ }
+
+ private setQuiet(quiet: boolean): void {
+ if (quiet === this.quiet) return;
+
+ this.quiet = quiet;
+
+ for (const listener of this.quietListeners) listener(quiet);
+ }
+
+ private teardown(): void {
+ removeEventListener('pagehide', this.handlePageHide);
+
+ if (this.frame !== null) {
+ cancelAnimationFrame(this.frame);
+ this.frame = null;
+ }
+
+ const graph = this.graph;
+
+ this.graph = null;
+ this.resampler = null;
+
+ if (!graph) return;
+
+ graph.tap.disconnect();
+ graph.source.disconnect();
+ graph.analyser.disconnect();
+ graph.mute.disconnect();
+ releaseDevice(graph.stream, graph.context);
+ }
+
+ /** Resolves with the first audio block, or after a timeout so a silent device still records. */
+ private waitForFirstAudio(): Promise<void> {
+ return new Promise((resolve) => {
+ const done = () => {
+ clearTimeout(timer);
+ this.firstAudio = null;
+ resolve();
+ };
+ const timer = setTimeout(done, VOICE_FIRST_AUDIO_TIMEOUT_MS);
+
+ this.firstAudio = done;
+ });
+ }
+
+ private watchLevels(analyser: AnalyserNode): void {
+ const buffer = new Float32Array(analyser.fftSize);
+
+ let loudest = 0;
+ let last = performance.now();
+
+ const tick = (now: number) => {
+ if (!this.graph) return;
+
+ analyser.getFloatTimeDomainData(buffer);
+ loudest = Math.max(loudest, rmsToLevel(rms(buffer)));
+
+ if (now - last >= this.levelsIntervalMs) {
+ const levels = this.history.push(loudest);
+
+ loudest = 0;
+ last = now;
+
+ for (const listener of this.levelListeners) listener(levels);
+ }
+
+ this.frame = requestAnimationFrame(tick);
+ };
+
+ this.frame = requestAnimationFrame(tick);
+ }
+}
+
+function releaseDevice(stream: MediaStream | null, context: AudioContext | null): void {
+ stopStream(stream);
+
+ if (context) idleCaptureContext(context);
+}
+
+const PROCESSING = ['echoCancellation', 'noiseSuppression', 'autoGainControl'] as const;
+
+/**
+ * What the browser gave for the microphone, e.g. "MacBook Pro Microphone, 48000 Hz, raw".
+ * Processing that was asked off but is on is warned about: the browser keeps one setting
+ * per device, so a page or app that has the same microphone open with echo cancellation
+ * (a call, voice mode) gets the recording processed too, and the system may lower other
+ * sound while it lasts.
+ */
+function checkProcessing(stream: MediaStream, asked: MediaTrackConstraints): string {
+ const track = stream.getAudioTracks()[0];
+
+ if (!track) return 'no audio track';
+
+ const settings = track.getSettings();
+ const on = PROCESSING.filter((key) => settings[key] === true);
+ const unwanted = on.filter((key) => asked[key] === false);
+
+ if (unwanted.length > 0) {
+ console.warn(
+ `[voice] ${track.label || 'The microphone'} is processed (${unwanted.join(', ')}) although the raw signal was asked for; another tab or app may have it open for a call.`
+ );
+ }
+
+ const rate = settings.sampleRate ? `${settings.sampleRate} Hz` : null;
+
+ return [track.label || 'microphone', rate, on.length > 0 ? on.join(' + ') : 'raw']
+ .filter(Boolean)
+ .join(', ');
+}
+
+/** How long each step of a start took, logged in development to see where a slow start goes. */
+function startTiming() {
+ const startedAt = performance.now();
+ const steps: string[] = [];
+ const notes: string[] = [];
+
+ let last = startedAt;
+
+ return {
+ mark(step: string) {
+ const now = performance.now();
+
+ steps.push(`${step} ${Math.round(now - last)} ms`);
+ last = now;
+ },
+ note(text: string) {
+ notes.push(text);
+ },
+ report() {
+ if (!import.meta.env?.DEV) return;
+
+ console.debug(
+ `[voice] recording after ${Math.round(performance.now() - startedAt)} ms: ${steps.join(', ')}${notes.length > 0 ? ` (${notes.join('; ')})` : ''}`
+ );
+ }
+ };
+}
diff --git a/ui/lib/voice/src/constants.ts b/ui/lib/voice/src/constants.ts
new file mode 100644
index 000000000..3b56c078a
--- /dev/null
+++ b/ui/lib/voice/src/constants.ts
@@ -0,0 +1,322 @@
+/** Voice engine defaults. */
+
+/** Type of the recorded clips: 16-bit PCM WAV, what speech encoders and `input_audio` take. */
+export const VOICE_WAV_MIME = 'audio/wav';
+
+/** Why the microphone can't record, as the app's microphone button and toasts tell the user. */
+export const VOICE_MICROPHONE_MESSAGE = {
+ busy: 'The microphone is in use by another app.',
+ denied: "Microphone access was denied. Allow it in the browser's site settings to record.",
+ failed: "The microphone couldn't start.",
+ insecure:
+ 'The microphone only works over HTTPS or on localhost. Open this page over HTTPS to record.',
+ notFound: 'No microphone was found.',
+ unsupported: "This browser can't record audio."
+} as const;
+
+/** Rate of the PCM chunks and of the recorded WAV, what speech encoders expect. */
+export const VOICE_SAMPLE_RATE = 16_000;
+
+/** Length of the PCM chunks handed to the stream. */
+export const VOICE_CHUNK_MS = 100;
+
+/** Number of values in the live level history, newest last. */
+export const VOICE_LEVELS_LENGTH = 48;
+
+/** Time between two values of the live level history. */
+export const VOICE_LEVELS_INTERVAL_MS = 50;
+
+/** Level floor in dBFS: quieter input reads as 0. */
+export const VOICE_LEVELS_FLOOR_DB = -60;
+
+/** Analyser window, about 20 ms at 48 kHz. */
+export const VOICE_ANALYSER_FFT_SIZE = 1024;
+
+/** Number of peaks of a clip's static waveform. */
+export const VOICE_PEAKS_COUNT = 64;
+
+/**
+ * Microphone constraints: the raw signal. Echo cancellation makes the system treat the
+ * recording as a call, which lowers or interrupts what else is playing (Apple's voice
+ * processing, Windows' communications ducking); nothing plays while we record, so it
+ * has nothing to cancel. Auto gain fades the input in, losing the first words, and
+ * speech models take unprocessed audio well.
+ */
+export const VOICE_MICROPHONE_CONSTRAINTS: MediaTrackConstraints = {
+ autoGainControl: false,
+ echoCancellation: false,
+ noiseSuppression: false
+};
+
+/**
+ * How live text is cut into pieces for a model that transcribes per commit: at a pause
+ * after speech, or at the next quiet moment once a piece is this long. Short pieces keep
+ * the text close behind the speaker; the cut always falls between words.
+ */
+export const VOICE_PIECE = {
+ MAX_MS: 3000,
+ MIN_SPEECH_MS: 250,
+ PAUSE_MS: 350
+} as const;
+
+/**
+ * Longest recording that is transcribed once more as a whole when it ends. Pieces cut at
+ * the speaker's pauses come back as sentences of their own ("I went. To the store."); one
+ * pass over the whole clip punctuates by meaning. Past this, the pieces' text is kept.
+ */
+export const VOICE_RETRANSCRIBE_MAX_MS = 120_000;
+
+/** How long a started capture waits for its first audio before it counts as recording anyway. */
+export const VOICE_FIRST_AUDIO_TIMEOUT_MS = 1500;
+
+/** Name of the PCM tap worklet processor. */
+export const VOICE_WORKLET_PROCESSOR = 'llama-voice-pcm-tap';
+
+/** Frames the worklet gathers before it posts them, about 43 ms at 48 kHz. */
+export const VOICE_WORKLET_BATCH_FRAMES = 2048;
+
+/** Buffer size of the ScriptProcessor fallback. */
+export const VOICE_SCRIPT_PROCESSOR_FRAMES = 2048;
+
+/** Bytes per String.fromCharCode call when encoding base64, below the argument limit. */
+export const VOICE_BASE64_SLICE = 0x8000;
+
+/** Server routes, relative so a server under a subpath still works. */
+export const VOICE_API_PATHS = {
+ REALTIME: './v1/realtime',
+ TRANSCRIPTIONS: './v1/audio/transcriptions'
+} as const;
+
+/** Query parameters of the realtime socket. */
+export const VOICE_REALTIME_QUERY = {
+ AUTOLOAD: 'autoload',
+ INTENT: 'intent',
+ INTENT_TRANSCRIPTION: 'transcription',
+ MODEL: 'model'
+} as const;
+
+/** Realtime events this client sends and reads, see tools/server/README.md. */
+export const VOICE_REALTIME_EVENTS = {
+ APPEND: 'input_audio_buffer.append',
+ COMMIT: 'input_audio_buffer.commit',
+ COMPLETED: 'conversation.item.input_audio_transcription.completed',
+ DELTA: 'conversation.item.input_audio_transcription.delta',
+ ERROR: 'error',
+ SESSION_CREATED: 'session.created',
+ SESSION_UPDATE: 'session.update'
+} as const;
+
+/**
+ * How Qwen3-ASR opens its answer: the language it heard, then the tag its text follows,
+ * e.g. `language French<asr_text>`.
+ */
+export const VOICE_ASR_TEXT_PREFIX_PATTERN = /^\s*(?:language\s+[^<\n]{0,40})?<asr_text>/i;
+
+/** The tag around Qwen3-ASR's text, wherever it stands. */
+export const VOICE_ASR_TEXT_TAG_PATTERN = /<\/?asr_text>/gi;
+
+/**
+ * How VibeVoice-ASR opens its answer: a JSON array of timed segments, or the start of
+ * one while it streams.
+ */
+export const VOICE_SEGMENTS_START_PATTERN = /^\s*\[\s*(?:\{\s*(?:"|$)|$)/;
+
+/** The text of each timed segment, its closing quote missing while the last one streams. */
+export const VOICE_SEGMENTS_CONTENT_PATTERN = /"content"\s*:\s*"((?:[^"\\]|\\.)*)/gi;
+
+/** An escape cut short at the end of a JSON string that is still streaming. */
+export const VOICE_UNFINISHED_ESCAPE_PATTERN = /\\(?:u[0-9a-f]{0,3})?$/i;
+
+/** The session type that transcribes; a conversation session is switched to it. */
+export const VOICE_REALTIME_SESSION_TRANSCRIPTION = 'transcription';
+
+/** What the server says when the model has no streaming audio encoder. */
+export const VOICE_REALTIME_UNSUPPORTED_PATTERN = /does not support realtime transcription/i;
+
+/** Deadlines of a realtime session. */
+export const VOICE_REALTIME_TIMEOUTS = {
+ /** Socket open until session.created; a busy slot can hold it back */
+ CONNECT_MS: 10_000,
+ /** Final commit until the last completed event */
+ FINISH_MS: 20_000
+} as const;
+
+/**
+ * Speech detection (speech-detector.ts): what tells speech from the room, for the pause
+ * detector, the speech gain and any feature listening for speech. Levels in dBFS.
+ */
+export const VOICE_SPEECH = {
+ /** How far above the noise floor a level is speech */
+ ABOVE_FLOOR_DB: 9,
+ /** How fast the floor creeps back up through louder levels */
+ FLOOR_RISE_DB_PER_S: 2,
+ /** Quieter than this is never speech, whatever the room */
+ MIN_DB: -54,
+ /** Up to this far above the floor a level is quiet: the room, or a breath between words */
+ QUIET_ABOVE_FLOOR_DB: 5
+} as const;
+
+/**
+ * Speech gain (gain.ts): how a quiet microphone is brought up for transcription. Levels
+ * are RMS of samples in -1..1.
+ */
+export const VOICE_GAIN = {
+ /** Frames a finished recording is measured in */
+ FRAME_MS: 20,
+ /** Where in a finished recording's frames, quietest first, its noise floor is read */
+ FLOOR_QUANTILE: 0.1,
+ /** The soft limiter's knee: louder samples bend towards full scale */
+ KNEE: 0.8,
+ /** How long speech must stay too quiet before the recording says so, as the microphone may settle */
+ QUIET_HOLD_MS: 2000,
+ /** The most it is raised: 8x, +18 dB, so a far-off voice does not bring the room up with it */
+ MAX: 8,
+ /** How much of the way to its target the streaming gain rises per speech block */
+ RISE: 0.3,
+ /** How much of each new speech block the streaming speech level takes in */
+ SPEECH_FOLLOW: 0.2,
+ /** Speech is raised towards this level, about -20 dBFS */
+ TARGET_RMS: 0.1,
+ /** Speech below this even at the highest gain (about -32 dBFS) reads as too quiet */
+ TOO_QUIET_RMS: 0.025
+} as const;
+
+/**
+ * How long a microphone opened ahead of a recording (MicrophoneWarmup) stays open once
+ * the pointer leaves the button, in case it comes back to press it.
+ */
+export const VOICE_WARMUP_RELEASE_MS = 1500;
+
+/** Messages of the errors a recording, a transcription or a realtime session ends with. */
+export const VOICE_ERROR_MESSAGE = {
+ captureAlreadyStarted: 'Voice capture already started',
+ captureCancelled: 'Voice capture cancelled',
+ captureNotStarted: 'No voice capture to stop',
+ noTranscript: 'The server returned no transcript.',
+ realtimeClosed: 'The realtime session was closed.',
+ realtimeClosedEarly: 'The realtime socket closed before the transcript completed.',
+ realtimeConnectTimeout: 'The realtime session did not start in time.',
+ realtimeFailed: 'Realtime transcription failed.',
+ realtimeFinishTimeout: 'The transcript did not complete in time.',
+ realtimeNotOpen: 'The realtime session is not open.',
+ realtimeRefused: (code: number) =>
+ `The realtime socket closed before the session started (code ${code}).`,
+ realtimeSocketFailed: 'The realtime socket could not be opened.',
+ recorderBusy: 'Voice recorder already running',
+ recorderIdle: 'No voice recording to stop',
+ transcriptionFailed: 'Transcription failed',
+ workletUnavailable: 'AudioWorklet unavailable'
+} as const;
+
+/** The DOMException names the engine raises or tells apart. */
+export const VOICE_DOM_ERROR = {
+ ABORT: 'AbortError',
+ NOT_ALLOWED: 'NotAllowedError',
+ NOT_FOUND: 'NotFoundError',
+ NOT_READABLE: 'NotReadableError',
+ OVERCONSTRAINED: 'OverconstrainedError',
+ SECURITY: 'SecurityError'
+} as const;
+
+/** AudioContext states the engine checks. */
+export const VOICE_AUDIO_CONTEXT_STATE = {
+ CLOSED: 'closed',
+ RUNNING: 'running',
+ SUSPENDED: 'suspended'
+} as const satisfies Record<string, AudioContextState>;
+
+/** Output of the capture context where the browser allows none: it never opens the speakers. */
+export const VOICE_AUDIO_SINK_NONE = { type: 'none' } as const;
+
+/** What the browser lists microphones as. */
+export const VOICE_AUDIO_INPUT_KIND = 'audioinput' satisfies MediaDeviceKind;
+
+/** Entries Chrome lists beside the microphones: the system default and the communications device. */
+export const VOICE_DEVICE_ALIAS = {
+ COMMUNICATIONS: 'communications',
+ DEFAULT: 'default'
+} as const;
+
+/** Microphone labels that name a Bluetooth headset, whose microphone switches it to call mode. */
+export const VOICE_BLUETOOTH_LABEL_PATTERN = /bluetooth|airpods|beats|hands-?free|headset|buds/i;
+
+/** Microphone labels that name the computer's own microphone. */
+export const VOICE_BUILT_IN_LABEL_PATTERN = /built-?in|internal|macbook|imac|laptop/i;
+
+/** The Permissions API entry for the microphone, and the state that needs no prompt. */
+export const VOICE_MICROPHONE_PERMISSION = {
+ GRANTED: 'granted' satisfies PermissionState,
+ NAME: 'microphone'
+} as const;
+
+/** State of a track that still records. */
+export const VOICE_TRACK_LIVE = 'live' satisfies MediaStreamTrackState;
+
+/** Messages between the main thread and the PCM tap worklet, and how long a flush may take. */
+export const VOICE_WORKLET_MESSAGE = {
+ FLUSH: 'flush',
+ FLUSHED: 'flushed'
+} as const;
+
+export const VOICE_WORKLET_FLUSH_TIMEOUT_MS = 250;
+
+/** Type of the worklet's inline source. */
+export const VOICE_WORKLET_SOURCE_MIME = 'text/javascript';
+
+/** Sample rate of the offline context that decodes stored clips: a placeholder, decoding keeps the file's own. */
+export const VOICE_DECODE_SAMPLE_RATE = 44_100;
+
+/** A data URL's header, before the base64 payload. */
+export const VOICE_DATA_URL_HEADER_PATTERN = /^data:[^,]*,/;
+
+/** Quieter than anything a microphone gives, for a block of digital silence (dBFS). */
+export const VOICE_SILENCE_DB = -120;
+
+/** Defaults of a PauseDetector, see PauseDetectorOptions. */
+export const VOICE_PAUSE_DEFAULTS = {
+ MAX_SEGMENT_MS: 10_000,
+ MIN_SPEECH_MS: 300,
+ /** How far past its longest length a piece waits for a quiet moment before it is cut anyway */
+ OVERTIME: 1.5,
+ PAUSE_MS: 500
+} as const;
+
+/** Fields of the `/v1/audio/transcriptions` form, and the response format asked for. */
+export const VOICE_TRANSCRIPTION_FORM = {
+ FILE: 'file',
+ LANGUAGE: 'language',
+ MODEL: 'model',
+ PROMPT: 'prompt',
+ RESPONSE_FORMAT: 'response_format'
+} as const;
+
+export const VOICE_TRANSCRIPTION_RESPONSE_FORMAT = 'json';
+
+/** File names the audio is sent under: a whole clip, or a piece of a recording still going. */
+export const VOICE_CLIP_FILENAME = {
+ CLIP: 'audio.wav',
+ PIECE: 'piece.wav'
+} as const;
+
+/** The header the browser sets itself for a multipart body. */
+export const VOICE_CONTENT_TYPE_HEADER = 'content-type';
+
+/** An error body that is an HTML page, which says nothing useful in a toast. */
+export const VOICE_HTML_BODY_PATTERN = /^</;
+
+/** Socket protocols for the page's HTTP protocol. */
+export const VOICE_SOCKET_PROTOCOL = {
+ HTTPS: 'https:',
+ WS: 'ws:',
+ WSS: 'wss:'
+} as const;
+
+/** Format of the PCM16 chunks sent to the realtime socket. */
+export const VOICE_REALTIME_AUDIO_FORMAT = 'audio/pcm';
+
+/** Whitespace at either end of a text, where two pieces join without another space. */
+export const VOICE_TRAILING_SPACE_PATTERN = /\s$/;
+export const VOICE_LEADING_SPACE_PATTERN = /^\s/;
+
+/** Punctuation that closes onto the word before it, so dictated text needs no space before it. */
+export const VOICE_CLOSING_PUNCTUATION_PATTERN = /^[,.;:!?)\]}]/;
diff --git a/ui/lib/voice/src/dictation-span.ts b/ui/lib/voice/src/dictation-span.ts
new file mode 100644
index 000000000..791909305
--- /dev/null
+++ b/ui/lib/voice/src/dictation-span.ts
@@ -0,0 +1,122 @@
+import {
+ VOICE_CLOSING_PUNCTUATION_PATTERN,
+ VOICE_LEADING_SPACE_PATTERN,
+ VOICE_TRAILING_SPACE_PATTERN
+} from './constants.js';
+
+/**
+ * The stretch of the message that dictation writes: it starts at the caret, and each
+ * new version of the dictated text replaces only that stretch, so what the user typed
+ * before and after it stays. An edit before the stretch moves it; an edit inside it
+ * hands that text over to the user, and dictation goes on after it.
+ */
+export class DictationSpan {
+ /**
+ * Edits inside the stretch stay in it, as when its text is a voice message's transcript:
+ * what it held is the user's from then on, and new text goes after it.
+ */
+ keepsEdits = false;
+ /** Whether the user edited inside the stretch, see keepsEdits. */
+ userEdited = false;
+ // characters of the dictated text already handed over to the user
+ private handedOver = 0;
+ // characters at the start of the stretch that the user edited, kept as they are
+ private kept = 0;
+ private length = 0;
+ // characters of the dictated text the stretch shows
+ private shown = 0;
+ private start: number;
+
+ /** Where the dictated text ends, e.g. to put the caret after it. */
+ get end(): number {
+ return this.start + this.length;
+ }
+
+ /** Whether the stretch shows any text. */
+ get isEmpty(): boolean {
+ return this.length === 0;
+ }
+
+ /**
+ * Writes the dictated text so far into `value` in place of the stretch, with a space
+ * on either side where it would otherwise run into the neighbouring text.
+ */
+ apply(value: string, text: string): string {
+ const body = text.slice(this.handedOver).trim();
+ const before = value.slice(0, this.start + this.kept);
+ const after = value.slice(this.start + this.length);
+ // no space before text that goes on with punctuation, e.g. ", and then"
+ const lead =
+ body &&
+ before &&
+ !VOICE_TRAILING_SPACE_PATTERN.test(before) &&
+ !VOICE_CLOSING_PUNCTUATION_PATTERN.test(body)
+ ? ' '
+ : '';
+ const trail = body && after && !VOICE_LEADING_SPACE_PATTERN.test(after) ? ' ' : '';
+ const inserted = lead + body + trail;
+
+ this.length = this.kept + inserted.length;
+ this.shown = text.length;
+
+ return before + inserted + after;
+ }
+
+ constructor(value: string, caret: number) {
+ this.start = Math.max(0, Math.min(caret, value.length));
+ }
+
+ /** Follows an edit the user made, `previous` to `next`. */
+ edited(previous: string, next: string): void {
+ const { from, insert, to } = changeBetween(previous, next);
+ const end = this.end;
+ const delta = insert.length - (to - from);
+
+ if (to <= this.start) {
+ this.start += delta;
+ } else if (from >= end) {
+ return;
+ } else if (this.keepsEdits && from >= this.start && to <= end) {
+ this.length += delta;
+ this.kept = this.length;
+ this.handedOver = this.shown;
+ this.userEdited = true;
+ } else {
+ // typed into the dictated text: it is theirs now, what comes next goes after it
+ this.handedOver = this.shown;
+ this.start = to <= end ? end + delta : from + insert.length;
+ this.length = 0;
+ }
+ }
+
+ /** Takes the stretch out of `value`, as a cancelled dictation leaves the message. */
+ remove(value: string): string {
+ const next = value.slice(0, this.start) + value.slice(this.start + this.length);
+
+ this.length = 0;
+
+ return next;
+ }
+
+ /** The text the stretch holds in `value`, with the user's edits inside it. */
+ text(value: string): string {
+ return value.slice(this.start, this.start + this.length);
+ }
+}
+
+/** The single change that turns `current` into `next`: their differing middle. */
+function changeBetween(current: string, next: string) {
+ let from = 0;
+
+ while (from < current.length && from < next.length && current[from] === next[from]) from++;
+
+ let currentTo = current.length;
+ let nextTo = next.length;
+
+ while (currentTo > from && nextTo > from && current[currentTo - 1] === next[nextTo - 1]) {
+ currentTo--;
+ nextTo--;
+ }
+
+ return { from, insert: next.slice(from, nextTo), to: currentTo };
+}
diff --git a/ui/lib/voice/src/dictation.ts b/ui/lib/voice/src/dictation.ts
new file mode 100644
index 000000000..c32291d1c
--- /dev/null
+++ b/ui/lib/voice/src/dictation.ts
@@ -0,0 +1,187 @@
+import { DictationSpan } from './dictation-span.js';
+
+/** The text field dictation writes into. */
+export interface DictationTarget {
+ getValue: () => string;
+ /** Writes the value as dictation changes it; not a user edit. */
+ setValue: (value: string) => void;
+ getCaretOffset: () => number;
+ /** Puts the caret after the dictated text once it is final. */
+ setCaretOffset: (offset: number) => void;
+}
+
+export interface DictationOptions extends DictationTarget {
+ /** Called when what the getters say changes, for a caller that keeps them reactive */
+ onChange?: () => void;
+ /** The pass over the whole recording failed; the live text stays */
+ onError?: (error: unknown) => void;
+}
+
+/** What a finished recording hands dictation: see VoiceSessionResult. */
+export interface DictationResult {
+ /** Text that streamed while recording, so the field shows it already; null when none did */
+ liveText: string | null;
+ /** The text of the whole recording, null when it is not transcribed */
+ transcript: Promise<string> | null;
+}
+
+export interface BeginDictationOptions {
+ /** Where the text goes, the caret by default */
+ at?: number;
+ /** Edits inside the dictated text stay in it, see DictationSpan.keepsEdits */
+ keepsEdits?: boolean;
+}
+
+export type Dictation = ReturnType<typeof createDictation>;
+
+/**
+ * Dictation into a text field, without the recording: the text streams in at the caret
+ * (write()), the user can type around it and into it as it goes (edited()), and when the
+ * recording stops the whole of it may be transcribed once more and replace the live text
+ * (finish()), after which the caret goes after it. Plain TypeScript, so any field can take
+ * it: start a voice session with `onText: dictation.write` and hand its result to finish().
+ */
+export function createDictation(options: DictationOptions) {
+ let span: DictationSpan | null = null;
+ // the value as dictation last left it, to tell the user's edits from its own
+ let last = '';
+ let refining: Promise<void> | null = null;
+ // bumped when a dictation starts or is cancelled, so text still on its way is dropped
+ let generation = 0;
+
+ const changed = () => options.onChange?.();
+
+ function write(text: string) {
+ if (!span) return;
+
+ last = span.apply(options.getValue(), text);
+ options.setValue(last);
+ }
+
+ return {
+ /** Whether dictation is writing into the field. */
+ get active() {
+ return span !== null;
+ },
+
+ /** Starts writing at the caret, or `at`, replacing any dictation under way. */
+ begin(begin: BeginDictationOptions = {}) {
+ generation++;
+ last = options.getValue();
+ span = new DictationSpan(last, begin.at ?? options.getCaretOffset());
+ span.keepsEdits = begin.keepsEdits ?? false;
+ changed();
+ },
+
+ /** Drops the dictated text, as a cancelled recording leaves the field. */
+ cancel() {
+ generation++;
+
+ if (!span) return;
+
+ if (!span.isEmpty) {
+ last = span.remove(options.getValue());
+ options.setValue(last);
+ }
+
+ span = null;
+ changed();
+ },
+
+ /** Follows an edit the user typed while dictation writes into the field. */
+ edited() {
+ const next = options.getValue();
+
+ span?.edited(last, next);
+ last = next;
+ },
+
+ /** Where the dictated text ends, e.g. to put the caret after it; -1 when none is active. */
+ get end() {
+ return span?.end ?? -1;
+ },
+
+ /**
+ * Ends the dictation with the recording's result: the whole recording's text replaces
+ * the live text (when there is any to replace it with), then the caret goes after it.
+ * Resolves once the text is final.
+ */
+ async finish({ liveText, transcript }: DictationResult): Promise<void> {
+ const run = generation;
+ const dictated = span;
+
+ if (!dictated) return;
+
+ if (transcript) {
+ const refined = (async () => {
+ try {
+ const text = await transcript;
+
+ if (run !== generation) return;
+
+ if (text.trim() || !liveText) write(text);
+ } catch (error) {
+ if (run === generation) options.onError?.(error);
+ }
+ })();
+
+ if (liveText) {
+ refining = refined;
+ changed();
+ }
+
+ await refined;
+
+ if (refining === refined) {
+ refining = null;
+ changed();
+ }
+ }
+
+ if (run !== generation) return;
+
+ span = null;
+ changed();
+
+ if (!dictated.isEmpty) options.setCaretOffset(dictated.end);
+ },
+
+ /** The live text is still being replaced by a pass over the whole recording. */
+ get refining() {
+ return refining !== null;
+ },
+
+ /** Resolves once the text in the field is final, at once when nothing is refining it. */
+ settled(): Promise<void> {
+ return refining ?? Promise.resolve();
+ },
+
+ /**
+ * Takes the dictated text back out of the field and ends the dictation, as a voice
+ * message's live text goes with its clip. Returns the text, and whether the user
+ * edited it; null when there was none.
+ */
+ take(): { text: string; userEdited: boolean } | null {
+ const taken = span;
+
+ generation++;
+ span = null;
+
+ if (!taken) return null;
+
+ changed();
+
+ if (taken.isEmpty) return null;
+
+ const previous = options.getValue();
+ const text = taken.text(previous).trim();
+
+ last = taken.remove(previous);
+ options.setValue(last);
+
+ return { text, userEdited: taken.userEdited };
+ },
+
+ write
+ };
+}
diff --git a/ui/lib/voice/src/enums.ts b/ui/lib/voice/src/enums.ts
new file mode 100644
index 000000000..f1ae8d49c
--- /dev/null
+++ b/ui/lib/voice/src/enums.ts
@@ -0,0 +1,35 @@
+/** Voice engine enums: the states and reasons the engine reports. */
+
+/** Where a VoiceCapture is. */
+export enum VoiceCaptureState {
+ IDLE = 'idle',
+ RECORDING = 'recording',
+ STARTING = 'starting',
+ STOPPED = 'stopped'
+}
+
+/** Where a VoiceRecorder (voice/svelte) is. */
+export enum VoiceRecorderStatus {
+ IDLE = 'idle',
+ RECORDING = 'recording',
+ STARTING = 'starting',
+ STOPPING = 'stopping'
+}
+
+/** Where a RealtimeTranscriber's socket is. */
+export enum RealtimeState {
+ CLOSED = 'closed',
+ CONNECTING = 'connecting',
+ FINISHING = 'finishing',
+ IDLE = 'idle',
+ OPEN = 'open'
+}
+
+/**
+ * Why realtime transcription is not available: the model has no streaming audio encoder,
+ * or the socket could not be opened.
+ */
+export enum RealtimeUnsupportedReason {
+ CONNECTION = 'connection',
+ MODEL = 'model'
+}
diff --git a/ui/lib/voice/src/env.d.ts b/ui/lib/voice/src/env.d.ts
new file mode 100644
index 000000000..261525147
--- /dev/null
+++ b/ui/lib/voice/src/env.d.ts
@@ -0,0 +1,4 @@
+// The bundler's flags, read when present: Vite sets import.meta.env.DEV in development.
+interface ImportMeta {
+ readonly env?: { readonly DEV?: boolean };
+}
diff --git a/ui/lib/voice/src/errors.ts b/ui/lib/voice/src/errors.ts
new file mode 100644
index 000000000..9d5be32bd
--- /dev/null
+++ b/ui/lib/voice/src/errors.ts
@@ -0,0 +1,43 @@
+import type { RealtimeUnsupportedReason } from './enums.js';
+
+/** A realtime session failed; the caller can transcribe the clip over HTTP instead. */
+export class RealtimeError extends Error {
+ constructor(message: string, options?: { cause?: unknown }) {
+ super(message, options);
+ this.name = 'RealtimeError';
+ }
+}
+
+/**
+ * Realtime transcription is not available: the model has no streaming audio encoder
+ * (`model`), or the socket could not be opened (`connection`), as with a server that
+ * checks an API key, which a browser websocket cannot send.
+ */
+export class RealtimeUnsupportedError extends RealtimeError {
+ readonly reason: RealtimeUnsupportedReason;
+
+ constructor(message: string, reason: RealtimeUnsupportedReason, options?: { cause?: unknown }) {
+ super(message, options);
+ this.name = 'RealtimeUnsupportedError';
+ this.reason = reason;
+ }
+}
+
+/** The transcription endpoint answered with an error; `message` is the server's own. */
+export class VoiceTranscriptionError extends Error {
+ readonly status: number;
+
+ constructor(message: string, status: number) {
+ super(message);
+ this.name = 'VoiceTranscriptionError';
+ this.status = status;
+ }
+}
+
+/** The microphone did not start; `message` says why in words for the user. */
+export class VoiceCaptureError extends Error {
+ constructor(message: string, options?: { cause?: unknown }) {
+ super(message, options);
+ this.name = 'VoiceCaptureError';
+ }
+}
diff --git a/ui/lib/voice/src/gain.ts b/ui/lib/voice/src/gain.ts
new file mode 100644
index 000000000..c0f56b2bd
--- /dev/null
+++ b/ui/lib/voice/src/gain.ts
@@ -0,0 +1,168 @@
+import { VOICE_GAIN, VOICE_SAMPLE_RATE } from './constants.js';
+import { rms, rmsToDb } from './levels.js';
+import { floatToPcm16, pcm16ToFloat } from './pcm.js';
+import { isSpeechLevel, SpeechDetector } from './speech-detector.js';
+
+/**
+ * Speech gain: brings a quiet microphone up to a level transcription models hear well.
+ * Only speech is measured, never the room's silence (see SpeechDetector), so a pause does not pump the gain
+ * up; the gain only ever raises the input (up to VOICE_GAIN.MAX), never lowers it, and
+ * a soft limiter keeps the raised peaks from clipping.
+ *
+ * Two forms: SpeechGain follows the input block by block for what streams while
+ * speaking, and normalizeSpeech() treats a finished recording in one pass, so its
+ * first words get the same gain as the rest.
+ */
+
+/** What normalizeSpeech() did to a recording. */
+export interface SpeechGainResult {
+ /** The recording, raised, or the same samples when no speech was found or it was loud enough */
+ samples: Int16Array;
+ /** The gain applied, 1 when none */
+ gain: number;
+ /** The speech is quiet even at the highest gain: a sign the microphone is too far or too low */
+ tooQuiet: boolean;
+}
+
+/** Gain that brings speech of this RMS to the target, within 1..max. */
+export function speechGainFor(speechRms: number, max: number = VOICE_GAIN.MAX): number {
+ if (speechRms <= 0) return 1;
+
+ return Math.max(1, Math.min(max, VOICE_GAIN.TARGET_RMS / speechRms));
+}
+
+/** Whether speech of this RMS stays quiet even at the highest gain. */
+export function isTooQuiet(speechRms: number): boolean {
+ return speechRms > 0 && speechRms * VOICE_GAIN.MAX < VOICE_GAIN.TOO_QUIET_RMS;
+}
+
+/**
+ * Soft limiter: samples up to the knee pass as they are, louder ones bend smoothly
+ * towards full scale and never reach it.
+ */
+export function softLimit(sample: number): number {
+ const magnitude = Math.abs(sample);
+
+ if (magnitude <= VOICE_GAIN.KNEE) return sample;
+
+ const room = 1 - VOICE_GAIN.KNEE;
+ const bent = VOICE_GAIN.KNEE + room * Math.tanh((magnitude - VOICE_GAIN.KNEE) / room);
+
+ return Math.sign(sample) * bent;
+}
+
+/** RMS of each frame of `samples`, frames of `frameLength` (the last one may be shorter). */
+function frameLevels(samples: Float32Array, frameLength: number): number[] {
+ const levels: number[] = [];
+
+ for (let start = 0; start < samples.length; start += frameLength) {
+ levels.push(rms(samples.subarray(start, Math.min(samples.length, start + frameLength))));
+ }
+
+ return levels;
+}
+
+/**
+ * The loudness of the speech in a recording: frames that are speech against its noise
+ * floor (its quietest tenth, see isSpeechLevel); their RMS together. 0 when nothing in it
+ * is speech.
+ */
+export function measureSpeech(samples: Float32Array, sampleRate: number): number {
+ const levels = frameLevels(
+ samples,
+ Math.max(1, Math.round((sampleRate * VOICE_GAIN.FRAME_MS) / 1000))
+ );
+
+ if (levels.length === 0) return 0;
+
+ const sorted = [...levels].sort((a, b) => a - b);
+ const floorDb = rmsToDb(sorted[Math.floor((sorted.length - 1) * VOICE_GAIN.FLOOR_QUANTILE)]);
+
+ let power = 0;
+ let count = 0;
+
+ for (const level of levels) {
+ if (!isSpeechLevel(rmsToDb(level), floorDb)) continue;
+
+ power += level * level;
+ count++;
+ }
+
+ return count > 0 ? Math.sqrt(power / count) : 0;
+}
+
+/** Raises a finished recording's speech to the target level in one pass; see SpeechGainResult. */
+export function normalizeSpeech(
+ samples: Int16Array,
+ sampleRate: number,
+ max: number = VOICE_GAIN.MAX
+): SpeechGainResult {
+ const floats = pcm16ToFloat(samples);
+ const speech = measureSpeech(floats, sampleRate);
+ const gain = speechGainFor(speech, max);
+ const tooQuiet = isTooQuiet(speech);
+
+ if (gain <= 1) return { gain: 1, samples, tooQuiet };
+
+ for (let i = 0; i < floats.length; i++) floats[i] = softLimit(floats[i] * gain);
+
+ return { gain, samples: floatToPcm16(floats), tooQuiet };
+}
+
+/**
+ * Speech gain that follows the input as it streams. A SpeechDetector tells which blocks
+ * are speech; a block of speech moves the speech level, and the
+ * gain heads for what that level needs: it falls at once when speech gets louder, so
+ * nothing clips, and rises over a few blocks when it gets quieter. Silence holds the
+ * gain where it is.
+ */
+export class SpeechGain {
+ /** The gain applied to the last block. */
+ gain = 1;
+
+ private readonly detector = new SpeechDetector();
+ private speech = 0;
+
+ /** Whether the speech heard so far stays quiet even at the highest gain. */
+ get tooQuiet(): boolean {
+ return isTooQuiet(this.speech);
+ }
+
+ /** `sampleRate` of the blocks, to follow the room at the same pace whatever their length. */
+ constructor(
+ private readonly max: number = VOICE_GAIN.MAX,
+ private readonly sampleRate: number = VOICE_SAMPLE_RATE
+ ) {}
+
+ /**
+ * Measures `block` and returns it raised by the gain, ramped from the previous block's
+ * gain so the change makes no click. `apply` false only measures, returning the block as is.
+ */
+ process(block: Float32Array, apply = true): Float32Array {
+ if (block.length === 0) return block;
+
+ const level = rms(block);
+ const isSpeech = this.detector.push(rmsToDb(level), (block.length / this.sampleRate) * 1000);
+ const previous = this.gain;
+
+ if (isSpeech) {
+ this.speech =
+ this.speech === 0 ? level : this.speech + (level - this.speech) * VOICE_GAIN.SPEECH_FOLLOW;
+
+ const target = speechGainFor(this.speech, this.max);
+
+ this.gain = target < this.gain ? target : this.gain + (target - this.gain) * VOICE_GAIN.RISE;
+ // a block much louder than the speech so far must not clip waiting for the level to follow
+ this.gain = Math.min(this.gain, Math.max(1, VOICE_GAIN.KNEE / level));
+ }
+
+ if (!apply || (previous === 1 && this.gain === 1)) return block;
+
+ const out = new Float32Array(block.length);
+ const step = (this.gain - previous) / block.length;
+
+ for (let i = 0; i < block.length; i++) out[i] = softLimit(block[i] * (previous + step * i));
+
+ return out;
+ }
+}
diff --git a/ui/lib/voice/src/http-pieces.ts b/ui/lib/voice/src/http-pieces.ts
new file mode 100644
index 000000000..a75fc0095
--- /dev/null
+++ b/ui/lib/voice/src/http-pieces.ts
@@ -0,0 +1,88 @@
+import { concatPcm16 } from './pcm.js';
+import { joinPieces } from './realtime.js';
+import { encodeWav } from './wav.js';
+
+export interface HttpPieceTranscriberOptions {
+ /** Transcribes one piece, e.g. with transcribeClip() */
+ transcribe: (wav: Blob, signal: AbortSignal) => Promise<string>;
+ /** Rate of the PCM chunks given to append() */
+ sampleRate: number;
+ /** The whole text so far, each time a piece adds to it */
+ onText?: (text: string) => void;
+}
+
+/**
+ * Live text without a realtime session: for a model that hears but cannot hold one
+ * (Qwen3-ASR), or a server a browser websocket cannot reach. The recording is cut into
+ * pieces at the speaker's pauses (see cut()), each sent to `/v1/audio/transcriptions`
+ * as a WAV of its own, one at a time, and their text joined in order.
+ */
+export class HttpPieceTranscriber {
+ /** The pieces' text so far */
+ text = '';
+
+ private chunks: Int16Array[] = [];
+ private closed = false;
+ private inFlight: AbortController | null = null;
+ private length = 0;
+ private readonly options: HttpPieceTranscriberOptions;
+
+ /**
+ * Whether a piece can be cut now: audio since the last one, and none still being
+ * transcribed, so pieces come back in order; meanwhile the next one grows.
+ */
+ get canCut(): boolean {
+ return !this.closed && this.inFlight === null && this.length > 0;
+ }
+
+ /** Takes a PCM16 mono chunk at `options.sampleRate`. */
+ append(chunk: Int16Array): void {
+ if (this.closed || chunk.length === 0) return;
+
+ this.chunks.push(chunk);
+ this.length += chunk.length;
+ }
+
+ /** Drops what is pending and stops a piece being transcribed. */
+ close(): void {
+ this.closed = true;
+ this.inFlight?.abort();
+ this.inFlight = null;
+ this.chunks = [];
+ this.length = 0;
+ }
+
+ constructor(options: HttpPieceTranscriberOptions) {
+ this.options = options;
+ }
+
+ /** Ends the current piece and transcribes it; see canCut. */
+ cut(): void {
+ if (!this.canCut) return;
+
+ const samples = concatPcm16(this.chunks);
+
+ this.chunks = [];
+ this.length = 0;
+
+ const controller = new AbortController();
+
+ this.inFlight = controller;
+ this.options
+ .transcribe(encodeWav(samples, this.options.sampleRate), controller.signal)
+ .then(
+ (piece) => {
+ if (controller.signal.aborted || !piece.trim()) return;
+
+ this.text = joinPieces(this.text, piece.trim());
+ this.options.onText?.(this.text);
+ },
+ (error) => {
+ if (!controller.signal.aborted) console.warn('Transcribing a piece failed:', error);
+ }
+ )
+ .finally(() => {
+ if (this.inFlight === controller) this.inFlight = null;
+ });
+ }
+}
diff --git a/ui/lib/voice/src/index.ts b/ui/lib/voice/src/index.ts
new file mode 100644
index 000000000..ee7d93ea2
--- /dev/null
+++ b/ui/lib/voice/src/index.ts
@@ -0,0 +1,61 @@
+/**
+ * Llama Voice: microphone capture with live levels and PCM streaming, clip peaks,
+ * transcription over HTTP or llama-server's realtime socket, and dictation into a text
+ * field. It knows no app: the app passes in how to reach a server (VoiceServerContext).
+ * Plain TypeScript: the Svelte adapters (VoiceRecorder) are in @ggml-org/llama-voice/svelte.
+ */
+
+export { warmUpVoiceCapture } from './audio-context.js';
+export { VoiceCapture } from './capture.js';
+export {
+ VOICE_LEVELS_INTERVAL_MS,
+ VOICE_LEVELS_LENGTH,
+ VOICE_MICROPHONE_CONSTRAINTS,
+ VOICE_MICROPHONE_MESSAGE,
+ VOICE_PEAKS_COUNT,
+ VOICE_SAMPLE_RATE,
+ VOICE_WAV_MIME
+} from './constants.js';
+export {
+ type BeginDictationOptions,
+ createDictation,
+ type Dictation,
+ type DictationOptions,
+ type DictationResult,
+ type DictationTarget
+} from './dictation.js';
+export { DictationSpan } from './dictation-span.js';
+export { RealtimeUnsupportedReason, VoiceCaptureState, VoiceRecorderStatus } from './enums.js';
+export {
+ RealtimeError,
+ RealtimeUnsupportedError,
+ VoiceCaptureError,
+ VoiceTranscriptionError
+} from './errors.js';
+export { SpeechGain } from './gain.js';
+export { levelToDb } from './levels.js';
+export {
+ listMicrophones,
+ type Microphone,
+ microphoneRequest,
+ pickMicrophone,
+ resolveMicrophone,
+ VOICE_MICROPHONE_AUTO,
+ VOICE_MICROPHONE_DEFAULT
+} from './microphone.js';
+export { PauseDetector, type PauseDetectorOptions } from './pauses.js';
+export { base64ToBytes } from './pcm.js';
+export { computePeaks, computePeaksFromAudio } from './peaks.js';
+export { RealtimeTranscriber } from './realtime.js';
+export { startVoiceSession } from './session.js';
+export { SpeechDetector, type SpeechDetectorOptions } from './speech-detector.js';
+export {
+ describeMicrophoneError,
+ isVoiceCaptureSupported,
+ microphoneUnavailableReason
+} from './support.js';
+export { transcribeClip } from './transcribe.js';
+export { cleanTranscript } from './transcript.js';
+export type * from './types.js';
+export { encodeWav } from './wav.js';
+export { MicrophoneWarmup } from './warm-microphone.js';
diff --git a/ui/lib/voice/src/levels.ts b/ui/lib/voice/src/levels.ts
new file mode 100644
index 000000000..03ecf1e2e
--- /dev/null
+++ b/ui/lib/voice/src/levels.ts
@@ -0,0 +1,59 @@
+import { VOICE_LEVELS_FLOOR_DB, VOICE_SILENCE_DB } from './constants.js';
+
+/** Root mean square of a block of samples. */
+export function rms(samples: Float32Array): number {
+ if (samples.length === 0) return 0;
+
+ let sum = 0;
+
+ for (let i = 0; i < samples.length; i++) sum += samples[i] * samples[i];
+
+ return Math.sqrt(sum / samples.length);
+}
+
+/** RMS to dBFS, digital silence as VOICE_SILENCE_DB rather than -Infinity. */
+export function rmsToDb(value: number): number {
+ return value > 0 ? Math.max(VOICE_SILENCE_DB, 20 * Math.log10(value)) : VOICE_SILENCE_DB;
+}
+
+/** A 0..1 level (see rmsToLevel) back to dBFS; 0 reads as the floor. */
+export function levelToDb(level: number, floorDb = VOICE_LEVELS_FLOOR_DB): number {
+ return (1 - Math.max(0, Math.min(1, level))) * floorDb;
+}
+
+/**
+ * RMS to a 0..1 level that follows loudness as heard: decibels mapped linearly from
+ * the floor (0) to full scale (1), so quiet speech still moves the waveform.
+ */
+export function rmsToLevel(value: number, floorDb = VOICE_LEVELS_FLOOR_DB): number {
+ if (value <= 0) return 0;
+
+ const db = 20 * Math.log10(value);
+
+ return Math.max(0, Math.min(1, 1 - db / floorDb));
+}
+
+/** Fixed-length level history, newest last. */
+export class LevelHistory {
+ private values: number[];
+
+ constructor(readonly length: number) {
+ this.values = new Array<number>(length).fill(0);
+ }
+
+ /** Adds a value, drops the oldest, and returns a fresh copy for listeners. */
+ push(level: number): number[] {
+ this.values.push(level);
+ this.values.shift();
+
+ return this.values.slice();
+ }
+
+ reset(): void {
+ this.values.fill(0);
+ }
+
+ snapshot(): number[] {
+ return this.values.slice();
+ }
+}
diff --git a/ui/lib/voice/src/microphone.ts b/ui/lib/voice/src/microphone.ts
new file mode 100644
index 000000000..6825e60a7
--- /dev/null
+++ b/ui/lib/voice/src/microphone.ts
@@ -0,0 +1,123 @@
+/**
+ * Which microphone to record from. `auto` keeps the system's default input, except a
+ * Bluetooth headset's: recording from its microphone switches the headset into its
+ * call mode, which turns what it plays to low-quality mono, so a built-in microphone
+ * is used instead when there is one. `default` always takes the system's input; any
+ * other value is a device id from enumerateDevices().
+ */
+
+import {
+ VOICE_AUDIO_INPUT_KIND,
+ VOICE_BLUETOOTH_LABEL_PATTERN,
+ VOICE_BUILT_IN_LABEL_PATTERN,
+ VOICE_DEVICE_ALIAS,
+ VOICE_MICROPHONE_CONSTRAINTS,
+ VOICE_MICROPHONE_PERMISSION
+} from './constants.js';
+
+export const VOICE_MICROPHONE_AUTO = 'auto';
+export const VOICE_MICROPHONE_DEFAULT = 'default';
+
+// Chrome lists the system default and the communications device as extra entries
+const ALIASES = new Set<string>(Object.values(VOICE_DEVICE_ALIAS));
+
+export interface Microphone {
+ deviceId: string;
+ label: string;
+}
+
+/** The microphones the browser lists; their labels are empty until access is granted. */
+export async function listMicrophones(): Promise<Microphone[]> {
+ if (!navigator.mediaDevices?.enumerateDevices) return [];
+
+ const devices = await navigator.mediaDevices.enumerateDevices();
+
+ return devices
+ .filter((device) => device.kind === VOICE_AUDIO_INPUT_KIND && !ALIASES.has(device.deviceId))
+ .map((device) => ({ deviceId: device.deviceId, label: device.label }));
+}
+
+/**
+ * The device id to record from for a choice, or undefined for the system default.
+ * `devices` is what enumerateDevices() returned.
+ */
+export function pickMicrophone(
+ devices: readonly Pick<MediaDeviceInfo, 'deviceId' | 'kind' | 'label'>[],
+ choice: string
+): string | undefined {
+ const inputs = devices.filter((device) => device.kind === VOICE_AUDIO_INPUT_KIND);
+
+ if (choice === VOICE_MICROPHONE_DEFAULT) return undefined;
+
+ if (choice !== VOICE_MICROPHONE_AUTO) {
+ // a microphone that is gone falls back to the default
+ return inputs.some((device) => device.deviceId === choice) ? choice : undefined;
+ }
+
+ // Chrome names the default entry after the device, e.g. "Default - AirPods (Bluetooth)";
+ // elsewhere the first input is the default
+ const fallback =
+ inputs.find((device) => device.deviceId === VOICE_DEVICE_ALIAS.DEFAULT) ?? inputs[0];
+
+ if (!fallback || !VOICE_BLUETOOTH_LABEL_PATTERN.test(fallback.label)) return undefined;
+
+ return inputs.find(
+ (device) =>
+ !ALIASES.has(device.deviceId) &&
+ VOICE_BUILT_IN_LABEL_PATTERN.test(device.label) &&
+ !VOICE_BLUETOOTH_LABEL_PATTERN.test(device.label)
+ )?.deviceId;
+}
+
+/** The device id for a choice, from the microphones listed now. */
+export async function resolveMicrophone(choice: string): Promise<string | undefined> {
+ if (choice === VOICE_MICROPHONE_DEFAULT || !navigator.mediaDevices?.enumerateDevices) {
+ return undefined;
+ }
+
+ try {
+ return pickMicrophone(await navigator.mediaDevices.enumerateDevices(), choice);
+ } catch {
+ return undefined;
+ }
+}
+
+/** Lets go of a microphone: its tracks stop, and the browser's recording indicator goes. */
+export function stopStream(stream: MediaStream | null | undefined): void {
+ for (const track of stream?.getTracks() ?? []) track.stop();
+}
+
+/**
+ * What a recording asks the browser for: the raw signal (VOICE_MICROPHONE_CONSTRAINTS)
+ * with the caller's processing on top, from `deviceId` when one is picked. The same
+ * options always give the same request, so a microphone opened ahead (MicrophoneWarmup)
+ * can be told apart from one the recording would ask for differently.
+ */
+export function microphoneRequest(options: {
+ constraints?: MediaTrackConstraints;
+ deviceId?: string;
+}): MediaStreamConstraints {
+ return {
+ audio: {
+ ...VOICE_MICROPHONE_CONSTRAINTS,
+ ...(options.deviceId ? { deviceId: options.deviceId } : {}),
+ ...options.constraints
+ }
+ };
+}
+
+/**
+ * Whether the page may use the microphone without asking: access was granted before.
+ * False where the browser can't tell (no Permissions API, or no `microphone` name in it).
+ */
+export async function microphoneAllowed(): Promise<boolean> {
+ try {
+ const status = await navigator.permissions?.query({
+ name: VOICE_MICROPHONE_PERMISSION.NAME as PermissionName
+ });
+
+ return status?.state === VOICE_MICROPHONE_PERMISSION.GRANTED;
+ } catch {
+ return false;
+ }
+}
diff --git a/ui/lib/voice/src/pauses.ts b/ui/lib/voice/src/pauses.ts
new file mode 100644
index 000000000..c036910e1
--- /dev/null
+++ b/ui/lib/voice/src/pauses.ts
@@ -0,0 +1,86 @@
+import { VOICE_PAUSE_DEFAULTS } from './constants.js';
+import { levelToDb } from './levels.js';
+import { SpeechDetector } from './speech-detector.js';
+
+/**
+ * Finds the pauses in speech from the live input levels (0..1, see rmsToLevel), so a
+ * recording can be transcribed piece by piece while it goes on, cut where the speaker
+ * breathes rather than inside a word. A SpeechDetector tells speech, and quiet, from the
+ * room's noise.
+ */
+export interface PauseDetectorOptions {
+ /** Time between two levels, as the capture sends them */
+ intervalMs: number;
+ /** Silence after speech that makes a pause */
+ pauseMs?: number;
+ /** Speech needed before a pause counts */
+ minSpeechMs?: number;
+ /**
+ * A piece longer than this is cut at its next quiet moment, pause or not; half as
+ * long again and it is cut where it is, as with music or talk in the room no moment
+ * is ever quiet
+ */
+ maxSegmentMs?: number;
+}
+
+export class PauseDetector {
+ private readonly detector = new SpeechDetector();
+ private readonly intervalMs: number;
+ private readonly maxSegmentMs: number;
+ private readonly minSpeechMs: number;
+ private readonly pauseMs: number;
+ private quietMs = 0;
+ private segmentMs = 0;
+ private speechMs = 0;
+
+ /** Whether speech was heard since the last cut. */
+ get heardSpeech(): boolean {
+ return this.speechMs > 0;
+ }
+
+ constructor(options: PauseDetectorOptions) {
+ this.intervalMs = options.intervalMs;
+ this.pauseMs = options.pauseMs ?? VOICE_PAUSE_DEFAULTS.PAUSE_MS;
+ this.minSpeechMs = options.minSpeechMs ?? VOICE_PAUSE_DEFAULTS.MIN_SPEECH_MS;
+ this.maxSegmentMs = options.maxSegmentMs ?? VOICE_PAUSE_DEFAULTS.MAX_SEGMENT_MS;
+ }
+
+ /** Takes the newest level; true when this is a place to cut the recording. */
+ push(level: number): boolean {
+ const db = levelToDb(level);
+ const loud = this.detector.push(db, this.intervalMs);
+
+ this.segmentMs += this.intervalMs;
+
+ if (loud) {
+ this.speechMs += this.intervalMs;
+ this.quietMs = 0;
+ }
+
+ const spoke = this.speechMs >= this.minSpeechMs;
+
+ // no quiet moment came: the text must not wait for one that a noisy room never has
+ if (spoke && this.segmentMs >= this.maxSegmentMs * VOICE_PAUSE_DEFAULTS.OVERTIME) {
+ this.reset();
+
+ return true;
+ }
+
+ if (loud || !this.detector.isQuiet(db)) return false;
+
+ this.quietMs += this.intervalMs;
+
+ const cut = spoke && (this.quietMs >= this.pauseMs || this.segmentMs >= this.maxSegmentMs);
+
+ if (cut) this.reset();
+
+ return cut;
+ }
+
+ /** Starts a new piece, e.g. after one was cut elsewhere. */
+ reset(): void {
+ this.quietMs = 0;
+ this.segmentMs = 0;
+ this.speechMs = 0;
+ }
+}
diff --git a/ui/lib/voice/src/pcm-tap.ts b/ui/lib/voice/src/pcm-tap.ts
new file mode 100644
index 000000000..37922c6b5
--- /dev/null
+++ b/ui/lib/voice/src/pcm-tap.ts
@@ -0,0 +1,166 @@
+import {
+ VOICE_ERROR_MESSAGE,
+ VOICE_SCRIPT_PROCESSOR_FRAMES,
+ VOICE_WORKLET_BATCH_FRAMES,
+ VOICE_WORKLET_FLUSH_TIMEOUT_MS,
+ VOICE_WORKLET_MESSAGE,
+ VOICE_WORKLET_PROCESSOR,
+ VOICE_WORKLET_SOURCE_MIME
+} from './constants.js';
+
+// runs on the audio thread; inline so no extra file has to be served next to the app
+const WORKLET_SOURCE = `
+class PcmTap extends AudioWorkletProcessor {
+ constructor(options) {
+ super();
+ this.size = options.processorOptions.batchFrames;
+ this.buffer = new Float32Array(this.size);
+ this.filled = 0;
+ this.port.onmessage = (event) => {
+ if (event.data !== '${VOICE_WORKLET_MESSAGE.FLUSH}') return;
+ this.post();
+ this.port.postMessage('${VOICE_WORKLET_MESSAGE.FLUSHED}');
+ };
+ }
+
+ post() {
+ if (this.filled === 0) return;
+ const block = this.buffer.slice(0, this.filled);
+ this.port.postMessage(block, [block.buffer]);
+ this.filled = 0;
+ }
+
+ process(inputs) {
+ const channel = inputs[0] && inputs[0][0];
+ if (!channel) return true;
+ let offset = 0;
+ while (offset < channel.length) {
+ const n = Math.min(channel.length - offset, this.size - this.filled);
+ this.buffer.set(channel.subarray(offset, offset + n), this.filled);
+ this.filled += n;
+ offset += n;
+ if (this.filled === this.size) this.post();
+ }
+ return true;
+ }
+}
+registerProcessor('${VOICE_WORKLET_PROCESSOR}', PcmTap);
+`;
+
+/** A node that hands mono float blocks of its input to the main thread. */
+export interface PcmTap {
+ node: AudioNode;
+ /** Posts what is still buffered, then resolves */
+ flush(): Promise<void>;
+ disconnect(): void;
+}
+
+/**
+ * Taps mono float samples from the graph: an AudioWorklet loaded from a Blob URL
+ * (fine under COOP/COEP, a blob inherits the page's origin), or a ScriptProcessor
+ * where AudioWorklet is missing.
+ */
+export async function createPcmTap(
+ context: AudioContext,
+ onBlock: (block: Float32Array) => void
+): Promise<PcmTap> {
+ if (context.audioWorklet && typeof AudioWorkletNode !== 'undefined') {
+ try {
+ return await createWorkletTap(context, onBlock);
+ } catch (error) {
+ console.warn('AudioWorklet unavailable, using ScriptProcessor:', error);
+ }
+ }
+
+ return createScriptProcessorTap(context, onBlock);
+}
+
+// a context loads the worklet once, however many recordings it serves
+const modules = new WeakMap<BaseAudioContext, Promise<void>>();
+
+/** Loads the tap's worklet into `context`, once; later calls wait for the same load. */
+export function loadPcmTapModule(context: BaseAudioContext): Promise<void> {
+ let loading = modules.get(context);
+
+ if (!loading) {
+ if (!context.audioWorklet)
+ return Promise.reject(new Error(VOICE_ERROR_MESSAGE.workletUnavailable));
+
+ const url = URL.createObjectURL(
+ new Blob([WORKLET_SOURCE], { type: VOICE_WORKLET_SOURCE_MIME })
+ );
+
+ loading = context.audioWorklet.addModule(url).finally(() => URL.revokeObjectURL(url));
+ // a failed load may be tried again
+ loading.catch(() => modules.delete(context));
+ modules.set(context, loading);
+ }
+
+ return loading;
+}
+
+async function createWorkletTap(
+ context: AudioContext,
+ onBlock: (block: Float32Array) => void
+): Promise<PcmTap> {
+ await loadPcmTapModule(context);
+
+ const node = new AudioWorkletNode(context, VOICE_WORKLET_PROCESSOR, {
+ channelCount: 1,
+ channelCountMode: 'explicit',
+ numberOfInputs: 1,
+ numberOfOutputs: 1,
+ processorOptions: { batchFrames: VOICE_WORKLET_BATCH_FRAMES }
+ });
+
+ let flushed: (() => void) | null = null;
+
+ node.port.onmessage = (event: MessageEvent<Float32Array | string>) => {
+ if (event.data === VOICE_WORKLET_MESSAGE.FLUSHED) {
+ flushed?.();
+ } else if (event.data instanceof Float32Array) {
+ onBlock(event.data);
+ }
+ };
+
+ return {
+ disconnect: () => {
+ node.port.onmessage = null;
+ node.port.close();
+ node.disconnect();
+ },
+ flush: () =>
+ new Promise<void>((resolve) => {
+ const timer = setTimeout(resolve, VOICE_WORKLET_FLUSH_TIMEOUT_MS);
+
+ flushed = () => {
+ clearTimeout(timer);
+ flushed = null;
+ resolve();
+ };
+ node.port.postMessage(VOICE_WORKLET_MESSAGE.FLUSH);
+ }),
+ node
+ };
+}
+
+function createScriptProcessorTap(
+ context: AudioContext,
+ onBlock: (block: Float32Array) => void
+): PcmTap {
+ const node = context.createScriptProcessor(VOICE_SCRIPT_PROCESSOR_FRAMES, 1, 1);
+
+ node.onaudioprocess = (event) => {
+ onBlock(event.inputBuffer.getChannelData(0).slice());
+ };
+
+ return {
+ disconnect: () => {
+ node.onaudioprocess = null;
+ node.disconnect();
+ },
+ // a ScriptProcessor holds nothing back: each block is posted as soon as it is full
+ flush: () => Promise.resolve(),
+ node
+ };
+}
diff --git a/ui/lib/voice/src/pcm.ts b/ui/lib/voice/src/pcm.ts
new file mode 100644
index 000000000..0030491c4
--- /dev/null
+++ b/ui/lib/voice/src/pcm.ts
@@ -0,0 +1,140 @@
+import { VOICE_BASE64_SLICE } from './constants.js';
+
+/** Float samples (-1..1) to PCM16, clipped. */
+export function floatToPcm16(samples: Float32Array): Int16Array {
+ const out = new Int16Array(samples.length);
+
+ for (let i = 0; i < samples.length; i++) {
+ const s = Math.max(-1, Math.min(1, samples[i]));
+
+ out[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
+ }
+
+ return out;
+}
+
+/** PCM16 samples back to floats (-1..1). */
+export function pcm16ToFloat(samples: Int16Array): Float32Array {
+ const out = new Float32Array(samples.length);
+
+ for (let i = 0; i < samples.length; i++) {
+ out[i] = samples[i] / 0x8000;
+ }
+
+ return out;
+}
+
+/** Joins PCM16 chunks into one array. */
+export function concatPcm16(chunks: readonly Int16Array[]): Int16Array {
+ let length = 0;
+
+ for (const chunk of chunks) length += chunk.length;
+
+ const out = new Int16Array(length);
+
+ let offset = 0;
+
+ for (const chunk of chunks) {
+ out.set(chunk, offset);
+ offset += chunk.length;
+ }
+
+ return out;
+}
+
+/** Bytes to base64, in slices so long inputs do not overflow the call stack. */
+export function bytesToBase64(bytes: Uint8Array): string {
+ let binary = '';
+
+ for (let i = 0; i < bytes.length; i += VOICE_BASE64_SLICE) {
+ binary += String.fromCharCode(...bytes.subarray(i, i + VOICE_BASE64_SLICE));
+ }
+
+ return btoa(binary);
+}
+
+/** Base64 to bytes. */
+export function base64ToBytes(base64: string): Uint8Array<ArrayBuffer> {
+ const binary = atob(base64);
+ const bytes = new Uint8Array(binary.length);
+
+ for (let i = 0; i < binary.length; i++) {
+ bytes[i] = binary.charCodeAt(i);
+ }
+
+ return bytes;
+}
+
+/** PCM16 samples as base64, little-endian as the realtime API expects. */
+export function pcm16ToBase64(samples: Int16Array): string {
+ // typed arrays use the platform byte order, which is little-endian on every browser target
+ return bytesToBase64(new Uint8Array(samples.buffer, samples.byteOffset, samples.byteLength));
+}
+
+/** Streaming resampler: feed blocks of any size, get the samples ready so far. */
+export interface Resampler {
+ process(input: Float32Array): Float32Array;
+}
+
+/**
+ * Streaming mono resampler. Going down, each output sample is a triangle-weighted mean
+ * of the input around it, two output periods wide: a cheap low-pass that keeps most
+ * aliasing out of speech. Going up, it interpolates linearly. Same rates pass through.
+ */
+export function createResampler(fromRate: number, toRate: number): Resampler {
+ if (fromRate === toRate) {
+ return { process: (input) => input };
+ }
+
+ const ratio = fromRate / toRate;
+ // input reach on each side of an output sample
+ const reach = Math.max(1, ratio);
+
+ // input samples kept for the next block, and the position of the next output in them
+ let pending = new Float32Array(0);
+ let position = 0;
+
+ return {
+ process(input: Float32Array): Float32Array {
+ const buffer = new Float32Array(pending.length + input.length);
+
+ buffer.set(pending);
+ buffer.set(input, pending.length);
+
+ const out: number[] = [];
+
+ if (ratio > 1) {
+ while (position + reach < buffer.length) {
+ let sum = 0;
+ let weight = 0;
+
+ for (let i = Math.max(0, Math.ceil(position - reach)); i <= position + reach; i++) {
+ const w = 1 - Math.abs(i - position) / reach;
+
+ sum += buffer[i] * w;
+ weight += w;
+ }
+
+ out.push(weight > 0 ? sum / weight : 0);
+ position += ratio;
+ }
+ } else {
+ while (position + 1 < buffer.length) {
+ const i = Math.floor(position);
+ const t = position - i;
+
+ out.push(buffer[i] * (1 - t) + buffer[i + 1] * t);
+ position += ratio;
+ }
+ }
+
+ // keep what the next output still reaches back to
+ const keep = Math.max(0, Math.floor(position - (ratio > 1 ? reach : 0)));
+
+ pending = buffer.slice(keep);
+ position -= keep;
+
+ return Float32Array.from(out);
+ }
+ };
+}
diff --git a/ui/lib/voice/src/peaks.ts b/ui/lib/voice/src/peaks.ts
new file mode 100644
index 000000000..fb3c0ca41
--- /dev/null
+++ b/ui/lib/voice/src/peaks.ts
@@ -0,0 +1,79 @@
+import {
+ VOICE_DATA_URL_HEADER_PATTERN,
+ VOICE_DECODE_SAMPLE_RATE,
+ VOICE_PEAKS_COUNT
+} from './constants.js';
+import { base64ToBytes } from './pcm.js';
+
+/** Mono samples of a buffer: the mean of its channels. */
+export function mixToMono(buffer: AudioBuffer): Float32Array {
+ if (buffer.numberOfChannels === 1) return buffer.getChannelData(0);
+
+ const out = new Float32Array(buffer.length);
+
+ for (let c = 0; c < buffer.numberOfChannels; c++) {
+ const channel = buffer.getChannelData(c);
+
+ for (let i = 0; i < out.length; i++) out[i] += channel[i] / buffer.numberOfChannels;
+ }
+
+ return out;
+}
+
+/**
+ * Peaks of a clip for a static waveform: the loudest sample of each of `count`
+ * equal spans, scaled so the loudest span is 1. Silence gives all zeros.
+ */
+export function computePeaks(
+ source: Float32Array | Int16Array | AudioBuffer,
+ count = VOICE_PEAKS_COUNT
+): number[] {
+ const samples =
+ source instanceof Float32Array || source instanceof Int16Array ? source : mixToMono(source);
+ const scale = samples instanceof Int16Array ? 1 / 0x8000 : 1;
+ const peaks = new Array<number>(count).fill(0);
+
+ if (samples.length === 0 || count <= 0) return peaks;
+
+ for (let p = 0; p < count; p++) {
+ const start = Math.floor((p * samples.length) / count);
+ // a clip shorter than `count` repeats samples rather than leaving gaps
+ const end = Math.max(start + 1, Math.floor(((p + 1) * samples.length) / count));
+
+ let max = 0;
+
+ for (let i = start; i < end && i < samples.length; i++) {
+ const value = Math.abs(samples[i] * scale);
+
+ if (value > max) max = value;
+ }
+
+ peaks[p] = max;
+ }
+
+ const loudest = Math.max(...peaks);
+
+ return loudest > 0 ? peaks.map((value) => value / loudest) : peaks;
+}
+
+/** Decodes an audio file (Blob, bytes, or base64 such as a stored attachment) to a buffer. */
+export async function decodeAudio(source: Blob | ArrayBuffer | string): Promise<AudioBuffer> {
+ const bytes =
+ typeof source === 'string'
+ ? base64ToBytes(source.replace(VOICE_DATA_URL_HEADER_PATTERN, '')).buffer
+ : source instanceof ArrayBuffer
+ ? source
+ : await source.arrayBuffer();
+ // an offline context decodes without a running audio device; the rate is a placeholder
+ const context = new OfflineAudioContext(1, 1, VOICE_DECODE_SAMPLE_RATE);
+
+ return context.decodeAudioData(bytes as ArrayBuffer);
+}
+
+/** Peaks of an encoded audio file, for stored clips that carry none. */
+export async function computePeaksFromAudio(
+ source: Blob | ArrayBuffer | string,
+ count = VOICE_PEAKS_COUNT
+): Promise<number[]> {
+ return computePeaks(await decodeAudio(source), count);
+}
diff --git a/ui/lib/voice/src/realtime.ts b/ui/lib/voice/src/realtime.ts
new file mode 100644
index 000000000..ec774e00b
--- /dev/null
+++ b/ui/lib/voice/src/realtime.ts
@@ -0,0 +1,474 @@
+import {
+ VOICE_API_PATHS,
+ VOICE_ERROR_MESSAGE,
+ VOICE_LEADING_SPACE_PATTERN,
+ VOICE_REALTIME_AUDIO_FORMAT,
+ VOICE_REALTIME_EVENTS,
+ VOICE_REALTIME_QUERY,
+ VOICE_REALTIME_SESSION_TRANSCRIPTION,
+ VOICE_REALTIME_TIMEOUTS,
+ VOICE_REALTIME_UNSUPPORTED_PATTERN,
+ VOICE_SOCKET_PROTOCOL,
+ VOICE_TRAILING_SPACE_PATTERN
+} from './constants.js';
+import { RealtimeState, RealtimeUnsupportedReason } from './enums.js';
+import { RealtimeError, RealtimeUnsupportedError } from './errors.js';
+import { pcm16ToBase64 } from './pcm.js';
+import { cleanTranscript } from './transcript.js';
+import type { RealtimeTranscriberOptions, VoiceServerContext } from './types.js';
+
+interface Pending<T> {
+ resolve: (value: T) => void;
+ reject: (error: Error) => void;
+}
+
+/** A server event, only the fields this client reads. */
+interface RealtimeServerEvent {
+ type?: string;
+ delta?: unknown;
+ transcript?: unknown;
+ error?: { message?: unknown };
+ session?: {
+ type?: unknown;
+ audio?: { input?: { format?: { rate?: unknown } } };
+ };
+}
+
+/**
+ * URL of the realtime socket for a server: the same target as its HTTP routes, with
+ * ws(s) for http(s), `intent=transcription`, and the model for a router.
+ * No API key goes in it: llama-server only reads the key from headers.
+ */
+export function realtimeUrl(
+ context: VoiceServerContext,
+ options: Pick<RealtimeTranscriberOptions, 'autoload' | 'model'> = {}
+): string {
+ const url = new URL(
+ context.resolveUrl(VOICE_API_PATHS.REALTIME),
+ context.baseHref ?? globalThis.location?.href
+ );
+
+ url.protocol =
+ url.protocol === VOICE_SOCKET_PROTOCOL.HTTPS
+ ? VOICE_SOCKET_PROTOCOL.WSS
+ : VOICE_SOCKET_PROTOCOL.WS;
+ url.searchParams.set(VOICE_REALTIME_QUERY.INTENT, VOICE_REALTIME_QUERY.INTENT_TRANSCRIPTION);
+
+ if (options.model) url.searchParams.set(VOICE_REALTIME_QUERY.MODEL, options.model);
+
+ if (options.autoload !== undefined) {
+ url.searchParams.set(VOICE_REALTIME_QUERY.AUTOLOAD, String(options.autoload));
+ }
+
+ return url.toString();
+}
+
+/**
+ * Live transcription over llama-server's `/v1/realtime` socket. Audio given to
+ * `append` before the session starts is queued and sent once it does.
+ *
+ * A model with no streaming audio encoder gets an `error` event and the socket
+ * closes: `connect()` then rejects with RealtimeUnsupportedError and the caller
+ * transcribes the clip over HTTP instead. A model that holds realtime conversations
+ * is switched to a transcription session, which transcribes each committed piece of
+ * audio: commitSegment() at the speaker's pauses gives its text while they go on, and
+ * the final commit the rest. One piece is transcribed at a time.
+ */
+export class RealtimeTranscriber {
+ /** True when the session streams text while the audio comes in */
+ live = false;
+ /** Input rate of the model, from session.created */
+ modelSampleRate: number | null = null;
+
+ // audio sent since the last commit: an empty commit is an error on the server
+ private audioSinceCommit = false;
+ private completedText = '';
+ private connecting: Promise<void> | null = null;
+ private readonly context: VoiceServerContext;
+ private finalSent = false;
+ private idleWaiters: (() => void)[] = [];
+
+ // commits whose transcript has not completed yet, and who waits for them
+ private inFlight = 0;
+ private itemText = '';
+ private readonly options: RealtimeTranscriberOptions;
+ private pendingConnect: Pending<void> | null = null;
+ private pendingFinish: Pending<string> | null = null;
+ private pieces = 0;
+ private queue: string[] = [];
+ private sentAudio = false;
+ private socket: WebSocket | null = null;
+ private state: RealtimeState = RealtimeState.IDLE;
+ private timer: ReturnType<typeof setTimeout> | null = null;
+
+ /**
+ * Whether a piece can be committed now: a session that transcribes per commit, with
+ * audio since the last commit and no piece still being transcribed.
+ */
+ get canCommitSegment(): boolean {
+ return (
+ this.state === RealtimeState.OPEN &&
+ !this.live &&
+ this.audioSinceCommit &&
+ this.inFlight === 0
+ );
+ }
+
+ /** Pieces committed with commitSegment(), the final commit not counted. */
+ get piecesCommitted(): number {
+ return this.pieces;
+ }
+
+ /** True between session.created and finish() or close(). */
+ get ready(): boolean {
+ return this.state === RealtimeState.OPEN;
+ }
+
+ /**
+ * The whole transcript so far, live text of the current item included, without the
+ * format a model wraps its text in (see cleanTranscript).
+ */
+ get text(): string {
+ const item = cleanTranscript(this.itemText);
+
+ // a model's own format carries no spacing between items; plain text streams with its own
+ return item === this.itemText
+ ? this.completedText + item
+ : joinPieces(this.completedText, item);
+ }
+
+ /** Streams a PCM16 mono chunk at `options.sampleRate`. */
+ append(chunk: Int16Array): void {
+ if (
+ chunk.length === 0 ||
+ this.state === RealtimeState.FINISHING ||
+ this.state === RealtimeState.CLOSED
+ )
+ return;
+
+ const message = JSON.stringify({
+ audio: pcm16ToBase64(chunk),
+ type: VOICE_REALTIME_EVENTS.APPEND
+ });
+
+ this.sentAudio = true;
+ this.audioSinceCommit = true;
+
+ if (this.state === RealtimeState.OPEN) {
+ this.socket?.send(message);
+ } else {
+ this.queue.push(message);
+ }
+ }
+
+ /** Closes the socket; pending connect or finish calls reject. */
+ close(): void {
+ if (this.state === RealtimeState.CLOSED) return;
+
+ this.fail(new RealtimeError(VOICE_ERROR_MESSAGE.realtimeClosed));
+ }
+
+ /** Commits the audio so far as a piece to transcribe now; see canCommitSegment. */
+ commitSegment(): void {
+ if (!this.canCommitSegment) return;
+
+ this.inFlight++;
+ this.pieces++;
+ this.audioSinceCommit = false;
+ this.send({ type: VOICE_REALTIME_EVENTS.COMMIT });
+ }
+
+ /**
+ * Opens the socket and resolves once the session is created.
+ * @throws RealtimeUnsupportedError when the model cannot stream or the socket does not open
+ */
+ connect(): Promise<void> {
+ if (this.connecting) return this.connecting;
+
+ this.connecting = new Promise<void>((resolve, reject) => {
+ this.pendingConnect = { reject, resolve };
+ });
+
+ this.state = RealtimeState.CONNECTING;
+
+ let socket: WebSocket;
+
+ try {
+ socket = new WebSocket(realtimeUrl(this.context, this.options));
+ } catch (error) {
+ this.fail(
+ new RealtimeUnsupportedError(
+ VOICE_ERROR_MESSAGE.realtimeSocketFailed,
+ RealtimeUnsupportedReason.CONNECTION,
+ { cause: error }
+ )
+ );
+
+ return this.connecting;
+ }
+
+ this.socket = socket;
+ socket.onmessage = (event: MessageEvent) => this.handleMessage(event.data);
+ socket.onclose = (event: CloseEvent) => this.handleClose(event);
+ this.startTimer(
+ this.options.connectTimeoutMs ?? VOICE_REALTIME_TIMEOUTS.CONNECT_MS,
+ VOICE_ERROR_MESSAGE.realtimeConnectTimeout
+ );
+
+ return this.connecting;
+ }
+
+ constructor(context: VoiceServerContext, options: RealtimeTranscriberOptions) {
+ this.context = context;
+ this.options = options;
+ }
+
+ /**
+ * Ends the input (`input_audio_buffer.commit` with `final: true`) and resolves with
+ * the whole transcript once the last item completes. With `commitRest: false`, a
+ * session that transcribes per commit leaves out the audio after the last piece,
+ * e.g. when it is silence, which such a model may hear a word in.
+ * @throws RealtimeError when the session fails, closes or times out first
+ */
+ async finish(options: { commitRest?: boolean } = {}): Promise<string> {
+ if (this.state === RealtimeState.CONNECTING) await this.connecting;
+
+ if (this.state !== RealtimeState.OPEN)
+ throw new RealtimeError(VOICE_ERROR_MESSAGE.realtimeNotOpen);
+
+ if (!this.sentAudio) {
+ this.close();
+
+ return '';
+ }
+
+ this.state = RealtimeState.FINISHING;
+
+ const done = new Promise<string>((resolve, reject) => {
+ this.pendingFinish = { reject, resolve };
+ });
+
+ this.startTimer(
+ this.options.finishTimeoutMs ?? VOICE_REALTIME_TIMEOUTS.FINISH_MS,
+ VOICE_ERROR_MESSAGE.realtimeFinishTimeout
+ );
+
+ // a piece still being transcribed finishes first: the server follows one at a time
+ while (this.inFlight > 0 && this.state === RealtimeState.FINISHING) {
+ await new Promise<void>((resolve) => this.idleWaiters.push(resolve));
+ }
+
+ if (this.state !== RealtimeState.FINISHING) return done;
+
+ const commitRest = this.live || options.commitRest !== false || this.completedText === '';
+
+ if (this.audioSinceCommit && commitRest) {
+ this.finalSent = true;
+ this.inFlight++;
+ this.audioSinceCommit = false;
+ this.send({ final: true, type: VOICE_REALTIME_EVENTS.COMMIT });
+ } else {
+ // everything was committed in pieces already, or what is left holds no speech
+ this.resolveFinish();
+ }
+
+ return done;
+ }
+
+ private clearTimer(): void {
+ if (this.timer !== null) {
+ clearTimeout(this.timer);
+ this.timer = null;
+ }
+ }
+
+ /** Rejects whatever is pending with `error` and releases the socket. */
+ private fail(error: Error): void {
+ this.state = RealtimeState.CLOSED;
+ this.clearTimer();
+
+ for (const wake of this.idleWaiters.splice(0)) wake();
+
+ this.queue = [];
+ this.pendingConnect?.reject(error);
+ this.pendingConnect = null;
+ this.pendingFinish?.reject(error);
+ this.pendingFinish = null;
+
+ const socket = this.socket;
+
+ this.socket = null;
+
+ if (socket) {
+ socket.onmessage = null;
+ socket.onclose = null;
+ socket.close();
+ }
+ }
+
+ private handleClose(event: CloseEvent): void {
+ if (this.state === RealtimeState.CLOSED) return;
+
+ // a refused upgrade (e.g. 401 from a server with an API key) closes before any session
+ const error =
+ this.state === RealtimeState.CONNECTING
+ ? new RealtimeUnsupportedError(
+ VOICE_ERROR_MESSAGE.realtimeRefused(event.code),
+ RealtimeUnsupportedReason.CONNECTION
+ )
+ : new RealtimeError(VOICE_ERROR_MESSAGE.realtimeClosedEarly);
+
+ if (this.state === RealtimeState.OPEN) this.options.onError?.(error);
+
+ this.fail(error);
+ }
+
+ private handleCompleted(transcript: string): void {
+ this.completedText = joinPieces(this.completedText, cleanTranscript(transcript));
+ this.itemText = '';
+ this.inFlight = Math.max(0, this.inFlight - 1);
+ this.options.onCompleted?.(this.text);
+
+ if (this.inFlight === 0) {
+ for (const wake of this.idleWaiters.splice(0)) wake();
+ }
+
+ // a live session completes once, on its final commit; a per-commit one once the last piece is in
+ if (
+ this.state === RealtimeState.FINISHING &&
+ (this.live || (this.finalSent && this.inFlight === 0))
+ ) {
+ this.resolveFinish();
+ }
+ }
+
+ private handleError(message: string): void {
+ if (this.state === RealtimeState.CONNECTING) {
+ this.fail(
+ VOICE_REALTIME_UNSUPPORTED_PATTERN.test(message)
+ ? new RealtimeUnsupportedError(message, RealtimeUnsupportedReason.MODEL)
+ : new RealtimeError(message)
+ );
+
+ return;
+ }
+
+ const error = new RealtimeError(message);
+
+ // a piece that failed gets no transcript: commit the next one, and let a finish go on
+ if (this.inFlight > 0) {
+ this.inFlight = 0;
+
+ for (const wake of this.idleWaiters.splice(0)) wake();
+ }
+
+ this.options.onError?.(error);
+
+ if (this.state === RealtimeState.FINISHING) this.fail(error);
+ }
+
+ private handleMessage(data: unknown): void {
+ if (typeof data !== 'string') return;
+
+ let event: RealtimeServerEvent;
+
+ try {
+ event = JSON.parse(data);
+ } catch {
+ return;
+ }
+
+ switch (event.type) {
+ case VOICE_REALTIME_EVENTS.SESSION_CREATED:
+ this.handleSessionCreated(event);
+
+ break;
+ case VOICE_REALTIME_EVENTS.DELTA:
+ if (typeof event.delta === 'string' && event.delta) {
+ this.itemText += event.delta;
+ this.options.onDelta?.(event.delta, this.text);
+ }
+
+ break;
+ case VOICE_REALTIME_EVENTS.COMPLETED:
+ this.handleCompleted(
+ typeof event.transcript === 'string' ? event.transcript : this.itemText
+ );
+
+ break;
+ case VOICE_REALTIME_EVENTS.ERROR:
+ this.handleError(
+ typeof event.error?.message === 'string'
+ ? event.error.message
+ : VOICE_ERROR_MESSAGE.realtimeFailed
+ );
+
+ break;
+ }
+ }
+
+ private handleSessionCreated(event: RealtimeServerEvent): void {
+ if (this.state !== RealtimeState.CONNECTING) return;
+
+ const rate = event.session?.audio?.input?.format?.rate;
+
+ this.modelSampleRate = typeof rate === 'number' ? rate : null;
+ this.live = event.session?.type === VOICE_REALTIME_SESSION_TRANSCRIPTION;
+ this.clearTimer();
+
+ // the rate of our chunks, and a transcription session for a model that would hold a conversation
+ this.send({
+ session: {
+ audio: {
+ input: {
+ format: {
+ rate: this.options.sampleRate,
+ type: VOICE_REALTIME_AUDIO_FORMAT
+ },
+ ...(this.options.language ? { transcription: { language: this.options.language } } : {})
+ }
+ },
+ type: VOICE_REALTIME_SESSION_TRANSCRIPTION
+ },
+ type: VOICE_REALTIME_EVENTS.SESSION_UPDATE
+ });
+
+ this.state = RealtimeState.OPEN;
+
+ for (const message of this.queue) this.socket?.send(message);
+
+ this.queue = [];
+ this.pendingConnect?.resolve();
+ this.pendingConnect = null;
+ }
+
+ private resolveFinish(): void {
+ if (!this.pendingFinish) return;
+
+ const text = this.text.trim();
+
+ this.pendingFinish.resolve(text);
+ this.pendingFinish = null;
+ this.close();
+ }
+
+ private send(event: object): void {
+ this.socket?.send(JSON.stringify(event));
+ }
+
+ private startTimer(ms: number, message: string): void {
+ this.clearTimer();
+ this.timer = setTimeout(() => this.fail(new RealtimeError(message)), ms);
+ }
+}
+
+/** Two transcribed pieces as one text, a space between them where neither has one. */
+export function joinPieces(before: string, piece: string): string {
+ if (!before) return piece;
+
+ if (!piece) return before;
+
+ return VOICE_TRAILING_SPACE_PATTERN.test(before) || VOICE_LEADING_SPACE_PATTERN.test(piece)
+ ? before + piece
+ : `${before} ${piece}`;
+}
diff --git a/ui/lib/voice/src/session.ts b/ui/lib/voice/src/session.ts
new file mode 100644
index 000000000..1d8d9624c
--- /dev/null
+++ b/ui/lib/voice/src/session.ts
@@ -0,0 +1,245 @@
+import { VoiceCapture } from './capture.js';
+import { VOICE_CLIP_FILENAME, VOICE_PIECE, VOICE_RETRANSCRIBE_MAX_MS } from './constants.js';
+import { RealtimeUnsupportedError } from './errors.js';
+import { HttpPieceTranscriber } from './http-pieces.js';
+import { PauseDetector } from './pauses.js';
+import { RealtimeTranscriber } from './realtime.js';
+import { transcribeClip } from './transcribe.js';
+import type { VoiceSession, VoiceSessionOptions, VoiceSessionResult } from './types.js';
+
+/**
+ * Starts capturing and, when asked to transcribe, tries the realtime socket at the
+ * same time. Realtime gives live text: a live ASR model streams it, any other model
+ * that hears transcribes the recording piece by piece, cut at the speaker's pauses.
+ * When the model cannot hold a realtime session (Qwen3-ASR) or the socket does not
+ * open, the pieces go over HTTP instead, one request each, so the text is live too.
+ * Either way, a recording cut into pieces gets one more pass over the whole clip
+ * at the end, punctuated by meaning rather than by the pauses.
+ * @throws VoiceCaptureError when the microphone does not start
+ */
+export async function startVoiceSession(options: VoiceSessionOptions = {}): Promise<VoiceSession> {
+ const capture = isCapture(options.capture) ? options.capture : new VoiceCapture(options.capture);
+ const unsubscribers: (() => void)[] = [];
+
+ if (options.onLevels) unsubscribers.push(capture.onLevels(options.onLevels));
+
+ if (options.onQuiet) unsubscribers.push(capture.onQuiet(options.onQuiet));
+
+ try {
+ await capture.start();
+ } catch (error) {
+ for (const unsubscribe of unsubscribers) unsubscribe();
+
+ throw error;
+ }
+
+ const transcribe = options.transcribe;
+
+ let transcriber: RealtimeTranscriber | null = null;
+ // speech heard since the last piece was committed; until pieces are cut, assume so
+ let speechSinceCommit = true;
+ let liveText: string | null = null;
+ let ended = false;
+
+ const showText = (text: string) => {
+ liveText = text;
+ options.onText?.(text);
+ };
+ // cuts the recording at the speaker's pauses: short ones, so the text keeps up with them
+ const cutAtPauses = (cut: () => boolean) => {
+ const pauses = new PauseDetector({
+ intervalMs: capture.levelsIntervalMs,
+ maxSegmentMs: VOICE_PIECE.MAX_MS,
+ minSpeechMs: VOICE_PIECE.MIN_SPEECH_MS,
+ pauseMs: VOICE_PIECE.PAUSE_MS
+ });
+
+ unsubscribers.push(
+ capture.onLevels((levels) => {
+ const atPause = pauses.push(levels[levels.length - 1] ?? 0);
+
+ if (pauses.heardSpeech) speechSinceCommit = true;
+
+ if (atPause && cut()) speechSinceCommit = false;
+ })
+ );
+ };
+ // pieces over HTTP, for when realtime is not to be had; the audio is kept from the start,
+ // so nothing said while the socket answers is lost if the model turns out unable to stream
+ const httpPieces = transcribe
+ ? new HttpPieceTranscriber({
+ onText: showText,
+ sampleRate: capture.sampleRate,
+ transcribe: (wav, signal) =>
+ transcribeClip(transcribe.context, wav, {
+ filename: VOICE_CLIP_FILENAME.PIECE,
+ language: transcribe.language,
+ model: transcribe.model,
+ signal
+ })
+ })
+ : null;
+
+ let piecesOverHttp = false;
+
+ if (httpPieces) unsubscribers.push(capture.onPcm((chunk) => httpPieces.append(chunk)));
+
+ const startHttpPieces = () => {
+ if (!httpPieces || ended) return;
+
+ piecesOverHttp = true;
+ cutAtPauses(() => {
+ if (!httpPieces.canCut) return false;
+
+ httpPieces.cut();
+
+ return true;
+ });
+ };
+
+ if (transcribe && options.realtime !== false) {
+ const realtime = new RealtimeTranscriber(transcribe.context, {
+ autoload: transcribe.autoload,
+ language: transcribe.language,
+ model: transcribe.model,
+ onCompleted: showText,
+ onDelta: (_delta, text) => showText(text),
+ sampleRate: capture.sampleRate
+ });
+ const unsubscribePcm = capture.onPcm((chunk) => realtime.append(chunk));
+
+ transcriber = realtime;
+ unsubscribers.push(unsubscribePcm);
+ realtime.connect().then(
+ () => {
+ // the socket has it all from here
+ httpPieces?.close();
+
+ if (ended) return;
+
+ options.onRealtime?.(realtime.live);
+
+ // a model that transcribes per commit gets the speech piece by piece, at its pauses
+ if (!realtime.live) {
+ cutAtPauses(() => {
+ if (!realtime.canCommitSegment) return false;
+
+ realtime.commitSegment();
+
+ return true;
+ });
+ }
+ },
+ (error) => {
+ unsubscribePcm();
+
+ if (ended) return;
+
+ if (!(error instanceof RealtimeUnsupportedError)) {
+ console.warn('Realtime transcription unavailable, transcribing over HTTP:', error);
+ }
+
+ options.onRealtime?.(false);
+ startHttpPieces();
+ }
+ );
+ } else if (transcribe) {
+ options.onRealtime?.(false);
+ startHttpPieces();
+ }
+
+ const release = () => {
+ ended = true;
+
+ for (const unsubscribe of unsubscribers) unsubscribe();
+
+ unsubscribers.length = 0;
+ };
+
+ return {
+ cancel() {
+ release();
+ capture.cancel();
+ transcriber?.close();
+ httpPieces?.close();
+ },
+ capture,
+ async stop(): Promise<VoiceSessionResult> {
+ let clip;
+
+ try {
+ // the last PCM chunk reaches the socket before stop() resolves
+ clip = await capture.stop();
+ } catch (error) {
+ release();
+ transcriber?.close();
+ httpPieces?.close();
+
+ throw error;
+ }
+
+ release();
+
+ if (!transcribe) return { clip, liveText: null, transcript: null };
+
+ const overHttp = () =>
+ transcribeClip(transcribe.context, clip.blob, {
+ language: transcribe.language,
+ model: transcribe.model
+ });
+ const realtime = transcriber;
+
+ let transcript: Promise<string>;
+
+ if (realtime?.ready) {
+ // more than one piece: the text was cut at pauses, maybe inside a sentence
+ const split = !realtime.live && realtime.piecesCommitted + (speechSinceCommit ? 1 : 0) > 1;
+ // a silent rest is left out: a model may hear a word ("yeah") in silence
+ const pieces = realtime.finish({ commitRest: speechSinceCommit }).catch((error) => {
+ console.warn('Realtime transcript failed, transcribing over HTTP:', error);
+
+ return overHttp();
+ });
+
+ transcript =
+ split && clip.durationMs <= VOICE_RETRANSCRIBE_MAX_MS
+ ? // the whole clip once more, punctuated by meaning, not by the pauses
+ overHttp().then(
+ (text) => text || pieces,
+ (error) => {
+ console.warn(
+ 'Transcribing the whole recording failed, keeping its pieces:',
+ error
+ );
+
+ return pieces;
+ }
+ )
+ : pieces;
+ } else if (piecesOverHttp && httpPieces) {
+ // the whole clip now, punctuated by meaning; a piece still on its way would hold the server up
+ const fromPieces = httpPieces.text;
+
+ httpPieces.close();
+ transcript = overHttp().catch((error) => {
+ if (!fromPieces) throw error;
+
+ console.warn('Transcribing the whole recording failed, keeping its pieces:', error);
+
+ return fromPieces;
+ });
+ } else {
+ // still connecting at the end: the HTTP request is the shorter wait
+ realtime?.close();
+ httpPieces?.close();
+ transcript = overHttp();
+ }
+
+ return { clip, liveText: liveText?.trim() || null, transcript };
+ }
+ };
+}
+
+function isCapture(value: VoiceSessionOptions['capture']): value is VoiceCapture {
+ return typeof (value as VoiceCapture | undefined)?.start === 'function';
+}
diff --git a/ui/lib/voice/src/speech-detector.ts b/ui/lib/voice/src/speech-detector.ts
new file mode 100644
index 000000000..03d2a5557
--- /dev/null
+++ b/ui/lib/voice/src/speech-detector.ts
@@ -0,0 +1,64 @@
+import { VOICE_SPEECH } from './constants.js';
+
+/**
+ * Tells speech from the room it is said in. Levels are in dBFS; the noise floor follows
+ * the room (it drops to any quieter level at once and creeps back up through louder
+ * ones), and a level clearly above it, and above the absolute minimum, is speech. So a
+ * quiet and a loud microphone both work, and so does a room with a fan running.
+ *
+ * Everything in the engine that needs to know whether someone is speaking asks one of
+ * these: the pause detector that cuts a recording, the gain that raises quiet speech, and
+ * a feature listening for speech over a reply. Tuning VOICE_SPEECH tunes them all.
+ */
+export interface SpeechDetectorOptions {
+ /** The room's noise level in dBFS when it is known already, so speech from the first level counts */
+ floorDb?: number;
+ /** How far above the floor a level is speech, in dB; higher for a stricter listener */
+ speechDb?: number;
+ /** How far above the floor a level still is quiet, in dB */
+ quietDb?: number;
+}
+
+/** Whether a level is speech, given the room's noise floor (both in dBFS). */
+export function isSpeechLevel(
+ db: number,
+ floorDb: number,
+ speechDb: number = VOICE_SPEECH.ABOVE_FLOOR_DB
+): boolean {
+ return db > VOICE_SPEECH.MIN_DB && db > floorDb + speechDb;
+}
+
+export class SpeechDetector {
+ /** The room's noise level in dBFS, as heard so far. */
+ floorDb: number;
+
+ private readonly quietDb: number;
+ private readonly speechDb: number;
+
+ constructor(options: SpeechDetectorOptions = {}) {
+ // full scale until the first level, which then sets it
+ this.floorDb = options.floorDb ?? 0;
+ this.quietDb = options.quietDb ?? VOICE_SPEECH.QUIET_ABOVE_FLOOR_DB;
+ this.speechDb = options.speechDb ?? VOICE_SPEECH.ABOVE_FLOOR_DB;
+ }
+
+ /** Whether `db` is quiet against the floor: the room, or a breath between words. */
+ isQuiet(db: number): boolean {
+ return db <= this.floorDb + this.quietDb;
+ }
+
+ /** Whether `db` is speech against the floor. */
+ isSpeech(db: number): boolean {
+ return isSpeechLevel(db, this.floorDb, this.speechDb);
+ }
+
+ /**
+ * Takes the level of the newest `ms` of input, follows the floor with it, and returns
+ * whether it is speech.
+ */
+ push(db: number, ms: number): boolean {
+ this.floorDb = Math.min(db, this.floorDb + (VOICE_SPEECH.FLOOR_RISE_DB_PER_S * ms) / 1000);
+
+ return this.isSpeech(db);
+ }
+}
diff --git a/ui/lib/voice/src/support.ts b/ui/lib/voice/src/support.ts
new file mode 100644
index 000000000..35f71232d
--- /dev/null
+++ b/ui/lib/voice/src/support.ts
@@ -0,0 +1,52 @@
+import { VOICE_DOM_ERROR, VOICE_MICROPHONE_MESSAGE } from './constants.js';
+
+/**
+ * What the browser offers for voice capture, and what to tell the user when it does not.
+ * Plain checks and messages, so the engine can say why a recording cannot start without
+ * the app's constants.
+ */
+
+/** Whether this browser has what voice capture uses: getUserMedia and an AudioContext. */
+export function isVoiceCaptureSupported(): boolean {
+ return !!(
+ typeof navigator !== 'undefined' &&
+ navigator.mediaDevices &&
+ typeof navigator.mediaDevices.getUserMedia === 'function' &&
+ typeof window !== 'undefined' &&
+ typeof window.AudioContext === 'function'
+ );
+}
+
+/**
+ * Why this page can't record at all, or null when it can: a reason no tap will fix. Browsers
+ * only expose the microphone to a secure context, HTTPS or localhost, so a llama-server
+ * opened over plain HTTP on the local network or over Tailscale can't record.
+ */
+export function microphoneUnavailableReason(): string | null {
+ if (typeof window === 'undefined') return null;
+
+ if (!window.isSecureContext) return VOICE_MICROPHONE_MESSAGE.insecure;
+
+ if (!isVoiceCaptureSupported()) return VOICE_MICROPHONE_MESSAGE.unsupported;
+
+ return null;
+}
+
+/** Why the microphone would not start, from the error the browser raised. */
+export function describeMicrophoneError(error: unknown): string {
+ switch (error instanceof DOMException ? error.name : '') {
+ case VOICE_DOM_ERROR.NOT_ALLOWED:
+ case VOICE_DOM_ERROR.SECURITY:
+ return VOICE_MICROPHONE_MESSAGE.denied;
+ case VOICE_DOM_ERROR.NOT_FOUND:
+ case VOICE_DOM_ERROR.OVERCONSTRAINED:
+ return VOICE_MICROPHONE_MESSAGE.notFound;
+ case VOICE_DOM_ERROR.ABORT:
+ case VOICE_DOM_ERROR.NOT_READABLE:
+ return VOICE_MICROPHONE_MESSAGE.busy;
+ default:
+ return error instanceof Error && error.message
+ ? `${VOICE_MICROPHONE_MESSAGE.failed} ${error.message}`
+ : VOICE_MICROPHONE_MESSAGE.failed;
+ }
+}
diff --git a/ui/lib/voice/src/svelte/index.ts b/ui/lib/voice/src/svelte/index.ts
new file mode 100644
index 000000000..8810add45
--- /dev/null
+++ b/ui/lib/voice/src/svelte/index.ts
@@ -0,0 +1,6 @@
+/**
+ * Svelte adapters of the voice engine: reactive wrappers, kept apart so the rest of
+ * lib/voice stays plain TypeScript that any framework (or none) can use.
+ */
+
+export { VoiceRecorder } from './voice-recorder.svelte.js';
diff --git a/ui/lib/voice/src/svelte/voice-recorder.svelte.ts b/ui/lib/voice/src/svelte/voice-recorder.svelte.ts
new file mode 100644
index 000000000..f70f24423
--- /dev/null
+++ b/ui/lib/voice/src/svelte/voice-recorder.svelte.ts
@@ -0,0 +1,107 @@
+import { VOICE_ERROR_MESSAGE, VOICE_LEVELS_LENGTH } from '../constants.js';
+import { VoiceRecorderStatus } from '../enums.js';
+import { startVoiceSession } from '../session.js';
+import type { VoiceSession, VoiceSessionOptions, VoiceSessionResult } from '../types.js';
+
+/**
+ * Reactive voice session for components: `levels`, `liveText`, `live`, `quiet` and
+ * `status` are runes that follow the running session.
+ */
+export class VoiceRecorder {
+ /** Level history for the live waveform, 0..1, newest last */
+ levels = $state.raw<number[]>(new Array(VOICE_LEVELS_LENGTH).fill(0));
+ /** True once realtime is known to stream text while speaking */
+ live = $state(false);
+ /** Text the realtime socket streamed so far */
+ liveText = $state('');
+ /** True while the speech is too quiet even at the highest gain */
+ quiet = $state(false);
+ status = $state<VoiceRecorderStatus>(VoiceRecorderStatus.IDLE);
+
+ // bumped by cancel(), so a start still waiting for the microphone drops its session
+ private generation = 0;
+ private session: VoiceSession | null = null;
+
+ /** Drops the recording. */
+ cancel(): void {
+ this.generation++;
+ this.session?.cancel();
+ this.session = null;
+ this.status = VoiceRecorderStatus.IDLE;
+ }
+
+ /**
+ * Starts a session; the callbacks in `options` still run.
+ * @throws VoiceCaptureError when the microphone does not start
+ */
+ async start(options: VoiceSessionOptions = {}): Promise<void> {
+ if (this.status !== VoiceRecorderStatus.IDLE) throw new Error(VOICE_ERROR_MESSAGE.recorderBusy);
+
+ const capture = options.capture;
+ const length =
+ capture && 'levels' in capture
+ ? capture.levels.length
+ : (capture?.levelsLength ?? VOICE_LEVELS_LENGTH);
+
+ this.levels = new Array(length).fill(0);
+ this.liveText = '';
+ this.live = false;
+ this.quiet = false;
+ this.status = VoiceRecorderStatus.STARTING;
+
+ const generation = ++this.generation;
+
+ try {
+ const session = await startVoiceSession({
+ ...options,
+ onLevels: (levels) => {
+ this.levels = levels as number[];
+ options.onLevels?.(levels);
+ },
+ onQuiet: (quiet) => {
+ this.quiet = quiet;
+ options.onQuiet?.(quiet);
+ },
+ onRealtime: (live) => {
+ this.live = live;
+ options.onRealtime?.(live);
+ },
+ onText: (text) => {
+ this.liveText = text;
+ options.onText?.(text);
+ }
+ });
+
+ if (generation !== this.generation) {
+ session.cancel();
+
+ return;
+ }
+
+ this.session = session;
+ this.status = VoiceRecorderStatus.RECORDING;
+ } catch (error) {
+ if (generation === this.generation) this.status = VoiceRecorderStatus.IDLE;
+
+ throw error;
+ }
+ }
+
+ /** Ends the recording; see VoiceSessionResult. */
+ async stop(): Promise<VoiceSessionResult> {
+ const session = this.session;
+
+ if (!session || this.status !== VoiceRecorderStatus.RECORDING) {
+ throw new Error(VOICE_ERROR_MESSAGE.recorderIdle);
+ }
+
+ this.status = VoiceRecorderStatus.STOPPING;
+
+ try {
+ return await session.stop();
+ } finally {
+ this.session = null;
+ this.status = VoiceRecorderStatus.IDLE;
+ }
+ }
+}
diff --git a/ui/lib/voice/src/transcribe.ts b/ui/lib/voice/src/transcribe.ts
new file mode 100644
index 000000000..043dfc5d6
--- /dev/null
+++ b/ui/lib/voice/src/transcribe.ts
@@ -0,0 +1,92 @@
+import {
+ VOICE_API_PATHS,
+ VOICE_CLIP_FILENAME,
+ VOICE_CONTENT_TYPE_HEADER,
+ VOICE_ERROR_MESSAGE,
+ VOICE_HTML_BODY_PATTERN,
+ VOICE_TRANSCRIPTION_FORM,
+ VOICE_TRANSCRIPTION_RESPONSE_FORMAT
+} from './constants.js';
+import { VoiceTranscriptionError } from './errors.js';
+import { cleanTranscript } from './transcript.js';
+import type { TranscribeClipOptions, VoiceServerContext } from './types.js';
+
+/**
+ * Transcribes a recorded clip with `POST /v1/audio/transcriptions`: any model with
+ * audio input can do it, the server turns it into a chat completion.
+ * @returns the transcript, trimmed and without the model's own format (see cleanTranscript)
+ * @throws VoiceTranscriptionError carrying the server's own message
+ */
+export async function transcribeClip(
+ context: VoiceServerContext,
+ blob: Blob,
+ options: TranscribeClipOptions = {}
+): Promise<string> {
+ const form = new FormData();
+
+ form.append(VOICE_TRANSCRIPTION_FORM.FILE, blob, options.filename ?? VOICE_CLIP_FILENAME.CLIP);
+
+ if (options.model) form.append(VOICE_TRANSCRIPTION_FORM.MODEL, options.model);
+
+ if (options.language) form.append(VOICE_TRANSCRIPTION_FORM.LANGUAGE, options.language);
+
+ if (options.prompt) form.append(VOICE_TRANSCRIPTION_FORM.PROMPT, options.prompt);
+
+ form.append(VOICE_TRANSCRIPTION_FORM.RESPONSE_FORMAT, VOICE_TRANSCRIPTION_RESPONSE_FORMAT);
+
+ // no Content-Type: the browser sets the multipart boundary itself
+ const headers = { ...context.getHeaders?.() };
+
+ for (const key of Object.keys(headers)) {
+ if (key.toLowerCase() === VOICE_CONTENT_TYPE_HEADER) delete headers[key];
+ }
+
+ const response = await (context.fetch ?? fetch)(
+ context.resolveUrl(VOICE_API_PATHS.TRANSCRIPTIONS),
+ {
+ body: form,
+ headers,
+ method: 'POST',
+ signal: options.signal
+ }
+ );
+
+ if (!response.ok) {
+ throw new VoiceTranscriptionError(await readErrorMessage(response), response.status);
+ }
+
+ const data = (await response.json()) as { text?: unknown };
+
+ if (typeof data?.text !== 'string') {
+ throw new VoiceTranscriptionError(VOICE_ERROR_MESSAGE.noTranscript, response.status);
+ }
+
+ return cleanTranscript(data.text).trim();
+}
+
+async function readErrorMessage(response: Response): Promise<string> {
+ const fallback = `${VOICE_ERROR_MESSAGE.transcriptionFailed} (${response.status}${response.statusText ? ` ${response.statusText}` : ''})`;
+
+ try {
+ const text = await response.text();
+
+ try {
+ const data = JSON.parse(text);
+
+ if (typeof data?.error?.message === 'string') return data.error.message;
+
+ if (typeof data?.error === 'string') return data.error;
+
+ if (typeof data?.message === 'string') return data.message;
+ } catch {
+ // not JSON: a proxy's plain text error
+ }
+
+ const plain = text.trim();
+
+ // an HTML error page says nothing useful in a toast
+ return plain && !VOICE_HTML_BODY_PATTERN.test(plain) ? plain : fallback;
+ } catch {
+ return fallback;
+ }
+}
diff --git a/ui/lib/voice/src/transcript.ts b/ui/lib/voice/src/transcript.ts
new file mode 100644
index 000000000..f735684d8
--- /dev/null
+++ b/ui/lib/voice/src/transcript.ts
@@ -0,0 +1,55 @@
+import {
+ VOICE_ASR_TEXT_PREFIX_PATTERN,
+ VOICE_ASR_TEXT_TAG_PATTERN,
+ VOICE_SEGMENTS_CONTENT_PATTERN,
+ VOICE_SEGMENTS_START_PATTERN,
+ VOICE_UNFINISHED_ESCAPE_PATTERN
+} from './constants.js';
+
+/**
+ * A transcript as the speaker's words alone. Some speech-to-text models wrap them in a
+ * format of their own, which the server passes on as it is: Qwen3-ASR names the language
+ * first (`language French<asr_text>Bonjour`), VibeVoice-ASR answers with timed segments
+ * (`[{"Start":0,"End":3.1,"Speaker":0,"Content":"Bonjour"}]`). Text in neither format is
+ * returned untouched, so text still streaming keeps its spaces.
+ *
+ * It takes text that is still streaming too: segments cut short give the words they
+ * hold so far, and nothing before the first one.
+ */
+export function cleanTranscript(text: string): string {
+ if (VOICE_SEGMENTS_START_PATTERN.test(text)) return segmentsText(text);
+
+ // search() keeps no position in the shared pattern
+ if (text.search(VOICE_ASR_TEXT_TAG_PATTERN) === -1) return text;
+
+ return text
+ .replace(VOICE_ASR_TEXT_PREFIX_PATTERN, '')
+ .replace(VOICE_ASR_TEXT_TAG_PATTERN, '')
+ .trim();
+}
+
+/** The words of timed segments, in order, whether or not the JSON is complete. */
+function segmentsText(text: string): string {
+ const contents: string[] = [];
+
+ for (const match of text.matchAll(VOICE_SEGMENTS_CONTENT_PATTERN)) {
+ const content = unescapeJsonString(match[1]).trim();
+
+ if (content) contents.push(content);
+ }
+
+ return contents.join(' ');
+}
+
+/** The value of a JSON string's inside, which may stop in the middle of an escape. */
+function unescapeJsonString(inside: string): string {
+ try {
+ return JSON.parse(`"${inside}"`);
+ } catch {
+ try {
+ return JSON.parse(`"${inside.replace(VOICE_UNFINISHED_ESCAPE_PATTERN, '')}"`);
+ } catch {
+ return inside;
+ }
+ }
+}
diff --git a/ui/lib/voice/src/types.ts b/ui/lib/voice/src/types.ts
new file mode 100644
index 000000000..ad8f9378c
--- /dev/null
+++ b/ui/lib/voice/src/types.ts
@@ -0,0 +1,129 @@
+import type { VoiceCapture } from './capture.js';
+
+/** Voice engine types. */
+
+/** A finished recording, ready to send and to draw. */
+export interface VoiceClip {
+ /** WAV, 16-bit PCM, mono */
+ blob: Blob;
+ durationMs: number;
+ /** Normalised 0..1, fixed count, for the static waveform */
+ peaks: number[];
+ sampleRate: number;
+}
+
+/** Called with the live level history: 0..1, fixed length, newest last. */
+export type VoiceLevelsListener = (levels: readonly number[]) => void;
+
+/** Called with each PCM16 mono chunk at the capture rate. */
+export type VoicePcmListener = (chunk: Int16Array) => void;
+
+/** Called when the speech starts or stops being too quiet even at the highest gain. */
+export type VoiceQuietListener = (quiet: boolean) => void;
+
+export interface VoiceCaptureOptions {
+ /** Rate of the PCM chunks and of the clip (default 16000) */
+ sampleRate?: number;
+ /** Length of a PCM chunk (default 100 ms) */
+ chunkMs?: number;
+ /** Values in the level history (default 48) */
+ levelsLength?: number;
+ /** Time between two level values (default 50 ms) */
+ levelsIntervalMs?: number;
+ /** Peaks of the clip (default 64) */
+ peaksCount?: number;
+ /** Extra constraints merged over the raw-signal ones (no echo cancellation, noise suppression or auto gain) */
+ constraints?: MediaTrackConstraints;
+ /** Microphone to record from, see resolveMicrophone(); left out, the system default */
+ deviceId?: string;
+ /** Use this stream instead of asking for the microphone; the capture stops its tracks too */
+ stream?: MediaStream;
+ /** Raise quiet speech (gain.ts): the streamed chunks as they go, the clip as a whole */
+ boost?: boolean;
+}
+
+/**
+ * How the engine reaches a server. The app wires its own URL resolution and
+ * auth in, so the engine never imports app stores.
+ */
+export interface VoiceServerContext {
+ /** Request target for an API path, e.g. `(path) => apiUrl(path, serverId)` */
+ resolveUrl: (path: string) => string;
+ /** HTTP headers such as the API key; a browser websocket cannot carry them */
+ getHeaders?: () => Record<string, string>;
+ /** Fetch to use, defaults to the global one */
+ fetch?: typeof fetch;
+ /** Base for relative URLs, defaults to `location.href` */
+ baseHref?: string;
+}
+
+export interface TranscribeClipOptions {
+ /** Model id, required by a router server */
+ model?: string;
+ /** ISO-639-1 language hint */
+ language?: string;
+ /** Replaces the model's own transcription prompt */
+ prompt?: string;
+ /** File name sent with the audio (default `audio.wav`) */
+ filename?: string;
+ signal?: AbortSignal;
+}
+
+export interface RealtimeTranscriberOptions {
+ /** Model id, sent as `?model=` (the router routes on it) */
+ model?: string;
+ /** Rate of the PCM16 chunks given to `append` */
+ sampleRate: number;
+ /** ISO-639-1 language hint, used by a conversation model switched to transcription */
+ language?: string;
+ /** Sent as `?autoload=`; false keeps a router from loading the model */
+ autoload?: boolean;
+ connectTimeoutMs?: number;
+ finishTimeoutMs?: number;
+ /** New text of the current item, and the whole live text */
+ onDelta?: (delta: string, text: string) => void;
+ /** The whole transcript so far, once an item completes */
+ onCompleted?: (text: string) => void;
+ /** Errors after the session started; the session goes on unless finish() fails */
+ onError?: (error: Error) => void;
+}
+
+export interface VoiceSessionOptions {
+ /** Capture options, or a capture made by the caller */
+ capture?: VoiceCaptureOptions | VoiceCapture;
+ /** Try the realtime socket first (default true); off for servers behind a tunnel */
+ realtime?: boolean;
+ /** Where and with which model to transcribe; leave out to record only */
+ transcribe?: {
+ context: VoiceServerContext;
+ model?: string;
+ language?: string;
+ autoload?: boolean;
+ };
+ onLevels?: VoiceLevelsListener;
+ /** Whether the speech is too quiet even at the highest gain, when that changes */
+ onQuiet?: VoiceQuietListener;
+ /** Live text while speaking, only when the realtime socket streams it */
+ onText?: (text: string) => void;
+ /** Once known: true when text streams while speaking, false when it comes after recording */
+ onRealtime?: (live: boolean) => void;
+}
+
+export interface VoiceSessionResult {
+ clip: VoiceClip;
+ /** Text the realtime socket gave while speaking, null when it streamed none */
+ liveText: string | null;
+ /**
+ * Resolves with the final transcript (realtime, else HTTP; a recording transcribed in pieces
+ * once more as a whole); null when not asked to transcribe
+ */
+ transcript: Promise<string> | null;
+}
+
+export interface VoiceSession {
+ readonly capture: VoiceCapture;
+ /** Ends the recording and resolves with the clip and the transcript (or its promise) */
+ stop(): Promise<VoiceSessionResult>;
+ /** Drops the recording and closes everything */
+ cancel(): void;
+}
diff --git a/ui/lib/voice/src/warm-microphone.ts b/ui/lib/voice/src/warm-microphone.ts
new file mode 100644
index 000000000..4d029c092
--- /dev/null
+++ b/ui/lib/voice/src/warm-microphone.ts
@@ -0,0 +1,92 @@
+import { VOICE_TRACK_LIVE, VOICE_WARMUP_RELEASE_MS } from './constants.js';
+import { microphoneAllowed, microphoneRequest, stopStream } from './microphone.js';
+import type { VoiceCaptureOptions } from './types.js';
+
+type MicrophoneOptions = Pick<VoiceCaptureOptions, 'constraints' | 'deviceId'>;
+
+interface Warm {
+ key: string;
+ stream: Promise<MediaStream | null>;
+}
+
+/**
+ * Opens the microphone a moment before a recording starts, e.g. while the pointer is on
+ * the record button, so the device has settled (its level, the system's input gain) by
+ * the time the first words come, and the recording starts at once. A recording then
+ * takes the open stream (take) instead of asking for one.
+ *
+ * It only opens a microphone the page may already use: it never makes the browser ask,
+ * as nothing was pressed yet. The browser shows its recording indicator while it is open,
+ * so it is let go shortly after the pointer leaves (release).
+ */
+export class MicrophoneWarmup {
+ private warm: Warm | null = null;
+ private timer: ReturnType<typeof setTimeout> | undefined;
+
+ /** Opens the microphone for these options, keeping one that is open for the same already. */
+ open(options: MicrophoneOptions = {}): void {
+ clearTimeout(this.timer);
+
+ const request = microphoneRequest(options);
+ const key = JSON.stringify(request);
+
+ if (this.warm?.key === key) return;
+
+ this.drop();
+
+ const stream = (async () => {
+ if (!navigator.mediaDevices?.getUserMedia || !(await microphoneAllowed())) return null;
+
+ try {
+ return await navigator.mediaDevices.getUserMedia(request);
+ } catch {
+ // the recording asks again, and reports what is wrong
+ return null;
+ }
+ })();
+
+ this.warm = { key, stream };
+ }
+
+ /**
+ * Hands the open stream over for a recording with these options, which then owns it;
+ * null when none is open for them (it is let go), so the recording asks for its own.
+ */
+ async take(options: MicrophoneOptions = {}): Promise<MediaStream | null> {
+ clearTimeout(this.timer);
+
+ const warm = this.warm;
+
+ this.warm = null;
+
+ if (!warm) return null;
+
+ const stream = await warm.stream;
+ const live =
+ stream?.getAudioTracks().some((track) => track.readyState === VOICE_TRACK_LIVE) ?? false;
+
+ if (warm.key !== JSON.stringify(microphoneRequest(options)) || !live) {
+ stopStream(stream);
+
+ return null;
+ }
+
+ return stream;
+ }
+
+ /** Lets the microphone go after `delayMs`, unless a recording takes it or it is opened again. */
+ release(delayMs: number = VOICE_WARMUP_RELEASE_MS): void {
+ clearTimeout(this.timer);
+ this.timer = setTimeout(() => this.drop(), delayMs);
+ }
+
+ /** Lets the microphone go now. */
+ drop(): void {
+ clearTimeout(this.timer);
+
+ const warm = this.warm;
+
+ this.warm = null;
+ void warm?.stream.then(stopStream);
+ }
+}
diff --git a/ui/lib/voice/src/wav.ts b/ui/lib/voice/src/wav.ts
new file mode 100644
index 000000000..ff37e721f
--- /dev/null
+++ b/ui/lib/voice/src/wav.ts
@@ -0,0 +1,36 @@
+import { VOICE_WAV_MIME } from './constants.js';
+
+const WAV_HEADER_BYTES = 44;
+
+/**
+ * Mono PCM16 samples as a WAV file: the format `/v1/audio/transcriptions` and the
+ * chat's `input_audio` parts accept.
+ */
+export function encodeWav(samples: Int16Array, sampleRate: number): Blob {
+ const dataSize = samples.length * 2;
+ const buffer = new ArrayBuffer(WAV_HEADER_BYTES + dataSize);
+ const view = new DataView(buffer);
+ const writeAscii = (offset: number, text: string) => {
+ for (let i = 0; i < text.length; i++) view.setUint8(offset + i, text.charCodeAt(i));
+ };
+
+ writeAscii(0, 'RIFF');
+ view.setUint32(4, WAV_HEADER_BYTES - 8 + dataSize, true);
+ writeAscii(8, 'WAVE');
+ writeAscii(12, 'fmt ');
+ view.setUint32(16, 16, true); // fmt chunk size
+ view.setUint16(20, 1, true); // PCM
+ view.setUint16(22, 1, true); // mono
+ view.setUint32(24, sampleRate, true);
+ view.setUint32(28, sampleRate * 2, true); // byte rate
+ view.setUint16(32, 2, true); // block align
+ view.setUint16(34, 16, true); // bits per sample
+ writeAscii(36, 'data');
+ view.setUint32(40, dataSize, true);
+
+ for (let i = 0; i < samples.length; i++) {
+ view.setInt16(WAV_HEADER_BYTES + i * 2, samples[i], true);
+ }
+
+ return new Blob([buffer], { type: VOICE_WAV_MIME });
+}
diff --git a/ui/lib/voice/tests/audio-context.test.ts b/ui/lib/voice/tests/audio-context.test.ts
new file mode 100644
index 000000000..a04d4d9e8
--- /dev/null
+++ b/ui/lib/voice/tests/audio-context.test.ts
@@ -0,0 +1,38 @@
+// Guards the shared capture context: it idles only once the last capture using it lets
+// go, so one still letting go cannot silence the next recording.
+
+import { captureContext, idleCaptureContext } from '../src/audio-context.js';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+class FakeAudioContext {
+ state: AudioContextState = 'running';
+ suspend = vi.fn(async () => {
+ this.state = 'suspended';
+ });
+ resume = vi.fn(async () => {});
+ close = vi.fn(async () => {});
+ audioWorklet = { addModule: async () => {} };
+}
+
+beforeEach(() => {
+ vi.stubGlobal('AudioContext', FakeAudioContext);
+});
+
+afterEach(() => {
+ vi.unstubAllGlobals();
+});
+
+describe('captureContext', () => {
+ it('idles once the last capture lets go', () => {
+ const first = captureContext().context;
+ const second = captureContext().context;
+
+ expect(second).toBe(first);
+
+ idleCaptureContext(first);
+ expect(first.suspend).not.toHaveBeenCalled();
+
+ idleCaptureContext(second);
+ expect(first.suspend).toHaveBeenCalledOnce();
+ });
+});
diff --git a/ui/lib/voice/tests/audio.test.ts b/ui/lib/voice/tests/audio.test.ts
new file mode 100644
index 000000000..431492703
--- /dev/null
+++ b/ui/lib/voice/tests/audio.test.ts
@@ -0,0 +1,195 @@
+// Guards the voice engine's audio math: PCM16 conversion, the streaming resampler,
+// the WAV the server accepts (16-bit PCM mono header fields), peaks and levels.
+
+import { LevelHistory, rms, rmsToLevel } from '../src/levels.js';
+import {
+ base64ToBytes,
+ concatPcm16,
+ createResampler,
+ floatToPcm16,
+ pcm16ToBase64,
+ pcm16ToFloat
+} from '../src/pcm.js';
+import { computePeaks } from '../src/peaks.js';
+import { encodeWav } from '../src/wav.js';
+import { describe, expect, it } from 'vitest';
+
+function sine(rate: number, seconds: number, frequency: number, amplitude = 0.5): Float32Array {
+ const out = new Float32Array(Math.round(rate * seconds));
+
+ for (let i = 0; i < out.length; i++) {
+ out[i] = amplitude * Math.sin((2 * Math.PI * frequency * i) / rate);
+ }
+
+ return out;
+}
+
+describe('PCM16 conversion', () => {
+ it('maps full scale to the int16 range and clips beyond it', () => {
+ expect(Array.from(floatToPcm16(Float32Array.from([0, 1, -1, 2, -2, 0.5])))).toEqual([
+ 0, 32767, -32768, 32767, -32768, 16383
+ ]);
+ });
+
+ it('round-trips through floats within one step', () => {
+ const input = Float32Array.from([0.25, -0.75, 0.1]);
+ const back = pcm16ToFloat(floatToPcm16(input));
+
+ for (let i = 0; i < input.length; i++) expect(back[i]).toBeCloseTo(input[i], 4);
+ });
+
+ it('joins chunks in order', () => {
+ expect(
+ Array.from(concatPcm16([Int16Array.from([1, 2]), Int16Array.from([]), Int16Array.from([3])]))
+ ).toEqual([1, 2, 3]);
+ });
+
+ it('encodes little-endian base64', () => {
+ const bytes = base64ToBytes(pcm16ToBase64(Int16Array.from([1, -2, 0x1234])));
+
+ expect(Array.from(bytes)).toEqual([0x01, 0x00, 0xfe, 0xff, 0x34, 0x12]);
+ });
+
+ it('encodes long inputs without overflowing the call stack', () => {
+ const samples = new Int16Array(200_000).fill(7);
+
+ expect(base64ToBytes(pcm16ToBase64(samples)).length).toBe(400_000);
+ });
+});
+
+describe('createResampler', () => {
+ it('passes the same rate through', () => {
+ const input = Float32Array.from([0.1, 0.2]);
+
+ expect(createResampler(16_000, 16_000).process(input)).toBe(input);
+ });
+
+ it('downsamples 48 kHz to 16 kHz in blocks, holding back at most the last sample', () => {
+ const resampler = createResampler(48_000, 16_000);
+ const input = sine(48_000, 1, 440);
+
+ let total = 0;
+
+ // odd block sizes, as the audio thread may deliver
+ for (let offset = 0; offset < input.length; offset += 1000) {
+ total += resampler.process(input.subarray(offset, offset + 1000)).length;
+ }
+
+ // the last output waits for the input after it
+ expect(total).toBeGreaterThanOrEqual(15_999);
+ expect(total).toBeLessThanOrEqual(16_000);
+ });
+
+ it('downsamples 44.1 kHz to within one sample of the expected length', () => {
+ const resampler = createResampler(44_100, 16_000);
+ const out = resampler.process(sine(44_100, 1, 300));
+
+ expect(Math.abs(out.length - 16_000)).toBeLessThanOrEqual(2);
+ });
+
+ it('keeps a speech-band tone and its level', () => {
+ const out = createResampler(48_000, 16_000).process(sine(48_000, 0.5, 440, 0.5));
+
+ expect(rms(out)).toBeCloseTo(0.5 / Math.SQRT2, 2);
+ });
+
+ it('averages away a tone above the new Nyquist rate', () => {
+ // 12 kHz aliases into speech at 16 kHz unless it is filtered out
+ const out = createResampler(48_000, 16_000).process(sine(48_000, 0.5, 12_000, 0.5));
+
+ expect(rms(out)).toBeLessThan(0.05);
+ });
+
+ it('upsamples by linear interpolation', () => {
+ const out = createResampler(8_000, 16_000).process(Float32Array.from([0, 1, 0, -1, 0]));
+
+ expect(Array.from(out)).toEqual([0, 0.5, 1, 0.5, 0, -0.5, -1, -0.5]);
+ });
+});
+
+describe('encodeWav', () => {
+ it('writes a 16-bit PCM mono header the server accepts', async () => {
+ const samples = Int16Array.from([0, 1000, -1000, 32767]);
+ const blob = encodeWav(samples, 16_000);
+ const view = new DataView(await blob.arrayBuffer());
+ const ascii = (offset: number) =>
+ String.fromCharCode(...new Uint8Array(view.buffer, offset, 4));
+
+ expect(blob.type).toBe('audio/wav');
+ expect(view.byteLength).toBe(44 + 8);
+ expect(ascii(0)).toBe('RIFF');
+ expect(view.getUint32(4, true)).toBe(36 + 8);
+ expect(ascii(8)).toBe('WAVE');
+ expect(ascii(12)).toBe('fmt ');
+ expect(view.getUint32(16, true)).toBe(16);
+ expect(view.getUint16(20, true)).toBe(1); // PCM
+ expect(view.getUint16(22, true)).toBe(1); // mono
+ expect(view.getUint32(24, true)).toBe(16_000);
+ expect(view.getUint32(28, true)).toBe(32_000);
+ expect(view.getUint16(32, true)).toBe(2);
+ expect(view.getUint16(34, true)).toBe(16);
+ expect(ascii(36)).toBe('data');
+ expect(view.getUint32(40, true)).toBe(8);
+ expect([0, 1, 2, 3].map((i) => view.getInt16(44 + i * 2, true))).toEqual([
+ 0, 1000, -1000, 32767
+ ]);
+ });
+});
+
+describe('computePeaks', () => {
+ it('gives the normalised loudest sample of each span', () => {
+ const samples = Float32Array.from([0.1, -0.2, 0.4, -0.1, 0, 0.05, -0.8, 0.2]);
+
+ expect(computePeaks(samples, 4)).toEqual([0.25, 0.5, 0.0625, 1]);
+ });
+
+ it('reads PCM16 the same as floats', () => {
+ const floats = sine(16_000, 0.2, 200);
+ const fromPcm = computePeaks(floatToPcm16(floats), 16);
+ const fromFloats = computePeaks(floats, 16);
+
+ fromPcm.forEach((value, i) => expect(value).toBeCloseTo(fromFloats[i], 3));
+ });
+
+ it('mixes the channels of a buffer', () => {
+ const left = Float32Array.from([1, 0]);
+ const right = Float32Array.from([0, 0.5]);
+ const buffer = {
+ getChannelData: (c: number) => (c === 0 ? left : right),
+ length: 2,
+ numberOfChannels: 2
+ } as unknown as AudioBuffer;
+
+ expect(computePeaks(buffer, 2)).toEqual([1, 0.5]);
+ });
+
+ it('keeps the count for silence, empty and short clips', () => {
+ expect(computePeaks(new Float32Array(100), 8)).toEqual(new Array(8).fill(0));
+ expect(computePeaks(new Float32Array(0), 8)).toEqual(new Array(8).fill(0));
+ expect(computePeaks(Float32Array.from([0.5, -1]), 4)).toEqual([0.5, 0.5, 1, 1]);
+ });
+});
+
+describe('levels', () => {
+ it('maps loudness in decibels from the floor to full scale', () => {
+ expect(rmsToLevel(0)).toBe(0);
+ expect(rmsToLevel(1)).toBe(1);
+ expect(rmsToLevel(2)).toBe(1);
+ expect(rmsToLevel(0.001)).toBe(0); // -60 dBFS
+ expect(rmsToLevel(0.0316)).toBeCloseTo(0.5, 2); // -30 dBFS
+ });
+
+ it('keeps a fixed-length history, newest last', () => {
+ const history = new LevelHistory(3);
+
+ history.push(0.1);
+ history.push(0.2);
+
+ const levels = history.push(0.3);
+
+ expect(levels).toEqual([0.1, 0.2, 0.3]);
+ expect(history.push(0.4)).toEqual([0.2, 0.3, 0.4]);
+ // listeners get copies they can keep
+ expect(levels).toEqual([0.1, 0.2, 0.3]);
+ });
+});
diff --git a/ui/lib/voice/tests/dictation-span.test.ts b/ui/lib/voice/tests/dictation-span.test.ts
new file mode 100644
index 000000000..4c599b0e0
--- /dev/null
+++ b/ui/lib/voice/tests/dictation-span.test.ts
@@ -0,0 +1,90 @@
+// Guards how dictation writes into the message: at the caret, replacing only its own text
+// as the transcript grows, and keeping what the user typed around it.
+
+import { DictationSpan } from '../src/dictation-span.js';
+import { describe, expect, it } from 'vitest';
+
+describe('DictationSpan', () => {
+ it('writes into an empty message and replaces its own text as it grows', () => {
+ const span = new DictationSpan('', 0);
+
+ let value = span.apply('', 'Hello');
+
+ expect(value).toBe('Hello');
+
+ value = span.apply(value, 'Hello world');
+ expect(value).toBe('Hello world');
+ expect(span.end).toBe(value.length);
+ });
+
+ it('inserts at the caret with spaces, keeping the text before and after', () => {
+ const start = 'Translate into German';
+ const span = new DictationSpan(start, 'Translate'.length);
+
+ let value = span.apply(start, 'good');
+
+ expect(value).toBe('Translate good into German');
+
+ value = span.apply(value, 'good morning');
+ expect(value).toBe('Translate good morning into German');
+ });
+
+ it('adds a space where the dictated text would run into a word', () => {
+ const span = new DictationSpan('Say', 3);
+
+ expect(span.apply('Say', 'hi')).toBe('Say hi');
+
+ const middle = new DictationSpan('ab', 1);
+
+ expect(middle.apply('ab', 'x')).toBe('a x b');
+ });
+
+ it('follows text typed before it while it streams', () => {
+ const span = new DictationSpan('', 0);
+
+ let value = span.apply('', 'one');
+
+ const typed = `Note: ${value}`;
+
+ span.edited(value, typed);
+ value = span.apply(typed, 'one two');
+
+ expect(value).toBe('Note: one two');
+ });
+
+ it('keeps text typed after it', () => {
+ const span = new DictationSpan('', 0);
+
+ let value = span.apply('', 'one');
+
+ const typed = `${value}!`;
+
+ span.edited(value, typed);
+ value = span.apply(typed, 'one two');
+
+ expect(value).toBe('one two !');
+ });
+
+ it('hands over text the user edits and dictates on after it', () => {
+ const span = new DictationSpan('', 0);
+
+ let value = span.apply('', 'helo');
+
+ const fixed = 'hello';
+
+ span.edited(value, fixed);
+ value = span.apply(fixed, 'helo there');
+
+ expect(value).toBe('hello there');
+ });
+
+ it('takes its text back out when the dictation is cancelled', () => {
+ const start = 'Before after';
+ const span = new DictationSpan(start, 6);
+ const value = span.apply(start, 'middle');
+
+ expect(value).toBe('Before middle after');
+ expect(span.remove(value)).toBe(start);
+ expect(span.isEmpty).toBe(true);
+ });
+});
diff --git a/ui/lib/voice/tests/dictation.test.ts b/ui/lib/voice/tests/dictation.test.ts
new file mode 100644
index 000000000..fa4377b75
--- /dev/null
+++ b/ui/lib/voice/tests/dictation.test.ts
@@ -0,0 +1,158 @@
+// Guards dictation into a plain text field, without the chat form: text streams in at the
+// caret, the user's typing around it stays, the whole recording's text replaces the live
+// text when it comes, and a cancel or a voice message takes the text back out.
+
+import { createDictation } from '../src/dictation.js';
+import { describe, expect, it, vi } from 'vitest';
+
+/** A text field: its value and caret, and a dictation writing into it. */
+function field(value = '', caret = value.length) {
+ const state = { caret, changes: 0, value };
+ const onError = vi.fn();
+ const dictation = createDictation({
+ getCaretOffset: () => state.caret,
+ getValue: () => state.value,
+ onChange: () => state.changes++,
+ onError,
+ setCaretOffset: (offset) => (state.caret = offset),
+ setValue: (next) => (state.value = next)
+ });
+
+ /** The user types: the field changes, and dictation is told. */
+ const type = (next: string) => {
+ state.value = next;
+ dictation.edited();
+ };
+
+ return { dictation, onError, state, type };
+}
+
+describe('createDictation', () => {
+ it('writes at the caret, keeping the text around it', () => {
+ const { dictation, state } = field('Dear Ana, thanks', 9);
+
+ dictation.begin();
+ dictation.write('the notes');
+ dictation.write('the notes were great');
+
+ expect(state.value).toBe('Dear Ana, the notes were great thanks');
+ expect(dictation.active).toBe(true);
+ });
+
+ it('keeps what the user types before the dictated text', () => {
+ const { dictation, state, type } = field('');
+
+ dictation.begin();
+ dictation.write('hello');
+ type('Oh, hello');
+ dictation.write('hello there');
+
+ expect(state.value).toBe('Oh, hello there');
+ });
+
+ it("replaces the live text with the whole recording's, then puts the caret after it", async () => {
+ const { dictation, state } = field('Note:');
+
+ dictation.begin();
+ dictation.write('buy milk an bred');
+
+ const finished = dictation.finish({
+ liveText: 'buy milk an bred',
+ transcript: Promise.resolve('buy milk and bread')
+ });
+
+ expect(dictation.refining).toBe(true);
+
+ await finished;
+
+ expect(state.value).toBe('Note: buy milk and bread');
+ expect(state.caret).toBe(state.value.length);
+ expect(dictation.refining).toBe(false);
+ expect(dictation.active).toBe(false);
+ });
+
+ it('keeps the live text when the whole recording comes back empty or fails', async () => {
+ const empty = field();
+
+ empty.dictation.begin();
+ empty.dictation.write('kept');
+ await empty.dictation.finish({ liveText: 'kept', transcript: Promise.resolve(' ') });
+ expect(empty.state.value).toBe('kept');
+
+ const failed = field();
+
+ failed.dictation.begin();
+ failed.dictation.write('kept');
+ await failed.dictation.finish({
+ liveText: 'kept',
+ transcript: Promise.reject(new Error('busy'))
+ });
+ expect(failed.state.value).toBe('kept');
+ expect(failed.onError).toHaveBeenCalledOnce();
+ });
+
+ it('settles once the text is final', async () => {
+ const { dictation } = field();
+ let resolve!: (text: string) => void;
+
+ dictation.begin();
+ dictation.write('draft');
+
+ const finished = dictation.finish({
+ liveText: 'draft',
+ transcript: new Promise((done) => (resolve = done))
+ });
+ const settled = vi.fn();
+
+ void dictation.settled().then(settled);
+ await Promise.resolve();
+ expect(settled).not.toHaveBeenCalled();
+
+ resolve('final');
+ await finished;
+ expect(settled).toHaveBeenCalled();
+ });
+
+ it('takes the text back out on cancel, and drops text still on its way', async () => {
+ const { dictation, state } = field('Hi');
+
+ dictation.begin();
+ dictation.write('there');
+
+ let resolve!: (text: string) => void;
+ const finished = dictation.finish({
+ liveText: 'there',
+ transcript: new Promise((done) => (resolve = done))
+ });
+
+ dictation.cancel();
+ resolve('there you go');
+ await finished;
+
+ expect(state.value).toBe('Hi');
+ expect(dictation.active).toBe(false);
+ });
+
+ it("hands a voice message's text over with whether the user edited it", () => {
+ const { dictation, state, type } = field('');
+
+ dictation.begin({ at: 0, keepsEdits: true });
+ dictation.write('send the file');
+ type('send that file');
+
+ expect(dictation.take()).toEqual({ text: 'send that file', userEdited: true });
+ expect(state.value).toBe('');
+ expect(dictation.take()).toBeNull();
+ });
+
+ it('tells the caller when its state changes', async () => {
+ const { dictation, state } = field();
+
+ dictation.begin();
+ dictation.write('a');
+ await dictation.finish({ liveText: 'a', transcript: Promise.resolve('a') });
+
+ // begin, refining on and off, and the end
+ expect(state.changes).toBe(4);
+ });
+});
diff --git a/ui/lib/voice/tests/gain.test.ts b/ui/lib/voice/tests/gain.test.ts
new file mode 100644
index 000000000..bb7293df9
--- /dev/null
+++ b/ui/lib/voice/tests/gain.test.ts
@@ -0,0 +1,219 @@
+// Guards how quiet speech is raised for transcription: only speech is measured, the gain
+// only raises and stops at its cap, loud peaks bend instead of clipping, and speech that
+// stays quiet even at the cap is told.
+
+import { VOICE_GAIN } from '../src/constants.js';
+import {
+ isTooQuiet,
+ measureSpeech,
+ normalizeSpeech,
+ softLimit,
+ SpeechGain,
+ speechGainFor
+} from '../src/gain.js';
+import { floatToPcm16, pcm16ToFloat } from '../src/pcm.js';
+import { describe, expect, it } from 'vitest';
+
+const RATE = 16_000;
+
+/** `ms` of a 220 Hz tone whose RMS is `level`, like a stretch of speech. */
+function tone(level: number, ms: number): Float32Array {
+ const out = new Float32Array(Math.round((RATE * ms) / 1000));
+ const amplitude = level * Math.SQRT2;
+
+ for (let i = 0; i < out.length; i++)
+ out[i] = amplitude * Math.sin((2 * Math.PI * 220 * i) / RATE);
+
+ return out;
+}
+
+/** `ms` of faint noise, a quiet room between words. */
+function room(ms: number, level = 0.0005): Float32Array {
+ const out = new Float32Array(Math.round((RATE * ms) / 1000));
+
+ // deterministic: the same "noise" every run
+ let seed = 7;
+
+ for (let i = 0; i < out.length; i++) {
+ seed = (seed * 16807) % 2147483647;
+ out[i] = ((seed / 2147483647) * 2 - 1) * level * Math.sqrt(3);
+ }
+
+ return out;
+}
+
+function join(...parts: Float32Array[]): Float32Array {
+ const out = new Float32Array(parts.reduce((n, part) => n + part.length, 0));
+
+ let offset = 0;
+
+ for (const part of parts) {
+ out.set(part, offset);
+ offset += part.length;
+ }
+
+ return out;
+}
+
+/** A recording: silence, speech at `level`, a pause, more speech, silence. */
+function recording(level: number): Float32Array {
+ return join(room(500), tone(level, 800), room(300), tone(level, 600), room(500));
+}
+
+function rmsOf(samples: Float32Array): number {
+ let sum = 0;
+
+ for (const sample of samples) sum += sample * sample;
+
+ return Math.sqrt(sum / samples.length);
+}
+
+describe('speechGainFor', () => {
+ it('raises quiet speech to the target and stops at the cap', () => {
+ expect(speechGainFor(VOICE_GAIN.TARGET_RMS / 4)).toBeCloseTo(4);
+ expect(speechGainFor(VOICE_GAIN.TARGET_RMS / 100)).toBe(VOICE_GAIN.MAX);
+ });
+
+ it('never lowers speech that is already loud enough', () => {
+ expect(speechGainFor(VOICE_GAIN.TARGET_RMS * 3)).toBe(1);
+ expect(speechGainFor(0)).toBe(1);
+ });
+});
+
+describe('softLimit', () => {
+ it('passes samples under the knee and bends louder ones below full scale', () => {
+ expect(softLimit(0.5)).toBe(0.5);
+ expect(softLimit(-0.5)).toBe(-0.5);
+ expect(softLimit(3)).toBeLessThan(1);
+ expect(softLimit(3)).toBeGreaterThan(VOICE_GAIN.KNEE);
+ expect(softLimit(-3)).toBeCloseTo(-softLimit(3));
+ });
+});
+
+describe('measureSpeech', () => {
+ it('measures the speech, not the silence around it', () => {
+ const speech = measureSpeech(recording(0.02), RATE);
+
+ expect(speech).toBeGreaterThan(0.018);
+ expect(speech).toBeLessThan(0.022);
+ });
+
+ it('finds no speech in a quiet room', () => {
+ expect(measureSpeech(room(2000), RATE)).toBe(0);
+ });
+});
+
+describe('normalizeSpeech', () => {
+ it('raises a quiet recording so its speech reaches the target', () => {
+ const result = normalizeSpeech(floatToPcm16(recording(0.02)), RATE);
+
+ expect(result.gain).toBeCloseTo(VOICE_GAIN.TARGET_RMS / 0.02, 0);
+ expect(result.tooQuiet).toBe(false);
+ expect(measureSpeech(pcm16ToFloat(result.samples), RATE)).toBeGreaterThan(0.08);
+ });
+
+ it('gives the first words the same gain as the rest', () => {
+ const result = normalizeSpeech(floatToPcm16(recording(0.02)), RATE);
+ const raised = pcm16ToFloat(result.samples);
+ // the first and the last 100 ms of speech
+ const first = raised.subarray(RATE * 0.5, RATE * 0.6);
+ const last = raised.subarray(RATE * 2.1, RATE * 2.2);
+
+ expect(rmsOf(first)).toBeCloseTo(rmsOf(last), 2);
+ });
+
+ it('leaves loud speech as it is', () => {
+ const samples = floatToPcm16(recording(0.2));
+ const result = normalizeSpeech(samples, RATE);
+
+ expect(result.gain).toBe(1);
+ expect(result.samples).toBe(samples);
+ });
+
+ it('stops at the cap and tells that the speech is still too quiet', () => {
+ const result = normalizeSpeech(floatToPcm16(recording(0.002)), RATE);
+
+ expect(result.gain).toBe(VOICE_GAIN.MAX);
+ expect(result.tooQuiet).toBe(true);
+ expect(isTooQuiet(0.002)).toBe(true);
+ });
+
+ it('does not clip a loud peak in quiet speech', () => {
+ const loudPeak = join(recording(0.02), tone(0.3, 50));
+ const raised = pcm16ToFloat(normalizeSpeech(floatToPcm16(loudPeak), RATE).samples);
+
+ expect(Math.max(...raised.map(Math.abs))).toBeLessThan(1);
+ });
+
+ it('leaves a recording with no speech alone', () => {
+ const samples = floatToPcm16(room(2000));
+
+ expect(normalizeSpeech(samples, RATE).samples).toBe(samples);
+ });
+});
+
+describe('SpeechGain', () => {
+ /** Feeds `samples` in 100 ms blocks; returns the gain after each. */
+ function feed(gain: SpeechGain, samples: Float32Array, apply = true): number[] {
+ const gains: number[] = [];
+
+ for (let i = 0; i < samples.length; i += RATE / 10) {
+ gain.process(samples.subarray(i, i + RATE / 10), apply);
+ gains.push(gain.gain);
+ }
+
+ return gains;
+ }
+
+ it('rises over a few blocks of quiet speech, and holds through the silence after', () => {
+ const gain = new SpeechGain();
+
+ feed(gain, room(500));
+ expect(gain.gain).toBe(1);
+
+ const rising = feed(gain, tone(0.02, 800));
+
+ expect(rising[0]).toBeGreaterThan(1);
+ expect(rising.at(-1)).toBeGreaterThan(rising[0]);
+ expect(rising.at(-1)).toBeGreaterThan(3.5);
+
+ const held = gain.gain;
+
+ feed(gain, room(500));
+ expect(gain.gain).toBe(held);
+ });
+
+ it('falls at once when the speech gets loud, so it does not clip', () => {
+ const gain = new SpeechGain();
+
+ feed(gain, join(room(300), tone(0.02, 800)));
+ expect(gain.gain).toBeGreaterThan(3);
+
+ const out = gain.process(tone(0.3, 100));
+
+ expect(gain.gain).toBeLessThan(3);
+ expect(Math.max(...out.map(Math.abs))).toBeLessThan(1);
+ });
+
+ it('only measures when asked not to apply', () => {
+ const gain = new SpeechGain();
+ const block = tone(0.02, 100);
+
+ feed(gain, room(300), false);
+
+ expect(gain.process(block, false)).toBe(block);
+ expect(gain.gain).toBeGreaterThan(1);
+ });
+
+ it('tells when speech stays too quiet even at the cap', () => {
+ const gain = new SpeechGain();
+
+ feed(gain, join(room(300), tone(0.025, 500)));
+ expect(gain.tooQuiet).toBe(false);
+
+ const faint = new SpeechGain();
+
+ feed(faint, join(room(300, 0.0001), tone(0.002, 500)));
+ expect(faint.tooQuiet).toBe(true);
+ });
+});
diff --git a/ui/lib/voice/tests/pauses.test.ts b/ui/lib/voice/tests/pauses.test.ts
new file mode 100644
index 000000000..daea87536
--- /dev/null
+++ b/ui/lib/voice/tests/pauses.test.ts
@@ -0,0 +1,64 @@
+// Guards where a recording is cut into pieces to transcribe while it goes on: at pauses
+// after speech, judged against the room's noise, and at the latest after a long stretch.
+
+import { PauseDetector } from '../src/pauses.js';
+import { describe, expect, it } from 'vitest';
+
+const INTERVAL = 50;
+
+/** Pushes `ms` worth of one level; true if any of them cut. */
+function hold(detector: PauseDetector, level: number, ms: number): boolean {
+ let cut = false;
+
+ for (let t = 0; t < ms; t += INTERVAL) cut = detector.push(level) || cut;
+
+ return cut;
+}
+
+describe('PauseDetector', () => {
+ it('cuts at a pause after speech', () => {
+ const detector = new PauseDetector({ intervalMs: INTERVAL });
+
+ expect(hold(detector, 0.1, 300)).toBe(false);
+ expect(hold(detector, 0.6, 600)).toBe(false);
+ expect(hold(detector, 0.1, 300)).toBe(false);
+ expect(hold(detector, 0.1, 300)).toBe(true);
+ });
+
+ it('does not cut silence alone, nor a short sound', () => {
+ const detector = new PauseDetector({ intervalMs: INTERVAL });
+
+ expect(hold(detector, 0.1, 2000)).toBe(false);
+ expect(hold(detector, 0.6, 100)).toBe(false);
+ expect(hold(detector, 0.1, 1000)).toBe(false);
+ });
+
+ it('follows the noise floor of the room', () => {
+ const detector = new PauseDetector({ intervalMs: INTERVAL });
+
+ // a loud room: its noise is not speech, and speech above it still counts
+ expect(hold(detector, 0.45, 500)).toBe(false);
+ expect(hold(detector, 0.8, 600)).toBe(false);
+ expect(hold(detector, 0.45, 600)).toBe(true);
+ });
+
+ it('cuts a long stretch of speech at its next quiet moment', () => {
+ const detector = new PauseDetector({ intervalMs: INTERVAL, maxSegmentMs: 2000 });
+
+ expect(hold(detector, 0.1, 100)).toBe(false);
+ expect(hold(detector, 0.6, 2500)).toBe(false);
+ expect(detector.push(0.1)).toBe(true);
+ });
+
+ it('cuts anyway when no quiet moment comes, as with music playing in the room', () => {
+ const detector = new PauseDetector({ intervalMs: INTERVAL, maxSegmentMs: 2000 });
+
+ expect(hold(detector, 0.1, 100)).toBe(false);
+ // speech over music: the level never comes back down
+ expect(hold(detector, 0.6, 2850)).toBe(false);
+ expect(detector.push(0.6)).toBe(true);
+ // and again for the next piece
+ expect(hold(detector, 0.6, 2950)).toBe(false);
+ expect(detector.push(0.6)).toBe(true);
+ });
+});
diff --git a/ui/lib/voice/tests/realtime.test.ts b/ui/lib/voice/tests/realtime.test.ts
new file mode 100644
index 000000000..5efe65a0a
--- /dev/null
+++ b/ui/lib/voice/tests/realtime.test.ts
@@ -0,0 +1,463 @@
+// Guards the realtime transcription client against llama-server's /v1/realtime
+// protocol (tools/server/README.md): the URL, the session flow, deltas and completed
+// events, the final commit, the "does not support" error, refused sockets and timeouts.
+
+import { RealtimeError, RealtimeUnsupportedError } from '../src/errors.js';
+import { base64ToBytes } from '../src/pcm.js';
+import { RealtimeTranscriber, realtimeUrl } from '../src/realtime.js';
+import { type VoiceServerContext } from '../src/types.js';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+class FakeWebSocket {
+ closed = false;
+
+ static instances: FakeWebSocket[] = [];
+ onclose: ((event: { code: number }) => void) | null = null;
+ onmessage: ((event: { data: string }) => void) | null = null;
+ sent: Record<string, unknown>[] = [];
+
+ close() {
+ this.closed = true;
+ }
+
+ constructor(public url: string) {
+ FakeWebSocket.instances.push(this);
+ }
+
+ /** The server (or the browser, on a refused upgrade) closes the socket. */
+ drop(code = 1006) {
+ this.onclose?.({ code });
+ }
+
+ /** An event from the server. */
+ emit(event: Record<string, unknown>) {
+ this.onmessage?.({ data: JSON.stringify(event) });
+ }
+
+ send(data: string) {
+ this.sent.push(JSON.parse(data));
+ }
+}
+
+const context: VoiceServerContext = {
+ baseHref: 'http://localhost:8080/',
+ resolveUrl: (path) => new URL(path, 'http://localhost:8080/').toString()
+};
+const sessionCreated = (type = 'transcription') => ({
+ session: { audio: { input: { format: { rate: 24_000, type: 'audio/pcm' } } }, type },
+ type: 'session.created'
+});
+
+function socket(): FakeWebSocket {
+ return FakeWebSocket.instances.at(-1)!;
+}
+
+beforeEach(() => {
+ vi.stubGlobal('WebSocket', FakeWebSocket);
+});
+
+afterEach(() => {
+ FakeWebSocket.instances = [];
+ vi.unstubAllGlobals();
+ vi.useRealTimers();
+});
+
+describe('realtimeUrl', () => {
+ it('turns the HTTP target into a websocket URL with the intent and model', () => {
+ expect(realtimeUrl(context, { model: 'org/model:Q4' })).toBe(
+ 'ws://localhost:8080/v1/realtime?intent=transcription&model=org%2Fmodel%3AQ4'
+ );
+ });
+
+ it('uses wss for https and resolves relative targets against the page', () => {
+ const relative: VoiceServerContext = {
+ baseHref: 'https://example.com/llama/',
+ resolveUrl: (path) => path
+ };
+
+ expect(realtimeUrl(relative, { autoload: false })).toBe(
+ 'wss://example.com/llama/v1/realtime?intent=transcription&autoload=false'
+ );
+ });
+});
+
+describe('RealtimeTranscriber', () => {
+ it('starts the session, sends the capture rate, and flushes audio queued while connecting', async () => {
+ const transcriber = new RealtimeTranscriber(context, { language: 'en', sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ transcriber.append(Int16Array.from([1, 2]));
+ expect(socket().sent).toEqual([]);
+
+ socket().emit(sessionCreated());
+ await connected;
+
+ expect(transcriber.ready).toBe(true);
+ expect(transcriber.live).toBe(true);
+ expect(transcriber.modelSampleRate).toBe(24_000);
+ expect(socket().sent[0]).toEqual({
+ session: {
+ audio: {
+ input: {
+ format: { rate: 16_000, type: 'audio/pcm' },
+ transcription: { language: 'en' }
+ }
+ },
+ type: 'transcription'
+ },
+ type: 'session.update'
+ });
+ expect(socket().sent[1].type).toBe('input_audio_buffer.append');
+ expect(Array.from(base64ToBytes(socket().sent[1].audio as string))).toEqual([1, 0, 2, 0]);
+
+ transcriber.append(Int16Array.from([3]));
+ expect(socket().sent).toHaveLength(3);
+ });
+
+ it('streams deltas, ends with a final commit and resolves with the completed transcript', async () => {
+ const onDelta = vi.fn();
+ const onCompleted = vi.fn();
+ const transcriber = new RealtimeTranscriber(context, {
+ onCompleted,
+ onDelta,
+ sampleRate: 16_000
+ });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated());
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+
+ socket().emit({ delta: 'Hello', type: 'conversation.item.input_audio_transcription.delta' });
+ socket().emit({ delta: ' world', type: 'conversation.item.input_audio_transcription.delta' });
+
+ expect(onDelta).toHaveBeenLastCalledWith(' world', 'Hello world');
+ expect(transcriber.text).toBe('Hello world');
+
+ const finished = transcriber.finish();
+
+ expect(socket().sent.at(-1)).toEqual({ final: true, type: 'input_audio_buffer.commit' });
+
+ socket().emit({
+ transcript: 'Hello world.',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+
+ await expect(finished).resolves.toBe('Hello world.');
+ expect(onCompleted).toHaveBeenCalledWith('Hello world.');
+ expect(socket().closed).toBe(true);
+ expect(transcriber.ready).toBe(false);
+ });
+
+ it('joins the items completed before the end', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated());
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+ socket().emit({
+ transcript: 'First line.\n',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+ socket().emit({ delta: 'Second', type: 'conversation.item.input_audio_transcription.delta' });
+
+ expect(transcriber.text).toBe('First line.\nSecond');
+
+ const finished = transcriber.finish();
+
+ socket().emit({
+ transcript: 'Second line.',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+
+ await expect(finished).resolves.toBe('First line.\nSecond line.');
+ });
+
+ it('shows the words alone while a model streams them in a format of its own', async () => {
+ const onDelta = vi.fn();
+ const transcriber = new RealtimeTranscriber(context, { onDelta, sampleRate: 16_000 });
+ const connected = transcriber.connect();
+ const delta = (text: string) =>
+ socket().emit({ delta: text, type: 'conversation.item.input_audio_transcription.delta' });
+ const completed = (transcript: string) =>
+ socket().emit({ transcript, type: 'conversation.item.input_audio_transcription.completed' });
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+ transcriber.commitSegment();
+
+ delta('[{"Start":0.0,"End":1.2,"Spe');
+ expect(transcriber.text).toBe('');
+ delta('aker":0,"Content":"Bonjour');
+ expect(onDelta).toHaveBeenLastCalledWith('aker":0,"Content":"Bonjour', 'Bonjour');
+ completed('[{"Start":0.0,"End":1.2,"Speaker":0,"Content":"Bonjour."}]');
+ expect(transcriber.text).toBe('Bonjour.');
+
+ // the next piece follows the first, a space between them
+ transcriber.append(Int16Array.from([2]));
+ transcriber.commitSegment();
+ delta('[{"Start":0.0,"End":0.9,"Speaker":0,"Content":"Ça va');
+ expect(transcriber.text).toBe('Bonjour. Ça va');
+
+ const finished = transcriber.finish({ commitRest: false });
+
+ completed('language French<asr_text>Ça va ?');
+ await expect(finished).resolves.toBe('Bonjour. Ça va ?');
+ });
+
+ it('transcribes a conversation model piece by piece, one at a time', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+ const commits = () =>
+ socket().sent.filter((event) => event.type === 'input_audio_buffer.commit');
+ const completed = (transcript: string) =>
+ socket().emit({ transcript, type: 'conversation.item.input_audio_transcription.completed' });
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+
+ // nothing to commit before any audio
+ expect(transcriber.canCommitSegment).toBe(false);
+ transcriber.append(Int16Array.from([1]));
+ transcriber.commitSegment();
+ expect(commits()).toEqual([{ type: 'input_audio_buffer.commit' }]);
+
+ // the next piece waits for the text of this one
+ transcriber.append(Int16Array.from([2]));
+ expect(transcriber.canCommitSegment).toBe(false);
+ completed('Hello there.');
+ expect(transcriber.text).toBe('Hello there.');
+ expect(transcriber.canCommitSegment).toBe(true);
+
+ transcriber.commitSegment();
+ transcriber.append(Int16Array.from([3]));
+
+ // the end waits for the piece in flight, then commits the rest
+ const finished = transcriber.finish();
+
+ await Promise.resolve();
+ expect(commits()).toHaveLength(2);
+ completed('How are you?');
+ await vi.waitFor(() => expect(commits()).toHaveLength(3));
+ expect(commits()[2]).toEqual({ final: true, type: 'input_audio_buffer.commit' });
+ completed('Fine.');
+
+ await expect(finished).resolves.toBe('Hello there. How are you? Fine.');
+ });
+
+ it('goes on with the next piece after one the server failed', async () => {
+ const onError = vi.fn();
+ const transcriber = new RealtimeTranscriber(context, { onError, sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+ transcriber.commitSegment();
+ socket().emit({ error: { message: 'decode failed' }, type: 'error' });
+
+ expect(onError).toHaveBeenCalledOnce();
+ transcriber.append(Int16Array.from([2]));
+ expect(transcriber.canCommitSegment).toBe(true);
+ });
+
+ it('ends without a final commit when every piece was committed already', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+ transcriber.commitSegment();
+ socket().emit({
+ transcript: 'Only this.',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+
+ await expect(transcriber.finish()).resolves.toBe('Only this.');
+ expect(
+ socket().sent.filter((event) => event.type === 'input_audio_buffer.commit')
+ ).toHaveLength(1);
+ });
+
+ it('leaves out a silent rest after the last piece, but never the only one', async () => {
+ const commits = () =>
+ socket().sent.filter((event) => event.type === 'input_audio_buffer.commit');
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+ transcriber.commitSegment();
+ socket().emit({
+ transcript: 'All said.',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+ // silence after the last piece: a model may hear "yeah" in it
+ transcriber.append(Int16Array.from([0]));
+
+ await expect(transcriber.finish({ commitRest: false })).resolves.toBe('All said.');
+ expect(commits()).toHaveLength(1);
+
+ // with nothing transcribed yet, the rest is all there is: it is committed
+ const second = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const secondConnected = second.connect();
+
+ socket().emit(sessionCreated('realtime'));
+ await secondConnected;
+ second.append(Int16Array.from([1]));
+
+ const finished = second.finish({ commitRest: false });
+
+ await vi.waitFor(() => expect(commits()).toHaveLength(1));
+ socket().emit({
+ transcript: 'Short.',
+ type: 'conversation.item.input_audio_transcription.completed'
+ });
+ await expect(finished).resolves.toBe('Short.');
+ });
+
+ it('marks a conversation session switched to transcription as not live', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated('realtime'));
+ await connected;
+
+ expect(transcriber.live).toBe(false);
+ expect((socket().sent[0].session as { type: string }).type).toBe('transcription');
+ });
+
+ it('rejects with RealtimeUnsupportedError when the model cannot stream', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit({
+ error: {
+ code: null,
+ message: 'The current model does not support realtime transcription.',
+ type: 'invalid_request_error'
+ },
+ type: 'error'
+ });
+ socket().drop(1000);
+
+ const error = await connected.catch((e) => e);
+
+ expect(error).toBeInstanceOf(RealtimeUnsupportedError);
+ expect(error.reason).toBe('model');
+ expect(socket().closed).toBe(true);
+ });
+
+ it('rejects with a connection RealtimeUnsupportedError when the upgrade is refused', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ // e.g. a 401 from a server with an API key: the browser only reports a close
+ socket().drop(1006);
+
+ const error = await connected.catch((e) => e);
+
+ expect(error).toBeInstanceOf(RealtimeUnsupportedError);
+ expect(error.reason).toBe('connection');
+ });
+
+ it('passes other errors before the session through as RealtimeError', async () => {
+ const transcriber = new RealtimeTranscriber(context, { model: 'x', sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit({
+ error: { message: 'model name=x is not running', type: 'server_error' },
+ type: 'error'
+ });
+
+ const error = await connected.catch((e) => e);
+
+ expect(error).toBeInstanceOf(RealtimeError);
+ expect(error).not.toBeInstanceOf(RealtimeUnsupportedError);
+ expect(error.message).toBe('model name=x is not running');
+ });
+
+ it('times out a session that never starts', async () => {
+ vi.useFakeTimers();
+
+ const transcriber = new RealtimeTranscriber(context, {
+ connectTimeoutMs: 1000,
+ sampleRate: 16_000
+ });
+ const connected = transcriber.connect().catch((e) => e);
+
+ vi.advanceTimersByTime(1000);
+
+ expect(await connected).toBeInstanceOf(RealtimeError);
+ expect(socket().closed).toBe(true);
+ });
+
+ it('times out a transcript that never completes', async () => {
+ vi.useFakeTimers();
+
+ const transcriber = new RealtimeTranscriber(context, {
+ finishTimeoutMs: 500,
+ sampleRate: 16_000
+ });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated());
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+
+ const finished = transcriber.finish().catch((e) => e);
+
+ vi.advanceTimersByTime(500);
+
+ const error = await finished;
+
+ expect(error).toBeInstanceOf(RealtimeError);
+ expect(error.message).toMatch(/did not complete/);
+ });
+
+ it('rejects finish when the socket closes or errors before the transcript', async () => {
+ const onError = vi.fn();
+ const transcriber = new RealtimeTranscriber(context, { onError, sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated());
+ await connected;
+ transcriber.append(Int16Array.from([1]));
+
+ const finished = transcriber.finish();
+
+ socket().emit({ error: { message: 'server error' }, type: 'error' });
+
+ await expect(finished).rejects.toThrow('server error');
+ expect(onError).toHaveBeenCalledOnce();
+ });
+
+ it('resolves an empty transcript without a commit when no audio was sent', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect();
+
+ socket().emit(sessionCreated());
+ await connected;
+
+ await expect(transcriber.finish()).resolves.toBe('');
+ expect(socket().sent.map((event) => event.type)).toEqual(['session.update']);
+ expect(socket().closed).toBe(true);
+ });
+
+ it('close() rejects a pending connect and stops sending', async () => {
+ const transcriber = new RealtimeTranscriber(context, { sampleRate: 16_000 });
+ const connected = transcriber.connect().catch((e) => e);
+ const ws = socket();
+
+ transcriber.close();
+ transcriber.append(Int16Array.from([1]));
+
+ expect(await connected).toBeInstanceOf(RealtimeError);
+ expect(ws.closed).toBe(true);
+ expect(ws.sent).toEqual([]);
+ });
+});
diff --git a/ui/lib/voice/tests/session.test.ts b/ui/lib/voice/tests/session.test.ts
new file mode 100644
index 000000000..42eed104b
--- /dev/null
+++ b/ui/lib/voice/tests/session.test.ts
@@ -0,0 +1,355 @@
+// Guards the voice session: audio goes to the realtime socket while recording and the
+// final transcript comes from it; when the model cannot stream, the session falls back
+// to one HTTP transcription of the clip without telling the user anything went wrong.
+
+import { type VoiceCapture } from '../src/capture.js';
+import { startVoiceSession } from '../src/session.js';
+import { type VoiceClip, type VoicePcmListener, type VoiceServerContext } from '../src/types.js';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+class FakeWebSocket {
+ closed = false;
+
+ static instances: FakeWebSocket[] = [];
+ onclose: ((event: { code: number }) => void) | null = null;
+ onmessage: ((event: { data: string }) => void) | null = null;
+ sent: Record<string, unknown>[] = [];
+
+ close() {
+ this.closed = true;
+ }
+
+ constructor(public url: string) {
+ FakeWebSocket.instances.push(this);
+ }
+
+ emit(event: Record<string, unknown>) {
+ this.onmessage?.({ data: JSON.stringify(event) });
+ }
+
+ send(data: string) {
+ this.sent.push(JSON.parse(data));
+ }
+}
+
+const clip: VoiceClip = {
+ blob: new Blob([new Uint8Array(44)], { type: 'audio/wav' }),
+ durationMs: 1000,
+ peaks: [0, 1],
+ sampleRate: 16_000
+};
+
+function fakeCapture() {
+ const pcm = new Set<VoicePcmListener>();
+ const levels = new Set<(levels: number[]) => void>();
+ const capture = {
+ cancel: vi.fn(),
+ levels: [0, 0],
+ levelsIntervalMs: 50,
+ onLevels: vi.fn((listener: (levels: number[]) => void) => {
+ levels.add(listener);
+
+ return () => levels.delete(listener);
+ }),
+ onPcm: (listener: VoicePcmListener) => {
+ pcm.add(listener);
+
+ return () => pcm.delete(listener);
+ },
+ sampleRate: 16_000,
+ start: vi.fn(async () => {}),
+ stop: vi.fn(async () => {
+ // the last chunk reaches the listeners before stop resolves, as in VoiceCapture
+ for (const listener of pcm) listener(Int16Array.from([9]));
+
+ return clip;
+ })
+ };
+
+ return {
+ capture: capture as unknown as VoiceCapture,
+ emitPcm: (chunk: Int16Array) => pcm.forEach((listener) => listener(chunk)),
+ /** Holds one input level for `ms`, as the capture reports it every 50 ms. */
+ holdLevel: (level: number, ms: number) => {
+ for (let t = 0; t < ms; t += 50) levels.forEach((listener) => listener([level]));
+ },
+ raw: capture
+ };
+}
+
+function serverContext(fetchImpl: typeof fetch): VoiceServerContext {
+ return {
+ baseHref: 'http://localhost/',
+ fetch: fetchImpl,
+ resolveUrl: (path) => new URL(path, 'http://localhost/').toString()
+ };
+}
+
+const flush = () => new Promise((resolve) => setTimeout(resolve, 0));
+
+beforeEach(() => {
+ vi.stubGlobal('WebSocket', FakeWebSocket);
+});
+
+afterEach(() => {
+ FakeWebSocket.instances = [];
+ vi.unstubAllGlobals();
+});
+
+describe('startVoiceSession', () => {
+ it('records only when not asked to transcribe', async () => {
+ const { capture } = fakeCapture();
+ const session = await startVoiceSession({ capture });
+
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ await expect(session.stop()).resolves.toEqual({ clip, liveText: null, transcript: null });
+ });
+
+ it('streams to the realtime socket and takes its transcript', async () => {
+ const fetchMock = vi.fn();
+ const onText = vi.fn();
+ const onRealtime = vi.fn();
+ const { capture, emitPcm } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ onRealtime,
+ onText,
+ transcribe: { context: serverContext(fetchMock as typeof fetch), model: 'asr' }
+ });
+ const ws = FakeWebSocket.instances[0];
+
+ expect(ws.url).toBe('ws://localhost/v1/realtime?intent=transcription&model=asr');
+
+ emitPcm(Int16Array.from([1]));
+ ws.emit({ session: { type: 'transcription' }, type: 'session.created' });
+ await flush();
+
+ expect(onRealtime).toHaveBeenCalledWith(true);
+
+ ws.emit({ delta: 'Hi', type: 'conversation.item.input_audio_transcription.delta' });
+ expect(onText).toHaveBeenLastCalledWith('Hi');
+
+ const result = await session.stop();
+
+ expect(result.clip).toBe(clip);
+ expect(result.liveText).toBe('Hi');
+ // the queued chunk, the last chunk from stop(), then the final commit
+ expect(ws.sent.map((event) => event.type)).toEqual([
+ 'session.update',
+ 'input_audio_buffer.append',
+ 'input_audio_buffer.append',
+ 'input_audio_buffer.commit'
+ ]);
+
+ ws.emit({ transcript: 'Hi.', type: 'conversation.item.input_audio_transcription.completed' });
+
+ await expect(result.transcript).resolves.toBe('Hi.');
+ expect(fetchMock).not.toHaveBeenCalled();
+ });
+
+ describe('a recording transcribed in pieces', () => {
+ const COMPLETED = 'conversation.item.input_audio_transcription.completed';
+
+ /** Speaks two pieces with a pause between them and stops. */
+ async function speakTwoPieces(fetchMock: ReturnType<typeof vi.fn>) {
+ const onText = vi.fn();
+ const { capture, emitPcm, holdLevel } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ onText,
+ transcribe: { context: serverContext(fetchMock as unknown as typeof fetch) }
+ });
+ const ws = FakeWebSocket.instances[0];
+
+ // a model that holds conversations transcribes per commit
+ ws.emit({ session: { type: 'realtime' }, type: 'session.created' });
+ await flush();
+
+ emitPcm(Int16Array.from([1]));
+ holdLevel(0.05, 200);
+ holdLevel(0.6, 500);
+ holdLevel(0.05, 400);
+
+ expect(ws.sent.filter((event) => event.type === 'input_audio_buffer.commit')).toHaveLength(1);
+
+ ws.emit({ transcript: 'I went.', type: COMPLETED });
+ expect(onText).toHaveBeenLastCalledWith('I went.');
+
+ emitPcm(Int16Array.from([2]));
+ holdLevel(0.6, 500);
+
+ const result = await session.stop();
+
+ ws.emit({ transcript: 'To the store.', type: COMPLETED });
+
+ return result;
+ }
+
+ it('is transcribed once more as a whole, punctuated by meaning', async () => {
+ const fetchMock = vi.fn(
+ async () => new Response(JSON.stringify({ text: 'I went to the store.' }))
+ );
+ const result = await speakTwoPieces(fetchMock);
+
+ expect(result.liveText).toBe('I went.');
+ await expect(result.transcript).resolves.toBe('I went to the store.');
+ expect(fetchMock).toHaveBeenCalledOnce();
+ });
+
+ it('keeps its pieces when the whole pass fails', async () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const fetchMock = vi.fn(async () => new Response('busy', { status: 503 }));
+ const result = await speakTwoPieces(fetchMock);
+
+ await expect(result.transcript).resolves.toBe('I went. To the store.');
+ warn.mockRestore();
+ });
+ });
+
+ it('falls back to HTTP when the model cannot stream', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ text: 'From HTTP.' })));
+ const onRealtime = vi.fn();
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const { capture } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ onRealtime,
+ transcribe: { context: serverContext(fetchMock as typeof fetch), model: 'omni' }
+ });
+
+ FakeWebSocket.instances[0].emit({
+ error: { message: 'The current model does not support realtime transcription.' },
+ type: 'error'
+ });
+ await flush();
+
+ expect(onRealtime).toHaveBeenCalledWith(false);
+ // an expected fallback, not worth a warning
+ expect(warn).not.toHaveBeenCalled();
+
+ const result = await session.stop();
+
+ expect(result.liveText).toBeNull();
+ await expect(result.transcript).resolves.toBe('From HTTP.');
+ expect(fetchMock).toHaveBeenCalledOnce();
+
+ const form = (fetchMock.mock.calls[0] as unknown as [string, RequestInit])[1].body as FormData;
+
+ expect(form.get('model')).toBe('omni');
+ warn.mockRestore();
+ });
+
+ describe('a model that cannot stream (Qwen3-ASR)', () => {
+ /** Answers a piece with "Piece N." and the whole clip with `whole`, or fails it. */
+ function server(whole: string | null) {
+ let pieces = 0;
+
+ return vi.fn(async (_url: string, init: RequestInit) => {
+ const file = (init.body as FormData).get('file') as File;
+
+ if (file.name === 'piece.wav') {
+ return new Response(JSON.stringify({ text: `Piece ${++pieces}.` }));
+ }
+
+ return whole === null
+ ? new Response('busy', { status: 503 })
+ : new Response(JSON.stringify({ text: whole }));
+ });
+ }
+
+ /** Speaks two pieces with a pause after each, over HTTP once the socket says no. */
+ async function speakOverHttp(fetchMock: ReturnType<typeof server>) {
+ const onText = vi.fn();
+ const { capture, emitPcm, holdLevel } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ onText,
+ transcribe: { context: serverContext(fetchMock as unknown as typeof fetch), model: 'asr' }
+ });
+
+ // said before the socket answered: kept for the first piece
+ emitPcm(Int16Array.from([1, 2]));
+ FakeWebSocket.instances[0].emit({
+ error: { message: 'The current model does not support realtime transcription.' },
+ type: 'error'
+ });
+ await flush();
+
+ holdLevel(0.05, 200);
+ holdLevel(0.6, 500);
+ holdLevel(0.05, 400);
+ await flush();
+ expect(onText).toHaveBeenLastCalledWith('Piece 1.');
+
+ emitPcm(Int16Array.from([3]));
+ holdLevel(0.6, 500);
+ holdLevel(0.05, 400);
+ await flush();
+ expect(onText).toHaveBeenLastCalledWith('Piece 1. Piece 2.');
+
+ return session.stop();
+ }
+
+ it('gives live text piece by piece over HTTP, then the whole clip', async () => {
+ const fetchMock = server('Whole.');
+ const result = await speakOverHttp(fetchMock);
+
+ expect(result.liveText).toBe('Piece 1. Piece 2.');
+ await expect(result.transcript).resolves.toBe('Whole.');
+ // two pieces and the whole clip
+ expect(fetchMock).toHaveBeenCalledTimes(3);
+
+ const first = (fetchMock.mock.calls[0][1].body as FormData).get('file') as File;
+
+ // the audio from before the socket answered went with the first piece
+ expect(first.size).toBeGreaterThan(44);
+ });
+
+ it('keeps the pieces when the whole clip fails', async () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const result = await speakOverHttp(server(null));
+
+ await expect(result.transcript).resolves.toBe('Piece 1. Piece 2.');
+ warn.mockRestore();
+ });
+ });
+
+ it('uses HTTP when realtime is still connecting at the end', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ text: 'Late.' })));
+ const { capture } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ transcribe: { context: serverContext(fetchMock as typeof fetch) }
+ });
+ const result = await session.stop();
+
+ expect(FakeWebSocket.instances[0].closed).toBe(true);
+ await expect(result.transcript).resolves.toBe('Late.');
+ });
+
+ it('skips the socket when realtime is off', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ text: 'ok' })));
+ const { capture } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ realtime: false,
+ transcribe: { context: serverContext(fetchMock as typeof fetch) }
+ });
+
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ await expect((await session.stop()).transcript).resolves.toBe('ok');
+ });
+
+ it('cancel drops the capture and the socket', async () => {
+ const { capture, raw } = fakeCapture();
+ const session = await startVoiceSession({
+ capture,
+ transcribe: { context: serverContext(vi.fn() as unknown as typeof fetch) }
+ });
+
+ session.cancel();
+
+ expect(raw.cancel).toHaveBeenCalledOnce();
+ expect(FakeWebSocket.instances[0].closed).toBe(true);
+ });
+});
diff --git a/ui/lib/voice/tests/speech-detector.test.ts b/ui/lib/voice/tests/speech-detector.test.ts
new file mode 100644
index 000000000..4a2ae195c
--- /dev/null
+++ b/ui/lib/voice/tests/speech-detector.test.ts
@@ -0,0 +1,73 @@
+// Guards the one place the engine tells speech from the room: against a noise floor that
+// drops at once and creeps back up, above an absolute minimum, with a stricter threshold
+// for a listener that asks for one.
+
+import { VOICE_SPEECH } from '../src/constants.js';
+import { levelToDb, rmsToDb } from '../src/levels.js';
+import { isSpeechLevel, SpeechDetector } from '../src/speech-detector.js';
+import { describe, expect, it } from 'vitest';
+
+/** Pushes `ms` of one level in 50 ms steps; true if the last one was speech. */
+function hold(detector: SpeechDetector, db: number, ms: number): boolean {
+ let speech = false;
+
+ for (let t = 0; t < ms; t += 50) speech = detector.push(db, 50);
+
+ return speech;
+}
+
+describe('SpeechDetector', () => {
+ it('hears speech above a quiet room, and not the room', () => {
+ const detector = new SpeechDetector();
+
+ expect(hold(detector, -60, 500)).toBe(false);
+ expect(hold(detector, -35, 300)).toBe(true);
+ expect(detector.isQuiet(-58)).toBe(true);
+ });
+
+ it('follows a loud room: its noise is not speech, but speech over it is', () => {
+ const detector = new SpeechDetector();
+
+ expect(hold(detector, -30, 1000)).toBe(false);
+ expect(hold(detector, -18, 300)).toBe(true);
+ });
+
+ it('lets the floor creep up through a long sound, so a steady one stops counting', () => {
+ const detector = new SpeechDetector();
+
+ hold(detector, -60, 200);
+
+ const riseMs = ((VOICE_SPEECH.ABOVE_FLOOR_DB + 10) / VOICE_SPEECH.FLOOR_RISE_DB_PER_S) * 1000;
+
+ expect(hold(detector, -41, 300)).toBe(true);
+ expect(hold(detector, -41, riseMs)).toBe(false);
+ });
+
+ it('starts from a known floor, so the first level can be speech', () => {
+ expect(new SpeechDetector({ floorDb: -50 }).push(-35, 50)).toBe(true);
+ expect(new SpeechDetector().push(-35, 50)).toBe(false);
+ });
+
+ it('takes a stricter threshold when asked', () => {
+ const strict = new SpeechDetector({ floorDb: -50, speechDb: 15 });
+
+ expect(strict.isSpeech(-40)).toBe(false);
+ expect(strict.isSpeech(-30)).toBe(true);
+ });
+
+ it('never hears speech below the absolute minimum, however quiet the room', () => {
+ expect(isSpeechLevel(VOICE_SPEECH.MIN_DB - 1, -120)).toBe(false);
+ expect(isSpeechLevel(VOICE_SPEECH.MIN_DB + 1, -120)).toBe(true);
+ });
+});
+
+describe('dB helpers', () => {
+ it('turns RMS and 0..1 levels into dBFS', () => {
+ expect(rmsToDb(1)).toBe(0);
+ expect(rmsToDb(0.1)).toBeCloseTo(-20);
+ expect(rmsToDb(0)).toBeLessThan(-100);
+ expect(levelToDb(1)).toBeCloseTo(0);
+ expect(levelToDb(0.5)).toBe(-30);
+ expect(levelToDb(0)).toBe(-60);
+ });
+});
diff --git a/ui/lib/voice/tests/transcribe.test.ts b/ui/lib/voice/tests/transcribe.test.ts
new file mode 100644
index 000000000..582392052
--- /dev/null
+++ b/ui/lib/voice/tests/transcribe.test.ts
@@ -0,0 +1,129 @@
+// Guards the HTTP transcription client: the multipart request
+// /v1/audio/transcriptions expects, auth headers, and the server's error messages.
+
+import { VoiceTranscriptionError } from '../src/errors.js';
+import { transcribeClip } from '../src/transcribe.js';
+import { type VoiceServerContext } from '../src/types.js';
+import { describe, expect, it, vi } from 'vitest';
+
+function contextWith(fetchImpl: typeof fetch): VoiceServerContext {
+ return {
+ fetch: fetchImpl,
+ getHeaders: () => ({ Authorization: 'Bearer secret', 'Content-Type': 'application/json' }),
+ resolveUrl: (path) => `http://server${path.replace(/^\./, '')}`
+ };
+}
+
+describe('transcribeClip', () => {
+ it('posts the clip as multipart with the model and json format', async () => {
+ const fetchMock = vi.fn(
+ async (_url: RequestInfo | URL, _init?: RequestInit) =>
+ new Response(JSON.stringify({ text: ' Hello there. ' }), { status: 200 })
+ );
+ const blob = new Blob([new Uint8Array([1, 2, 3])], { type: 'audio/wav' });
+ const text = await transcribeClip(contextWith(fetchMock as typeof fetch), blob, {
+ language: 'en',
+ model: 'org/model:Q8_0'
+ });
+
+ expect(text).toBe('Hello there.');
+
+ const [url, init] = fetchMock.mock.calls[0];
+ const form = init!.body as FormData;
+ const file = form.get('file') as File;
+
+ expect(url).toBe('http://server/v1/audio/transcriptions');
+ expect(init!.method).toBe('POST');
+ // the browser sets the multipart Content-Type with its boundary
+ expect(init!.headers).toEqual({ Authorization: 'Bearer secret' });
+ expect(file.name).toBe('audio.wav');
+ expect(file.type).toBe('audio/wav');
+ expect(new Uint8Array(await file.arrayBuffer())).toEqual(new Uint8Array([1, 2, 3]));
+ expect(form.get('model')).toBe('org/model:Q8_0');
+ expect(form.get('language')).toBe('en');
+ expect(form.get('response_format')).toBe('json');
+ expect(form.has('prompt')).toBe(false);
+ });
+
+ it('returns the words alone when the model wraps them in a format of its own', async () => {
+ const answers = [
+ 'language French<asr_text>Bonjour.',
+ '[{"Start":0.0,"End":3.1,"Speaker":0,"Content":"Bonjour."}]'
+ ];
+
+ for (const answer of answers) {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ text: answer })));
+
+ await expect(
+ transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([]))
+ ).resolves.toBe('Bonjour.');
+ }
+ });
+
+ it('leaves the model out for a single-model server', async () => {
+ const fetchMock = vi.fn(
+ async (_url: RequestInfo | URL, _init?: RequestInit) =>
+ new Response(JSON.stringify({ text: 'ok' }))
+ );
+
+ await transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([]));
+
+ expect((fetchMock.mock.calls[0][1]!.body as FormData).has('model')).toBe(false);
+ });
+
+ it("surfaces the server's error message and status", async () => {
+ const fetchMock = vi.fn(
+ async () =>
+ new Response(
+ JSON.stringify({
+ error: {
+ code: 501,
+ message: 'The current model does not support audio input.',
+ type: 'not_supported_error'
+ }
+ }),
+ { status: 501 }
+ )
+ );
+ const error = await transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([])).catch(
+ (e) => e
+ );
+
+ expect(error).toBeInstanceOf(VoiceTranscriptionError);
+ expect(error.message).toBe('The current model does not support audio input.');
+ expect(error.status).toBe(501);
+ });
+
+ it('falls back to the status for bodies that say nothing useful', async () => {
+ const fetchMock = vi.fn(
+ async () =>
+ new Response('<html>Bad Gateway</html>', { status: 502, statusText: 'Bad Gateway' })
+ );
+
+ await expect(
+ transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([]))
+ ).rejects.toThrow('Transcription failed (502 Bad Gateway)');
+ });
+
+ it('rejects a response without text', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ choices: [] })));
+
+ await expect(
+ transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([]))
+ ).rejects.toBeInstanceOf(VoiceTranscriptionError);
+ });
+
+ it('passes the abort signal on', async () => {
+ const controller = new AbortController();
+ const fetchMock = vi.fn(
+ async (_url: RequestInfo | URL, _init?: RequestInit) =>
+ new Response(JSON.stringify({ text: '' }))
+ );
+
+ await transcribeClip(contextWith(fetchMock as typeof fetch), new Blob([]), {
+ signal: controller.signal
+ });
+
+ expect(fetchMock.mock.calls[0][1]!.signal).toBe(controller.signal);
+ });
+});
diff --git a/ui/lib/voice/tests/transcript.test.ts b/ui/lib/voice/tests/transcript.test.ts
new file mode 100644
index 000000000..92cb1b871
--- /dev/null
+++ b/ui/lib/voice/tests/transcript.test.ts
@@ -0,0 +1,53 @@
+// Guards that a transcript is the speaker's words alone: the formats speech-to-text models
+// wrap them in (Qwen3-ASR's language tag, VibeVoice-ASR's timed segments) are taken off,
+// also while the text still streams, and plain text is left as it is.
+
+import { cleanTranscript } from '../src/transcript.js';
+import { describe, expect, it } from 'vitest';
+
+describe('cleanTranscript', () => {
+ it('leaves plain text as it is, spaces included', () => {
+ expect(cleanTranscript(' Hello there. ')).toBe(' Hello there. ');
+ expect(cleanTranscript('')).toBe('');
+ // a bracket or the word "language" alone is not a format
+ expect(cleanTranscript('[laughs] language matters')).toBe('[laughs] language matters');
+ });
+
+ it('takes off the language and tag Qwen3-ASR opens with', () => {
+ expect(cleanTranscript('language French<asr_text>Bonjour à tous.')).toBe('Bonjour à tous.');
+ expect(cleanTranscript('<asr_text>Hello.</asr_text>')).toBe('Hello.');
+ // silence: the model names no language and says nothing
+ expect(cleanTranscript('language None<asr_text>')).toBe('');
+ });
+
+ it('cleans the same text again and again', () => {
+ // the tag pattern is shared and global: it must keep no position between calls
+ for (let i = 0; i < 3; i++) {
+ expect(cleanTranscript('language English<asr_text>Hi.')).toBe('Hi.');
+ }
+ });
+
+ it('joins the words of the timed segments VibeVoice-ASR answers with', () => {
+ const segments = JSON.stringify([
+ { Content: 'Bonjour, ça va ?', End: 3.1, Speaker: 0, Start: 0 },
+ { Content: ' Oui, "très" bien.', End: 5.4, Speaker: 1, Start: 3.1 }
+ ]);
+
+ expect(cleanTranscript(segments)).toBe('Bonjour, ça va ? Oui, "très" bien.');
+ expect(cleanTranscript('[{"Start":0.0,"End":1.0,"Speaker":0,"Content":""}]')).toBe('');
+ });
+
+ it('shows only the words of segments that are still streaming', () => {
+ expect(cleanTranscript('[')).toBe('');
+ expect(cleanTranscript('[{"Start":0.0,"End":3.1,"Spe')).toBe('');
+ expect(cleanTranscript('[{"Start":0.0,"End":3.1,"Speaker":0,"Content":"Bonjour, ça')).toBe(
+ 'Bonjour, ça'
+ );
+ // cut inside an escape: the half of it is left out
+ expect(cleanTranscript('[{"Start":0,"Content":"Line one\\')).toBe('Line one');
+ expect(cleanTranscript('[{"Start":0,"Content":"caf\\u00')).toBe('caf');
+ expect(cleanTranscript('[{"Start":0,"Content":"caf\\u00e9"},{"Start":1,"Content":"oui')).toBe(
+ 'café oui'
+ );
+ });
+});
diff --git a/ui/lib/voice/tests/warm-microphone.test.ts b/ui/lib/voice/tests/warm-microphone.test.ts
new file mode 100644
index 000000000..931235f11
--- /dev/null
+++ b/ui/lib/voice/tests/warm-microphone.test.ts
@@ -0,0 +1,121 @@
+// Guards opening the microphone ahead of a recording: only when the page may already use
+// it, handed over to a recording with the same options, and let go otherwise.
+
+import { VOICE_MICROPHONE_CONSTRAINTS } from '../src/constants.js';
+import { microphoneRequest } from '../src/microphone.js';
+import { MicrophoneWarmup } from '../src/warm-microphone.js';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+interface FakeTrack {
+ readyState: 'live' | 'ended';
+ stop: () => void;
+}
+
+function fakeStream(): MediaStream & { track: FakeTrack } {
+ const track: FakeTrack = {
+ readyState: 'live',
+ stop() {
+ track.readyState = 'ended';
+ }
+ };
+
+ return {
+ getAudioTracks: () => [track],
+ getTracks: () => [track],
+ track
+ } as unknown as MediaStream & { track: FakeTrack };
+}
+
+let permission: PermissionState;
+let getUserMedia: ReturnType<typeof vi.fn>;
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ permission = 'granted';
+ getUserMedia = vi.fn(async () => fakeStream());
+ vi.stubGlobal('navigator', {
+ mediaDevices: { getUserMedia },
+ permissions: { query: async () => ({ state: permission }) }
+ });
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ vi.unstubAllGlobals();
+});
+
+describe('microphoneRequest', () => {
+ it('asks for the raw signal, the caller’s processing on top, from the picked device', () => {
+ expect(microphoneRequest({})).toEqual({ audio: VOICE_MICROPHONE_CONSTRAINTS });
+ expect(microphoneRequest({ constraints: { echoCancellation: true }, deviceId: 'usb' })).toEqual(
+ {
+ audio: { ...VOICE_MICROPHONE_CONSTRAINTS, deviceId: 'usb', echoCancellation: true }
+ }
+ );
+ });
+});
+
+describe('MicrophoneWarmup', () => {
+ it('hands the open microphone to a recording with the same options', async () => {
+ const warmup = new MicrophoneWarmup();
+
+ warmup.open({ deviceId: 'usb' });
+
+ const stream = (await warmup.take({ deviceId: 'usb' })) as MediaStream & { track: FakeTrack };
+
+ expect(getUserMedia).toHaveBeenCalledOnce();
+ expect(stream.track.readyState).toBe('live');
+ // handed over: nothing is left to take
+ expect(await warmup.take({ deviceId: 'usb' })).toBeNull();
+ });
+
+ it('opens it once for the same options, however often the pointer comes back', async () => {
+ const warmup = new MicrophoneWarmup();
+
+ warmup.open();
+ warmup.release();
+ warmup.open();
+
+ expect(await warmup.take()).not.toBeNull();
+ expect(getUserMedia).toHaveBeenCalledOnce();
+ });
+
+ it('never asks for access: without it, nothing opens', async () => {
+ permission = 'prompt';
+
+ const warmup = new MicrophoneWarmup();
+
+ warmup.open();
+
+ expect(await warmup.take()).toBeNull();
+ expect(getUserMedia).not.toHaveBeenCalled();
+ });
+
+ it('lets go of a microphone opened for other options', async () => {
+ const warmup = new MicrophoneWarmup();
+ const opened = fakeStream();
+
+ getUserMedia.mockResolvedValueOnce(opened);
+ warmup.open({ deviceId: 'usb' });
+
+ expect(await warmup.take({ deviceId: 'built-in' })).toBeNull();
+ expect(opened.track.readyState).toBe('ended');
+ });
+
+ it('lets go a moment after release, unless a recording takes it first', async () => {
+ const warmup = new MicrophoneWarmup();
+ const opened = fakeStream();
+
+ getUserMedia.mockResolvedValueOnce(opened);
+ warmup.open();
+ await Promise.resolve();
+ warmup.release(1000);
+
+ await vi.advanceTimersByTimeAsync(999);
+ expect(opened.track.readyState).toBe('live');
+
+ await vi.advanceTimersByTimeAsync(1);
+ expect(opened.track.readyState).toBe('ended');
+ expect(await warmup.take()).toBeNull();
+ });
+});
diff --git a/ui/lib/voice/tsconfig.build.json b/ui/lib/voice/tsconfig.build.json
new file mode 100644
index 000000000..cb6a3121d
--- /dev/null
+++ b/ui/lib/voice/tsconfig.build.json
@@ -0,0 +1,10 @@
+{
+ "extends": "./tsconfig.json",
+ "compilerOptions": {
+ "rootDir": "src",
+ "outDir": "dist",
+ "declaration": true,
+ "sourceMap": true
+ },
+ "include": ["src"]
+}
diff --git a/ui/lib/voice/tsconfig.json b/ui/lib/voice/tsconfig.json
new file mode 100644
index 000000000..5bef2d7ae
--- /dev/null
+++ b/ui/lib/voice/tsconfig.json
@@ -0,0 +1,13 @@
+{
+ "compilerOptions": {
+ "target": "ES2022",
+ "lib": ["ES2022", "DOM", "DOM.Iterable"],
+ "module": "NodeNext",
+ "moduleResolution": "NodeNext",
+ "strict": true,
+ "verbatimModuleSyntax": true,
+ "skipLibCheck": true,
+ "types": ["node", "svelte"]
+ },
+ "include": ["src", "tests"]
+}
diff --git a/ui/lib/voice/vitest.config.ts b/ui/lib/voice/vitest.config.ts
new file mode 100644
index 000000000..d49f1d4d6
--- /dev/null
+++ b/ui/lib/voice/vitest.config.ts
@@ -0,0 +1,5 @@
+import { defineConfig } from 'vitest/config';
+
+export default defineConfig({
+ test: { environment: 'node', include: ['tests/**/*.test.ts'] }
+});