[bug] Application doesn't honor selected language

[rusak47]

This is a nice app—thanks for sharing it.

I'm currently using Nerd Dictation, but the available Vosk models aren't very accurate, so I've been looking for better options.

Whisper looks interesting, but I've found it to be unstable—it often inserts words that were never spoken.

I also noticed an issue with the speech language recognition: it ignores the manually selected language preference and always auto-detects the language instead. About half the time, it chooses incorrectly.

• Unrelated to the main problem, but regarding portable packaging: the application unexpectedly looks for models in /usr/lib without any UI indication or warning when they are not found there.

whisper_init_from_file_with_params_no_state: loading model from '/usr/lib/S2Tui/models/ggml-small.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
ggml_vulkan: Found 1 Vulkan devices:

[cbarange]

Hello! I feel so bad because I didn't have any notification about your submitted issue. I just discovered them today. So first thank you for your message. I will look at it very soon.
Also you are right the Whisper model often inserts non spoken word. I don't know where did he get this word. I suspect this is from the training part of the model and I don't know how to fix that yet.

[cbarange]

Hi @rusak47, thank you for the detailed report — and for the patience.

I've identified the root cause and shipped a fix in v0.1.5 (just tagged).

What was happening

The language picker in the UI was correctly saving the selected language to the in-memory app settings, but the value was never propagated to the Whisper engine itself. The WhisperWorker.engine.config.language was kept at its default (None = auto-detect) regardless of what was selected, so every transcription auto-detected the language. That matches your observation that "about half the time, it chooses incorrectly."

What changed

• set_language now actually pushes the selected language to the Whisper engine.
• The persisted language is re-synced to the backend on every app start (so a language chosen yesterday is still active today without re-selecting).
• After a model load, the current language is re-applied to keep settings and engine in sync.
• transcribe() now logs the active language for diagnostics — you can verify it in the logs (Transcribing N samples — language: fr).

Bonus: the hallucination point

You also mentioned Whisper inserting words that were never spoken. This is a known Whisper behaviour, especially on silence and low-energy audio. I tuned the inference parameters in the same release to mitigate it:

• temperature = 0.0 and temperature_inc = 0.0 → fully deterministic decoding (no random sampling, no fallback ladder)
• no_speech_thold = 0.6 → drop segments the model itself flags as likely non-speech
• suppress_blank = true → suppress blank tokens at decode

These are sensible defaults and should noticeably reduce phantom words. If you still see issues, please share a short audio clip + the model used, and I'll iterate.

Out of scope (for follow-up)

The /usr/lib/S2Tui/models/... packaging point you raised at the bottom is a real concern but a different topic — I'll track it as a separate issue about model-path discovery and user-visible feedback when models are missing.

Closing this issue as fixed by v0.1.5. Thanks again for the report — it was a great catch.

[cbarange]

Fix in release : https://github.com/AccessDevops/S2Tui/releases/tag/v0.1.5

[rusak47]

Hi @cbarange!

Thanks for the great work—it's really cool to see such a detailed issue investigation report.

I checked the release tag, but sadly this time it lacks a portable binary (AppImage my bad—actually I was referring to the S2Tui_xxx_linux_x64_portable.tar.gz archive). Please consider adding it.

I also noticed that every build ships with the Whisper models. While that's convenient for a first-time installation, it unnecessarily consumes bandwidth for subsequent installs. Speaking specifically about the aforementioned binary (not the AppImage), s2tui takes up roughly 1% of the total build size.

Please consider shipping the binary separately, at least for portable builds.

Thanks again!

[cbarange]

Hello @rusak47 , thanks you :). I did have a major issue in me github action, some bundle don't work anymore I have to check that. And you are right about the build size, initially I wanted to make S2Tui fully independent and don't reliable on any external resources. For example the download link of the Whisper model is not really stable over the months sometimes the file name or path change. So that's why I wanted to embed it into the app. But what I can do is maybe have my own Whisper model store somewhere with a static path and name. I will look for implementing such a thing. I will reopen the issue until I fix the CI bundle generation thanks you :). I'm working on the shortcut that lets you change models and languages. I'll come back to this issue soon

[rusak47]

@cbarange Sorry to hear about the issues with GitHub Actions—I hope it's nothing too difficult to fix.

Regarding the model: that's totally understandable. Have you considered hosting it on GitHub as a separate archive? That way, you could update the model only when necessary, while still keeping it reliably stored and without overusing Git storage. Plus users wouldn't be confused about where to find it.

Cool to hear that you're working on that feature! Definitely looking forward with curiosity to see how it evolves.

[rusak47]

Some early observations:

• When the language is forced, I don't see it reflected in the terminal logs at the coding stage—or maybe I'm looking in the wrong place?

whisper_backend_init_gpu: using Vulkan backend
whisper_init_state: kv self size  =   83,89 MB
whisper_init_state: kv cross size =  251,66 MB
whisper_init_state: kv pad  size  =    7,86 MB
whisper_init_state: compute buffer (conv)   =   37,67 MB
whisper_init_state: compute buffer (encode) =  212,29 MB
whisper_init_state: compute buffer (cross)  =    9,25 MB
whisper_init_state: compute buffer (decode) =  100,03 MB

• Sadly, hallucinations still happen, although less frequently. They seem to occur especially during silent pauses between words. However, they're hard to reproduce, so I'm not sure there's much you can do about it.

• I've seen somewhere that a lightweight model specifically designed to detect silence could help mitigate this issue, allowing the heavier model to be called only when truly needed. But I suspect this would require changing the audio streaming mode in the application.
• With a specific language now selected, it no longer jumps between languages as often. Instead, it either translates into the selected language or transliterates. and sometime produces garbage...

• The main downside from the language selection perspective is that the list of available languages is very limited—for example, it lacks Latvian. While it may correctly recognize Latvian in auto‑detection mode, it's still a guess, and as I initially showed, it can be wrong.

The main problem is definitely solved - thank you!

[cbarange]

Hi, thanks again, your share is precious and with many details that's very helpful for me to understand and keep working on.

For the log it's seem's to be normal according to the whisper.cpp log documentation it's only contains "whisper_init_state, compute buffers, kv self size," but not for language change

For the hallucination part, I'm kind of stuck at the moment. I'm missing a technical solution, so I am doing some research. And reverse engineering on other implementations to understand how these issues are resolved by other developments. Because it seems to be very based on the language, I did have some different types of hallucinations when switching language. Some are just nonsense. Some others are words repeating like the "thank you" at the end. for example I have for French language: the text: "Thank you for listening Radio Canada" that I never said at the end of half my text. So it seems to be the training part of the model that are causing problems. Your suggestion about using a small model is interesting. I will look to understand how it works. Thank you :)

For the list of languages, I'm really agreeing with you that this is missing so many of them. So I did want to implement 60 languages, the 60 most speaking languages :)

I will add a comment here when a new release with all this fix or at least the language missing part fix will be published.

Thanks for your comments :)

[cbarange]

Hello,
here is the release https://github.com/AccessDevops/S2Tui/releases/tag/v0.1.8 , This release includes support for custom models. I also added model parameter configuration options that should reduce noise and hallucinations. Most hallucinations come from the short silent segment at the end of each recording. The settings I implemented should help the model better detect silent moments and reduce hallucinations during those parts :)