Skip to content

fix version startup and imported encodings - #42

Merged
matveynator merged 1 commit into
mainfrom
agent/fix-version-and-encoding
Jul 30, 2026
Merged

fix version startup and imported encodings#42
matveynator merged 1 commit into
mainfrom
agent/fix-version-and-encoding

Conversation

@matveynator

Copy link
Copy Markdown
Owner

What changed

  • handle clean --version, -version, and -v invocations before storage-path and server flag initialization
  • update NetChan to the lazy-logger change from start logger only when used netchan#4
  • remove package-level fallback workers for registration confirmations, import cancellation, trial previews, and publish progress
  • keep valid UTF-8 bytes as UTF-8 before heuristic scoring can reinterpret them
  • normalize accidental MySQL UTF-8 charset/collation labels such as utf8mb4_unicode_ci
  • normalize heuristic scores by document length and expose the selected encoding source/certainty in download results
  • rewrite charset declarations only inside HTML <meta> tags, leaving JavaScript/JSON text untouched
  • log uncertain import encoding decisions with their URL and source

Root cause

https://kmv.ru/for-home.html currently returns 170,977 valid UTF-8 bytes and declares UTF-8 in both HTTP and HTML metadata. The old whole-document scoring still selected KOI8-U because its length-dependent score outweighed the fixed UTF-8 preference, turning Для дома into п■п╩я▐.

Separately, NetChan started its logging goroutine from package init, so it ran before SiteBrush could process --version.

User impact

Version commands now print exactly one version line with no SiteBrush storage creation or NetChan startup log. Large UTF-8 pages retain their original text, legacy Cyrillic fallbacks remain supported, and charset-like strings embedded in scripts are no longer modified.

Checks

  • GOCACHE=/private/tmp/sitebrush-gocache go test ./pkg/...
  • GOCACHE=/private/tmp/sitebrush-gocache go test ./pkg/crawler -count=1
  • targeted root tests for version command, legacy Windows-1251 import, and charset rewriting
  • GOCACHE=/private/tmp/sitebrush-gocache go vet ./pkg/crawler
  • live KMV check: encoding=utf-8 source=http certain=true expected_text=true mojibake=false bytes=170977
  • built binary returned only version-live-check for all three version flags
  • git diff --check

The full root suite has the same nine pre-existing failing tests on this branch and clean origin/main; no additional failing test was introduced.

@matveynator
matveynator marked this pull request as ready for review July 30, 2026 20:26
@matveynator
matveynator merged commit 318aa30 into main Jul 30, 2026
6 checks passed
@matveynator
matveynator deleted the agent/fix-version-and-encoding branch July 30, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant