tokenizer: Add BPE vocabulary serialization support

[Eamon2009]

Implement the complete Byte-Pair Encoding (BPE) tokenizer infrastructure in
tokenizer.h to power sub-word tokenization, serialization, and text streaming
for the llm.cpp r epo

Key features and implementation details:

• BPE Vocabulary and Merges:
  • Added data structures to manage the vocabulary map (vocab), reverse
    lookup tables, and merge ranks (merge_ranks) loaded from training artifacts.
  • Implemented custom byte-level fallback encodings to handle raw string sequences
    gracefully without out-of-vocabulary (OOV) crashes.
• Encoding & Decoding Pipelines:
  • encode(): Converts raw input text strings into sequences of integer token IDs
    by iteratively merging adjacent byte pairs based on learned priority ranks.
  • decode(): Reconstructs human-readable text streams from vector token arrays.
• Serialization and File I/O:
  • Added binary and text file loading/saving routines to persist vocabulary
    weights and merge dictionaries across runtime sessions.
  • Included error handling and stream validation for missing or corrupted tokenizer files.
    [Feature]: Complete BPE Tokenizer Infrastructure for Sub-word Tokenization and Serialization #70

[Eamon2009]

/run-checks

🚀 Strict PR checks triggered by @Eamon2009... Running analysis and build steps.

[github-actions]

📋 Maintainer Check Results

Check Category	Status
C++ Strict Quality	❌ Failed
Python Strict Quality	❌ Failed
Build & Tests	❌ Failed

Overall Result: ❌ CHECK FAILURES DETECTED

View Full Execution Logs

[Eamon2009]

Checking ./main.cpp
./main.cpp:22:10: fatal error: include/lm.h: No such file or directory
   22 | #include "include/lm.h"
      |          ^~~~~~~~~~~~~~

The compiler is failing on #include "include/lm.h" because the header changes are currently in a local branch and have not been committed yet. A follow-up commit containing these files is on the way.