All issues

Search results

byte_fallback=true with no <0xHH> pieces rejects a ByteLevel BPE tokenizer

Status: Open.

#55 In marcelroed/gigatoken;

· alexnails opened on Aug 12, 2026

Less memmove in gigatoken_rs::bpe_train::update_word

Status: Open.

#54 In marcelroed/gigatoken;

· qsliu2017 opened on Aug 6, 2026

Make Python bindings optional for Rust consumers

enhancementNew feature or requestNew feature or request

Status: Open.

#53 In marcelroed/gigatoken;

· pjb157 opened on Aug 4, 2026

Cargo.toml cannot be parsed on current nightly due to profile.profiling.rustflags

Status: Open.

#51 In marcelroed/gigatoken;

· artiprice opened on Aug 3, 2026

discussion: exploring possible collaboration between gigatoken and ffbpe

performance

Status: Open.

#50 In marcelroed/gigatoken;

· clouds56 opened on Jul 29, 2026

Security: unsafe Hub cache paths and unchecked UTF-8 in native APIs

security

Status: Open.

#45 In marcelroed/gigatoken;

· GhimBoon opened on Jul 26, 2026

HFCompat uses _token_to_id whilst Huggingface Tokenizers uses token_to_id

Status: Open.

#44 In marcelroed/gigatoken;

· EclipsedStar0 opened on Jul 26, 2026

Better report issues due to Unicode version differences (HF outdated)

hf-bugBug is in the HuggingFace Tokenizers implementation, not in GigatokenBug is in the HuggingFace Tokenizers implementation, not in Gigatoken

Status: Open.

#43 In marcelroed/gigatoken;

· marcelroed opened on Jul 25, 2026

Eliminate remaining allocator overhead for large documents

performance

Status: Open.

#38 In marcelroed/gigatoken;

· jetpks opened on Jul 24, 2026

Standalone pretokenization for all supported schemes from Python (implementation ready)

Status: Open.

#34 In marcelroed/gigatoken;

· AmitMY opened on Jul 23, 2026

Improve caveat at bottom of README

Status: Open.

#33 In marcelroed/gigatoken;

· stephantul opened on Jul 22, 2026

Fix o200k special-token handling in the tiktoken loader

Status: Open.

#31 In marcelroed/gigatoken;

· laudney opened on Jul 22, 2026