**Problem** - Sparse index models (BMX, BM25) maintain the vocabulary to map tokens to token ids, while building sparse matrices only needs token ids. - BaseSparseIndex stores the corpus tokens for index rebuilding. It takes up a lot of memory if the corpus is big. **Suggestion** - Maintain the vocabulary and perform token mapping in BaseSparseIndex to simplify the model logic. - Save token ids for corpus instead of tokens for index rebuilding to save up memory, as only token ids are necessary for matrix construction.
Problem
Suggestion