DCLM Style Deduplications

[revbucket]

General updates to the dedupe command to do deduplication using a joint paragraph/document flow in the same way that DCLM does.

Nuanced update list:
Bloom Filter updates:

• Used better binary search to get optimal BF size
• Initialize Bloom Filters with multicore parallelism in mind

Deduper updates:

• Switched out threadpool for rayon (cleaner, but equivalently performant)
• Added optional read/write of bloom filter file (usually it's not necessary to save this, right?)
• Made the main rust fxn more modular, easier to add in different types of dedupe methods
• Logged some after-dedupe stats: {sparsity, removal rate}
• Added DCLM style deduplication

Other stuff:

• Modified the dedupe config to want a "dedupe.dedupe_method" attribute to specify which type {documents, paragraphs, dclm} of deduplication we do
• Updated tutorial/etc to include this modified config^

[Whattabatt]

Run make style to pass the linter and style check . Also, please add tests in https://github.com/allenai/dolma/blob/main/tests/python/test_deduper.py

[Whattabatt]

Why this change?

[revbucket]

0 => means we do the max parallelism (processes becomes number of cores available). I just assumed that we want this behavior almost all of the time

This might not actually play nice with beaker nodes and how CPU's get allocated here. I'll fall back on ai2-best-practices here

[Whattabatt]

Update the help string to reflect this since it's non-obvious