Pile Dedupe
What it is
A Python script collection for deduplicating the Pile dataset using MinHash and Locality-Sensitive Hashing (LSH). It provides a pipeline to generate minhashes, identify duplicates, and yield a deduplicated data stream for training.
The problem it solves
Large text datasets like the Pile contain many near-duplicate documents, which can lead to data leakage and overfitting if not properly deduplicated.
What you could use it for
- Deduplicating the Pile dataset before training language models
- Identifying near-duplicate documents in large text corpora
- Generating a clean, deduplicated data stream for model training
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.