| | --- |
| | title: Semantic Deduplication |
| | emoji: 🧹 |
| | colorFrom: green |
| | colorTo: green |
| | sdk: gradio |
| | sdk_version: 5.32.1 |
| | app_file: app.py |
| | pinned: false |
| | license: mit |
| | short_description: Deduplicate HuggingFace datasets in seconds |
| | hf_oauth: true |
| | hf_oauth_scopes: |
| | - write-repos |
| | - manage-repos |
| | --- |
| | |
| | # Semantic Text Deduplication Using SemHash |
| |
|
| | This Gradio application performs **semantic deduplication** on HuggingFace datasets using [SemHash](https://github.com/MinishLab/semhash) with [Model2Vec](https://github.com/MinishLab/model2vec) embeddings. |
| |
|
| | ## Features |
| |
|
| | - **Two deduplication modes**: |
| | - **Single dataset**: Find and remove duplicates within one dataset |
| | - **Cross-dataset**: Remove entries from Dataset 2 that are similar to entries in Dataset 1 |
| |
|
| | - **Customizable similarity threshold**: Control how strict the deduplication should be (0.0 = very loose, 1.0 = exact matches only) |
| |
|
| | - **Detailed results**: View statistics and examples of found duplicates with word-level differences highlighted |
| |
|
| | - **Hub Integration**: 🆕 **Push deduplicated datasets directly to the Hugging Face Hub** after logging in |
| |
|
| | ## How to Use |
| |
|
| | ### 1. Choose Deduplication Type |
| | - **Cross-dataset**: Useful for removing training data contamination from test sets |
| | - **Single dataset**: Clean up duplicate entries within a single dataset |
| |
|
| | ### 2. Configure Datasets |
| | - Enter the HuggingFace dataset names (e.g., `SetFit/amazon_massive_scenario_en-US`) |
| | - Specify the dataset splits (e.g., `train`, `test`, `validation`) |
| | - Set the text column name (usually `text`, `sentence`, or `content`) |
| |
|
| | ### 3. Set Similarity Threshold |
| | - **0.9** (default): Good balance between precision and recall |
| | - **Higher values** (0.95-0.99): More conservative, only removes very similar texts |
| | - **Lower values** (0.7-0.85): More aggressive, may remove semantically similar but different texts |
| |
|
| | ### 4. Run Deduplication |
| | Click **"Deduplicate"** to start the process. You'll see: |
| | - Loading progress for datasets |
| | - Deduplication progress |
| | - Results with statistics and example duplicates |
| |
|
| | ### 5. Push to Hub (New!) |
| | After deduplication completes: |
| | 1. **Log in** with your Hugging Face account using the login button |
| | 2. Enter a **dataset name** for your cleaned dataset |
| | 3. Click **"Push to Hub"** to upload the deduplicated dataset |
| |
|
| | The dataset will be saved as `your-username/dataset-name` and be publicly available. |
| |
|
| |
|
| | ## Notes |
| |
|
| | - The app preserves all original columns from the datasets |
| | - Only the text similarity is used for deduplication decisions |
| | - Deduplicated datasets maintain the same structure as the original |
| | - OAuth login is required only for pushing to the Hub, not for deduplication |
| |
|
