## Fuzzy Deduplication Steps
This directory consists of scripts that can be invoked directly via the command line for finding fuzzy duplicates from a group of Jsonl files consisting of text & unique ID's that are specifically formatted using the `add_id` script included as a part of NeMo-Curator.

> [!IMPORTANT]
> The scripts are helper utilities that wrap the fuzzy_dedup API for handling multiple jsonl directories and the id format generated by [add_id](../add_id.py). For most cases we recommend working with the fuzzy_deduplication API directly.

### Usage
1. Compute Minhashes
    - Input: Data Directories
    -  Output: minhashes.parquet for each data dir.
    - Example call:
    ```bash
        # same as `python compute_minhashes.py`
        gpu_compute_minhashes \
          --input-data-dirs /path/to/jsonl/dir1 /path/to/jsonl/dir2 \
          --output-minhash-dir /path/to/output_minhashes \
          --input-json-text-field text_column_name \
          --input-json-id-field id_column_name \
          --minhash-length number_of_hashes \
          --char-ngram char_ngram_size \
          --hash-bytes 4(or 8 byte hashes) \
          --seed 42 \
          --log-dir ./
          # --scheduler-file /path/to/file.json
    ```
2. Buckets (Minhash Buckets)
    - Input: Minhash directories
    - Output: Buckets.parquet
    - Example call:
    ```bash
        # same as `python minhash_lsh.py`
        minhash_buckets \
          --input-data-dirs /path/to/output_minhashes/dir1 /path/to/output_minhashes/dir2 \
          --output-bucket-dir /path/to/dedup_output \
          --input-minhash-field _minhash_signature \
          --input-json-id-field id_column_name \
          --minhash-length number_of_hashes \
          --num-bands num_bands \
          --buckets-per-shuffle 1 `#Value b/w [1-num_bands]. Higher is better but might lead to oom` \
          --log-dir ./
          # --scheduler-file /path/to/file.json
    ```
3. Jaccard Map Buckets
    - Input: Buckets.parquet + Data Dir
    - Output: anchor_docs_with_bk.parquet
    - Example call:
    ```bash
        # same as `python map_buckets.py`
        jaccard_map_buckets \
          --input-data-dirs /path/to/jsonl/dir1 /path/to/jsonl/dir2 \
          --input-bucket-dir /path/to/dedup_output/_buckets.parquet \
          --output-dir /path/to/dedup_output \
          --input-json-text-field text_column_name \
          --input-json-id-field id_column_name \
          # --scheduler-file /path/to/file.json
    ```
4. Jaccard Shuffle
    - Input: anchor_docs_with_bk.parquet + Data Dir
    - Output: shuffled_docs.parquet
    - Example call:
    ```bash
        # same as `python jaccard_shuffle.py`
        jaccard_shuffle \
          --input-data-dirs /path/to/jsonl/dir1 /path/to/jsonl/dir2 \
          --input-bucket-mapping-dir /path/to/dedup_output/anchor_docs_with_bk.parquet \
          --output-dir /path/to/dedup_output \
          --input-json-text-field text_column_name \
          --input-json-id-field id_column_name \
          # --scheduler-file /path/to/file.json
    ```
5. Jaccard compute
    - Input: Shuffled docs.parquet
    - Output: jaccard_similarity_results.parquet
    - Example call:
    ```bash
        # same as `python jaccard_compute.py`
        jaccard_compute \
          --shuffled-docs-path /path/to/dedup_output/shuffled_docs.parquet \
          --output-dir /path/to/dedup_output \
          --ngram-size char_ngram_size_for_similarity \
          # --scheduler-file /path/to/file.json
    ```
6. Connected Components
    - Input: jaccard_similarity_results.parquet
    - Output: connected_components.parquet
    - Example call:
    ```bash
        # same as `python connected_components.py`
        gpu_connected_component \
          --jaccard-pairs_path /path/to/dedup_output/jaccard_similarity_results.parquet \
          --output-dir /path/to/dedup_output \
          --cache-dir /path/to/cc_cache \
          --jaccard-threshold 0.8
          # --scheduler-file /path/to/file.json
    ```

> [!TIP]
> When using these scripts in a multi-node environment (like Slurm, K8's etc.) it is recommended to start up a Dask cluster prior to execution and connect to the existing cluster via the `--scheduler-address` or `--scheduler-file` flag.
> Use the `--help` flag to view all possible CLI options for the scripts and details on what they do.
