cerebras.modelzoo.data_preparation.nlp.data_dedup.to_hash

Functions

clean
custom_progress_bar
generate_hashes
get_documents
get_features
output_results
preprocess_string
print_docs_processed
to_minhash

previous

cerebras.modelzoo.data_preparation.nlp.data_dedup.generate_duplicate_pairs.split_files

next

cerebras.modelzoo.data_preparation.nlp.data_dedup.to_hash.clean