cerebras.modelzoo.data_preparation.nlp.pile.download.download_tokenizer_files#

cerebras.modelzoo.data_preparation.nlp.pile.download.download_tokenizer_files(args)[source]#

Download files needed for tokenization for dataset creation.

Parameters

args (argparse namespace) – Arguments for downloading the tokenizer files.

previous

cerebras.modelzoo.data_preparation.nlp.pile.download.download_pile

next

cerebras.modelzoo.data_preparation.nlp.pile.download.get_urls_for_tokenizer_files