cerebras.modelzoo.data_preparation.data_preprocessing.preprocess_data#

Script to generate an HDF5 dataset for GPT Models.

Functions

main

Main function for execution.

preprocess_data

previous

cerebras.modelzoo.data_preparation.data_preprocessing.nlg_token_generator.NLGTokenGenerator

next

cerebras.modelzoo.data_preparation.data_preprocessing.preprocess_data.main