cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.hdf5_base_preprocessor#

Classes

HDF5BasePreprocessor

This module defines how to process a dataset, tokenize it and write into HDF5 format.

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.create_hdf5_dataset.main

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.hdf5_base_preprocessor.HDF5BasePreprocessor