cerebras.modelzoo.data_preparation.raw_dataset_processor#
This is Dataset process for processing Raw data set on the fly This contains methods for loading the dataset, tokenizing the dataset and all data transformations are handled as part of the collator function |
|