cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.VerificationArgs#

class cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.VerificationArgs(processes: int, files_per_record: int, max_seq_length: int, tokenizer_obj: object, eos_id: int, pad_id: int, num_features: int)[source]#

Bases: object

Methods

Attributes

processes

files_per_record

max_seq_length

tokenizer_obj

eos_id

pad_id

num_features

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.Reader

next

cerebras.modelzoo.data_preparation.nlp.pile