cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.VerificationArgs#

class cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.VerificationArgs[source]#

Bases: object

VerificationArgs(processes: int, files_per_record: int, max_seq_length: int, tokenizer_obj: object, eos_id: int, pad_id: int, num_features: int)

Methods

Attributes

`processes`
`files_per_record`
`max_seq_length`
`tokenizer_obj`
`eos_id`
`pad_id`
`num_features`

__init__(processes: int, files_per_record: int, max_seq_length: int, tokenizer_obj: object, eos_id: int, pad_id: int, num_features: int) → None#

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.Reader

next

cerebras.modelzoo.data_preparation.nlp.pile