cerebras.modelzoo.data.nlp.t5.T5HDF5DataProcessor.T5HDF5DataProcessor#

class cerebras.modelzoo.data.nlp.t5.T5HDF5DataProcessor.T5HDF5DataProcessor(config)[source]#

Bases: cerebras.modelzoo.data.common.HDF5IterableDataProcessor.HDF5IterableDataProcessor

A HDF5 dataset processor for T5 training. Loads data from HDF5 files.

Methods

collate_fn

create_dataloader

Classmethod to create the dataloader object.
create_dataloader()#

Classmethod to create the dataloader object.

previous

cerebras.modelzoo.data.nlp.t5.T5HDF5DataProcessor

next

cerebras.modelzoo.data.nlp.t5.T5HDF5DataProcessor.T5HDF5DataProcessorConfig