cerebras.modelzoo.data.nlp.dpo.DpoHDF5DataProcessor.DpoHDF5DataProcessor#

class cerebras.modelzoo.data.nlp.dpo.DpoHDF5DataProcessor.DpoHDF5DataProcessor(config)[source]#

Bases: cerebras.modelzoo.data.common.HDF5IterableDataProcessor.HDF5IterableDataProcessor

A HDF5 dataset processor for DPO. Loads data from HDF5 files.

Methods

collate_fn

create_dataloader

Classmethod to create the dataloader object.
create_dataloader()#

Classmethod to create the dataloader object.

previous

cerebras.modelzoo.data.nlp.dpo.DpoHDF5DataProcessor

next

cerebras.modelzoo.data.nlp.dpo.DpoHDF5DataProcessor.DpoHDF5DataProcessorConfig