cerebras.modelzoo.data_preparation.huggingface.HuggingFaceDataProcessor.HuggingFaceDataProcessor#

class cerebras.modelzoo.data_preparation.huggingface.HuggingFaceDataProcessor.HuggingFaceDataProcessor(config, dataset)[source]#

Bases: object

A HuggingFace map-style Data Processor. :param dict params: dict containing training

input parameters for creating dataset.

Expects the following fields:

Methods

create_dataloader

Classmethod to create the dataloader object.
create_dataloader()[source]#

Classmethod to create the dataloader object.

previous

cerebras.modelzoo.data_preparation.huggingface.HuggingFaceDataProcessor

next

cerebras.modelzoo.data_preparation.huggingface.HuggingFaceDataProcessor.HuggingFaceDataProcessorConfig