cerebras.modelzoo.data.nlp.gpt

DummyDataProcessor	Pytorch Generic Dataloader
DummyIterableDataProcessor	Pytorch Generic Iterable Dataloader
GptHDF5DataProcessor	Pytorch GPT2/3 Dataloader.
HuggingFaceDataProcessorEli5	Pytorch HuggingFace Eli5 map-style Dataloader
HuggingFaceIterableDataProcessorEli5	Pytorch HuggingFace Eli5 Iterable Dataloader.
InferenceDataProcessor	This module defines the InferenceDataProcessor class, its subclasses and the EvalHarnessDataset class for preprocessing and loading eval harness data
config	Config classes of T5 data Configs.

previous

cerebras.modelzoo.data.nlp.dpo.DpoHDF5DataProcessor.DpoHDF5DataProcessorConfig

next

cerebras.modelzoo.data.nlp.gpt.DummyDataProcessor