modelzoo.transformers.data_processing.huggingface#

CSDataCollatorForLanguageModeling

HF_converter_example_BookCorpus

Example script to convert HF BookCorpus dataset to HDF5 dataset

HF_converter_example_Eli5

Example script to convert HF Eli5 dataset to HDF5 dataset

HuggingFaceDataProcessor

Pytorch HuggingFace Dataloader

HuggingFace_BookCorpus

HuggingFace BookCorpus Dataset

HuggingFace_Eli5

HuggingFace Eli5 Dataset

previous

modelzoo.transformers.data_processing.h5_map_dataset.samplers.Sharder

next

modelzoo.transformers.data_processing.huggingface.CSDataCollatorForLanguageModeling