cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.RedPajamaCommonCrawlDataset

class cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.RedPajamaCommonCrawlDataset

Bases: cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.Dataset

Methods

already_shuffled	Datasets where the source is already shuffled should override this to return True so that it isn't shuffled again.
dir_path	Path to the directory
documents	A generator producing all documents in the dataset.
name	Human-readable name of tfhe dataset
num_docs
num_duplicate_docs
num_short_docs
short_documents_path	Path to the file with short documents
size	Return an estimate of the dataset size.
size_duplicate_docs
size_short_docs
stem_dir_path

__init__(input_dir)

dir_path()

Path to the directory

name()

Human-readable name of tfhe dataset

size()

Return an estimate of the dataset size. Implementations may use a faster, less accurate estimate.

already_shuffled()

Datasets where the source is already shuffled should override this to return True so that it isn’t shuffled again.

documents(process_id, n_process, dup_sh, short_sh)

A generator producing all documents in the dataset.

short_documents_path()

Path to the file with short documents

previous

cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.RedPajamaC4Dataset

next

cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.RedPajamaGithubDataset