cerebras.modelzoo.data_preparation.utils.count_total_documents#

cerebras.modelzoo.data_preparation.utils.count_total_documents(metadata_files)[source]#

Counts total number of documents in metadata_files. :param str or list[str] metadata_files: Path or list of paths

to metadata files.

Returns

Number of documents whose paths are contained in the metadata files.

previous

cerebras.modelzoo.data_preparation.utils.convert_to_unicode

next

cerebras.modelzoo.data_preparation.utils.create_masked_lm_predictions