cerebras.modelzoo.data_preparation.data_preprocessing.utils.wikitext_detokenizer

cerebras.modelzoo.data_preparation.data_preprocessing.utils.wikitext_detokenizer(string)

Detokenizer for wikitext. Used for special handling of data for substrings.

Parameters

string (str) – String to detoknize before tokenization.

Returns

Detokenized string

previous

cerebras.modelzoo.data_preparation.data_preprocessing.utils.update_params

next

cerebras.modelzoo.data_preparation.data_preprocessing.vsl_finetuning_token_generator