cerebras.modelzoo.data_preparation.nlp.tokenizers.BPETokenizer.BPETokenizer#

class cerebras.modelzoo.data_preparation.nlp.tokenizers.BPETokenizer.BPETokenizer[source]#

Bases: object

Methods

`add_token`
`bpe`
`decode`
`encode`
`get_token_id`

__init__(vocab_file, encoder_file, errors='replace', special_tokens=None)[source]#

previous

cerebras.modelzoo.data_preparation.nlp.tokenizers.BPETokenizer.get_pairs

next

cerebras.modelzoo.data_preparation.nlp.tokenizers.HFTokenizer