cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.Tokenizer#

class cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.Tokenizer[source]#

Bases: object

Tokenizes files from the input path into output path. Stanford CoreNLP is used for tokenization. :param params: dict params: Tokenizer configuration parameters.

Methods

process

__init__(params)[source]#

Tokenizes files from the input path into output path. Stanford CoreNLP is used for tokenization. :param params: dict params: Tokenizer configuration parameters.

previous

cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.RougeBasedLabelsFormatter

next

cerebras.modelzoo.data_preparation.nlp.bert.create_csv