cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.tokenize#

cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.tokenize(params)[source]#

Split sentences and perform tokenization. Takes params.input_path, tokenize it and store it under params.output_path.

previous

cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.create_parser

next

cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.BertData