cerebras.modelzoo.data.nlp.bert.bert_utils.parse_text

cerebras.modelzoo.data.nlp.bert.bert_utils.parse_text(text, do_lower)

Postprocessing of the CSV file. This code should parse commas that are part of the token strings. :param: str text: String with an input text.

Returns

List of parsed tokens.

previous

cerebras.modelzoo.data.nlp.bert.bert_utils.get_whole_word_span

next

cerebras.modelzoo.data.nlp.bert.bert_utils.shard_and_shuffle_data