cerebras.modelzoo.data.nlp.bert.bert_utils.get_whole_word_span

cerebras.modelzoo.data.nlp.bert.bert_utils.get_whole_word_span(tokens, start_index)

Returns the whole word start and end indices.

Param

list tokens: Tokens to process.

Param

int start_index: Start index.

Returns

tuple with start and end index of the word from the token list.

previous

cerebras.modelzoo.data.nlp.bert.bert_utils.get_meta_data

next

cerebras.modelzoo.data.nlp.bert.bert_utils.parse_text