cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor.BertTokenClassifierDataProcessor#

class cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor.BertTokenClassifierDataProcessor(*args, **kwargs)[source]#

Bases: torch.utils.data.IterableDataset

Reads csv file containing the input token ids, and label_ids. Creates attention_masks and sedment_ids on the fly

Methods

create_dataloader

Classmethod to create the dataloader object.

load_buffer

Generator to read the data in chunks of size of data_buffer.
create_dataloader()[source]#

Classmethod to create the dataloader object.

load_buffer()[source]#

Generator to read the data in chunks of size of data_buffer.

Returns

Yields the data stored in the data_buffer.

previous

cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor.parse_ner_row

next

cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor.BertTokenClassifierDataProcessorConfig