cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor#

class cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor[source]#

Bases: torch.utils.data.IterableDataset

Reads csv file containing the input_token_ids, and label_ids. Creates attention_mask and segment_ids on the fly :param <dict> params: dict containing input parameters for creating dataset.

Methods

create_dataloader

Classmethod to create the dataloader object.

load_buffer

Generator to read the data in chunks of size of data_buffer.
__init__(params)[source]#
create_dataloader()[source]#

Classmethod to create the dataloader object.

load_buffer()[source]#

Generator to read the data in chunks of size of data_buffer.

Returns

Yields the data stored in the data_buffer.

__call__(*args: Any, **kwargs: Any) → Any#

Call self as a function.

static __new__(cls, *args: Any, **kwargs: Any) → Any#

previous

cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.create_bertsum_feature

next

cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor