cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.DataFrame

class cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.DataFrame(keys=None, read_hook_fn=None)

Bases: object

Initialize the DataFrame object.

Parameters

keys (Dict) – Keys for the data entries.

Methods

add	Add an entry to the DataFrame.
clear	Clear the raw data after tokenizing.
tokenize	Tokenize the data values.

add(value)

Add an entry to the DataFrame.

Parameters

value (Union[Dict[str, Any], Any]) – Entry to be added.

clear()

Clear the raw data after tokenizing.

tokenize(token_generator)

Tokenize the data values.

Parameters

token_generator (Any) – Token generator to be used for processing the data.

previous

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.split_entry_by_paragraph_or_sentence

next

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.Reader