cerebras.modelzoo.data_preparation.raw_dataset_processor.RawDatasetProcessor.RawDatasetProcessorConfig#

class cerebras.modelzoo.data_preparation.raw_dataset_processor.RawDatasetProcessor.RawDatasetProcessorConfig(*args, **kwargs)[source]#

Bases: cerebras.modelzoo.config.data_config.DataConfig

Configuration class for RawDatasetProcessor.

Methods

check_for_deprecated_fields

check_literal_discriminator_field

copy

get_orig_class

get_orig_class_args

model_copy

model_post_init

post_init

Attributes

batch_size

The dataset preprocessing configuration.

discriminator

discriminator_value

drop_last

model_config

num_workers

persistent_workers

prefetch_factor

preprocessing

seed

shuffle

shuffle_seed

data_processor

batch_size = Ellipsis#

The dataset preprocessing configuration.

previous

cerebras.modelzoo.data_preparation.raw_dataset_processor.RawDatasetProcessor.RawDatasetProcessor

next

cerebras.modelzoo.data_preparation.raw_dataset_processor.utils