cerebras.modelzoo.data_preparation.data_preprocessing.data_preprocessor#

This module implements a generic data preprocessor called DataPreprocessor. It internally uses DataFrame and DataReader to read and process data.

Functions

`check_and_create_dir`	Ensures a directory exists, optionally handling a subdirectory.
`default_hook`
`format_time`	Format seconds into a human-readable string showing hours:minutes:seconds, minutes:seconds, or seconds.
`get_available_memory`	Returns available memory in bytes.
`get_compression_factor`	Calculate and return the compression factor based on a file's extension.
`save_image_locally`
`update_data_stats`
`update_progress`	Update the progress bar based on the current progress.

Classes

Initialize the class with given parameters.

cerebras.modelzoo.data_preparation.data_preprocessing.data_dedup.to_hash.to_minhash

cerebras.modelzoo.data_preparation.data_preprocessing.data_preprocessor.check_and_create_dir