cerebras.modelzoo.data.common.h5_map_dataset.dataset.MultiModalHDF5Dataset#

class cerebras.modelzoo.data.common.h5_map_dataset.dataset.MultiModalHDF5Dataset[source]#

Bases: cerebras.modelzoo.data.common.h5_map_dataset.dataset.HDF5Dataset

Specialized HDF5 dataset class to handle image preprocessing in multimodal datasets Functionality is largely the same as HDF5Dataset except with added image loading and preprocessing :param params: a dictionary containing the following added fields:

  • “img_data_dir” (str): the path to the directory containing

    the images.

  • “fp16_type” (str): the half dtype cast for the image

  • “image_data_size” (list[int]): the final C x H x W shape of

    the image

  • “transforms” (list[dict]): a specification of the torchvision

    transforms

Methods

generate_sample

Generates an empty tensor with the same shape and dtype as a sample from its dataset.

map

preprocess_img

Attributes

by_sample

seed

__init__(params)[source]#
generate_sample()[source]#

Generates an empty tensor with the same shape and dtype as a sample from its dataset.

__call__(*args: Any, **kwargs: Any) → Any#

Call self as a function.

static __new__(cls, *args: Any, **kwargs: Any) → Any#

previous

cerebras.modelzoo.data.common.h5_map_dataset.dataset.HDF5Dataset

next

cerebras.modelzoo.data.common.h5_map_dataset.dataset.RestartableDataLoader