cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.collect_stats#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.collect_stats(data_arr, args)[source]#

Collect statistics of the dataset.

Parameters

  • data_arr (numpy.ndarray) – Numpy array containing the dataset.

  • args (ValidationArgs) – Arguments for verifying HDF5 dataset.

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.chunk

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm