cerebras.modelzoo.data_preparation.data_preprocessing.utils.args_to_params#

cerebras.modelzoo.data_preparation.data_preprocessing.utils.args_to_params(args)[source]#

Process data preprocessing CLI arguments to parameters :returns:

Dictionary contains the parameters used to configure

the data processing.

Return type

params (Dict)

previous

cerebras.modelzoo.data_preparation.data_preprocessing.utils.append_eos_to_multiple_semantic_regions

next

cerebras.modelzoo.data_preparation.data_preprocessing.utils.check_fim_special_tokens