cerebras.modelzoo.data_preparation.data_preprocessing.hooks

Functions

chat_read_hook	Transforms chat data for reading.
dpo_read_hook	Transforms data for the Direct Preference Optimization (DPO) task into a semantic data array format.
finetuning_image_captions_hook	Process finetuning image captions data into a semantic_data_array format.
finetuning_llava_hook	Transforms conversation data for finetuning LLaVA.
finetuning_llava_hook_prompt_completion	Transforms conversation data for finetuning LLaVA into SDA format.
nlg_read_hook	Transforms natural language generation (NLG) data for reading.
pretraining_image_captions_hook	Transforms image and caption data for pretraining.
prompt_completion_chat_read_hook	Process prompt and completion data from a chat into a semantic_data_array format.
prompt_completion_text_read_hook	Process prompt and completion text data into a semantic_data_array format.
text_read_hook	Transforms text data for reading.

previous

cerebras.modelzoo.data_preparation.data_preprocessing.finetuning_token_generator.FinetuningTokenGenerator

next

cerebras.modelzoo.data_preparation.data_preprocessing.hooks.chat_read_hook