cerebras.modelzoo.data.nlp.gpt.InferenceDataProcessor#

Functions

get_token_ids

Get encoded token ids from a string using the specified tokenizer.

Classes

InferenceDataProcessor

InferenceDataProcessorGU

Subclass for processing EEH generate_until requests.

InferenceDataProcessorLL

Subclass for processing EEH loglikelihood requests.

RequestType

An enumeration.

previous

cerebras.modelzoo.data.nlp.gpt.HuggingFaceIterableDataProcessorEli5.HuggingFaceIterableDataProcessorEli5

next

cerebras.modelzoo.data.nlp.gpt.InferenceDataProcessor.get_token_ids