Cerebras Model Zoo Extensions#

This module contains integrations of external tools to the Trainer.

Downstream Validation Callbacks#

The set of callbacks that implement eval harness, i.e. external frameworks for running downstream validation with the Trainer.

BigCodeEvalHarness

class cerebras.modelzoo.trainer.extensions.bigcode.BigCodeEvalHarness(bigcode_args, keep_data_dir=False, every_n_vals=1, flags=None, name_scope=None, batch_size=None, data_dir=None, max_sequence_length=None, tokenizer_file_path=None, eos_id=None, **dataloader_args)[source]#

Bases: cerebras.modelzoo.trainer.callbacks.callback.ValidationCallback

ValidationCallback class to run BigCode’s Evaluation Harness.

Parameters

  • bigcode_args (Union[cerebras.modelzoo.trainer.extensions.bigcode.bigcode_eval_harness.BigCodeCLIArgs, Dict[str, Any]]) – BigCodeCLIArgs dataclass or dict capturing BCEH’s CLI args

  • keep_data_dir (bool) – Specifies whether dumped data samples should be kept for reuse. Defaults to False, i.e. data samples are deleted after the run.

  • every_n_vals (int) –

    Run the BigCode eval harness script every N validations. e.g. If the eval_frequency is set to 200 and N=2,

    then BigCode eval harness runs every 400 training steps.

    The BigCode eval harness script will also always run after the final training iteration.

  • flags (Optional[dict]) – A optional dictionary of scoped global flags to set during the BigCode eval harness run.

  • name_scope (Optional[str]) – An optional string that gets added to the trainer’s name scope.

  • batch_size (Optional[int]) – Batch size to BigCodeEvalHarness to preprocess input data samples from the specified eval harness tasks.

  • data_dir (Optional[str]) – Path to data directory

  • max_sequence_length (Optional[int]) – Maximum sequence length

  • tokenizer_file_path (Optional[str]) – Path to tokenizer file

  • eos_id (Optional[int]) – End of sentence token id

  • dataloader_args – Any additional dataloader args, e.g. num_workers.

run(trainer)[source]#

Run BigCode Eval Harness.

Parameters

trainer – the Trainer object

EleutherEvalHarness

class cerebras.modelzoo.trainer.extensions.eleuther.EleutherEvalHarness(eeh_args, keep_data_dir=False, every_n_vals=1, flags=None, name_scope=None, batch_size=None, data_dir=None, max_sequence_length=None, tokenizer_file_path=None, eos_id=None, **dataloader_args)[source]#

Bases: cerebras.modelzoo.trainer.callbacks.callback.ValidationCallback

Callback class to run EleutherAI’s Evaluation Harness.

Parameters

  • eeh_args (Union[cerebras.modelzoo.trainer.extensions.eleuther.eval_harness_utils.EleutherCLIArgs, Dict[str, Any]]) – EleutherCLIArgs dataclass or dict capturing EEH’s CLI args

  • keep_data_dir (bool) – Specifies whether dumped data samples should be kept for reuse. Defaults to False, i.e. data samples are deleted after the run.

  • every_n_vals (int) –

    Run the EEH script every N validations e.g. If the eval_frequency is set to 200 and N=2,

    then EEH runs every 400 training steps.

    The EEH script will also always run after the final training iteration.

  • flags (Optional[dict]) – An optional dictionary of scoped global flags to set during the EEH run.

  • name_scope (Optional[str]) – An optional string that gets added to the trainer’s name scope.

  • batch_size (Optional[int]) – Batch size to EleutherEvalHarness to preprocess input data samples from the specified eval harness tasks.

  • data_dir (Optional[str]) – Path to data directory

  • max_sequence_length (Optional[int]) – Maximum sequence length

  • tokenizer_file_path (Optional[str]) – Path to tokenizer file

  • eos_id (Optional[int]) – End of sentence token id

  • dataloader_args – Any additional dataloader args, e.g. num_workers.

property has_generative_task#

Returns True if the task dictionary contains a generative task.

property has_non_generative_task#

Returns True if the task dictionary contains a non-generative task.

run(trainer)[source]#

Run the EleutherAI Evaluation Harness.

Parameters

trainer – the Trainer object

Eval Harness Utils#

Util classes capturing the command line interface arguments for the supported eval harness frameworks.

BigCodeCLIArgs

class cerebras.modelzoo.trainer.extensions.bigcode.BigCodeCLIArgs(prefix='', do_sample=True, temperature=None, top_k=None, top_p=None, n_samples=1, seed=0, tasks=None, instruction_tokens=None, max_tokens=None, limit=None, limit_start=0, save_every_k_tasks=- 1, postprocess=True, allow_code_execution=False, generation_only=True, load_generations_path=None, load_data_path=None, metric_output_path='evaluation_results.json', save_generations=True, load_generations_intermediate_paths=None, save_generations_path='generations.json', save_references=True, save_references_path='references.json', prompt='prompt', check_references=False)[source]#

Captures BigCode EH’s CLI arguments with defaults.

Fields:

prefix: Prefix to add to the prompt. For example InCoder needs prefix=’<| file ext=.py |>n’ do_sample: Sample from the language model’s output distribution. temperature: Sampling temperature used for generation. top_k: Top-k parameter used for generation. top_p: Top-p parameter used for nucleus sampling. n_samples: Number of completions to generate for each sample. seed: Random seed used for evaluation. tasks: List of tasks to evaluate code evals instruction_tokens: A series of instruction tokens used for instruction-tuning

benchamrks separated by comma e.g. <user_message>,<end_user_message>,<assistant_message>

max_tokens: Maximum number of tokens to generate. limit: Number of samples to solve and evaluate from the benchmark limit_start: Optional offset to start from when limiting the number of samples save_every_k_tasks: Optional saving after every k tasks postprocess: Postprocess model outputs before execution, always on except

during generation tests

allow_code_execution: Allow code evaluation to execute external/untrusted Python

code on your machine

generation_only: Do code generation but no evaluation load_generations_path: Path of file with previously generated solutions, if

provided generation is skipped and only evaluation is done

load_data_path: Path of additional data to load for the tasks metric_output_path: Path to save the results save_generations: Whether to save code generations load_generations_intermediate_paths: List of paths for saving the

intermediate code generations

save_generations_path: Path for saving the code generations save_references: Whether to save reference solutions/tests save_references_path: Path for saving the references solutions/tests prompt: Prompt type to use for generation in HumanEvalPack tasks check_references: Don’t run generation but benchmark groundtruth (useful for debugging)

EleutherCLIArgs

class cerebras.modelzoo.trainer.extensions.eleuther.EleutherCLIArgs(tasks, num_fewshot=None, output_path=None, limit=None, use_cache=None, cache_requests=None, check_integrity=False, write_out=False, log_samples=False, system_instruction=None, apply_chat_template=False, fewshot_as_multiturn=False, show_config=False, include_path=None, predict_only=False, seed='0,1234,1234,1234', trust_remote_code=False, verbosity='INFO', max_tokens=None, temperature=None, top_k=None, top_p=None)[source]#

Captures EEH’s CLI arguments with defaults.

Fields:
tasks: List of tasks to evaluate

To get full list of tasks, use the command lm-eval --tasks list

num_fewshot: Number of examples in few-shot context output_path: The path to the output file where the result metrics

will be saved. If the path is a directory and log_samples is true, the results will be saved in the directory. Else the parent directory will be used.

limit: Limit the number of examples per task.

If <1, limit is a percentage of the total number of examples.

use_cache: A path to a sqlite db file for caching model responses.

None if not caching.

cache_requests: Speed up evaluation by caching the building of

dataset requests. None if not caching.

check_integrity: Whether to run the relevant part of the test suite

for the tasks.

write_out: Prints the prompt for the first few documents. log_samples: If True, write out all model outputs and documents for

per-sample measurement and post-hoc analysis. Use with –output_path.

system_instruction: System instruction to be used in the prompt apply_chat_template: If True, applies the chat template to the prompt fewshot_as_multiturn: If True, uses the fewshot as a multi-turn conversation show_config: If True, shows the the full config of all tasks at the

end of the evaluation.

include_path: Additional path to include if there are external tasks

to include.

predict_only: Use with –log_samples. Only model outputs will be

saved and metrics will not be evaluated.

seed: Set seed for python’s random, numpy and torch.

Accepts a comma-separated list of 3 values for python’s random, numpy, and torch seeds, respectively, or a single integer to set the same seed for all three. The values are either an integer or None to not set the seed. Default is 0,1234,1234 (for backward compatibility). E.g. --seed 0,None,8 sets random.seed(0) and torch.manual_seed(8). Here numpy’s seed is not set since the second value is None. E.g, --seed 42 sets all three seeds to 42.

trust_remote_code: Sets trust_remote_code to True to execute code to

create HF Datasets from the Hub

verbosity: EEH logging level max_tokens: Maximum number of tokens to generate. temperature: Sampling temperature used for generation. top_k: Top-k parameter used for generation. top_p: Top-p parameter used for nucleus sampling.

Other Extensions#

Other extensions implemented as callbacks that can be used to enhance the Trainer.

HFCacheDir#

class cerebras.modelzoo.trainer.extensions.HFCacheDir(cache_dir)[source]#

Bases: cerebras.modelzoo.trainer.callbacks.callback.Callback

A callback that sets up the HuggingFace cache directory.

Parameters

cache_dir (str) – The cache directory to use for HuggingFace utilities.

WandbLogger#

class cerebras.modelzoo.trainer.extensions.WandbLogger(project=None, group=None, run_id=None, run_name=None, job_type=None, tags=None, resume='auto', entity=None)[source]#

Bases: cerebras.modelzoo.trainer.loggers.logger.Logger

Logger class for logging metrics to Weights and Biases.

Parameters

  • project (Optional[str]) – The name of the project to which the run belongs.

  • group (Optional[str]) – The name of the group to which the run belongs.

  • run_id (Optional[str]) – The unique identifier for the run.

  • run_name (Optional[str]) – The name of the run.

  • job_type (Optional[str]) – The type of job.

  • tags (Optional[List[str]]) – List of tags to be associated with the run.

  • resume (Literal['never', 'allow', 'auto', 'must']) – Resume mode for the run. It can be one of the following: - “never”: Do not resume the run. - “allow”: Allow the run to resume if a previous run exists. - “auto”: Automatically resume the run if a previous run exists. - “must”: Resume the run if a previous run exists.

  • entity (Optional[str]) – An entity is a username or team name where you’re sending runs. This entity must exist before you can send runs there, so make sure to create your account or team in the UI before starting to log runs.

check_presence_of_wandb_dir(rundir)[source]#

Check if the wandb directory is present in the run directory.

Parameters

rundir – The directory where the run is being stored.

previous

Cerebras Model Zoo Callbacks

next

Cerebras Model Zoo Loggers