common.run_utils package#
Submodules#
common.run_utils.cli_parser module#
CLI Utilities
- common.run_utils.cli_parser.add_csx_arguments(parser: argparse.ArgumentParser, first_parse: bool = True)[source]#
Injects Cerebras System specific parser arguments.
- Parameters
parser – Parser into which the arguments are being added.
first_parse – Boolean indicating whether this is the first time processing the arguments. If True, default args are ignored in case the other has them stored elsewhere. If False, default args are set.
- common.run_utils.cli_parser.add_general_arguments(parser: argparse.ArgumentParser, default_model_dir: str, first_parse: bool = True)[source]#
Injects general parser arguments.
- Parameters
parser – Parser into which the arguments are being added.
default_model_dir – String containing model directory path.
first_parse – Boolean indicating whether this is the first time processing the arguments. If True, the “params” arg is required to get additional parameters, if False then the params file has already been read and is not required.
- common.run_utils.cli_parser.add_gpu_arguments(gpu_parser: argparse.ArgumentParser, first_parse: bool = True)[source]#
Injects GPU specific parser arguments.
- Parameters
gpu_parser – Parser into which the arguments are being added.
- common.run_utils.cli_parser.assemble_disallowlist(params: dict, extra_args_parser_fn: Optional[Callable[[], List[argparse.ArgumentParser]]] = None) list[source]#
Determine invalid parameters for the current device type.
- common.run_utils.cli_parser.discard_params(device: str, extra_args_parser_fn: Optional[Callable[[], List[argparse.ArgumentParser]]] = None)[source]#
External utility for determining invalid parameters for the current device type.
- common.run_utils.cli_parser.get_all_args(extra_args_parser_fn: Optional[Callable[[], List[argparse.ArgumentParser]]] = None)[source]#
Helper for returning all valid params for each device.
- common.run_utils.cli_parser.get_params(params_file: str) dict[source]#
Reads params from file and returns them as a dict.
- Parameters
params_file – The YAML file holding the params.
config – Optional config to load from the params. If None, the default config is returned. Defaults to None.
- Returns
A dict containing the params.
- common.run_utils.cli_parser.get_params_from_args(run_dir: Optional[str] = None, argv: Optional[List] = None, extra_args_parser_fn: Optional[Callable[[], List[argparse.ArgumentParser]]] = None) dict[source]#
Parse the arguments and get the params dict from the resulting args
- Parameters
run_dir – The path to the run.py file
argv – The args to be parse. Defaults to sys.argv if not provided
- common.run_utils.cli_parser.get_parser(run_dir: Optional[str] = None, first_parse: bool = True, extra_args_parser_fn: Optional[Callable[[], List[argparse.ArgumentParser]]] = None) argparse.ArgumentParser[source]#
Returns an ArgumentParser for parsing commandline options.
- Parameters
run_dir – String to be used to determine model directory.
first_parse – Boolean indicating whether this is the first time processing the arguments. If True, the parser is being used to collect commandline inputs. If False, it is only being used for verification on existing params.
extra_args_parser_fn – Parent parser passed in by models with unique specific arguments.
- Returns
A parser instance.
- common.run_utils.cli_parser.post_process_params(params: dict, valid_arguments: list, invalid_arguments: list) list[source]#
Removes arguments that are not used by this target device.
- common.run_utils.cli_parser.read_params_file(params_file: str) dict[source]#
Helper for loading params file.
- common.run_utils.cli_parser.update_defaults(params: dict, default_params: dict) dict[source]#
Updates the params dict with global default for a key if a key is not present. Works on nested dictionaries and recursively updates defaults for nested dictionaries. All other types, apart from dict are considered as base type and aren’t updated recursively. :param params: dict holding the params. :param default_params: dict holding the default params.
- Returns
A dict containing the params, with the defaults updated
- common.run_utils.cli_parser.update_params_from_args(args: argparse.Namespace, params: dict, sysadmin_params: dict)[source]#
Update params in-place with the arguments from args.
- Parameters
args – The namespace containing args from the commandline.
params – The params to be updated.
first_parse – Indicates whether to keep in the params path.
common.run_utils.cli_pytorch module#
PyTorch CLI Utilities
common.run_utils.utils module#
Contains common run related utilies