common.pytorch.summaries package#

Submodules#

common.pytorch.summaries.cb_summary module#

Base class for creating summaries compatible with CPU/GPU/CS-X.

class common.pytorch.summaries.cb_summary.CBSummary[source]#

Bases: object

Base class for creating summaries on CS devices.

Subclasses must override methods to provide the full functionality of the metric. These methods are meant to split the summary operations into 3 portions:

run_on_device: Compiles and runs on the device (i.e., Cerebras).

run_on_host: Runs on the host (i.e., CPU).

save: Saves the results as summaries.

These summaries also support running on CPU and GPU.

Constructs a CBSummary instance.

This also registers the summary in the global pool of summaries. Therefore, it is import for subclasses to call super().__init__(). Otherwise, the summaries will not run.

Parameters: name – Name of the summary. Generally, this is the name used to tag the summaries with in TensorBoard.

__init__(name: str)[source]#

Constructs a CBSummary instance.

This also registers the summary in the global pool of summaries. Therefore, it is import for subclasses to call super().__init__(). Otherwise, the summaries will not run.

Parameters: name – Name of the summary. Generally, this is the name used to tag the summaries with in TensorBoard.

discard() → None#: Discards cached activations on host.

property name#: Returns the name of the metric.

run_on_device(*args, **kwargs) → common.pytorch.summaries.cb_summary.DeviceOutputs[source]#

Define the portion of the summary computation that runs on device.

This method must return a DeviceOutputs object whose args/kwargs can only contain a item/list/tuple/dict of torch tensors or Nones. These tensors may be converted to CPU tensors at the step boundary and passed to run_on_host to do the host (i.e. CPU) portion of the computation.

The default implementation is just a passthrough where the arguments are converted to host tensors as is.

This method is called for every iteration.

NOTE: No tensors should be evaluated in this method. This method merely defines the operations in the graph that runs on device.

Returns: An instance of DeviceOutputs.

abstract run_on_host(*args, **kwargs) → Any[source]#

Define the portion of the summary computation that runs on host.

This methods takes as inputs the outputs of run_on_device whose tensors have been evaluated and converted to CPU tensors. It can do any sort of computation on the host (e.g., applying reduction). Its return value is passed to save_on_host.

This method is called only when the summary is to be saved.

Returns: The computed summary value.

save(*args, **kwargs) → None#

Saves the results.

This method is intended to be called inside the training loop whenever the summary results need to be saved. The arguments are passed directly to the save_on_host method.

NOTE: This method should not be overriden. Instead, run_on_host and save_on_host should be overriden to provide full functionality.

abstract save_on_host(host_outputs: Any, writer: torch.utils.tensorboard.SummaryWriter, step: int) → None[source]#

Save the computed summary into events files.

Parameters

host_outputs – The return value of run_on_host method. This is the computed summary value.
writer – A writer for writing summaries to events files.
step – The current global step.

class common.pytorch.summaries.cb_summary.CBSummaryMeta[source]#

Bases: abc.ABCMeta

Metaclass for CBSummary to handle instance creation.

class common.pytorch.summaries.cb_summary.DeviceOutputs[source]#

Bases: object

Class for encapsulating the outputs of CBSummary.run_on_device.

Parameters

args – postional arguments which are passed to CBSummary.run_on_host once they are converted to CPU tensors.
kwargs – keyword arguments which are passed to CBSummary.run_on_host once they are converted to CPU tensors.

__init__(args: typing.List[typing.Any] = <factory>, kwargs: typing.Dict[str, typing.Any] = <factory>) → None#

args: List[Any]#

kwargs: Dict[str, Any]#

common.pytorch.summaries.cb_summary.discard_cached_summaries() → None[source]#: Discards cached activations for all summaries.

common.pytorch.summaries.cb_summary.get_all_summaries() → Dict[str, common.pytorch.summaries.cb_summary.CBSummary][source]#: Returns all registered summaries.

common.pytorch.summaries.cb_summary.save_all_summaries(writer: torch.utils.tensorboard.SummaryWriter, step: int) → None[source]#

Calls save on all registered summaries.

Parameters

writer – A writer for writing summaries to events files.
step – The current global step.

common.pytorch.summaries.scalar_summary module#

Utilities for saving scalar summaries.

common.pytorch.summaries.scalar_summary.scalar_summary(name: str, tensor: torch.Tensor, reduction: Optional[str] = None)[source]#

Convenience method for creating and running scalar summaries.

This method searches registered summaries for the given name. If one is found, it uses it. Otherwise, it creates a new summary and runs the tensor through that summary.

Parameters

name – Name of the summary. This is the tag that appears in TensorBoard.
tensor – The tensor to be summarized.
reduction – The reduction function to apply to the tensor. Defaults to no reduction.

common.pytorch.summaries.tensor_summary module#

Utilities for saving tensor summaries.

class common.pytorch.summaries.tensor_summary.TensorDescriptor[source]#

Bases: object

Descriptor for a summarized tensor.

Parameters

step – Step at which the tensor was summarized.
ns_since_epoch – Nanoseconds since “epoch” (e.g., UNIX time).
tensor – The summarized tensor.

__init__(step: int, ns_since_epoch: int, tensor: torch.Tensor) → None#

static from_dict(values) → common.pytorch.summaries.tensor_summary.TensorDescriptor[source]#: Returns a descriptor from a dict of values.

ns_since_epoch: int#

step: int#

tensor: torch.Tensor#

to_dict() → dict[source]#: Returns the descriptor converted to a dict.

property utctime: datetime.datetime#: Returns the UTC time when this tensor was saved.

class common.pytorch.summaries.tensor_summary.TensorSummary[source]#

Bases: modelzoo.common.pytorch.summaries.cb_summary.CBSummary

A class for providing tensor summaries on CS/CPU/GPU devices.

In constrast to other summaries, such as scalar summaries, Tensor summaries are not written to Tensorboard events files and are not viewable in Tensorboard. Instead, they are written to files and a convenience API class is provided to load the values (see TensorSummaryReader class below).

This class still takes in a SummaryWriter, like other summary classes, but instead of using it to save values into events files, it identifies the events file path, creates a sibling directory with a similar naming scheme, and places summarized tensor data in that directory. Users are discouraged from inspecting this directory as the implementation might change. Instead, users are encouraged to use the API provided below for loading summarized tensors.

Constructs a TensorSummary instance.

Parameters: name – Name of the summary. This is the tag that appears in TensorBoard.

__init__(name: str)[source]#

Constructs a TensorSummary instance.

Parameters: name – Name of the summary. This is the tag that appears in TensorBoard.

run_on_device(tensor: torch.Tensor) → modelzoo.common.pytorch.summaries.cb_summary.DeviceOutputs[source]#

Define the portion of the summary computation that runs on device.

Parameters: tensor – The tensor to be summarized.
Returns: An instance of DeviceOutputs.

run_on_host(tensor: torch.Tensor) → torch.Tensor[source]#

Runs the host portion of the summary computation.

Parameters: tensor – The tensor to be summarized.
Returns: The summarized tensor.

save_on_host(host_outputs: torch.Tensor, writer: torch.utils.tensorboard.SummaryWriter, step: int) → None[source]#

Saves the tensor summary to events file.

Parameters

host_outputs – The summarized tensor to write to events file.
writer – A writer for writing summaries to events files.
step – The current global step.

class common.pytorch.summaries.tensor_summary.TensorSummaryReader[source]#

Bases: object

Class for reading summarized tensors.

This class works in tandem with TensorSummary defined above. It provides general convenience APIs for inspecting tensor summaries produced by a run.

Currently this class does not do any caching. So it can be used to inspect a live run. As more data becomes available, calling the APIs will reload the latest values.

Constructs a TensorSummaryReader instance.

Parameters: path – Path to a Tensorboard events file or a directory containing Tensorboard events files. Location of tensor summaries are inferred from these events files as there is a one-to-one mapping from Tensorboard events files and tensor summary directories.

__init__(path: str)[source]#

Constructs a TensorSummaryReader instance.

Parameters: path – Path to a Tensorboard events file or a directory containing Tensorboard events files. Location of tensor summaries are inferred from these events files as there is a one-to-one mapping from Tensorboard events files and tensor summary directories.

load(name: str, step: int, latest_only: bool = True) → Optional[Union[common.pytorch.summaries.tensor_summary.TensorDescriptor, List[common.pytorch.summaries.tensor_summary.TensorDescriptor]]][source]#

Loads and returns tensor(s) with given name at the given step.

Parameters

name – Name of the tensor.
step – Step at which the tensor was summarized.
latest_only – If False, return all if there are multiple tensors with the same name and step. If True, only return the latest value.

Returns

A single tensor, multiple tensors, or no tensors matching the given: name and step.

names() → List[str][source]#: Returns a list of available tensor names.

common.pytorch.summaries.tensor_summary.tensor_summary(name: str, tensor: torch.Tensor)[source]#

Convenience method for creating and running tensor summaries.

This method searches registered summaries for the given name. If one is found, it uses it. Otherwise, it creates a new summary and runs the tensor through that summary.

Parameters

name – Name of the summary. This is the tag that appears in TensorBoard.
tensor – The tensor to be summarized.

Module contents#

Package for providing summaries that work on CPU/GPU/CS-X.

common.pytorch.sparsity package

common.run_utils package