cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriterView#

class cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriterView[source]#

Bases: object

StreamingCSWriterView allows for checkpoints with arbitrarily nested dictionaries/lists to be written in a streaming (incremental) manner by offering a “view” into a StreamingCSWriter. For example, in a checkpoint with the structure {“model”: {<model state>}}, we can obtain a view into the model state via checkpoint[“model”]. This view has state <model state> and prefix [“model”]. The view acts like a dict (offers __getitem__, __setitem__, etc operations) which incrementally saves/loads from an H5 checkpoint under the hood.

Parameters

  • checkpoint_file – Path to H5 checkpoint

  • state – (Sub)state dictionary corresponding to the current view of the checkpoint.

  • prefix – Chain of keys that were accessed in the checkpoint that yielded the current view

Methods

get

items

keys

pop

values

__init__(checkpoint_file, state, prefix=None) → None[source]#

previous

cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriter

next

cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingShardedHFReader