Software Release Notes#

The following are the release notes for the Cerebras software.

Release 1.9.1#

New features and enhancements#

Large Language Models#

Maximal Update Parameterization (muP), used for improving training stability and transferring hyperparameters from smaller language models to Larger Language Models (including CerebrasGPT), is now available for GPT-2 and GPT-3 style models. See the How-to guide for usage.
New checkpoint converters between Hugging Face and Cerebras formats have been added. See more at Convert checkpoints and configurations.
Gradient accumulation is enabled for all transformer language models in 1.9.1 through YAML config.
Pre-trained Falcon 7B is supported in Model Zoo.
Pre-trained LLaMA 7B, 13B, and 33B are supported in Model Zoo.
BLOOM 7B is available in Model Zoo. ALiBi positional encodings can be enabled in all GPT-style models through the model section in the configuration yaml.

position_embedding_type: 'alibi'

alibi_trainable_slopes: False # whether the slopes of the alibi embedding is trainable (default to False).

alibi_implementation: 'expand' # We support `embedding` and `expand` with default set to `expand`.

Computer vision models#

Fixed bugs and improved performance for computer vision models.

Other features#

Improved stdout messages. Added console progress bars to provide more detail about the operations on the Wafer Scale Cluster.
Improved tooling for cluster resource management, job monitoring, and performance monitoring. For more information, see csctl: CLI tool for job monitoring.
Pipeline mode and TensorFlow support is deprecated in 1.9.1. All models must use PyTorch and weight streaming functionality. There is no longer a need to specify a {pipelined,weight_streaming} argument in run.py because all models will run in weight_streaming mode by default. All models previously supported in Pipeline are now supported for Weight Streaming.
The batch_size parameter in Model Zoo yaml configuration files now represents the total effective batch size of the model and is divided evenly across the specified num_csx CSX systems. This differs from pre-1.9.0 behavior, where the batch size parameter defined the batch size per CSX, not globally. Note that batch_size must now be divisible by num_csx.
Custom worker containers are enabled by default, allowing user environment to be replicated into the worker servers running inside the appliance. The deployment/admin user needs to specify the cluster volume that can be used for virtual environment copy. This feature can also be turned off during deployment if needed.

Known Issues#

Some dataloader implementations from Model Zoo require evaluation to be done on a single CS-2 rather than multiple CS-2s. Multibox evaluation has no explicit limitation, but these dataloaders require the dataset to be sharded in such a way that each worker gets at least one file. Evaluation datasets are often small and not split into many files.
All T5 limitations from Release 1.8 remain.
Loss scaling by number of tokens (num_tokens) is not yet fully supported and requires coordination with the Cerebras team.
GPT NeoX suffers NaNs when trained with extremely long sequence lengths (30k, 50k).
The base, pre-trained Falcon and LLaMA variants are supported. Other variants, such as those with long sequence lengths or different numbers of heads, may not be supported.
Running several jobs in parallel with the same model_dir can cause issues.

Release 1.9.0#

Note

1.9.0 was a special, small-distribution release. 1.9.1 is our general release.

Release 1.8.0#

New features and enhancements#

Large language models#

Added support for T5 models up to 11B parameters with Weight Streaming execution mode. T5 is supported with source and target inputs up to 2K tokens.
Added support for BERT pre-training with Weight Streaming execution mode. BERT is supported with input sequences up to 30K tokens.
Added support for gradient accumulation for GPT-style and BERT-style language models, allowing for larger effective batch sizes. See Train with gradient accumulation for more details.
Added support for deterministic checkpointing of dataloaders for language models to enable pausing and restarting of training runs without using duplicate samples or batches. See Deterministically restart a dataloader for more details.
In past releases pre-layer normalization in our T5 & Transformer models required setting use_pre_encoder_decoder_layer_norm: False. This was confusing, and we have changed the behavior in 1.8. To enable pre-layer normalization you should instead set use_pre_encoder_decoder_layer_norm: True. This update better aligns the naming of the parameter to its usage. To use release 1.7 checkpoints in release 1.8, you’ll need to update the config to reflect this change. Directions for converting configuration files can be found in our checkpoint conversion documentation.
You may now control the activation function used by the BERT pooler (pooler_nonlinearity) and masked language model head (mlm_nonlinearity) independently of the activation used for the rest of the model (encoder_nonlinearity). Both will default to encoder_nonlinearity if not explicitly set. Use the checkpoint conversion documentation to convert 1.7 configuration files to 1.8 to have access to this feature.

Computer vision models#

Added support for 3D UNet model in Weight Streaming execution to enable segmentation of large volumes of up to 512 x 512 x 160 pixels. Single CS-2 system support only. Check our reference implementation and learn more in the Model Zoo.
Multi-channel multi-class segmentation is now supported for 2D UNet.
2D UNet now supports image sizes up to 7k x 7k
Improved 2D CNN robustness & support, for example ResNet 2D on up to 1k x 1k images
2D UNet now runs at high performance on 256 x 256 resolution

Other features#

We are releasing scripts and instrucions for checkpoint and configuration conversion to and from corresponding Hugging Face models and between Cerebras software versions. More details in Convert checkpoints and configurations.
Added support for eval_all and train_and_eval to enable users to evaluate models throughout long training runs or evaluate all checkpoints after training has completed. Mode details in Evaluate your model during training.
The Cerebras Model Zoo run scripts have been updated with a more informative and explicit command line interface. For more details please read Launch your job.
The script run_appliance.py has now been deprecated for TensorFlow in favour of one single script called run.py that employs the aforementioned run script changes.
More custom losses and non-linearities are now supported with AutoGen feature. It allows users to swap operations (losses, nonlinearities, positional encodings) for language models and improves performance of loss functions with fused kernels. Learn more about Kernel autogeneration with AutoGen.
You can now use scalar and tensor summaries in PyTorch to track various tensors of interest during training. Internally we heavily rely on these features for example to track parameters like gradient norms during training of large language models. Learn mode about how to Summarize scalars and tensors.

Known Issues#

T5 with input or output sequences longer than 1024 tokens (src_max_sequence_length and tgt_max_sequence_length parameters in model yaml config file) may have compile times of over 3 hours. T5 is only supported with input and output sequences up to 2048 tokens.
T5 has limitations with respect to gradient accumulation and batch sizes (BS).
- Gradient accumulation is not supported for T5.
- At precision optimization level 0 (POL0), the largest supported batch size for T5 model with 11B parameters is 220.
- At precision optimization levels 1 and 2 (POL1 and POL2) batch sizes over 770 for T5 3B and over 260 for T5 11B will result in a long compile time.
- Models will not compile if (vocabulary V / (heads * Greatest_Common_Divisor(Sin, Sout)) > 2^11.
Maximum supported vocabulary size for language models is 1 million.
Downloading data sets from the internet within data loaders is not supported. As a workaround, please download data sets and prepare them outside the dataloader function. See PyTorch FC MNIST implementation for an example and additional details.
Users creating their own models and training scripts must separate the dataloader or input_fn into a separate Python file from the rest of the training script, in order to avoid the error described here.
The experimental PyTorch API does not save checkpoints at steps 0 and 1 in the correct format. No issues with checkpoints at other steps or outside the experimental API.
When share_embedding_weights parameter is set to True for PyTorch GPT-style models (e.g. here), custom settings for embedding_initializer in the yaml configuration (like here), are ignored. Embedding weights are initialized with default configuration (normal distribution, std=0.02). This is done by the __rest_parameters function, which is called at the end of the GPT2LMHeadModel initialization

Release 1.7.1#

New features and enhancements#

Unified user workflow on the Wafer-Scale Cluster#

User workflow on the Wafer-Scale Cluster is now the same for Pipelined and Weight Streaming execution. The same launching scripts and the same environment can now be used to run larger models with Weight Streaming execution and smaller models with Pipelined execution. Use additional command line argument with run.py for PyTorch and with run_appliance.py for Tensorflow: set --execution_strategy argument to pipeline or weight_streaming to specify execution mode.
Note that Pipelined execution only supports using one CS-2 for each run. It is only valid to specify --num_csx=1 in the run command for Pipelined execution. Weight Streaming does not have such requirement.

Known issues#

Specifying number of workers#

--num_workers_per_csx denotes the maximum number of Kubernetes worker pods per CS-X. Note that these are distributed across physical worker nodes attached to each CS-X on the Wafer-Scale Cluster.
For this release, please specify --num_workers_per_csx=8 as a command line argument in the run command for Pipelined execution. Weight Streaming execution does not need to specify this argument.

Specifying path in the configuration file#

Please specify absolute paths for any configuration parameters that are used by dataloaders in PyTorch or the input_fn in TensorFlow. More specifically, this applies to all variables in train_input and/or eval_input sections of the configuration yamls.

Padding token in loss calculation#

TensorFlow BERT fine-tuning token classification model does not support padding tokens in loss on the Wafer-Scale Cluster for both Pipelined and Weight Streaming execution. Please set include_padding_in_loss: False in the configuration yaml. We believe it makes the most sense to exclude padding tokens in the loss calculation. Such setting differs from the original public implementation where token padding is included in the loss, which is most likely used for performance optimization on GPUs, leading to our eval accuracy being potentially different from published numbers. This does not apply to the PyTorch version or the Original Cerebras Installation.

TensorFlow BERT SQuAD Unsupported#

TensorFlow BERT fine-tuning model for SQuAD is not supported in appliance (neither Pipelined nor Weight Streaming). If you would like to fine-tune BERT with SQuAD, please use the PyTorch version.

Release 1.7.0#

New features and enhancements#

Large language models#

Added support for GPT-J-style models up to 20B parameters in PyTorch, e.g. GPT-J 6B, GPT-NeoX 20B. More example configs can be found in Cerebras Model Zoo.
Added new GPT-3 variants up to 20B parameters both in TensorFlow and PyTorch.
Improved performance (throughput) for GPT-style models by up to 1.5x compared to the previous release.

Computer vision models#

Added support for UNet 2D in Weight Streaming execution to enable large input sizes of up to 5k x 5k resolutions. Both Training and Eval workflows are supported. Single CS-2 system support only. Single-channel single-class segmentation only in this release.

Other features#

Added support for bfloat16 data type for large language models in PyTorch and enabled precision knobs for performance optimization.
Added support for training language models with sparse weights in PyTorch; training with static sparsity masks is now enabled, and scripts to convert a dense PyTorch checkpoint into a sparse version are available.
Improved error messages for easy-to-understand, user-actionable errors.
Expanded support for PyTorch learning rate schedules and loss functions.

Known issues#

Running eval with UNet in PyTorch, Weight Streaming execution#

UNet model eval with metrics mIOU, DSC on images of size 4096 x 4096 pixels causes failures and is a known issue. All default configurations and image sizes published in the ModelZoo have been tested and expected to work without issues. The issue may manifest itself with other non-tested image sizes. The expected error is as follows:
status = StatusCode.INTERNAL details = "KM to RT IR Translation Failed" debug_error_string = "UNKNOWN:Error received from peer ipv4:10.254.104.16:443 {grpc_message:"KM to RT IR Translation Failed", grpc_status:13, created_time:"2022-12-15T00:27:47.134671257-08:00"}"
Please contact Cerebras support if you run into failures.

Release 1.6.1#

New features and enhancements#

Weight Streaming models in PyTorch#

Early access support for GPT model variants, up to 1.5B parameters in PyTorch on single CS-2 systems.

Release 1.6.0#

New features and enhancements#

Weight streaming models#

Up to 2x performance improvement for GPT-style models with weight streaming training on CS-2 systems.
Early access support for additional GPT model variants, up to 20B parameters on single CS-2 systems.

Cerebras Wafer-Scale Cluster#

This is the first production release of software support for weight streaming on the Cerebras Wafer-Scale Cluster. This includes a simple workflow that allows users to easily submit and scale large training jobs to a Cerebras cluster. Refer to our TF and PyTorch getting started guides on how to run your pipeline and weight streaming models on the cluster.

Multi-node CS-2 support#

Expands Cerebras Wafer-Scale Cluster to support 8x CS-2s on GPT-style models and achieves near linear scaling performance.

PyTorch support#

Introduces Cerebras PyTorch Layer API that implements a subset of PyTorch APIs with Cerebras custom implementations that take advantage of our high-performance kernels and provides drop-in replacement to the native PyTorch version.
Includes a demo of GPT2 model that uses the Layer API and implements the model.
Added support for label-smoothed cross entropy for pipelined models in PyTorch.

Cerebras Model Zoo#

Launched a public version of Cerebras sample model implementations through Cerebras Model Zoo github repository.

Release 1.5.0#

New features and enhancements#

Weight streaming#

Significant performance boost in the evaluation of accuracy for large-scale GPT models. Evaluation performance is now on par with training performance.
Added support for vocab sizes up to 2³¹ for GPT-style models when run with weight streaming execution.

Multi-node CS-2 support#

Expands Cerebras Wafer-Scale Cluster to support 4x CS-2s on GPT-style models.
Cerebras Wafer-Scale Clusters are composed of CS-2 systems, MemoryX and SwarmX nodes, input pre-processing servers and associated internal network switches. End-user workflow is supported via appliance model, where the user submits a job to the cluster as if it were a single device.

Note

To learn more and get a demo of our cluster capabilities and appliance workflow, contact Cerebras support by sending email to support@cerebras.net.

Note

CSoft 1.5 deprecates some of our experimental models that were brittle and not suitable to run with many variants of model implementation. The deprecated models are: RNN, GNN, CTR, Audio and RevBERT which were previously supported in demo mode.

Release 1.4.0#

New features and enhancements#

Weight Streaming#

Single CS-2 System now supports training of multi-billion parameter NLP models including GPT-3XL 1.3 billion, GPT-J 6 billion, GPT-3 13 billion and more via the weight streaming mode of execution.
Training on GPT-J (6B parameters) is now supported for longer sequences up to 30K tokens on single CS-2 system.
Switching between these extreme scale models can be achieved by just a few changes in config file.

PyTorch models#

We have made performance Improvements for small- to medium-sized PyTorch models through support for Multi-Replica (MR) + Variable Tensor Shape (VTS), including: - Multi-replica improves fabric utilization and throughput performance. - VTS improves performance for dataset with variable length sequence.
Models now support MR + VTS: BERT, Transformer (AIAYN) and T5.
We added an adafactor optimizer support to T5 model in PyTorch to achieve robust convergence.

TensorFlow models#

We have added support for a multi-replica mode and variable sequence length to T5 model in pipeline execution to further boost performance on CS-2 System.

Multi-node CS-2 support#

This release introduces 2X CS-2 cluster in demo mode that leverages a weight streaming support cluster (composed of CS2 systems, MemoryX and SwarmX systems, worker servers and associated internal network switches) to efficiently run extreme scale NLP models.
The end-to-end user workflow can be thought of and handled as a single network-attached service appliance. In this appliance model, the user submits a job to the weight streaming cluster as if it were a single device.

Note

To learn more and get a demo of our cluster capabilities and appliance workflow, contact Cerebras support by sending email to support@cerebras.net.

Known issues#

Running eval on Cerebras system#

When running in eval mode on CS-1 system, if --nodes and --tasks_per_node value pairs are not set to one of the following, then the session may hang. This issue exists for both TensorFlow and PyTorch.

--nodes==1 --tasks_per_node=2, or

--nodes==2 --tasks_per_node=1

Workaound: Make sure that you use one of the above settings for --nodes and --tasks_per_node. For example:
--nodes=1 --tasks_per_node=2
The eval performance is not affected by these Slurm resource settings. See the example command below:
csrun_wse python run.py --mode=eval \
    --nodes=1 --tasks_per_node=2 \
    --params configs/your-params-file.yaml \
    --model_dir your-model-dir \
    --cs_ip=10.255.253.0

Release 1.3.0#

New features and enhancements#

PyTorch models#

Supports Variable Tensor Shapes (VTS) for Transformer, T5 and BERT models, which boosts performance significantly
Added support for BERT Finetuning tasks: SQUAD (Q&A), Classifier (SST) and Summarization (SUM)
Supports fixed positional embeddings.
Upgrades to the latest version of PyTorch 1.11.

Weight streaming mode#

GPT-J 6B-parameter model in Tensorflow is supported for pretraining on single CS-2 system
The abstractive summarization fine-tuning task supported for GPT-J (6B parameters).
Eval metrics supported for GPT-2, GPT-3 variants, GPT-J. Metrics include perplexity, accuracy and BPB, BPC, BPW.

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Multi-replica mode#

Multi-replica mode is now supported across Transformer and BERT Tensorflow models.
Multi-replica mode also adds Variable Tensor Shape support to further boost performance for these models.

Known issues#

GPT-J (6B parameters) model#

There is a non-determinism on the GPU side we are currently debugging, so in order to match the GPU reference, CS-2 run should start from the same initial checkpoint.
There is an unexplained shuffle happening when the input function runs out of data and needs to repeat the dataset. So, in order to get the exact match, the reference should run for less number of steps than the dataset, or the dataset needs to be extended, so that repeat doesn’t happen.
When running the GPT-J 6B model, each weight streaming server should be configured to have 512 GB of total memory. It is recommended to have at least 128 GB of physical memory and any remainder as swap space.

Running eval on Cerebras system#

--nodes==1 --tasks_per_node=2, or

--nodes==2 --tasks_per_node=1

Workaound: Make sure that you use one of the above settings for --nodes and --tasks_per_node. For example:
--nodes=1 --tasks_per_node=2
The eval performance is not affected by these Slurm resource settings. See the example command below:
csrun_wse python run.py --mode=eval \
    --nodes=1 --tasks_per_node=2 \
    --params configs/your-params-file.yaml \
    --model_dir your-model-dir \
    --cs_ip=10.255.253.0

Release 1.2.0#

New features and enhancements#

PyTorch models#

Train and Eval mode is now supported for PyTorch BERT Base with sequences upto 4k tokens and BERT Large with sequences upto 2k tokens. Includes support for common eval metrics (eval loss, MLM accuracy, NSP accuracy, perplexity).
Train and Eval mode is now supported for RoBERTa configuration in PyTorch BERT.
Adds support for BERT-NER finetuning
Train and Eval mode is now supported for the PyTorch Transformer-Attention is All You Need model.
Train and Eval mode is now supported for PyTorch T5 model with configurations up to ~500M parameters, e.g., T5-Small 60M and T5-Base 220M
Train and Eval mode is now supported for PyTorch GPT-2 model with configurations with up to ~770M parameters, e.g., GPT-2 Small 117M, GPT-2 Medium 345M, GPT-2 Large 774M

Weight streaming execution mode#

A new execution mode, called weight streaming mode, to run extremely large models, is introduced as an early release. See Weight Streaming Execution for a detailed explanation of the weight streaming concept.
In weight streaming mode, support is added for eval on GPU.
In weight streaming mode, support is added to store checkpoints and resume training from checkpoints.
Support is added in weight streaming mode to track training runs with TensorBoard.

Weight streaming models#

The following models support weight streaming mode. These models are in early beta.

GPT-3 XL (1.3 billion total parameters) running on a single CS-2 system.

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Input analyzer for Slurm resources#

The cs_input_analyzer is a new Bash script that recommends Slurm resource settings you need to run on Cerebras system. These recommendations are generated by this script for a given input_fn and model. To use this tool, run it manually. See The cs_input_analyzer Script.

Known issues#

Running eval on Cerebras system#

--nodes==1 --tasks_per_node=2, or

--nodes==2 --tasks_per_node=1

Workaound: Make sure that you use one of the above settings for --nodes and --tasks_per_node. For example:
--nodes=1 --tasks_per_node=2
The eval performance is not affected by these Slurm resource settings. See the example command below:
csrun_wse python run.py --mode=eval \
    --nodes=1 --tasks_per_node=2 \
    --params configs/your-params-file.yaml \
    --model_dir your-model-dir \
    --cs_ip=10.255.253.0

Release 1.1.0#

New features and enhancements#

PyTorch#

The PyTorch support is enhanced. Key changes include but not limited to:
- Support for eval mode is added for BERT and FC-MNIST PyTorch models. These models now support both train and eval modes.
- Enhanced the flexibility in specifying the cerebras.framework.torch.initialize().
- Use of cbfloat16 data format (see CB16 Half-Precision) is now supported.
- Made mixed precision interface more intuitive, via GradScaler (see Train with dynamic loss scaling).
- Fixed several bugs in the areas of numerics, convergence and performance.

PyTorch models#

The following PyTorch models are supported.

A PyTorch version of FC-MNIST.
The PyTorch versions of BERT Base and BERT Large.
- RoBERTa (Next Sentence Prediction (NSP) only) configurations are supported. See roberta_base.yaml and roberta_large.yaml.
- Longer Maximum Sequence Length (MSL) configurations are supported, at least up to MSL 4096.
The PyTorch Transformer-Attention is All You Need model is added as a Beta feature.
This model can be compiled using run.py with the --compile_only flag, as well as run on CPU or GPU using run_cpu_gpu.py.

To train this model on the Cerebras System at your own risk, comment out the following lines from run.py:
if not runconfig_params["compile_only"]: raise ValueError( "Running the Transformer model on the Cerebras System is in beta." "Convergence is not guaranteed. Remove this exception to proceed." )

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Supported PyTorch ops#

A preliminary list of supported PyTorch ops is released.

Multi-replica data parallel training#

A new feature called multi-replica data parallel training is released. Currently this feature is available only for TensorFlow models. When you use this feature, the Cerebras compiler uses several copies (replicas) of the same model to run data parallel training.

Known issues#

T5 and Transformer (Attention is All You Need)#

The TensorFlow versions of the T5 and Transformer models are not guaranteed to converge. These models can still be compiled to the Cerebras system. However, to train these models on the Cerebras System at your own risk, comment out the following lines from run.py of the model:
if not runconfig_params["compile_only"]: raise ValueError( "Running the Transformer model on the Cerebras System is in beta." "Convergence is not guaranteed. Remove this exception to proceed." )
When you train the TensorFlow Transformer model on Cerebras system, you will see a modest increase in loss volatility, compared to the runs on GPUs. This is due to numerical differences. The pre-training eval accuracy is expected to be within a few percent of the equivalent model trained on a GPU.

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Running eval on Cerebras system#

--nodes==1 --tasks_per_node=2, or

--nodes==2 --tasks_per_node=1

Workaound: Make sure that you use one of the above settings for --nodes and --tasks_per_node. For example:
--nodes=1 --tasks_per_node=2
The eval performance is not affected by these Slurm resource settings. See the example command below:
csrun_wse python run.py --mode=eval \
    --nodes=1 --tasks_per_node=2 \
    --params configs/your-params-file.yaml \
    --model_dir your-model-dir \
    --cs_ip=10.255.253.0

Multi-replica data parallel training#

Eval on Cerebras system is not yet supported for multi-replica data parallel trained models. You can run eval on CPU or GPU for these models.

PyTorch#

For PyTorch, when you are targeting GPU, the following warning will be displayed. This can be safely ignored. This issue does not exist when you target Cerebras system for your acceleration.

UserWarning: Detected call of ``lr_scheduler.step()`` before
``optimizer.step()``. In PyTorch 1.1.0 and later, you should
call them in the opposite order: ``optimizer.step()`` before
``lr_scheduler.step()``.  Failure to do this will result in
PyTorch skipping the first value of the learning rate schedule.

For PyTorch models only, to run the training on the Cerebras system, the cs_ip flag must include both the IP address and the port number of the CS system. Only the IP address, for example: --cs_ip 192.168.1.1, will not be sufficient. You must also include the port number, for example: --cs_ip 192.168.1.1:9000.

Release 1.0.0#

New features and enhancements#

PyTorch (BETA)#

Support is added, in beta phase only, for the PyTorch framework. The models and quickstart provided are strictly intended as advanced information only.

A PyTorch version of FC-MNIST is added as a part of PyTorch (BETA) support.

This version only supports compiling on a CPU node with the train mode. To train this model on the Cerebras System at your own risk, edit the run.py file and comment out the entire raise ValueError() function, as shown below:

elif runconfig_params["mode"] == TRAIN:
        # raise ValueError(
        #    "Training PyTorch models on the Cerebras System is in beta "
        #    "and is only validated with the default config provided in the "
        #    "Model Zoo. Remove this exception and use the provided config to"
        #    "proceed."
        #)
        runner.train(train_loader)

The PyTorch versions of BERT Base and BERT Large are added as a part of PyTorch (BETA) support.

These versions only support compiling on a CPU node with the train mode. To train these models on the Cerebras System at your own risk, edit the run.py file and comment out the entire raise ValueError() function, as shown below:

elif runconfig_params["mode"] == TRAIN:
        #raise ValueError(
        #"Training PyTorch models on the Cerebras System is in beta "
        #"and is only validated with the default configs provided in the "
        #"Model Zoo. Remove this exception and use one of the provided "
        #"configs to proceed."
        #)
        runner.train(train_loader)

RevBERT#

A new TensorFlow model, the RevBERT is introduced. The RevBERT is a Cerebras-specific BERT model that improves the BERT performance on Cerebras accelerator. Using the RevBERT model you can run up to 20x larger batch sizes and 2.7x larger models on the Cerebras System. This version of RevBERT is only supported with TensorFlow and only supports the train mode.

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Transformer (Attention Is All You Need)#

Support is added in the train mode for Variable Sequence Length (VSL) on the CS system.

T5 model#

Support is enhanced from loss-only eval to full eval metrics.
Support is added in the train mode for Variable Sequence Length (VSL) on the CS system.

GPT-2#

Support is added in the train mode for Variable Sequence Length (VSL) on the CS system.

Release 0.9.0#

New features and enhancements#

Two new Slurm wrapper scripts are introduced to make it easy to run on CS system and on the CPU. These scripts will replace srun_train and salloc_node. See below:

The csrun_wse script can be used to execute training, evaluation and prediction on the CS system.

The csrun_cpu script can be used to launch a given user command on a CPU, within the Cerebras Singularity container.

Support is added for the Transformer (Attention Is All You Need), with the following capabilities:

Example dataset and preprocessing scripts for English-to-German translation included.
On CS system: Training, and Eval (loss only).
On GPU: Train, Eval (eval and eval_all).

Support is added for the following T5 family of models:

Small model:
- d_model = 512
- d_ff = 2,048.
- 8-headed attention.
- 6 layers each in the encoder and decoder.
- About 60 million parameters.
Model:
- Base, BERT Base-sized encoder and decoder.
- About ~ 220 million parameters.
Model: Large, BERT Large-sized encoder and decoder.
- d_model = 1,024.
- d_ff = 4,096.
- d_kv = 64.
- 16-headed attention.
- 24 layers each in the encoder and decoder.
- Around 770 million parameters.
Sample dataset: Colossal Clean Crawled Corpus (C4) dataset.
On CS system: Pre-training, Eval (loss only).
On GPU: Train, Eval (eval and eval_all).

The variable sequence length (VSL) performance of BERT-style encoder-decoder models is enhanced. Previously, a sequence of less than pre-defined maximum sequence length is padded up to the maximum sequence length. The compute and memory are also spent on processing these tokens used for padding, resulting in a significant loss of performance.

With this enhancement, by taking advantage of the sparsity the tokens used for padding are not processed, thereby enhancing the performance of the variable length sequences.

The performance-optimized variable sequence length is now available for the following models on the CS system:

BERT Pre-training (training only).
RNN Language Model (LM) (training only).
RNN Sentiment (training only).

Performance is enhanced for long sequences (MSL up to 8K for smaller models) for BERT- and GPT-style models. This is accomplished by making use of sparse attention to reduce memory requirements.

Known issues#

When you use AdamW Optimizer and if both the following conditions are true:
- The parameter weight_decay is set to a non-zero value, and
- The parameter loss_scaling_factor is not set to “dynamic”.
then the execution will stop with the following error message:

Error

“When using the AdamW optimizer with weight decay, set the loss_scaling_factor to dynamic.”
For the models T5 and Transformer (Attention Is All You Need), the performance in samples-per-sec is optimal when the source max_seq_len and the target max_seq_len are equal.
When running evaluation with a BERT model, if the max_predictions_per_seq parameter is set to an odd value and if the following conditions are true:
- The tensor is multi-dimensional (>1D).
- The inner dimension is an odd value.
- The datatype is < 4 bytes, i.e., FP16 or INT16 or UINT16.
then this leads to a compile failure in 0.9.0 and execution failure in 0.8.0.

Workaround: Set the max_predictions_per_seq parameter to an even value.

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

Release 0.8.0#

New features and enhancements#

Inference is now supported for the following models:
- Graph Convolutional Network
- Graph Attention Network

Note

If you are interested in these models, contact Cerebras support by sending email to support@cerebras.net.

A new feature, multi-model inference, is introduced. Using this you can run multiple neural network models on the CS system, send inference requests to these models and receive prediction responses.

Early stopping is now supported using a custom hook called CerebrasEarlyStoppingHook. Using this hook, you can terminate early a neural network training based on some logic.

Release 0.7.1#

New features and enhancements#

Evaluation and prediction are now supported on the CS system for BERT networks. While executing the run.py, you can run evaluation or prediction with your network as follows:
- Evaluation: Use --mode eval to use the evaluation feature.
- Prediction: Use --mode predict to use the prediction feature.

Release 0.7.0#

New features and enhancements#

Performance is improved for BERT Large models with MSL 512. This is accomplished by making a tradeoff that mitigates the need for large buffer memory.

Support is added for combined Dice loss and Softmax Cross-entropy (CE) loss.

Support for TensorFlow summaries is added.

A new datatype called CB16 is introduced. The CB16 is Cerebras’ 16-bit format, also referred to as cbfloat16. The CB16 is a floating-point format with 6-bit exponent and 9-bit explicit mantissa. This allows for double the dynamic range of FP16. See Control numerical precision level.

A new feature that projects the performance of your network is added to the Cerebras Graph Compiler (CGC). Now when your compile is successful, the generated report includes projections on how your network might perform on the CS system.:

A new feature called incremental compile is added to the Cerebras Graph Compiler (CGC). After you compile your model the first time, the incremental compile feature of CGC will automatically speed up the subsequent compile runs of your model by reusing, wherever possible, the optimizations already performed.

The input function analyzer is enhanced. Now called analyze_input_fn_compile, this tool provides a detailed log identifying any missing functions and provides recommendations on parameter values to enhance the training performance on the CS system.

Introduced a new method called Cerebras AUTOTUNE (CS_AUTOTUNE), which is similar to the TensorFlow tf.data.AUTOTUNE. When you are targeting the CS system, using CS_AUTOTUNE instead of tf.data.AUTOTUNE will result in a better specification of parameters such as:
- num_parallel_calls
- cycle_length
- num_parallel_reads

A new function, KerasModelToCerebrasEstimator, is provided to convert a Keras model so the model can be run using the CerebrasEstimator.

While setting the runtime configuration options, in v0.6.3 and earlier versions you were required to add the following code for the Slurm cluster resolver.

from cerebras.tf.cs_slurm_cluster_resolver import CSSlurmClusterResolver
slurm_cluster_resolver = CSSlurmClusterResolver()
cluster_spec = slurm_cluster_resolver.cluster_spec()
task_type, task_id = slurm_cluster_resolver.get_task_info()
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': cluster_spec.as_dict(),
    'task': {'type': task_type, 'index': task_id}
})

Now this is done automatically. This means that your Slurm-orchestrated TensorFlow code that contains the above statements should be edited as follows:

# Do not remove the following import statement.

from cerebras.tf.cs_slurm_cluster_resolver import CSSlurmClusterResolver

# Remove the following lines starting CGC v0.7.0.

 slurm_cluster_resolver = CSSlurmClusterResolver()
 cluster_spec = slurm_cluster_resolver.cluster_spec()
 task_type, task_id = slurm_cluster_resolver.get_task_info()
 os.environ['TF_CONFIG'] = json.dumps({
     'cluster': cluster_spec.as_dict(),
     'task': {'type': task_type, 'index': task_id}
 })

Breaking changes#

The use_cs parameter in the cerebras estimator interface is removed and will result in compiler error if used in this API. The target hardware will now be automatically determined from a combination of the runtime configuration parameter cs_ip and the use_cs parameter setting in the method definitions for train.
The format of the YAML config files for all the models is changed as follows:
- All the training-related parameters have been moved to the runconfig section.
- The max_steps parameter is added as a default parameter to control the duration of training.

Known issues#

For BERT, a change in the max_gradient_norm hyperparameter value will not result in reduced incremental compile times.

In v0.6.3, in some cases, when you enable dynamic loss scaling and an arbitrary operation is performed on the computed loss, then the Cerebras compiler may give error and fail to compile.

Workaround: In v0.7.0 you can workaround this error by disabling the dynamic loss scaling by setting loss_scaling_factor to a constant value either equal to or greater than 1.0.

Release 0.6.3#

New features and enhancements#

The overall performance is improved for BERT for max sequence length 128 (MSL128) variants. This improvement varies based on the fabric and model configuration. Enable the following custom Cerebras configuration flag only for BERT MSL128 variants to see this performance improvement:
```
config.matching.kernel.no_dcache_spill_splits = True
```
Tip

The Cerebras implementation sets this flag by default for BERT runs with MSL128.
The kernel matching phase of the Cerebras Graph Compiler (CGC) is enhanced to significantly reduce the kernel matching time and improve flexibility. With this enhancement, the kernel matching phase is completed within 60 seconds in a majority of cases. As a result, the overall compile time will be reduced in these cases.

Resolved issues#

Resolved a kernel matching issue with 1DConv models with embeddings, when a Conv1D layer is stacked on top of an embedding layer.

Vocabulary Size Troubleshooting

Documentation Updates

Software Release Notes#

Release 1.9.1#

New features and enhancements#

Large Language Models#

Computer vision models#

Other features#

Known Issues#

Release 1.9.0#

Release 1.8.0#

New features and enhancements#

Large language models#

Computer vision models#

Other features#

Known Issues#

Release 1.7.1#

New features and enhancements#

Unified user workflow on the Wafer-Scale Cluster#

Known issues#

Specifying number of workers#

Specifying path in the configuration file#

Padding token in loss calculation#

TensorFlow BERT SQuAD Unsupported#

Release 1.7.0#

New features and enhancements#

Large language models#

Computer vision models#

Other features#

Known issues#

Running eval with UNet in PyTorch, Weight Streaming execution#

Release 1.6.1#

New features and enhancements#

Weight Streaming models in PyTorch#

Security-related Updates and Error fixes#

Release 1.6.0#

New features and enhancements#

Weight streaming models#

Cerebras Wafer-Scale Cluster#

Multi-node CS-2 support#

PyTorch support#

Cerebras Model Zoo#

Release 1.5.0#

New features and enhancements#

Weight streaming#

Multi-node CS-2 support#

Release 1.4.0#

New features and enhancements#

Weight Streaming#

PyTorch models#

TensorFlow models#

Multi-node CS-2 support#

Known issues#

Running eval on Cerebras system#

Release 1.3.0#

New features and enhancements#

PyTorch models#

Weight streaming mode#

Multi-replica mode#

Known issues#

GPT-J (6B parameters) model#

Running eval on Cerebras system#

Release 1.2.0#

New features and enhancements#

PyTorch models#

Weight streaming execution mode#

Weight streaming models#

Input analyzer for Slurm resources#

Known issues#

Running eval on Cerebras system#

Release 1.1.0#

New features and enhancements#

PyTorch#

PyTorch models#

Supported PyTorch ops#

Multi-replica data parallel training#

Known issues#

T5 and Transformer (Attention is All You Need)#

Running eval on Cerebras system#

Multi-replica data parallel training#

PyTorch#

Release 1.0.0#