.. _pytorch-cs-workflow:

Workflow for PyTorch on CS
==========================

When you are targeting the Cerebras system for your neural network jobs, start with the high-level workflow described here.

.. raw:: html

    <div class="container-fluid">

    <div class="row">
        <div class="col">
            <div class="card h-100" >
                <div class="card-body">
                    <h3 class="card-title">1</h3>
                    <a href="#port-pytorch-to-cs" class="card-link"><h4>Port PyTorch to CS</h4></a>
                </div>
            </div>
        </div>
        <div class="col">
            <div class="card h-100" >
                <div class="card-body">
                    <h3 class="card-title">2</h3>
                    <a href="#load-the-data" class="card-link"><h4>Load the data</h4></a>
                </div>
            </div>
        </div>
        <div class="col">
            <div class="card h-100" >
                <div class="card-body">
                    <h3 class="card-title">3</h3>
                    <a href="#compile-on-cpu" class="card-link"><h4>Compile on CPU</h4></a>
                </div>
            </div>
        </div>

        <div class="col">
            <div class="card h-100" >
                <div class="card-body">
                    <h3 class="card-title">4</h3>
                    <a href="#train-on-the-cs-system" class="card-link"><h4>Train on CS</h4></a>
                </div>
            </div>
        </div>

        <!-- break -->

    </div>
    </div>
    <br \>


.. attention::

	Several PyTorch models, such as the following, and run scripts, are provided in `Cerebras Model Zoo <https://github.com/Cerebras/modelzoo/tree/master/modelzoo>`_:

        - `PyTorch version of FC-MNIST <https://github.com/Cerebras/modelzoo/tree/master/modelzoo/fc_mnist/pytorch>`_, and

        - `PyTorch versions of BERT Base and BERT Large <https://github.com/Cerebras/modelzoo/tree/master/modelzoo/transformers/pytorch/bert>`_.

.. note::

	Cerebras has moved away from huggingface model implementations in favor for our own PyTorch layer API. One of the many benefits of using our PyTorch layer API is that it is designed to be (near) drop-in compatible with the transformer layers that are included in PyTorch. It is not possible (at least for T5 and Transformer) to maintain the same naming scheme in the migrated model as in the original.


1. Port PyTorch to CS
---------------------

See :ref:`porting-pytorch-to-cs` for detailed documentation on how to port your
PyTorch model to work on a Cerebras System.


2. Load the data
----------------

As mentioned in :ref:`porting-pytorch-to-cs`, you need to provide
:code:`get_train_dataloader` for training and :code:`get_eval_dataloader` for
evaluation. See :ref:`cbtorch-dataloader` for details.

3. Compile on CPU
-----------------

Before you run it on the Cerebras system, we recommend that you iterate until
your model first compiles successfully on a CPU node. Make sure that this CPU
node has the Cerebras Singularity container client software.

As described in :ref:`porting-pytorch-to-cs`, you can verify that your model
will compile by passing in the :code:`--compile_only` flag to the :code:`run`
function's CLI.

4. Train on the Cerebras system
-------------------------------

As described in :ref:`porting-pytorch-to-cs`, you can train or evaluate your
model by providing the IP address of the Cerebras system and the mode you want
to run.