Troubleshooting

• Enable kernel generalizability with Autogen
  • Example use cases
• Failing to save checkpoints using experimental PyTorch API
  • Observed Error
  • Explanation
  • Work around
• Error Receiving Activation
  • cerebras_appliance.errors.ApplianceUnknownError: Ran into error while receiving activation tensor <custom-call …>
• Failed mount directory during execution
  • Observed Error
  • Work around
• Throughput spike after saving checkpoints
• Input Starvation
• Failing to automatically load checkpoints
  • Explanation
  • Work around
• Cannot load Cerebras checkpoints in GPUs
  • Work around
• ModuleNotFoundError
  • ModuleNotFoundError: No module named <’_bz2’, ‘_sqlite3’>
  • ModuleNotFoundError: No module named <…>
• Training fails when logged-in as root
  • Observed Error
  • Explanation
• Out of memory errors and system resources
  • Determining if your job is queued
  • Determining if job failed because of an OOM error
  • Determining if job failed because of system could not fit requested memory
  • Troubleshooting OOM errors
• Running jobs in parallel with the same model_dir causes issues
  • Observed error
  • Explanation
  • Workaround
• Error parsing metadata
  • Observed Error
  • Explanation
  • Work around
• Model is too large to fit on the device
  • Observed Error
• Vocabulary Size Troubleshooting
  • Large vocabulary size
  • Small vocabulary size

previous

Train with gradient accumulation

next

Enable kernel generalizability with Autogen