InverseSquareRootDecayLR
InverseSquareRootDecayLR¶
class modelzoo.common.pytorch.optim.lr_scheduler.InverseSquareRootDecayLR
(optimizer: torch.optim.optimizer.Optimizer, initial_learning_rate: float, scale: float, warmup_steps: int, disable_lr_steps_reset: bool = False)
Decays the learning rate inverse-squareroot over time, as described in the following equation:
- Parameters:
optimizer – The optimizer to schedule
initial_learning_rate – The initial learning rate
scale – Multiplicative factor to scale the result
warmup_steps – use initial_learning_rate for the first warmup_steps