Shortcuts

MADGRAD

class madgrad.MADGRAD(params: Any, lr: float = 0.01, momentum: float = 0.9, weight_decay: float = 0, eps: float = 1e-06)

MADGRAD: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization.

MADGRAD is a general purpose optimizer that can be used in place of SGD or Adam may converge faster and generalize better. Currently GPU-only. Typically, the same learning rate schedule that is used for SGD or Adam may be used. The overall learning rate is not comparable to either method and should be determined by a hyper-parameter sweep.

MADGRAD requires less weight decay than other methods, often as little as zero. Momentum values used for SGD or Adam’s beta1 should work here also.

On sparse problems both weight_decay and momentum should be set to 0.

Arguments:
params (iterable):

Iterable of parameters to optimize or dicts defining parameter groups.

lr (float):

Learning rate (default: 1e-2).

momentum (float):

Momentum value in the range [0,1) (default: 0.9).

weight_decay (float):

Weight decay, i.e. a L2 penalty (default: 0).

eps (float):

Term added to the denominator outside of the root operation to improve numerical stability. (default: 1e-6).

step(closure: Optional[Callable[], float]] = None) → Optional[float]

Performs a single optimization step.

Arguments:
closure (callable, optional): A closure that reevaluates the model

and returns the loss.

property supports_flat_params
property supports_memory_efficient_fp16
Read the Docs v: latest
Versions
latest
Downloads
pdf
html
epub
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources