Skip to content

Adagrad — Adaptive Gradient Descent Optimizer

An adaptive optimizer that accumulates squared gradients for each parameter, scaling the learning rate inversely with the magnitude of past gradients.


Overview

Adagrad (Adaptive Gradient Descent) is an adaptive optimization algorithm that maintains a running sum of squared gradients for each parameter, automatically scaling the learning rate inversely with the magnitude of past gradients. This helps with sparse data and handles parameters that receive infrequent updates. For MAGNET models, Adagrad provides specialized parameter scaling for s, z0, *_s, and *_z0 parameters.


Key Features

  • Adaptive Learning Rates: Per‑parameter learning rates based on accumulated squared gradients
  • Sqrt Scaling: Learning rate scaling by the square root of accumulated gradients
  • Magnet Integration: Specialized scaling for MAGNET parameters (s, z0, *_s, *_z0)
  • Epsilon Control: Numerical stability parameter for denominator calculations
  • Learning Rate Decay: Optional learning rate decay over time

Parameters

Parameter Type Default Description
lr float 0.01 Base learning rate for parameter updates
eps float 1e-8 Small constant for numerical stability
weight_decay float 0.0 L2 regularization coefficient (weight decay)
lr_decay float 0.0 Learning rate decay factor (per step)

Methods

__init__(lr=0.01, eps=1e-8, weight_decay=0.0, lr_decay=0.0)

Initialize the Adagrad optimizer with specified hyperparameters.

step(param, grad, **kwargs)

Update parameter param using gradient grad. For MAGNET parameters (s, z0, *_s, *_z0), applies specialized scaling based on their parameter type. The update uses the accumulated squared gradient history.

state_dict()

Return optimizer state (accumulated squared gradients, step count) for serialization.

load_state_dict(state)

Restore optimizer state from a saved dictionary.

zero_grad()

Zero out gradients for all parameters being optimized.

get_lr()

Return current learning rate for the optimizer.

set_lr(lr)

Update the learning rate for the optimizer.


Example Usage

from machinegnostics.magnet import Sequential, Dense, ReLU, MSE, Adagrad
import numpy as np

model = Sequential(layers=[
    Dense(in_features=4, out_features=8), ReLU(),
    Dense(in_features=8, out_features=1)
])

model.compile(loss=MSE(), optimizer=Adagrad(lr=0.001))
model.fit(X_train, y_train, epochs=10, batch_size=32)
loss, metric = model.evaluate(X_val, y_val)
print(f"Validation Loss: {loss:.4f}")

Author: Nirmal Parmar