A blog by Robert M. Gower

Random Search

Notes on optimization, machine learning, and ideas worth trying. A little theory, a little experimentation.

A series of short posts

Stochastic optimization methods through statistical whitening

What can the statistics of a gradient tell us about how to take a step? Starting with variance, we explore a statistical viewpoint on modern optimizers.

Based on our paper with Tetiana Parshakova and Paulo Orenstein.

  1. Whiten out variance of the gradients