Papers
(Order of completion)
R. M. G., Dirk A. Lorenz, Maximilian Winkler
A Bregman-Kaczmarz method for nonlinear systems of equations, arXiv 2303.08549, 2023
A Bregman-Kaczmarz method for nonlinear systems of equations, arXiv 2303.08549, 2023
Guillaume Garrigos, Robert M. Gower
Handbook of Convergence Theorems for (Stochastic) Gradient Methods, arXiv:2301.11235 2023.
Handbook of Convergence Theorems for (Stochastic) Gradient Methods, arXiv:2301.11235 2023.
Rui Yuan, Simon S. Du, R. M. G.
Alessandro Lazaric, Lin Xiao
Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies, ICLR 2023.
Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies, ICLR 2023.
Shuang Li, William J. Swartworth, Martin Takáč, Deanna Needell, R. M. G.
SP2 : A Second Order Stochastic Polyak Method, ICLR 2023.
SP2 : A Second Order Stochastic Polyak Method, ICLR 2023.
R. M. G. , Mathieu Blondel, Nidham Gazagnadou, Fabian Pedregosa.
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes, 2022.
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes, 2022.
R. M. G. , Aaron Defazio, Michael Rabbat.
Stochastic Polyak Stepsize with a Moving Target, 2021.
Stochastic Polyak Stepsize with a Moving Target, 2021.
Rui Yuan, R. M. G., Alessandro Lazaric.
A general sample complexity analysis of vanilla policy gradient, AISTATS 2022.
A general sample complexity analysis of vanilla policy gradient, AISTATS 2022.
Jiabin Chen, Rui Yuan, Guillaume Garrigos, R. M. G.
SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums, AISTATS 2022.
Nidham Gazagnadou, Mark Ibrahim, R. M. G.
RidgeSketch: A Fast sketching based solver for large scale ridge regression, SIAM Journal of Matrix Analysis,
Vol. 43, Iss. 3, 2022.
R. M. G., Mark Schmidt, Francis Bach, Peter Richtarik
Variance-Reduced Methods for Machine Learning, Proceedings of the IEEE, vol. 108, no. 11, pp. 1968-1983, Nov. 2020.
Rui Yuan, Alessandro Lazaric, R. M. G.
Sketched Newton-Raphson, SIAM Journal of Optimization, Vol. 32, Iss. 3, 2022.
Ahmed Khaled, Othmane Sebbouh, Nicolas Loizou, R. M. G., Peter Richtárik.
Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization, 2020.
R. M. G., Othmane Sebbouh, Nicolas Loizou
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation, AISTATS 2021.
Aaron Defazio, R. M. G.
Factorial Powers for Stochastic Optimization, Asian Conference on Machine Learning, 2021.
Othmane Sebbouh, R. M. G., Aaron Defazio.
Almost sure convergence rates for Stochastic Gradient Descent and
Stochastic Heavy Ball, COLT 2021.
Dmitry Kovalev, R. M. G., Peter Richtárik, Alexander Rogozin.
Fast Linear Convergence of Randomized BFGS, 2020.
R. M. G., Denali Molitor, Jacob Moorman, Deanna Needell.
Adaptive Sketch-and-Project Methods for Solving Linear Systems, SIAM Journal on Matrix Analysis and Applications, 2019.
O. Sebbouh, N. Gazagnadou, S. Jelassi, F. Bach, R. M. G.
Towards closing the gap between the theory and practice of SVRG, Neurips 2019.
R. M. G., D. Kovalev, F. Lieder, P. Richtárik.
RSN: Randomized Subspace Newton, Neurips 2019.
N. Gazagnadou, R. M. G., J. Salmon.
Optimal mini-batch and step sizes for SAGA, ICML 2019.
R. M. G., N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, P. Richtárik.
SGD: general analysis and improved rates, (extended oral presentation) ICML 2019.
A. L. Gower, R. M. G., J. Deakin, W. J. Parnell, I. D. Abrahams.
Characterising particulate random media from near-surface backscattering: A machine learning approach to predict particle size and concentration .
EPL (Europhysics Letters), 2018.
A. Bibi, A. Sailanbayev, B. Ghanem, R. M. G. and P. Richtárik.
Improving SAGA via a probabilistic interpolation with gradient descent, 2018.
R. M. G., P. Richtárik and F. Bach.
Stochastic quasi-gradient methods: variance reduction via Jacobian sketching, Mathematical Programming 2020.
R. M. G., F. Hanzely, P. Richtárik and S. Stich.
Accelerated stochastic matrix inversion: general theory and speeding up BFGS rules for faster second-order optimization, NIPS, 2018.
B. K. Abid and R. M. G..
Greedy stochastic algorithms for entropy-regularized optimal transport problems, AISTATS, 2018.
R. M. G., N. Le Roux and F. Bach.
Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods, AISTATS (Oral presenation), 2018.
R. M. G. and P. Richtárik.
Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms, SIAM Journal on Matrix Analysis and Applications, 2017.
R. M. G. and P. Richtárik.
Linearly Convergent Randomized Iterative Methods for Computing the Pseudoinverse, 2016.
R. M. G.
Sketch and Project: Randomized Iterative Methods for Linear Systems and Inverting Matrices, PhD Dissertation, School of Mathematics, The University of Edinburgh, 2016.
R. M. G., D. Goldfarb and P. Richtárik.
Stochastic Block BFGS: Squeezing More Curvature out of Data, ICML, 2016.
R. M. G. and P. Richtárik.
Stochastic dual ascent for solving linear systems, 2015.
R. M. G. and P. Richtárik.
Randomized iterative methods for linear systems, SIAM Journal on Matrix Analysis and Applications, 2015.
R. M. G. and A. L. Gower .
High order reverse automatic differentiation with emphasis on the third order, Mathematical Programming, 2014.
R. M. G. and M. P. Mello.
Computing the sparsity pattern of Hessians using automatic differentiation, ACM Transactions on Mathematical Software, 2014.
Reports and Notes
R. M. G.
A Very Simple Introduction to Diffusion Models and The Standard Loss Function
R. M. G., R. Whittaker, M.D. Wykes, J. Christmas, P. Browne, J. Van lent, A.L. Gower and S. Ghosh.
Train Positioning Using Video Odometry, 2014.
R. M. G.and J. Gondzio.
Action constrained quasi-Newton methods, Technical Report ERGO 14-020, 2014
Action constrained quasi-Newton methods, Technical Report ERGO 14-020, 2014
R. M. G.
Conjugate Gradients: The short and painful explanation with oblique projections
R. M. G. and M.P. Mello
Hessian matrices via automatic differentiation, State University of Campinas technical report and Msc Thesis 2011
Recent & Upcoming Talks
ICCOPT 2019
Aug 5, 2019
Expected smoothness is the key to understanding the mini-batch complexity of stochastic gradient methods
Slides
Teaching
Flatiron: Machine Learning X Science Summer School (Spring 2022)
1) Lecture slides.2) Jupiter notebook for generating (most) of figures in slides SGD_figure.ipynb, and Gradient_momentum.ipynb
3) Exercises on convexity and smoothness, complexity and convergence rates, ridge regression and gradient descent, stochastc methods for ridge regression, SGD proof
Cornell lecture: Optimization for Machine Learning (Spring 2020)
In case you need it, here is a short probability revision.1) Details of the course.
2) Lecture slides for SGD.
3) Exercises on stochastc methods for ridge regression (solutions) and SGD convergence proof (solutions).
4) Lecture slides for Variance Reduction.
Master IASD: AI Systems and Data Science (Fall 2019)
For prerequisites see here . For revision of vector calculus see here .The course information can be found here
1) Slides on introduction to SGD and ERM
2) Lecture notes on probability revision
3) Exercise list on stochastc methods for ridge regression (solutions)
4) Slides on SGD and variants
5) Exercise on SGD proof (solutions)
6) Python notebook on SGD (solutions)
Telecom Paris IA317: Large scale machine learning (Fall 2019)
The course information can be found here.For prerequisites and revision material see here.
1) Lecture notes on dimension reduction tools and sparse matrices
2) Exercise list on dimension reduction tools and sparse matrices
3) Python Notebook graded homework on dimension reduction tools and sparse matrices. Data sets needed for homework: colon-cancer, anthracyclineTaxaneChemotherapy and sector.scale.
MDI210 Optimization et Analise númeric (Fall 2020)
The complete lectures notes with examples and exercises (by Irene Charon Olivier Hudry) are here. Here are my notes and slides (WARNING: these are a work in progress!)1) Lecture notes on numerical linear algebra
2) Lecture notes on linear and nonlinear optimization (Updated 12/10/2020)
3) Slides on Linear systems and Eigenvalues
4) Slides on Linear Programming
5) Slides on Nonlinear Linear Programming (Updated 12/10/2020)
Master2 Optimization for Data Science (Fall 2019)
For prerequisites see here . For revision of vector calculus see here .Lecture notes on gradient descent proofs.
0) Exercises on convexity and smoothness (solutions)
1) Exercises on complexity and convergence rates (solutions)
2) Lecture I: intro to ML, convexity, smoothness and gradient descent
3) Exercises ridge regression and gradient descent (solutions)
4) Lecture II: proximal gradient methods
5) Exercises on proximal operator (solutions)
6) Lab1: Proximal gradient methods
7) Lecture III: Stochastic gradient descent
8) Exercises on stochastc methods for ridge regression (solutions)
9) Exercise on SGD proof (solutions)
10) Lecture IV: Stochastic variance reduced gradient methods
11) Exercise on variance reduction, proof of convergence of SVRG
12) Lecture V: Sampling and momentum
13) Exercise on sampling and momentum
14) Python notebook on momentum
African Masters of Machine Intelligence (AMMI) (Winter 2019)
1) Lecture I: Introduction into ML and optimization2) Exercises on convexity, smoothness and gradient descent
3) Lecture II: proximal gradient methods
4) Exercises on proximal operator
5) Lecture III: Stochastic gradient descent
6) Exercises on stochastc methods
7) Lecture IV: Stochastic variance reduced gradient methods
8) Notes on stochastic variance reduced methods
Contact
- gowerrobert$@$gmail.com
- Flatiron Institute, 162 5th Ave, New York, NY 10010, United States. Office: 411
- email for appointment