Optimization for Machine Learning Course Project
We consider ridge regression problem with randomly generated data. The goal is to implement gradient descent and experiment with different strong-convexity settings and different learning rates. Test with different weights of regularizer.
Implement functions:
- smoothness parameter
- closed form solution of the ridge regression problem
- compute the objective function value
- compute the gradient of the objective function
- gradient descent algorithm for t iterations
We consider optimization with the smoothed hinge loss, and randomly generated data. The goal is to implement gradient descent and experiment with different strong-convexity settings and different learning rates.
Implement functions:
- implement the smoothed hinge loss (SVM) objective function
- compute the gradient of the smoothed hinge loss object function
- heavy ball method with adaptive beta (practically simplified)
- Nesterov's acceleration method
- Nesterov's acceleration with adaptive beta (practically simplified)
- Nesterov's general acceleration method (applicable for smooth and non-strongly convex case)
Download data in the mnist directory (which contains class 1 (positive) versus 7 (negative) from the MNIST data). We consider the composite convex optimization with the L1-L2 regularized logistic regression objective function.
Implement functions:
- compute prox gradient of the objective f(w) + g(w)
- compute the proximal mapping
- gradient of the dual regularizer g^* (alpha)
- regularized dual averaging algorithm
- primal dual ascent method
- proximal gradient descent algorithm
- proximal descent with AG learning rate
- Nesterov's accelerated proximal gradient algorithm
- Nesterov's accelerated proximal gradient algorithm with AG learning rate
Download data in the mnist directory (which contains class 1 (positive) versus 7 (negative) from the MNIST data). We consider the smoothed hinge-loss (SVM) objective function with L1-L2 regularizer.
Implement functions:
- compute dual objective of L1-L2 regularizer -g^* (alpha)
- compute the dual coordinate ascent at a data point (x, y) of the smoothed hinge-loss (SVM) objective function
- compute dual objective of the smoothed hinge-loss (SVM) objective function
- implement primal proximal coordinate descent
- implement stochastic primal-dual coordinate method
- implement accelerated linearized alternating direction method of multipliers