Fabian Schaipp
PhD student in Optimization for Machine Learning
- Munich
- Github
- Google Scholar
- ORCID
You May Also Enjoy
How to jointly tune learning rate and weight decay for AdamW
14 minute read
Published:
TL;DR: AdamW is often considered a method that decouples weight decay and learning rate. In this blog post, we show that this is not true for the specific way AdamW is implemented in Pytorch. We also show how to adapt the tuning strategy in order to fix this: when doubling the learning rate, the weight decay should be halved.
Optimization Nuggets: Stochastic Polyak Step-size, Part 2
less than 1 minute read
Published:
Fabian Pedregosa invited me to write a joint blog post on a convergence proof for the stochastic Polyak step size (SPS).
Solve it all and solve it fast: using numba for optimization in Python
6 minute read
Published:
When implementing optimization algorithms, we typically have to balance the following goals:
A collection of resources for creating open-source software packages
7 minute read
Published:
Making your research code open-source, tested and documented is quite simple nowadays. This post gives an overview of the most important steps and collects useful ressources, e.g. tutorials for Readthedocs, Sphinx (Gallery) and unit testing in Python.