Category Archives: Memos

Thresholding over the l1-ball

Since long ago I wanted to write a post with an elementary introduction to soft / hard thresholding. The time has come to pay the dues, so here we go.

Posted in Memos, Sparsity, Statistics | Leave a comment

Formulations of Mirror Descent

A very nice reference for this post is [this blog post]. Mirror Descent is a first-order algorithm to solve a convex optimization problem    In this post we will learn several equivalent formulations of this algorithm and discuss how they are related to each other.

Posted in Course on Convex Optimization, Memos | Leave a comment

Brunn-Minkowski, Prékopa-Leindler, and isoperimetry

Found a great overview of the subject with a lot of insight. Another brief intro is here.

Posted in Memos | Tagged | Leave a comment

Fourier transforms

Here is a recap of proper normalizations for the Fourier series/DTFT (i.e. torus/ domains), DFT (), and also a brief reminder of Plancherel’s theorem.

Posted in Memos | Tagged | Leave a comment

Projection on a convex set is closer to any point of the set

A useful little fact for constrained optimization. Let    be a convex set, and consider      and its projection on   As quickly follows from the separation theorem with hyperplane containing  for any    the angle between    and    is obtuse. … Continue reading

Posted in Convex Optimization, Course on Convex Optimization, Memos | Tagged | 3 Comments

Sandwiching smooth convex functions

If a function has a Lipschitz gradient, i.e. for any and  then

Posted in Convex Optimization, Course on Convex Optimization, Memos | 2 Comments

Farkas’ lemma

Farkas’ lemma  is a simple (and very useful) statement in convex analysis. Theorem (Farkas’ lemma) For any matrix    and vector  ,  exactly one of the following holds: (1)     linear system    has a solution    in the first … Continue reading

Posted in Convex Optimization, Memos, Optimization tricks | Leave a comment