This last three weeks I was in Utah for a summer math school or conference whichever you prefer that was organized by the Institute of Advanced Studies (IAS), Princeton, New Jersey. Basically we had three weeks of lectures, problem sessions, endless fun discussions day in day out with free food, free stay – the resort was amazing, we stayed at the Zermatt resort (if you’re using headphones you might wanna turn the volume down a bit). Anyway, the point of this post is lectures given by Roman Vershynin that I sat through on Random Matrices and applications (basically this is a scribe). Specifically, about the famous problem called as matrix completion. Later (hopefully soon) I’ll write about another topic that definitely got my attention called as Differential Privacy.
The problem is very simple to state: let’s say we have a matrix of of size , this simply means we have a 2-D data array with rows and columns. The rows and columns are indexed over, say, users and movies respectively. The famous Netflix prize problem had a matrix like this and of course, not every user has seen every movie. So the questions is, can we fill in the entries of the matrix otherwise incomplete matrix? Indeed, they had a test dataset (or simply test set) on which your algorithms will be tested on, whoever gets the most accuracy wins the challenge, so they get lots of $$$. This problem is obviously not well posed (why?) but as with most machine learning problems, mathematical ill-posedness doesn’t really matter as far as we can do good on the test set (this is not 100% true). In fact, the name of the game is to find mathematical theory/algorithms/models that can explain the problem/solution in practice. There are actually very many cool examples of this, probably we will see about this in later posts. Anyway, to analyze matix completion problem, we will need some tools from, of course, random matrix theory (I was actually surprised that these tools were actually developed very recently, I mean within the last 10-15 years atmost, if I’m wrong, please comment). So in this post we will look at some basic concentration inequalities.
Definition of subgaussian random variable: Let be a random variable. Then, the following are equivalent (TFAE):
3. MGF of :
4. MGF of :
We will refer to these numbers before the statements through out this post. denotes the probability and denotes the expectation.
In these statements, s can be multiplicative factors of each other but this really doesn’t matter since we are interested in the asymptotics. Examples include: gaussian, bernoulli and nonexamples include exponential, poisson etc.. Finally we will denote smallest value of in 3 (MGF of ).
Proposition 1: If ‘s are independent subgaussian random variables with mean 0, then is also a subgaussian random variable.
Proof: We will show 4 for .
The first equality is true because of the independence of the random variables , the inequality is due to 4. Choosing to be we are done.
Hoeffding’s Inequality: Let ‘s be independent subgaussian random variables with mean 0. Then
Proof: This follows from proposition 1 and the equivalence of the definitions.
OK, now we need to know a bit about subexponential random variables.
Definition of subexponential random variable: Let be a random variable. Then, the following are equivalent (TFAE):
3′. If , then
In this case we denote = smallest .
Bernstein’s Inequality: Let ‘s be independent subexponential random variables with mean 0. Then
Proof (sketch): This requires the use of Markov inequality on the sum and then we use 3′.
Corallary: where .
Synopsis: We have Hoeffding’s inequality for subgaussians and Bernstein’s inequality for subexponentials, so what? well, if you look at the stuff inside the exp, then you’ll notice that the subgaussians are much behaved than subexponentials since they go down (concentrate) faster or better (quadratic in subgaussians vs linear in subexponentials). In the next post we will see how this can be extended to the case where are matrices.