What is Hoeffding’s Inequality ?

Given a sequence of Random Variables that act as I.I.Ds (Identical Independently Distributed) $Z_{1}, Z_{2}, Z_{3} \dots Z_{n}$ with mean $\mu$ and $\mathbf{P}(a \leq Z_{i} \leq b) = 1$ then for any $\epsilon > 0$ we have the following:

where $\bar{Z_{n}}=\frac{1}{n} \sum_{i=1}^{n}Z_{i}$.

Why should anyone care about this at all?? This is well explained in Abu Mostafa’s book: Learning from data: A Short Course. This is an excellent book for anyone to get started on theoretical machine learning and is a must read. This book was based on the course that he had taught at CalTech. Although this book doesn’t go into the actual proof of Hoeffding’s in-equality this blog aims of explaining Hoeffding’s inequality in a rigorous manner while also providing great intuition. Most of this blog is originally taken from Stanford CS229: Supplemental Lecture notes: Hoeffding’s inequality, John Duchi. and CMU 36-705 Intermediate Statistics by Larry Wasserman. I highly recommend that the readers go through their courses as they are stellar and are freely available on the internet.

One can interpret this as follows: the probability that the mean of the independent samples one collects from a distribution deviates from the mean of the original distribution with at tolerance $\epsilon$ is exponentially decreasing with the number of samples one collects $n$. This is very important thing for folk in machine learning and data science. This is one of the founding basis for the belief that: More data gives better results. We will see in the next few blogs how this plays into the idea of generalisation in Machine Learning Theory.

Proof of Hoeffding’s inequality:

In order to prove Hoeffding’s inequality we prove some other smaller theorems who consequence in what we require:

Check Point 1: Markov’s inequality

Let $Z \geq 0$ be a non-negative random variable. Then $\forall\ t>0$ this \(\mathbf{P}(Z \geq t) \leq \frac{\mathbf{E}[Z]}{t}\) Proof: This is simple and probably familiar to most of you, but let’s go over this anyway: \(\begin{aligned} \mathbf{P} (Z \geq t) & = \int_{t}^{\infty} p(x)dx \\ & \leq \int_{t}^{\infty} p(x)dx \leq \int_{t}^{\infty} \frac{x}{t} p(x)dx \\ \\ & \leq \frac{1}{t} \int_{t}^{\infty} x p(x)dx \\ \\ & \leq \frac{1}{t}\int_{0}^{\infty} xp(x)dx = \frac{\mathbf{E}[Z]}{t}\\ \end{aligned}\) Hence we have Markov’s: $\mathbf{P}(Z \geq t) \leq \frac{\mathbf{E}[Z]}{t}$

Check Point 2: Chebyshev’s Inequality

Let $Z$ be a random variable with $\mathbf{Var}(Z)$ being finite. The for $t > 0$ we have: \(\mathbf{P}(|Z - \mathbf{E}[Z]| \geq \epsilon) < \frac{\mathbf{Var}(Z)}{\epsilon^{2}}\) The proof for this is a consequence of the previous Markov Inequality: \(\begin{aligned} \mathbf{P}(|Z - \mathbf{E}(Z)| \geq \epsilon) & = \mathbf{P} ((Z - \mathbf{E}[Z])^{2} \geq \epsilon^{2}) \\ & \leq \frac{\mathbf{E} [(Z-E[Z])^{2}]}{\epsilon^{2}} = \frac{\mathbf{Var(Z)}}{\epsilon^{2}} \end{aligned}\) Hence we have the following: $\mathbf{P}(|Z - \mathbf{E}[Z]| \geq \epsilon) < \frac{\mathbf{Var}(Z)}{\epsilon^{2}}$. You need to note this strategy which works based on derivations of Markov’s Inequality. Works wary well and proves a lot of great things.

Some further observations from this is: When ever we have $Z_{1}, Z_{2}, Z_{3}\dots Z_{n}$ are I.I.Ds with $\mathbf{E}[Z_{i}]=0$ we have the following: \(\mathbf{Var} (\bar{Z}) = \frac{1}{n}\sum_{i,j\leq n}\mathbf{E}[Z_{i}Z_{j}]\) Proof of this is: \(\begin{aligned} \sum_{i,j \leq n} \mathbf{E}[Z_{i} Z_{j}] &= \sum_{i=1}^{n} \mathbf{E}[Z_{i}^{2}] + \sum_{i \neq j} \mathbf{E}[Z_{i} Z_{j}] \\ & = \sum_{i=1}^{n} \mathbf{E}[Z_{i}^{2}] + \sum_{i \neq j} \mathbf{E}[Z_{i}].\mathbf{E}[Z_{j}] = \sum_{i=1}^{n} \mathbf{E}[Z_{i}^{2}] + \sum_{i \neq j} (\mathbf{E}[Z_{i}])^{2} \\ & =n\mathbf{E}[Z_{i}^{2}] + n(n-1)\mathbf{E}[Z]^{2} \\ & = n.\mathbf{E}(Z_{i}^{2}) \end{aligned}\) This gives us: \(\mathbf{E} (Z_{i}^{2}) = \mathbf{Var}(Z_{i}) = \frac{1}{n} \sum_{i,j} \mathbf{E}[Z_{i} Z_{j}]\) Now using Chebyshev’s inequality on this result we get: \(\mathbf{P}(|\bar{Z}| \geq \epsilon) \leq \frac{\mathbf{Var}(Z_{i})}{\epsilon^{2}} = \frac{1}{n\epsilon^{2}} \sum_{i, j} \mathbf{E}[Z_{i}Z_{j}]\) We will come back to these results back again.

Check Point 3: Moment Generating Functions and Chernoff Bounds

Given a random variable $Z$, the corresponding M.G.F is: \(M_{Z}(\lambda) = \mathbf{E}[\exp(\lambda Z)]\) The Chernoff bound is a statistical bound stated as follows, given a random variable we have: \(\mathbf{P}(|Z - \mathbf{E}[Z]| \geq \epsilon) \leq \min_{\lambda\geq {0}} \mathbf{E}[\exp(\ \lambda\ |Z - \mathbf{E}[Z]|)] e^{-\lambda\epsilon} = \min_ M_{|Z - \mathbf{E}[Z]|}(\lambda) e^{-\lambda t}\) Proof for the same is, using Markov’s inequality (same trick again … see it’s pretty useful): \(\begin{aligned} \mathbf{P} (|Z - \mathbf{E}[Z]| \geq \epsilon) & = \mathbf{P} (\exp(\lambda|Z-\mathbf{E}[Z]|) \geq e^{\lambda \epsilon}) \\ & \leq \mathbf{E}[\exp(\lambda|Z- \mathbf{E}[Z]|)].e^{-\lambda\epsilon} \end{aligned}\) Since this is true for all $\lambda > 0$, we can replace this with a minima over varying positive $\lambda’s$. \(\mathbf{P}(|Z - \mathbf{E}[Z]| \geq \epsilon) \leq \min_{\lambda} = \min_ \mathbf{E}[\exp(\lambda|Z- \mathbf{E}[Z]|)].e^{-\lambda\epsilon} = \min_{\lambda > 0} M_{|Z - \mathbf{E}[Z]|}(\lambda) e^{-\lambda t}\) This is the chernoff bound. The reason we care about this is that it plays well with summations over Independent distributions, which is a consequence of moment generating functions. We get the following results: \(M_{Z_{1} + Z_{2} +\dots +Z_{n}}(\lambda) = \prod_{i=1}^{n} M_{Z_{i}}(\lambda)\) It’s useful to see that from this we will have: \(\begin{aligned} \mathbf{P}(|\bar{Z_{i}} - \mu| > \epsilon) &= P\left( \left|\sum_{i=1}^{n} Z_{i} - n\mu\right| > n\epsilon \right) \\ &=P\left(\exp\left(t.\left|\sum_{i=1}^{n} Z_{i} - n\mu\right|\right) > \exp(nt\epsilon)\right) \\ \implies \mathbf{P}(|\bar{Z_{i}} - \mu| > \epsilon) &\leq \exp(-tn\epsilon) \prod_{i=1}^{n}M_{|Z_{i}-\mu|}(t) \\ &= \exp(-tn\epsilon) (M_{|Z_{i}-\mu|}(t))^n \end{aligned}\)

Small Helper Function:

There is one final inequality we use to prove Hoeffding’s inequality: Suppose that we have a random variable $Z$, which is bounded, i.e., $Z \in [a,b]$, with mean $\mu$: \(\mathbf{E}(e^{tX}) \leq e^{t\mu} \exp\left( \frac{t^2 (b-a)^2 }{8} \right)\) The link to this proof is in a different blog entry:

Final Step: Hoeffding’s Inequality!!

The true form of Hoeffding’s Inequality is not as we mentioned!! The original statement actually only speaks about the Independent Variables but not I.I.Ds, this is because I.I.Ds are just are just one version of Independent Variables. The true statement is:

Here’s the final proof of Hoeffding’s Inequality, alas: Suppose we have independent random variables $Z_{1}, Z_{2}\dots Zn$ such that each one of them is bounded, i.e., $Z_{i} \in [a_{i}, b_{i}]$: \(\begin{aligned} \mathbf{P}(\bar{Z}_{n} > \epsilon) &\leq e^{-nt \epsilon} \mathbf{E}\left( \exp\left( t \sum_{i=1}^{n} Z_{i} \right) \right)\\ &\leq e^{-nt\epsilon} \prod_{i=1}^{n} \mathbf{E}(\exp(tZ_{i})) \\ &\leq e^{-nt\epsilon} \prod_{i=1}^{n} \exp\left( nt\mu + \frac{t^2(b-a)^2}{8} \right) \end{aligned}\) This is essentially Hoeffding’s inequality. The more familiar version of this is when we have $Z_i$s are all identical, aka, they are iids. \(\mathbf{P}(\bar{Y}_{n} > \epsilon) \leq \inf_{t\geq 0} \exp\left( -n\epsilon t + nt\mu+\frac{nt^2(b-a)^2}{8} \right)\) The minima for this occurs when $t = \frac{4\epsilon}{(b-a)^2}$. Hence we have: (when taken with absolute value $\mu$ becomes 0) \(\mathbf{P}(|\bar{Y}_{n}-\mu| > \epsilon) \leq 2 \exp\left( \frac{-2n\epsilon^2}{(b-a)^2} \right)\)