Learning

Logistics

• General announcements

  • Assignment 6 due December 7, 2021 for bonus, deadline December 10, 2021

  • Last lecture!

  • Let me know what’s missing

  • Expect an email from me tonight

• Midterm 2 statistics

  • Overall: AVG: 72% - MIN: 29% - MAX: 98%

What we have learned this Fall

• Hilbert spaces

  • Spaces of functions can be manipulated almost just as easily

  • Finite dimensional is fairly natural

  • Infinite dimensional can be manipulated just as well using orthobases

  • With orthobases, vectors in infinite dimensional separates Hilbert spaces are like square summable sequences

• Regression

  • Who knew solving $\mathbf{y}=\mathbf{A}\mathbf{x}$ could be so useful?

  • SVD provides lots of insights

• Regression in Hilbert spaces

  • Perhaps biggest lesson of the course
  • Representer theorem allows us to do regression in infinite dimensional Hilbert spaces
  • RKHS provide the kind of Hilbert spaces that naturally embed our data

What’s on the agenda for today?

• More on learning and Bayes classifiers

• Lecture notes 17 and 23

A simpler supervised learning problem

Consider a special case of the general supervised learning problem

1. Dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

   • $\{{\mathbf{x}}_i{\}}_{i=1}^N$ drawn i.i.d. from unknown $P_{\mathbf{x}}$ on $\mathcal{X}$
   • $\{y_i{\}}_{i=1}^N$ labels with $\mathcal{Y}=\{0,1\}$ (binary classification)

2. Unknown $f:\mathcal{X}\to \mathcal{Y}$, no noise.

3. Finite set of hypotheses $\mathcal{H}$, $|\mathcal{H}|=M<\mathrm{\infty }$

   • $\mathcal{H}\triangleq \{h_i{\}}_{i=1}^M$

4. Binary loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}^{+}:(y_1,y_2)\mapsto \mathbf{1}\{y_1\ne y_2\}$

• In this very specific case, the true risk simplifies $$R(h)\triangleq {\mathbb{E}}_{\mathbf{x}y}\left[\mathbf{1}\{h(\mathbf{x})\ne y\}\right]={\mathbb{P}}_{\mathbf{x}y}(h(\mathbf{x})\ne y)$$

• The empirical risk becomes $${\hat{R}}_N(h)=\frac{1}{N}\sum _{i=1}^N\mathbf{1}\{h({\mathbf{x}}_i)\ne y_i\}$$

Can we learn?

• Our objective is to find a hypothesis $h^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{\hat{R}}_N(h)$ that ensures a small risk

• For a fixed $h_j\in \mathcal{H}$, how does ${\hat{R}}_N(h_j)$ compares to $R(h_j)$?

• Observe that for $h_j\in \mathcal{H}$

  • The empirical risk is a sum of iid random variables $${\hat{R}}_N(h_j)=\frac{1}{N}\sum _{i=1}^N\mathbf{1}\{h_j({\mathbf{x}}_i)\ne y_i\}$$

  • ${\mathbb{E}}_{}[{\hat{R}}_N(h_j)]=R(h_j)$

• ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|>ϵ)$ is a statement about the deviation of a normalized sum of iid random variables from its mean

• We’re in luck! Such bounds, a.k.a, known as concentration inequalities, are a well studied subject

Concentration inequalities: basics

• Let $X$ be a non-negative real-valued random variable. Then for all $t>0$ $${\mathbb{P}}_{}(X\ge t)\le \frac{{\mathbb{E}}_{}\left[X\right]}{t}.$$

• Let $X$ be a real-valued random variable. Then for all $t>0$ $${\mathbb{P}}_{}(\left|X-{\mathbb{E}}_{}\left[X\right]\right|\ge t)\le \frac{\text{Var}\left(X\right)}{t^2}.$$

• Let $\{X_i{\}}_{i=1}^N$ be i.i.d. real-valued random variables with finite mean $\mu$ and finite variance ${\sigma }^2$. Then $${\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i-\mu \right|\ge ϵ)\le \frac{{\sigma }^2}{N{ϵ}^2}\underset{N\to \mathrm{\infty }}{lim}{\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i-\mu \right|\ge ϵ)=0.$$

Back to learning

• By the law of large number, we know that $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{\{({\mathbf{x}}_i,y_i)\}}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le \frac{\text{Var}\left(\mathbf{1}\{h_j({\mathbf{x}}_1)\ne y_1\}\right)}{N{ϵ}^2}\le \frac{1}{N{ϵ}^2}$$

• Given enough data, we can generalize

• How much data? $N=\frac{1}{\delta {ϵ}^2}$ to ensure ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le \delta$.

• That’s not quite enough! We care about ${\hat{R}}_N(h^{\ast })$ where $h^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{\hat{R}}_N(h)$

  • If $M=|\mathcal{H}|$ is large we should expect the existence of $h_k\in \mathcal{H}$ such that ${\hat{R}}_N(h_k)\ll R(h_k)$

• $${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le {\mathbb{P}}_{}(\mathrm{\exists }j:\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)$$

• $${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le \frac{M}{N{ϵ}^2}$$

• If we choose $N\ge \lceil \frac{M}{\delta {ϵ}^2}\rceil$ we can ensure ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le \delta$.

  • That’s a lot of samples!

Concentration inequalities: not so basic

• We can obtain much better bounds than with Chebyshev

• Let $\{X_i{\}}_{i=1}^N$ be i.i.d. real-valued zero-mean random variables such that $X_i\in [a_i;b_i]$ with $a_i<b_i$. Then for all $ϵ>0$ $${\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i\right|\ge ϵ)\le 2\exp (-\frac{2N^2{ϵ}^2}{\sum _{i=1}^N(b_i-a_i{)}^2}).$$

• In our learning problem $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le 2\exp (-2N{ϵ}^2)$$

• $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le 2M\exp (-2N{ϵ}^2)$$

• We can now choose $N\ge \lceil \frac{1}{2{ϵ}^2}(\ln \frac{2M}{\delta })\rceil$

• $M$ can be quite large (almost exponential in $N$) and, with enough data, we can generalize $h^{\ast }$.

• How about learning $h^{\mathrm{♯}}\triangleq {\text{argmin}}_{h\in \mathcal{H}}R(h)$?

Learning can work!

• If $\mathrm{\forall }j\in \mathcal{H}\left|{\hat{R}}_N(h_j)-R(h_j)\right|\le ϵ$ then $\left|R(h^{\ast })-R(h^{\mathrm{♯}})\right|\le 2ϵ$.

• How do we make $R(h^{\mathrm{♯}})$ small?

  • Need bigger hypothesis class $\mathcal{H}$! (could we take $M\to \mathrm{\infty }$?)
  • Fundamental trade-off of learning

Probably Approximately Correct Learnability

• A hypothesis set $\mathcal{H}$ is (agnostic) PAC learnable if there exists a function $N_{\mathcal{H}}:]0;1{[}^2\to \mathbb{N}$ and a learning algorithm such that:

  • for very $ϵ,\delta \in ]0;1[$,
  • for every $P_{\mathbf{x}}$, $P_{y|\mathbf{x}}$,
  • when running the algorithm on at least $N_{\mathcal{H}}(ϵ,\delta )$ i.i.d. examples, the algorithm returns a hypothesis $h\in \mathcal{H}$ such that $${\mathbb{P}}_{\mathbf{x}y}(\left|R(h)-R(h^{\mathrm{♯}})\right|\le ϵ)\ge 1-\delta$$

• The function $N_{\mathcal{H}}(ϵ,\delta )$ is called sample complexity

• We have effectively already proved the following result

• A finite hypothesis set $\mathcal{H}$ is PAC learnable with the Empirical Risk Minimization algorithm and with sample complexity $$N_{\mathcal{H}}(ϵ,\delta )=\lceil \frac{2\ln (2|\mathcal{H}|/\delta )}{{ϵ}^2}\rceil$$

What is a good hypothesis set?

• Ideally we want $|\mathcal{H}|$ small so that $R(h^{\ast })\approx R(h^{\mathrm{♯}})$ and get lucky so that $R(h^{\ast })\approx 0$

• In general this is not possible

• Remember, we usually have to learn $P_{y|\mathbf{x}}$, not a function $f$

• Questions

  • What is the optimal binary classification hypothesis class?
  • How small can $R(h^{\ast })$ be?

Supervised learning model

We revisit the supervised learning setup (slight change in notation)

1. Dataset $\mathcal{D}\triangleq \{(X_1,Y_1),\cdots ,(X_N,Y_N)\}$

   • $\{X_i{\}}_{i=1}^N$ drawn i.i.d. from unknown $P_X$ on $\mathcal{X}={\mathbb{R}}^d$
   • $\{Y_i{\}}_{i=1}^N$ labels with $\mathcal{Y}=\{0,1,\cdots ,K-1\}$ (multiclass classification)

2. Unknown $P_{Y|X}$

3. Binary loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}^{+}:(y_1,y_2)\mapsto \mathbf{1}\{y_1\ne y_2\}$

• The risk of a classifier $h$ is $$R(h)\triangleq {\mathbb{E}}_{XY}\left[\mathbf{1}\{h(X)\ne Y\}\right]={\mathbb{P}}_{XY}(h(X)\ne Y)$$

• We will not directly worry about $\mathcal{H}$, but rather about $R({\hat{h}}_N)$ for some ${\hat{h}}_N$ that we will estimate from the data

Bayes classifier

• What is the best risk (smallest) that we can achieve?
  • Assume that we actually know $P_X$ and $P_{Y|X}$
  • Denote the a posteriori class probabilities of $\mathbf{x}\in \mathcal{X}$ by $${\eta }_k(\mathbf{x})\triangleq {\mathbb{P}}_{}(Y=k|X=\mathbf{x})$$
  • Denote the a priori class probabilities by $${\pi }_k\triangleq {\mathbb{P}}_{}(Y=k)$$

• The classifier $h^{\text{B}}(\mathbf{x})\triangleq {\text{argmax}}_{k\in [0;K-1]}{\eta }_k(\mathbf{x})$ is optimal, i.e., for any classifier $h$, we have $R(h^{\text{B}})\le R(h)$. $$R(h^{\text{B}})={\mathbb{E}}_X[1-\underset{k}{max}{\eta }_k(X)]$$

• Terminology
  • $h^B$ is called the Bayes classifier
  • $R_B\triangleq R(h^B)$ is called the Bayes risk

Other forms of the Bayes classifier

• $h^{\text{B}}(\mathbf{x})\triangleq {\text{argmax}}_{k\in [0;K-1]}{\eta }_k(\mathbf{x})$

• $h^{\text{B}}(\mathbf{x})\triangleq {\text{argmax}}_{k\in [0;K-1]}{\pi }_k{p}_{X|Y}(\mathbf{x}|k)$

• For $K=2$ (binary classification): log-likelihood ratio test $$\log \frac{p_{X|Y}(\mathbf{x}|1)}{p_{X|Y}(\mathbf{x}|0)}\gtrless \log \frac{{\pi }_0}{{\pi }_1}$$

• If all classes are equally likely ${\pi }_0={\pi }_1=\cdots ={\pi }_{K-1}$ $$h^{\text{B}}(\mathbf{x})\triangleq \underset{k\in [0;K-1]}{\text{argmax}}{p}_{X|Y}(\mathbf{x}|k)$$

• Assume $X|Y=0\sim \mathcal{N}(0,1)$ and $X|Y=1\sim \mathcal{N}(1,1)$. The Bayes risk for ${\pi }_0={\pi }_1$ is $R(h^{\text{B}})=\mathrm{\Phi }(-\frac{1}{2})$ with $\mathrm{\Phi }\triangleq \text{Normal CDF}$

• In practice we do not know $P_X$ and $P_{Y|X}$

  • Plugin methods: use the data to learn the distributions and plug result in Bayes classifier

Nearest neighbor classifier

• Back to our training dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

• The nearest-neighbor (NN) classifier is $h^{\text{NN}}(\mathbf{x})\triangleq y_{\text{NN}(\mathbf{x})}$ where $\text{NN}(\mathbf{x})\triangleq {\text{argmin}}_i{\Vert {\mathbf{x}}_i-\mathbf{x}\Vert }_{}$

• Risk of NN classifier conditioned on $\mathbf{x}$ and ${\mathbf{x}}_{\text{NN}(\mathbf{x})}$ $$R_{\text{NN}}(\mathbf{x},{\mathbf{x}}_{\text{NN}(\mathbf{x})})=\sum _k{\eta }_k({\mathbf{x}}_{\text{NN}(\mathbf{x})})(1-{\eta }_k(\mathbf{x}))=\sum _k{\eta }_k(\mathbf{x})(1-{\eta }_k({\mathbf{x}}_{\text{NN}(\mathbf{x})})).$$

  • How well does the average risk $R_{\text{NN}}=R(h^{\text{NN}})$ compare to the Bayes risk for large $N$?

• Let $\mathbf{x}$, $\{{\mathbf{x}}_i{\}}_{i=1}^N$ be i.i.d. $\sim P_{\mathbf{x}}$ in a separable metric space $\mathcal{X}$. Let ${\mathbf{x}}_{\text{NN}(\mathbf{x})}$ be the nearest neighbor of $\mathbf{x}$. Then ${\mathbf{x}}_{\text{NN}(\mathbf{x})}\to \mathbf{x}$ with probability one as $N\to \mathrm{\infty }$

• Let $\mathcal{X}$ be a separable metric space. Let $p(\mathbf{x}|y=0)$, $p(\mathbf{x}|y=1)$ be such that, with probability one, $\mathbf{x}$ is either a continuity point of $p(\mathbf{x}|y=0)$ and $p(\mathbf{x}|y=1)$ or a point of non-zero probability measure. As $N\to \mathrm{\infty }$, $$R(h^{\text{B}})\le R(h^{\text{NN}})\le 2R(h^{\text{B}})(1-R(h^{\text{B}}))$$

K Nearest neighbors classifier

• Can drive the risk of the NN classifier to the Bayes risk by increasing the size of the neighborhood
  • Assign label to $\mathbf{x}$ by taking majority vote among $K$ nearest neighbors $h^{K\text{-NN}}$ $$\underset{N\to \mathrm{\infty }}{lim}{\mathbb{E}}_{}[R(h^{K\text{-NN}})]\le (1+\sqrt{\frac{2}{K}})R(h^{\text{B}})$$

• Let ${\hat{h}}_N$ be a classifier learned from $N$ data points; ${\hat{h}}_N$ is consistent if ${\mathbb{E}}_{}[R({\hat{h}}_N)]\to R_B$ as $N\to \mathrm{\infty }$.

• If $N\to \mathrm{\infty }$, $K\to \mathrm{\infty }$, $K/N\to 0$, then $h^{K\text{-NN}}$ is consistent

• Choosing $K$ is a problem of model selection
  • Do not choose $K$ by minimizing the empirical risk on training: $${\hat{R}}_N(h^{1\text{-NN}})=\frac{1}{N}\sum _{i=1}^N\mathbf{1}\{h_1({\mathbf{x}}_i)=y_i\}=0$$
  • Need to rely on estimates from model selection techniques (more later!)