Learning

Logistics

• General announcements

  • Assignment 6 posted (last assignment)

  • Due December 7, 2021 for bonus, deadline December 10, 2021

  • 2 lectures left

  • Let me know what’s missing

• Assignment 5 grades posted

• Reviewing Midterm2 grades one last time

What’s on the agenda for today?

• The learning problem and why we need probabilities.

• Lecture notes 17 and 23

Components of supervised machine learning

1. An unknown function $f:\mathcal{X}\to \mathcal{Y}:\mathbf{x}\mapsto y=f(\mathbf{x})$ to learn
   • The formula to distinguish cats from dogs
2. A dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$
   • ${\mathbf{x}}_i\in \mathcal{X}\triangleq {\mathbb{R}}^d$: picture of cat/dog
   • $y_i\in \mathcal{Y}\triangleq \mathbb{R}$: the corresponding label cat/dog
3. A set of hypotheses $\mathcal{H}$ as to what the function could be
   • Example: deep neural nets with AlexNet architecture
4. An algorithm $\mathtt{\text{ALG}}$ to find the best $h\in \mathcal{H}$ that explains $f$

• Terminology:
  • $\mathcal{Y}=\mathbb{R}$: regression problem
  • $|\mathcal{Y}|<\mathrm{\infty }$: classification problem
  • $|\mathcal{Y}|=2$: binary classification problem
• The goal is to generalize, i.e., be able to classify inputs we have not seen.

A learning puzzle

• Learning seems impossible without additional assumptions!

Possible vs probable

• Flip a biased coin, lands on head with unknown probability $p\in [0,1]$

• ${\mathbb{P}}_{}\left(\text{head}\right)=p$ and ${\mathbb{P}}_{}\left(\text{tail}\right)=1-p$

• Say we flip the coin $N$ times, can we estimate $p$?

• $$\hat{p}=\frac{\text{\# head}}{N}$$

• Can we relate $\hat{p}$ to $p$?

  • The law of large numbers tells us that $\hat{p}$ converges in probability to $p$ as $N$ gets large $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{}(\left|\hat{p}-p\right|>ϵ)\underset{N\to \mathrm{\infty }}{⟶}0.$$

• It is possible that $\hat{p}$ is completely off but it is not probable

Components of supervised machine learning

1. An unknown function $f:\mathcal{X}\to \mathcal{Y}:\mathbf{x}\mapsto y=f(\mathbf{x})$ to learn

2. A dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

   • $\{{\mathbf{x}}_i{\}}_{i=1}^N$ i.i.d. from unknown distribution $P_{\mathbf{x}}$ on $\mathcal{X}$
   • $\{y_i{\}}_{i=1}^N$ are the corresponding labels $y_i\in \mathcal{Y}\triangleq \mathbb{R}$

3. A set of hypotheses $\mathcal{H}$ as to what the function could be

4. An algorithm $\mathtt{\text{ALG}}$ to find the best $h\in \mathcal{H}$ that explains $f$

Another learning puzzle

Components of supervised machine learning

1. An unknown conditional distribution $P_{y|\mathbf{x}}$ to learn

   • $P_{y|\mathbf{x}}$ models $f:\mathcal{X}\to \mathcal{Y}$ with noise

2. A dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

   • $\{{\mathbf{x}}_i{\}}_{i=1}^N$ i.i.d. from distribution $P_{\mathbf{x}}$ on $\mathcal{X}$
   • $\{y_i{\}}_{i=1}^N$ are the corresponding labels $y_i\sim P_{y|\mathbf{x}={\mathbf{x}}_i}$

3. A set of hypotheses $\mathcal{H}$ as to what the function could be

4. An algorithm $\mathtt{\text{ALG}}$ to find the best $h\in \mathcal{H}$ that explains $f$

• The roles of $P_{y|\mathbf{x}}$ and $P_{\mathbf{x}}$ are different
  • $P_{y|\mathbf{x}}$ is what we want to learn, captures the underlying function and the noise added to it
  • $P_{\mathbf{x}}$ models sampling of dataset, need not be learned

Yet another learning puzzle

• Assume that you are designing a fingerprint authentication system
  • You trained your system with a fancy machine learning system
  • The probability of wrongly authenticating is 1%
  • The probability of correctly authenticating is 60%
  • Is this a good system?
• It depends!
  • If you are GTRI, this might be ok (security matters more)
  • If you are Apple, this is not acceptable (convenience matters more)
• There is an application dependent cost that can affect the design

Components of supervised machine learning

1. A dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

   • $\{{\mathbf{x}}_i{\}}_{i=1}^N$ i.i.d. from an unknown distribution $P_{\mathbf{x}}$ on $\mathcal{X}$

2. An unknown conditional distribution $P_{y|\mathbf{x}}$

   • $P_{y|\mathbf{x}}$ models $f:\mathcal{X}\to \mathcal{Y}$ with noise
   • $\{y_i{\}}_{i=1}^N$ are the corresponding labels $y_i\sim P_{y|\mathbf{x}={\mathbf{x}}_i}$

3. A set of hypotheses $\mathcal{H}$ as to what the function could be

4. A loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}^{+}$ capturing the “cost” of prediction

5. An algorithm $\mathtt{\text{ALG}}$ to find the best $h\in \mathcal{H}$ that explains $f$

The supervised learning problem

• Learning is not memorizing

  • Our goal is not to find $h\in \mathcal{H}$ that accurately assigns values to elements of $\mathcal{D}$
  • Our goal is to find the best $h\in \mathcal{H}$ that accurately predicts values of unseen samples

• Consider hypothesis $h\in \mathcal{H}$. We can easily compute the empirical risk (a.k.a. in-sample error) $${\hat{R}}_N(h)\triangleq \frac{1}{N}\sum _{i=1}^N\ell (y_i,h({\mathbf{x}}_i))$$

• What we really care about is the true risk (a.k.a. out-sample error) $R(h)\triangleq {\mathbb{E}}_{\mathbf{x}y}[\ell (y,h(\mathbf{x}))]$

• Question #1: Can we generalize?

  • For a given $h$, is ${\hat{R}}_N(h)$ close to $R(h)$?

• Question #2: Can we learn well?

  • The best hypothesis is $h^{\mathrm{♯}}\triangleq {\text{argmin}}_{h\in \mathcal{H}}R(h)$ but we can only find $h^{\ast }\triangleq {\text{argmin}}_{h\in \mathcal{H}}{\hat{R}}_N(h)$
  • Is ${\hat{R}}_N(h^{\ast })$ close to $R(h^{\mathrm{♯}})$?
  • Is $R(h^{\mathrm{♯}})\approx 0$?

A simpler supervised learning problem

Consider a special case of the general supervised learning problem

1. Dataset $\mathcal{D}\triangleq \{({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)\}$

   • $\{{\mathbf{x}}_i{\}}_{i=1}^N$ drawn i.i.d. from unknown $P_{\mathbf{x}}$ on $\mathcal{X}$
   • $\{y_i{\}}_{i=1}^N$ labels with $\mathcal{Y}=\{0,1\}$ (binary classification)

2. Unknown $f:\mathcal{X}\to \mathcal{Y}$, no noise.

3. Finite set of hypotheses $\mathcal{H}$, $|\mathcal{H}|=M<\mathrm{\infty }$

   • $\mathcal{H}\triangleq \{h_i{\}}_{i=1}^M$

4. Binary loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}^{+}:(y_1,y_2)\mapsto \mathbf{1}\{y_1\ne y_2\}$

• In this very specific case, the true risk simplifies $$R(h)\triangleq {\mathbb{E}}_{\mathbf{x}y}\left[\mathbf{1}\{h(\mathbf{x})\ne y\}\right]={\mathbb{P}}_{\mathbf{x}y}(h(\mathbf{x})\ne y)$$

• The empirical risk becomes $${\hat{R}}_N(h)=\frac{1}{N}\sum _{i=1}^N\mathbf{1}\{h({\mathbf{x}}_i)\ne y_i\}$$

Can we learn?

• Our objective is to find a hypothesis $h^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{\hat{R}}_N(h)$ that ensures a small risk

• For a fixed $h_j\in \mathcal{H}$, how does ${\hat{R}}_N(h_j)$ compares to $R(h_j)$?

• Observe that for $h_j\in \mathcal{H}$

  • The empirical risk is a sum of iid random variables $${\hat{R}}_N(h_j)=\frac{1}{N}\sum _{i=1}^N\mathbf{1}\{h_j({\mathbf{x}}_i)\ne y_i\}$$

  • ${\mathbb{E}}_{}[{\hat{R}}_N(h_j)]=R(h_j)$

• ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|>ϵ)$ is a statement about the deviation of a normalized sum of iid random variables from its mean

• We’re in luck! Such bounds, a.k.a, known as concentration inequalities, are a well studied subject

Concentration inequalities: basics

• Let $X$ be a non-negative real-valued random variable. Then for all $t>0$ $${\mathbb{P}}_{}(X\ge t)\le \frac{{\mathbb{E}}_{}\left[X\right]}{t}.$$

• Let $X$ be a real-valued random variable. Then for all $t>0$ $${\mathbb{P}}_{}(\left|X-{\mathbb{E}}_{}\left[X\right]\right|\ge t)\le \frac{\text{Var}\left(X\right)}{t^2}.$$

• Let $\{X_i{\}}_{i=1}^N$ be i.i.d. real-valued random variables with finite mean $\mu$ and finite variance ${\sigma }^2$. Then $${\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i-\mu \right|\ge ϵ)\le \frac{{\sigma }^2}{N{ϵ}^2}\underset{N\to \mathrm{\infty }}{lim}{\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i-\mu \right|\ge ϵ)=0.$$

Back to learning

• By the law of large number, we know that $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{\{({\mathbf{x}}_i,y_i)\}}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le \frac{\text{Var}\left(\mathbf{1}\{h_j({\mathbf{x}}_1)\ne y_1\}\right)}{N{ϵ}^2}\le \frac{1}{N{ϵ}^2}$$

• Given enough data, we can generalize

• How much data? $N=\frac{1}{\delta {ϵ}^2}$ to ensure ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le \delta$.

• That’s not quite enough! We care about ${\hat{R}}_N(h^{\ast })$ where $h^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{\hat{R}}_N(h)$

  • If $M=|\mathcal{H}|$ is large we should expect the existence of $h_k\in \mathcal{H}$ such that ${\hat{R}}_N(h_k)\ll R(h_k)$

• $${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le {\mathbb{P}}_{}(\mathrm{\exists }j:\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)$$

• $${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le \frac{M}{N{ϵ}^2}$$

• If we choose $N\ge \lceil \frac{M}{\delta {ϵ}^2}\rceil$ we can ensure ${\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le \delta$.

  • That’s a lot of samples!

Concentration inequalities: not so basic

• We can obtain much better bounds than with Chebyshev

• Let $\{X_i{\}}_{i=1}^N$ be i.i.d. real-valued zero-mean random variables such that $X_i\in [a_i;b_i]$ with $a_i<b_i$. Then for all $ϵ>0$ $${\mathbb{P}}_{}(\left|\frac{1}{N}\sum _{i=1}^N{X}_i\right|\ge ϵ)\le 2\exp (-\frac{2N^2{ϵ}^2}{\sum _{i=1}^N(b_i-a_i{)}^2}).$$

• In our learning problem $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{}(\left|{\hat{R}}_N(h_j)-R(h_j)\right|\ge ϵ)\le 2\exp (-2N{ϵ}^2)$$

• $$\mathrm{\forall }ϵ>0{\mathbb{P}}_{}(\left|{\hat{R}}_N(h^{\ast })-R(h^{\ast })\right|\ge ϵ)\le 2M\exp (-2N{ϵ}^2)$$

• We can now choose $N\ge \lceil \frac{1}{2{ϵ}^2}(\ln \frac{2M}{\delta })\rceil$

• $M$ can be quite large (almost exponential in $N$) and, with enough data, we can generalize $h^{\ast }$.

• How about learning $h^{\mathrm{♯}}\triangleq {\text{argmin}}_{h\in \mathcal{H}}R(h)$?

Learning can work!

• If $\mathrm{\forall }j\in \mathcal{H}\left|{\hat{R}}_N(h_j)-R(h_j)\right|\le ϵ$ then $\left|R(h^{\ast })-R(h^{\mathrm{♯}})\right|\le 2ϵ$.

• How do we make $R(h^{\mathrm{♯}})$ small?

  • Need bigger hypothesis class $\mathcal{H}$! (could we take $M\to \mathrm{\infty }$?)
  • Fundamental trade-off of learning

Probably Approximately Correct Learnability

• A hypothesis set $\mathcal{H}$ is (agnostic) PAC learnable if there exists a function $N_{\mathcal{H}}:]0;1{[}^2\to \mathbb{N}$ and a learning algorithm such that:

  • for very $ϵ,\delta \in ]0;1[$,
  • for every $P_{\mathbf{x}}$, $P_{y|\mathbf{x}}$,
  • when running the algorithm on at least $N_{\mathcal{H}}(ϵ,\delta )$ i.i.d. examples, the algorithm returns a hypothesis $h\in \mathcal{H}$ such that $${\mathbb{P}}_{\mathbf{x}y}(\left|R(h)-R(h^{\mathrm{♯}})\right|\le ϵ)\ge 1-\delta$$

• The function $N_{\mathcal{H}}(ϵ,\delta )$ is called sample complexity

• We have effectively already proved the following result

• A finite hypothesis set $\mathcal{H}$ is PAC learnable with the Empirical Risk Minimization algorithm and with sample complexity $$N_{\mathcal{H}}(ϵ,\delta )=\lceil \frac{2\ln (2|\mathcal{H}|/\delta )}{{ϵ}^2}\rceil$$

What is a good hypothesis set?

• Ideally we want $|\mathcal{H}|$ small so that $R(h^{\ast })\approx R(h^{\mathrm{♯}})$ and get lucky so that $R(h^{\ast })\approx 0$

• In general this is not possible

• Remember, we usually have to learn $P_{y|\mathbf{x}}$, not a function $f$

• Questions

  • What is the optimal binary classification hypothesis class?
  • How small can $R(h^{\ast })$ be?