Mathematical Foundations of Machine Learning

Last time

• Last class: Wednesday August 28, 2024
  • We talked about the notion of completeness
  • We talked about the orthogonal projection theorem
  • We proved it in infinite dimension
  • We explicitly used the notion of completeness in our proof
• To be effectively prepared for today's class, you should have:
  1. Gone over last week's slides and read associated lecture notes
  2. Turned in Homework 1
  3. Started looking at Homework 2

Computing approximations

• The orthogonality principle gives us a procedure for computing the closest point

Orthobases

• Notes
  • If the last condition is not met, this is just called an orthogonal basis
  • Orthobases are useful because we can write $x=\sum _{i=1}^n{⟨x,v_i⟩}_{}{v}_i$ (what happens in a non-orthonormal basis?)
• We would like to extend this idea to infinite dimension and happily write $x=\sum _{i=1}^{\mathrm{\infty }}{⟨x,v_i⟩}_{}{v}_i$
  • We have to be a bit careful
  • With a little bit of machinery, this works: separability + completeness

What we know works in infinite dimension

• Consider a Hilbert space $\mathcal{H}$. The following notions we encountered work without problems:
  • Norm induced by inner product
  • Cauchy-Schwartz inequality
  • Orthogonality + Pythagorean theorem + projection theorem

• Notes
  • We do not even require $\mathcal{I}$ to be countable at this stage.
  • We are not saying anything about this being a basis

Towards orthobases in infinite dimension

• This does not say anything about the existence of such a nice orthornormal set

Separable space

• Separability is the key property to deal with sequences instead of collections

• Many useful Hilbert spaces are separable! We won't worry about non-separable Hilbert spaces

• Key take away for separable Hilbert spaces
  • $x=\sum _{i=1}^{\mathrm{\infty }}{⟨x,v_i⟩}_{}{v}_i$ is perfectly well defined for an orthonormal basis
  • Parseval's identity tell us that ${\Vert x\Vert }_{}^2=\sum _{k\ge 1}{\left|{⟨x,v_k⟩}_{}\right|}^2$.
  • We don't need to even worry about the nature of $\mathcal{H}$ and only think about coefficients ${⟨x,v_i⟩}_{}$

Next time

• Next class: Monday September 9, 2024
  • We will talk about orthobases
  • We will prove a fundamental result called Parseval's theorem
• To be effectively prepared for next class, you should:
  1. Go over today's slides and read associated lecture notes
  2. Work on Homework 2 (due date: Friday September 13, 2024)
• Optional
  • Export slides for next lecture as PDF (be on the lookout for an announcement when they're ready)