Mathematical Foundations of Machine Learning

Enrollment updates

• ISyE and BME sections were added to canvas

• Enrollment update
  • 141 students across 9 sections
  • Will likely be some movement this week
  • See here to request permits
  • All restrictions are lifted on Friday August 23, 2024
  • Email me if you need the materials

Last time

• Last class: Monday August 19, 2024
  • We covered class logistics
  • We discussed the problem of representing functions and introduced the notion of basis expansion
  • We talked briefly about how to write proofs
• To be effectively prepared for today's class, you should have:
  1. Read the syllabus in detail
  2. Gone over the previous slides and read associated lecture notes
  3. Made sure you have access to Canvas, Piazza, Gradescope (if you could enroll)
  4. Started Homework 0 (due date: Monday August 26, 2024)

Example: Lagrange Polynomials

• Question: How do we find a basis expansion given ${\{(x_i,y_i)\}}_{i=1}^n$?

• Lagrange polynomials $$p(x)=\sum _{j=1}^n{\alpha }_j{L}_j(x)L_j(x)\triangleq \prod _{1\le i\le n,i\ne j}\frac{x-x_i}{x_j-x_i}$$

Example: Polynomial splines

• B-splines: the splines can be viewed as a linear basis expansion on a basis called B-splines
  • What are the B-splines for $\ell =0$?
  • What are the B-splines for $\ell =1$?
  • What are the B-splines for $\ell =2$?
• Moving forward, we will need linear algebra to describe this precisely and systematically
  • How do we compute a linear basis expansion?
  • What is a good basis for machine learning?
  • How do we measure the quality of approximation?

Vector (linear) space

• You are probably used to working in ${\mathbb{R}}^n$ … we'll deal with weirder objects

Examples of vector spaces

• Which of these spaces are vector spaces?

Subspaces

• Which of these examples correspond to subspaces?

Linear combinations and span

• Let ${\{v_i\}}_{i=1}^n$ be a set of vectors in a vector space $\mathcal{V}$.

Linear independence

• Let ${\{v_i\}}_{i=1}^n$ be a set of vectors in a vector space $\mathcal{V}$

Bases

• You should be somewhat familiar with bases (at least in ${\mathbb{R}}^n$):
  • the representation of a vector on a basis is unique
  • every subspace has a basis
  • having a basis reduces the operations on vectors to operations on their components
• Things sort of work in infinite dimensions, but we have to be bit more careful

Next time

• Next class: Monday August 26, 2024 3:30pm
  • We will talk about norms and inner products
  • We will work towards infinite dimensional spaces of functions
  • We will talk about the notion of completeness
• To be effectively prepared for next class, you should:
  1. Go over today's slides and read associated lecture notes
  2. Make sure you have access to Canvas, Piazza, Gradescope
  3. Work on Homework 0 (due date: Monday August 26, 2024)
• Optional
  • Export slides for next lecture as PDF (be on the lookout for an announcement when they're ready)

Inner product and norm

• An inner product space is also called a pre-Hilbert space

Induced norm and orthogonality

Hilbert space

• Intuition: a Hilbert space has no "missing point"

• A finite dimensional inner product space is complete (many Hilbert spaces are infinite dimensional!)