Deterministic Least Squares

Today in ECE 6555

Don’t forget
1. Read the syllabus (for real!) and ask for clarifications as needed
2. Register on Piazza
3. Check Gradescope access
4. Check out the self assessment (ungraded, see canvas later today)
Announcements
1. Mathematics of ECE workshops (more on this soon)
2. Problem set 1 posted (reinforcing linear algebra, formal thinking, and extending some of what we’ve seen in class)
Today’s plan
- Geometry in (finite) dimensional spaces
- Normal equations
- Deterministic least squares
- Why? Geometric intuition is incredibly powerful and will carry over to more complex settings
Questions?

An inner product space over $R$ is a vector space $V$ equipped with a positive definite symmetric bilinear form ${⟨ \cdot, \cdot ⟩}_{} : V \times V \to R$ called an inner product
An inner product space is also called a pre-Hilbert space
An inner product satisfies $\forall x, y \in V$ ${⟨ x, y ⟩}_{}^{2} \leq {⟨ x, x ⟩}_{} {⟨ y, y ⟩}_{}$
A norm on a vector space $V$ over $R$ is a function ${‖ \cdot ‖}_{} : V \to R$ that satisfies:
- Positive definiteness: $\forall x \in V$ ${‖ x ‖}_{} \geq 0$ with equality iff $x = 0$
- Homogeneity: $\forall x \in V$ $\forall α \in R$ ${‖ α x ‖}_{} = | α | {‖ x ‖}_{}$
- Subadditivity: $\forall x, y \in V$ ${‖ x + y ‖}_{} \leq {‖ x ‖}_{} + {‖ y ‖}_{}$
$x \in R^{d} {‖ x ‖}_{0} ≜ | {i : x_{i} \neq 0} | {‖ x ‖}_{1} ≜ \sum_{i = 1}^{d} | x_{i} | {‖ x ‖}_{2} ≜ \sqrt{\sum_{i = 1}^{d} x_{i}^{2}}$

In an inner product space, an inner product induces a norm ${‖ x ‖}_{} ≜ \sqrt{{⟨ x, x ⟩}_{}}$
A norm ${‖ \cdot ‖}_{}$ is induced by an inner product on $V$ iff $\forall x, y \in V$ ${‖ x ‖}_{}^{2} + {‖ y ‖}_{}^{2} = \frac{1}{2} ({‖ x + y ‖}_{}^{2} + {‖ x - y ‖}_{}^{2})$
If this is the case, the inner product is given by the polarization identity ${⟨ x, y ⟩}_{} = \frac{1}{2} ({‖ x ‖}_{}^{2} + {‖ y ‖}_{}^{2} - {‖ x - y ‖}_{}^{2})$

- Two vectors $x, y \in V$ are orthogonal if ${⟨ x, y ⟩}_{} = 0$ . We write $x ⊥ y$ for simplicity.
- A vector $x \in V$ is orthogonal to a set $S \subset V$ if $\forall s \in S$ ${⟨ x, s ⟩}_{} = 0$ . We write $x ⊥ S$ for simplicity.
- Two vector space $V$ and $W$ are orthogonal if $\forall v \in V$ and $\forall w \in W$ we have $v ⊥ w$ .
If $x ⊥ y$ then ${‖ x + y ‖}_{}^{2} = {‖ x ‖}_{}^{2} + {‖ y ‖}_{}^{2}$
The orthogonal complement of vector space $W \subset V \subset R^{n}$ is $W^{⊥} ≜ {x \in V : {⟨ x, y ⟩}_{} = 0 \forall y \in W}$
Homework problem: check that $W^{⊥}$ is a vector space, $(W^{⊥})^{⊥} = W$
$Ker (H) = Im (H^{T})^{⊥} Im (H) = Ker (H^{T})^{⊥}$

Consider vectors subspaces $U, V, W$ or $R^{n}$ . Then $W = U \oplus V$ iff for every $w \in W$ there exists a unique pair $(u, v) \in U \times V$ such that $w = u + v$
$Ker (H) \oplus Im (H^{⊺}) = R^{n} Ker (H^{⊺}) \oplus Im (H) = R^{m}$
Homework problem: prove that $W$ and $W^{⊥}$ are in direct sum
That’s about it for our review our linear algebra for now!