The Mathematics of ECE

Logistics

The Mathematics of ECE
- Review sessions not tied to any class coverting probabilities, linear algebra, and real analysis
- Objective: small curriculum but opportunity to ask anything!
- No grade, no pressure, no stakes
Instructor
- Brighton Ancelin, Ph.D. student in Machine Learning
- Piazza for course: sign-up
- Slides available here
Today’s class: Linear algebra fundamentals
- Objective #1: review concepts that you might be already familiar with
- Objective #2: get used to the level of rigor expected in grad school
- Objective #3: meet classmates!
Next workshop: Friday September 02, 2022
Please leave your email! We’d like to hear from you to do a better job

Let ${v_{i}}_{i = 1}^{n}$ be a set of vectors in a vector space $V$ .
For ${a_{i}}_{i = 1}^{n} \in R^{n}$ , $\sum_{i = 1}^{n} a_{i} v_{i}$ is called a linear combination of the vectors ${v_{i}}_{i = 1}^{n}$ .
The span of the vectors ${v_{i}}_{i = 1}^{n}$ is the set $span ({v_{i}}_{i = 1}^{n}) ≜ {\sum_{i = 1}^{n} a_{i} v_{i} : {a_{i}}_{i = 1}^{n} \in R^{n}}$
Drill: show that the span of the vectors ${v_{i}}_{i = 1}^{n} \in V^{n}$ is a vector subspace of $V$ .
Let ${v_{i}}_{i = 1}^{n}$ be a set of vectors in a vector space $V$
${v_{i}}_{i = 1}^{n}$ is linearly independent (or the vectors ${v_{i}}_{i = 1}^{n}$ are linearly independent ) if (and only if) $\sum_{i = 1}^{n} a_{i} v_{i} = 0 \Rightarrow \forall i \in [1; n] a_{i} = 0$ Otherwise the set is (or the vectors are) linearly dependent.

An inner product space over $R$ is a vector space $V$ equipped with a positive definite symmetric bilinear form ${⟨ \cdot, \cdot ⟩}_{} : V \times V \to R$ called an inner product
An inner product space is also called a pre-Hilbert space
An inner product satisfies $\forall x, y \in V$ ${⟨ x, y ⟩}_{}^{2} \leq {⟨ x, x ⟩}_{} {⟨ y, y ⟩}_{}$
A norm on a vector space $V$ over $R$ is a function ${‖ \cdot ‖}_{} : V \to R$ that satisfies:
- Positive definiteness: $\forall x \in V$ ${‖ x ‖}_{} \geq 0$ with equality iff $x = 0$
- Homogeneity: $\forall x \in V$ $\forall α \in R$ ${‖ α x ‖}_{} = | α | {‖ x ‖}_{}$
- Subadditivity: $\forall x, y \in V$ ${‖ x + y ‖}_{} \leq {‖ x ‖}_{} + {‖ y ‖}_{}$
In an inner product space, an inner product induces a norm ${‖ x ‖}_{} ≜ \sqrt{{⟨ x, x ⟩}_{}}$

Two vectors $x, y \in V$ are orthogonal if ${⟨ x, y ⟩}_{} = 0$ . We write $x ⊥ y$ for simplicity.

A vector $x \in V$ is orthogonal to a set $S \subset V$ if $\forall s \in S$ ${⟨ x, s ⟩}_{} = 0$ . We write $x ⊥ S$ for simplicity.
Every linearly independent subset can be orthonormalized.

A basis of vector subspace $W$ of a vector space $V$ is a countable set of vectors $B$ such that:
1. $span (B) = W$
2. $B$ is linearly independent
You should be somewhat familiar with this in $R^{n}$ , there are lots of nice features
- every subspace has a basis
- every basis for a subspace has the same number of elements
- the number of elements in a basis is called the dimension
- the representation of a vector on a basis is unique
- having a basis reduces the operations on vectors to operations on their components.

A subset $W$ of a vector space $V$ is a linear subspace if $\forall x, y \in W \forall λ, μ \in R$ $λ x + μ y \in W$
Every linear subspace of $R^{n}$ has a basis
if $x \in span ({v_{i}}_{i = 1}^{n})$ and $\forall i x^{⊺} v_{i} = 0$ , then $x = 0$

A matrix $A \in R^{m \times n}$ is a collection of vectors concatenated together $A ≜ [\begin{matrix} - r_{1} - \\ - r_{2} - \\ ⋮ \\ - r_{m} - \end{matrix}] ≜ [\begin{array}{cccc} | & | & | \\ c_{1} & c_{2} & \dots & c_{n} \\ | & | & | \end{array}]$
Let $A \in R^{m \times n}$ be a matrix with columns ${c_{j}}_{j = 1}^{n}$ and rows ${r_{i}}_{i = 1}^{m}$ . The column (image) space $col (A)$ of $A$ is $span ({c_{i}}_{i = 1}^{n})$ . The row space $row (A)$ of $A$ is $span ({r_{i}}_{i = 1}^{m})$ . The null (kernel) space of $A$ is $null ≜ (A) {x \in R^{n} : A x = 0}$
Note that $col (A) = row (A^{⊺})$
These spaces play an important role, in particular to solve systems of equations $A x = y$ (You’ll be surprised how often this shows up in ECE!)
$null (A)$ is a subvector space.

In the following $A$ is an $m \times n$ real-valued matrix
$dim (row (A)) = dim (col (A))$ , which is called the rank of $A$ denoted $rank (A)$
$row (A)$ and $null (A)$ are orthogonal complements, i.e.,
- for any $u \in row (A)$ , $v \in null (A)$ $u^{⊺} v = 0$
- $dim (row (A)) = n - dim (null (A))$
- for any $x \in R^{n}$ , there exists $x_{2} \in row (A)$ , $x_{2} \in null (A)$ such that $x = x_{1} + x_{2}$