Dr. Matthieu R Bloch
Tuesday, August 23, 2022
Many engineering problems reduce to solving a sytem of linear equations \[ \mathbf{y} = \matH \mathbf{x} \]
The system may have unique of multiple solutions: what determines it?
Rewrite \(\matH\) in terms of its columns \(\set{\vech_i}_{i=0}^{n-1}\) \[ \matH\eqdef \left[\begin{array}{ccc}|&&|\\\vech_0&\cdots &\vech_{n-1}\\|&&|\end{array}\right] \]
When does the system \(\mathbf{y} = \matH \mathbf{x}\) have a solution? (when is the system \(\bfy\) /consistent/?)
The system has a solution if and only if \(\mathbf{y}\) is a linear combination of the columns \(\set{\vech_i}_{i=0}^{n-1}\)
These sets are (sub) linear vector spaces… we’ll review very quickly
A subset \(\calW\) of a vector space \(\calV\) is a vector subspace if \(\forall x,y\in\calW\forall \lambda,\mu\in\bbR\) \(\lambda x+\mu y \in\calW\)
If \(\calW\) is a vector subspace of a vector space \(\calV\), \(\calW\) is a vector space.
\(\text{Im}(\matH)\) and \(\text{Ker}(\matH)\) are sub vector spaces
Let \(\set{v_i}_{i=1}^n\) be a set of vectors in a vector space \(\calV\).
For \(\set{a_i}_{i=1}^n\in\bbR^n\), \(\sum_{i=1}^na_iv_i\) is called a linear combination of the vectors \(\set{v_i}_{i=1}^n\).
The span of the vectors \(\set{v_i}_{i=1}^n\) is the set \[ \text{span}(\set{v_i}_{i=1}^n)\eqdef \{\sum_{i=1}^na_iv_i:\set{a_i}_{i=1}^n\in\bbR^n\} \]
The span of the vectors \(\set{v_i}_{i=1}^n\in\calV^n\) is a vector subspace of \(\calV\).
Let \(\set{v_i}_{i=1}^n\) be a set of vectors in a vector space \(\calV\)
\(\set{v_i}_{i=1}^n\) is linearly independent (or the vectors \(\set{v_i}_{i=1}^n\) are linearly independent ) if (and only if) \[ \sum_{i=1}^na_iv_i = 0\Rightarrow \forall i\in\intseq{1}{n}\,a_i=0 \] Otherwise the set is (or the vectors are) linearly dependent.
The inverse of a matrix can only exist is \(\matH\) is square and full rank
An inner product space over \(\bbR\) is a vector space \(\calV\) equipped with a positive definite symmetric bilinear form \(\dotp{\cdot}{\cdot}:\calV\times\calV\to\bbR\) called an inner product
An inner product space is also called a pre-Hilbert space
An inner product satisfies \(\forall x,y\in\calV\) \(\dotp{x}{y}^2\leq\dotp{x}{x}\dotp{y}{y}\)
\[\bfx\in\bbR^d\qquad\norm[0]{\bfx}\eqdef\card{\set{i:x_i\neq 0}}\quad\norm[1]{\bfx}\eqdef\sum_{i=1}^d\abs{x_i}\quad \norm[2]{\bfx}\eqdef\sqrt{\sum_{i=1}^d x_i^2}\]
In an inner product space, an inner product induces a norm \(\norm{x} \eqdef \sqrt{\dotp{x}{x}}\)
A norm \(\norm{\cdot}\) is induced by an inner product on \(\calV\) iff \(\forall x,y\in\calV\) \(\norm{x}^2+\norm{y}^2 = \frac{1}{2}\left(\norm{x+y}^2+\norm{x-y}^2\right)\)
If this is the case, the inner product is given by the polarization identity \[\dotp{x}{y}=\frac{1}{2}\left(\norm{x}^2+\norm{y}^2-\norm{x-y}^2\right)\]
Two vectors \(x,y\in\calV\) are orthogonal if \(\dotp{x}{y}=0\). We write \(x\perp y\) for simplicity.
A vector \(x\in\calV\) is orthogonal to a set \(\calS\subset\calV\) if \(\forall s\in\calS\) \(\dotp{x}{s}=0\). We write \(x\perp \calS\) for simplicity.
If \(x\perp y\) then \(\norm{x+y}^2=\norm{x}^2+\norm{y}^2\)
The orthogonal complement of vector space \(\calW\subset\calV\subset\bbR^n\) is \[ \calW^\perp\eqdef \set{x\in\calV:\dotp{x}{y}=0 \forall y\in\calW} \]
\[ \text{Ker}(\matH) = \text{Im}(\matH^T)^\perp\quad\text{Im}(\matH) = \text{Ker}(\matH^T)^\perp \]
Consider vectors subspaces \(\calU,\calV,\calW\) or \(\bbR^n\). Then \(\calW=\calU\oplus\calV\) iff for every \(w\in\calW\) there exists a unique pair \((u,v)\in\calU\times\calV\) such that \(w=u+v\)
\[ \text{Ker}(\matH)\oplus\text{Im}(\matH^\intercal) = \bbR^n\qquad \text{Ker}(\matH^\intercal)\oplus\text{Im}(\matH) = \bbR^m \]