The Mathematics of ECE

Linear Algebra - \(\bbR^n\), bases, matrices, rank

Monday August 30, 2021

Logistics

Image
https://xkcd.com/1050/
  • The Mathematics of ECE
    • Last Friday was well attented! We might have to spread the sessions.
    • No food, no drink, we want to keep using the auditorium!
    • No review on Wednesday September 01, 2021 (Grad student picnic)
  • Websites
  • Today’s class: Linear algebra
    • Focus on \(\bbR^n\) (finite dimension)
    • Review of key principles: bases, matrices, image, kernel, rank

Linear algebra for \(\bbR^n\)

  • The focus will be on \(\bbR^n\) (\(n\in\bbN^*\)) as the canonical (and very useful) examplar of a finite dimensional space.

  • \(\bbR^n\) is the collection of column vectors consisting of \(n\) elements \(\{v_i\}_{i=1}^n\in\bbR\), i.e., \[ \bfv=\mat{c}{v_1\\v_2\\\vdots\\v_n} = \mat{cccc}{v_1&v_2&\cdots&v_n}^\intercal \]

  • \(\bbR^n\) is formally a vector space, but what matters is that you can scale (multiply by real number) and add vectors together, and that the scaling and additions work well together.

  • A subvector space \(\calV\subset\bbR^n\) is such that:
    • For all \(\bfv,\bfw\in\calV\), \(\bfv+\bfw\in\calV\)
    • For all \(\lambda\in\bbR, \bfv\in\calV\), \(\lambda\bfv\in\calV\)

  • In other words, a subvector space \(\calV\) is a vector space that is a stable (for addition and scale multiplication) subset of \(\bbR^n\)

Span and linear independence

  • The span of set of vectors \(\set{\bfv_i}_{i=1}^m\) is \[ \text{span}\left(\set{\bfv_i}_{i=1}^m\right)\eqdef \set{\sum_{i=1}^m\alpha_i\bfv_i:\set{\alpha_i}_{i=1}^m\in\bbR^m}, \] i.e., it is the set of all possible linear combinations of the vectors.

  • The span of set of vectors \(\set{\bfv_i}_{i=1}^m\) is a subvector space.

  • The vectors \(\set{\bfv_i}_{i=1}^m\) are linearly independent if and only if \[ \sum_{i=1}^m\alpha_i\bfv_i \Rightarrow \forall i\in\intseq{1}{n}\quad\alpha_i=0. \] Otherwise the vectors are linearly dependent.

Span, linear independence

  • Any set of vectors \(\set{\bfv_i}_{i=1}^m\) is linearly dependent if and only if one vector can be expressed as a linear combination of the others.

  • Any set of vectors \(\set{\bfv_i}_{i=1}^m\) contains a subset of linearly independent vectors with the same span.

Bases

  • A set \(\set{\bfv_i}_{i=1}^m\) is a basis for a subvector space \(\calV\subset\bbR^n\) if

    • \(\text{span}(\set{\bfv_i}_{i=1}^m)=\calV\)
    • the vectors \(\set{\bfv_i}_{i=1}^m\) are linearly independent
  • Let \(\set{\bfv_i}_{i=1}^m\) be a basis for \(\calV\). Then \(\forall \vecv\in\calV\), there exists a unique set \(\set{\alpha_i}_{i=1}^m\) such that \[ \bfv = \sum_{i=1}^m \alpha_i \bfv_i \]

  • The decomposition on the basis is unique, but the basis is not!

  • A basis is useful to reduce the operation on vectors to operations on components of the vectors w.r.t to the basis (crucial to do things beyond \(\bbR^n\))

    • Dimension

    • In the following, \(\calV\) denotes a subvector space of \(\bbR^n\)

    • If a \(\calV\) contains a set of \(k\) linearly independent vectors and any \(k+1\) vectors are linearly dependent, then the dimension of \(\calV\) is \(k\), denoted \(\text{dim}(\calV)=k\).
    • If \(\text{dim}(\calV)=k\), any set of \(k\) linearly independent vectors is a basis for \(\calV\).
    • If there is basis for \(\calV\), then \(\text{dim}(\calV)\) is equal to the number of basis vectors.
    • \(\bbR^n\) is of dimension \(n\).

    Inner product, orthogonality

    • \(\bbR^n\) has other nice properties too!

    • For \(\bfu,\bfv\in\bbR^n\), \(\bfu^\intercal\bfv \eqdef \sum_{i=1}^n u_i v_i = \bfv^\intercal\bfu\) is an inner product such that

      • \(\bfu^\intercal\bfu=0\Rightarrow \bfu=\mathbf{0}\)
      • \(\bfu^\intercal (\alpha_1\bfv_1+\alpha_2\bfv_2) = \alpha_1\bfu^\intercal\bfv_1+\alpha_2\bfu^\intercal\bfv_2\)

  • Striclty speaking, this is not a definition, we’re checking that \(\bfu^\intercal\bfv\) defines a symmetric homogeneous bilinear form.

  • For \(\bfu,\bfv\in\bbR^n\), \(\bfu\) is orthogonal to \(\bfv\) if \(\bfu^\intercal\bfv=0\). Given a subvector space \(\calV\subset\bbR^n\), we define the orthogonal space \[ \calA^\perp \eqdef\set{\bfx\in\bbR^n:\forall \bfu\in\calA\,\bfu^\intercal\bfx=0} \]

    • Matrices

    • A matrix \(\matA\in\bbR^{m\times n}\) is a collection of vectors concatenated together \[ \matA \eqdef\mat{c}{-\vecr_1-\\-\vecr_2-\\\vdots\\-\vecr_m-} \eqdef \mat{cccc}{\vert&\vert&&\vert\\\vecc_1&\vecc_2&\cdots&\vecc_n\\\vert&\vert&&\vert} \]

    • Let \(\matA\in\bbR^{m\times n}\) be a matrix with columns \(\set{\vecc_j}_{j=1}^n\) and rows \(\set{\vecr_i}_{i=1}^m\).

      • The column (image) space \(\text{col}(\matA)\) of \(\matA\) is \(\text{span}\left(\set{\bfc_i}_{i=1}^n\right)\)
      • The row space \(\text{row}(\matA)\)of \(\matA\) is \(\text{span}\left(\set{\bfr_i}_{i=1}^m\right)\)
      • The null (kernel) space of \(\matA\) is \(\text{null}\eqdef(\matA)\set{\bfx\in\bbR^n:\matA\vecx=\mathbf{0}}\)

  • Note that \(\text{col}(\matA)=\text{row}(\matA^\intercal)\)

  • These spaces play an important role, in particular to solve systems of equations \(\matA\bfx=\bfy\) (You’ll be surprised how often this shows up in ECE!)

  • \(\text{null}(\matA)\) is a subvector space.

    • Rank

    • In the following \(\matA\) is an \(m\times n\) real-valued matrix

    • \(\text{dim}(\text{row}(\matA))=\text{dim}(\text{col}(\matA))\), which is called the rank of \(\matA\) denoted \(\text{rank}(\matA)\)

    • \(\text{row}(\matA)\) and \(\text{null}(\matA)\) are orthogonal complements, i.e.,

      • for any \(\bfu\in\text{row}(\matA)\), \(\bfv\in\text{null}(\matA)\) \(\bfu^\intercal\bfv=0\)
      • \(\text{dim}(\text{row}(\matA)) = n-\text{dim}(\text{null}(\matA))\)
      • for any \(\bfx\in\bbR^n\), there exists \(\bfx_2\in\text{row}(\matA)\), \(\bfx_2\in\text{null}(\matA)\) such that \(\bfx=\bfx_1+\bfx_2\)