Advanced Linear Algebra: Foundations to Frontiers

Subsection 2.5.2 Summary

Given \(x, y \in \Cm \)

• their dot product (inner product) is defined as

  \begin{equation*} x^H y = \overline x^T y = \overline{x^T} y = \overline \chi_0 \psi_0 + \overline \chi_1 \psi_1 + \cdots + \overline \chi_{m-1} \psi_{m-1} = \sum_{i=0}^{m-1} \overline \chi_i \psi_i . \end{equation*}

• These vectors are said to be orthogonal (perpendicular) iff \(x^H y = 0 \text{.}\)

• The component of \(y \) in the direction of \(x \) is given by

  \begin{equation*} \frac{x^H y}{x^H x} x = \frac{x x^H }{x^H x} y. \end{equation*}

  The matrix that projects a vector onto the space spanned by \(x \) is given by

  \begin{equation*} \frac{x x^H } {x^H x}. \end{equation*}

• The component of \(y \) orthogonal to \(x \) is given by

  \begin{equation*} y - \frac{x^H y}{x^H x} x = \left( I - \frac{x x^H }{x^H x}\right) y. \end{equation*}

  Thus, the matrix that projects a vector onto the space orthogonal to \(x \) is given by

  \begin{equation*} I - \frac{x x^H } {x^H x}. \end{equation*}

Given \(u,v \in \Cm \) with \(u \) of unit length

• The component of \(v \) in the direction of \(u \) is given by

  \begin{equation*} u^H v u = u u^H v. \end{equation*}

• The matrix that projects a vector onto the space spanned by \(u \) is given by

  \begin{equation*} u u^H \end{equation*}

• The component of \(v \) orthogonal to \(u \) is given by

  \begin{equation*} v - u^H v u = \left( I - u u^H \right) v. \end{equation*}

• The matrix that projects a vector onto the space that is orthogonal to \(x \) is given by

  \begin{equation*} I - u u^H \end{equation*}

Let \(u_0, u_1, \ldots, u_{n-1} \in \C^m \text{.}\) These vectors are said to be mutually orthonormal if for all \(0 \leq i,j \lt n \)

Let \(Q \in \C^{m \times n} \) (with \(n \leq m \)). Then \(Q \) is said to be

• an orthonormal matrix iff \(Q^H Q = I \text{.}\)

• a unitary matrix iff \(Q^H Q = I \) and \(m = n \text{..}\)

• an orthogonal matrix iff it is a unitary matrix and is real-valued.

Let \(Q \in \C^{m \times n} \) (with \(n \leq m \)). Then \(Q = \left( \begin{array}{c | c | c} q_0 \amp \cdots q_{n-1} \end{array} \right) \) is orthonormal iff \(\{ q_0, \ldots, q_{n-1} \) are mutually orthonormal.

If \(U, V \in \C^{m \times m} \) are unitary, then

• \(U^H U = I \text{.}\)

• \(U U^H = I \text{.}\)

• \(U^{-1} = U^H \text{.}\)

• \(U^H \) is unitary.

• \(U V \) is unitary.

If \(U \in \Cmxm \) and \(V \in \C^{n \times n} \) are unitary, \(x \in \Cm \text{,}\) and \(A \in \Cmxn \text{,}\) then

• \(\| U x \|_2 = \| x \|_2 \text{.}\)

• \(\| U^H A \|_2 = \| U A \|_2 = \| A V \|_2 = \| A V^H \|_2 = \| U^H A V \|_2 = \| U A V^H \|_2 = \| A \|_2 \text{.}\)

• \(\| U^H A \|_F = \| U A \|_F = \| A V \|_F = \| A V^H \|_F = \| U^H A V \|_F = \| U A V^H \|_F = \| A \|_F \text{.}\)

• \(\| U\|_2 = 1 \)

• \(\kappa_2( U ) = 1 \)

Examples of unitary matrices:

• Rotation in 2D: \(\left( \begin{array}{r r} c \amp -s \\ s \amp c \end{array} \right) \text{.}\)

• Reflection: \(I - 2 u u^H \) where \(u \in \Cm\) and \(\| u \|_2 = 1 \text{.}\)

Change of orthonormal basis: If \(x \in \Cm \) and \(U = \left( \begin{array}{c | c | c} u_0 \amp \cdots \amp u_{m-1} \end{array} \right) \) is unitary, then

Let \(A \in \Cnxn \) be nonsingular and \(x \in \Cn \) a nonzero vector. Consider

Then

where \(\| \cdot \| \) is an induced matrix norm.

Let \(A \in \C^{m \times n} \) and \(A = U \Sigma V^H \) its SVD with

and

Here \(U_L \in \C^{m \times r} \text{,}\) \(V_L \in \C^{n \times r} \) and \(\Sigma_{TL} \in \R^{r \times r } \text{.}\) Then

• \(\| A \|_2 = \sigma_0 \text{.}\) (The 2-norm of a matrix equals the largest singular value.)

• \(\rank( A ) = r \text{.}\)

• \(\Col( A ) = \Col( U_L ) \text{.}\)

• \(\Null( A ) = \Col( V_R ) \text{.}\)

• \(\Rowspace( A ) = \Col( V_L ) \text{.}\)

• Left null-space of \(A = \Col( U_R ) \text{.}\)

• \(A^H = V \Sigma^T U^H \text{.}\)

• SVD: \(A^H = V \Sigma U^H \text{.}\)

• Reduced SVD: \(A = U_L \Sigma_{TL} V_L^H \text{.}\)

• \begin{equation*} A = \begin{array}[t]{c} \underbrace{ ~~~~~~~~~\sigma_0 u_0 v_0^H ~~~~~~~~~ } \\ \sigma_0 \!\!\!\! \begin{array}[t]{c|c} ~\amp~\\ ~\amp~\\ \end{array} \!\!\!\! \begin{array}[t]{c}\hline ~~~~~~~~~ \\ ~ \end{array} \end{array} + \begin{array}[t]{c} \underbrace{ ~~~~~~~~~\sigma_1 u_1 v_1^H ~~~~~~~~~ } \\ \sigma_1 \!\!\!\! \begin{array}[t]{c|c} ~\amp~\\ ~\amp~\\ \end{array} \!\!\!\! \begin{array}[t]{c} \hline ~~~~~~~~~ \\ ~ \end{array} \end{array} + \cdots + \begin{array}[t]{c} \underbrace{ ~~~~~~~~~\sigma_{r-1} u_{r-1} v_{r-1}^H ~~~~~~~~~ } \\ \sigma_{r-1} \!\!\!\! \begin{array}[t]{c|c} ~\amp~\\ ~\amp~\\ \end{array} \!\!\!\! \begin{array}[t]{c} \hline ~~~~~~~~~ \\ ~ \end{array} \end{array} . \end{equation*}

• Reduced SVD: \(A^H = V_L \Sigma U_L^H \text{.}\)

• If \(m \times m \) matrix \(A \) is nonsingular: \(A^{-1} = V \Sigma^{-1} U^H \text{.}\)

• If \(A \in \Cmxm \) then \(A \) is nonsingular if and only if \(\sigma_{m-1} \neq 0 \text{.}\)

• If \(A \in \Cmxm \) is nonsingular then \(\kappa_2( A ) = \sigma_0 / \sigma_{m-1} \text{.}\)

• (Left) pseudo inverse: if \(A \) has linearly independent columns, then \(A^\dagger = ( A^H A )^{-1} A^H = V \Sigma_{TL}^{-1} U_L^H \text{.}\)

• \(v_0 \) is the direction of maximal magnification.

• \(v_{n-1} \) is is the direction of minimal magnification.

• If \(n \leq m \text{,}\) then \(A v_j = \sigma_j u_j \text{,}\) for \(0 \leq j \lt n \text{.}\)