ALAFF Orthonormal vectors and matrices

Subsection 2.2.3 Orthonormal vectors and matrices

A lot of the formulae in the last unit become simpler if the length of the vector equals one: If \(\| u \|_2 = 1 \) then

the component of \(v \) in the direction of \(u \) equals

\begin{equation*} \frac{u^H v}{u^H u} u = u^H v u. \end{equation*}
the matrix that projects a vector onto the vector \(u \) is given by

\begin{equation*} \frac{u u^H}{u^H u} = u u^H. \end{equation*}
the component of \(v \) orthogonal to \(u \) equals

\begin{equation*} v - \frac{u^H v}{u^H u} u = v - u^H v u. \end{equation*}
the matrix that projects a vector onto the space orthogonal to \(u \) is given by

\begin{equation*} I - \frac{u u^H}{u^H u} = I - u u^H. \end{equation*}

Homework 2.2.3.1.

Let \(u \neq 0 \in \Cm \text{.}\)

ALWAYS/SOMETIMES/NEVER \(u / \| u \|_2 \) has unit length.

Answer

ALWAYS.

Now prove it.

Solution

\begin{equation*} \begin{array}{l} \left\| \frac{u}{\| u \|_2} \right\|_2 \\ ~~~ = ~~~~ \lt \mbox{ homogeneity of norms } \gt \\ \frac{ \| u \|_2} {\| u \|_2} \\ ~~~ = ~~~~ \lt \mbox{ algebra } \gt \\ 1 \end{array} \end{equation*}

This last exercise shows that any nonzero vector can be scaled (normalized) to have unit length.

Definition 2.2.3.1. Orthonormal vectors.

Let \(u_0, u_1, \ldots, u_{n-1} \in \C^m \text{.}\) These vectors are said to be mutually orthonormal if for all \(0 \leq i,j \lt n \)

\begin{equation*} u_i^H u_j = \left\{ \begin{array}{c l} 1 \amp {\rm if~} i = j \\ 0 \amp {\rm otherwise} \end{array} \right. . \end{equation*}

The definition implies that \(\| u_i \|_2 = \sqrt{ u_i^H u_i } = 1 \) and hence each of the vectors is of unit length in addition to being orthogonal to each other.

The standard basis vectors (Definition 1.3.1.3)

\begin{equation*} \{ e_j \}_{i=0}^{m-1} \subset \Cm, \end{equation*}

where

\begin{equation*} e_j = \left( \begin{array}{c} 0 \\ \vdots \\ 0 \\ 1 \\ 0 \\ \vdots \\ 0 \end{array} \right) \longleftarrow \mbox{ entry indexed with } j \end{equation*}

are mutually orthonormal since, clearly,

\begin{equation*} e_i^H e_j = \left\{ \begin{array}{c l} 1 \amp {\rm if~} i = j \\ 0 \amp {\rm otherwise.} \end{array} \right. \end{equation*}

Naturally, any subset of the standard basis vectors is a set of mutually orthonormal vectors.

Remark 2.2.3.2.

For \(n \) vectors of size \(m \) to be mutually orthonormal, \(n \) must be less than or equal to \(m \text{.}\) This is because \(n \) mutually orthonormal vectors are linearly independent and there can be at most \(m \) linearly independent vectors of size \(m \text{.}\)

A very concise way of indicating that a set of vectors are mutually orthonormal is to view them as the columns of a matrix, which then has a very special property:

Definition 2.2.3.3. Orthonormal matrix.

Let \(Q \in \C^{m \times n} \) (with \(n \leq m \)). Then \(Q \) is said to be an orthonormal matrix iff \(Q^H Q = I \text{.}\)

The subsequent exercise makes the connection between mutually orthonormal vectors and an orthonormal matrix.

Homework 2.2.3.2.

Let \(Q \in \C^{m \times n} \) (with \(n \leq m \)). Partition \(Q = \left( \begin{array}{c | c | c | c} q_0 \amp q_1 \amp \cdots \amp q_{n-1} \end{array} \right) \text{.}\)

TRUE/FALSE: \(Q \) is an orthonormal matrix if and only if \(q_0, q_1, \ldots , q_{n-1} \) are mutually orthonormal.

Answer

TRUE

Now prove it!

Solution

Let \(Q \in \C^{m \times n} \) (with \(n \leq m \)). Partition \(Q = \left( \begin{array}{c | c | c | c} q_0 \amp q_1 \amp \cdots \amp q_{n-1} \end{array} \right)\text{.}\) Then

\begin{equation*} \begin{array}{rcl} Q^H Q \amp=\amp \left( \begin{array}{c | c | c | c} q_0 \amp q_1 \amp \cdots \amp q_{n-1} \end{array} \right)^H \left( \begin{array}{c | c | c | c} q_0 \amp q_1 \amp \cdots \amp q_{n-1} \end{array} \right) \\ \amp=\amp \left( \begin{array}{c} q_0^H \\ q_1^H \\ \vdots \\ q_{n-1}^H \end{array} \right) \left( \begin{array}{c | c | c | c} q_0 \amp q_1 \amp \cdots \amp q_{n-1} \end{array} \right) \\ \amp=\amp \left( \begin{array}{c | c | c | c} q_0^H q_0 \amp q_0^H q_1 \amp \cdots \amp q_0^H q_{n-1} \\ \hline q_1^H q_0 \amp q_1^H q_1 \amp \cdots \amp q_1^H q_{n-1} \\ \hline \vdots \amp \vdots \amp \amp \vdots \\ \hline q_{n-1}^H q_0 \amp q_{n-1}^H q_1 \amp \cdots \amp q_{n-1}^H q_{n-1} \end{array} \right). \end{array} \end{equation*}

Now consider that \(Q^H Q = I \text{:}\)

\begin{equation*} \begin{array}{rcl} \left( \begin{array}{c | c | c | c} q_0^H q_0 \amp q_0^H q_1 \amp \cdots \amp q_0^H q_{n-1} \\ \hline q_1^H q_0 \amp q_1^H q_1 \amp \cdots \amp q_1^H q_{n-1} \\ \hline \vdots \amp \vdots \amp \amp \vdots \\ \hline q_{n-1}^H q_0 \amp q_{n-1}^H q_1 \amp \cdots \amp q_{n-1}^H q_{n-1} \end{array} \right) = \left( \begin{array}{c | c | c | c} 1 \amp 0 \amp \cdots \amp 0 \\ \hline 0 \amp 1 \amp \cdots \amp 0 \\ \hline \vdots \amp \vdots \amp \amp \vdots \\ \hline 0 \amp 0 \amp \cdots \amp 1 \end{array} \right). \end{array} \end{equation*}

Clearly \(Q \) is orthonormal if and only if \(q_0, q_1, \ldots, q_{n-1} \) are mutually orthonormal.

Homework 2.2.3.3.

Let \(Q \in \Cmxn \text{.}\)

ALWAYS/SOMETIMES/NEVER: If \(Q^H Q = I \) then \(Q Q^H = I \text{.}\)

Answer

SOMETIMES.

Now explain why.

Solution

If \(Q \) is a square matrix (\(m = n \)) then \(Q^H Q = I \) means \(Q^{-1} = Q^H \text{.}\) But then \(Q Q^{-1} = I \) and hence \(Q Q^H = I \text{.}\)
If \(Q \) is not square, then \(Q^H Q = I \) means \(m \gt n \text{.}\) Hence \(Q \) has rank equal to \(n \) which in turn means \(Q Q^H\) is a matrix with rank at most equal to \(n \text{.}\) (Actually, its rank equals \(n \text{.}\)). Since \(I \) has rank equal to \(m \) (it is an \(m \times m \) matrix with linearly independent columns), \(Q Q^H \) cannot equal \(I \text{.}\)

More concretely: let \(m \gt 1 \) and \(n = 1 \text{.}\) Choose \(Q = \left( \begin{array}{c} e_0 \end{array} \right) \text{.}\) Then \(Q^H Q = e_0^H e_0 = 1 = I \text{.}\) But

\begin{equation*} Q Q^H = e_0 e_0^H = \left( \begin{array}{c c c } 1 \amp 0 \amp \cdots \\ 0 \amp 0 \amp \cdots \\ \vdots \amp \vdots \amp \end{array} \right). \end{equation*}