AI TutorHigh School
SAT
ACT
University
MCAT
LSAT
DAT
Wize University Linear Algebra Textbook > Orthogonality

Orthogonal Complement and Orthogonal Projection (COMING SOON)

Orthonormal Basis and Gram-Schmidt ProcessPrevious Section
Orthogonal Diagonalization of Symmetric Matrices (COMING SOON)Next Section
Popular Courses
Find My Course

Orthogonal Matrices

Orthogonal Matrix

A matrix is orthogonal if its columns form an orthonormal set.
I.e. Every column is a unit vector and the dot product of any two columns is zero.
An equivalent definition: QQ is an orthogonal matrix if and only if QTQ=IQ^TQ = I.
Properties
Suppose QQ is square (n×nn \times n) and x⃗,y⃗∈Rn\vec x, \vec y \in \reals^n. The following are equivalent:
  1. QQ is an orthogonal matrix
  2. Q−1=QTQ^{-1} = Q^T
  3. ∥Qx⃗∥=∥x⃗∥\lVert Q \vec x \rVert = \lVert \vec x \rVert
  4. Qx⃗⋅Qy⃗ = x⃗⋅y⃗Q \vec x \cdot Q \vec y \ =\ \vec x \cdot \vec y

Orthogonal Complement

Let WW be a subspace of Rn\reals^n.
The orthogonal complement of WW, denoted W⊥W^\perp, is the set of vectors that are orthogonal to every vector in WW.
That is,
W⊥={v⃗∈Rn ∣ v⃗⋅w⃗=0, ∀w⃗∈W}W^\perp = \{ \vec v \in \reals^n \ | \ \vec v \cdot \vec w =0, \ \forall \vec w \in W \}
Example
Suppose WW is a plane in R3\reals^3 with normal vector n⃗\vec n. What is W⊥W^\perp?
Then n⃗\vec n must be in W⊥W^\perp since, by definition, vector n⃗\vec n is orthogonal to the plane (meaning: orthogonal to every vector that lies in the plane).
Moreover, every scalar multiple of n⃗\vec n is orthogonal to WW.

The set of scalar multiples of n⃗\vec n is a line, call it LL, and this line is the orthogonal complement of WW.
Conversely, the orthogonal complement of the line is the plane.
W⊥=L , L⊥=WW^\perp = L \ ,\ \quad L^\perp = W

PAGE BREAK
Properties
Given a subspace WW of Rn\reals^n:
  • W⊥W^\perp is a subspace of Rn\reals^n
  • W∩W⊥={0⃗}W \cap W^\perp = \{\vec 0\} (no vector except 0⃗\vec 0 can be in both)
  • Dim W+Dim W⊥=n{\rm Dim}\ W + {\rm Dim}\ W^\perp = n
  • (Row A)⊥=Null A,  (Col A)⊥=Null AT({\rm Row}\ A)^\perp = {\rm Null}\ A , \ \ ({\rm Col}\ A)^\perp = {\rm Null}\ A^T

Orthogonal Projection

We have seen projection of one vector onto another. Now we'll see projection onto an entire subspace!
Let WW be a subspace of Rn\reals^n with orthogonal basis {v⃗1,…,v⃗n}\{ \vec v_1, \dots, \vec v_n \}.
The orthogonal projection of vector x⃗\vec x onto WW is the sum of the projections onto each basis vector:
projW x⃗=projv⃗1 x⃗+⋯+projv⃗n x⃗{\rm proj}_W \ \vec x = {\rm proj}_{\normalsize \vec v_1} \ \vec x + \dots + {\rm proj}_{\normalsize \vec v_n} \ \vec x
Or, written out in full,
projW x⃗=(x⃗⋅v⃗1v⃗1⋅v⃗1)v⃗1+⋯+(x⃗⋅v⃗nv⃗n⋅v⃗n)v⃗n{\rm proj}_W \ \vec x = \left( \frac{\vec x \cdot \vec v_1}{\vec v_1 \cdot \vec v_1}\right) \vec v_1 + \dots + \left(\frac{\vec x \cdot \vec v_n}{\vec v_n \cdot \vec v_n}\right) \vec v_n
The projection must lie in WW (it's a linear combination of its basis vectors).
What is the complement of the line formed by this projection vector? The perp!
perpW x⃗=x⃗−projW x⃗\boxed{ {\rm perp}_W\ \vec x = \vec x - {\rm proj}_W\ \vec x }

This is the basis (no pun intended) of the Gram-Schmidt orthogonalization process!
PAGE BREAK
Why is this useful?
The projection of a vector onto a subspace WW is the best approximation of the vector in the subspace.
It also allows us to decompose a vector into two parts: one part in WW, and one orthogonal part in W⊥W^\perp.
That is, for every vector in x∈Rnx \in \reals^n,
x⃗=w⃗+v⃗ , where  w⃗∈W,   v⃗∈W⊥\boxed{\vec x = \vec w + \vec v}\ , \ \text{where }\ \vec w \in W \text{, }\ \ \vec v \in W^\perp

Extra Practice
19.4F_Final_Builder_Ch_17.3_Least_Squares_$\tkco{eg4}$_$\key{Final}$_Builder_$\tkcth{17.3.}\tkcf{4}$_
The figure below illustrates the relationship between the Im (A‾)\bct{\text{Im}\, (\A)} and b⃗\bco{\color{cyan}\vb} for the system given by:
{x−2y=−42x−4y=7⇔[  1−2    2−4  ][abc] ⁣ ⁣ ⁣ ⁣⏟:= A‾[xy]=[−47][abc]⏟:= b⃗ \left\{ \begin{array}{rr} x - 2y \quad =& -4 \\ 2x - 4y \quad =& 7 \end{array} \right. % \qquad \Leftrightarrow \qquad % \underbrace{ \begin{bmatrix} \; 1 & -2 \; \\ \; 2 & -4 \; \end{bmatrix} \vphantom{\colvecth{a}{b}{c}\!\!\!\!} }_{:= \, \bct{\A}} % \colvec{x}{y} % = % \underbrace{ \colvec{-4}{7} \vphantom{\colvecth{a}{b}{c}} }_{:=\, \bco{\color{cyan}\vb}}
Let b⃗Im(A‾)=proj⇀Im(A‾)(b⃗)\vb_{_{\textrm{Im}(\A)}} = \overrightharpoon{\text{proj}}_{_{\textrm{Im}(\A)}}(\vb), i.e. the (vector) component of b⃗\vb which is in the image of A‾\A. Identify this vector on the figure; we'll call this component b⃗∣∣\vb_{_{||}} for short.
The projection of a vector b⃗\vb onto the image of a set of vectors {v⃗1,v⃗2}\big\{ \vv_1, \vv_2 \big\} is equal to the sum of the projection of the vector b⃗\vb onto each of the individual vectors, i.e.:
proj⇀Im(v⃗1,v⃗2)(b⃗)=proȷ⇀v1⃗ ⁣(b⃗)+proȷ⇀v2⃗ ⁣(b⃗) \overrightharpoon{\textrm{proj}}_{_{\textrm{Im}(\vv_1,\vv_2)}}(\vb) = \proj{b}{v_1} + \proj{b}{v_2}
Span
Let B:={(1,1,0,2),(0,1,0,−1),(0,0,1,1)}B:=\{(1,1,0,2),(0,1,0,-1),(0,0,1,1)\} , and U:=spanBU:=spanB .
a) Find a pairwise orthogonal set of nonzero vectors OO so that U=spanOU=spanO .
b) Write the Fourier expansion of (0,0,1,0)(0,0,1,0) in terms of the basis OO .
19.4F_Final_Builder_Ch_17.2_Orthogonal_Complement_$\tkco{eg4}$_$\key{Final}$_Builder_$\tkcth{17.2.}\tkcf{4}$_
Suppose U={v⃗1, … , v⃗n}U = \left\{ \vv_1,\, \dots\,,\, \vv_n \right\} is an orthogonal basis for Rn\mathbb{R}^{n}, then any vector w⃗\vw in Rn\mathbb{R}^{n} can be written as a linear combination of these vectors, i.e.
w⃗=∑inci v⃗i=c1 v⃗1+c2 v⃗2+ … +cn v⃗n. \vw = \sum_{i}^{n} c_i \, \vv_i = c_1 \, \vv_1 + c_2 \, \vv_2 + \, \dots \, + c_n \, \vv_n.
Find an expression for the constants c1c_1, c2c_2, …\dots, cnc_n in terms of w⃗\vw and the vectors in UU.
19.4F_WML_6_$\tkco{eg10}$_$\key{Final}$_Builder_$\tkcth{17.1.}\tkct{11}$_
Find the set of all vectors perpendicular to the vectors {a⃗1, a⃗2 }={[1101], [0111]}\left\{ \va_1,\, \va_2\ \right\} = \left\{ \colvecf{1}{1}{0}{1}, \, \colvecf{0}{1}{1}{1} \right\}
Consider the subspace W={(x,y,z)∈R3  ∣  x−y+3z=0}W=\Big\{(x,y,z)\in \mathbb{R}^3 \; | \; x-y+3z=0 \Big\} of R\mathbb{R}.
Find a basis for WW that contains the vector v⃗=[0  3  1]\vec{v}=[0 \; 3 \; 1].
Find orthogonal complement of WW in R3\mathbb{R}^3.
19.4F_WML_6_$\tkco{eg7}$_$\key{Final}$_Builder_$\tkcth{17.1.}\tkct{7}$_
Find a basis for the set of all vectors that are perpendicular to both {[1102], [1010]}\left\{ \colvecf{1}{1}{0}{2}, \, \colvecf{1}{0}{1}{0} \right\}
19.4F_WML_6_$\tkco{eg13}$_$\key{Final}$_Builder_$\tkcth{17.1.}\tkct{14}$_
Show that any vector v⃗\vv in the span of the set
V={v⃗1, v⃗2, v⃗3}={[102], [214], [112]} V = \left\{ \vv_1,\, \vv_2,\, \vv_3 \right\} = \left\{ \colvecth{1}{0}{2}, \, \colvecth{2}{1}{4}, \, \colvecth{1}{1}{2} \right\}
will be perpendicular to any vector w⃗\vw in the span of the orthogonal complement W=V⊥W = V^{\perp} of the set.
19.4F_WML_6_$\tkco{eg12}$_$\key{Final}$_Builder_$\tkcth{17.1.}\tkct{13}$_
Describe the orthogonal complement of the matrix whose columns are given by the vectors {v⃗1, v⃗2, v⃗3}={[102], [214], [112]}\left\{ \vv_1,\, \vv_2,\, \vv_3 \right\} = \left\{ \colvecth{1}{0}{2}, \, \colvecth{2}{1}{4}, \, \colvecth{1}{1}{2} \right\} by finding a basis for the orthogonal complement, and describing the span of the basis in geometric terms.
Example: Fourier Expansion
Example: Fourier Expansion
Write the vector (1,1,1)(1,1,1) as a linear combination of {(1,0,1),(1,1,−1),(−1,2,1)}\{(1,0,1),(1,1,-1), (-1,2,1)\}
Basis of Orthogonal Complement
Find an orthonormal basis for the orthogonal complement of the intersection of the planes 2x+y−z=02x+y-z=0 and 4x−3y=04x-3y=0 in R3\mathbb{R}^3 .
Orthogonal Complements
Suppose UU is a subspace of Rn\mathbb{R}^n . Prove that (U⊥)⊥=U(U^\perp)^\perp=U .
Orthogonal Complement
If UU is a subspace of Rn\mathbb{R}^n , show that dim(U)+dim(U⊥)=ndim(U)+dim(U^\perp)=n .
Basis of Orthogonal Complement
Find a basis for the orthogonal complement of the intersection of the planes 2x+y−z=02x+y-z=0 and 4x−3y=04x-3y=0 in R3\mathbb{R}^3 .
Projection onto a subspace
Suppose u⃗,v⃗∈Rn\vec u,\vec v\in\mathbb{R}^n satisfy u⃗⋅v⃗=0,∣∣u⃗∣∣=∣∣v⃗∣∣=1\vec u\cdot\vec v=0, ||\vec u||=||\vec v||=1, and define U:=span{u⃗,v⃗}U := span\{\vec u,\vec v\} .
Using the definition that projUx⃗=A(ATA)−1ATx⃗proj_U\vec x=A(A^TA)^{-1}A^T\vec x , where A=[u⃗v⃗]A=\left[\begin{array}{rr}\vec u&\vec v\end{array}\right] , show that for all x⃗∈Rn\vec x\in\mathbb{R}^n we have that projUx⃗=proju⃗x⃗+projv⃗x⃗proj_U\vec x=proj_{\vec u}\vec x+proj_{\vec v}\vec x .
Projection onto a Subspace
Suppose S:={a⃗1,...,a⃗k}S:=\{\vec a_1,...,\vec a_k\} is a set of orthonormal vectors in Rn\mathbb{R}^n , U:=span(S)U:=span(S) , and suppose that the n×kn\times k matrix AA is defined by A:=[a⃗1a⃗2...a⃗k]A:=\left[\begin{array}{rrrr}\vec a_1&\vec a_2&...&\vec a_k\end{array}\right] .
Show that AATAA^T is the matrix of the linear transformation: Rn→Rnx⃗→projUx⃗\begin{array}{l}\mathbb{R}^n\rightarrow\mathbb{R}^n\\ \vec x\rightarrow proj_U\vec x\end{array} .