This short post addresses the title question, a common query from my students. First, we will explain the singular value decomposition (SVD) using the definition as a starting point, the way it is introduced in lectures. We will then go through a derivation to arrive at the definition by analogy with eigenvalues and eigenvectors for a square matrix.

Fait accompli

In my Maths Data Book, the SVD of a real $m$ by $n$ matrix $\bf{A}$ is defined as, $$\newcommand{\tra}[1]{\bf{#1}^\mathrm{t}} \newcommand{\Qtwo}{{\bf V}} \newcommand{\Qtt}{{\bf V}^\mathrm{t}} \newcommand{\Qone}{{\bf U}} \newcommand{\Rn}{\mathbb{R}^n} \newcommand{\Rm}{\mathbb{R}^m} \newcommand{\Sg}{{\bf \Sigma}} {\bf{A}} = {\Qone \Sg \Qtt} \ , \tag{I}$$ where:

• $\Qone$ contains the $m$ unit eigenvectors of $\bf{A}\tra{A}$ as columns;
• $\Qtwo$ contains the $n$ unit eigenvectors of $\tra{A}\bf{A}$ as columns;
• $\Sg$ is an $m$ by $n$ diagonal matrix containing the square roots of the eigenvalues shared between $\tra{A}\bf{A}$ and $\bf{A}\tra{A}$, conventionally in descending order.

If we consider $\bf{A}$ as a linear transformation from $\Rn$ to $\Rm$, then we can see the SVD as a decomposition into three separate transformations: $\Qtwo$ is a reflection or rotation in $\Rn$, $\Sg$ is a scaling along each coordinate direction, and $\Qone$ is a reflection or rotation in $\Rm$.

The SVD also reveals orthonormal bases for the four fundamental subspaces of $\bf{A}$. Because eigenvectors are by definition orthogonal, we can form a vector space in $\Rn$ by using as many columns of $\Qtwo$ as we desire dimensions, and similarly for $\Rm$ with the columns of $\Qone$. If the rank of $\bf{A}$ is $r$, we need $r$ vectors for each of the column and row space, $n-r$ for the null space, and $m-r$ for the left null space.

It turns out that exactly $r$ of the singular values are non-zero, allowing a two-way mapping between $\Rn$ and $\Rm$ of the first $r$ columns of $\Qtwo$ and $\Qone$. These are bases for the row and column spaces respectively.

If we multiply out the decomposition, Eqn. $\rm(I)$, the remaining $n-r$ columns of $\Qtwo$, corresponding to zero singular values, are mapped to $\vec{0}$. That is, they are solutions to ${\bf{A}}\vec{x} = \vec{0}$ and and a basis for the null space. Taking the transpose of the definition, we also see that the last $m-r$ columns of $\Qone$ are mapped to $\vec{0}$, are solutions of ${\tra{A}}\vec{b}=\vec{0}$, and hence a basis for the left null space.

Ab initio

This is all very well, but how did we get here? We will now take a different perspective, that links back to eigenvalues and illustrates the significance of $\bf{A}\tra{A}$.

An eigenvector $\vec{a}$ of the matrix $\bf{C}$ maps to itself scaled by the eigenvalue $\lambda$, or $${\bf{C}}\vec{a} = \lambda \vec{a} \ .$$ Eigenvectors and eigenvalues are neat, but they only work for square matrices mapping within the same $\Rn$. For an $m$ by $n$ rectangular matrix, we need two sets of special unit vectors $\vec{v}$ and $\vec{u}$ in $\Rn$ and $\Rm$, but we can get close to the spirit of eigenvectors by defining, $${{\bf{A}}\vec{v}} = \sigma \vec{u} \ , \tag{II}$$ where $\sigma$ is a scaling factor. Stacking special vectors column-wise into matrices $\bf V$ and $\bf U$, $${\bf{A}\bf{V}} = {\bf{U} \Sg }\ ,$$ where $\Sg$ is a diagonal matrix of $\sigma$ values for each $\vec{v}$, $\vec{u}$ pair, post-multiplying to scale the columns of $\bf{U}$.

If the special vectors are unit, then $\bf U$ and $\bf V$ are orthonormal, $${\bf{U}}^{-1} = {\tra{U}} \ , \quad {\bf{V}}^{-1} = {\tra{V}} \ , \quad \Rightarrow \quad {{\bf{U}}\tra{U}} = 1 \ , \quad {{\bf{V}}\tra{V}} = 1$$ and we can rearrange to get to our original definition of Eqn. $\rm (I)$, ${\bf{A}} = {\bf U\Sg\tra{V}}$. Then after simplifying, $$\begin{align} {\bf{A}\tra{A}} &= {\bf U\Sg\tra{V}V\tra{\Sg}\tra{U}} = {\bf U} ({\Sg\tra{\Sg}}) {\tra{U}} \\ {\tra{A}\bf{A}} &= {\bf V\tra{\Sg}\tra{U}U\Sg\tra{V}} = {\bf V} ({\tra{\Sg}\Sg}) {\tra{V}} \end{align}$$ These equations are diagonalised forms of the square matrices $\bf{A}\tra{A}$ and $\tra{A}\bf{A}$. Our special matrices $\vec{u}$ are in fact the left-singular vectors, eigenvectors of $\bf{A}\tra{A}$. Our special matrices $\vec{v}$ are the right-singular vectors, eigenvectors of $\tra{A}\bf{A}$. Because $\Sg$ is a diagonal matrix, both $\bf \Sg\tra{\Sg}$ and $\tra{\Sg}\Sg$ are also diagonal with elements $\sigma^2$, hence why we have to square root the eigenvalues of $\bf{A}\tra{A}$.

The point is, Eqn. $\rm (II)$ implies our original definition in Eqn. $\rm (I)$.