Linear Algebra/Inner product spaces

Recall that in your study of vectors, we looked at an operation known as the dot product, and that if we have two vectors in ${ℝ}^n$, we simply multiply the components together and sum them up. With the dot product, it becomes possible to introduce important new ideas like length and angle. The length of a vector $𝐚$, is just $\Vert 𝐚\Vert =\sqrt{𝐚\cdot 𝐚}$. The angle between two vectors, $𝐚$ and $𝐛$, is related to the dot product by

$\cos \theta =\frac{𝐚\cdot 𝐛}{\Vert 𝐚\Vert \cdot \Vert 𝐛\Vert }$

It turns out that only a few properties of the dot product are necessary to define similar ideas in vector spaces other than ${ℝ}^n$, such as the spaces of $m\times n$ matrices, or polynomials. The more general operation that will take the place of the dot product in these other spaces is called the "inner product".

The inner product

Say we have two vectors:

$𝐚=\left(\begin{array}{c}2\\ 1\\ 4\end{array}\right),𝐛=\left(\begin{array}{c}6\\ 3\\ 0\end{array}\right)$

If we want to take their dot product, we would work as follows

$𝐚\cdot 𝐛=a_1{b}_1+a_2{b}_2+a_3{b}_3=(2)(6)+(1)(3)+(4)(0)=15$

Because in this case multiplication is commutative, we then have $𝐚\cdot 𝐛=𝐛\cdot 𝐚$.

But then, we observe

$𝐯\cdot (\alpha 𝐚+\beta 𝐛)=\alpha 𝐯\cdot 𝐚+\beta 𝐯\cdot 𝐛$

much like the regular algebraic equality $v(aA+bB)=avA+bvB$. For regular dot products this is true since, for ${ℝ}^3$, for example, one can expand both sides out to obtain

$\begin{array}{c}(\alpha v_1{a}_1+\beta v_1{b}_1)+(\alpha v_2{a}_2+\beta v_2{b}_2)+(\alpha v_3{a}_3+\beta v_3{b}_3)=\\ (\alpha v_1{a}_1+\alpha v_2{a}_2+\alpha v_3{a}_3)+(\beta v_1{b}_1+\beta v_2{b}_2+\beta v_3{b}_3)\end{array}$

Finally, we can notice that $𝐯\cdot 𝐯$ is always positive or greater than zero - checking this for ${ℝ}^3$ gives this as

$𝐯\cdot 𝐯=v_1^2+v_2^2+v_3^2$

which can never be less than zero since a real number squared is positive. Note $𝐯\cdot 𝐯=0$ if and only if $𝐯=0$.

In generalizing this sort of behaviour, we want to keep these three behaviours. We can then move on to a definition of a generalization of the dot product, which we call the inner product. An inner product of two vectors in some vector space $V$, written $⟨𝐱,𝐲⟩$ is a function $V\times V\to ℝ$, which obeys the properties

• $⟨𝐱,𝐲⟩=⟨𝐲,𝐱⟩$
• $⟨𝐯,\alpha 𝐚+\beta 𝐛⟩=\alpha ⟨𝐯,𝐚⟩+\beta ⟨𝐯,𝐛⟩$
• $⟨𝐚,𝐚⟩\ge 0$ with equality if and only if $𝐚=0$.

The vector space $V$ and some inner product together are known as an inner product space.

The dot product in ${ℂ}^n$

Given two vectors $𝐚=a_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n\in {ℂ}^n$ and $𝐛=b_1{\overrightarrow{e}}_1+b_2{\overrightarrow{e}}_2+\cdots +b_n{\overrightarrow{e}}_n\in {ℂ}^n$, the dot product generalized to complex numbers is:

$𝐚\cdot 𝐛=\sum _{i=1}^n{a}_i^{\ast }{b}_i=a_1^{\ast }{b}_1+a_2^{\ast }{b}_2+\cdots +a_n^{\ast }{b}_n$

where $z^{\ast }$ for an arbitrary complex number $z=c+di$ is the complex conjugate: $z^{\ast }=c-di$.

The dot product is "conjugate commutative": $𝐚\cdot 𝐛=(𝐛\cdot 𝐚{)}^{\ast }$. One immediate consequence of the definition of the dot product is that the dot product of a vector with itself is always a non-negative real number: $𝐚\cdot 𝐚\ge 0$.

$𝐚\cdot 𝐚=0$ if and only if $𝐚=\overrightarrow{0}$

The Cauchy-Schwarz Inequality for ${ℂ}^n$

Cauchy-Schwarz Inequality

Given two vectors $𝐚,𝐛\in {ℂ}^n$, it is the case that $|𝐚\cdot 𝐛|\le |𝐚||𝐛|$

In ${ℝ}^n$, the Cauchy-Schwarz inequality can be proven from the triangle inequality. Here, the Cauchy-Schwarz inequality will be proven algebraically.

To make the proof more intuitive, the algebraic proof for $𝐚,𝐛\in {ℝ}^n$ will be given first.

Proof for $𝐚,𝐛\in {ℝ}^n$

$|𝐚\cdot 𝐛|\le |𝐚||𝐛|$ follows from $|𝐚\cdot 𝐛{|}^2\le |𝐚{|}^2|𝐛{|}^2$ which is equivalent to

${\left(\sum _{i=1}^n{a}_i{b}_i\right)}^2\le \left(\sum _{i=1}^n{a}_i^2\right)\left(\sum _{j=1}^n{b}_j^2\right)$

expanding both sides gives:

$\sum _{i=1}^n\sum _{j=1}^n{a}_i{b}_i{a}_j{b}_j\le \sum _{i=1}^n\sum _{j=1}^n{a}_i^2{b}_j^2$

$⟺\sum _{i=1}^n\sum _{j=1}^n(a_i{b}_j)(a_j{b}_i)\le \sum _{i=1}^n\sum _{j=1}^n(a_i{b}_j{)}^2$

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

$⟺\sum _{i=2}^n\sum _{j=1}^{i-1}2(a_i{b}_j)(a_j{b}_i)\le \sum _{i=2}^n\sum _{j=1}^{i-1}((a_i{b}_j{)}^2+(a_j{b}_i{)}^2)$

$⟺0\le \sum _{i=2}^n\sum _{j=1}^{i-1}(a_i{b}_j-a_j{b}_i{)}^2$

The above inequality is clearly true, therefore the Cauchy-Schwarz inequality holds for $𝐚,𝐛\in {ℝ}^n$.

Proof for $𝐚,𝐛\in {ℂ}^n$

Note that $|𝐚\cdot 𝐛|\le |𝐚||𝐛|$ follows from $|𝐚\cdot 𝐛{|}^2\le |𝐚{|}^2|𝐛{|}^2$ which is equivalent to $(𝐚\cdot 𝐛{)}^{\ast }(𝐚\cdot 𝐛)\le (𝐚\cdot 𝐚)(𝐛\cdot 𝐛)$. Expanding both sides yields:

${\left(\sum _{i=1}^n{a}_i^{\ast }{b}_i\right)}^{\ast }\left(\sum _{i=1}^n{a}_i^{\ast }{b}_i\right)\le (\sum _{i=1}^n|a_i{|}^2)(\sum _{i=1}^n|b_i{|}^2)$

$⟺\sum _{i=1}^n\sum _{j=1}^n(a_i^{\ast }{b}_i{)}^{\ast }(a_j^{\ast }{b}_j)\le \sum _{i=1}^n\sum _{j=1}^n|a_i{|}^2|b_j{|}^2$

$⟺\sum _{i=1}^n\sum _{j=1}^n(a_j{b}_i{)}^{\ast }(a_i{b}_j)\le \sum _{i=1}^n\sum _{j=1}^n|a_i{b}_j{|}^2$

"Folding" the double sums along the diagonal, and cancelling out the diagonal terms which are equivalent on both sides, gives:

$⟺\sum _{i=2}^n\sum _{j=1}^{i-1}((a_j{b}_i{)}^{\ast }(a_i{b}_j)+(a_i{b}_j{)}^{\ast }(a_j{b}_i))\le \sum _{i=2}^n\sum _{j=1}^{i-1}(|a_i{b}_j{|}^2+|a_j{b}_i{|}^2)$

$⟺\sum _{i=2}^n\sum _{j=1}^{i-1}2\mathrm{\Re }((a_i{b}_j{)}^{\ast }(a_j{b}_i))\le \sum _{i=2}^n\sum _{j=1}^{i-1}(|a_i{b}_j{|}^2+|a_j{b}_i{|}^2)$

Given complex numbers $z$ and $w$, it can be proven that $2\mathrm{\Re }(z^{\ast }w)\le |z{|}^2+|w{|}^2$ (this is similar to $2xy\le x^2+y^2$ for real numbers). The above inequality holds, and therefore the Cauchy-Schwarz inequality holds for complex numbers.