10.4 Dot product and cosine similarity

Two vectors can have similar directions even when their lengths differ greatly. The dot product combines their components; cosine similarity removes the effect of length. These ideas appear in geometry and in comparing numeric feature vectors.

This is intermediate vector algebra. Study the dot-product section of OpenStax's vector lesson after practising vector components and norms.

A basic idea: alignment

For $u=(u_1,\ldots,u_n)$ and $v=(v_1,\ldots,v_n)$,

$$u\cdot v=\sum_{i=1}^{n}u_i v_i,\qquad \|u\|=\sqrt{\sum_{i=1}^{n}u_i^2}.$$

For nonzero vectors, $u\cdot v=\|u\|\|v\|\cos\theta$, where $\theta$ is the angle between them. A positive dot product means the angle is acute, zero means perpendicular, and a negative one means obtuse.

A quick tour of properties

For nonzero vectors, cosine similarity is $(u\cdot v)/(\|u\|\|v\|)$, from $-1$ to $1$ inclusive. It is undefined if either vector is zero. Scaling a vector by a positive number leaves its cosine similarity with another unchanged; the dot product is multiplied by that number. Dot product is symmetric and distributes over addition.

Application 1: geometric projection

Let $u=(3,4)$ and $v=(1,0)$. Then $u\cdot v=3$ and $\|u\|=5$, $\|v\|=1$. Cosine similarity is $3/5=0.6$. Because $v$ is a unit horizontal vector, the dot product $3$ is also the signed horizontal component of $u$.

Application 2: compare feature directions

Let a query feature vector be $q=(1,1)$, and two candidates be $a=(2,2)$ and $b=(3,0)$. The dot products are $q\cdot a=4$ and $q\cdot b=3$. Their cosine similarities are $1$ for $a$ and $3/(3\sqrt2)=1/\sqrt2\approx0.707$ for $b$. Candidate $a$ points in exactly the query's direction. If $b$ were $(30,0)$, its dot product with $q$ would be $30$, far above $a$'s $4$, but its cosine similarity would still be $1/\sqrt2$. Similarity alone does not tell you whether either candidate is useful for the real task.