Covariance

What is Covariance?

This post assumes you already know what variance is but if you are unfamiliar, check Variance out first.

Theoretically speaking, for the population, the covariance is defined as $\text{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]$. Oftentimes, we don’t know the full probability distribution of the random variables X and Y so we use the sample covariance to approximate the population covariance.

Then, by defition, sample covariance between two random variables X and Y, for sample size $n$ is defined as $Cov(X,Y)=\frac{1}{n-1}{\sum }_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$ where $\bar{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ and $\bar{Y}=\frac{1}{n}{\sum }_{i=1}^n{Y}_i$ are the sample means of X and Y, respectively. Note that here we use the factor $\frac{1}{n-1}$ in calculating the sample covariance to account for the fact that the sample means are just estimates not the true means.

So we have the definition of the covariance but what does it actually represent? In a nutshell, covariance tells us how the two variables change together on average. If X and Y are linearly related, their covariance tells us the strength and direction of the linear relationship. Positive covariance means X and Y tend to increase or decrease together whereas negative covariance indicates that X and Y tend to move in opposite directions. If the covariance is $0$, that means there is no relationship between the two variables.

Warning

This is not a completed post