kullback-leibler divergence

what is KL divergence? §

KL divergence is a metric that measures the difference between two probabilities.

for a discrete random variable, $d_{KL}(p(x)||q(x))={\sum }_{x\in x}p(x)\ln \frac{p(x)}{q(x)}$ and for a continuous random variable $x$, $d_{KL}(p(x)||q(x))={\int }_{-\infty }\infty p(x)\ln \frac{p(x)}{q(x)}dx$

where $p(x)>0,q(x)>0\forall x\text{and}\sum p(x)=1,\sum q(x)=1$

• KL divergence is a non-symmetric measure: $d_{KL}(p(x)\Vert \Vert q(x))\ne d_{KL}(q(x)\Vert \Vert p(x))$
• pretty much used to measure how much information is lost when $q(x)$ is used to approximate $p(x)$
• generally, $p(x)$ is true distribution of data that we can really never know and $q(x)$ is the approximation of $p(x)$ from a neural network.
• it’s not a distance metric and doesn’t need to satisfy the triangular inequality
• it’s a non-negative measure
• $d_{KL}(p(x)||q(x))\ge 0,d_{KL}(p(x)||q(x))=0⟷p(x)=q(x)$
• ${\lim }_{p\to 0}p\log (p)=0$ when $p\ne 0$. but when $q=0,d_{KL}(p||q)=\infty$
• if $p(z)>0\text{ and }q(z)=0$ $\to$ the 2 distributions are absolutely different

KL divergence and maximum likelihood estimator §

before we identify the relationship between KL divergence metric and the maximum likelihood estimator, we have to look into what exactly maximum likelihood estimation is. in short, maximum likelihood estimation is finding the optimal parameter of a distribution that best describes a set of data. likelihood can be written in the form of $l(\theta )={\pi }_{i=1}nf(x_i;\theta )$ and finding the parameter $\theta$ that maximizes the likelihood is the maximum likelihood estimation. the log likelihood can be represented as $\log (l(\theta ))={\sum }_{i=1}n\log (f(x_i;\theta ))$. now we can take a look at how KL divergence relates to maximum likelihood estimation.

$$\begin{gathered} \underset{\hat{\theta }}{\mathrm{argmin}}{d}_{KL}(p\Vert q(x;\theta )) \\ =\underset{\hat{\theta }}{\mathrm{argmin}}{\mathbb{E}}_{x\sim p}[\log \frac{p(x)}{q(x;\hat{\theta })}] \\ =\underset{\hat{\theta }}{\mathrm{argmin}}{\mathbb{E}}_{x\sim p}[\log (p(x))-\log (q(x;\hat{\theta }))] \\ =\underset{\hat{\theta }}{\mathrm{argmin}}{\mathbb{E}}_{x\sim p}[-\log (q(x;\hat{\theta }))] \\ =\underset{\hat{\theta }}{\mathrm{argmin}}-{\mathbb{E}}_{x\sim p}[\log (q(x;\hat{\theta }))] \\ =\underset{\hat{\theta }}{\mathrm{argmax}}{\mathbb{E}}_{x\sim p}[\log (q(x;\hat{\theta }))] \\ =\underset{\hat{\theta }}{\mathrm{argmax}}{\mathbb{E}}_{x\sim p}[\log (q(x;\hat{\theta }))] \\ =\underset{\hat{\theta }}{\mathrm{argmax}}\frac{1}{n}{\sum }_{i=1}^n\log (q(xi;\hat{\theta })) \\ =\underset{\hat{\theta }}{\mathrm{argmax}}{\sum }_{i=1}^n\log (q(xi;\hat{\theta })) \\ =\underset{\hat{\theta }}{\mathrm{argmax}}\log \left({\prod }_{i=1}^{n}q(x_i;\hat{\theta })\right) \end{gathered}$$

here we’ve derived that minimizing KL divergence is essentially the same thing as maximizing log likelihood estimator

KL divergence and cross entropy §

now moving onto the relationship between KL divergence and cross entropy.

entropy is a measure of the level of uncertainty/disorder/unpredictability in a given dataset/system. it’s a metric that quantifies the amount of information in a dataset and it’s often used to evaluate the qualtiy of a model and its ability to make accurate predictions.
⬆️ uncertainty = ⬆️ entropy

the formula for entropy is the following:

$$\begin{gathered} h(x)=-{\sum }_{x}p(x)log(p(x)) \\ ={\sum }_{x}p(x)log(\frac{1}{p(x)}) \end{gathered}$$

then what is cross entropy? cross entropy is another loss function that can be used to quantify the difference between two probability distributions. $h(p,q)=-{\sum }_{i=1}^{n}p(x_i)log(q(x_i))$ where $p(x_i)$ represents the true probability distribution (usually one-hot) and $q(x_i)$ represents model’s predicted probability distribution. cross entropy is usually used in classification problem to determine how well the model performs. a good classifier would usually have $h(p,q)\approx h(p)$. essentially, minimizing KL divergence is equivalent to minimizing cross entropy under specific conditions (for more take a look at this link)