Language Model

$$p(d|q) \propto p(q|d)p(d)$$

$$p(q|d) = \prod_i p(q_i|d)$$

1. 一类是对于在文档中出现的词的模型$p_s(w|d)$
2. 另一个是没有出现在文档中的词的模型$p_u(w|d)$

$$\begin{split}log p(q|d)&= \sum_i log p(q_i|d) \\ &= \sum_{i:c(q_i;d)>0} log p_s(q_i|d) + \sum_{i:c(q_i|d)=0} log p_u(q_i|d) \\ &= \sum_{i:c(q_i;d)>0} log p_s(q_i|d) - \sum_{i:c(q_i;d)>0} log p_u(q_i|d) + \sum_{i:c(q_i;d)>0} log p_u(q_i|d) + \sum_{i:c(q_i|d)=0} log p_u(q_i|d) \\ &= \sum_{i:c(q_i|d)>0} log \frac{p_s(q_i|d)}{p_u(q_i|d)} + \sum_i log p_u(q_i|d) \\ \end{split}$$

$$p_u(q_i|d) = \alpha_d p(q_i|C)$$

$$log p(q|d) = \sum_{i:c(q_i:d)>0} log \frac{p_s(q_i|d)}{\alpha_d p(q_i|C)} + n log \alpha_d + \sum_i log p(q_i|C)$$

Smoothing Methods

$$p_{ml}(w|d) = \frac{w;d}{\sum_w c(w;d)}$$

p(w|d)=\left\{ \begin{aligned} p_s(w|d) & \quad if \quad word \quad w \quad is \quad seen \\ \alpha_d p(w|C) & \quad otherwise\\ \end{aligned} \right.

$$\alpha_d = \frac{1-\sum_{w:c(w:d)>0} p_s(w|d)}{1-\sum_{w:c(w:d)>0}p(w|C)}$$

Jelinek-Mercer

$$p_{\lambda} = (1-\lambda)p_{ml}(w|d) + \lambda p(w|C)$$

Dirichlet

$$p_{\mu}(w|d) = \frac{c(w;d) + \mu p(w|C)}{\sum_w c(w;d)+ \mu}$$

Absolute discount

$$p_{\delta } = \frac{max(c(w:d)-\delta,0)}{\sum_w c(w:d)} + \sigma p(w|C)$$

Jelinek-Mercer $(1-\lambda)p_{ml}(w | d) + \lambda p(w | C)$ $\lambda$ $\lambda$
Dirichlet $\frac{c(w;d) + \mu p(w | C)}{\sum_w c(w;d)+ \mu}$ $\frac{\mu}{\sum_w c(w;d)+\mu}$ $\mu$
Absolute discount $\frac{max(c(w:d)-\delta,0)}{\sum_w c(w:d)} + \frac{\delta | d | _{u}}{ | d |} p(w | C)$ $\frac{\delta | d | _{u}}{ | d |}$ $\delta$

参考

1. Zhai, Chengxiang, and John Lafferty. “A study of smoothing methods for language models applied to ad hoc information retrieval.” Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2001.