How do I find the probability mass function of an individual observation for a multinomial logit model?

Question

I know what the probability mass function of a multinomial logit model is. However, I do not know what the probability mass function of an individual observation i, f(yi|xi), is for a multinomial logit model. I would really appreciate if someone could explain how to get the answer and knows the correct answer.

Answer 1

A useful way to view the model is as follows. For each observation $i$, there are $K$ different possible observations (choices). So, there is a random variable $\varepsilon_{i,k}$ for each possible choice. We observe which choice is made and the values of the explanatory variables $\boldsymbol X_i$. Then, for a given distribution of the random variables $\varepsilon_{i,k}$ (given parameter values of the Type-1 EV), we know the conditional probability of picking choice $k$ conditional on $\boldsymbol X_i$.

This is explained in the following part of the relevant Wikipedia article:

It is also possible to formulate multinomial logistic regression as a latent variable model, following the two-way latent variable model described for binary logistic regression. This formulation is common in the theory of discrete choice models, and makes it easier to compare multinomial logistic regression to the related multinomial probit model, as well as to extend it to more complex models.

Imagine that, for each data point $i$ and possible outcome $k$, there is a continuous latent variable $Y_{i,k}^*$ (i.e. an unobserved random variable) that is distributed as follows:

$$\begin{align} Y_{i,1}^{\ast} &= \boldsymbol\beta_1 \cdot \mathbf{X}_i + \varepsilon_1 \, \\ Y_{i,2}^{\ast} &= \boldsymbol\beta_2 \cdot \mathbf{X}_i + \varepsilon_2 \, \\ \cdots & \\ Y_{i,K}^{\ast} &= \boldsymbol\beta_K \cdot \mathbf{X}_i + \varepsilon_K \, \\ \end{align} $$

where $\varepsilon_k \sim \operatorname{EV}_1(0,1),$ i.e. a standard type-1 extreme value distribution.

This latent variable can be thought of as the utility associated with data point $i$ choosing outcome $k$, where there is some randomness in the actual amount of utility obtained, which accounts for other unmodeled factors that go into the choice. The value of the actual variable $Y_i$ is then determined in a non-random fashion from these latent variables (i.e. the randomness has been moved from the observed outcomes into the latent variables), where outcome $k$ is chosen if and only if the associated utility (the value of $Y_{i,k}^{\ast}$) is greater than the utilities of all the other choices, i.e. if the utility associated with outcome $k$ is the maximum of all the utilities. Since the latent variables are continuous, the probability of two having exactly the same value is 0, so we ignore the scenario. That is:

\begin{align} \Pr(Y_i = 1) &= \Pr(Y_{i,1}^{\ast} > Y_{i,2}^{\ast} \text{ and } Y_{i,1}^{\ast} > Y_{i,3}^{\ast}\text{ and } \cdots \text{ and } Y_{i,1}^{\ast} > Y_{i,K}^{\ast}) \\ \Pr(Y_i = 2) &= \Pr(Y_{i,2}^{\ast} > Y_{i,1}^{\ast} \text{ and } Y_{i,2}^{\ast} > Y_{i,3}^{\ast}\text{ and } \cdots \text{ and } Y_{i,2}^{\ast} > Y_{i,K}^{\ast}) \\ \cdots & \\ \Pr(Y_i = K) &= \Pr(Y_{i,K}^{\ast} > Y_{i,1}^{\ast} \text{ and } Y_{i,K}^{\ast} > Y_{i,2}^{\ast}\text{ and } \cdots \text{ and } Y_{i,K}^{\ast} > Y_{i,K-1}^{\ast}) \\ \end{align}

Or equivalently:

\begin{align} \Pr(Y_i = 1) &= \Pr(\max(Y_{i,1}^{\ast},Y_{i,2}^{\ast},\ldots,Y_{i,K}^{\ast})=Y_{i,1}^{\ast}) \\ \Pr(Y_i = 2) &= \Pr(\max(Y_{i,1}^{\ast},Y_{i,2}^{\ast},\ldots,Y_{i,K}^{\ast})=Y_{i,2}^{\ast}) \\ \cdots & \\ \Pr(Y_i = K) &= \Pr(\max(Y_{i,1}^{\ast},Y_{i,2}^{\ast},\ldots,Y_{i,K}^{\ast})=Y_{i,K}^{\ast}) \\ \end{align}

Edit: Sometimes you'll see notation like this: $$ Pr( i \rightarrow k) = \Pr(\max(Y_{i,1}^{\ast},Y_{i,2}^{\ast},\ldots,Y_{i,K}^{\ast})=Y_{i,k}^{*}), $$ where $i \rightarrow k$ means that individual $i$ chooses choice $k$, is fairly concise once the appropriate notation is defined. If you're looking for a useful way to put this into a likelihood function, you might see something like this: $$ P_i = \prod_{k=1}^K Pr( i \rightarrow k)^{\mathbb 1_{i \rightarrow k}}, $$ where $\mathbb 1_{i \rightarrow k}$ is an indicator function that is equal to 1 when $i$ chooses choice $k$ and zero otherwise and $P_i$ is the PMF of the observation associated with individual $i$.

Edit 2:

For an explicit formula, see the same Wikipedia article. Check out, for instance, the section with the following:

As a result, it is conventional to set $C = -\boldsymbol\beta_K$ (or alternatively, one of the other coefficient vectors). Essentially, we set the constant so that one of the vectors becomes 0, and all of the other vectors get transformed into the difference between those vectors and the vector we chose. This is equivalent to "pivoting" around one of the $K$ choices, and examining how much better or worse all of the other $K-1$ choices are, relative to the choice are pivoting around. Mathematically, we transform the coefficients as follows:

\begin{align} \boldsymbol\beta'_1 &= \boldsymbol\beta_1 - \boldsymbol\beta_K \\ \cdots & \cdots \\ \boldsymbol\beta'_{K-1} &= \boldsymbol\beta_{K-1} - \boldsymbol\beta_K \\ \boldsymbol\beta'_K &= 0 \end{align}

This leads to the following equations:

\begin{align} \Pr(Y_i=1) &= \frac{e^{\boldsymbol\beta'_1 \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta'_k \cdot \mathbf{X}_i}} \, \\ \cdots & \cdots \\ \Pr(Y_i=K-1) &= \frac{e^{\boldsymbol\beta'_{K-1} \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta'_k \cdot \mathbf{X}_i}} \, \\ \Pr(Y_i=K) &= \frac{1}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta'_k \cdot \mathbf{X}_i}} \, \\ \end{align}