How do I begin to approach this dynamic discrete choice model?

Question

I'm working through an old problem set (that sadly I don't have solutions for) and I got stuck. It is a dynamic model of entrepreneurship and invention. I'm looking for guidance on this model as well as references or papers that discuss it. Here's the model.

Every period you can choose to start a business. The choice in period $t$ is $b_t \in \{0, 1\}$, where $b_t=1$ is trying to invent something, so it's a discrete choice problem. You have some skill parameter $p$. In every period that you invent, your invention ``succeeds'' with probability $p$ and you get $v=1$. Otherwise it fails and you get $v=0$.

If you don't invent in period $t$ you work and get some wage $w$. The person has linear lifetime utility like this

\begin{equation} \sum_{t=0}^\infty \beta^t [b_t I\{\text{successful invention}\} + (1 - b_t)w] \end{equation}

$I$ is the indicator function. The problem is we (and the agent) don't know $p$. They just know its distributed with a beta distribution with parameters $a$ and $b$ and they have to learn about it over time. I know how to update the prior depending on if the invention succeeds or not (if the person chose to invent in period $t$).

The problem set asks me to set up the Bellman equation and use value function iteration to solve the problem numerically, but that's where I got stuck. Can someone give me a push in the right direction for how to start?

---

The general Bellman equation is something like this

\begin{equation} V(b) = \max_{b'} (u(b) + \beta E V(b')) \end{equation} $b$ is the agent's choice in the current period and $b'$ is the agents choice in the next period. I can't figure out how to incorporate the Bayesian updating of the prior into the expectation, though, since the value of the prior going into period $t$ and thus the value of the posterior at the end of that period depends on the history of successful or failed inventions.

For ex. the person starts out with the prior on $p$ as just the mean of the beta distribution, which is $a/(a+b)$, but if they choose to invent in the next period and are successful, the beta distribution updates to the posterior that has mean $(a+1)/(a+1 + b)$. Etc.

Answer 1

$b_t$ is the decision variable, and the value function is formulated as a function of the state variable of the problem. In your case, the role of the state variable is played by the $p$ parameter, which may change in each period (updated). So your Bellman equation is

$$V(p_t) = \max_{b_t} \{u(b_t) + \beta E [V(p_{t+1})\mid t]\}$$

and I guess now you see where the updating of $p$ enters the picture.

On "Value function iteration", or "iteration on the Bellman equation", the book "Recursive Macroeconomic Theory" of Ljungqvist & Sargent contains clear and practical guidance.