The von Neumann-Morgenstern (vNM) utility function takes the form
\begin{equation}
U(p,x)=\sum_{i=1}^np_iu(x_i)
\end{equation}
where $x=(x_1,\dots,x_n)$ with $x_i$ being the (monetary) payoff associated with outcome $i$ and $p=(p_1,\dots,p_n)$ with $p_i$ being the probability that $i$ occurs.
In behavioral economics, generalizations of the vNM utility usually happen in either (or both) of the following two channels:
Non-linear probability weighting: instead of using $p_i$'s, which can be considered objective probabilities, a model may use decision weights $w_i$'s which map the objective probability distribution into a new distribution.
Reference dependent utility function: instead of $u(x_i)$, which depends only on the payoff in outcome $i$, a model may consider a utility function $u(x_i,r)$ that depends on both $x_i$ and some reference payoff $r$.
Modifications made through either of these will give rise to a non-expected utility function, which is supposed to improve the model's descriptive accuracy of people's decision under risk.
Example of 1: Rank-Dependent Utility
Suppose $x_1<x_2<\cdots<x_n$, and calculate decision weights ($w_i$'s) as follows:
\begin{equation}
w_i=\pi\left(\sum_{j=i}^np_j\right)-\pi\left(\sum_{j=i+1}^np_j\right),
\end{equation}
where $\pi(\cdot)$ is called a probability weighting function (PWF).
A common PWF is the Tversky-Kahneman PWF:
\begin{equation}
\pi(t)=\frac{s^\gamma}{(s^\gamma+(1-s)^\gamma)^{1/\gamma}}.
\end{equation}
Then the rank-dependent utility with T-K PWF is
\begin{equation}
RDU(p,x;\gamma)=\sum_{i=1}^nw_iu(x_i).
\end{equation}
Note that $w_i=p_i$ if $\gamma=1$, so that vNM EU is a special case of RDU.
Example of 2: Disappointment Aversion
\begin{equation}
DAU(p,x;\overline U)=\sum_{i=1}^np_i[u(x_i)-D(u(x_i)-\overline U)],
\end{equation}
where $D(\cdot)$ is a function that captures how disappointment ($u(x_i)<\overline U$) or elation ($u(x_i)>\overline U$) affects an individual's evaluation of a prospect. Here, the reference utility level $\overline U$ can be something like the certainty equivalent level of utility. EU can also be derived as a special case from DAU by requiring that $D(\cdot)$ be a constant function.
Example of 1+2: Prospect Theory
Prospect theory (PT) incorporates both non-linear probability weighting and reference dependence. Again, assume $x_1<\cdots<x_n$. PT distinguishes between gains ($x_i>r$) and losses ($x_i<r$). For gains, decision weights are given by
\begin{equation}
w_i=\pi^+(p_i+\cdots+p_n)-\pi^+(p_{i+1}+\cdots+p_n),
\end{equation}
while for losses the decision weights are
\begin{equation}
w_i=\pi^-(p_i+\cdots+p_n)-\pi^-(p_{i+1}+\cdots+p_n),
\end{equation}
where $\pi^+(\cdot)$ and $\pi^-(\cdot)$ are both PWFs. For instance, one can have both PWFs be the Tversky-Kahneman form, but with different parameters. In addition, the utility function also takes a reference dependent form:
\begin{equation}
u(x_i,r)=
\begin{cases}
(x_i-r)^\alpha &\text{if }x_i\ge r\\
-\lambda(r-x_i)^\beta &\text{if }x_i<r
\end{cases}
\end{equation}
where $\alpha,\beta\in(0,1)$ are parameters of risk aversion (differentiated between gains and losses) and $\lambda>0$ is the coefficient of loss aversion.
Thus, the non-expected utility function under PT is
\begin{equation}
V(p,x;\text{parameters})=\sum_{i=1}^nw_iu(x_i,r)
\end{equation}
