The "trick" is to identify an equilibrium action where you know (or can pin-down) the payoff. You can then impose indifference between this action's known payoff and the payoffs from other actions (which will depend on the mixing probabilities) to determine those equilibrium mixing probabilities.
A nice example can be found in a Varian's "A Model of Sales". Let me describe a simple version of that model before discussing what it teaches us about finding continuous mixed equilibria.
Set up
Two firms each set a price, $p_i$, for a homogeneous product they product at zero cost. There is a continuum of consumers, each willing to pay up to $v$ for the product.
A fraction $a$ of the consumers (call them "shoppers") will buy from whichever firm sets the lowest price (breaking ties randomly). A fraction $\frac{1-a}{2}$ of the consumers (call them "1's captives") always buy from firm $1$ (as long as $p_1<v$). The final $\frac{1-a}{2}$ of the consumers (call them "2's captives") always buy from firm $2$ (as long as $p_2<v$).
You should be able to quickly check that this game has no pure strategy equilibrium (think about why we can't have an equilibrium with $0<p_1<p_2$, $0<p_1=p_2$, or $0=p_1=p_2$).
Thus, we look for a mixed strategy equilibrium where prices are drawn from the distribution $F$. Let the support of $F$ (i.e., the range of prices over which firms mix) be $[\underline{p},\overline{p}]$.
No firm will ever choose $p_i>v$ because it then serves no consumers, but could earn a profit of $v(1-a)/2$ by serving its captives at a price of $v$. Thus, $\overline{p}\leq v$.
A firm's profit if it sets price $p$ is
$$\frac{1-a}{2}p+ap[1-F(p)].$$
The first term is the profit from captives (who the firm serves for sure). The second term is the profit from shoppers, who are served only if the rival firm sets a higher prices (which happens with probability $1-F(p)$).
Finding the mixed-strategy equilibrium
We know that if the firm is to mix over prices $p$ and $\overline{p}$ then it must be indifferent:
$$\frac{1-a}{2}p+ap[1-F(p)]=\frac{1-a}{2}\overline{p}+a\overline{p}[1-F(\overline{p})].$$
Notice that if we could eliminate the $\overline{p}$ terms from the right-hand side then we would have an equation that could be solved for the equilibrium strategy, $F$.
The next part is the key to being able to find the equilibrium, so I'll put it in a box:
Let's think about what happens when a firm sets $p_i=\overline{p}$. Because $\overline{p}$ is the highest price ever charged in equilibrium, we know that the firm never serves the shoppers at that price (i.e., $F(\overline{p})=1$). Moreover, since the firm serves only its captives who will buy at any price below $v$, it must be the case that $\overline{p}=v$.
We therefore know that:
- $\overline{p}$ is in the support of $F$, meaning firms must be indifferent between $\overline{p}$ and any other price
- $\overline{p}$ results in profit $\frac{1-a}{2}\overline{p}$
- $\overline{p}=v.$
Putting these facts together with the indifference condition, we have
$$\frac{1-a}{2}p+ap[1-F(p)]=\frac{1-a}{2}v,$$
which implies
$$F(p)=1-\frac{\frac{1-a}{2}(v-p)}{ap},$$
which is our mixed strategy equilibrium. The only part of the equilibrium we haven't solved is the value of $\underline{p}$. But this is easily found by solving $F(\underline{p})=0$ for $\underline{p}$ now that we know $F(\cdot)$.
Final remarks
The trick used in the above example is pretty useful more generally.
Unless the equilibrium strategy, $F$, has an atom, we know by definition that $F(\underline{p})=0$ and $F(\overline{p})=1$. Thus, if we can use economic reasoning to determine the value of either $\overline{p}$ or $\underline{p}$ (as in the box above) then we can eliminate all of the terms that depend on $\overline{p}$ (or $\underline{p}$) on the right hand side of our indifference equation and then solve for $F$.