In the given scenario, the optimal social outcome from a Coasian bargain is not in general the same as the optimal social outcome from a Pigouvian tax, although there is a special case in which this is so.
The scenario outlined in this question has several features that make it rather more complicated than some textbook discussions of policy towards negative externalities.
a) It focuses not on benefits and costs but on utility functions (cf (A)).
b) It considers optimality, defined as a state in which the sum of the utility of the two parties is maximised, rather than efficiency (a state in which the utility of one party cannot be increased without lowering that of the other, also known as Pareto-optimality) (cf (B)).
c) No direct link is specified between utility and any monetary quantity (cf (C)). Payment or receipt of money by either party affects utility indirectly via the quantity each can afford of a second good not subject to an externality.
d) It uses general utility functions with no restrictions on their functional form.
e) Although both parties can consume the second good, as a consequence of (d) above a given quantity of that good does not necessarily affect each party’s utility in the same way. It is possible therefore that a monetary transfer between the parties (whether via tax revenue paid on to the other party, or a Coasian bargain) could raise or lower the sum of utility not for reasons relating to the externality but simply by transferring spending power to or from the party which gains more utility from the same quantity of the good.
This answer is concerned, as the question asks, with social optima. Whether a government is likely to set a tax at a rate that yields a social optimum, and whether Coasian bargaining is likely to lead to an agreement that yields a social optimum, are further questions which I do not address. I make one simplifying assumption not in the question, namely, that the utility functions are additively separable, so that cross-partial derivatives can be ignored.
Case 1: $a$’s consumption of $y$ taxed at a rate $t$, the use of the tax revenue being considered outside the scope of the optimisation
This is the case considered in the first half of the question. The analysis of $a$’s optimisation is correct, but this is how I would specify the social optimisation:
$\max\limits_{x_a,y_a,t} \ u_a(x_a,y_a)+u_b(x_b,y_b)$
subject to $\ p_xx_a+(p_y+t)y_a=e_a\qquad(1)$
and $\ \dfrac{\partial u_a}{\partial y_a}=\dfrac{p_y+t}{p_x}\dfrac{\partial u_a}{\partial x_a}\qquad(2)$
The maximisation does not need to be over $x_b$ and $b$’s income constraint $p_xx_b=e_b$ need not be included since, with no tax or other payment to $b$, and no consumption of $y$ by $b$, $x_b$ is fixed at $e_b/p_x$. But it does need to be over $t$, and $t$ needs to be included in $a$’s income constraint (1), since the tax has both an income and a substitution effect on $a$, with consequences for both parties’ utility. The first-order condition (2) from $a$’s optimisation also needs to be included as a constraint: we want to maximise welfare only over those combinations of the goods that $a$ would actually choose in response to given tax rates. Also, $x_a$, $y_a$ and $t$ must be non-negative.
The Lagrangian is then:
$L=u_a+u_b+\lambda_1[p_xx_a+(p_y+t)y_a-e_a]+\lambda_2\Bigg[\dfrac{\partial u_a}{\partial y_a}-\dfrac{p_y+t}{p_x}\dfrac{\partial u_a}{\partial x_a}\Bigg]$
This leads to first-order conditions:
$\dfrac{\partial L}{\partial x_a}=\dfrac{\partial u_a}{\partial x_a}+\lambda_1p_x-\lambda_2\dfrac{p_y+t}{p_x}\dfrac{\partial^2u_a}{\partial x_a^2}=0\qquad(3)$
$\dfrac{\partial L}{\partial y_a}=\dfrac{\partial u_a}{\partial y_a}+\dfrac{\partial u_b}{\partial y_a}+\lambda_1(p_y+t)+\lambda_2\dfrac{\partial^2u_a}{\partial y_a^2}=0\qquad(4)$
$\dfrac{\partial L}{\partial t}=\lambda_1y_a-\dfrac{\lambda_2}{p_x}\dfrac{\partial u_a}{\partial x_a}=0\qquad(5)$
For this case only I will outline a method of solution. Using (3) to (5) it is possible to eliminate $\lambda_1$ and $\lambda_2$ and obtain the relation:
$(p_y+t)\Bigg[y_a\dfrac{\partial^2u_a}{\partial x_a^2}\bigg(\dfrac{\partial u_a}{\partial y_a}+\dfrac{\partial u_b}{\partial y_a}\bigg)+\bigg(\dfrac{\partial u_a}{\partial x_a}\bigg)^2\Bigg]=p_x \dfrac{\partial u_a}{\partial x_a}\Bigg[\bigg(\dfrac{\partial u_a}{\partial y_a}+\dfrac{\partial u_b}{\partial y_a}\bigg)-y_a\dfrac{\partial^2u_a}{\partial y_a^2}\Bigg]\qquad(6)$
This can be rearranged as a formula for $t$ (note the difference from the formula in the question):
$t=\dfrac{p_x\frac{\partial u_a}{\partial x_a}\Big[\big(\frac{\partial u_a}{\partial y_a}+\frac{\partial u_b}{\partial y_a}\big)- y_a\frac{\partial^2u_a}{\partial y_a^2}\Big]}{y_a\frac{\partial^2u_a}{\partial x_a^2}\big(\frac{\partial u_a}{\partial y_a}+\frac{\partial u_b}{\partial y_a}\big)+\big(\frac{\partial u_a}{\partial x_a}\big)^2}-p_y\qquad(7)$
From (2) we also have:
$t=\dfrac{p_x\frac{\partial u_a}{\partial y_a}}{\frac{\partial u_a}{\partial x_a}}-p_y\qquad(8)$
Setting the fractions in (7) and (8) equal, as they must be, and rearranging, we can infer:
$y_a=\dfrac{\big(\frac{\partial u_a}{\partial x_a}\big)^2\frac{\partial u_b}{\partial y_a}}{\frac{\partial u_a}{\partial y_a}\frac{\partial^2u_a}{\partial x_a^2}\big(\frac{\partial u_a}{\partial y_a}+\frac{\partial u_b}{\partial y_a}\big)+\big(\frac{\partial u_a}{\partial x_a}\big)^2\frac{\partial^2u_a}{\partial y_a^2}}\qquad(9)$
At this level of generality it’s hard to make further progress, but clearly if the utility functions were specified and we could express the various derivatives as functions of $x_a$ and $y_a$, then (9) would yield an equation in $x_a$ and $y_a$. Together with (1) and (7) this would give three equations which could be solved for the three variables $x_a$, $y_a$ and $t$. All of the above of course is subject to the caveat that these variables must be non-negative.
Case 2: $a$’s consumption of $y$ taxed at rate $t$, the tax revenue being paid on by the government to $b$
In this case $x_b$ depends on how much tax revenue $b$ receives, so we need to optimise over $x_b$ as well as $x_a$ and $y_a$, and include $b$’s income constraint. So the social optimisation should be specified as:
$\max\limits_{x_a,y_a,t,x_b}u_a+u_b$
subject to $\ p_xx_a+(p_y+t)y_a=e_a\qquad(10)$
and $\ \dfrac{\partial u_a}{\partial y_a}=\dfrac{p_y+t}{p_x}\dfrac{\partial u_a}{\partial x_a}\qquad(11)$
and $\ p_xx_b=e_b+ty_a\qquad(12)$
The Lagrangian is:
$L=u_a+u_b+\lambda_1[p_xx_a+(p_y+t)y_a-e_a]+\lambda_2\Bigg[\dfrac{\partial u_a}{\partial y_a}-\dfrac{p_y+t}{p_x}\dfrac{\partial u_a}{\partial x_a}\Bigg]+\lambda_3[p_xx_b-e_b-ty_a]$
This leads to first-order conditions:
$\dfrac{\partial L}{\partial x_a}=\dfrac{\partial u_a}{\partial x_a}+\lambda_1p_x-\lambda_2\dfrac{p_y+t}{p_x}\dfrac{\partial^2u_a}{\partial x_a^2}=0\qquad(13)$
$\dfrac{\partial L}{\partial y_a}=\dfrac{\partial u_a}{\partial y_a}+\dfrac{\partial u_b}{\partial y_a}+\lambda_1(p_y+t)+\lambda_2\dfrac{\partial^2u_a}{\partial y_a^2}-\lambda_3t=0\qquad(14)$
$\dfrac{\partial L}{\partial t}=\lambda_1y_a-\dfrac{\lambda_2}{p_x}\dfrac{\partial u_a}{\partial x_a}-\lambda_3y_a=0\qquad(15)$
$\dfrac{\partial L}{\partial x_b}=\dfrac{\partial u_b}{\partial x_b}+\lambda_3x_b=0\qquad(16)$
Using these four equations it should be possible to eliminate $\lambda_1$, $\lambda_2$ and $\lambda_3$ and then, using (11) and (12) and if the utility functions are specified, solve for $x_a$, $y_a$, $t$ and $x_b$.
Case 3: No tax, a Coasian bargain in the context that $b$ is the right-holder and the parties agree to negotiate only on the basis of a flat rate $p_z$ per unit of $y$ consumed by $a$
If the parties agree to negotiate only on the basis of a flat rate $p_z$ per unit of $y$, the optimal rate will be obtainable exactly as for Case 2, albeit replacing $t$ by $p_z$ throughout. Why? Because for any $y_a$, transfer of the sum $p_zy_a=ty_a$ from $a$ to $b$ will have exactly the same effect on the behaviour and therefore the utility of each party regardless of whether it is a tax collected from $a$ by the governemnt and then paid to $b$, or a direct payment from $a$ to $b$.
Note what is not being said here. It is not claimed that the social optimum that can be achieved via any Coasian bargain on any basis the parties may choose is the same as for Case 2. Such a claim would be false, as is shown in considering Case 4 below.
Case 4: No tax, a Coasian bargain in the context that $b$ is the right-holder and the parties agree to negotiate on the basis of a lump sum payment for consumption by $a$ of specific quantity of $y$.
This is one example of a bargain not involving a flat rate charge per unit of $y$ (others are possible, eg an increasing rate per unit, but will not be considered here).
It is convenient to illustrate this case by a numerical example. Suppose that: $u_a= \ln(x_a+1)+\ln(y_a+1)$; $\ u_b=\ln(x_b+1)-0.1y_a^2$; $\ p_x=p_y=1$; $\ e_a=e_b=4$. Suppose further that the parties agree that $a$ will consume $y_a=0.9$ in return for payment to $b$ of $0.4$. Because the quantity $y_a$ is specified in the agreement, there is no opportunity for $a$ to optimise within the scope of the agreement, other than by spending on $x_a$ all the funds not required to pay the lump sum and to purchase the $0.9$ $y_a$.
The quantities of $x$ consumed by each party will then be:
$x_a=4-0.4-0.9=2.7\qquad(17)$
$x_b=4+0.4=4.4\qquad(18)$
and social utility will be:
$u_a+u_b=\ln(3.7)+\ln(1.9)+\ln(5.4)-0.1(0.9^2)\approx1.308+0.642+1.686-0.081=3.555\quad(19)$
I shall show that that is more social utility than can be obtained via any flat rate charge. This may seem surprising since the lump sum payment of $0.4$ for consumption of $0.9$ can be viewed as equivalent to a per unit charge $p_z$ of $0.4/0.9\approx 0.444$. The point however is that once the parties agree to the lump sum, $a$ is committed to consuming $0.9$ of $y$, whereas if the parties agree to a rate per unit, $a$ can still choose to consume the quantity of $y$ that will maximise $a$’s utility given that rate. In fact, substituting into (1) and (2) with $p_z$ replacing $t$ and $p_z=0.444$ we have:
$x_a+1.444y_a=4\qquad(20)$
$\dfrac{1}{y_a+1}=\dfrac{1.444}{x_a+1}=\dfrac{1.444}{5-1.444y_a}\qquad(21)$
from which we can infer:
$5-1.444y_a=1.444y_a+1.444\qquad(22)$
$3.556=2.888y_a$ and so $\ y_a\approx 1.231\qquad(23)$
In response to a flat rate of $0.444$, therefore, $a$ will choose to consume not $0.9$ but $1.231$ of $y$. Furthermore, we can infer:
$x_a=4-(1+0.444)(1.231)=2.222\qquad(24)$
$x_b=4+0.444(1.231)=4.547\qquad(25)$
$u_a+u_b=\ln(3.222)+\ln(2.231)+\ln(5.547)-0.1(1.231^2)\approx1.170+0.802+1.713-0.152=3.533\qquad(26)$
Comparing (26) with (19), social utility is somewhat less with the flat rate.
Making a similar calculation for various values of $p_z$ I obtained the results in the table below.
It can be seen that, with a flat rate charge, the social optimum is achieved when the charge $p_z$ is approximately $0.50$, with social utility approximately $3.536$ which is less than the $3.555$ achieved with a lump sum of $0.4$ for consumption of $0.9$ of $y$.
Thus for the specific example illustrated, a lump sum payment for a specific level of consumption of y can yield more social utility than any flat rate. Given the equivalence of Cases 2 and 3, it follows that the socially optimal outcome from Coasian bargaining will not in general equal the socially optimal outcome from a tax paid on to $b$.
References
A. Hanley N, Shogren J F & White B (2nd edn 2007) Environmental Economics in Theory and Practice (pp 45-8 discuss Coasian bargaining in the context of given marginal benefit and marginal cost curves, without mentioning utility).
B. Wikipedia – Coase Theorem (introduces the Coase Theorem in terms of economic efficiency).
C. Perman R, Ma Y, McGilvray & Common M (3rd edn 2003) Natural Resources and Environmental Economics (pp 137-9 discuss Coasian bargaining in the context of marginal benefit and marginal cost curves supported by utility functions which include a monetary variable (wealth)).
