Efficiency in general means that the item being traded should be given to the party that values it the most.
In a competitive market, this means trade should continue as long as a consumer's value of a good, as captured by the demand curve, is greater than a seller's value of that good, captured by the supply curve.
In a monopoly, seller's value is captured by its marginal cost curve. But since price is greater than marginal cost in equilibrium, the units for which a consumer's value is higher that MC but lower than equilibrium price don't get traded. Hence there is inefficiency: those units of the good are not in the hands of the party (i.e. the consumers) that values them the most. The story for monopsony is similar.
In a bilateral trade setting with one seller and one buyer, we need to discuss two cases. In the first case, the traders' values are common knowledge. In this case efficiency can be attained, though price depends on the relative bargaining power of the two parties. In the second case, suppose neither trader knows the other party's value, but they do know that gains from trade are possible but not certain. Then the Myerson-Satterthwaite theorem tells us that no bilateral trading mechanism that is individually rational and incentive compatible can guarantee efficiency in the ex post sense; that is, the party with a lower value for the good being traded may end up having the good.