Suppose the state raises taxes in order to internalise externalities. Doesn't the consumer have to pay the additional external costs then, because the prices are raised due to the taxes?
If negative externalities are priced into the market via a pigouvian tax, then those responsible for the negative externalities pay.
As a whole, the public is better off. Firstly because the market will now move to a more efficient situation, and the amount of negative externality will decrease (assuming something other than perfect inelasticity). And secondly, it will also benefit in other ways, through improved public services paid for by the taxes; and/or decreased deadweight loss, achieved by lowering non-pigouvian taxes.
Some market participants will be worse off - the polluter will be paying. Exactly how the burden is spread among those responsible, will depend on the specifics of the situation. Consumers, equity-owners, managers, employees, land-owners, in short anyone involved in the relevant polluting supply chain - each of them might bear some, none or all of the burden, and between them, they'll bear it all.