I'd like to read a more detailed/mathematical analysis on the subject of Coase's theorem. Could you perhaps suggest any advanced textbook/paper on the matter?
The Public Choice III by D. C. Mueller has whole chapter with mathematical exposition of the Coase theorem. His exposition includes both the general case and practical examples. Also I think Mueller strikes very good balance between rigor and simplicity.
You will also find some treatment of the theorem from Game Theory perspective in Game Theory: An Introduction by Steven Tadelis or the Tirol's Game Theory but only relatively briefly when compared to Public Choice III. Also Tirol's The Theory of Industrial Organization has some treatment but again relatively short. Nonetheless despite that these other texts only deal with the theorem briefly expositions of the theorem in different situations which itself has wide application may help you.
If you would prefer some paper then for example you can see application and exposition of Coase Theorem here, but papers are rarely written with didactic motive in mind so if you are self-studying I would recommend the textbook sources above.