tl;dr
Your description of an invisible hand is a straw man, and academic literature typically avoids straw man ideas.
When it comes to proper non-strawman version of the invisible hand argument there are several interpretations of the invisible hand in the literature as describing:
- First Fundamental Theorem of Welfare Economics. For derivation and description of the Arrow (1951) and Debreu (1951)
- Spontaneous beneficial order that is a result of unintended individual action - this is covered by cornucopia of various models that have this property. Even Ricardian model of trade would technically apply.
- There could be other interpretations such as just making relatively vague argument that the economic growth raises income of the poor/economics is generally not zero-sum game.
Full Answer
Correction of misconceptions in the question:
Let me start by correcting some misconceptions
The idea of the invisible hand could be expressed as: in a competitive market, agents acting selfishly will make choices that benefit others.
That is not an expression of invisible hand idea, it is a straw man of invisible hand. Academic literature, at least the high quality portion of it, does not examine straw men of ideas (e.g. you would not expect serious climate paper to address straw men like "if there is a climate change how is it possible there is a snow in a winter?").
First, Adam Smith used the metaphor invisible hand very sparingly, only 3 times in the whole corpus of his writing and only once in Wealth of Nations (WoN), which is the most relevant, where he states that:
But the annual revenue of every society is always precisely equal to the exchangeable value of the whole annual produce of its industry, or rather is precisely the same thing with that exchangeable value. As every individual, therefore, endeavours as much as he can, both to employ his capital in the support of domestic industry, and so to direct that industry that its produce maybe of the greatest value; every individual necessarily labours to render the annual revenue of the society as great as he can. He generally, indeed, neither intends to promote the public interest, nor knows how much he is promoting it. By preferring the support of domestic to that of foreign industry, he intends only his own security; and by directing that industry in such a manner as its produce may be of the greatest value, he intends only his own gain; and he is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention.
Adam Smith never mentions selfishness here (nor self-interest), but person's own gain, i.e. in modern lingo that person is trying to maximize their own utility. What everyone's greatest gain is depends on their preferences and pursuing charitable activity fully qualifies.
Adam Smith does say also say in other passages of WoN that purely selfish activity can increase public welfare, but in those passages he does it to emphasize that that his claim does hold even if people would be selfish. No fair interpretation of Smith would say that he stated that it is selfish action in particular, that is required here. Rather, any pursuit of one's self-interest (greatest value judged by one's own preferences) would qualify whether it is one that is selfish or not.
Second, Adam Smith never referred to competitive markets (the notion of perfect competition did not even exist at the time of Smith's writing). Adam Smith referred to free market, which is not the same as competitive market. Perfect competition was first developed by Cournot in 1838.
Third, Adam Smith does not make a claim that this necessarily increases welfare of others (there are other passages where he says it often does). In fact, Smith clearly states that because:
annual revenue of every society is always precisely equal to the exchangeable value of the whole annual produce of its industry ... As every individual, therefore, endeavours as much as he can, both to employ his capital in the support of domestic industry, and so to direct that industry that its produce maybe of the greatest value; every individual necessarily labours to render the annual revenue of the society as great as he can
In essence, Smith simply states here that the welfare/revenue of society is sum of individual welfare/revenue, that is $W = \sum_i w_i$, and thus if you maximize your own welfare you will contribute to the maximization of public welfare as well. But there is no requirement for your welfare improvements to also help others (although it could). For example, if one person gains but no-one else looses anything that counts as well as situations where one person gains, but other gain as a result as well.
Research Relating to Invisible Hand
The style of 18$^{\text{th}}$ century academic writing was less precise than we use at the present, and Adam Smith used the phrase "invisible hand" only 2 times in all of his published works, and 3 times if we count all of his unpublished works (Blaug 2008). Hence, there is a bit of ambiguity in the meaning of the term, but we can certainly exclude straw man interpretations such as made in the question.
Next, the following few interpretations are the ones that are most widely accepted are:
- Adam Smith was implicitly describing the first fundamental welfare theorem. This is the claim made by Arrow and Hahn (1971) and MasColell et al. (1995) Microeconomic Theory (pp. 308, 327, 524, 545, 549).
You can see the original work on first fundamental theorems of welfare economics in Arrow (1951) and Debreu (1951).
The above interpretation is popular but a criticism of this interpretation is that it seems to read too much into Smith's work. The first fundamental theorem of welfare economics requires perfect markets, no externalities and other assumptions that Smith was never explicitly discussing (see Berg 2008). Thus some economic historians would rather say that Smith was not describing fundamental welfare theorem.
- According to Vaughn (1987) and Hayek (1973), Adam Smith was describing
the principle by which a beneficent social order emerged as the unintended consequence of individual human action
This principle is much wider that the one suggested in 1, and it is difficult to just recommend single paper. Technically, the interpretation of the first welfare theorem is an example of emergence of beneficial social order, so the same models can be used. The difference between this and the first interpretation is that this second one allows for a much wider set of models and conditions than those of the first theorem of welfare economics. For example, classical Ricardian model of trade can be considered a model of beneficial spontaneous order emerging as an unintended consequence of individual action.
- There could also be some other, less popular interpretations. The statement could be interpreted much more narrowly just as Smith stating that increases in economic output generally increase incomes of the poor as well (Blaug 2008). There are growth models in growth literature that (depending on their parameters) show this. You can see an overview of various growth models in Introduction to Modern Economic Growth by Acemoglu.
