While a lack of infrastructure and public goods may explain poverty in a short-term perspective, it is widely considered that institutions are the key factor in the long term (how could infrastructure and public goods be provided without suitable institutions?).
As well as Acemoglu & Robinson, referred to in FooBar's answer, two important writers on this topic are Hernando de Soto and Douglass North.
De Soto's book The Mystery of Capital argues that people in poor countries are often constrained by the absence of institutions providing effective property rights over capital. Thus the land farmed by a villager may be informally recognised by other villagers as being his land, but his lack of formal property rights over the land means that he cannot use it as security for a loan to buy farm equipment, and cannot sell it if he wants to start a different business or move to a city. As a result his economic options are heavily constrained.
North, in his paper Institutions, emphasizes the importance for economic development of institutions which:
- facilitate trade and specialisation, both within countries and
internationally, by lowering transactions costs between people in
circumstances where the informal constraints of village life do not
apply;
- facilitate mobility of capital, by developing a legal framework for
financial instruments and removing legal obstacles (eg usury laws);
- facilitate the spreading of risk, reducing the consequences of bad
luck for individuals and firms.
While such institutions are largely taken for granted in developed countries, North suggests that their growth is not an inevitable development, and indeed that the historical norm is the absence of such a development (p 98 of Institutions).