Necessary Caveats When Discussing Slavery:
First before tackling this question it is important to note that this issue is broad and complex. This is because there is no single 'slavery'. For example, slavery under the Roman Empire was not the same as slavery practiced in US south (see Tamin; 2017). Furthermore, even within a given society there might be different parallel forms of slavery. This is especially true if you count as slavery all 'unfree labor' such as indentured servitude or serfdom which were similar to slavery (Russian serfs could even be bought and sold see Kolchin 1990). Hence it is extremely difficult to generalize about an institution that has been practiced differently all across world and time until very recently, and making exhaustive review of all forms of slavery and unfree labor is beyond scope of stack exchange. As a consequence I will mainly focus in my answer on the sort of slavery that existed in the US although I will try to keep thigs as general as possible.
Second, when we discuss whether slavery made an economic sense we have to distinguish between economic sense from point of view of society and point of view of few private individuals. For example, mercantilism is demonstrably not economically beneficial to society and generally tends to make economies poorer than they could be without it, which you can learn from any international economics textbook (see Krugman et al (2018) for example). Yet despite of that you could historically find many merchants who became fabulously rich thanks to mercantilist doctrines. North Korean or Venezuelan institutions also make local elites extremely rich but are arguably disastrous to the general level of economic welfare in their societies. If government would give a company monopoly on apple production it would greatly benefit the owners of said apple company but hurt consumers much more so on net basis it would be not economically desirable to have it.
Slavery from Private Perspective:
From narrow private perspective slavery was profitable. For example, according estimates provided by Conrad & Meyer (1958) the slave owners in antebellum US could expect to see return in excess of $10\%$ during certain time periods. According to Fogel & Engerman (1974) returns of plantation owners using slave labor were comparable to those in manufacturing.
Generally speaking if slavery was not privately profitable for people engaged in the slave business they would not continue to do it on any large scale. Yes even if it would not be profitable some individuals could have kept slaves as a some sort of Veblen good to signal their high status, and I am sure historically some wealthy individuals kept slaves for only this purpose, but on any economy-wide scale slavery would not persist if it would not bring benefits to the slaveholders themselves.
Slavery from Societal Perspective
From a societal point of view the economic literature suggest slavery had more costs than benefits. For example, Anderson & Gallman (1977) argue that slave ownership is detriment to specialization. The reason for this is that if you own slaves you own them whether demand for your goods is high and low. Slaves are essentially a fixed cost for a business because you need to shelter them and feed them even if you dont have any work for them whereas free laborer can be just fired and hired at will. As a response slave owners were often not engaged in just one single line of business (like just picking cotton) but had to also diversify and use their slaves for various tasks.
However, such diversification is detrimental to economic welfare and even economic growth as specialization leads to both higher efficiency and hence allows economies to produce more, but what even more specialization can also have dynamic effects that allow economies to grow more faster (inventions & discoveries are usually made by specialist not by 'jacks of all trades'). Consequently, Anderson & Gallman argue that slavery in long run stifled the economic growth in south and hence been detrimental to southern economies even if for the slave owning elite benefited from such system. Other authors such as Genovese (1976) went even so far as to claiming that slave ownership was detrimental to the development of 'capitalism' as it discouraged development of towns and commercial centers.
Furthermore, generally from a development perspective we recognize two types of institutions (see Acemoglu and Robinson Why Nations Fail);
Inclusive Institutions - “inclusive economic institutions… are those that allow and encourage participation by the great mass of people in economic activities that make the best use of their talents and skills”
Extractive Institutions - are defined as the opposite of the above inclusive institutions (and authors even mention slavery as prime example of extractive institution).
And developmental research shows (again see Acemoglu and Robinson) that extractive institutions are detrimental to economic development. In fact the authors argue that most of the differences between 'rich' and 'poor' countries today can be explained by the fact that countries that are high income today tended to adopt inclusive institutions while the low income countries mostly relied on extractive institutions. Now slavery is only one of myriad of institutions so it is possible that a country would manage to develop even despite of it. Also the contrary holds a country where most institutions are highly extractive but does not have slavery will still not be able to develop itself. Nonetheless, their research suggests that slavery would be detrimental to economic development.
An concrete example of study that looks at unfree labor would be study of Dell (2010) that using regression discontinuity (in order to ensure exogeneity) examines the long‐run impacts of the mita, an extensive forced mining labor system (which is very close to slavery) in effect in Peru and Bolivia between 1573 and 1812. Results show that this system had persistent effect on lowering household consumption by around 25% and increases the prevalence of stunted growth in children by around 6 percentage points in subjected districts today (long after the system was abolished). Furthermore, the study shows this system had also persistent negative effects on other developmental measures such as integration of road networks and that also the regions that used to had mita are less industrialized and focus more on subsistence farming even today.
There are also some studies that highlight that slavery also had some positive economic impacts. For example, Williams (1944) argues that even though slavery was overall detrimental to economies of Americas it helped to develop international financial networks and in turn international banking has first order effect on economic development. Nonetheless, international financial systems would most likely develop thanks to the international trade eventually anyway so the benefits of slave trade were likely just indirect.
Furthermore, all of the above does not even take into account the welfare cost associated with the loss of utility experienced with by the slaves themselves. This utility loss was undoubtedly tremendous and any economic analysis from societal point of view has to take it into consideration.
Lastly, as pointed out by Giskard in his +1 comment, slavery was not abolished just due to economic reasons (and the same can be said about serfdom or myriad of other extractive institutions). Moral and socio-political factors play role as well. For example, top US public economists are generally in favor of well regulated market for kidneys as this poll shows, yet US does not have such market and this is most likely because most voters would consider such market immoral regardless of any general welfare benefits. Consequently, one should not assume that some economic policy exists or ceases to exist solely because it is or ceases to be economically efficient. You should also note that slavery was most prevalent in the past when democracy was not ubiquitous as today, and even countries that had democracy extended franchise only to the land holding elites (which were also most likely the ones to own slaves) and was further restricted based on other ethic or gender characteristics. Thus extension of franchise probably played big role in abolition of slavery and other forms of unfree labor but these are questions for political scientists, historians, sociologist and anthropologists to answer.
