It's not about riches.
According to the CIA World Fact Book, World Gross Domestic Product per person in Purchasing-Power-Parity USD was estimated at 17,500 in 2017. And it was only 7,200 PPP USD in 2000.
Now assume away all inequality. That would mean that a family of four, say two adults and two children, would have 70.000 USD living in USA... sounds a lot? No, if one realizes that this figure is NOT about discretionary disposable income, but income out of which this family should pay also for all the infrastructures that structure our lives -law, security, transportation, education, public and private health, scientific research, etc.
In other words, two hundred and thirty years after the Industrial Revolution we are still far away from the "affluent society" (viewing humanity as a whole), proving the grim axiom of economics that resources are always scarce, relative to what (most) humans have in their minds and imagination.
From a psychological point of view, growth was always important because it represents an "escape to the the future", compared to the nasty realities of distribution and inequality, which are a "what about today?" issue. "Look how to make the future brilliant" would be the schematic answer of growth theory. Growth is also viewed as an answer to social stability in the face of inequality: at least it gives the opportunity that all will improve their material well-being, even if to different degrees.
From a history-of-science angle, economics largely ignored distribution and inequality in order to become as much of a "positive" social science as possible -avoiding the hot and often bloody fights we fight over distribution of income and wealth.
Only relatively recently pragmatic models started linking inequality to reduced growth -and again, without arguing that inequality is morally/philosophically/ideologically bad, but that, even if it is fully "justifiable", it is "bad for growth". Ah, growth again.