This (paraphrase) appears in Book III, Chapter IV of The Wealth of Nations. Smith is discussing how, in his opinion, towns engaged in commerce and manufacturing have contributed to the improvement and cultivation of their country.
Specifically, the quote appears in the context of a discussion about feudalism, and kings being unable to control the violence of feudal lords on page 448 (line breaks and emphasis mine for clarity):
But what all the violence of the feudal institutions could never have
effected, the silent and insensible operation of foreign commerce and
manufactures gradually brought about.
These gradually furnished the
great proprietors with something for which they could exchange the
whole surplus produce of their lands, and which they could consume
themselves, without sharing it either with tenants or retainers. All
for ourselves, and nothing for other people, seems, in every age of
the world, to have been the vile maxim of the masters of mankind.
Smith continues:
As
soon, therefore, as they could find a method of consuming the whole
value of their rents themselves, they had no disposition to share them
with any other persons. For a pair of diamond buckles, perhaps, or for
something as frivolous and useless, they exchanged the maintenance,
or, what is the same thing, the price of the maintenance of 1000 men
for a year, and with it the whole weight and authority which it could
give them. The buckles, however, were to be all their own, and no
other human creature was to have any share of them; whereas, in the
more ancient method of expense, they must have shared with at least
1000 people.
With the judges that were to determine the preference,
this difference was perfectly decisive; and thus, for the
gratification of the most childish, the meanest, and the most sordid
of all vanities they gradually bartered their whole power and
authority.
So, Smith seems to be making a greater historical observation about civilization while specifically discussing the concentration of power, greed, and wealth as it was in feudal society. Hopefully this helps explain the context of the observation a bit.