When it comes to economic growth there are not many reasons. As already pointed out in the question, increase in working age population would lead to increase in economic growth (see Romer Advanced Macroeconomics Ch 1). As pointed by Jasper in his comments that does not automatically mean it leads to higher per capita economic growth, but actually under endogenous growth theory it does (again see Romer Advanced Economics Ch 3). Endogenous growth theory is not 100% accepted as it is somewhat difficult to test but even endogenous theory aside Clemens (2011) even estimates that free migration could increase world output by factor of 0.5-1.5. So even if we take the mean estimate of 1 it would effectively double world GDP. This is simply because productivity of the same person differs by country. The same unskilled worker from Yemen will be more productive in the US than in Yemen just because US has more capital, better institutions, infrastructure and so on. Generally all major studies report overall positive benefits even if exact estimates vary (see Hamilton and Whalley 1984; Moses and letnes 2004, 2005; Klein and Ventura 2009; Kennan 2013 and Delogu et al 2018).
However, there are some (public/political) economic arguments arguments one could levy against completely unrestricted mass migration:
- Inequality arguments
Migration can affect both income and wealth inequality. Starting with income inequality, influx of new migrants has effect on local wages. If immigrant with complementary skill enters it will have a positive effect on local wage (e.g. if you are scientist and thanks to immigration you can hire cheap gardener you have more time to focus on research and earn better pay) but in the industry in which migrants enter increase in labor supply can temporarily suppress wages. Borjas and Katz 2007, find that in the US Mexican immigration between 1980-2000 decreased wages of US native dropouts by 4.8% but increased wages of high school graduates by 1.2% and people with some college education by additional 0.7%. Even though the overall effect is positive (most people in US have at least high school diploma), it will have negative distributional effects on US native high school drop outs.
Now, in a (very) long term the effects of migration should be positive across the board because of dynamic and general equilibrium effects but even if overall all wages would rise they might not rise at the same proportion. If you have strong preference for equality then you might prefer scenario A where poor people earn \$100 and rich people \$1000, to scenario B where poor people earn \$200 and rich \$10,000. Economics, cannot tell you which scenario is better, but given the research cited above in long run this is more or less analogue to the the sort of choice one faces when it comes to migration.
Next, immigration puts pressure on housing prices, the estimates show that 1% increase in immigration rises housing prices by about 1% (see especially Saiz 2007 but also Ottaviano and Peri 2006 and Gonzales and Ortega 2013), there is some limited evidence that low skilled immigration may lower housing prices as it prompts exit of wealthier natives from area (who presumably prefer to live in communities with other wealthier people e.g. see Saiz and Wachter 2011 and Sa 2015). However, on net studies show positive effect on house prices. Consequently, immigration will have positive effect on wealth of home owners and this can very well exacerbate wealth inequality too. Again economics itself cannot tell you whether wealth inequality is bad or not, but the the extent you care about such issue this would be an economic argument against immigration.
However, it is extremely important to note the above holds when we discus inequality on national level. The biggest economic inequalities are on international level (for example even middle class Americans are globally in the top 1% of income distribution, and even poor Americans in the global top 20%-10%). Similarly in order to be in top 10% of wealth institution you just need to own net assets (assets in excess of any debt) above \$71,560 (see the Economist explainer on global inequality patterns), which is doable for most (even lower) middle class people in the West. Consequently while immigration would worse national wealth and income inequality, it would almost certainly improve the global wealth and income inequality, since as mentioned in the first paragraph those immigrants will generally be much more productive in host country than in their countries of origin.
- Fiscal Arguments
Next economic argument can be made on fiscal reasons. Economy is not the same as government finances and government finances are not the same as economy. In principle, something can have positive effect on economy while having negative fiscal effect (e.g. government spending with heavy borrowing erodes governments fiscal position but it helps economy recover from recessions).
Now the fiscal effects on immigration depend crucially on country's policy, so one cannot just make blanket statements here. Government can always adopt policy where immigration won't have direct fiscal consequences, lets say establishing some two tier system where only native born are eligible for any fiscal spending (although indirect ones are harder to avoid). So these will differ widely country by country and year by year.
To take USA as an one example Blau and Mackie 2017 p343 estimate that under optimistic scenarios migrants are net fiscal benefit, but under pessimistic scenarios there would be net fiscal burden. The fiscal effect is most likely most negative for immigration of people who are not working age, like children and elderly.
- Precautionary Principle
Even though major studies show positive net effects one can evoke precautionary principle. The studies measure generally benefit of migration based on data from our current situation where immigration is quite restricted. Just because there is a clear benefit of more migration on economy when we move from our current state to more migration, that does not mean we can extrapolate these results to completely unrestricted migration. Depending, on how you qualify mass migration (is it our current level or something several orders of magnitude higher?), you could invoke precautionary principle. Note this is not good argument for not increasing migration from our current level, as the studies mentioned in the first paragraph clearly support the argument that more migration will have incredible economic benefits, it would be a reasonable argument against going all the way to the extreme of having completely borderless world with as much migration as people want.
- Social Trust & Diversity Issue
Note here when talking about diversity, nor I or research cited herein does necessarily mean ethnic or racial diversity, but rather diversity of values, ideology or cultures (even if the ethnic diversity can sometimes be used as proxy for diversity of values etc).
There is some research that suggest that having too much diversity in terms of ideology, values and culture can erode social trust, and in turn social trust has economic benefits (e.g. see Berggen et al 2008; Roth 2009; Bjornskov 2007). It is well documented that poorer countries often have low levels of social trust (e.g. see Algan and Cahuc 2014 or Ortiz-Ospina and Roser 2017) although causality is not easy to disentangle.
This is not an argument about immigration per se but rather about limiting immigration of people who are too much ideologically or culturally distant native population, given the assimilation rate, since migrants typically assimilate to local values & culture sooner or later. Indeed studies show that typically assimilation rate is quick (e.g. see Dinesen 2012; Dinesen an Hooghe 2010), although many studies focus on US and western liberal democracies not every country might be such good 'melting pot' as these. But depending on your definition of mass immigration it would be an argument against having completely unrestricted cross border migration.
However, an important caveat here is that estimates of effect of social trust on economic outcomes, although significant, are economically not too large and some studies also show that having too much social trust might be issue as well, so rather there might a sweet spot a country would like to hit. In addition, trust is not affected just by immigration Balliet et al (2018) show that for example in the US political polarization of Democrats and Republicans is associated with high levels of distrust, so while immigration can contribute to this issue it is not necessarily even the most important cause of it (studies show that immigrants tend to lean more democrat than republican but only very slightly so see Nowrasteh and Wilson 2017).
Final caveat:
While the above economic arguments against immigration have merit, you should be aware of the fact that general consensus is that economic benefits of increasing migration (from current level) in most of the developed world outweigh the economic cost if we look at it narrowly from looking at size of output and economic growth (e.g. points 2 and 4 are serious counterarguments but even in pessimistic scenarios they would likely not create that much cost to outweigh large positive benefits). In order for this conclusion to change one has to include additional arguments for disutility of inequality, and in addition argue that only welfare of natives counts or that welfare of natives counts more (since as soon as we include welfare of migrants themselves in the calculation the benefits are enormous) and so on, which is still legitimate since in economics we care about welfare not just money or output, but at the same time these issues are contentious as they go well beyond economics and are rooted in different moral and political philosophies.
