I think by today the arguments you mention are completely outdated. Nowadays, using combination of psychometry and econometrics companies can predict whether you are pregnant (from your shopping patterns) earlier than it is possible for humans to notice the pregnancy (see here). If companies being able to use data with combination of social sciences to predict pregnancy does not count as "unreasonably effective" I don't know what does.
there has been much joking in recent times about "unreasonable ineffectiveness of mathematics"
I do not know where you heard such jokes, but they must have come from some people who are not up to date on modern research. While economics or other social sciences are not yet as precise as some areas of physics they are not far behind. Such jokes would ring true before the 'credibility revolution' in economics (see Angrist and Pischke 2010). After the credibility revolution such jokes are certainly not accurate.
what could be the reasons why mathematics seems to work very well in certain areas (physics or engineering), but is not as useful or accurate in in social sciences such as Economics?
In past (pre 80s) social science models were very ineffective mainly for the following reasons:
- lack of data to verify/quantify relationships.
- there is more noise in social science data which requires large sample sizes for disentangling relationships from noise.
- lack of computing power.
- lack of statistical techniques crafted for social sciences and lack of experimentation.
lets take these points one by one:
- lack of data to verify/quantify relationships.
in the past it was very difficult to find high quality data. You will find a lot of social science papers from that era using sample sizes of 40-60, which by modern standards would be laughable. In fact even as recently as in 90s you could see published papers using sample sizes as small as 117 observations (e.g. see Dollar 1992).
For example, even in physics it would be difficult if not impossible to predict path of an asteroid if there are no good data on its velocity, position etc available.
- there is more noise in social science data which requires large sample sizes for disentangling relationships from noise.
In connection to the previous point there is usually much more noise in social science data, hence it goes without saying that social science requires much larger data sets to be precise. This is why the stories such as that about the prediction of pregnancy are popping up just now as they are result of large data being employed in social sciences.
lack of computing power.
Lack of computing power was large issue in the past, and it is still limiting nowadays. For example, analogue hydraulic computer (e.g. MONIAC) was used in economics even after regular computers were invented as early computers did not had computing power to model even simple macro models.
Modern computers are better but still computing power is big limiting factor. Recently I took graduate class from Ben Moll on distributional macro where we were building relatively simple macro models with multiple agents (e.g. see examples of the models here), yet even modern PCs have still some trouble running such models (they can take quite long to solve).
Nonetheless, outside area of simulations (that are usually very intensive computationally) the computing power now is sufficient to run wide array of statistical models which would be impossible to run in the past.
- lack of statistical techniques crafted for social sciences and lack of experimentation.
In the past big issue with social sciences was lack of appropriate statistical models. In physics most relationships are exogenous, you have typically very simple chains of causality. In social sciences most relationships are endogenous. Such relationships cannot be as easily analyzed with (comparatively) basic methods that are sufficient for many natural sciences.
As a result before the 'creditability revolution' most empirical research was in very bad state. However, the development of new statistical techniques such as diff-in-diff, synthetic control, TSLS, RD etc as well as greater focus on running randomized controlled trials brought credibility and much greater predictive power to social science research (see Angrist and Pischke 2010).
Even if social science research is still not as precise as some areas of physics, it would be absurd to say that mathematics is somehow 'unreasonable ineffective' at present day.
Is it reasonable to assume that economics or sociology will one day have as intensive and predictive a use of mathematics as is currently the case in physics, or can these sciences as we know them today simply not be formalized to that degree?
It is reasonable to assume that. In fact I would say that present day economics can be as accurate as many areas of physics were decades ago. With better data, more computing power, better statistical techniques social sciences can be frighteningly precise. For example, recently university of Chicago researchers developed algorithm that can predict crimes weeks in advance with about 90% accuracy (see Rotaru et al 2022 or here) which almost sounds like the "precogs from the minority report.