This is a historical trend whether it may 18th century Britain or 19th century America or what's currently happening in China. What I am confused is by the fact that manufacturing jobs such as working on an assembly line are not that well paying. Still the trend holds true.
I want to know what are the causes for the shift.
This is a common area of study in Development Economics. There is for example the Dual-sector model, first developed in 1954. It is very well explained in the link provided, but basically:
[the] agricultural sector is typically characterized by low wages, an abundance of labour, and low productivity through a labour-intensive production process. In contrast, the capitalist manufacturing sector is defined by higher wage rates as compared to the subsistence sector, higher marginal productivity, and a demand for more workers. Also, the capitalist sector is assumed to use a production process that is capital intensive, so investment and capital formation in the manufacturing sector are possible over time as capitalists' profits are reinvested in the capital stock. [...]
The primary relationship between the two sectors is that when the capitalist sector expands, it extracts or draws labour from the subsistence sector. This causes the output per head of labourers who move from the subsistence sector to the capitalist sector to increase. [...]
The agricultural sector has a limited amount of land to cultivate, the marginal product of an additional farmer is assumed to be zero as the law of diminishing marginal returns has run its course due to the fixed input, land. As a result, the agricultural sector has a quantity of farm workers that are not contributing to agricultural output since their marginal productivities are zero. This group of farmers that is not producing any output is termed surplus labour since this cohort could be moved to another sector with no effect on agricultural output. [...]
The end result of this transition process is that the agricultural wage equals the manufacturing wage, the agricultural marginal product of labour equals the manufacturing marginal product of labour, and no further manufacturing sector enlargement takes place as workers no longer have a monetary incentive to transition.
In other words,, low productivity in agriculture due to unlimited land and workers and low use of capital means low agricultural wages, whereas high productivity in new capital intensive industries means high wages, thereby leading to a migration process that continues until wages equalise.
It might be worth noticing that this model was also used by Simon Kuznets to explain why industrialised countries saw a non-monotonic evolution of wage inequality between 1870 and 1950 (i.e. an increase and then a decrease in inequality), pattern that came to be known as the Kuznets Curve. As the article above states:
The Kuznets curve implies that as a nation undergoes industrialization – and especially the mechanization of agriculture – the center of the nation’s economy will shift to the cities. As internal migration by farmers looking for better-paying jobs in urban hubs causes a significant rural-urban inequality gap (the owners of firms would be profiting, while laborers from those industries would see their incomes rise at a much slower rate and agricultural workers would possibly see their incomes decrease), rural populations decrease as urban populations increase. Inequality is then expected to decrease when a certain level of average income is reached and the processes of industrialization – democratization and the rise of the welfare state – allow for the trickle-down of the benefits from rapid growth, and increase the per-capita income.
A schematic picture is as follows:
Industrialization also means industrialization of Agriculture. This kills land-related jobs in the sector while at the same time it increases output. Inceased output requires preservation since it can't be immediately consumed as fresh product. So jobs are created in the "packaging" business-end of Agriculture, which turns farm-workers into factory-workers already.
Moreover, the job-transformation is not one-to-one so a surplus of ex-farm workers is created, and they have to seek work in factories outside the Agriculture sector.
In more abstract terms, capital displaces labor from the traditional Agricultural activities, while also it creates demand for labor in the agriculture-related and other manufacturing activities.
Aside from the economic arguments, a big difference was the amount of autonomy for the workers.
Consider agriculture and similar rural employment (as servants etc) in the UK, even as recently as 1900. The status of agricultural workers in society was little changed from a medieval serf. They lived in housing provided by their employer. Their employer provided much of the resources (including time) for them to grow their own food and keep a few animals for their own use (e.g. a pig to fatten, kill, and preserve the meat each year, and maybe a couple of cows for milk as well). If that was not practical (e.g. for unmarried female domestic servants) food was provided "free" by the employer - with no personal choice available, of course. Workers were employed on annual contracts with no long term security, and paid in cash just once per year - with no cash payment at all if their employer was dissatisfied with them for any reason, or no good reason at all.
Compare all that with the relative freedom of paid factory work in a town, even allowing for the appalling living conditions of many factory workers and the extortionate systems of payment by company tokens that were only valid in company-run stores, rather than in standard currency. "The American Dream" wasn't invented in the USA, but in the European Industrial Revolution!
Food is the most necessary thing in life. It is not uncommon that undeveloped societies can have as large as 80% of the economy in agriculture. One of the first steps to an industrialized society therefore becomes to make agriculture more efficient. To produce more food per worker. Why? Because otherwise no matter how efficient technology you have outside of agriculture, you don't have enough human resources available to allocate to work with it ( either that, or or force-reallocate with the trade-off that the population will starve )! You can have as advanced science and high tech machinery but if people have to work to get food for the day they don't have time to learn it and to work with it, those machines & that knowledge will stand mostly unused.
So the effectivization of agriculture is a necessary prerequisite of industrialization.
