One important reason why it is better to have activity be done formally rather than informally (grey or black markets) is access to proper institutions.
Property rights, ability to enforce contractual obligations, above mentioned safety laws etc, are vital for economic activity (see Stiglitz Economics of Public Sector 3rd ed).
Some economic activity does not require institutions. You might hire someone for mowing your lawn, where you can easily observe if the job is done properly. At the same time the worker has little in stake since even if they can't force you to pay without contract they will not do business with you in future. But when it comes to more complex economic interactions which are often necessary as economic activity expands, such as establishing supply chains, having several workers at different locations etc it becomes more and more limiting if people do not have access to proper instiutions.
What even worse, in absence of government imposing institutions, gangs, mafia or local paramilitary might. For example, one reason why gangs exist in drug trade is that drugs are expensive and easily portable, so as all expensive things they attract thieves. If selling drugs was legal firms would be just able to either rely on police or in worst case scenario private security forces to defend their labs, storages or shipments. However, without access to this institutions they have rely on hiring thugs and gangsters. Moreover, in absence of access to court & legal system any disputes between drug producers about quality of drugs or payments between producers and dealers will be settled by gang conflict, intimidation and so on.
Second, you mention that formalizing sector brings in more government revenue, but then you just brush this point off as if it would be nothing even though this is crucial argument.
If millions of people work in informal economy (as you argue in your question), this seriously retards government ability to extract revenue. Even if you would happen to libertarian and reject any redistribution concerns government is still necessary to provide wide range of public goods (or in some cases quasi-public) goods such as national defense, infrastructure, law and order enforcement. If government can't efficiently raise enough revenue to properly fund these functions then whole economy will suffer. As a matter of fact provision of public goods and access to well designed institutions, economic development is thought to be virtually impossible (see Acemoglu & Robinson. Why Nations Fail).
Consequently, it is vital for the country to have most of its economic activity be done in a formal way. That does not mean all grey economic activity should be completely eliminated, it would probably not be worth while to require even occasional babysitting should be only done by formally organized business, and have some space for economic activity between relatives. However, grey economic activity should not be encouraged and if as you put it millions of people are employed in informal economy then that would be a problem.
Lastly, you seem to assume that formalizing economic activity has to be costly but that is not true. Formalizing economic activity does not mean that people there have to follow some draconian regulations. Maybe in your place government plans that, but many advanced countries will have lower regulation/administration/tax burden imposed on self-employed people or small businesses. Consequently, there is also no reason to think that formalizing informal sector should involve large costs a priori.
